LLM application development: architecture, API costs and evaluation, a practical guide

Retrieval-augmented generation (RAG)

A technique that pairs a language model with search over a document collection, so each answer is grounded in passages retrieved at query time. The term comes from a 2020 research paper describing models that combine the model's built-in knowledge with an external, searchable memory.

What does LLM application development involve?

It involves everything around the model rather than the model itself: the instructions, the data it can see, the tools it can call, the interface, and the checks that tell you whether it works. Very few startups train their own models; almost all build on APIs from providers such as OpenAI, Anthropic, Google or Mistral, so the engineering effort goes into context, cost and quality.

Prompting, RAG or fine-tuning: which approach do you need?

Start with prompting, add RAG when the model needs knowledge it does not have, and consider fine-tuning only to change behaviour that prompts cannot fix. Our rule of thumb, summarised:

There is a fourth option for small knowledge bases: put everything in the prompt. In its post on Contextual Retrieval, Anthropic suggests this for knowledge bases under 200,000 tokens, roughly 500 pages, using prompt caching to keep costs down.

How does a RAG pipeline work?

A RAG pipeline has a preparation stage, run once and refreshed when documents change, and a query stage, run on every question.

1. Chunk documents into passages of a few hundred words, keeping headings and context.
2. Embed each chunk, turning it into a vector of numbers that captures its meaning, and store it in an index.
3. Retrieve at query time by embedding the question and finding the closest chunks, often combined with keyword search.
4. Generate by sending the question and the retrieved chunks to the model, instructed to answer only from them and cite its sources.
5. Log the answer, the sources used and the cost of the request.

How much do LLM APIs cost?

LLM APIs bill per token, with one rate for input and a higher one for output. Below is a selection from the official pricing pages of OpenAI, Anthropic and Google. Prices change often, so use them to reason about orders of magnitude and check again before you commit.

How do you estimate the cost of an LLM feature?

Add two parts: indexing, paid once and again on each update, and cost per query, paid every time. Here is an example we built on hypothetical volumes using the prices above.

The real driver is the trade-off between input tokens and quality: more chunks cost more and do not always produce a better answer, which makes retrieval quality a cost lever as well. If the feature is an agent making several model calls per request, multiply accordingly; our breakdown of AI agent development cost covers that case.

How do you cut LLM API costs?

Mostly by matching the model to each step and never paying full price twice for the same text. The main levers:

• The smallest model that passes your evals. Within a single provider the price gap per token can reach a hundred times.
• Prompt caching. Repeated content such as instructions and examples costs far less when read from cache: Anthropic bills cache reads at 10% of the input price by default, 5% on Opus 5.5 and Sonnet 5.5.
• Batch for anything that can wait. Overnight summaries, classification, re-indexing: OpenAI's Batch API runs at half price with results within a day, and Anthropic and Google offer the same discount.
• Fewer, better chunks. Precise retrieval lets you send less context per question.
• Output limits. Shorter answers cost less and are often easier to read.

How do you evaluate an LLM application?

Evaluate the two halves separately: retrieval (did it find the right passages?) and generation (did it use them faithfully?). If retrieval fails, no model can rescue the answer; if retrieval works and the answer does not, the problem is in the instructions or the model. Anthropic's own tests show how much retrieval matters: combining contextual embeddings, keyword search and reranking cut top-20 retrieval failures by 67%, from 5.7% to 1.9%, without changing the model. Those are vendor-reported results on the vendor's own datasets.

OpenAI's evaluation guide notes that human judgement gives the highest-quality evals but is slow and expensive, while model-graded evals are cheaper and scale better, and that no strategy is perfect. Anthropic's documentation adds a practical bias: favour many automatically graded cases over a few graded by hand. A workable set-up looks like this:

• a list of real user questions, each mapped to the documents that hold the answer;
• for retrieval, the share of questions where the right passage appears among those retrieved;
• for answers, correctness, faithfulness to sources and citations, graded by a second model and spot-checked by a person;
• the whole suite rerun on every change to chunking, index, prompts or model.

Which mistakes push up cost and error rates?

• Chunks that are too big or stripped of context. The model gets noise, or sentences cut off from their heading.
• No eval set before launch. Without real questions you cannot tell whether a change helped.
• Stale documents in the index. The answer matches the source, but the source is out of date.
• Free tiers with customer data. Some let the provider use your content; see our guide to GDPR and LLM apps.
• No cost dashboard. Track cost per query weekly, the way you track latency.

If the model also needs to act, opening tickets or updating a CRM, you are building an agent: start with what AI agents are and how to build an AI agent for business. Still choosing where AI fits? How startups can use AI helps you pick the first use case.