RAG Cost Structure: Embedding + Search + LLM

TL;DR — A RAG pipeline has three cost components: embedding (one-time, ~$1.30 per million chunks with text-embedding-v4), search ($0.001–0.02 per query), and LLM generation ($0.01–0.15 per answer). For most knowledge bases, RAG costs $0.02–0.05 per question. Long-context models win below ~50 pages but lose badly above ~200 pages. This guide provides exact calculations for planning.

The three cost layers

Every RAG (Retrieval-Augmented Generation) pipeline has the same fundamental cost structure:

Total cost per answer = Embedding (amortized) + Search + LLM generation

Each layer has different characteristics:

Embedding — mostly one-time, amortized across all queries
Search — per-query, usually the cheapest component
LLM generation — per-query, usually the most expensive component

Let’s break each down with real 2026 pricing.

Layer 1: Embedding cost

Embedding converts your documents into vectors for similarity search. This is primarily a one-time cost when you ingest documents, plus incremental cost for updates.

Pricing landscape (2026)

Model	Provider	Price per 1M tokens	Dimensions	Notes
text-embedding-v4	Alibaba/SandBase	$0.0015/1K tokens	1024/2048	Latest, high quality
text-embedding-3-large	OpenAI	$0.00013/1K tokens	3072	Most cost-effective
text-embedding-3-small	OpenAI	$0.00002/1K tokens	1536	Budget option
embed-v4	Cohere	$0.0001/1K tokens	1024	Good multilingual
Voyage-3-large	Voyage AI	$0.00018/1K tokens	1024	Specialized for code

Calculating embedding cost for 1M documents

Assumptions:

Average document: 500 words ≈ 650 tokens
Chunking strategy: 512 tokens per chunk with 64 token overlap
Average chunks per document: 1.5 (accounting for short docs)
Total chunks: 1,500,000

With text-embedding-v4 ($0.0015/1K tokens):
  Total tokens: 1,500,000 chunks × 512 tokens = 768,000,000 tokens
  Cost: 768,000 × $0.0015 = $1,152

With text-embedding-3-large ($0.00013/1K tokens):
  Cost: 768,000 × $0.00013 = $99.84

With text-embedding-3-small ($0.00002/1K tokens):
  Cost: 768,000 × $0.00002 = $15.36

Amortization

Embedding is a one-time cost (per document version). If your knowledge base is queried 100,000 times before the next full re-embed:

Amortized embedding cost per query:
  text-embedding-v4: $1,152 / 100,000 = $0.01152
  text-embedding-3-large: $99.84 / 100,000 = $0.001
  text-embedding-3-small: $15.36 / 100,000 = $0.00015

For most applications, amortized embedding cost is negligible — a fraction of a cent per query.

Re-embedding triggers

You’ll need to re-embed when:

Documents are updated (re-embed changed docs only)
You switch embedding models (full re-embed)
You change chunk size strategy (full re-embed)
New documents are added (embed new only)

Budget for monthly incremental embedding based on your update frequency:

Update pattern	Monthly embedding cost (1M doc base)
Static (no updates)	$0
1% updated/month	~$12 (text-embedding-v4)
10% updated/month	~$115
Full monthly refresh	~$1,152

Layer 2: Search cost

Search finds relevant chunks for a query. This has two sub-components: query embedding and vector similarity search.

Query embedding

Every query must be embedded using the same model as your documents:

Query embedding cost per search:
  Average query: 20 tokens
  text-embedding-v4: 20/1000 × $0.0015 = $0.00003
  text-embedding-3-large: 20/1000 × $0.00013 = $0.0000026

Effectively free at per-query level.

Vector database costs

The significant search cost is the vector database hosting:

Solution	Monthly cost (1.5M vectors, 1024 dims)	Per-query cost at 10K queries/day
Pinecone (Starter)	$70/month	$0.00023
Pinecone (Standard)	$200/month	$0.00067
Weaviate Cloud	$150/month	$0.0005
Qdrant Cloud	$100/month	$0.00033
pgvector (self-hosted)	$50–100/month (VM cost)	$0.00017–0.00033
ChromaDB (self-hosted)	$30–80/month (VM cost)	$0.0001–0.00027

Search API services

Alternatively, use a search API that handles embedding + vector search:

SandBase search APIs (e.g., Cloudsway):
  $0.001–0.02 per search query
  Includes: query embedding + retrieval + re-ranking
  No infrastructure to manage

Total search cost per query

Self-managed (Pinecone Standard + text-embedding-3-large):
  Query embedding: $0.0000026
  Vector search: $0.00067
  Total: ~$0.0007/query

Managed search API:
  $0.001–0.02/query (all-inclusive)

Layer 3: LLM generation cost

The most expensive and most variable component. This is where the retrieved context gets sent to an LLM along with the user’s question.

Token math

A typical RAG prompt includes:

System prompt: ~200 tokens
Retrieved context: 5 chunks × 512 tokens = 2,560 tokens
User question: ~50 tokens
Total input: ~2,810 tokens
Generated answer: ~300–500 tokens

LLM pricing comparison (2026)

Model	Input $/1M tokens	Output $/1M tokens	Cost per RAG answer
GPT-4.1	$2.00	$8.00	$0.0096
GPT-4.1-mini	$0.40	$1.60	$0.0019
Claude Sonnet 5	$3.00	$15.00	$0.0159
Claude Haiku 4	$0.80	$4.00	$0.0042
Kimi K3	$1.50	$6.00	$0.0072
DeepSeek V4	$0.50	$2.00	$0.0024
Qwen 3.6	$0.30	$1.20	$0.0015

Cost per RAG answer assumes 2,810 input tokens + 400 output tokens

Context window optimization

More retrieved chunks = better answers but higher cost:

Chunks retrieved	Input tokens	GPT-4.1 cost	Claude Sonnet 5 cost
3 chunks	1,786 tokens	$0.0068	$0.0113
5 chunks	2,810 tokens	$0.0096	$0.0159
10 chunks	5,370 tokens	$0.0167	$0.0281
20 chunks	10,490 tokens	$0.0310	$0.0525

The sweet spot for most applications is 5–10 chunks. Beyond 10, you get diminishing returns on answer quality while costs scale linearly.

Total RAG cost per answer

Combining all three layers for a typical setup:

Budget configuration

Embedding: text-embedding-3-large (amortized ~$0.001/query)
Search: Self-managed Pinecone ($0.0007/query)
LLM: GPT-4.1-mini with 5 chunks ($0.0019/query)

Total: $0.0036/query (~$0.004)

Mid-range configuration

Embedding: text-embedding-v4 (amortized ~$0.012/query)
Search: Managed API ($0.005/query)
LLM: GPT-4.1 with 5 chunks ($0.0096/query)

Total: $0.0266/query (~$0.03)

Premium configuration

Embedding: text-embedding-v4 (amortized ~$0.012/query)
Search: Managed API with re-ranking ($0.02/query)
LLM: Claude Sonnet 5 with 10 chunks ($0.0281/query)

Total: $0.0601/query (~$0.06)

Monthly cost projections

Daily queries	Budget ($0.004)	Mid-range ($0.03)	Premium ($0.06)
100	$12/month	$90/month	$180/month
1,000	$120/month	$900/month	$1,800/month
10,000	$1,200/month	$9,000/month	$18,000/month
100,000	$12,000/month	$90,000/month	$180,000/month

Optimization strategies

1. Chunk size tuning

Larger chunks mean fewer chunks needed but more tokens per chunk:

Chunk size	Chunks for same coverage	Tokens per RAG call	Trade-off
256 tokens	10 chunks	2,810	More precision, more search cost
512 tokens	5 chunks	2,810	Balanced
1024 tokens	3 chunks	3,322	Less precision, fewer searches

Recommendation: Start with 512 tokens, 64 token overlap. Adjust based on answer quality evaluation.

2. Two-stage retrieval

# Stage 1: Fast, cheap retrieval (get 20 candidates)
candidates = vector_db.search(query_embedding, top_k=20)

# Stage 2: Re-rank with a cross-encoder (keep top 5)
reranked = reranker.rank(query, candidates, top_k=5)

# Only send top 5 to LLM
answer = llm.generate(context=reranked, question=query)

Re-ranking costs ~$0.001–0.005 per query but significantly improves which chunks reach the LLM, reducing the need for large context windows.

3. Caching

import hashlib

def get_cached_answer(query: str, cache: dict) -> str | None:
    # Exact match cache
    key = hashlib.sha256(query.encode()).hexdigest()
    return cache.get(key)

def get_semantic_cache(query_embedding, cache_embeddings, threshold=0.95):
    # Semantic similarity cache
    similarities = cosine_similarity(query_embedding, cache_embeddings)
    if max(similarities) > threshold:
        return cached_answers[argmax(similarities)]
    return None

Cache hit rates depend on query repetition:

Customer support: 40–60% cache hit rate (many repeated questions)
Research/analysis: 5–15% cache hit rate (unique questions)
Internal knowledge base: 20–40% cache hit rate

A 40% cache hit rate reduces effective per-query cost by 40%.

4. Model routing

Use a cheaper model for simple questions, premium model for complex ones:

def route_query(query: str, context_chunks: list) -> str:
    # Simple heuristic: short answers likely simple
    if len(context_chunks) <= 3 and len(query.split()) < 15:
        return "gpt-4.1-mini"  # $0.0019/answer
    else:
        return "claude-sonnet-5"  # $0.0159/answer

With 60% simple / 40% complex split:

Blended cost: 0.6 × $0.0019 + 0.4 × $0.0159 = $0.0075/answer
vs. always using premium: $0.0159/answer
Savings: 53%

5. Embedding model selection by use case

For deeper analysis of embedding model options including text-embedding-v4, consider:

English-only knowledge base: text-embedding-3-large (cheapest, excellent quality)
Multilingual (including Chinese): text-embedding-v4 (best CJK performance)
Code repositories: Voyage-3-large (specialized)
Budget-constrained: text-embedding-3-small (20x cheaper, 90% quality)

See also our comprehensive embedding model comparison for benchmark data.

RAG vs Long-Context: When to use which

The advent of 1M-token context windows raises the question: when is RAG still worth the complexity?

Cost comparison by document size

Assume: Answering questions about a document collection.

Collection size	RAG cost/query	Long-context cost/query (GPT-4.1)	Winner
10 pages (~5K tokens)	$0.03	$0.012	Long-context
50 pages (~25K tokens)	$0.03	$0.054	RAG
200 pages (~100K tokens)	$0.03	$0.204	RAG (7x cheaper)
1000 pages (~500K tokens)	$0.03	$1.004	RAG (33x cheaper)
10,000 pages (~5M tokens)	$0.03	Impossible (exceeds context)	RAG (only option)

The crossover point

RAG becomes cheaper than long-context when your source material exceeds approximately 15,000–25,000 tokens (~8-12 pages), depending on model pricing.

When long-context wins

Small document sets (< 50 pages): Cheaper and simpler to stuff everything in context
Questions requiring holistic understanding: “Summarize the main themes across all documents”
Sequential reasoning: When the answer requires following a thread across the full document
One-off analysis: No need to build embedding infrastructure for a single use

When RAG wins

Large knowledge bases (> 100 pages): Cost scales with retrieved chunks, not total collection
Frequently queried: Amortized embedding cost approaches zero
Precise retrieval needed: Questions about specific facts in large collections
Multi-source: Combining data from different document types/sources
Dynamic content: New documents added regularly without re-processing everything

Implementation: Minimal RAG pipeline

from openai import OpenAI
import numpy as np

client = OpenAI(base_url="https://api.sandbase.ai/v1", api_key="...")

# Step 1: Embed documents (one-time)
def embed_chunks(chunks: list[str]) -> list[list[float]]:
    response = client.embeddings.create(
        model="text-embedding-v4",
        input=chunks
    )
    return [item.embedding for item in response.data]

# Step 2: Search (per query)
def search(query: str, chunk_embeddings: np.ndarray, chunks: list[str], top_k=5):
    query_resp = client.embeddings.create(model="text-embedding-v4", input=[query])
    query_vec = np.array(query_resp.data[0].embedding)
    
    similarities = np.dot(chunk_embeddings, query_vec)
    top_indices = np.argsort(similarities)[-top_k:][::-1]
    return [chunks[i] for i in top_indices]

# Step 3: Generate answer (per query)
def answer(query: str, context_chunks: list[str]) -> str:
    context = "\n\n---\n\n".join(context_chunks)
    response = client.chat.completions.create(
        model="gpt-4.1-mini",
        messages=[
            {"role": "system", "content": f"Answer based on this context:\n\n{context}"},
            {"role": "user", "content": query}
        ]
    )
    return response.choices[0].message.content

# Full pipeline
chunks = load_and_chunk_documents("./docs/")
embeddings = np.array(embed_chunks(chunks))

# Per-query cost: ~$0.004
query = "What is the refund policy?"
relevant = search(query, embeddings, chunks)
result = answer(query, relevant)

Cost monitoring for RAG agents

For agents using RAG pipelines, track these metrics:

class RAGCostTracker:
    def __init__(self):
        self.embedding_tokens = 0
        self.search_queries = 0
        self.llm_input_tokens = 0
        self.llm_output_tokens = 0
    
    def log_query(self, input_tokens: int, output_tokens: int):
        self.search_queries += 1
        self.llm_input_tokens += input_tokens
        self.llm_output_tokens += output_tokens
    
    @property
    def total_cost(self):
        embedding_cost = self.embedding_tokens / 1000 * 0.0015
        search_cost = self.search_queries * 0.001
        llm_input_cost = self.llm_input_tokens / 1_000_000 * 2.00
        llm_output_cost = self.llm_output_tokens / 1_000_000 * 8.00
        return embedding_cost + search_cost + llm_input_cost + llm_output_cost

For a deeper dive on API pricing models and budget controls, see LLM API pricing guide.

Conclusion

RAG cost structure is predictable and optimizable:

Embedding: One-time cost, amortizes to near-zero per query. Choose model based on language needs and quality requirements.
Search: Cheapest component. Self-hosted vector DB or managed API, either way it’s < $0.01/query.
LLM generation: Dominant cost. Optimize by routing models, tuning chunk count, and caching.

The total: $0.004–0.06 per answer depending on configuration. At 1,000 queries/day, that’s $120–1,800/month — a fraction of what a human analyst costs for the same knowledge retrieval task.

RAG remains the economically correct choice for any knowledge base exceeding ~25K tokens where questions are frequent and answers require precision. Below that threshold, long-context is simpler and often cheaper. Above it, RAG’s fixed-cost-per-query regardless of collection size makes it the clear winner.