TestForge Blog
← All Posts
#ai #rag #prompt #llm #service

RAG Development Part 4 — Answer Generation, Prompt Design, and Citations

Retrieval is only half of RAG. This post explains how to structure prompts, select and compress context, design citations, and make the system answer safely when evidence is weak.

TestForge Team ·

Retrieval Alone Does Not Finish the Job

Even if search finds the right documents, the final answer can still be misleading.

Typical issues:

  • The model overstates confidence
  • Citations are missing or inconsistent
  • It answers from general knowledge instead of retrieved evidence
  • It gives a confident answer when evidence is weak

So retrieval and generation should be designed as separate layers.

A Practical Generation Flow

User Query
 -> Retrieved Context
 -> Context Selection / Compression
 -> Prompt Assembly
 -> LLM Generation
 -> Citation Formatting
 -> Final Answer

The important question is not just what documents to send, but what rules the model must follow while using them.

What Good Prompts Usually Contain

1. Role

You are a technical support assistant that answers using product documentation.

2. Evidence Rules

Answer only from the provided documents.
If evidence is insufficient, say so explicitly.

3. Output Format

Structure the answer as summary -> detailed explanation -> references.

4. Uncertainty Handling

If retrieved evidence is weak or conflicting, avoid firm conclusions.

Without these rules, models tend to fill gaps with plausible-sounding guesses.

How to Feed Context

Passing all retrieved chunks directly is rarely ideal.

Typical improvements:

  • choose the strongest 3 to 5 chunks
  • merge adjacent chunks from the same source
  • preserve title and section labels
  • compress context when needed

Example:

[Document 1]
Title: Authentication API Guide
Section: Token Refresh
Body: ...

[Document 2]
Title: Authentication Error Codes
Section: 401 Unauthorized
Body: ...

This structure makes the evidence much easier for the model to use correctly.

Citations Should Be Structured From the Start

Do not treat citations as an afterthought.

A good pattern is:

  • store source_url, title, section, and chunk_id
  • use numbered citations in the prompt
  • render them as links after generation

Example:

[1] Authentication API Guide - Token Refresh
[2] Authentication Error Codes - 401 Unauthorized

Users trust RAG systems far more when they can inspect the source directly.

How to Make the Model Say “I Don’t Know”

Prompt instructions help, but application logic should help too.

Useful signals:

  • low top retrieval score
  • too few good hits
  • conflicting documents
  • weak confidence heuristic before generation

Example:

if not hits or hits[0].score < 0.45:
    return {
        "answer": "The available documents do not provide enough evidence for a reliable answer.",
        "citations": []
    }

Output Format Should Match the Use Case

Different RAG products need different output shapes.

Customer Support

  • short answer
  • action steps
  • related docs

Operational Guidance

  • symptoms
  • cause
  • diagnosis
  • recovery

Product Docs Assistant

  • concept explanation
  • examples
  • constraints
  • reference links

Templates create consistency and reduce drift.

Practical Ways to Reduce Hallucinations

  • Ask the model to summarize evidence before final answering
  • Limit it to question-relevant content only
  • Block unsupported feature claims

Example:

Explain only what is directly supported by the retrieved documents.
Do not mention product behavior not found in the provided evidence.

Multi-turn RAG Needs Extra Care

Conversation history and retrieved evidence can conflict.

Example:

  • Earlier turns discussed the dev environment
  • Retrieved docs are about production operations

Useful separation:

  • conversation summary memory
  • current retrieved documents
  • the current user question

Output Post-processing Matters

Before returning the answer, validate:

  • citations exist
  • links are valid
  • no sensitive data leaked
  • the answer length is reasonable
  • banned or unsafe patterns are filtered

Generation systems need quality control at the output layer too.

Closing Thoughts

RAG answer generation is not only prompt writing.

It is about:

  • selecting context carefully
  • constraining the model clearly
  • formatting citations consistently
  • failing safely when evidence is weak

That is what makes a RAG response useful and trustworthy.