{ "metadata": { "id": "ch12", "title": "第12章：RAG增强Agent", "volume": "vol4", "volume_title": "高级篇", "word_count": 4980, "difficulty": "intermediate", "prerequisites": [ "ch04", "ch07" ], "key_concepts": [ "概述", "LLM 的知识困境", "什么是 RAG", "为什么 Agent 需要 RAG", "RAG 的发展历程", "RAG 架构深入", "完整架构总览", "Document Loader（文档加载器）", "Text Splitter（文本分割器）", "Embedding（嵌入模型）", "Vector Store（向量存储）", "文档处理管线", "完整的索引构建流水线", "分块策略的深入探讨", "检索策略" ], "learning_objectives": [], "estimated_tokens": 2988, "source_file": "vol4/ch12_RAG增强Agent.md" }, "overview": "", "sections": [ { "id": "12.1", "title": "12.1 概述", "level": 2, "content": "", "subsections": [ { "id": "12.1.1", "title": "12.1.1 LLM 的知识困境", "content": "大语言模型（LLM）有一个根本性的局限：**它的知识在训练完成的那一刻就被冻结了**。无论 GPT-4o、Claude 3.5 还是 Gemini，它们都无法回答训练截止日期之后发生的事件，也无法访问你公司的内部文档、私有数据库或实时业务数据。\n\n更关键的问题在于**幻觉（Hallucination）**。当 LLM 遇到知识边界时，它不会坦诚地说\"我不知道\"，而是会\"自信地编造\"。在企业场景中，一个给出错误技术方案或财务数据的 Agent，其危害远比直接回答\"不知道\"要大得多。" }, { "id": "12.1.2", "title": "12.1.2 什么是 RAG", "content": "RAG（Retrieval-Augmented Generation，检索增强生成）的核心思想极其优雅：\n\n1. **检索（Retrieval）**：当用户提问时，先从一个大型文档库中检索出与问题最相关的文档片段\n2. **增强（Augmentation）**：将这些文档片段作为上下文，注入到 LLM 的提示词中\n3. **生成（Generation）**：LLM 基于这些真实的上下文信息来生成回答" }, { "id": "12.1.3", "title": "12.1.3 为什么 Agent 需要 RAG", "content": "单纯的 RAG 系统回答问题是被动的——用户问什么就答什么。但 **RAG 增强 Agent** 则赋予了系统主动性和多步推理能力：\n\n| 特性 | 传统 RAG | RAG 增强 Agent |\n|------|---------|---------------|\n| 交互模式 | 单轮问答 | 多轮对话 |\n| 推理能力 | 直接检索+回答 | 规划→检索→分析→综合 |\n| 工具使用 | 仅检索 | 检索 + 计算 + API调用 |\n| 自我反思 | 无 | 可评估回答质量并重试 |\n| 多源融合 | 单一知识库 | 跨知识库、数据库、Web |\n| 主动学习 | 无 | 可主动更新知识库 |\n\n一个 RAG 增强 Agent 不仅能回答\"你的退货政策是什么？\"，还能回答\"帮我对比三家供应商的价格，并给出采购建议\"。后者需要多次检索、数据计算和综合推理。" }, { "id": "12.1.4", "title": "12.1.4 RAG 的发展历程", "content": "RAG 技术自 2020 年由 Meta（Facebook AI Research）提出以来，已经经历了多代演进：\n\n\n---" } ] }, { "id": "12.2", "title": "12.2 RAG 架构深入", "level": 2, "content": "", "subsections": [ { "id": "12.2.1", "title": "12.2.1 完整架构总览", "content": "一个生产级 RAG 系统包含多个协作组件，每个组件都有多种实现选择：" }, { "id": "12.2.2", "title": "12.2.2 Document Loader（文档加载器）", "content": "文档加载器负责将各种格式的文档统一转换为纯文本或结构化文本：" }, { "id": "12.2.3", "title": "12.2.3 Text Splitter（文本分割器）", "content": "文本分割是 RAG 中最关键也最容易被忽视的环节。分块策略直接影响检索质量。\n\n\n**分块策略对比：**\n\n| 策略 | 优点 | 缺点 | 适用场景 |\n|------|------|------|---------|\n| 固定大小 | 简单、可控 | 切断语义、效率低 | 日志文件、结构化数据 |\n| 递归分割 | 保持语义边界、通用性强 | 仍可能切断长段落 | 通用文档 |\n| 语义分割 | 最佳语义完整性 | 计算成本高 | 学术论文、技术文档 |\n| 按标题分割 | 结构清晰 | 依赖文档格式 | Markdown、HTML |" }, { "id": "12.2.4", "title": "12.2.4 Embedding（嵌入模型）", "content": "嵌入模型将文本转换为高维向量，是 RAG 系统的核心组件：\n\n\n**主流嵌入模型对比：**\n\n| 模型 | 维度 | 中文支持 | 特点 |\n|------|------|---------|------|\n| OpenAI text-embedding-3-small | 1536 | ✅ | 性价比高，API调用 |\n| OpenAI text-embedding-3-large | 3072 | ✅ | 最高质量，成本较高 |\n| BGE-large-zh-v1.5 | 1024 | ✅✅ | 中文最优开源 |\n| Cohere embed-v3 | 1024 | ✅ | 多语言，内建重排 |\n| GTE-Qwen2 | 1536 | ✅✅ | 阿里开源，长文本支持好 |" }, { "id": "12.2.5", "title": "12.2.5 Vector Store（向量存储）", "content": "向量数据库存储文档的向量表示，支持高效的相似度检索：\n\n\n---" } ] }, { "id": "12.3", "title": "12.3 文档处理管线", "level": 2, "content": "", "subsections": [ { "id": "12.3.1", "title": "12.3.1 完整的索引构建流水线", "content": "" }, { "id": "12.3.2", "title": "12.3.2 分块策略的深入探讨", "content": "分块策略的选择对最终检索效果有决定性影响。这里总结一些实践中的关键经验：\n\n**分块大小的黄金法则**：\n\n\n**重要实践技巧**：\n\n1. **父子块策略（Parent-Child Chunking）**：用小块做检索（高精度），返回对应的大块作为上下文（高完整性）\n2. **上下文增强**：每个块附加上下文（标题、章节号、前一段摘要），帮助理解\n3. **元数据附加**：添加文档层级元数据（作者、日期、部门），支持过滤检索\n\n\n---" } ] }, { "id": "12.4", "title": "12.4 检索策略", "level": 2, "content": "", "subsections": [ { "id": "12.4.1", "title": "12.4.1 稠密检索（Dense Retrieval）", "content": "稠密检索是 RAG 的默认方案——通过 embedding 的语义相似度来查找相关文档：\n\n\n**稠密检索的优势与局限：**\n\n- ✅ 能理解语义（\"手机\"和\"智能手机\"能匹配）\n- ✅ 跨语言检索（如果嵌入模型支持）\n- ❌ 对专有名词、产品编号等精确匹配不够好\n- ❌ 长尾查询的效果可能不理想" }, { "id": "12.4.2", "title": "12.4.2 稀疏检索（BM25）", "content": "BM25 是经典的信息检索算法，基于词频统计：" }, { "id": "12.4.3", "title": "12.4.3 混合检索（Hybrid Search）", "content": "混合检索结合稠密检索和稀疏检索的优势：\n\n\n**混合检索的参数调优建议：**\n\n| 场景 | alpha（稠密权重） | 说明 |\n|------|------------------|------|\n| 通用问答 | 0.7 | 语义理解更重要 |\n| 技术文档搜索 | 0.5 | 关键词匹配很重要 |\n| 产品/错误码查询 | 0.3 | 精确匹配是关键 |\n| 法律/合规文档 | 0.6 | 语义和精确都需要 |" }, { "id": "12.4.4", "title": "12.4.4 重排序（Reranking）", "content": "重排序是 RAG 系统的\"杀手锏\"——先用快速检索获取候选集，再用精确的交叉编码器精排：\n\n\n---" } ] }, { "id": "12.5", "title": "12.5 知识库维护", "level": 2, "content": "", "subsections": [ { "id": "12.5.1", "title": "12.5.1 增量更新", "content": "生产环境的知识库不是一成不变的——文档会被新增、修改和删除。增量更新是保持知识库时效性的关键：" }, { "id": "12.5.2", "title": "12.5.2 过期与清理策略", "content": "---" } ] }, { "id": "12.6", "title": "12.6 GraphRAG", "level": 2, "content": "", "subsections": [ { "id": "12.6.1", "title": "12.6.1 从向量到图：知识的新维度", "content": "传统的 RAG 基于向量相似度检索，能回答\"与 X 相关的内容是什么\"，但难以回答需要多跳推理的问题，比如\"A 公司的 CEO 的母校位于哪个城市？\"。这类问题需要沿着实体关系链进行推理——这正是知识图谱的强项。\n\n**GraphRAG = 向量检索 + 图遍历**，将两种知识表示方式的优势互补：" }, { "id": "12.6.2", "title": "12.6.2 知识图谱构建", "content": "" }, { "id": "12.6.3", "title": "12.6.3 GraphRAG 检索", "content": "---" } ] }, { "id": "12.7", "title": "12.7 代码示例：完整的 RAG Agent 实现", "level": 2, "content": "", "subsections": [ { "id": "12.7.1", "title": "12.7.1 基于 LangChain 和 ChromaDB 的完整实现", "content": "---" } ] }, { "id": "12.8", "title": "12.8 最佳实践与常见陷阱", "level": 2, "content": "", "subsections": [ { "id": "12.8.1", "title": "12.8.1 最佳实践清单", "content": "" }, { "id": "12.8.2", "title": "12.8.2 常见陷阱与解决方案", "content": "| 陷阱 | 症状 | 解决方案 |\n|------|------|---------|\n| **分块太大** | 检索结果不精准，包含大量无关信息 | 减小 chunk_size，使用语义分割 |\n| **分块太小** | 缺乏上下文，回答不完整 | 增大 chunk_size，使用父子块策略 |\n| **只用稠密检索** | 产品编号、专有名词搜不到 | 加入 BM25 混合检索 |\n| **不设 overlap** | 关键信息正好被切断 | 设置 10-20% 的 overlap |\n| **忽略元数据** | 无法按部门/日期/类型过滤 | 始终保存和利用 metadata |\n| **幻觉问题** | Agent 编造不存在的答案 | 强化 Prompt 约束 + 引用溯源 |\n| **全量重建索引** | 文档更新代价高昂 | 实现增量更新机制 |\n| **不考虑查询质量** | 用户查询模糊导致检索差 | 加入查询重写/查询扩展 |" }, { "id": "12.8.3", "title": "12.8.3 查询重写与扩展", "content": "用户原始查询往往不够精准，查询重写能显著提升检索效果：\n\n\n---" } ] }, { "id": "12.9", "title": "12.9 小结与延伸阅读", "level": 2, "content": "", "subsections": [ { "id": "12.9.1", "title": "12.9.1 核心要点回顾", "content": "本章我们深入探讨了 RAG 增强 Agent 的完整技术栈：\n\n1. **RAG 的核心价值**：通过检索外部知识来弥补 LLM 的知识过期、私有数据缺失和幻觉问题\n2. **六步管线**：Document Loading → Text Splitting → Embedding → Vector Store → Retrieval → Generation\n3. **分块是关键**：分块策略直接影响检索质量，递归分割+父子块是推荐方案\n4. **混合检索优于单一检索**：稠密检索（语义）+ BM25（关键词）的组合效果最好\n5. **重排序是杀手锏**：交叉编码器能显著提升 Top-K 精度\n6. **知识库需要维护**：增量更新、版本管理、过期清理是生产环境的必备能力\n7. **GraphRAG 拓展了 RAG 的能力边界**：知识图谱赋予了多跳推理和关系查询能力" }, { "id": "12.9.2", "title": "12.9.2 RAG 评估框架", "content": "评估 RAG 系统的质量需要专门的指标：\n\n| 指标 | 类型 | 含义 |\n|------|------|------|\n| **Faithfulness** | 忠实度 | 回答是否基于检索到的上下文 |\n| **Answer Relevancy** | 答案相关性 | 回答与问题的相关程度 |\n| **Context Precision** | 上下文精确率 | 检索到的文档中相关文档的占比 |\n| **Context Recall** | 上下文召回率 | 相关文档被检索到的比例 |\n| **Hit Rate** | 命中率 | 正确答案出现在 Top-K 中的比例 |\n| **MRR** | 平均倒数排名 | 正确答案排名的倒数均值 |\n\n推荐使用 [RAGAS](https://github.com/explodinggradients/ragas) 框架进行自动化评估。" }, { "id": "12.9.3", "title": "12.9.3 延伸阅读", "content": "1. **论文**\n - Lewis et al. (2020). \"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks\" — RAG 开山之作\n - Gao et al. (2024). \"Retrieval-Augmented Generation for Large Language Models: A Survey\" — RAG 全面综述\n - Es et al. (2024). \"Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection\" — 自我反思式 RAG\n - Edge et al. (2024). \"From Local to Global: A Graph RAG Approach to Query-Focused Summarization\" — Microsoft 的 GraphRAG\n\n2. **工具与框架**\n - [LangChain](https://python.langchain.com/) — 最流行的 LLM 应用框架\n - [LlamaIndex](https://www.llamaindex.ai/) — 专注 RAG 的数据框架\n - [ChromaDB](https://www.trychroma.com/) — 轻量级向量数据库\n - [RAGAS](https://github.com/explodinggradients/ragas) — RAG 评估框架\n - [Haystack](https://haystack.deepset.ai/) — 端到端 NLP 框架\n\n3. **进阶主题**\n - **Agentic RAG**：让 Agent 自主决定何时检索、检索什么、是否需要补充检索\n - **CRAG (Corrective RAG)**：检索后自我评估，不满足则触发纠正策略\n - **自适应 RAG**：根据查询复杂度动态选择检索策略\n - **多模态 RAG**：支持图片、表格、公式的检索与理解\n - **实时 RAG**：结合 Web 搜索的实时知识检索" } ] } ], "code_blocks": [ { "id": "code-1", "language": "text", "description": "更关键的问题在于幻觉（Hallucination）。当 LLM 遇到知识边界时，它不会坦诚地说\"我不知道\"，而是会\"自信地编造\"。在企业场景中，一个给出错误技术方案或财务数据的 Agent，其危害远比", "code": "┌──────────────────────────────────────────────────────────────┐\n│ LLM 知识的三重困境 │\n│ │\n│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │\n│ │ 知识过期 │ │ 私有数据缺失 │ │ 幻觉问题 │ │\n│ │ │ │ │ │ │ │\n│ │ 训练数据 │ │ 企业内部 │ │ \"自信地编造\" │ │\n│ │ 截止在 │ │ 文档不在 │ │ 看似合理实则 │ │\n│ │ 2025年X月 │ │ 训练集中 │ │ 错误的答案 │ │\n│ └─────────────┘ └─────────────┘ └─────────────────────┘ │\n│ ▲ ▲ ▲ │\n│ │ │ │ │\n│ └────────────────┼────────────────────┘ │\n│ ▼ │\n│ ┌──────────────────────┐ │\n│ │ RAG: 检索增强生成 │ │\n│ │ (Retrieval │ │\n│ │ Augmented │ │\n│ │ Generation) │ │\n│ │ │ │\n│ │ 让 LLM 基于检索到的 │ │\n│ │ 真实文档来回答问题 │ │\n│ └──────────────────────┘ │\n└──────────────────────────────────────────────────────────────┘", "section_ref": "12.1.1", "runnable": false, "dependencies": [] }, { "id": "code-2", "language": "python", "description": "3. 生成（Generation）：LLM 基于这些真实的上下文信息来生成回答", "code": "# RAG 的核心流程（伪代码）\ndef rag_answer(user_question: str) -> str:\n # Step 1: 检索相关文档\n relevant_docs = vector_store.search(\n query=embed(user_question),\n top_k=5\n )\n \n # Step 2: 构建增强提示\n context = \"\\n\\n\".join(doc.content for doc in relevant_docs)\n prompt = f\"\"\"\n 基于以下参考资料回答用户问题。\n 如果参考资料中没有相关信息，请直接回答\"我不知道\"。\n \n 参考资料：\n {context}\n \n 用户问题：{user_question}\n \"\"\"\n \n # Step 3: LLM 生成回答\n answer = llm.generate(prompt)\n return answer", "section_ref": "12.1.2", "runnable": true, "dependencies": [] }, { "id": "code-3", "language": "text", "description": "RAG 技术自 2020 年由 Meta（Facebook AI Research）提出以来，已经经历了多代演进：", "code": "RAG 技术演进时间线\n═══════════════════════════════════════════════════════\n\n2020 RAG 论文发表\n │ Meta 提出标准 RAG 范式\n │ (Lewis et al., \"Retrieval-Augmented Generation\")\n │\n2021 朴素 RAG (Naive RAG)\n │ Document → Split → Embed → Store → Retrieve → Generate\n │ 简单粗暴，效果有限\n │\n2022 高级 RAG (Advanced RAG)\n │ 引入：重排序(Reranking)、查询重写(Query Rewriting)\n │ 混合检索(Hybrid Search)、多跳检索(Multi-hop)\n │\n2023 模块化 RAG (Modular RAG)\n │ RAG 组件可插拔替换\n │ LangChain/LlamaIndex 生态繁荣\n │\n2024 Agent化 RAG\n │ RAG + Agent 推理能力\n │ 自适应检索、Self-RAG、CRAG\n │\n2025 GraphRAG + Agentic RAG\n │ 知识图谱与 RAG 深度融合\n │ 多 Agent 协作检索、自动知识库管理\n═══════════════════════════════════════════════════════", "section_ref": "12.1.4", "runnable": false, "dependencies": [] }, { "id": "code-4", "language": "text", "description": "一个生产级 RAG 系统包含多个协作组件，每个组件都有多种实现选择：", "code": "┌─────────────────────────────────────────────────────────────────────┐\n│ RAG 系统完整架构 │\n│ │\n│ ┌──── 索引阶段 (Indexing) ────────────────────────────────────┐ │\n│ │ │ │\n│ │ 文档源 文档处理 向量化 存储 │ │\n│ │ ┌──────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │\n│ │ │PDF │───▶│ Loader │───▶│ Splitter │───▶│ Embedding│ │ │\n│ │ │HTML │ │ │ │ │ │ │ │ │\n│ │ │Word │ └──────────┘ └──────────┘ └────┬─────┘ │ │\n│ │ │Markdown│ │ │ │\n│ │ │Database│ ▼ │ │\n│ │ │API │ ┌──────────┐ │ │\n│ │ └──────┘ │ Vector │ │ │\n│ │ │ Store │ │ │\n│ │ ┌──────┐ │ │ │ │\n│ │ │ Meta │───────────────────────────────────▶│ Metadata │ │ │\n│ │ │ data │ │ Index │ │ │\n│ │ └──────┘ └──────────┘ │ │\n│ └──────────────────────────────────────────────────────────────┘ │\n│ │\n│ ┌──── 查询阶段 (Querying) ─────────────────────────────────────┐ │\n│ │ │ │\n│ │ 用户查询 │ │\n│ │ │ │ │\n│ │ ▼ │ │\n│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │\n│ │ │ Query │───▶│ Retriever│───▶│Reranker │───▶ Top-K 文档 │ │\n│ │ │ Rewrite │ │ │ │ │ │ │\n│ │ └──────────┘ │ 稠密检索 │ │ 交叉编码 │ │ │\n│ │ │ 稀疏检索 │ │ 精排 │ │ │\n│ │ │ 混合检索 │ └──────────┘ │ │\n│ │ └──────────┘ │ │\n│ └──────────────────────────────────────────────────────────────┘ │\n│ │\n│ ┌──── 生成阶段 (Generation) ────────────────────────────────────┐ │\n│ │ │ │\n│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │\n│ │ │ Context │───▶│ LLM │───▶│ Response │ │ │\n│ │ │ Assembly │ │ Generate │ │ & Cite │ │ │\n│ │ └──────────┘ └──────────┘ └──────────┘ │ │\n│ │ │ ▲ │ │\n│ │ │ │ │ │\n│ │ └── 反馈循环 ────┘ (Self-RAG: 自我评估 + 补充检索) │ │\n│ └──────────────────────────────────────────────────────────────┘ │\n└─────────────────────────────────────────────────────────────────────┘", "section_ref": "12.2.1", "runnable": false, "dependencies": [] }, { "id": "code-5", "language": "python", "description": "文档加载器负责将各种格式的文档统一转换为纯文本或结构化文本：", "code": "from pathlib import Path\nfrom typing import AsyncIterator\nimport json\n\nclass Document:\n \"\"\"统一文档表示\"\"\"\n def __init__(\n self,\n content: str,\n metadata: dict,\n source: str,\n doc_id: str | None = None\n ):\n self.content = content\n self.metadata = metadata # 标题、作者、日期、标签等\n self.source = source\n self.doc_id = doc_id\n \n def to_dict(self) -> dict:\n return {\n \"content\": self.content,\n \"metadata\": self.metadata,\n \"source\": self.source,\n \"doc_id\": self.doc_id\n }\n\n\nclass DocumentLoader:\n \"\"\"统一文档加载器\"\"\"\n \n def __init__(self):\n self._loaders = {\n \".pdf\": self._load_pdf,\n \".md\": self._load_markdown,\n \".txt\": self._load_text,\n \".html\": self._load_html,\n \".docx\": self._load_docx,\n \".json\": self._load_json,\n }\n \n def load(self, path: str) -> Document:\n \"\"\"加载单个文档\"\"\"\n suffix = Path(path).suffix.lower()\n loader = self._loaders.get(suffix)\n if loader is None:\n raise ValueError(f\"Unsupported format: {suffix}\")\n return loader(path)\n \n async def load_directory(\n self,\n directory: str,\n recursive: bool = True\n ) -> AsyncIterator[Document]:\n \"\"\"批量加载目录中的文档\"\"\"\n dir_path = Path(directory)\n pattern = \"**/*\" if recursive else \"*\"\n \n for file_path in dir_path.glob(pattern):\n if file_path.is_file() and file_path.suffix.lower() in self._loaders:\n try:\n doc = self.load(str(file_path))\n yield doc\n except Exception as e:\n print(f\"Failed to load {file_path}: {e}\")\n \n def _load_pdf(self, path: str) -> Document:\n \"\"\"加载 PDF 文档\"\"\"\n import pdfplumber\n \n text_parts = []\n metadata = {}\n \n with pdfplumber.open(path) as pdf:\n metadata = {\n \"title\": pdf.metadata.get(\"Title\", \"\"),\n \"author\": pdf.metadata.get(\"Author\", \"\"),\n \"pages\": len(pdf.pages),\n \"format\": \"pdf\"\n }\n for page in pdf.pages:\n page_text = page.extract_text()\n if page_text:\n text_parts.append(page_text)\n \n return Document(\n content=\"\\n\\n\".join(text_parts),\n metadata=metadata,\n source=path,\n doc_id=f\"pdf_{Path(path).stem}\"\n )\n \n def _load_markdown(self, path: str) -> Document:\n \"\"\"加载 Markdown 文档\"\"\"\n with open(path, \"r\", encoding=\"utf-8\") as f:\n content = f.read()\n \n # 提取 Front Matter（如果有）\n metadata = {\"format\": \"markdown\"}\n if content.startswith(\"---\"):\n parts = content.split(\"---\", 2)\n if len(parts) >= 3:\n try:\n metadata.update(json.loads(parts[1]))\n except json.JSONDecodeError:\n # YAML 格式的 front matter\n import yaml\n try:\n metadata.update(yaml.safe_load(parts[1]) or {})\n except:\n pass\n content = parts[2].strip()\n \n return Document(\n content=content,\n metadata=metadata,\n source=path,\n doc_id=f\"md_{Path(path).stem}\"\n )\n \n def _load_text(self, path: str) -> Document:\n with open(path, \"r\", encoding=\"utf-8\") as f:\n content = f.read()\n return Document(\n content=content,\n metadata={\"format\": \"text\"},\n source=path,\n doc_id=f\"txt_{Path(path).stem}\"\n )\n \n def _load_html(self, path: str) -> Document:\n from bs4 import BeautifulSoup\n \n with open(path, \"r\", encoding=\"utf-8\") as f:\n soup = BeautifulSoup(f.read(), \"html.parser\")\n \n # 移除 script 和 style\n for tag in soup([\"script\", \"style\", \"nav\", \"footer\"]):\n tag.decompose()\n \n return Document(\n content=soup.get_text(separator=\"\\n\", strip=True),\n metadata={\"format\": \"html\", \"title\": soup.title.string if soup.title else \"\"},\n source=path,\n doc_id=f\"html_{Path(path).stem}\"\n )\n \n def _load_docx(self, path: str) -> Document:\n from docx import Document as DocxDocument\n \n doc = DocxDocument(path)\n content = \"\\n\\n\".join(para.text for para in doc.paragraphs if para.text.strip())\n \n return Document(\n content=content,\n metadata={\"format\": \"docx\"},\n source=path,\n doc_id=f\"docx_{Path(path).stem}\"\n )\n \n def _load_json(self, path: str) -> Document:\n with open(path, \"r\", encoding=\"utf-8\") as f:\n data = json.load(f)\n \n # 递归展平 JSON 为文本\n text_parts = []\n def flatten(obj, prefix=\"\"):\n if isinstance(obj, dict):\n for k, v in obj.items():\n flatten(v, f\"{prefix}{k}.\")\n elif isinstance(obj, list):\n for i, v in enumerate(obj):\n flatten(v, f\"{prefix}[{i}].\")\n else:\n text_parts.append(f\"{prefix.rstrip('.')}: {obj}\")\n \n flatten(data)\n \n return Document(\n content=\"\\n\".join(text_parts),\n metadata={\"format\": \"json\"},\n source=path,\n doc_id=f\"json_{Path(path).stem}\"\n )", "section_ref": "12.2.2", "runnable": true, "dependencies": [] }, { "id": "code-6", "language": "python", "description": "文本分割是 RAG 中最关键也最容易被忽视的环节。分块策略直接影响检索质量。", "code": "from dataclasses import dataclass\nfrom typing import List\nimport re\n\n@dataclass\nclass Chunk:\n \"\"\"文档块\"\"\"\n content: str\n chunk_id: str\n metadata: dict\n start_index: int\n end_index: int\n\n\nclass BaseSplitter:\n \"\"\"分块器基类\"\"\"\n def split(self, text: str, metadata: dict = None) -> List[Chunk]:\n raise NotImplementedError\n \n def split_documents(self, documents: List[Document]) -> List[Chunk]:\n chunks = []\n for doc in documents:\n chunks.extend(self.split(doc.content, doc.metadata))\n return chunks\n\n\nclass FixedSizeSplitter(BaseSplitter):\n \"\"\"固定大小分块器 — 最简单的方案\"\"\"\n \n def __init__(self, chunk_size: int = 512, chunk_overlap: int = 50):\n self.chunk_size = chunk_size\n self.chunk_overlap = chunk_overlap\n \n def split(self, text: str, metadata: dict = None) -> List[Chunk]:\n if not text:\n return []\n \n metadata = metadata or {}\n chunks = []\n start = 0\n chunk_idx = 0\n \n while start < len(text):\n end = start + self.chunk_size\n chunk_text = text[start:end]\n \n chunks.append(Chunk(\n content=chunk_text,\n chunk_id=f\"chunk_{chunk_idx}\",\n metadata={**metadata, \"chunk_index\": chunk_idx, \"splitter\": \"fixed\"},\n start_index=start,\n end_index=end\n ))\n \n start += self.chunk_size - self.chunk_overlap\n chunk_idx += 1\n \n return chunks\n\n\nclass RecursiveCharacterSplitter(BaseSplitter):\n \"\"\"递归字符分块器 — LangChain 的默认方案\n \n 优先级：段落 → 句子 → 单词 → 字符\n 尽量在自然边界处分割，保持语义完整性\n \"\"\"\n \n def __init__(\n self,\n chunk_size: int = 1000,\n chunk_overlap: int = 200,\n separators: List[str] = None\n ):\n self.chunk_size = chunk_size\n self.chunk_overlap = chunk_overlap\n self.separators = separators or [\n \"\\n\\n\", # 段落\n \"\\n\", # 换行\n \"。\", # 中文句号\n \"！\", # 中文感叹号\n \"？\", # 中文问号\n \"；\", # 中文分号\n \". \", # 英文句号\n \"? \", # 英文问号\n \"! \", # 英文感叹号\n \"; \", # 英文分号\n \", \", # 英文逗号\n \"，\", # 中文逗号\n \" \", # 空格\n \"\", # 字符级\n ]\n \n def split(self, text: str, metadata: dict = None) -> List[Chunk]:\n metadata = metadata or {}\n final_chunks = []\n self._recursive_split(text, self.separators, self.chunk_size, final_chunks)\n \n # 添加 overlap 和元数据\n result = []\n for i, chunk_text in enumerate(final_chunks):\n result.append(Chunk(\n content=chunk_text,\n chunk_id=f\"chunk_{i}\",\n metadata={**metadata, \"chunk_index\": i, \"splitter\": \"recursive\"},\n start_index=text.find(chunk_text) if chunk_text in text else 0,\n end_index=text.find(chunk_text) + len(chunk_text) if chunk_text in text else 0\n ))\n \n return result\n \n def _recursive_split(self, text, separators, chunk_size, chunks):\n \"\"\"递归分割\"\"\"\n if len(text) <= chunk_size:\n if text.strip():\n chunks.append(text.strip())\n return\n \n # 找到合适的分隔符\n for sep in separators:\n if sep == \"\":\n # 字符级分割\n for i in range(0, len(text), chunk_size):\n chunks.append(text[i:i + chunk_size].strip())\n return\n \n if sep in text:\n parts = text.split(sep)\n current_chunk = \"\"\n \n for part in parts:\n if len(current_chunk) + len(sep) + len(part) <= chunk_size:\n current_chunk = current_chunk + sep + part if current_chunk else part\n else:\n if current_chunk.strip():\n chunks.append(current_chunk.strip())\n current_chunk = part\n \n if current_chunk.strip():\n chunks.append(current_chunk.strip())\n return\n \n # 没找到分隔符，强制分割\n chunks.append(text[:chunk_size])\n if len(text) > chunk_size:\n self._recursive_split(text[chunk_size:], separators, chunk_size, chunks)\n\n\nclass SemanticSplitter(BaseSplitter):\n \"\"\"语义分块器 — 基于 Embedding 相似度分割\n \n 思路：如果相邻两个句子的 embedding 相似度骤降，\n 说明它们属于不同的语义段落，应该在此处分割。\n \"\"\"\n \n def __init__(self, embedding_fn, max_chunk_size: int = 1500, \n similarity_threshold: float = 0.3, buffer_size: int = 3):\n self.embedding_fn = embedding_fn\n self.max_chunk_size = max_chunk_size\n self.similarity_threshold = similarity_threshold\n self.buffer_size = buffer_size\n \n def split(self, text: str, metadata: dict = None) -> List[Chunk]:\n import numpy as np\n \n metadata = metadata or {}\n \n # Step 1: 将文本拆分为句子\n sentences = re.split(r'(?<=[。！？.!?])\\s*', text)\n sentences = [s.strip() for s in sentences if s.strip()]\n \n if len(sentences) <= 1:\n return [Chunk(content=text, chunk_id=\"chunk_0\",\n metadata=metadata, start_index=0, end_index=len(text))]\n \n # Step 2: 计算每个句子的 embedding\n embeddings = self.embedding_fn(sentences)\n \n # Step 3: 计算相邻句子的余弦相似度\n similarities = []\n for i in range(len(embeddings) - 1):\n sim = np.dot(embeddings[i], embeddings[i+1]) / (\n np.linalg.norm(embeddings[i]) * np.linalg.norm(embeddings[i+1]) + 1e-8\n )\n similarities.append(sim)\n \n # Step 4: 找到相似度骤降的分割点\n split_points = [0] # 文档开头始终是一个分割点\n for i, sim in enumerate(similarities):\n if sim < self.similarity_threshold:\n split_points.append(i + 1)\n split_points.append(len(sentences)) # 文档结尾\n \n # Step 5: 组装 chunks\n chunks = []\n for i in range(len(split_points) - 1):\n start = split_points[i]\n end = split_points[i + 1]\n chunk_text = \" \".join(sentences[start:end])\n \n # 如果 chunk 太大，用递归分割进一步切分\n if len(chunk_text) > self.max_chunk_size:\n sub_splitter = RecursiveCharacterSplitter(\n chunk_size=self.max_chunk_size,\n chunk_overlap=200\n )\n sub_chunks = sub_splitter.split(chunk_text, metadata)\n chunks.extend(sub_chunks)\n else:\n chunks.append(Chunk(\n content=chunk_text,\n chunk_id=f\"chunk_{len(chunks)}\",\n metadata={**metadata, \"splitter\": \"semantic\",\n \"sentences\": f\"{start}-{end}\"},\n start_index=0, # 简化处理\n end_index=len(chunk_text)\n ))\n \n return chunks", "section_ref": "12.2.3", "runnable": true, "dependencies": [] }, { "id": "code-7", "language": "python", "description": "嵌入模型将文本转换为高维向量，是 RAG 系统的核心组件：", "code": "from abc import ABC, abstractmethod\nfrom typing import List\nimport numpy as np\n\n\nclass EmbeddingModel(ABC):\n \"\"\"嵌入模型抽象接口\"\"\"\n \n @abstractmethod\n def embed_text(self, text: str) -> List[float]:\n \"\"\"将单条文本转为向量\"\"\"\n pass\n \n @abstractmethod\n def embed_batch(self, texts: List[str]) -> List[List[float]]:\n \"\"\"批量嵌入\"\"\"\n pass\n \n @property\n @abstractmethod\n def dimension(self) -> int:\n \"\"\"向量维度\"\"\"\n pass\n\n\nclass OpenAIEmbedding(EmbeddingModel):\n \"\"\"OpenAI 嵌入模型\"\"\"\n \n def __init__(self, model: str = \"text-embedding-3-small\", \n api_key: str = None, dimensions: int = 1536):\n from openai import OpenAI\n self.client = OpenAI(api_key=api_key)\n self.model = model\n self._dimensions = dimensions\n \n def embed_text(self, text: str) -> List[float]:\n response = self.client.embeddings.create(\n input=text,\n model=self.model,\n dimensions=self._dimensions\n )\n return response.data[0].embedding\n \n def embed_batch(self, texts: List[str]) -> List[List[float]]:\n # OpenAI 支持批量，但限制 2048 条\n all_embeddings = []\n batch_size = 2048\n \n for i in range(0, len(texts), batch_size):\n batch = texts[i:i + batch_size]\n response = self.client.embeddings.create(\n input=batch,\n model=self.model,\n dimensions=self._dimensions\n )\n batch_embeddings = [item.embedding for item in response.data]\n all_embeddings.extend(batch_embeddings)\n \n return all_embeddings\n \n @property\n def dimension(self) -> int:\n return self._dimensions\n\n\nclass LocalEmbedding(EmbeddingModel):\n \"\"\"本地嵌入模型（使用 sentence-transformers）\"\"\"\n \n def __init__(self, model_name: str = \"BAAI/bge-large-zh-v1.5\"):\n from sentence_transformers import SentenceTransformer\n self.model = SentenceTransformer(model_name)\n self._dimension = self.model.get_sentence_embedding_dimension()\n \n def embed_text(self, text: str) -> List[float]:\n embedding = self.model.encode(text, normalize_embeddings=True)\n return embedding.tolist()\n \n def embed_batch(self, texts: List[str]) -> List[List[float]]:\n embeddings = self.model.encode(texts, normalize_embeddings=True)\n return [e.tolist() for e in embeddings]\n \n @property\n def dimension(self) -> int:\n return self._dimension\n\n\ndef cosine_similarity(a: List[float], b: List[float]) -> float:\n \"\"\"余弦相似度计算\"\"\"\n a_vec = np.array(a)\n b_vec = np.array(b)\n return float(np.dot(a_vec, b_vec) / \n (np.linalg.norm(a_vec) * np.linalg.norm(b_vec) + 1e-8))", "section_ref": "12.2.4", "runnable": true, "dependencies": [ "numpy" ] }, { "id": "code-8", "language": "python", "description": "向量数据库存储文档的向量表示，支持高效的相似度检索：", "code": "from abc import ABC, abstractmethod\nfrom typing import List, Optional\nfrom dataclasses import dataclass\nimport json\nimport os\n\n@dataclass\nclass SearchResult:\n \"\"\"检索结果\"\"\"\n content: str\n score: float\n metadata: dict\n chunk_id: str\n source: str\n\n\nclass VectorStore(ABC):\n \"\"\"向量存储抽象接口\"\"\"\n \n @abstractmethod\n def add_documents(self, chunks: List[Chunk], embeddings: List[List[float]]) -> None:\n \"\"\"添加文档\"\"\"\n pass\n \n @abstractmethod\n def search(self, query_embedding: List[float], top_k: int = 5,\n filters: dict = None) -> List[SearchResult]:\n \"\"\"相似度搜索\"\"\"\n pass\n \n @abstractmethod\n def delete(self, doc_ids: List[str]) -> None:\n \"\"\"删除文档\"\"\"\n pass\n\n\nclass ChromaDBStore(VectorStore):\n \"\"\"ChromaDB 向量存储 — 轻量级，适合开发和小规模部署\"\"\"\n \n def __init__(self, collection_name: str = \"rag_documents\",\n persist_directory: str = \"./chroma_db\"):\n import chromadb\n from chromadb.config import Settings\n \n self.client = chromadb.Client(Settings(\n chroma_db_impl=\"duckdb+parquet\",\n persist_directory=persist_directory\n ))\n self.collection = self.client.get_or_create_collection(\n name=collection_name,\n metadata={\"hnsw:space\": \"cosine\"}\n )\n \n def add_documents(self, chunks: List[Chunk], embeddings: List[List[float]]) -> None:\n if not chunks:\n return\n \n ids = [c.chunk_id for c in chunks]\n documents = [c.content for c in chunks]\n metadatas = [\n {**c.metadata, \"source\": c.metadata.get(\"source\", \"\")}\n for c in chunks\n ]\n \n self.collection.add(\n ids=ids,\n documents=documents,\n embeddings=embeddings,\n metadatas=metadatas\n )\n \n def search(self, query_embedding: List[float], top_k: int = 5,\n filters: dict = None) -> List[SearchResult]:\n where = None\n if filters:\n where = filters\n \n results = self.collection.query(\n query_embeddings=[query_embedding],\n n_results=top_k,\n where=where,\n include=[\"documents\", \"metadatas\", \"distances\"]\n )\n \n search_results = []\n if results[\"documents\"] and results[\"documents\"][0]:\n for i, doc in enumerate(results[\"documents\"][0]):\n search_results.append(SearchResult(\n content=doc,\n score=1 - results[\"distances\"][0][i], # 距离转相似度\n metadata=results[\"metadatas\"][0][i],\n chunk_id=results[\"ids\"][0][i],\n source=results[\"metadatas\"][0][i].get(\"source\", \"\")\n ))\n \n return search_results\n \n def delete(self, doc_ids: List[str]) -> None:\n self.collection.delete(ids=doc_ids)\n\n\nclass InMemoryVectorStore(VectorStore):\n \"\"\"内存向量存储 — 仅用于测试和演示\"\"\"\n \n def __init__(self):\n self._documents: List[dict] = []\n \n def add_documents(self, chunks: List[Chunk], embeddings: List[List[float]]) -> None:\n for chunk, embedding in zip(chunks, embeddings):\n self._documents.append({\n \"content\": chunk.content,\n \"embedding\": embedding,\n \"metadata\": chunk.metadata,\n \"chunk_id\": chunk.chunk_id,\n \"source\": chunk.metadata.get(\"source\", \"\")\n })\n \n def search(self, query_embedding: List[float], top_k: int = 5,\n filters: dict = None) -> List[SearchResult]:\n import numpy as np\n \n query_vec = np.array(query_embedding)\n \n scored = []\n for doc in self._documents:\n # 过滤\n if filters:\n match = True\n for k, v in filters.items():\n if doc[\"metadata\"].get(k) != v:\n match = False\n break\n if not match:\n continue\n \n doc_vec = np.array(doc[\"embedding\"])\n score = float(np.dot(query_vec, doc_vec) / \n (np.linalg.norm(query_vec) * np.linalg.norm(doc_vec) + 1e-8))\n scored.append({\n \"content\": doc[\"content\"],\n \"score\": score,\n \"metadata\": doc[\"metadata\"],\n \"chunk_id\": doc[\"chunk_id\"],\n \"source\": doc[\"source\"]\n })\n \n scored.sort(key=lambda x: x[\"score\"], reverse=True)\n top_results = scored[:top_k]\n \n return [SearchResult(**r) for r in top_results]\n \n def delete(self, doc_ids: List[str]) -> None:\n id_set = set(doc_ids)\n self._documents = [d for d in self._documents if d[\"chunk_id\"] not in id_set]", "section_ref": "12.2.5", "runnable": true, "dependencies": [] }, { "id": "code-9", "language": "python", "description": "", "code": "import asyncio\nfrom typing import Optional\nfrom dataclasses import dataclass, field\nfrom datetime import datetime\n\n\n@dataclass\nclass IndexingConfig:\n \"\"\"索引配置\"\"\"\n chunk_size: int = 1000\n chunk_overlap: int = 200\n splitter_type: str = \"recursive\" # \"fixed\", \"recursive\", \"semantic\"\n embedding_model: str = \"BAAI/bge-large-zh-v1.5\"\n batch_size: int = 32\n similarity_threshold: float = 0.3 # 语义分割阈值\n\n\n@dataclass \nclass IndexingStats:\n \"\"\"索引统计\"\"\"\n total_documents: int = 0\n total_chunks: int = 0\n failed_documents: int = 0\n total_time_seconds: float = 0.0\n errors: list = field(default_factory=list)\n\n\nclass RAGIndexer:\n \"\"\"RAG 索引构建器\"\"\"\n \n def __init__(\n self,\n config: IndexingConfig,\n embedding_model: EmbeddingModel,\n vector_store: VectorStore\n ):\n self.config = config\n self.embedding_model = embedding_model\n self.vector_store = vector_store\n self.loader = DocumentLoader()\n self._stats = IndexingStats()\n \n def _create_splitter(self) -> BaseSplitter:\n if self.config.splitter_type == \"fixed\":\n return FixedSizeSplitter(\n chunk_size=self.config.chunk_size,\n chunk_overlap=self.config.chunk_overlap\n )\n elif self.config.splitter_type == \"recursive\":\n return RecursiveCharacterSplitter(\n chunk_size=self.config.chunk_size,\n chunk_overlap=self.config.chunk_overlap\n )\n elif self.config.splitter_type == \"semantic\":\n return SemanticSplitter(\n embedding_fn=self.embedding_model.embed_text,\n similarity_threshold=self.config.similarity_threshold\n )\n else:\n raise ValueError(f\"Unknown splitter: {self.config.splitter_type}\")\n \n def index_documents(self, document_paths: list[str]) -> IndexingStats:\n \"\"\"索引一批文档\"\"\"\n start_time = datetime.now()\n splitter = self._create_splitter()\n \n all_chunks = []\n \n for path in document_paths:\n try:\n doc = self.loader.load(path)\n chunks = splitter.split(doc.content, doc.metadata)\n all_chunks.extend(chunks)\n self._stats.total_documents += 1\n except Exception as e:\n self._stats.failed_documents += 1\n self._stats.errors.append(f\"{path}: {str(e)}\")\n \n # 批量嵌入\n if all_chunks:\n self._batch_embed_and_store(all_chunks)\n \n elapsed = (datetime.now() - start_time).total_seconds()\n self._stats.total_time_seconds = elapsed\n \n print(f\"索引完成: {self._stats.total_documents} 文档 → \"\n f\"{self._stats.total_chunks} 块, \"\n f\"耗时 {elapsed:.2f}s\")\n \n if self._stats.errors:\n print(f\"失败: {self._stats.failed_documents} 个文档\")\n for err in self._stats.errors:\n print(f\" ⚠ {err}\")\n \n return self._stats\n \n def _batch_embed_and_store(self, chunks: list[Chunk]):\n \"\"\"批量嵌入并存储\"\"\"\n batch_size = self.config.batch_size\n \n for i in range(0, len(chunks), batch_size):\n batch = chunks[i:i + batch_size]\n texts = [c.content for c in batch]\n \n print(f\" 嵌入批次 {i//batch_size + 1}/{(len(chunks)-1)//batch_size + 1} \"\n f\"({len(batch)} 块)...\")\n \n embeddings = self.embedding_model.embed_batch(texts)\n self.vector_store.add_documents(batch, embeddings)\n self._stats.total_chunks += len(batch)\n \n async def index_directory(self, directory: str, recursive: bool = True) -> IndexingStats:\n \"\"\"异步索引整个目录\"\"\"\n paths = []\n dir_path = Path(directory)\n pattern = \"**/*\" if recursive else \"*\"\n \n for file_path in dir_path.glob(pattern):\n if file_path.is_file() and file_path.suffix.lower() in self.loader._loaders:\n paths.append(str(file_path))\n \n print(f\"发现 {len(paths)} 个文档待索引\")\n return self.index_documents(paths)\n\n\n# 使用示例\nif __name__ == \"__main__\":\n config = IndexingConfig(\n chunk_size=1000,\n chunk_overlap=200,\n splitter_type=\"recursive\",\n embedding_model=\"BAAI/bge-large-zh-v1.5\"\n )\n \n embedding_model = LocalEmbedding(\"BAAI/bge-large-zh-v1.5\")\n vector_store = ChromaDBStore(\"knowledge_base\", \"./chroma_db\")\n \n indexer = RAGIndexer(config, embedding_model, vector_store)\n stats = indexer.index_documents([\"./docs/policy.pdf\", \"./docs/faq.md\"])", "section_ref": "12.3.1", "runnable": true, "dependencies": [] }, { "id": "code-10", "language": "text", "description": "分块大小的黄金法则：", "code": "分块大小与检索质量的关系\n═══════════════════════════════════════════\n 太小 (100-200 字符)\n ├── ✅ 精准匹配好\n ├── ❌ 缺乏上下文\n ├── ❌ 语义不完整\n └── ❌ 噪声多\n\n 适中 (500-1500 字符) ← 推荐区间\n ├── ✅ 语义完整\n ├── ✅ 上下文充分\n ├── ✅ 检索效率好\n └── ✅ 嵌入质量高\n\n 太大 (2000+ 字符)\n ├── ✅ 上下文丰富\n ├── ❌ 稀释关键信息\n ├── ❌ 嵌入质量下降\n └── ❌ Token 消耗大\n═══════════════════════════════════════════", "section_ref": "12.3.2", "runnable": false, "dependencies": [] }, { "id": "code-11", "language": "python", "description": "3. 元数据附加：添加文档层级元数据（作者、日期、部门），支持过滤检索", "code": "class ParentChildSplitter:\n \"\"\"父子块分割器\n \n 检索时用小块提高精度，返回时附带父块提供完整上下文\n \"\"\"\n \n def __init__(self, child_size: int = 200, parent_size: int = 1000, \n child_overlap: int = 50):\n self.child_splitter = RecursiveCharacterSplitter(\n chunk_size=child_size, chunk_overlap=child_overlap\n )\n self.parent_splitter = RecursiveCharacterSplitter(\n chunk_size=parent_size, chunk_overlap=200\n )\n \n def split(self, text: str, metadata: dict = None) -> List[dict]:\n \"\"\"返回包含父子关系的块\"\"\"\n metadata = metadata or {}\n \n # 先分父块\n parent_chunks = self.parent_splitter.split(text, metadata)\n \n result = []\n for parent in parent_chunks:\n # 再对每个父块分子块\n children = self.child_splitter.split(parent.content, parent.metadata)\n \n for child in children:\n result.append({\n \"child\": child,\n \"parent\": parent,\n \"type\": \"parent_child\"\n })\n \n return result", "section_ref": "12.3.2", "runnable": true, "dependencies": [] }, { "id": "code-12", "language": "python", "description": "稠密检索是 RAG 的默认方案——通过 embedding 的语义相似度来查找相关文档：", "code": "class DenseRetriever:\n \"\"\"稠密检索器\"\"\"\n \n def __init__(self, embedding_model: EmbeddingModel, \n vector_store: VectorStore, top_k: int = 5):\n self.embedding_model = embedding_model\n self.vector_store = vector_store\n self.top_k = top_k\n \n def retrieve(self, query: str, top_k: int = None, \n filters: dict = None) -> List[SearchResult]:\n \"\"\"检索相关文档\"\"\"\n k = top_k or self.top_k\n \n # 将查询转为向量\n query_embedding = self.embedding_model.embed_text(query)\n \n # 在向量数据库中搜索\n results = self.vector_store.search(\n query_embedding, top_k=k, filters=filters\n )\n \n return results", "section_ref": "12.4.1", "runnable": true, "dependencies": [] }, { "id": "code-13", "language": "python", "description": "BM25 是经典的信息检索算法，基于词频统计：", "code": "import math\nfrom collections import Counter, defaultdict\nfrom typing import List, Tuple\nimport re\n\n\nclass BM25Retriever:\n \"\"\"BM25 稀疏检索器\n \n 基于词频和逆文档频率的经典检索算法，\n 对精确匹配和关键词搜索特别有效\n \"\"\"\n \n def __init__(self, k1: float = 1.5, b: float = 0.75, epsilon: float = 0.25):\n self.k1 = k1 # 词频饱和参数\n self.b = b # 文档长度归一化\n self.epsilon = epsilon\n self.corpus = [] # 原始文档\n self.tokenized_corpus = [] # 分词后的文档\n self.doc_freqs = defaultdict(int) # 文档频率\n self.avg_doc_len = 0\n self.idf = {} # 逆文档频率\n \n def _tokenize(self, text: str) -> List[str]:\n \"\"\"简单分词（中文按字，英文按词）\"\"\"\n # 中英文混合分词\n tokens = []\n # 提取英文单词\n en_words = re.findall(r'[a-zA-Z0-9]+', text.lower())\n tokens.extend(en_words)\n # 提取中文字符（简化处理，实际应使用 jieba 等分词器）\n cn_chars = re.findall(r'[\\u4e00-\\u9fff]{2,}', text)\n tokens.extend(cn_chars)\n return tokens\n \n def fit(self, documents: List[dict]):\n \"\"\"构建索引\n \n Args:\n documents: [{\"content\": str, \"metadata\": dict, \"chunk_id\": str}, ...]\n \"\"\"\n self.corpus = documents\n doc_lens = []\n N = len(documents)\n \n for doc in documents:\n tokens = self._tokenize(doc[\"content\"])\n self.tokenized_corpus.append(tokens)\n doc_lens.append(len(tokens))\n \n # 统计文档频率\n unique_terms = set(tokens)\n for term in unique_terms:\n self.doc_freqs[term] += 1\n \n self.avg_doc_len = sum(doc_lens) / N if N > 0 else 0\n \n # 计算 IDF\n for term, df in self.doc_freqs.items():\n idf = math.log((N - df + 0.5) / (df + 0.5) + 1)\n self.idf[term] = max(idf, self.epsilon)\n \n def retrieve(self, query: str, top_k: int = 5) -> List[SearchResult]:\n \"\"\"BM25 检索\"\"\"\n query_tokens = self._tokenize(query)\n query_freq = Counter(query_tokens)\n \n scores = []\n for i, doc_tokens in enumerate(self.tokenized_corpus):\n score = 0.0\n doc_len = len(doc_tokens)\n doc_freq = Counter(doc_tokens)\n \n for term, qf in query_freq.items():\n if term not in self.idf:\n continue\n \n tf = doc_freq.get(term, 0)\n idf = self.idf[term]\n \n # BM25 评分公式\n numerator = tf * (self.k1 + 1)\n denominator = tf + self.k1 * (1 - self.b + self.b * doc_len / self.avg_doc_len)\n score += idf * (numerator / denominator)\n \n scores.append((i, score))\n \n # 排序取 Top-K\n scores.sort(key=lambda x: x[1], reverse=True)\n \n results = []\n for idx, score in scores[:top_k]:\n doc = self.corpus[idx]\n results.append(SearchResult(\n content=doc[\"content\"],\n score=score,\n metadata=doc.get(\"metadata\", {}),\n chunk_id=doc.get(\"chunk_id\", \"\"),\n source=doc.get(\"metadata\", {}).get(\"source\", \"\")\n ))\n \n return results\n\n\n# 使用示例\nbm25 = BM25Retriever(k1=1.5, b=0.75)\nbm25.fit([\n {\"content\": \"Python 是一种广泛使用的编程语言\", \"chunk_id\": \"c1\", \"metadata\": {}},\n {\"content\": \"Java 是企业级开发的主流语言\", \"chunk_id\": \"c2\", \"metadata\": {}},\n {\"content\": \"Go 语言以高并发著称\", \"chunk_id\": \"c3\", \"metadata\": {}},\n])\nresults = bm25.retrieve(\"Python 编程\", top_k=2)", "section_ref": "12.4.2", "runnable": true, "dependencies": [] }, { "id": "code-14", "language": "python", "description": "混合检索结合稠密检索和稀疏检索的优势：", "code": "class HybridRetriever:\n \"\"\"混合检索器\n \n 结合稠密检索（语义理解）和 BM25（关键词匹配），\n 通过加权融合两种检索结果\n \"\"\"\n \n def __init__(\n self,\n dense_retriever: DenseRetriever,\n bm25_retriever: BM25Retriever,\n alpha: float = 0.7 # 稠密检索权重\n ):\n self.dense_retriever = dense_retriever\n self.bm25_retriever = bm25_retriever\n self.alpha = alpha # alpha 控制稠密/稀疏的权重\n \n def retrieve(self, query: str, top_k: int = 5) -> List[SearchResult]:\n \"\"\"混合检索\"\"\"\n # 并行执行两种检索\n dense_results = self.dense_retriever.retrieve(query, top_k=top_k * 2)\n sparse_results = self.bm25_retriever.retrieve(query, top_k=top_k * 2)\n \n # 归一化分数\n dense_scores = self._normalize_scores(dense_results)\n sparse_scores = self._normalize_scores(sparse_results)\n \n # 加权融合\n fused = {}\n \n for result in dense_results:\n cid = result.chunk_id\n fused[cid] = {\n \"result\": result,\n \"dense_score\": dense_scores.get(cid, 0),\n \"sparse_score\": 0\n }\n \n for result in sparse_results:\n cid = result.chunk_id\n if cid in fused:\n fused[cid][\"sparse_score\"] = sparse_scores.get(cid, 0)\n else:\n fused[cid] = {\n \"result\": result,\n \"dense_score\": 0,\n \"sparse_score\": sparse_scores.get(cid, 0)\n }\n \n # 计算融合分数\n for cid, data in fused.items():\n data[\"final_score\"] = (\n self.alpha * data[\"dense_score\"] +\n (1 - self.alpha) * data[\"sparse_score\"]\n )\n \n # 排序\n sorted_results = sorted(\n fused.values(),\n key=lambda x: x[\"final_score\"],\n reverse=True\n )[:top_k]\n \n return [\n SearchResult(\n content=item[\"result\"].content,\n score=item[\"final_score\"],\n metadata=item[\"result\"].metadata,\n chunk_id=item[\"result\"].chunk_id,\n source=item[\"result\"].source\n )\n for item in sorted_results\n ]\n \n def _normalize_scores(self, results: List[SearchResult]) -> dict:\n \"\"\"Min-Max 归一化\"\"\"\n if not results:\n return {}\n \n scores = [r.score for r in results]\n min_score = min(scores)\n max_score = max(scores)\n range_score = max_score - min_score\n \n if range_score < 1e-8:\n return {r.chunk_id: 1.0 for r in results}\n \n return {\n r.chunk_id: (r.score - min_score) / range_score\n for r in results\n }", "section_ref": "12.4.3", "runnable": true, "dependencies": [] }, { "id": "code-15", "language": "python", "description": "重排序是 RAG 系统的\"杀手锏\"——先用快速检索获取候选集，再用精确的交叉编码器精排：", "code": "class CrossEncoderReranker:\n \"\"\"交叉编码器重排序\n \n 与双编码器（embedding 检索）不同，交叉编码器同时处理 query 和 document，\n 能捕捉更细粒度的相关性信号，但计算成本更高\n \"\"\"\n \n def __init__(self, model_name: str = \"cross-encoder/ms-marco-MiniLM-L-6-v2\"):\n from sentence_transformers import CrossEncoder\n self.model = CrossEncoder(model_name)\n \n def rerank(\n self, query: str, documents: List[SearchResult], top_k: int = 5\n ) -> List[SearchResult]:\n \"\"\"对检索结果重排序\"\"\"\n if not documents:\n return []\n \n pairs = [(query, doc.content) for doc in documents]\n scores = self.model.predict(pairs)\n \n # 按新分数排序\n reranked = []\n for doc, score in zip(documents, scores):\n reranked.append(SearchResult(\n content=doc.content,\n score=float(score),\n metadata=doc.metadata,\n chunk_id=doc.chunk_id,\n source=doc.source\n ))\n \n reranked.sort(key=lambda x: x.score, reverse=True)\n return reranked[:top_k]\n\n\n# 完整的检索管道\nclass RetrievalPipeline:\n \"\"\"检索管道：混合检索 + 重排序\"\"\"\n \n def __init__(\n self,\n hybrid_retriever: HybridRetriever,\n reranker: CrossEncoderReranker,\n initial_top_k: int = 20,\n final_top_k: int = 5\n ):\n self.retriever = hybrid_retriever\n self.reranker = reranker\n self.initial_top_k = initial_top_k\n self.final_top_k = final_top_k\n \n def retrieve(self, query: str) -> List[SearchResult]:\n \"\"\"完整检索流程\"\"\"\n # Stage 1: 混合检索，获取候选集\n candidates = self.retriever.retrieve(query, top_k=self.initial_top_k)\n \n # Stage 2: 交叉编码器重排序\n final_results = self.reranker.rerank(\n query, candidates, top_k=self.final_top_k\n )\n \n return final_results", "section_ref": "12.4.4", "runnable": true, "dependencies": [] }, { "id": "code-16", "language": "python", "description": "生产环境的知识库不是一成不变的——文档会被新增、修改和删除。增量更新是保持知识库时效性的关键：", "code": "from dataclasses import dataclass\nfrom datetime import datetime\nimport hashlib\n\n\n@dataclass\nclass DocumentVersion:\n \"\"\"文档版本记录\"\"\"\n doc_id: str\n source_path: str\n content_hash: str\n indexed_at: datetime\n metadata: dict\n is_deleted: bool = False\n\n\nclass KnowledgeBaseManager:\n \"\"\"知识库管理器\n \n 负责知识库的增量更新、版本管理和过期清理\n \"\"\"\n \n def __init__(self, vector_store: VectorStore, \n embedding_model: EmbeddingModel,\n version_store_path: str = \"./kb_versions.json\"):\n self.vector_store = vector_store\n self.embedding_model = embedding_model\n self.version_store_path = version_store_path\n self._versions: dict[str, DocumentVersion] = {}\n self._load_versions()\n \n def _load_versions(self):\n \"\"\"加载版本记录\"\"\"\n if os.path.exists(self.version_store_path):\n with open(self.version_store_path, \"r\") as f:\n data = json.load(f)\n for doc_id, v in data.items():\n self._versions[doc_id] = DocumentVersion(\n doc_id=v[\"doc_id\"],\n source_path=v[\"source_path\"],\n content_hash=v[\"content_hash\"],\n indexed_at=datetime.fromisoformat(v[\"indexed_at\"]),\n metadata=v[\"metadata\"],\n is_deleted=v.get(\"is_deleted\", False)\n )\n \n def _save_versions(self):\n \"\"\"保存版本记录\"\"\"\n data = {}\n for doc_id, v in self._versions.items():\n data[doc_id] = {\n \"doc_id\": v.doc_id,\n \"source_path\": v.source_path,\n \"content_hash\": v.content_hash,\n \"indexed_at\": v.indexed_at.isoformat(),\n \"metadata\": v.metadata,\n \"is_deleted\": v.is_deleted\n }\n with open(self.version_store_path, \"w\") as f:\n json.dump(data, f, ensure_ascii=False, indent=2)\n \n @staticmethod\n def _compute_hash(content: str) -> str:\n \"\"\"计算文档内容哈希\"\"\"\n return hashlib.sha256(content.encode(\"utf-8\")).hexdigest()[:16]\n \n def check_updates(self, documents: List[Document]) -> dict:\n \"\"\"检查哪些文档需要更新\n \n Returns:\n {\n \"new\": [需要新索引的文档],\n \"modified\": [需要重新索引的文档],\n \"unchanged\": [无变化的文档],\n \"deleted\": [已删除但未清理的文档]\n }\n \"\"\"\n current_sources = {doc.source for doc in documents}\n result = {\"new\": [], \"modified\": [], \"unchanged\": [], \"deleted\": []}\n \n # 检查现有文档\n for doc in documents:\n doc_id = doc.doc_id or f\"doc_{self._compute_hash(doc.content)}\"\n content_hash = self._compute_hash(doc.content)\n \n if doc_id not in self._versions:\n result[\"new\"].append(doc)\n else:\n version = self._versions[doc_id]\n if version.content_hash != content_hash:\n result[\"modified\"].append(doc)\n else:\n result[\"unchanged\"].append(doc)\n \n # 检查已删除的文档\n for doc_id, version in self._versions.items():\n if version.source_path not in current_sources and not version.is_deleted:\n result[\"deleted\"].append(version)\n \n return result\n \n def incremental_update(self, documents: List[Document]) -> dict:\n \"\"\"增量更新知识库\"\"\"\n update_info = self.check_updates(documents)\n \n splitter = RecursiveCharacterSplitter(chunk_size=1000, chunk_overlap=200)\n \n # 处理新增文档\n for doc in update_info[\"new\"]:\n chunks = splitter.split(doc.content, doc.metadata)\n embeddings = self.embedding_model.embed_batch([c.content for c in chunks])\n self.vector_store.add_documents(chunks, embeddings)\n \n doc_id = doc.doc_id or f\"doc_{self._compute_hash(doc.content)}\"\n self._versions[doc_id] = DocumentVersion(\n doc_id=doc_id,\n source_path=doc.source,\n content_hash=self._compute_hash(doc.content),\n indexed_at=datetime.now(),\n metadata=doc.metadata\n )\n print(f\" ✅ 新增: {doc.source} ({len(chunks)} 块)\")\n \n # 处理修改文档：先删后增\n for doc in update_info[\"modified\"]:\n doc_id = doc.doc_id or f\"doc_{self._compute_hash(doc.content)}\"\n # 删除旧版本的所有块\n old_chunk_ids = [f\"chunk_{i}\" for i in range(100)] # 简化处理\n self.vector_store.delete(old_chunk_ids)\n \n # 重新索引\n chunks = splitter.split(doc.content, doc.metadata)\n embeddings = self.embedding_model.embed_batch([c.content for c in chunks])\n self.vector_store.add_documents(chunks, embeddings)\n \n self._versions[doc_id] = DocumentVersion(\n doc_id=doc_id,\n source_path=doc.source,\n content_hash=self._compute_hash(doc.content),\n indexed_at=datetime.now(),\n metadata=doc.metadata\n )\n print(f\" 🔄 更新: {doc.source} ({len(chunks)} 块)\")\n \n # 处理已删除文档\n for version in update_info[\"deleted\"]:\n self.vector_store.delete([f\"{version.doc_id}_*\"])\n version.is_deleted = True\n print(f\" 🗑️ 删除: {version.source_path}\")\n \n self._save_versions()\n \n return {\n \"new_count\": len(update_info[\"new\"]),\n \"modified_count\": len(update_info[\"modified\"]),\n \"deleted_count\": len(update_info[\"deleted\"]),\n \"unchanged_count\": len(update_info[\"unchanged\"])\n }", "section_ref": "12.5.1", "runnable": true, "dependencies": [] }, { "id": "code-17", "language": "python", "description": "", "code": "class KnowledgeBaseCleaner:\n \"\"\"知识库清理器\"\"\"\n \n def __init__(self, vector_store: VectorStore,\n version_store_path: str = \"./kb_versions.json\"):\n self.vector_store = vector_store\n self.version_store_path = version_store_path\n \n def clean_expired(self, max_age_days: int = 180) -> dict:\n \"\"\"清理过期文档\n \n Args:\n max_age_days: 文档最大保留天数\n \"\"\"\n cutoff = datetime.now().timestamp() - max_age_days * 86400\n expired = []\n \n # 扫描版本记录\n if os.path.exists(self.version_store_path):\n with open(self.version_store_path, \"r\") as f:\n versions = json.load(f)\n \n for doc_id, info in versions.items():\n indexed_at = datetime.fromisoformat(info[\"indexed_at\"]).timestamp()\n if indexed_at < cutoff:\n expired.append(doc_id)\n \n # 删除过期文档\n for doc_id in expired:\n self.vector_store.delete([doc_id])\n print(f\" 清理过期文档: {doc_id}\")\n \n return {\n \"cleaned_count\": len(expired),\n \"max_age_days\": max_age_days\n }\n \n def deduplicate(self, similarity_threshold: float = 0.98) -> dict:\n \"\"\"去重：删除高度相似的文档块\"\"\"\n # 获取所有文档\n all_docs = self.vector_store.get_all() # 假设 VectorStore 有此方法\n \n duplicates = []\n seen = set()\n \n for doc in all_docs:\n if doc.chunk_id in seen:\n continue\n \n # 检查是否与已有文档高度相似\n for other in all_docs:\n if other.chunk_id == doc.chunk_id or other.chunk_id in seen:\n continue\n \n if doc.score >= similarity_threshold: # 需要比较逻辑\n duplicates.append(other.chunk_id)\n seen.add(other.chunk_id)\n \n if duplicates:\n self.vector_store.delete(duplicates)\n \n return {\"duplicate_count\": len(duplicates)}", "section_ref": "12.5.2", "runnable": true, "dependencies": [] }, { "id": "code-18", "language": "text", "description": "GraphRAG = 向量检索 + 图遍历，将两种知识表示方式的优势互补：", "code": "传统 RAG vs GraphRAG\n═══════════════════════════════════════════════════════\n\n传统 RAG (基于向量)\n Query: \"张三在哪里工作？\"\n ──▶ 向量检索 ──▶ 找到包含\"张三\"的文档片段\n ✅ 简单事实查询\n ❌ 多跳推理困难\n ❌ 关系查询弱\n\nGraphRAG (向量 + 图)\n Query: \"张三的直属领导的母校是哪所大学？\"\n ──▶ 实体识别: 张三(人), 直属领导(关系), 母校(属性)\n ──▶ 图遍历: 张三 →[直属]→ 李四 →[母校]→ 清华大学\n ──▶ 向量检索补充: 清华大学相关的上下文\n ✅ 多跳推理\n ✅ 关系查询\n ✅ 结构化知识\n═══════════════════════════════════════════════════════", "section_ref": "12.6.1", "runnable": false, "dependencies": [] }, { "id": "code-19", "language": "python", "description": "", "code": "from dataclasses import dataclass, field\nfrom typing import List, Dict, Optional, Set, Tuple\nimport networkx as nx\n\n\n@dataclass\nclass Entity:\n \"\"\"知识实体\"\"\"\n name: str\n entity_type: str # 人、组织、地点、产品、概念...\n properties: Dict = field(default_factory=dict)\n description: str = \"\"\n\n\n@dataclass\nclass Relation:\n \"\"\"实体关系\"\"\"\n source: str # 源实体名称\n target: str # 目标实体名称\n relation_type: str # 关系类型：任职、位于、生产、属于...\n properties: Dict = field(default_factory=dict)\n\n\nclass KnowledgeGraph:\n \"\"\"轻量级知识图谱\"\"\"\n \n def __init__(self):\n self.graph = nx.DiGraph()\n self.entities: Dict[str, Entity] = {}\n self.relations: List[Relation] = []\n \n def add_entity(self, entity: Entity):\n \"\"\"添加实体\"\"\"\n self.entities[entity.name] = entity\n self.graph.add_node(\n entity.name,\n type=entity.entity_type,\n description=entity.description,\n **entity.properties\n )\n \n def add_relation(self, relation: Relation):\n \"\"\"添加关系\"\"\"\n self.relations.append(relation)\n \n # 如果实体不存在，自动创建\n if relation.source not in self.entities:\n self.add_entity(Entity(name=relation.source, entity_type=\"unknown\"))\n if relation.target not in self.entities:\n self.add_entity(Entity(name=relation.target, entity_type=\"unknown\"))\n \n self.graph.add_edge(\n relation.source,\n relation.target,\n type=relation.relation_type,\n **relation.properties\n )\n \n def get_neighbors(self, entity_name: str, relation_type: str = None) -> List[Tuple[str, str]]:\n \"\"\"获取相邻实体\n \n Returns:\n [(邻居实体名, 关系类型), ...]\n \"\"\"\n neighbors = []\n for _, target, data in self.graph.out_edges(entity_name, data=True):\n if relation_type is None or data.get(\"type\") == relation_type:\n neighbors.append((target, data.get(\"type\", \"\")))\n return neighbors\n \n def multi_hop_search(\n self, start_entity: str, max_hops: int = 3\n ) -> List[List[Tuple[str, str, str]]]:\n \"\"\"多跳图遍历\n \n Returns:\n [[(源, 关系, 目标), ...], ...] 所有可达路径\n \"\"\"\n paths = []\n visited = {start_entity}\n \n def dfs(current: str, path: list, depth: int):\n if depth >= max_hops:\n if path:\n paths.append(path[:])\n return\n \n for _, target, data in self.graph.out_edges(current, data=True):\n if target not in visited or depth < max_hops - 1:\n edge = (current, data.get(\"type\", \"\"), target)\n path.append(edge)\n dfs(target, path, depth + 1)\n path.pop()\n \n dfs(start_entity, [], 0)\n return paths\n \n def get_subgraph(self, entity_name: str, radius: int = 2) -> 'KnowledgeGraph':\n \"\"\"获取以某实体为中心的子图\"\"\"\n sub_nodes = set()\n sub_nodes.add(entity_name)\n \n # BFS 扩展\n current_level = {entity_name}\n for _ in range(radius):\n next_level = set()\n for node in current_level:\n for _, target in self.graph.out_edges(node):\n next_level.add(target)\n for source, _ in self.graph.in_edges(node):\n next_level.add(source)\n sub_nodes.update(next_level)\n current_level = next_level\n \n # 构建子图\n sub_kg = KnowledgeGraph()\n for node in sub_nodes:\n if node in self.entities:\n sub_kg.add_entity(self.entities[node])\n \n for rel in self.relations:\n if rel.source in sub_nodes and rel.target in sub_nodes:\n sub_kg.add_relation(rel)\n \n return sub_kg\n \n def to_context_text(self, entity_name: str, radius: int = 2) -> str:\n \"\"\"将子图转为文本上下文（供 LLM 使用）\"\"\"\n sub_kg = self.get_subgraph(entity_name, radius)\n \n lines = [f\"与「{entity_name}」相关的知识图谱信息：\\n\"]\n \n for rel in sub_kg.relations:\n source_desc = sub_kg.entities.get(rel.source, Entity(rel.source, \"\"))\n target_desc = sub_kg.entities.get(rel.target, Entity(rel.target, \"\"))\n lines.append(\n f\"- {source_desc.name}({source_desc.entity_type}) \"\n f\"--[{rel.relation_type}]--> \"\n f\"{target_desc.name}({target_desc.entity_type})\"\n )\n \n return \"\\n\".join(lines)\n\n\nclass SimpleEntityExtractor:\n \"\"\"简单的基于规则的实体关系抽取器\n \n 生产环境应使用 LLM 或专门的 NER 模型\n \"\"\"\n \n def __init__(self):\n self.entity_patterns = {\n \"person\": r'[\\u4e00-\\u9fff]{2,4}(?=是|在|任职|毕业于)',\n \"organization\": r'[\\u4e00-\\u9fff]{2,10}(?=公司|集团|大学|研究院|部门)',\n \"location\": r'[\\u4e00-\\u9fff]{2,6}(?=市|省|区|县|镇)',\n }\n \n def extract(self, text: str) -> Tuple[List[Entity], List[Relation]]:\n \"\"\"从文本中抽取实体和关系\"\"\"\n import re\n \n entities = []\n relations = []\n \n # 提取实体\n entity_names = set()\n for etype, pattern in self.entity_patterns.items():\n matches = re.findall(pattern, text)\n for match in matches:\n if match not in entity_names:\n entities.append(Entity(name=match, entity_type=etype))\n entity_names.add(match)\n \n # 简单关系抽取（基于模板）\n relation_patterns = [\n (r'([\\u4e00-\\u9fff]{2,4})在([\\u4e00-\\u9fff]+)(公司|集团)', \"任职于\"),\n (r'([\\u4e00-\\u9fff]{2,4})毕业于([\\u4e00-\\u9fff]+)(大学|学院)', \"就读于\"),\n (r'([\\u4e00-\\u9fff]+)(公司|集团)位于([\\u4e00-\\u9fff]+)', \"位于\"),\n ]\n \n for pattern, rel_type in relation_patterns:\n matches = re.findall(pattern, text)\n for match in matches:\n source = match[0]\n target = \"\".join(match[1:])\n relations.append(Relation(source=source, target=target, relation_type=rel_type))\n \n return entities, relations", "section_ref": "12.6.2", "runnable": true, "dependencies": [ "networkx" ] }, { "id": "code-20", "language": "python", "description": "", "code": "class GraphRAGRetriever:\n \"\"\"GraphRAG 混合检索器\n \n 结合知识图谱的关系遍历和向量语义检索\n \"\"\"\n \n def __init__(\n self,\n knowledge_graph: KnowledgeGraph,\n entity_extractor,\n dense_retriever: DenseRetriever,\n graph_weight: float = 0.4,\n vector_weight: float = 0.6\n ):\n self.kg = knowledge_graph\n self.extractor = entity_extractor\n self.dense_retriever = dense_retriever\n self.graph_weight = graph_weight\n self.vector_weight = vector_weight\n \n def retrieve(self, query: str, top_k: int = 5) -> List[SearchResult]:\n \"\"\"GraphRAG 检索\"\"\"\n import numpy as np\n \n # Step 1: 从查询中识别实体\n entities, _ = self.extractor.extract(query)\n \n # Step 2: 知识图谱遍历\n graph_context = \"\"\n graph_results = []\n \n for entity in entities:\n if entity.name in self.kg.entities:\n # 获取实体周围的子图\n context_text = self.kg.to_context_text(entity.name, radius=2)\n graph_context += context_text + \"\\n\"\n graph_results.append(SearchResult(\n content=context_text,\n score=0.8, # 图检索的基础分数\n metadata={\"source\": \"knowledge_graph\", \"entity\": entity.name},\n chunk_id=f\"graph_{entity.name}\",\n source=\"knowledge_graph\"\n ))\n \n # Step 3: 向量检索\n vector_results = self.dense_retriever.retrieve(query, top_k=top_k * 2)\n \n # Step 4: 融合结果\n all_results = []\n \n # 添加图检索结果\n for result in graph_results:\n all_results.append(SearchResult(\n content=result.content,\n score=result.score * self.graph_weight,\n metadata=result.metadata,\n chunk_id=result.chunk_id,\n source=result.source\n ))\n \n # 添加向量检索结果\n for result in vector_results:\n all_results.append(SearchResult(\n content=result.content,\n score=result.score * self.vector_weight,\n metadata=result.metadata,\n chunk_id=result.chunk_id,\n source=result.source\n ))\n \n # 排序\n all_results.sort(key=lambda x: x.score, reverse=True)\n \n return all_results[:top_k]", "section_ref": "12.6.3", "runnable": true, "dependencies": [] }, { "id": "code-21", "language": "python", "description": "", "code": "\"\"\"\n完整的 RAG Agent 实现\n使用 LangChain + ChromaDB + OpenAI\n\n功能：\n- 文档加载与分块\n- 向量存储与检索\n- 混合检索 + 重排序\n- 多轮对话\n- 引用溯源\n\"\"\"\n\nimport os\nfrom typing import List, Dict, Optional\nfrom dataclasses import dataclass\n\nfrom langchain_community.document_loaders import (\n PyPDFLoader, TextLoader, UnstructuredMarkdownLoader,\n WebBaseLoader, DirectoryLoader\n)\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\nfrom langchain_community.embeddings import OpenAIEmbeddings, HuggingFaceEmbeddings\nfrom langchain_community.vectorstores import Chroma\nfrom langchain_openai import ChatOpenAI\nfrom langchain_core.prompts import ChatPromptTemplate\nfrom langchain_core.output_parsers import StrOutputParser\nfrom langchain_core.runnables import RunnablePassthrough, RunnableParallel\nfrom langchain_core.messages import HumanMessage, AIMessage, SystemMessage\n\n\n# ═══════════════════════════════════════════════════\n# 第一步：配置\n# ═══════════════════════════════════════════════════\n\n@dataclass\nclass RAGAgentConfig:\n \"\"\"RAG Agent 配置\"\"\"\n # LLM 配置\n llm_model: str = \"gpt-4o\"\n llm_temperature: float = 0.1\n \n # Embedding 配置\n embedding_model: str = \"text-embedding-3-small\"\n use_local_embedding: bool = False\n local_embedding_model: str = \"BAAI/bge-large-zh-v1.5\"\n \n # 分块配置\n chunk_size: int = 1000\n chunk_overlap: int = 200\n \n # 检索配置\n retrieval_top_k: int = 5\n rerank_top_k: int = 3\n use_hybrid_search: bool = True\n \n # 向量数据库\n persist_directory: str = \"./chroma_db\"\n collection_name: str = \"knowledge_base\"\n\n\n# ═══════════════════════════════════════════════════\n# 第二步：文档索引\n# ═══════════════════════════════════════════════════\n\nclass RAGDocumentIndexer:\n \"\"\"文档索引器\"\"\"\n \n def __init__(self, config: RAGAgentConfig):\n self.config = config\n self._setup_components()\n \n def _setup_components(self):\n \"\"\"初始化组件\"\"\"\n # Embedding 模型\n if self.config.use_local_embedding:\n self.embeddings = HuggingFaceEmbeddings(\n model_name=self.config.local_embedding_model\n )\n else:\n self.embeddings = OpenAIEmbeddings(\n model=self.config.embedding_model\n )\n \n # 文本分割器\n self.text_splitter = RecursiveCharacterTextSplitter(\n chunk_size=self.config.chunk_size,\n chunk_overlap=self.config.chunk_overlap,\n separators=[\"\\n\\n\", \"\\n\", \"。\", \".\", \" \", \"\"],\n length_function=len\n )\n \n def index_directory(self, directory: str) -> Chroma:\n \"\"\"索引整个目录\"\"\"\n loader = DirectoryLoader(\n directory,\n glob=\"**/*.{pdf,md,txt,html}\",\n loader_cls={\n \".pdf\": PyPDFLoader,\n \".md\": UnstructuredMarkdownLoader,\n \".txt\": TextLoader,\n },\n silent_errors=True\n )\n \n documents = loader.load()\n print(f\"加载了 {len(documents)} 个文档\")\n \n # 分块\n splits = self.text_splitter.split_documents(documents)\n print(f\"分割为 {len(splits)} 个文档块\")\n \n # 构建向量索引\n vectorstore = Chroma.from_documents(\n documents=splits,\n embedding=self.embeddings,\n persist_directory=self.config.persist_directory,\n collection_name=self.config.collection_name\n )\n \n print(\"索引构建完成！\")\n return vectorstore\n \n def index_url(self, url: str) -> Chroma:\n \"\"\"索引网页\"\"\"\n loader = WebBaseLoader(url)\n documents = loader.load()\n \n splits = self.text_splitter.split_documents(documents)\n \n vectorstore = Chroma.from_documents(\n documents=splits,\n embedding=self.embeddings,\n persist_directory=self.config.persist_directory,\n collection_name=self.config.collection_name\n )\n \n return vectorstore\n\n\n# ═══════════════════════════════════════════════════\n# 第三步：RAG Agent 核心\n# ═══════════════════════════════════════════════════\n\nclass RAGAgent:\n \"\"\"RAG 增强 Agent\n \n 集成检索增强生成、多轮对话、引用溯源的完整 Agent\n \"\"\"\n \n def __init__(self, config: RAGAgentConfig, vectorstore: Chroma):\n self.config = config\n self.vectorstore = vectorstore\n self._setup_llm()\n self._setup_retriever()\n self._setup_chain()\n self._conversation_history: List[dict] = []\n \n def _setup_llm(self):\n \"\"\"初始化 LLM\"\"\"\n self.llm = ChatOpenAI(\n model=self.config.llm_model,\n temperature=self.config.llm_temperature\n )\n \n def _setup_retriever(self):\n \"\"\"初始化检索器\"\"\"\n self.retriever = self.vectorstore.as_retriever(\n search_type=\"mmr\", # 最大边际相关性，避免重复\n search_kwargs={\n \"k\": self.config.retrieval_top_k,\n \"fetch_k\": self.config.retrieval_top_k * 3,\n \"lambda_mult\": 0.7\n }\n )\n \n def _setup_chain(self):\n \"\"\"构建 RAG Chain\"\"\"\n \n # 系统提示\n system_prompt = \"\"\"你是一个专业的知识问答助手。请基于提供的参考资料来回答用户问题。\n\n规则：\n1. 只使用参考资料中的信息来回答\n2. 如果参考资料中没有相关信息，请明确告知用户\n3. 在回答中标注引用来源 [来源X]\n4. 回答应结构清晰、信息准确\n5. 对于不确定的信息，请说明不确定的原因\n\n参考资料：\n{context}\n\n对话历史：\n{history}\"\"\"\n\n self.prompt = ChatPromptTemplate.from_messages([\n (\"system\", system_prompt),\n (\"human\", \"{question}\")\n ])\n \n # 构建处理链\n def format_docs(docs):\n \"\"\"格式化检索到的文档\"\"\"\n formatted = []\n for i, doc in enumerate(docs, 1):\n source = doc.metadata.get(\"source\", \"未知来源\")\n page = doc.metadata.get(\"page\", \"\")\n page_info = f\" 第{page}页\" if page else \"\"\n formatted.append(\n f\"[来源{i}] {source}{page_info}\\n\"\n f\"{doc.page_content}\"\n )\n return \"\\n\\n---\\n\\n\".join(formatted)\n \n def format_history(history):\n \"\"\"格式化对话历史\"\"\"\n if not history:\n return \"（无对话历史）\"\n lines = []\n for msg in history[-6:]: # 保留最近 3 轮\n role = \"用户\" if msg[\"role\"] == \"user\" else \"助手\"\n lines.append(f\"{role}: {msg['content']}\")\n return \"\\n\".join(lines)\n \n # RAG Chain\n self.rag_chain = (\n {\n \"context\": self.retriever | format_docs,\n \"question\": RunnablePassthrough(),\n \"history\": RunnablePassthrough()\n }\n | self.prompt\n | self.llm\n | StrOutputParser()\n )\n \n def ask(self, question: str) -> dict:\n \"\"\"提问\n \n Returns:\n {\n \"answer\": str,\n \"sources\": list[dict],\n \"history\": list[dict]\n }\n \"\"\"\n # 格式化历史\n history_text = self._format_history()\n \n # 检索相关文档\n docs = self.retriever.invoke(question)\n \n # 生成回答\n answer = self.rag_chain.invoke(question)\n \n # 提取来源\n sources = []\n for i, doc in enumerate(docs[:self.config.rerank_top_k], 1):\n sources.append({\n \"index\": i,\n \"source\": doc.metadata.get(\"source\", \"未知\"),\n \"content_preview\": doc.page_content[:200] + \"...\",\n \"score\": doc.metadata.get(\"score\", 0)\n })\n \n # 更新对话历史\n self._conversation_history.append({\"role\": \"user\", \"content\": question})\n self._conversation_history.append({\"role\": \"assistant\", \"content\": answer})\n \n return {\n \"answer\": answer,\n \"sources\": sources,\n \"history\": self._conversation_history.copy()\n }\n \n def _format_history(self) -> str:\n if not self._conversation_history:\n return \"（无对话历史）\"\n lines = []\n for msg in self._conversation_history[-6:]:\n role = \"用户\" if msg[\"role\"] == \"user\" else \"助手\"\n lines.append(f\"{role}: {msg['content'][:200]}\")\n return \"\\n\".join(lines)\n \n def clear_history(self):\n \"\"\"清空对话历史\"\"\"\n self._conversation_history = []\n \n def conversation_mode(self):\n \"\"\"进入多轮对话模式\"\"\"\n print(\"=\" * 60)\n print(\"RAG Agent 多轮对话模式\")\n print(\"输入 'quit' 退出, 'clear' 清空历史, 'source' 查看来源\")\n print(\"=\" * 60)\n \n while True:\n question = input(\"\\n👤 你: \").strip()\n \n if question.lower() == \"quit\":\n break\n elif question.lower() == \"clear\":\n self.clear_history()\n print(\"✅ 对话历史已清空\")\n continue\n \n result = self.ask(question)\n \n print(f\"\\n🤖 助手: {result['answer']}\")\n \n if result['sources']:\n print(\"\\n📚 参考来源:\")\n for s in result['sources']:\n print(f\" [{s['index']}] {s['source']}\")\n\n\n# ═══════════════════════════════════════════════════\n# 第四步：运行\n# ═══════════════════════════════════════════════════\n\ndef main():\n \"\"\"主函数\"\"\"\n # 配置\n config = RAGAgentConfig(\n llm_model=\"gpt-4o\",\n embedding_model=\"text-embedding-3-small\",\n chunk_size=1000,\n chunk_overlap=200,\n retrieval_top_k=5,\n persist_directory=\"./chroma_db\",\n collection_name=\"knowledge_base\"\n )\n \n # 索引文档\n indexer = RAGDocumentIndexer(config)\n vectorstore = indexer.index_directory(\"./knowledge_docs\")\n \n # 创建 Agent\n agent = RAGAgent(config, vectorstore)\n \n # 对话\n agent.conversation_mode()\n\n\nif __name__ == \"__main__\":\n main()", "section_ref": "12.7.1", "runnable": true, "dependencies": [ "langchain_community", "langchain", "langchain_openai", "langchain_core" ] }, { "id": "code-22", "language": "text", "description": "", "code": "╔══════════════════════════════════════════════════════════════╗\n║ RAG 最佳实践清单 ║\n╠══════════════════════════════════════════════════════════════╣\n║ ║\n║ 📄 文档处理 ║\n║ ├── ✅ 清洗文档：移除页眉页脚水印噪音 ║\n║ ├── ✅ 保留元数据：来源、作者、日期、章节 ║\n║ ├── ✅ 使用递归分割，在自然边界处分块 ║\n║ ├── ✅ 分块大小 500-1500 字符（中文约 200-500 字） ║\n║ ├── ✅ overlap 设为 chunk_size 的 10-20% ║\n║ └── ✅ 考虑父子块策略，小块检索大块返回 ║\n║ ║\n║ 🔍 检索策略 ║\n║ ├── ✅ 混合检索（稠密 + BM25）优于单一检索 ║\n║ ├── ✅ 使用 MMR 替代纯相似度排序，增加多样性 ║\n║ ├── ✅ 重排序能显著提升 Top-K 精度 ║\n║ ├── ✅ 查询重写/扩展能改善召回率 ║\n║ └── ✅ metadata filtering 减少搜索空间 ║\n║ ║\n║ 🤖 生成与交互 ║\n║ ├── ✅ 明确告知 LLM \"只基于参考资料回答\" ║\n║ ├── ✅ 要求标注引用来源，可溯源 ║\n║ ├── ✅ 保留对话历史，支持多轮追问 ║\n║ ├── ✅ 设置 temperature=0.1，降低幻觉 ║\n║ └── ✅ 限制回答长度，聚焦核心信息 ║\n║ ║\n║ 🏗️ 架构与运维 ║\n║ ├── ✅ 实现增量索引，避免全量重建 ║\n║ ├── ✅ 监控检索延迟和 LLM Token 消耗 ║\n║ ├── ✅ 记录每次检索的分数，便于调试 ║\n║ ├── ✅ A/B 测试不同分块策略和检索参数 ║\n║ └── ✅ 定期评估检索质量（使用 RAGAS 等框架） ║\n║ ║\n╚══════════════════════════════════════════════════════════════╝", "section_ref": "12.8.1", "runnable": false, "dependencies": [] }, { "id": "code-23", "language": "python", "description": "用户原始查询往往不够精准，查询重写能显著提升检索效果：", "code": "class QueryRewriter:\n \"\"\"查询重写器 — 用 LLM 优化用户查询\"\"\"\n \n def __init__(self, llm):\n self.llm = llm\n \n def rewrite(self, original_query: str, conversation_history: list = None) -> str:\n \"\"\"重写查询，使其更适合检索\"\"\"\n history_context = \"\"\n if conversation_history:\n last_qa = conversation_history[-2:] if len(conversation_history) >= 2 else conversation_history\n history_context = \"\\n\".join(\n f\"{'用户' if m['role']=='user' else '助手'}: {m['content']}\"\n for m in last_qa\n )\n \n prompt = f\"\"\"请将用户的查询重写为一个更适合信息检索的查询。\n\n规则：\n1. 补全指代和省略（如果用户说\"它的价格\"，请替换为具体对象）\n2. 移除无关的寒暄词语\n3. 保持核心语义不变\n4. 添加可能有用的同义词或相关术语\n\n对话历史：\n{history_context if history_context else '（无）'}\n\n原始查询：{original_query}\n\n重写后的查询：\"\"\"\n \n response = self.llm.invoke(prompt)\n return response.content.strip()\n \n def expand(self, query: str, n_variations: int = 3) -> list[str]:\n \"\"\"生成查询变体，提高召回率\"\"\"\n prompt = f\"\"\"为以下查询生成 {n_variations} 个语义等价但表达不同的变体。\n\n原始查询：{query}\n\n变体列表（每行一个）：\"\"\"\n \n response = self.llm.invoke(prompt)\n variations = [line.strip() for line in response.content.strip().split(\"\\n\") if line.strip()]\n return variations[:n_variations]\n\n\nclass MultiQueryRetriever:\n \"\"\"多查询检索器 — 用多个变体查询提高召回\"\"\"\n \n def __init__(self, retriever, query_rewriter: QueryRewriter):\n self.retriever = retriever\n self.rewriter = query_rewriter\n \n def retrieve(self, query: str, top_k: int = 5) -> List[SearchResult]:\n \"\"\"使用多个查询变体检索\"\"\"\n # 生成查询变体\n variations = self.rewriter.expand(query, n_variations=3)\n all_queries = [query] + variations\n \n # 对每个变体执行检索\n all_results = {}\n for q in all_queries:\n results = self.retriever.retrieve(q, top_k=top_k * 2)\n for r in results:\n cid = r.chunk_id\n if cid not in all_results or r.score > all_results[cid].score:\n all_results[cid] = r\n \n # 去重并取 Top-K\n unique_results = list(all_results.values())\n unique_results.sort(key=lambda x: x.score, reverse=True)\n return unique_results[:top_k]", "section_ref": "12.8.3", "runnable": true, "dependencies": [] } ], "tables": [ { "headers": [ "特性", "传统 RAG", "RAG 增强 Agent" ], "data": [ [ "交互模式", "单轮问答", "多轮对话" ], [ "推理能力", "直接检索+回答", "规划→检索→分析→综合" ], [ "工具使用", "仅检索", "检索 + 计算 + API调用" ], [ "自我反思", "无", "可评估回答质量并重试" ], [ "多源融合", "单一知识库", "跨知识库、数据库、Web" ], [ "主动学习", "无", "可主动更新知识库" ] ] }, { "headers": [ "策略", "优点", "缺点", "适用场景" ], "data": [ [ "固定大小", "简单、可控", "切断语义、效率低", "日志文件、结构化数据" ], [ "递归分割", "保持语义边界、通用性强", "仍可能切断长段落", "通用文档" ], [ "语义分割", "最佳语义完整性", "计算成本高", "学术论文、技术文档" ], [ "按标题分割", "结构清晰", "依赖文档格式", "Markdown、HTML" ] ] }, { "headers": [ "模型", "维度", "中文支持", "特点" ], "data": [ [ "OpenAI text-embedding-3-small", "1536", "✅", "性价比高，API调用" ], [ "OpenAI text-embedding-3-large", "3072", "✅", "最高质量，成本较高" ], [ "BGE-large-zh-v1.5", "1024", "✅✅", "中文最优开源" ], [ "Cohere embed-v3", "1024", "✅", "多语言，内建重排" ], [ "GTE-Qwen2", "1536", "✅✅", "阿里开源，长文本支持好" ] ] }, { "headers": [ "场景", "alpha（稠密权重）", "说明" ], "data": [ [ "通用问答", "0.7", "语义理解更重要" ], [ "技术文档搜索", "0.5", "关键词匹配很重要" ], [ "产品/错误码查询", "0.3", "精确匹配是关键" ], [ "法律/合规文档", "0.6", "语义和精确都需要" ] ] }, { "headers": [ "陷阱", "症状", "解决方案" ], "data": [ [ "**分块太大**", "检索结果不精准，包含大量无关信息", "减小 chunk_size，使用语义分割" ], [ "**分块太小**", "缺乏上下文，回答不完整", "增大 chunk_size，使用父子块策略" ], [ "**只用稠密检索**", "产品编号、专有名词搜不到", "加入 BM25 混合检索" ], [ "**不设 overlap**", "关键信息正好被切断", "设置 10-20% 的 overlap" ], [ "**忽略元数据**", "无法按部门/日期/类型过滤", "始终保存和利用 metadata" ], [ "**幻觉问题**", "Agent 编造不存在的答案", "强化 Prompt 约束 + 引用溯源" ], [ "**全量重建索引**", "文档更新代价高昂", "实现增量更新机制" ], [ "**不考虑查询质量**", "用户查询模糊导致检索差", "加入查询重写/查询扩展" ] ] }, { "headers": [ "指标", "类型", "含义" ], "data": [ [ "**Faithfulness**", "忠实度", "回答是否基于检索到的上下文" ], [ "**Answer Relevancy**", "答案相关性", "回答与问题的相关程度" ], [ "**Context Precision**", "上下文精确率", "检索到的文档中相关文档的占比" ], [ "**Context Recall**", "上下文召回率", "相关文档被检索到的比例" ], [ "**Hit Rate**", "命中率", "正确答案出现在 Top-K 中的比例" ], [ "**MRR**", "平均倒数排名", "正确答案排名的倒数均值" ] ] } ], "key_takeaways": [], "common_pitfalls": [], "related_chapters": [ "ch04", "ch07" ] }