{ "metadata": { "id": "ch17", "title": "第17章：数据分析 Agent", "volume": "vol5", "volume_title": "专项篇", "word_count": 2466, "difficulty": "intermediate", "prerequisites": [ "ch06", "ch09" ], "key_concepts": [ "概述：从\"写 SQL\"到\"说人话\"", "数据分析 Agent 的能力边界", "数据分析 Agent 的技术栈", "自然语言 SQL 查询引擎", "Schema 感知的 Prompt 工程", "SQL 安全检查与验证", "NL-to-SQL Agent 完整实现", "数据清洗与预处理", "智能数据清洗框架", "统计分析与可视化", "智能统计分析引擎", "自动可视化生成", "报告自动生成", "结构化分析报告引擎", "异常检测系统" ], "learning_objectives": [], "estimated_tokens": 1480, "source_file": "vol5/ch17_数据分析Agent.md" }, "overview": "", "sections": [ { "id": "17.1", "title": "17.1 概述：从\"写 SQL\"到\"说人话\"", "level": 2, "content": "传统数据分析流程中，业务人员需要将问题翻译成 SQL、Python 脚本或 Excel 公式，再由数据工程师执行、验证、可视化。这个流程的核心瓶颈在于**意图翻译**——业务需求与数据实现之间的鸿沟。\n\n数据分析 Agent 的核心价值就是消除这道鸿沟。用户只需用自然语言描述需求，Agent 自动完成查询构建、数据获取、清洗转换、分析计算和可视化展示。", "subsections": [ { "id": "17.1.1", "title": "17.1.1 数据分析 Agent 的能力边界", "content": "- **自然语言 SQL 查询**：用户说\"上个月销售额最高的前10个城市\"，Agent 生成并执行 SQL\n- **数据清洗与预处理**：自动识别缺失值、异常值、重复数据，智能填充或剔除\n- **统计分析与可视化**：描述性统计、相关性分析、趋势分析，自动选择最佳图表类型\n- **报告自动生成**：将分析结果转化为结构化的 Markdown/PDF/HTML 报告\n- **异常检测**：基于统计规则和机器学习模型的实时异常识别" }, { "id": "17.1.2", "title": "17.1.2 数据分析 Agent 的技术栈", "content": "| 层次 | 技术组件 | 说明 |\n|------|---------|------|\n| 意图理解 | LLM + Schema 感知 Prompt | 将自然语言映射为数据库操作 |\n| 查询生成 | Text-to-SQL 引擎 | 支持单表/多表/嵌套查询 |\n| 数据处理 | Pandas / PySpark | 清洗、转换、聚合 |\n| 统计分析 | SciPy / Statsmodels | 假设检验、回归分析 |\n| 可视化 | Matplotlib / Plotly | 自动图表生成 |\n| 报告生成 | Jinja2 模板引擎 | 结构化文档输出 |" } ] }, { "id": "17.2", "title": "17.2 自然语言 SQL 查询引擎", "level": 2, "content": "", "subsections": [ { "id": "17.2.1", "title": "17.2.1 Schema 感知的 Prompt 工程", "content": "Text-to-SQL 的核心挑战在于让 LLM 理解数据库的表结构、字段含义和业务语义。下面我们从 Schema 定义开始，逐步构建完整的 Text-to-SQL 引擎。\n\n{{\n \"sql\": \"生成的 SQL 语句\",\n \"explanation\": \"查询逻辑说明\",\n \"complexity\": \"simple|aggregate|join|nested|window|complex\",\n \"tables_used\": [\"使用的表名\"],\n \"assumptions\": [\"做出的假设\"]\n}}" }, { "id": "17.2.2", "title": "17.2.2 SQL 安全检查与验证", "content": "在将 LLM 生成的 SQL 交给数据库执行之前，必须经过严格的安全检查。Text-to-SQL 场景尤其危险——LLM 可能生成包含 `DROP TABLE`、`DELETE` 等破坏性操作的语句。" }, { "id": "17.2.3", "title": "17.2.3 NL-to-SQL Agent 完整实现", "content": "" } ] }, { "id": "17.3", "title": "17.3 数据清洗与预处理", "level": 2, "content": "", "subsections": [ { "id": "17.3.1", "title": "17.3.1 智能数据清洗框架", "content": "数据清洗是数据分析中最耗时也最容易出错的环节。一个智能的数据清洗 Agent 需要能够自动检测数据质量问题、选择合适的处理策略、并记录每一步操作以便追溯。" } ] }, { "id": "17.4", "title": "17.4 统计分析与可视化", "level": 2, "content": "", "subsections": [ { "id": "17.4.1", "title": "17.4.1 智能统计分析引擎", "content": "统计分析引擎是数据分析 Agent 的大脑。它需要根据用户的数据特征和查询意图，自动选择最合适的分析方法。" }, { "id": "17.4.2", "title": "17.4.2 自动可视化生成", "content": "图表类型的选择往往需要丰富的数据可视化经验。自动可视化引擎通过分析数据特征（时间序列、类别对比、数值分布等）自动选择最佳图表类型。" } ] }, { "id": "17.5", "title": "17.5 报告自动生成", "level": 2, "content": "", "subsections": [ { "id": "17.5.1", "title": "17.5.1 结构化分析报告引擎", "content": "分析的最终目的是产出可供决策参考的报告。报告生成引擎将统计分析结果封装为结构化的 Markdown 或 HTML 文档，支持执行摘要、数据概览、分析章节和建议行动项。" } ] }, { "id": "17.6", "title": "17.6 异常检测系统", "level": 2, "content": "", "subsections": [ { "id": "17.6.1", "title": "17.6.1 多层异常检测框架", "content": "异常检测是数据分析的高阶能力。一个好的异常检测系统需要同时支持统计规则（Z-Score、IQR）和趋势分析，并能够自动判断异常严重性。" } ] }, { "id": "17.7", "title": "17.7 完整集成：数据分析 Agent", "level": 2, "content": "", "subsections": [ { "id": "17.7.1", "title": "17.7.1 统一入口", "content": "" }, { "id": "17.7.2", "title": "17.7.2 端到端示例", "content": "" } ] }, { "id": "17.8", "title": "17.8 生产级考量", "level": 2, "content": "", "subsections": [ { "id": "17.8.1", "title": "17.8.1 性能优化", "content": "| 优化点 | 策略 | 预期效果 |\n|--------|------|---------|\n| Schema 检索 | 向量相似度 + 关键词混合 | 减少 50% Prompt Token |\n| SQL 缓存 | 语义相似度匹配的查询缓存 | 重复查询 0 延迟 |\n| 数据处理 | 分块读取 + 增量计算 | 支持亿级数据 |\n| 可视化 | 采样渲染 + 懒加载 | 大数据集不卡顿 |" }, { "id": "17.8.2", "title": "17.8.2 安全与权限", "content": "- **SQL 白名单**：只允许 SELECT/WITH/EXPLAIN，禁止任何写操作\n- **行级权限**：根据用户角色过滤可访问的数据行\n- **查询超时**：设置最大执行时间，防止资源耗尽\n- **结果脱敏**：敏感字段（手机号、身份证等）自动脱敏" }, { "id": "17.8.3", "title": "17.8.3 评测指标", "content": "" } ] }, { "id": "本章小结", "title": "本章小结", "level": 2, "content": "本章从自然语言 SQL 查询出发，逐步构建了完整的数据分析 Agent 体系：\n\n1. **Schema 感知的 Text-to-SQL**：通过结构化 Prompt 让 LLM 理解数据库语义\n2. **智能数据清洗**：自动检测缺失值、异常值、重复数据，智能选择处理策略\n3. **统计分析引擎**：描述性统计、趋势分析、相关性分析，自动选择分析方法\n4. **自动可视化**：根据数据特征自动匹配最佳图表类型\n5. **报告自动生成**：将分析结果转化为结构化的 Markdown/HTML 报告\n6. **异常检测系统**：Z-Score、IQR、趋势突变三重检测机制\n\n核心思想：**将数据分析从\"技能\"降维为\"对话\"**。用户不需要会 SQL、Python 或统计学，只需要会用自然语言提问。", "subsections": [] } ], "code_blocks": [ { "id": "code-1", "language": "text", "description": "数据分析 Agent 的核心价值就是消除这道鸿沟。用户只需用自然语言描述需求，Agent 自动完成查询构建、数据获取、清洗转换、分析计算和可视化展示。", "code": "自然语言查询 → 数据清洗 → 统计分析 → 可视化 → 自动报告 → 异常检测\n │ │ │ │ │ │\n Text-to-SQL 自动化 描述性/ 图表自动 结构化 规则+统计\n 语义理解 缺失处理 推断性分析 生成 文档输出 双重检测", "section_ref": "17.1.1", "runnable": false, "dependencies": [] }, { "id": "code-2", "language": "python", "description": "Text-to-SQL 的核心挑战在于让 LLM 理解数据库的表结构、字段含义和业务语义。下面我们从 Schema 定义开始，逐步构建完整的 Text-to-SQL 引擎。", "code": "\"\"\"\n数据分析 Agent - 自然语言 SQL 查询引擎\n支持多表关联、聚合函数、时间窗口查询\n\"\"\"\n\nfrom typing import Optional\nfrom dataclasses import dataclass, field\nfrom enum import Enum\nimport json\nimport re\n\n\nclass QueryComplexity(Enum):\n SIMPLE = \"simple\" # 单表简单查询\n AGGREGATE = \"aggregate\" # 聚合查询\n JOIN = \"join\" # 多表关联\n NESTED = \"nested\" # 嵌套子查询\n WINDOW = \"window\" # 窗口函数\n COMPLEX = \"complex\" # 复合查询\n\n\n@dataclass\nclass ColumnSchema:\n \"\"\"数据库列元信息\"\"\"\n name: str\n data_type: str\n description: str\n is_primary_key: bool = False\n is_foreign_key: bool = False\n foreign_table: Optional[str] = None\n examples: list = field(default_factory=list)\n nullable: bool = True\n\n\n@dataclass\nclass TableSchema:\n \"\"\"数据库表元信息\"\"\"\n name: str\n description: str\n columns: list = field(default_factory=list)\n row_count: int = 0\n relationships: list = field(default_factory=list)\n\n def to_prompt_text(self) -> str:\n \"\"\"生成用于 Prompt 的表结构描述\"\"\"\n lines = [f\"表名: {self.name} ({self.description})\"]\n lines.append(f\"行数: {self.row_count}\")\n lines.append(\"字段:\")\n for col in self.columns:\n pk_mark = \" [PK]\" if col.is_primary_key else \"\"\n fk_mark = f\" [FK -> {col.foreign_table}]\" if col.is_foreign_key else \"\"\n null_mark = \" (可空)\" if col.nullable else \"\"\n lines.append(\n f\" - {col.name} ({col.data_type}){pk_mark}{fk_mark}{null_mark}: \"\n f\"{col.description}\"\n )\n if col.examples:\n lines.append(f\" 示例值: {', '.join(str(e) for e in col.examples[:3])}\")\n return \"\\n\".join(lines)\n\n\n@dataclass\nclass DatabaseSchema:\n \"\"\"数据库完整 Schema\"\"\"\n tables: dict = field(default_factory=dict)\n\n def add_table(self, table: TableSchema):\n self.tables[table.name] = table\n\n def get_relevant_tables(self, query: str) -> list:\n \"\"\"根据查询意图找到相关表（简化版，生产环境可用向量检索）\"\"\"\n relevant = []\n query_lower = query.lower()\n for table in self.tables.values():\n keywords = f\"{table.name} {table.description} \".lower()\n for col in table.columns:\n keywords += f\"{col.name} {col.description} \"\n overlap = sum(1 for w in query_lower.split() if w in keywords)\n if overlap > 0:\n relevant.append((table, overlap))\n relevant.sort(key=lambda x: x[1], reverse=True)\n return [t for t, _ in relevant]\n\n def build_schema_prompt(self, query: str) -> str:\n \"\"\"构建 Schema 感知的 Prompt\"\"\"\n relevant_tables = self.get_relevant_tables(query)\n schema_text = \"\\n\\n\".join(t.to_prompt_text() for t in relevant_tables)\n\n return f\"\"\"你是一个专业的 SQL 分析师。根据用户的自然语言问题，生成准确的 SQL 查询。\n\n## 数据库 Schema\n\n{schema_text}\n\n## 重要规则\n\n1. 只使用上述表和字段，不要编造不存在的表或列\n2. 使用标准 SQL 语法（兼容 PostgreSQL / MySQL）\n3. 处理时间范围查询时，注意时区问题\n4. 聚合查询必须包含 GROUP BY\n5. 注意 NULL 值处理\n\n## 用户问题\n\n{query}\n\n## 请生成 SQL 查询\n\n请以 JSON 格式返回:", "section_ref": "17.2.1", "runnable": true, "dependencies": [] }, { "id": "code-3", "language": "\"\"\"", "description": "}}", "code": "\n\n# ==================== 示例：构建电商数据库 Schema ====================\n\ndef build_ecommerce_schema() -> DatabaseSchema:\n \"\"\"构建电商数据库 Schema 示例\"\"\"\n db = DatabaseSchema()\n\n users = TableSchema(\n name=\"users\",\n description=\"用户信息表\",\n row_count=1_200_000,\n columns=[\n ColumnSchema(\"user_id\", \"BIGINT\", \"用户唯一标识\", is_primary_key=True),\n ColumnSchema(\"username\", \"VARCHAR(50)\", \"用户名\", examples=[\"张三\", \"shopper_01\"]),\n ColumnSchema(\"email\", \"VARCHAR(100)\", \"邮箱地址\", examples=[\"user@example.com\"]),\n ColumnSchema(\"city\", \"VARCHAR(50)\", \"所在城市\", examples=[\"北京\", \"上海\", \"深圳\"]),\n ColumnSchema(\"province\", \"VARCHAR(30)\", \"所在省份\"),\n ColumnSchema(\"register_date\", \"DATE\", \"注册日期\"),\n ColumnSchema(\"vip_level\", \"INT\", \"VIP 等级 1-5\", examples=[1, 3, 5]),\n ]\n )\n\n products = TableSchema(\n name=\"products\",\n description=\"商品信息表\",\n row_count=85_000,\n columns=[\n ColumnSchema(\"product_id\", \"BIGINT\", \"商品唯一标识\", is_primary_key=True),\n ColumnSchema(\"name\", \"VARCHAR(200)\", \"商品名称\", examples=[\"iPhone 15\", \"机械键盘\"]),\n ColumnSchema(\"category\", \"VARCHAR(50)\", \"商品类目\", examples=[\"电子产品\", \"服装\"]),\n ColumnSchema(\"sub_category\", \"VARCHAR(50)\", \"子类目\"),\n ColumnSchema(\"price\", \"DECIMAL(10,2)\", \"售价\"),\n ColumnSchema(\"cost_price\", \"DECIMAL(10,2)\", \"成本价\"),\n ColumnSchema(\"stock\", \"INT\", \"库存数量\"),\n ColumnSchema(\"brand\", \"VARCHAR(50)\", \"品牌\"),\n ColumnSchema(\"status\", \"VARCHAR(20)\", \"上架状态\", examples=[\"on_sale\"]),\n ]\n )\n\n orders = TableSchema(\n name=\"orders\",\n description=\"订单主表\",\n row_count=5_600_000,\n columns=[\n ColumnSchema(\"order_id\", \"BIGINT\", \"订单唯一标识\", is_primary_key=True),\n ColumnSchema(\"user_id\", \"BIGINT\", \"用户 ID\", is_foreign_key=True, foreign_table=\"users\"),\n ColumnSchema(\"order_date\", \"TIMESTAMP\", \"下单时间\"),\n ColumnSchema(\"total_amount\", \"DECIMAL(12,2)\", \"订单总金额\"),\n ColumnSchema(\"discount_amount\", \"DECIMAL(10,2)\", \"优惠金额\"),\n ColumnSchema(\"final_amount\", \"DECIMAL(12,2)\", \"实付金额\"),\n ColumnSchema(\"status\", \"VARCHAR(20)\", \"订单状态\",\n examples=[\"pending\", \"paid\", \"shipped\", \"completed\"]),\n ColumnSchema(\"shipping_city\", \"VARCHAR(50)\", \"收货城市\"),\n ColumnSchema(\"payment_method\", \"VARCHAR(30)\", \"支付方式\",\n examples=[\"alipay\", \"wechat\", \"credit_card\"]),\n ]\n )\n\n order_items = TableSchema(\n name=\"order_items\",\n description=\"订单明细表\",\n row_count=15_200_000,\n columns=[\n ColumnSchema(\"item_id\", \"BIGINT\", \"明细唯一标识\", is_primary_key=True),\n ColumnSchema(\"order_id\", \"BIGINT\", \"订单 ID\", is_foreign_key=True, foreign_table=\"orders\"),\n ColumnSchema(\"product_id\", \"BIGINT\", \"商品 ID\", is_foreign_key=True, foreign_table=\"products\"),\n ColumnSchema(\"quantity\", \"INT\", \"购买数量\"),\n ColumnSchema(\"unit_price\", \"DECIMAL(10,2)\", \"成交单价\"),\n ColumnSchema(\"subtotal\", \"DECIMAL(12,2)\", \"小计金额\"),\n ]\n )\n\n for table in [users, products, orders, order_items]:\n db.add_table(table)\n return db", "section_ref": "17.2.1", "runnable": false, "dependencies": [] }, { "id": "code-4", "language": "python", "description": "在将 LLM 生成的 SQL 交给数据库执行之前，必须经过严格的安全检查。Text-to-SQL 场景尤其危险——LLM 可能生成包含 DROP TABLE、DELETE 等破坏性操作的语句。", "code": "\"\"\"\nSQL 安全检查器 - 防止危险操作\n\"\"\"\n\nclass SQLSafetyChecker:\n \"\"\"SQL 安全检查器\"\"\"\n\n FORBIDDEN_KEYWORDS = [\n \"DROP\", \"DELETE\", \"TRUNCATE\", \"ALTER\", \"GRANT\", \"REVOKE\",\n \"CREATE\", \"INSERT\", \"UPDATE\", \"REPLACE\", \"EXEC\", \"EXECUTE\"\n ]\n ALLOWED_PREFIXES = [\"SELECT\", \"WITH\", \"EXPLAIN\"]\n\n @classmethod\n def check(cls, sql: str) -> tuple[bool, str]:\n \"\"\"检查 SQL 安全性，返回 (is_safe, reason)\"\"\"\n sql_upper = sql.strip().upper()\n if not any(sql_upper.startswith(p) for p in cls.ALLOWED_PREFIXES):\n return False, f\"查询必须以 {cls.ALLOWED_PREFIXES} 开头\"\n for keyword in cls.FORBIDDEN_KEYWORDS:\n if re.search(rf'\\b{keyword}\\b', sql_upper):\n return False, f\"禁止使用 {keyword} 操作\"\n if re.search(r'\\bCALL\\b|\\bEXEC\\b', sql_upper):\n return False, \"禁止调用存储过程\"\n return True, \"安全检查通过\"\n\n\nclass SQLValidator:\n \"\"\"SQL 语法验证器\"\"\"\n\n @staticmethod\n def validate_basic(sql: str) -> list[str]:\n \"\"\"基础语法验证，返回错误列表\"\"\"\n errors = []\n sql_clean = sql.strip().rstrip(\";\")\n\n # 括号匹配\n if sql_clean.count(\"(\") != sql_clean.count(\")\"):\n errors.append(\"括号不匹配\")\n\n # 引号匹配\n single_quotes = sql_clean.count(\"'\") - sql_clean.count(\"\\\\'\")\n if single_quotes % 2 != 0:\n errors.append(\"单引号不匹配\")\n\n # GROUP BY 一致性检查\n aggregate_funcs = re.findall(\n r'\\b(SUM|AVG|COUNT|MIN|MAX|STDDEV)\\s*\\(', sql_clean, re.IGNORECASE)\n has_group_by = bool(re.search(r'\\bGROUP\\s+BY\\b', sql_clean, re.IGNORECASE))\n\n if aggregate_funcs and not has_group_by:\n select_part = re.search(r'SELECT\\s+(.*?)\\s+FROM', sql_clean, re.IGNORECASE)\n if select_part:\n cols = [c.strip() for c in select_part.group(1).split(\",\")]\n non_agg = [c for c in cols\n if not re.search(r'(SUM|AVG|COUNT|MIN|MAX)\\s*\\(', c, re.IGNORECASE)\n and c.strip() != \"*\"]\n if non_agg:\n errors.append(f\"聚合查询存在非聚合字段 {non_agg}，请添加 GROUP BY\")\n return errors", "section_ref": "17.2.2", "runnable": true, "dependencies": [] }, { "id": "code-5", "language": "python", "description": "", "code": "\"\"\"\nNLToSQL Agent - 将自然语言转换为可执行的 SQL\n\"\"\"\n\nimport logging\nfrom datetime import datetime\n\nlogger = logging.getLogger(__name__)\n\n\nclass NLToSQLAgent:\n \"\"\"自然语言转 SQL 的 Agent\"\"\"\n\n def __init__(self, db_schema: DatabaseSchema, llm_client=None):\n self.db_schema = db_schema\n self.llm_client = llm_client\n self.safety_checker = SQLSafetyChecker()\n self.validator = SQLValidator()\n self.query_history: list[dict] = []\n\n def _call_llm(self, prompt: str) -> str:\n \"\"\"调用 LLM（生产环境替换为实际 API）\"\"\"\n if self.llm_client:\n return self.llm_client.chat(prompt)\n # 模拟返回 - 实际使用时替换为 API 调用\n return json.dumps({\n \"sql\": \"SELECT city, COUNT(*) as order_count, \"\n \"SUM(final_amount) as total_sales \"\n \"FROM orders WHERE order_date >= '2025-03-01' \"\n \"GROUP BY city ORDER BY total_sales DESC LIMIT 10\",\n \"explanation\": \"查询上个月销售额最高的前10个城市\",\n \"complexity\": \"aggregate\",\n \"tables_used\": [\"orders\"],\n \"assumptions\": [\"上个月指2025年3月\", \"使用 final_amount 计算销售额\"]\n }, ensure_ascii=False)\n\n def generate_query(self, user_query: str) -> dict:\n \"\"\"生成 SQL 查询的完整流程\"\"\"\n # 1. 构建 Schema 感知 Prompt\n prompt = self.db_schema.build_schema_prompt(user_query)\n\n # 2. 调用 LLM 生成 SQL\n response = self._call_llm(prompt)\n\n # 3. 解析 JSON 响应\n try:\n json_match = re.search(r'```json\\s*(.*?)\\s*```', response, re.DOTALL)\n result = json.loads(json_match.group(1)) if json_match else json.loads(response)\n except json.JSONDecodeError as e:\n return {\"success\": False, \"error\": f\"LLM 返回格式错误: {e}\"}\n\n # 4. 安全检查\n is_safe, reason = self.safety_checker.check(result[\"sql\"])\n if not is_safe:\n return {\"success\": False, \"error\": f\"安全检查未通过: {reason}\"}\n\n # 5. 语法验证\n errors = self.validator.validate_basic(result[\"sql\"])\n if errors:\n return {\"success\": False, \"error\": f\"语法验证失败: {'; '.join(errors)}\"}\n\n # 6. 记录查询历史\n self.query_history.append({\n \"user_query\": user_query,\n \"generated_sql\": result[\"sql\"],\n \"explanation\": result.get(\"explanation\", \"\"),\n \"complexity\": result.get(\"complexity\", \"unknown\"),\n \"timestamp\": datetime.now().isoformat()\n })\n\n return {\n \"success\": True,\n \"sql\": result[\"sql\"],\n \"explanation\": result.get(\"explanation\", \"\"),\n \"complexity\": result.get(\"complexity\", \"simple\"),\n \"tables_used\": result.get(\"tables_used\", []),\n \"assumptions\": result.get(\"assumptions\", [])\n }\n\n def explain_results(self, sql: str, results: list[dict], user_query: str) -> str:\n \"\"\"用自然语言解释查询结果\"\"\"\n prompt = f\"\"\"你是数据分析师。用简洁中文解释以下查询结果。\n\n用户问题: {user_query}\nSQL: {sql}\n结果（前20条）: {json.dumps(results[:20], ensure_ascii=False, default=str)}\n\n请用 2-3 句话总结关键发现。\"\"\"\n return self._call_llm(prompt)", "section_ref": "17.2.3", "runnable": true, "dependencies": [] }, { "id": "code-6", "language": "python", "description": "数据清洗是数据分析中最耗时也最容易出错的环节。一个智能的数据清洗 Agent 需要能够自动检测数据质量问题、选择合适的处理策略、并记录每一步操作以便追溯。", "code": "\"\"\"\n智能数据清洗 Agent - 自动检测并处理数据质量问题\n\"\"\"\n\nimport pandas as pd\nimport numpy as np\nfrom typing import Optional\nfrom dataclasses import dataclass, field\n\n\n@dataclass\nclass DataQualityReport:\n \"\"\"数据质量报告\"\"\"\n total_rows: int = 0\n total_columns: int = 0\n missing_values: dict = field(default_factory=dict)\n missing_percentage: dict = field(default_factory=dict)\n duplicate_rows: int = 0\n duplicate_percentage: float = 0.0\n data_types: dict = field(default_factory=dict)\n outliers: dict = field(default_factory=dict)\n quality_score: float = 0.0\n\n def to_markdown(self) -> str:\n \"\"\"生成 Markdown 格式的质量报告\"\"\"\n lines = [\n \"## 数据质量报告\\n\",\n f\"- **总行数**: {self.total_rows:,}\",\n f\"- **总列数**: {self.total_columns}\",\n f\"- **重复行数**: {self.duplicate_rows:,} ({self.duplicate_percentage:.1f}%)\",\n f\"- **质量评分**: {self.quality_score:.1f}/100\\n\",\n \"### 缺失值统计\\n\",\n \"| 列名 | 缺失数量 | 缺失比例 | 建议处理方式 |\",\n \"|------|---------|---------|-------------|\"\n ]\n for col, pct in self.missing_percentage.items():\n if pct > 0:\n suggestion = (\"填充中位数\" if pct < 0.1 else\n \"填充众数\" if pct < 0.3 else \"删除或单独分析\")\n lines.append(\n f\"| {col} | {self.missing_values[col]:,} \"\n f\"| {pct:.1%} | {suggestion} |\")\n\n if self.outliers:\n lines.append(\"\\n### 异常值检测\\n\")\n for col, info in self.outliers.items():\n lines.append(\n f\"- **{col}**: {info['count']} 个异常值 \"\n f\"(方法: {info['method']})\")\n return \"\\n\".join(lines)\n\n\nclass DataCleaner:\n \"\"\"智能数据清洗器 - 支持自动策略选择\"\"\"\n\n def __init__(self, df: pd.DataFrame):\n self.df = df.copy()\n self.report = DataQualityReport(\n total_rows=len(df), total_columns=len(df.columns))\n self.cleaning_log: list[str] = []\n\n def analyze(self) -> DataQualityReport:\n \"\"\"分析数据质量，生成完整报告\"\"\"\n df = self.df\n self.report.data_types = {col: str(dtype) for col, dtype in df.dtypes.items()}\n\n # 缺失值分析\n for col in df.columns:\n missing = df[col].isna().sum()\n self.report.missing_values[col] = missing\n self.report.missing_percentage[col] = missing / len(df) if len(df) > 0 else 0\n\n # 重复行分析\n self.report.duplicate_rows = df.duplicated().sum()\n self.report.duplicate_percentage = (\n self.report.duplicate_rows / len(df) if len(df) > 0 else 0)\n\n # 异常值分析（IQR 方法）\n for col in df.select_dtypes(include=[np.number]).columns:\n Q1, Q3 = df[col].quantile(0.25), df[col].quantile(0.75)\n IQR = Q3 - Q1\n outlier_count = ((df[col] < Q1 - 1.5 * IQR) | (df[col] > Q3 + 1.5 * IQR)).sum()\n if outlier_count > 0:\n self.report.outliers[col] = {\n \"count\": int(outlier_count), \"method\": \"IQR (1.5倍)\",\n \"lower_bound\": float(Q1 - 1.5 * IQR),\n \"upper_bound\": float(Q3 + 1.5 * IQR)\n }\n\n # 质量评分计算\n missing_penalty = sum(min(p * 100, 20) for p in self.report.missing_percentage.values())\n dup_penalty = min(self.report.duplicate_percentage * 50, 15)\n outlier_penalty = min(\n sum(v[\"count\"] / max(self.report.total_rows, 1)\n for v in self.report.outliers.values()) * 100, 15)\n self.report.quality_score = max(0, 100 - missing_penalty - dup_penalty - outlier_penalty)\n return self.report\n\n def handle_missing(self, strategy: str = \"auto\") -> pd.DataFrame:\n \"\"\"处理缺失值 - 支持 auto/median/mode/ffill/drop_rows/drop_column\"\"\"\n for col in self.df.columns:\n missing_pct = self.df[col].isna().sum() / len(self.df)\n if missing_pct == 0:\n continue\n\n s = (strategy if strategy != \"auto\"\n else self._auto_strategy(self.df[col], missing_pct))\n\n if s == \"drop_column\" and missing_pct > 0.5:\n self.df = self.df.drop(columns=[col])\n self.cleaning_log.append(f\"删除列 '{col}': 缺失率 {missing_pct:.1%} 过高\")\n elif s == \"drop_rows\":\n before = len(self.df)\n self.df = self.df.dropna(subset=[col])\n self.cleaning_log.append(f\"删除 '{col}' 缺失行: {before - len(self.df)} 行\")\n elif s == \"fill_median\" and pd.api.types.is_numeric_dtype(self.df[col]):\n val = self.df[col].median()\n self.df[col] = self.df[col].fillna(val)\n self.cleaning_log.append(f\"填充 '{col}' 缺失值: 中位数 = {val}\")\n elif s == \"fill_mode\":\n val = (self.df[col].mode().iloc[0]\n if not self.df[col].mode().empty else \"未知\")\n self.df[col] = self.df[col].fillna(val)\n self.cleaning_log.append(f\"填充 '{col}' 缺失值: 众数 = {val}\")\n elif s == \"fill_ffill\":\n self.df[col] = self.df[col].ffill().bfill()\n self.cleaning_log.append(f\"前向填充 '{col}' 缺失值\")\n return self.df\n\n def _auto_strategy(self, series: pd.Series, missing_pct: float) -> str:\n \"\"\"根据数据特征自动选择填充策略\"\"\"\n if missing_pct > 0.5:\n return \"drop_column\"\n if pd.api.types.is_numeric_dtype(series):\n return \"fill_median\"\n if pd.api.types.is_datetime64_any_dtype(series):\n return \"fill_ffill\"\n return \"fill_mode\"\n\n def remove_duplicates(self, subset: Optional[list] = None) -> pd.DataFrame:\n \"\"\"删除重复行\"\"\"\n before = len(self.df)\n self.df = self.df.drop_duplicates(subset=subset)\n removed = before - len(self.df)\n if removed > 0:\n self.cleaning_log.append(f\"删除重复行: {removed} 行\")\n return self.df\n\n def handle_outliers(self, method: str = \"clip\",\n columns: Optional[list] = None) -> pd.DataFrame:\n \"\"\"处理异常值 - clip（截断）或 remove（移除）\"\"\"\n numeric_cols = self.df.select_dtypes(include=[np.number]).columns\n if columns:\n numeric_cols = [c for c in numeric_cols if c in columns]\n\n for col in numeric_cols:\n Q1, Q3 = self.df[col].quantile(0.25), self.df[col].quantile(0.75)\n IQR = Q3 - Q1\n lower, upper = Q1 - 1.5 * IQR, Q3 + 1.5 * IQR\n mask = (self.df[col] < lower) | (self.df[col] > upper)\n count = mask.sum()\n if count == 0:\n continue\n if method == \"clip\":\n self.df[col] = self.df[col].clip(lower, upper)\n self.cleaning_log.append(\n f\"截断 '{col}' 异常值: {count} 个值限制在 [{lower:.2f}, {upper:.2f}]\")\n elif method == \"remove\":\n self.df = self.df[~mask]\n self.cleaning_log.append(f\"移除 '{col}' 异常行: {count} 行\")\n return self.df\n\n def get_cleaning_summary(self) -> str:\n \"\"\"获取清洗摘要日志\"\"\"\n if not self.cleaning_log:\n return \"无需清洗操作。\"\n lines = [\"## 数据清洗日志\\n\"]\n for i, log in enumerate(self.cleaning_log, 1):\n lines.append(f\"{i}. {log}\")\n if self.report.total_rows != len(self.df):\n lines.append(\n f\"\\n**数据量变化**: {self.report.total_rows:,} → {len(self.df):,}\")\n return \"\\n\".join(lines)", "section_ref": "17.3.1", "runnable": true, "dependencies": [ "pandas", "numpy" ] }, { "id": "code-7", "language": "python", "description": "统计分析引擎是数据分析 Agent 的大脑。它需要根据用户的数据特征和查询意图，自动选择最合适的分析方法。", "code": "\"\"\"\n统计分析引擎 - 自动选择分析方法和可视化图表\n\"\"\"\n\nfrom dataclasses import dataclass\nfrom enum import Enum\n\n\nclass AnalysisType(Enum):\n DESCRIPTIVE = \"descriptive\"\n TREND = \"trend\"\n COMPARISON = \"comparison\"\n CORRELATION = \"correlation\"\n DISTRIBUTION = \"distribution\"\n\n\n@dataclass\nclass AnalysisResult:\n \"\"\"分析结果统一封装\"\"\"\n analysis_type: AnalysisType\n chart_type: str\n chart_config: dict\n summary: str\n key_findings: list[str]\n statistics: dict\n\n\nclass StatisticalAnalyzer:\n \"\"\"统计分析器\"\"\"\n\n def __init__(self, df: pd.DataFrame):\n self.df = df\n\n def descriptive_analysis(self, columns: Optional[list] = None) -> AnalysisResult:\n \"\"\"描述性统计分析 - 均值、中位数、标准差、偏度等\"\"\"\n numeric_cols = self.df.select_dtypes(include=[np.number]).columns\n if columns:\n numeric_cols = [c for c in numeric_cols if c in columns]\n\n stats = {}\n for col in numeric_cols:\n series = self.df[col].dropna()\n stats[col] = {\n \"mean\": float(series.mean()),\n \"median\": float(series.median()),\n \"std\": float(series.std()),\n \"min\": float(series.min()),\n \"max\": float(series.max()),\n \"skewness\": float(series.skew()),\n \"cv\": float(series.std() / series.mean()) if series.mean() != 0 else None\n }\n\n key_findings = []\n for col, s in stats.items():\n if s[\"cv\"] and s[\"cv\"] > 1:\n key_findings.append(f\"'{col}' 数据离散度极高 (CV={s['cv']:.2f})\")\n elif s[\"cv\"] and s[\"cv\"] < 0.1:\n key_findings.append(f\"'{col}' 数据分布非常均匀 (CV={s['cv']:.2f})\")\n if s[\"skewness\"] > 1:\n key_findings.append(f\"'{col}' 呈右偏分布 (偏度={s['skewness']:.2f})\")\n elif s[\"skewness\"] < -1:\n key_findings.append(f\"'{col}' 呈左偏分布 (偏度={s['skewness']:.2f})\")\n\n return AnalysisResult(\n analysis_type=AnalysisType.DESCRIPTIVE, chart_type=\"box\",\n chart_config={\"type\": \"box\", \"title\": \"数值字段分布概览\"},\n summary=f\"对 {len(numeric_cols)} 个数值字段进行了描述性统计分析。\",\n key_findings=key_findings or [\"各字段分布较为正常\"],\n statistics=stats\n )\n\n def trend_analysis(self, date_col: str, value_col: str,\n freq: str = \"D\") -> AnalysisResult:\n \"\"\"趋势分析 - 按时间聚合并检测变化趋势\"\"\"\n df = self.df.copy()\n df[date_col] = pd.to_datetime(df[date_col])\n\n if freq == \"D\":\n grouped = df.groupby(df[date_col].dt.date)[value_col].agg([\"sum\", \"mean\"])\n elif freq == \"M\":\n grouped = df.groupby(df[date_col].dt.to_period(\"M\"))[value_col].agg([\"sum\", \"mean\"])\n else:\n grouped = df.groupby(df[date_col].dt.date)[value_col].agg([\"sum\", \"mean\"])\n\n grouped = grouped.reset_index()\n grouped.columns = [\"period\", \"total\", \"average\"]\n\n if len(grouped) >= 2:\n first = grouped[\"total\"].iloc[:len(grouped)//2].mean()\n second = grouped[\"total\"].iloc[len(grouped)//2:].mean()\n change_pct = (second - first) / first * 100 if first != 0 else 0\n trend = (\"呈上升趋势 (+{:.1f}%)\".format(change_pct) if change_pct > 10 else\n \"呈下降趋势 ({:.1f}%)\".format(change_pct) if change_pct < -10 else\n \"基本平稳 ({:+.1f}%)\".format(change_pct))\n else:\n trend = \"数据不足\"\n\n peak_idx = grouped[\"total\"].idxmax()\n valley_idx = grouped[\"total\"].idxmin()\n\n return AnalysisResult(\n analysis_type=AnalysisType.TREND, chart_type=\"line\",\n chart_config={\"type\": \"line\", \"title\": f\"{value_col} 趋势\"},\n summary=f\"{value_col} 在分析周期内{trend}。\",\n key_findings=[\n trend,\n f\"峰值: {grouped.loc[peak_idx, 'period']} \"\n f\"= {grouped.loc[peak_idx, 'total']:,.0f}\",\n f\"谷值: {grouped.loc[valley_idx, 'period']} \"\n f\"= {grouped.loc[valley_idx, 'total']:,.0f}\"\n ],\n statistics={\"trend\": trend, \"change_pct\": change_pct}\n )\n\n def correlation_analysis(self, columns: Optional[list] = None) -> AnalysisResult:\n \"\"\"相关性分析 - 找出强相关特征对\"\"\"\n numeric_cols = self.df.select_dtypes(include=[np.number]).columns\n if columns:\n numeric_cols = [c for c in numeric_cols if c in columns]\n corr_matrix = self.df[numeric_cols].corr()\n\n strong = []\n cols_list = list(numeric_cols)\n for i in range(len(cols_list)):\n for j in range(i + 1, len(cols_list)):\n val = corr_matrix.iloc[i, j]\n if abs(val) > 0.7:\n strong.append({\n \"col1\": cols_list[i], \"col2\": cols_list[j],\n \"correlation\": round(val, 3),\n \"type\": \"强正相关\" if val > 0 else \"强负相关\"\n })\n strong.sort(key=lambda x: abs(x[\"correlation\"]), reverse=True)\n\n findings = [f\"发现 {len(strong)} 对强相关特征 (|r| > 0.7)\"]\n for c in strong[:5]:\n findings.append(f\" {c['col1']} ↔ {c['col2']}: r={c['correlation']:.3f}\")\n\n return AnalysisResult(\n analysis_type=AnalysisType.CORRELATION, chart_type=\"heatmap\",\n chart_config={\"type\": \"heatmap\", \"title\": \"特征相关性热力图\"},\n summary=f\"分析了 {len(numeric_cols)} 个数值特征的相关性。\",\n key_findings=findings or [\"未发现强相关特征\"],\n statistics={\"strong_count\": len(strong)}\n )", "section_ref": "17.4.1", "runnable": true, "dependencies": [] }, { "id": "code-8", "language": "python", "description": "图表类型的选择往往需要丰富的数据可视化经验。自动可视化引擎通过分析数据特征（时间序列、类别对比、数值分布等）自动选择最佳图表类型。", "code": "\"\"\"\n自动可视化引擎 - 根据数据特征自动选择最佳图表类型\n\"\"\"\n\nimport matplotlib\nmatplotlib.use(\"Agg\") # 非交互式后端\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport re\nimport os\nimport tempfile\nfrom typing import Optional\n\n# 中文字体设置\nplt.rcParams['font.sans-serif'] = ['Arial Unicode MS', 'SimHei', 'DejaVu Sans']\nplt.rcParams['axes.unicode_minus'] = False\n\n\nclass AutoVisualizer:\n \"\"\"自动可视化引擎\"\"\"\n\n # 图表类型决策矩阵\n CHART_DECISION = {\n (\"numeric\", \"time\"): \"line\",\n (\"numeric\", \"category\"): \"bar\",\n (\"category\", \"category\"): \"bar\",\n (\"numeric\", \"numeric\"): \"scatter\",\n (\"distribution\", None): \"histogram\",\n (\"proportion\", None): \"pie\",\n }\n\n def __init__(self, style: str = \"seaborn-v0_8-whitegrid\"):\n self.style = style\n\n def _infer_type(self, data) -> str:\n \"\"\"推断数据类型: time / numeric / category\"\"\"\n sample = data[:min(20, len(data))]\n time_n = sum(1 for v in sample\n if isinstance(v, (np.datetime64, pd.Timestamp))\n or (isinstance(v, str) and re.match(r'\\d{4}[-/]\\d{2}[-/]\\d{2}', v)))\n if time_n > len(sample) * 0.5:\n return \"time\"\n num_n = sum(1 for v in sample if isinstance(v, (int, float, np.number)))\n return \"numeric\" if num_n > len(sample) * 0.7 else \"category\"\n\n def _best_chart(self, x_data, y_data, intent: Optional[str] = None) -> str:\n \"\"\"自动选择图表类型\"\"\"\n if intent:\n return intent\n xt = self._infer_type(x_data)\n yt = self._infer_type(y_data) if y_data else None\n return self.CHART_DECISION.get((xt, yt), \"bar\")\n\n def plot(self, x_data: list, y_data: Optional[list] = None,\n title: str = \"\", x_label: str = \"\", y_label: str = \"\",\n chart_type: Optional[str] = None, figsize: tuple = (10, 6),\n save_path: Optional[str] = None) -> str:\n \"\"\"生成图表并保存，返回保存路径\"\"\"\n ct = chart_type or self._best_chart(x_data, y_data)\n plt.style.use(self.style)\n fig, ax = plt.subplots(figsize=figsize)\n\n if ct == \"line\" and y_data is not None:\n ax.plot(x_data, y_data, linewidth=2, color=\"#4A90D9\")\n ax.fill_between(range(len(x_data)), y_data, alpha=0.1, color=\"#4A90D9\")\n step = max(1, len(x_data) // 10)\n ax.set_xticks(range(0, len(x_data), step))\n ax.set_xticklabels([str(x_data[i]) for i in range(0, len(x_data), step)],\n rotation=45, ha='right')\n\n elif ct == \"bar\" and y_data is not None:\n colors = plt.cm.Set2(np.linspace(0, 1, len(x_data)))\n bars = ax.bar(range(len(x_data)), y_data, color=colors, edgecolor='white')\n ax.set_xticks(range(len(x_data)))\n ax.set_xticklabels([str(x) for x in x_data], rotation=45, ha='right')\n for bar, val in zip(bars, y_data):\n ax.text(bar.get_x() + bar.get_width() / 2, bar.get_height(),\n f'{val:,.0f}', ha='center', va='bottom', fontsize=8)\n\n elif ct == \"scatter\" and y_data is not None:\n ax.scatter(x_data, y_data, alpha=0.5, s=30, c=\"#4A90D9\")\n if len(x_data) > 2:\n z = np.polyfit([float(x) for x in x_data], y_data, 1)\n xr = np.linspace(min(float(x) for x in x_data),\n max(float(x) for x in x_data), 100)\n ax.plot(xr, np.poly1d(z)(xr), \"--\", color=\"red\", alpha=0.7)\n\n elif ct == \"pie\" and y_data is not None:\n if len(x_data) > 8:\n x_data, y_data = x_data[:8] + [\"其他\"], y_data[:8] + [sum(y_data[8:])]\n ax.pie(y_data, labels=x_data, autopct='%1.1f%%',\n colors=plt.cm.Set3(np.linspace(0, 1, len(x_data))), startangle=90)\n\n elif ct == \"histogram\":\n bins = min(30, max(10, len(set(x_data)) // 10))\n ax.hist(x_data, bins=bins, color=\"#4A90D9\", edgecolor=\"white\", alpha=0.8)\n\n ax.set_title(title, fontsize=14, fontweight='bold', pad=15)\n ax.set_xlabel(x_label, fontsize=11)\n ax.set_ylabel(y_label, fontsize=11)\n ax.grid(True, alpha=0.3)\n plt.tight_layout()\n\n if save_path is None:\n save_path = os.path.join(tempfile.gettempdir(),\n f\"chart_{abs(hash(title)) % 10000}.png\")\n fig.savefig(save_path, dpi=150, bbox_inches='tight', facecolor='white')\n plt.close(fig)\n return save_path", "section_ref": "17.4.2", "runnable": true, "dependencies": [ "matplotlib", "numpy", "tempfile" ] }, { "id": "code-9", "language": "python", "description": "分析的最终目的是产出可供决策参考的报告。报告生成引擎将统计分析结果封装为结构化的 Markdown 或 HTML 文档，支持执行摘要、数据概览、分析章节和建议行动项。", "code": "\"\"\"\n分析报告自动生成引擎\n将数据分析结果转化为结构化的 Markdown/HTML 报告\n\"\"\"\n\nfrom datetime import datetime\nfrom dataclasses import dataclass, field\nfrom typing import Optional\n\n\n@dataclass\nclass ReportSection:\n \"\"\"报告章节\"\"\"\n title: str\n content: str\n chart_paths: list = field(default_factory=list)\n level: int = 2\n\n\nclass AnalysisReportGenerator:\n \"\"\"分析报告生成器\"\"\"\n\n def __init__(self, title: str, author: str = \"数据分析 Agent\"):\n self.title = title\n self.author = author\n self.sections: list[ReportSection] = []\n self.generated_at = datetime.now().strftime(\"%Y-%m-%d %H:%M:%S\")\n\n def add_executive_summary(self, data_source: str, analysis_period: str,\n key_findings: list[str], quality_score: float):\n \"\"\"添加执行摘要\"\"\"\n content = (f\"本报告基于 **{data_source}**，\"\n f\"对 **{analysis_period}** 的数据进行了全面分析。\\n\\n\"\n f\"**数据质量评分**: {quality_score:.1f}/100\\n\\n\"\n \"### 关键发现\\n\\n\")\n for f in key_findings:\n content += f\"- {f}\\n\"\n self.sections.append(ReportSection(title=\"执行摘要\", content=content))\n\n def add_data_overview(self, total_rows: int, total_columns: int,\n column_types: dict, quality_md: str):\n \"\"\"添加数据概览\"\"\"\n content = (f\"### 基本信息\\n\"\n f\"- 数据量: **{total_rows:,}** 行\\n\"\n f\"- 字段数: **{total_columns}** 列\\n\\n\"\n \"### 字段类型分布\\n\\n| 类型 | 字段数 |\\n|------|--------|\\n\")\n groups = {}\n for col, dtype in column_types.items():\n groups.setdefault(dtype, []).append(col)\n for dtype, cols in groups.items():\n content += f\"| {dtype} | {len(cols)} |\\n\"\n content += f\"\\n{quality_md}\"\n self.sections.append(ReportSection(title=\"数据概览\", content=content))\n\n def add_analysis_section(self, title: str, result: AnalysisResult,\n chart_path: Optional[str] = None):\n \"\"\"添加分析章节\"\"\"\n content = f\"### {result.summary}\\n\\n### 关键发现\\n\\n\"\n for f in result.key_findings:\n content += f\"- {f}\\n\"\n if result.statistics:\n content += \"\\n### 统计指标\\n\\n\"\n for k, v in result.statistics.items():\n if isinstance(v, dict) and \"mean\" in v:\n content += (f\"| 指标 | {k} |\\n|------|------|\\n\"\n f\"| 均值 | {v['mean']:.2f} |\\n\"\n f\"| 中位数 | {v['median']:.2f} |\\n\"\n f\"| 标准差 | {v['std']:.2f} |\\n\\n\")\n self.sections.append(ReportSection(\n title=title, content=content,\n chart_paths=[chart_path] if chart_path else []))\n\n def add_recommendations(self, recommendations: list[str]):\n \"\"\"添加建议章节\"\"\"\n content = \"\"\n for i, rec in enumerate(recommendations, 1):\n content += f\"### 建议 {i}\\n\\n{rec}\\n\\n\"\n self.sections.append(ReportSection(title=\"建议与行动项\", content=content))\n\n def to_markdown(self) -> str:\n \"\"\"生成 Markdown 格式报告\"\"\"\n lines = [f\"# {self.title}\\n\",\n f\"> 生成时间: {self.generated_at} | 生成工具: 数据分析 Agent\\n\",\n \"---\\n\"]\n for section in self.sections:\n prefix = \"#\" * (section.level + 1)\n lines.append(f\"\\n{prefix} {section.title}\\n{section.content}\")\n for p in section.chart_paths:\n lines.append(f\"\\n![{section.title}]({p})\\n\")\n lines.append(\"\\n---\\n*本报告由数据分析 Agent 自动生成*\\n\")\n return \"\\n\".join(lines)\n\n def to_html(self) -> str:\n \"\"\"生成 HTML 格式报告\"\"\"\n md = self.to_markdown()\n html = md\n for pat, repl in [\n (r'^# (.*$)', r'

\\1

'),\n (r'^## (.*$)', r'

\\1

'),\n (r'^### (.*$)', r'

\\1

'),\n (r'\\*\\*(.*?)\\*\\*', r'\\1'),\n (r'- (.*$)', r'
  • \\1
    • '),\n (r'^> (.*)$', r'
    \\1
    '),\n (r'^---$', r'
    '),\n ]:\n html = re.sub(pat, repl, html, flags=re.MULTILINE)\n\n return f\"\"\"\n\n\n \n {self.title}\n \n\n{html}\"\"\"", "section_ref": "17.5.1", "runnable": true, "dependencies": [] }, { "id": "code-10", "language": "python", "description": "异常检测是数据分析的高阶能力。一个好的异常检测系统需要同时支持统计规则（Z-Score、IQR）和趋势分析，并能够自动判断异常严重性。", "code": "\"\"\"\n异常检测系统 - 结合统计规则与趋势分析\n\"\"\"\n\nfrom dataclasses import dataclass\nfrom typing import Optional\nimport numpy as np\nfrom datetime import datetime\n\n\n@dataclass\nclass Anomaly:\n \"\"\"异常记录\"\"\"\n timestamp: str\n metric_name: str\n value: float\n expected_range: tuple\n severity: str # \"low\" / \"medium\" / \"high\" / \"critical\"\n description: str\n suggestion: str = \"\"\n\n\nclass AnomalyDetector:\n \"\"\"多层异常检测器\"\"\"\n\n def __init__(self, sensitivity: float = 2.0):\n \"\"\"\n Args:\n sensitivity: Z-Score 灵敏度阈值，值越小越敏感\n \"\"\"\n self.sensitivity = sensitivity\n self.baseline: dict[str, dict] = {}\n\n def fit_baseline(self, data: dict[str, list[float]]):\n \"\"\"从历史数据建立基线统计\"\"\"\n for metric, values in data.items():\n arr = np.array(values, dtype=float)\n self.baseline[metric] = {\n \"mean\": float(np.mean(arr)),\n \"std\": float(np.std(arr)) or 1e-9,\n \"median\": float(np.median(arr)),\n \"q1\": float(np.percentile(arr, 25)),\n \"q3\": float(np.percentile(arr, 75)),\n \"iqr\": float(np.percentile(arr, 75) - np.percentile(arr, 25)) or 1e-9,\n \"p5\": float(np.percentile(arr, 5)),\n \"p95\": float(np.percentile(arr, 95)),\n }\n\n def detect_zscore(self, metric: str, value: float) -> Optional[Anomaly]:\n \"\"\"Z-Score 异常检测\"\"\"\n if metric not in self.baseline:\n return None\n bl = self.baseline[metric]\n z = abs(value - bl[\"mean\"]) / bl[\"std\"]\n\n if z > self.sensitivity:\n sev = (\"critical\" if z > 4 else \"high\" if z > 3 else\n \"medium\" if z > 2.5 else \"low\")\n return Anomaly(\n timestamp=datetime.now().isoformat(), metric_name=metric,\n value=value, expected_range=(\n bl[\"mean\"] - self.sensitivity * bl[\"std\"],\n bl[\"mean\"] + self.sensitivity * bl[\"std\"]),\n severity=sev,\n description=f\"{metric} 值 {value:.2f} 偏离基线 {z:.1f} 个标准差\",\n suggestion=\"建议检查数据源和采集流程，确认是否为真实异常\"\n )\n return None\n\n def detect_iqr(self, metric: str, value: float) -> Optional[Anomaly]:\n \"\"\"IQR 异常检测（对非正态分布更鲁棒）\"\"\"\n if metric not in self.baseline:\n return None\n bl = self.baseline[metric]\n lower, upper = bl[\"q1\"] - 1.5 * bl[\"iqr\"], bl[\"q3\"] + 1.5 * bl[\"iqr\"]\n\n if value < lower or value > upper:\n sev = \"high\" if value < bl[\"p5\"] or value > bl[\"p95\"] else \"medium\"\n return Anomaly(\n timestamp=datetime.now().isoformat(), metric_name=metric,\n value=value, expected_range=(lower, upper),\n severity=sev,\n description=f\"{metric} 值 {value:.2f} 超出 IQR 范围\",\n suggestion=\"该值超出正常四分位范围，建议人工确认\"\n )\n return None\n\n def detect_trend_change(self, metric: str, recent: list[float],\n window: int = 7) -> Optional[Anomaly]:\n \"\"\"趋势突变检测 - 对比前后窗口的均值变化\"\"\"\n if len(recent) < window * 2:\n return None\n\n old_avg = np.mean(recent[-window*2:-window])\n new_avg = np.mean(recent[-window:])\n if old_avg == 0:\n return None\n\n ratio = (new_avg - old_avg) / abs(old_avg)\n if abs(ratio) > 0.3:\n direction = \"上升\" if ratio > 0 else \"下降\"\n sev = (\"critical\" if abs(ratio) > 0.5 else\n \"high\" if abs(ratio) > 0.4 else \"medium\")\n return Anomaly(\n timestamp=datetime.now().isoformat(), metric_name=metric,\n value=new_avg, expected_range=(old_avg * 0.7, old_avg * 1.3),\n severity=sev,\n description=f\"{metric} 近期均值{direction} {abs(ratio):.1%}\",\n suggestion=\"指标出现显著趋势变化，建议排查原因\"\n )\n return None\n\n def batch_detect(self, metric: str, values: list[float]) -> list[Anomaly]:\n \"\"\"批量检测 - 综合使用多种检测方法\"\"\"\n anomalies = []\n for i, value in enumerate(values):\n a = self.detect_zscore(metric, value)\n if not a:\n a = self.detect_iqr(metric, value)\n if a:\n anomalies.append(a)\n if i >= 13:\n ta = self.detect_trend_change(metric, values[:i+1])\n if ta:\n anomalies.append(ta)\n return anomalies\n\n def generate_report(self, anomalies: list[Anomaly]) -> str:\n \"\"\"生成异常检测报告\"\"\"\n if not anomalies:\n return \"## 异常检测报告\\n\\n未发现异常。\\n\"\n\n lines = [\"## 异常检测报告\\n\", f\"**检测到 {len(anomalies)} 个异常点**\\n\",\n \"| 严重性 | 数量 |\\n|--------|------|\"]\n counts = {}\n for a in anomalies:\n counts[a.severity] = counts.get(a.severity, 0) + 1\n for s in [\"critical\", \"high\", \"medium\", \"low\"]:\n if s in counts:\n lines.append(f\"| {s} | {counts[s]} |\")\n\n lines.append(\"\\n### 异常详情\\n\")\n for i, a in enumerate(anomalies, 1):\n lines.append(f\"#### 异常 {i}: {a.metric_name}\")\n lines.append(f\"- **严重性**: {a.severity} | **值**: {a.value:.2f}\")\n lines.append(f\"- **描述**: {a.description}\")\n lines.append(f\"- **建议**: {a.suggestion}\\n\")\n return \"\\n\".join(lines)", "section_ref": "17.6.1", "runnable": true, "dependencies": [ "numpy" ] }, { "id": "code-11", "language": "python", "description": "", "code": "\"\"\"\n数据分析 Agent - 完整集成版\n将 SQL 引擎、数据清洗、统计分析、可视化和报告生成整合为一体\n\"\"\"\n\nclass DataAnalysisAgent:\n \"\"\"\n 数据分析 Agent 核心入口\n\n 使用示例:\n db = build_ecommerce_schema()\n agent = DataAnalysisAgent(db_schema=db)\n result = agent.analyze(\"分析上个月各城市的销售趋势\")\n \"\"\"\n\n def __init__(self, db_schema: Optional[DatabaseSchema] = None, llm_client=None):\n self.sql_agent = NLToSQLAgent(db_schema or DatabaseSchema(), llm_client)\n\n def analyze_from_file(self, csv_path: str, query: str) -> dict:\n \"\"\"文件分析模式 - 读取 CSV 并生成完整报告\"\"\"\n df = pd.read_csv(csv_path)\n\n # 数据质量分析\n cleaner = DataCleaner(df)\n quality = cleaner.analyze()\n\n # 数据清洗\n cleaner.handle_missing(strategy=\"auto\")\n cleaner.remove_duplicates()\n cleaner.handle_outliers(method=\"clip\")\n\n # 统计分析\n analyzer = StatisticalAnalyzer(cleaner.df)\n desc = analyzer.descriptive_analysis()\n corr = analyzer.correlation_analysis()\n\n # 趋势分析（自动检测日期列）\n trend = None\n for col in cleaner.df.columns:\n try:\n pd.to_datetime(cleaner.df[col])\n nums = cleaner.df.select_dtypes(include=[np.number]).columns\n if len(nums) > 0:\n trend = analyzer.trend_analysis(col, nums[0])\n break\n except (ValueError, TypeError):\n continue\n\n # 生成报告\n gen = AnalysisReportGenerator(title=f\"分析报告: {query}\")\n findings = desc.key_findings[:5] + (trend.key_findings if trend else []) + corr.key_findings[:3]\n gen.add_executive_summary(csv_path, \"全量数据\", findings[:8], quality.quality_score)\n gen.add_data_overview(len(df), len(df.columns), quality.data_types, quality.to_markdown())\n gen.add_analysis_section(\"描述性统计\", desc)\n if trend:\n gen.add_analysis_section(\"趋势分析\", trend)\n gen.add_analysis_section(\"相关性分析\", corr)\n gen.add_recommendations([\n \"定期监控数据质量，重点关注缺失率较高的字段\",\n \"对强相关特征进行因果分析以发现业务洞察\",\n \"引入更多外部特征以丰富分析维度\"\n ])\n\n return {\n \"query\": query,\n \"quality_score\": quality.quality_score,\n \"cleaning_log\": cleaner.cleaning_log,\n \"report_markdown\": gen.to_markdown(),\n \"report_html\": gen.to_html()\n }\n\n def analyze_from_db(self, query: str, executor=None) -> dict:\n \"\"\"数据库查询模式 - NL-to-SQL + 结果分析\"\"\"\n # 生成 SQL\n sql_result = self.sql_agent.generate_query(query)\n if not sql_result[\"success\"]:\n return {\"success\": False, \"error\": sql_result[\"error\"]}\n\n # 执行 SQL（需要外部传入 executor）\n if executor:\n try:\n columns, rows = executor(sql_result[\"sql\"])\n df = pd.DataFrame(rows, columns=columns)\n except Exception as e:\n return {\"success\": False, \"error\": f\"SQL 执行失败: {e}\"}\n else:\n return {\"success\": False, \"error\": \"未提供数据库执行器\"}\n\n # 对查询结果进行分析\n analyzer = StatisticalAnalyzer(df)\n desc = analyzer.descriptive_analysis()\n\n return {\n \"success\": True,\n \"sql\": sql_result[\"sql\"],\n \"explanation\": sql_result[\"explanation\"],\n \"statistics\": desc.statistics,\n \"findings\": desc.key_findings\n }", "section_ref": "17.7.1", "runnable": true, "dependencies": [] }, { "id": "code-12", "language": "python", "description": "", "code": "\"\"\"\n端到端使用示例\n\"\"\"\n\n# 示例 1: 文件分析\nif __name__ == \"__main__\":\n # 创建模拟数据\n import pandas as pd\n import numpy as np\n\n np.random.seed(42)\n dates = pd.date_range(\"2025-01-01\", periods=90, freq=\"D\")\n mock_df = pd.DataFrame({\n \"date\": dates,\n \"city\": np.random.choice([\"北京\", \"上海\", \"深圳\", \"广州\", \"杭州\"], 90),\n \"sales\": np.random.lognormal(10, 0.5, 90),\n \"orders\": np.random.randint(50, 500, 90),\n \"customers\": np.random.randint(20, 200, 90),\n \"refund_rate\": np.random.uniform(0, 0.1, 90)\n })\n\n # 保存临时 CSV\n csv_path = \"/tmp/mock_sales.csv\"\n mock_df.to_csv(csv_path, index=False)\n\n # 创建 Agent 并分析\n agent = DataAnalysisAgent()\n result = agent.analyze_from_file(csv_path, \"分析各城市的销售趋势和异常情况\")\n\n print(\"=== 质量评分 ===\")\n print(f\"{result['quality_score']:.1f}/100\")\n\n print(\"\\n=== 清洗日志 ===\")\n for log in result['cleaning_log']:\n print(f\" {log}\")\n\n print(\"\\n=== 报告预览（前500字）===\")\n print(result['report_markdown'][:500] + \"...\")\n\n # 示例 2: 数据库查询\n db = build_ecommerce_schema()\n sql_agent = NLToSQLAgent(db)\n\n queries = [\n \"上个月销售额最高的前10个城市\",\n \"各商品类目的订单数量和平均客单价\",\n \"VIP 等级与消费金额的关系\"\n ]\n\n for q in queries:\n print(f\"\\n### 问题: {q}\")\n r = sql_agent.generate_query(q)\n if r[\"success\"]:\n print(f\"SQL: {r['sql']}\")\n print(f\"说明: {r['explanation']}\")\n else:\n print(f\"错误: {r['error']}\")", "section_ref": "17.7.2", "runnable": true, "dependencies": [] }, { "id": "code-13", "language": "text", "description": "- 结果脱敏：敏感字段（手机号、身份证等）自动脱敏", "code": "准确率 = 生成的正确 SQL 数 / 总查询数\n召回率 = 成功处理的查询数 / 总查询数\n延迟 = P95 查询响应时间\n用户满意度 = 用户确认结果准确的比例", "section_ref": "17.8.3", "runnable": false, "dependencies": [] } ], "tables": [ { "headers": [ "层次", "技术组件", "说明" ], "data": [ [ "意图理解", "LLM + Schema 感知 Prompt", "将自然语言映射为数据库操作" ], [ "查询生成", "Text-to-SQL 引擎", "支持单表/多表/嵌套查询" ], [ "数据处理", "Pandas / PySpark", "清洗、转换、聚合" ], [ "统计分析", "SciPy / Statsmodels", "假设检验、回归分析" ], [ "可视化", "Matplotlib / Plotly", "自动图表生成" ], [ "报告生成", "Jinja2 模板引擎", "结构化文档输出" ] ] }, { "headers": [ "优化点", "策略", "预期效果" ], "data": [ [ "Schema 检索", "向量相似度 + 关键词混合", "减少 50% Prompt Token" ], [ "SQL 缓存", "语义相似度匹配的查询缓存", "重复查询 0 延迟" ], [ "数据处理", "分块读取 + 增量计算", "支持亿级数据" ], [ "可视化", "采样渲染 + 懒加载", "大数据集不卡顿" ] ] } ], "key_takeaways": [], "common_pitfalls": [], "related_chapters": [ "ch06", "ch09", "ch28" ] }