{ "metadata": { "id": "ch14", "title": "第14章：多模态Agent", "volume": "vol4", "volume_title": "高级篇", "word_count": 1810, "difficulty": "intermediate", "prerequisites": [ "ch04" ], "key_concepts": [ "多模态AI概述", "什么是多模态", "多模态Agent的核心价值", "视觉Agent", "图像理解", "OCR与文档解析", "图表分析", "语音Agent", "语音识别（ASR）", "语音合成（TTS）", "实时语音对话", "多模态融合策略", "融合策略分类", "Agent中的实用融合方案", "多模态工具链" ], "learning_objectives": [], "estimated_tokens": 1086, "source_file": "vol4/ch14_多模态Agent.md" }, "overview": "现实世界的信息不只有文本——图像、语音、视频、文档构成了人类感知的完整图景。多模态 Agent（Multimodal Agent）能够像人类一样，同时理解和处理多种类型的信息输入，并生成跨模态的输出。本章将深入讲解多模态 Agent 的架构设计、实现技术和优化策略，涵盖视觉、语音、文档处理等核心模态，以及它们在 Agent 系统中的融合应用。", "sections": [ { "id": "14.1", "title": "14.1 多模态AI概述", "level": 2, "content": "", "subsections": [ { "id": "14.1.1", "title": "14.1.1 什么是多模态", "content": "多模态（Multimodal）指系统同时处理两种或两种以上信息模态（Modality）的能力。常见的模态包括：\n\n\n| 模态 | 输入能力 | 输出能力 | 典型模型 |\n|------|---------|---------|---------|\n| 文本 | ✅ | ✅ | GPT-4, Claude, Gemini |\n| 图像 | ✅ | ✅ | GPT-4V, Gemini Pro Vision |\n| 音频 | ✅ | ✅ | Whisper, TTS |\n| 视频 | ✅（逐帧） | ⚠️（有限） | Gemini, Video-LLaMA |\n| 文档 | ✅ | ✅ | Claude, GPT-4V |\n| 代码 | ✅ | ✅ | GPT-4, Claude |" }, { "id": "14.1.2", "title": "14.1.2 多模态Agent的核心价值", "content": "" } ] }, { "id": "14.2", "title": "14.2 视觉Agent", "level": 2, "content": "", "subsections": [ { "id": "14.2.1", "title": "14.2.1 图像理解", "content": "" }, { "id": "14.2.2", "title": "14.2.2 OCR与文档解析", "content": "" }, { "id": "14.2.3", "title": "14.2.3 图表分析", "content": "" } ] }, { "id": "14.3", "title": "14.3 语音Agent", "level": 2, "content": "", "subsections": [ { "id": "14.3.1", "title": "14.3.1 语音识别（ASR）", "content": "" }, { "id": "14.3.2", "title": "14.3.2 语音合成（TTS）", "content": "" }, { "id": "14.3.3", "title": "14.3.3 实时语音对话", "content": "" } ] }, { "id": "14.4", "title": "14.4 多模态融合策略", "level": 2, "content": "", "subsections": [ { "id": "14.4.1", "title": "14.4.1 融合策略分类", "content": "| 策略 | 原理 | 优势 | 劣势 |\n|------|------|------|------|\n| 早期融合 | 将原始多模态数据拼接后送入模型 | 模态间交互充分 | 计算量大，需要重新训练 |\n| 晚期融合 | 各模态独立处理，合并结果 | 实现简单，模块化 | 模态间交互有限 |\n| 交叉注意力 | 模态间通过注意力机制交互 | 平衡效果与效率 | 架构复杂 |" }, { "id": "14.4.2", "title": "14.4.2 Agent中的实用融合方案", "content": "" } ] }, { "id": "14.5", "title": "14.5 多模态工具链", "level": 2, "content": "", "subsections": [ { "id": "14.5.1", "title": "14.5.1 文件处理工具链", "content": "" } ] }, { "id": "14.6", "title": "14.6 多模态Agent架构设计", "level": 2, "content": "", "subsections": [ { "id": "14.6.1", "title": "14.6.1 完整架构", "content": "" }, { "id": "14.6.2", "title": "14.6.2 输出合成", "content": "" } ] }, { "id": "14.7", "title": "14.7 实战：构建多模态文档分析Agent", "level": 2, "content": "", "subsections": [] }, { "id": "14.8", "title": "14.8 性能优化与成本控制", "level": 2, "content": "", "subsections": [ { "id": "14.8.1", "title": "14.8.1 模型选择策略", "content": "" }, { "id": "14.8.2", "title": "14.8.2 缓存与去重", "content": "" } ] }, { "id": "最佳实践", "title": "最佳实践", "level": 2, "content": "1. **模态路由优化**：在处理前先判断输入模态，避免对文本输入调用视觉模型\n2. **渐进式处理**：先用低成本模型快速预判，再决定是否需要高成本精细分析\n3. **结果缓存**：对相同文件的分析结果进行缓存，避免重复调用昂贵的视觉模型\n4. **错误降级**：视觉模型失败时降级为纯文本处理，保证基本可用\n5. **批量处理**：多张图片可以合并为一次API调用，降低延迟和成本", "subsections": [] }, { "id": "常见陷阱", "title": "常见陷阱", "level": 2, "content": "1. **图片过大**：超过模型限制的图片会导致API错误。先压缩到合理尺寸\n2. **忽略音频质量**：低质量音频会导致STT结果很差，影响后续处理\n3. **过度依赖单一模态**：只看图不看文字，或只看文字不看图，导致信息遗漏\n4. **成本失控**：多模态API调用成本远高于纯文本。必须设置预算和缓存\n5. **时序信息丢失**：将视频拆成图片处理时，丢失了帧间时序信息", "subsections": [] }, { "id": "小结", "title": "小结", "level": 2, "content": "多模态 Agent 将 AI 的能力从文本世界扩展到了感知层面。通过视觉、语音、文档等多种模态的融合处理，Agent 能够理解和操作更丰富的信息。本章介绍了各模态的处理技术、融合策略和架构设计。多模态是 Agent 技术发展的重要方向，随着模型能力的持续提升，未来的 Agent 将越来越接近人类的感知能力。", "subsections": [] }, { "id": "延伸阅读", "title": "延伸阅读", "level": 2, "content": "1. **论文**: \"GPT-4 Technical Report\" — 多模态能力的里程碑\n2. **论文**: \"LLaVA: Visual Instruction Tuning\" — 开源多模态模型\n3. **Whisper文档**: https://github.com/openai/whisper\n4. **PyMuPDF文档**: https://pymupdf.readthedocs.io/\n5. **OpenAI Vision指南**: https://platform.openai.com/docs/guides/vision", "subsections": [] } ], "code_blocks": [ { "id": "code-1", "language": "mermaid", "description": "多模态（Multimodal）指系统同时处理两种或两种以上信息模态（Modality）的能力。常见的模态包括：", "code": "graph TB\n A[多模态AI] --> B[文本 Text]\n A --> C[图像 Image]\n A --> D[音频 Audio]\n A --> E[视频 Video]\n A --> F[文档 Document]\n A --> G[结构化数据]\n A --> H[3D/传感器]\n \n B --> I[理解 + 生成]\n C --> I\n D --> I\n E --> I\n F --> I", "section_ref": "14.1.1", "runnable": false, "dependencies": [] }, { "id": "code-2", "language": "python", "description": "| 代码 | ✅ | ✅ | GPT-4, Claude |", "code": "class MultimodalAgentUseCases:\n \"\"\"多模态Agent的典型应用场景\"\"\"\n \n use_cases = {\n \"文档智能\": {\n \"input\": [\"PDF\", \"图片\", \"扫描件\"],\n \"output\": [\"摘要\", \"问答\", \"数据提取\", \"分类\"],\n \"example\": \"上传合同PDF，Agent提取关键条款并生成摘要\"\n },\n \"视觉质检\": {\n \"input\": [\"产品照片\", \"质检标准\"],\n \"output\": [\"缺陷检测\", \"质检报告\", \"改进建议\"],\n \"example\": \"Agent分析生产线照片，自动识别产品缺陷\"\n },\n \"内容创作\": {\n \"input\": [\"文字描述\", \"参考图片\"],\n \"output\": [\"图片生成\", \"视频剪辑\", \"文案\"],\n \"example\": \"根据产品描述和品牌指南生成营销素材\"\n },\n \"无障碍服务\": {\n \"input\": [\"图片\", \"环境音\"],\n \"output\": [\"语音描述\", \"文字转述\"],\n \"example\": \"为视障用户描述图片内容\"\n },\n \"数据分析\": {\n \"input\": [\"图表截图\", \"数据表\"],\n \"output\": [\"数据解读\", \"趋势分析\", \"报告\"],\n \"example\": \"上传销售图表，Agent自动解读趋势并给出建议\"\n }\n }", "section_ref": "14.1.2", "runnable": true, "dependencies": [] }, { "id": "code-3", "language": "python", "description": "", "code": "import base64\nfrom openai import AsyncOpenAI\n\nclass VisionAgent:\n \"\"\"视觉理解Agent\"\"\"\n \n def __init__(self, api_key: str):\n self.client = AsyncOpenAI(api_key=api_key)\n \n def _encode_image(self, image_path: str) -> str:\n \"\"\"将图片编码为base64\"\"\"\n with open(image_path, \"rb\") as f:\n return base64.standard_b64encode(f.read()).decode()\n \n async def analyze_image(self, image_path: str, \n prompt: str = \"描述这张图片\") -> str:\n \"\"\"分析单张图片\"\"\"\n base64_image = self._encode_image(image_path)\n \n response = await self.client.chat.completions.create(\n model=\"gpt-4o\",\n messages=[{\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"text\", \"text\": prompt},\n {\n \"type\": \"image_url\",\n \"image_url\": {\n \"url\": f\"data:image/jpeg;base64,{base64_image}\"\n }\n }\n ]\n }]\n )\n return response.choices[0].message.content\n \n async def compare_images(self, image_paths: list[str],\n comparison_prompt: str) -> str:\n \"\"\"对比多张图片\"\"\"\n content = [{\"type\": \"text\", \"text\": comparison_prompt}]\n \n for path in image_paths:\n base64_image = self._encode_image(path)\n content.append({\n \"type\": \"image_url\",\n \"image_url\": {\n \"url\": f\"data:image/jpeg;base64,{base64_image}\"\n }\n })\n \n response = await self.client.chat.completions.create(\n model=\"gpt-4o\",\n messages=[{\"role\": \"user\", \"content\": content}]\n )\n return response.choices[0].message.content", "section_ref": "14.2.1", "runnable": true, "dependencies": [ "base64", "openai" ] }, { "id": "code-4", "language": "python", "description": "", "code": "import fitz # PyMuPDF\nfrom PIL import Image\nimport io\n\nclass DocumentAgent:\n \"\"\"文档解析Agent\"\"\"\n \n def __init__(self, vision_agent: VisionAgent):\n self.vision = vision_agent\n \n def pdf_to_images(self, pdf_path: str, \n dpi: int = 200) -> list[str]:\n \"\"\"将PDF每页转为图片路径\"\"\"\n doc = fitz.open(pdf_path)\n image_paths = []\n \n for page_num in range(len(doc)):\n page = doc[page_num]\n mat = fitz.Matrix(dpi / 72, dpi / 72)\n pix = page.get_pixmap(matrix=mat)\n \n img_path = f\"/tmp/page_{page_num}.png\"\n pix.save(img_path)\n image_paths.append(img_path)\n \n return image_paths\n \n async def extract_text_from_pdf(self, pdf_path: str) -> str:\n \"\"\"从PDF中提取文本（混合策略）\"\"\"\n # 策略1：尝试直接提取文本\n doc = fitz.open(pdf_path)\n text = \"\"\n for page in doc:\n page_text = page.get_text()\n if page_text.strip():\n text += page_text\n \n if len(text) > 100:\n return text\n \n # 策略2：文本提取不足，使用OCR\n image_paths = self.pdf_to_images(pdf_path)\n ocr_results = []\n for img_path in image_paths:\n result = await self.vision.analyze_image(\n img_path, \n \"请提取这张图片中的所有文字，保持原始格式。\"\n )\n ocr_results.append(result)\n \n return \"\\n\\n\".join(ocr_results)\n \n async def analyze_document(self, file_path: str, \n analysis_type: str = \"summary\") -> dict:\n \"\"\"智能文档分析\"\"\"\n if file_path.endswith(\".pdf\"):\n pages = self.pdf_to_images(file_path)\n else:\n pages = [file_path]\n \n results = {}\n for i, page_path in enumerate(pages):\n prompts = {\n \"summary\": \"请总结这一页的主要内容。\",\n \"key_info\": \"请提取这一页中的关键信息点。\",\n \"tables\": \"请识别并提取这一页中的表格数据，格式化为JSON。\",\n \"structure\": \"请分析这一页的文档结构（标题、段落、列表等）。\",\n }\n result = await self.vision.analyze_image(\n page_path, prompts.get(analysis_type, prompts[\"summary\"])\n )\n results[f\"page_{i}\"] = result\n \n return results", "section_ref": "14.2.2", "runnable": true, "dependencies": [ "fitz", "PIL" ] }, { "id": "code-5", "language": "python", "description": "", "code": "class ChartAnalysisAgent:\n \"\"\"图表分析Agent\"\"\"\n \n async def analyze_chart(self, image_path: str) -> dict:\n \"\"\"分析图表并提取数据\"\"\"\n prompt = \"\"\"\n 请分析这张图表，提供以下信息：\n 1. 图表类型（折线图、柱状图、饼图等）\n 2. 标题和坐标轴标签\n 3. 主要趋势和模式\n 4. 关键数据点（尽可能精确）\n 5. 异常值或显著变化\n \n 请以JSON格式返回。\n \"\"\"\n result = await self.vision.analyze_image(image_path, prompt)\n return json.loads(result)\n \n async def generate_chart_description(self, image_path: str,\n language: str = \"zh\") -> str:\n \"\"\"为图表生成自然语言描述\"\"\"\n prompt = f\"\"\"\n 请用{language}语言详细描述这张图表，包括：\n - 图表展示的主题\n - 主要数据和趋势\n - 值得注意的关键发现\n 描述要专业且易于理解。\n \"\"\"\n return await self.vision.analyze_image(image_path, prompt)", "section_ref": "14.2.3", "runnable": true, "dependencies": [] }, { "id": "code-6", "language": "python", "description": "", "code": "import whisper\nimport asyncio\n\nclass SpeechToTextAgent:\n \"\"\"语音识别Agent\"\"\"\n \n def __init__(self, model_size: str = \"base\"):\n self.model = whisper.load_model(model_size)\n \n async def transcribe(self, audio_path: str,\n language: str | None = None) -> str:\n \"\"\"转录音频文件\"\"\"\n loop = asyncio.get_event_loop()\n result = await loop.run_in_executor(\n None,\n lambda: self.model.transcribe(\n audio_path, \n language=language,\n verbose=False\n )\n )\n return result[\"text\"]\n \n async def transcribe_stream(self, audio_stream: bytes) -> str:\n \"\"\"转录音频流\"\"\"\n # 保存临时文件再转录\n import tempfile\n with tempfile.NamedTemporaryFile(suffix=\".wav\", delete=False) as f:\n f.write(audio_stream)\n temp_path = f.name\n \n try:\n return await self.transcribe(temp_path)\n finally:\n os.unlink(temp_path)\n \n async def detect_language(self, audio_path: str) -> str:\n \"\"\"检测音频语言\"\"\"\n loop = asyncio.get_event_loop()\n result = await loop.run_in_executor(\n None,\n lambda: self.model.transcribe(audio_path, verbose=False)\n )\n return result[\"language\"]", "section_ref": "14.3.1", "runnable": true, "dependencies": [ "whisper" ] }, { "id": "code-7", "language": "python", "description": "", "code": "from openai import AsyncOpenAI\n\nclass TextToSpeechAgent:\n \"\"\"语音合成Agent\"\"\"\n \n def __init__(self, api_key: str):\n self.client = AsyncOpenAI(api_key=api_key)\n \n async def synthesize(self, text: str, voice: str = \"alloy\",\n output_path: str = \"output.mp3\") -> str:\n \"\"\"将文本转为语音\"\"\"\n response = await self.client.audio.speech.create(\n model=\"tts-1\",\n voice=voice,\n input=text\n )\n \n with open(output_path, \"wb\") as f:\n for chunk in response.iter_bytes():\n f.write(chunk)\n \n return output_path\n \n async def synthesize_conversation(self, messages: list[dict],\n output_dir: str = \"conv_audio\"\n ) -> list[str]:\n \"\"\"将对话转为语音序列\"\"\"\n os.makedirs(output_dir, exist_ok=True)\n audio_files = []\n \n voice_map = {\"user\": \"alloy\", \"assistant\": \"nova\"}\n \n for i, msg in enumerate(messages):\n role = msg[\"role\"]\n text = msg[\"content\"]\n output_path = f\"{output_dir}/{i:03d}_{role}.mp3\"\n \n await self.synthesize(\n text, voice=voice_map.get(role, \"alloy\"),\n output_path=output_path\n )\n audio_files.append(output_path)\n \n return audio_files", "section_ref": "14.3.2", "runnable": true, "dependencies": [ "openai" ] }, { "id": "code-8", "language": "python", "description": "", "code": "import websockets\nimport json\n\nclass RealtimeVoiceAgent:\n \"\"\"实时语音对话Agent\"\"\"\n \n def __init__(self, stt: SpeechToTextAgent, \n tts: TextToSpeechAgent, \n llm: Any):\n self.stt = stt\n self.tts = tts\n self.llm = llm\n self.conversation_history = []\n \n async def handle_audio_stream(self, websocket):\n \"\"\"处理WebSocket音频流\"\"\"\n async for message in websocket:\n audio_data = json.loads(message)[\"audio\"]\n \n # 1. 语音转文字\n text = await self.stt.transcribe_stream(\n bytes.fromhex(audio_data)\n )\n \n # 2. LLM生成回复\n self.conversation_history.append(\n {\"role\": \"user\", \"content\": text}\n )\n reply = await self.llm.chat(self.conversation_history)\n self.conversation_history.append(\n {\"role\": \"assistant\", \"content\": reply}\n )\n \n # 3. 文字转语音\n audio_path = await self.tts.synthesize(reply)\n with open(audio_path, \"rb\") as f:\n audio_response = f.read()\n \n await websocket.send(json.dumps({\n \"audio\": audio_response.hex(),\n \"text\": reply\n }))", "section_ref": "14.3.3", "runnable": true, "dependencies": [ "websockets" ] }, { "id": "code-9", "language": "mermaid", "description": "", "code": "graph TB\n A[多模态融合] --> B[早期融合 Early Fusion]\n A --> C[晚期融合 Late Fusion]\n A --> D[交叉注意力 Cross-Attention]\n \n B --> B1[\"原始数据拼接\\n→ 统一模型\"]\n C --> C1[\"各模态独立处理\\n→ 结果合并\"]\n D --> D1[\"模态间交互\\n→ 联合表示\"]", "section_ref": "14.4.1", "runnable": false, "dependencies": [] }, { "id": "code-10", "language": "python", "description": "| 交叉注意力 | 模态间通过注意力机制交互 | 平衡效果与效率 | 架构复杂 |", "code": "class MultimodalFusionAgent:\n \"\"\"多模态融合Agent（晚期融合策略）\"\"\"\n \n def __init__(self, vision: VisionAgent, \n stt: SpeechToTextAgent,\n llm: Any):\n self.vision = vision\n self.stt = stt\n self.llm = llm\n \n async def process_multimodal_input(\n self, \n text: str | None = None,\n image_path: str | None = None,\n audio_path: str | None = None,\n instruction: str = \"请综合分析以下信息\"\n ) -> str:\n \"\"\"处理多模态输入并生成综合回答\"\"\"\n context_parts = []\n \n # 处理文本模态\n if text:\n context_parts.append(f\"文本内容: {text}\")\n \n # 处理图像模态\n if image_path:\n image_desc = await self.vision.analyze_image(\n image_path, \n \"详细描述这张图片的内容\"\n )\n context_parts.append(f\"图像描述: {image_desc}\")\n \n # 处理音频模态\n if audio_path:\n transcript = await self.stt.transcribe(audio_path)\n context_parts.append(f\"语音转录: {transcript}\")\n \n # 综合推理\n full_context = \"\\n\\n\".join(context_parts)\n prompt = f\"\"\"\n {instruction}\n \n 以下是多模态输入信息：\n {full_context}\n \n 请综合以上信息给出回答。\n \"\"\"\n \n return await self.llm.generate(prompt)", "section_ref": "14.4.2", "runnable": true, "dependencies": [] }, { "id": "code-11", "language": "python", "description": "", "code": "from pathlib import Path\nfrom typing import Literal\n\nclass FileProcessingTool:\n \"\"\"多模态文件处理工具\"\"\"\n \n SUPPORTED_FORMATS = {\n \"image\": [\".jpg\", \".jpeg\", \".png\", \".gif\", \".webp\"],\n \"audio\": [\".mp3\", \".wav\", \".m4a\", \".ogg\"],\n \"video\": [\".mp4\", \".avi\", \".mov\"],\n \"document\": [\".pdf\", \".docx\", \".xlsx\", \".pptx\"],\n \"text\": [\".txt\", \".md\", \".csv\", \".json\"],\n }\n \n def detect_type(self, file_path: str) -> str:\n \"\"\"检测文件类型\"\"\"\n ext = Path(file_path).suffix.lower()\n for file_type, extensions in self.SUPPORTED_FORMATS.items():\n if ext in extensions:\n return file_type\n return \"unknown\"\n \n async def process(self, file_path: str, \n operation: str) -> dict:\n \"\"\"处理文件\"\"\"\n file_type = self.detect_type(file_path)\n \n processors = {\n \"image\": self._process_image,\n \"audio\": self._process_audio,\n \"document\": self._process_document,\n \"text\": self._process_text,\n }\n \n processor = processors.get(file_type)\n if not processor:\n return {\"error\": f\"不支持的文件类型: {file_type}\"}\n \n return await processor(file_path, operation)\n \n async def _process_image(self, path: str, op: str) -> dict:\n \"\"\"图像处理\"\"\"\n if op == \"describe\":\n return {\"description\": await self.vision.analyze_image(path)}\n elif op == \"ocr\":\n return {\"text\": await self.vision.analyze_image(\n path, \"提取图片中的所有文字\"\n )}\n elif op == \"resize\":\n # 使用PIL处理\n img = Image.open(path)\n img.thumbnail((800, 800))\n output = path.replace(\".\", \"_resized.\")\n img.save(output)\n return {\"output\": output}\n \n async def _process_audio(self, path: str, op: str) -> dict:\n \"\"\"音频处理\"\"\"\n if op == \"transcribe\":\n return {\"transcript\": await self.stt.transcribe(path)}\n elif op == \"detect_language\":\n return {\"language\": await self.stt.detect_language(path)}", "section_ref": "14.5.1", "runnable": true, "dependencies": [] }, { "id": "code-12", "language": "python", "description": "", "code": "class ProductionMultimodalAgent:\n \"\"\"生产级多模态Agent\"\"\"\n \n def __init__(self, config: dict):\n self.config = config\n self.vision = VisionAgent(config[\"openai_key\"])\n self.stt = SpeechToTextAgent(config[\"whisper_model\"])\n self.tts = TextToSpeechAgent(config[\"openai_key\"])\n self.llm = get_llm(config[\"llm_config\"])\n self.file_processor = FileProcessingTool()\n self.memory = AgentMemory()\n \n async def handle_request(self, request: dict) -> dict:\n \"\"\"统一处理多模态请求\"\"\"\n # 1. 输入预处理\n preprocessed = await self._preprocess(request)\n \n # 2. 模态路由\n modality = self._route_modality(preprocessed)\n \n # 3. 各模态处理\n results = await self._process_modalities(\n preprocessed, modality\n )\n \n # 4. 融合推理\n answer = await self._fuse_and_reason(results, request)\n \n # 5. 输出后处理\n response = await self._postprocess(answer, request)\n \n return response\n \n async def _preprocess(self, request: dict) -> dict:\n \"\"\"输入预处理：统一格式化\"\"\"\n processed = {\n \"text\": request.get(\"text\", \"\"),\n \"images\": [],\n \"audio\": [],\n \"documents\": [],\n }\n \n # 处理文件上传\n for file_info in request.get(\"files\", []):\n file_type = self.file_processor.detect_type(\n file_info[\"path\"]\n )\n if file_type == \"image\":\n processed[\"images\"].append(file_info[\"path\"])\n elif file_type == \"audio\":\n processed[\"audio\"].append(file_info[\"path\"])\n elif file_type == \"document\":\n processed[\"documents\"].append(file_info[\"path\"])\n \n return processed\n \n def _route_modality(self, data: dict) -> str:\n \"\"\"模态路由：判断主要模态\"\"\"\n if data[\"images\"]:\n return \"visual\"\n elif data[\"audio\"]:\n return \"audio\"\n elif data[\"documents\"]:\n return \"document\"\n else:\n return \"text_only\"\n \n async def _process_modalities(self, data: dict, \n modality: str) -> dict:\n \"\"\"分模态处理\"\"\"\n results = {}\n \n if modality in (\"visual\", \"document\"):\n for img_path in data[\"images\"]:\n results[f\"image_{img_path}\"] = \\\n await self.vision.analyze_image(img_path)\n \n if data[\"audio\"]:\n for audio_path in data[\"audio\"]:\n results[f\"audio_{audio_path}\"] = \\\n await self.stt.transcribe(audio_path)\n \n if data[\"documents\"]:\n for doc_path in data[\"documents\"]:\n results[f\"doc_{doc_path}\"] = \\\n await self.file_processor.process(\n doc_path, \"extract\"\n )\n \n return results", "section_ref": "14.6.1", "runnable": true, "dependencies": [] }, { "id": "code-13", "language": "python", "description": "", "code": "class OutputSynthesizer:\n \"\"\"多模态输出合成器\"\"\"\n \n def __init__(self, tts: TextToSpeechAgent):\n self.tts = tts\n \n async def synthesize(self, answer: str, \n output_format: str = \"text\",\n **kwargs) -> dict:\n \"\"\"根据请求格式合成输出\"\"\"\n if output_format == \"text\":\n return {\"type\": \"text\", \"content\": answer}\n elif output_format == \"audio\":\n audio_path = await self.tts.synthesize(answer)\n return {\n \"type\": \"audio\", \n \"content\": answer,\n \"audio_url\": audio_path\n }\n elif output_format == \"both\":\n audio_path = await self.tts.synthesize(answer)\n return {\n \"type\": \"both\",\n \"text\": answer,\n \"audio_url\": audio_path\n }", "section_ref": "14.6.2", "runnable": true, "dependencies": [] }, { "id": "code-14", "language": "python", "description": "", "code": "class DocumentAnalysisAgent:\n \"\"\"多模态文档分析Agent\"\"\"\n \n def __init__(self):\n self.vision = VisionAgent(api_key=os.environ[\"OPENAI_API_KEY\"])\n self.llm = LLMClient()\n \n async def analyze_contract(self, pdf_path: str) -> dict:\n \"\"\"分析合同文档\"\"\"\n # 1. PDF转图片\n pages = self.pdf_to_images(pdf_path)\n \n # 2. 逐页分析\n page_analyses = []\n for page_path in pages:\n analysis = await self.vision.analyze_image(\n page_path,\n \"\"\"\n 请分析这份合同页面，提取以下信息：\n 1. 合同各方（甲方、乙方）\n 2. 关键条款（金额、期限、违约责任）\n 3. 需要注意的风险点\n 以JSON格式返回。\n \"\"\"\n )\n page_analyses.append(analysis)\n \n # 3. 综合分析\n summary_prompt = f\"\"\"\n 以下是一份合同的逐页分析结果：\n {json.dumps(page_analyses, ensure_ascii=False, indent=2)}\n \n 请综合分析这份合同，提供：\n 1. 合同概要\n 2. 关键条款总结\n 3. 潜在风险提示\n 4. 谈判建议\n \"\"\"\n comprehensive_analysis = await self.llm.generate(summary_prompt)\n \n return {\n \"page_analyses\": page_analyses,\n \"comprehensive_analysis\": comprehensive_analysis,\n \"total_pages\": len(pages)\n }\n \n async def analyze_financial_report(self, pdf_path: str) -> dict:\n \"\"\"分析财务报告\"\"\"\n pages = self.pdf_to_images(pdf_path)\n \n # 重点分析含图表的页面\n data_points = []\n for page_path in pages:\n analysis = await self.vision.analyze_image(\n page_path,\n \"\"\"\n 如果此页包含财务数据或图表：\n 1. 识别数据类型（收入、利润、现金流等）\n 2. 提取关键数值\n 3. 识别趋势（增长/下降/持平）\n 4. 标注异常数据点\n \n 以JSON格式返回。\n \"\"\"\n )\n data_points.append(json.loads(analysis))\n \n # 生成分析报告\n report = await self.llm.generate(f\"\"\"\n 基于以下财务数据提取结果，生成一份简洁的财务分析报告：\n {json.dumps(data_points, ensure_ascii=False)}\n \n 报告应包含：\n 1. 财务状况概览\n 2. 关键财务指标\n 3. 趋势分析\n 4. 风险与机会\n \"\"\")\n \n return {\"data_points\": data_points, \"report\": report}", "section_ref": "14.7", "runnable": true, "dependencies": [] }, { "id": "code-15", "language": "python", "description": "", "code": "class ModelSelector:\n \"\"\"根据任务复杂度选择模型\"\"\"\n \n def __init__(self):\n self.model_tiers = {\n \"simple\": {\n \"vision\": \"gpt-4o-mini\",\n \"cost_per_image\": 0.001,\n \"suitable_for\": [\"简单OCR\", \"文本检测\", \"分类\"]\n },\n \"standard\": {\n \"vision\": \"gpt-4o\",\n \"cost_per_image\": 0.01,\n \"suitable_for\": [\"图像描述\", \"图表分析\", \"文档理解\"]\n },\n \"complex\": {\n \"vision\": \"gpt-4o\",\n \"cost_per_image\": 0.01,\n \"suitable_for\": [\"复杂推理\", \"多图对比\", \"合同分析\"]\n }\n }\n \n def select_model(self, task_type: str, \n complexity: str = \"auto\") -> str:\n \"\"\"选择合适的模型\"\"\"\n if complexity == \"auto\":\n # 简单任务用小模型\n simple_tasks = [\"ocr\", \"classify\", \"detect\"]\n if any(t in task_type.lower() for t in simple_tasks):\n complexity = \"simple\"\n else:\n complexity = \"standard\"\n \n return self.model_tiers[complexity][\"vision\"]", "section_ref": "14.8.1", "runnable": true, "dependencies": [] }, { "id": "code-16", "language": "python", "description": "", "code": "import hashlib\n\nclass MultimodalCache:\n \"\"\"多模态结果缓存\"\"\"\n \n def __init__(self, redis_client):\n self.redis = redis_client\n self.ttl = 3600 * 24 # 24小时缓存\n \n def _compute_hash(self, file_path: str) -> str:\n \"\"\"计算文件内容哈希\"\"\"\n with open(file_path, \"rb\") as f:\n content = f.read()\n return hashlib.md5(content).hexdigest()\n \n async def get_cached(self, file_path: str, \n operation: str) -> str | None:\n \"\"\"获取缓存结果\"\"\"\n file_hash = self._compute_hash(file_path)\n cache_key = f\"multimodal:{operation}:{file_hash}\"\n return await self.redis.get(cache_key)\n \n async def set_cached(self, file_path: str, \n operation: str, result: str):\n \"\"\"设置缓存\"\"\"\n file_hash = self._compute_hash(file_path)\n cache_key = f\"multimodal:{operation}:{file_hash}\"\n await self.redis.setex(cache_key, self.ttl, result)", "section_ref": "14.8.2", "runnable": true, "dependencies": [] } ], "tables": [ { "headers": [ "模态", "输入能力", "输出能力", "典型模型" ], "data": [ [ "文本", "✅", "✅", "GPT-4, Claude, Gemini" ], [ "图像", "✅", "✅", "GPT-4V, Gemini Pro Vision" ], [ "音频", "✅", "✅", "Whisper, TTS" ], [ "视频", "✅（逐帧）", "⚠️（有限）", "Gemini, Video-LLaMA" ], [ "文档", "✅", "✅", "Claude, GPT-4V" ], [ "代码", "✅", "✅", "GPT-4, Claude" ] ] }, { "headers": [ "策略", "原理", "优势", "劣势" ], "data": [ [ "早期融合", "将原始多模态数据拼接后送入模型", "模态间交互充分", "计算量大，需要重新训练" ], [ "晚期融合", "各模态独立处理，合并结果", "实现简单，模块化", "模态间交互有限" ], [ "交叉注意力", "模态间通过注意力机制交互", "平衡效果与效率", "架构复杂" ] ] } ], "key_takeaways": [], "common_pitfalls": [], "related_chapters": [ "ch04" ] }