Multimodal RAG with the Gemini API File Search Tool: A Developer Guide

✍ Google AI Studio🕐 2026-05-06📦 12.3 KB 🟢 已读 𝕏 文章列表

本文介绍了 Gemini API 中的 File Search 工具，该工具现已通过 Gemini Embedding 2 支持多模态检索。文章详细讲解了如何创建存储、上传图文文档、执行具备依据生成的查询以及检索引用和图像。File Search 提供了全托管的 RAG 解决方案，具备原生图像搜索、内置引用和成本效益高等优势。

Gemini API RAG 多模态文件检索开发指南 Google AI Python SDK

# Multimodal RAG with the Gemini API File Search Tool: A Developer Guide **作者**: Google AI Studio **日期**: 2026-05-05T20:22:26.000Z **来源**: [https://x.com/GoogleAIStudio/status/2051759446888698149](https://x.com/GoogleAIStudio/status/2051759446888698149) --- ![Image](/doc/1721/image/img_991482ed1127.jpg) The File Search tool in the Gemini API now supports multimodal retrieval by adding support for Gemini Embedding 2. This update allows images, such as charts, product photos, and diagrams, to be natively indexed and searched in the same store as your text-based documents. This post covers how to use the File Search tool end-to-end: creating a store, uploading documents and images, querying with grounded generation, and retrieving image citations. ## What is File Search? Here's an example app you can try in AI Studio that lets you chat with your documents and image library File Search is the Gemini API's built-in RAG tool. When you upload your documents, the API takes care of the heavy lifting: chunking, embedding, indexing, and retrieval. At query time, pass a file_search tool alongside your prompt, and the model automatically retrieves relevant chunks from your data to generate a grounded response. Compared to rolling your own RAG pipeline, File Search offers: - Fully managed: No vector databases to provision or embedding pipeline to maintain. - Cost-effective: Storage and query-time embeddings are free. You only pay for the initial indexing embeddings and the standard Gemini input/output tokens. - Built-in citations: Every response includes grounding metadata that links the answer to specific documents and pages. For multimodal stores, citations also include downloadable image references. - Native image search: With the gemini-embedding-2 model, images are embedded directly rather than relying on OCR, enabling true visual retrieval. ## Try It in AI Studio Want to see multimodal File Search in action before writing any code? We built an example app in AI Studio that lets you chat with your documents and image library. Upload PDFs and images, then ask questions. The app retrieves relevant text and visuals in real time, complete with citations and page numbers so you can trace every answer back to its source. ![图片](/doc/1721/image/img_f260b9c77c82.jpg) ## Getting Started Step 1: Create a File Search Store A File Search Store is a persistent container for your document embeddings. Think of it as a managed vector database scoped to a project. To enable multimodal search over images, specify gemini-embedding-2 as the embedding model. This parameter is optional; if omitted, the store defaults to gemini-embedding-001, which is cost-optimized for text-only workloads, and cannot be changed later. To use the new features, make sure to install the latest Python SDK: pip install -U google-genai. ``` from google import genai from google.genai import types client = genai.Client() # Create a multimodal store with gemini-embedding-2 # Omit embedding_model to use the default text-only model (gemini-embedding-001) file_search_store = client.file_search_stores.create( config={ "display_name": "product-catalog", "embedding_model": "models/gemini-embedding-2" } ) print(f"Created store: {file_search_store.name}") ``` ![Image](/doc/1721/image/img_80b58528e01e.jpg) Step 2: Upload Documents and Images The simplest path is the upload_to_file_search_store method, which uploads and indexes a file in one step. With gemini-embedding-2, this works for both documents and images: Note: Audio and video formats are currently not supported. ``` import time # Upload a PDF document operation = client.file_search_stores.upload_to_file_search_store( file_search_store_name=file_search_store.name, file="product_catalog.pdf", config={"display_name": "Product Catalog"} ) # Wait for ingestion to complete while not operation.done: time.sleep(5) operation = client.operations.get(operation) # Upload product images directly for image_file in ["sneaker_red.png", "sneaker_blue.jpeg", "sneaker_white.png"]: op = client.file_search_stores.upload_to_file_search_store( file_search_store_name=file_search_store.name, file=image_file, config={"display_name": image_file} ) while not op.done: time.sleep(5) op = client.operations.get(op) print("All files indexed!") ``` Behind the scenes, the API chunks documents, generates embeddings, and indexes the content. When using gemini-embedding-2, images within PDFs are also natively embedded alongside the text. You can also import existing files from the Files API into a store. Step 3: Query with File Search Query your data by passing the file_search tool to generate_content: ``` response = client.models.generate_content( model="gemini-3-flash-preview", contents="Which sneakers come in red?", config={ "tools": [{ "file_search": { "file_search_store_names": [file_search_store.name] } }] } ) print(response.text) ``` The system performs a file search to find the most similar and relevant chunks from the File Search store , and uses them to generate a grounded response. Step 4: Inspect Citations and Retrieve Images Every File Search response includes grounding metadata — essentially, a bibliography for the model's answer. It captures page numbers for the indexed information, allowing applications to point users directly to the right spot in a document. This is especially useful for rigorous fact-checking over large PDFs. With multimodal stores, citations can include a media_id for referenced images, which can be downloaded directly: ``` grounding = response.candidates[0].grounding_metadata for chunk in grounding.grounding_chunks: ctx = chunk.retrieved_context if ctx.media_id: # This is an image citation — download it print(f"Cited image: {ctx.title}") print(f" Media ID: {ctx.media_id}") blob = client.file_search_stores.download_media( media_id=ctx.media_id ) with open(f"cited_{ctx.title}.png", "wb") as f: f.write(blob) else: # Text citation with exact page number print(f"Cited text: {ctx.title}") if ctx.page_number: print(f" Page: {ctx.page_number}") print(f" {ctx.text[:200]}...") # See which parts of the response are grounded in which sources for support in grounding.grounding_supports: print(f"Claim: '{support.segment.text}'") print(f" Grounded in chunks: {support.grounding_chunk_indices}") ``` This is powerful for building user-facing applications. It's now possible to show users the actual images the model used in its reasoning, not just a text description. ## Managing Stores Here's a quick reference for managing stores and documents: ``` # List all stores for store in client.file_search_stores.list(): print(f"{store.name} — {store.display_name}") # List documents in a store for doc in client.file_search_stores.documents.list(parent=file_search_store.name): print(f" {doc.name}") # Delete a specific document client.file_search_stores.documents.delete( name="fileSearchStores/my-store/documents/old_doc" ) # Delete an entire store (force=True also deletes all contained documents) client.file_search_stores.delete( name=file_search_store.name, config={"force": True} ) ``` ## Power Features Custom Metadata and Filtering You can attach metadata to documents at upload time and use it to filter at query time. This is essential when a store contains diverse documents and searches need to be scoped: ``` # Upload with metadata op = client.file_search_stores.upload_to_file_search_store( file_search_store_name=file_search_store.name, file="shoes_collection.pdf", config={ "display_name": "Spring 2026 Shoes", "custom_metadata": [ {"key": "category", "string_value": "footwear"}, {"key": "season", "string_value": "spring-2026"}, {"key": "price_tier", "numeric_value": 2} ] } ) # Query with a metadata filter response = client.models.generate_content( model="gemini-3-flash-preview", contents="Do you have blue spring shoes?", config={ "tools": [{ "file_search": { "file_search_store_names": [file_search_store.name], "metadata_filter": 'category="footwear" AND season="spring-2026"', } }] } ) ``` Structured Output Starting with Gemini 3 models, File Search can be combined with structured output. This is perfect for extracting structured data from grounded responses: ``` from pydantic import BaseModel, Field class ProductMatch(BaseModel): name: str = Field(description="Product name") description: str = Field(description="Brief product description") confidence: str = Field(description="How confident the match is") response = client.models.generate_content( model="gemini-3-flash-preview", contents="Find products similar to a red running shoe", config={ "tools": [{ "file_search": { "file_search_store_names": [file_search_store.name] } }], "response_mime_type": "application/json", "response_schema": ProductMatch.model_json_schema() } ) ``` Chunking Configuration For more control over how documents are split, the chunking strategy can be configured: ``` operation = client.file_search_stores.upload_to_file_search_store( file_search_store_name=file_search_store.name, file="long_document.pdf", config={ "display_name": "Technical Manual", "chunking_config": { "white_space_config": { "max_tokens_per_chunk": 200, "max_overlap_tokens": 20 } } } ) ``` ## Use Cases With multimodal retrieval, File Search opens up scenarios that text-only RAG can't handle: - Visual product search: Index catalogs with images and spec sheets, then search by visual similarity or natural language descriptions. - Research and technical documentation: Retrieve specific charts, architecture diagrams, or data visualizations from papers and reports. - Insurance and claims processing: Combine structured forms with damage photos for unified document and visual assessment. - Design systems: Make component libraries searchable by visual appearance, not just naming conventions. - Real estate and property listings: Match properties based on floor plans, interior photos, and visual preferences. ## Pricing File Search is designed to be cost-effective: - Indexing: You pay for embeddings at indexing time (embeddings pricing). - Storage: Free. - Query-time embeddings: Free. - Retrieved tokens: Charged as regular context tokens. ## Get Started Here's everything needed to get started: - File Search documentation - File Search quickstart notebook - The latest Python SDK: Install it with pip install -U google-genai - Get an API key Create your store with gemini-embedding-2, upload some images, and start building multimodal RAG applications. ## 相关链接 - [Google AI Studio](https://x.com/GoogleAIStudio) - [@GoogleAIStudio](https://x.com/GoogleAIStudio) - [19K](https://x.com/GoogleAIStudio/status/2051759446888698149/analytics) - [Gemini Embedding 2](https://developers.googleblog.com/en/building-with-gemini-embedding-2/) - [File Search](https://ai.google.dev/gemini-api/docs/file-search) - [example app in AI Studio](https://ai.studio/apps/acb0ca81-7130-43ae-a31f-bedd96d28294) - [import existing files](https://ai.google.dev/gemini-api/docs/file-search#importing-files) - [embeddings pricing](https://ai.google.dev/gemini-api/docs/pricing#gemini-embedding-2) - [File Search documentation](https://ai.google.dev/gemini-api/docs/file-search) - [File Search quickstart notebook](https://github.com/google-gemini/cookbook/blob/main/quickstarts/File_Search.ipynb) - [The latest Python SDK](https://github.com/googleapis/python-genai) - [Get an API key](https://aistudio.google.com/apikey) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [4:22 AM · May 6, 2026](https://x.com/GoogleAIStudio/status/2051759446888698149) - [19.2K Views](https://x.com/GoogleAIStudio/status/2051759446888698149/analytics) - [View quotes](https://x.com/GoogleAIStudio/status/2051759446888698149/quotes) --- *导出时间: 2026/5/6 23:05:27* --- ## 中文翻译 # 使用 Gemini API 文件搜索工具构建多模态 RAG：开发者指南 **作者**: Google AI Studio **日期**: 2026-05-05T20:22:26.000Z **来源**: [https://x.com/GoogleAIStudio/status/2051759446888698149](https://x.com/GoogleAIStudio/status/2051759446888698149) --- ![图片](/doc/1721/image/img_991482ed1127.jpg) Gemini API 中的文件搜索工具现在支持多模态检索，新增了对 Gemini Embedding 2 的支持。此次更新允许图表、产品照片和示意图等图像与文本文档在同一存储中原生地进行索引和搜索。本文将涵盖如何端到端地使用文件搜索工具：包括创建存储、上传文档和图像、使用基于事实的生成进行查询，以及检索图像引用。 ## 什么是文件搜索？这里有一个您可以在 AI Studio 中尝试的示例应用，它让您能与文档和图像库进行对话文件搜索是 Gemini API 内置的 RAG（检索增强生成）工具。当您上传文档时，API 会处理繁重的工作：分块、嵌入、索引和检索。在查询时，将 `file_search` 工具与您的提示词一起传递，模型会自动从您的数据中检索相关的分块，以生成基于事实的回复。与自行构建 RAG 流程相比，文件搜索提供了以下优势： - **完全托管**：无需配置向量数据库或维护嵌入流程。 - **成本效益高**：存储和查询时的嵌入是免费的。您只需支付初始索引嵌入的费用以及标准的 Gemini 输入/输出 Token 费用。 - **内置引用**：每个回复都包含将答案链接到特定文档和页面的事实依据元数据。对于多模态存储，引用还包括可下载的图像引用。 - **原生图像搜索**：利用 `gemini-embedding-2` 模型，图像是直接被嵌入的，而不依赖 OCR（光学字符识别），从而实现真正的视觉检索。 ## 在 AI Studio 中试用想在编写代码之前先体验多模态文件搜索的实际效果吗？我们在 AI Studio 中构建了一个示例应用，让您可以与文档和图像库聊天。上传 PDF 和图片，然后提问。该应用会实时检索相关的文本和视觉内容，并提供引用和页码，以便您将每个答案追溯至其来源。 ![图片](/doc/1721/image/img_f260b9c77c82.jpg) ## 快速开始 **步骤 1：创建文件搜索存储** 文件搜索存储是您文档嵌入的持久化容器。可以将其视为特定于项目的托管向量数据库。要启用针对图像的多模态搜索，请将 `gemini-embedding-2` 指定为嵌入模型。此参数是可选的；如果省略，存储默认为 `gemini-embedding-001`，这是针对仅文本工作负载进行成本优化的模型，且事后无法更改。要使用新功能，请确保安装最新的 Python SDK：`pip install -U google-genai`。 ``` from google import genai from google.genai import types client = genai.Client() # Create a multimodal store with gemini-embedding-2 # Omit embedding_model to use the default text-only model (gemini-embedding-001) file_search_store = client.file_search_stores.create( config={ "display_name": "product-catalog", "embedding_model": "models/gemini-embedding-2" } ) print(f"Created store: {file_search_store.name}") ``` ![图片](/doc/1721/image/img_80b58528e01e.jpg) **步骤 2：上传文档和图像** 最简单的方法是使用 `upload_to_file_search_store` 方法，它一步即可完成文件上传和索引。使用 `gemini-embedding-2` 时，这适用于文档和图像：注意：目前不支持音频和视频格式。 ``` import time # Upload a PDF document operation = client.file_search_stores.upload_to_file_search_store( file_search_store_name=file_search_store.name, file="product_catalog.pdf", config={"display_name": "Product Catalog"} ) # Wait for ingestion to complete while not operation.done: time.sleep(5) operation = client.operations.get(operation) # Upload product images directly for image_file in ["sneaker_red.png", "sneaker_blue.jpeg", "sneaker_white.png"]: op = client.file_search_stores.upload_to_file_search_store( file_search_store_name=file_search_store.name, file=image_file, config={"display_name": image_file} ) while not op.done: time.sleep(5) op = client.operations.get(op) print("All files indexed!") ``` 在幕后，API 会对文档进行分块、生成嵌入并索引内容。当使用 `gemini-embedding-2` 时，PDF 内的图像也会与文本一起被原生嵌入。您还可以将文件 API 中的现有文件导入到存储中。 **步骤 3：使用文件搜索进行查询** 通过将 `file_search` 工具传递给 `generate_content` 来查询您的数据： ``` response = client.models.generate_content( model="gemini-3-flash-preview", contents="Which sneakers come in red?", config={ "tools": [{ "file_search": { "file_search_store_names": [file_search_store.name] } }] } ) print(response.text) ``` 系统会执行文件搜索，从文件搜索存储中找到最相似和相关的分块，并使用它们生成基于事实的回复。 **步骤 4：检查引用并检索图像** 每个文件搜索回复都包含事实依据元数据——本质上是模型答案的参考文献。它会捕获索引信息的页码，允许应用程序将用户直接指向文档中的正确位置。这对于对大型 PDF 进行严格的事实核查特别有用。对于多模态存储，引用可以包含所引用图像的 `media_id`，可以直接下载： ``` grounding = response.candidates[0].grounding_metadata for chunk in grounding.grounding_chunks: ctx = chunk.retrieved_context if ctx.media_id: # This is an image citation — download it print(f"Cited image: {ctx.title}") print(f" Media ID: {ctx.media_id}") blob = client.file_search_stores.download_media( media_id=ctx.media_id ) with open(f"cited_{ctx.title}.png", "wb") as f: f.write(blob) else: # Text citation with exact page number print(f"Cited text: {ctx.title}") if ctx.page_number: print(f" Page: {ctx.page_number}") print(f" {ctx.text[:200]}...") # See which parts of the response are grounded in which sources for support in grounding.grounding_supports: print(f"Claim: '{support.segment.text}'") print(f" Grounded in chunks: {support.grounding_chunk_indices}") ``` 这对于构建面向用户的应用程序非常强大。现在可以向用户展示模型在其推理中使用的实际图像，而不仅仅是文字描述。 ## 管理存储以下是管理存储和文档的快速参考： ``` # List all stores for store in client.file_search_stores.list(): print(f"{store.name} — {store.display_name}") # List documents in a store for doc in client.file_search_stores.documents.list(parent=file_search_store.name): print(f" {doc.name}") # Delete a specific document client.file_search_stores.documents.delete( name="fileSearchStores/my-store/documents/old_doc" ) # Delete an entire store (force=True also deletes all contained documents) client.file_search_stores.delete( name=file_search_store.name, config={"force": True} ) ``` ## 高级功能 **自定义元数据和过滤** 您可以在上传时将元数据附加到文档，并在查询时使用它进行过滤。当存储包含多样化的文档且需要限定搜索范围时，这至关重要： ``` # Upload with metadata op = client.file_search_stores.upload_to_file_search_store( file_search_store_name=file_search_store.name, file="shoes_collection.pdf", config={ "display_name": "Spring 2026 Shoes", "custom_metadata": [ {"key": "category", "string_value": "footwear"}, {"key": "season", "string_value": "spring-2026"}, {"key": "price_tier", "numeric_value": 2} ] } ) # Query with a metadata filter response = client.models.generate_content( model="gemini-3-flash-preview", contents="Do you have blue spring shoes?", config={ "tools": [{ "file_search": { "file_search_store_names": [file_search_store.name], "metadata_filter": 'category="footwear" AND season="spring-2026"', } }] } ) ``` **结构化输出** 从 Gemini 3 模型开始，文件搜索可以与结构化输出结合使用。这对于从基于事实的回复中提取结构化数据非常完美： ``` from pydantic import BaseModel, Field class ProductMatch(BaseModel): name: str = Field(description="Product name") description: str = Field(description="Brief product description") confidence: str = Field(description="How confident the match is") response = client.models.generate_content( model="gemini-3-flash-preview", contents="Find products similar to a red running shoe", config={ "tools": [{ "file_search": { "file_search_store_names": [file_search_store.name] } }], "response_mime_type": "application/json", "response_schema": ProductMatch.model_json_schema() } ) ``` **分块配置** 为了更好地控制文档的分割方式，可以配置分块策略： ``` operation = client.file_search_stores.upload_to_file_search_store( file_search_store_name=file_search_store.name, file="long_document.pdf", config={ "display_name": "Technical Manual", "chunking_config": { "white_space_config": { "max_tokens_per_chunk": 200, "max_overlap_tokens": 20 } } } ) ``` ## 用例借助多模态检索，文件搜索开启了仅文本 RAG 无法处理的场景： - **视觉产品搜索**：索引包含图像和规格表的目录，然后通过视觉相似性或自然语言描述进行搜索。 - **研究和技术文档**：从论文和报告中检索特定的图表、架构图或数据可视化。 - **保险和理赔处理**：将结构化表单与损坏照片结合，进行统一的文档和视觉评估。 - **设计系统**：使组件库可以通过视觉外观进行搜索，而不仅仅是命名约定。 - **房地产和房源列表**：根据平面图、室内照片和视觉偏好匹配房源。 ## 定价文件搜索的设计旨在具有成本效益： - **索引**：您需支付索引时的嵌入费用（嵌入定价）。 - **存储**：免费。 - **查询时嵌入**：免费。 - **检索的 Token**：作为常规上下文 Token 收费。 ## 立即开始以下是开始使用所需的一切： - [文件搜索文档](https://ai.google.dev/gemini-api/docs/file-search) - [文件搜索快速入门笔记本](https://github.com/google-gemini/cookbook/blob/main/quickstarts/File_Search.ipynb) - 最新的 Python SDK：使用 `pip install -U google-genai` 安装 - [获取 API 密钥](https://aistudio.google.com/apikey) 使用 `gemini-embedding-2` 创建您的存储，上传一些图像，然后开始构建多模态 RAG 应用程序。 ## 相关链接 - [Google AI Studio](https://x.com/GoogleAIStudio) - [@GoogleAIStudio](https://x.com/GoogleAIStudio) - [19K](https://x.com/GoogleAIStudio/status/2051759446888698149/analytics) - [Gemini Embedding 2](https://developers.googleblog.com/en/building-with-gemini-embedding-2/) - [File Search](https://ai.google.dev/gemini-api/docs/file-search) - [AI Studio 中的示例应用](https://ai.studio/apps/acb0ca81-7130-43ae-a31f-bedd96d28294) - [导入现有文件](https://ai.google.dev/gemini-api/docs/file-search#importing-files) - [嵌入定价](https://ai.google.dev/gemini-api/docs/pricing#gemini-embedding-2) - [文件搜索文档](https://ai.google.dev/gemini-api/docs/file-search) - [文件搜索快速入门笔记本](https://github.com/google-gemini/cookbook/blob/main/quickstarts/File_Search.ipynb) - [最新的 Python SDK](https://github.com/googleapis/python-genai) - [获取 API 密钥](https://aistudio.google.com/apikey) - [升级到 Premium](https://x.com/i/premium_sign_up) - [2026年5月6日上午4:22](https://x.com/GoogleAIStudio/status/2051759446888698149) - [1.92万次观看](https://x.com/GoogleAIStudio/status/2051759446888698149/analytics) - [查看引用](https://x.com/GoogleAIStudio/status/2051759446888698149/quotes) --- *导出时间: 2026/5/6 23:05:27*

🔗 相关文档