diff --git a/docs/features/chat-features/code-execution/index.md b/docs/features/chat-features/code-execution/index.md index 74d2660ee..3468f506c 100644 --- a/docs/features/chat-features/code-execution/index.md +++ b/docs/features/chat-features/code-execution/index.md @@ -7,7 +7,9 @@ Open WebUI offers powerful code execution capabilities directly within your chat ## Key Features -- **Python Code Execution**: Run Python scripts directly in your browser using Pyodide, with support for popular libraries like pandas and matplotlib with no setup required. +- **Code Interpreter Capability**: Enable models to autonomously write and execute Python code as part of their responses. Works with both Default Mode (XML-based) and Native Mode (tool calling via `execute_code`). + +- **Python Code Execution**: Run Python scripts directly in your browser using Pyodide, or on a server using Jupyter. Supports popular libraries like pandas and matplotlib with no setup required. - **MermaidJS Rendering**: Create and visualize flowcharts, diagrams, and other visual representations with MermaidJS syntax that automatically renders in your chat. diff --git a/docs/features/chat-features/code-execution/python.md b/docs/features/chat-features/code-execution/python.md index 8c5bbc617..c2a2a05c4 100644 --- a/docs/features/chat-features/code-execution/python.md +++ b/docs/features/chat-features/code-execution/python.md @@ -50,6 +50,20 @@ These settings can be configured at **Admin Panel → Settings → Code Executio For Jupyter configuration, see the [Jupyter Notebook Integration](/tutorials/integrations/jupyter) tutorial. +### Native Function Calling (Native Mode) + +When using **Native function calling mode** with a capable model (e.g., GPT-5, Claude 4.5, MiniMax M2.1), the code interpreter is available as a builtin tool called `execute_code`. This provides a more integrated experience: + +- **No XML tags required**: The model calls `execute_code(code)` directly +- **Same image handling**: Base64 image URLs in output are replaced with file URLs; model embeds via markdown + +**Requirements:** +1. `ENABLE_CODE_INTERPRETER` must be enabled globally +2. Model must have `code_interpreter` capability enabled +3. Model must use **Native** function calling mode (set in model's advanced params) + +For more details on builtin tools and native mode, see the [Tool Development Guide](/features/plugin/tools/development#built-in-system-tools-nativeagentic-mode). + ## Displaying Images Inline (matplotlib, etc.) When using matplotlib or other visualization libraries, images can be displayed directly in the chat. For this to work correctly, the code must output the image as a **base64 data URL**. @@ -92,9 +106,7 @@ If you see raw base64 text appearing in chat responses, the model is incorrectly ### Example Prompt -> Create a bar chart showing quarterly sales: Q1: 150, Q2: 230, Q3: 180, Q4: 310. -> Use matplotlib, save the figure to a BytesIO buffer, encode it as base64, and print the data URL. -> After the code runs, use the resulting file URL from the output to display the image in your response. +> Create a bar chart showing quarterly sales: Q1: 150, Q2: 230, Q3: 180, Q4: 310. **Expected model behavior:** 1. Model writes Python code using the base64 pattern above @@ -175,6 +187,32 @@ plt.close() The image will be automatically uploaded and displayed inline in your chat. +## Browser Compatibility + +### Microsoft Edge: Pyodide Crashes + +If Pyodide-based code execution causes Microsoft Edge to crash with a `STATUS_ACCESS_VIOLATION` error, this is caused by Edge's enhanced security mode. + +**Symptom:** The browser tab or entire browser crashes when attempting to run Python code, with no useful error message. + +**Cause:** Edge's "Enhance your security on the web" setting (found at `edge://settings/privacy/security`) enables stricter security mitigations that are incompatible with WebAssembly-based runtimes like Pyodide. + +**Solutions:** + +1. **Disable enhanced security in Edge:** + - Go to `edge://settings/privacy/security` + - Turn off **"Enhance your security on the web"** + +2. **Use a different browser:** + - Chrome and Firefox do not have this issue + +3. **Use Jupyter backend:** + - Switch `CODE_INTERPRETER_ENGINE` to `jupyter` to avoid browser-based execution entirely + +:::note +This is a known compatibility issue between Edge's enhanced security mode and WebAssembly. The same crash occurs on the official [Pyodide console](https://pyodide.org/en/stable/console.html) when this setting is enabled. +::: + ## Tips for Better Results - **Mention the environment**: Tell the LLM it's running in a "Pyodide environment" or "code interpreter" for better code generation diff --git a/docs/features/image-generation-and-editing/usage.md b/docs/features/image-generation-and-editing/usage.md index 4a132db1d..aea079cc5 100644 --- a/docs/features/image-generation-and-editing/usage.md +++ b/docs/features/image-generation-and-editing/usage.md @@ -18,9 +18,12 @@ Before you can use image generation, you must ensure that the **Image Generation If your model is configured with **Native Function Calling** (see the [**Central Tool Calling Guide**](/features/plugin/tools#tool-calling-modes-default-vs-native)), it can invoke image generation directly as a tool. ### How it works: -- **Requirement**: The **Image Generation** feature must be toggled **ON** for the chat or model. This grants the model "permission" to use the tool. +- **Requirements**: + - **Image Generation** must be enabled globally in **Admin Panel → Settings → Images** + - The model must have the **Image Generation** capability enabled +- **No Chat Toggle Needed**: With Native Mode, the `generate_image` tool is automatically included when the model has the `image_generation` capability. You don't need to manually toggle it on per chat. - **Natural Language**: You can simply ask the model: *"Generate an image of a cybernetic forest."* -- **Action**: If **Native Mode** is active and the feature is enabled, the model will invoke the `generate_image` tool. +- **Action**: If **Native Mode** is active and the model has the capability, it will invoke the `generate_image` tool. - **Display**: The generated image is displayed directly in the chat interface. - **Editing**: This also supports **Image Editing** (inpainting) via the `edit_image` tool (e.g., *"Make the sky in this image red"*). diff --git a/docs/features/plugin/tools/development.mdx b/docs/features/plugin/tools/development.mdx index d636f622c..46cd1e32f 100644 --- a/docs/features/plugin/tools/development.mdx +++ b/docs/features/plugin/tools/development.mdx @@ -145,60 +145,8 @@ You can configure the function calling mode in two places: If the model seems to be unable to call the tool, make sure it is enabled (either via the Model page or via the `+` sign next to the chat input field). -### Built-in System Tools (Native/Agentic Mode) - -When **Native Mode (Agentic Mode)** is enabled, Open WebUI automatically injects built-in system tools based on the features enabled for the chat. This enables powerful agentic behaviors where capable models (like GPT-5, Claude 4.5 Sonnet, Gemini 3 Flash, or MiniMax M2.1) can perform multi-step research, explore knowledge bases autonomously, or manage user memory dynamically. - -:::warning Quality Models Required for Agentic Behavior -Agentic tool calling requires **high-quality frontier models** to work reliably. Small local models often struggle with the complex reasoning, proper JSON formatting, and multi-step planning required for effective tool use. For production agentic workflows, use models like **GPT-5**, **Claude 4.5+**, **Gemini 3+**, or **MiniMax M2.1**. Small local models may work better with **Default Mode** instead. -::: - -#### Available Built-in Tools - -| Tool | Purpose | Requirements | -|------|---------|--------------| -| **Search & Web** | | | -| `search_web` | Search the public web for information. Best for current events, external references, or topics not covered in internal documents. | `ENABLE_WEB_SEARCH` enabled. | -| `fetch_url` | Visits a URL and extracts text content via the Web Loader. | Part of Web Search feature. | -| **Knowledge Base** | | | -| `list_knowledge_bases` | List the user's accessible knowledge bases with file counts. | Always available. | -| `search_knowledge_bases` | Search knowledge bases by name and description. | Always available. | -| `search_knowledge_files` | Search files across accessible knowledge bases by filename. | Always available. | -| `view_knowledge_file` | Get the full content of a file from a knowledge base. | Always available. | -| `query_knowledge_bases` | Search knowledge bases using semantic/vector search (default: 5 results). **Note:** Does not use hybrid search or reranking. | Always available. | -| **Image Gen** | | | -| `generate_image` | Generates a new image based on a prompt (supports `steps`). | `ENABLE_IMAGE_GENERATION` enabled. | -| `edit_image` | Edits an existing image based on a prompt and URL. | `ENABLE_IMAGE_EDIT` enabled.| -| **Memory** | | | -| `search_memories` | Searches the user's personal memory/personalization bank. | Memory feature enabled. | -| `add_memory` | Stores a new fact in the user's personalization memory. | Memory feature enabled. | -| `replace_memory_content` | Updates an existing memory record by its unique ID. | Memory feature enabled. | -| **Notes** | | | -| `search_notes` | Search the user's notes by title and content. | `ENABLE_NOTES` enabled. | -| `view_note` | Get the full markdown content of a specific note. | `ENABLE_NOTES` enabled. | -| `write_note` | Create a new private note for the user. | `ENABLE_NOTES` enabled. | -| `replace_note_content` | Update an existing note's content or title. | `ENABLE_NOTES` enabled. | -| **Chat History** | | | -| `search_chats` | Simple text search across chat titles and message content. Returns matching chat IDs and snippets. | Always available. | -| `view_chat` | Reads and returns the full message history of a specific chat by ID. | Always available. | -| **Channels** | | | -| `search_channels` | Find public or accessible channels by name/description. | `ENABLE_CHANNELS` enabled. | -| `search_channel_messages` | Search for specific messages inside accessible channels. | `ENABLE_CHANNELS` enabled. | -| `view_channel_message` | View a specific message or its details in a channel. | `ENABLE_CHANNELS` enabled. | -| `view_channel_thread` | View a full message thread/replies in a channel. | `ENABLE_CHANNELS` enabled. | -| **Time Tools** | | | -| `get_current_timestamp` | Get the current UTC Unix timestamp and ISO date. | Always available. | -| `calculate_timestamp` | Calculate relative timestamps (e.g., "3 days ago"). | Always available. | - -#### Why Use Built-in Tools? -- **Agentic Research**: Models can invoke `search_web` multiple times to refine results, then use `fetch_url` to read specific deep-dive articles. -- **Knowledge Base Search**: Models can use `query_knowledge_bases` to search internal documents semantically, or `list_knowledge_bases` and `view_knowledge_file` to browse and read specific files. -- **Contextual Awareness**: Models can search your previous chat history or notes to find specific details without manual copy-pasting. -- **Dynamic Personalization**: Models can proactively store important facts about the user using `add_memory` during the conversation. -- **Improved Context Selection**: Instead of forcing a search before every prompt, the model decides *when* a search or retrieval is actually necessary. - -:::info Complete Tool Reference -This table provides a quick reference for developers. For the complete user-facing guide on how to enable and use these tools, see the [**Tool Calling Modes Guide**](/features/plugin/tools#tool-calling-modes-default-vs-native). +:::info Native Mode & Built-in Tools +When writing custom tools, be aware that Open WebUI also provides **built-in system tools** when Native Mode is enabled. For details on built-in tools, function calling modes, and model requirements, see the [**Tool Calling Modes Guide**](/features/plugin/tools#tool-calling-modes-default-vs-native). ::: diff --git a/docs/features/plugin/tools/index.mdx b/docs/features/plugin/tools/index.mdx index 0c56469c4..89d0caa0d 100644 --- a/docs/features/plugin/tools/index.mdx +++ b/docs/features/plugin/tools/index.mdx @@ -148,41 +148,41 @@ These models excel at multi-step reasoning, proper JSON formatting, and autonomo 🛠️ When **Native Mode (Agentic Mode)** is enabled, Open WebUI automatically injects powerful system tools based on the features toggled for the chat. This unlocks truly agentic behaviors where capable models (like GPT-5, Claude 4.5 Sonnet, Gemini 3 Flash, or MiniMax M2.1) can perform multi-step research, explore knowledge bases, or manage user memory autonomously. -| Tool | Purpose | Requirements | -|------|---------|--------------| -| **Search & Web** | | | -| `search_web` | Search the public web for information. Best for current events, external references, or topics not covered in internal documents. | `ENABLE_WEB_SEARCH` enabled. | -| `fetch_url` | Visits a URL and extracts text content via the Web Loader. | Part of Web Search feature. | -| **Knowledge Base** | | | -| `list_knowledge_bases` | List the user's accessible knowledge bases with file counts. | Always available. | -| `query_knowledge_bases` | Search knowledge bases by semantic similarity to query. Finds KBs whose name/description match the meaning of your query. Use this to discover relevant knowledge bases before querying their files. | Always available. | -| `search_knowledge_bases` | Search knowledge bases by name and description. | Always available. | -| `query_knowledge_files` | Search knowledge base files using simple vector search. **Note:** Does not use hybrid search or reranking—for full RAG pipeline with reranking, use File Context instead by attaching files via `#` or assigning knowledge bases. | Always available. | -| `search_knowledge_files` | Search files across accessible knowledge bases by filename. | Always available. | -| `view_knowledge_file` | Get the full content of a file from a knowledge base. | Always available. | -| **Image Gen** | | | -| `generate_image` | Generates a new image based on a prompt. | `ENABLE_IMAGE_GENERATION` enabled. | -| `edit_image` | Edits existing images based on a prompt and image URLs. | `ENABLE_IMAGE_EDIT` enabled. | -| **Memory** | | | -| `search_memories` | Searches the user's personal memory/personalization bank. | Memory feature enabled. | -| `add_memory` | Stores a new fact in the user's personalization memory. | Memory feature enabled. | -| `replace_memory_content` | Updates an existing memory record by its unique ID. | Memory feature enabled. | -| **Notes** | | | -| `search_notes` | Search the user's notes by title and content. | `ENABLE_NOTES` enabled. | -| `view_note` | Get the full markdown content of a specific note. | `ENABLE_NOTES` enabled. | -| `write_note` | Create a new private note for the user. | `ENABLE_NOTES` enabled. | -| `replace_note_content` | Update an existing note's content or title. | `ENABLE_NOTES` enabled. | -| **Chat History** | | | -| `search_chats` | Simple text search across chat titles and message content. Returns matching chat IDs and snippets. | Always available. | -| `view_chat` | Reads and returns the full message history of a specific chat by ID. | Always available. | -| **Channels** | | | -| `search_channels` | Find public or accessible channels by name/description. | `ENABLE_CHANNELS` enabled. | -| `search_channel_messages` | Search for specific messages inside accessible channels. | `ENABLE_CHANNELS` enabled. | -| `view_channel_message` | View a specific message or its details in a channel. | `ENABLE_CHANNELS` enabled. | -| `view_channel_thread` | View a full message thread/replies in a channel. | `ENABLE_CHANNELS` enabled. | -| **Time Tools** | | | -| `get_current_timestamp` | Get the current UTC Unix timestamp and ISO date. | Always available. | -| `calculate_timestamp` | Calculate relative timestamps (e.g., "3 days ago"). | Always available. | +| Tool | Purpose | +|------|---------| +| **Search & Web** | *Requires `ENABLE_WEB_SEARCH` enabled.* | +| `search_web` | Search the public web for information. Best for current events, external references, or topics not covered in internal documents. | +| `fetch_url` | Visits a URL and extracts text content via the Web Loader. | +| **Knowledge Base** | *Always available.* | +| `list_knowledge_bases` | List the user's accessible knowledge bases with file counts. | +| `query_knowledge_bases` | Search knowledge bases by semantic similarity to query. Finds KBs whose name/description match the meaning of your query. Use this to discover relevant knowledge bases before querying their files. | +| `search_knowledge_bases` | Search knowledge bases by name and description. | +| `query_knowledge_files` | Search knowledge base files using simple vector search. **Note:** Does not use hybrid search or reranking—for full RAG pipeline with reranking, use File Context instead by attaching files via `#` or assigning knowledge bases. | +| `search_knowledge_files` | Search files across accessible knowledge bases by filename. | +| `view_knowledge_file` | Get the full content of a file from a knowledge base. | +| **Image Gen** | *Requires image generation enabled (per-tool).* | +| `generate_image` | Generates a new image based on a prompt. Requires `ENABLE_IMAGE_GENERATION`. | +| `edit_image` | Edits existing images based on a prompt and image URLs. Requires `ENABLE_IMAGE_EDIT`. | +| **Memory** | *Requires Memory feature enabled.* | +| `search_memories` | Searches the user's personal memory/personalization bank. | +| `add_memory` | Stores a new fact in the user's personalization memory. | +| `replace_memory_content` | Updates an existing memory record by its unique ID. | +| **Notes** | *Requires `ENABLE_NOTES` enabled.* | +| `search_notes` | Search the user's notes by title and content. | +| `view_note` | Get the full markdown content of a specific note. | +| `write_note` | Create a new private note for the user. | +| `replace_note_content` | Update an existing note's content or title. | +| **Chat History** | *Always available.* | +| `search_chats` | Simple text search across chat titles and message content. Returns matching chat IDs and snippets. | +| `view_chat` | Reads and returns the full message history of a specific chat by ID. | +| **Channels** | *Requires `ENABLE_CHANNELS` enabled.* | +| `search_channels` | Find public or accessible channels by name/description. | +| `search_channel_messages` | Search for specific messages inside accessible channels. | +| `view_channel_message` | View a specific message or its details in a channel. | +| `view_channel_thread` | View a full message thread/replies in a channel. | +| **Time Tools** | *Always available.* | +| `get_current_timestamp` | Get the current UTC Unix timestamp and ISO date. | +| `calculate_timestamp` | Calculate relative timestamps (e.g., "3 days ago"). | #### Tool Parameters Reference diff --git a/docs/features/web-search/agentic-search.mdx b/docs/features/web-search/agentic-search.mdx index a4fdbdeeb..5f9f39cbd 100644 --- a/docs/features/web-search/agentic-search.mdx +++ b/docs/features/web-search/agentic-search.mdx @@ -29,13 +29,19 @@ For comprehensive information about all built-in agentic tools (including web se To unlock these features, your model must support native tool calling and have strong reasoning capabilities (e.g., GPT-5, Claude 4.5 Sonnet, Gemini 3 Flash, MiniMax M2.1). Administrator-level configuration for these built-in system tools is handled via the [**Central Tool Calling Guide**](/features/plugin/tools#tool-calling-modes-default-vs-native). -1. **Enable Web Search**: Ensure a search engine is configured in **Admin Panel > Settings > Web Search**. -2. **Enable Native Mode (Agentic Mode)**: - * Go to **Admin Panel > Settings > Models**. - * Navigate to **Model Specific Settings** for your target model. - * Under **Advanced Parameters**, set **Function Calling** to `Native`. -3. **Use a Quality Model**: Ensure you're using a frontier model with strong reasoning capabilities for best results. -4. **Chat Features**: Ensure the **Web Search** feature is toggled **ON** for your chat session. +1. **Enable Web Search Globally**: Ensure a search engine is configured in **Admin Panel → Settings → Web Search**. +2. **Enable Model Capability**: In **Admin Panel → Settings → Models**, select your model and enable the **Web Search** capability. +3. **Enable Native Mode (Agentic Mode)**: + * In the same model settings, under **Advanced Parameters**, set **Function Calling** to `Native`. +4. **Use a Quality Model**: Ensure you're using a frontier model with strong reasoning capabilities for best results. + +:::tip Model Capability vs. Chat Toggle +In **Native Mode**, the `search_web` and `fetch_url` tools are automatically included based on the model's `web_search` capability setting. The chat toggle is not required. + +In **Default Mode** (non-native), the chat toggle still controls whether web search is performed via RAG-style injection. + +**Important**: If you disable the `web_search` capability on a model but use Native Mode, the tools won't be available even if you manually toggle Web Search on in the chat. +::: ## How Native Tools Handle Data (Agentic Mode) 🔗 It is important to understand that Native Mode (Agentic Mode) works fundamentally differently from the global "Web Search" toggle found in standard models. diff --git a/docs/features/workspace/models.md b/docs/features/workspace/models.md index 62f0361fa..923325f89 100644 --- a/docs/features/workspace/models.md +++ b/docs/features/workspace/models.md @@ -81,7 +81,7 @@ You can transform a generic model into a specialized agent by toggling specific - **Web Search**: Enable the model to access the configured search provider (e.g., Google, SearxNG) for real-time information. - **File Upload**: Allow users to upload files to this model. - **File Context**: When enabled (default), attached files are processed via RAG and their content is injected into the conversation. When disabled, file content is **not** extracted or injected—the model receives no file content unless it retrieves it via builtin tools. Only visible when File Upload is enabled. See [File Context vs Builtin Tools](../rag/index.md#file-context-vs-builtin-tools) for details. - - **Code Interpreter**: Enable Python code execution. + - **Code Interpreter**: Enable Python code execution. See [Python Code Execution](../chat-features/code-execution/python.md) for details. - **Image Generation**: Enable image generation integration. - **Usage / Citations**: Toggle usage tracking or source citations. - **Status Updates**: Show visible progress steps in the chat UI (e.g., "Searching web...", "Reading file...") during generation. Useful for slower, complex tasks. diff --git a/docs/getting-started/env-configuration.mdx b/docs/getting-started/env-configuration.mdx index 688d3a86f..6ea71e11a 100644 --- a/docs/getting-started/env-configuration.mdx +++ b/docs/getting-started/env-configuration.mdx @@ -2262,11 +2262,25 @@ When using Pinecone as the vector store, the following environment variables are ### Weaviate +:::info + +**Self-Hosted and Cloud Deployments** + +Open WebUI uses `connect_to_custom` for Weaviate connections, which supports both locally hosted and remote Weaviate instances. This is essential for self-hosted deployments where HTTP and gRPC endpoints may be on different ingresses or hostnames, which is common in container orchestration platforms like Kubernetes or Azure Container Apps. + +::: + #### `WEAVIATE_HTTP_HOST` - Type: `str` - Default: Empty string (' ') -- Description: Specifies the hostname of the Weaviate server for HTTP connections. +- Description: Specifies the hostname of the Weaviate server for HTTP connections. For self-hosted deployments, this is typically your Weaviate HTTP endpoint hostname. + +#### `WEAVIATE_GRPC_HOST` + +- Type: `str` +- Default: Empty string (' ') +- Description: Specifies the hostname of the Weaviate server for gRPC connections. This can be different from `WEAVIATE_HTTP_HOST` when HTTP and gRPC are served on separate ingresses, which is common in container orchestration environments. #### `WEAVIATE_HTTP_PORT` @@ -2280,12 +2294,30 @@ When using Pinecone as the vector store, the following environment variables are - Default: `50051` - Description: Specifies the gRPC port for connecting to the Weaviate server. +#### `WEAVIATE_HTTP_SECURE` + +- Type: `bool` +- Default: `False` +- Description: Enables HTTPS for HTTP connections to the Weaviate server. Set to `true` when connecting to a Weaviate instance with TLS enabled on the HTTP endpoint. + +#### `WEAVIATE_GRPC_SECURE` + +- Type: `bool` +- Default: `False` +- Description: Enables TLS for gRPC connections to the Weaviate server. Set to `true` when connecting to a Weaviate instance with TLS enabled on the gRPC endpoint. + #### `WEAVIATE_API_KEY` - Type: `str` - Default: `None` - Description: Sets the API key for authenticating with Weaviate server. +#### `WEAVIATE_SKIP_INIT_CHECKS` + +- Type: `bool` +- Default: `False` +- Description: Skips Weaviate initialization checks when connecting. This can be useful in certain network configurations where the checks may fail but the connection itself works. + ### Oracle 23ai Vector Search (oracle23ai) #### `ORACLE_DB_USE_WALLET`