Skip to content
Draft

0.7.3 #999

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/features/chat-features/code-execution/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@ Open WebUI offers powerful code execution capabilities directly within your chat

## Key Features

- **Python Code Execution**: Run Python scripts directly in your browser using Pyodide, with support for popular libraries like pandas and matplotlib with no setup required.
- **Code Interpreter Capability**: Enable models to autonomously write and execute Python code as part of their responses. Works with both Default Mode (XML-based) and Native Mode (tool calling via `execute_code`).

- **Python Code Execution**: Run Python scripts directly in your browser using Pyodide, or on a server using Jupyter. Supports popular libraries like pandas and matplotlib with no setup required.

- **MermaidJS Rendering**: Create and visualize flowcharts, diagrams, and other visual representations with MermaidJS syntax that automatically renders in your chat.

Expand Down
44 changes: 41 additions & 3 deletions docs/features/chat-features/code-execution/python.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,20 @@ These settings can be configured at **Admin Panel → Settings → Code Executio

For Jupyter configuration, see the [Jupyter Notebook Integration](/tutorials/integrations/jupyter) tutorial.

### Native Function Calling (Native Mode)

When using **Native function calling mode** with a capable model (e.g., GPT-5, Claude 4.5, MiniMax M2.1), the code interpreter is available as a builtin tool called `execute_code`. This provides a more integrated experience:

- **No XML tags required**: The model calls `execute_code(code)` directly
- **Same image handling**: Base64 image URLs in output are replaced with file URLs; model embeds via markdown

**Requirements:**
1. `ENABLE_CODE_INTERPRETER` must be enabled globally
2. Model must have `code_interpreter` capability enabled
3. Model must use **Native** function calling mode (set in model's advanced params)

For more details on builtin tools and native mode, see the [Tool Development Guide](/features/plugin/tools/development#built-in-system-tools-nativeagentic-mode).

## Displaying Images Inline (matplotlib, etc.)

When using matplotlib or other visualization libraries, images can be displayed directly in the chat. For this to work correctly, the code must output the image as a **base64 data URL**.
Expand Down Expand Up @@ -92,9 +106,7 @@ If you see raw base64 text appearing in chat responses, the model is incorrectly

### Example Prompt

> Create a bar chart showing quarterly sales: Q1: 150, Q2: 230, Q3: 180, Q4: 310.
> Use matplotlib, save the figure to a BytesIO buffer, encode it as base64, and print the data URL.
> After the code runs, use the resulting file URL from the output to display the image in your response.
> Create a bar chart showing quarterly sales: Q1: 150, Q2: 230, Q3: 180, Q4: 310.

**Expected model behavior:**
1. Model writes Python code using the base64 pattern above
Expand Down Expand Up @@ -175,6 +187,32 @@ plt.close()

The image will be automatically uploaded and displayed inline in your chat.

## Browser Compatibility

### Microsoft Edge: Pyodide Crashes

If Pyodide-based code execution causes Microsoft Edge to crash with a `STATUS_ACCESS_VIOLATION` error, this is caused by Edge's enhanced security mode.

**Symptom:** The browser tab or entire browser crashes when attempting to run Python code, with no useful error message.

**Cause:** Edge's "Enhance your security on the web" setting (found at `edge://settings/privacy/security`) enables stricter security mitigations that are incompatible with WebAssembly-based runtimes like Pyodide.

**Solutions:**

1. **Disable enhanced security in Edge:**
- Go to `edge://settings/privacy/security`
- Turn off **"Enhance your security on the web"**

2. **Use a different browser:**
- Chrome and Firefox do not have this issue

3. **Use Jupyter backend:**
- Switch `CODE_INTERPRETER_ENGINE` to `jupyter` to avoid browser-based execution entirely

:::note
This is a known compatibility issue between Edge's enhanced security mode and WebAssembly. The same crash occurs on the official [Pyodide console](https://pyodide.org/en/stable/console.html) when this setting is enabled.
:::

## Tips for Better Results

- **Mention the environment**: Tell the LLM it's running in a "Pyodide environment" or "code interpreter" for better code generation
Expand Down
7 changes: 5 additions & 2 deletions docs/features/image-generation-and-editing/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,12 @@ Before you can use image generation, you must ensure that the **Image Generation
If your model is configured with **Native Function Calling** (see the [**Central Tool Calling Guide**](/features/plugin/tools#tool-calling-modes-default-vs-native)), it can invoke image generation directly as a tool.

### How it works:
- **Requirement**: The **Image Generation** feature must be toggled **ON** for the chat or model. This grants the model "permission" to use the tool.
- **Requirements**:
- **Image Generation** must be enabled globally in **Admin Panel → Settings → Images**
- The model must have the **Image Generation** capability enabled
- **No Chat Toggle Needed**: With Native Mode, the `generate_image` tool is automatically included when the model has the `image_generation` capability. You don't need to manually toggle it on per chat.
- **Natural Language**: You can simply ask the model: *"Generate an image of a cybernetic forest."*
- **Action**: If **Native Mode** is active and the feature is enabled, the model will invoke the `generate_image` tool.
- **Action**: If **Native Mode** is active and the model has the capability, it will invoke the `generate_image` tool.
- **Display**: The generated image is displayed directly in the chat interface.
- **Editing**: This also supports **Image Editing** (inpainting) via the `edit_image` tool (e.g., *"Make the sky in this image red"*).

Expand Down
56 changes: 2 additions & 54 deletions docs/features/plugin/tools/development.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -145,60 +145,8 @@ You can configure the function calling mode in two places:

If the model seems to be unable to call the tool, make sure it is enabled (either via the Model page or via the `+` sign next to the chat input field).

### Built-in System Tools (Native/Agentic Mode)

When **Native Mode (Agentic Mode)** is enabled, Open WebUI automatically injects built-in system tools based on the features enabled for the chat. This enables powerful agentic behaviors where capable models (like GPT-5, Claude 4.5 Sonnet, Gemini 3 Flash, or MiniMax M2.1) can perform multi-step research, explore knowledge bases autonomously, or manage user memory dynamically.

:::warning Quality Models Required for Agentic Behavior
Agentic tool calling requires **high-quality frontier models** to work reliably. Small local models often struggle with the complex reasoning, proper JSON formatting, and multi-step planning required for effective tool use. For production agentic workflows, use models like **GPT-5**, **Claude 4.5+**, **Gemini 3+**, or **MiniMax M2.1**. Small local models may work better with **Default Mode** instead.
:::

#### Available Built-in Tools

| Tool | Purpose | Requirements |
|------|---------|--------------|
| **Search & Web** | | |
| `search_web` | Search the public web for information. Best for current events, external references, or topics not covered in internal documents. | `ENABLE_WEB_SEARCH` enabled. |
| `fetch_url` | Visits a URL and extracts text content via the Web Loader. | Part of Web Search feature. |
| **Knowledge Base** | | |
| `list_knowledge_bases` | List the user's accessible knowledge bases with file counts. | Always available. |
| `search_knowledge_bases` | Search knowledge bases by name and description. | Always available. |
| `search_knowledge_files` | Search files across accessible knowledge bases by filename. | Always available. |
| `view_knowledge_file` | Get the full content of a file from a knowledge base. | Always available. |
| `query_knowledge_bases` | Search knowledge bases using semantic/vector search (default: 5 results). **Note:** Does not use hybrid search or reranking. | Always available. |
| **Image Gen** | | |
| `generate_image` | Generates a new image based on a prompt (supports `steps`). | `ENABLE_IMAGE_GENERATION` enabled. |
| `edit_image` | Edits an existing image based on a prompt and URL. | `ENABLE_IMAGE_EDIT` enabled.|
| **Memory** | | |
| `search_memories` | Searches the user's personal memory/personalization bank. | Memory feature enabled. |
| `add_memory` | Stores a new fact in the user's personalization memory. | Memory feature enabled. |
| `replace_memory_content` | Updates an existing memory record by its unique ID. | Memory feature enabled. |
| **Notes** | | |
| `search_notes` | Search the user's notes by title and content. | `ENABLE_NOTES` enabled. |
| `view_note` | Get the full markdown content of a specific note. | `ENABLE_NOTES` enabled. |
| `write_note` | Create a new private note for the user. | `ENABLE_NOTES` enabled. |
| `replace_note_content` | Update an existing note's content or title. | `ENABLE_NOTES` enabled. |
| **Chat History** | | |
| `search_chats` | Simple text search across chat titles and message content. Returns matching chat IDs and snippets. | Always available. |
| `view_chat` | Reads and returns the full message history of a specific chat by ID. | Always available. |
| **Channels** | | |
| `search_channels` | Find public or accessible channels by name/description. | `ENABLE_CHANNELS` enabled. |
| `search_channel_messages` | Search for specific messages inside accessible channels. | `ENABLE_CHANNELS` enabled. |
| `view_channel_message` | View a specific message or its details in a channel. | `ENABLE_CHANNELS` enabled. |
| `view_channel_thread` | View a full message thread/replies in a channel. | `ENABLE_CHANNELS` enabled. |
| **Time Tools** | | |
| `get_current_timestamp` | Get the current UTC Unix timestamp and ISO date. | Always available. |
| `calculate_timestamp` | Calculate relative timestamps (e.g., "3 days ago"). | Always available. |

#### Why Use Built-in Tools?
- **Agentic Research**: Models can invoke `search_web` multiple times to refine results, then use `fetch_url` to read specific deep-dive articles.
- **Knowledge Base Search**: Models can use `query_knowledge_bases` to search internal documents semantically, or `list_knowledge_bases` and `view_knowledge_file` to browse and read specific files.
- **Contextual Awareness**: Models can search your previous chat history or notes to find specific details without manual copy-pasting.
- **Dynamic Personalization**: Models can proactively store important facts about the user using `add_memory` during the conversation.
- **Improved Context Selection**: Instead of forcing a search before every prompt, the model decides *when* a search or retrieval is actually necessary.

:::info Complete Tool Reference
This table provides a quick reference for developers. For the complete user-facing guide on how to enable and use these tools, see the [**Tool Calling Modes Guide**](/features/plugin/tools#tool-calling-modes-default-vs-native).
:::info Native Mode & Built-in Tools
When writing custom tools, be aware that Open WebUI also provides **built-in system tools** when Native Mode is enabled. For details on built-in tools, function calling modes, and model requirements, see the [**Tool Calling Modes Guide**](/features/plugin/tools#tool-calling-modes-default-vs-native).
:::


Expand Down
70 changes: 35 additions & 35 deletions docs/features/plugin/tools/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -148,41 +148,41 @@ These models excel at multi-step reasoning, proper JSON formatting, and autonomo

🛠️ When **Native Mode (Agentic Mode)** is enabled, Open WebUI automatically injects powerful system tools based on the features toggled for the chat. This unlocks truly agentic behaviors where capable models (like GPT-5, Claude 4.5 Sonnet, Gemini 3 Flash, or MiniMax M2.1) can perform multi-step research, explore knowledge bases, or manage user memory autonomously.

| Tool | Purpose | Requirements |
|------|---------|--------------|
| **Search & Web** | | |
| `search_web` | Search the public web for information. Best for current events, external references, or topics not covered in internal documents. | `ENABLE_WEB_SEARCH` enabled. |
| `fetch_url` | Visits a URL and extracts text content via the Web Loader. | Part of Web Search feature. |
| **Knowledge Base** | | |
| `list_knowledge_bases` | List the user's accessible knowledge bases with file counts. | Always available. |
| `query_knowledge_bases` | Search knowledge bases by semantic similarity to query. Finds KBs whose name/description match the meaning of your query. Use this to discover relevant knowledge bases before querying their files. | Always available. |
| `search_knowledge_bases` | Search knowledge bases by name and description. | Always available. |
| `query_knowledge_files` | Search knowledge base files using simple vector search. **Note:** Does not use hybrid search or reranking—for full RAG pipeline with reranking, use File Context instead by attaching files via `#` or assigning knowledge bases. | Always available. |
| `search_knowledge_files` | Search files across accessible knowledge bases by filename. | Always available. |
| `view_knowledge_file` | Get the full content of a file from a knowledge base. | Always available. |
| **Image Gen** | | |
| `generate_image` | Generates a new image based on a prompt. | `ENABLE_IMAGE_GENERATION` enabled. |
| `edit_image` | Edits existing images based on a prompt and image URLs. | `ENABLE_IMAGE_EDIT` enabled. |
| **Memory** | | |
| `search_memories` | Searches the user's personal memory/personalization bank. | Memory feature enabled. |
| `add_memory` | Stores a new fact in the user's personalization memory. | Memory feature enabled. |
| `replace_memory_content` | Updates an existing memory record by its unique ID. | Memory feature enabled. |
| **Notes** | | |
| `search_notes` | Search the user's notes by title and content. | `ENABLE_NOTES` enabled. |
| `view_note` | Get the full markdown content of a specific note. | `ENABLE_NOTES` enabled. |
| `write_note` | Create a new private note for the user. | `ENABLE_NOTES` enabled. |
| `replace_note_content` | Update an existing note's content or title. | `ENABLE_NOTES` enabled. |
| **Chat History** | | |
| `search_chats` | Simple text search across chat titles and message content. Returns matching chat IDs and snippets. | Always available. |
| `view_chat` | Reads and returns the full message history of a specific chat by ID. | Always available. |
| **Channels** | | |
| `search_channels` | Find public or accessible channels by name/description. | `ENABLE_CHANNELS` enabled. |
| `search_channel_messages` | Search for specific messages inside accessible channels. | `ENABLE_CHANNELS` enabled. |
| `view_channel_message` | View a specific message or its details in a channel. | `ENABLE_CHANNELS` enabled. |
| `view_channel_thread` | View a full message thread/replies in a channel. | `ENABLE_CHANNELS` enabled. |
| **Time Tools** | | |
| `get_current_timestamp` | Get the current UTC Unix timestamp and ISO date. | Always available. |
| `calculate_timestamp` | Calculate relative timestamps (e.g., "3 days ago"). | Always available. |
| Tool | Purpose |
|------|---------|
| **Search & Web** | *Requires `ENABLE_WEB_SEARCH` enabled.* |
| `search_web` | Search the public web for information. Best for current events, external references, or topics not covered in internal documents. |
| `fetch_url` | Visits a URL and extracts text content via the Web Loader. |
| **Knowledge Base** | *Always available.* |
| `list_knowledge_bases` | List the user's accessible knowledge bases with file counts. |
| `query_knowledge_bases` | Search knowledge bases by semantic similarity to query. Finds KBs whose name/description match the meaning of your query. Use this to discover relevant knowledge bases before querying their files. |
| `search_knowledge_bases` | Search knowledge bases by name and description. |
| `query_knowledge_files` | Search knowledge base files using simple vector search. **Note:** Does not use hybrid search or reranking—for full RAG pipeline with reranking, use File Context instead by attaching files via `#` or assigning knowledge bases. |
| `search_knowledge_files` | Search files across accessible knowledge bases by filename. |
| `view_knowledge_file` | Get the full content of a file from a knowledge base. |
| **Image Gen** | *Requires image generation enabled (per-tool).* |
| `generate_image` | Generates a new image based on a prompt. Requires `ENABLE_IMAGE_GENERATION`. |
| `edit_image` | Edits existing images based on a prompt and image URLs. Requires `ENABLE_IMAGE_EDIT`. |
| **Memory** | *Requires Memory feature enabled.* |
| `search_memories` | Searches the user's personal memory/personalization bank. |
| `add_memory` | Stores a new fact in the user's personalization memory. |
| `replace_memory_content` | Updates an existing memory record by its unique ID. |
| **Notes** | *Requires `ENABLE_NOTES` enabled.* |
| `search_notes` | Search the user's notes by title and content. |
| `view_note` | Get the full markdown content of a specific note. |
| `write_note` | Create a new private note for the user. |
| `replace_note_content` | Update an existing note's content or title. |
| **Chat History** | *Always available.* |
| `search_chats` | Simple text search across chat titles and message content. Returns matching chat IDs and snippets. |
| `view_chat` | Reads and returns the full message history of a specific chat by ID. |
| **Channels** | *Requires `ENABLE_CHANNELS` enabled.* |
| `search_channels` | Find public or accessible channels by name/description. |
| `search_channel_messages` | Search for specific messages inside accessible channels. |
| `view_channel_message` | View a specific message or its details in a channel. |
| `view_channel_thread` | View a full message thread/replies in a channel. |
| **Time Tools** | *Always available.* |
| `get_current_timestamp` | Get the current UTC Unix timestamp and ISO date. |
| `calculate_timestamp` | Calculate relative timestamps (e.g., "3 days ago"). |

#### Tool Parameters Reference

Expand Down
20 changes: 13 additions & 7 deletions docs/features/web-search/agentic-search.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,19 @@ For comprehensive information about all built-in agentic tools (including web se

To unlock these features, your model must support native tool calling and have strong reasoning capabilities (e.g., GPT-5, Claude 4.5 Sonnet, Gemini 3 Flash, MiniMax M2.1). Administrator-level configuration for these built-in system tools is handled via the [**Central Tool Calling Guide**](/features/plugin/tools#tool-calling-modes-default-vs-native).

1. **Enable Web Search**: Ensure a search engine is configured in **Admin Panel > Settings > Web Search**.
2. **Enable Native Mode (Agentic Mode)**:
* Go to **Admin Panel > Settings > Models**.
* Navigate to **Model Specific Settings** for your target model.
* Under **Advanced Parameters**, set **Function Calling** to `Native`.
3. **Use a Quality Model**: Ensure you're using a frontier model with strong reasoning capabilities for best results.
4. **Chat Features**: Ensure the **Web Search** feature is toggled **ON** for your chat session.
1. **Enable Web Search Globally**: Ensure a search engine is configured in **Admin Panel → Settings → Web Search**.
2. **Enable Model Capability**: In **Admin Panel → Settings → Models**, select your model and enable the **Web Search** capability.
3. **Enable Native Mode (Agentic Mode)**:
* In the same model settings, under **Advanced Parameters**, set **Function Calling** to `Native`.
4. **Use a Quality Model**: Ensure you're using a frontier model with strong reasoning capabilities for best results.

:::tip Model Capability vs. Chat Toggle
In **Native Mode**, the `search_web` and `fetch_url` tools are automatically included based on the model's `web_search` capability setting. The chat toggle is not required.

In **Default Mode** (non-native), the chat toggle still controls whether web search is performed via RAG-style injection.

**Important**: If you disable the `web_search` capability on a model but use Native Mode, the tools won't be available even if you manually toggle Web Search on in the chat.
:::

## How Native Tools Handle Data (Agentic Mode)
🔗 It is important to understand that Native Mode (Agentic Mode) works fundamentally differently from the global "Web Search" toggle found in standard models.
Expand Down
Loading