Skip to content

Conversation

@leeweisern
Copy link

Summary

Custom tools can now return images alongside text, allowing the LLM to actually see and analyze visual data.

  • Custom tools return { content: [{ type: "text", text }, { type: "image", mimeType, data }] }
  • Backwards compatible: plain string returns still work
  • Images converted to attachments for model vision input

Changes

  • Add ToolContentPart and ToolResult types to plugin
  • Add content normalization helper with validation (max 10 images, ~5MB each, PNG/JPEG/GIF/WebP)
  • Handle rich content in tool registry
  • Add attachments field to tool.execute.after hook
  • Add 8 unit tests with 100% coverage on content module
  • Update docs with usage examples and limits

Verification

  • All 8 content tests pass: bun test test/tool/registry.test.ts --test-name-pattern "tool.content"
  • 100% function and line coverage on content.ts

Fixes #9539

Custom tools can now return images alongside text using a rich content format:

  return {
    content: [
      { type: "text", text: "Screenshot captured" },
      { type: "image", mimeType: "image/png", data: base64 }
    ]
  }

- Add ToolContentPart and ToolResult types to plugin
- Add content normalization helper with validation
- Handle rich content in tool registry (backwards compatible)
- Add attachments field to tool.execute.after hook
- Add unit tests with 100% coverage on content module
- Update docs with usage examples and limits

Limits: max 10 images, ~5MB per image, PNG/JPEG/GIF/WebP only.

Fixes anomalyco#9539
@github-actions
Copy link
Contributor

The following comment was made by an LLM, it may be inaccurate:

No duplicate PRs found

@leeweisern
Copy link
Author

leeweisern commented Jan 20, 2026

@rekram1-node Could you help add the discussion and enhancement labels? This is a new feature that extends the custom tool API to support image returns.

@rekram1-node
Copy link
Collaborator

this seems overcomplicated, we already have the attachments notion I think someone else has a similar pr that accomplishes this

@leeweisern
Copy link
Author

this seems overcomplicated, we already have the attachments notion I think someone else has a similar pr that accomplishes this

you mean it can support returning images in custom tool?

@nathanl
Copy link

nathanl commented Jan 20, 2026

I think this PR would solve a problem I'm having. I'm having Claude (via opencode) write Playwright tests, but when a test fails, Claude can't see the screenshots in the trace. I've written a command-line tool that supports commands like pwtrace screenshot --step 1 --base64 path/to/trace, so Claude is now able to get the base64 image data from any step, but it can't "see" what's in the image as it can if I manually drag in a file to upload. I want to take myself out of that loop so that Claude can iterate on failing tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: support images in custom tool responses

3 participants