feat(vscode-lm): re-apply image support with types version fix #11069
Conversation
Re-applies the image support feature from PR #11065 that was reverted, with a fix for the @types/vscode version issue. The fix keeps @types/vscode at ^1.84.0 in package.json (matching engines.vscode) while the pnpm-lock.yaml resolves to version 1.108.1 which provides the LanguageModelDataPart API. This approach: - Maintains compatibility with VS Code 1.84.0+ - Provides TypeScript types for the new LanguageModelDataPart API - Resolves the vsix build failure from the original PR Changes: - Add convertImageToDataPart() for image-to-LanguageModelDataPart conversion - Add checkModelSupportsImages() for model image capability detection - Update useSelectedModel to use models actual supportsImages capability - Add comprehensive tests for image support Relates to #11064, #11065
dosubot
bot
added
size:L
Reviewed the VS Code LM API image support changes. Found one issue regarding consistency between dynamic and static model definitions.
Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues. |
![roomote[bot]](https://avatars.githubusercontent.com/in/1546624?s=60&v=4){kind=small}
| export const IMAGE_CAPABLE_MODEL_PREFIXES = [ | ||
| "gpt", // All GPT models (gpt-4o, gpt-4.1, gpt-5, gpt-5.1, gpt-5.2, gpt-5-mini, gpt-5.1-codex, etc.) | ||
| "claude", // All Claude models (claude-haiku-4.5, claude-opus-4.5, claude-sonnet-4, claude-sonnet-4.5) | ||
| "gemini", // All Gemini models (gemini-2.5-pro, gemini-3-flash-preview, gemini-3-pro-preview) | ||
| "o1", // OpenAI o1 reasoning models | ||
| "o3", // OpenAI o3 reasoning models | ||
| ] | ||
|
|
||
| /** | ||
| * Checks if a model supports image inputs based on its model ID. | ||
| * Uses prefix matching against known image-capable model families. | ||
| * | ||
| * @param _family The model family (unused, kept for API compatibility) | ||
| * @param id The model ID | ||
| * @returns true if the model supports image inputs | ||
| */ | ||
| export function checkModelSupportsImages(_family: string, id: string): boolean { | ||
| const idLower = id.toLowerCase() | ||
| return IMAGE_CAPABLE_MODEL_PREFIXES.some((prefix) => idLower.startsWith(prefix)) | ||
| } |
There was a problem hiding this comment.
The broad prefix matching here creates an inconsistency with the static vscodeLlmModels definitions in packages/types/src/providers/vscode-llm.ts. For example, gpt-3.5-turbo, gpt-4, gpt-4o-mini, o1, and o3-mini all have supportsImages: false in the static definitions, but this function returns true for them because they match the "gpt", "o1", or "o3" prefixes. This means VsCodeLmHandler.getModel() may report supportsImages: true while useSelectedModel (which uses the static definitions) reports supportsImages: false for the same model. Consider either updating the static definitions to match the dynamic behavior, or making this function more granular to exclude older models that don't support images.
Fix it with Roo Code or mention @roomote and request a fix.
There was a problem hiding this comment.
@NaccOll @hannesrudolph what do you think of this?
There was a problem hiding this comment.
@NaccOll @hannesrudolph what do you think of this?
Should use exact matching, and even consider adding a supportImage attribute to the statically defined models in vscode-lm instead of matching on the provider side. That would make adding and maintaining new models much easier.
3 participants
Summary
Re-applies the image support feature for VS Code LM API provider from PR #11065 that was reverted in PR #11068, with a fix for the
@types/vscodeversion mismatch issue.The Problem
The original PR upgraded
@types/vscodefrom^1.84.0to^1.106.0insrc/package.json. This caused a mismatch with theengines.vscodefield which specifies^1.84.0, resulting inpnpm vsixbuild failures.The Solution
This PR keeps
@types/vscodeat^1.84.0inpackage.json(matchingengines.vscode) while thepnpm-lock.yamlresolves to version1.108.1which provides theLanguageModelDataPartAPI needed for image support. This approach:LanguageModelDataPartAPIChanges
convertImageToDataPart()for image-to-LanguageModelDataPart conversioncheckModelSupportsImages()for model image capability detectionTesting
Relates to #11064
Supersedes the reverted #11065
Important
Re-applies image support for VS Code LM API with a fix for
@types/vscodeversion mismatch, ensuring compatibility and adding comprehensive tests.convertImageToDataPart()invscode-lm-format.tsfor image conversion toLanguageModelDataPart.checkModelSupportsImages()invscode-lm.tsfor detecting model image capability.useSelectedModel.tsto use model's actualsupportsImagescapability.vscode-lm.spec.tsandvscode-lm-format.spec.ts.@types/vscodeat^1.84.0inpackage.json.pnpm-lock.yamlto resolve@types/vscodeto version1.108.1.This description was created by
for 7ec9664. You can customize this summary. It will automatically update as commits are pushed.