[ENHANCEMENT] Image support for VS Code LM API (may require API update) #8123 #11064
Description
Problem (one or two sentences)
GitHub copilot supports uploading images, while RooCode cannot upload images via vscode-lm.
Context (who is affected and when)
Users of the vscode-lm provider in RooCode
Desired behavior (conceptual, not technical)
Model upload is supported when using the vscode-lm provider.
Constraints / preferences (optional)
No response
Request checklist
- I've searched existing Issues and Discussions for duplicates
- This describes a specific problem with clear context and impact
Roo Code Task Links (optional)
No response
Acceptance criteria (optional)
No response
Proposed approach (optional)
- Upgrade the @types/vscode dependency to version 1.106 or higher. This version changes some data structures, which are prerequisites for supporting image uploads.
export class LanguageModelChatMessage {
/**
* Utility to create a new user message.
*
* @param content The content of the message.
* @param name The optional name of a user for the message.
*/
static User(content: string | Array<LanguageModelTextPart | LanguageModelToolResultPart | LanguageModelDataPart>, name?: string): LanguageModelChatMessage;
/**
* Utility to create a new assistant message.
*
* @param content The content of the message.
* @param name The optional name of a user for the message.
*/
static Assistant(content: string | Array<LanguageModelTextPart | LanguageModelToolCallPart | LanguageModelDataPart>, name?: string): LanguageModelChatMessage;
/**
* The role of this message.
*/
role: LanguageModelChatMessageRole;
/**
* A string or heterogeneous array of things that a message can contain as content. Some parts may be message-type
* specific for some models.
*/
content: Array<LanguageModelInputPart>;
/**
* The optional name of a user for this message.
*/
name: string | undefined;
/**
* Create a new user message.
*
* @param role The role of the message.
* @param content The content of the message.
* @param name The optional name of a user for the message.
*/
constructor(role: LanguageModelChatMessageRole, content: string | Array<LanguageModelInputPart>, name?: string);
}
/**
* A language model response part containing arbitrary data. Can be used in {@link LanguageModelChatResponse responses},
* {@link LanguageModelChatMessage chat messages}, {@link LanguageModelToolResult tool results}, and other language model interactions.
*/
export class LanguageModelDataPart {
/**
* Create a new {@linkcode LanguageModelDataPart} for an image.
* @param data Binary image data
* @param mime The MIME type of the image. Common values are `image/png` and `image/jpeg`.
*/
static image(data: Uint8Array, mime: string): LanguageModelDataPart;
/**
* Create a new {@linkcode LanguageModelDataPart} for a json.
*
* *Note* that this function is not expecting "stringified JSON" but
* an object that can be stringified. This function will throw an error
* when the passed value cannot be JSON-stringified.
* @param value A JSON-stringifyable value.
* @param mime Optional MIME type, defaults to `application/json`
*/
static json(value: any, mime?: string): LanguageModelDataPart;
/**
* Create a new {@linkcode LanguageModelDataPart} for text.
*
* *Note* that an UTF-8 encoder is used to create bytes for the string.
* @param value Text data
* @param mime The MIME type if any. Common values are `text/plain` and `text/markdown`.
*/
static text(value: string, mime?: string): LanguageModelDataPart;
/**
* The mime type which determines how the data property is interpreted.
*/
mimeType: string;
/**
* The byte data for this part.
*/
data: Uint8Array;
/**
* Construct a generic data part with the given content.
* @param data The byte data for this part.
* @param mimeType The mime type of the data.
*/
constructor(data: Uint8Array, mimeType: string);
}- Modify vscode-lm and vscode-lm-format according to the new data structure to enable providers to support image uploads.
- Modify useSelectedModel and vscode-llm.ts to enable image upload capabilities for the model.
Trade-offs / risks (optional)
No response