Legacy encoding [BUG]

[coolcoldice-cmyk]

Problem (one or two sentences)

I know this one is odd one. One legacy application that I mantain uses CP 852 encoding. Files must be saved as such, and with VS Code I am editing and opening fine - Roo Code also works fine with that specific language, and Claude is delivering just fine, except it messes up the document and does not save in expected encoding.

I've written in system prompt CRITICAL: This project uses CP 852 (Code Page 852) encoding. All code generation and file modifications must preserve CP 852 characters (č, ć, ž, š, đ). Do not convert files to UTF-8;

It attempts, but fails.

Context (who is affected and when)

Just for few of us in legacy applications...

Reproduction steps

Latest versions of VS Code and Roo Code; Windows 11

Expected result

Should save in CP852

Actual result

Saves in Unicode

Variations tried (optional)

No response

App Version

3.43.0

API Provider (optional)

None

Model Used (optional)

No response

Roo Code Task Links (optional)

No response

Relevant logs or errors (optional)

[roomote]

I've analyzed this issue and here's my proposed implementation plan:

Problem Analysis

Roo Code currently hardcodes UTF-8 encoding for all file read/write operations. When editing files with non-UTF-8 encodings (like CP 852), the content is read as UTF-8, modified, and written back as UTF-8, which corrupts special characters (č, ć, ž, š, đ).

Affected code locations:

1. src/integrations/editor/DiffViewProvider.ts - Lines 70, 81, 656 use fs.readFile/writeFile with utf-8
2. src/core/tools/ApplyPatchTool.ts - Lines 62, 393 use fs.readFile/writeFile with utf8

Proposed Solution

I propose implementing encoding detection and preservation:

1. Add encoding detection utility

• Create a new utility src/utils/fileEncoding.ts that:
  • Uses VSCode's files.encoding setting as a baseline
  • Optionally uses the jschardet library for automatic encoding detection
  • Provides functions to detect, read, and write files with proper encoding

2. Modify DiffViewProvider

• Store the detected encoding when opening a file
• Convert content to UTF-8 internally for processing
• Convert back to original encoding when saving

3. Modify ApplyPatchTool

• Use the encoding utility to read files with correct encoding
• Write files back with the original encoding preserved

4. Add dependencies

• Add iconv-lite for encoding conversion (already commonly used in Node.js ecosystem)
• Optionally add jschardet for encoding detection

Technical Considerations

• AI models always output UTF-8 text, so we need to convert the model's output to the target encoding when saving
• We should respect VSCode's files.autoGuessEncoding and files.encoding settings
• Need to handle edge cases where encoding conversion might fail

Questions

Before proceeding, I'd like to clarify:

1. Should encoding detection be automatic (using jschardet), or should we only rely on VSCode's configured file encodings?
2. Should there be a fallback behavior if encoding conversion fails (e.g., save as UTF-8 with a warning)?
3. Are there specific file patterns or workspace settings that should override the detected encoding?

Please mention @roomote in a comment saying that you approve this plan (or answer the questions above), and I'll proceed with the implementation.