Track `Span` for other tokens on parsing stage

[gerau]

Project

compiler

Describe the feature

Currently, the compiler parser discards Span information for some tokens, such as Identifier, and converts it directly to str. This is perfectly normal for the compiler, but for LSP it is important information. Since LSP takes information from the original parser, there are implementations of language server features only for elements that contain Span. This means that there are two ways without changing the original parser:

1. Use the original parser and search for Span analyzing the string;
2. Use another parser that stores the necessary information.

The first option is quite difficult to maintain and quite easy to introduce errors.

The implementation of the second option was presented in #176, which uses tree-sitter, but since it already introduces fourth parser, it will add complexity to support and maintenance. There is also an option to use the pest grammar used here, but as far as I understand, this can be considered as writing another parser.

So, it would be great if we could add Span support for tokens that are directly converted to str. If that introduces performance issues, it could be optional.

Or if you have a better idea on how to use existing parsers for this task, I would love to hear it!

[uncomputable]

The chumsky parser is pretty good. I was considering to move from pest to chumsky when I was still actively maintaining the project. Here is an example project similar to SimplicityHL that uses chumsky.