Add unicode support for string literals

[kwvanderlinde]

Unicode is everywhere nowadays, but the parser is lagging behind. Support for unicode in strings would greatly expand its usefulness.

Unfortunately antlr2 does not directly support unicode, so an update is required.

[kwvanderlinde]

Been testing the new parser out in MapTool and found a couple issues. Will work on a fix.

[Baaaaaz]

Not sure if this is the right place to put this comment (let me know if not!), but the MapTool nightly build seems to have some stricter parsing rules than in the current 1.16.3 release so what was considered parseable/runnable MTScript in 1.16.3 (and before) no longer runs in the nightly build.

If it helps, here are a couple of examples of MTScript which can parse/run in 1.16.3 but not in the current nightly releases (e.g. 20250510), which both error in the chat with:

org.antlr.v4.runtime.misc.ParseCancellationException
Error trace : (new)@{name='(new), location='Dragon, source='token'}

• Example 1:

[json.get("", 0))]

Cause: Unmatched closing bracket

• Example 2:

["x"+"y",]

Cause: Unnecessary comma

---

While I think the improved syntax checking is great, trying to troubleshoot these newly caught parse errors based on the current antlr.v4 error message is a bit tricky! Maybe better as a feature request (in this repo?), but could the error message be more specific such as providing a line number and/or a more meaningful reason for raising the exception for the above examples beyond the current "org.antlr.v4.runtime.misc.ParseCancellationException"?

[kwvanderlinde]

tl;dr: The parser is certainly stricter now, but I don't believe it is worth trying to approximate the previous behaviour.

---

I've toyed with trying to relax the parser to allow trailing characters as in MT 1.17 and before. But this was a really bizarre aspect of the previous parser implementation where it tries to recover from invalid parses instead of reporting them. That results in some really arbitrary rules about which expressions can have trailing input and which cannot. E.g.,

• a || 5,) is allowed (equivalent to a || 5
• a,) is not allowed

And there's other odd inconsistencies related to this, too:

• "a" = 5 is not allowed since we can only assign to identifiers.
• "a"(5) is allowed, because even though we can only call identifiers as functions, the (5) gets ignored as trailing input.

So right now my leaning is to leave the stricter parsing in place as there's no way this was intended.

There is one legitimate difference with the new parser: quotes inside of variable values used to be stripped in some cases, but no longer are. But the old behaviour here is also a bug, as the intention is clearly to strip quotes from string literals, not to strip quotes embedded inside of strings. So I think I'll leave this behaviour in as well.

While I think the improved syntax checking is great, trying to troubleshoot these newly caught parse errors based on the current antlr.v4 error message is a bit tricky! Maybe better as a feature request (in this repo?), but could the error message be more specific such as providing a line number and/or a more meaningful reason for raising the exception for the above examples beyond the current "org.antlr.v4.runtime.misc.ParseCancellationException"?

I agree, this is not ideal and we should handle it better. It's not really a new issue, although it is a new error (I think) and does show up in some different situations. As you say, if we can manage it, it would be good to have both better location information and we should properly handle the error to better communicate the issue to the user.