fix: let caseInsensitive: false keep a rule case-sensitive

[ybai08]

Issue #, if available: #4887

Description of changes:

Each tokenizer state is compiled to one regexp, so caseInsensitive: true on any rule turns on the i flag for the whole state. A rule with an explicit caseInsensitive: false is matched case-insensitively too, as in the example from the issue:

start: [
    {token: "keyword", regex: /FOO/, caseInsensitive: false},
    {token: "keyword", regex: /BAR/, caseInsensitive: true}
]

On master foo FOO bar BAR gives four keyword tokens. With this change foo is plain text, and FOO, bar and BAR are keywords.

How it works: when a state has a caseInsensitive rule and also a rule with caseInsensitive: false, the state is compiled without the i flag. The other rules are rewritten by the new Tokenizer#caseInsensitiveRegex so that their letters match either case, as suggested in the issue thread:

• a letter outside a class becomes [aA];
• a class gets the other case of its letters and ASCII letter ranges, e.g. [a-f0-9] becomes [A-Fa-f0-9]. The letters go at the start of the class, or at the end when the class starts with -. When it starts and ends with -, they go in a separate class next to it ((?:[-a-]|[A])), or in a negative lookahead for a negated class;
• escapes (\w, \x41, A, \u{61}, \p{Lu} with unicode, \cJ, \k<name>, backreferences) and group syntax ((?:, (?=, (?<=, (?<name>) are left alone.

The rewrite is used for the state regexp and for the splitter regexp of rules with token arrays.

Compatibility: a state that has no caseInsensitive: false rule is compiled exactly as before, so caseInsensitive on one rule still applies to the rules that don't set it (a test covers this). Across the 200 bundled modes, no state has both a caseInsensitive rule and a caseInsensitive: false rule. The only mode that uses caseInsensitive: false (logtalk) has no caseInsensitive: true rules, so no bundled mode tokenizes differently.

Limits of the rewrite: backreferences still match case-sensitively, and ranges only get the other case for ASCII letters, plus single non-ASCII letters whose other case is one character.

types/ace-modules.d.ts was regenerated with npm run update-types for the new method. ace.d.ts, ace-modules.d.ts and ace-extensions.d.ts did not change.

Tests: three tests in src/tokenizer_test.js:

• the example from the issue, plus a token-array rule (which uses the splitter regexp) in the same state;
• the existing behaviour for a state without a caseInsensitive: false rule;
• unit cases for caseInsensitiveRegex.

What was run:

• npx mocha src/tokenizer_test.js: 8 passing. With src/tokenizer.js reset to master, the issue-example test fails on its token assertion (foo comes out as a keyword) and the helper test fails because the method doesn't exist yet.
• npm test: 1498 passing, 3 pending, 2 failing. The same 2 tests fail on master in the same environment (1495 passing): mouse_handler_test.js "multiselect" and vim_test.js "copy".
• npx eslint src/tokenizer.js src/tokenizer_test.js, npm run update-types (only the types/ace-modules.d.ts addition above), npm run typecheck, and the strict tsc ... ace.d.ts check from CI.
• Not run: tool/test-npm-package.sh and coverage upload.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Pull Request Checklist:

• No backwards incompatible changes were made to Ace's public interface which is defined through the main typings file (ace.d.ts) and its references:
  • https://github.com/ajaxorg/ace/blob/master/ace.d.ts
  • https://github.com/ajaxorg/ace/blob/master/ace-modes.d.ts
  • https://github.com/ajaxorg/ace/blob/master/ace-extensions.d.ts

Open kitchen-sink @ a3f779626bcb366f9f7591d6af48dda253f174b2