Skip to content

Parser improvements - #9016

Open
Jenson3210 wants to merge 4 commits into
openrewrite:mainfrom
Jenson3210:parser-improvements
Open

Jenson3210 wants to merge 4 commits into
openrewrite:mainfrom
Jenson3210:parser-improvements

Conversation

@Jenson3210

@Jenson3210 Jenson3210 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Parser fixes for failures found while analyzing a parse-failure report (org.openrewrite.table.ParseFailures) across ~18 open source repositories.

Docker (rewrite-docker)

  • Whitespace before a comma in JSON arrays was lost (CMD ["a", "b" , "c"]), so the printed output differed from the input. A new CommaPrefix marker stores the whitespace before the comma on the preceding element, and DockerPrinter prints it back. Covers VOLUME, SHELL and the exec forms of CMD/ENTRYPOINT/RUN/HEALTHCHECK/COPY/ADD.
  • = in ADD/COPY paths (ADD https://host/r?filepath=org/x.jar /dest) split the token, and ANTLR error recovery dropped text. sourcePath and destinationPath now accept EQUALS, and tokens touching an = are regrouped into one argument. The lexer is unchanged, so ENV/LABEL/ARG behave as before.
  • DockerParser.java / .interp are regenerated.

JavaScript / TypeScript (rewrite-javascript)

  • Source files are now decoded by decodeSourceBytes: a FF FE BOM is read as UTF-16LE, FE FF as UTF-16BE, and everything else as UTF-8 as before. Previously a UTF-16 file was read as UTF-8 and TypeScript reported diagnostic 1490 ("File appears to be binary").
  • The BOM stays in the text as a leading U+FEFF, the same way a UTF-8 BOM already did, so the existing BOM handling round-trips it.
  • Known limitation: the tree records the charset as utf8 for a UTF-16 file because the charset is detected after decoding. Printing is unaffected.

JSON (rewrite-json)

  • Invalid JSON (YAML saved as .json, template placeholders, a missing closing brace, ...) previously produced either an opaque NullPointerException in JsonParserVisitor or a mangled tree that failed requirePrintEqualsInput.
  • JsonParser now records ANTLR syntax errors and, before the visitor runs, fails the file with a ParseError: Syntax error in <path> at line L:C <msg> (plus a count of further errors).
  • Behavior change: any input with an ANTLR syntax error is now a ParseError; before, some could still yield a document if it happened to print back identically. onError now receives one JsonParsingException per file. The existing suite (comments, single quotes, trailing commas, hex numbers) passes unchanged.

YAML (rewrite-yaml)

  • Tagged block mappings (UserData: !Base64 followed by a newline and Fn::Sub:, !workspace, !!python/object/apply:...) re-printed the first key with extra indentation. The tag suffix was cut with a hard-coded 2-character indent, so it only worked for a mapping nested exactly two deep (a root-level tag threw StringIndexOutOfBoundsException). The suffix now runs up to the last newline before the first key.
  • Helm templates: a new findHelmTemplateEnd() replaces the single-line regex. It handles multi-line {{/* */}} comments, multi-line {{- ... }} actions and ... -}} closers, and braces inside quoted strings (e.g. "${2}"). A {{ it cannot delimit falls back to the old regex. A standalone template line also no longer loses its leading/trailing whitespace, which caused print-idempotence failures.
  • Not addressed: templates that share a line with other content (include ... | nindent), and non-YAML such as Jinja {% %}.

Testing

  • New tests for every fix, written first and confirmed failing before the fix. rewrite-docker: 10 new tests (415 total, 0 failures). rewrite-json: parameterized syntax-error test over 9 inputs plus an empty-file test. rewrite-yaml: 8 tag tests and 13 Helm tests (737 total, 0 failures). rewrite-javascript: encoding.test.ts (UTF-16LE/BE, UTF-8 with and without BOM).
  • Corpus check of rewrite-yaml against 5,809 YAML files containing {{ from the cloned repos: parse + print-identical went from 5,744 to 5,782, non-idempotent from 4 to 0, parse errors from 61 to 27, with no regressions. All 16 tag-bearing files now round-trip.
  • Only each changed module's own tests were run; modules depending on the JSON/YAML parsers were not.

@Jenson3210

Copy link
Copy Markdown
Contributor Author

@timtebeek @greg-at-moderne, I identified these issues while testing a parser on my end, no rush / direct need for it. It's just this ParseFailures cluttered my output.

Feel free to close / cherry pick fixes that are worth picking.

@Jenson3210

Copy link
Copy Markdown
Contributor Author

on the failed pipeline:

  1. The CI run had empty credentials. The job log shows both ORG_GRADLE_PROJECT_codegenomeUsername: and ORG_GRADLE_PROJECT_codegenomePassword: with no value
  2. The codegenome repo needs those credentials and serves jsonrpc.
  3. The version that works isn't on Maven Central. I queried Central directly. jsonrpc-1.0.12.pom returns 200 and jsonrpc-1.3.0.pom returns 404. latest.release therefore resolves to 1.0.12 without codegenome
    access, and JsonRpc.tags(...), which arrived in 1.3.0, doesn't compile. My local runs had the credentials and got 1.3.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

2 participants