Skip to content

Fix quadratic-time ReDoS in timestamp (X) format parsing - #1361

Open
kratos0718 wants to merge 1 commit into
arrow-py:masterfrom
kratos0718:fix-timestamp-redos
Open

kratos0718 wants to merge 1 commit into
arrow-py:masterfrom
kratos0718:fix-timestamp-redos

Conversation

@kratos0718

Copy link
Copy Markdown

Pull Request Checklist

  • 🧪 Added tests for changed code.
  • 🛠️ All tests pass when run locally (pytest — 1905 passed, 99.93% coverage).
  • 🧹 All linting checks pass when run locally (pre-commit run — isort, pyupgrade, black, flake8, mypy all clean).
  • 📚 Updated documentation for changed code.
  • ⏩ Code is up-to-date with the master branch.

Description of Changes

Closes: #1353

_TIMESTAMP_RE (^\-?\d+\.?\d+$) used two independently-greedy \d+ groups either side of an optional ".". For input with no valid split point between the two groups (e.g. a long run of digits with no decimal point that still fails to match for some other reason, or simply enough digits that the backtracking search itself dominates), the regex engine tries every possible split before giving up — quadratic in the input length. Verified independently: at n=1000/5000/10000 digits the old regex took 0.003s/0.065s/0.25s; growth continues quadratically past that (the issue's own report measured ~15s at 40k digits). A service calling arrow.get(user_string, "X") on unsanitized input (a common pattern for webhook/query-param timestamps) is exposed to this with no auth needed.

Fix: wrap the fractional part as a single non-capturing optional group (^\-?\d+(?:\.\d+)?$). There's only one way to consume the . once it's found, so there's no ambiguous split to backtrack over. Verified this stays linear (all under ~20ms) at up to 1,000,000 characters, well past where the old regex was already unusable.

One side effect worth calling out: the old regex also rejected single-digit timestamps (e.g. "0", i.e. the Unix epoch itself) because it required two separate \d+ matches even with no decimal point at all. The new pattern correctly accepts these. I checked the existing test suite for anything relying on that rejection — nothing does (all existing timestamp tests use real-world multi-digit values from time.time()) — and added an explicit test for it alongside the ReDoS regression test.

Test Plan

  • pytest (full suite): 1905 passed, coverage 99.93% (gate is 99%)
  • pre-commit run --files arrow/parser.py tests/test_parser.py: all hooks pass (isort, pyupgrade, black, flake8, mypy)
  • Added test_timestamp_does_not_backtrack_quadratically: times matching a 200k-character adversarial string against _TIMESTAMP_RE and asserts it completes in under 5s (generous margin; actual time is a few ms) — this test fails on the pre-fix regex well before the timeout
  • Extended test_timestamp with the single-digit case ("0", "-5")
  • Independently verified (outside the test suite) functional equivalence between the old and new regex across representative inputs (integers, decimals, negatives, trailing/leading dot, empty string), confirming the only behavioral difference is the single-digit fix described above

_TIMESTAMP_RE used two independently-greedy \d+ groups either side of
an optional ".", so non-matching digit-heavy input (e.g. a crafted
Unix-timestamp string with no valid split point) made the regex engine
try every possible split before failing, which is quadratic in the
input length. A single ~50k character string could tie up a call to
arrow.get(s, "X") for seconds to minutes - a realistic exposure for any
service parsing timestamps from unsanitized input.

Wrapped the fractional part as a single non-capturing optional group
instead: there's only one way to consume the "." once found, removing
the ambiguity. As a side effect this also fixes single-digit timestamps
(e.g. "0", the epoch) being rejected, since the old pattern required
two separate \d+ matches even without a decimal point.
@codecov

codecov Bot commented Sep 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (2224255) to head (2415b9d).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff            @@
##            master     #1361   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           10        10           
  Lines         2315      2315           
  Branches       358       358           
=========================================
  Hits          2315      2315           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@kratos0718

Copy link
Copy Markdown
Author

The one red check (windows-latest (pypy-3.11)) is an unrelated CI infra flake, not a real failure: pip's install of build dependencies for pyyaml hit Connection broken: IncompleteRead on that runner, so pytest never actually ran on this job. All 25 other platform/version combinations pass. Happy to push an update if a maintainer would rather see it re-triggered.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Quadratic-time ReDoS in Unix-timestamp (X) format token parsing

1 participant