Repository navigation
Fix quadratic-time ReDoS in timestamp (X) format parsing - #1361
Open
kratos0718 wants to merge 1 commit into
Open
kratos0718 wants to merge 1 commit into
kratos0718 wants to merge 1 commit into
Conversation
_TIMESTAMP_RE used two independently-greedy \d+ groups either side of an optional ".", so non-matching digit-heavy input (e.g. a crafted Unix-timestamp string with no valid split point) made the regex engine try every possible split before failing, which is quadratic in the input length. A single ~50k character string could tie up a call to arrow.get(s, "X") for seconds to minutes - a realistic exposure for any service parsing timestamps from unsanitized input. Wrapped the fractional part as a single non-capturing optional group instead: there's only one way to consume the "." once found, removing the ambiguity. As a side effect this also fixes single-digit timestamps (e.g. "0", the epoch) being rejected, since the old pattern required two separate \d+ matches even without a decimal point.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1361 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 10 10
Lines 2315 2315
Branches 358 358
=========================================
Hits 2315 2315 ☔ View full report in Codecov by Harness. |
Author
|
The one red check ( |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Checklist
pytest— 1905 passed, 99.93% coverage).pre-commit run— isort, pyupgrade, black, flake8, mypy all clean).masterbranch.Description of Changes
Closes: #1353
_TIMESTAMP_RE(^\-?\d+\.?\d+$) used two independently-greedy\d+groups either side of an optional".". For input with no valid split point between the two groups (e.g. a long run of digits with no decimal point that still fails to match for some other reason, or simply enough digits that the backtracking search itself dominates), the regex engine tries every possible split before giving up — quadratic in the input length. Verified independently: at n=1000/5000/10000 digits the old regex took 0.003s/0.065s/0.25s; growth continues quadratically past that (the issue's own report measured ~15s at 40k digits). A service callingarrow.get(user_string, "X")on unsanitized input (a common pattern for webhook/query-param timestamps) is exposed to this with no auth needed.Fix: wrap the fractional part as a single non-capturing optional group (
^\-?\d+(?:\.\d+)?$). There's only one way to consume the.once it's found, so there's no ambiguous split to backtrack over. Verified this stays linear (all under ~20ms) at up to 1,000,000 characters, well past where the old regex was already unusable.One side effect worth calling out: the old regex also rejected single-digit timestamps (e.g.
"0", i.e. the Unix epoch itself) because it required two separate\d+matches even with no decimal point at all. The new pattern correctly accepts these. I checked the existing test suite for anything relying on that rejection — nothing does (all existing timestamp tests use real-world multi-digit values fromtime.time()) — and added an explicit test for it alongside the ReDoS regression test.Test Plan
pytest(full suite): 1905 passed, coverage 99.93% (gate is 99%)pre-commit run --files arrow/parser.py tests/test_parser.py: all hooks pass (isort, pyupgrade, black, flake8, mypy)test_timestamp_does_not_backtrack_quadratically: times matching a 200k-character adversarial string against_TIMESTAMP_REand asserts it completes in under 5s (generous margin; actual time is a few ms) — this test fails on the pre-fix regex well before the timeouttest_timestampwith the single-digit case ("0","-5")