Repository navigation
Speed up logcat_processor file scanning and polling loops - #1057
Open
xpconanfan wants to merge 3 commits into
Open
xpconanfan wants to merge 3 commits into
xpconanfan wants to merge 3 commits into
Conversation
Behavior-preserving efficiency pass over
mobly/controllers/android_device_lib/logcat_processor.py:
* `_iter_lines` now reads the file in binary mode and derives byte offsets
by summing raw line lengths instead of calling text-mode `f.tell()` twice
per line (an opaque cookie computation that dominated scan time). Lines
are decoded as UTF-8 with `errors='replace'` exactly as before, and the
resulting offsets are the same byte offsets `tail()` already computes, so
positions from forward scans and `tail()` remain interchangeable.
* Filter criteria are normalised once per query instead of once per line:
a new internal `_LineFilter` compiles the pattern and builds the level
sets a single time, and `_TimestampCutoff` pre-parses the `since`
timestamp. `LogLine.matches()` keeps its public signature and now
delegates to `_LineFilter`, so matching semantics are unchanged.
* `LogcatListenerContext._listen_loop` and `wait_for()` keep a single file
handle open for the lifetime of the listen/wait (via a new internal
`_LineReader`) instead of re-opening the file every 50ms/100ms. The
reader lazily opens a file that does not exist yet, re-tries after I/O
errors, and is always closed through a context manager so handles are
not held past the wait. Truncation behaves as before (reads stall until
the file grows past the saved offset). In-order `wait_for` reuses the
same reader across patterns.
* `_parse_timestamp` uses precompiled split regexes.
No public API changes. The old in-order timeout message semantics
(reporting remaining budget) are kept as-is.
Micro-benchmark (Python 3.13, 200k-line / 12.8 MB logcat file, best of 3,
master's module loaded side-by-side via `git show origin/master:...`):
_iter_lines (full scan) 2.394s -> 0.709s (3.4x)
get_lines(level=[E,F], tag=[...]) 2.467s -> 0.738s (3.3x)
get_lines(pattern, since=<timestamp>) 3.505s -> 1.150s (3.0x)
tail(num_lines=50000, level=E) 0.860s -> 0.800s
Adds tests/mobly/controllers/android_device_lib/logcat_processor_test.py
covering: filter/cutoff equivalence against the public API, byte offsets
with multi-byte characters, CRLF endings and invalid UTF-8, tail vs
forward-scan offset consistency, reader handle reuse / lazy open, and
listen/wait_for behaviour including files created after the wait starts.
All behavioural tests in the new file also pass against the master
implementation.
* `_iter_lines`: read in binary and track byte offsets by summing line lengths instead of two text-mode `f.tell()` calls per line (offsets stay compatible with `tail()`). * Normalise filter criteria once per query (`_LineFilter` compiles the pattern / builds level sets) instead of once per line; `LogLine.matches()` keeps its signature and delegates to it. Memoize `_parse_timestamp` so `since=` comparisons don't re-parse the bound per line. * `listen()` / `wait_for()` keep one file handle open (`_LineReader`) instead of re-opening the file every poll. * Fix a race in `listen()`: the start offset was snapshotted inside the listener thread, so a line appended right after `listen()` returned could be missed. Take it in `__enter__` before the thread starts. 200k-line log, best of 3: `_iter_lines` 1.84s -> 0.71s, `get_lines(pattern, since)` 2.94s -> 1.24s. Adds `logcat_processor_test.py` (offsets with multi-byte/CRLF/invalid UTF-8, tail vs scan consistency, filter equivalence, listen/wait_for).
xpconanfan
force-pushed
the
eff-logcat-processor-perf
branch
from
October 6, 2026 20:41
f25cd3b to
41cc778
Compare
adb logcat output is block-buffered, so wait_for()/listen() can observe a line whose tail has not been flushed yet. Consuming it split one log line into two fragments that neither matched the pattern. Polling readers now leave a trailing line without a newline for the next poll; one-shot get_lines() is unchanged.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_iter_lines: read in binary and track byte offsets by summing line lengths instead of two text-modef.tell()calls per line (offsets stay compatible withtail())._LineFiltercompiles the pattern / builds level sets) instead of once per line;LogLine.matches()keeps its signature and delegates to it. Memoize_parse_timestampsosince=comparisons don't re-parse the bound per line.listen()/wait_for()keep one file handle open (_LineReader) instead of re-opening the file every poll.listen(): the start offset was snapshotted inside the listener thread, so a line appended right afterlisten()returned could be missed. Take it in__enter__before the thread starts.200k-line log, best of 3:
_iter_lines1.84s -> 0.71s,get_lines(pattern, since)2.94s -> 1.24s.Adds
logcat_processor_test.py(offsets with multi-byte/CRLF/invalid UTF-8, tail vs scan consistency, filter equivalence, listen/wait_for).Also fixes a latent bug in the polling paths:
adb logcatoutput is block-buffered, sowait_for()/listen()could observe a half-written line and match neither fragment. Polling readers now defer a trailing line without a newline to the next poll; one-shotget_lines()is unchanged.