Skip to content

fix(web): recover the tmux repaint gap above the live screen on the first scroll-up (#814) - #815

Open
HadrienKerlero wants to merge 2 commits into
Ark0N:masterfrom
HadrienKerlero:fix/814-early-history-repull
Open

HadrienKerlero wants to merge 2 commits into
Ark0N:masterfrom
HadrienKerlero:fix/814-early-history-repull

Conversation

@HadrienKerlero

Copy link
Copy Markdown

Fixes #814.

Problem

In a non-shell session (seen with OMP), scrolling up from the live screen shows a hole: recent output just above the screen is missing and the rows jump to older history. tmux still holds those rows. When output outpaces tmux's flush interval, tmux repaints the pane rectangle instead of emitting linefeeds, so rows that were on screen get overwritten and never reach xterm's scrollback. The repair for this (#205, _maybeRefetchFullHistory) only runs when the viewport reaches viewportY === 0. The gap sits near the bottom of the buffer, so the user has to scroll past it to the top before the history is fixed.

What this changes

Frontend only: src/web/public/app.js and src/web/public/terminal-ui.js.

  1. A per-session "live output since last full pull" flag. flushPendingWrites sets _liveOutputSinceFullPull and increments _liveOutputGen[sessionId] for every live chunk it writes, but not while _isLoadingBuffer is set, because a replay cannot open a new gap. A full-history replay clears the flag only if the generation has not changed since its capture could have been taken. The scroll re-pull samples the generation before its fetch, because nothing gates live output while the capture downloads. The tab-switch ?full=1 load samples it at the replay, because it already queues live output during its fetch, and it clears the flag only if it completed without falling back to the tail. Live output newer than the capture therefore keeps the session marked, and the next scroll-up pulls again.
  2. An early trigger in _maybeLoadMoreHistoryOnScroll. If the gesture is upward, started at the live bottom (viewportY >= baseY - 2 - |lines|, which allows for the travel scrollLines already applied), the session is not shell and is marked, and no pull or load is running, it calls _maybeRefetchFullHistory({ anchor: 'bottom' }). This hook is only reached from Codeman's own wheel, touch and smooth-scroll routes. Scrollbar drags and keyboard scrolling do not get the early repair, and still depend on the unchanged top-of-buffer path, as before.
  3. anchor: 'bottom' in _maybeRefetchFullHistory. The existing restore keeps the position measured from the top, which is right when the user is at row 0. Here the user is a few rows above the live screen, so the restore keeps the distance from the bottom instead. While the replay runs, the reset buffer cannot scroll. Wheel and touch moves are recorded in order in _earlyReplayTravel and replayed once the replay ends, each clamped to [0, baseY] as it would have been live, so scrolling down to the bottom and back up does not cancel out. That way the gesture is neither lost nor fed into the top-of-buffer path partway through. If that travel reaches the top, the partial-history notice is still shown.
  4. Cleanup: both new maps are cleared with the other per-session state when a session is removed.

The existing guards still apply unchanged: the 4 s cooldown, the 60 s latch after a useless pull, _replayWouldShrinkBuffer, the select-generation check, detached sessions, and tiles.

Why this shape, and what I rejected

  • The issue suggested comparing baseY with _paneHistoryLines. I did not do that. _paneHistoryLines is recorded at tab switch, so it is stale exactly when it matters, after a burst. A fresh #{history_size} would need a probe round trip, and probably a new server route, for every scroll-up. The dirty flag is free and always current. The cost is that it is a coarser proxy (see below).
  • Trying to detect repaint bursts in the stream. The client sees a PTY byte stream, and nothing in it reliably separates "tmux repainted because it fell behind" from a TUI that redraws in place on purpose, which OMP does all the time. Any heuristic for that would be fragile and hard to test.
  • Showing the "Load full history" notice earlier. That still asks the user to act on something that should not happen. It would also mean detecting the gap, which is the same problem as above.
  • Shell is excluded. Shell captures can be many MB, and shell already gets a bounded 1 MiB window on scroll-to-top. An unbounded pull on every first scroll-up would be a regression there.

Downsides and risks

  • The trigger over-fires by design. Any live write marks the session, not just a burst that actually opened a gap. So in practice, in an active Claude, Codex or OMP session, the first scroll-up from the bottom after new output costs an unbounded ?full=1 capture: tmux capture-pane over up to history-limit lines on the server, plus a full reset and chunked replay in the browser. The cooldown limits this to one pull every 4 s, and the flag limits it to once per batch of output. On a long session this is still far more capture traffic than before, when it happened only at the top of the buffer. On mobile data it also costs bandwidth. I think this is the main thing to review.
  • It is visible. The terminal resets and replays right as the user starts scrolling, not after they scroll to the top. On a large history the replay runs over several animation frames, and the buffer briefly shows a partial rewrite. The restore puts the viewport back at the right distance from the bottom, but it can flicker.
  • Repaint-mode panes pay for a useless pull. Where tmux keeps less history than the browser, the pull is refused by _replayWouldShrinkBuffer and latched for 60 s. That is correct, but it still costs one wasted capture per minute per session on scroll-up. The test for this case checks that the buffer is not reset and the flag stays set.
  • The repair is not guaranteed. The early trigger needs the gesture to start at the bottom. If the user scrolls up a little, stops, new output marks the session, and they scroll further, nothing fires early, and the top-of-buffer path is the fallback as before. Scrollbar and keyboard scrolling never trigger it (see item 2).
  • It can still reset while the user reads. The request starts only near the bottom, but a slow capture of a large history can finish after the user has scrolled further and started reading. The reset and replay then happen under them. Their position is restored, counted from the bottom, but the flicker lands mid-read. A check after the fetch could drop the replay in that case. I did not add one, because it would throw away the capture the user scrolled for.
  • _earlyReplayTravel is one app-level field, not per session. It works because only one pull can run at a time (_fullHistoryRepullInFlight) and the finally always resets it. It is still shared state that a future concurrent path could break.

Tests

New describe block in test/shell-scroll-history-pull.test.ts ("early full-history recovery above live output (#814)"). It covers:

  • the first scroll-up after live output pulls once and keeps the distance from the new bottom
  • no second pull without new output
  • travel during an unfinished replay is replayed in order with a clamp after each move: up, down, past the top, and reversals after hitting the bottom and the top
  • live output that lands while the capture downloads keeps the session marked
  • shell never takes the early path
  • no early pull while loading or already pulling
  • the cooldown still applies
  • a refused (shrinking) pull does not reset the buffer and keeps the flag

The shared makeApp fixture now sets viewportY: 0, because the scroll path reads it. The reversal and download-time tests fail on the previous version of this branch and pass now.

These are VM-loaded unit tests with a mocked xterm. They pin the logic but not real xterm behavior during a reset and replay, nor the real buffer-load queue. They do not cover an early replay cancelled by a tab switch, or the tab-switch path clearing the flag. No browser test covers this path. I did not run npm run test:browser.

Run on this branch (based on master @ e99b42c3):

  • npm test (on the first commit): 757 files, 17130 passed, 13 skipped, 1 failed. The failure is frontend-public-tooling.test.ts > runs the public asset check script. It failed because my worktree lacked the gitignored src/web/public/vendor/* bundles that postinstall builds. With those copied in, the file passes 4/4. It is unrelated to this change.
  • After the review follow-up commit: the 33 scroll, history, flush and selection test files pass (778 tests). I did not rerun the full suite on that commit.
  • npm run lint, format:check, typecheck, check:frontend-syntax, check:frontend-dead, check:browser-excludes: pass on both commits. The pre-push hook passed on the second push.
  • npm run build: passes.

Manual verification: not done

The first commit was built into a local 1.51.0 install (Linux, tmux 3.4, OMP 18.20.0, Zen Browser), but I have no recorded before/after observation that it removes the gap. The follow-up commit has not been run in a browser at all. Please treat the behavior as unverified until someone reproduces #814 and checks that the first scroll-up after an OMP burst shows the recent rows without a jump.

…irst scroll-up (Ark0N#814)

A tmux repaint burst overwrites rows that never reach xterm scrollback,
leaving a hole just above the live screen. The Ark0N#205 re-pull only fired at
viewportY 0, so the user had to scroll past the hole first.

Mark a non-shell session dirty on each live write; the first upward scroll
from the bottom of a dirty session runs the existing full-history re-pull
anchored to the bottom, keeping the viewport's distance from live output and
replaying travel made during the replay. Cooldown, downgrade guard and the
top-of-buffer path are unchanged.
…ve (Ark0N#814)

Review follow-up. The early re-pull sampled the live-output generation after
its fetch, but nothing gates live output while the capture downloads, so
output newer than the capture was overwritten and the session marked clean.
Sample before the fetch.

Travel made during the replay was one signed sum clamped once, so scrolling
to the bottom and back up cancelled out. Record moves in order and clamp
after each, as live scrolling does.
@Ark0N

Ark0N commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Thanks for this, Hadrien, and for the careful write-up. The PR repairs the tmux repaint gap from #814 by re-pulling the full tmux history on the first scroll-up from the live screen, instead of waiting until the scroll reaches the top of the buffer.

The open design question

One decision on #814 is still open, and this PR makes it: whether the history repair may fire before the top of the buffer, and on which signal. That has to be settled before more work goes into the code.

The signal here is any live output. flushPendingWrites marks the session on every chunk it writes (src/web/public/terminal-ui.js:4689), which includes spinner ticks, status-line repaints and the echo of what the user types. The generation check then keeps it marked whenever output arrives during the pull. So in a session that is producing output, every scroll-up that starts at the bottom more than 4 s after the last pull runs GET /api/sessions/:id/terminal?full=1. On the server that is capturePaneBuffer running execSync over up to 100k lines of history, and it blocks the event loop for every client and session while it runs (docs/architecture-invariants.md gives about 0.3 s at 100k lines). In the browser it is a reset and a replay of that payload over several frames, just as the user starts reading. The same doc gives this cost as the reason the pull only runs at the top today ("On-demand rather than automatic because at a 100k history limit the capture can be megabytes").

There are narrower options, and picking one is the open part: keep the top-only behaviour, use a stream signal that only fires on a burst (the reproduction on #814 found that tmux sends a full-screen CSI <rows> S and then repaints when it falls behind), or do a bounded early pull, since the gap sits at the bottom of the buffer. Your notes in the PR description on the cost and the over-firing feed straight into that decision.

Code points, if the any-output trigger is kept

  1. Output that arrives during the download disappears from the screen (src/web/public/app.js:9654 and :9751). The pull opens the buffer load only inside chunkedTerminalWrite, after the fetch. Live output that reaches the browser while the capture downloads is written into the old buffer and then cleared by _resetTerminalForReplay(). The capture doesn't contain it, and the since replay in _finishBufferLoad only covers the queue opened later. Your comment at app.js:9648 keeps the session marked for this case, but the rows are still missing on screen, right above the prompt, which is the Scrollback has a gap right above the live screen until scrolling to the very top triggers a full-history re-pull (OMP session) #814 symptom. I confirmed this with a scratch test that drives the real _maybeRefetchFullHistory, batchTerminalWrite, flushPendingWrites and chunkedTerminalWrite into @xterm/headless. With the change below patched in, the same test keeps the line. The fix is to open the load before the fetch, the way selectSession does (app.js:10195): call _beginBufferLoad() before _fetchTerminalCapture, pass that owner to chunkedTerminalWrite, and on every return that does not replay (no buffer, tab switch, bounded skip, downgrade refusal, catch) call _finishBufferLoad(owner, { flushQueued: true, since: 0 }). Please add a test for the 128 KiB queue cap in _onSessionTerminal (app.js:2564), since a long download in a busy session now queues there.
  2. Run it in a browser. The new tests load the methods in a VM with a mocked xterm and a mocked chunkedTerminalWrite, so they can't see the reset and replay, and point 1 is that kind of problem. CONTRIBUTING asks that changes have been run, so please reproduce Scrollback has a gap right above the live screen until scrolling to the very top triggers a full-history re-pull (OMP session) #814 before and after in a non-shell session (the recipe on the issue: 40 lines printed slowly, then 500 at once) and add a browser test in the style of test/capture-load-window.browser.test.ts.
  3. Clear the mark based on what came back, not on what was requested (src/web/public/app.js:9764 and :10440). When the capture fails, the route still answers full=1, with the byte history and source: 'history', and that carries the same hole. Please key both clears on source === 'mux-full-history'.
  4. Docs. docs/architecture-invariants.md:254 (Full-scrollback replay) still says other modes re-pull "at the TOP" and "on demand rather than automatic", and so does the JSDoc of _maybeRefetchFullHistory (app.js:9586) and _maybeLoadMoreHistoryOnScroll (terminal-ui.js:4053). These change along with whichever trigger is accepted.
  5. Tests for the tab-switch half (src/web/public/app.js:10440). The select path's clear and an early replay cancelled by a tab switch have no tests, as your description says.
  6. A small one: _earlyReplayTravel (src/web/public/terminal-ui.js:4067) isn't tied to its session. After a tab switch supersedes an early replay, moves on the new tab are swallowed until the old replay settles. Storing the session id with the moves, and only pushing while that session is active, fixes it.

Typecheck, lint, format, the frontend syntax and dead-code checks, and the full test suite (17134 tests) pass on this branch, and 9 of the new tests fail on master as intended.

The trigger question is open, so the PR stays as it is for now. The code points above only apply if the any-output trigger is kept.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scrollback has a gap right above the live screen until scrolling to the very top triggers a full-history re-pull (OMP session)

2 participants