feat: apply Studio's indexing requests from the Redpanda topic - #693
Open
MoonBoi9001 wants to merge 7 commits into
Conversation
MoonBoi9001
force-pushed
the
mb9/consume-studio-indexing-requests-into-the-registry
branch
from
August 24, 2026 15:33
ba137d1 to
ed22895
Compare
The Kafka client the dipper uses has no consumer groups, so the broker cannot remember how far a consumer got. A new table records the next offset to fetch per topic partition, written after each record is processed, so a restarted dipper resumes where it left off.
Setting an indexing target does 3 things: upsert the request, emit the request-received event for a new one, and queue a reassessment. That sequence lived inside the admin RPC handler; it moves to a shared function so the upcoming Kafka consumer applies requests identically.
A new background service reads the propose events Studio publishes and applies each through the same path as the admin RPC, so a developer's request in Studio reaches indexer selection with no RPC call. It resumes from offsets stored in the dipper's database and skips bad messages.
CI starts a Redpanda container and 2 test layers use it: producer-to-consumer roundtrips with GZIP batches, and a full service run showing offsets persist after processing and a restarted consumer resumes without re-applying old events. Without a broker the tests skip with a note.
Setting an indexing target commits the row change first, then queues a reassessment job. If that second step failed, retrying the whole record would hit the registry's no-op path (count unchanged) and quietly drop the job; the consumer now retries just the queue push instead.
Review hardening: a topic that gains partitions now restarts the consumer so nothing reads from the new ones unseen; a cursor past a truncated log re-anchors to the nearest live edge instead of replaying everything; wire counts are capped; the consumer now defaults to off.
MoonBoi9001
force-pushed
the
mb9/consume-studio-indexing-requests-into-the-registry
branch
from
August 24, 2026 15:42
ed22895 to
6affafa
Compare
Kafka partitions only ever grow, so a metadata response reporting fewer than we serve is a stale read, not a topology change; it is now logged instead of restarting the whole process. The read itself is bounded to 3 seconds so it cannot stall shutdown, and a 0 connect timeout is rejected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes a developer's indexing request in Studio reach the dipper with no RPC call: a new background service consumes the propose events Studio publishes on Redpanda and applies each through the same shared path as the signed admin RPC, emitting the request-received lifecycle event and queueing indexer selection. It builds on the consumer plumbing in #692. A propose carries a target indexer count per deployment; an unchanged count is a no-op, a changed count triggers reassessment, a count of 0 cancels the request's agreements, and oversized counts are clamped since Kafka records carry no signature.
The Kafka client has no consumer groups, so a new table records the next offset per partition, written only after a record is fully processed: at-least-once, safe because redelivered counts are no-ops. Unreadable messages are logged and skipped; a topic that grows partitions restarts the consumer so the new ones are not silently unread. The section ships disabled: until Studio's schema carries the indexed network field from #692, every request would be skipped.
CI now starts a Redpanda container; a service test proves a restarted consumer resumes from stored offsets without re-applying old events.