Skip to content

Pass only the required Kafka consumer properties to the spout lag monitor - #8988

Open
rzo1 wants to merge 1 commit into
masterfrom
fix/spout-lag-consumer-property-filter
Open

Pass only the required Kafka consumer properties to the spout lag monitor#8988
rzo1 wants to merge 1 commit into
masterfrom
fix/spout-lag-consumer-property-filter

Conversation

@rzo1

@rzo1 rzo1 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

TopologySpoutLag copied every config.-prefixed key out of the submitted spout json_conf into the properties file handed to the storm-kafka-monitor process.

Only the properties the monitor actually reads are now forwarded, and anything dropped is logged by name so a lag display that stops working can be diagnosed. topics, groupid, bootstrap.servers and security.protocol are unaffected, since they travel via argv. Lag monitoring remains off by default. Adds TopologySpoutLagTest.

@rzo1 rzo1 added this to the 3.1.0 milestone Aug 21, 2026
@rzo1 rzo1 self-assigned this Aug 21, 2026

@reiabreu reiabreu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disclaimer: this comment was generated with the help of an LLM.

Approving — the allowlist approach is sound (forwarding only connection-relevant consumer properties is safe by construction). One compatibility note worth calling out in the release notes: because topology-supplied config.* properties are now filtered, any lag-monitor setup that relied on a topology-provided sasl.jaas.config (or another now-dropped property) to authenticate to the brokers will stop reporting lag after upgrade, until the monitor is given its own credentials on the UI host. It's the correct behavior, just a silent change for those setups.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants