Skip to content

Report a refused connection once it has persisted - #183

Merged
CodeDrivenMitch merged 2 commits into
mainfrom
fix/quieter-auth-failure-logging-af5
Sep 23, 2026
Merged

CodeDrivenMitch merged 2 commits into
mainfrom
fix/quieter-auth-failure-logging-af5

Conversation

@CodeDrivenMitch

Copy link
Copy Markdown
Collaborator

Ports #181 to the AF5 line.

The problem

The client announced an authentication failure on the first refusal, at ERROR, and repeated it on every attempt:

ERROR Could not connect to Axoniq Platform due to connection error: ...

During a GKE node rotation our gateway was replaced and, for 2.3 seconds, Axon Server still routed ValidateAccessToken to the dead pod. Inspector turned that into a refusal, and every connected customer application logged errors telling operators to check credentials that were perfectly fine — for something the retry loop resolved on its own, in under three seconds.

What changes

retrieveSettings no longer logs. It maps the error and leaves the decision to connectSafely, which now asks whether a failure is worth reporting:

internal fun shouldReport(refusedCredentials: Boolean, hasEverConnected: Boolean, retryCount: Int): Boolean {
    val firstEverAttempt = refusedCredentials && !hasEverConnected && retryCount == 1
    return firstEverAttempt || (retryCount > 0 && retryCount % RETRIES_BETWEEN_REPORTS == 0)
}
Situation When it speaks
Bad token at startup immediately, then every ~10 min
Refusal after it had been working attempt 10 (~4 min), then every ~10 min
Platform unreachable, any time attempt 10, then every ~10 min

A refusal on the very first attempt an application ever makes is said at once: nothing has ever worked, so a misconfigured token is far likelier than a blip, and whoever is starting the application is watching. Once a connection has been established the same refusal is most likely transient.

isAuthenticationFailure walks the cause chain (depth-bounded — a self-referential cause would otherwise spin) because RSocket wraps the server's error.

Levels

Everything that remains is info. The client is not essential to the application it runs in, and an outage on our side should not trip a customer's alerting.

Differences from #181

  • main already gated at retryCount == 4 then % 10; this unifies both lines on % 10, so the first report lands at attempt 10 rather than 4
  • main's registrar already dropped the authentication_failed WARN in 75093c6, so only the client needed changing

On the server side

Inspector now distinguishes denied from could not ask and, when it cannot reach the gateway, drops the connection instead of sending authentication_failed (axoniq-platform#1105). So the stronger "check the access token" wording here only ever fires on a genuine denial. The two changes are independent — this one stands alone, and already-deployed clients benefit from the server change without it.

Tests

10 new unit tests covering the classifier and the reporting policy, including the exact incident (refused, hasEverConnected = true, retryCount = 1 → silent) and the startup case.

176 of 180 pass. The four failures are AxoniqConsoleRSocketClientToxiproxyIntegrationTest, which fail identically on unmodified main — verified on a clean checkout. Pre-existing and worth a separate look.

The client announced an authentication failure on the very first refusal, at
ERROR, and repeated it on every attempt. A platform that refuses a perfectly
good token for a moment - as ours did for 2.3 seconds during a node rotation -
therefore filled customer logs with errors telling them to check credentials
that were fine, for a condition the retry loop resolved unaided.

retrieveSettings no longer logs; it maps the error and leaves the decision to
connectSafely, which now asks whether a failure is worth reporting at all. A
refusal on the very first attempt an application ever makes is said at once,
since nothing has ever worked and whoever is starting it is usually watching.
After that, a refusal is most likely transient and waits for the periodic
report, as connection failures already did.

Everything that remains is info rather than warn or error: the client is not
essential to the application it runs in, and an outage on our side should not
trip a customer's alerting.

Ports #181 to the AF5 line.
@sonarqubecloud

Copy link
Copy Markdown

@stefanmirkovic
stefanmirkovic self-requested a review September 23, 2026 13:21
@CodeDrivenMitch
CodeDrivenMitch merged commit ce55499 into main Sep 23, 2026
3 checks passed
@CodeDrivenMitch CodeDrivenMitch mentioned this pull request Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants