Repository navigation
feat(python): add ElastiCacheIAMAuthTokenManager (auto-refresh) - #15
Merged
Merged
Conversation
…auth tokens The stateless ElastiCacheIAMAuthTokenProvider signs on every call. The manager wraps it with a thread-safe cache and refreshes the token on a background daemon thread `refresh_after` seconds after signing (default 300), so clients that reconnect often or run connection pools always get a valid token without signing per call. Behaviour - Config validated and region resolved eagerly at construction; first token signed lazily on the first get_token()/get_credentials(). - A token is served for 880s after signing: the 900s lifetime minus a 20s margin for clock skew and the AUTH round trip. refresh_after must be below 880. Expiry is tracked on the wall clock, since the server judges X-Amz-Date against its own wall clock and a monotonic clock stands still through a host suspend. - get_token() returns the cached token while it is valid. A token found past its refresh due time (missed timer, suspended process) is returned immediately and refreshed in the background; callers holding a valid token never wait. - Single flight: concurrent callers with no valid token share one signing cycle and receive the same result or the same failure (concurrent.futures.Future). - Retry: up to 8 attempts, exponential backoff from 100ms with a nominal 5s cap, +/-20% jitter. Every error is retried, including ConfigurationError: on an instance role a transient IMDS failure surfaces as "no credentials". - Failure semantics: while the cached token is valid a failed cycle is invisible to callers, logs a warning, and schedules another cycle after the maximum backoff (get_token() does not restart it early); with no token the last error is raised; with an expired token TokenRefreshError is raised with the last error as __cause__. - on_token_changed(token) is called synchronously outside the lock for each newly installed token; a callback failure is logged and the token kept. - close() (or the context manager) cancels the timer, wakes a sleeping retry, drops the cached secret, and makes later calls raise TokenRefreshError. Idempotent. A scheduling failure never loses a signed token. Also - TokenRefreshError (not a ToolkitInputError) added to errors and exported with the manager and DEFAULT_REFRESH_AFTER_SECONDS. - _CachedToken keeps the bearer token out of its repr. - Tests: shared session fake and fixtures moved to tests/helpers.py and tests/conftest.py; 53 manager tests with injected clock, scheduler, sleep and jitter so no real timers run, plus real-thread single-flight tests. - README "Caching and background refresh" section and CHANGELOG entry.
mwaisaida-13
force-pushed
the
python/auto-refresh
branch
from
September 21, 2026 11:52
fc9572a to
a5c6810
Compare
mwaisaida-13
marked this pull request as ready for review
September 21, 2026 11:52
alexr-bq
reviewed
Sep 21, 2026
| try: | ||
| self._on_token_changed(token) | ||
| except Exception: # a callback failure must not discard the token | ||
| _logger.warning("on_token_changed callback raised", exc_info=True) |
Collaborator
There was a problem hiding this comment.
Because the callback receives the bearer token, its exception message or chained exception may contain that token. Should emit a sanitized message
Collaborator
Author
There was a problem hiding this comment.
good catch, will remove
Collaborator
Author
There was a problem hiding this comment.
Fixed in e407df0: the log line now carries only the exception type, no message or traceback. Test asserts the callback's message text is absent from the log.
edlng
reviewed
Sep 21, 2026
- Callback failures are logged by exception type only: the callback received the token, so its message or traceback might contain it. - Re-check expiry under the lock before serving the still-valid token at the end of a failed cycle; a waiter that joined because its token expired must not get it back if the clock crossed expires_at while the lock was released. - Complete and clear the signing flight before calling on_token_changed, so a timer that fires during a slow callback starts its own cycle instead of joining the old one and skipping the reschedule. Reuse decisions now compare token identity rather than issued_at. - Add refresh_token(): sign a replacement immediately for a client whose AUTH was rejected with a token the manager still considers valid; raises TokenRefreshError if none could be signed. - Tests: daemon workers with liveness asserts; shared-flight tests hold the owner open until every waiter has joined (join-counting Future); re-entrant callback test runs bounded; real _wait interrupted while blocked; real-timer end-to-end test with a callback slower than refresh_after; regression tests for the two races above.
ogasarah
approved these changes
Sep 22, 2026
alexr-bq
approved these changes
Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
ElastiCacheIAMAuthTokenManager: a thread-safe cache around the statelessElastiCacheIAMAuthTokenProviderthat refreshes the token on a background daemonthread
refresh_afterseconds after signing (default 300).ElastiCacheIAMAuthTokenProviderElastiCacheIAMAuthTokenManagerget_token()refresh_afterafter signingon_token_changedclose()/ context managerBehaviour
clock skew and the
AUTHround trip);refresh_aftermust be below 880. Expiry istracked on the wall clock, since the server judges
X-Amz-Dateagainst its ownwall clock and a monotonic clock stands still through a host suspend.
get_token()returns the cached token while valid; a token past its refresh duetime (missed timer) is returned immediately and refreshed in the background.
receive the same result or failure (
concurrent.futures.Future).±20 % jitter. Every error is retried, including
ConfigurationError: on aninstance role a transient IMDS failure surfaces as "no credentials".
is logged and another cycle is scheduled after the max backoff (
get_token()doesnot restart it early). With no token the last error is raised; with an expired
token
TokenRefreshErroris raised with the last error as__cause__.on_token_changed(token)is called synchronously, outside the lock, for each newtoken; a callback failure is logged and the token kept.
close()cancels the timer, wakes a sleeping retry, drops the cached secret;later calls raise
TokenRefreshError. Idempotent.get_credentials()returns(user_id, token)— the no-arg provider hook.Also in this change
TokenRefreshError(not aToolkitInputError), exported with the manager andDEFAULT_REFRESH_AFTER_SECONDS.tests/helpers.py/tests/conftest.py; 53 managertests with injected clock/scheduler/sleep/jitter (no real timers), plus
real-thread single-flight tests (success and failure sharing).
Unreleasedentry.Verification
python scripts/python_dev.py ci— lint clean, 128 passed, wheel + sdist built,twine check --strictpassed. 128 passed on 3.10 as well.all findings addressed with regression tests.
GETs, 18 background refreshes at an exact 300 s cadence. A second 90-minute soak on this exact revision is running now; result to follow in a comment.Notes for reviewers
noqas, each with a reason on the line:PLR0913(7 keyword-only args),PYI034(typing.Selfis 3.11+; we support 3.10),BLE001×3 (retry any error,scheduling-failure guard, timer-thread catch).
WARNINGper exhausted refresh cycle and peron_token_changedexception. Note thatbotocore.authat DEBUG level logs theSigV4 signature, i.e. the token — a README security note for that is a follow-up.