Repository navigation
fix(ssm): add opt-in session restart after instance reboot - #719
Open
hariom-hashicorp wants to merge 2 commits into
Open
hariom-hashicorp wants to merge 2 commits into
hariom-hashicorp wants to merge 2 commits into
Conversation
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Process cancellation and reboot recovery across both SDK implementations need human confirmation with live AWS acceptance testing.
No open issues 🎉
4 files not reviewed
builder/ebs/builder.hcl2spec.go: Generated filebuilder/ebssurrogate/builder.hcl2spec.go: Generated filebuilder/ebsvolume/builder.hcl2spec.go: Generated filebuilder/instance/builder.hcl2spec.go: Generated file
What changed in this PR
Adds opt-in Session Manager tunnel recovery after instance reboots across four Amazon builders, addressing the stalled-session scenarios in #699 and #289.
Changes:
- Adds configuration and agent-disconnection monitoring to both AWS SDK implementations.
- Adds unit tests and re-enables two reboot acceptance tests.
- Documents configuration, permissions, and reboot behavior.
| File | Description |
|---|---|
| docs-partials/common/RunConfig-not-required.mdx | Documents the reconnect option. |
| docs-partials/builders/aws-session-manager.mdx | Adds reboot guidance and example. |
| docs-partials/builder/common/RunConfig-not-required.mdx | Documents the legacy configuration option. |
| common/step_create_ssm_tunnel.go | Passes reconnect configuration to sessions. |
| common/ssm/session.go | Adds SDK v2 monitoring and process restart. |
| common/ssm/session_test.go | Tests monitoring and session restart behavior. |
| common/run_config.go | Defines the opt-in setting. |
| builder/instance/builder.hcl2spec.go | Exposes the setting in HCL. |
| builder/instance/builder.go | Wires instance-builder configuration. |
| builder/ebsvolume/builder.hcl2spec.go | Exposes the setting in HCL. |
| builder/ebsvolume/builder.go | Wires volume-builder configuration. |
| builder/ebssurrogate/builder.hcl2spec.go | Exposes the setting in HCL. |
| builder/ebssurrogate/builder.go | Wires surrogate-builder configuration. |
| builder/ebs/builder.hcl2spec.go | Exposes the setting in HCL. |
| builder/ebs/builder.go | Wires EBS-builder configuration. |
| builder/ebs/builder_acc_test.go | Enables reconnect in unskipped reboot tests. |
| builder/common/step_create_ssm_tunnel.go | Passes reconnect configuration to legacy sessions. |
| builder/common/ssm/session.go | Adds SDK v1 monitoring and process restart. |
| builder/common/ssm/session_test.go | Tests SDK v1 connection monitoring. |
| builder/common/run_config.go | Defines the legacy opt-in setting. |
| .web-docs/components/builder/instance/README.md | Updates instance-builder documentation. |
| .web-docs/components/builder/ebsvolume/README.md | Updates volume-builder documentation. |
| .web-docs/components/builder/ebssurrogate/README.md | Updates surrogate-builder documentation. |
| .web-docs/components/builder/ebs/README.md | Updates EBS-builder documentation. |
Review effort: Balanced
hariom-hashicorp
force-pushed
the
fix/ssm-reconnect-after-reboot
branch
from
October 21, 2026 05:31
101486f to
8d497bf
Compare
anurag5sh
reviewed
Oct 21, 2026
| - `session_manager_reconnect_on_reboot` (bool) - Start a new Session Manager session when the SSM agent on the instance | ||
| disconnects, for example when a provisioner reboots the instance. AWS | ||
| does not resume a port forwarding session after a reboot, and the | ||
| session-manager-plugin process does not always exit, so without this |
Collaborator
There was a problem hiding this comment.
Do we need to mention this session-manager-plugin in the docs? This can be omitted.
|
|
||
| AWS does not resume a Session Manager port forwarding session after the instance reboots, and the session-manager-plugin process does not always exit when its tunnel stops working. A build that reboots the instance, for example with a `shell` provisioner using `expect_disconnect = true`, can then wait on the dead tunnel until the next provisioner times out. | ||
|
|
||
| Set `session_manager_reconnect_on_reboot = true` to have Packer check the instance's SSM agent every 5 seconds with [GetConnectionStatus](https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_GetConnectionStatus.html). After two consecutive `notconnected` results, Packer stops the current session-manager-plugin process and starts a new session on the same local port once the agent is back. The credentials Packer runs with need the `ssm:GetConnectionStatus` permission. Without it, Packer logs the denied call and keeps the current session, as when the option is not set. |
|
|
||
| const ( | ||
| // How often the SSM agent connection is checked while a session is open. | ||
| agentConnectionCheckInterval = 5 * time.Second |
Collaborator
There was a problem hiding this comment.
Review if these defaults could be useful if given the ability to configure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AWS does not resume a Session Manager port forwarding session after the
instance reboots, and session-manager-plugin does not always exit when
its tunnel dies. Packer only starts a new session after that process
exits, so a build that reboots the instance can wait on a dead tunnel
until the next provisioner times out.
Add session_manager_reconnect_on_reboot (default false). When set,
Packer checks the instance's SSM agent every 5 seconds with
ssm:GetConnectionStatus. After two consecutive "notconnected" results it
stops the session-manager-plugin process and starts a new session on the
same local port. If GetConnectionStatus is denied, Packer logs it and
keeps the current session, as when the option is not set. With the
option unset, no status calls are made and behavior is unchanged.
The option is available in amazon-ebs, amazon-ebsvolume,
amazon-ebssurrogate (SDK v2 session code) and amazon-instance (SDK v1
session code).
Enable the option in TestAccBuilder_EbsSSMRebootProvisioner and
TestAccBuilder_PrivateKeyFileWithReboot and stop skipping them.
Related: #699, #289