DSC support: Capture logs from containers; retry on startup - #870
Open
mpobrien wants to merge 2 commits into
Open
DSC support: Capture logs from containers; retry on startup#870mpobrien wants to merge 2 commits into
mpobrien wants to merge 2 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds support for capturing Docker Compose container logs to disk when mongodb-runner starts a disaggregated storage backend, so those logs are preserved independently of Docker’s log rotation and (optionally) the parent process lifetime.
Changes:
- Forward
MongoClusterOptions.logDirintoDockerComposeProject.start()for disaggregated storage projects. - Add
logDirsupport toDockerComposeProjectto continuously streamdocker compose logs --followto a file while the project is up. - Add a fallback
dumpLogs()snapshot on teardown if the log follower is not running, and include log settings in serialization.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| packages/mongodb-runner/src/mongocluster.ts | Pass logDir into Docker Compose project startup for disaggregated storage. |
| packages/mongodb-runner/src/docker-compose.ts | Implement continuous log streaming to file, fallback snapshot dumping, and serialization for log capture. |
Suppressed comments (4)
packages/mongodb-runner/src/docker-compose.ts:104
startLogFollower()awaits thespawnevent, but a failed spawn emitserrorand never emitsspawn, which can deadlock startup and also keep the log file descriptor open indefinitely. Racingspawnagainsterrormakes the failure explicit and ensures thefinallyblock can run.
env: { ...process.env, ...env },
detached: true,
},
);
await once(proc, 'spawn');
proc.unref();
packages/mongodb-runner/src/docker-compose.ts:178
dumpLogs()includesthis.projectNameverbatim in the output filename. As withstartLogFollower(), this can contain path separators (viaoptions.projectName/COMPOSE_PROJECT_NAME) and write outside the intended directory. Sanitize it before building the filename.
const outFile = path.join(
logDir,
`docker-compose-${this.projectName}-${new Date()
.toISOString()
.replace(/[^-_a-zA-Z0-9.]/g, '')}.log`,
);
packages/mongodb-runner/src/docker-compose.ts:196
- Like the other docker spawns,
dumpLogs()awaits thespawnevent without handling theerrorevent. Ifdockercan't be spawned, this can hang and leak the opened file descriptor until process exit. Racespawnvserrorso thefinallyblock reliably closes the fd.
const proc = spawn(
'docker',
dockerComposeArgs(this.composeFile, this.projectName, [
'logs',
'--no-color',
'--timestamps',
]),
{
stdio: ['ignore', fd, fd],
env: { ...process.env, ...this.env },
},
);
await once(proc, 'spawn');
const [code] = await once(proc, 'exit');
debug('dumped docker compose logs', { outFile, code });
packages/mongodb-runner/src/docker-compose.ts:223
close()usesthis.logDir !== undefined, which will treat an empty string as enabled and then attempt to dump logs to''(throwing). This should match the truthy check used inMongoServer.start()so an empty string behaves like "unset".
// If the log follower died (or was never started) while the project kept
// running, fall back to a one-off snapshot before teardown destroys the
// container logs.
if (this.logDir !== undefined && !this.isLogFollowerRunning()) {
try {
await this.dumpLogs(this.logDir);
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
43
to
47
| const proc = spawn( | ||
| 'docker', | ||
| ['compose', '-f', composeFile, '-p', projectName, ...args], | ||
| dockerComposeArgs(composeFile, projectName, args), | ||
| { | ||
| stdio: ['inherit', 'pipe', 'pipe'], |
Comment on lines
+83
to
+86
| await fs.mkdir(logDir, { recursive: true }); | ||
| const logFile = path.join(logDir, `docker-compose-${projectName}.log`); | ||
| const fd = openSync(logFile, 'a'); | ||
| try { |
Comment on lines
+145
to
+157
| let logFollowerPid: number | undefined; | ||
| if (options.logDir !== undefined) { | ||
| try { | ||
| ({ pid: logFollowerPid } = await startLogFollower( | ||
| composeFile, | ||
| projectName, | ||
| options.env, | ||
| options.logDir, | ||
| )); | ||
| } catch (err) { | ||
| debug('failed to start docker compose log follower', err); | ||
| } | ||
| } |
Comment on lines
553
to
556
| cluster.dockerComposeProject = await DockerComposeProject.start( | ||
| disaggregatedStorage.composeFile, | ||
| { env: disaggregatedStorage.env }, | ||
| { env: disaggregatedStorage.env, logDir: options.logDir }, | ||
| ); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The log-create flow may need to retry if the SLS containers are still coming up, so this supports retries/retryInterval with sensible defaults.
To help debugging, an additional
logDirparam will also allow capturing the docker container output from SLS.