[SVLS-9883] feat: add DD_LOG_LEVEL_BY_TARGET to set log levels per target

[lym953]

Stacked on #1412. Review that PR first. The new variable is parsed by the crate datadog-serverless-logging, from DataDog/serverless-components#180.

Problem

DD_LOG_LEVEL=debug turns on debug logs from all parts of the extension. In a test, three cold starts at debug level logged 117 debug lines, when only 33 of them were cold start durations. Too many logs can be noisy for users (either customers or Datadog engineers) who just want to debug cold starts.

This PR

This PR adds DD_LOG_LEVEL_BY_TARGET, a comma-separated list of <target>=<level> entries. Each entry sets the log level of one tracing target, and DD_LOG_LEVEL stays the level of all other targets. Both are environment variables of the Lambda function. For example:

Environment variable	Value
DD_LOG_LEVEL	info
DD_LOG_LEVEL_BY_TARGET	cold_start_duration=debug,dogstatsd=debug

Suppose there are logging statements with target cold_start_duration, like:

debug!(
    target: "cold_start_duration",
    "Datadog Next-Gen Extension ready in {:}ms",
    start_time.elapsed().as_millis().to_string()
);

Then here are some examples (each row is one scenario):

DD_LOG_LEVEL	DD_LOG_LEVEL_BY_TARGET	Logs that you get
info	cold_start_duration=debug	Info logs, plus the cold start step durations from #1412
debug	cold_start_duration=off	All debug logs except the cold start step durations
info	dogstatsd=debug	Info logs, plus debug logs from the dogstatsd crate of serverless-components
not set	cold_start_duration=debug	Info logs, plus the cold start step durations. Info is the default level.

A target can be a custom target that the logging statement sets, a module path such as bottlecap::traces, or a crate name such as dogstatsd. An entry matches every target that starts with its text. The README of datadog-serverless-logging lists all the kinds of targets.

The extension logs a warning for an entry that is not valid, and skips it. Examples are a level with no target such as debug, which would otherwise replace DD_LOG_LEVEL, and an unknown level such as cold_start_duration=verbose.

Why not DD_LOG_LEVEL

The Datadog libraries and tracers in the function also read DD_LOG_LEVEL, and they accept only a plain level. For example, datadog-lambda-python logs "Invalid log level" for any other value, and dd-trace-rs logs a parse error. Support for <target>=<level> entries in DD_LOG_LEVEL would need changes in several repositories. These include tracers such as dd-trace-java and dd-trace-dotnet, which Serverless teams do not own and which run in many environments other than Lambda.

Testing

I built an arm64 layer of this branch at 4a71d023 and deployed it on a Python 3.13 arm64 function. Since then, the branch was rebased onto main, changed one code comment, and moved to the merge commit of #180, which has the same code. For each row, I forced 2 cold starts and counted the extension's log lines in CloudWatch. Each cold start logs 11 cold start duration lines.

DD_LOG_LEVEL	DD_LOG_LEVEL_BY_TARGET	Debug lines	Cold start duration lines	Warnings	ERROR lines
info	cold_start_duration=debug	22	22	0	0
debug	cold_start_duration=off	56	0	0	0
info	debug,cold_start_duration=verbose,cold_start_duration=debug	22	22	2	0
info	dogstatsd=debug	10	0	0	0
not set	cold_start_duration=debug	22	22	0	0

In the third row, the extension ignored debug and cold_start_duration=verbose with a warning, and debug did not turn on the other debug logs. In the fourth row, all the debug messages come from the dogstatsd crate. I did not test on amd64 or FIPS builds.

🤖 Generated with Claude Code

[lym953]

Changing back to draft. Will make the change in serverless-components and make DD_LOG_LEVEL support values like: warn,cold_start_duration=debug,trace_flush_duration=debug.

[lym953]

Publishing this again since the serverless-components PR is published: DataDog/serverless-components#178

[Copilot]

🟡 Changes recommended

The bundled wrapper does not support directive lists, and the dependencies remain pinned to a temporary upstream revision.

2 open findings

• Replace temporary serverless-components pins with final merge commit · New
• Wrapper fails to parse combined DD_LOG_LEVEL directives · New

What changed in this PR

Adds target-specific DD_LOG_LEVEL filtering while preserving default logging behavior.

Changes:

• Parses comma-separated tracing filter directives and reports invalid entries.
• Adds unit coverage for filtering behavior.
• Updates serverless-components dependencies.

File	Description
bottlecap/​src/​logger.rs	Documents the cold-start logging target.
bottlecap/​src/​bin/​bottlecap/​main.rs	Implements and tests directive-based filtering.
bottlecap/​Cargo.toml	Updates shared component revisions.
bottlecap/​Cargo.lock	Locks the updated revisions.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟢 Approval recommended

The implementation matches the documented behavior and includes focused coverage for valid, invalid, and overriding directives.

0 open findings

2 resolved since last review

• Wrapper fails to parse combined DD_LOG_LEVEL directives
• Replace temporary serverless-components pins with final merge commit

🧠 Review effort: Balanced

[lym953]

Converting to draft again. Will move the helper function to a crate in serverless-components, as discussed with @duncanpharvey .

[Copilot]

🟡 Changes recommended

The temporary upstream revision must be replaced with the final merge commit before approval.

1 open finding

• Repin dependencies and regenerate Cargo.lock after upstream merge · New

🧠 Review effort: Balanced

[clifordshelton]

Thanks for moving the filter logic into a shared crate. Keeping it out of DD_LOG_LEVEL makes sense too, since the tracers and libraries also read that variable. I have two inline comments and two broader ones here.

1. Decide whether DD_LOG_LEVEL_BY_TARGET is for customers or for us

This adds a new environment variable that customers can set, but it isn't documented anywhere they'd see it. The README in serverless-components#180 is written for developers. Before we document it, it's worth noticing what the setting depends on. Besides custom targets like cold_start_duration, it accepts our internal Rust module paths, like bottlecap::traces, and crate names with underscores. Those can change any time we move or rename code. If customers start using them, a refactor on our side would quietly change what their setting does, and nobody would notice.

My suggestion is to treat it as a support and debugging tool for now. Only document the custom targets we add on purpose (cold_start_duration, and later ones like trace_flush_duration), since those are names we can keep stable. Module paths still work, but we don't promote them. That way we can add or rename modules without breaking anyone. It would be good to confirm this with the team before merging, since the name and behavior are hard to change once customers rely on them.

2. Bring back the note about target names

An entry matches every target that starts with its text, so trace_flush=debug would also turn on trace_flush_duration. There was a code comment warning about this, but the "Drop the DD_LOG_LEVEL_BY_TARGET notes" commit removed it, and now it's only in the serverless-components README. Whoever adds the next target will be working in logger.rs, not reading that README. It's better to put the note back next to COLD_START_DURATION in logger.rs, so new target names don't accidentally start with an existing one.

[clifordshelton]

The base filter turns off h2, hyper, reqwest and rustls for a reason, and with this change a DD_LOG_LEVEL_BY_TARGET entry can turn them back on. The extension reads its own stdout and sends it to Datadog as logs (see the "Be careful adding logs inside the logs pipeline" note in AGENTS.md). These HTTP crates log on every request, and that includes the requests the extension makes to send logs. So with hyper=debug, every flush creates more log lines, which get sent in the next flush, which creates more lines, and so on. That adds a lot of noise to the customer's CloudWatch and Datadog logs, and it costs them money.

Since this is an environment variable customers can set, I'd keep those four crates locked to off. You can skip any entry for them and add it to invalid_entries, so the user gets the same warning as for a typo. The cleanest place for this is probably the crate in serverless-components#180, for example by letting the caller pass a list of targets that can't be changed. That would mean making the change before #180 merges. The overrides_base_filter test that sets h2=debug would then change to check that the entry is rejected.