Skip to content

[SVLS-9883] feat: add DD_LOG_LEVEL_BY_TARGET to set log levels per target #1413

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lym953 wants to merge 14 commits into
base: yiming.luo/cold-start-duration-logs
Choose a base branch
from
Open

[SVLS-9883] feat: add DD_LOG_LEVEL_BY_TARGET to set log levels per target #1413

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
fdb365f
[SVLS-9883] Add DD_LOG_DEBUG_TARGETS to enable debug logs per target
lym953 Oct 7, 2026
246a2e9
[SVLS-9883] Use trace_flush_duration in the env filter test
lym953 Oct 7, 2026
981ee9d
[SVLS-9883] Replace DD_LOG_DEBUG_TARGETS with DD_LOG_LEVEL_BY_TARGET
lym953 Oct 7, 2026
16f683a
[SVLS-9883] Document that target names must not be prefixes of each o…
lym953 Oct 7, 2026
225e496
[SVLS-9883] Accept filter directives in DD_LOG_LEVEL
lym953 Oct 7, 2026
0e706b3
[SVLS-9883] Bump serverless-components for directive-aware DD_LOG_LEVEL
lym953 Oct 7, 2026
35ea498
[SVLS-9883] Bump serverless-components to the latest commit of #178
lym953 Oct 7, 2026
015e783
[SVLS-9883] Bump serverless-components to the rebased #178
lym953 Oct 8, 2026
bf67a21
[SVLS-9883] Go back to DD_LOG_LEVEL_BY_TARGET
lym953 Oct 9, 2026
6f1eab7
[SVLS-9883] Accept only target=level entries in DD_LOG_LEVEL_BY_TARGET
lym953 Oct 9, 2026
e47df3c
[SVLS-9883] Use datadog-serverless-logging for DD_LOG_LEVEL_BY_TARGET
lym953 Oct 9, 2026
c2aa611
[SVLS-9883] Drop the DD_LOG_LEVEL_BY_TARGET notes from the target con…
lym953 Oct 9, 2026
f9fd99a
[SVLS-9883] Bump serverless-components to the merge commit of #180
lym953 Oct 13, 2026
fcb610e
[SVLS-9883] Bring back the note about target names next to the target…
lym953 Oct 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 13 additions & 4 deletions bottlecap/Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 5 additions & 4 deletions bottlecap/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,10 +95,11 @@ libdd-trace-normalization = "3.0.1"
libdd-trace-obfuscation = { version = "6.0.0", default-features = false }
libdd-trace-stats = { version = "7.0.0", default-features = false }
datadog-opentelemetry = { version = "0.5.2", default-features = false, features = ["_unstable_propagation"] }
dogstatsd = { git = "https://github.com/DataDog/serverless-components", rev = "8156757470d0843a99fc5d19809ae021cd2c7178", default-features = false }
datadog-fips = { git = "https://github.com/DataDog/serverless-components", rev = "8156757470d0843a99fc5d19809ae021cd2c7178", default-features = false }
datadog-agent-config = { git = "https://github.com/DataDog/serverless-components", rev = "8156757470d0843a99fc5d19809ae021cd2c7178", default-features = false }
datadog-agent-trace-sampler = { git = "https://github.com/DataDog/serverless-components", rev = "8156757470d0843a99fc5d19809ae021cd2c7178", default-features = false }
dogstatsd = { git = "https://github.com/DataDog/serverless-components", rev = "9538fc7f08811e8912a7ce32c2ee7b887f90e9e2", default-features = false }
datadog-fips = { git = "https://github.com/DataDog/serverless-components", rev = "9538fc7f08811e8912a7ce32c2ee7b887f90e9e2", default-features = false }
datadog-agent-config = { git = "https://github.com/DataDog/serverless-components", rev = "9538fc7f08811e8912a7ce32c2ee7b887f90e9e2", default-features = false }
datadog-agent-trace-sampler = { git = "https://github.com/DataDog/serverless-components", rev = "9538fc7f08811e8912a7ce32c2ee7b887f90e9e2", default-features = false }
datadog-serverless-logging = { git = "https://github.com/DataDog/serverless-components", rev = "9538fc7f08811e8912a7ce32c2ee7b887f90e9e2", default-features = false }
libddwaf = { version = "1.28.1", git = "https://github.com/DataDog/libddwaf-rust", rev = "d1534a158d976bd4f747bf9fcc58e0712d2d17fc", default-features = false, features = ["serde"] }

[dev-dependencies]
Expand Down
1 change: 1 addition & 0 deletions bottlecap/LICENSE-3rdparty.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ datadog-agent-trace-sampler,https://github.com/DataDog/serverless-components,Apa
datadog-fips,https://github.com/DataDog/serverless-components,Apache-2.0,The datadog-fips Authors
datadog-opentelemetry,https://github.com/DataDog/dd-trace-rs/tree/main/datadog-opentelemetry,Apache-2.0,Datadog Inc. <info@datadoghq.com>
datadog-protos,https://github.com/DataDog/saluki,Apache-2.0,The datadog-protos Authors
datadog-serverless-logging,https://github.com/DataDog/serverless-components,Apache-2.0,The datadog-serverless-logging Authors
ddsketch-agent,https://github.com/DataDog/saluki,Apache-2.0,The ddsketch-agent Authors
defmt,https://github.com/knurling-rs/defmt,MIT OR Apache-2.0,The Knurling-rs developers
defmt-macros,https://github.com/knurling-rs/defmt,MIT OR Apache-2.0,The Knurling-rs developers
Expand Down
19 changes: 12 additions & 7 deletions bottlecap/src/bin/bottlecap/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ use bottlecap::{
},
};
use datadog_fips::reqwest_adapter::create_reqwest_client_builder;
use datadog_serverless_logging::{LEVEL_BY_TARGET_ENV_VAR, build_env_filter};
use decrypt::{resolve_additional_endpoints_secrets, resolve_secrets};
use dogstatsd::{
aggregator::{
Expand All @@ -103,7 +104,6 @@ use tokio::time::Instant;
use tokio::{sync::Mutex as TokioMutex, sync::mpsc::Sender};
use tokio_util::sync::CancellationToken;
use tracing::{debug, error, warn};
use tracing_subscriber::EnvFilter;
use ustr::Ustr;

fn resolve_dsm_service(
Expand Down Expand Up @@ -239,13 +239,15 @@ fn enable_logging_subsystem() {
)
.unwrap_or(LogLevel::Info);

let env_filter = format!(
"h2=off,hyper=off,reqwest=off,rustls=off,datadog-trace-mini-agent=off,{log_level:?}",
);
let (env_filter, invalid_entries) = build_env_filter(

@clifordshelton clifordshelton Oct 9, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The base filter turns off h2, hyper, reqwest and rustls for a reason, and with this change a DD_LOG_LEVEL_BY_TARGET entry can turn them back on. The extension reads its own stdout and sends it to Datadog as logs (see the "Be careful adding logs inside the logs pipeline" note in AGENTS.md). These HTTP crates log on every request, and that includes the requests the extension makes to send logs. So with hyper=debug, every flush creates more log lines, which get sent in the next flush, which creates more lines, and so on. That adds a lot of noise to the customer's CloudWatch and Datadog logs, and it costs them money.

Since this is an environment variable customers can set, I'd keep those four crates locked to off. You can skip any entry for them and add it to invalid_entries, so the user gets the same warning as for a typo. The cleanest place for this is probably the crate in serverless-components#180, for example by letting the caller pass a list of targets that can't be changed. That would mean making the change before #180 merges. The overrides_base_filter test that sets h2=debug would then change to check that the entry is rejected.

&format!(
"h2=off,hyper=off,reqwest=off,rustls=off,datadog-trace-mini-agent=off,{log_level:?}"
),
&std::env::var(LEVEL_BY_TARGET_ENV_VAR).unwrap_or_default(),
)
.expect("could not parse log level in configuration");
let subscriber = tracing_subscriber::fmt::Subscriber::builder()
.with_env_filter(
EnvFilter::try_new(env_filter).expect("could not parse log level in configuration"),
)
.with_env_filter(env_filter)
.with_level(true)
.with_thread_names(false)
.with_thread_ids(false)
Expand All @@ -257,6 +259,9 @@ fn enable_logging_subsystem() {
.finish();
tracing::subscriber::set_global_default(subscriber).expect("setting default subscriber failed");

if !invalid_entries.is_empty() {
warn!("Ignoring invalid {LEVEL_BY_TARGET_ENV_VAR} entries: {invalid_entries:?}");
}
debug!("Logging subsystem enabled");
}

Expand Down
3 changes: 3 additions & 0 deletions bottlecap/src/logger.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ use tracing_subscriber::fmt::{
};
use tracing_subscriber::registry::LookupSpan;

// `DD_LOG_LEVEL_BY_TARGET` matches a target by text prefix, so no target name may start with
// another one: an entry for `trace_flush` would also match `trace_flush_duration`.

/// Tracing target for the duration of each cold start step.
pub const COLD_START_DURATION: &str = "cold_start_duration";

Expand Down
Loading