diff --git a/AGENTS.md b/AGENTS.md index 72e3a05..8ca0799 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ Token Pilot is evolving from a Spring AI usage-tracking starter into a framework-independent Java LLM control and accounting core with optional framework and observability adapters. -Current truth: post-call usage normalization, cost calculation, ledger events, Micrometer publishing, Clock-based monthly budget windows, pure budget decisions, typed missing-pricing policies, pricing snapshots, framework-independent token count results, a UTF-8 byte heuristic estimator, a preflight cost-bound projection, versioned model metadata, conservative context admission, a plain-Java core consumer verification path, framework-independent in-memory atomic reservations, and estimate/actual reconciliation with best-effort accounting events are implemented. Candidate-aware request production and Spring AI lifecycle integration remain 30-day MVP targets, not current capabilities. +Current truth: post-call usage normalization, cost calculation, ledger events, Token Pilot-owned Micrometer control/accounting metrics, Clock-based monthly budget windows, pure budget decisions, typed missing-pricing policies, pricing snapshots, framework-independent token count results, a UTF-8 byte heuristic estimator, a preflight cost-bound projection, versioned model metadata, conservative context admission, a plain-Java core consumer verification path, framework-independent in-memory atomic reservations, and estimate/actual reconciliation with observable best-effort accounting events are implemented. Candidate-aware request production and Spring AI lifecycle integration remain 30-day MVP targets, not current capabilities. Distribution direction: publish a framework-independent core and an optional Spring AI convenience starter from the same repository and release train. The existing starter artifact is `token-pilot-starter`; `token-pilot-spring-ai-starter` is only a target name until a compatibility ADR and module change land. @@ -71,12 +71,12 @@ Token Pilot의 제품 포지션은 framework-independent Java LLM control and ac | Module | Status | Notes | | --- | --- | --- | -| `token-pilot-core` | Basic implementation complete | Domain records, pricing, calculator, registry, ledger manager, pricing snapshots, versioned model catalog, token count results, UTF-8 byte heuristic estimation, preflight cost-bound projection, conservative context admission, and public plain-Java consumer verification | -| `token-pilot-spring-ai` | Basic implementation complete | Spring AI 2.0.0 `UsageExtractor`, `LedgerAdvisor`, pricing snapshot resolution, response usage recording, reconciliation decisions, and legacy provider-boundary BLOCK enforcement | -| `token-pilot-micrometer` | Basic implementation complete | `MetricsOptions`, tag whitelist, and metric metadata exist; metric ownership must be narrowed | -| `token-pilot-budget` | Atomic reservation and reconciliation implemented | Typed monthly keys, Clock/ZoneId windows, safe-upper-bound reservations, commit/release/write-off lifecycle, pending reconciliation liability, estimate/actual token and cost deltas, duplicate callback protection, and framework-independent best-effort accounting events implemented; candidate production and durable stores remain | -| `token-pilot-notification` | Basic implementation complete | Event API and deduplication exist; not yet connected to the full advisor/budget lifecycle | -| `token-pilot-autoconfigure` | Basic implementation complete | Bean registration, property binding, pricing/budget/notification wiring, and `ChatClientBuilderCustomizer` implemented | +| `token-pilot-core` | Basic implementation complete | Domain records, pricing, calculator, registry, ledger manager, pricing snapshots, versioned model catalog, token count results, UTF-8 byte heuristic estimation, preflight cost-bound projection, conservative context admission, decision/pricing-miss listener contracts, and public plain-Java consumer verification | +| `token-pilot-spring-ai` | Basic implementation complete | Spring AI 2.0.0 `UsageExtractor`, `LedgerAdvisor`, pricing snapshot/miss observation, response usage recording, reconciliation decisions, and legacy provider-boundary BLOCK enforcement | +| `token-pilot-micrometer` | Token Pilot-owned metrics implemented | Optional owner-specific publishers cover cost, preflight, reservation, pricing-miss, reconciliation, listener-failure, and notification outcomes; legacy `ai.token.*` publishing is opt-in | +| `token-pilot-budget` | Atomic reservation and reconciliation implemented | Typed monthly keys, safe-upper-bound reservations, reconciliation, duplicate callback protection, bounded listener-failure observation, and framework-independent best-effort accounting events implemented; candidate production and durable stores remain | +| `token-pilot-notification` | Basic implementation complete | Event API, deduplication, and handler success/failure/dedup lifecycle observation exist; full atomic advisor/budget connection remains #48 | +| `token-pilot-autoconfigure` | Basic implementation complete | Bean registration, property binding, owner-specific metric listener wiring, pricing/budget/notification wiring, and `ChatClientBuilderCustomizer` implemented | | `token-pilot-starter` | Basic implementation complete | Thin final user entrypoint that brings runtime modules together | | `token-pilot-sample-app` | Basic E2E complete | Direct ledger metrics, budget, and fake Spring AI advisor E2E implemented | @@ -99,13 +99,14 @@ Gradle dependency cleanup has landed. Library modules should not regain app-only Current Micrometer status: -- `MetricsOptions` exists as the small Micrometer options object. -- Default allowed tag keys remain `tenant_id`. -- The existing `MicroCostMetricsPublisher(MeterRegistry)` constructor is preserved. -- Tests cover null/empty tags and multiple allowed tags. -- Metric descriptions and base units should remain stable. -- Current `ai.token.*` metrics may overlap Spring AI Observability; treat renaming/default suppression as an explicit compatibility decision. -- New default metrics should describe Token Pilot-owned cost, preflight, reservation, pricing-miss, and reconciliation outcomes. +- Default Token Pilot metrics describe cost, preflight, reservation, pricing-miss, reconciliation, isolated listener-failure, and notification lifecycle outcomes. +- Default user tag keys are empty, and Token Pilot-owned metrics use only bounded enum/registered identifier values. +- `tokenpilot.cost.total` consumes newly applied actual reservation commits, never legacy ledger estimates or missing-pricing zero values. +- Duplicate/reused reconciliation callbacks do not republish accounting metrics; actual-unavailable transitions publish `reconciliation_required` without zero cost or error samples. +- The existing `MicroCostMetricsPublisher` constructors remain available for direct compatibility. +- Legacy `ai.token.*` metrics are off in starter/autoconfigure unless `token-pilot.metrics.legacy-ai-token-metrics-enabled=true`. +- The legacy metric opt-in, including its raw `model` compatibility tag, is a 0.1.x bridge planned for removal in 0.2.0. +- Micrometer `double` counters are operational telemetry; exact `BigDecimal` ledger values remain the accounting source of truth. ## Distribution Contract @@ -149,6 +150,7 @@ The autoconfigure module provides: - `META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports` - `TokenPilotAutoConfiguration` +- Owner-specific core metrics, budget, Spring AI, notification, and metrics auto-configurations - `TokenPilotProperties` - Pricing property binding - Budget property binding @@ -192,10 +194,14 @@ Default bean graph: | --- | --- | --- | | `CostCalculator` | missing bean | Core cost calculation | | `PricingRegistry` | missing bean | Pricing plan lookup | +| `ModelRegistry` | missing bean | Versioned default model/context metadata | +| `TokenEstimator` | missing bean | UTF-8 byte heuristic token estimate | +| `TokenBudget` | missing bean | Context admission and preflight decision events | | `LedgerManager` | missing bean | Cost and usage recording | | `UsageExtractor` | Spring AI classpath + missing bean | Spring AI response usage extraction | | `LedgerAdvisor` | Spring AI classpath + missing bean | ChatClient advisor | -| `MicroCostMetricsPublisher` | Micrometer classpath + `token-pilot.metrics.enabled` | Cost/token metrics listener | +| owner-specific Token Pilot metric publishers | Micrometer + owner contract classpath + `token-pilot.metrics.enabled` | Low-cardinality control/accounting metrics | +| `MicroCostMetricsPublisher` | Micrometer + explicit legacy flag | Compatibility `ai.token.*` metrics listener | | `BudgetStateStore` | `token-pilot.budget.enabled` + missing bean | Default in-memory budget state | | `BudgetEvaluator` | `token-pilot.budget.enabled` + missing bean | Default budget evaluator | | `ChatClientBuilderCustomizer` | Spring AI classpath + `LedgerAdvisor` bean | Adds advisor to ChatClient builders | @@ -251,9 +257,8 @@ token-pilot: COMPLETION: 0.00060 metrics: enabled: true - tag-whitelist: - - tenant_id - - model + legacy-ai-token-metrics-enabled: false + tag-whitelist: [] budget: enabled: false policy-id: default-monthly @@ -285,7 +290,7 @@ Test-only E2E endpoint: - `GET /test/token-pilot/chat`: exercises the Spring AI `ChatClient` advisor path with a fake/mock provider or documented real provider setup. -The direct ledger E2E test verifies that `/actuator/prometheus` contains token-pilot metrics after a ledger event is recorded. The fake ChatClient E2E test verifies that Spring AI `ChatClient` calls flow through `LedgerAdvisor` into token-pilot metrics without requiring a real provider API key. +The default starter E2E verifies a Token Budget preflight metric while legacy `ai.token.*` meters and sensitive tags remain absent. The fake ChatClient E2E verifies the Spring AI advisor path, explicit legacy compatibility mode, and a bounded pricing-miss metric without requiring a real provider API key. ## Maven Publishing Direction @@ -330,8 +335,10 @@ The active checklist is in `docs/30_DAY_MVP_REPORT.md`; detailed long-term works - Spring AI usage extraction converts map/JSON-compatible native usage objects into the normalized core model. Real-provider compatibility fixtures remain required because provider and Spring AI usage shapes can change independently. - The legacy provider boundary blocks an already-exhausted budget decision before provider invocation. Its candidate-free `STATUS` input is a regression guard, not admission evidence; the flow remains check-then-add and is not connected to the new atomic reservation lifecycle until #39. - In-memory reservation reconciliation uses the reservation-time pricing snapshot, accepts only provider-reported or provider-derived actual usage, moves estimate liability atomically between active, pending, and committed totals, and skips cost calculation for exact duplicate callbacks. Legacy reservations without pricing/token metadata have an explicit cost-only settlement path; new reservations should use the usage-based API. Spring AI callback integration remains #39. -- Accounting listeners run synchronously after the bucket lock is released. Runtime listener failures do not roll back a committed transition, stop later listeners, or trigger redelivery on duplicate callbacks, but delivery remains best-effort at-most-once without a durable outbox; failure observation remains #40. -- Current Micrometer `ai.token.*` metrics may duplicate Spring AI Observability; preserve compatibility while deciding default suppression or replacement. +- Accounting listeners run synchronously after the bucket lock is released. Runtime listener failures do not roll back a committed transition, stop later listeners, or trigger redelivery on duplicate callbacks; bounded failure events feed `tokenpilot.listener.failures`, but delivery remains best-effort at-most-once without a durable outbox. +- Legacy `LedgerListener` runtime failures are likewise isolated from ledger/provider results and later listeners; JVM `Error`s still propagate. +- Compatibility cost-only reservation commits do not carry token/model correlation and therefore do not publish accounting metrics; new reservations should use usage-based reconciliation. +- Legacy Micrometer `ai.token.*` metrics may duplicate Spring AI Observability and are therefore disabled by default; their explicit migration opt-in still permits application-provided tag values with caller-owned cardinality. - The verified Spring AI 2.0.0 path is synchronous `ChatClient` usage recording with a fake provider. Streaming cancellation and reconciliation remain outside the current compatibility guarantee. - The repository, README, JReleaser configuration, and every published module POM use the MIT License. `verifyPublicationMetadata` guards this release contract and ensures the sample app is not published. - Sample app E2E uses a fake Spring AI `ChatModel`; real provider API behavior is not yet verified. @@ -356,6 +363,12 @@ Verify the fixed Java/Spring compatibility matrix and generated core-only consum ./gradlew verifyCompatibilityMatrix verifyCoreConsumer ``` +Verify the published Micrometer artifact with optional owners absent/composed and core autoconfiguration without optional adapters: + +```bash +./gradlew verifyMicrometerConsumer +``` + Run sample app after implementation work: ```bash @@ -402,6 +415,13 @@ Stage and deploy a Central release: ## Update History +### 2026-08-25 + +- Added owner-specific Micrometer publishers for Token Pilot cost, preflight, atomic reservation, pricing-miss, reconciliation, listener-failure, and notification lifecycle outcomes with bounded default tags. +- Connected framework-independent decision/lifecycle listener contracts while preserving duplicate-safe applied accounting semantics and observable best-effort listener failure isolation. +- Disabled legacy `ai.token.*` metrics and the default user tag whitelist in starter/autoconfigure; retained explicit compatibility opt-in and existing direct publisher constructors. +- Added published-artifact consumer checks for Micrometer with optional owners both absent and explicitly composed. + ### 2026-08-22 - Restricted usage-based reservation reconciliation to provider-reported or provider-derived usage so local and heuristic estimates cannot be committed as actual spend. diff --git a/README.md b/README.md index 28de07d..aa1efd1 100644 --- a/README.md +++ b/README.md @@ -57,6 +57,65 @@ atomic budget reservation/idempotency foundation. The MVP is extending this foundation with provider-connected preflight control and actual usage reconciliation. +## Micrometer metrics + +When a `MeterRegistry` is available, the Spring Boot starter publishes +TokenPilot-owned, low-cardinality metrics for control and accounting outcomes: + +| Metric | Tags | Meaning | +| --- | --- | --- | +| `tokenpilot.cost.total` | `currency` | Newly committed actual cost from usage-based reconciliation | +| `tokenpilot.preflight.requests` | `decision`, `reason` | Context admission decisions | +| `tokenpilot.budget.reservations` | `state` | Atomic reservation results | +| `tokenpilot.reconciliation.error.tokens` | `direction` | Absolute estimate/actual token error | +| `tokenpilot.reconciliation.outcomes` | `outcome`, `reason` | Applied reconciliation outcomes | +| `tokenpilot.pricing.missing` | `policy` | Missing pricing observed at the provider boundary | +| `tokenpilot.listener.failures` | `listener`, `phase` | Isolated accounting-listener failures | +| `tokenpilot.notification.events` | `outcome`, `threshold` | Notification delivery and deduplication outcomes | + +The default user-tag whitelist is empty. The metrics above never include raw +model, tenant, user, request, reservation, or idempotency identifiers. Their +tag values come from bounded domain enums or registered currency codes. + +```yaml +token-pilot: + metrics: + enabled: true + tag-whitelist: [] + legacy-ai-token-metrics-enabled: false +``` + +The former `ai.token.*` meters are disabled by default because Spring AI +Observability may already publish standard token telemetry. Set +`token-pilot.metrics.legacy-ai-token-metrics-enabled=true` to opt in during +migration. This is a 0.1.x compatibility bridge, including the legacy raw +`model` tag, and is planned for removal in 0.2.0. Migrate dashboards to Spring +AI token telemetry and the `tokenpilot.*` control/accounting meters before +then. `tag-whitelist` applies only to that legacy path and limits keys, not the +cardinality of application-provided values. Existing direct +`MicroCostMetricsPublisher` constructors retain their legacy `tenant_id` +allowlist behavior; the starter default remains empty. + +Accounting metrics consume newly applied reservation transitions, so reused +callbacks do not add cost twice and unavailable actual usage is recorded as +`reconciliation_required`, not as zero cost or zero error. Listener delivery +is synchronous, best-effort, and at-most-once without a durable outbox. +Micrometer counters use `double` internally and are operational telemetry, not +the monetary source of truth; the ledger's `BigDecimal` values remain +authoritative. The legacy cost-only commit methods cannot carry token/model +correlation and do not emit these accounting metrics; new reservations should +use the usage-based reconciliation API. + +`LedgerListener` and other optional observer `RuntimeException`s are isolated: +they do not change ledger/provider results and later listeners still run. +JVM `Error`s are not swallowed. + +For direct autoconfiguration composition, use +`TokenPilotBudgetPolicyFactory.from(properties)` instead of the former +`TokenPilotProperties.toBudgetPolicy()`. Keeping the budget return type out of +the shared properties class allows autoconfiguration to start when the optional +budget module is absent. + ## Framework-independent core Applications that do not use Spring can depend on `token-pilot-core` alone: diff --git a/build.gradle b/build.gradle index b0a3f97..20afbf5 100644 --- a/build.gradle +++ b/build.gradle @@ -13,7 +13,7 @@ plugins { def publishedModuleDescriptions = [ 'token-pilot-core': 'Framework-independent LLM usage normalization, pricing, cost calculation, and ledger contracts.', 'token-pilot-spring-ai': 'Optional Spring AI adapter for recording provider-reported LLM usage through Token Pilot.', - 'token-pilot-micrometer': 'Optional Micrometer publisher for Token Pilot cost and usage accounting events.', + 'token-pilot-micrometer': 'Optional Micrometer publishers for Token Pilot control, cost, budget, reconciliation, and notification events.', 'token-pilot-budget': 'Budget policy evaluation and in-memory budget state components for Token Pilot.', 'token-pilot-notification': 'Budget notification events, handlers, and deduplication components for Token Pilot.', 'token-pilot-autoconfigure': 'Spring Boot auto-configuration for Token Pilot core and optional adapters.', @@ -371,6 +371,37 @@ tasks.register('verifyPublicationMetadata') { ) requirePomValue(moduleName, 'SCM URL', pom.scm.url.text(), 'https://github.com/tokenpliot/tokenpilot') } + + def micrometerPomFile = project(':token-pilot-micrometer') + .layout.buildDirectory + .file('publications/mavenJava/pom-default.xml') + .get() + .asFile + def micrometerPom = new XmlSlurper(false, false).parse(micrometerPomFile) + def micrometerDependencies = micrometerPom.dependencies.dependency.collect { + "${it.groupId.text()}:${it.artifactId.text()}:${it.scope.text()}".toString() + } as Set + if (!micrometerDependencies.contains( + 'cloud.token-pilot:token-pilot-core:compile' + ) || !micrometerDependencies.contains( + 'io.micrometer:micrometer-core:compile' + )) { + throw new GradleException( + 'token-pilot-micrometer POM must expose core and micrometer-core as compile dependencies.' + ) + } + def forbiddenMicrometerOwners = micrometerPom.dependencies.dependency.findAll { + it.groupId.text() == 'cloud.token-pilot' && + it.artifactId.text() in [ + 'token-pilot-budget', + 'token-pilot-notification' + ] + } + if (!forbiddenMicrometerOwners.isEmpty()) { + throw new GradleException( + 'token-pilot-micrometer POM must keep budget and notification owners optional.' + ) + } } } @@ -737,7 +768,291 @@ tasks.register('verifyCoreConsumer', Exec) { commandLine rootProject.file('gradlew').absolutePath, '--no-daemon', 'run' } +def micrometerConsumerDirectory = layout.buildDirectory.dir( + 'compatibility/micrometer-consumer' +) + +tasks.register('prepareMicrometerConsumer') { + group = 'verification' + description = 'Generates external consumers for optional Micrometer owners and core-only autoconfiguration.' + dependsOn ':token-pilot-core:publishMavenJavaPublicationToStagingRepository' + dependsOn ':token-pilot-autoconfigure:publishMavenJavaPublicationToStagingRepository' + dependsOn ':token-pilot-micrometer:publishMavenJavaPublicationToStagingRepository' + dependsOn ':token-pilot-budget:publishMavenJavaPublicationToStagingRepository' + dependsOn ':token-pilot-notification:publishMavenJavaPublicationToStagingRepository' + outputs.dir(micrometerConsumerDirectory) + outputs.upToDateWhen { false } + + doLast { + def consumerDirectory = micrometerConsumerDirectory.get().asFile + delete(consumerDirectory) + + def coreOnlySource = new File( + consumerDirectory, + 'core-only/src/main/java/compatibility' + ) + def allOwnersSource = new File( + consumerDirectory, + 'all-owners/src/main/java/compatibility' + ) + def autoConfigureSource = new File( + consumerDirectory, + 'autoconfigure-core-only/src/main/java/compatibility' + ) + coreOnlySource.mkdirs() + allOwnersSource.mkdirs() + autoConfigureSource.mkdirs() + + new File(consumerDirectory, 'settings.gradle').text = """ +rootProject.name = 'token-pilot-micrometer-consumer' +include 'core-only', 'all-owners', 'autoconfigure-core-only' +""".stripIndent().trim() + System.lineSeparator() + + def stagingRepositories = [ + project(':token-pilot-core'), + project(':token-pilot-autoconfigure'), + project(':token-pilot-micrometer'), + project(':token-pilot-budget'), + project(':token-pilot-notification') + ].collect { moduleProject -> + "maven { url = uri('${moduleProject.layout.buildDirectory.dir('staging-deploy').get().asFile.toURI()}') }" + }.join(System.lineSeparator() + ' ') + + new File(consumerDirectory, 'build.gradle').text = """ +subprojects { + apply plugin: 'application' + + configurations.configureEach { + resolutionStrategy.cacheChangingModulesFor 0, 'seconds' + } + + repositories { + ${stagingRepositories} + mavenCentral() + } + + java { + toolchain { + languageVersion = JavaLanguageVersion.of(${compatibilityMatrix.java}) + } + } + + tasks.withType(JavaCompile).configureEach { + options.release = ${compatibilityMatrix.java} + } +} +""".stripIndent().trim() + System.lineSeparator() + + new File(consumerDirectory, 'core-only/build.gradle').text = """ +dependencies { + implementation 'cloud.token-pilot:token-pilot-micrometer:${project.version}' +} + +application { + mainClass = 'compatibility.MicrometerCoreConsumer' +} + +tasks.register('verifyOptionalOwnersAbsent') { + doLast { + def forbidden = configurations.runtimeClasspath.incoming.resolutionResult + .allComponents + .findAll { component -> + component.moduleVersion?.group == 'cloud.token-pilot' && + component.moduleVersion?.name in [ + 'token-pilot-budget', + 'token-pilot-notification' + ] + } + if (!forbidden.isEmpty()) { + throw new GradleException('Optional owners leaked into micrometer runtime: ' + forbidden) + } + } +} + +tasks.named('run') { + dependsOn tasks.named('verifyOptionalOwnersAbsent') +} +""".stripIndent().trim() + System.lineSeparator() + + new File(coreOnlySource, 'MicrometerCoreConsumer.java').text = """ +package compatibility; + +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; +import io.tokenpilot.core.domain.MissingPricingPolicy; +import io.tokenpilot.core.domain.PricingMissingEvent; +import io.tokenpilot.core.domain.PricingResolution; +import io.tokenpilot.micrometer.internal.CoreMetricsPublisher; +import io.tokenpilot.micrometer.internal.LedgerMicrometerComponents; +import java.lang.reflect.Method; +import java.util.Arrays; + +public final class MicrometerCoreConsumer { + private MicrometerCoreConsumer() { + } + + public static void main(String[] args) throws Exception { + for (Method method : LedgerMicrometerComponents.class.getDeclaredMethods()) { + String signature = method.toGenericString(); + if (signature.contains("io.tokenpilot.budget") + || signature.contains("io.tokenpilot.notification")) { + throw new IllegalStateException("Optional owner leaked into core factory method table"); + } + } + for (String optionalType : Arrays.asList( + "io.tokenpilot.budget.ReservationAccountingListener", + "io.tokenpilot.notification.BudgetNotificationLifecycleListener")) { + try { + Class.forName(optionalType); + throw new IllegalStateException("Optional owner is unexpectedly present: " + optionalType); + } catch (ClassNotFoundException expected) { + // Expected for the core + micrometer-only consumer. + } + } + + var registry = new SimpleMeterRegistry(); + CoreMetricsPublisher publisher = + LedgerMicrometerComponents.coreMetricsPublisher(registry); + publisher.onPricingMissing(new PricingMissingEvent( + MissingPricingPolicy.FAIL_CLOSED, + PricingResolution.MISSING_PLAN + )); + double count = registry.get("tokenpilot.pricing.missing") + .tag("policy", "fail_closed") + .counter() + .count(); + if (count != 1.0d) { + throw new IllegalStateException("Core Micrometer publisher verification failed"); + } + System.out.println("token-pilot-micrometer core-only consumer OK"); + } +} +""".stripIndent().trim() + System.lineSeparator() + + new File(consumerDirectory, 'all-owners/build.gradle').text = """ +dependencies { + implementation 'cloud.token-pilot:token-pilot-micrometer:${project.version}' + implementation 'cloud.token-pilot:token-pilot-budget:${project.version}' + implementation 'cloud.token-pilot:token-pilot-notification:${project.version}' +} + +application { + mainClass = 'compatibility.MicrometerOwnerConsumer' +} +""".stripIndent().trim() + System.lineSeparator() + + new File(allOwnersSource, 'MicrometerOwnerConsumer.java').text = """ +package compatibility; + +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; +import io.tokenpilot.budget.ReservationAccountingListener; +import io.tokenpilot.micrometer.internal.BudgetMicrometerComponents; +import io.tokenpilot.micrometer.internal.NotificationMicrometerComponents; +import io.tokenpilot.notification.BudgetNotificationLifecycleListener; + +public final class MicrometerOwnerConsumer { + private MicrometerOwnerConsumer() { + } + + public static void main(String[] args) { + var registry = new SimpleMeterRegistry(); + Object budget = BudgetMicrometerComponents.budgetMetricsPublisher(registry); + Object notification = NotificationMicrometerComponents + .notificationMetricsPublisher(registry); + if (!(budget instanceof ReservationAccountingListener) + || !(notification instanceof BudgetNotificationLifecycleListener)) { + throw new IllegalStateException("Owner Micrometer composition verification failed"); + } + System.out.println("token-pilot-micrometer all-owners consumer OK"); + } +} +""".stripIndent().trim() + System.lineSeparator() + + new File( + consumerDirectory, + 'autoconfigure-core-only/build.gradle' + ).text = """ +dependencies { + implementation 'cloud.token-pilot:token-pilot-autoconfigure:${project.version}' + implementation 'org.springframework.boot:spring-boot:${compatibilityMatrix.springBoot}' + implementation 'org.springframework.boot:spring-boot-autoconfigure:${compatibilityMatrix.springBoot}' +} + +application { + mainClass = 'compatibility.AutoConfigureCoreConsumer' +} + +tasks.register('verifyOptionalAdaptersAbsent') { + doLast { + def forbidden = configurations.runtimeClasspath.incoming.resolutionResult + .allComponents + .findAll { component -> + component.moduleVersion?.group == 'cloud.token-pilot' && + component.moduleVersion?.name in [ + 'token-pilot-budget', + 'token-pilot-notification', + 'token-pilot-spring-ai', + 'token-pilot-micrometer', + 'token-pilot-starter' + ] + } + if (!forbidden.isEmpty()) { + throw new GradleException('Optional adapters leaked into core autoconfigure runtime: ' + forbidden) + } + } +} + +tasks.named('run') { + dependsOn tasks.named('verifyOptionalAdaptersAbsent') +} +""".stripIndent().trim() + System.lineSeparator() + + new File(autoConfigureSource, 'AutoConfigureCoreConsumer.java').text = """ +package compatibility; + +import io.tokenpilot.core.PricingRegistry; +import io.tokenpilot.core.TokenBudget; +import io.tokenpilot.core.TokenEstimator; +import org.springframework.boot.SpringApplication; +import org.springframework.boot.WebApplicationType; +import org.springframework.boot.autoconfigure.EnableAutoConfiguration; +import org.springframework.boot.SpringBootConfiguration; +import org.springframework.context.ConfigurableApplicationContext; + +@SpringBootConfiguration +@EnableAutoConfiguration +public class AutoConfigureCoreConsumer { + public static void main(String[] args) { + SpringApplication application = new SpringApplication( + AutoConfigureCoreConsumer.class + ); + application.setWebApplicationType(WebApplicationType.NONE); + application.setLogStartupInfo(false); + try (ConfigurableApplicationContext context = application.run(args)) { + context.getBean(PricingRegistry.class); + context.getBean(TokenBudget.class); + context.getBean(TokenEstimator.class); + } + System.out.println("token-pilot-autoconfigure core-only consumer OK"); + } +} +""".stripIndent().trim() + System.lineSeparator() + } +} + +tasks.register('verifyMicrometerConsumer', Exec) { + group = 'verification' + description = 'Runs Micrometer owner consumers and core-only autoconfiguration without optional adapters.' + dependsOn tasks.named('prepareMicrometerConsumer') + workingDir micrometerConsumerDirectory + commandLine rootProject.file('gradlew').absolutePath, + '--no-daemon', + ':core-only:run', + ':all-owners:run', + ':autoconfigure-core-only:run' +} + tasks.named('check') { dependsOn tasks.named('verifyCompatibilityMatrix') dependsOn tasks.named('verifyCoreConsumer') + dependsOn tasks.named('verifyMicrometerConsumer') } diff --git a/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotAutoConfiguration.java b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotAutoConfiguration.java index e3d2735..fe8f3c1 100644 --- a/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotAutoConfiguration.java +++ b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotAutoConfiguration.java @@ -1,45 +1,28 @@ package io.tokenpilot.autoconfigure; -import io.micrometer.core.instrument.MeterRegistry; -import io.tokenpilot.budget.BudgetEvaluator; -import io.tokenpilot.budget.BudgetStateStore; -import io.tokenpilot.budget.internal.LedgerBudgetComponents; import io.tokenpilot.core.CostCalculator; import io.tokenpilot.core.LedgerListener; import io.tokenpilot.core.LedgerManager; +import io.tokenpilot.core.ModelRegistry; +import io.tokenpilot.core.PreflightDecisionListener; import io.tokenpilot.core.PricingEvaluator; import io.tokenpilot.core.PricingProvider; import io.tokenpilot.core.PricingRegistry; -import io.tokenpilot.core.domain.MissingPricingPolicy; +import io.tokenpilot.core.TokenBudget; +import io.tokenpilot.core.TokenEstimator; +import io.tokenpilot.core.CoreComponents; import io.tokenpilot.core.internal.LedgerComponents; -import io.tokenpilot.micrometer.internal.LedgerMicrometerComponents; -import io.tokenpilot.springai.LedgerAdvisor; -import io.tokenpilot.springai.UsageExtractor; -import io.tokenpilot.springai.internal.LedgerSpringAiComponents; -import org.springframework.ai.chat.client.ChatClient; import org.springframework.beans.factory.ObjectProvider; import org.springframework.boot.autoconfigure.AutoConfiguration; -import org.springframework.boot.autoconfigure.condition.ConditionalOnBean; -import org.springframework.boot.autoconfigure.condition.ConditionalOnClass; import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; import org.springframework.boot.context.properties.EnableConfigurationProperties; import org.springframework.context.annotation.Bean; -import io.tokenpilot.notification.BudgetNotificationHandler; -import io.tokenpilot.notification.BudgetNotificationService; -import io.tokenpilot.notification.InMemoryNotificationStateStore; -import io.tokenpilot.notification.NotificationStateStore; - -import java.time.Clock; /** * Token Pilot 라이브러리의 자동 설정을 담당하는 클래스. */ -@AutoConfiguration(afterName = { - "org.springframework.boot.micrometer.metrics.autoconfigure.MetricsAutoConfiguration", - "org.springframework.boot.micrometer.metrics.autoconfigure.CompositeMeterRegistryAutoConfiguration", - "org.springframework.boot.micrometer.metrics.autoconfigure.export.prometheus.PrometheusMetricsExportAutoConfiguration" -}) +@AutoConfiguration @ConditionalOnProperty(prefix = "token-pilot", name = "enabled", havingValue = "true", matchIfMissing = true) @EnableConfigurationProperties(TokenPilotProperties.class) public class TokenPilotAutoConfiguration { @@ -82,149 +65,52 @@ public PricingEvaluator pricingEvaluator() { } /** - * 비용 기록 및 리스너 관리를 담당하는 LedgerManager를 등록합니다. + * 버전이 고정된 기본 model metadata catalog를 등록합니다. */ @Bean @ConditionalOnMissingBean - public LedgerManager ledgerManager(PricingRegistry pricingRegistry, - CostCalculator costCalculator, - ObjectProvider ledgerListeners) { - return LedgerComponents.defaultLedgerManager( - pricingRegistry, - costCalculator, - ledgerListeners.orderedStream() - .toList() - ); + public ModelRegistry modelRegistry() { + return CoreComponents.defaultModelRegistry(); } /** - * ChatClientResponse에서 토큰 사용량을 추출하는 UsageExtractor를 등록합니다. + * framework-independent UTF-8 휴리스틱 estimator를 등록합니다. */ @Bean @ConditionalOnMissingBean - @ConditionalOnClass({ChatClient.class, LedgerSpringAiComponents.class}) - public UsageExtractor usageExtractor() { - return LedgerSpringAiComponents.defaultUsageExtractor(); + public TokenEstimator tokenEstimator() { + return CoreComponents.utf8ByteHeuristicTokenEstimator(); } /** - * Spring AI 호출 전후로 비용을 측정하고 기록하는 LedgerAdvisor를 등록합니다. + * context admission 결과를 등록된 observer와 함께 발행합니다. */ @Bean @ConditionalOnMissingBean - @ConditionalOnClass({ChatClient.class, LedgerSpringAiComponents.class}) - public LedgerAdvisor ledgerAdvisor( - LedgerManager ledgerManager, - UsageExtractor usageExtractor, - ObjectProvider budgetEvaluator, - ObjectProvider budgetStateStore, - CostCalculator costCalculator, - PricingRegistry pricingRegistry, - PricingEvaluator pricingEvaluator + public TokenBudget tokenBudget( + ModelRegistry modelRegistry, + ObjectProvider decisionListeners ) { - BudgetEvaluator evaluator = budgetEvaluator.getIfAvailable(); - BudgetStateStore stateStore = budgetStateStore.getIfAvailable(); - - if (evaluator != null && stateStore != null) { - return LedgerSpringAiComponents.defaultLedgerAdvisor( - ledgerManager, - usageExtractor, - evaluator, - stateStore, - costCalculator, - pricingRegistry, - pricingEvaluator, - MissingPricingPolicy.FAIL_CLOSED - ); - } - - return LedgerSpringAiComponents.defaultLedgerAdvisor( - ledgerManager, - usageExtractor, - costCalculator, - pricingRegistry, - pricingEvaluator + return CoreComponents.tokenBudget( + modelRegistry, + decisionListeners.orderedStream().toList() ); } /** - * LedgerAdvisor가 빈으로 등록되어 있을 경우, ChatClient.Builder를 위한 커스터마이저를 생성합니다. - */ - @Bean - @ConditionalOnBean(LedgerAdvisor.class) - @ConditionalOnClass(ChatClient.class) - public LedgerChatClientBuilderCustomizer ledgerChatClientBuilderCustomizer(LedgerAdvisor ledgerAdvisor) { - return new LedgerChatClientBuilderCustomizer(ledgerAdvisor); - } - - /** - * Micrometer 메트릭 발행을 위한 리스너를 등록합니다. - */ - @Bean - @ConditionalOnClass({MeterRegistry.class, LedgerMicrometerComponents.class}) - @ConditionalOnBean(MeterRegistry.class) - @ConditionalOnProperty(prefix = "token-pilot.metrics", name = "enabled", havingValue = "true", matchIfMissing = true) - public LedgerListener microCostMetricsPublisher(MeterRegistry meterRegistry, TokenPilotProperties properties) { - return LedgerMicrometerComponents.microCostMetricsPublisher(meterRegistry, properties.getMetrics() - .getTagWhitelist()); - } - - /** - * 예산 관리 모듈이 있을 경우 기본 예산 저장소를 등록합니다. - */ - @Bean - @ConditionalOnMissingBean - @ConditionalOnClass(LedgerBudgetComponents.class) - @ConditionalOnProperty(prefix = "token-pilot.budget", name = "enabled", havingValue = "true") - public BudgetStateStore budgetStateStore() { - return LedgerBudgetComponents.inMemoryBudgetStateStore(); - } - - /** - * 예산 관리 모듈이 있을 경우 예산 평가기를 등록합니다. + * 비용 기록 및 리스너 관리를 담당하는 LedgerManager를 등록합니다. */ @Bean @ConditionalOnMissingBean - @ConditionalOnClass(LedgerBudgetComponents.class) - @ConditionalOnProperty(prefix = "token-pilot.budget", name = "enabled", havingValue = "true") - public BudgetEvaluator budgetEvaluator( - BudgetStateStore budgetStateStore, - TokenPilotProperties properties, - ObjectProvider clock - ) { - return LedgerBudgetComponents.defaultBudgetEvaluator( - budgetStateStore, - properties.toBudgetPolicy(), - clock.getIfAvailable(Clock::systemUTC) + public LedgerManager ledgerManager(PricingRegistry pricingRegistry, + CostCalculator costCalculator, + ObjectProvider ledgerListeners) { + return LedgerComponents.defaultLedgerManager( + pricingRegistry, + costCalculator, + ledgerListeners.orderedStream() + .toList() ); } - /** - * 알림 상태 저장소를 등록합니다. - * - 중복 알림 방지를 위해 window 단위로 상태를 저장 - * - token-pilot.notification.enabled=true 일 때만 등록 - */ - @Bean - @ConditionalOnMissingBean - @ConditionalOnProperty(prefix = "token-pilot.notification", name = "enabled", havingValue = "true") - public NotificationStateStore notificationStateStore() { - return new InMemoryNotificationStateStore(); - } - - /** - * 예산 알림 서비스를 등록합니다. - * - BudgetNotificationHandler 빈이 있을 때만 등록 - * - 없으면 no-op으로 동작 (알림 서비스 자체가 등록되지 않음) - * - token-pilot.notification.enabled=true 일 때만 등록 - */ - @Bean - @ConditionalOnMissingBean - @ConditionalOnBean(BudgetNotificationHandler.class) - @ConditionalOnProperty(prefix = "token-pilot.notification", name = "enabled", havingValue = "true") - public BudgetNotificationService budgetNotificationService( - BudgetNotificationHandler handler, - NotificationStateStore notificationStateStore - ) { - return new BudgetNotificationService(handler, notificationStateStore); - } } diff --git a/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotBudgetAutoConfiguration.java b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotBudgetAutoConfiguration.java new file mode 100644 index 0000000..b5d3d4c --- /dev/null +++ b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotBudgetAutoConfiguration.java @@ -0,0 +1,61 @@ +package io.tokenpilot.autoconfigure; + +import io.tokenpilot.budget.BudgetEvaluator; +import io.tokenpilot.budget.BudgetStateStore; +import io.tokenpilot.budget.ReservationAccountingListener; +import io.tokenpilot.budget.ReservationAccountingListenerErrorHandler; +import io.tokenpilot.budget.internal.LedgerBudgetComponents; +import org.springframework.beans.factory.ObjectProvider; +import org.springframework.boot.autoconfigure.AutoConfiguration; +import org.springframework.boot.autoconfigure.condition.ConditionalOnClass; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.context.annotation.Bean; + +import java.time.Clock; + +/** Optional budget module의 bean graph를 격리하는 자동 설정입니다. */ +@AutoConfiguration(after = { + TokenPilotAutoConfiguration.class, + TokenPilotBudgetMetricsAutoConfiguration.class +}) +@ConditionalOnClass(LedgerBudgetComponents.class) +@ConditionalOnProperty( + prefix = "token-pilot", + name = "enabled", + havingValue = "true", + matchIfMissing = true +) +@ConditionalOnProperty( + prefix = "token-pilot.budget", + name = "enabled", + havingValue = "true" +) +public class TokenPilotBudgetAutoConfiguration { + + @Bean + @ConditionalOnMissingBean + public BudgetStateStore budgetStateStore( + ObjectProvider accountingListeners, + ObjectProvider listenerErrorHandlers + ) { + return LedgerBudgetComponents.inMemoryBudgetStateStore( + () -> accountingListeners.orderedStream().toList(), + () -> listenerErrorHandlers.orderedStream().toList() + ); + } + + @Bean + @ConditionalOnMissingBean + public BudgetEvaluator budgetEvaluator( + BudgetStateStore budgetStateStore, + TokenPilotProperties properties, + ObjectProvider clock + ) { + return LedgerBudgetComponents.defaultBudgetEvaluator( + budgetStateStore, + TokenPilotBudgetPolicyFactory.from(properties), + clock.getIfAvailable(Clock::systemUTC) + ); + } +} diff --git a/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotBudgetMetricsAutoConfiguration.java b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotBudgetMetricsAutoConfiguration.java new file mode 100644 index 0000000..28a5726 --- /dev/null +++ b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotBudgetMetricsAutoConfiguration.java @@ -0,0 +1,43 @@ +package io.tokenpilot.autoconfigure; + +import io.micrometer.core.instrument.MeterRegistry; +import io.tokenpilot.budget.ReservationAccountingListener; +import io.tokenpilot.micrometer.internal.BudgetMicrometerComponents; +import io.tokenpilot.micrometer.internal.BudgetMetricsPublisher; +import org.springframework.boot.autoconfigure.AutoConfiguration; +import org.springframework.boot.autoconfigure.condition.ConditionalOnBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnClass; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.context.annotation.Bean; + +/** Atomic reservation/accounting metrics의 조건부 자동 설정입니다. */ +@AutoConfiguration(after = TokenPilotCoreMetricsAutoConfiguration.class) +@ConditionalOnClass({ + MeterRegistry.class, + ReservationAccountingListener.class, + BudgetMetricsPublisher.class +}) +@ConditionalOnBean(MeterRegistry.class) +@ConditionalOnProperty( + prefix = "token-pilot", + name = "enabled", + havingValue = "true", + matchIfMissing = true +) +@ConditionalOnProperty( + prefix = "token-pilot.metrics", + name = "enabled", + havingValue = "true", + matchIfMissing = true +) +public class TokenPilotBudgetMetricsAutoConfiguration { + + @Bean + @ConditionalOnMissingBean + public BudgetMetricsPublisher tokenPilotBudgetMetricsPublisher( + MeterRegistry meterRegistry + ) { + return BudgetMicrometerComponents.budgetMetricsPublisher(meterRegistry); + } +} diff --git a/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotBudgetPolicyFactory.java b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotBudgetPolicyFactory.java new file mode 100644 index 0000000..0e63186 --- /dev/null +++ b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotBudgetPolicyFactory.java @@ -0,0 +1,40 @@ +package io.tokenpilot.autoconfigure; + +import io.tokenpilot.budget.BudgetPolicy; +import io.tokenpilot.core.domain.Cost; + +import java.time.ZoneId; +import java.util.Currency; +import java.util.Objects; + +/** + * Token Pilot 설정을 optional budget 정책으로 변환하는 공개 진입점입니다. + * + *

budget 타입을 {@link TokenPilotProperties}에서 분리해 budget 모듈이 없는 + * core-only 자동 설정도 class loading에 실패하지 않도록 합니다.

+ */ +public final class TokenPilotBudgetPolicyFactory { + + private TokenPilotBudgetPolicyFactory() { + } + + public static BudgetPolicy from(TokenPilotProperties properties) { + Objects.requireNonNull(properties, "properties must not be null"); + return from(properties.getBudget()); + } + + public static BudgetPolicy from(TokenPilotProperties.BudgetProperties properties) { + Objects.requireNonNull(properties, "properties must not be null"); + return new BudgetPolicy( + properties.getPolicyId(), + properties.getTargetType(), + properties.getTargetTagKey(), + properties.getFallbackTargetId(), + Cost.of( + properties.getMonthlyLimit(), + Currency.getInstance(properties.getCurrency()) + ), + ZoneId.of(properties.getZoneId()) + ); + } +} diff --git a/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotCoreMetricsAutoConfiguration.java b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotCoreMetricsAutoConfiguration.java new file mode 100644 index 0000000..5d09d94 --- /dev/null +++ b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotCoreMetricsAutoConfiguration.java @@ -0,0 +1,65 @@ +package io.tokenpilot.autoconfigure; + +import io.micrometer.core.instrument.MeterRegistry; +import io.tokenpilot.micrometer.MetricsOptions; +import io.tokenpilot.micrometer.internal.CoreMetricsPublisher; +import io.tokenpilot.micrometer.internal.LedgerMicrometerComponents; +import io.tokenpilot.micrometer.internal.MicroCostMetricsPublisher; +import org.springframework.boot.autoconfigure.AutoConfiguration; +import org.springframework.boot.autoconfigure.condition.ConditionalOnBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnClass; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.boot.context.properties.EnableConfigurationProperties; +import org.springframework.context.annotation.Bean; + +/** Core control events와 legacy ledger metrics의 조건부 자동 설정입니다. */ +@AutoConfiguration(after = TokenPilotAutoConfiguration.class, afterName = { + "org.springframework.boot.micrometer.metrics.autoconfigure.MetricsAutoConfiguration", + "org.springframework.boot.micrometer.metrics.autoconfigure.CompositeMeterRegistryAutoConfiguration" +}) +@ConditionalOnClass({MeterRegistry.class, CoreMetricsPublisher.class}) +@ConditionalOnBean(MeterRegistry.class) +@ConditionalOnProperty( + prefix = "token-pilot", + name = "enabled", + havingValue = "true", + matchIfMissing = true +) +@ConditionalOnProperty( + prefix = "token-pilot.metrics", + name = "enabled", + havingValue = "true", + matchIfMissing = true +) +@EnableConfigurationProperties(TokenPilotProperties.class) +public class TokenPilotCoreMetricsAutoConfiguration { + + @Bean + @ConditionalOnMissingBean + public CoreMetricsPublisher tokenPilotCoreMetricsPublisher( + MeterRegistry meterRegistry + ) { + return LedgerMicrometerComponents.coreMetricsPublisher(meterRegistry); + } + + @Bean(name = "microCostMetricsPublisher") + @ConditionalOnMissingBean(MicroCostMetricsPublisher.class) + @ConditionalOnProperty( + prefix = "token-pilot.metrics", + name = "legacy-ai-token-metrics-enabled", + havingValue = "true" + ) + public MicroCostMetricsPublisher microCostMetricsPublisher( + MeterRegistry meterRegistry, + TokenPilotProperties properties + ) { + return LedgerMicrometerComponents.legacyMicroCostMetricsPublisher( + meterRegistry, + new MetricsOptions( + properties.getMetrics().getTagWhitelist(), + true + ) + ); + } +} diff --git a/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotNotificationAutoConfiguration.java b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotNotificationAutoConfiguration.java new file mode 100644 index 0000000..9795fe1 --- /dev/null +++ b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotNotificationAutoConfiguration.java @@ -0,0 +1,55 @@ +package io.tokenpilot.autoconfigure; + +import io.tokenpilot.notification.BudgetNotificationHandler; +import io.tokenpilot.notification.BudgetNotificationLifecycleListener; +import io.tokenpilot.notification.BudgetNotificationService; +import io.tokenpilot.notification.InMemoryNotificationStateStore; +import io.tokenpilot.notification.NotificationStateStore; +import org.springframework.beans.factory.ObjectProvider; +import org.springframework.boot.autoconfigure.AutoConfiguration; +import org.springframework.boot.autoconfigure.condition.ConditionalOnBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnClass; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.context.annotation.Bean; + +/** Optional notification module의 bean graph를 격리하는 자동 설정입니다. */ +@AutoConfiguration(after = { + TokenPilotAutoConfiguration.class, + TokenPilotNotificationMetricsAutoConfiguration.class +}) +@ConditionalOnClass(NotificationStateStore.class) +@ConditionalOnProperty( + prefix = "token-pilot", + name = "enabled", + havingValue = "true", + matchIfMissing = true +) +@ConditionalOnProperty( + prefix = "token-pilot.notification", + name = "enabled", + havingValue = "true" +) +public class TokenPilotNotificationAutoConfiguration { + + @Bean + @ConditionalOnMissingBean + public NotificationStateStore notificationStateStore() { + return new InMemoryNotificationStateStore(); + } + + @Bean + @ConditionalOnMissingBean + @ConditionalOnBean(BudgetNotificationHandler.class) + public BudgetNotificationService budgetNotificationService( + BudgetNotificationHandler handler, + NotificationStateStore notificationStateStore, + ObjectProvider lifecycleListeners + ) { + return new BudgetNotificationService( + handler, + notificationStateStore, + lifecycleListeners.orderedStream().toList() + ); + } +} diff --git a/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotNotificationMetricsAutoConfiguration.java b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotNotificationMetricsAutoConfiguration.java new file mode 100644 index 0000000..0b39766 --- /dev/null +++ b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotNotificationMetricsAutoConfiguration.java @@ -0,0 +1,45 @@ +package io.tokenpilot.autoconfigure; + +import io.micrometer.core.instrument.MeterRegistry; +import io.tokenpilot.micrometer.internal.NotificationMetricsPublisher; +import io.tokenpilot.micrometer.internal.NotificationMicrometerComponents; +import io.tokenpilot.notification.BudgetNotificationLifecycleListener; +import org.springframework.boot.autoconfigure.AutoConfiguration; +import org.springframework.boot.autoconfigure.condition.ConditionalOnBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnClass; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.context.annotation.Bean; + +/** Budget notification lifecycle metrics의 조건부 자동 설정입니다. */ +@AutoConfiguration(after = TokenPilotCoreMetricsAutoConfiguration.class) +@ConditionalOnClass({ + MeterRegistry.class, + BudgetNotificationLifecycleListener.class, + NotificationMetricsPublisher.class +}) +@ConditionalOnBean(MeterRegistry.class) +@ConditionalOnProperty( + prefix = "token-pilot", + name = "enabled", + havingValue = "true", + matchIfMissing = true +) +@ConditionalOnProperty( + prefix = "token-pilot.metrics", + name = "enabled", + havingValue = "true", + matchIfMissing = true +) +public class TokenPilotNotificationMetricsAutoConfiguration { + + @Bean + @ConditionalOnMissingBean + public NotificationMetricsPublisher tokenPilotNotificationMetricsPublisher( + MeterRegistry meterRegistry + ) { + return NotificationMicrometerComponents.notificationMetricsPublisher( + meterRegistry + ); + } +} diff --git a/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotProperties.java b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotProperties.java index 44a7209..3d48bf4 100644 --- a/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotProperties.java +++ b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotProperties.java @@ -1,14 +1,10 @@ package io.tokenpilot.autoconfigure; -import io.tokenpilot.budget.BudgetPolicy; -import io.tokenpilot.core.domain.Cost; import io.tokenpilot.core.domain.PricingPlan; import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.boot.context.properties.NestedConfigurationProperty; -import java.time.ZoneId; import java.util.ArrayList; -import java.util.Currency; import java.util.HashSet; import java.util.List; import java.util.Set; @@ -84,20 +80,6 @@ public List toPricingPlans() { .toList(); } - public BudgetPolicy toBudgetPolicy() { - return new BudgetPolicy( - budget.getPolicyId(), - budget.getTargetType(), - budget.getTargetTagKey(), - budget.getFallbackTargetId(), - Cost.of( - budget.getMonthlyLimit(), - Currency.getInstance(budget.getCurrency()) - ), - ZoneId.of(budget.getZoneId()) - ); - } - public static class PricingProperties { private List plans = new ArrayList<>(); @@ -112,7 +94,8 @@ public void setPlans(List plans) { public static class MetricsProperties { private boolean enabled = true; - private Set tagWhitelist = new HashSet<>(List.of("tenant_id")); + private Set tagWhitelist = new HashSet<>(); + private boolean legacyAiTokenMetricsEnabled = false; public boolean isEnabled() { return enabled; @@ -129,6 +112,16 @@ public Set getTagWhitelist() { public void setTagWhitelist(Set tagWhitelist) { this.tagWhitelist = tagWhitelist; } + + public boolean isLegacyAiTokenMetricsEnabled() { + return legacyAiTokenMetricsEnabled; + } + + public void setLegacyAiTokenMetricsEnabled( + boolean legacyAiTokenMetricsEnabled + ) { + this.legacyAiTokenMetricsEnabled = legacyAiTokenMetricsEnabled; + } } public static class BudgetProperties { diff --git a/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotSpringAiAutoConfiguration.java b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotSpringAiAutoConfiguration.java new file mode 100644 index 0000000..95b887d --- /dev/null +++ b/token-pilot-autoconfigure/src/main/java/io/tokenpilot/autoconfigure/TokenPilotSpringAiAutoConfiguration.java @@ -0,0 +1,85 @@ +package io.tokenpilot.autoconfigure; + +import io.tokenpilot.budget.BudgetEvaluator; +import io.tokenpilot.budget.BudgetStateStore; +import io.tokenpilot.core.CostCalculator; +import io.tokenpilot.core.LedgerManager; +import io.tokenpilot.core.PricingEvaluator; +import io.tokenpilot.core.PricingMissingListener; +import io.tokenpilot.core.PricingRegistry; +import io.tokenpilot.core.domain.MissingPricingPolicy; +import io.tokenpilot.springai.LedgerAdvisor; +import io.tokenpilot.springai.UsageExtractor; +import io.tokenpilot.springai.internal.LedgerSpringAiComponents; +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.beans.factory.ObjectProvider; +import org.springframework.boot.autoconfigure.AutoConfiguration; +import org.springframework.boot.autoconfigure.condition.ConditionalOnBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnClass; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.context.annotation.Bean; + +/** Optional Spring AI adapter의 bean graph를 격리하는 자동 설정입니다. */ +@AutoConfiguration(after = { + TokenPilotAutoConfiguration.class, + TokenPilotBudgetAutoConfiguration.class, + TokenPilotCoreMetricsAutoConfiguration.class +}) +@ConditionalOnClass({ChatClient.class, LedgerSpringAiComponents.class}) +@ConditionalOnProperty( + prefix = "token-pilot", + name = "enabled", + havingValue = "true", + matchIfMissing = true +) +public class TokenPilotSpringAiAutoConfiguration { + + @Bean + @ConditionalOnMissingBean + public UsageExtractor usageExtractor() { + return LedgerSpringAiComponents.defaultUsageExtractor(); + } + + @Bean + @ConditionalOnMissingBean + public LedgerAdvisor ledgerAdvisor( + LedgerManager ledgerManager, + UsageExtractor usageExtractor, + ObjectProvider budgetEvaluator, + ObjectProvider budgetStateStore, + CostCalculator costCalculator, + PricingRegistry pricingRegistry, + PricingEvaluator pricingEvaluator, + ObjectProvider pricingMissingListeners + ) { + BudgetEvaluator evaluator = budgetEvaluator.getIfAvailable(); + BudgetStateStore stateStore = budgetStateStore.getIfAvailable(); + boolean completeBudgetGraph = evaluator != null && stateStore != null; + BudgetEvaluator resolvedEvaluator = completeBudgetGraph ? evaluator : null; + BudgetStateStore resolvedStateStore = completeBudgetGraph ? stateStore : null; + MissingPricingPolicy missingPricingPolicy = completeBudgetGraph + ? MissingPricingPolicy.FAIL_CLOSED + : MissingPricingPolicy.FAIL_OPEN; + + return LedgerSpringAiComponents.defaultLedgerAdvisor( + ledgerManager, + usageExtractor, + resolvedEvaluator, + resolvedStateStore, + costCalculator, + pricingRegistry, + pricingEvaluator, + missingPricingPolicy, + pricingMissingListeners.orderedStream().toList() + ); + } + + @Bean + @ConditionalOnBean(LedgerAdvisor.class) + public LedgerChatClientBuilderCustomizer ledgerChatClientBuilderCustomizer( + LedgerAdvisor ledgerAdvisor + ) { + return new LedgerChatClientBuilderCustomizer(ledgerAdvisor); + } +} diff --git a/token-pilot-autoconfigure/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports b/token-pilot-autoconfigure/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports index f531281..3f7af7a 100644 --- a/token-pilot-autoconfigure/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports +++ b/token-pilot-autoconfigure/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports @@ -1 +1,7 @@ io.tokenpilot.autoconfigure.TokenPilotAutoConfiguration +io.tokenpilot.autoconfigure.TokenPilotCoreMetricsAutoConfiguration +io.tokenpilot.autoconfigure.TokenPilotBudgetMetricsAutoConfiguration +io.tokenpilot.autoconfigure.TokenPilotBudgetAutoConfiguration +io.tokenpilot.autoconfigure.TokenPilotSpringAiAutoConfiguration +io.tokenpilot.autoconfigure.TokenPilotNotificationMetricsAutoConfiguration +io.tokenpilot.autoconfigure.TokenPilotNotificationAutoConfiguration diff --git a/token-pilot-autoconfigure/src/test/java/io/tokenpilot/autoconfigure/ChatClientBuilderCustomizerTest.java b/token-pilot-autoconfigure/src/test/java/io/tokenpilot/autoconfigure/ChatClientBuilderCustomizerTest.java index 8d39441..4d02eec 100644 --- a/token-pilot-autoconfigure/src/test/java/io/tokenpilot/autoconfigure/ChatClientBuilderCustomizerTest.java +++ b/token-pilot-autoconfigure/src/test/java/io/tokenpilot/autoconfigure/ChatClientBuilderCustomizerTest.java @@ -14,7 +14,10 @@ class ChatClientBuilderCustomizerTest { private final ApplicationContextRunner contextRunner = new ApplicationContextRunner() - .withConfiguration(AutoConfigurations.of(TokenPilotAutoConfiguration.class)); + .withConfiguration(AutoConfigurations.of( + TokenPilotAutoConfiguration.class, + TokenPilotSpringAiAutoConfiguration.class + )); @Test @DisplayName("Advisor가 존재할 때 ChatClementCustomizer를 등록해야 합니다.") diff --git a/token-pilot-autoconfigure/src/test/java/io/tokenpilot/autoconfigure/TokenPilotAutoConfigurationTest.java b/token-pilot-autoconfigure/src/test/java/io/tokenpilot/autoconfigure/TokenPilotAutoConfigurationTest.java index e1e89c0..8e3f0c9 100644 --- a/token-pilot-autoconfigure/src/test/java/io/tokenpilot/autoconfigure/TokenPilotAutoConfigurationTest.java +++ b/token-pilot-autoconfigure/src/test/java/io/tokenpilot/autoconfigure/TokenPilotAutoConfigurationTest.java @@ -2,18 +2,30 @@ import io.micrometer.core.instrument.MeterRegistry; import io.micrometer.core.instrument.simple.SimpleMeterRegistry; +import io.tokenpilot.budget.ActualUsageCommand; import io.tokenpilot.budget.BudgetDecision; import io.tokenpilot.budget.BudgetEvaluator; import io.tokenpilot.budget.BudgetKey; +import io.tokenpilot.budget.BudgetReservationRequest; +import io.tokenpilot.budget.BudgetReservationResult; import io.tokenpilot.budget.BudgetState; import io.tokenpilot.budget.BudgetStateStore; import io.tokenpilot.budget.BudgetThreshold; import io.tokenpilot.budget.BudgetWindow; +import io.tokenpilot.budget.IdempotencyKey; +import io.tokenpilot.budget.ReservationAccounting; +import io.tokenpilot.budget.ReservationAccountingEvent; +import io.tokenpilot.budget.ReservationAccountingListener; +import io.tokenpilot.budget.ReservationId; +import io.tokenpilot.budget.ReservationTokenEstimate; import io.tokenpilot.core.CostCalculator; import io.tokenpilot.core.LedgerManager; +import io.tokenpilot.core.ModelRegistry; import io.tokenpilot.core.PricingEvaluator; import io.tokenpilot.core.PricingProvider; import io.tokenpilot.core.PricingRegistry; +import io.tokenpilot.core.TokenBudget; +import io.tokenpilot.core.TokenEstimator; import io.tokenpilot.core.domain.Cost; import io.tokenpilot.core.domain.PricingPlan; import io.tokenpilot.core.domain.PricingReconciliationResult; @@ -22,6 +34,10 @@ import io.tokenpilot.core.domain.TokenType; import io.tokenpilot.core.domain.TokenUsage; import io.tokenpilot.core.exception.MissingPricingException; +import io.tokenpilot.micrometer.internal.BudgetMetricsPublisher; +import io.tokenpilot.micrometer.internal.CoreMetricsPublisher; +import io.tokenpilot.micrometer.internal.MicroCostMetricsPublisher; +import io.tokenpilot.micrometer.internal.NotificationMetricsPublisher; import io.tokenpilot.notification.BudgetNotificationHandler; import io.tokenpilot.notification.BudgetNotificationService; import io.tokenpilot.notification.NotificationStateStore; @@ -34,10 +50,12 @@ import org.junit.jupiter.params.provider.Arguments; import org.junit.jupiter.params.provider.MethodSource; import org.springframework.ai.chat.client.ChatClientRequest; +import org.springframework.ai.chat.client.ChatClientResponse; import org.springframework.ai.chat.client.advisor.api.AdvisorChain; import org.springframework.ai.chat.prompt.ChatOptions; import org.springframework.ai.chat.prompt.Prompt; import org.springframework.boot.autoconfigure.AutoConfigurations; +import org.springframework.boot.test.context.FilteredClassLoader; import org.springframework.boot.test.context.runner.ApplicationContextRunner; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; @@ -49,6 +67,7 @@ import java.util.Currency; import java.util.Map; import java.util.Optional; +import java.util.concurrent.atomic.AtomicInteger; import java.util.stream.Stream; import static io.tokenpilot.core.domain.TokenType.COMPLETION; @@ -68,7 +87,15 @@ class TokenPilotAutoConfigurationTest { private static final String PROP_CURRENCY = PREFIX + ".currency"; private final ApplicationContextRunner contextRunner = new ApplicationContextRunner() - .withConfiguration(AutoConfigurations.of(TokenPilotAutoConfiguration.class)); + .withConfiguration(AutoConfigurations.of( + TokenPilotAutoConfiguration.class, + TokenPilotCoreMetricsAutoConfiguration.class, + TokenPilotBudgetMetricsAutoConfiguration.class, + TokenPilotNotificationMetricsAutoConfiguration.class, + TokenPilotBudgetAutoConfiguration.class, + TokenPilotSpringAiAutoConfiguration.class, + TokenPilotNotificationAutoConfiguration.class + )); @Test @DisplayName("기본 설정에서 Core 및 Spring AI 빈은 등록되고, Budget 빈은 등록되지 않아야 한다") @@ -78,6 +105,9 @@ void shouldRegisterDefaultBeans() { assertThat(context).hasSingleBean(PricingRegistry.class); assertThat(context).hasSingleBean(CostCalculator.class); assertThat(context).hasSingleBean(PricingEvaluator.class); + assertThat(context).hasSingleBean(ModelRegistry.class); + assertThat(context).hasSingleBean(TokenEstimator.class); + assertThat(context).hasSingleBean(TokenBudget.class); assertThat(context).hasSingleBean(LedgerManager.class); assertThat(context).hasSingleBean(UsageExtractor.class); @@ -88,9 +118,40 @@ void shouldRegisterDefaultBeans() { assertThat(context).doesNotHaveBean(BudgetEvaluator.class); assertThat(context).doesNotHaveBean(NotificationStateStore.class); assertThat(context).doesNotHaveBean(BudgetNotificationService.class); + assertThat(context).doesNotHaveBean(CoreMetricsPublisher.class); + assertThat(context).doesNotHaveBean(BudgetMetricsPublisher.class); + assertThat(context).doesNotHaveBean(NotificationMetricsPublisher.class); }); } + @Test + @DisplayName("선택 adapter 모듈이 없어도 core 자동 설정은 시작되어야 한다") + void shouldStartCoreOnlyWithoutOptionalAdapterModules() { + this.contextRunner + .withClassLoader(new FilteredClassLoader( + "io.tokenpilot.budget", + "io.tokenpilot.notification", + "io.tokenpilot.springai", + "io.tokenpilot.micrometer", + "org.springframework.ai", + "io.micrometer" + )) + .run(context -> { + assertThat(context).hasNotFailed(); + assertThat(context).hasSingleBean(PricingProvider.class); + assertThat(context).hasSingleBean(PricingRegistry.class); + assertThat(context).hasSingleBean(CostCalculator.class); + assertThat(context).hasSingleBean(ModelRegistry.class); + assertThat(context).hasSingleBean(TokenEstimator.class); + assertThat(context).hasSingleBean(TokenBudget.class); + assertThat(context).hasSingleBean(LedgerManager.class); + assertThat(context).doesNotHaveBean("ledgerAdvisor"); + assertThat(context).doesNotHaveBean("budgetStateStore"); + assertThat(context).doesNotHaveBean("notificationStateStore"); + assertThat(context).doesNotHaveBean("tokenPilotCoreMetricsPublisher"); + }); + } + @Test @DisplayName("Ledger-only advisor는 Prompt model과 기본 policy로 pricing snapshot을 resolve해야 한다") void shouldResolvePricingSnapshotInLedgerOnlyAdvisor() { @@ -139,6 +200,33 @@ void shouldResolvePricingSnapshotInLedgerOnlyAdvisor() { }); } + @Test + @DisplayName("pricing miss는 advisor에서 Token Pilot 고유 meter로 연결되어야 한다") + void shouldPublishPricingMissingMetricFromAdvisor() { + this.contextRunner + .withUserConfiguration(MeterRegistryConfiguration.class) + .run(context -> { + LedgerAdvisor advisor = context.getBean(LedgerAdvisor.class); + ChatClientRequest request = new ChatClientRequest( + new Prompt( + "test", + ChatOptions.builder().model("missing-model").build() + ), + Map.of() + ); + + advisor.before(request, mock(AdvisorChain.class)); + + MeterRegistry registry = context.getBean(MeterRegistry.class); + assertThat(registry.get("tokenpilot.pricing.missing") + .tag("policy", "fail_open") + .counter() + .count()).isEqualTo(1.0); + assertThat(registry.get("tokenpilot.pricing.missing") + .counter().getId().getTag("model")).isNull(); + }); + } + @Test @DisplayName("설정 값이 없을 경우 빈 목록을 가진 PricingProvider가 생성되어야 한다") void shouldRegisterDefaultPricingProviderWhenNoProperties() { @@ -251,6 +339,120 @@ void shouldRegisterBudgetBeansWhenEnabled() { }); } + @Test + @DisplayName("BudgetStateStore를 의존하는 listener도 순환 참조 없이 지연 연결되어야 한다") + void shouldLazilyResolveStoreDependentAccountingListener() { + this.contextRunner + .withUserConfiguration(StoreDependentAccountingListenerConfiguration.class) + .withPropertyValues("token-pilot.budget.enabled=true") + .run(context -> { + assertThat(context).hasNotFailed(); + BudgetStateStore stateStore = context.getBean(BudgetStateStore.class); + StoreDependentAccountingListener listener = context.getBean( + StoreDependentAccountingListener.class + ); + Currency usd = Currency.getInstance("USD"); + Cost limit = Cost.of(new BigDecimal("10.00"), usd); + BudgetKey key = new BudgetKey( + "policy-a", + "tenant", + "tenant-a", + BudgetWindow.parse("2026-08") + ); + + stateStore.checkAndReserve(new BudgetReservationRequest( + key, + limit, + Cost.of(BigDecimal.ONE, usd), + "request-lazy", + new IdempotencyKey("dedupe-lazy"), + null, + null, + null, + Optional.empty(), + Optional.empty() + )); + + assertThat(listener.stateStore()).isSameAs(stateStore); + assertThat(listener.evaluations()).isEqualTo(1); + }); + } + + @Test + @DisplayName("자동 설정 budget store가 reservation, commit, listener failure metric을 연결해야 한다") + void shouldWireBudgetAccountingMetricsIntoAutoConfiguredStore() { + this.contextRunner + .withUserConfiguration( + MeterRegistryConfiguration.class, + FailingAccountingListenerConfiguration.class + ) + .withPropertyValues( + "token-pilot.budget.enabled=true", + "token-pilot.budget.monthly-limit=100.00" + ) + .run(context -> { + Currency usd = Currency.getInstance("USD"); + BudgetKey key = new BudgetKey( + "policy-a", + "tenant", + "tenant-a", + BudgetWindow.parse("2026-08") + ); + Cost limit = Cost.of(new BigDecimal("100.00"), usd); + PricingSnapshot snapshot = new PricingSnapshot( + "gpt-4o-mini", + "pricing-v1", + "catalog-v1", + Instant.parse("2026-08-25T00:00:00Z"), + Map.of( + PROMPT, new BigDecimal("0.001"), + COMPLETION, new BigDecimal("0.002") + ), + usd + ); + BudgetReservationRequest request = new BudgetReservationRequest( + key, + limit, + Cost.of(new BigDecimal("1.00"), usd), + "request-a", + new IdempotencyKey("dedupe-a"), + snapshot, + new ReservationTokenEstimate(100, 120, 50) + ); + BudgetStateStore stateStore = context.getBean(BudgetStateStore.class); + ReservationAccounting accounting = (ReservationAccounting) stateStore; + + BudgetReservationResult reservation = stateStore.checkAndReserve(request); + ReservationId reservationId = reservation.reservationId(); + accounting.markInFlight(reservationId); + ActualUsageCommand command = new ActualUsageCommand( + "request-a", + "attempt-a", + reservationId, + TokenUsage.from(110, 60), + "gpt-4o-mini" + ); + var applied = accounting.commit(command); + var reused = accounting.commit(command); + + MeterRegistry registry = context.getBean(MeterRegistry.class); + assertThat(applied.transition().status().isApplied()).isTrue(); + assertThat(reused.transition().status().isApplied()).isFalse(); + assertThat(registry.get("tokenpilot.budget.reservations") + .tag("state", "created").counter().count()).isEqualTo(1.0); + assertThat(registry.get("tokenpilot.cost.total") + .tag("currency", "USD").counter().count()).isEqualTo(0.00023); + assertThat(registry.get("tokenpilot.reconciliation.outcomes") + .tag("outcome", "committed").counter().count()).isEqualTo(1.0); + assertThat(registry.get("tokenpilot.listener.failures") + .tags("listener", "custom", "phase", "reservation_evaluated") + .counter().count()).isEqualTo(1.0); + assertThat(registry.get("tokenpilot.listener.failures") + .tags("listener", "custom", "phase", "accounting_applied") + .counter().count()).isEqualTo(1.0); + }); + } + @Test @DisplayName("Budget 빈은 사용자 Clock 기준의 월별 window를 공유해야 한다") void shouldUseUserClockForMonthlyBudgetWindow() { @@ -351,6 +553,63 @@ void shouldUseFailClosedMissingPricingPolicyWhenBudgetEnabled() { }); } + @Test + @DisplayName("BudgetEvaluator만 있으면 advisor는 budget graph를 비활성화하고 FAIL_OPEN을 사용해야 한다") + void shouldNormalizeEvaluatorOnlyBudgetGraphToFailOpen() { + this.contextRunner + .withUserConfiguration(RecordingBudgetEvaluatorConfiguration.class) + .run(context -> { + LedgerAdvisor advisor = context.getBean(LedgerAdvisor.class); + RecordingBudgetEvaluator evaluator = context.getBean( + RecordingBudgetEvaluator.class + ); + ChatClientRequest request = new ChatClientRequest( + new Prompt( + "test", + ChatOptions.builder().model("missing-model").build() + ), + Map.of() + ); + + ChatClientRequest resolved = advisor.before( + request, + mock(AdvisorChain.class) + ); + + assertThat(evaluator.evaluateCalls()).isZero(); + assertThat(resolved.context().values()) + .contains(PricingResolution.MISSING_PLAN); + }); + } + + @Test + @DisplayName("BudgetStateStore만 있으면 advisor는 budget graph를 비활성화하고 FAIL_OPEN을 사용해야 한다") + void shouldNormalizeStoreOnlyBudgetGraphToFailOpen() { + this.contextRunner + .withUserConfiguration(StoreOnlyBudgetConfiguration.class) + .withPropertyValues(buildProperties( + "gpt-4o", + "0.005", + "0.015", + "USD" + )) + .run(context -> { + LedgerAdvisor advisor = context.getBean(LedgerAdvisor.class); + ChatClientResponse response = mock(ChatClientResponse.class); + when(response.context()).thenReturn(Map.of( + "tokenpilot.model.id", + "gpt-4o" + )); + + ChatClientResponse result = advisor.after( + response, + mock(AdvisorChain.class) + ); + + assertThat(result).isSameAs(response); + }); + } + @Test @DisplayName("token-pilot.budget.enabled=false 일 때 Budget 관련 빈이 등록되지 않아야 한다") void shouldNotRegisterBudgetBeansWhenDisabled() { @@ -363,12 +622,121 @@ void shouldNotRegisterBudgetBeansWhenDisabled() { } @Test - @DisplayName("MeterRegistry가 존재할 때 Micrometer 관련 빈이 등록되어야 한다") + @DisplayName("MeterRegistry가 존재하면 Token Pilot 고유 publisher만 기본 등록되어야 한다") void shouldRegisterMicrometerBeanWhenMeterRegistryExists() { this.contextRunner .withUserConfiguration(MeterRegistryConfiguration.class) .run(context -> { + assertThat(context).hasSingleBean(CoreMetricsPublisher.class); + assertThat(context).hasSingleBean(BudgetMetricsPublisher.class); + assertThat(context).hasSingleBean(NotificationMetricsPublisher.class); + assertThat(context).doesNotHaveBean(MicroCostMetricsPublisher.class); + assertThat(context.getBean(TokenPilotProperties.class) + .getMetrics().getTagWhitelist()).isEmpty(); + assertThat(context.getBean(TokenPilotProperties.class) + .getMetrics().isLegacyAiTokenMetricsEnabled()).isFalse(); + }); + } + + @Test + @DisplayName("자동 설정 TokenBudget의 admission 결과가 preflight meter로 연결되어야 한다") + void shouldPublishPreflightMetricFromAutoConfiguredTokenBudget() { + this.contextRunner + .withUserConfiguration(MeterRegistryConfiguration.class) + .run(context -> { + TokenEstimator estimator = context.getBean(TokenEstimator.class); + TokenBudget tokenBudget = context.getBean(TokenBudget.class); + + tokenBudget.check( + "gpt-4o-mini", + estimator.estimate("preflight metric"), + 256 + ); + + MeterRegistry registry = context.getBean(MeterRegistry.class); + assertThat(registry.get("tokenpilot.preflight.requests") + .tags("decision", "indeterminate", "reason", "incomplete_scope") + .counter().count()).isEqualTo(1.0); + }); + } + + @Test + @DisplayName("legacy ai.token metrics는 명시적으로 활성화할 때만 등록되어야 한다") + void shouldRegisterLegacyMicrometerBeanOnlyWhenExplicitlyEnabled() { + this.contextRunner + .withUserConfiguration(MeterRegistryConfiguration.class) + .withPropertyValues( + "token-pilot.metrics.legacy-ai-token-metrics-enabled=true", + "token-pilot.metrics.tag-whitelist[0]=tenant_id" + ) + .run(context -> { + assertThat(context).hasSingleBean(MicroCostMetricsPublisher.class); assertThat(context).hasBean("microCostMetricsPublisher"); + assertThat(context.getBean(TokenPilotProperties.class) + .getMetrics().getTagWhitelist()).containsExactly("tenant_id"); + assertThat(context.getBean(TokenPilotProperties.class) + .getMetrics().isLegacyAiTokenMetricsEnabled()).isTrue(); + }); + } + + @Test + @DisplayName("legacy flag 기본값에서는 Ledger 기록이 ai.token meter를 만들지 않아야 한다") + void shouldNotPublishLegacyMetersByDefault() { + this.contextRunner + .withUserConfiguration(MeterRegistryConfiguration.class) + .withPropertyValues(buildProperties( + "gpt-4o", + "0.005", + "0.015", + "USD" + )) + .run(context -> { + context.getBean(LedgerManager.class).record( + "gpt-4o", + TokenUsage.from(10, 20), + Map.of("tenant_id", "raw-tenant") + ); + + MeterRegistry registry = context.getBean(MeterRegistry.class); + assertThat(registry.find("ai.token.cost.total").counter()).isNull(); + assertThat(registry.find("ai.token.usage.total").counter()).isNull(); + }); + } + + @Test + @DisplayName("사용자 정의 metrics publisher가 있으면 기본 publisher가 물러나야 한다") + void shouldBackOffForUserDefinedMetricsPublisher() { + this.contextRunner + .withUserConfiguration(UserMetricsPublisherConfiguration.class) + .run(context -> { + assertThat(context).hasSingleBean(CoreMetricsPublisher.class); + assertThat(context).hasBean("customCoreMetricsPublisher"); + assertThat(context).doesNotHaveBean("tokenPilotCoreMetricsPublisher"); + }); + } + + @Test + @DisplayName("legacy flag를 활성화하면 기존 ai.token meter를 발행해야 한다") + void shouldPublishLegacyMetersWhenEnabled() { + this.contextRunner + .withUserConfiguration(MeterRegistryConfiguration.class) + .withPropertyValues( + "token-pilot.metrics.legacy-ai-token-metrics-enabled=true", + PROP_MODEL_ID + "=gpt-4o", + PROP_PROMPT + "=0.005", + PROP_COMPLETION + "=0.015", + PROP_CURRENCY + "=USD" + ) + .run(context -> { + context.getBean(LedgerManager.class).record( + "gpt-4o", + TokenUsage.from(10, 20), + Map.of() + ); + + MeterRegistry registry = context.getBean(MeterRegistry.class); + assertThat(registry.find("ai.token.cost.total").counter()).isNotNull(); + assertThat(registry.find("ai.token.usage.total").counters()).isNotEmpty(); }); } @@ -377,8 +745,14 @@ void shouldRegisterMicrometerBeanWhenMeterRegistryExists() { void shouldNotRegisterMicrometerBeanWhenMetricsDisabled() { this.contextRunner .withUserConfiguration(MeterRegistryConfiguration.class) - .withPropertyValues("token-pilot.metrics.enabled=false") + .withPropertyValues( + "token-pilot.metrics.enabled=false", + "token-pilot.metrics.legacy-ai-token-metrics-enabled=true" + ) .run(context -> { + assertThat(context).doesNotHaveBean(CoreMetricsPublisher.class); + assertThat(context).doesNotHaveBean(BudgetMetricsPublisher.class); + assertThat(context).doesNotHaveBean(NotificationMetricsPublisher.class); assertThat(context).doesNotHaveBean("microCostMetricsPublisher"); }); } @@ -434,6 +808,49 @@ void shouldRegisterNotificationServiceWhenEnabledAndHandlerExists() { }); } + @Test + @DisplayName("notification success와 dedup 결과가 lifecycle meter로 연결되어야 한다") + void shouldPublishNotificationLifecycleMetrics() { + this.contextRunner + .withUserConfiguration( + MeterRegistryConfiguration.class, + FakeBudgetNotificationHandlerConfiguration.class + ) + .withPropertyValues("token-pilot.notification.enabled=true") + .run(context -> { + Currency usd = Currency.getInstance("USD"); + BudgetDecision decision = new BudgetDecision( + new BudgetKey( + "policy-a", + "tenant", + "tenant-a", + BudgetWindow.parse("2026-08") + ), + BudgetDecision.EvaluationType.STATUS, + BudgetState.WARN, + BudgetThreshold.HALF, + "half reached", + Cost.of(new BigDecimal("5.00"), usd), + Cost.of(new BigDecimal("5.00"), usd), + Cost.of(new BigDecimal("10.00"), usd) + ); + BudgetNotificationService service = context.getBean( + BudgetNotificationService.class + ); + + service.notifyIfNeeded(decision, Map.of()); + service.notifyIfNeeded(decision, Map.of()); + + MeterRegistry registry = context.getBean(MeterRegistry.class); + assertThat(registry.get("tokenpilot.notification.events") + .tags("outcome", "success", "threshold", "half") + .counter().count()).isEqualTo(1.0); + assertThat(registry.get("tokenpilot.notification.events") + .tags("outcome", "deduplicated", "threshold", "half") + .counter().count()).isEqualTo(1.0); + }); + } + private static Stream providePricingConfigs() { return Stream.of( argumentSet( @@ -505,6 +922,19 @@ public RecordingBudgetEvaluator budgetEvaluator() { } } + @Configuration(proxyBeanMethods = false) + static class StoreOnlyBudgetConfiguration { + @Bean + public BudgetStateStore budgetStateStore() { + return mock(BudgetStateStore.class); + } + + @Bean + public UsageExtractor usageExtractor() { + return response -> TokenUsage.from(10, 20); + } + } + @Configuration(proxyBeanMethods = false) static class FixedClockConfiguration { @Bean @@ -560,6 +990,83 @@ public MeterRegistry meterRegistry() { } } + @Configuration(proxyBeanMethods = false) + static class UserMetricsPublisherConfiguration { + @Bean + public MeterRegistry meterRegistry() { + return new SimpleMeterRegistry(); + } + + @Bean + public CoreMetricsPublisher customCoreMetricsPublisher( + MeterRegistry meterRegistry + ) { + return new CoreMetricsPublisher(meterRegistry); + } + } + + @Configuration(proxyBeanMethods = false) + static class FailingAccountingListenerConfiguration { + @Bean + public ReservationAccountingListener failingAccountingListener() { + return new ReservationAccountingListener() { + @Override + public void onCommitted(ReservationAccountingEvent event) { + throw new IllegalStateException("accounting listener failed"); + } + + @Override + public void onReservationEvaluated( + BudgetReservationRequest request, + BudgetReservationResult result + ) { + throw new IllegalStateException("reservation listener failed"); + } + }; + } + } + + @Configuration(proxyBeanMethods = false) + static class StoreDependentAccountingListenerConfiguration { + @Bean + StoreDependentAccountingListener storeDependentAccountingListener( + BudgetStateStore stateStore + ) { + return new StoreDependentAccountingListener(stateStore); + } + } + + static final class StoreDependentAccountingListener + implements ReservationAccountingListener { + + private final BudgetStateStore stateStore; + private final AtomicInteger evaluations = new AtomicInteger(); + + private StoreDependentAccountingListener(BudgetStateStore stateStore) { + this.stateStore = stateStore; + } + + @Override + public void onCommitted(ReservationAccountingEvent event) { + } + + @Override + public void onReservationEvaluated( + BudgetReservationRequest request, + BudgetReservationResult result + ) { + evaluations.incrementAndGet(); + } + + BudgetStateStore stateStore() { + return stateStore; + } + + int evaluations() { + return evaluations.get(); + } + } + // 테스트용 no-op handler - 실제 알림 전송 없이 빈 등록 여부만 검증 @Configuration(proxyBeanMethods = false) static class FakeBudgetNotificationHandlerConfiguration { diff --git a/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListener.java b/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListener.java index 57d206b..b16d335 100644 --- a/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListener.java +++ b/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListener.java @@ -1,8 +1,50 @@ package io.tokenpilot.budget; -/** 예약 정산 이벤트를 수신하는 framework-independent 계약입니다. */ +/** + * 예약 admission과 정산 결과를 수신하는 framework-independent 계약입니다. + * + *

기존 lambda listener와의 source 호환성을 위해 {@link #onCommitted}만 추상 + * 메서드로 유지합니다. 나머지 callback은 예약 저장소가 확정한 결과를 관찰하기 위한 + * 선택적 확장점입니다.

+ */ @FunctionalInterface public interface ReservationAccountingListener { void onCommitted(ReservationAccountingEvent event); + + /** + * 적용된 accounting transition과 같은 linearization point의 bucket snapshot을 전달합니다. + * 기존 listener는 {@link #onCommitted(ReservationAccountingEvent)}로 위임됩니다. + */ + default void onAccountingApplied( + ReservationAccountingEvent event, + BudgetSnapshot snapshot + ) { + onCommitted(event); + } + + /** actual 미확정 estimate가 pending liability로 이동한 결과를 전달합니다. */ + default void onReconciliationRequired( + ReservationReconciliationRequiredEvent event + ) { + } + + /** 원자적 예약 시도의 모든 결과를 전달합니다. */ + default void onReservationEvaluated( + BudgetReservationRequest request, + BudgetReservationResult result + ) { + } + + /** 원자적 admission에서 상태 변경 없이 차단된 결과를 전달합니다. */ + default void onReservationBlocked( + BudgetReservationRequest request, + BudgetReservationResult result + ) { + } + + /** metric tag에 안전한 제한된 listener 종류를 반환합니다. */ + default ReservationAccountingListenerType listenerType() { + return ReservationAccountingListenerType.CUSTOM; + } } diff --git a/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListenerErrorHandler.java b/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListenerErrorHandler.java new file mode 100644 index 0000000..a97c4a4 --- /dev/null +++ b/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListenerErrorHandler.java @@ -0,0 +1,12 @@ +package io.tokenpilot.budget; + +/** bounded 회계 listener 실패를 관찰하는 best-effort hook입니다. */ +@FunctionalInterface +public interface ReservationAccountingListenerErrorHandler { + + void onFailure(ReservationAccountingListenerFailureEvent event); + + static ReservationAccountingListenerErrorHandler noOp() { + return ignored -> { }; + } +} diff --git a/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListenerFailureEvent.java b/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListenerFailureEvent.java new file mode 100644 index 0000000..89684b0 --- /dev/null +++ b/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListenerFailureEvent.java @@ -0,0 +1,20 @@ +package io.tokenpilot.budget; + +import java.util.Objects; + +/** + * 회계 listener callback 실패를 bounded 정보만으로 전달하는 이벤트입니다. + * + *

원본 예외와 메시지를 포함하지 않아 prompt, provider 응답 또는 자격 증명이 + * 관측 경로로 전달되지 않습니다.

+ */ +public record ReservationAccountingListenerFailureEvent( + ReservationAccountingListenerType listenerType, + ReservationAccountingListenerPhase phase +) { + + public ReservationAccountingListenerFailureEvent { + Objects.requireNonNull(listenerType, "listenerType must not be null"); + Objects.requireNonNull(phase, "phase must not be null"); + } +} diff --git a/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListenerPhase.java b/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListenerPhase.java new file mode 100644 index 0000000..93642ce --- /dev/null +++ b/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListenerPhase.java @@ -0,0 +1,9 @@ +package io.tokenpilot.budget; + +/** 회계 listener 실패 관측에 사용하는 제한된 callback 단계입니다. */ +public enum ReservationAccountingListenerPhase { + ACCOUNTING_APPLIED, + RECONCILIATION_REQUIRED, + RESERVATION_EVALUATED, + RESERVATION_BLOCKED +} diff --git a/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListenerType.java b/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListenerType.java new file mode 100644 index 0000000..8fac5d8 --- /dev/null +++ b/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationAccountingListenerType.java @@ -0,0 +1,8 @@ +package io.tokenpilot.budget; + +/** 회계 listener 실패 관측에 사용하는 제한된 listener 종류입니다. */ +public enum ReservationAccountingListenerType { + METRICS, + NOTIFICATION, + CUSTOM +} diff --git a/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationReconciliationRequiredEvent.java b/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationReconciliationRequiredEvent.java new file mode 100644 index 0000000..ce80644 --- /dev/null +++ b/token-pilot-budget/src/main/java/io/tokenpilot/budget/ReservationReconciliationRequiredEvent.java @@ -0,0 +1,39 @@ +package io.tokenpilot.budget; + +import java.util.Objects; + +/** actual usage를 확보하지 못해 pending liability로 이동한 원자적 회계 결과입니다. */ +public record ReservationReconciliationRequiredEvent( + ReservationId reservationId, + BudgetKey budgetKey, + ReservationTransition transition, + ReservationAccountingReason reason, + BudgetSnapshot snapshot +) { + + public ReservationReconciliationRequiredEvent { + Objects.requireNonNull(reservationId, "reservationId must not be null"); + Objects.requireNonNull(budgetKey, "budgetKey must not be null"); + Objects.requireNonNull(transition, "transition must not be null"); + Objects.requireNonNull(reason, "reason must not be null"); + Objects.requireNonNull(snapshot, "snapshot must not be null"); + if (!transition.status().isApplied() + || transition.resultingState() + != ReservationState.RECONCILIATION_REQUIRED) { + throw new IllegalArgumentException( + "event requires an applied RECONCILIATION_REQUIRED transition" + ); + } + if (reason != ReservationAccountingReason.ACTUAL_USAGE_UNAVAILABLE + && reason != ReservationAccountingReason.CALLBACK_TIMED_OUT) { + throw new IllegalArgumentException( + "reason must require later actual reconciliation" + ); + } + if (!budgetKey.equals(snapshot.key())) { + throw new IllegalArgumentException( + "event and snapshot must use the same budget key" + ); + } + } +} diff --git a/token-pilot-budget/src/main/java/io/tokenpilot/budget/internal/InMemoryBudgetStateStore.java b/token-pilot-budget/src/main/java/io/tokenpilot/budget/internal/InMemoryBudgetStateStore.java index 7d5e1a1..c214edc 100644 --- a/token-pilot-budget/src/main/java/io/tokenpilot/budget/internal/InMemoryBudgetStateStore.java +++ b/token-pilot-budget/src/main/java/io/tokenpilot/budget/internal/InMemoryBudgetStateStore.java @@ -12,11 +12,17 @@ import io.tokenpilot.budget.ReservationAccounting; import io.tokenpilot.budget.ReservationAccountingEvent; import io.tokenpilot.budget.ReservationAccountingListener; +import io.tokenpilot.budget.ReservationAccountingListenerErrorHandler; +import io.tokenpilot.budget.ReservationAccountingListenerFailureEvent; +import io.tokenpilot.budget.ReservationAccountingListenerPhase; +import io.tokenpilot.budget.ReservationAccountingListenerType; import io.tokenpilot.budget.ReservationAccountingReason; import io.tokenpilot.budget.ReservationActualTokens; import io.tokenpilot.budget.ReservationId; import io.tokenpilot.budget.ReservationReconciliation; +import io.tokenpilot.budget.ReservationReconciliationRequiredEvent; import io.tokenpilot.budget.ReservationState; +import io.tokenpilot.budget.ReservationStatus; import io.tokenpilot.budget.ReservationStateMachine; import io.tokenpilot.budget.ReservationTransition; import io.tokenpilot.budget.ReservationTokenEstimate; @@ -53,7 +59,9 @@ public class InMemoryBudgetStateStore implements BudgetStateStore, ReservationAc private final Clock clock; private final Supplier reservationIdGenerator; private final CostCalculator costCalculator; - private final List accountingListeners; + private final Supplier> accountingListenerSupplier; + private final Supplier> + listenerErrorHandlerSupplier; public InMemoryBudgetStateStore() { this( @@ -79,7 +87,7 @@ public InMemoryBudgetStateStore( Supplier reservationIdGenerator, CostCalculator costCalculator ) { - this(clock, reservationIdGenerator, costCalculator, List.of()); + this(clock, reservationIdGenerator, costCalculator, List.of(), List.of()); } public InMemoryBudgetStateStore( @@ -87,6 +95,45 @@ public InMemoryBudgetStateStore( Supplier reservationIdGenerator, CostCalculator costCalculator, List accountingListeners + ) { + this( + clock, + reservationIdGenerator, + costCalculator, + accountingListeners, + List.of() + ); + } + + public InMemoryBudgetStateStore( + Clock clock, + Supplier reservationIdGenerator, + CostCalculator costCalculator, + List accountingListeners, + List listenerErrorHandlers + ) { + this( + clock, + reservationIdGenerator, + costCalculator, + fixedListSupplier( + accountingListeners, + "accountingListeners must not be null" + ), + fixedListSupplier( + listenerErrorHandlers, + "listenerErrorHandlers must not be null" + ) + ); + } + + public InMemoryBudgetStateStore( + Clock clock, + Supplier reservationIdGenerator, + CostCalculator costCalculator, + Supplier> accountingListenerSupplier, + Supplier> + listenerErrorHandlerSupplier ) { this.clock = Objects.requireNonNull(clock, "clock must not be null"); this.reservationIdGenerator = Objects.requireNonNull( @@ -97,11 +144,13 @@ public InMemoryBudgetStateStore( costCalculator, "costCalculator must not be null" ); - this.accountingListeners = List.copyOf( - Objects.requireNonNull( - accountingListeners, - "accountingListeners must not be null" - ) + this.accountingListenerSupplier = Objects.requireNonNull( + accountingListenerSupplier, + "accountingListenerSupplier must not be null" + ); + this.listenerErrorHandlerSupplier = Objects.requireNonNull( + listenerErrorHandlerSupplier, + "listenerErrorHandlerSupplier must not be null" ); } @@ -145,7 +194,13 @@ public BudgetReservationResult checkAndReserve(BudgetReservationRequest request) ) ); - return Objects.requireNonNull(result.get(), "reservation result must be set"); + BudgetReservationResult reservationResult = Objects.requireNonNull( + result.get(), + "reservation result must be set" + ); + publishReservationEvaluated(request, reservationResult); + publishBlockedReservation(request, reservationResult); + return reservationResult; } @Override @@ -260,13 +315,14 @@ public ReservationTransition markReconciliationRequired( ) { requireReconciliationRequiredReason(reason); Bucket bucket = bucketFor(reservationId); + ReservationTransition transition; + ReservationReconciliationRequiredEvent event; synchronized (bucket) { ReservationAccountingState accountingState = accountingState( bucket, reservationId ); - ReservationTransition transition = - accountingState.evaluateReconciliationRequired(); + transition = accountingState.evaluateReconciliationRequired(); if (!transition.status().isApplied()) { return transition; } @@ -276,8 +332,16 @@ public ReservationTransition markReconciliationRequired( accountingState, transition.resultingState() ); - return transition; + event = new ReservationReconciliationRequiredEvent( + reservationId, + accountingState.reservation().key(), + transition, + reason, + bucket.snapshot(accountingState.reservation().key()) + ); } + publishReconciliationRequired(event); + return transition; } @Override @@ -425,6 +489,7 @@ private ReservationReconciliation reconcileUsage( Objects.requireNonNull(reason, "reason must not be null"); Bucket bucket = bucketFor(command.reservationId()); ReservationReconciliation reconciliation; + BudgetSnapshot accountingSnapshot; synchronized (bucket) { reconciliation = reconcileUsageInBucket( bucket, @@ -432,14 +497,18 @@ private ReservationReconciliation reconcileUsage( type, reason ); + accountingSnapshot = bucket.snapshot(reconciliation.budgetKey()); } - publishAccountingEvent(reconciliation); + publishAccountingEvent(reconciliation, accountingSnapshot); return reconciliation; } private void publishAccountingEvent( - ReservationReconciliation reconciliation + ReservationReconciliation reconciliation, + BudgetSnapshot accountingSnapshot ) { + List accountingListeners = + accountingListeners(); if (!reconciliation.transition().status().isApplied() || accountingListeners.isEmpty()) { return; @@ -448,18 +517,124 @@ private void publishAccountingEvent( reconciliation ); for (ReservationAccountingListener listener : accountingListeners) { - notifyBestEffort(listener, event); + notifyBestEffort( + listener, + ReservationAccountingListenerPhase.ACCOUNTING_APPLIED, + () -> listener.onAccountingApplied(event, accountingSnapshot) + ); } } - private static void notifyBestEffort( + private void publishReservationEvaluated( + BudgetReservationRequest request, + BudgetReservationResult result + ) { + for (ReservationAccountingListener listener : accountingListeners()) { + notifyBestEffort( + listener, + ReservationAccountingListenerPhase.RESERVATION_EVALUATED, + () -> listener.onReservationEvaluated(request, result) + ); + } + } + + private void publishBlockedReservation( + BudgetReservationRequest request, + BudgetReservationResult result + ) { + if (result.status() != ReservationStatus.BLOCKED) { + return; + } + for (ReservationAccountingListener listener : accountingListeners()) { + notifyBestEffort( + listener, + ReservationAccountingListenerPhase.RESERVATION_BLOCKED, + () -> listener.onReservationBlocked(request, result) + ); + } + } + + private void publishReconciliationRequired( + ReservationReconciliationRequiredEvent event + ) { + for (ReservationAccountingListener listener : accountingListeners()) { + notifyBestEffort( + listener, + ReservationAccountingListenerPhase.RECONCILIATION_REQUIRED, + () -> listener.onReconciliationRequired(event) + ); + } + } + + private void notifyBestEffort( ReservationAccountingListener listener, - ReservationAccountingEvent event + ReservationAccountingListenerPhase phase, + Runnable callback + ) { + try { + callback.run(); + } catch (RuntimeException ignored) { + publishListenerFailure(listenerType(listener), phase); + } + } + + private ReservationAccountingListenerType listenerType( + ReservationAccountingListener listener + ) { + try { + ReservationAccountingListenerType listenerType = listener.listenerType(); + return listenerType == null + ? ReservationAccountingListenerType.CUSTOM + : listenerType; + } catch (RuntimeException ignored) { + return ReservationAccountingListenerType.CUSTOM; + } + } + + private void publishListenerFailure( + ReservationAccountingListenerType listenerType, + ReservationAccountingListenerPhase phase + ) { + List listenerErrorHandlers = + listenerErrorHandlers(); + if (listenerErrorHandlers.isEmpty()) { + return; + } + ReservationAccountingListenerFailureEvent event = + new ReservationAccountingListenerFailureEvent(listenerType, phase); + for (ReservationAccountingListenerErrorHandler errorHandler + : listenerErrorHandlers) { + try { + errorHandler.onFailure(event); + } catch (RuntimeException ignored) { + // Error handler도 best-effort이며 회계/admission 결과에 영향을 주지 않습니다. + } + } + } + + private List accountingListeners() { + return suppliedList(accountingListenerSupplier); + } + + private List listenerErrorHandlers() { + return suppliedList(listenerErrorHandlerSupplier); + } + + private static Supplier> fixedListSupplier( + List values, + String message ) { + List copy = List.copyOf(Objects.requireNonNull(values, message)); + return () -> copy; + } + + private static List suppliedList(Supplier> supplier) { try { - listener.onCommitted(event); + List values = supplier.get(); + return values == null ? List.of() : List.copyOf(values); } catch (RuntimeException ignored) { - // Listener 실패는 이미 적용된 회계 상태를 되돌리지 않습니다. + // Optional observer resolution cannot change reservation/accounting results. + return List.of(); } } diff --git a/token-pilot-budget/src/main/java/io/tokenpilot/budget/internal/LedgerBudgetComponents.java b/token-pilot-budget/src/main/java/io/tokenpilot/budget/internal/LedgerBudgetComponents.java index d0380c1..7f625c3 100644 --- a/token-pilot-budget/src/main/java/io/tokenpilot/budget/internal/LedgerBudgetComponents.java +++ b/token-pilot-budget/src/main/java/io/tokenpilot/budget/internal/LedgerBudgetComponents.java @@ -5,8 +5,10 @@ import io.tokenpilot.budget.BudgetStateStore; import io.tokenpilot.budget.ReservationAccounting; import io.tokenpilot.budget.ReservationAccountingListener; +import io.tokenpilot.budget.ReservationAccountingListenerErrorHandler; import io.tokenpilot.budget.ReservationId; import io.tokenpilot.core.CostCalculator; +import io.tokenpilot.core.internal.LedgerComponents; import java.time.Clock; import java.util.List; @@ -25,6 +27,47 @@ public static BudgetStateStore inMemoryBudgetStateStore() { return new InMemoryBudgetStateStore(); } + /** PR #68 accounting/notification listener 연결과 호환되는 convenience factory입니다. */ + public static BudgetStateStore inMemoryBudgetStateStore( + List accountingListeners + ) { + return new InMemoryBudgetStateStore( + Clock.systemUTC(), + ReservationId::random, + LedgerComponents.defaultCostCalculator(), + accountingListeners + ); + } + + /** listener 실패 관측 hook까지 연결하는 convenience factory입니다. */ + public static BudgetStateStore inMemoryBudgetStateStore( + List accountingListeners, + List listenerErrorHandlers + ) { + return new InMemoryBudgetStateStore( + Clock.systemUTC(), + ReservationId::random, + LedgerComponents.defaultCostCalculator(), + accountingListeners, + listenerErrorHandlers + ); + } + + /** Spring 등 DI container가 listener를 store 생성 뒤 지연 해석할 수 있는 factory입니다. */ + public static BudgetStateStore inMemoryBudgetStateStore( + Supplier> accountingListenerSupplier, + Supplier> + listenerErrorHandlerSupplier + ) { + return new InMemoryBudgetStateStore( + Clock.systemUTC(), + ReservationId::random, + LedgerComponents.defaultCostCalculator(), + accountingListenerSupplier, + listenerErrorHandlerSupplier + ); + } + public static BudgetStateStore inMemoryBudgetStateStore( Clock clock, Supplier reservationIdGenerator @@ -58,6 +101,22 @@ public static BudgetStateStore inMemoryBudgetStateStore( ); } + public static BudgetStateStore inMemoryBudgetStateStore( + Clock clock, + Supplier reservationIdGenerator, + CostCalculator costCalculator, + List accountingListeners, + List listenerErrorHandlers + ) { + return new InMemoryBudgetStateStore( + clock, + reservationIdGenerator, + costCalculator, + accountingListeners, + listenerErrorHandlers + ); + } + /** * 예약을 생성한 store와 동일한 객체의 회계 명령 진입점을 반환합니다. */ diff --git a/token-pilot-budget/src/test/java/io/tokenpilot/budget/internal/ReservationAccountingListenerLifecycleTest.java b/token-pilot-budget/src/test/java/io/tokenpilot/budget/internal/ReservationAccountingListenerLifecycleTest.java new file mode 100644 index 0000000..2f1f8eb --- /dev/null +++ b/token-pilot-budget/src/test/java/io/tokenpilot/budget/internal/ReservationAccountingListenerLifecycleTest.java @@ -0,0 +1,375 @@ +package io.tokenpilot.budget.internal; + +import io.tokenpilot.budget.ActualUsageCommand; +import io.tokenpilot.budget.BudgetKey; +import io.tokenpilot.budget.BudgetReservationRequest; +import io.tokenpilot.budget.BudgetReservationResult; +import io.tokenpilot.budget.BudgetSnapshot; +import io.tokenpilot.budget.BudgetWindow; +import io.tokenpilot.budget.IdempotencyKey; +import io.tokenpilot.budget.ReservationAccountingEvent; +import io.tokenpilot.budget.ReservationAccountingListener; +import io.tokenpilot.budget.ReservationAccountingListenerFailureEvent; +import io.tokenpilot.budget.ReservationAccountingListenerPhase; +import io.tokenpilot.budget.ReservationAccountingListenerType; +import io.tokenpilot.budget.ReservationId; +import io.tokenpilot.budget.ReservationReconciliation; +import io.tokenpilot.budget.ReservationReconciliationRequiredEvent; +import io.tokenpilot.budget.ReservationStatus; +import io.tokenpilot.budget.ReservationTokenEstimate; +import io.tokenpilot.core.domain.Cost; +import io.tokenpilot.core.domain.PricingSnapshot; +import io.tokenpilot.core.domain.TokenType; +import io.tokenpilot.core.domain.TokenUsage; +import org.junit.jupiter.api.Test; + +import java.math.BigDecimal; +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.Currency; +import java.util.List; +import java.util.Map; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +import static io.tokenpilot.budget.AccountingTransitionStatus.APPLIED; +import static io.tokenpilot.budget.AccountingTransitionStatus.REUSED; +import static io.tokenpilot.budget.ReservationState.COMMITTED; +import static io.tokenpilot.budget.ReservationState.RECONCILIATION_REQUIRED; +import static org.assertj.core.api.Assertions.assertThat; + +class ReservationAccountingListenerLifecycleTest { + + private static final Currency USD = Currency.getInstance("USD"); + private static final Currency KRW = Currency.getInstance("KRW"); + private static final Cost LIMIT = usd("100.00"); + private static final BudgetKey KEY = new BudgetKey( + "budget-policy", + "tenant", + "tenant-a", + BudgetWindow.parse("2026-08") + ); + private static final Clock CLOCK = Clock.fixed( + Instant.parse("2026-08-15T12:34:56Z"), + ZoneOffset.UTC + ); + private static final ReservationTokenEstimate TOKEN_ESTIMATE = + new ReservationTokenEstimate(90, 100, 50); + + @Test + void 모든_예약_결과를_평가_callback으로_전달하고_BLOCKED만_별도_전달한다() { + List evaluated = new ArrayList<>(); + List blocked = new ArrayList<>(); + ReservationAccountingListener listener = new ReservationAccountingListener() { + @Override + public void onCommitted(ReservationAccountingEvent event) { + } + + @Override + public void onReservationEvaluated( + BudgetReservationRequest request, + BudgetReservationResult result + ) { + evaluated.add(result.status()); + } + + @Override + public void onReservationBlocked( + BudgetReservationRequest request, + BudgetReservationResult result + ) { + blocked.add(result.status()); + } + }; + InMemoryBudgetStateStore store = store(List.of(listener), List.of()); + + store.checkAndReserve(legacyRequest("request-1", usd("20.00"))); + store.checkAndReserve(legacyRequest("request-1", usd("20.00"))); + store.checkAndReserve(legacyRequest("request-1", usd("30.00"))); + store.checkAndReserve(legacyRequest("request-2", usd("80.00"))); + store.checkAndReserve(legacyRequest( + "request-3", + Cost.of(new BigDecimal("10.00"), KRW) + )); + + assertThat(evaluated).containsExactly( + ReservationStatus.CREATED, + ReservationStatus.REUSED, + ReservationStatus.CONFLICT, + ReservationStatus.BLOCKED, + ReservationStatus.CURRENCY_MISMATCH + ); + assertThat(blocked).containsExactly(ReservationStatus.BLOCKED); + } + + @Test + void RECONCILIATION_REQUIRED는_APPLIED_전이만_snapshot과_함께_한번_전달한다() { + List events = new ArrayList<>(); + ReservationAccountingListener listener = new ReservationAccountingListener() { + @Override + public void onCommitted(ReservationAccountingEvent event) { + } + + @Override + public void onReconciliationRequired( + ReservationReconciliationRequiredEvent event + ) { + events.add(event); + } + }; + InMemoryBudgetStateStore store = store(List.of(listener), List.of()); + ReservationId reservationId = reserveUsage(store, "request-1", "dedupe-1", "60.00"); + store.markInFlight(reservationId); + + var applied = store.markReconciliationRequired(reservationId); + var duplicate = store.markReconciliationRequired(reservationId); + + assertThat(applied.status()).isEqualTo(APPLIED); + assertThat(duplicate.status()).isEqualTo(REUSED); + assertThat(events).hasSize(1); + ReservationReconciliationRequiredEvent event = events.getFirst(); + assertThat(event.transition()).isEqualTo(applied); + assertThat(event.snapshot().activeReservedCost()).isEqualTo(usd("0.00")); + assertThat(event.snapshot().pendingReconciliationLiability()) + .isEqualTo(usd("60.00")); + } + + @Test + void APPLIED_commit은_같은_linearization_point의_snapshot과_함께_한번_전달한다() { + List events = new ArrayList<>(); + AtomicReference snapshot = new AtomicReference<>(); + ReservationAccountingListener listener = new ReservationAccountingListener() { + @Override + public void onCommitted(ReservationAccountingEvent event) { + } + + @Override + public void onAccountingApplied( + ReservationAccountingEvent event, + BudgetSnapshot accountingSnapshot + ) { + events.add(event); + snapshot.set(accountingSnapshot); + } + }; + InMemoryBudgetStateStore store = store(List.of(listener), List.of()); + ReservationId reservationId = reserveUsage(store, "request-1", "dedupe-1", "60.00"); + store.markInFlight(reservationId); + + ReservationReconciliation applied = store.commit(command("request-1", reservationId)); + ReservationReconciliation duplicate = store.commit(command("request-1", reservationId)); + + assertThat(applied.transition().status()).isEqualTo(APPLIED); + assertThat(duplicate.transition().status()).isEqualTo(REUSED); + assertThat(events).containsExactly(new ReservationAccountingEvent(applied)); + assertThat(snapshot.get().committedCost()).isEqualTo(usd("40.00")); + assertThat(snapshot.get().activeReservedCost()).isEqualTo(usd("0.00")); + assertThat(snapshot.get().pendingReconciliationLiability()) + .isEqualTo(usd("0.00")); + } + + @Test + void callback과_error_handler_실패를_격리하고_bounded_failure를_계속_전달한다() { + List failures = new ArrayList<>(); + AtomicInteger observedEvaluations = new AtomicInteger(); + AtomicInteger observedBlocks = new AtomicInteger(); + AtomicInteger observedPending = new AtomicInteger(); + AtomicInteger observedCommits = new AtomicInteger(); + + ReservationAccountingListener failing = failingMetricsListener(); + ReservationAccountingListener following = new ReservationAccountingListener() { + @Override + public void onCommitted(ReservationAccountingEvent event) { + observedCommits.incrementAndGet(); + } + + @Override + public void onReconciliationRequired( + ReservationReconciliationRequiredEvent event + ) { + observedPending.incrementAndGet(); + } + + @Override + public void onReservationEvaluated( + BudgetReservationRequest request, + BudgetReservationResult result + ) { + observedEvaluations.incrementAndGet(); + } + + @Override + public void onReservationBlocked( + BudgetReservationRequest request, + BudgetReservationResult result + ) { + observedBlocks.incrementAndGet(); + } + }; + InMemoryBudgetStateStore store = store( + List.of(failing, following), + List.of( + event -> { + throw new IllegalStateException("error handler failed"); + }, + failures::add + ) + ); + + BudgetReservationResult blocked = store.checkAndReserve( + legacyRequest("blocked", usd("100.00")) + ); + ReservationId pendingId = reserveUsage( + store, + "pending-request", + "pending-dedupe", + "60.00" + ); + store.markInFlight(pendingId); + store.markReconciliationRequired(pendingId); + + ReservationId committedId = reserveUsage( + store, + "committed-request", + "committed-dedupe", + "20.00" + ); + store.markInFlight(committedId); + ReservationReconciliation committed = store.commit( + command("committed-request", committedId) + ); + store.commit(command("committed-request", committedId)); + + assertThat(blocked.status()).isEqualTo(ReservationStatus.BLOCKED); + assertThat(committed.transition().resultingState()).isEqualTo(COMMITTED); + assertThat(store.snapshot(KEY, LIMIT).pendingReconciliationLiability()) + .isEqualTo(usd("60.00")); + assertThat(store.snapshot(KEY, LIMIT).committedCost()).isEqualTo(usd("40.00")); + assertThat(observedEvaluations).hasValue(3); + assertThat(observedBlocks).hasValue(1); + assertThat(observedPending).hasValue(1); + assertThat(observedCommits).hasValue(1); + assertThat(failures) + .extracting(ReservationAccountingListenerFailureEvent::listenerType) + .containsOnly(ReservationAccountingListenerType.METRICS); + assertThat(failures) + .extracting(ReservationAccountingListenerFailureEvent::phase) + .contains( + ReservationAccountingListenerPhase.RESERVATION_EVALUATED, + ReservationAccountingListenerPhase.RESERVATION_BLOCKED, + ReservationAccountingListenerPhase.RECONCILIATION_REQUIRED, + ReservationAccountingListenerPhase.ACCOUNTING_APPLIED + ); + assertThat(failures.stream() + .filter(event -> event.phase() + == ReservationAccountingListenerPhase.ACCOUNTING_APPLIED)) + .hasSize(1); + } + + private static ReservationAccountingListener failingMetricsListener() { + return new ReservationAccountingListener() { + @Override + public void onCommitted(ReservationAccountingEvent event) { + throw new IllegalStateException("commit listener failed"); + } + + @Override + public void onReconciliationRequired( + ReservationReconciliationRequiredEvent event + ) { + throw new IllegalStateException("pending listener failed"); + } + + @Override + public void onReservationEvaluated( + BudgetReservationRequest request, + BudgetReservationResult result + ) { + throw new IllegalStateException("reservation listener failed"); + } + + @Override + public void onReservationBlocked( + BudgetReservationRequest request, + BudgetReservationResult result + ) { + throw new IllegalStateException("blocked listener failed"); + } + + @Override + public ReservationAccountingListenerType listenerType() { + return ReservationAccountingListenerType.METRICS; + } + }; + } + + private static InMemoryBudgetStateStore store( + List listeners, + List errorHandlers + ) { + AtomicInteger sequence = new AtomicInteger(); + return new InMemoryBudgetStateStore( + CLOCK, + () -> new ReservationId("reservation-" + sequence.incrementAndGet()), + (usage, plan) -> usd("40.00"), + listeners, + errorHandlers + ); + } + + private static BudgetReservationRequest legacyRequest(String id, Cost amount) { + return new BudgetReservationRequest(KEY, LIMIT, amount, id); + } + + private static ReservationId reserveUsage( + InMemoryBudgetStateStore store, + String requestId, + String deduplicationId, + String amount + ) { + return store.checkAndReserve(new BudgetReservationRequest( + KEY, + LIMIT, + usd(amount), + requestId, + new IdempotencyKey(deduplicationId), + pricingSnapshot(), + TOKEN_ESTIMATE + )).reservationId(); + } + + private static ActualUsageCommand command( + String requestId, + ReservationId reservationId + ) { + return new ActualUsageCommand( + requestId, + "attempt-1", + reservationId, + TokenUsage.from(100, 50), + "gpt-4o-mini-response" + ); + } + + private static PricingSnapshot pricingSnapshot() { + return new PricingSnapshot( + "gpt-4o-mini-request", + "pricing-v1", + "catalog-v1", + CLOCK.instant(), + Map.of( + TokenType.PROMPT, + new BigDecimal("0.10"), + TokenType.COMPLETION, + new BigDecimal("0.20") + ), + USD + ); + } + + private static Cost usd(String amount) { + return Cost.of(new BigDecimal(amount), USD); + } +} diff --git a/token-pilot-core/src/main/java/io/tokenpilot/core/CoreComponents.java b/token-pilot-core/src/main/java/io/tokenpilot/core/CoreComponents.java index 63a8cc7..7b4826b 100644 --- a/token-pilot-core/src/main/java/io/tokenpilot/core/CoreComponents.java +++ b/token-pilot-core/src/main/java/io/tokenpilot/core/CoreComponents.java @@ -4,6 +4,7 @@ import io.tokenpilot.core.internal.LedgerComponents; import java.util.Collection; +import java.util.List; /** * framework-independent core component entrypoint입니다. @@ -28,4 +29,11 @@ public static ModelRegistry defaultModelRegistry() { public static TokenBudget tokenBudget(ModelRegistry modelRegistry) { return LedgerComponents.tokenBudget(modelRegistry); } + + public static TokenBudget tokenBudget( + ModelRegistry modelRegistry, + List listeners + ) { + return LedgerComponents.tokenBudget(modelRegistry, listeners); + } } diff --git a/token-pilot-core/src/main/java/io/tokenpilot/core/PreflightDecisionListener.java b/token-pilot-core/src/main/java/io/tokenpilot/core/PreflightDecisionListener.java new file mode 100644 index 0000000..4621305 --- /dev/null +++ b/token-pilot-core/src/main/java/io/tokenpilot/core/PreflightDecisionListener.java @@ -0,0 +1,10 @@ +package io.tokenpilot.core; + +import io.tokenpilot.core.domain.PreflightDecisionEvent; + +/** context admission 판정을 best-effort로 관찰하는 framework-independent 계약입니다. */ +@FunctionalInterface +public interface PreflightDecisionListener { + + void onDecision(PreflightDecisionEvent event); +} diff --git a/token-pilot-core/src/main/java/io/tokenpilot/core/PricingMissingListener.java b/token-pilot-core/src/main/java/io/tokenpilot/core/PricingMissingListener.java new file mode 100644 index 0000000..6d23f32 --- /dev/null +++ b/token-pilot-core/src/main/java/io/tokenpilot/core/PricingMissingListener.java @@ -0,0 +1,10 @@ +package io.tokenpilot.core; + +import io.tokenpilot.core.domain.PricingMissingEvent; + +/** missing-pricing 판정을 best-effort로 관찰하는 framework-independent 계약입니다. */ +@FunctionalInterface +public interface PricingMissingListener { + + void onPricingMissing(PricingMissingEvent event); +} diff --git a/token-pilot-core/src/main/java/io/tokenpilot/core/domain/PreflightDecisionEvent.java b/token-pilot-core/src/main/java/io/tokenpilot/core/domain/PreflightDecisionEvent.java new file mode 100644 index 0000000..efd3eea --- /dev/null +++ b/token-pilot-core/src/main/java/io/tokenpilot/core/domain/PreflightDecisionEvent.java @@ -0,0 +1,15 @@ +package io.tokenpilot.core.domain; + +import java.util.Objects; + +/** + * provider 호출 전 context admission 판정을 전달하는 framework-independent 이벤트입니다. + * + * @param result bounded status/reason과 판정 metadata를 포함한 immutable 결과 + */ +public record PreflightDecisionEvent(BudgetResult result) { + + public PreflightDecisionEvent { + Objects.requireNonNull(result, "result must not be null"); + } +} diff --git a/token-pilot-core/src/main/java/io/tokenpilot/core/domain/PricingMissingEvent.java b/token-pilot-core/src/main/java/io/tokenpilot/core/domain/PricingMissingEvent.java new file mode 100644 index 0000000..94580f1 --- /dev/null +++ b/token-pilot-core/src/main/java/io/tokenpilot/core/domain/PricingMissingEvent.java @@ -0,0 +1,23 @@ +package io.tokenpilot.core.domain; + +import java.util.Objects; + +/** + * 명시적인 missing-pricing 판정을 전달하는 framework-independent 이벤트입니다. + * + * @param policy missing pricing 처리 정책 + * @param resolution missing plan/rate 판정 + */ +public record PricingMissingEvent( + MissingPricingPolicy policy, + PricingResolution resolution +) { + + public PricingMissingEvent { + Objects.requireNonNull(policy, "policy must not be null"); + Objects.requireNonNull(resolution, "resolution must not be null"); + if (!resolution.isMissing()) { + throw new IllegalArgumentException("resolution must identify missing pricing"); + } + } +} diff --git a/token-pilot-core/src/main/java/io/tokenpilot/core/domain/PricingResolution.java b/token-pilot-core/src/main/java/io/tokenpilot/core/domain/PricingResolution.java index 2a6b6fe..48153fd 100644 --- a/token-pilot-core/src/main/java/io/tokenpilot/core/domain/PricingResolution.java +++ b/token-pilot-core/src/main/java/io/tokenpilot/core/domain/PricingResolution.java @@ -9,4 +9,8 @@ public enum PricingResolution { public boolean isResolved() { return this == RESOLVED; } + + public boolean isMissing() { + return this == MISSING_PLAN || this == MISSING_RATE; + } } diff --git a/token-pilot-core/src/main/java/io/tokenpilot/core/internal/DefaultLedgerManager.java b/token-pilot-core/src/main/java/io/tokenpilot/core/internal/DefaultLedgerManager.java index e9e66f4..3cba97f 100644 --- a/token-pilot-core/src/main/java/io/tokenpilot/core/internal/DefaultLedgerManager.java +++ b/token-pilot-core/src/main/java/io/tokenpilot/core/internal/DefaultLedgerManager.java @@ -80,7 +80,13 @@ private void publish(String modelId, TokenUsage usage, Cost cost, Map listener.onRecord(event)); + for (LedgerListener listener : listeners) { + try { + listener.onRecord(event); + } catch (RuntimeException ignored) { + // Optional observers do not change an already calculated ledger result. + } + } } } } diff --git a/token-pilot-core/src/main/java/io/tokenpilot/core/internal/DefaultTokenBudget.java b/token-pilot-core/src/main/java/io/tokenpilot/core/internal/DefaultTokenBudget.java index 01f9165..da34bed 100644 --- a/token-pilot-core/src/main/java/io/tokenpilot/core/internal/DefaultTokenBudget.java +++ b/token-pilot-core/src/main/java/io/tokenpilot/core/internal/DefaultTokenBudget.java @@ -1,14 +1,17 @@ package io.tokenpilot.core.internal; import io.tokenpilot.core.ModelRegistry; +import io.tokenpilot.core.PreflightDecisionListener; import io.tokenpilot.core.TokenBudget; import io.tokenpilot.core.domain.AdmissionReason; import io.tokenpilot.core.domain.AdmissionStatus; import io.tokenpilot.core.domain.BudgetResult; import io.tokenpilot.core.domain.ModelDefinition; +import io.tokenpilot.core.domain.PreflightDecisionEvent; import io.tokenpilot.core.domain.TokenCountResult; import io.tokenpilot.core.domain.TokenCountScope; +import java.util.List; import java.util.Objects; import java.util.Optional; import java.util.OptionalLong; @@ -19,9 +22,20 @@ final class DefaultTokenBudget implements TokenBudget { private final ModelRegistry modelRegistry; + private final List listeners; DefaultTokenBudget(ModelRegistry modelRegistry) { + this(modelRegistry, List.of()); + } + + DefaultTokenBudget( + ModelRegistry modelRegistry, + List listeners + ) { this.modelRegistry = Objects.requireNonNull(modelRegistry, "modelRegistry must not be null"); + this.listeners = List.copyOf( + Objects.requireNonNull(listeners, "listeners must not be null") + ); } @Override @@ -146,7 +160,7 @@ private BudgetResult buildResult( OptionalLong maxContextTokens, OptionalLong remainingTokens ) { - return new BudgetResult( + BudgetResult result = new BudgetResult( status, reason, canonicalModelId, @@ -158,5 +172,21 @@ private BudgetResult buildResult( input.estimatorDescriptor(), input.tokenizationBasis() ); + publishBestEffort(result); + return result; + } + + private void publishBestEffort(BudgetResult result) { + if (listeners.isEmpty()) { + return; + } + PreflightDecisionEvent event = new PreflightDecisionEvent(result); + for (PreflightDecisionListener listener : listeners) { + try { + listener.onDecision(event); + } catch (RuntimeException ignored) { + // Optional observers do not change a completed admission decision. + } + } } } diff --git a/token-pilot-core/src/main/java/io/tokenpilot/core/internal/LedgerComponents.java b/token-pilot-core/src/main/java/io/tokenpilot/core/internal/LedgerComponents.java index 461f71c..6267212 100644 --- a/token-pilot-core/src/main/java/io/tokenpilot/core/internal/LedgerComponents.java +++ b/token-pilot-core/src/main/java/io/tokenpilot/core/internal/LedgerComponents.java @@ -8,6 +8,7 @@ import io.tokenpilot.core.PricingProvider; import io.tokenpilot.core.PricingRegistry; import io.tokenpilot.core.PreflightCostEstimator; +import io.tokenpilot.core.PreflightDecisionListener; import io.tokenpilot.core.TokenBudget; import io.tokenpilot.core.TokenEstimator; import io.tokenpilot.core.domain.ModelDefinition; @@ -63,6 +64,16 @@ public static TokenBudget tokenBudget(ModelRegistry modelRegistry) { return new DefaultTokenBudget(modelRegistry); } + /** + * model registry와 best-effort decision listener를 사용하는 context admission evaluator를 생성합니다. + */ + public static TokenBudget tokenBudget( + ModelRegistry modelRegistry, + List listeners + ) { + return new DefaultTokenBudget(modelRegistry, listeners); + } + /** * 등록할 model definition으로 in-memory model registry를 생성합니다. * diff --git a/token-pilot-core/src/test/java/io/tokenpilot/core/domain/PricingMissingEventTest.java b/token-pilot-core/src/test/java/io/tokenpilot/core/domain/PricingMissingEventTest.java new file mode 100644 index 0000000..9f9c13e --- /dev/null +++ b/token-pilot-core/src/test/java/io/tokenpilot/core/domain/PricingMissingEventTest.java @@ -0,0 +1,38 @@ +package io.tokenpilot.core.domain; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +class PricingMissingEventTest { + + @Test + @DisplayName("missing pricing policy와 bounded resolution을 보존한다") + void preservesPolicyAndResolution() { + PricingMissingEvent event = new PricingMissingEvent( + MissingPricingPolicy.FAIL_CLOSED, + PricingResolution.MISSING_RATE + ); + + assertThat(event.policy()).isEqualTo(MissingPricingPolicy.FAIL_CLOSED); + assertThat(event.resolution()).isEqualTo(PricingResolution.MISSING_RATE); + } + + @Test + @DisplayName("resolved와 currency mismatch는 missing pricing event가 될 수 없다") + void rejectsNonMissingPricing() { + for (PricingResolution resolution : new PricingResolution[] { + PricingResolution.RESOLVED, + PricingResolution.CURRENCY_MISMATCH + }) { + assertThatThrownBy(() -> new PricingMissingEvent( + MissingPricingPolicy.FAIL_OPEN, + resolution + )) + .isInstanceOf(IllegalArgumentException.class) + .hasMessage("resolution must identify missing pricing"); + } + } +} diff --git a/token-pilot-core/src/test/java/io/tokenpilot/core/internal/DefaultLedgerManagerTest.java b/token-pilot-core/src/test/java/io/tokenpilot/core/internal/DefaultLedgerManagerTest.java index ab7d258..56617ca 100644 --- a/token-pilot-core/src/test/java/io/tokenpilot/core/internal/DefaultLedgerManagerTest.java +++ b/token-pilot-core/src/test/java/io/tokenpilot/core/internal/DefaultLedgerManagerTest.java @@ -132,4 +132,40 @@ void shouldRecordZeroCostWithExplicitZeroRateSnapshot() { event.cost().equals(cost) )); } + + @Test + @DisplayName("한 리스너의 RuntimeException은 비용 결과와 다음 리스너 호출에 영향을 주지 않아야 한다") + void shouldIsolateListenerRuntimeExceptionAndContinuePublishing() { + PricingPlan plan = new PricingPlan( + "gpt-4o", + new BigDecimal("5.0"), + new BigDecimal("15.0") + ); + registry.registerPlan(plan); + LedgerListener failingListener = mock(LedgerListener.class); + LedgerListener succeedingListener = mock(LedgerListener.class); + DefaultLedgerManager isolatedManager = new DefaultLedgerManager( + registry, + calculator, + List.of(failingListener, succeedingListener) + ); + TokenUsage usage = TokenUsage.from(1_000, 1_000); + doThrow(new IllegalStateException("listener failed")) + .when(failingListener) + .onRecord(any(CostRecordedEvent.class)); + + Cost cost = isolatedManager.record("gpt-4o", usage, Map.of()); + + assertThat(cost).isEqualTo(Cost.of( + new BigDecimal("20.000000"), + Currency.getInstance("USD") + )); + var inOrder = inOrder(failingListener, succeedingListener); + inOrder.verify(failingListener).onRecord(any(CostRecordedEvent.class)); + inOrder.verify(succeedingListener).onRecord(argThat(event -> + event.modelId().equals("gpt-4o") + && event.usage().equals(usage) + && event.cost().equals(cost) + )); + } } diff --git a/token-pilot-core/src/test/java/io/tokenpilot/core/internal/DefaultTokenBudgetTest.java b/token-pilot-core/src/test/java/io/tokenpilot/core/internal/DefaultTokenBudgetTest.java index 4771dcb..e826809 100644 --- a/token-pilot-core/src/test/java/io/tokenpilot/core/internal/DefaultTokenBudgetTest.java +++ b/token-pilot-core/src/test/java/io/tokenpilot/core/internal/DefaultTokenBudgetTest.java @@ -1,11 +1,13 @@ package io.tokenpilot.core.internal; +import io.tokenpilot.core.CoreComponents; import io.tokenpilot.core.ModelRegistry; import io.tokenpilot.core.TokenBudget; import io.tokenpilot.core.domain.AdmissionReason; import io.tokenpilot.core.domain.AdmissionStatus; import io.tokenpilot.core.domain.BudgetResult; import io.tokenpilot.core.domain.ModelDefinition; +import io.tokenpilot.core.domain.PreflightDecisionEvent; import io.tokenpilot.core.domain.TokenCountAccuracy; import io.tokenpilot.core.domain.TokenCountResult; import io.tokenpilot.core.domain.TokenCountScope; @@ -18,6 +20,8 @@ import java.time.Instant; import java.util.List; import java.util.Set; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.atomic.AtomicInteger; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; @@ -160,6 +164,49 @@ void rejectsNegativeReservedOutputAndRequiresFits() { .hasMessageContaining("INCOMPLETE_SCOPE"); } + @Test + @DisplayName("모든 admission 결과를 immutable event로 한 번씩 발행한다") + void publishesEveryAdmissionResultOnce() { + List events = new CopyOnWriteArrayList<>(); + TokenBudget observed = CoreComponents.tokenBudget( + registry, + List.of(events::add) + ); + + BudgetResult fits = observed.check("model-v1", request(6, 6), 4); + BudgetResult exceeds = observed.check("model-v1", request(6, 6), 5); + BudgetResult indeterminate = observed.check("unknown", request(1, 1), 0); + + assertThat(events).containsExactly( + new PreflightDecisionEvent(fits), + new PreflightDecisionEvent(exceeds), + new PreflightDecisionEvent(indeterminate) + ); + } + + @Test + @DisplayName("한 preflight listener 실패가 결과나 다음 listener를 바꾸지 않는다") + void isolatesPreflightListenerFailures() { + AtomicInteger failedDeliveries = new AtomicInteger(); + List received = new CopyOnWriteArrayList<>(); + TokenBudget observed = LedgerComponents.tokenBudget( + registry, + List.of( + event -> { + failedDeliveries.incrementAndGet(); + throw new IllegalStateException("listener failed"); + }, + received::add + ) + ); + + BudgetResult result = observed.check("model-v1", request(6, 6), 4); + + assertThat(result.status()).isEqualTo(AdmissionStatus.FITS); + assertThat(failedDeliveries).hasValue(1); + assertThat(received).containsExactly(new PreflightDecisionEvent(result)); + } + private static TokenCountResult request(long tokens, long safeUpperBound) { return counted(tokens, safeUpperBound, TokenCountScope.REQUEST); } diff --git a/token-pilot-micrometer/build.gradle b/token-pilot-micrometer/build.gradle index 4f836b2..9a31486 100644 --- a/token-pilot-micrometer/build.gradle +++ b/token-pilot-micrometer/build.gradle @@ -1,5 +1,9 @@ dependencies { api project(':token-pilot-core') - implementation 'io.micrometer:micrometer-observation:1.17.0' - implementation 'io.micrometer:micrometer-core:1.17.0' + api 'io.micrometer:micrometer-core:1.17.0' + compileOnly project(':token-pilot-budget') + compileOnly project(':token-pilot-notification') + + testImplementation project(':token-pilot-budget') + testImplementation project(':token-pilot-notification') } diff --git a/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/MetricsOptions.java b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/MetricsOptions.java index 01cf3ea..343e470 100644 --- a/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/MetricsOptions.java +++ b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/MetricsOptions.java @@ -6,23 +6,49 @@ * Micrometer metric publishing options. * * @param allowedTagKeys metric tag keys allowed beyond the built-in low-cardinality tags + * @param legacyAiTokenMetricsEnabled whether the legacy {@code ai.token.*} meters are enabled */ -public record MetricsOptions(Set allowedTagKeys) { +public record MetricsOptions( + Set allowedTagKeys, + boolean legacyAiTokenMetricsEnabled +) { - public static final Set DEFAULT_ALLOWED_TAG_KEYS = Set.of("tenant_id"); + public static final Set DEFAULT_ALLOWED_TAG_KEYS = Set.of(); + private static final Set LEGACY_DEFAULT_ALLOWED_TAG_KEYS = + Set.of("tenant_id"); public MetricsOptions { allowedTagKeys = normalize(allowedTagKeys); } + /** + * 기존 직접 생성 코드의 source/binary 호환성을 유지하는 생성자입니다. + * + *

이 생성자를 직접 사용하는 것은 legacy {@code ai.token.*} meter에 대한 명시적 + * opt-in으로 취급됩니다. Starter 자동 설정 기본값은 {@link #defaults()}를 사용하여 + * legacy meter를 비활성화합니다.

+ */ + public MetricsOptions(Set allowedTagKeys) { + this( + allowedTagKeys == null + ? LEGACY_DEFAULT_ALLOWED_TAG_KEYS + : allowedTagKeys, + true + ); + } + public static MetricsOptions defaults() { - return new MetricsOptions(DEFAULT_ALLOWED_TAG_KEYS); + return new MetricsOptions(DEFAULT_ALLOWED_TAG_KEYS, false); } public static MetricsOptions withAllowedTagKeys(Set allowedTagKeys) { return new MetricsOptions(allowedTagKeys); } + public static MetricsOptions legacyDefaults() { + return new MetricsOptions(LEGACY_DEFAULT_ALLOWED_TAG_KEYS, true); + } + private static Set normalize(Set allowedTagKeys) { if (allowedTagKeys == null) { return DEFAULT_ALLOWED_TAG_KEYS; diff --git a/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/BudgetMetricsPublisher.java b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/BudgetMetricsPublisher.java new file mode 100644 index 0000000..8b46648 --- /dev/null +++ b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/BudgetMetricsPublisher.java @@ -0,0 +1,161 @@ +package io.tokenpilot.micrometer.internal; + +import io.micrometer.core.instrument.Counter; +import io.micrometer.core.instrument.DistributionSummary; +import io.micrometer.core.instrument.MeterRegistry; +import io.tokenpilot.budget.AccountingTransitionStatus; +import io.tokenpilot.budget.BudgetReservationRequest; +import io.tokenpilot.budget.BudgetReservationResult; +import io.tokenpilot.budget.BudgetSnapshot; +import io.tokenpilot.budget.ReservationAccountingEvent; +import io.tokenpilot.budget.ReservationAccountingListener; +import io.tokenpilot.budget.ReservationAccountingListenerErrorHandler; +import io.tokenpilot.budget.ReservationAccountingListenerFailureEvent; +import io.tokenpilot.budget.ReservationAccountingListenerType; +import io.tokenpilot.budget.ReservationAccountingReason; +import io.tokenpilot.budget.ReservationReconciliation; +import io.tokenpilot.budget.ReservationReconciliationRequiredEvent; +import io.tokenpilot.budget.ReservationState; + +import java.math.BigInteger; +import java.util.Objects; + +/** Atomic budget/accounting 결과를 Token Pilot 고유 Micrometer meter로 투영합니다. */ +public final class BudgetMetricsPublisher + implements ReservationAccountingListener, ReservationAccountingListenerErrorHandler { + + private final MeterRegistry meterRegistry; + + public BudgetMetricsPublisher(MeterRegistry meterRegistry) { + this.meterRegistry = Objects.requireNonNull( + meterRegistry, + "meterRegistry must not be null" + ); + } + + @Override + public void onCommitted(ReservationAccountingEvent event) { + recordCommitted(event); + } + + @Override + public void onAccountingApplied( + ReservationAccountingEvent event, + BudgetSnapshot snapshot + ) { + Objects.requireNonNull(snapshot, "snapshot must not be null"); + recordCommitted(event); + } + + @Override + public void onReconciliationRequired( + ReservationReconciliationRequiredEvent event + ) { + Objects.requireNonNull(event, "event must not be null"); + if (event.transition().status() != AccountingTransitionStatus.APPLIED + || event.transition().resultingState() + != ReservationState.RECONCILIATION_REQUIRED) { + return; + } + recordReconciliationOutcome( + event.transition().resultingState(), + event.reason() + ); + } + + @Override + public void onReservationEvaluated( + BudgetReservationRequest request, + BudgetReservationResult result + ) { + Objects.requireNonNull(request, "request must not be null"); + Objects.requireNonNull(result, "result must not be null"); + Counter.builder(TokenPilotMeterMetadata.BUDGET_RESERVATIONS) + .description(TokenPilotMeterMetadata.BUDGET_RESERVATIONS_DESCRIPTION) + .baseUnit(TokenPilotMeterMetadata.RESERVATIONS_BASE_UNIT) + .tag("state", TokenPilotMeterMetadata.tagValue(result.status())) + .register(meterRegistry) + .increment(); + } + + @Override + public ReservationAccountingListenerType listenerType() { + return ReservationAccountingListenerType.METRICS; + } + + @Override + public void onFailure(ReservationAccountingListenerFailureEvent event) { + Objects.requireNonNull(event, "event must not be null"); + Counter.builder(TokenPilotMeterMetadata.LISTENER_FAILURES) + .description(TokenPilotMeterMetadata.LISTENER_FAILURES_DESCRIPTION) + .baseUnit(TokenPilotMeterMetadata.FAILURES_BASE_UNIT) + .tag("listener", TokenPilotMeterMetadata.tagValue(event.listenerType())) + .tag("phase", TokenPilotMeterMetadata.tagValue(event.phase())) + .register(meterRegistry) + .increment(); + } + + private void recordCommitted(ReservationAccountingEvent event) { + Objects.requireNonNull(event, "event must not be null"); + ReservationReconciliation reconciliation = event.reconciliation(); + if (reconciliation.transition().status() != AccountingTransitionStatus.APPLIED + || reconciliation.transition().resultingState() + != ReservationState.COMMITTED) { + return; + } + + Counter.builder(TokenPilotMeterMetadata.COST_TOTAL) + .description(TokenPilotMeterMetadata.COST_TOTAL_DESCRIPTION) + .baseUnit(TokenPilotMeterMetadata.COST_BASE_UNIT) + .tag("currency", reconciliation.currency().getCurrencyCode()) + .register(meterRegistry) + .increment(reconciliation.actual().value().doubleValue()); + + BigInteger delta = totalTokenDelta(reconciliation); + String direction = delta.signum() > 0 + ? "underestimate" + : delta.signum() < 0 ? "overestimate" : "exact"; + DistributionSummary.builder(TokenPilotMeterMetadata.RECONCILIATION_ERROR_TOKENS) + .description(TokenPilotMeterMetadata.RECONCILIATION_ERROR_TOKENS_DESCRIPTION) + .baseUnit(TokenPilotMeterMetadata.TOKENS_BASE_UNIT) + .tag("direction", direction) + .register(meterRegistry) + .record(delta.abs().doubleValue()); + + recordReconciliationOutcome( + reconciliation.transition().resultingState(), + reconciliation.reason() + ); + } + + private void recordReconciliationOutcome( + ReservationState outcome, + ReservationAccountingReason reason + ) { + Counter.builder(TokenPilotMeterMetadata.RECONCILIATION_OUTCOMES) + .description(TokenPilotMeterMetadata.RECONCILIATION_OUTCOMES_DESCRIPTION) + .baseUnit(TokenPilotMeterMetadata.RECONCILIATIONS_BASE_UNIT) + .tag("outcome", TokenPilotMeterMetadata.tagValue(outcome)) + .tag("reason", TokenPilotMeterMetadata.tagValue(reason)) + .register(meterRegistry) + .increment(); + } + + private static BigInteger totalTokenDelta( + ReservationReconciliation reconciliation + ) { + BigInteger actual = BigInteger.valueOf( + reconciliation.actualTokens().inputTokens() + ) + .add(BigInteger.valueOf( + reconciliation.actualTokens().outputTokens() + )); + BigInteger estimated = BigInteger.valueOf( + reconciliation.tokenEstimate().inputEstimatedTokens() + ) + .add(BigInteger.valueOf( + reconciliation.tokenEstimate().reservedOutputTokens() + )); + return actual.subtract(estimated); + } +} diff --git a/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/BudgetMicrometerComponents.java b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/BudgetMicrometerComponents.java new file mode 100644 index 0000000..754185a --- /dev/null +++ b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/BudgetMicrometerComponents.java @@ -0,0 +1,16 @@ +package io.tokenpilot.micrometer.internal; + +import io.micrometer.core.instrument.MeterRegistry; + +/** Atomic budget owner의 선택 Micrometer publisher 팩토리입니다. */ +public final class BudgetMicrometerComponents { + + private BudgetMicrometerComponents() { + } + + public static BudgetMetricsPublisher budgetMetricsPublisher( + MeterRegistry meterRegistry + ) { + return new BudgetMetricsPublisher(meterRegistry); + } +} diff --git a/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/CoreMetricsPublisher.java b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/CoreMetricsPublisher.java new file mode 100644 index 0000000..7672c76 --- /dev/null +++ b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/CoreMetricsPublisher.java @@ -0,0 +1,48 @@ +package io.tokenpilot.micrometer.internal; + +import io.micrometer.core.instrument.Counter; +import io.micrometer.core.instrument.MeterRegistry; +import io.tokenpilot.core.PreflightDecisionListener; +import io.tokenpilot.core.PricingMissingListener; +import io.tokenpilot.core.domain.PreflightDecisionEvent; +import io.tokenpilot.core.domain.PricingMissingEvent; + +import java.util.Objects; + +/** Token Pilot core의 preflight와 pricing-miss 이벤트를 Micrometer meter로 투영합니다. */ +public final class CoreMetricsPublisher + implements PreflightDecisionListener, PricingMissingListener { + + private final MeterRegistry meterRegistry; + + public CoreMetricsPublisher(MeterRegistry meterRegistry) { + this.meterRegistry = Objects.requireNonNull( + meterRegistry, + "meterRegistry must not be null" + ); + } + + @Override + public void onDecision(PreflightDecisionEvent event) { + Objects.requireNonNull(event, "event must not be null"); + var result = event.result(); + Counter.builder(TokenPilotMeterMetadata.PREFLIGHT_REQUESTS) + .description(TokenPilotMeterMetadata.PREFLIGHT_REQUESTS_DESCRIPTION) + .baseUnit(TokenPilotMeterMetadata.REQUESTS_BASE_UNIT) + .tag("decision", TokenPilotMeterMetadata.tagValue(result.status())) + .tag("reason", TokenPilotMeterMetadata.tagValue(result.reason())) + .register(meterRegistry) + .increment(); + } + + @Override + public void onPricingMissing(PricingMissingEvent event) { + Objects.requireNonNull(event, "event must not be null"); + Counter.builder(TokenPilotMeterMetadata.PRICING_MISSING) + .description(TokenPilotMeterMetadata.PRICING_MISSING_DESCRIPTION) + .baseUnit(TokenPilotMeterMetadata.EVENTS_BASE_UNIT) + .tag("policy", TokenPilotMeterMetadata.tagValue(event.policy())) + .register(meterRegistry) + .increment(); + } +} diff --git a/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/LedgerMicrometerComponents.java b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/LedgerMicrometerComponents.java index 8fde10a..404d760 100644 --- a/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/LedgerMicrometerComponents.java +++ b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/LedgerMicrometerComponents.java @@ -21,4 +21,16 @@ public static LedgerListener microCostMetricsPublisher(MeterRegistry meterRegist public static LedgerListener microCostMetricsPublisher(MeterRegistry meterRegistry, MetricsOptions options) { return new MicroCostMetricsPublisher(meterRegistry, options); } + + public static MicroCostMetricsPublisher legacyMicroCostMetricsPublisher( + MeterRegistry meterRegistry, + MetricsOptions options + ) { + return new MicroCostMetricsPublisher(meterRegistry, options); + } + + public static CoreMetricsPublisher coreMetricsPublisher(MeterRegistry meterRegistry) { + return new CoreMetricsPublisher(meterRegistry); + } + } diff --git a/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/MicroCostMetricsPublisher.java b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/MicroCostMetricsPublisher.java index 1744581..8c533e6 100644 --- a/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/MicroCostMetricsPublisher.java +++ b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/MicroCostMetricsPublisher.java @@ -10,7 +10,9 @@ import io.tokenpilot.core.domain.TokenType; import io.tokenpilot.micrometer.MetricsOptions; +import java.util.Locale; import java.util.Map; +import java.util.Objects; import java.util.Set; /** @@ -22,9 +24,10 @@ public class MicroCostMetricsPublisher implements LedgerListener { private final MeterRegistry meterRegistry; private final Set allowedTagKeys; + private final boolean enabled; public MicroCostMetricsPublisher(MeterRegistry meterRegistry) { - this(meterRegistry, MetricsOptions.defaults()); + this(meterRegistry, MetricsOptions.legacyDefaults()); } public MicroCostMetricsPublisher(MeterRegistry meterRegistry, Set allowedTagKeys) { @@ -32,13 +35,23 @@ public MicroCostMetricsPublisher(MeterRegistry meterRegistry, Set allowe } public MicroCostMetricsPublisher(MeterRegistry meterRegistry, MetricsOptions options) { - this.meterRegistry = meterRegistry; - MetricsOptions resolvedOptions = (options != null) ? options : MetricsOptions.defaults(); + this.meterRegistry = Objects.requireNonNull( + meterRegistry, + "meterRegistry must not be null" + ); + MetricsOptions resolvedOptions = (options != null) + ? options + : MetricsOptions.legacyDefaults(); this.allowedTagKeys = resolvedOptions.allowedTagKeys(); + this.enabled = resolvedOptions.legacyAiTokenMetricsEnabled(); } @Override public void onRecord(CostRecordedEvent event) { + if (!enabled) { + return; + } + Objects.requireNonNull(event, "event must not be null"); Tags commonTags = Tags.of("model", event.modelId()) .and(allowedTags(event.tags())); @@ -47,7 +60,7 @@ public void onRecord(CostRecordedEvent event) { for (TokenType tokenType : TokenType.values()) { long count = event.usage().getCount(tokenType); if (count > 0) { - String typeName = tokenType.name().toLowerCase(); + String typeName = tokenType.name().toLowerCase(Locale.ROOT); DistributionSummary.builder("ai.token.usage.distribution") .description("Distribution of AI token usage per recorded model call") .baseUnit("tokens") diff --git a/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/NotificationMetricsPublisher.java b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/NotificationMetricsPublisher.java new file mode 100644 index 0000000..04e35b3 --- /dev/null +++ b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/NotificationMetricsPublisher.java @@ -0,0 +1,34 @@ +package io.tokenpilot.micrometer.internal; + +import io.micrometer.core.instrument.Counter; +import io.micrometer.core.instrument.MeterRegistry; +import io.tokenpilot.notification.BudgetNotificationLifecycleEvent; +import io.tokenpilot.notification.BudgetNotificationLifecycleListener; + +import java.util.Objects; + +/** Budget notification lifecycle 이벤트를 Micrometer meter로 투영합니다. */ +public final class NotificationMetricsPublisher + implements BudgetNotificationLifecycleListener { + + private final MeterRegistry meterRegistry; + + public NotificationMetricsPublisher(MeterRegistry meterRegistry) { + this.meterRegistry = Objects.requireNonNull( + meterRegistry, + "meterRegistry must not be null" + ); + } + + @Override + public void onNotificationLifecycle(BudgetNotificationLifecycleEvent event) { + Objects.requireNonNull(event, "event must not be null"); + Counter.builder(TokenPilotMeterMetadata.NOTIFICATION_EVENTS) + .description(TokenPilotMeterMetadata.NOTIFICATION_EVENTS_DESCRIPTION) + .baseUnit(TokenPilotMeterMetadata.EVENTS_BASE_UNIT) + .tag("outcome", TokenPilotMeterMetadata.tagValue(event.outcome())) + .tag("threshold", TokenPilotMeterMetadata.tagValue(event.threshold())) + .register(meterRegistry) + .increment(); + } +} diff --git a/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/NotificationMicrometerComponents.java b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/NotificationMicrometerComponents.java new file mode 100644 index 0000000..023e2ee --- /dev/null +++ b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/NotificationMicrometerComponents.java @@ -0,0 +1,16 @@ +package io.tokenpilot.micrometer.internal; + +import io.micrometer.core.instrument.MeterRegistry; + +/** Budget notification owner의 선택 Micrometer publisher 팩토리입니다. */ +public final class NotificationMicrometerComponents { + + private NotificationMicrometerComponents() { + } + + public static NotificationMetricsPublisher notificationMetricsPublisher( + MeterRegistry meterRegistry + ) { + return new NotificationMetricsPublisher(meterRegistry); + } +} diff --git a/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/TokenPilotMeterMetadata.java b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/TokenPilotMeterMetadata.java new file mode 100644 index 0000000..e12ee29 --- /dev/null +++ b/token-pilot-micrometer/src/main/java/io/tokenpilot/micrometer/internal/TokenPilotMeterMetadata.java @@ -0,0 +1,54 @@ +package io.tokenpilot.micrometer.internal; + +import java.util.Locale; + +final class TokenPilotMeterMetadata { + + static final String COST_TOTAL = "tokenpilot.cost.total"; + static final String COST_TOTAL_DESCRIPTION = + "Total actual LLM cost committed by Token Pilot"; + static final String COST_BASE_UNIT = "currency"; + + static final String PREFLIGHT_REQUESTS = "tokenpilot.preflight.requests"; + static final String PREFLIGHT_REQUESTS_DESCRIPTION = + "Total preflight admission decisions made by Token Pilot"; + static final String REQUESTS_BASE_UNIT = "requests"; + + static final String BUDGET_RESERVATIONS = "tokenpilot.budget.reservations"; + static final String BUDGET_RESERVATIONS_DESCRIPTION = + "Total budget reservation results produced by Token Pilot"; + static final String RESERVATIONS_BASE_UNIT = "reservations"; + + static final String RECONCILIATION_ERROR_TOKENS = + "tokenpilot.reconciliation.error.tokens"; + static final String RECONCILIATION_ERROR_TOKENS_DESCRIPTION = + "Absolute difference between estimated and actual total tokens per reconciliation"; + static final String TOKENS_BASE_UNIT = "tokens"; + + static final String RECONCILIATION_OUTCOMES = + "tokenpilot.reconciliation.outcomes"; + static final String RECONCILIATION_OUTCOMES_DESCRIPTION = + "Total reservation reconciliation outcomes applied by Token Pilot"; + static final String RECONCILIATIONS_BASE_UNIT = "reconciliations"; + + static final String PRICING_MISSING = "tokenpilot.pricing.missing"; + static final String PRICING_MISSING_DESCRIPTION = + "Total pricing misses observed by Token Pilot"; + + static final String LISTENER_FAILURES = "tokenpilot.listener.failures"; + static final String LISTENER_FAILURES_DESCRIPTION = + "Total listener failures isolated by Token Pilot"; + static final String FAILURES_BASE_UNIT = "failures"; + + static final String NOTIFICATION_EVENTS = "tokenpilot.notification.events"; + static final String NOTIFICATION_EVENTS_DESCRIPTION = + "Total budget notification lifecycle events observed by Token Pilot"; + static final String EVENTS_BASE_UNIT = "events"; + + private TokenPilotMeterMetadata() { + } + + static String tagValue(Enum value) { + return value.name().toLowerCase(Locale.ROOT); + } +} diff --git a/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/BudgetMetricsPublisherTest.java b/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/BudgetMetricsPublisherTest.java new file mode 100644 index 0000000..ca2eb41 --- /dev/null +++ b/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/BudgetMetricsPublisherTest.java @@ -0,0 +1,379 @@ +package io.tokenpilot.micrometer.internal; + +import io.micrometer.core.instrument.Meter; +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; +import io.tokenpilot.budget.AccountingTransitionStatus; +import io.tokenpilot.budget.BudgetKey; +import io.tokenpilot.budget.BudgetReservationRequest; +import io.tokenpilot.budget.BudgetReservationResult; +import io.tokenpilot.budget.BudgetSnapshot; +import io.tokenpilot.budget.BudgetWindow; +import io.tokenpilot.budget.IdempotencyKey; +import io.tokenpilot.budget.ReservationAccountingEvent; +import io.tokenpilot.budget.ReservationAccountingListenerFailureEvent; +import io.tokenpilot.budget.ReservationAccountingListenerPhase; +import io.tokenpilot.budget.ReservationAccountingListenerType; +import io.tokenpilot.budget.ReservationAccountingReason; +import io.tokenpilot.budget.ReservationActualTokens; +import io.tokenpilot.budget.ReservationId; +import io.tokenpilot.budget.ReservationReconciliation; +import io.tokenpilot.budget.ReservationReconciliationRequiredEvent; +import io.tokenpilot.budget.ReservationState; +import io.tokenpilot.budget.ReservationTokenEstimate; +import io.tokenpilot.budget.ReservationTransition; +import io.tokenpilot.core.domain.Cost; +import io.tokenpilot.core.domain.PricingSnapshot; +import io.tokenpilot.core.domain.TokenType; +import io.tokenpilot.core.domain.TokenUsageDetails; +import io.tokenpilot.core.domain.UsageSource; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.math.BigDecimal; +import java.time.Instant; +import java.util.Currency; +import java.util.Map; +import java.util.Optional; +import java.util.Set; + +import static org.assertj.core.api.Assertions.assertThat; + +class BudgetMetricsPublisherTest { + + private static final Currency USD = Currency.getInstance("USD"); + private static final ReservationId RESERVATION_ID = + new ReservationId("sensitive-reservation-id"); + private static final BudgetKey BUDGET_KEY = new BudgetKey( + "monthly", + "tenant", + "sensitive-tenant-id", + BudgetWindow.parse("2026-08") + ); + private static final Cost LIMIT = usd("10.00"); + + private SimpleMeterRegistry meterRegistry; + private BudgetMetricsPublisher publisher; + + @BeforeEach + void setUp() { + meterRegistry = new SimpleMeterRegistry(); + publisher = new BudgetMetricsPublisher(meterRegistry); + } + + @Test + void publishesActualCostErrorAndOutcomeOnlyForAppliedCommit() { + ReservationAccountingEvent applied = accountingEvent( + ReservationTransition.applied( + ReservationState.IN_FLIGHT, + ReservationState.COMMITTED + ), + 110, + 60, + "0.80" + ); + + publisher.onAccountingApplied(applied, snapshot()); + + var cost = meterRegistry.find("tokenpilot.cost.total") + .tag("currency", "USD") + .counter(); + assertThat(cost).isNotNull(); + assertThat(cost.count()).isEqualTo(0.8); + assertThat(cost.getId().getType()).isEqualTo(Meter.Type.COUNTER); + assertThat(cost.getId().getBaseUnit()).isEqualTo("currency"); + assertThat(cost.getId().getDescription()) + .isEqualTo("Total actual LLM cost committed by Token Pilot"); + assertThat(cost.getId().getTags()).extracting(tag -> tag.getKey()) + .containsExactly("currency"); + + var error = meterRegistry.find("tokenpilot.reconciliation.error.tokens") + .tag("direction", "underestimate") + .summary(); + assertThat(error).isNotNull(); + assertThat(error.count()).isEqualTo(1L); + assertThat(error.totalAmount()).isEqualTo(20.0); + assertThat(error.getId().getType()).isEqualTo(Meter.Type.DISTRIBUTION_SUMMARY); + assertThat(error.getId().getBaseUnit()).isEqualTo("tokens"); + assertThat(error.getId().getDescription()).isEqualTo( + "Absolute difference between estimated and actual total tokens per reconciliation" + ); + + var outcome = meterRegistry.find("tokenpilot.reconciliation.outcomes") + .tag("outcome", "committed") + .tag("reason", "actual_usage_reported") + .counter(); + assertThat(outcome).isNotNull(); + assertThat(outcome.count()).isEqualTo(1.0); + assertThat(outcome.getId().getBaseUnit()).isEqualTo("reconciliations"); + assertThat(outcome.getId().getDescription()) + .isEqualTo("Total reservation reconciliation outcomes applied by Token Pilot"); + } + + @Test + void ignoresReusedCommitSoCostAndOutcomeAreNotDuplicated() { + ReservationAccountingEvent applied = accountingEvent( + ReservationTransition.applied( + ReservationState.IN_FLIGHT, + ReservationState.COMMITTED + ), + 100, + 50, + "0.80" + ); + ReservationAccountingEvent reused = accountingEvent( + ReservationTransition.unchanged( + ReservationState.COMMITTED, + AccountingTransitionStatus.REUSED + ), + 100, + 50, + "0.80" + ); + + publisher.onCommitted(applied); + publisher.onCommitted(reused); + + assertThat(meterRegistry.find("tokenpilot.cost.total").counter().count()) + .isEqualTo(0.8); + assertThat(meterRegistry.find("tokenpilot.reconciliation.outcomes") + .counter().count()).isEqualTo(1.0); + assertThat(meterRegistry.find("tokenpilot.reconciliation.error.tokens") + .summary().count()).isEqualTo(1L); + } + + @Test + void explicitZeroCommitRemainsDistinctFromMissingPricing() { + publisher.onCommitted(accountingEvent( + ReservationTransition.applied( + ReservationState.IN_FLIGHT, + ReservationState.COMMITTED + ), + 100, + 50, + "0.00" + )); + + var cost = meterRegistry.find("tokenpilot.cost.total") + .tag("currency", "USD") + .counter(); + assertThat(cost).isNotNull(); + assertThat(cost.count()).isZero(); + assertThat(meterRegistry.find("tokenpilot.reconciliation.outcomes") + .tag("outcome", "committed") + .counter().count()).isEqualTo(1.0); + assertThat(meterRegistry.find("tokenpilot.pricing.missing").meter()) + .isNull(); + } + + @Test + void distinguishesOverestimateAndExactTokenErrors() { + publisher.onCommitted(accountingEvent( + ReservationTransition.applied( + ReservationState.IN_FLIGHT, + ReservationState.COMMITTED + ), + 90, + 50, + "0.40" + )); + publisher.onCommitted(accountingEvent( + ReservationTransition.applied( + ReservationState.IN_FLIGHT, + ReservationState.COMMITTED + ), + 100, + 50, + "0.40" + )); + + var overestimate = meterRegistry.find("tokenpilot.reconciliation.error.tokens") + .tag("direction", "overestimate") + .summary(); + var exact = meterRegistry.find("tokenpilot.reconciliation.error.tokens") + .tag("direction", "exact") + .summary(); + + assertThat(overestimate.count()).isEqualTo(1L); + assertThat(overestimate.totalAmount()).isEqualTo(10.0); + assertThat(exact.count()).isEqualTo(1L); + assertThat(exact.totalAmount()).isZero(); + } + + @Test + void reconciliationRequiredPublishesOutcomeWithoutZeroCostOrError() { + ReservationTransition transition = ReservationTransition.applied( + ReservationState.IN_FLIGHT, + ReservationState.RECONCILIATION_REQUIRED + ); + publisher.onReconciliationRequired(new ReservationReconciliationRequiredEvent( + RESERVATION_ID, + BUDGET_KEY, + transition, + ReservationAccountingReason.ACTUAL_USAGE_UNAVAILABLE, + snapshot() + )); + + var outcome = meterRegistry.find("tokenpilot.reconciliation.outcomes") + .tag("outcome", "reconciliation_required") + .tag("reason", "actual_usage_unavailable") + .counter(); + assertThat(outcome).isNotNull(); + assertThat(outcome.count()).isEqualTo(1.0); + assertThat(meterRegistry.find("tokenpilot.cost.total").meter()).isNull(); + assertThat(meterRegistry.find("tokenpilot.reconciliation.error.tokens") + .meter()).isNull(); + } + + @Test + void publishesEveryReservationEvaluationWithBoundedStateOnly() { + BudgetReservationRequest request = new BudgetReservationRequest( + BUDGET_KEY, + LIMIT, + usd("1.00"), + "sensitive-request-id", + new IdempotencyKey("sensitive-idempotency-id"), + null, + null, + null, + Optional.empty() + ); + BudgetReservationResult result = BudgetReservationResult.blocked( + snapshot(), + "raw rejection message that must not become a tag" + ); + + publisher.onReservationEvaluated(request, result); + + var counter = meterRegistry.find("tokenpilot.budget.reservations") + .tag("state", "blocked") + .counter(); + assertThat(counter).isNotNull(); + assertThat(counter.count()).isEqualTo(1.0); + assertThat(counter.getId().getBaseUnit()).isEqualTo("reservations"); + assertThat(counter.getId().getDescription()) + .isEqualTo("Total budget reservation results produced by Token Pilot"); + assertThat(counter.getId().getTags()).extracting(tag -> tag.getKey()) + .containsExactly("state"); + } + + @Test + void publishesBoundedListenerFailureAndIdentifiesAsMetricsListener() { + publisher.onFailure(new ReservationAccountingListenerFailureEvent( + ReservationAccountingListenerType.NOTIFICATION, + ReservationAccountingListenerPhase.RECONCILIATION_REQUIRED + )); + + var counter = meterRegistry.find("tokenpilot.listener.failures") + .tag("listener", "notification") + .tag("phase", "reconciliation_required") + .counter(); + assertThat(counter).isNotNull(); + assertThat(counter.count()).isEqualTo(1.0); + assertThat(counter.getId().getBaseUnit()).isEqualTo("failures"); + assertThat(counter.getId().getDescription()) + .isEqualTo("Total listener failures isolated by Token Pilot"); + assertThat(publisher.listenerType()) + .isEqualTo(ReservationAccountingListenerType.METRICS); + } + + @Test + void neverCopiesRawIdentifiersModelsOrMessagesIntoMeterMetadata() { + publisher.onCommitted(accountingEvent( + ReservationTransition.applied( + ReservationState.IN_FLIGHT, + ReservationState.COMMITTED + ), + 110, + 60, + "0.80" + )); + BudgetReservationRequest request = new BudgetReservationRequest( + BUDGET_KEY, + LIMIT, + usd("1.00"), + "sensitive-request-id", + new IdempotencyKey("sensitive-idempotency-id"), + null, + null, + null, + Optional.empty() + ); + publisher.onReservationEvaluated( + request, + BudgetReservationResult.blocked( + snapshot(), + "raw rejection message that must not become metadata" + ) + ); + + assertThat(meterRegistry.getMeters()) + .extracting(meter -> meter.getId().toString()) + .allSatisfy(metadata -> assertThat(metadata) + .doesNotContain( + "sensitive-reservation-id", + "sensitive-tenant-id", + "sensitive-request-id", + "sensitive-idempotency-id", + "raw-request-model", + "raw-response-model", + "raw rejection message" + )); + } + + private static ReservationAccountingEvent accountingEvent( + ReservationTransition transition, + long actualInput, + long actualOutput, + String actualCost + ) { + return new ReservationAccountingEvent(new ReservationReconciliation( + "sensitive-request-id", + "sensitive-attempt-id", + RESERVATION_ID, + BUDGET_KEY, + "raw-response-model", + pricingSnapshot(), + new ReservationTokenEstimate(100, 120, 50), + new ReservationActualTokens( + actualInput, + actualOutput, + TokenUsageDetails.unreported(), + UsageSource.PROVIDER_REPORTED + ), + usd("1.00"), + usd(actualCost), + false, + transition, + ReservationAccountingReason.ACTUAL_USAGE_REPORTED + )); + } + + private static PricingSnapshot pricingSnapshot() { + return new PricingSnapshot( + "raw-request-model", + "pricing-policy", + "catalog-version", + Instant.parse("2026-08-25T00:00:00Z"), + Map.of( + TokenType.PROMPT, new BigDecimal("0.001"), + TokenType.COMPLETION, new BigDecimal("0.002") + ), + USD + ); + } + + private static BudgetSnapshot snapshot() { + return new BudgetSnapshot( + BUDGET_KEY, + LIMIT, + Cost.zero(USD), + Cost.zero(USD), + Cost.zero(USD), + Set.of() + ); + } + + private static Cost usd(String amount) { + return Cost.of(new BigDecimal(amount), USD); + } +} diff --git a/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/CoreMetricsPublisherTest.java b/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/CoreMetricsPublisherTest.java new file mode 100644 index 0000000..7c047d8 --- /dev/null +++ b/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/CoreMetricsPublisherTest.java @@ -0,0 +1,87 @@ +package io.tokenpilot.micrometer.internal; + +import io.micrometer.core.instrument.Meter; +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; +import io.tokenpilot.core.domain.AdmissionReason; +import io.tokenpilot.core.domain.AdmissionStatus; +import io.tokenpilot.core.domain.BudgetResult; +import io.tokenpilot.core.domain.MissingPricingPolicy; +import io.tokenpilot.core.domain.PreflightDecisionEvent; +import io.tokenpilot.core.domain.PricingMissingEvent; +import io.tokenpilot.core.domain.PricingResolution; +import io.tokenpilot.core.domain.TokenEstimatorDescriptor; +import io.tokenpilot.core.domain.TokenizationBasis; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.util.Optional; +import java.util.OptionalLong; + +import static org.assertj.core.api.Assertions.assertThat; + +class CoreMetricsPublisherTest { + + private SimpleMeterRegistry meterRegistry; + private CoreMetricsPublisher publisher; + + @BeforeEach + void setUp() { + meterRegistry = new SimpleMeterRegistry(); + publisher = new CoreMetricsPublisher(meterRegistry); + } + + @Test + void publishesBoundedPreflightDecision() { + publisher.onDecision(new PreflightDecisionEvent(indeterminateResult())); + + var counter = meterRegistry.find("tokenpilot.preflight.requests") + .tag("decision", "indeterminate") + .tag("reason", "unknown_model") + .counter(); + + assertThat(counter).isNotNull(); + assertThat(counter.count()).isEqualTo(1.0); + assertThat(counter.getId().getType()).isEqualTo(Meter.Type.COUNTER); + assertThat(counter.getId().getBaseUnit()).isEqualTo("requests"); + assertThat(counter.getId().getDescription()) + .isEqualTo("Total preflight admission decisions made by Token Pilot"); + assertThat(counter.getId().getTags()).extracting(tag -> tag.getKey()) + .containsExactly("decision", "reason"); + } + + @Test + void publishesPricingMissWithoutResolutionOrRequestTags() { + publisher.onPricingMissing(new PricingMissingEvent( + MissingPricingPolicy.FAIL_CLOSED, + PricingResolution.MISSING_PLAN + )); + + var counter = meterRegistry.find("tokenpilot.pricing.missing") + .tag("policy", "fail_closed") + .counter(); + + assertThat(counter).isNotNull(); + assertThat(counter.count()).isEqualTo(1.0); + assertThat(counter.getId().getType()).isEqualTo(Meter.Type.COUNTER); + assertThat(counter.getId().getBaseUnit()).isEqualTo("events"); + assertThat(counter.getId().getDescription()) + .isEqualTo("Total pricing misses observed by Token Pilot"); + assertThat(counter.getId().getTags()).extracting(tag -> tag.getKey()) + .containsExactly("policy"); + } + + private static BudgetResult indeterminateResult() { + return new BudgetResult( + AdmissionStatus.INDETERMINATE, + AdmissionReason.UNKNOWN_MODEL, + Optional.empty(), + OptionalLong.empty(), + OptionalLong.empty(), + 0, + OptionalLong.empty(), + OptionalLong.empty(), + new TokenEstimatorDescriptor("test-estimator", "1"), + new TokenizationBasis("TEST") + ); + } +} diff --git a/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/LedgerMicrometerComponentsTest.java b/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/LedgerMicrometerComponentsTest.java new file mode 100644 index 0000000..293a0c4 --- /dev/null +++ b/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/LedgerMicrometerComponentsTest.java @@ -0,0 +1,40 @@ +package io.tokenpilot.micrometer.internal; + +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; +import io.tokenpilot.micrometer.MetricsOptions; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +class LedgerMicrometerComponentsTest { + + @Test + void createsOwnerSpecificPublishers() { + var meterRegistry = new SimpleMeterRegistry(); + + assertThat(LedgerMicrometerComponents.coreMetricsPublisher(meterRegistry)) + .isInstanceOf(CoreMetricsPublisher.class); + assertThat(BudgetMicrometerComponents.budgetMetricsPublisher(meterRegistry)) + .isInstanceOf(BudgetMetricsPublisher.class); + assertThat(NotificationMicrometerComponents.notificationMetricsPublisher(meterRegistry)) + .isInstanceOf(NotificationMetricsPublisher.class); + assertThat(LedgerMicrometerComponents.legacyMicroCostMetricsPublisher( + meterRegistry, + MetricsOptions.legacyDefaults() + )).isInstanceOf(MicroCostMetricsPublisher.class); + } + + @Test + void coreFactoryMethodTableDoesNotReferenceOptionalOwners() { + assertThat(LedgerMicrometerComponents.class.getDeclaredMethods()) + .allSatisfy(method -> { + assertThat(method.getReturnType().getName()) + .doesNotContain("BudgetMetricsPublisher") + .doesNotContain("NotificationMetricsPublisher"); + assertThat(method.getParameterTypes()) + .extracting(Class::getName) + .noneMatch(name -> name.startsWith("io.tokenpilot.budget")) + .noneMatch(name -> name.startsWith("io.tokenpilot.notification")); + }); + } +} diff --git a/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/MicroCostMetricsPublisherTest.java b/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/MicroCostMetricsPublisherTest.java index 319d63b..314bd9a 100644 --- a/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/MicroCostMetricsPublisherTest.java +++ b/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/MicroCostMetricsPublisherTest.java @@ -43,13 +43,11 @@ void shouldPublishMetricsWhenEventRecorded() { assertThat(meterRegistry.find("ai.token.usage.total") .tag("model", "gpt-4o") .tag("token_type", "prompt") - .tag("tenant_id", "tenant-1") .counter().count()).isEqualTo(100.0); assertThat(meterRegistry.find("ai.token.usage.total") .tag("model", "gpt-4o") .tag("token_type", "completion") - .tag("tenant_id", "tenant-1") .counter().count()).isEqualTo(200.0); // Then: 토큰 사용량 분포(Summary) 확인 @@ -61,7 +59,6 @@ void shouldPublishMetricsWhenEventRecorded() { // Then: 비용 카운터 확인 var costCounter = meterRegistry.find("ai.token.cost.total") .tag("model", "gpt-4o") - .tag("tenant_id", "tenant-1") .tag("currency", "USD") .counter(); @@ -71,13 +68,14 @@ void shouldPublishMetricsWhenEventRecorded() { var promptSummary = meterRegistry.find("ai.token.usage.distribution") .tag("model", "gpt-4o") - .tag("tenant_id", "tenant-1") .tag("token_type", "prompt") .summary(); assertThat(promptSummary.getId().getDescription()) .isEqualTo("Distribution of AI token usage per recorded model call"); assertThat(promptSummary.getId().getBaseUnit()).isEqualTo("tokens"); + assertThat(costCounter.getId().getTag("tenant_id")) + .isEqualTo("tenant-1"); } @Test @@ -172,4 +170,58 @@ void shouldPublishMetricsWhenTagsAreEmpty() { .tag("currency", "USD") .counter()).isNotNull(); } + + @Test + @DisplayName("기본 MetricsOptions는 사용자 태그와 legacy meter를 비활성화해야 한다") + void shouldUseSafeDefaults() { + MetricsOptions options = MetricsOptions.defaults(); + + assertThat(options.allowedTagKeys()).isEmpty(); + assertThat(options.legacyAiTokenMetricsEnabled()).isFalse(); + } + + @Test + @DisplayName("legacy 비활성 options를 사용하면 ai.token meter를 발행하지 않아야 한다") + void shouldNotPublishLegacyMetricsWhenDisabled() { + publisher = new MicroCostMetricsPublisher( + meterRegistry, + MetricsOptions.defaults() + ); + + publisher.onRecord(new CostRecordedEvent( + "gpt-4o", + TokenUsage.from(100, 200), + new Cost(new BigDecimal("0.5"), Currency.getInstance("USD")), + Map.of("tenant_id", "tenant-1") + )); + + assertThat(meterRegistry.find("ai.token.usage.total").meter()).isNull(); + assertThat(meterRegistry.find("ai.token.usage.distribution").meter()).isNull(); + assertThat(meterRegistry.find("ai.token.cost.total").meter()).isNull(); + } + + @Test + @DisplayName("1-arg MetricsOptions 생성자는 기존 직접 생성의 legacy opt-in을 유지해야 한다") + void shouldPreserveOneArgumentOptionsCompatibility() { + MetricsOptions options = new MetricsOptions(Set.of("tenant_id")); + + assertThat(options.allowedTagKeys()).containsExactly("tenant_id"); + assertThat(options.legacyAiTokenMetricsEnabled()).isTrue(); + } + + @Test + @DisplayName("기존 direct 기본 생성자는 tenant_id 허용 동작을 유지해야 한다") + void shouldPreserveDirectPublisherDefaultTagCompatibility() { + publisher.onRecord(new CostRecordedEvent( + "gpt-4o", + TokenUsage.from(1, 1), + Cost.of(new BigDecimal("0.1"), Currency.getInstance("USD")), + Map.of("tenant_id", "tenant-legacy", "user_id", "user-sensitive") + )); + + var counter = meterRegistry.get("ai.token.cost.total") + .tag("tenant_id", "tenant-legacy") + .counter(); + assertThat(counter.getId().getTag("user_id")).isNull(); + } } diff --git a/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/NotificationMetricsPublisherTest.java b/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/NotificationMetricsPublisherTest.java new file mode 100644 index 0000000..c48b398 --- /dev/null +++ b/token-pilot-micrometer/src/test/java/io/tokenpilot/micrometer/internal/NotificationMetricsPublisherTest.java @@ -0,0 +1,68 @@ +package io.tokenpilot.micrometer.internal; + +import io.micrometer.core.instrument.Meter; +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; +import io.tokenpilot.budget.BudgetThreshold; +import io.tokenpilot.notification.BudgetNotificationLifecycleEvent; +import io.tokenpilot.notification.BudgetNotificationOutcome; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +class NotificationMetricsPublisherTest { + + private SimpleMeterRegistry meterRegistry; + private NotificationMetricsPublisher publisher; + + @BeforeEach + void setUp() { + meterRegistry = new SimpleMeterRegistry(); + publisher = new NotificationMetricsPublisher(meterRegistry); + } + + @Test + void publishesBoundedNotificationLifecycleOutcome() { + publisher.onNotificationLifecycle(new BudgetNotificationLifecycleEvent( + BudgetNotificationOutcome.FAILURE, + BudgetThreshold.WARNING + )); + + var counter = meterRegistry.find("tokenpilot.notification.events") + .tag("outcome", "failure") + .tag("threshold", "warning") + .counter(); + + assertThat(counter).isNotNull(); + assertThat(counter.count()).isEqualTo(1.0); + assertThat(counter.getId().getType()).isEqualTo(Meter.Type.COUNTER); + assertThat(counter.getId().getBaseUnit()).isEqualTo("events"); + assertThat(counter.getId().getDescription()) + .isEqualTo("Total budget notification lifecycle events observed by Token Pilot"); + assertThat(counter.getId().getTags()).extracting(tag -> tag.getKey()) + .containsExactly("outcome", "threshold"); + } + + @Test + void distinguishesSuccessFailureAndDeduplicatedOutcomes() { + publisher.onNotificationLifecycle(new BudgetNotificationLifecycleEvent( + BudgetNotificationOutcome.SUCCESS, + BudgetThreshold.HALF + )); + publisher.onNotificationLifecycle(new BudgetNotificationLifecycleEvent( + BudgetNotificationOutcome.FAILURE, + BudgetThreshold.HALF + )); + publisher.onNotificationLifecycle(new BudgetNotificationLifecycleEvent( + BudgetNotificationOutcome.DEDUPLICATED, + BudgetThreshold.HALF + )); + + assertThat(meterRegistry.find("tokenpilot.notification.events") + .tag("outcome", "success").counter().count()).isEqualTo(1.0); + assertThat(meterRegistry.find("tokenpilot.notification.events") + .tag("outcome", "failure").counter().count()).isEqualTo(1.0); + assertThat(meterRegistry.find("tokenpilot.notification.events") + .tag("outcome", "deduplicated").counter().count()).isEqualTo(1.0); + } +} diff --git a/token-pilot-notification/src/main/java/io/tokenpilot/notification/BudgetNotificationLifecycleEvent.java b/token-pilot-notification/src/main/java/io/tokenpilot/notification/BudgetNotificationLifecycleEvent.java new file mode 100644 index 0000000..3ebabea --- /dev/null +++ b/token-pilot-notification/src/main/java/io/tokenpilot/notification/BudgetNotificationLifecycleEvent.java @@ -0,0 +1,25 @@ +package io.tokenpilot.notification; + +import io.tokenpilot.budget.BudgetThreshold; + +import java.util.Objects; + +/** + * notification handler 처리 또는 dedup 결과를 전달하는 framework-independent 이벤트입니다. + * + * @param outcome success, failure 또는 process-local dedup 결과 + * @param threshold 처리 대상인 bounded budget threshold + */ +public record BudgetNotificationLifecycleEvent( + BudgetNotificationOutcome outcome, + BudgetThreshold threshold +) { + + public BudgetNotificationLifecycleEvent { + Objects.requireNonNull(outcome, "outcome must not be null"); + Objects.requireNonNull(threshold, "threshold must not be null"); + if (threshold == BudgetThreshold.NONE) { + throw new IllegalArgumentException("threshold must identify a notification boundary"); + } + } +} diff --git a/token-pilot-notification/src/main/java/io/tokenpilot/notification/BudgetNotificationLifecycleListener.java b/token-pilot-notification/src/main/java/io/tokenpilot/notification/BudgetNotificationLifecycleListener.java new file mode 100644 index 0000000..d1d6e0b --- /dev/null +++ b/token-pilot-notification/src/main/java/io/tokenpilot/notification/BudgetNotificationLifecycleListener.java @@ -0,0 +1,8 @@ +package io.tokenpilot.notification; + +/** Budget notification lifecycle을 best-effort로 관찰하는 framework-independent 계약입니다. */ +@FunctionalInterface +public interface BudgetNotificationLifecycleListener { + + void onNotificationLifecycle(BudgetNotificationLifecycleEvent event); +} diff --git a/token-pilot-notification/src/main/java/io/tokenpilot/notification/BudgetNotificationOutcome.java b/token-pilot-notification/src/main/java/io/tokenpilot/notification/BudgetNotificationOutcome.java new file mode 100644 index 0000000..0056630 --- /dev/null +++ b/token-pilot-notification/src/main/java/io/tokenpilot/notification/BudgetNotificationOutcome.java @@ -0,0 +1,8 @@ +package io.tokenpilot.notification; + +/** Budget notification 처리의 bounded 운영 결과입니다. */ +public enum BudgetNotificationOutcome { + SUCCESS, + FAILURE, + DEDUPLICATED +} diff --git a/token-pilot-notification/src/main/java/io/tokenpilot/notification/BudgetNotificationService.java b/token-pilot-notification/src/main/java/io/tokenpilot/notification/BudgetNotificationService.java index 43c3ffd..876c9ea 100644 --- a/token-pilot-notification/src/main/java/io/tokenpilot/notification/BudgetNotificationService.java +++ b/token-pilot-notification/src/main/java/io/tokenpilot/notification/BudgetNotificationService.java @@ -3,7 +3,9 @@ import io.tokenpilot.budget.BudgetDecision; import io.tokenpilot.budget.BudgetThreshold; +import java.util.List; import java.util.Map; +import java.util.Objects; /** * 예산 판단 결과를 기반으로 알림 이벤트를 발행하는 서비스 @@ -12,13 +14,25 @@ public class BudgetNotificationService { private final BudgetNotificationHandler handler; private final NotificationStateStore store; + private final List lifecycleListeners; public BudgetNotificationService( BudgetNotificationHandler handler, NotificationStateStore store ) { - this.handler = handler; - this.store = store; + this(handler, store, List.of()); + } + + public BudgetNotificationService( + BudgetNotificationHandler handler, + NotificationStateStore store, + List lifecycleListeners + ) { + this.handler = Objects.requireNonNull(handler, "handler must not be null"); + this.store = Objects.requireNonNull(store, "store must not be null"); + this.lifecycleListeners = List.copyOf( + Objects.requireNonNull(lifecycleListeners, "lifecycleListeners must not be null") + ); } /** @@ -38,6 +52,10 @@ public void notifyIfNeeded( // 같은 window에서 중복 방지 if (current.compareTo(last) <= 0) { + publishLifecycleBestEffort( + BudgetNotificationOutcome.DEDUPLICATED, + current + ); return; } @@ -52,11 +70,43 @@ public void notifyIfNeeded( tags ); - handler.handle(event); + try { + handler.handle(event); + } catch (RuntimeException failure) { + publishLifecycleBestEffort( + BudgetNotificationOutcome.FAILURE, + current + ); + throw failure; + } store.updateLastNotifiedThreshold( decision.key(), current ); + publishLifecycleBestEffort( + BudgetNotificationOutcome.SUCCESS, + current + ); + } + + private void publishLifecycleBestEffort( + BudgetNotificationOutcome outcome, + BudgetThreshold threshold + ) { + if (lifecycleListeners.isEmpty()) { + return; + } + BudgetNotificationLifecycleEvent event = new BudgetNotificationLifecycleEvent( + outcome, + threshold + ); + for (BudgetNotificationLifecycleListener listener : lifecycleListeners) { + try { + listener.onNotificationLifecycle(event); + } catch (RuntimeException ignored) { + // Optional observers do not alter notification delivery or failure propagation. + } + } } } diff --git a/token-pilot-notification/src/test/java/io/tokenpilot/notification/BudgetNotificationLifecycleEventTest.java b/token-pilot-notification/src/test/java/io/tokenpilot/notification/BudgetNotificationLifecycleEventTest.java new file mode 100644 index 0000000..e0dea8f --- /dev/null +++ b/token-pilot-notification/src/test/java/io/tokenpilot/notification/BudgetNotificationLifecycleEventTest.java @@ -0,0 +1,31 @@ +package io.tokenpilot.notification; + +import io.tokenpilot.budget.BudgetThreshold; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +class BudgetNotificationLifecycleEventTest { + + @Test + void bounded_outcome과_threshold를_보존한다() { + BudgetNotificationLifecycleEvent event = new BudgetNotificationLifecycleEvent( + BudgetNotificationOutcome.SUCCESS, + BudgetThreshold.WARNING + ); + + assertThat(event.outcome()).isEqualTo(BudgetNotificationOutcome.SUCCESS); + assertThat(event.threshold()).isEqualTo(BudgetThreshold.WARNING); + } + + @Test + void NONE_threshold를_거부한다() { + assertThatThrownBy(() -> new BudgetNotificationLifecycleEvent( + BudgetNotificationOutcome.DEDUPLICATED, + BudgetThreshold.NONE + )) + .isInstanceOf(IllegalArgumentException.class) + .hasMessage("threshold must identify a notification boundary"); + } +} diff --git a/token-pilot-notification/src/test/java/io/tokenpilot/notification/BudgetNotificationServiceTest.java b/token-pilot-notification/src/test/java/io/tokenpilot/notification/BudgetNotificationServiceTest.java index 6bc580e..ef5b6cc 100644 --- a/token-pilot-notification/src/test/java/io/tokenpilot/notification/BudgetNotificationServiceTest.java +++ b/token-pilot-notification/src/test/java/io/tokenpilot/notification/BudgetNotificationServiceTest.java @@ -12,11 +12,16 @@ import java.math.BigDecimal; import java.util.Currency; +import java.util.List; import java.util.Map; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.atomic.AtomicInteger; import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; import static org.mockito.Mockito.any; import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; import static org.mockito.Mockito.same; import static org.mockito.Mockito.times; import static org.mockito.Mockito.verify; @@ -76,6 +81,114 @@ class BudgetNotificationServiceTest { .isEqualTo(event.getValue().projectedUsage()); } + @Test + void handler_성공과_중복을_lifecycle_event로_발행한다() { + BudgetNotificationHandler handler = mock(BudgetNotificationHandler.class); + List lifecycleEvents = new CopyOnWriteArrayList<>(); + BudgetNotificationService service = new BudgetNotificationService( + handler, + new InMemoryNotificationStateStore(), + List.of(lifecycleEvents::add) + ); + BudgetDecision half = decision(key("2026-08"), BudgetThreshold.HALF, "50"); + + service.notifyIfNeeded(half, Map.of()); + service.notifyIfNeeded(half, Map.of()); + + assertThat(lifecycleEvents).containsExactly( + new BudgetNotificationLifecycleEvent( + BudgetNotificationOutcome.SUCCESS, + BudgetThreshold.HALF + ), + new BudgetNotificationLifecycleEvent( + BudgetNotificationOutcome.DEDUPLICATED, + BudgetThreshold.HALF + ) + ); + verify(handler).handle(any()); + } + + @Test + void handler_실패를_발행하고_기존처럼_호출자에게_전파한다() { + RuntimeException failure = new IllegalStateException("handler failed"); + BudgetNotificationHandler handler = event -> { + throw failure; + }; + NotificationStateStore store = mock(NotificationStateStore.class); + List lifecycleEvents = new CopyOnWriteArrayList<>(); + BudgetNotificationService service = new BudgetNotificationService( + handler, + store, + List.of(lifecycleEvents::add) + ); + BudgetKey key = key("2026-08"); + when(store.getLastNotifiedThreshold(key)).thenReturn(BudgetThreshold.NONE); + + assertThatThrownBy(() -> service.notifyIfNeeded( + decision(key, BudgetThreshold.WARNING, "80"), + Map.of() + )).isSameAs(failure); + + assertThat(lifecycleEvents).containsExactly( + new BudgetNotificationLifecycleEvent( + BudgetNotificationOutcome.FAILURE, + BudgetThreshold.WARNING + ) + ); + verify(store, never()).updateLastNotifiedThreshold(any(), any()); + } + + @Test + void lifecycle_listener_실패는_handler와_다음_listener를_바꾸지_않는다() { + BudgetNotificationHandler handler = mock(BudgetNotificationHandler.class); + AtomicInteger failedDeliveries = new AtomicInteger(); + List received = new CopyOnWriteArrayList<>(); + BudgetNotificationService service = new BudgetNotificationService( + handler, + new InMemoryNotificationStateStore(), + List.of( + event -> { + failedDeliveries.incrementAndGet(); + throw new IllegalStateException("listener failed"); + }, + received::add + ) + ); + + service.notifyIfNeeded( + decision(key("2026-08"), BudgetThreshold.HALF, "50"), + Map.of() + ); + + assertThat(failedDeliveries).hasValue(1); + assertThat(received).containsExactly( + new BudgetNotificationLifecycleEvent( + BudgetNotificationOutcome.SUCCESS, + BudgetThreshold.HALF + ) + ); + verify(handler).handle(any()); + } + + @Test + void NONE은_lifecycle_event를_발행하지_않는다() { + List lifecycleEvents = new CopyOnWriteArrayList<>(); + BudgetNotificationHandler handler = mock(BudgetNotificationHandler.class); + BudgetNotificationService service = new BudgetNotificationService( + handler, + new InMemoryNotificationStateStore(), + List.of(lifecycleEvents::add) + ); + + service.notifyIfNeeded( + decision(key("2026-08"), BudgetThreshold.NONE, "0"), + Map.of() + ); + + assertThat(lifecycleEvents).isEmpty(); + verify(handler, never()).handle(any()); + } + private static BudgetKey key(String window) { return new BudgetKey("policy-a", "tenant", "tenant-a", BudgetWindow.parse(window)); } diff --git a/token-pilot-sample-app/src/main/java/io/tokenpilot/sample/SampleController.java b/token-pilot-sample-app/src/main/java/io/tokenpilot/sample/SampleController.java index 1065052..8f9f89e 100644 --- a/token-pilot-sample-app/src/main/java/io/tokenpilot/sample/SampleController.java +++ b/token-pilot-sample-app/src/main/java/io/tokenpilot/sample/SampleController.java @@ -53,6 +53,9 @@ public Map tokenLedgerBeans() { "ledgerManager", applicationContext.containsBean("ledgerManager"), "ledgerAdvisor", applicationContext.containsBean("ledgerAdvisor"), "pricingRegistry", applicationContext.containsBean("pricingRegistry"), + "tokenPilotCoreMetricsPublisher", applicationContext.containsBean("tokenPilotCoreMetricsPublisher"), + "tokenPilotBudgetMetricsPublisher", applicationContext.containsBean("tokenPilotBudgetMetricsPublisher"), + "tokenPilotNotificationMetricsPublisher", applicationContext.containsBean("tokenPilotNotificationMetricsPublisher"), "microCostMetricsPublisher", applicationContext.containsBean("microCostMetricsPublisher"), "budgetEvaluator", applicationContext.containsBean("budgetEvaluator"), "budgetStateStore", applicationContext.containsBean("budgetStateStore") diff --git a/token-pilot-sample-app/src/main/resources/application.yml b/token-pilot-sample-app/src/main/resources/application.yml index ae3ce58..53589fd 100644 --- a/token-pilot-sample-app/src/main/resources/application.yml +++ b/token-pilot-sample-app/src/main/resources/application.yml @@ -21,5 +21,5 @@ token-pilot: COMPLETION: 0.00060 metrics: enabled: true - tag-whitelist: - - tenant_id + legacy-ai-token-metrics-enabled: false + tag-whitelist: [] diff --git a/token-pilot-sample-app/src/test/java/io/tokenpilot/sample/SampleApplicationChatClientE2ETest.java b/token-pilot-sample-app/src/test/java/io/tokenpilot/sample/SampleApplicationChatClientE2ETest.java index 2298e39..1dd5c9e 100644 --- a/token-pilot-sample-app/src/test/java/io/tokenpilot/sample/SampleApplicationChatClientE2ETest.java +++ b/token-pilot-sample-app/src/test/java/io/tokenpilot/sample/SampleApplicationChatClientE2ETest.java @@ -7,10 +7,14 @@ import io.tokenpilot.core.domain.PricingReconciliationResult; import io.tokenpilot.core.domain.PricingResolution; import io.tokenpilot.core.domain.PricingSnapshot; +import io.tokenpilot.core.exception.MissingPricingException; +import io.tokenpilot.springai.LedgerAdvisor; import org.junit.jupiter.api.Test; import org.springframework.ai.chat.client.ChatClient; import org.springframework.ai.chat.client.ChatClientBuilderCustomizer; +import org.springframework.ai.chat.client.ChatClientRequest; import org.springframework.ai.chat.client.ChatClientResponse; +import org.springframework.ai.chat.client.advisor.api.AdvisorChain; import org.springframework.ai.chat.messages.AssistantMessage; import org.springframework.ai.chat.metadata.ChatResponseMetadata; import org.springframework.ai.chat.metadata.DefaultUsage; @@ -38,6 +42,8 @@ import java.util.Map; import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.Mockito.mock; @SpringBootTest( webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT, @@ -48,6 +54,7 @@ "token-pilot.pricing.plans[0].rates.PROMPT=0.00015", "token-pilot.pricing.plans[0].rates.COMPLETION=0.00060", "token-pilot.metrics.enabled=true", + "token-pilot.metrics.legacy-ai-token-metrics-enabled=true", "token-pilot.metrics.tag-whitelist[0]=tenant_id", "token-pilot.budget.enabled=true", "token-pilot.budget.monthly-limit=10.00", @@ -69,12 +76,18 @@ class SampleApplicationChatClientE2ETest { @Autowired private BudgetStateStore budgetStateStore; + @Autowired + private LedgerAdvisor ledgerAdvisor; + @Test void chatClientAdvisorRecordsTokenPilotMetricsEndToEnd() throws Exception { HttpResponse beans = get("/test/token-pilot/beans"); assertThat(beans.statusCode()).isEqualTo(200); assertThat(beans.body()) .contains("\"ledgerAdvisor\":true") + .contains("\"tokenPilotCoreMetricsPublisher\":true") + .contains("\"tokenPilotBudgetMetricsPublisher\":true") + .contains("\"tokenPilotNotificationMetricsPublisher\":true") .contains("\"microCostMetricsPublisher\":true"); HttpResponse chat = get("/test/token-pilot/chat"); @@ -124,6 +137,29 @@ void budgetAdvisorResolvesModelAndPolicyFromRegularChatClientCall() { assertThat(accumulatedCost.currency()).isEqualTo(Currency.getInstance("USD")); } + @Test + void missingPricingPublishesTokenPilotMetricBeforeProviderInvocation() throws Exception { + ChatClientRequest request = new ChatClientRequest( + new Prompt( + "missing pricing metric", + ChatOptions.builder().model("missing-model").build() + ), + Map.of("tenant_id", "metric-tenant") + ); + + assertThatThrownBy(() -> ledgerAdvisor.before( + request, + mock(AdvisorChain.class) + )).isInstanceOf(MissingPricingException.class); + + HttpResponse prometheus = get("/actuator/prometheus"); + assertThat(prometheus.statusCode()).isEqualTo(200); + assertThat(prometheus.body()) + .contains("tokenpilot_pricing_missing_events_total") + .contains("policy=\"fail_closed\"") + .doesNotContain("missing-model"); + } + private T contextValue(ChatClientResponse response, Class type) { return response.context().values().stream() .filter(type::isInstance) diff --git a/token-pilot-sample-app/src/test/java/io/tokenpilot/sample/SampleApplicationE2ETest.java b/token-pilot-sample-app/src/test/java/io/tokenpilot/sample/SampleApplicationE2ETest.java index 6c73807..5d95472 100644 --- a/token-pilot-sample-app/src/test/java/io/tokenpilot/sample/SampleApplicationE2ETest.java +++ b/token-pilot-sample-app/src/test/java/io/tokenpilot/sample/SampleApplicationE2ETest.java @@ -1,6 +1,9 @@ package io.tokenpilot.sample; +import io.tokenpilot.core.TokenBudget; +import io.tokenpilot.core.TokenEstimator; import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.context.SpringBootTest; import org.springframework.boot.test.web.server.LocalServerPort; @@ -21,7 +24,6 @@ "token-pilot.pricing.plans[0].rates.PROMPT=0.00015", "token-pilot.pricing.plans[0].rates.COMPLETION=0.00060", "token-pilot.metrics.enabled=true", - "token-pilot.metrics.tag-whitelist[0]=tenant_id", "management.endpoints.web.exposure.include=prometheus,health" } ) @@ -32,6 +34,12 @@ class SampleApplicationE2ETest { @LocalServerPort private int port; + @Autowired + private TokenEstimator tokenEstimator; + + @Autowired + private TokenBudget tokenBudget; + @Test void starterEndpointsAndPrometheusMetricsWorkEndToEnd() throws Exception { HttpResponse smoke = get("/test/token-pilot/smoke"); @@ -46,7 +54,10 @@ void starterEndpointsAndPrometheusMetricsWorkEndToEnd() throws Exception { .contains("\"ledgerManager\":true") .contains("\"ledgerAdvisor\":true") .contains("\"pricingRegistry\":true") - .contains("\"microCostMetricsPublisher\":true"); + .contains("\"tokenPilotCoreMetricsPublisher\":true") + .contains("\"tokenPilotBudgetMetricsPublisher\":true") + .contains("\"tokenPilotNotificationMetricsPublisher\":true") + .contains("\"microCostMetricsPublisher\":false"); HttpResponse record = get("/test/token-pilot/record"); assertThat(record.statusCode()).isEqualTo(200); @@ -55,13 +66,22 @@ void starterEndpointsAndPrometheusMetricsWorkEndToEnd() throws Exception { .contains("\"cost\":\"0.001350\"") .contains("\"currency\":\"USD\""); + tokenBudget.check( + "gpt-4o-mini", + tokenEstimator.estimate("sample preflight"), + 256 + ); + HttpResponse prometheus = get("/actuator/prometheus"); assertThat(prometheus.statusCode()).isEqualTo(200); assertThat(prometheus.body()) - .contains("ai_token_usage_total") - .contains("ai_token_usage_distribution") - .contains("ai_token_cost_total") - .contains("tenant_id=\"sample-tenant\"") + .contains("tokenpilot_preflight_requests_total") + .contains("decision=\"indeterminate\"") + .contains("reason=\"incomplete_scope\"") + .doesNotContain("ai_token_usage_total") + .doesNotContain("ai_token_usage_distribution") + .doesNotContain("ai_token_cost_total") + .doesNotContain("tenant_id=\"sample-tenant\"") .doesNotContain("user_id=\"sample-user\""); } diff --git a/token-pilot-spring-ai/src/main/java/io/tokenpilot/springai/internal/DefaultLedgerAdvisor.java b/token-pilot-spring-ai/src/main/java/io/tokenpilot/springai/internal/DefaultLedgerAdvisor.java index 441d574..707892d 100644 --- a/token-pilot-spring-ai/src/main/java/io/tokenpilot/springai/internal/DefaultLedgerAdvisor.java +++ b/token-pilot-spring-ai/src/main/java/io/tokenpilot/springai/internal/DefaultLedgerAdvisor.java @@ -16,6 +16,7 @@ import org.springframework.ai.chat.model.ChatResponse; import java.util.HashMap; +import java.util.List; import java.util.Map; import java.util.Objects; import java.util.Optional; @@ -49,6 +50,7 @@ public class DefaultLedgerAdvisor implements LedgerAdvisor { private final PricingRegistry pricingRegistry; private final PricingEvaluator pricingEvaluator; private final MissingPricingPolicy missingPricingPolicy; + private final List pricingMissingListeners; public DefaultLedgerAdvisor(LedgerManager ledgerManager, UsageExtractor usageExtractor) { this(ledgerManager, usageExtractor, null, null, null, null); @@ -89,6 +91,25 @@ public DefaultLedgerAdvisor(LedgerManager ledgerManager, UsageExtractor usageExt CostCalculator costCalculator, PricingRegistry pricingRegistry, PricingEvaluator pricingEvaluator, MissingPricingPolicy missingPricingPolicy) { + this( + ledgerManager, + usageExtractor, + budgetEvaluator, + budgetStateStore, + costCalculator, + pricingRegistry, + pricingEvaluator, + missingPricingPolicy, + List.of() + ); + } + + public DefaultLedgerAdvisor(LedgerManager ledgerManager, UsageExtractor usageExtractor, + BudgetEvaluator budgetEvaluator, BudgetStateStore budgetStateStore, + CostCalculator costCalculator, PricingRegistry pricingRegistry, + PricingEvaluator pricingEvaluator, + MissingPricingPolicy missingPricingPolicy, + List pricingMissingListeners) { this.ledgerManager = ledgerManager; this.usageExtractor = usageExtractor; this.budgetEvaluator = budgetEvaluator; @@ -103,6 +124,12 @@ public DefaultLedgerAdvisor(LedgerManager ledgerManager, UsageExtractor usageExt missingPricingPolicy, "missingPricingPolicy must not be null" ); + this.pricingMissingListeners = List.copyOf( + Objects.requireNonNull( + pricingMissingListeners, + "pricingMissingListeners must not be null" + ) + ); } @Override @@ -151,6 +178,7 @@ public ChatClientResponse after(ChatClientResponse response, AdvisorChain chain) try { cost = ledgerManager.record(resolvedSnapshot, usage, tags); } catch (MissingPricingException exception) { + publishPricingMissingBestEffort(exception.getResolution()); return handleActualPricingFailure(response, exception); } ChatClientResponse reconciledResponse = withReconciliationResult( @@ -167,7 +195,12 @@ public ChatClientResponse after(ChatClientResponse response, AdvisorChain chain) } return reconciledResponse; } else { - ledgerManager.record(modelId, usage, tags); + try { + ledgerManager.record(modelId, usage, tags); + } catch (MissingPricingException exception) { + publishPricingMissingBestEffort(exception.getResolution()); + throw exception; + } recordLegacyBudgetCost(modelId, usage, response); } @@ -223,6 +256,9 @@ private ChatClientRequest resolvePricing(ChatClientRequest request) { ? Optional.empty() : pricingRegistry.resolveSnapshot(modelId, pricingPolicyId); PricingResolution resolution = pricingEvaluator.validateSnapshotRates(snapshot); + if (!resolution.isResolved()) { + publishPricingMissingBestEffort(resolution); + } rejectMissingPricingIfFailClosed(resolution); return withPricingContext(request, pricingPolicyId, resolution, snapshot); @@ -253,6 +289,23 @@ private void rejectMissingPricingIfFailClosed(PricingResolution resolution) { throw new MissingPricingException(resolution); } + private void publishPricingMissingBestEffort(PricingResolution resolution) { + if (!resolution.isMissing() || pricingMissingListeners.isEmpty()) { + return; + } + PricingMissingEvent event = new PricingMissingEvent( + missingPricingPolicy, + resolution + ); + for (PricingMissingListener listener : pricingMissingListeners) { + try { + listener.onPricingMissing(event); + } catch (RuntimeException ignored) { + // Optional observers do not change pricing or provider-call semantics. + } + } + } + private String extractModelId(ChatClientRequest request) { Object contextValue = request.context().get(MODEL_ID_CONTEXT); if (contextValue instanceof String modelId && !modelId.isBlank()) { diff --git a/token-pilot-spring-ai/src/main/java/io/tokenpilot/springai/internal/LedgerSpringAiComponents.java b/token-pilot-spring-ai/src/main/java/io/tokenpilot/springai/internal/LedgerSpringAiComponents.java index 2ee86cd..a946df4 100644 --- a/token-pilot-spring-ai/src/main/java/io/tokenpilot/springai/internal/LedgerSpringAiComponents.java +++ b/token-pilot-spring-ai/src/main/java/io/tokenpilot/springai/internal/LedgerSpringAiComponents.java @@ -5,12 +5,15 @@ import io.tokenpilot.core.CostCalculator; import io.tokenpilot.core.LedgerManager; import io.tokenpilot.core.PricingEvaluator; +import io.tokenpilot.core.PricingMissingListener; import io.tokenpilot.core.PricingRegistry; import io.tokenpilot.core.domain.MissingPricingPolicy; import io.tokenpilot.core.internal.LedgerComponents; import io.tokenpilot.springai.LedgerAdvisor; import io.tokenpilot.springai.UsageExtractor; +import java.util.List; + /** * Spring AI 어댑터 컴포넌트 생성을 위한 팩토리 클래스입니다. */ @@ -115,6 +118,30 @@ public static LedgerAdvisor defaultLedgerAdvisor( PricingRegistry pricingRegistry, PricingEvaluator pricingEvaluator, MissingPricingPolicy missingPricingPolicy + ) { + return defaultLedgerAdvisor( + ledgerManager, + usageExtractor, + budgetEvaluator, + budgetStateStore, + costCalculator, + pricingRegistry, + pricingEvaluator, + missingPricingPolicy, + List.of() + ); + } + + public static LedgerAdvisor defaultLedgerAdvisor( + LedgerManager ledgerManager, + UsageExtractor usageExtractor, + BudgetEvaluator budgetEvaluator, + BudgetStateStore budgetStateStore, + CostCalculator costCalculator, + PricingRegistry pricingRegistry, + PricingEvaluator pricingEvaluator, + MissingPricingPolicy missingPricingPolicy, + List pricingMissingListeners ) { return new DefaultLedgerAdvisor( ledgerManager, @@ -124,7 +151,8 @@ public static LedgerAdvisor defaultLedgerAdvisor( costCalculator, pricingRegistry, pricingEvaluator, - missingPricingPolicy + missingPricingPolicy, + pricingMissingListeners ); } } diff --git a/token-pilot-spring-ai/src/test/java/io/tokenpilot/springai/internal/DefaultLedgerAdvisorTest.java b/token-pilot-spring-ai/src/test/java/io/tokenpilot/springai/internal/DefaultLedgerAdvisorTest.java index 1aa03d7..7a9fe90 100644 --- a/token-pilot-spring-ai/src/test/java/io/tokenpilot/springai/internal/DefaultLedgerAdvisorTest.java +++ b/token-pilot-spring-ai/src/test/java/io/tokenpilot/springai/internal/DefaultLedgerAdvisorTest.java @@ -13,6 +13,7 @@ import io.tokenpilot.core.domain.*; import io.tokenpilot.core.exception.MissingPricingException; import io.tokenpilot.core.internal.LedgerComponents; +import io.tokenpilot.springai.LedgerAdvisor; import io.tokenpilot.springai.UsageExtractor; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; @@ -663,6 +664,179 @@ void failOpenAllowsProviderCallAndMarksMissingPricingAsUnpriced() { verifyNoInteractions(ledgerManager); } + @Test + @DisplayName("pre-call missing pricing은 provider 전 한 번만 발행하고 after에서 중복하지 않는다") + void publishesPreCallPricingMissExactlyOnce() { + LedgerManager ledgerManager = mock(LedgerManager.class); + UsageExtractor extractor = mock(UsageExtractor.class); + PricingRegistry pricingRegistry = mock(PricingRegistry.class); + List events = new java.util.concurrent.CopyOnWriteArrayList<>(); + + when(extractor.extract(any())).thenReturn(TokenUsage.from(100, 200)); + when(pricingRegistry.resolveSnapshot("missing-model", PricingPlan.DEFAULT_PRICING_POLICY_ID)) + .thenReturn(Optional.empty()); + + LedgerAdvisor advisor = LedgerSpringAiComponents.defaultLedgerAdvisor( + ledgerManager, + extractor, + null, + null, + mock(CostCalculator.class), + pricingRegistry, + LedgerComponents.defaultPricingEvaluator(), + MissingPricingPolicy.FAIL_OPEN, + List.of(events::add) + ); + ChatClientRequest request = new ChatClientRequest( + new Prompt("test"), + Map.of(DefaultLedgerAdvisor.MODEL_ID_CONTEXT, "missing-model") + ); + + ChatClientRequest resolved = advisor.before(request, mock(AdvisorChain.class)); + advisor.after( + response("missing-model", resolved.context()), + mock(AdvisorChain.class) + ); + + assertThat(events).containsExactly(new PricingMissingEvent( + MissingPricingPolicy.FAIL_OPEN, + PricingResolution.MISSING_PLAN + )); + } + + @Test + @DisplayName("actual cost 계산의 MissingPricingException도 한 번 발행한다") + void publishesActualPricingMissExactlyOnce() { + LedgerManager ledgerManager = mock(LedgerManager.class); + UsageExtractor extractor = mock(UsageExtractor.class); + List events = new java.util.concurrent.CopyOnWriteArrayList<>(); + TokenUsage usage = TokenUsage.from(100, 200); + PricingSnapshot snapshot = PricingSnapshot.from( + new PricingPlan( + "gpt-4o", + new BigDecimal("0.01"), + new BigDecimal("0.03"), + Currency.getInstance("USD") + ), + PricingSnapshot.DEFAULT_CATALOG_VERSION, + Instant.parse("2026-07-30T00:00:00Z") + ); + + when(extractor.extract(any())).thenReturn(usage); + when(ledgerManager.record(same(snapshot), same(usage), anyMap())) + .thenThrow(new MissingPricingException(PricingResolution.MISSING_RATE)); + + DefaultLedgerAdvisor advisor = new DefaultLedgerAdvisor( + ledgerManager, + extractor, + null, + null, + mock(CostCalculator.class), + null, + LedgerComponents.defaultPricingEvaluator(), + MissingPricingPolicy.FAIL_OPEN, + List.of(events::add) + ); + + ChatClientResponse result = advisor.after( + response( + "gpt-4o", + Map.of( + DefaultLedgerAdvisor.PRICING_SNAPSHOT_CONTEXT, snapshot, + DefaultLedgerAdvisor.PRICING_RESOLUTION_CONTEXT, PricingResolution.RESOLVED + ) + ), + mock(AdvisorChain.class) + ); + + assertThat(result.context().get(DefaultLedgerAdvisor.PRICING_RECONCILIATION_RESULT_CONTEXT)) + .isEqualTo(PricingReconciliationResult.UNPRICED); + assertThat(events).containsExactly(new PricingMissingEvent( + MissingPricingPolicy.FAIL_OPEN, + PricingResolution.MISSING_RATE + )); + } + + @Test + @DisplayName("legacy actual pricing miss는 한 번 발행하고 기존 예외를 그대로 전파한다") + void publishesLegacyActualPricingMissWithoutChangingFailureSemantics() { + LedgerManager ledgerManager = mock(LedgerManager.class); + UsageExtractor extractor = mock(UsageExtractor.class); + List events = new java.util.concurrent.CopyOnWriteArrayList<>(); + TokenUsage usage = TokenUsage.from(100, 200); + MissingPricingException failure = new MissingPricingException(PricingResolution.MISSING_PLAN); + + when(extractor.extract(any())).thenReturn(usage); + when(ledgerManager.record(eq("missing-model"), same(usage), anyMap())) + .thenThrow(failure); + + DefaultLedgerAdvisor advisor = new DefaultLedgerAdvisor( + ledgerManager, + extractor, + null, + null, + null, + null, + LedgerComponents.defaultPricingEvaluator(), + MissingPricingPolicy.FAIL_OPEN, + List.of(events::add) + ); + + assertThatThrownBy(() -> advisor.after( + response("missing-model", Map.of()), + mock(AdvisorChain.class) + )).isSameAs(failure); + assertThat(events).containsExactly(new PricingMissingEvent( + MissingPricingPolicy.FAIL_OPEN, + PricingResolution.MISSING_PLAN + )); + } + + @Test + @DisplayName("pricing miss listener 실패는 fail-closed 판정과 다음 listener를 바꾸지 않는다") + void isolatesPricingMissListenerFailures() { + LedgerManager ledgerManager = mock(LedgerManager.class); + PricingRegistry pricingRegistry = mock(PricingRegistry.class); + AtomicInteger failedDeliveries = new AtomicInteger(); + List received = new java.util.concurrent.CopyOnWriteArrayList<>(); + + when(pricingRegistry.resolveSnapshot("missing-model", PricingPlan.DEFAULT_PRICING_POLICY_ID)) + .thenReturn(Optional.empty()); + + DefaultLedgerAdvisor advisor = new DefaultLedgerAdvisor( + ledgerManager, + mock(UsageExtractor.class), + null, + null, + mock(CostCalculator.class), + pricingRegistry, + LedgerComponents.defaultPricingEvaluator(), + MissingPricingPolicy.FAIL_CLOSED, + List.of( + event -> { + failedDeliveries.incrementAndGet(); + throw new IllegalStateException("listener failed"); + }, + received::add + ) + ); + ChatClientRequest request = new ChatClientRequest( + new Prompt("test"), + Map.of(DefaultLedgerAdvisor.MODEL_ID_CONTEXT, "missing-model") + ); + + assertThatThrownBy(() -> advisor.before(request, mock(AdvisorChain.class))) + .isInstanceOf(MissingPricingException.class) + .extracting(exception -> ((MissingPricingException) exception).getResolution()) + .isEqualTo(PricingResolution.MISSING_PLAN); + assertThat(failedDeliveries).hasValue(1); + assertThat(received).containsExactly(new PricingMissingEvent( + MissingPricingPolicy.FAIL_CLOSED, + PricingResolution.MISSING_PLAN + )); + verifyNoInteractions(ledgerManager); + } + @Test @DisplayName("FAIL_OPEN은 model id가 없어도 MISSING_PLAN을 보존하고 UNPRICED로 남겨야 한다") void failOpenPreservesMissingPlanWhenModelIdIsMissing() {