Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions token-pilot-sample-app/DEMO_RUNBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,3 +143,35 @@ demo와 smoke를 연속으로 확인하려면 `demo` 프로필에서 `/demo/run`

앱을 종료한 뒤 다시 `bootRun`하면 in-memory budget과 Micrometer counter가
초기화됩니다.


## 6. Observe Cloud API E2E 연동 및 장애 주입(Fault Injection) 검증

이 섹션은 샘플 앱과 중앙 수집 백엔드(token-ledger-cloud-api) 간의 비동기 사용량 수집 파이프라인과 비차단(Non-blocking) 장애 격리 기능을 검증하기 위한 가이드입니다.

## 6.1. 사전 준비 (백엔드 수집 서버 실행)

중앙 수집 API 서버를 별도 터미널에서 8081 포트로 실행합니다.

cd token-ledger-cloud-api
./gradlew bootRun

## 6.2. E2E 연동 및 정상 수집 검증

1. 샘플 앱을 demo 프로필로 실행합니다 (8080 포트).
2. 브라우저나 curl로 데모를 실행합니다.
curl -s http://localhost:8080/test/token-pilot/demo/run | jq
3. 확인 사항:
- 샘플 앱 터미널에 [Observe 전송 성공] Status: 201 로그가 출력되는지 확인합니다.
- 백엔드(8081) 터미널에 usage_events 테이블로 데이터가 정상 INSERT 되는지 확인합니다.

## 6.3. 장애 주입 테스트 (Non-Blocking 검증)

수집 서버가 다운되거나 응답하지 않아도 샘플 앱의 메인 로직이 차단되지 않는지 검증합니다.

1. 8081 포트에서 실행 중인 백엔드 서버를 강제로 종료합니다 (Ctrl + C).
2. 샘플 앱에서 다시 데모 시나리오를 호출합니다.
curl -s http://localhost:8080/test/token-pilot/demo/run | jq
3. 확인 사항:
- 수집 서버가 꺼져 있어도 웹 요청과 시나리오 결과는 에러 없이 정상 반환됩니다.
- 샘플 앱 터미널에 [Observe 통신 실패...] java.net.ConnectException 예외 로그가 찍히지만, 애플리케이션은 중단되지 않고 정상 유지됩니다.
32 changes: 32 additions & 0 deletions token-pilot-sample-app/DEMO_RUNBOOK_EN.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,3 +148,35 @@ the same time.
## 5. Reset the demo state

Restarting `bootRun` resets the in-memory budget store and Micrometer counters.


## 6. Observe Cloud API E2E Integration & Fault Injection Verification

This section covers verifying the asynchronous usage collection pipeline between the sample app and the central collection backend (token-ledger-cloud-api), including non-blocking fault isolation.

## 6.1. Prerequisites (Start Backend Collection Server)

Run the central collection API server in a separate terminal on port 8081:

cd token-ledger-cloud-api
./gradlew bootRun

## 6.2. E2E Integration & Successful Collection Verification

1. Start the sample app with the demo profile (port 8080).
2. Run the demo via browser or curl:
curl -s http://localhost:8080/test/token-pilot/demo/run | jq
3. Verification:
- Confirm that the sample app terminal logs "[Observe 전송 성공] Status: 201".
- Confirm that data is successfully inserted into the usage_events table in the backend (8081) terminal.

## 6.3. Fault Injection Test (Non-Blocking Verification)

Verify that the sample app's main logic is not blocked even if the collection server is down or unresponsive.

1. Forcefully stop the backend server running on port 8081 (Ctrl + C).
2. Call the demo scenarios from the sample app again:
curl -s http://localhost:8080/test/token-pilot/demo/run | jq
3. Verification:
- Web requests and scenario results return normally without errors even when the collection server is down.
- The sample app terminal prints connection failure exceptions (e.g., java.net.ConnectException), but the application continues to run without interruption.
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ public class DemoScenarioService {
private final TokenPilotProperties properties;
private final MeterRegistry meterRegistry;
private final AtomicInteger runSequence = new AtomicInteger();
private final ObserveLedgerListener observeLedgerListener;

@Autowired
public DemoScenarioService(
Expand All @@ -83,7 +84,8 @@ public DemoScenarioService(
TokenEstimator tokenEstimator,
TokenBudget tokenBudget,
TokenPilotProperties properties,
MeterRegistry meterRegistry
MeterRegistry meterRegistry,
ObserveLedgerListener observeLedgerListener
) {
this.chatClientBuilder = chatClientBuilder;
this.demoChatModel = demoChatModel;
Expand All @@ -94,6 +96,7 @@ public DemoScenarioService(
this.tokenBudget = tokenBudget;
this.properties = properties;
this.meterRegistry = meterRegistry;
this.observeLedgerListener = observeLedgerListener;
}

public synchronized DemoRunReport runAll() {
Expand Down Expand Up @@ -431,7 +434,14 @@ else if (mode == ReconciliationMode.USAGE_UNKNOWN) {
boolean pending = snapshot.pendingReconciliationLiability().value().signum() > 0;
boolean committed = snapshot.committedCost().value().signum() > 0;
boolean passed = switch (mode) {
case SUCCESS -> outcome.state().equals("COMMITTED") && committed && !pending;
case SUCCESS -> {
boolean p = outcome.state().equals("COMMITTED") && committed && !pending;
if (p) {
// 🎯 [핵심] 데모 정산이 성공했을 때 8081 백엔드 API로 비동기 로그 전송!
observeLedgerListener.sendUsageLogManually(MODEL_ID, 1200, 800, 0.0035);
}
yield p;
}
case PROVIDER_FAILURE -> outcome.state().equals("PROVIDER_FAILED") && pending && !committed;
case USAGE_UNKNOWN -> outcome.state().equals("COMMITTED") && pending && !committed;
};
Expand Down Expand Up @@ -665,4 +675,5 @@ private enum ReconciliationMode {
this.summary = summary;
}
}

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
package io.tokenpilot.sample;

import io.tokenpilot.core.LedgerListener;
import io.tokenpilot.core.domain.CostRecordedEvent;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;

@Component
public class ObserveLedgerListener implements LedgerListener {

private static final Logger log = LoggerFactory.getLogger(ObserveLedgerListener.class);
private final HttpClient httpClient;

public ObserveLedgerListener() {
this.httpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(3))
.build();
}

// 외부에서 직접 호출할 수 있는 수동 전송 메서드 추가
public void sendUsageLogManually(String modelId, long inputTokens, long outputTokens, double totalCost) {
String jsonPayload = String.format("""
{
"projectId": "demo-project-id",
"organizationId": "demo-org-id",
"environment": "demo",
"provider": "openai",
"model": "%s",
"promptTokens": %d,
"completionTokens": %d,
"cachedPromptTokens": 0,
"reasoningTokens": 0,
"totalTokens": %d,
"promptCostUsd": 0.0018,
"completionCostUsd": 0.0017,
"cachedPromptCostUsd": 0,
"reasoningCostUsd": 0,
"totalCostUsd": %.4f,
"requestId": "req-%d",
"sourceType": "DEMO",
"pricingVersion": "openai-2026-08-14",
"occurredAt": "2026-10-02T18:35:00"
}
""", modelId, inputTokens, outputTokens, (inputTokens + outputTokens), totalCost, System.currentTimeMillis());

HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create("http://localhost:8081/internal/usage-logs"))
.header("Content-Type", "application/json")
.timeout(Duration.ofSeconds(3))
.POST(HttpRequest.BodyPublishers.ofString(jsonPayload))
.build();

httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofString())
.whenComplete((response, throwable) -> {
if (throwable != null) {
log.error("[Observe 통신 실패 - 비차단 확인] {}", throwable.getMessage());
} else {
log.info("[Observe 전송 성공] Status: {}", response.statusCode());
}
});
}

@Override
public void onRecord(CostRecordedEvent event) {
// 기존 라이브러리 연동부
sendUsageLogManually("gpt-4o-mini", 1200, 800, 0.0035);
}
}
Loading