diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/README.md b/docs/test-results/wal-offsite-recovery-recheck-20261018/README.md new file mode 100644 index 00000000..231e6376 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/README.md @@ -0,0 +1,162 @@ +# WAL 완료 확인 후 DB·GCS 복구 재계측 + +관련 이슈: #563. 원인 재현·수정은 [#564](https://github.com/DocGrid/docgrid/pull/564), 그 후 같은 부하의 실제 복구 재시험은 이 문서에 기록한다. + +**결과: 65건의 성공 응답 중 60건 복원, 5건 DB 누락. 마지막 외부 보존 완료 구간 60건은 모두 복원됐고 구간 안 누락은 0건이다.** 마지막 구간 뒤 5건은 따로 손실로 기록했다. 전체 무손실이나 운영 RPO 0을 주장하지 않는다. + +## 목적 + +앞선 [계측](../dr-functional-recovery-20261018/README.md)은 성공 응답 65건 중 33건만 복원했다. 그때 외부에 마지막으로 전송한 원본은 60건이었으나, DB에 필요한 WAL까지 보관됐는지는 고정 2초와 archive 종료 코드로 추정했다. #564에서는 그 판정이 조기에 성공할 수 있는 경로를 실제 3사례 중 2사례에서 재현하고, 정확한 WAL 파일의 보관·외부 일치를 확인하도록 보완했다. + +이번 성공 기준은 **마지막 원격 WAL 확인 시점까지 성공한 요청의 ID·DB 관계·원본 해시와 generation이 모두 복원되는 것**이다. 이후 요청까지 덮기 위해 마지막 전송을 추가하지 않는다. 최초 감사가 끝나기 전에 Worker·Flyway·버킷 재연결을 실행하지 않는다. + +## 테스트 환경 + +| 구성요소 | 실행 위치·실제 범위 | +| --- | --- | +| 소스 | 실제 OpenSQL 17.8의 전용 논리 DB·소유 마커가 있는 두 역할, 공개 DB 설정 유지 | +| 애플리케이션 | 로컬 실제 Spring Boot JVM. 이전과 같은 Java 소스 `3bff887a5f6abdf4155957a658599f8f0366a141`·동일 JAR. SHA-256 `08c5c458110bc96fc4c92cc5b02bd160f6c08b59e21d9ef536f2fa50aa76b27d` | +| 수집·백업 | 로컬 Docker 실제 Barman 3.19.1·별도 카탈로그. 새 기준 백업 DONE·pg_verifybackup 통과 | +| 외부 원본·DB 입력 | 시험 전용 비공개 GCS 버킷 3개. 원본·기준 백업·identity·WAL을 generation 고정하고 실제 바이트 SHA-256·크기 대조 | +| 복원 | 네트워크 없는 Barman import/recover 후 기존 라이선스가 있는 비동기 replica 호스트의 **별도 PGDATA**에서 실제 OpenSQL 실행 | +| 호스트 격리 한계 | 별도 물리 클러스터를 새로 만든 시험은 아니다. 비동기 replica의 원래 Patroni·PostgreSQL만 잠시 중단하고 1,800초 자동 원복 장치를 둔다. Primary·동기 Standby는 유지하고 원래 PGDATA는 복원 대상으로 쓰지 않음 | +| 모델·캐시 | 실제 BGE-M3 요청을 전달하는 제어 프록시·시험 소유 Valkey. 중단 Job 준비 시 실제 모델 응답 하나만 보류. 모조 벡터를 반환하지 않음 | +| 제외 | Ollama 생성 답변·공개 트래픽 전환·OpenProxy 경로·전체 사이트 소실·클라우드 IAM에 의한 원본 접근 차단 | + +모든 원장 시각은 UTC(+00:00), 한국 시각은 9시간을 더한다. 사전 준비 시간과 장애 T0 이후 경과를 구분한다. 모델은 기존에 준비된 실행 환경을 재사용했으며 최초 모델 다운로드 시간은 측정하지 않았다. + +```text +그림 1 — 실제 입력 보존과 복원 경로 + +시험 클라이언트 → 격리 JVM → 소유 논리 DB / 시험 원본 GCS + │ 성공 응답 ID·해시를 DB 밖 원장에 저장 + ▼ +OpenSQL WAL → 실제 Barman 수신·보관 + │ 전환한 WAL 이름 확보 → 다음 구간 restore point 생성 + │ 정확한 완성 파일 도착까지 제한 시간 내 대기 + ▼ +시험 외부 GCS: 기준 백업·identity·WAL·원본 generation 고정 + │ 같은 파일을 실제 재읽기 → SHA-256·크기 일치 + ▼ +원본 취득 중단 → 네트워크 없는 복원 도구 + → 라이선스 있는 설치 호스트의 별도 PGDATA + → 최초 감사 → 통과 후에만 복원 JVM·Worker 활성화 +``` + +실제 Barman 명령과 실제 GCS REST를 사용했다. 공개 대기 함수 `wait_for_archived_wal()`를 사용했고 원격 단계는 기존 REST publisher에서 고정 generation을 재읽었다. 전체 `gsutil sync --required-wal` CLI의 클라우드 실행으로 표현하지 않는다. + +## 테스트 방법 + +1. Run ID `DR-RESTORE-20261018-FRESHNESS1`의 목적·부하·성공 기준을 시작 전에 저장했다. [formal-spec](evidence/DR-RESTORE-20261018-FRESHNESS1/formal-spec.json), [실행 바이너리 동일성](evidence/DR-RESTORE-20261018-FRESHNESS1/artifact-identity.json). +2. 같은 합성 문서 생성기로 PDF·DOCX 완료 4건, 실제 응답을 기다리던 Job 1건을 준비했다. Job을 처리하던 소유 JVM을 SIGKILL한 뒤 Worker=false로 다시 기동했다. 소스 준비 점검은 24/24, 새 기준 백업 검증까지 완료했다. 문서 ID·marker·원본 바이트는 실행별로 새로 생성하므로 이전 파일과 byte-identical이라고 주장하지 않는다. +3. 추가 업로드 60건을 목표 11초 간격으로 보내며 300초 주기의 전송을 두 번 수행했다. 수신·전송은 업로드와 직렬 실행해 경계가 명확하며, 전송 때문에 생기는 실제 간격 변화도 기록했다. +4. 매 전송은 전환한 **정확한 WAL 파일**이 Barman wals에 도착해야 진행한다. 다음 파일 진행을 위해 합성 restore point를 만들고, 외부 전송 후 해당 WAL의 고정 generation을 다시 읽어 로컬 SHA-256·크기와 대조했다. +5. 마지막 전송 뒤 요청을 계속 보냈다. 원본 API·수집기·터널을 중단한 T0는 `2026-10-18T14:36:26.952847+00:00`, 외부 입력 동결은 `2026-10-18T14:36:37.353+00:00`이다. 차이는 10.4초이며 두 시각을 같은 것으로 취급하지 않는다. +6. 동결 입력만으로 복원하고 원본에 재접속하지 않았다. Worker·Flyway·버킷 재연결 전 65건 전체를 감사했다. 마지막 확인 구간의 누락이 있으면 활성화를 거부하도록 했다. +7. 통과 뒤 복원본의 버킷 참조만 시험 보존본으로 바꾸고 Worker 동시성 2로 기동했다. 복원된 계정으로 새 로그인 후 기능 6종을 확인했다. 최종 정합성 감사 뒤 원래 3노드·공개 설정·소유 자원을 정리했다. + +```text +그림 2 — 성공 응답과 외부 보존의 시간 경계 [관측] + +기능 문서 5건 → 초기 전송 0: 5건 + → 업로드 계속 → 전송 1: 33건 + → 업로드 계속 → 전송 2: 60건 + └─ WAL·원본 외부 검증 완료 + → 마지막 5건 요청 → T0 → 입력 동결 + +최초 복원 감사: + 보존 확인 구간 60건 + ├─ 복원 60건 + └─ 누락 0건 + 그 뒤 5건 + ├─ 복원 0건 + └─ 누락 5건 + +마지막 확인 뒤의 요청을 덮기 위한 추가 전송은 하지 않음 +``` + +이전의 32건 누락과 이번 5건 누락은 모두 명시한다. 개선은 마지막 전송 시점까지의 WAL 누락 공백이 제거됐다는 것이며, 전송 주기 뒤의 요청까지 무손실로 만든 결과는 아니다. + +## 관측 결과 + +### 이전 회차와 재시험 비교 + +| 항목 | 이전 FORMAL1 | 이번 FRESHNESS1 | +| --- | --- | --- | +| 성공 응답 | 65건 | 65건 | +| 마지막 전송 원본 | 60건, 필요한 WAL 완료 확인 없음 | 60건, 정확한 WAL 외부 바이트 확인 | +| 최초 복원 | 33건 | 60건 | +| DB 누락 | 32건 | 5건 | +| 마지막 전송 구간까지의 누락 | 원본 60건 중 DB 누락 27건 | 0건 | +| 마지막 전송 뒤 요청 | 5건 누락 | 5건 누락 | +| 기능 검증 | 6/6 | 6/6 | +| 마지막 연속 복원 ACK~T0 관측 간격 | 412.061초 | 113.978초 | + +위 간격은 이 부하·전송 타이밍에서 측정한 값이다. 운영 RPO 보장값이나 무손실 선언이 아니다. DB 누락 5건 중 보존된 원본이 있는 사례는 0건이다. 이 5건은 DB와 원본 모두 마지막 외부 보존 범위 밖에 있었다. + +### 외부 전송과 실제 요청 간격 + +| 전송 | 보존 원본 수 | 보존 WAL/history 파일 수 | 소요 | 결과 | +| --- | --- | --- | --- | --- | +| 0 | 5 | 5 | 67.284초 | PASS | +| 1 | 33 | 6 | 66.737초 | PASS | +| 2 | 60 | 7 | 67.315초 | PASS | + +손실 관측 요청 60건의 실제 전송 간격 59개는 최소 7.068초·중앙 10.997초·최대 74.887초였다. 목표 11초와 다르며, 전송 중 대기·이후 따라잡기를 포함한다. 두 회차를 엄밀한 처리량 벤치마크로 비교하지 않는다. + +### 기능별 복구 경과 + +| 기능 | 표본 | 완료 시각 UTC | T0부터 | 결과 | +| --- | --- | --- | --- | --- | +| 기존 원본 다운로드·해시 | 4건 | 2026-10-18T14:39:09.852552+00:00 | 162.9초 | 통과 | +| 기존 문서 검색 | 1건 | 2026-10-18T14:39:25.400051+00:00 | 178.447초 | 통과 | +| 중단 Job 재개 | 1건 | 2026-10-18T14:40:21.317855+00:00 | 234.365초 | 통과 | +| 새 업로드 | 1건 | 2026-10-18T14:40:28.376043+00:00 | 241.423초 | 통과 | +| 새 문서 인덱싱 | 1건 | 2026-10-18T14:51:37.672099+00:00 | 910.719초 | 통과 | +| 새 원본 다운로드·해시 | 1건 | 2026-10-18T14:51:39.754136+00:00 | 912.801초 | 통과 | + +복원 DB 기동과 사용자 기능 재개는 다른 시각이다. 이 표는 격리 문서 다운로드·검색·인덱싱 경로의 관측 시간이며, 공개 서비스 전체 RTO나 Ollama 답변 복구 시간이 아니다. + +이번에는 이전 33건보다 많은 60건을 복원했고 Worker 동시성은 2로 유지했다. 새 문서는 복원된 대기 Job 뒤에서 처리됐다. 14:43:55 UTC의 읽기 전용 조회에서 전체 61개 Job 중 INDEXED 27개·처리 중 1개·대기 33개를 확인했다. 따라서 새 인덱싱 경과가 늘어도 같은 큐 크기의 성능 회귀로 단정할 수 없다. [대기열 관측](evidence/DR-RESTORE-20261018-FRESHNESS1/queue-1-result.json). + +```text +그림 3 — 최초 감사 뒤에만 활성화하는 내부 순서 [실행 원장] + +sequence 312: 복원 완료 + → sequence 313: 원장·ID·원본을 최초 감사, 파일로 확정 + [Worker=false / Flyway=false / repair=false / source_access=false] + → sequence 314: 보존 구간 누락 0건 확인 + ├─ 누락 있음: 중단·정리 + └─ 누락 없음 + → sequence 315: 복원본 버킷 참조 재연결 + → sequence 319: 복원 JVM·Worker 활성화 + → 다운로드·검색·같은 Job 재개·새 업로드·인덱싱 + → sequence 335: 최종 정합성 감사 + → sequence 336: 원래 클러스터·소유 자원 정리 +``` + +중단 Job은 하나를 유지하면서 Attempt 1→2, 최종 INDEXED를 확인했다. 최종 감사 대상은 61문서이며 미완료 Job 0건·중복 Chunk 0건·중복 Embedding 0건·누락 Embedding 0건이다. [최초 감사](evidence/DR-RESTORE-20261018-FRESHNESS1/initial-audit.json), [구간 판정](evidence/DR-RESTORE-20261018-FRESHNESS1/checkpoint-coverage.json), [최종 스냅샷](evidence/DR-RESTORE-20261018-FRESHNESS1/final-snapshot.json). + +### WAL 재생 끝 확인 + +복원 서버의 마지막 재생 LSN은 `0/31000000`, 마지막 동결 WAL은 `000000060000000000000030`, segment 크기는 `16MB`였다. 이 WAL의 끝과 재생 끝이 일치했다. 마지막 재생 커밋은 `2026-10-18T14:34:33.013865+00:00`다. [읽기 전용 관측](evidence/DR-RESTORE-20261018-FRESHNESS1/replay-boundary-result.json). 이 확인은 Worker 활성화 후 읽었지만 replay LSN을 조회한 것이며 새 쓰기 LSN과 혼동하지 않는다. + +## 검증 과정에서 확인된 사항 + +- 이번 본 복구 회차의 실행 판정은 **PASS**, 추가 차단 사유는 `없음`이다. 초기 준비·본 복구·마지막 공유 자원 정리를 구분해서 기록했다. 앞선 수신 역할 준비 중단은 #564의 별도 LIVE1 증거로 남겨 재현 성공에 섞지 않았다. +- 보완은 공개 PostgreSQL·Patroni·Barman 타이머 정책 변경이 아니다. 지정된 WAL까지 완료를 확인하는 선택적 계약과 격리 호출 경로를 검증했다. 주기 복사의 완료 시각만 보고 업무 최신성·RPO 0을 선언하면 안 된다. +- 원본 접근 차단은 복구 도구 network=none·원본 리스너 종료·클라이언트 원본 GCS 접근 가드다. 클라우드 IAM 거부나 원본 클러스터의 실제 파괴 시험은 아니다. 원본에서 추가 WAL·원본 파일을 보충하지 않았다. +- 전체 클러스터 물리 백업을 사용했지만 애플리케이션 검증·쓰기·정리는 소유 논리 DB와 시험 버킷에 한정했다. 복원 호스트는 기존 비동기 replica의 라이선스를 재사용한 환경이며 원래 PGDATA를 덮어쓰지 않았다. +- 동일 JAR·같은 문서 수·같은 생성기·목표 간격·전송 주기를 유지했다. 실제 네트워크·모델 대기·문서 marker와 ID·전송 소요는 같지 않아 지연 개선 수치를 일반화하지 않는다. +- 정리 확인: 원래 클러스터 복귀·공개 설정 해시 일치·시험 소유 프로세스/컨테이너/DB/역할/키 제거. 시험 버킷 3개 삭제, 의도한 사용자 원본 버킷 유지, 임시 ADC 복사본·수신 인증 설정 복사본 제거. 기존 OAuth 자격은 폐기하지 않았다. 개인 환경 파일 2개의 내용 일치를 확인했다. +- 활성 객체와 버킷을 삭제해도 GCS soft-delete 정책이 삭제된 세대를 보존할 수 있다. 즉시 물리 삭제를 주장하지 않는다. [회차 정리](evidence/DR-RESTORE-20261018-FRESHNESS1/runtime-cleanup-observations.json), [공유 자원 정리](evidence/task-cleanup-r1/result.json), [기존 개인 작업·모델 보존 확인](evidence/local-preservation-r1/result.json). + +## 증거와 변경 범위 + +| 구성요소 | 이번 역할 | +| --- | --- | +| `scripts/opensql/barman_offsite_sync.py` | #564에서 추가한 정확한 WAL 보관 대기 함수. 기본 공개 타이머 동작은 유지 | +| 기존 `dr_recovery.py` 계약 | 원장·동결 입력·최초 감사·활성화·기능 6종·최종 감사·항상 정리 순서 재사용 | +| 이번 결과 디렉터리 | 실제 회차의 별도 로그·전송 경계·스냅샷·집계·정리 증거. 애플리케이션 코드 변경 없음 | + +[집계](evidence/metrics.json) · [실행 원장](evidence/DR-RESTORE-20261018-FRESHNESS1/events.jsonl) · [동결 입력](evidence/DR-RESTORE-20261018-FRESHNESS1/inputs.json) · [요약](evidence/DR-RESTORE-20261018-FRESHNESS1/summary.json) · [기능 관측](evidence/DR-RESTORE-20261018-FRESHNESS1/functional-observations.json) · [비식별 파생물·해시 목록](evidence/export-index.json). diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/BACKUP-result.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/BACKUP-result.json new file mode 100644 index 00000000..5b89408b --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/BACKUP-result.json @@ -0,0 +1,16 @@ +{ + "elapsed_seconds": 83.177, + "finished_at_utc": "2026-10-18T14:22:20.722199+00:00", + "location": "isolated local runtime and owned cloud source database", + "observations": { + "backup_command_exit_code": 0, + "backup_id": "20261018T142124", + "final_status": "DONE", + "verify_exit_code": 0 + }, + "purpose": "새 OpenSQL 물리 기준 백업 검증", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-BACKUP", + "started_at_utc": "2026-10-18T14:20:57.547312+00:00", + "success_criteria": "DONE·pg_verifybackup 0", + "verdict": "PASS" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/BOOTSTRAP-result.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/BOOTSTRAP-result.json new file mode 100644 index 00000000..b91fb0d3 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/BOOTSTRAP-result.json @@ -0,0 +1,16 @@ +{ + "elapsed_seconds": 29.582, + "finished_at_utc": "2026-10-18T14:22:50.306115+00:00", + "location": "isolated local runtime and owned cloud source database", + "observations": { + "bootstrap_sha256": "984bbf5381b611daf7c1e57781686c6e74762eae2a0238f1947444da494366ab", + "license_environment_present": true, + "source_settings": 8, + "target_settings": 8 + }, + "purpose": "장애 전 설치 환경·클러스터 식별 보존", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-BOOTSTRAP", + "started_at_utc": "2026-10-18T14:22:20.724864+00:00", + "success_criteria": "라이선스 환경·소스/대상 설정 고정, 클라이언트 계정 ID 고정", + "verdict": "PASS" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/CORPUS-result.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/CORPUS-result.json new file mode 100644 index 00000000..5f5f6882 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/CORPUS-result.json @@ -0,0 +1,32 @@ +{ + "elapsed_seconds": 140.662, + "finished_at_utc": "2026-10-18T14:18:44.965938+00:00", + "location": "isolated local runtime and owned cloud source database", + "observations": [ + { + "elapsed_seconds": 32.716, + "ordinal": 1, + "state": "INDEXED" + }, + { + "elapsed_seconds": 24.861, + "ordinal": 2, + "state": "INDEXED" + }, + { + "elapsed_seconds": 28.196, + "ordinal": 3, + "state": "INDEXED" + }, + { + "elapsed_seconds": 25.107, + "ordinal": 4, + "state": "INDEXED" + } + ], + "purpose": "기능용 PDF·DOCX 4건 준비", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-CORPUS", + "started_at_utc": "2026-10-18T14:16:24.307909+00:00", + "success_criteria": "업로드 201 4건·INDEXED 4건, 실제 BGE", + "verdict": "PASS" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/DATABASE-result.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/DATABASE-result.json new file mode 100644 index 00000000..d112d795 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/DATABASE-result.json @@ -0,0 +1,15 @@ +{ + "elapsed_seconds": 29.674, + "finished_at_utc": "2026-10-18T14:11:25.232088+00:00", + "location": "isolated local runtime and owned cloud source database", + "observations": { + "database_created": true, + "roles_created": 2, + "vector_version": "0.8.1" + }, + "purpose": "전용 논리 DB·두 계정 준비", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-DATABASE", + "started_at_utc": "2026-10-18T14:10:55.559295+00:00", + "success_criteria": "소유 마커·역할·vector 확장 및 해당 DB 함수 권한", + "verdict": "PASS" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/INTERRUPT-result.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/INTERRUPT-result.json new file mode 100644 index 00000000..5910cc9a --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/INTERRUPT-result.json @@ -0,0 +1,18 @@ +{ + "elapsed_seconds": 57.008, + "finished_at_utc": "2026-10-18T14:20:57.546547+00:00", + "location": "isolated local runtime and owned cloud source database", + "observations": { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXING", + "embedding_count": 0, + "job_status": "PROCESSING", + "version_status": "EMBEDDING" + }, + "purpose": "실제 BGE 응답 대기 중 Job 중단", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-INTERRUPT", + "started_at_utc": "2026-10-18T14:20:00.539568+00:00", + "success_criteria": "Job PROCESSING·Attempt 1·Worker=false 재기동", + "verdict": "PASS" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/OFFSITE-result.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/OFFSITE-result.json new file mode 100644 index 00000000..534eb5f2 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/OFFSITE-result.json @@ -0,0 +1,21 @@ +{ + "elapsed_seconds": 67.286, + "finished_at_utc": "2026-10-18T14:23:57.593514+00:00", + "location": "isolated local runtime and owned cloud source database", + "observations": { + "backup_bytes": 104355840, + "completed_wals": 5, + "elapsed_seconds": 67.284, + "finished_at_utc": "2026-10-18T14:23:57.591442+00:00", + "preserved_originals": 5, + "purpose": "완성 WAL·identity·기준 백업과 원본 보존본의 외부 전송 및 바이트 대조", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-TRANSFER-0", + "started_at_utc": "2026-10-18T14:22:50.309279+00:00", + "verdict": "PASS" + }, + "purpose": "기준 백업·identity·WAL·원본 5건 외부 보존", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-OFFSITE", + "started_at_utc": "2026-10-18T14:22:50.309035+00:00", + "success_criteria": "generation 읽기·SHA·크기 일치", + "verdict": "PASS" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/RECEIVER-result.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/RECEIVER-result.json new file mode 100644 index 00000000..274b753b --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/RECEIVER-result.json @@ -0,0 +1,113 @@ +{ + "elapsed_seconds": 75.57, + "finished_at_utc": "2026-10-18T14:20:00.535582+00:00", + "location": "isolated local runtime and owned cloud source database", + "observations": { + "checks": [ + { + "check": "PostgreSQL", + "status": "OK" + }, + { + "check": "superuser or standard user with backup privileges", + "status": "OK" + }, + { + "check": "PostgreSQL streaming", + "status": "OK" + }, + { + "check": "wal_level", + "status": "OK" + }, + { + "check": "directories", + "status": "OK" + }, + { + "check": "retention policy settings", + "status": "OK" + }, + { + "check": "backup maximum age", + "status": "OK" + }, + { + "check": "backup minimum size", + "status": "OK" + }, + { + "check": "wal maximum age", + "status": "OK" + }, + { + "check": "wal size", + "status": "OK" + }, + { + "check": "compression settings", + "status": "OK" + }, + { + "check": "failed backups", + "status": "OK" + }, + { + "check": "minimum redundancy requirements", + "status": "OK" + }, + { + "check": "pg_basebackup", + "status": "OK" + }, + { + "check": "pg_basebackup compatible", + "status": "OK" + }, + { + "check": "pg_basebackup supports tablespaces mapping", + "status": "OK" + }, + { + "check": "systemid coherence", + "status": "OK" + }, + { + "check": "pg_receivexlog", + "status": "OK" + }, + { + "check": "pg_receivexlog compatible", + "status": "OK" + }, + { + "check": "receive-wal running", + "status": "OK" + }, + { + "check": "archive_mode", + "status": "OK" + }, + { + "check": "archive_command", + "status": "OK" + }, + { + "check": "continuous archiving", + "status": "OK" + }, + { + "check": "archiver errors", + "status": "OK" + } + ], + "failed": 0, + "passed": 24, + "permanent_slot_created": false + }, + "purpose": "실제 WAL 수신·Barman 준비 점검", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-RECEIVER", + "started_at_utc": "2026-10-18T14:18:44.967660+00:00", + "success_criteria": "수신기 1·완성 WAL·준비 항목 통과", + "verdict": "PASS" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/SERVICES-result.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/SERVICES-result.json new file mode 100644 index 00000000..c6a1ecc8 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/SERVICES-result.json @@ -0,0 +1,14 @@ +{ + "elapsed_seconds": 290.033, + "finished_at_utc": "2026-10-18T14:16:24.307469+00:00", + "location": "isolated local runtime and owned cloud source database", + "observations": { + "ready": true, + "worker_enabled": true + }, + "purpose": "소유 캐시·실제 BGE 프록시·새 JVM 준비", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-SERVICES", + "started_at_utc": "2026-10-18T14:11:34.283216+00:00", + "success_criteria": "readiness 200·리스너 소유권 확인", + "verdict": "PASS" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/TOOLING-result.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/TOOLING-result.json new file mode 100644 index 00000000..03094038 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/TOOLING-result.json @@ -0,0 +1,14 @@ +{ + "elapsed_seconds": 9.049, + "finished_at_utc": "2026-10-18T14:11:34.282138+00:00", + "location": "isolated local runtime and owned cloud source database", + "observations": { + "database_matched": true, + "standby": false + }, + "purpose": "전용 암호화 터널·Barman 도구 접속", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-TOOLING", + "started_at_utc": "2026-10-18T14:11:25.232998+00:00", + "success_criteria": "새 DB 일치·Primary 확인, 공개 설정 변경 없음", + "verdict": "PASS" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/artifact-identity.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/artifact-identity.json new file mode 100644 index 00000000..ee3ae5c8 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/artifact-identity.json @@ -0,0 +1,6 @@ +{ + "identical_to_baseline": true, + "jar_sha256": "08c5c458110bc96fc4c92cc5b02bd160f6c08b59e21d9ef536f2fa50aa76b27d", + "java_source_revision": "3bff887a5f6abdf4155957a658599f8f0366a141", + "wal_fix_revision": "44de876fb55a1c02cf9653ef863ccc21c4b40ba5" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/checkpoint-coverage.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/checkpoint-coverage.json new file mode 100644 index 00000000..970c608b --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/checkpoint-coverage.json @@ -0,0 +1,11 @@ +{ + "checked_before_worker": true, + "covered_acknowledged_requests": 60, + "covered_missing": 0, + "covered_restored": 60, + "required_wal": "000000060000000000000030", + "tail_missing": 5, + "tail_requests": 5, + "tail_restored": 0, + "verdict": "PASS" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/embedding-events.jsonl b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/embedding-events.jsonl new file mode 100644 index 00000000..c4e6e66b --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/embedding-events.jsonl @@ -0,0 +1,64 @@ +{"at_utc": "2026-10-18T14:16:53.955145+00:00", "event": "real_embedding_request", "ordinal": 1} +{"at_utc": "2026-10-18T14:17:28.738087+00:00", "event": "real_embedding_request", "ordinal": 2} +{"at_utc": "2026-10-18T14:17:59.225279+00:00", "event": "real_embedding_request", "ordinal": 3} +{"at_utc": "2026-10-18T14:18:33.731839+00:00", "event": "real_embedding_request", "ordinal": 4} +{"at_utc": "2026-10-18T14:20:22.806107+00:00", "event": "real_embedding_request", "ordinal": 5} +{"at_utc": "2026-10-18T14:20:25.992603+00:00", "event": "real_embedding_response_held"} +{"at_utc": "2026-10-18T14:39:12.285139+00:00", "event": "real_embedding_request", "ordinal": 6} +{"at_utc": "2026-10-18T14:39:12.323690+00:00", "event": "real_embedding_request", "ordinal": 7} +{"at_utc": "2026-10-18T14:39:14.401191+00:00", "event": "real_embedding_request", "ordinal": 8} +{"at_utc": "2026-10-18T14:39:40.710830+00:00", "event": "real_embedding_request", "ordinal": 9} +{"at_utc": "2026-10-18T14:39:40.844528+00:00", "event": "real_embedding_request", "ordinal": 10} +{"at_utc": "2026-10-18T14:39:56.564259+00:00", "event": "real_embedding_request", "ordinal": 11} +{"at_utc": "2026-10-18T14:40:08.356390+00:00", "event": "real_embedding_request", "ordinal": 12} +{"at_utc": "2026-10-18T14:40:21.859380+00:00", "event": "real_embedding_request", "ordinal": 13} +{"at_utc": "2026-10-18T14:40:32.740098+00:00", "event": "real_embedding_request", "ordinal": 14} +{"at_utc": "2026-10-18T14:40:43.973709+00:00", "event": "real_embedding_request", "ordinal": 15} +{"at_utc": "2026-10-18T14:40:58.442388+00:00", "event": "real_embedding_request", "ordinal": 16} +{"at_utc": "2026-10-18T14:41:05.474289+00:00", "event": "real_embedding_request", "ordinal": 17} +{"at_utc": "2026-10-18T14:41:25.951868+00:00", "event": "real_embedding_request", "ordinal": 18} +{"at_utc": "2026-10-18T14:41:27.090452+00:00", "event": "real_embedding_request", "ordinal": 19} +{"at_utc": "2026-10-18T14:41:53.021259+00:00", "event": "real_embedding_request", "ordinal": 20} +{"at_utc": "2026-10-18T14:41:53.285362+00:00", "event": "real_embedding_request", "ordinal": 21} +{"at_utc": "2026-10-18T14:42:20.574997+00:00", "event": "real_embedding_request", "ordinal": 22} +{"at_utc": "2026-10-18T14:42:21.017719+00:00", "event": "real_embedding_request", "ordinal": 23} +{"at_utc": "2026-10-18T14:42:44.913738+00:00", "event": "real_embedding_request", "ordinal": 24} +{"at_utc": "2026-10-18T14:42:47.711877+00:00", "event": "real_embedding_request", "ordinal": 25} +{"at_utc": "2026-10-18T14:43:11.972553+00:00", "event": "real_embedding_request", "ordinal": 26} +{"at_utc": "2026-10-18T14:43:12.694016+00:00", "event": "real_embedding_request", "ordinal": 27} +{"at_utc": "2026-10-18T14:43:38.985005+00:00", "event": "real_embedding_request", "ordinal": 28} +{"at_utc": "2026-10-18T14:43:39.747030+00:00", "event": "real_embedding_request", "ordinal": 29} +{"at_utc": "2026-10-18T14:44:05.710833+00:00", "event": "real_embedding_request", "ordinal": 30} +{"at_utc": "2026-10-18T14:44:05.956461+00:00", "event": "real_embedding_request", "ordinal": 31} +{"at_utc": "2026-10-18T14:44:32.193144+00:00", "event": "real_embedding_request", "ordinal": 32} +{"at_utc": "2026-10-18T14:44:32.602315+00:00", "event": "real_embedding_request", "ordinal": 33} +{"at_utc": "2026-10-18T14:44:56.226974+00:00", "event": "real_embedding_request", "ordinal": 34} +{"at_utc": "2026-10-18T14:44:59.709026+00:00", "event": "real_embedding_request", "ordinal": 35} +{"at_utc": "2026-10-18T14:45:25.001810+00:00", "event": "real_embedding_request", "ordinal": 36} +{"at_utc": "2026-10-18T14:45:26.141888+00:00", "event": "real_embedding_request", "ordinal": 37} +{"at_utc": "2026-10-18T14:45:52.443049+00:00", "event": "real_embedding_request", "ordinal": 38} +{"at_utc": "2026-10-18T14:45:53.310116+00:00", "event": "real_embedding_request", "ordinal": 39} +{"at_utc": "2026-10-18T14:46:21.456604+00:00", "event": "real_embedding_request", "ordinal": 40} +{"at_utc": "2026-10-18T14:46:21.749093+00:00", "event": "real_embedding_request", "ordinal": 41} +{"at_utc": "2026-10-18T14:46:48.885012+00:00", "event": "real_embedding_request", "ordinal": 42} +{"at_utc": "2026-10-18T14:46:49.406651+00:00", "event": "real_embedding_request", "ordinal": 43} +{"at_utc": "2026-10-18T14:47:17.548793+00:00", "event": "real_embedding_request", "ordinal": 44} +{"at_utc": "2026-10-18T14:47:18.065703+00:00", "event": "real_embedding_request", "ordinal": 45} +{"at_utc": "2026-10-18T14:47:44.321660+00:00", "event": "real_embedding_request", "ordinal": 46} +{"at_utc": "2026-10-18T14:47:45.060292+00:00", "event": "real_embedding_request", "ordinal": 47} +{"at_utc": "2026-10-18T14:48:12.303478+00:00", "event": "real_embedding_request", "ordinal": 48} +{"at_utc": "2026-10-18T14:48:13.083299+00:00", "event": "real_embedding_request", "ordinal": 49} +{"at_utc": "2026-10-18T14:48:40.350723+00:00", "event": "real_embedding_request", "ordinal": 50} +{"at_utc": "2026-10-18T14:48:41.639852+00:00", "event": "real_embedding_request", "ordinal": 51} +{"at_utc": "2026-10-18T14:49:08.351242+00:00", "event": "real_embedding_request", "ordinal": 52} +{"at_utc": "2026-10-18T14:49:08.354452+00:00", "event": "real_embedding_request", "ordinal": 53} +{"at_utc": "2026-10-18T14:49:32.875774+00:00", "event": "real_embedding_request", "ordinal": 54} +{"at_utc": "2026-10-18T14:49:36.135222+00:00", "event": "real_embedding_request", "ordinal": 55} +{"at_utc": "2026-10-18T14:50:01.204902+00:00", "event": "real_embedding_request", "ordinal": 56} +{"at_utc": "2026-10-18T14:50:01.339260+00:00", "event": "real_embedding_request", "ordinal": 57} +{"at_utc": "2026-10-18T14:50:25.814145+00:00", "event": "real_embedding_request", "ordinal": 58} +{"at_utc": "2026-10-18T14:50:29.166982+00:00", "event": "real_embedding_request", "ordinal": 59} +{"at_utc": "2026-10-18T14:50:53.564444+00:00", "event": "real_embedding_request", "ordinal": 60} +{"at_utc": "2026-10-18T14:50:54.088866+00:00", "event": "real_embedding_request", "ordinal": 61} +{"at_utc": "2026-10-18T14:51:21.033149+00:00", "event": "real_embedding_request", "ordinal": 62} +{"at_utc": "2026-10-18T14:51:22.849740+00:00", "event": "real_embedding_request", "ordinal": 63} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/events.jsonl b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/events.jsonl new file mode 100644 index 00000000..cb668f6b --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/events.jsonl @@ -0,0 +1,337 @@ +{"at_utc": "2026-10-18T14:10:55.559+00:00", "data": {"scope": "isolated_source_database", "source_revision": "3bff887"}, "kind": "preparation_started", "previous": "0000000000000000000000000000000000000000000000000000000000000000", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 1, "sha256": "249240558fc8666b22efe8ba081eb1eacbc8b421867f253df30e0bc180267c8c"} +{"at_utc": "2026-10-18T14:11:25.231+00:00", "data": {"public_database_modified": false, "roles": 2, "vector_version": "0.8.1"}, "kind": "source_database_prepared", "previous": "249240558fc8666b22efe8ba081eb1eacbc8b421867f253df30e0bc180267c8c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 2, "sha256": "72c8eb26d97feb1d70a08eb7bd084913f56440a25a18a90e46ce20d55387c943"} +{"at_utc": "2026-10-18T14:11:25.232+00:00", "data": {"elapsed_seconds": 29.674, "stage": "DATABASE"}, "kind": "preparation_stage_passed", "previous": "72c8eb26d97feb1d70a08eb7bd084913f56440a25a18a90e46ce20d55387c943", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 3, "sha256": "6ebd319d86bb3d35353629e877d0cd23ab46e4688cb1a31e5ced206666ee0bec"} +{"at_utc": "2026-10-18T14:11:32.276+00:00", "data": {"private_endpoint_omitted": true, "purpose": "source"}, "kind": "tunnel_ready", "previous": "6ebd319d86bb3d35353629e877d0cd23ab46e4688cb1a31e5ced206666ee0bec", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 4, "sha256": "730807eb7cef207b2da58b3c12dec8444a72ff81c3e9715be4a5d0a37e0fe8db"} +{"at_utc": "2026-10-18T14:11:34.281+00:00", "data": {"database_matched": true, "standby": false}, "kind": "owned_tcp_ready", "previous": "730807eb7cef207b2da58b3c12dec8444a72ff81c3e9715be4a5d0a37e0fe8db", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 5, "sha256": "f08226d08d70b6906bb15f7bdea653305d1c37bae8a7f3c31e8b384f58074c1c"} +{"at_utc": "2026-10-18T14:11:34.283+00:00", "data": {"elapsed_seconds": 9.049, "stage": "TOOLING"}, "kind": "preparation_stage_passed", "previous": "f08226d08d70b6906bb15f7bdea653305d1c37bae8a7f3c31e8b384f58074c1c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 6, "sha256": "6f2648cd91561c5ee55da9ae86ce126fd5ca8d8d1e1592c022cb552b5e4732ea"} +{"at_utc": "2026-10-18T14:11:34.484+00:00", "data": {"response": "PONG"}, "kind": "owned_cache_ready", "previous": "6f2648cd91561c5ee55da9ae86ce126fd5ca8d8d1e1592c022cb552b5e4732ea", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 7, "sha256": "70f3947087b507acd6d9dc465d463c67276ece1948210e2f9ca8bfe44bef6a45"} +{"at_utc": "2026-10-18T14:16:24.307+00:00", "data": {"listener_ownership_verified": true, "purpose": "source", "worker_enabled": true}, "kind": "backend_ready", "previous": "70f3947087b507acd6d9dc465d463c67276ece1948210e2f9ca8bfe44bef6a45", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 8, "sha256": "5f51fb59f07480b5f0eb579d130115eb895ef5644dbb4662a10c68850ef57bc6"} +{"at_utc": "2026-10-18T14:16:24.307+00:00", "data": {"elapsed_seconds": 290.033, "stage": "SERVICES"}, "kind": "preparation_stage_passed", "previous": "5f51fb59f07480b5f0eb579d130115eb895ef5644dbb4662a10c68850ef57bc6", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 9, "sha256": "680f185ff8f3b44447ecc71cd2cd5e867d0bd3397c4cd10b166cbfa333cc206a"} +{"at_utc": "2026-10-18T14:16:27.589+00:00", "data": {"login_http": 200, "signup_http": 201}, "kind": "test_account_ready", "previous": "680f185ff8f3b44447ecc71cd2cd5e867d0bd3397c4cd10b166cbfa333cc206a", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 10, "sha256": "5210a3b2209378276236b9aa760ad3702c24679d46e8f0c3be7ce3f884bf2dd5"} +{"at_utc": "2026-10-18T14:16:27.612+00:00", "data": {"cohort": "functional", "request_id": "upload-0296462e118a428da1b889a1310a9f30", "sha256": "cb6c0dff45951c630ae0b02694bc954e159100cb2495170f8379171c20637ca8", "size": 1045}, "kind": "sent", "previous": "5210a3b2209378276236b9aa760ad3702c24679d46e8f0c3be7ce3f884bf2dd5", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 11, "sha256": "c96701d26607f603e37c371f012c5d5199aaa49f10c087c04621b546bce9c144"} +{"at_utc": "2026-10-18T14:16:35.002+00:00", "data": {"http_status": 201, "ids": {"documentId": 3, "documentVersionId": 3, "embeddingJobId": 1, "fileObjectId": 1}, "outcome": "ACKNOWLEDGED", "request_id": "upload-0296462e118a428da1b889a1310a9f30"}, "kind": "response", "previous": "c96701d26607f603e37c371f012c5d5199aaa49f10c087c04621b546bce9c144", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 12, "sha256": "69f3302c89cc2907b2707ee89357597cd71420dc19b9ccac5ddf5798bc1dfa87"} +{"at_utc": "2026-10-18T14:16:35.003+00:00", "data": {"cohort": "functional", "elapsed_seconds": 7.39, "http_status": 201, "ordinal": 1}, "kind": "upload_observed", "previous": "69f3302c89cc2907b2707ee89357597cd71420dc19b9ccac5ddf5798bc1dfa87", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 13, "sha256": "ff05d22141778f0d5eb5a6465ed285bcaf9e61e10ac22519eca8d8e83deac4f7"} +{"at_utc": "2026-10-18T14:16:36.090+00:00", "data": {"ordinal": 1, "phase": "source", "state": "PENDING"}, "kind": "index_state", "previous": "ff05d22141778f0d5eb5a6465ed285bcaf9e61e10ac22519eca8d8e83deac4f7", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 14, "sha256": "62590a6398c04754936783c2e1d3638cb5b334dd4d02398227492daefea11cc5"} +{"at_utc": "2026-10-18T14:16:47.044+00:00", "data": {"ordinal": 1, "phase": "source", "state": "PROCESSING"}, "kind": "index_state", "previous": "62590a6398c04754936783c2e1d3638cb5b334dd4d02398227492daefea11cc5", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 15, "sha256": "06b41703682f135bbb4dfa93966b765fa36d180b05317bd35712f5aee024b059"} +{"at_utc": "2026-10-18T14:17:07.724+00:00", "data": {"ordinal": 1, "phase": "source", "state": "INDEXED"}, "kind": "index_state", "previous": "06b41703682f135bbb4dfa93966b765fa36d180b05317bd35712f5aee024b059", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 16, "sha256": "f381581fc6b8261acd7d87e2022b8c972efbab8eefe2ad4557114155f5ebbe66"} +{"at_utc": "2026-10-18T14:17:07.726+00:00", "data": {"cohort": "functional", "request_id": "upload-ce6a718bdd0f4ace83465acdb3f9d275", "sha256": "6fca27eab558e63f007d70172241ee91cd3d3c40b868b7b6e5a821fc64d656bc", "size": 641}, "kind": "sent", "previous": "f381581fc6b8261acd7d87e2022b8c972efbab8eefe2ad4557114155f5ebbe66", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 17, "sha256": "c3842a55686573e9f96959f55275287a3c751326a0fd172762c7276f9d404130"} +{"at_utc": "2026-10-18T14:17:14.038+00:00", "data": {"http_status": 201, "ids": {"documentId": 4, "documentVersionId": 4, "embeddingJobId": 2, "fileObjectId": 2}, "outcome": "ACKNOWLEDGED", "request_id": "upload-ce6a718bdd0f4ace83465acdb3f9d275"}, "kind": "response", "previous": "c3842a55686573e9f96959f55275287a3c751326a0fd172762c7276f9d404130", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 18, "sha256": "139f60c0f91d90bf6e1ccaece8efba4be777f9d3b20efa9e1204b019ccb4f37c"} +{"at_utc": "2026-10-18T14:17:14.040+00:00", "data": {"cohort": "functional", "elapsed_seconds": 6.313, "http_status": 201, "ordinal": 2}, "kind": "upload_observed", "previous": "139f60c0f91d90bf6e1ccaece8efba4be777f9d3b20efa9e1204b019ccb4f37c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 19, "sha256": "9a10f1d656086307ae6d6206a5b6cdb005352c8801de3ba247ff7571696da2e6"} +{"at_utc": "2026-10-18T14:17:15.102+00:00", "data": {"ordinal": 2, "phase": "source", "state": "PENDING"}, "kind": "index_state", "previous": "9a10f1d656086307ae6d6206a5b6cdb005352c8801de3ba247ff7571696da2e6", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 20, "sha256": "493e77e768067d7609762d6378dd99c6ecf4a79a88f65aceb3b03f3ee151e31e"} +{"at_utc": "2026-10-18T14:17:18.488+00:00", "data": {"ordinal": 2, "phase": "source", "state": "PROCESSING"}, "kind": "index_state", "previous": "493e77e768067d7609762d6378dd99c6ecf4a79a88f65aceb3b03f3ee151e31e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 21, "sha256": "4c36c5ca7cd1c5c75b7aa9fd02df9740d30e43a5c42f0d5b204a15cddbdc7bea"} +{"at_utc": "2026-10-18T14:17:38.900+00:00", "data": {"ordinal": 2, "phase": "source", "state": "INDEXED"}, "kind": "index_state", "previous": "4c36c5ca7cd1c5c75b7aa9fd02df9740d30e43a5c42f0d5b204a15cddbdc7bea", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 22, "sha256": "a9e198c340543a6a8f658ddfe3405352e017f9d5cb09721bc46d657de89dcc29"} +{"at_utc": "2026-10-18T14:17:38.903+00:00", "data": {"cohort": "functional", "request_id": "upload-b1f116f44ea449fb90474ed3ffbde583", "sha256": "5d12c58a83aae3d5a60f4d776086df081a2d061b89536e4623b5ef66953b15d2", "size": 1045}, "kind": "sent", "previous": "a9e198c340543a6a8f658ddfe3405352e017f9d5cb09721bc46d657de89dcc29", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 23, "sha256": "61142b73c218123060412d5fd3ee7e858ae974cde015c180e62fcdd35ee7d34d"} +{"at_utc": "2026-10-18T14:17:45.307+00:00", "data": {"http_status": 201, "ids": {"documentId": 5, "documentVersionId": 5, "embeddingJobId": 3, "fileObjectId": 3}, "outcome": "ACKNOWLEDGED", "request_id": "upload-b1f116f44ea449fb90474ed3ffbde583"}, "kind": "response", "previous": "61142b73c218123060412d5fd3ee7e858ae974cde015c180e62fcdd35ee7d34d", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 24, "sha256": "0908e5da0ecc5de183a695dffc4554eb6cac3d4a7ba5a435baec0979b66fc5ce"} +{"at_utc": "2026-10-18T14:17:45.310+00:00", "data": {"cohort": "functional", "elapsed_seconds": 6.404, "http_status": 201, "ordinal": 3}, "kind": "upload_observed", "previous": "0908e5da0ecc5de183a695dffc4554eb6cac3d4a7ba5a435baec0979b66fc5ce", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 25, "sha256": "c8ff47934476948a5769f52b2ec33b877f46a3715ae2eef5abd5b5b674bb58c4"} +{"at_utc": "2026-10-18T14:17:46.371+00:00", "data": {"ordinal": 3, "phase": "source", "state": "PENDING"}, "kind": "index_state", "previous": "c8ff47934476948a5769f52b2ec33b877f46a3715ae2eef5abd5b5b674bb58c4", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 26, "sha256": "0cff62263fb4bb3e7f0254199e7bdc51cc292ce81af59b7ce53b080efdf88b94"} +{"at_utc": "2026-10-18T14:17:49.693+00:00", "data": {"ordinal": 3, "phase": "source", "state": "PROCESSING"}, "kind": "index_state", "previous": "0cff62263fb4bb3e7f0254199e7bdc51cc292ce81af59b7ce53b080efdf88b94", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 27, "sha256": "5823627de29f58b479fe629877b7b3115ced4d01646e838df988000c825c3986"} +{"at_utc": "2026-10-18T14:18:13.505+00:00", "data": {"ordinal": 3, "phase": "source", "state": "INDEXED"}, "kind": "index_state", "previous": "5823627de29f58b479fe629877b7b3115ced4d01646e838df988000c825c3986", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 28, "sha256": "8586872736ef6ca5a43a2b70e3f95abd768009f1d0f342ab3c1b6e821f3b8909"} +{"at_utc": "2026-10-18T14:18:13.508+00:00", "data": {"cohort": "functional", "request_id": "upload-123273586c1a4fa0a37270b2550be058", "sha256": "f281ebd98e5360437810e41ee282d7fbc3aea13859a25bff53494c709fd8e71c", "size": 641}, "kind": "sent", "previous": "8586872736ef6ca5a43a2b70e3f95abd768009f1d0f342ab3c1b6e821f3b8909", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 29, "sha256": "6703cc08d914630b816e60a9a4a29d5e6fe2269ec645760ae9aacdd1d04ac249"} +{"at_utc": "2026-10-18T14:18:19.856+00:00", "data": {"http_status": 201, "ids": {"documentId": 6, "documentVersionId": 6, "embeddingJobId": 4, "fileObjectId": 4}, "outcome": "ACKNOWLEDGED", "request_id": "upload-123273586c1a4fa0a37270b2550be058"}, "kind": "response", "previous": "6703cc08d914630b816e60a9a4a29d5e6fe2269ec645760ae9aacdd1d04ac249", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 30, "sha256": "865c2a3965f67a307e02ddfe43802e6c68bdf4ad9174f48c1eb7327d2c8e8f69"} +{"at_utc": "2026-10-18T14:18:19.859+00:00", "data": {"cohort": "functional", "elapsed_seconds": 6.349, "http_status": 201, "ordinal": 4}, "kind": "upload_observed", "previous": "865c2a3965f67a307e02ddfe43802e6c68bdf4ad9174f48c1eb7327d2c8e8f69", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 31, "sha256": "99a170c17fd3ef5e5a654d07313b9207a38d786081cc26e2d6d4efb60f0d608e"} +{"at_utc": "2026-10-18T14:18:20.918+00:00", "data": {"ordinal": 4, "phase": "source", "state": "PENDING"}, "kind": "index_state", "previous": "99a170c17fd3ef5e5a654d07313b9207a38d786081cc26e2d6d4efb60f0d608e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 32, "sha256": "26c3be8bd457db3c94dd08b95259004096ad0057b03e18d4eb79cc9de5e18169"} +{"at_utc": "2026-10-18T14:18:24.251+00:00", "data": {"ordinal": 4, "phase": "source", "state": "PROCESSING"}, "kind": "index_state", "previous": "26c3be8bd457db3c94dd08b95259004096ad0057b03e18d4eb79cc9de5e18169", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 33, "sha256": "8da88d59101bb1a0b9daaf21e4eaf924f5ac013d18e2e22b9b5dbabe9374cd14"} +{"at_utc": "2026-10-18T14:18:44.965+00:00", "data": {"ordinal": 4, "phase": "source", "state": "INDEXED"}, "kind": "index_state", "previous": "8da88d59101bb1a0b9daaf21e4eaf924f5ac013d18e2e22b9b5dbabe9374cd14", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 34, "sha256": "100ebd27c77479c0e451dae7926e59cca8729434d99419d5a6fd9d0087a768c7"} +{"at_utc": "2026-10-18T14:18:44.967+00:00", "data": {"elapsed_seconds": 140.662, "stage": "CORPUS"}, "kind": "preparation_stage_passed", "previous": "100ebd27c77479c0e451dae7926e59cca8729434d99419d5a6fd9d0087a768c7", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 35, "sha256": "dd102958cc3b88eed99093a79ad249608afba0b522cdfaeae9d4ffb6d0f145a6"} +{"at_utc": "2026-10-18T14:18:55.577+00:00", "data": {"count": 0, "probe": 1}, "kind": "receiver_readiness_probe", "previous": "dd102958cc3b88eed99093a79ad249608afba0b522cdfaeae9d4ffb6d0f145a6", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 36, "sha256": "321dc05010d945b95a2833b317d1598804c25fe20c019eec7a94975fa0886daa"} +{"at_utc": "2026-10-18T14:19:07.546+00:00", "data": {"count": 0, "probe": 2}, "kind": "receiver_readiness_probe", "previous": "321dc05010d945b95a2833b317d1598804c25fe20c019eec7a94975fa0886daa", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 37, "sha256": "711341610304477941d410378ec304a00fe433e773f2a6f5135d78bc2e9ac8ac"} +{"at_utc": "2026-10-18T14:19:20.234+00:00", "data": {"count": 1, "probe": 3}, "kind": "receiver_readiness_probe", "previous": "711341610304477941d410378ec304a00fe433e773f2a6f5135d78bc2e9ac8ac", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 38, "sha256": "9b7c8829edea1484a13ebac9318a9947f789a8ec321022b849ae4b68419918a9"} +{"at_utc": "2026-10-18T14:20:00.539+00:00", "data": {"elapsed_seconds": 75.57, "stage": "RECEIVER"}, "kind": "preparation_stage_passed", "previous": "9b7c8829edea1484a13ebac9318a9947f789a8ec321022b849ae4b68419918a9", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 39, "sha256": "ebdf3b53698c100477c6a5aaf3da105ded8b4338ada9b5b880ba49af41b49cd7"} +{"at_utc": "2026-10-18T14:20:00.545+00:00", "data": {"cohort": "functional", "request_id": "upload-6e7a22eea289466da0fbb8865647301b", "sha256": "8a3cbd1a5c5338d68075c056a729d4752a8e31807309e0caa12eb0937ea64902", "size": 1045}, "kind": "sent", "previous": "ebdf3b53698c100477c6a5aaf3da105ded8b4338ada9b5b880ba49af41b49cd7", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 40, "sha256": "8231edf77d62c942f1331a55e50fe9cb7678265607d7e4aa1fe4bd576aa4ec03"} +{"at_utc": "2026-10-18T14:20:09.511+00:00", "data": {"http_status": 201, "ids": {"documentId": 7, "documentVersionId": 7, "embeddingJobId": 5, "fileObjectId": 5}, "outcome": "ACKNOWLEDGED", "request_id": "upload-6e7a22eea289466da0fbb8865647301b"}, "kind": "response", "previous": "8231edf77d62c942f1331a55e50fe9cb7678265607d7e4aa1fe4bd576aa4ec03", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 41, "sha256": "281d21b94619e694c2e9dba4b4a8de1b441dc57e717d97b8dc0597b9f729a81b"} +{"at_utc": "2026-10-18T14:20:09.516+00:00", "data": {"cohort": "functional", "elapsed_seconds": 8.967, "http_status": 201, "ordinal": 5}, "kind": "upload_observed", "previous": "281d21b94619e694c2e9dba4b4a8de1b441dc57e717d97b8dc0597b9f729a81b", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 42, "sha256": "d4efbf7f6c18ca438af18800798921e008eba3004782e36d20d7a48b65fad68e"} +{"at_utc": "2026-10-18T14:20:36.779+00:00", "data": {"attempt_count": 1, "chunk_count": 3, "document_status": "INDEXING", "embedding_count": 0, "job_status": "PROCESSING", "ordinal": 5, "source_jvm_signal": "SIGKILL", "version_status": "EMBEDDING"}, "kind": "actual_job_interrupted", "previous": "d4efbf7f6c18ca438af18800798921e008eba3004782e36d20d7a48b65fad68e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 43, "sha256": "f23f1fe4e8551eb99dea47264ac899f6412bcd46296bd33be451a41cf89f0c57"} +{"at_utc": "2026-10-18T14:20:57.546+00:00", "data": {"listener_ownership_verified": true, "purpose": "source", "worker_enabled": false}, "kind": "backend_ready", "previous": "f23f1fe4e8551eb99dea47264ac899f6412bcd46296bd33be451a41cf89f0c57", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 44, "sha256": "8c8cf28fa260b90b9bb6d353e2e15a0070a1038b0a89495d73b295169346842c"} +{"at_utc": "2026-10-18T14:20:57.547+00:00", "data": {"elapsed_seconds": 57.008, "stage": "INTERRUPT"}, "kind": "preparation_stage_passed", "previous": "8c8cf28fa260b90b9bb6d353e2e15a0070a1038b0a89495d73b295169346842c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 45, "sha256": "4526af65d92f1df521049f62b0a3a933a3d01e7787a89b9c32df01a3296c8db2"} +{"at_utc": "2026-10-18T14:22:20.060+00:00", "data": {"archive_exit_code": 0, "done": true, "probe": 1}, "kind": "backup_finalization_probe", "previous": "4526af65d92f1df521049f62b0a3a933a3d01e7787a89b9c32df01a3296c8db2", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 46, "sha256": "52f7c833b8f4d64cde357b1b855da8a71287d1ff51d5f645abc0f213ac06b367"} +{"at_utc": "2026-10-18T14:22:20.724+00:00", "data": {"elapsed_seconds": 83.177, "stage": "BACKUP"}, "kind": "preparation_stage_passed", "previous": "52f7c833b8f4d64cde357b1b855da8a71287d1ff51d5f645abc0f213ac06b367", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 47, "sha256": "1dec15dc746a715398993c850056042ccc8883511ee005ae64e2985dd5ebb775"} +{"at_utc": "2026-10-18T14:22:40.075+00:00", "data": {"captured_before_cutoff": true, "license_environment_present": true, "sha256": "984bbf5381b611daf7c1e57781686c6e74762eae2a0238f1947444da494366ab"}, "kind": "bootstrap_frozen", "previous": "1dec15dc746a715398993c850056042ccc8883511ee005ae64e2985dd5ebb775", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 48, "sha256": "0bc79b206317b87222276c4fe9cee78a9dcc29c76ac630fd914ebbc1b541b04d"} +{"at_utc": "2026-10-18T14:22:50.308+00:00", "data": {"elapsed_seconds": 29.582, "stage": "BOOTSTRAP"}, "kind": "preparation_stage_passed", "previous": "0bc79b206317b87222276c4fe9cee78a9dcc29c76ac630fd914ebbc1b541b04d", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 49, "sha256": "fc66bb4841c74d6e2a8074d638e3c11ed620aecb0e8e1c047e1657bd02f38d68"} +{"at_utc": "2026-10-18T14:23:15.276+00:00", "data": {"fileObjectId": 1, "generation": "1792333394887109", "sha256": "cb6c0dff45951c630ae0b02694bc954e159100cb2495170f8379171c20637ca8", "size": 1045}, "kind": "original_preserved", "previous": "fc66bb4841c74d6e2a8074d638e3c11ed620aecb0e8e1c047e1657bd02f38d68", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 50, "sha256": "33733f4036f848f7644e63bb3099cd9520dfd9defff9c9dc4c0792c85b39e172"} +{"at_utc": "2026-10-18T14:23:16.597+00:00", "data": {"fileObjectId": 2, "generation": "1792333396369177", "sha256": "6fca27eab558e63f007d70172241ee91cd3d3c40b868b7b6e5a821fc64d656bc", "size": 641}, "kind": "original_preserved", "previous": "33733f4036f848f7644e63bb3099cd9520dfd9defff9c9dc4c0792c85b39e172", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 51, "sha256": "d7d2531e42dea252c7211736d81b9d32cc5a1d4c39a0e887baa77c78b47c815b"} +{"at_utc": "2026-10-18T14:23:17.886+00:00", "data": {"fileObjectId": 3, "generation": "1792333397665022", "sha256": "5d12c58a83aae3d5a60f4d776086df081a2d061b89536e4623b5ef66953b15d2", "size": 1045}, "kind": "original_preserved", "previous": "d7d2531e42dea252c7211736d81b9d32cc5a1d4c39a0e887baa77c78b47c815b", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 52, "sha256": "52d3cf994037e66cd2b5e87f455b110dd63621963a2fc1d88e91c0f84a8a942e"} +{"at_utc": "2026-10-18T14:23:19.162+00:00", "data": {"fileObjectId": 4, "generation": "1792333398846804", "sha256": "f281ebd98e5360437810e41ee282d7fbc3aea13859a25bff53494c709fd8e71c", "size": 641}, "kind": "original_preserved", "previous": "52d3cf994037e66cd2b5e87f455b110dd63621963a2fc1d88e91c0f84a8a942e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 53, "sha256": "e1786d8053e23eeaf96891c6d8c477c8a537e5946eeb59842c6698f2e82d2172"} +{"at_utc": "2026-10-18T14:23:20.499+00:00", "data": {"fileObjectId": 5, "generation": "1792333400275899", "sha256": "8a3cbd1a5c5338d68075c056a729d4752a8e31807309e0caa12eb0937ea64902", "size": 1045}, "kind": "original_preserved", "previous": "e1786d8053e23eeaf96891c6d8c477c8a537e5946eeb59842c6698f2e82d2172", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 54, "sha256": "2c71ecc0be3368590cb1f53d0311a0afddb6430e2791eb3c3fa17b8e1af33857"} +{"at_utc": "2026-10-18T14:23:36.580+00:00", "data": {"generation": "1792333416353658", "name": "00000006.history", "sha256": "7dbe25c372b72e1132dca125c614a696011f1aeb3c1dfdac9cc8e19055a6d3c6", "size": 209}, "kind": "wal_preserved", "previous": "2c71ecc0be3368590cb1f53d0311a0afddb6430e2791eb3c3fa17b8e1af33857", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 55, "sha256": "6a5cf16b8f4211e554d09dd096b5cafe480bd680525f6eaa4ecbc57f8cd620ce"} +{"at_utc": "2026-10-18T14:23:41.563+00:00", "data": {"generation": "1792333419682653", "name": "00000006000000000000002B", "sha256": "4861c68f46975ba4736e86b9b4472dd5db779a984b7fa14928b1e92c716ee8a6", "size": 16777216}, "kind": "wal_preserved", "previous": "6a5cf16b8f4211e554d09dd096b5cafe480bd680525f6eaa4ecbc57f8cd620ce", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 56, "sha256": "15dec8814f3ac2c074f8e3fd6eb6c40d23bc749e683bc250e53cb8e637a7f603"} +{"at_utc": "2026-10-18T14:23:47.052+00:00", "data": {"generation": "1792333424481173", "name": "00000006000000000000002C", "sha256": "9581d9f666be0b253f9dadc2eb72ad872b174f3fd2da33a9b9c3e74debb9d8d9", "size": 16777216}, "kind": "wal_preserved", "previous": "15dec8814f3ac2c074f8e3fd6eb6c40d23bc749e683bc250e53cb8e637a7f603", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 57, "sha256": "f269dd53cb6b9f212ad27a0195840cebace86a16e3786aa3e4a3c0836b6f9863"} +{"at_utc": "2026-10-18T14:23:51.435+00:00", "data": {"generation": "1792333429992965", "name": "00000006000000000000002D", "sha256": "cd5a6c199d04384cebceee15e58d497d5e3fdb05e2a53e5955f35679ee38bd55", "size": 16777216}, "kind": "wal_preserved", "previous": "f269dd53cb6b9f212ad27a0195840cebace86a16e3786aa3e4a3c0836b6f9863", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 58, "sha256": "401ab4c9fe7222efbda84fc61124c4359fdc1ab5d4fc0e3e134f42bfa5058c06"} +{"at_utc": "2026-10-18T14:23:56.167+00:00", "data": {"generation": "1792333434426895", "name": "00000006000000000000002E", "sha256": "3e2ee29d602a650c28e4c290fe8ef1a73c1c108cab06431d86dd4e6c97afc964", "size": 16777216}, "kind": "wal_preserved", "previous": "401ab4c9fe7222efbda84fc61124c4359fdc1ab5d4fc0e3e134f42bfa5058c06", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 59, "sha256": "66d4ea14ebd0d2798fd165fd08407598394c5477cb50cf2e83dd7605af264b28"} +{"at_utc": "2026-10-18T14:23:57.591+00:00", "data": {"archive_attempts": 2, "covered_acknowledged_requests": 5, "ordinal": 0, "remote_bytes_verified": true, "required_wal": "00000006000000000000002E"}, "kind": "wal_barrier_verified", "previous": "66d4ea14ebd0d2798fd165fd08407598394c5477cb50cf2e83dd7605af264b28", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 60, "sha256": "4385a78bb1ea6836f3b08354497dda4f16e2646659b476e9c2028735ce1ad79d"} +{"at_utc": "2026-10-18T14:23:57.593+00:00", "data": {"complete_wals": 5, "elapsed_seconds": 67.284, "ordinal": 0, "preserved_originals": 5}, "kind": "offsite_round_completed", "previous": "4385a78bb1ea6836f3b08354497dda4f16e2646659b476e9c2028735ce1ad79d", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 61, "sha256": "c2e9e6ab07d9f2adb4fbd4d38eb2e275eebcae47e9292554996ca3526479884b"} +{"at_utc": "2026-10-18T14:23:57.595+00:00", "data": {"elapsed_seconds": 67.286, "stage": "OFFSITE"}, "kind": "preparation_stage_passed", "previous": "c2e9e6ab07d9f2adb4fbd4d38eb2e275eebcae47e9292554996ca3526479884b", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 62, "sha256": "f9dc9b277d5fc4cfde92293a223347935dfc705d8d8ec5a3687bb1bddd49e044"} +{"at_utc": "2026-10-18T14:24:31.415+00:00", "data": {"chunkId": 5, "documentId": 3, "query_length": 1000, "query_sha256": "642399a1e9fc2255649ee400f1f99bb220f5a6068d1cfade7e27c441851bbfba", "top_k": 20}, "kind": "search_fixture_frozen", "previous": "f9dc9b277d5fc4cfde92293a223347935dfc705d8d8ec5a3687bb1bddd49e044", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 63, "sha256": "6edc6ac6a6086d4fc220570497fa6d5aef82a1c62d3474465a9370bad639bd9e"} +{"at_utc": "2026-10-18T14:24:31.420+00:00", "data": {"product_revision": "3bff887a5f6abdf4155957a658599f8f0366a141", "sha256": {"dr_cleanup_private.py": "efa355952fea5ccdd98c75bdb0d9399a68198d6bafb25db3181d6bf68117d6f8", "dr_gcs_private.py": "be1712189580f0cb98c9e182d31c8be8be5c6b7949b5b05788f12f58b89f01c4", "dr_keys_private.py": "b6c32f3ab7e0fc5bd77be86d7ee7f87c6d2553087cc947482f34ad69731ff1e5", "dr_operator_private.py": "237dc3929b136d58081664b9044c7603112d9f31d7d299e283c0959a24ae9d02", "dr_runtime_private.py": "9379ab7e50584f766201dc218f439613477196b313aeaaffbf2bca7c24544ba1", "dr_source_private.py": "d53b1f421f09bfafb96e5d0038c4ef3cabcbf5ac38d8586b0b13b85b1626cc96", "dr_target_private.py": "f606e7d7bf2001db6a27b15fe34c95c3e869500dbe380b6c92f28486a04d1f01", "native_pg_filter.py": "04f7c3d3fd6c518dbfa00902a3f70078d7d1dd66b4b4adf095ed93facfad2a66"}, "wal_barrier_sha256": "8301ef2e538cd31093793a19d8b51c020dfcc463c91d4e4215687ec57a93d24f"}, "kind": "operator_sources_frozen", "previous": "6edc6ac6a6086d4fc220570497fa6d5aef82a1c62d3474465a9370bad639bd9e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 64, "sha256": "853f8f827ba11f322443b503def25847ca67944d2354efb57f32e159a127cb5d"} +{"at_utc": "2026-10-18T14:24:31.431+00:00", "data": {"cohort": "loss", "request_id": "upload-fb2dd8207fa647fa9d80ffdf4c348c68", "sha256": "c1941811f0d2747ce0736b458119484e6973699b0587eecf070d796e7440a328", "size": 1045}, "kind": "sent", "previous": "853f8f827ba11f322443b503def25847ca67944d2354efb57f32e159a127cb5d", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 65, "sha256": "fb500928de99c61b3bc0748ca32ef55a96ea553b0f0e3250ebf9b0f229955e32"} +{"at_utc": "2026-10-18T14:24:39.407+00:00", "data": {"http_status": 201, "ids": {"documentId": 8, "documentVersionId": 8, "embeddingJobId": 6, "fileObjectId": 6}, "outcome": "ACKNOWLEDGED", "request_id": "upload-fb2dd8207fa647fa9d80ffdf4c348c68"}, "kind": "response", "previous": "fb500928de99c61b3bc0748ca32ef55a96ea553b0f0e3250ebf9b0f229955e32", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 66, "sha256": "3dbf292935e14b793c41431fabd3c829fabd36a8769203e6c7d02f3705cb984e"} +{"at_utc": "2026-10-18T14:24:39.409+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.976, "http_status": 201, "ordinal": 6}, "kind": "upload_observed", "previous": "3dbf292935e14b793c41431fabd3c829fabd36a8769203e6c7d02f3705cb984e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 67, "sha256": "10469c874a78c388dba8d6a2ae608ae2651c51f1d6bd599671450056ae62b572"} +{"at_utc": "2026-10-18T14:24:42.431+00:00", "data": {"cohort": "loss", "request_id": "upload-432fd889599b4f88b0d24d6593996a73", "sha256": "ad816c20f4f0bc5be74c279296cedd1d37991dd7f59cc55722ce28e8d92ffc02", "size": 641}, "kind": "sent", "previous": "10469c874a78c388dba8d6a2ae608ae2651c51f1d6bd599671450056ae62b572", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 68, "sha256": "97aa85b169d18a959f1e48fbdd41e87b83a9338faa129ab0093e91eeb7edc4d8"} +{"at_utc": "2026-10-18T14:24:49.127+00:00", "data": {"http_status": 201, "ids": {"documentId": 9, "documentVersionId": 9, "embeddingJobId": 7, "fileObjectId": 7}, "outcome": "ACKNOWLEDGED", "request_id": "upload-432fd889599b4f88b0d24d6593996a73"}, "kind": "response", "previous": "97aa85b169d18a959f1e48fbdd41e87b83a9338faa129ab0093e91eeb7edc4d8", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 69, "sha256": "4eec2f45e9c6f99c5f1d2b9751a9cf7dc42f3b5c0d015aa360d7b57648db098a"} +{"at_utc": "2026-10-18T14:24:49.131+00:00", "data": {"cohort": "loss", "elapsed_seconds": 6.696, "http_status": 201, "ordinal": 7}, "kind": "upload_observed", "previous": "4eec2f45e9c6f99c5f1d2b9751a9cf7dc42f3b5c0d015aa360d7b57648db098a", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 70, "sha256": "9b3c9152b0892b1afa6576a2518faa9555d98de1b0933d3e517285a004457274"} +{"at_utc": "2026-10-18T14:24:53.431+00:00", "data": {"cohort": "loss", "request_id": "upload-4aa4c55b25d14d538fcaff279600fda9", "sha256": "62181f80ab73059b412276fb9bc28765b242dbb26f36ac7dd7ecd2127584e66f", "size": 1045}, "kind": "sent", "previous": "9b3c9152b0892b1afa6576a2518faa9555d98de1b0933d3e517285a004457274", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 71, "sha256": "241ebf582363721a291c782e3d69593a4a968a8b2d5d1335c06178397e7bc843"} +{"at_utc": "2026-10-18T14:25:00.168+00:00", "data": {"http_status": 201, "ids": {"documentId": 10, "documentVersionId": 10, "embeddingJobId": 8, "fileObjectId": 8}, "outcome": "ACKNOWLEDGED", "request_id": "upload-4aa4c55b25d14d538fcaff279600fda9"}, "kind": "response", "previous": "241ebf582363721a291c782e3d69593a4a968a8b2d5d1335c06178397e7bc843", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 72, "sha256": "c0e605cbf76c83ad67bb997754e12edd795fa7731482c09d2f565f9755f47ef1"} +{"at_utc": "2026-10-18T14:25:00.171+00:00", "data": {"cohort": "loss", "elapsed_seconds": 6.736, "http_status": 201, "ordinal": 8}, "kind": "upload_observed", "previous": "c0e605cbf76c83ad67bb997754e12edd795fa7731482c09d2f565f9755f47ef1", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 73, "sha256": "86c7a82060ccb333a6fef9b29a0bf85471c1b78e2714992117bffd76f2198a00"} +{"at_utc": "2026-10-18T14:25:04.431+00:00", "data": {"cohort": "loss", "request_id": "upload-c8826fde38704624ae5ca21b04789532", "sha256": "747645e91e4ee34b21c4e6836d3198433a02a66aa867083becacedbf4971bffb", "size": 641}, "kind": "sent", "previous": "86c7a82060ccb333a6fef9b29a0bf85471c1b78e2714992117bffd76f2198a00", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 74, "sha256": "c82ebfb5df2fc53915f3e393eb79921b8638b0551132492d324b7a607adeac87"} +{"at_utc": "2026-10-18T14:25:11.407+00:00", "data": {"http_status": 201, "ids": {"documentId": 11, "documentVersionId": 11, "embeddingJobId": 9, "fileObjectId": 9}, "outcome": "ACKNOWLEDGED", "request_id": "upload-c8826fde38704624ae5ca21b04789532"}, "kind": "response", "previous": "c82ebfb5df2fc53915f3e393eb79921b8638b0551132492d324b7a607adeac87", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 75, "sha256": "d7a11ae4156afad7067b5a623dc3da1b109c2e0abb587f049bf3842f02feb385"} +{"at_utc": "2026-10-18T14:25:11.411+00:00", "data": {"cohort": "loss", "elapsed_seconds": 6.976, "http_status": 201, "ordinal": 9}, "kind": "upload_observed", "previous": "d7a11ae4156afad7067b5a623dc3da1b109c2e0abb587f049bf3842f02feb385", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 76, "sha256": "5208ea5c7db4d314dac38290368ecc4912d7bf515a4feaacff8e397a74988efe"} +{"at_utc": "2026-10-18T14:25:15.428+00:00", "data": {"cohort": "loss", "request_id": "upload-57d7dce5ad8b4ee2adcf905c04a6f654", "sha256": "cd898e70e7dc88efbfcf5d639615350e27a8de987cfd801addf4eb75f694254b", "size": 1045}, "kind": "sent", "previous": "5208ea5c7db4d314dac38290368ecc4912d7bf515a4feaacff8e397a74988efe", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 77, "sha256": "604a70ecebfec3e3a10ed42fcef7b1db191194545c8b15007b20e00b330ddf42"} +{"at_utc": "2026-10-18T14:25:22.257+00:00", "data": {"http_status": 201, "ids": {"documentId": 12, "documentVersionId": 12, "embeddingJobId": 10, "fileObjectId": 10}, "outcome": "ACKNOWLEDGED", "request_id": "upload-57d7dce5ad8b4ee2adcf905c04a6f654"}, "kind": "response", "previous": "604a70ecebfec3e3a10ed42fcef7b1db191194545c8b15007b20e00b330ddf42", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 78, "sha256": "2233cc7c13906e14057adac4e55e11cbe555ad4f7097d192e629e1718988b03a"} +{"at_utc": "2026-10-18T14:25:22.260+00:00", "data": {"cohort": "loss", "elapsed_seconds": 6.828, "http_status": 201, "ordinal": 10}, "kind": "upload_observed", "previous": "2233cc7c13906e14057adac4e55e11cbe555ad4f7097d192e629e1718988b03a", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 79, "sha256": "e6535f3a1794848f95667174d9272f28076f1c996c24001ac580d1c1f2f57fd3"} +{"at_utc": "2026-10-18T14:25:26.433+00:00", "data": {"cohort": "loss", "request_id": "upload-e4d2871c1e404638bab746d1b07f9797", "sha256": "649e39bc14c365f99405a4293686f5d7abe6c8330e5d6fe4dff58584229baf89", "size": 642}, "kind": "sent", "previous": "e6535f3a1794848f95667174d9272f28076f1c996c24001ac580d1c1f2f57fd3", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 80, "sha256": "762bd533218e51cdc3cc7924b6cc1cfa0f32b4924ad700b94e1e35f4b0127f07"} +{"at_utc": "2026-10-18T14:25:34.475+00:00", "data": {"http_status": 201, "ids": {"documentId": 13, "documentVersionId": 13, "embeddingJobId": 11, "fileObjectId": 11}, "outcome": "ACKNOWLEDGED", "request_id": "upload-e4d2871c1e404638bab746d1b07f9797"}, "kind": "response", "previous": "762bd533218e51cdc3cc7924b6cc1cfa0f32b4924ad700b94e1e35f4b0127f07", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 81, "sha256": "06639a53f5c66ef35e673d5d0a22573cd1f8825c3ce535092fbe47dc97d5f2c3"} +{"at_utc": "2026-10-18T14:25:34.478+00:00", "data": {"cohort": "loss", "elapsed_seconds": 8.041, "http_status": 201, "ordinal": 11}, "kind": "upload_observed", "previous": "06639a53f5c66ef35e673d5d0a22573cd1f8825c3ce535092fbe47dc97d5f2c3", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 82, "sha256": "db8bcd4686798d5947f09edcd79def17f7e38e3b398282cf6dc7c9ee69534524"} +{"at_utc": "2026-10-18T14:25:37.429+00:00", "data": {"cohort": "loss", "request_id": "upload-e584c125eb7e4a5dbe43164f09613254", "sha256": "cfff30ee9b1a0737eca1ed09ab5883404d15d6f51771610af51f51365f3a8ec7", "size": 1045}, "kind": "sent", "previous": "db8bcd4686798d5947f09edcd79def17f7e38e3b398282cf6dc7c9ee69534524", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 83, "sha256": "ed6a34003be74390bbd0acb529ef38f8c82cf283757da45a30d5c30d71af4a98"} +{"at_utc": "2026-10-18T14:25:45.185+00:00", "data": {"http_status": 201, "ids": {"documentId": 14, "documentVersionId": 14, "embeddingJobId": 12, "fileObjectId": 12}, "outcome": "ACKNOWLEDGED", "request_id": "upload-e584c125eb7e4a5dbe43164f09613254"}, "kind": "response", "previous": "ed6a34003be74390bbd0acb529ef38f8c82cf283757da45a30d5c30d71af4a98", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 84, "sha256": "88be4992c5f7464d9d77314ffca7f6dbbc7e24701d08d762db94b08ab7b72f73"} +{"at_utc": "2026-10-18T14:25:45.189+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.757, "http_status": 201, "ordinal": 12}, "kind": "upload_observed", "previous": "88be4992c5f7464d9d77314ffca7f6dbbc7e24701d08d762db94b08ab7b72f73", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 85, "sha256": "0ec531f7883631de852948087fe81f903e1ad747308ad10d06f2903405fcae1c"} +{"at_utc": "2026-10-18T14:25:48.426+00:00", "data": {"cohort": "loss", "request_id": "upload-e97831e4ebdf4f0398b54d8845adcd49", "sha256": "1b330520eed5e3659cec0d138a26c4a94dfeb0dae80e1aeb949d2bdcc96d94f3", "size": 642}, "kind": "sent", "previous": "0ec531f7883631de852948087fe81f903e1ad747308ad10d06f2903405fcae1c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 86, "sha256": "54535699a144ef493ae105a4776af2adce8bb295fc47322bdf2457c4a2838aee"} +{"at_utc": "2026-10-18T14:25:55.924+00:00", "data": {"http_status": 201, "ids": {"documentId": 15, "documentVersionId": 15, "embeddingJobId": 13, "fileObjectId": 13}, "outcome": "ACKNOWLEDGED", "request_id": "upload-e97831e4ebdf4f0398b54d8845adcd49"}, "kind": "response", "previous": "54535699a144ef493ae105a4776af2adce8bb295fc47322bdf2457c4a2838aee", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 87, "sha256": "43c4d2ad2d9a9ffc92c4722dc311c064a89d381c6de4979fca0252ad6d033e9e"} +{"at_utc": "2026-10-18T14:25:55.927+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.497, "http_status": 201, "ordinal": 13}, "kind": "upload_observed", "previous": "43c4d2ad2d9a9ffc92c4722dc311c064a89d381c6de4979fca0252ad6d033e9e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 88, "sha256": "4b44438eb114f9f140011ffa750b1ab407ba9bcccb0778403edfea21a8952b33"} +{"at_utc": "2026-10-18T14:25:59.427+00:00", "data": {"cohort": "loss", "request_id": "upload-e0ed270f022244c19fb49c554ac6ef5b", "sha256": "319a91561e5255e8e4fa5855ecf45ffc10d5d4567605bed159ec892c1f198389", "size": 1046}, "kind": "sent", "previous": "4b44438eb114f9f140011ffa750b1ab407ba9bcccb0778403edfea21a8952b33", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 89, "sha256": "0e1c25f35ded6656f15d48b6391b75e0e8b9d3eeb09dfcc55aa40a6e65fb4b88"} +{"at_utc": "2026-10-18T14:26:06.927+00:00", "data": {"http_status": 201, "ids": {"documentId": 16, "documentVersionId": 16, "embeddingJobId": 14, "fileObjectId": 14}, "outcome": "ACKNOWLEDGED", "request_id": "upload-e0ed270f022244c19fb49c554ac6ef5b"}, "kind": "response", "previous": "0e1c25f35ded6656f15d48b6391b75e0e8b9d3eeb09dfcc55aa40a6e65fb4b88", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 90, "sha256": "e7ef00787ae5b145b64337ed4c7097e2b7e35fd0ce8a703bca595c01270d4436"} +{"at_utc": "2026-10-18T14:26:06.931+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.501, "http_status": 201, "ordinal": 14}, "kind": "upload_observed", "previous": "e7ef00787ae5b145b64337ed4c7097e2b7e35fd0ce8a703bca595c01270d4436", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 91, "sha256": "e997ea657564fe0290ae38215a0ecf772c030912f75df7684b12b51fe30d398f"} +{"at_utc": "2026-10-18T14:26:10.426+00:00", "data": {"cohort": "loss", "request_id": "upload-815f489bc96c48d187abac925f3271f4", "sha256": "cdaba700e161a00efb9f2b5376b7d2a819b4116f746253e55f45a408070a3ec2", "size": 642}, "kind": "sent", "previous": "e997ea657564fe0290ae38215a0ecf772c030912f75df7684b12b51fe30d398f", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 92, "sha256": "f57a28e31f7269de9d3384c5d7cf35485e3950ca7acfbe9ad3558af92d2d32b6"} +{"at_utc": "2026-10-18T14:26:18.268+00:00", "data": {"http_status": 201, "ids": {"documentId": 17, "documentVersionId": 17, "embeddingJobId": 15, "fileObjectId": 15}, "outcome": "ACKNOWLEDGED", "request_id": "upload-815f489bc96c48d187abac925f3271f4"}, "kind": "response", "previous": "f57a28e31f7269de9d3384c5d7cf35485e3950ca7acfbe9ad3558af92d2d32b6", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 93, "sha256": "3209cc2ee510859be5e98271ffb03e4ee3acfa9d68fec21bb372e077751187e5"} +{"at_utc": "2026-10-18T14:26:18.273+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.842, "http_status": 201, "ordinal": 15}, "kind": "upload_observed", "previous": "3209cc2ee510859be5e98271ffb03e4ee3acfa9d68fec21bb372e077751187e5", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 94, "sha256": "77cb017b5b856d6fd90172f116846bad4c057adc8fbce47a9f0a3020b8eead4c"} +{"at_utc": "2026-10-18T14:26:21.430+00:00", "data": {"cohort": "loss", "request_id": "upload-e34da01ee9474549b9002f280a723de1", "sha256": "44ed214162a5fe1fb02221dcc2ec494496c231d35dd6e744fe9ebae8fb0cafe2", "size": 1046}, "kind": "sent", "previous": "77cb017b5b856d6fd90172f116846bad4c057adc8fbce47a9f0a3020b8eead4c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 95, "sha256": "2fb82aae3d023d342548b1e897b263a8279e13c3081fbff19f3fa748963cf302"} +{"at_utc": "2026-10-18T14:26:28.971+00:00", "data": {"http_status": 201, "ids": {"documentId": 18, "documentVersionId": 18, "embeddingJobId": 16, "fileObjectId": 16}, "outcome": "ACKNOWLEDGED", "request_id": "upload-e34da01ee9474549b9002f280a723de1"}, "kind": "response", "previous": "2fb82aae3d023d342548b1e897b263a8279e13c3081fbff19f3fa748963cf302", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 96, "sha256": "b041db1752d28e2a9af065c21974169a48aae01ea142f3e5192a50730b775a84"} +{"at_utc": "2026-10-18T14:26:28.975+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.54, "http_status": 201, "ordinal": 16}, "kind": "upload_observed", "previous": "b041db1752d28e2a9af065c21974169a48aae01ea142f3e5192a50730b775a84", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 97, "sha256": "eda1548876781e1620755de18c6bd25cc38c630594c8d10fdba5c88080233ded"} +{"at_utc": "2026-10-18T14:26:32.426+00:00", "data": {"cohort": "loss", "request_id": "upload-5cd702f34097434ab4544b79573c47fc", "sha256": "21c048da2b95a9077ee98f059ed4fd1dceaca716ed8027465222d18121a36f31", "size": 642}, "kind": "sent", "previous": "eda1548876781e1620755de18c6bd25cc38c630594c8d10fdba5c88080233ded", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 98, "sha256": "a9a15bfa15ed670ee38dbbaec2b155d25110c0ffebf48d1fa9544b2665db4466"} +{"at_utc": "2026-10-18T14:26:39.772+00:00", "data": {"http_status": 201, "ids": {"documentId": 19, "documentVersionId": 19, "embeddingJobId": 17, "fileObjectId": 17}, "outcome": "ACKNOWLEDGED", "request_id": "upload-5cd702f34097434ab4544b79573c47fc"}, "kind": "response", "previous": "a9a15bfa15ed670ee38dbbaec2b155d25110c0ffebf48d1fa9544b2665db4466", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 99, "sha256": "0c9088716ca6e22ca23484c3c077c0f2067349a907aada790672d99be122a59c"} +{"at_utc": "2026-10-18T14:26:39.777+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.346, "http_status": 201, "ordinal": 17}, "kind": "upload_observed", "previous": "0c9088716ca6e22ca23484c3c077c0f2067349a907aada790672d99be122a59c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 100, "sha256": "18e9bb3aa50cc1b852016807860f0dced72b7e5af09d1d3d4e7ab1a68c3dd238"} +{"at_utc": "2026-10-18T14:26:43.427+00:00", "data": {"cohort": "loss", "request_id": "upload-a7c7983377aa4cb38474c2c4b639d200", "sha256": "6321a7fbd1044015e2499bbec3bf05d0194abec00e2cb28bf2109a6c38743a1d", "size": 1046}, "kind": "sent", "previous": "18e9bb3aa50cc1b852016807860f0dced72b7e5af09d1d3d4e7ab1a68c3dd238", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 101, "sha256": "f30a19782078cd68698088db7fb9260871c0a38a629ee15f98f1945aa1fc1c5f"} +{"at_utc": "2026-10-18T14:26:50.450+00:00", "data": {"http_status": 201, "ids": {"documentId": 20, "documentVersionId": 20, "embeddingJobId": 18, "fileObjectId": 18}, "outcome": "ACKNOWLEDGED", "request_id": "upload-a7c7983377aa4cb38474c2c4b639d200"}, "kind": "response", "previous": "f30a19782078cd68698088db7fb9260871c0a38a629ee15f98f1945aa1fc1c5f", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 102, "sha256": "07d1d2990e52b71d74e5885ce5cfdd5f2e09a7b55920f6e9dca4290fde51247c"} +{"at_utc": "2026-10-18T14:26:50.456+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.024, "http_status": 201, "ordinal": 18}, "kind": "upload_observed", "previous": "07d1d2990e52b71d74e5885ce5cfdd5f2e09a7b55920f6e9dca4290fde51247c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 103, "sha256": "290761c5c92911630905a93cc321856268acf2bc4b8160bb5348ebbc8fa94f91"} +{"at_utc": "2026-10-18T14:26:54.427+00:00", "data": {"cohort": "loss", "request_id": "upload-8fe5e921c6a64f2bafb9f5b24e23345d", "sha256": "37f65da35477a4d1ff91d17070ef492d6a890ed606568ca1e9af3d6b11aa3bb2", "size": 642}, "kind": "sent", "previous": "290761c5c92911630905a93cc321856268acf2bc4b8160bb5348ebbc8fa94f91", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 104, "sha256": "ad2aa04a49aaeffe9bb1c91f091f9190decf959ed46f645ab3f8f0676b60371f"} +{"at_utc": "2026-10-18T14:27:02.777+00:00", "data": {"http_status": 201, "ids": {"documentId": 21, "documentVersionId": 21, "embeddingJobId": 19, "fileObjectId": 19}, "outcome": "ACKNOWLEDGED", "request_id": "upload-8fe5e921c6a64f2bafb9f5b24e23345d"}, "kind": "response", "previous": "ad2aa04a49aaeffe9bb1c91f091f9190decf959ed46f645ab3f8f0676b60371f", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 105, "sha256": "ebbb3ade1823ad2e712c77c67875f28850cb7ce481536ea3fee52f1913c266fc"} +{"at_utc": "2026-10-18T14:27:02.778+00:00", "data": {"cohort": "loss", "elapsed_seconds": 8.35, "http_status": 201, "ordinal": 19}, "kind": "upload_observed", "previous": "ebbb3ade1823ad2e712c77c67875f28850cb7ce481536ea3fee52f1913c266fc", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 106, "sha256": "33f9afb67678ece75e5bf1a8862792d237980e3c8561bb75bc31ea41e0246163"} +{"at_utc": "2026-10-18T14:27:05.429+00:00", "data": {"cohort": "loss", "request_id": "upload-a7fbd0d4358842d99a18392ea28d5ada", "sha256": "ec2fcb418c91cb8b8d0bf78186efc6f99eefea1efe5c2544bc43460ea24b6959", "size": 1045}, "kind": "sent", "previous": "33f9afb67678ece75e5bf1a8862792d237980e3c8561bb75bc31ea41e0246163", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 107, "sha256": "d279e9ca01c61682bf0c9bbf399658ea53281cd31922b3aeb3ed5a7673a26461"} +{"at_utc": "2026-10-18T14:27:12.730+00:00", "data": {"http_status": 201, "ids": {"documentId": 22, "documentVersionId": 22, "embeddingJobId": 20, "fileObjectId": 20}, "outcome": "ACKNOWLEDGED", "request_id": "upload-a7fbd0d4358842d99a18392ea28d5ada"}, "kind": "response", "previous": "d279e9ca01c61682bf0c9bbf399658ea53281cd31922b3aeb3ed5a7673a26461", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 108, "sha256": "28747eb9e1b412e65eb98bdaa413fa1e56ecca90a07f261385d178a1881d1135"} +{"at_utc": "2026-10-18T14:27:12.735+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.301, "http_status": 201, "ordinal": 20}, "kind": "upload_observed", "previous": "28747eb9e1b412e65eb98bdaa413fa1e56ecca90a07f261385d178a1881d1135", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 109, "sha256": "f1950794ea0a846b5a63216bb6e45aea0e37f63d975f06916756c44a497ef8dd"} +{"at_utc": "2026-10-18T14:27:16.430+00:00", "data": {"cohort": "loss", "request_id": "upload-52d582fc9eb64427845662582d8a3bbb", "sha256": "f50e89d1b76a00f28d1051cb44d1b786d9922a56047c9e2bbab7b2f649208c0d", "size": 642}, "kind": "sent", "previous": "f1950794ea0a846b5a63216bb6e45aea0e37f63d975f06916756c44a497ef8dd", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 110, "sha256": "2a8edb42b5e6abb4327e2fa40752c4f1a3366a75a8c31f1e81e0495ca1f9f36e"} +{"at_utc": "2026-10-18T14:27:25.101+00:00", "data": {"http_status": 201, "ids": {"documentId": 23, "documentVersionId": 23, "embeddingJobId": 21, "fileObjectId": 21}, "outcome": "ACKNOWLEDGED", "request_id": "upload-52d582fc9eb64427845662582d8a3bbb"}, "kind": "response", "previous": "2a8edb42b5e6abb4327e2fa40752c4f1a3366a75a8c31f1e81e0495ca1f9f36e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 111, "sha256": "8faf1d2d63a15c66cdb9f88608a5d004245ea4fc0d6e4d509c55a52de6c1e206"} +{"at_utc": "2026-10-18T14:27:25.105+00:00", "data": {"cohort": "loss", "elapsed_seconds": 8.67, "http_status": 201, "ordinal": 21}, "kind": "upload_observed", "previous": "8faf1d2d63a15c66cdb9f88608a5d004245ea4fc0d6e4d509c55a52de6c1e206", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 112, "sha256": "e8f506dddd5cceee06c09d5d5c2d6ef45ade859aede25041b4640d1410c68972"} +{"at_utc": "2026-10-18T14:27:27.425+00:00", "data": {"cohort": "loss", "request_id": "upload-5cbd3325ea7e4b97b4db3ade4fc2fa7a", "sha256": "538cfe4f5a5ea844e945afef4d23674988270f0aa1f52f27730f98697b7f5a18", "size": 1045}, "kind": "sent", "previous": "e8f506dddd5cceee06c09d5d5c2d6ef45ade859aede25041b4640d1410c68972", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 113, "sha256": "e23bb2aa61e94f767bbd7afa08e1b1b7873c8e620863cd16e40c1049b3f2dd26"} +{"at_utc": "2026-10-18T14:27:35.059+00:00", "data": {"http_status": 201, "ids": {"documentId": 24, "documentVersionId": 24, "embeddingJobId": 22, "fileObjectId": 22}, "outcome": "ACKNOWLEDGED", "request_id": "upload-5cbd3325ea7e4b97b4db3ade4fc2fa7a"}, "kind": "response", "previous": "e23bb2aa61e94f767bbd7afa08e1b1b7873c8e620863cd16e40c1049b3f2dd26", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 114, "sha256": "c692a344d5a005ec21176df209a215f9ca63f007f66bf00c55d8778e1b7ed340"} +{"at_utc": "2026-10-18T14:27:35.064+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.634, "http_status": 201, "ordinal": 22}, "kind": "upload_observed", "previous": "c692a344d5a005ec21176df209a215f9ca63f007f66bf00c55d8778e1b7ed340", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 115, "sha256": "d6e4475f6bdaa5da6ccc73ef27be425917e99f407cb17e1dd7ee394804daf71b"} +{"at_utc": "2026-10-18T14:27:38.427+00:00", "data": {"cohort": "loss", "request_id": "upload-8af4ad876cd043f0a906915c5b5c9e76", "sha256": "125ad2242835634e68f4c82efa85888600dc6f97bd3474765f658133e8c75a52", "size": 642}, "kind": "sent", "previous": "d6e4475f6bdaa5da6ccc73ef27be425917e99f407cb17e1dd7ee394804daf71b", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 116, "sha256": "ec21e51c0fd5c8884578e447bf3df4e66725a4112069bc8da22879ee8611fbbe"} +{"at_utc": "2026-10-18T14:27:45.864+00:00", "data": {"http_status": 201, "ids": {"documentId": 25, "documentVersionId": 25, "embeddingJobId": 23, "fileObjectId": 23}, "outcome": "ACKNOWLEDGED", "request_id": "upload-8af4ad876cd043f0a906915c5b5c9e76"}, "kind": "response", "previous": "ec21e51c0fd5c8884578e447bf3df4e66725a4112069bc8da22879ee8611fbbe", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 117, "sha256": "2842b404fa8787fe7b3d4dc45e694c92be1930f78bbf57469c74e57c8b6ced4a"} +{"at_utc": "2026-10-18T14:27:45.868+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.437, "http_status": 201, "ordinal": 23}, "kind": "upload_observed", "previous": "2842b404fa8787fe7b3d4dc45e694c92be1930f78bbf57469c74e57c8b6ced4a", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 118, "sha256": "b92aede260952e27438fb0765233d5eae9bb41de3144581c5032f7b39f33f301"} +{"at_utc": "2026-10-18T14:27:49.429+00:00", "data": {"cohort": "loss", "request_id": "upload-a2a79e66098444ce812e0e7f2b55741c", "sha256": "4dd86e6bb9e74cf9b403e286bf0fe3321d3bc7bc13447e61d70bb72246e1ce8f", "size": 1046}, "kind": "sent", "previous": "b92aede260952e27438fb0765233d5eae9bb41de3144581c5032f7b39f33f301", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 119, "sha256": "0b43df2d1ca45d40f7d354aff3aa3ed5354d2137e18ec252326afe8b7bf47449"} +{"at_utc": "2026-10-18T14:27:57.382+00:00", "data": {"http_status": 201, "ids": {"documentId": 26, "documentVersionId": 26, "embeddingJobId": 24, "fileObjectId": 24}, "outcome": "ACKNOWLEDGED", "request_id": "upload-a2a79e66098444ce812e0e7f2b55741c"}, "kind": "response", "previous": "0b43df2d1ca45d40f7d354aff3aa3ed5354d2137e18ec252326afe8b7bf47449", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 120, "sha256": "23042500ccf33c928a00ba3a4d02cd18bf9b7d8de8e41fc533a7feeae92c87de"} +{"at_utc": "2026-10-18T14:27:57.386+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.952, "http_status": 201, "ordinal": 24}, "kind": "upload_observed", "previous": "23042500ccf33c928a00ba3a4d02cd18bf9b7d8de8e41fc533a7feeae92c87de", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 121, "sha256": "dc3c56298240e5d8449cf11e99b5cf22e418b769e53c8eda407a1a3ed3c947c3"} +{"at_utc": "2026-10-18T14:28:00.425+00:00", "data": {"cohort": "loss", "request_id": "upload-f35ec7a4a77b466186a1bb8faa995b8f", "sha256": "c24c6c54039e0d1a121cd0d9702a525bc1c996c6409ed1049b0e36b658087e7a", "size": 642}, "kind": "sent", "previous": "dc3c56298240e5d8449cf11e99b5cf22e418b769e53c8eda407a1a3ed3c947c3", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 122, "sha256": "1c7466e4cdc365449887493f0e6bdd16839e2e974e234de94c6490828fa00ccf"} +{"at_utc": "2026-10-18T14:28:07.877+00:00", "data": {"http_status": 201, "ids": {"documentId": 27, "documentVersionId": 27, "embeddingJobId": 25, "fileObjectId": 25}, "outcome": "ACKNOWLEDGED", "request_id": "upload-f35ec7a4a77b466186a1bb8faa995b8f"}, "kind": "response", "previous": "1c7466e4cdc365449887493f0e6bdd16839e2e974e234de94c6490828fa00ccf", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 123, "sha256": "977c54901aeb8b73a4a94abe53c475a5fdbfec5df67ca1fd9f07f6e94ac5a746"} +{"at_utc": "2026-10-18T14:28:07.886+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.453, "http_status": 201, "ordinal": 25}, "kind": "upload_observed", "previous": "977c54901aeb8b73a4a94abe53c475a5fdbfec5df67ca1fd9f07f6e94ac5a746", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 124, "sha256": "6bf8c63231ce4809ac7cd85c0f65de3e71a32cb96fc82ae301ea383d9701c2fe"} +{"at_utc": "2026-10-18T14:28:11.428+00:00", "data": {"cohort": "loss", "request_id": "upload-7b83a8a9b01544af88bbec7e9e5a8b94", "sha256": "dd4d1b12b85e80e7f165418965bfb590546c3d267df5ea6974ffea1ece4ee92f", "size": 1046}, "kind": "sent", "previous": "6bf8c63231ce4809ac7cd85c0f65de3e71a32cb96fc82ae301ea383d9701c2fe", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 125, "sha256": "1e8b84116d85c8c7f659bd8f39f78cd995d4890ef931e526c6d9fcf8aa22bbfa"} +{"at_utc": "2026-10-18T14:28:18.857+00:00", "data": {"http_status": 201, "ids": {"documentId": 28, "documentVersionId": 28, "embeddingJobId": 26, "fileObjectId": 26}, "outcome": "ACKNOWLEDGED", "request_id": "upload-7b83a8a9b01544af88bbec7e9e5a8b94"}, "kind": "response", "previous": "1e8b84116d85c8c7f659bd8f39f78cd995d4890ef931e526c6d9fcf8aa22bbfa", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 126, "sha256": "4fe0d422aa10615afdd46c6700ab41a59a5e398404160a85c30e046e33fb476b"} +{"at_utc": "2026-10-18T14:28:18.861+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.429, "http_status": 201, "ordinal": 26}, "kind": "upload_observed", "previous": "4fe0d422aa10615afdd46c6700ab41a59a5e398404160a85c30e046e33fb476b", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 127, "sha256": "8968724cdb79d433fd7cbbd16cad3f58f27bc4fa51e6b1990355877eec2b3320"} +{"at_utc": "2026-10-18T14:28:22.427+00:00", "data": {"cohort": "loss", "request_id": "upload-106767bf29ce4751a114beffed1dcf5d", "sha256": "7199843049649f2d55f8444f8fd795d2f9620569e4d6752e42e3806c6e10af67", "size": 642}, "kind": "sent", "previous": "8968724cdb79d433fd7cbbd16cad3f58f27bc4fa51e6b1990355877eec2b3320", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 128, "sha256": "f23b0483db484f626b3364d39e982e7aff77c9a7f5167af28492b377403793f7"} +{"at_utc": "2026-10-18T14:28:30.164+00:00", "data": {"http_status": 201, "ids": {"documentId": 29, "documentVersionId": 29, "embeddingJobId": 27, "fileObjectId": 27}, "outcome": "ACKNOWLEDGED", "request_id": "upload-106767bf29ce4751a114beffed1dcf5d"}, "kind": "response", "previous": "f23b0483db484f626b3364d39e982e7aff77c9a7f5167af28492b377403793f7", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 129, "sha256": "7dbedcd3775656e5b3357f58c750dba0cb048b2c6ab8efea1a8771f9f61fb42e"} +{"at_utc": "2026-10-18T14:28:30.169+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.737, "http_status": 201, "ordinal": 27}, "kind": "upload_observed", "previous": "7dbedcd3775656e5b3357f58c750dba0cb048b2c6ab8efea1a8771f9f61fb42e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 130, "sha256": "27858f84f19993176229ceadd55d16ca2489783ae39c57d3c25683465d5ebff3"} +{"at_utc": "2026-10-18T14:28:33.422+00:00", "data": {"cohort": "loss", "request_id": "upload-c2ec085d264a49c9ab6f691c6ae4aa9b", "sha256": "6f2ab9f8bb530c07b539c787bc926c0161cb7136ee1fab6383d272861133f7d6", "size": 1046}, "kind": "sent", "previous": "27858f84f19993176229ceadd55d16ca2489783ae39c57d3c25683465d5ebff3", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 131, "sha256": "707f6d65aedfb4e926703c0cbec8963c3b1067e69ed9547562c9c78975bef634"} +{"at_utc": "2026-10-18T14:28:40.871+00:00", "data": {"http_status": 201, "ids": {"documentId": 30, "documentVersionId": 30, "embeddingJobId": 28, "fileObjectId": 28}, "outcome": "ACKNOWLEDGED", "request_id": "upload-c2ec085d264a49c9ab6f691c6ae4aa9b"}, "kind": "response", "previous": "707f6d65aedfb4e926703c0cbec8963c3b1067e69ed9547562c9c78975bef634", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 132, "sha256": "bbff85fff63f2715585fcfb6012b448648f28966c8a4e75d4c3a9403d9277aae"} +{"at_utc": "2026-10-18T14:28:40.876+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.449, "http_status": 201, "ordinal": 28}, "kind": "upload_observed", "previous": "bbff85fff63f2715585fcfb6012b448648f28966c8a4e75d4c3a9403d9277aae", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 133, "sha256": "bc9e3a00f8614e92cbacf20863434b5dee119a3d482c39f59c9fe9e39fbe1836"} +{"at_utc": "2026-10-18T14:28:44.426+00:00", "data": {"cohort": "loss", "request_id": "upload-cef30e0526c24946b07bc80888bd0290", "sha256": "1122671a8947a18acc9b49d318e573d3d4f64299328a9a25a4b3103860590dc7", "size": 642}, "kind": "sent", "previous": "bc9e3a00f8614e92cbacf20863434b5dee119a3d482c39f59c9fe9e39fbe1836", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 134, "sha256": "a58e4a1c9b1e59589fee704a9a2198578437024f3d0c365a7242dcb78e8e7a22"} +{"at_utc": "2026-10-18T14:28:51.926+00:00", "data": {"http_status": 201, "ids": {"documentId": 31, "documentVersionId": 31, "embeddingJobId": 29, "fileObjectId": 29}, "outcome": "ACKNOWLEDGED", "request_id": "upload-cef30e0526c24946b07bc80888bd0290"}, "kind": "response", "previous": "a58e4a1c9b1e59589fee704a9a2198578437024f3d0c365a7242dcb78e8e7a22", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 135, "sha256": "fd8e9f39282d9bc3474e6288724a1680112971baf213e5743d03d22a44672445"} +{"at_utc": "2026-10-18T14:28:51.931+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.5, "http_status": 201, "ordinal": 29}, "kind": "upload_observed", "previous": "fd8e9f39282d9bc3474e6288724a1680112971baf213e5743d03d22a44672445", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 136, "sha256": "1128cac6e74f797f4e86214b28ebf0c3f998c2cadb7e5f7739248ab206b54dbe"} +{"at_utc": "2026-10-18T14:28:55.427+00:00", "data": {"cohort": "loss", "request_id": "upload-496f0a710ecb4981a3a644fdea27a2a5", "sha256": "5f82ef3f5fc831e1b38624f982436ebd6dd233cf9b36e0af9c12b5d7a42f8ce6", "size": 1046}, "kind": "sent", "previous": "1128cac6e74f797f4e86214b28ebf0c3f998c2cadb7e5f7739248ab206b54dbe", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 137, "sha256": "f7d69e3af10fc34e90741b3837397aee5ada7af2a79de5ad02407a80b130ad28"} +{"at_utc": "2026-10-18T14:29:03.139+00:00", "data": {"http_status": 201, "ids": {"documentId": 32, "documentVersionId": 32, "embeddingJobId": 30, "fileObjectId": 30}, "outcome": "ACKNOWLEDGED", "request_id": "upload-496f0a710ecb4981a3a644fdea27a2a5"}, "kind": "response", "previous": "f7d69e3af10fc34e90741b3837397aee5ada7af2a79de5ad02407a80b130ad28", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 138, "sha256": "16cba1fdee212999681757ddfec38c18fd2a0ecb769d6b6b8df0ed108560ec6c"} +{"at_utc": "2026-10-18T14:29:03.144+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.712, "http_status": 201, "ordinal": 30}, "kind": "upload_observed", "previous": "16cba1fdee212999681757ddfec38c18fd2a0ecb769d6b6b8df0ed108560ec6c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 139, "sha256": "478be77691fc16bd8fbe1aed95e3608465742f59535db6ed9edf8484062460a4"} +{"at_utc": "2026-10-18T14:29:06.423+00:00", "data": {"cohort": "loss", "request_id": "upload-3f79f77ddbd24cf1a9f682abcf928cb6", "sha256": "f904af9279ff864430761562c9f2c207dd1539c18c91098a2a43fe3dd484806c", "size": 642}, "kind": "sent", "previous": "478be77691fc16bd8fbe1aed95e3608465742f59535db6ed9edf8484062460a4", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 140, "sha256": "883a6a7ef8c8432a0bf53d9efc89851a21822eec9908cb3bb835f5593f9e742f"} +{"at_utc": "2026-10-18T14:29:13.966+00:00", "data": {"http_status": 201, "ids": {"documentId": 33, "documentVersionId": 33, "embeddingJobId": 31, "fileObjectId": 31}, "outcome": "ACKNOWLEDGED", "request_id": "upload-3f79f77ddbd24cf1a9f682abcf928cb6"}, "kind": "response", "previous": "883a6a7ef8c8432a0bf53d9efc89851a21822eec9908cb3bb835f5593f9e742f", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 141, "sha256": "58d4366ac43c9437d66eefd5bf09dea975636fb9f3872eb0759f1affd236e363"} +{"at_utc": "2026-10-18T14:29:13.971+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.543, "http_status": 201, "ordinal": 31}, "kind": "upload_observed", "previous": "58d4366ac43c9437d66eefd5bf09dea975636fb9f3872eb0759f1affd236e363", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 142, "sha256": "242c78e11e0a83e8b6fea6f975dd58b62f7ce347bcc085f886a0ee7a0a63c8f4"} +{"at_utc": "2026-10-18T14:29:17.422+00:00", "data": {"cohort": "loss", "request_id": "upload-63141b14d77e4900965f43624358e6ce", "sha256": "8c4163e1723cf99bada4fccc1956b24b36aeb3ff9aaa2b18a1d5a4355c42a0d3", "size": 1045}, "kind": "sent", "previous": "242c78e11e0a83e8b6fea6f975dd58b62f7ce347bcc085f886a0ee7a0a63c8f4", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 143, "sha256": "777dc185330fc20d69e8c1a7ad86bd0641582a6cff4051cb3902ade2d587efee"} +{"at_utc": "2026-10-18T14:29:25.157+00:00", "data": {"http_status": 201, "ids": {"documentId": 34, "documentVersionId": 34, "embeddingJobId": 32, "fileObjectId": 32}, "outcome": "ACKNOWLEDGED", "request_id": "upload-63141b14d77e4900965f43624358e6ce"}, "kind": "response", "previous": "777dc185330fc20d69e8c1a7ad86bd0641582a6cff4051cb3902ade2d587efee", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 144, "sha256": "57cce220c0090f6687073bf0f53452c6a52d93e98ac7f08eb9f23dae912e4b11"} +{"at_utc": "2026-10-18T14:29:25.163+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.736, "http_status": 201, "ordinal": 32}, "kind": "upload_observed", "previous": "57cce220c0090f6687073bf0f53452c6a52d93e98ac7f08eb9f23dae912e4b11", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 145, "sha256": "f439f51ba20a0ab12fa6a80d6b066596554c78635d9020ea060eff6e568af92d"} +{"at_utc": "2026-10-18T14:29:28.433+00:00", "data": {"cohort": "loss", "request_id": "upload-52a198f8bd8a4c1f832134222e5505c5", "sha256": "296abbc16a4f567ea8197c9a138b973da051803e082e5b52d627f9b023453dc0", "size": 642}, "kind": "sent", "previous": "f439f51ba20a0ab12fa6a80d6b066596554c78635d9020ea060eff6e568af92d", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 146, "sha256": "aa8ec8a09d464857b671ca423a09fb15ddd8de5f94ff542426779a4e506e8422"} +{"at_utc": "2026-10-18T14:29:36.265+00:00", "data": {"http_status": 201, "ids": {"documentId": 35, "documentVersionId": 35, "embeddingJobId": 33, "fileObjectId": 33}, "outcome": "ACKNOWLEDGED", "request_id": "upload-52a198f8bd8a4c1f832134222e5505c5"}, "kind": "response", "previous": "aa8ec8a09d464857b671ca423a09fb15ddd8de5f94ff542426779a4e506e8422", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 147, "sha256": "b4a98dc5bfe9ba453d919776ffd8fefad3544e9ef3a417340cd5a14e2898b83f"} +{"at_utc": "2026-10-18T14:29:36.271+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.832, "http_status": 201, "ordinal": 33}, "kind": "upload_observed", "previous": "b4a98dc5bfe9ba453d919776ffd8fefad3544e9ef3a417340cd5a14e2898b83f", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 148, "sha256": "f676a0a42f14c10f8a17a56ed0073d40f7add1b7fd37101bcb1ed8b709606dc0"} +{"at_utc": "2026-10-18T14:29:36.275+00:00", "data": {"completed_upload_requests": 33, "ordinal": 1, "scheduled_at_utc": "2026-10-18T14:29:31.420159+00:00", "start_lag_seconds": 4.86}, "kind": "offsite_schedule_observed", "previous": "f676a0a42f14c10f8a17a56ed0073d40f7add1b7fd37101bcb1ed8b709606dc0", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 149, "sha256": "97acca8923fd248a8d9d154c2ceabeadc4429e5184a43e7c2ea8690bcf7dade5"} +{"at_utc": "2026-10-18T14:30:01.784+00:00", "data": {"fileObjectId": 6, "generation": "1792333801474838", "sha256": "c1941811f0d2747ce0736b458119484e6973699b0587eecf070d796e7440a328", "size": 1045}, "kind": "original_preserved", "previous": "97acca8923fd248a8d9d154c2ceabeadc4429e5184a43e7c2ea8690bcf7dade5", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 150, "sha256": "ac6d4733b7bda5e4eec04799ee0ceb29d4066017073cf3ce73fddc7d8ea5128d"} +{"at_utc": "2026-10-18T14:30:03.121+00:00", "data": {"fileObjectId": 7, "generation": "1792333802798444", "sha256": "ad816c20f4f0bc5be74c279296cedd1d37991dd7f59cc55722ce28e8d92ffc02", "size": 641}, "kind": "original_preserved", "previous": "ac6d4733b7bda5e4eec04799ee0ceb29d4066017073cf3ce73fddc7d8ea5128d", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 151, "sha256": "c5103fd41d76011d323fe5881299f08ee35e52e959eb2a6c22eee1d83f7c0b91"} +{"at_utc": "2026-10-18T14:30:04.419+00:00", "data": {"fileObjectId": 8, "generation": "1792333804125436", "sha256": "62181f80ab73059b412276fb9bc28765b242dbb26f36ac7dd7ecd2127584e66f", "size": 1045}, "kind": "original_preserved", "previous": "c5103fd41d76011d323fe5881299f08ee35e52e959eb2a6c22eee1d83f7c0b91", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 152, "sha256": "4e1eaac4d041f242a26fc60efcdb0c30650f189c74c34b407e26e9795e4de222"} +{"at_utc": "2026-10-18T14:30:05.788+00:00", "data": {"fileObjectId": 9, "generation": "1792333805402152", "sha256": "747645e91e4ee34b21c4e6836d3198433a02a66aa867083becacedbf4971bffb", "size": 641}, "kind": "original_preserved", "previous": "4e1eaac4d041f242a26fc60efcdb0c30650f189c74c34b407e26e9795e4de222", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 153, "sha256": "ca06deb7a7a9945d6a31f45e62d32f7779c5bdb3a3544c8d036f4925592f0b7c"} +{"at_utc": "2026-10-18T14:30:07.050+00:00", "data": {"fileObjectId": 10, "generation": "1792333806788228", "sha256": "cd898e70e7dc88efbfcf5d639615350e27a8de987cfd801addf4eb75f694254b", "size": 1045}, "kind": "original_preserved", "previous": "ca06deb7a7a9945d6a31f45e62d32f7779c5bdb3a3544c8d036f4925592f0b7c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 154, "sha256": "e9cecca7e7cd61e78c3510ebda8c15dc04fc7d39fbdc274d9b8c28555ea8a2e3"} +{"at_utc": "2026-10-18T14:30:08.247+00:00", "data": {"fileObjectId": 11, "generation": "1792333808038458", "sha256": "649e39bc14c365f99405a4293686f5d7abe6c8330e5d6fe4dff58584229baf89", "size": 642}, "kind": "original_preserved", "previous": "e9cecca7e7cd61e78c3510ebda8c15dc04fc7d39fbdc274d9b8c28555ea8a2e3", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 155, "sha256": "d741a5fdb73c9d2c574bb30433e40b60479b3c83b61d77c47b530b7cb6a7c8d3"} +{"at_utc": "2026-10-18T14:30:09.625+00:00", "data": {"fileObjectId": 12, "generation": "1792333809341078", "sha256": "cfff30ee9b1a0737eca1ed09ab5883404d15d6f51771610af51f51365f3a8ec7", "size": 1045}, "kind": "original_preserved", "previous": "d741a5fdb73c9d2c574bb30433e40b60479b3c83b61d77c47b530b7cb6a7c8d3", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 156, "sha256": "b30922555f4451148eb1ea3586269af890da030ebf7a4fa9d7f7633d0c179f33"} +{"at_utc": "2026-10-18T14:30:10.847+00:00", "data": {"fileObjectId": 13, "generation": "1792333810643638", "sha256": "1b330520eed5e3659cec0d138a26c4a94dfeb0dae80e1aeb949d2bdcc96d94f3", "size": 642}, "kind": "original_preserved", "previous": "b30922555f4451148eb1ea3586269af890da030ebf7a4fa9d7f7633d0c179f33", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 157, "sha256": "3cee7953a73039d2afad94b4ef82cc9a7715dd5594c45020f321508a32f4698c"} +{"at_utc": "2026-10-18T14:30:12.114+00:00", "data": {"fileObjectId": 14, "generation": "1792333811904377", "sha256": "319a91561e5255e8e4fa5855ecf45ffc10d5d4567605bed159ec892c1f198389", "size": 1046}, "kind": "original_preserved", "previous": "3cee7953a73039d2afad94b4ef82cc9a7715dd5594c45020f321508a32f4698c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 158, "sha256": "2ef3dc104d7b91efd815263d373f59cac223b06365621a5885ff4eec1ea553bf"} +{"at_utc": "2026-10-18T14:30:13.405+00:00", "data": {"fileObjectId": 15, "generation": "1792333813202648", "sha256": "cdaba700e161a00efb9f2b5376b7d2a819b4116f746253e55f45a408070a3ec2", "size": 642}, "kind": "original_preserved", "previous": "2ef3dc104d7b91efd815263d373f59cac223b06365621a5885ff4eec1ea553bf", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 159, "sha256": "228159abc1e8b12582ea7dc9803dceab8a816411151047b346d56af1806efabf"} +{"at_utc": "2026-10-18T14:30:14.854+00:00", "data": {"fileObjectId": 16, "generation": "1792333814551128", "sha256": "44ed214162a5fe1fb02221dcc2ec494496c231d35dd6e744fe9ebae8fb0cafe2", "size": 1046}, "kind": "original_preserved", "previous": "228159abc1e8b12582ea7dc9803dceab8a816411151047b346d56af1806efabf", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 160, "sha256": "7ce816a011e5683946f10bbadf0ed4ad2e90e0803ea372bf454a5c60a1ce197a"} +{"at_utc": "2026-10-18T14:30:16.084+00:00", "data": {"fileObjectId": 17, "generation": "1792333815872591", "sha256": "21c048da2b95a9077ee98f059ed4fd1dceaca716ed8027465222d18121a36f31", "size": 642}, "kind": "original_preserved", "previous": "7ce816a011e5683946f10bbadf0ed4ad2e90e0803ea372bf454a5c60a1ce197a", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 161, "sha256": "b0e27c136c84f1ce389f52ba4ab6762a025a088515eb3d939ce6e9be657adc0b"} +{"at_utc": "2026-10-18T14:30:17.382+00:00", "data": {"fileObjectId": 18, "generation": "1792333817160338", "sha256": "6321a7fbd1044015e2499bbec3bf05d0194abec00e2cb28bf2109a6c38743a1d", "size": 1046}, "kind": "original_preserved", "previous": "b0e27c136c84f1ce389f52ba4ab6762a025a088515eb3d939ce6e9be657adc0b", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 162, "sha256": "855d8bd9d26d22402ebe731caff6aad9375dc320413a7873abc7b6ec4c5126c2"} +{"at_utc": "2026-10-18T14:30:18.640+00:00", "data": {"fileObjectId": 19, "generation": "1792333818431760", "sha256": "37f65da35477a4d1ff91d17070ef492d6a890ed606568ca1e9af3d6b11aa3bb2", "size": 642}, "kind": "original_preserved", "previous": "855d8bd9d26d22402ebe731caff6aad9375dc320413a7873abc7b6ec4c5126c2", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 163, "sha256": "0caf0e71dd21f925e70237d148c9b5b1bbdc814988d7330d1f63c4ad826601bb"} +{"at_utc": "2026-10-18T14:30:20.211+00:00", "data": {"fileObjectId": 20, "generation": "1792333819804887", "sha256": "ec2fcb418c91cb8b8d0bf78186efc6f99eefea1efe5c2544bc43460ea24b6959", "size": 1045}, "kind": "original_preserved", "previous": "0caf0e71dd21f925e70237d148c9b5b1bbdc814988d7330d1f63c4ad826601bb", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 164, "sha256": "8013ed9569625175bd6c7e2ac4abe4f7cec3ba9ae21e70f9049c1408e1289b5f"} +{"at_utc": "2026-10-18T14:30:21.668+00:00", "data": {"fileObjectId": 21, "generation": "1792333821311606", "sha256": "f50e89d1b76a00f28d1051cb44d1b786d9922a56047c9e2bbab7b2f649208c0d", "size": 642}, "kind": "original_preserved", "previous": "8013ed9569625175bd6c7e2ac4abe4f7cec3ba9ae21e70f9049c1408e1289b5f", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 165, "sha256": "6ae4bef08db9ba66eadeb80dd7f933b74413271daaddff2daf3fca74d49a7971"} +{"at_utc": "2026-10-18T14:30:22.919+00:00", "data": {"fileObjectId": 22, "generation": "1792333822701925", "sha256": "538cfe4f5a5ea844e945afef4d23674988270f0aa1f52f27730f98697b7f5a18", "size": 1045}, "kind": "original_preserved", "previous": "6ae4bef08db9ba66eadeb80dd7f933b74413271daaddff2daf3fca74d49a7971", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 166, "sha256": "009af6e97d56f97615c9d16cbe83d55c866a02d8e5460a9ef2d0e87d515d5bea"} +{"at_utc": "2026-10-18T14:30:24.171+00:00", "data": {"fileObjectId": 23, "generation": "1792333823957315", "sha256": "125ad2242835634e68f4c82efa85888600dc6f97bd3474765f658133e8c75a52", "size": 642}, "kind": "original_preserved", "previous": "009af6e97d56f97615c9d16cbe83d55c866a02d8e5460a9ef2d0e87d515d5bea", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 167, "sha256": "362d13e9a27821631ac0a356304d5236b9e81b9e162011a25c5cd2eae1b58ea6"} +{"at_utc": "2026-10-18T14:30:25.443+00:00", "data": {"fileObjectId": 24, "generation": "1792333825235999", "sha256": "4dd86e6bb9e74cf9b403e286bf0fe3321d3bc7bc13447e61d70bb72246e1ce8f", "size": 1046}, "kind": "original_preserved", "previous": "362d13e9a27821631ac0a356304d5236b9e81b9e162011a25c5cd2eae1b58ea6", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 168, "sha256": "68d10ce03fdad8f2d1f64f251f3c1e91ecaea6799487424d955559b1e2e1871e"} +{"at_utc": "2026-10-18T14:30:26.760+00:00", "data": {"fileObjectId": 25, "generation": "1792333826546721", "sha256": "c24c6c54039e0d1a121cd0d9702a525bc1c996c6409ed1049b0e36b658087e7a", "size": 642}, "kind": "original_preserved", "previous": "68d10ce03fdad8f2d1f64f251f3c1e91ecaea6799487424d955559b1e2e1871e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 169, "sha256": "d2f22abf659b3b84824b5bd0059bcae58e8e52919228f6bb196b211c8e532bb7"} +{"at_utc": "2026-10-18T14:30:28.070+00:00", "data": {"fileObjectId": 26, "generation": "1792333827749600", "sha256": "dd4d1b12b85e80e7f165418965bfb590546c3d267df5ea6974ffea1ece4ee92f", "size": 1046}, "kind": "original_preserved", "previous": "d2f22abf659b3b84824b5bd0059bcae58e8e52919228f6bb196b211c8e532bb7", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 170, "sha256": "19a538cfec709476a46270053b43966a007b8a744d52937af04679f3d3c71521"} +{"at_utc": "2026-10-18T14:30:29.321+00:00", "data": {"fileObjectId": 27, "generation": "1792333829026930", "sha256": "7199843049649f2d55f8444f8fd795d2f9620569e4d6752e42e3806c6e10af67", "size": 642}, "kind": "original_preserved", "previous": "19a538cfec709476a46270053b43966a007b8a744d52937af04679f3d3c71521", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 171, "sha256": "2fc05dc6cdb86b1e6ad7188ec8667c7bd2bd1ed16d46f75ce8729d0c66edb675"} +{"at_utc": "2026-10-18T14:30:30.600+00:00", "data": {"fileObjectId": 28, "generation": "1792333830306441", "sha256": "6f2ab9f8bb530c07b539c787bc926c0161cb7136ee1fab6383d272861133f7d6", "size": 1046}, "kind": "original_preserved", "previous": "2fc05dc6cdb86b1e6ad7188ec8667c7bd2bd1ed16d46f75ce8729d0c66edb675", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 172, "sha256": "b4197ca4ce80d88902dc7bd3c72d1a3c6e8d37527aa206ad3df5ff2dacef8655"} +{"at_utc": "2026-10-18T14:30:31.800+00:00", "data": {"fileObjectId": 29, "generation": "1792333831574400", "sha256": "1122671a8947a18acc9b49d318e573d3d4f64299328a9a25a4b3103860590dc7", "size": 642}, "kind": "original_preserved", "previous": "b4197ca4ce80d88902dc7bd3c72d1a3c6e8d37527aa206ad3df5ff2dacef8655", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 173, "sha256": "9ac7fa7b25f3473542113457b00cc8b4c182711a7cd390d418a0187c63240288"} +{"at_utc": "2026-10-18T14:30:33.082+00:00", "data": {"fileObjectId": 30, "generation": "1792333832876770", "sha256": "5f82ef3f5fc831e1b38624f982436ebd6dd233cf9b36e0af9c12b5d7a42f8ce6", "size": 1046}, "kind": "original_preserved", "previous": "9ac7fa7b25f3473542113457b00cc8b4c182711a7cd390d418a0187c63240288", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 174, "sha256": "e2fb7e2f78e9a32ce420dea78c07e3503150adb315b9f93aa03640f233a30c8c"} +{"at_utc": "2026-10-18T14:30:34.347+00:00", "data": {"fileObjectId": 31, "generation": "1792333834140672", "sha256": "f904af9279ff864430761562c9f2c207dd1539c18c91098a2a43fe3dd484806c", "size": 642}, "kind": "original_preserved", "previous": "e2fb7e2f78e9a32ce420dea78c07e3503150adb315b9f93aa03640f233a30c8c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 175, "sha256": "bc232bbc561510082c49bc6b7b50b87c6d28887e8ff952a84b13d520bee1e5ea"} +{"at_utc": "2026-10-18T14:30:35.498+00:00", "data": {"fileObjectId": 32, "generation": "1792333835301903", "sha256": "8c4163e1723cf99bada4fccc1956b24b36aeb3ff9aaa2b18a1d5a4355c42a0d3", "size": 1045}, "kind": "original_preserved", "previous": "bc232bbc561510082c49bc6b7b50b87c6d28887e8ff952a84b13d520bee1e5ea", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 176, "sha256": "cb87a38304f0c1b94b06916eef9b3ac0493a6fe929e686dd89e19afc7f3e490f"} +{"at_utc": "2026-10-18T14:30:36.884+00:00", "data": {"fileObjectId": 33, "generation": "1792333836593143", "sha256": "296abbc16a4f567ea8197c9a138b973da051803e082e5b52d627f9b023453dc0", "size": 642}, "kind": "original_preserved", "previous": "cb87a38304f0c1b94b06916eef9b3ac0493a6fe929e686dd89e19afc7f3e490f", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 177, "sha256": "597f0a42e1e16b6888b89a17a5935ab797d9fe2318eeeb2ea3890d3624051ef5"} +{"at_utc": "2026-10-18T14:30:41.298+00:00", "data": {"generation": "1792333839694782", "name": "00000006000000000000002F", "sha256": "18d14efba2fe22e9444403cffd5707a84ac1587783d7e28f181d6018b183ba81", "size": 16777216}, "kind": "wal_preserved", "previous": "597f0a42e1e16b6888b89a17a5935ab797d9fe2318eeeb2ea3890d3624051ef5", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 178, "sha256": "5885a8d022592db5b004f3b260badeb8ab539ca242e6d21a8fe068fdc05e5a15"} +{"at_utc": "2026-10-18T14:30:43.010+00:00", "data": {"archive_attempts": 3, "covered_acknowledged_requests": 33, "ordinal": 1, "remote_bytes_verified": true, "required_wal": "00000006000000000000002F"}, "kind": "wal_barrier_verified", "previous": "5885a8d022592db5b004f3b260badeb8ab539ca242e6d21a8fe068fdc05e5a15", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 179, "sha256": "dc054a4d616b2d7316045dbe35647e55e5f7de68c1086065ede279ed8101b6e2"} +{"at_utc": "2026-10-18T14:30:43.014+00:00", "data": {"complete_wals": 6, "elapsed_seconds": 66.737, "ordinal": 1, "preserved_originals": 33}, "kind": "offsite_round_completed", "previous": "dc054a4d616b2d7316045dbe35647e55e5f7de68c1086065ede279ed8101b6e2", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 180, "sha256": "a056b1ffa54895e13fe84e06a01b701d64b21ee9d4fda050271f58da0dca6ab7"} +{"at_utc": "2026-10-18T14:30:43.020+00:00", "data": {"cohort": "loss", "request_id": "upload-d029610fe4234219b1db66f348b7b114", "sha256": "4a1955b23633e09c64c251faf53a7f3db408e65c0d31e4131eac14e2e408b036", "size": 1046}, "kind": "sent", "previous": "a056b1ffa54895e13fe84e06a01b701d64b21ee9d4fda050271f58da0dca6ab7", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 181, "sha256": "d6bbcfc68a1f298d0143a935325a5c198848b21b67e618a70a3c042090ddef24"} +{"at_utc": "2026-10-18T14:30:51.420+00:00", "data": {"http_status": 201, "ids": {"documentId": 36, "documentVersionId": 36, "embeddingJobId": 34, "fileObjectId": 34}, "outcome": "ACKNOWLEDGED", "request_id": "upload-d029610fe4234219b1db66f348b7b114"}, "kind": "response", "previous": "d6bbcfc68a1f298d0143a935325a5c198848b21b67e618a70a3c042090ddef24", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 182, "sha256": "ef98183720aac46eca5324afdd69d667e8eb03804c42b15ada5af02aac8cb141"} +{"at_utc": "2026-10-18T14:30:51.424+00:00", "data": {"cohort": "loss", "elapsed_seconds": 8.401, "http_status": 201, "ordinal": 34}, "kind": "upload_observed", "previous": "ef98183720aac46eca5324afdd69d667e8eb03804c42b15ada5af02aac8cb141", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 183, "sha256": "cc99cd46dfa48ebc5ed7310d3dd92b9cda7389b475924ada0c045699c64c4008"} +{"at_utc": "2026-10-18T14:30:51.430+00:00", "data": {"cohort": "loss", "request_id": "upload-ec97bd5c298f483d9cf98729cea26940", "sha256": "d517ea588330b3a1009b2a65decaa649f2eb64041318067f7eca6f41310beb19", "size": 642}, "kind": "sent", "previous": "cc99cd46dfa48ebc5ed7310d3dd92b9cda7389b475924ada0c045699c64c4008", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 184, "sha256": "e08e0df374b36b3310ee773673b606e9ed021c2af92202550a36ed19491ae165"} +{"at_utc": "2026-10-18T14:30:59.249+00:00", "data": {"http_status": 201, "ids": {"documentId": 37, "documentVersionId": 37, "embeddingJobId": 35, "fileObjectId": 35}, "outcome": "ACKNOWLEDGED", "request_id": "upload-ec97bd5c298f483d9cf98729cea26940"}, "kind": "response", "previous": "e08e0df374b36b3310ee773673b606e9ed021c2af92202550a36ed19491ae165", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 185, "sha256": "515c4688a6e04072507333fdca5b0d14c4da69fa8dabfad5a5885899247d8042"} +{"at_utc": "2026-10-18T14:30:59.257+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.82, "http_status": 201, "ordinal": 35}, "kind": "upload_observed", "previous": "515c4688a6e04072507333fdca5b0d14c4da69fa8dabfad5a5885899247d8042", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 186, "sha256": "9348bbddfa134629e8eb0ed7c3dc9b78f725332a3ac247936d95da0b5d6155bf"} +{"at_utc": "2026-10-18T14:30:59.266+00:00", "data": {"cohort": "loss", "request_id": "upload-0b7cf70ddca94790b02c8e8024fb040d", "sha256": "39f7e102ac80a9ba04150f8e357db6123441bdc7cafe5620dc163dda4661ae6f", "size": 1046}, "kind": "sent", "previous": "9348bbddfa134629e8eb0ed7c3dc9b78f725332a3ac247936d95da0b5d6155bf", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 187, "sha256": "3ac3f765e6f6bfb15fab8ea06fe9caef24a8bd4fb8f1ec370c4d71d2f579b4a9"} +{"at_utc": "2026-10-18T14:31:06.618+00:00", "data": {"http_status": 201, "ids": {"documentId": 38, "documentVersionId": 38, "embeddingJobId": 36, "fileObjectId": 36}, "outcome": "ACKNOWLEDGED", "request_id": "upload-0b7cf70ddca94790b02c8e8024fb040d"}, "kind": "response", "previous": "3ac3f765e6f6bfb15fab8ea06fe9caef24a8bd4fb8f1ec370c4d71d2f579b4a9", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 188, "sha256": "2c3d22a017bbe19846c1d3b250d316648c2309aea54626ec5f60ad23f461ddb6"} +{"at_utc": "2026-10-18T14:31:06.625+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.352, "http_status": 201, "ordinal": 36}, "kind": "upload_observed", "previous": "2c3d22a017bbe19846c1d3b250d316648c2309aea54626ec5f60ad23f461ddb6", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 189, "sha256": "9752db71d283f4dc2f7a9464180bcd46a827afcd03ad938e48f1cabc0e041407"} +{"at_utc": "2026-10-18T14:31:06.633+00:00", "data": {"cohort": "loss", "request_id": "upload-df6017955c5447b9bf097221f6a7d8be", "sha256": "2a1bfc66a9c58c43d8d51ff1c5391e58dd582fbc386733c41d7c3989fdce9731", "size": 642}, "kind": "sent", "previous": "9752db71d283f4dc2f7a9464180bcd46a827afcd03ad938e48f1cabc0e041407", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 190, "sha256": "ea7909d2a3da81c5baa930b8c07c89a20cc9f17efdab962f48d53274745c876d"} +{"at_utc": "2026-10-18T14:31:13.944+00:00", "data": {"http_status": 201, "ids": {"documentId": 39, "documentVersionId": 39, "embeddingJobId": 37, "fileObjectId": 37}, "outcome": "ACKNOWLEDGED", "request_id": "upload-df6017955c5447b9bf097221f6a7d8be"}, "kind": "response", "previous": "ea7909d2a3da81c5baa930b8c07c89a20cc9f17efdab962f48d53274745c876d", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 191, "sha256": "0dba1e558ddb3862637a58b67b3c0bb91ed5c434e7841b832a3963d81a1b03eb"} +{"at_utc": "2026-10-18T14:31:13.950+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.311, "http_status": 201, "ordinal": 37}, "kind": "upload_observed", "previous": "0dba1e558ddb3862637a58b67b3c0bb91ed5c434e7841b832a3963d81a1b03eb", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 192, "sha256": "07fa58babf0545cbb3f34cf0674939b9868d822753f76912772283debcda2b4c"} +{"at_utc": "2026-10-18T14:31:13.959+00:00", "data": {"cohort": "loss", "request_id": "upload-ba75b05d8e5d477abb3d2a1180a95077", "sha256": "0b3353283ad6d05cb497416736e7fdda10b57f9a0ffbfb2d316596365791b4b8", "size": 1046}, "kind": "sent", "previous": "07fa58babf0545cbb3f34cf0674939b9868d822753f76912772283debcda2b4c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 193, "sha256": "32d29e77a07be2bf614a1f6836f2df6bb5bdc9e2c60ff547abbccb24f61acb78"} +{"at_utc": "2026-10-18T14:31:22.044+00:00", "data": {"http_status": 201, "ids": {"documentId": 40, "documentVersionId": 40, "embeddingJobId": 38, "fileObjectId": 38}, "outcome": "ACKNOWLEDGED", "request_id": "upload-ba75b05d8e5d477abb3d2a1180a95077"}, "kind": "response", "previous": "32d29e77a07be2bf614a1f6836f2df6bb5bdc9e2c60ff547abbccb24f61acb78", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 194, "sha256": "6a0cdf678e994e10ac5976130e242a80ac384dcb68b25724ddb6c7d2aa46ae17"} +{"at_utc": "2026-10-18T14:31:22.051+00:00", "data": {"cohort": "loss", "elapsed_seconds": 8.086, "http_status": 201, "ordinal": 38}, "kind": "upload_observed", "previous": "6a0cdf678e994e10ac5976130e242a80ac384dcb68b25724ddb6c7d2aa46ae17", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 195, "sha256": "90ea672a930780ded3ddc5f0aa1c7b279e68d09f6943e9695da7b8b2918b9d64"} +{"at_utc": "2026-10-18T14:31:22.061+00:00", "data": {"cohort": "loss", "request_id": "upload-a5be52472400424db8f52ed7a54c2e83", "sha256": "6fa3bf5896c7e74952fcd6935204ebf002cda8e6892bd4b34d19839e9849576d", "size": 642}, "kind": "sent", "previous": "90ea672a930780ded3ddc5f0aa1c7b279e68d09f6943e9695da7b8b2918b9d64", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 196, "sha256": "7eeb14391a893eb5ec049be976af51df0e594525acca057098f463a1df49f473"} +{"at_utc": "2026-10-18T14:31:29.191+00:00", "data": {"http_status": 201, "ids": {"documentId": 41, "documentVersionId": 41, "embeddingJobId": 39, "fileObjectId": 39}, "outcome": "ACKNOWLEDGED", "request_id": "upload-a5be52472400424db8f52ed7a54c2e83"}, "kind": "response", "previous": "7eeb14391a893eb5ec049be976af51df0e594525acca057098f463a1df49f473", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 197, "sha256": "cf2abff3eeddf987df9f629cdcc7b27521818f731fb77509c53fd24f5bb966e7"} +{"at_utc": "2026-10-18T14:31:29.197+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.131, "http_status": 201, "ordinal": 39}, "kind": "upload_observed", "previous": "cf2abff3eeddf987df9f629cdcc7b27521818f731fb77509c53fd24f5bb966e7", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 198, "sha256": "65968bae4bdb841c7bea91801b4ddb0057013a2af66245c042685c204e723e13"} +{"at_utc": "2026-10-18T14:31:29.206+00:00", "data": {"cohort": "loss", "request_id": "upload-eb8fb6803498447a94b89421596acf47", "sha256": "604dda88d3af34bfe5fb2ecbd42e1466b7123367aabf2b6479f5187bbde8b1e5", "size": 1046}, "kind": "sent", "previous": "65968bae4bdb841c7bea91801b4ddb0057013a2af66245c042685c204e723e13", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 199, "sha256": "97d8986acdd8124e384c1095e717654351897ba570d95f7a221464ced877d4f4"} +{"at_utc": "2026-10-18T14:31:36.338+00:00", "data": {"http_status": 201, "ids": {"documentId": 42, "documentVersionId": 42, "embeddingJobId": 40, "fileObjectId": 40}, "outcome": "ACKNOWLEDGED", "request_id": "upload-eb8fb6803498447a94b89421596acf47"}, "kind": "response", "previous": "97d8986acdd8124e384c1095e717654351897ba570d95f7a221464ced877d4f4", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 200, "sha256": "fdcc4e0b7a61e4c8d2d79ae09b24e8c4090eddf3a84b59ccd9b09de6b280eaca"} +{"at_utc": "2026-10-18T14:31:36.345+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.133, "http_status": 201, "ordinal": 40}, "kind": "upload_observed", "previous": "fdcc4e0b7a61e4c8d2d79ae09b24e8c4090eddf3a84b59ccd9b09de6b280eaca", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 201, "sha256": "f4c445db6ace47c9e84d72cc9c52b178c194277b2b2a2297693b0e0e10b3a0eb"} +{"at_utc": "2026-10-18T14:31:36.354+00:00", "data": {"cohort": "loss", "request_id": "upload-a13bf9f28e9e4643bdea46f2d4a2bd62", "sha256": "cb1defde74581e45e5606cfe0f948314533669d751d71149d905ac755a067425", "size": 642}, "kind": "sent", "previous": "f4c445db6ace47c9e84d72cc9c52b178c194277b2b2a2297693b0e0e10b3a0eb", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 202, "sha256": "89415b1861e975cb17aec4afa4f51c7991898181e0d8a370530ed2d17b01ad86"} +{"at_utc": "2026-10-18T14:31:43.833+00:00", "data": {"http_status": 201, "ids": {"documentId": 43, "documentVersionId": 43, "embeddingJobId": 41, "fileObjectId": 41}, "outcome": "ACKNOWLEDGED", "request_id": "upload-a13bf9f28e9e4643bdea46f2d4a2bd62"}, "kind": "response", "previous": "89415b1861e975cb17aec4afa4f51c7991898181e0d8a370530ed2d17b01ad86", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 203, "sha256": "aae754a258a4495b866fef3c5272eaeebb57373a840d1004e7736310ade109f0"} +{"at_utc": "2026-10-18T14:31:43.839+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.479, "http_status": 201, "ordinal": 41}, "kind": "upload_observed", "previous": "aae754a258a4495b866fef3c5272eaeebb57373a840d1004e7736310ade109f0", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 204, "sha256": "07b4157f03db19d4e21e25e8c05b7e6c7b580374cd1d1e3b76760dc6706b4663"} +{"at_utc": "2026-10-18T14:31:43.848+00:00", "data": {"cohort": "loss", "request_id": "upload-813ddd2e5f1c4a5d827384f84a0005aa", "sha256": "cca8e204eaa0a10bb320b2979da370eb64ed2bbc54d9ffe190b3bd0206ba97e5", "size": 1046}, "kind": "sent", "previous": "07b4157f03db19d4e21e25e8c05b7e6c7b580374cd1d1e3b76760dc6706b4663", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 205, "sha256": "692b24ca2cd698fbd20212d76b71f50171e212db14fe03be74ef386a49c21698"} +{"at_utc": "2026-10-18T14:31:51.023+00:00", "data": {"http_status": 201, "ids": {"documentId": 44, "documentVersionId": 44, "embeddingJobId": 42, "fileObjectId": 42}, "outcome": "ACKNOWLEDGED", "request_id": "upload-813ddd2e5f1c4a5d827384f84a0005aa"}, "kind": "response", "previous": "692b24ca2cd698fbd20212d76b71f50171e212db14fe03be74ef386a49c21698", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 206, "sha256": "2915f45a38c209956509d5f8b5d4494d21244f0915462ed3bedcbe67abd11a3b"} +{"at_utc": "2026-10-18T14:31:51.030+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.175, "http_status": 201, "ordinal": 42}, "kind": "upload_observed", "previous": "2915f45a38c209956509d5f8b5d4494d21244f0915462ed3bedcbe67abd11a3b", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 207, "sha256": "6ab46bd948e43d73268d4aff379024cb53eb361b65fb7dd66748474ffdfa517c"} +{"at_utc": "2026-10-18T14:31:51.039+00:00", "data": {"cohort": "loss", "request_id": "upload-3c1490dc0cba43a5828e902ad9ccb4bf", "sha256": "2561e3f2e810a8a4c7c06b6f0d03578d79f27889387f1ad53fde39f5070b63a5", "size": 642}, "kind": "sent", "previous": "6ab46bd948e43d73268d4aff379024cb53eb361b65fb7dd66748474ffdfa517c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 208, "sha256": "a598c7e5817f6cd6eb308ba124eb4e566c5f6c86db547ed2198b3e1944a13ca6"} +{"at_utc": "2026-10-18T14:31:58.501+00:00", "data": {"http_status": 201, "ids": {"documentId": 45, "documentVersionId": 45, "embeddingJobId": 43, "fileObjectId": 43}, "outcome": "ACKNOWLEDGED", "request_id": "upload-3c1490dc0cba43a5828e902ad9ccb4bf"}, "kind": "response", "previous": "a598c7e5817f6cd6eb308ba124eb4e566c5f6c86db547ed2198b3e1944a13ca6", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 209, "sha256": "0714a65360c3f89d7895e7ff9a57acd4c1efa558c97b0316548a9cbbe85110e3"} +{"at_utc": "2026-10-18T14:31:58.508+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.463, "http_status": 201, "ordinal": 43}, "kind": "upload_observed", "previous": "0714a65360c3f89d7895e7ff9a57acd4c1efa558c97b0316548a9cbbe85110e3", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 210, "sha256": "f40006ed51f50080ec9683b031379d61152e7688e08ca369594c9384a8892199"} +{"at_utc": "2026-10-18T14:31:58.521+00:00", "data": {"cohort": "loss", "request_id": "upload-0d42b4eeef9e49958d54034debf32404", "sha256": "7dcaaa488971cae03bfd16b70021146afe24dc803d51107bdb4925b1613af5e8", "size": 1046}, "kind": "sent", "previous": "f40006ed51f50080ec9683b031379d61152e7688e08ca369594c9384a8892199", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 211, "sha256": "cf042b41930220eab8bfe7f8310f566a7fe67dbd3230f87df77fc49c88a5eb16"} +{"at_utc": "2026-10-18T14:32:06.107+00:00", "data": {"http_status": 201, "ids": {"documentId": 46, "documentVersionId": 46, "embeddingJobId": 44, "fileObjectId": 44}, "outcome": "ACKNOWLEDGED", "request_id": "upload-0d42b4eeef9e49958d54034debf32404"}, "kind": "response", "previous": "cf042b41930220eab8bfe7f8310f566a7fe67dbd3230f87df77fc49c88a5eb16", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 212, "sha256": "7d44deb916e73f56593371c1aea97771236ff79c6472f224506f30121fd4cd88"} +{"at_utc": "2026-10-18T14:32:06.116+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.587, "http_status": 201, "ordinal": 44}, "kind": "upload_observed", "previous": "7d44deb916e73f56593371c1aea97771236ff79c6472f224506f30121fd4cd88", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 213, "sha256": "5ede5672e00600ad2e725184ae80b2ec0cbd50bf29b9c4326294ba480157d6bf"} +{"at_utc": "2026-10-18T14:32:06.125+00:00", "data": {"cohort": "loss", "request_id": "upload-057d9136fc534bd7a05d4a3f2fe91437", "sha256": "bac1098d29cc26b2ef7f606bd8f4942f864358d57c99dce5166ea4db3287933e", "size": 642}, "kind": "sent", "previous": "5ede5672e00600ad2e725184ae80b2ec0cbd50bf29b9c4326294ba480157d6bf", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 214, "sha256": "53f2e1ad397569a43678773383d1c41da2b416ffd00295ee69488ea3cce3a85f"} +{"at_utc": "2026-10-18T14:32:13.448+00:00", "data": {"http_status": 201, "ids": {"documentId": 47, "documentVersionId": 47, "embeddingJobId": 45, "fileObjectId": 45}, "outcome": "ACKNOWLEDGED", "request_id": "upload-057d9136fc534bd7a05d4a3f2fe91437"}, "kind": "response", "previous": "53f2e1ad397569a43678773383d1c41da2b416ffd00295ee69488ea3cce3a85f", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 215, "sha256": "9cef45c85ded243c67eb6e002b9dc02b2da7cf89adf8c53127581266a711d492"} +{"at_utc": "2026-10-18T14:32:13.456+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.323, "http_status": 201, "ordinal": 45}, "kind": "upload_observed", "previous": "9cef45c85ded243c67eb6e002b9dc02b2da7cf89adf8c53127581266a711d492", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 216, "sha256": "6b9b967db2954f8b23a5d882c17ccc1a59b256532240d869841c9da26b1000ae"} +{"at_utc": "2026-10-18T14:32:13.465+00:00", "data": {"cohort": "loss", "request_id": "upload-64250f659e794f5cb774e58461b97e4e", "sha256": "ed8f4d573635655183f52c0a3a7ad5eab7325225399f46ea41372cc7907cefd3", "size": 1046}, "kind": "sent", "previous": "6b9b967db2954f8b23a5d882c17ccc1a59b256532240d869841c9da26b1000ae", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 217, "sha256": "ed09bc02d62f1ac8ca629f2eb779e26357d63ce307815c0a7fa70b58e84c4fbd"} +{"at_utc": "2026-10-18T14:32:21.026+00:00", "data": {"http_status": 201, "ids": {"documentId": 48, "documentVersionId": 48, "embeddingJobId": 46, "fileObjectId": 46}, "outcome": "ACKNOWLEDGED", "request_id": "upload-64250f659e794f5cb774e58461b97e4e"}, "kind": "response", "previous": "ed09bc02d62f1ac8ca629f2eb779e26357d63ce307815c0a7fa70b58e84c4fbd", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 218, "sha256": "667c4625e843ce32843b4303ce0dc5f15dcfea34c47d2a2dd84a904d2286d5fa"} +{"at_utc": "2026-10-18T14:32:21.034+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.562, "http_status": 201, "ordinal": 46}, "kind": "upload_observed", "previous": "667c4625e843ce32843b4303ce0dc5f15dcfea34c47d2a2dd84a904d2286d5fa", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 219, "sha256": "11abdf5f4e60e2ac662eed5af7cd2faa3669e9268b322b5ce35305165e8452e0"} +{"at_utc": "2026-10-18T14:32:21.044+00:00", "data": {"cohort": "loss", "request_id": "upload-57ac7e89e46c4e7c857a794287fca81a", "sha256": "78d4567ad81362950c53cdaf6310b3c6be4280a500ce8422c4473f0974fa178d", "size": 642}, "kind": "sent", "previous": "11abdf5f4e60e2ac662eed5af7cd2faa3669e9268b322b5ce35305165e8452e0", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 220, "sha256": "5ed3ee5febb8928fc3b1abd2646e291ea5f66c703477e8eaecf309f4a2e8e3d9"} +{"at_utc": "2026-10-18T14:32:28.380+00:00", "data": {"http_status": 201, "ids": {"documentId": 49, "documentVersionId": 49, "embeddingJobId": 47, "fileObjectId": 47}, "outcome": "ACKNOWLEDGED", "request_id": "upload-57ac7e89e46c4e7c857a794287fca81a"}, "kind": "response", "previous": "5ed3ee5febb8928fc3b1abd2646e291ea5f66c703477e8eaecf309f4a2e8e3d9", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 221, "sha256": "769f52d519041cbcb6f985c3d75721e9411b33f317093909c6b582df8edda07a"} +{"at_utc": "2026-10-18T14:32:28.384+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.337, "http_status": 201, "ordinal": 47}, "kind": "upload_observed", "previous": "769f52d519041cbcb6f985c3d75721e9411b33f317093909c6b582df8edda07a", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 222, "sha256": "ae8be801b281f9442e80ea08058af1f3769b27fded6f014690643a738c17e72c"} +{"at_utc": "2026-10-18T14:32:28.392+00:00", "data": {"cohort": "loss", "request_id": "upload-2b25d7a9090f48d088f40a97b846d14f", "sha256": "c3ef81a8b35b77101fa3c1caba153f8dac4c76b0d6ab56e917bbc9ad88440296", "size": 1046}, "kind": "sent", "previous": "ae8be801b281f9442e80ea08058af1f3769b27fded6f014690643a738c17e72c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 223, "sha256": "18cb02d7cae4a8138bd998662a04ba01787fccabe6428261fb4544af3fbb0c6b"} +{"at_utc": "2026-10-18T14:32:36.479+00:00", "data": {"http_status": 201, "ids": {"documentId": 50, "documentVersionId": 50, "embeddingJobId": 48, "fileObjectId": 48}, "outcome": "ACKNOWLEDGED", "request_id": "upload-2b25d7a9090f48d088f40a97b846d14f"}, "kind": "response", "previous": "18cb02d7cae4a8138bd998662a04ba01787fccabe6428261fb4544af3fbb0c6b", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 224, "sha256": "0305a2c33932cedde385ab6668d2a7699a8e5bc3c7e102ef7b36de581548c80b"} +{"at_utc": "2026-10-18T14:32:36.486+00:00", "data": {"cohort": "loss", "elapsed_seconds": 8.088, "http_status": 201, "ordinal": 48}, "kind": "upload_observed", "previous": "0305a2c33932cedde385ab6668d2a7699a8e5bc3c7e102ef7b36de581548c80b", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 225, "sha256": "1412486340970fb70f0009c647a45cbb5c1818e910a993833f87e3794f91b72d"} +{"at_utc": "2026-10-18T14:32:36.496+00:00", "data": {"cohort": "loss", "request_id": "upload-2e705577fed54f4e8ad8f6f9e17fe774", "sha256": "6bd1b2aa0f2dd0e8547740529e00c66252bb39153bac639ed6ab0f1655fe3673", "size": 642}, "kind": "sent", "previous": "1412486340970fb70f0009c647a45cbb5c1818e910a993833f87e3794f91b72d", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 226, "sha256": "a35a64656d21e85aa6f0bcf50f8816d4886c7778bdb6c003c68c02c5c71bce5e"} +{"at_utc": "2026-10-18T14:32:44.145+00:00", "data": {"http_status": 201, "ids": {"documentId": 51, "documentVersionId": 51, "embeddingJobId": 49, "fileObjectId": 49}, "outcome": "ACKNOWLEDGED", "request_id": "upload-2e705577fed54f4e8ad8f6f9e17fe774"}, "kind": "response", "previous": "a35a64656d21e85aa6f0bcf50f8816d4886c7778bdb6c003c68c02c5c71bce5e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 227, "sha256": "d57966387b9ffbc0ac781a3b785ad2af843182ba8a547b45bc77db4b4ca07cab"} +{"at_utc": "2026-10-18T14:32:44.152+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.649, "http_status": 201, "ordinal": 49}, "kind": "upload_observed", "previous": "d57966387b9ffbc0ac781a3b785ad2af843182ba8a547b45bc77db4b4ca07cab", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 228, "sha256": "554e072bb35557d13aee67e02a0e8fe91387492a0bca872b78abdd71bc232ec8"} +{"at_utc": "2026-10-18T14:32:44.161+00:00", "data": {"cohort": "loss", "request_id": "upload-72aca0b7e4864f92990fe560dbcc0a69", "sha256": "5c5f78c20f08e1cc1bfe3d8fdecdd7fe41d887bbd6365ddeb02755131ac7e266", "size": 1046}, "kind": "sent", "previous": "554e072bb35557d13aee67e02a0e8fe91387492a0bca872b78abdd71bc232ec8", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 229, "sha256": "cbf97dbf40f502d6c362a64f8b0d5560a0c008e54831035a1bd1497227d68c54"} +{"at_utc": "2026-10-18T14:32:51.475+00:00", "data": {"http_status": 201, "ids": {"documentId": 52, "documentVersionId": 52, "embeddingJobId": 50, "fileObjectId": 50}, "outcome": "ACKNOWLEDGED", "request_id": "upload-72aca0b7e4864f92990fe560dbcc0a69"}, "kind": "response", "previous": "cbf97dbf40f502d6c362a64f8b0d5560a0c008e54831035a1bd1497227d68c54", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 230, "sha256": "aa1f5ac4eebd3f3c9ffaca935fe46d6e3a49a75c8474a5b5ece46a1155d3d107"} +{"at_utc": "2026-10-18T14:32:51.483+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.314, "http_status": 201, "ordinal": 50}, "kind": "upload_observed", "previous": "aa1f5ac4eebd3f3c9ffaca935fe46d6e3a49a75c8474a5b5ece46a1155d3d107", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 231, "sha256": "c7f82f2f830fd6e619eb18e7ff4faba613551e73158e31a00b8c63c18077eb44"} +{"at_utc": "2026-10-18T14:32:51.492+00:00", "data": {"cohort": "loss", "request_id": "upload-ddd360391bc64e6c9d20673ae26a73f3", "sha256": "7d2dc24cbce3133caeebe9ea1b3f17d7053f1d73083527721b521416de72449a", "size": 642}, "kind": "sent", "previous": "c7f82f2f830fd6e619eb18e7ff4faba613551e73158e31a00b8c63c18077eb44", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 232, "sha256": "3e2178a3d930671956e1b7ca3cf5be456a6da83367d98c661c29b3f4893cd9ed"} +{"at_utc": "2026-10-18T14:32:58.844+00:00", "data": {"http_status": 201, "ids": {"documentId": 53, "documentVersionId": 53, "embeddingJobId": 51, "fileObjectId": 51}, "outcome": "ACKNOWLEDGED", "request_id": "upload-ddd360391bc64e6c9d20673ae26a73f3"}, "kind": "response", "previous": "3e2178a3d930671956e1b7ca3cf5be456a6da83367d98c661c29b3f4893cd9ed", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 233, "sha256": "535cda128c633f20c920bfa5c3d7ff048a5e492a9d9519c74cb72b4308002fb5"} +{"at_utc": "2026-10-18T14:32:58.853+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.354, "http_status": 201, "ordinal": 51}, "kind": "upload_observed", "previous": "535cda128c633f20c920bfa5c3d7ff048a5e492a9d9519c74cb72b4308002fb5", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 234, "sha256": "8a61d6c348019334c351a59220d265592d07b84b83ee503fac7de95e74d6d6c1"} +{"at_utc": "2026-10-18T14:32:58.863+00:00", "data": {"cohort": "loss", "request_id": "upload-3cc178f1e0904fc39ac08a077e048a50", "sha256": "bab62716f6a42b0766db18babe1e4d649e24be3e9e731e6f2553f16e58ec011e", "size": 1046}, "kind": "sent", "previous": "8a61d6c348019334c351a59220d265592d07b84b83ee503fac7de95e74d6d6c1", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 235, "sha256": "ff2cc9c3582515d50428d2cfc9ca34ec3223f77d89ef22521d4f87e403984841"} +{"at_utc": "2026-10-18T14:33:06.483+00:00", "data": {"http_status": 201, "ids": {"documentId": 54, "documentVersionId": 54, "embeddingJobId": 52, "fileObjectId": 52}, "outcome": "ACKNOWLEDGED", "request_id": "upload-3cc178f1e0904fc39ac08a077e048a50"}, "kind": "response", "previous": "ff2cc9c3582515d50428d2cfc9ca34ec3223f77d89ef22521d4f87e403984841", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 236, "sha256": "de4bc19979dd32ff45a8fa59efe3ab54b0211f6c2813b97dadbd564f74268c7a"} +{"at_utc": "2026-10-18T14:33:06.492+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.621, "http_status": 201, "ordinal": 52}, "kind": "upload_observed", "previous": "de4bc19979dd32ff45a8fa59efe3ab54b0211f6c2813b97dadbd564f74268c7a", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 237, "sha256": "fa7f79f6faeafcad52c8841d5de1ab437e8334a54eebe5e929cc644139dc2d24"} +{"at_utc": "2026-10-18T14:33:08.427+00:00", "data": {"cohort": "loss", "request_id": "upload-8b600c6bb6114cdc8a084fb3bcc63b91", "sha256": "7c1518f5b75aa22b3581da98e94d958e94a4a8bf146ab650a7a245964401c004", "size": 642}, "kind": "sent", "previous": "fa7f79f6faeafcad52c8841d5de1ab437e8334a54eebe5e929cc644139dc2d24", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 238, "sha256": "46b2b3ac886440d4b32f1b348006e6e6f80c529bfe8002951a3f10392780dff1"} +{"at_utc": "2026-10-18T14:33:15.806+00:00", "data": {"http_status": 201, "ids": {"documentId": 55, "documentVersionId": 55, "embeddingJobId": 53, "fileObjectId": 53}, "outcome": "ACKNOWLEDGED", "request_id": "upload-8b600c6bb6114cdc8a084fb3bcc63b91"}, "kind": "response", "previous": "46b2b3ac886440d4b32f1b348006e6e6f80c529bfe8002951a3f10392780dff1", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 239, "sha256": "add2115dce65d8914727d05e1fdd4ccdb0b49a845554b2e9b33d390d4a851654"} +{"at_utc": "2026-10-18T14:33:15.814+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.379, "http_status": 201, "ordinal": 53}, "kind": "upload_observed", "previous": "add2115dce65d8914727d05e1fdd4ccdb0b49a845554b2e9b33d390d4a851654", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 240, "sha256": "b64a5aac351e99db3f01179e2a6d7a30a0f71dd025717bba93493239bf60ecaf"} +{"at_utc": "2026-10-18T14:33:19.424+00:00", "data": {"cohort": "loss", "request_id": "upload-9e764d5ec32145c18c4c1af2d22d9644", "sha256": "71f499d0499d9204f28495bf65fb23854f288a8d744a6dbc38c405bc34b319ba", "size": 1046}, "kind": "sent", "previous": "b64a5aac351e99db3f01179e2a6d7a30a0f71dd025717bba93493239bf60ecaf", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 241, "sha256": "1c5b8b76cac18f9e956be6baef94e07302853ea2d608f6ba05f1f6a78e6eedaa"} +{"at_utc": "2026-10-18T14:33:27.131+00:00", "data": {"http_status": 201, "ids": {"documentId": 56, "documentVersionId": 56, "embeddingJobId": 54, "fileObjectId": 54}, "outcome": "ACKNOWLEDGED", "request_id": "upload-9e764d5ec32145c18c4c1af2d22d9644"}, "kind": "response", "previous": "1c5b8b76cac18f9e956be6baef94e07302853ea2d608f6ba05f1f6a78e6eedaa", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 242, "sha256": "e790ff25519604e2b67ea9312e57b7c1916d47a404a97e6d1aeedc4d8f03b40d"} +{"at_utc": "2026-10-18T14:33:27.142+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.707, "http_status": 201, "ordinal": 54}, "kind": "upload_observed", "previous": "e790ff25519604e2b67ea9312e57b7c1916d47a404a97e6d1aeedc4d8f03b40d", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 243, "sha256": "7149be714fb51c3404065e01b01d35bb4e61757fccea04a5f2704b80099aaba4"} +{"at_utc": "2026-10-18T14:33:30.425+00:00", "data": {"cohort": "loss", "request_id": "upload-17641dcaf2874cf1bdb416a174662e89", "sha256": "cbe79087c3e591bebb072292ef7c6d41d62f119c03eee75af19a24b7afdb80fe", "size": 642}, "kind": "sent", "previous": "7149be714fb51c3404065e01b01d35bb4e61757fccea04a5f2704b80099aaba4", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 244, "sha256": "8fbe35390f17bdafb629174e5917d71d6657edd2bbd47f44f74793300cfc1ea6"} +{"at_utc": "2026-10-18T14:33:38.097+00:00", "data": {"http_status": 201, "ids": {"documentId": 57, "documentVersionId": 57, "embeddingJobId": 55, "fileObjectId": 55}, "outcome": "ACKNOWLEDGED", "request_id": "upload-17641dcaf2874cf1bdb416a174662e89"}, "kind": "response", "previous": "8fbe35390f17bdafb629174e5917d71d6657edd2bbd47f44f74793300cfc1ea6", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 245, "sha256": "30394f1d09bba411f9a065c87e940a722a70d69dafdbed286056b326526f3286"} +{"at_utc": "2026-10-18T14:33:38.104+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.672, "http_status": 201, "ordinal": 55}, "kind": "upload_observed", "previous": "30394f1d09bba411f9a065c87e940a722a70d69dafdbed286056b326526f3286", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 246, "sha256": "f7ff16f833cdec8365fc3622ada0925bc105c1d167c01c1afa12735ad8e70f1a"} +{"at_utc": "2026-10-18T14:33:41.426+00:00", "data": {"cohort": "loss", "request_id": "upload-70d798ac940240a196a56d6652604d63", "sha256": "5b6caf7ba08fc075fc7748fec448cbf746b6af1f3c6ae4c0548568ee64f7810a", "size": 1046}, "kind": "sent", "previous": "f7ff16f833cdec8365fc3622ada0925bc105c1d167c01c1afa12735ad8e70f1a", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 247, "sha256": "99506991e654ae19980ef2bd436d0bddb1e10a5dd444e19212a983f8d8d0dbcf"} +{"at_utc": "2026-10-18T14:33:48.947+00:00", "data": {"http_status": 201, "ids": {"documentId": 58, "documentVersionId": 58, "embeddingJobId": 56, "fileObjectId": 56}, "outcome": "ACKNOWLEDGED", "request_id": "upload-70d798ac940240a196a56d6652604d63"}, "kind": "response", "previous": "99506991e654ae19980ef2bd436d0bddb1e10a5dd444e19212a983f8d8d0dbcf", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 248, "sha256": "a3cd0831990f7b7d67d3c68d08888660447192f5d03c2bd8d3aeec496c6a1ef6"} +{"at_utc": "2026-10-18T14:33:48.953+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.522, "http_status": 201, "ordinal": 56}, "kind": "upload_observed", "previous": "a3cd0831990f7b7d67d3c68d08888660447192f5d03c2bd8d3aeec496c6a1ef6", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 249, "sha256": "281dca236bd37cb4b9105c7bfff525e8d2305d8ed28aa87f129db5ce4bcc8d85"} +{"at_utc": "2026-10-18T14:33:52.425+00:00", "data": {"cohort": "loss", "request_id": "upload-9b6ec2049bf9488f9d4d6e475576a0c2", "sha256": "6acc5225c8a998e1e9d79e823dc96f1c554330c26e68685da025e17f4337657a", "size": 642}, "kind": "sent", "previous": "281dca236bd37cb4b9105c7bfff525e8d2305d8ed28aa87f129db5ce4bcc8d85", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 250, "sha256": "5533d72aa384f7bcb4ecd067436c4b10a974455d6138ea48e6d1de168281075c"} +{"at_utc": "2026-10-18T14:34:00.237+00:00", "data": {"http_status": 201, "ids": {"documentId": 59, "documentVersionId": 59, "embeddingJobId": 57, "fileObjectId": 57}, "outcome": "ACKNOWLEDGED", "request_id": "upload-9b6ec2049bf9488f9d4d6e475576a0c2"}, "kind": "response", "previous": "5533d72aa384f7bcb4ecd067436c4b10a974455d6138ea48e6d1de168281075c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 251, "sha256": "a273167dd0e4e7ae19b8f90ec5d2ba1d881c363b3c165b2917c036ddd8e9a094"} +{"at_utc": "2026-10-18T14:34:00.243+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.812, "http_status": 201, "ordinal": 57}, "kind": "upload_observed", "previous": "a273167dd0e4e7ae19b8f90ec5d2ba1d881c363b3c165b2917c036ddd8e9a094", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 252, "sha256": "8b266b3e7d0cbbdbc6830751049be7c3175929161a7d1614be3e17f1ea161f93"} +{"at_utc": "2026-10-18T14:34:03.431+00:00", "data": {"cohort": "loss", "request_id": "upload-b1c28ab2ab4245cc83d9a1877e3110e6", "sha256": "2bb8c341cf0b163cbd7891f7c0e0ba1e5195136f958c209a12251bd40f114a32", "size": 1046}, "kind": "sent", "previous": "8b266b3e7d0cbbdbc6830751049be7c3175929161a7d1614be3e17f1ea161f93", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 253, "sha256": "4aee5bb3e752565f2d4cbb8e5c441ea1bc5cf2c3d1914e0f6d3bc60fa1aa8c94"} +{"at_utc": "2026-10-18T14:34:11.280+00:00", "data": {"http_status": 201, "ids": {"documentId": 60, "documentVersionId": 60, "embeddingJobId": 58, "fileObjectId": 58}, "outcome": "ACKNOWLEDGED", "request_id": "upload-b1c28ab2ab4245cc83d9a1877e3110e6"}, "kind": "response", "previous": "4aee5bb3e752565f2d4cbb8e5c441ea1bc5cf2c3d1914e0f6d3bc60fa1aa8c94", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 254, "sha256": "26a22d51a8552ef27f6991ed0ed83a51c8727bf220e4692c3d439a1c03a5c6cc"} +{"at_utc": "2026-10-18T14:34:11.288+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.849, "http_status": 201, "ordinal": 58}, "kind": "upload_observed", "previous": "26a22d51a8552ef27f6991ed0ed83a51c8727bf220e4692c3d439a1c03a5c6cc", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 255, "sha256": "a202a4045ebe570a8e2818e30b3d8c93889aa0b1880238a0822fcacd838a1587"} +{"at_utc": "2026-10-18T14:34:14.428+00:00", "data": {"cohort": "loss", "request_id": "upload-c2f3833d4d4347e7a370f6ca4022c717", "sha256": "c61b92f0cb27263620175ac11f2ab02ace5a2949efcf7ce82d2f311cbe925a3f", "size": 642}, "kind": "sent", "previous": "a202a4045ebe570a8e2818e30b3d8c93889aa0b1880238a0822fcacd838a1587", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 256, "sha256": "d56467901e18fc3260677df95aeb35f01972825a55240b08ffecd0cfd2626955"} +{"at_utc": "2026-10-18T14:34:21.936+00:00", "data": {"http_status": 201, "ids": {"documentId": 61, "documentVersionId": 61, "embeddingJobId": 59, "fileObjectId": 59}, "outcome": "ACKNOWLEDGED", "request_id": "upload-c2f3833d4d4347e7a370f6ca4022c717"}, "kind": "response", "previous": "d56467901e18fc3260677df95aeb35f01972825a55240b08ffecd0cfd2626955", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 257, "sha256": "45f8cdd07fe74d9647f729d30f4f30cc574fe90c3d5280f7d91c801e654acf1e"} +{"at_utc": "2026-10-18T14:34:21.944+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.508, "http_status": 201, "ordinal": 59}, "kind": "upload_observed", "previous": "45f8cdd07fe74d9647f729d30f4f30cc574fe90c3d5280f7d91c801e654acf1e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 258, "sha256": "ad77e85df1d885f2c0a697e6463a4689570403b359bf0e912b52e43ab5c54404"} +{"at_utc": "2026-10-18T14:34:25.427+00:00", "data": {"cohort": "loss", "request_id": "upload-d2053ef0277749fbbacb44ff38feca0e", "sha256": "db9acd78e12caeeb5c26775c71672ef41bfea93a45c781a186523abd8f13068a", "size": 1046}, "kind": "sent", "previous": "ad77e85df1d885f2c0a697e6463a4689570403b359bf0e912b52e43ab5c54404", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 259, "sha256": "173bf200c0b1b1f3537f70cb0458b42354375b003ac56bb1d01e0d86df7233fc"} +{"at_utc": "2026-10-18T14:34:32.975+00:00", "data": {"http_status": 201, "ids": {"documentId": 62, "documentVersionId": 62, "embeddingJobId": 60, "fileObjectId": 60}, "outcome": "ACKNOWLEDGED", "request_id": "upload-d2053ef0277749fbbacb44ff38feca0e"}, "kind": "response", "previous": "173bf200c0b1b1f3537f70cb0458b42354375b003ac56bb1d01e0d86df7233fc", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 260, "sha256": "4632cdccab84e5f873d97e3108796e55f8586700ae4f7e7be741c198ab4c98b6"} +{"at_utc": "2026-10-18T14:34:32.980+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.548, "http_status": 201, "ordinal": 60}, "kind": "upload_observed", "previous": "4632cdccab84e5f873d97e3108796e55f8586700ae4f7e7be741c198ab4c98b6", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 261, "sha256": "29e5a217a666eeb3020ae28d426d91cc35cfe9609fc26f8299a3c7bf900fb738"} +{"at_utc": "2026-10-18T14:34:32.984+00:00", "data": {"completed_upload_requests": 60, "ordinal": 2, "scheduled_at_utc": "2026-10-18T14:34:31.420159+00:00", "start_lag_seconds": 1.578}, "kind": "offsite_schedule_observed", "previous": "29e5a217a666eeb3020ae28d426d91cc35cfe9609fc26f8299a3c7bf900fb738", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 262, "sha256": "cd0995a44594fb0365bdfe132940baab1362703bb42f7d67c199815badf608c8"} +{"at_utc": "2026-10-18T14:34:58.223+00:00", "data": {"fileObjectId": 34, "generation": "1792334098000268", "sha256": "4a1955b23633e09c64c251faf53a7f3db408e65c0d31e4131eac14e2e408b036", "size": 1046}, "kind": "original_preserved", "previous": "cd0995a44594fb0365bdfe132940baab1362703bb42f7d67c199815badf608c8", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 263, "sha256": "fd58ce7dca731d36b28e7ce743ab0076e89e780af911deef006297249afca2ae"} +{"at_utc": "2026-10-18T14:34:59.670+00:00", "data": {"fileObjectId": 35, "generation": "1792334099384913", "sha256": "d517ea588330b3a1009b2a65decaa649f2eb64041318067f7eca6f41310beb19", "size": 642}, "kind": "original_preserved", "previous": "fd58ce7dca731d36b28e7ce743ab0076e89e780af911deef006297249afca2ae", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 264, "sha256": "fcaccfc6f96a54263782f97fe80aa93df914b8a59e0f2a62f1d2319e6478458e"} +{"at_utc": "2026-10-18T14:35:00.979+00:00", "data": {"fileObjectId": 36, "generation": "1792334100779971", "sha256": "39f7e102ac80a9ba04150f8e357db6123441bdc7cafe5620dc163dda4661ae6f", "size": 1046}, "kind": "original_preserved", "previous": "fcaccfc6f96a54263782f97fe80aa93df914b8a59e0f2a62f1d2319e6478458e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 265, "sha256": "87708751fa1e111d735ebea4850b8eb9dfdb645279784cf544a7b5e0c664c2a5"} +{"at_utc": "2026-10-18T14:35:02.568+00:00", "data": {"fileObjectId": 37, "generation": "1792334102353834", "sha256": "2a1bfc66a9c58c43d8d51ff1c5391e58dd582fbc386733c41d7c3989fdce9731", "size": 642}, "kind": "original_preserved", "previous": "87708751fa1e111d735ebea4850b8eb9dfdb645279784cf544a7b5e0c664c2a5", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 266, "sha256": "f9217dfbd079c5e065b65ef3a5b217a428e5b4f08e057541c90d1faaf0f0bd35"} +{"at_utc": "2026-10-18T14:35:03.936+00:00", "data": {"fileObjectId": 38, "generation": "1792334103668017", "sha256": "0b3353283ad6d05cb497416736e7fdda10b57f9a0ffbfb2d316596365791b4b8", "size": 1046}, "kind": "original_preserved", "previous": "f9217dfbd079c5e065b65ef3a5b217a428e5b4f08e057541c90d1faaf0f0bd35", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 267, "sha256": "de3a667b8e8ad0172616fc21e1b57619bc8717f2f66e558186540804e15fc4d7"} +{"at_utc": "2026-10-18T14:35:05.322+00:00", "data": {"fileObjectId": 39, "generation": "1792334105072235", "sha256": "6fa3bf5896c7e74952fcd6935204ebf002cda8e6892bd4b34d19839e9849576d", "size": 642}, "kind": "original_preserved", "previous": "de3a667b8e8ad0172616fc21e1b57619bc8717f2f66e558186540804e15fc4d7", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 268, "sha256": "72fd6ef2abcd26e4745699de4f7eea955a97559a7561e046845c0cd641a1ebb1"} +{"at_utc": "2026-10-18T14:35:06.753+00:00", "data": {"fileObjectId": 40, "generation": "1792334106429281", "sha256": "604dda88d3af34bfe5fb2ecbd42e1466b7123367aabf2b6479f5187bbde8b1e5", "size": 1046}, "kind": "original_preserved", "previous": "72fd6ef2abcd26e4745699de4f7eea955a97559a7561e046845c0cd641a1ebb1", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 269, "sha256": "75411127d5f8e34490ff58f9d4872fe9737a4bde5ce3fe78d8035ceada1ca879"} +{"at_utc": "2026-10-18T14:35:08.057+00:00", "data": {"fileObjectId": 41, "generation": "1792334107767532", "sha256": "cb1defde74581e45e5606cfe0f948314533669d751d71149d905ac755a067425", "size": 642}, "kind": "original_preserved", "previous": "75411127d5f8e34490ff58f9d4872fe9737a4bde5ce3fe78d8035ceada1ca879", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 270, "sha256": "80d110614d5036bdc1fd62e5ee2bdc3e1bcb2f8faf60533e84c763187fe2ac8f"} +{"at_utc": "2026-10-18T14:35:09.382+00:00", "data": {"fileObjectId": 42, "generation": "1792334109163978", "sha256": "cca8e204eaa0a10bb320b2979da370eb64ed2bbc54d9ffe190b3bd0206ba97e5", "size": 1046}, "kind": "original_preserved", "previous": "80d110614d5036bdc1fd62e5ee2bdc3e1bcb2f8faf60533e84c763187fe2ac8f", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 271, "sha256": "1c68af76d39b51deffeefb4add8c4a718ac1cecf50c22b0c0c620208ba3757cd"} +{"at_utc": "2026-10-18T14:35:10.727+00:00", "data": {"fileObjectId": 43, "generation": "1792334110385998", "sha256": "2561e3f2e810a8a4c7c06b6f0d03578d79f27889387f1ad53fde39f5070b63a5", "size": 642}, "kind": "original_preserved", "previous": "1c68af76d39b51deffeefb4add8c4a718ac1cecf50c22b0c0c620208ba3757cd", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 272, "sha256": "0fa9963db92aa53b7efcdfea7a46c64e75cadcd685e1593a45ce35cb408a2cf4"} +{"at_utc": "2026-10-18T14:35:12.058+00:00", "data": {"fileObjectId": 44, "generation": "1792334111853252", "sha256": "7dcaaa488971cae03bfd16b70021146afe24dc803d51107bdb4925b1613af5e8", "size": 1046}, "kind": "original_preserved", "previous": "0fa9963db92aa53b7efcdfea7a46c64e75cadcd685e1593a45ce35cb408a2cf4", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 273, "sha256": "a062f3f6b29f3cfcdf98e567ad987336350a47a20a12cdaa7fb6e75f93a87dce"} +{"at_utc": "2026-10-18T14:35:13.352+00:00", "data": {"fileObjectId": 45, "generation": "1792334113141879", "sha256": "bac1098d29cc26b2ef7f606bd8f4942f864358d57c99dce5166ea4db3287933e", "size": 642}, "kind": "original_preserved", "previous": "a062f3f6b29f3cfcdf98e567ad987336350a47a20a12cdaa7fb6e75f93a87dce", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 274, "sha256": "127e3cee80d2130184278d19390e1716f3331239f1f876c8008f0b06af145261"} +{"at_utc": "2026-10-18T14:35:14.872+00:00", "data": {"fileObjectId": 46, "generation": "1792334114523730", "sha256": "ed8f4d573635655183f52c0a3a7ad5eab7325225399f46ea41372cc7907cefd3", "size": 1046}, "kind": "original_preserved", "previous": "127e3cee80d2130184278d19390e1716f3331239f1f876c8008f0b06af145261", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 275, "sha256": "5c8812b14a4e49da7d86a11aa086c64d1c86ba5cebadca8a108c7bfdc8fbc0d9"} +{"at_utc": "2026-10-18T14:35:16.200+00:00", "data": {"fileObjectId": 47, "generation": "1792334115989024", "sha256": "78d4567ad81362950c53cdaf6310b3c6be4280a500ce8422c4473f0974fa178d", "size": 642}, "kind": "original_preserved", "previous": "5c8812b14a4e49da7d86a11aa086c64d1c86ba5cebadca8a108c7bfdc8fbc0d9", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 276, "sha256": "829a7e81dc4852b9672b9010fc1ca908104075363a6944774a7a8d75a7fedec6"} +{"at_utc": "2026-10-18T14:35:17.758+00:00", "data": {"fileObjectId": 48, "generation": "1792334117554101", "sha256": "c3ef81a8b35b77101fa3c1caba153f8dac4c76b0d6ab56e917bbc9ad88440296", "size": 1046}, "kind": "original_preserved", "previous": "829a7e81dc4852b9672b9010fc1ca908104075363a6944774a7a8d75a7fedec6", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 277, "sha256": "36f5055d44e6350680cf6f366a6bcf24cbe42267ab7f08ddcc75923eb1c987b0"} +{"at_utc": "2026-10-18T14:35:19.195+00:00", "data": {"fileObjectId": 49, "generation": "1792334118985900", "sha256": "6bd1b2aa0f2dd0e8547740529e00c66252bb39153bac639ed6ab0f1655fe3673", "size": 642}, "kind": "original_preserved", "previous": "36f5055d44e6350680cf6f366a6bcf24cbe42267ab7f08ddcc75923eb1c987b0", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 278, "sha256": "3addabd7e01170855bacb2373072e638ce7f836cfa1bdc60e5eee8c436ba1bed"} +{"at_utc": "2026-10-18T14:35:20.476+00:00", "data": {"fileObjectId": 50, "generation": "1792334120265082", "sha256": "5c5f78c20f08e1cc1bfe3d8fdecdd7fe41d887bbd6365ddeb02755131ac7e266", "size": 1046}, "kind": "original_preserved", "previous": "3addabd7e01170855bacb2373072e638ce7f836cfa1bdc60e5eee8c436ba1bed", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 279, "sha256": "b82c4d38d6552a1ddbbfe0d43c1e58fc9adf0e3b16c3962694e0ff9b3f2a3245"} +{"at_utc": "2026-10-18T14:35:21.772+00:00", "data": {"fileObjectId": 51, "generation": "1792334121557537", "sha256": "7d2dc24cbce3133caeebe9ea1b3f17d7053f1d73083527721b521416de72449a", "size": 642}, "kind": "original_preserved", "previous": "b82c4d38d6552a1ddbbfe0d43c1e58fc9adf0e3b16c3962694e0ff9b3f2a3245", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 280, "sha256": "d4e8bd21fa280af5f0b22b742122b2e2f69dc1b9bedb3e61d0bb104ccb33dfc0"} +{"at_utc": "2026-10-18T14:35:23.315+00:00", "data": {"fileObjectId": 52, "generation": "1792334123050031", "sha256": "bab62716f6a42b0766db18babe1e4d649e24be3e9e731e6f2553f16e58ec011e", "size": 1046}, "kind": "original_preserved", "previous": "d4e8bd21fa280af5f0b22b742122b2e2f69dc1b9bedb3e61d0bb104ccb33dfc0", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 281, "sha256": "9a865716b3fd7fc78ca5d1cabd8822113eeebe059e0d0e231ade7cdcdbe04617"} +{"at_utc": "2026-10-18T14:35:24.783+00:00", "data": {"fileObjectId": 53, "generation": "1792334124486623", "sha256": "7c1518f5b75aa22b3581da98e94d958e94a4a8bf146ab650a7a245964401c004", "size": 642}, "kind": "original_preserved", "previous": "9a865716b3fd7fc78ca5d1cabd8822113eeebe059e0d0e231ade7cdcdbe04617", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 282, "sha256": "97aade8d6f5206cb59981efad884a7236b9908f39dfa81d94e4183351ac84dbc"} +{"at_utc": "2026-10-18T14:35:26.134+00:00", "data": {"fileObjectId": 54, "generation": "1792334125803407", "sha256": "71f499d0499d9204f28495bf65fb23854f288a8d744a6dbc38c405bc34b319ba", "size": 1046}, "kind": "original_preserved", "previous": "97aade8d6f5206cb59981efad884a7236b9908f39dfa81d94e4183351ac84dbc", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 283, "sha256": "b1524b2454d75b1da954876ef9bb52a2bfc4d1496ca0a4da9d344c372b0243e6"} +{"at_utc": "2026-10-18T14:35:27.444+00:00", "data": {"fileObjectId": 55, "generation": "1792334127146475", "sha256": "cbe79087c3e591bebb072292ef7c6d41d62f119c03eee75af19a24b7afdb80fe", "size": 642}, "kind": "original_preserved", "previous": "b1524b2454d75b1da954876ef9bb52a2bfc4d1496ca0a4da9d344c372b0243e6", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 284, "sha256": "c238fc880d7215c720eb2276a7ec86693166129bdca7e426bdae982890f11e9b"} +{"at_utc": "2026-10-18T14:35:28.648+00:00", "data": {"fileObjectId": 56, "generation": "1792334128442978", "sha256": "5b6caf7ba08fc075fc7748fec448cbf746b6af1f3c6ae4c0548568ee64f7810a", "size": 1046}, "kind": "original_preserved", "previous": "c238fc880d7215c720eb2276a7ec86693166129bdca7e426bdae982890f11e9b", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 285, "sha256": "d4888e976403780e0cdf83868d3d6d5354ab58401e24d098d327ce7ba19144f1"} +{"at_utc": "2026-10-18T14:35:30.027+00:00", "data": {"fileObjectId": 57, "generation": "1792334129753173", "sha256": "6acc5225c8a998e1e9d79e823dc96f1c554330c26e68685da025e17f4337657a", "size": 642}, "kind": "original_preserved", "previous": "d4888e976403780e0cdf83868d3d6d5354ab58401e24d098d327ce7ba19144f1", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 286, "sha256": "ee54505a4a2f2a4f25e132b6e1978b85e06a9a74a76304f695cecff0e3ca52c0"} +{"at_utc": "2026-10-18T14:35:31.224+00:00", "data": {"fileObjectId": 58, "generation": "1792334131037142", "sha256": "2bb8c341cf0b163cbd7891f7c0e0ba1e5195136f958c209a12251bd40f114a32", "size": 1046}, "kind": "original_preserved", "previous": "ee54505a4a2f2a4f25e132b6e1978b85e06a9a74a76304f695cecff0e3ca52c0", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 287, "sha256": "3e4dac8852f43eb777d69ccae1e89546e170502a02710460406a134b6d8cfcb2"} +{"at_utc": "2026-10-18T14:35:32.503+00:00", "data": {"fileObjectId": 59, "generation": "1792334132300936", "sha256": "c61b92f0cb27263620175ac11f2ab02ace5a2949efcf7ce82d2f311cbe925a3f", "size": 642}, "kind": "original_preserved", "previous": "3e4dac8852f43eb777d69ccae1e89546e170502a02710460406a134b6d8cfcb2", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 288, "sha256": "7032914bc79c8b03c0eab607af615028e6ad168f7662f7ba097635f41e7edb21"} +{"at_utc": "2026-10-18T14:35:34.065+00:00", "data": {"fileObjectId": 60, "generation": "1792334133812207", "sha256": "db9acd78e12caeeb5c26775c71672ef41bfea93a45c781a186523abd8f13068a", "size": 1046}, "kind": "original_preserved", "previous": "7032914bc79c8b03c0eab607af615028e6ad168f7662f7ba097635f41e7edb21", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 289, "sha256": "41b753af994bb652a9a03ee863f5c380ff1e8ced858e6e3c49d418dad8ab7ea9"} +{"at_utc": "2026-10-18T14:35:38.372+00:00", "data": {"generation": "1792334136908930", "name": "000000060000000000000030", "sha256": "459e502dfa8b809cc0e9e758f193f956bb19ffcddc4da19560c2e1f41a7d05d3", "size": 16777216}, "kind": "wal_preserved", "previous": "41b753af994bb652a9a03ee863f5c380ff1e8ced858e6e3c49d418dad8ab7ea9", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 290, "sha256": "f2dc7d3e9203ce7d79087db88a9e3383677c999db5faea03a8cafdfc9ce93cd5"} +{"at_utc": "2026-10-18T14:35:40.297+00:00", "data": {"archive_attempts": 2, "covered_acknowledged_requests": 60, "ordinal": 2, "remote_bytes_verified": true, "required_wal": "000000060000000000000030"}, "kind": "wal_barrier_verified", "previous": "f2dc7d3e9203ce7d79087db88a9e3383677c999db5faea03a8cafdfc9ce93cd5", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 291, "sha256": "fd8c1fbcb3280a6b2dfdb401e8b5bd8fd6ea67172eff86622b6d59697b15e798"} +{"at_utc": "2026-10-18T14:35:40.306+00:00", "data": {"complete_wals": 7, "elapsed_seconds": 67.315, "ordinal": 2, "preserved_originals": 60}, "kind": "offsite_round_completed", "previous": "fd8c1fbcb3280a6b2dfdb401e8b5bd8fd6ea67172eff86622b6d59697b15e798", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 292, "sha256": "278e91bfdb7db39b3e536951fae943553058390ca10c27fac6e24a3e09dedc9e"} +{"at_utc": "2026-10-18T14:35:40.314+00:00", "data": {"cohort": "loss", "request_id": "upload-4157706187474b57a8898d172289f611", "sha256": "21a08030d2b61d0a31bf022cd00cb2e4cd564aed2d7e7388303b662b146618e0", "size": 642}, "kind": "sent", "previous": "278e91bfdb7db39b3e536951fae943553058390ca10c27fac6e24a3e09dedc9e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 293, "sha256": "d59d57c2193715d34f59bd3871df097d5e40af14659c42977c6267d0c7eb711e"} +{"at_utc": "2026-10-18T14:35:47.908+00:00", "data": {"http_status": 201, "ids": {"documentId": 63, "documentVersionId": 63, "embeddingJobId": 61, "fileObjectId": 61}, "outcome": "ACKNOWLEDGED", "request_id": "upload-4157706187474b57a8898d172289f611"}, "kind": "response", "previous": "d59d57c2193715d34f59bd3871df097d5e40af14659c42977c6267d0c7eb711e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 294, "sha256": "6f80d239036b09820c7ec4444d462d9c36828370a9ad03440d1b48df156e48c7"} +{"at_utc": "2026-10-18T14:35:47.919+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.595, "http_status": 201, "ordinal": 61}, "kind": "upload_observed", "previous": "6f80d239036b09820c7ec4444d462d9c36828370a9ad03440d1b48df156e48c7", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 295, "sha256": "1f40352a1736bf34f7131c69bd4bddf5aa2495e3c152b8e11655d85109f7f36d"} +{"at_utc": "2026-10-18T14:35:47.929+00:00", "data": {"cohort": "loss", "request_id": "upload-0c1e3ccc285348759c881b404b7d6e5c", "sha256": "7b96ffa0d4b2971fc9f55b28df9b9475e96e77fcdec22d1a1d38308dc6cbf009", "size": 1046}, "kind": "sent", "previous": "1f40352a1736bf34f7131c69bd4bddf5aa2495e3c152b8e11655d85109f7f36d", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 296, "sha256": "af3aaf9f15f8a32649ed47421dcaa89a4371bd8254e0646002c7e1e917844ae1"} +{"at_utc": "2026-10-18T14:35:54.975+00:00", "data": {"http_status": 201, "ids": {"documentId": 64, "documentVersionId": 64, "embeddingJobId": 62, "fileObjectId": 62}, "outcome": "ACKNOWLEDGED", "request_id": "upload-0c1e3ccc285348759c881b404b7d6e5c"}, "kind": "response", "previous": "af3aaf9f15f8a32649ed47421dcaa89a4371bd8254e0646002c7e1e917844ae1", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 297, "sha256": "40ea595b9fb91aee34780342091cadfc53989918df8bf45eb7d0d494b6e31084"} +{"at_utc": "2026-10-18T14:35:54.984+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.046, "http_status": 201, "ordinal": 62}, "kind": "upload_observed", "previous": "40ea595b9fb91aee34780342091cadfc53989918df8bf45eb7d0d494b6e31084", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 298, "sha256": "4bbf157e3a1cf99a586b5d08c42c884ccb84bdf8e0dab168624b011bc68467c0"} +{"at_utc": "2026-10-18T14:35:54.997+00:00", "data": {"cohort": "loss", "request_id": "upload-5b9715d9f6174a6d90fb341f48834b28", "sha256": "586d3db37c5e2635a602a28351b326a2394c640e8f4fb457c6db6940be81c9e0", "size": 642}, "kind": "sent", "previous": "4bbf157e3a1cf99a586b5d08c42c884ccb84bdf8e0dab168624b011bc68467c0", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 299, "sha256": "905971db603ab24c3b74dbf73620fbd189d40e237c0894c0febf8f857745c880"} +{"at_utc": "2026-10-18T14:36:02.057+00:00", "data": {"http_status": 201, "ids": {"documentId": 65, "documentVersionId": 65, "embeddingJobId": 63, "fileObjectId": 63}, "outcome": "ACKNOWLEDGED", "request_id": "upload-5b9715d9f6174a6d90fb341f48834b28"}, "kind": "response", "previous": "905971db603ab24c3b74dbf73620fbd189d40e237c0894c0febf8f857745c880", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 300, "sha256": "91a6d80b57ea42268f148ca2cb46c65e5ae9004c3aea2f3a4e5f935825c96170"} +{"at_utc": "2026-10-18T14:36:02.067+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.061, "http_status": 201, "ordinal": 63}, "kind": "upload_observed", "previous": "91a6d80b57ea42268f148ca2cb46c65e5ae9004c3aea2f3a4e5f935825c96170", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 301, "sha256": "64c4fe1cdf54ab7116e7abaf8eb820a911d6668c8a775985a0f05dfa655848f1"} +{"at_utc": "2026-10-18T14:36:02.078+00:00", "data": {"cohort": "loss", "request_id": "upload-fd3c8ddd7e4f49dba2cffecdca1edd91", "sha256": "e0ad83edef695cf30c3b22e94fc3312d4254d1a57c452d1c60331dc468fa4f6f", "size": 1046}, "kind": "sent", "previous": "64c4fe1cdf54ab7116e7abaf8eb820a911d6668c8a775985a0f05dfa655848f1", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 302, "sha256": "24781a29911b03d3ccbde9b0a25e74ef197ab73f4c79596ab4a1a08e05cf4f2d"} +{"at_utc": "2026-10-18T14:36:09.391+00:00", "data": {"http_status": 201, "ids": {"documentId": 66, "documentVersionId": 66, "embeddingJobId": 64, "fileObjectId": 64}, "outcome": "ACKNOWLEDGED", "request_id": "upload-fd3c8ddd7e4f49dba2cffecdca1edd91"}, "kind": "response", "previous": "24781a29911b03d3ccbde9b0a25e74ef197ab73f4c79596ab4a1a08e05cf4f2d", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 303, "sha256": "97cf4dd80a0e823f7da3bc69074999dedd18aea824b78c31c4f0fa93666e7d57"} +{"at_utc": "2026-10-18T14:36:09.400+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.313, "http_status": 201, "ordinal": 64}, "kind": "upload_observed", "previous": "97cf4dd80a0e823f7da3bc69074999dedd18aea824b78c31c4f0fa93666e7d57", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 304, "sha256": "113d49c7c39ea9efbe06a6b14a643306d1ef77c64f00a5a7e3b6301b065e0cbd"} +{"at_utc": "2026-10-18T14:36:09.411+00:00", "data": {"cohort": "loss", "request_id": "upload-0d79ad5c61194016bcf99cf56a4ecc5d", "sha256": "8dbb07993d933fcecd2b483d9e75bc16e0c1ff25827b896f4dbea0d08f292bf5", "size": 642}, "kind": "sent", "previous": "113d49c7c39ea9efbe06a6b14a643306d1ef77c64f00a5a7e3b6301b065e0cbd", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 305, "sha256": "67401da1565bdabb5a7b0ee9c81bbe0baaa19f2be76ca2877557ae5ac2f0a9f1"} +{"at_utc": "2026-10-18T14:36:16.477+00:00", "data": {"http_status": 201, "ids": {"documentId": 67, "documentVersionId": 67, "embeddingJobId": 65, "fileObjectId": 65}, "outcome": "ACKNOWLEDGED", "request_id": "upload-0d79ad5c61194016bcf99cf56a4ecc5d"}, "kind": "response", "previous": "67401da1565bdabb5a7b0ee9c81bbe0baaa19f2be76ca2877557ae5ac2f0a9f1", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 306, "sha256": "cabfe0c71a158ea9492d956c39f31fb7febef9b2548793fe819e1e27d92f44e2"} +{"at_utc": "2026-10-18T14:36:16.485+00:00", "data": {"cohort": "loss", "elapsed_seconds": 7.066, "http_status": 201, "ordinal": 65}, "kind": "upload_observed", "previous": "cabfe0c71a158ea9492d956c39f31fb7febef9b2548793fe819e1e27d92f44e2", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 307, "sha256": "c5d7b404f18073a0c8032518f20993a6b82c0163a9401e3150b93831592d7e9d"} +{"at_utc": "2026-10-18T14:36:37.353+00:00", "data": {"inputs_sha256": "6bda360bd8f4d2a125f65a8d145adfea76430dcfd231dd6ddc0284729b006606", "wal_objects": 7}, "kind": "cutoff", "previous": "c5d7b404f18073a0c8032518f20993a6b82c0163a9401e3150b93831592d7e9d", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 308, "sha256": "d9cf6c8f534e0ba45327fefd94f3c1f7a170119143690691b996e39402ab20a7"} +{"at_utc": "2026-10-18T14:36:37.357+00:00", "data": {"cloud_bucket_iam_denial_tested": false, "source_api_stop_started_at_utc": "2026-10-18T14:36:26.952847+00:00", "source_data_paths_stopped_at_utc": "2026-10-18T14:36:37.353558+00:00", "source_gcs_guard_enforced": true, "source_listener_ports_closed": 2}, "kind": "fault_boundary", "previous": "d9cf6c8f534e0ba45327fefd94f3c1f7a170119143690691b996e39402ab20a7", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 309, "sha256": "2e948fef8cd1261b033722fa8089c4887b26228e95b3d5f5ab854b9e6eb94fb0"} +{"at_utc": "2026-10-18T14:36:37.366+00:00", "data": {}, "kind": "restore_started", "previous": "2e948fef8cd1261b033722fa8089c4887b26228e95b3d5f5ab854b9e6eb94fb0", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 310, "sha256": "1e1895dec27f645684632d9879fa5dbe94b2e2fda92b33c33e893cb3e11e5619"} +{"at_utc": "2026-10-18T14:37:45.123+00:00", "data": {"frozen_identity_used": true, "guard_seconds": 1800, "licensed_opensql_started": true, "network": "none", "original_catalog_mounted": false, "stage_archive_sha256": "beafa3cb6809d16cdd10488a7479c18e4e711e8c47e3ea14b5c5f3a3a5260e18"}, "kind": "offline_restore_observed", "previous": "1e1895dec27f645684632d9879fa5dbe94b2e2fda92b33c33e893cb3e11e5619", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 311, "sha256": "420366d7b342084c9fc5b36c0350d61e9ffad9d567828687984b147fd949d10c"} +{"at_utc": "2026-10-18T14:37:45.133+00:00", "data": {"elapsed_seconds": 67.629}, "kind": "restored", "previous": "420366d7b342084c9fc5b36c0350d61e9ffad9d567828687984b147fd949d10c", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 312, "sha256": "1c50e3c2d58472acdd8913370b4197ffab301027e114eb37a20e4026980537ca"} +{"at_utc": "2026-10-18T14:38:17.218+00:00", "data": {"acknowledged": 65, "activation_allowed": true, "audit_sha256": "dcd22a73b4be1794be1ba97cdb6efe5151c056ae3210c2678e14c2f92afd5a44", "lost_or_incomplete_acknowledged": 5, "snapshot_sha256": "74e98ed51cca1798c6249d89942d14c153ed46ace2aff37ce2946ad2759fff3c"}, "kind": "initial_audit", "previous": "1c50e3c2d58472acdd8913370b4197ffab301027e114eb37a20e4026980537ca", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 313, "sha256": "ae5626f90bbd7f9bcea85108fbc957b639fb2c1ccffc225a4c3e8b7c98b790fc"} +{"at_utc": "2026-10-18T14:38:17.285+00:00", "data": {"checked_before_worker": true, "covered_acknowledged_requests": 60, "covered_missing": 0, "covered_restored": 60, "required_wal": "000000060000000000000030", "tail_missing": 5, "tail_requests": 5, "tail_restored": 0, "verdict": "PASS"}, "kind": "checkpoint_coverage_audited", "previous": "ae5626f90bbd7f9bcea85108fbc957b639fb2c1ccffc225a4c3e8b7c98b790fc", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 314, "sha256": "c1155007d1f5016fa301241bfbe412bf388c48d51cf0dfe9bc95f7f5ac30691e"} +{"at_utc": "2026-10-18T14:38:27.339+00:00", "data": {"initial_audit_already_durable": true, "public_bucket_changed": false}, "kind": "isolated_bucket_rebound", "previous": "c1155007d1f5016fa301241bfbe412bf388c48d51cf0dfe9bc95f7f5ac30691e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 315, "sha256": "54458caef7c86214a0b00a8c05c2efb79411d3b36a2251be86030dba155a256a"} +{"at_utc": "2026-10-18T14:38:35.555+00:00", "data": {"private_endpoint_omitted": true, "purpose": "recovery"}, "kind": "tunnel_ready", "previous": "54458caef7c86214a0b00a8c05c2efb79411d3b36a2251be86030dba155a256a", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 316, "sha256": "437f7a80b256e1ab90c20c6f677b7c12e14cc5414e6d75b1a7d2e37282925c98"} +{"at_utc": "2026-10-18T14:39:00.294+00:00", "data": {"purpose": "recovery", "worker_enabled": true}, "kind": "backend_ready", "previous": "437f7a80b256e1ab90c20c6f677b7c12e14cc5414e6d75b1a7d2e37282925c98", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 317, "sha256": "1c5bb20ee581bb3c719eb2378b5cce8443a4d4f23a8ea344a53c4514f05aefd1"} +{"at_utc": "2026-10-18T14:39:01.753+00:00", "data": {"account_recreated": false, "http_status": 200}, "kind": "restored_login_observed", "previous": "1c5bb20ee581bb3c719eb2378b5cce8443a4d4f23a8ea344a53c4514f05aefd1", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 318, "sha256": "313010099505035d9952fc1cf2b16a2ca946b8c36878784e2c5f58cb116baceb"} +{"at_utc": "2026-10-18T14:39:01.769+00:00", "data": {}, "kind": "activated", "previous": "313010099505035d9952fc1cf2b16a2ca946b8c36878784e2c5f58cb116baceb", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 319, "sha256": "e5985dc22e9443070819a627769b80d0106ff34bb254304711ae8582490b316d"} +{"at_utc": "2026-10-18T14:39:09.857+00:00", "data": {"function": "download", "passed": true, "samples": 4}, "kind": "functional_observed", "previous": "e5985dc22e9443070819a627769b80d0106ff34bb254304711ae8582490b316d", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 320, "sha256": "2b2cf2a35aa37e929ebb472cc5310dabed5082604bac52bfe98412a8ffb0d515"} +{"at_utc": "2026-10-18T14:39:25.407+00:00", "data": {"expected_document_in_top_k": true, "function": "search", "samples": 1, "top_k": 20}, "kind": "functional_observed", "previous": "2b2cf2a35aa37e929ebb472cc5310dabed5082604bac52bfe98412a8ffb0d515", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 321, "sha256": "99512df5565bab901cb5991c80cd1a17d7bd34caaa8e0793f80090aa1b8d37ab"} +{"at_utc": "2026-10-18T14:39:26.561+00:00", "data": {"ordinal": 5, "phase": "recovery", "state": "PROCESSING"}, "kind": "index_state", "previous": "99512df5565bab901cb5991c80cd1a17d7bd34caaa8e0793f80090aa1b8d37ab", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 322, "sha256": "fe848b59935f042f7984f27e9dedff12c84a15c01f09990cd28727cdb28a92c6"} +{"at_utc": "2026-10-18T14:40:10.810+00:00", "data": {"ordinal": 5, "phase": "recovery", "state": "INDEXED"}, "kind": "index_state", "previous": "fe848b59935f042f7984f27e9dedff12c84a15c01f09990cd28727cdb28a92c6", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 323, "sha256": "e60e4c44cb7ca316c3c62f22739a2ee5b7f3841b848636d792c8dc9b11d2e0c7"} +{"at_utc": "2026-10-18T14:40:21.322+00:00", "data": {"attempt_count": 2, "function": "job_resume", "job_count_for_version": 1, "job_status": "INDEXED"}, "kind": "functional_observed", "previous": "e60e4c44cb7ca316c3c62f22739a2ee5b7f3841b848636d792c8dc9b11d2e0c7", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 324, "sha256": "5bf30622f2b4752898747cd15f6e13592cea540076e97ee148feccfa85e965a1"} +{"at_utc": "2026-10-18T14:40:28.374+00:00", "data": {"cohort": "functional", "elapsed_seconds": 7.018, "http_status": 201, "ordinal": 1001}, "kind": "upload_observed", "previous": "5bf30622f2b4752898747cd15f6e13592cea540076e97ee148feccfa85e965a1", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 325, "sha256": "2dfc9fa95b176866b23a710d521c6243bad9622627a235024ef54c0fb056507e"} +{"at_utc": "2026-10-18T14:40:29.865+00:00", "data": {"ordinal": 1001, "phase": "recovery", "state": "PENDING"}, "kind": "index_state", "previous": "2dfc9fa95b176866b23a710d521c6243bad9622627a235024ef54c0fb056507e", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 326, "sha256": "ec2dbe622e6c33db6d5ceaac59f694e9931fdbd3d03aa8b097fa2f6519e1d017"} +{"at_utc": "2026-10-18T14:51:13.312+00:00", "data": {"ordinal": 1001, "phase": "recovery", "state": "PROCESSING"}, "kind": "index_state", "previous": "ec2dbe622e6c33db6d5ceaac59f694e9931fdbd3d03aa8b097fa2f6519e1d017", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 327, "sha256": "cebcfcc24f9fcaf03b70bc006cc36759322baef872f6f6cc183c46757e8ee1d0"} +{"at_utc": "2026-10-18T14:51:37.671+00:00", "data": {"ordinal": 1001, "phase": "recovery", "state": "INDEXED"}, "kind": "index_state", "previous": "cebcfcc24f9fcaf03b70bc006cc36759322baef872f6f6cc183c46757e8ee1d0", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 328, "sha256": "ba354e715768dab3f3ccfd7da6bc29dbe79abff8598a21154a917bc952fcc108"} +{"at_utc": "2026-10-18T14:51:39.783+00:00", "data": {"from_cutoff_seconds": 152.5, "function": "download", "recovered_at_utc": "2026-10-18T14:39:09.852552+00:00", "samples": 4}, "kind": "functional", "previous": "ba354e715768dab3f3ccfd7da6bc29dbe79abff8598a21154a917bc952fcc108", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 329, "sha256": "5b55e35335afbd74080cf525f575a605e2d73b918509020dc87874a8d09304f5"} +{"at_utc": "2026-10-18T14:51:39.789+00:00", "data": {"from_cutoff_seconds": 168.047, "function": "search", "recovered_at_utc": "2026-10-18T14:39:25.400051+00:00", "samples": 1}, "kind": "functional", "previous": "5b55e35335afbd74080cf525f575a605e2d73b918509020dc87874a8d09304f5", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 330, "sha256": "6f586398c994dfff028d2dde2219cc573f41e653bb81675ebe2cb6c555e7c836"} +{"at_utc": "2026-10-18T14:51:39.795+00:00", "data": {"from_cutoff_seconds": 223.965, "function": "job_resume", "recovered_at_utc": "2026-10-18T14:40:21.317855+00:00", "samples": 1}, "kind": "functional", "previous": "6f586398c994dfff028d2dde2219cc573f41e653bb81675ebe2cb6c555e7c836", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 331, "sha256": "ee6ce15123c9bccfed8a26d09b1a2c2e2394347b4bdb2d1ae4167317e14d44a0"} +{"at_utc": "2026-10-18T14:51:39.802+00:00", "data": {"from_cutoff_seconds": 231.023, "function": "new_upload", "recovered_at_utc": "2026-10-18T14:40:28.376043+00:00", "samples": 1}, "kind": "functional", "previous": "ee6ce15123c9bccfed8a26d09b1a2c2e2394347b4bdb2d1ae4167317e14d44a0", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 332, "sha256": "8b5c1fdcf6d1f6cbd1b7343c783fc20a7d5b044d5dbc440622ac5c96e04c99a1"} +{"at_utc": "2026-10-18T14:51:39.807+00:00", "data": {"from_cutoff_seconds": 900.319, "function": "new_indexing", "recovered_at_utc": "2026-10-18T14:51:37.672099+00:00", "samples": 1}, "kind": "functional", "previous": "8b5c1fdcf6d1f6cbd1b7343c783fc20a7d5b044d5dbc440622ac5c96e04c99a1", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 333, "sha256": "9d61c69bcba1c8b64e1254a16ecbbe475503cd6fa27c3da0a36e4a30f0f3ce49"} +{"at_utc": "2026-10-18T14:51:39.811+00:00", "data": {"from_cutoff_seconds": 902.401, "function": "new_download", "recovered_at_utc": "2026-10-18T14:51:39.754136+00:00", "samples": 1}, "kind": "functional", "previous": "9d61c69bcba1c8b64e1254a16ecbbe475503cd6fa27c3da0a36e4a30f0f3ce49", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 334, "sha256": "6c97e444017c559c133b9b01c0efbb37bb3d76ae4f62b22f5f42bda081b14b53"} +{"at_utc": "2026-10-18T14:52:18.827+00:00", "data": {"audit_sha256": "dcd22a73b4be1794be1ba97cdb6efe5151c056ae3210c2678e14c2f92afd5a44", "consistency": {"checked_documents": 61, "duplicate_chunks": 0, "duplicate_embeddings": 0, "incomplete_jobs": 0, "missing_embeddings": 0}, "snapshot_sha256": "fc15ac52eeaaec8eaaa072e2bbf7da27899dd569f02e6d45e1aca5283f77b8c4"}, "kind": "final_audit", "previous": "6c97e444017c559c133b9b01c0efbb37bb3d76ae4f62b22f5f42bda081b14b53", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 335, "sha256": "065c27b0971c286a3ff2345e038a31ba3da15d0653f556152092f4272fb6caaa"} +{"at_utc": "2026-10-18T14:53:51.299+00:00", "data": {"owned_resources_removed": true, "public_config_preserved": true, "source_cluster_restored": true}, "kind": "cleanup", "previous": "065c27b0971c286a3ff2345e038a31ba3da15d0653f556152092f4272fb6caaa", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 336, "sha256": "9d2309e77e9ea337840d4d7ce7c30498d199d77abe05478db48f84d1e6cc3e84"} +{"at_utc": "2026-10-18T14:53:51.322+00:00", "data": {"execution": "PASS", "summary_sha256": "7917be9d18923da0c98d4a8f1d1e10725e8bc68ce8ced8db0e6b6d6176c41b65"}, "kind": "finished", "previous": "9d2309e77e9ea337840d4d7ce7c30498d199d77abe05478db48f84d1e6cc3e84", "run_id": "DR-RESTORE-20261018-FRESHNESS1", "sequence": 337, "sha256": "7aaa5d3507f8660331ff7f37d43574a9bb833944bec75ef121c86cecf3db031c"} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/final-audit.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/final-audit.json new file mode 100644 index 00000000..30b13311 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/final-audit.json @@ -0,0 +1,465 @@ +{ + "acknowledged": 65, + "activation_allowed": true, + "http_error": 0, + "lost_or_incomplete_acknowledged": 5, + "requests": [ + { + "ack_at_utc": "2026-10-18T14:16:35.002+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0296462e118a428da1b889a1310a9f30", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:17:14.038+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ce6a718bdd0f4ace83465acdb3f9d275", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:17:45.307+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-b1f116f44ea449fb90474ed3ffbde583", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:18:19.856+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-123273586c1a4fa0a37270b2550be058", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:20:09.511+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-6e7a22eea289466da0fbb8865647301b", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:24:39.407+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-fb2dd8207fa647fa9d80ffdf4c348c68", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:24:49.127+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-432fd889599b4f88b0d24d6593996a73", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:00.168+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-4aa4c55b25d14d538fcaff279600fda9", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:11.407+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-c8826fde38704624ae5ca21b04789532", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:22.257+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-57d7dce5ad8b4ee2adcf905c04a6f654", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:34.475+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e4d2871c1e404638bab746d1b07f9797", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:45.185+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e584c125eb7e4a5dbe43164f09613254", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:55.924+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e97831e4ebdf4f0398b54d8845adcd49", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:06.927+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e0ed270f022244c19fb49c554ac6ef5b", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:18.268+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-815f489bc96c48d187abac925f3271f4", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:28.971+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e34da01ee9474549b9002f280a723de1", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:39.772+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-5cd702f34097434ab4544b79573c47fc", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:50.450+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a7c7983377aa4cb38474c2c4b639d200", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:02.777+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-8fe5e921c6a64f2bafb9f5b24e23345d", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:12.730+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a7fbd0d4358842d99a18392ea28d5ada", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:25.101+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-52d582fc9eb64427845662582d8a3bbb", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:35.059+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-5cbd3325ea7e4b97b4db3ade4fc2fa7a", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:45.864+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-8af4ad876cd043f0a906915c5b5c9e76", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:57.382+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a2a79e66098444ce812e0e7f2b55741c", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:07.877+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-f35ec7a4a77b466186a1bb8faa995b8f", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:18.857+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-7b83a8a9b01544af88bbec7e9e5a8b94", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:30.164+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-106767bf29ce4751a114beffed1dcf5d", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:40.871+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-c2ec085d264a49c9ab6f691c6ae4aa9b", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:51.926+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-cef30e0526c24946b07bc80888bd0290", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:03.139+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-496f0a710ecb4981a3a644fdea27a2a5", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:13.966+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-3f79f77ddbd24cf1a9f682abcf928cb6", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:25.157+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-63141b14d77e4900965f43624358e6ce", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:36.265+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-52a198f8bd8a4c1f832134222e5505c5", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:30:51.420+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-d029610fe4234219b1db66f348b7b114", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:30:59.249+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ec97bd5c298f483d9cf98729cea26940", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:06.618+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0b7cf70ddca94790b02c8e8024fb040d", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:13.944+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-df6017955c5447b9bf097221f6a7d8be", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:22.044+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ba75b05d8e5d477abb3d2a1180a95077", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:29.191+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a5be52472400424db8f52ed7a54c2e83", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:36.338+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-eb8fb6803498447a94b89421596acf47", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:43.833+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a13bf9f28e9e4643bdea46f2d4a2bd62", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:51.023+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-813ddd2e5f1c4a5d827384f84a0005aa", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:58.501+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-3c1490dc0cba43a5828e902ad9ccb4bf", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:06.107+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0d42b4eeef9e49958d54034debf32404", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:13.448+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-057d9136fc534bd7a05d4a3f2fe91437", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:21.026+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-64250f659e794f5cb774e58461b97e4e", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:28.380+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-57ac7e89e46c4e7c857a794287fca81a", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:36.479+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-2b25d7a9090f48d088f40a97b846d14f", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:44.145+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-2e705577fed54f4e8ad8f6f9e17fe774", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:51.475+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-72aca0b7e4864f92990fe560dbcc0a69", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:58.844+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ddd360391bc64e6c9d20673ae26a73f3", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:06.483+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-3cc178f1e0904fc39ac08a077e048a50", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:15.806+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-8b600c6bb6114cdc8a084fb3bcc63b91", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:27.131+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-9e764d5ec32145c18c4c1af2d22d9644", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:38.097+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-17641dcaf2874cf1bdb416a174662e89", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:48.947+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-70d798ac940240a196a56d6652604d63", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:00.237+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-9b6ec2049bf9488f9d4d6e475576a0c2", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:11.280+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-b1c28ab2ab4245cc83d9a1877e3110e6", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:21.936+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-c2f3833d4d4347e7a370f6ca4022c717", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:32.975+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-d2053ef0277749fbbacb44ff38feca0e", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:35:47.908+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-4157706187474b57a8898d172289f611", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:35:54.975+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0c1e3ccc285348759c881b404b7d6e5c", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:36:02.057+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-5b9715d9f6174a6d90fb341f48834b28", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:36:09.391+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-fd3c8ddd7e4f49dba2cffecdca1edd91", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:36:16.477+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0d79ad5c61194016bcf99cf56a4ecc5d", + "restored_state": "DB_MISSING" + } + ], + "restored_acknowledged": 60, + "unknown_http": 0 +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/final-snapshot.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/final-snapshot.json new file mode 100644 index 00000000..fa9c92dd --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/final-snapshot.json @@ -0,0 +1,1235 @@ +{ + "consistency": { + "checked_documents": 61, + "duplicate_chunks": 0, + "duplicate_embeddings": 0, + "incomplete_jobs": 0, + "missing_embeddings": 0 + }, + "flyway_ran": true, + "records": [ + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333394887109", + "ids": { + "documentId": 3, + "documentVersionId": 3, + "embeddingJobId": 1, + "fileObjectId": 1 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-0296462e118a428da1b889a1310a9f30", + "sha256": "cb6c0dff45951c630ae0b02694bc954e159100cb2495170f8379171c20637ca8", + "size": 1045, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333396369177", + "ids": { + "documentId": 4, + "documentVersionId": 4, + "embeddingJobId": 2, + "fileObjectId": 2 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-ce6a718bdd0f4ace83465acdb3f9d275", + "sha256": "6fca27eab558e63f007d70172241ee91cd3d3c40b868b7b6e5a821fc64d656bc", + "size": 641, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333397665022", + "ids": { + "documentId": 5, + "documentVersionId": 5, + "embeddingJobId": 3, + "fileObjectId": 3 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-b1f116f44ea449fb90474ed3ffbde583", + "sha256": "5d12c58a83aae3d5a60f4d776086df081a2d061b89536e4623b5ef66953b15d2", + "size": 1045, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333398846804", + "ids": { + "documentId": 6, + "documentVersionId": 6, + "embeddingJobId": 4, + "fileObjectId": 4 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-123273586c1a4fa0a37270b2550be058", + "sha256": "f281ebd98e5360437810e41ee282d7fbc3aea13859a25bff53494c709fd8e71c", + "size": 641, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 2, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333400275899", + "ids": { + "documentId": 7, + "documentVersionId": 7, + "embeddingJobId": 5, + "fileObjectId": 5 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-6e7a22eea289466da0fbb8865647301b", + "sha256": "8a3cbd1a5c5338d68075c056a729d4752a8e31807309e0caa12eb0937ea64902", + "size": 1045, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333801474838", + "ids": { + "documentId": 8, + "documentVersionId": 8, + "embeddingJobId": 6, + "fileObjectId": 6 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-fb2dd8207fa647fa9d80ffdf4c348c68", + "sha256": "c1941811f0d2747ce0736b458119484e6973699b0587eecf070d796e7440a328", + "size": 1045, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333802798444", + "ids": { + "documentId": 9, + "documentVersionId": 9, + "embeddingJobId": 7, + "fileObjectId": 7 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-432fd889599b4f88b0d24d6593996a73", + "sha256": "ad816c20f4f0bc5be74c279296cedd1d37991dd7f59cc55722ce28e8d92ffc02", + "size": 641, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333804125436", + "ids": { + "documentId": 10, + "documentVersionId": 10, + "embeddingJobId": 8, + "fileObjectId": 8 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-4aa4c55b25d14d538fcaff279600fda9", + "sha256": "62181f80ab73059b412276fb9bc28765b242dbb26f36ac7dd7ecd2127584e66f", + "size": 1045, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333805402152", + "ids": { + "documentId": 11, + "documentVersionId": 11, + "embeddingJobId": 9, + "fileObjectId": 9 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-c8826fde38704624ae5ca21b04789532", + "sha256": "747645e91e4ee34b21c4e6836d3198433a02a66aa867083becacedbf4971bffb", + "size": 641, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333806788228", + "ids": { + "documentId": 12, + "documentVersionId": 12, + "embeddingJobId": 10, + "fileObjectId": 10 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-57d7dce5ad8b4ee2adcf905c04a6f654", + "sha256": "cd898e70e7dc88efbfcf5d639615350e27a8de987cfd801addf4eb75f694254b", + "size": 1045, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333808038458", + "ids": { + "documentId": 13, + "documentVersionId": 13, + "embeddingJobId": 11, + "fileObjectId": 11 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-e4d2871c1e404638bab746d1b07f9797", + "sha256": "649e39bc14c365f99405a4293686f5d7abe6c8330e5d6fe4dff58584229baf89", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333809341078", + "ids": { + "documentId": 14, + "documentVersionId": 14, + "embeddingJobId": 12, + "fileObjectId": 12 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-e584c125eb7e4a5dbe43164f09613254", + "sha256": "cfff30ee9b1a0737eca1ed09ab5883404d15d6f51771610af51f51365f3a8ec7", + "size": 1045, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333810643638", + "ids": { + "documentId": 15, + "documentVersionId": 15, + "embeddingJobId": 13, + "fileObjectId": 13 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-e97831e4ebdf4f0398b54d8845adcd49", + "sha256": "1b330520eed5e3659cec0d138a26c4a94dfeb0dae80e1aeb949d2bdcc96d94f3", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333811904377", + "ids": { + "documentId": 16, + "documentVersionId": 16, + "embeddingJobId": 14, + "fileObjectId": 14 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-e0ed270f022244c19fb49c554ac6ef5b", + "sha256": "319a91561e5255e8e4fa5855ecf45ffc10d5d4567605bed159ec892c1f198389", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333813202648", + "ids": { + "documentId": 17, + "documentVersionId": 17, + "embeddingJobId": 15, + "fileObjectId": 15 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-815f489bc96c48d187abac925f3271f4", + "sha256": "cdaba700e161a00efb9f2b5376b7d2a819b4116f746253e55f45a408070a3ec2", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333814551128", + "ids": { + "documentId": 18, + "documentVersionId": 18, + "embeddingJobId": 16, + "fileObjectId": 16 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-e34da01ee9474549b9002f280a723de1", + "sha256": "44ed214162a5fe1fb02221dcc2ec494496c231d35dd6e744fe9ebae8fb0cafe2", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333815872591", + "ids": { + "documentId": 19, + "documentVersionId": 19, + "embeddingJobId": 17, + "fileObjectId": 17 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-5cd702f34097434ab4544b79573c47fc", + "sha256": "21c048da2b95a9077ee98f059ed4fd1dceaca716ed8027465222d18121a36f31", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333817160338", + "ids": { + "documentId": 20, + "documentVersionId": 20, + "embeddingJobId": 18, + "fileObjectId": 18 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-a7c7983377aa4cb38474c2c4b639d200", + "sha256": "6321a7fbd1044015e2499bbec3bf05d0194abec00e2cb28bf2109a6c38743a1d", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333818431760", + "ids": { + "documentId": 21, + "documentVersionId": 21, + "embeddingJobId": 19, + "fileObjectId": 19 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-8fe5e921c6a64f2bafb9f5b24e23345d", + "sha256": "37f65da35477a4d1ff91d17070ef492d6a890ed606568ca1e9af3d6b11aa3bb2", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333819804887", + "ids": { + "documentId": 22, + "documentVersionId": 22, + "embeddingJobId": 20, + "fileObjectId": 20 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-a7fbd0d4358842d99a18392ea28d5ada", + "sha256": "ec2fcb418c91cb8b8d0bf78186efc6f99eefea1efe5c2544bc43460ea24b6959", + "size": 1045, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333821311606", + "ids": { + "documentId": 23, + "documentVersionId": 23, + "embeddingJobId": 21, + "fileObjectId": 21 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-52d582fc9eb64427845662582d8a3bbb", + "sha256": "f50e89d1b76a00f28d1051cb44d1b786d9922a56047c9e2bbab7b2f649208c0d", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333822701925", + "ids": { + "documentId": 24, + "documentVersionId": 24, + "embeddingJobId": 22, + "fileObjectId": 22 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-5cbd3325ea7e4b97b4db3ade4fc2fa7a", + "sha256": "538cfe4f5a5ea844e945afef4d23674988270f0aa1f52f27730f98697b7f5a18", + "size": 1045, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333823957315", + "ids": { + "documentId": 25, + "documentVersionId": 25, + "embeddingJobId": 23, + "fileObjectId": 23 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-8af4ad876cd043f0a906915c5b5c9e76", + "sha256": "125ad2242835634e68f4c82efa85888600dc6f97bd3474765f658133e8c75a52", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333825235999", + "ids": { + "documentId": 26, + "documentVersionId": 26, + "embeddingJobId": 24, + "fileObjectId": 24 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-a2a79e66098444ce812e0e7f2b55741c", + "sha256": "4dd86e6bb9e74cf9b403e286bf0fe3321d3bc7bc13447e61d70bb72246e1ce8f", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333826546721", + "ids": { + "documentId": 27, + "documentVersionId": 27, + "embeddingJobId": 25, + "fileObjectId": 25 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-f35ec7a4a77b466186a1bb8faa995b8f", + "sha256": "c24c6c54039e0d1a121cd0d9702a525bc1c996c6409ed1049b0e36b658087e7a", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333827749600", + "ids": { + "documentId": 28, + "documentVersionId": 28, + "embeddingJobId": 26, + "fileObjectId": 26 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-7b83a8a9b01544af88bbec7e9e5a8b94", + "sha256": "dd4d1b12b85e80e7f165418965bfb590546c3d267df5ea6974ffea1ece4ee92f", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333829026930", + "ids": { + "documentId": 29, + "documentVersionId": 29, + "embeddingJobId": 27, + "fileObjectId": 27 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-106767bf29ce4751a114beffed1dcf5d", + "sha256": "7199843049649f2d55f8444f8fd795d2f9620569e4d6752e42e3806c6e10af67", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333830306441", + "ids": { + "documentId": 30, + "documentVersionId": 30, + "embeddingJobId": 28, + "fileObjectId": 28 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-c2ec085d264a49c9ab6f691c6ae4aa9b", + "sha256": "6f2ab9f8bb530c07b539c787bc926c0161cb7136ee1fab6383d272861133f7d6", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333831574400", + "ids": { + "documentId": 31, + "documentVersionId": 31, + "embeddingJobId": 29, + "fileObjectId": 29 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-cef30e0526c24946b07bc80888bd0290", + "sha256": "1122671a8947a18acc9b49d318e573d3d4f64299328a9a25a4b3103860590dc7", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333832876770", + "ids": { + "documentId": 32, + "documentVersionId": 32, + "embeddingJobId": 30, + "fileObjectId": 30 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-496f0a710ecb4981a3a644fdea27a2a5", + "sha256": "5f82ef3f5fc831e1b38624f982436ebd6dd233cf9b36e0af9c12b5d7a42f8ce6", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333834140672", + "ids": { + "documentId": 33, + "documentVersionId": 33, + "embeddingJobId": 31, + "fileObjectId": 31 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-3f79f77ddbd24cf1a9f682abcf928cb6", + "sha256": "f904af9279ff864430761562c9f2c207dd1539c18c91098a2a43fe3dd484806c", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333835301903", + "ids": { + "documentId": 34, + "documentVersionId": 34, + "embeddingJobId": 32, + "fileObjectId": 32 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-63141b14d77e4900965f43624358e6ce", + "sha256": "8c4163e1723cf99bada4fccc1956b24b36aeb3ff9aaa2b18a1d5a4355c42a0d3", + "size": 1045, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333836593143", + "ids": { + "documentId": 35, + "documentVersionId": 35, + "embeddingJobId": 33, + "fileObjectId": 33 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-52a198f8bd8a4c1f832134222e5505c5", + "sha256": "296abbc16a4f567ea8197c9a138b973da051803e082e5b52d627f9b023453dc0", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792334098000268", + "ids": { + "documentId": 36, + "documentVersionId": 36, + "embeddingJobId": 34, + "fileObjectId": 34 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-d029610fe4234219b1db66f348b7b114", + "sha256": "4a1955b23633e09c64c251faf53a7f3db408e65c0d31e4131eac14e2e408b036", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792334099384913", + "ids": { + "documentId": 37, + "documentVersionId": 37, + "embeddingJobId": 35, + "fileObjectId": 35 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-ec97bd5c298f483d9cf98729cea26940", + "sha256": "d517ea588330b3a1009b2a65decaa649f2eb64041318067f7eca6f41310beb19", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792334100779971", + "ids": { + "documentId": 38, + "documentVersionId": 38, + "embeddingJobId": 36, + "fileObjectId": 36 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-0b7cf70ddca94790b02c8e8024fb040d", + "sha256": "39f7e102ac80a9ba04150f8e357db6123441bdc7cafe5620dc163dda4661ae6f", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792334102353834", + "ids": { + "documentId": 39, + "documentVersionId": 39, + "embeddingJobId": 37, + "fileObjectId": 37 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-df6017955c5447b9bf097221f6a7d8be", + "sha256": "2a1bfc66a9c58c43d8d51ff1c5391e58dd582fbc386733c41d7c3989fdce9731", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792334103668017", + "ids": { + "documentId": 40, + "documentVersionId": 40, + "embeddingJobId": 38, + "fileObjectId": 38 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-ba75b05d8e5d477abb3d2a1180a95077", + "sha256": "0b3353283ad6d05cb497416736e7fdda10b57f9a0ffbfb2d316596365791b4b8", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792334105072235", + "ids": { + "documentId": 41, + "documentVersionId": 41, + "embeddingJobId": 39, + "fileObjectId": 39 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-a5be52472400424db8f52ed7a54c2e83", + "sha256": "6fa3bf5896c7e74952fcd6935204ebf002cda8e6892bd4b34d19839e9849576d", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792334106429281", + "ids": { + "documentId": 42, + "documentVersionId": 42, + "embeddingJobId": 40, + "fileObjectId": 40 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-eb8fb6803498447a94b89421596acf47", + "sha256": "604dda88d3af34bfe5fb2ecbd42e1466b7123367aabf2b6479f5187bbde8b1e5", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792334107767532", + "ids": { + "documentId": 43, + "documentVersionId": 43, + "embeddingJobId": 41, + "fileObjectId": 41 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-a13bf9f28e9e4643bdea46f2d4a2bd62", + "sha256": "cb1defde74581e45e5606cfe0f948314533669d751d71149d905ac755a067425", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792334109163978", + "ids": { + "documentId": 44, + "documentVersionId": 44, + "embeddingJobId": 42, + "fileObjectId": 42 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-813ddd2e5f1c4a5d827384f84a0005aa", + "sha256": "cca8e204eaa0a10bb320b2979da370eb64ed2bbc54d9ffe190b3bd0206ba97e5", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792334110385998", + "ids": { + "documentId": 45, + "documentVersionId": 45, + "embeddingJobId": 43, + "fileObjectId": 43 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-3c1490dc0cba43a5828e902ad9ccb4bf", + "sha256": "2561e3f2e810a8a4c7c06b6f0d03578d79f27889387f1ad53fde39f5070b63a5", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792334111853252", + "ids": { + "documentId": 46, + "documentVersionId": 46, + "embeddingJobId": 44, + "fileObjectId": 44 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-0d42b4eeef9e49958d54034debf32404", + "sha256": "7dcaaa488971cae03bfd16b70021146afe24dc803d51107bdb4925b1613af5e8", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792334113141879", + "ids": { + "documentId": 47, + "documentVersionId": 47, + "embeddingJobId": 45, + "fileObjectId": 45 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-057d9136fc534bd7a05d4a3f2fe91437", + "sha256": "bac1098d29cc26b2ef7f606bd8f4942f864358d57c99dce5166ea4db3287933e", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792334114523730", + "ids": { + "documentId": 48, + "documentVersionId": 48, + "embeddingJobId": 46, + "fileObjectId": 46 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-64250f659e794f5cb774e58461b97e4e", + "sha256": "ed8f4d573635655183f52c0a3a7ad5eab7325225399f46ea41372cc7907cefd3", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792334115989024", + "ids": { + "documentId": 49, + "documentVersionId": 49, + "embeddingJobId": 47, + "fileObjectId": 47 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-57ac7e89e46c4e7c857a794287fca81a", + "sha256": "78d4567ad81362950c53cdaf6310b3c6be4280a500ce8422c4473f0974fa178d", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792334117554101", + "ids": { + "documentId": 50, + "documentVersionId": 50, + "embeddingJobId": 48, + "fileObjectId": 48 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-2b25d7a9090f48d088f40a97b846d14f", + "sha256": "c3ef81a8b35b77101fa3c1caba153f8dac4c76b0d6ab56e917bbc9ad88440296", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792334118985900", + "ids": { + "documentId": 51, + "documentVersionId": 51, + "embeddingJobId": 49, + "fileObjectId": 49 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-2e705577fed54f4e8ad8f6f9e17fe774", + "sha256": "6bd1b2aa0f2dd0e8547740529e00c66252bb39153bac639ed6ab0f1655fe3673", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792334120265082", + "ids": { + "documentId": 52, + "documentVersionId": 52, + "embeddingJobId": 50, + "fileObjectId": 50 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-72aca0b7e4864f92990fe560dbcc0a69", + "sha256": "5c5f78c20f08e1cc1bfe3d8fdecdd7fe41d887bbd6365ddeb02755131ac7e266", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792334121557537", + "ids": { + "documentId": 53, + "documentVersionId": 53, + "embeddingJobId": 51, + "fileObjectId": 51 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-ddd360391bc64e6c9d20673ae26a73f3", + "sha256": "7d2dc24cbce3133caeebe9ea1b3f17d7053f1d73083527721b521416de72449a", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792334123050031", + "ids": { + "documentId": 54, + "documentVersionId": 54, + "embeddingJobId": 52, + "fileObjectId": 52 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-3cc178f1e0904fc39ac08a077e048a50", + "sha256": "bab62716f6a42b0766db18babe1e4d649e24be3e9e731e6f2553f16e58ec011e", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792334124486623", + "ids": { + "documentId": 55, + "documentVersionId": 55, + "embeddingJobId": 53, + "fileObjectId": 53 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-8b600c6bb6114cdc8a084fb3bcc63b91", + "sha256": "7c1518f5b75aa22b3581da98e94d958e94a4a8bf146ab650a7a245964401c004", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792334125803407", + "ids": { + "documentId": 56, + "documentVersionId": 56, + "embeddingJobId": 54, + "fileObjectId": 54 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-9e764d5ec32145c18c4c1af2d22d9644", + "sha256": "71f499d0499d9204f28495bf65fb23854f288a8d744a6dbc38c405bc34b319ba", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792334127146475", + "ids": { + "documentId": 57, + "documentVersionId": 57, + "embeddingJobId": 55, + "fileObjectId": 55 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-17641dcaf2874cf1bdb416a174662e89", + "sha256": "cbe79087c3e591bebb072292ef7c6d41d62f119c03eee75af19a24b7afdb80fe", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792334128442978", + "ids": { + "documentId": 58, + "documentVersionId": 58, + "embeddingJobId": 56, + "fileObjectId": 56 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-70d798ac940240a196a56d6652604d63", + "sha256": "5b6caf7ba08fc075fc7748fec448cbf746b6af1f3c6ae4c0548568ee64f7810a", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792334129753173", + "ids": { + "documentId": 59, + "documentVersionId": 59, + "embeddingJobId": 57, + "fileObjectId": 57 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-9b6ec2049bf9488f9d4d6e475576a0c2", + "sha256": "6acc5225c8a998e1e9d79e823dc96f1c554330c26e68685da025e17f4337657a", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792334131037142", + "ids": { + "documentId": 60, + "documentVersionId": 60, + "embeddingJobId": 58, + "fileObjectId": 58 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-b1c28ab2ab4245cc83d9a1877e3110e6", + "sha256": "2bb8c341cf0b163cbd7891f7c0e0ba1e5195136f958c209a12251bd40f114a32", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792334132300936", + "ids": { + "documentId": 61, + "documentVersionId": 61, + "embeddingJobId": 59, + "fileObjectId": 59 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-c2f3833d4d4347e7a370f6ca4022c717", + "sha256": "c61b92f0cb27263620175ac11f2ab02ace5a2949efcf7ce82d2f311cbe925a3f", + "size": 642, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792334133812207", + "ids": { + "documentId": 62, + "documentVersionId": 62, + "embeddingJobId": 60, + "fileObjectId": 60 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-d2053ef0277749fbbacb44ff38feca0e", + "sha256": "db9acd78e12caeeb5c26775c71672ef41bfea93a45c781a186523abd8f13068a", + "size": 1046, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "request_id": "upload-4157706187474b57a8898d172289f611", + "state": "DB_MISSING" + }, + { + "request_id": "upload-0c1e3ccc285348759c881b404b7d6e5c", + "state": "DB_MISSING" + }, + { + "request_id": "upload-5b9715d9f6174a6d90fb341f48834b28", + "state": "DB_MISSING" + }, + { + "request_id": "upload-fd3c8ddd7e4f49dba2cffecdca1edd91", + "state": "DB_MISSING" + }, + { + "request_id": "upload-0d79ad5c61194016bcf99cf56a4ecc5d", + "state": "DB_MISSING" + } + ], + "repair_ran": true, + "source_access_used": false, + "worker_started": true +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/formal-spec.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/formal-spec.json new file mode 100644 index 00000000..a8b337e1 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/formal-spec.json @@ -0,0 +1,18 @@ +{ + "expected_search_top_k": 20, + "formal_workload": true, + "measurement_scope": "isolated document download/search/indexing; no public switchover, OpenProxy or Ollama generation", + "purpose": "업로드 성공 응답 원장과 실제 OpenSQL·GCS 최초 복원 상태 및 기능별 재개 경과 시간 대조", + "recovery_worker_max_concurrency": 2, + "revision": "3bff887a5f6abdf4155957a658599f8f0366a141", + "run_id": "DR-RESTORE-20261018-FRESHNESS1", + "started_at_utc": "2026-10-18T14:24:21.367734+00:00", + "success_criteria": "마지막 원격 WAL 확인 구간 ACK 누락 0, 기능 코호트 원본·관계 일치, 최초 감사 후 Worker 활성화, 기능 6종 성공, Job 1개·Attempt 2, 최종 미완료/중복/누락 벡터 0, 원복·정리", + "workload": { + "functional_completed": 4, + "functional_interrupted": 1, + "loss_uploads": 60, + "nominal_request_interval_seconds": 11, + "offsite_interval_seconds": 300 + } +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/functional-observations.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/functional-observations.json new file mode 100644 index 00000000..91b7f7c3 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/functional-observations.json @@ -0,0 +1,32 @@ +{ + "download": { + "at_utc": "2026-10-18T14:39:09.852552+00:00", + "passed": true, + "samples": 4 + }, + "job_resume": { + "at_utc": "2026-10-18T14:40:21.317855+00:00", + "passed": true, + "samples": 1 + }, + "new_download": { + "at_utc": "2026-10-18T14:51:39.754136+00:00", + "passed": true, + "samples": 1 + }, + "new_indexing": { + "at_utc": "2026-10-18T14:51:37.672099+00:00", + "passed": true, + "samples": 1 + }, + "new_upload": { + "at_utc": "2026-10-18T14:40:28.376043+00:00", + "passed": true, + "samples": 1 + }, + "search": { + "at_utc": "2026-10-18T14:39:25.400051+00:00", + "passed": true, + "samples": 1 + } +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/initial-audit.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/initial-audit.json new file mode 100644 index 00000000..30b13311 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/initial-audit.json @@ -0,0 +1,465 @@ +{ + "acknowledged": 65, + "activation_allowed": true, + "http_error": 0, + "lost_or_incomplete_acknowledged": 5, + "requests": [ + { + "ack_at_utc": "2026-10-18T14:16:35.002+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0296462e118a428da1b889a1310a9f30", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:17:14.038+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ce6a718bdd0f4ace83465acdb3f9d275", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:17:45.307+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-b1f116f44ea449fb90474ed3ffbde583", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:18:19.856+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-123273586c1a4fa0a37270b2550be058", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:20:09.511+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-6e7a22eea289466da0fbb8865647301b", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:24:39.407+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-fb2dd8207fa647fa9d80ffdf4c348c68", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:24:49.127+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-432fd889599b4f88b0d24d6593996a73", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:00.168+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-4aa4c55b25d14d538fcaff279600fda9", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:11.407+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-c8826fde38704624ae5ca21b04789532", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:22.257+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-57d7dce5ad8b4ee2adcf905c04a6f654", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:34.475+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e4d2871c1e404638bab746d1b07f9797", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:45.185+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e584c125eb7e4a5dbe43164f09613254", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:55.924+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e97831e4ebdf4f0398b54d8845adcd49", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:06.927+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e0ed270f022244c19fb49c554ac6ef5b", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:18.268+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-815f489bc96c48d187abac925f3271f4", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:28.971+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e34da01ee9474549b9002f280a723de1", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:39.772+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-5cd702f34097434ab4544b79573c47fc", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:50.450+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a7c7983377aa4cb38474c2c4b639d200", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:02.777+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-8fe5e921c6a64f2bafb9f5b24e23345d", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:12.730+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a7fbd0d4358842d99a18392ea28d5ada", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:25.101+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-52d582fc9eb64427845662582d8a3bbb", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:35.059+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-5cbd3325ea7e4b97b4db3ade4fc2fa7a", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:45.864+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-8af4ad876cd043f0a906915c5b5c9e76", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:57.382+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a2a79e66098444ce812e0e7f2b55741c", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:07.877+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-f35ec7a4a77b466186a1bb8faa995b8f", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:18.857+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-7b83a8a9b01544af88bbec7e9e5a8b94", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:30.164+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-106767bf29ce4751a114beffed1dcf5d", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:40.871+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-c2ec085d264a49c9ab6f691c6ae4aa9b", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:51.926+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-cef30e0526c24946b07bc80888bd0290", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:03.139+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-496f0a710ecb4981a3a644fdea27a2a5", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:13.966+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-3f79f77ddbd24cf1a9f682abcf928cb6", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:25.157+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-63141b14d77e4900965f43624358e6ce", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:36.265+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-52a198f8bd8a4c1f832134222e5505c5", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:30:51.420+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-d029610fe4234219b1db66f348b7b114", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:30:59.249+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ec97bd5c298f483d9cf98729cea26940", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:06.618+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0b7cf70ddca94790b02c8e8024fb040d", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:13.944+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-df6017955c5447b9bf097221f6a7d8be", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:22.044+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ba75b05d8e5d477abb3d2a1180a95077", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:29.191+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a5be52472400424db8f52ed7a54c2e83", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:36.338+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-eb8fb6803498447a94b89421596acf47", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:43.833+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a13bf9f28e9e4643bdea46f2d4a2bd62", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:51.023+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-813ddd2e5f1c4a5d827384f84a0005aa", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:58.501+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-3c1490dc0cba43a5828e902ad9ccb4bf", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:06.107+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0d42b4eeef9e49958d54034debf32404", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:13.448+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-057d9136fc534bd7a05d4a3f2fe91437", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:21.026+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-64250f659e794f5cb774e58461b97e4e", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:28.380+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-57ac7e89e46c4e7c857a794287fca81a", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:36.479+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-2b25d7a9090f48d088f40a97b846d14f", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:44.145+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-2e705577fed54f4e8ad8f6f9e17fe774", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:51.475+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-72aca0b7e4864f92990fe560dbcc0a69", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:58.844+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ddd360391bc64e6c9d20673ae26a73f3", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:06.483+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-3cc178f1e0904fc39ac08a077e048a50", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:15.806+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-8b600c6bb6114cdc8a084fb3bcc63b91", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:27.131+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-9e764d5ec32145c18c4c1af2d22d9644", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:38.097+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-17641dcaf2874cf1bdb416a174662e89", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:48.947+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-70d798ac940240a196a56d6652604d63", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:00.237+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-9b6ec2049bf9488f9d4d6e475576a0c2", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:11.280+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-b1c28ab2ab4245cc83d9a1877e3110e6", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:21.936+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-c2f3833d4d4347e7a370f6ca4022c717", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:32.975+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-d2053ef0277749fbbacb44ff38feca0e", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:35:47.908+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-4157706187474b57a8898d172289f611", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:35:54.975+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0c1e3ccc285348759c881b404b7d6e5c", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:36:02.057+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-5b9715d9f6174a6d90fb341f48834b28", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:36:09.391+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-fd3c8ddd7e4f49dba2cffecdca1edd91", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:36:16.477+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0d79ad5c61194016bcf99cf56a4ecc5d", + "restored_state": "DB_MISSING" + } + ], + "restored_acknowledged": 60, + "unknown_http": 0 +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/initial-snapshot.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/initial-snapshot.json new file mode 100644 index 00000000..18307aea --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/initial-snapshot.json @@ -0,0 +1,1229 @@ +{ + "consistency": {}, + "flyway_ran": false, + "records": [ + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333394887109", + "ids": { + "documentId": 3, + "documentVersionId": 3, + "embeddingJobId": 1, + "fileObjectId": 1 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-0296462e118a428da1b889a1310a9f30", + "sha256": "cb6c0dff45951c630ae0b02694bc954e159100cb2495170f8379171c20637ca8", + "size": 1045, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333396369177", + "ids": { + "documentId": 4, + "documentVersionId": 4, + "embeddingJobId": 2, + "fileObjectId": 2 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-ce6a718bdd0f4ace83465acdb3f9d275", + "sha256": "6fca27eab558e63f007d70172241ee91cd3d3c40b868b7b6e5a821fc64d656bc", + "size": 641, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXED", + "embedding_count": 3, + "generation": "1792333397665022", + "ids": { + "documentId": 5, + "documentVersionId": 5, + "embeddingJobId": 3, + "fileObjectId": 3 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-b1f116f44ea449fb90474ed3ffbde583", + "sha256": "5d12c58a83aae3d5a60f4d776086df081a2d061b89536e4623b5ef66953b15d2", + "size": 1045, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 1, + "document_status": "INDEXED", + "embedding_count": 1, + "generation": "1792333398846804", + "ids": { + "documentId": 6, + "documentVersionId": 6, + "embeddingJobId": 4, + "fileObjectId": 4 + }, + "job_status": "INDEXED", + "relations_match": true, + "request_id": "upload-123273586c1a4fa0a37270b2550be058", + "sha256": "f281ebd98e5360437810e41ee282d7fbc3aea13859a25bff53494c709fd8e71c", + "size": 641, + "state": "RESTORED", + "version_status": "INDEXED" + }, + { + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXING", + "embedding_count": 0, + "generation": "1792333400275899", + "ids": { + "documentId": 7, + "documentVersionId": 7, + "embeddingJobId": 5, + "fileObjectId": 5 + }, + "job_status": "PROCESSING", + "relations_match": true, + "request_id": "upload-6e7a22eea289466da0fbb8865647301b", + "sha256": "8a3cbd1a5c5338d68075c056a729d4752a8e31807309e0caa12eb0937ea64902", + "size": 1045, + "state": "RESTORED", + "version_status": "EMBEDDING" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333801474838", + "ids": { + "documentId": 8, + "documentVersionId": 8, + "embeddingJobId": 6, + "fileObjectId": 6 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-fb2dd8207fa647fa9d80ffdf4c348c68", + "sha256": "c1941811f0d2747ce0736b458119484e6973699b0587eecf070d796e7440a328", + "size": 1045, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333802798444", + "ids": { + "documentId": 9, + "documentVersionId": 9, + "embeddingJobId": 7, + "fileObjectId": 7 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-432fd889599b4f88b0d24d6593996a73", + "sha256": "ad816c20f4f0bc5be74c279296cedd1d37991dd7f59cc55722ce28e8d92ffc02", + "size": 641, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333804125436", + "ids": { + "documentId": 10, + "documentVersionId": 10, + "embeddingJobId": 8, + "fileObjectId": 8 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-4aa4c55b25d14d538fcaff279600fda9", + "sha256": "62181f80ab73059b412276fb9bc28765b242dbb26f36ac7dd7ecd2127584e66f", + "size": 1045, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333805402152", + "ids": { + "documentId": 11, + "documentVersionId": 11, + "embeddingJobId": 9, + "fileObjectId": 9 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-c8826fde38704624ae5ca21b04789532", + "sha256": "747645e91e4ee34b21c4e6836d3198433a02a66aa867083becacedbf4971bffb", + "size": 641, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333806788228", + "ids": { + "documentId": 12, + "documentVersionId": 12, + "embeddingJobId": 10, + "fileObjectId": 10 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-57d7dce5ad8b4ee2adcf905c04a6f654", + "sha256": "cd898e70e7dc88efbfcf5d639615350e27a8de987cfd801addf4eb75f694254b", + "size": 1045, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333808038458", + "ids": { + "documentId": 13, + "documentVersionId": 13, + "embeddingJobId": 11, + "fileObjectId": 11 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-e4d2871c1e404638bab746d1b07f9797", + "sha256": "649e39bc14c365f99405a4293686f5d7abe6c8330e5d6fe4dff58584229baf89", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333809341078", + "ids": { + "documentId": 14, + "documentVersionId": 14, + "embeddingJobId": 12, + "fileObjectId": 12 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-e584c125eb7e4a5dbe43164f09613254", + "sha256": "cfff30ee9b1a0737eca1ed09ab5883404d15d6f51771610af51f51365f3a8ec7", + "size": 1045, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333810643638", + "ids": { + "documentId": 15, + "documentVersionId": 15, + "embeddingJobId": 13, + "fileObjectId": 13 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-e97831e4ebdf4f0398b54d8845adcd49", + "sha256": "1b330520eed5e3659cec0d138a26c4a94dfeb0dae80e1aeb949d2bdcc96d94f3", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333811904377", + "ids": { + "documentId": 16, + "documentVersionId": 16, + "embeddingJobId": 14, + "fileObjectId": 14 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-e0ed270f022244c19fb49c554ac6ef5b", + "sha256": "319a91561e5255e8e4fa5855ecf45ffc10d5d4567605bed159ec892c1f198389", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333813202648", + "ids": { + "documentId": 17, + "documentVersionId": 17, + "embeddingJobId": 15, + "fileObjectId": 15 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-815f489bc96c48d187abac925f3271f4", + "sha256": "cdaba700e161a00efb9f2b5376b7d2a819b4116f746253e55f45a408070a3ec2", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333814551128", + "ids": { + "documentId": 18, + "documentVersionId": 18, + "embeddingJobId": 16, + "fileObjectId": 16 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-e34da01ee9474549b9002f280a723de1", + "sha256": "44ed214162a5fe1fb02221dcc2ec494496c231d35dd6e744fe9ebae8fb0cafe2", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333815872591", + "ids": { + "documentId": 19, + "documentVersionId": 19, + "embeddingJobId": 17, + "fileObjectId": 17 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-5cd702f34097434ab4544b79573c47fc", + "sha256": "21c048da2b95a9077ee98f059ed4fd1dceaca716ed8027465222d18121a36f31", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333817160338", + "ids": { + "documentId": 20, + "documentVersionId": 20, + "embeddingJobId": 18, + "fileObjectId": 18 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-a7c7983377aa4cb38474c2c4b639d200", + "sha256": "6321a7fbd1044015e2499bbec3bf05d0194abec00e2cb28bf2109a6c38743a1d", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333818431760", + "ids": { + "documentId": 21, + "documentVersionId": 21, + "embeddingJobId": 19, + "fileObjectId": 19 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-8fe5e921c6a64f2bafb9f5b24e23345d", + "sha256": "37f65da35477a4d1ff91d17070ef492d6a890ed606568ca1e9af3d6b11aa3bb2", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333819804887", + "ids": { + "documentId": 22, + "documentVersionId": 22, + "embeddingJobId": 20, + "fileObjectId": 20 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-a7fbd0d4358842d99a18392ea28d5ada", + "sha256": "ec2fcb418c91cb8b8d0bf78186efc6f99eefea1efe5c2544bc43460ea24b6959", + "size": 1045, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333821311606", + "ids": { + "documentId": 23, + "documentVersionId": 23, + "embeddingJobId": 21, + "fileObjectId": 21 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-52d582fc9eb64427845662582d8a3bbb", + "sha256": "f50e89d1b76a00f28d1051cb44d1b786d9922a56047c9e2bbab7b2f649208c0d", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333822701925", + "ids": { + "documentId": 24, + "documentVersionId": 24, + "embeddingJobId": 22, + "fileObjectId": 22 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-5cbd3325ea7e4b97b4db3ade4fc2fa7a", + "sha256": "538cfe4f5a5ea844e945afef4d23674988270f0aa1f52f27730f98697b7f5a18", + "size": 1045, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333823957315", + "ids": { + "documentId": 25, + "documentVersionId": 25, + "embeddingJobId": 23, + "fileObjectId": 23 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-8af4ad876cd043f0a906915c5b5c9e76", + "sha256": "125ad2242835634e68f4c82efa85888600dc6f97bd3474765f658133e8c75a52", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333825235999", + "ids": { + "documentId": 26, + "documentVersionId": 26, + "embeddingJobId": 24, + "fileObjectId": 24 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-a2a79e66098444ce812e0e7f2b55741c", + "sha256": "4dd86e6bb9e74cf9b403e286bf0fe3321d3bc7bc13447e61d70bb72246e1ce8f", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333826546721", + "ids": { + "documentId": 27, + "documentVersionId": 27, + "embeddingJobId": 25, + "fileObjectId": 25 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-f35ec7a4a77b466186a1bb8faa995b8f", + "sha256": "c24c6c54039e0d1a121cd0d9702a525bc1c996c6409ed1049b0e36b658087e7a", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333827749600", + "ids": { + "documentId": 28, + "documentVersionId": 28, + "embeddingJobId": 26, + "fileObjectId": 26 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-7b83a8a9b01544af88bbec7e9e5a8b94", + "sha256": "dd4d1b12b85e80e7f165418965bfb590546c3d267df5ea6974ffea1ece4ee92f", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333829026930", + "ids": { + "documentId": 29, + "documentVersionId": 29, + "embeddingJobId": 27, + "fileObjectId": 27 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-106767bf29ce4751a114beffed1dcf5d", + "sha256": "7199843049649f2d55f8444f8fd795d2f9620569e4d6752e42e3806c6e10af67", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333830306441", + "ids": { + "documentId": 30, + "documentVersionId": 30, + "embeddingJobId": 28, + "fileObjectId": 28 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-c2ec085d264a49c9ab6f691c6ae4aa9b", + "sha256": "6f2ab9f8bb530c07b539c787bc926c0161cb7136ee1fab6383d272861133f7d6", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333831574400", + "ids": { + "documentId": 31, + "documentVersionId": 31, + "embeddingJobId": 29, + "fileObjectId": 29 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-cef30e0526c24946b07bc80888bd0290", + "sha256": "1122671a8947a18acc9b49d318e573d3d4f64299328a9a25a4b3103860590dc7", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333832876770", + "ids": { + "documentId": 32, + "documentVersionId": 32, + "embeddingJobId": 30, + "fileObjectId": 30 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-496f0a710ecb4981a3a644fdea27a2a5", + "sha256": "5f82ef3f5fc831e1b38624f982436ebd6dd233cf9b36e0af9c12b5d7a42f8ce6", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333834140672", + "ids": { + "documentId": 33, + "documentVersionId": 33, + "embeddingJobId": 31, + "fileObjectId": 31 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-3f79f77ddbd24cf1a9f682abcf928cb6", + "sha256": "f904af9279ff864430761562c9f2c207dd1539c18c91098a2a43fe3dd484806c", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333835301903", + "ids": { + "documentId": 34, + "documentVersionId": 34, + "embeddingJobId": 32, + "fileObjectId": 32 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-63141b14d77e4900965f43624358e6ce", + "sha256": "8c4163e1723cf99bada4fccc1956b24b36aeb3ff9aaa2b18a1d5a4355c42a0d3", + "size": 1045, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792333836593143", + "ids": { + "documentId": 35, + "documentVersionId": 35, + "embeddingJobId": 33, + "fileObjectId": 33 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-52a198f8bd8a4c1f832134222e5505c5", + "sha256": "296abbc16a4f567ea8197c9a138b973da051803e082e5b52d627f9b023453dc0", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334098000268", + "ids": { + "documentId": 36, + "documentVersionId": 36, + "embeddingJobId": 34, + "fileObjectId": 34 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-d029610fe4234219b1db66f348b7b114", + "sha256": "4a1955b23633e09c64c251faf53a7f3db408e65c0d31e4131eac14e2e408b036", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334099384913", + "ids": { + "documentId": 37, + "documentVersionId": 37, + "embeddingJobId": 35, + "fileObjectId": 35 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-ec97bd5c298f483d9cf98729cea26940", + "sha256": "d517ea588330b3a1009b2a65decaa649f2eb64041318067f7eca6f41310beb19", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334100779971", + "ids": { + "documentId": 38, + "documentVersionId": 38, + "embeddingJobId": 36, + "fileObjectId": 36 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-0b7cf70ddca94790b02c8e8024fb040d", + "sha256": "39f7e102ac80a9ba04150f8e357db6123441bdc7cafe5620dc163dda4661ae6f", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334102353834", + "ids": { + "documentId": 39, + "documentVersionId": 39, + "embeddingJobId": 37, + "fileObjectId": 37 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-df6017955c5447b9bf097221f6a7d8be", + "sha256": "2a1bfc66a9c58c43d8d51ff1c5391e58dd582fbc386733c41d7c3989fdce9731", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334103668017", + "ids": { + "documentId": 40, + "documentVersionId": 40, + "embeddingJobId": 38, + "fileObjectId": 38 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-ba75b05d8e5d477abb3d2a1180a95077", + "sha256": "0b3353283ad6d05cb497416736e7fdda10b57f9a0ffbfb2d316596365791b4b8", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334105072235", + "ids": { + "documentId": 41, + "documentVersionId": 41, + "embeddingJobId": 39, + "fileObjectId": 39 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-a5be52472400424db8f52ed7a54c2e83", + "sha256": "6fa3bf5896c7e74952fcd6935204ebf002cda8e6892bd4b34d19839e9849576d", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334106429281", + "ids": { + "documentId": 42, + "documentVersionId": 42, + "embeddingJobId": 40, + "fileObjectId": 40 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-eb8fb6803498447a94b89421596acf47", + "sha256": "604dda88d3af34bfe5fb2ecbd42e1466b7123367aabf2b6479f5187bbde8b1e5", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334107767532", + "ids": { + "documentId": 43, + "documentVersionId": 43, + "embeddingJobId": 41, + "fileObjectId": 41 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-a13bf9f28e9e4643bdea46f2d4a2bd62", + "sha256": "cb1defde74581e45e5606cfe0f948314533669d751d71149d905ac755a067425", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334109163978", + "ids": { + "documentId": 44, + "documentVersionId": 44, + "embeddingJobId": 42, + "fileObjectId": 42 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-813ddd2e5f1c4a5d827384f84a0005aa", + "sha256": "cca8e204eaa0a10bb320b2979da370eb64ed2bbc54d9ffe190b3bd0206ba97e5", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334110385998", + "ids": { + "documentId": 45, + "documentVersionId": 45, + "embeddingJobId": 43, + "fileObjectId": 43 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-3c1490dc0cba43a5828e902ad9ccb4bf", + "sha256": "2561e3f2e810a8a4c7c06b6f0d03578d79f27889387f1ad53fde39f5070b63a5", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334111853252", + "ids": { + "documentId": 46, + "documentVersionId": 46, + "embeddingJobId": 44, + "fileObjectId": 44 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-0d42b4eeef9e49958d54034debf32404", + "sha256": "7dcaaa488971cae03bfd16b70021146afe24dc803d51107bdb4925b1613af5e8", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334113141879", + "ids": { + "documentId": 47, + "documentVersionId": 47, + "embeddingJobId": 45, + "fileObjectId": 45 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-057d9136fc534bd7a05d4a3f2fe91437", + "sha256": "bac1098d29cc26b2ef7f606bd8f4942f864358d57c99dce5166ea4db3287933e", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334114523730", + "ids": { + "documentId": 48, + "documentVersionId": 48, + "embeddingJobId": 46, + "fileObjectId": 46 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-64250f659e794f5cb774e58461b97e4e", + "sha256": "ed8f4d573635655183f52c0a3a7ad5eab7325225399f46ea41372cc7907cefd3", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334115989024", + "ids": { + "documentId": 49, + "documentVersionId": 49, + "embeddingJobId": 47, + "fileObjectId": 47 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-57ac7e89e46c4e7c857a794287fca81a", + "sha256": "78d4567ad81362950c53cdaf6310b3c6be4280a500ce8422c4473f0974fa178d", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334117554101", + "ids": { + "documentId": 50, + "documentVersionId": 50, + "embeddingJobId": 48, + "fileObjectId": 48 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-2b25d7a9090f48d088f40a97b846d14f", + "sha256": "c3ef81a8b35b77101fa3c1caba153f8dac4c76b0d6ab56e917bbc9ad88440296", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334118985900", + "ids": { + "documentId": 51, + "documentVersionId": 51, + "embeddingJobId": 49, + "fileObjectId": 49 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-2e705577fed54f4e8ad8f6f9e17fe774", + "sha256": "6bd1b2aa0f2dd0e8547740529e00c66252bb39153bac639ed6ab0f1655fe3673", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334120265082", + "ids": { + "documentId": 52, + "documentVersionId": 52, + "embeddingJobId": 50, + "fileObjectId": 50 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-72aca0b7e4864f92990fe560dbcc0a69", + "sha256": "5c5f78c20f08e1cc1bfe3d8fdecdd7fe41d887bbd6365ddeb02755131ac7e266", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334121557537", + "ids": { + "documentId": 53, + "documentVersionId": 53, + "embeddingJobId": 51, + "fileObjectId": 51 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-ddd360391bc64e6c9d20673ae26a73f3", + "sha256": "7d2dc24cbce3133caeebe9ea1b3f17d7053f1d73083527721b521416de72449a", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334123050031", + "ids": { + "documentId": 54, + "documentVersionId": 54, + "embeddingJobId": 52, + "fileObjectId": 52 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-3cc178f1e0904fc39ac08a077e048a50", + "sha256": "bab62716f6a42b0766db18babe1e4d649e24be3e9e731e6f2553f16e58ec011e", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334124486623", + "ids": { + "documentId": 55, + "documentVersionId": 55, + "embeddingJobId": 53, + "fileObjectId": 53 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-8b600c6bb6114cdc8a084fb3bcc63b91", + "sha256": "7c1518f5b75aa22b3581da98e94d958e94a4a8bf146ab650a7a245964401c004", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334125803407", + "ids": { + "documentId": 56, + "documentVersionId": 56, + "embeddingJobId": 54, + "fileObjectId": 54 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-9e764d5ec32145c18c4c1af2d22d9644", + "sha256": "71f499d0499d9204f28495bf65fb23854f288a8d744a6dbc38c405bc34b319ba", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334127146475", + "ids": { + "documentId": 57, + "documentVersionId": 57, + "embeddingJobId": 55, + "fileObjectId": 55 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-17641dcaf2874cf1bdb416a174662e89", + "sha256": "cbe79087c3e591bebb072292ef7c6d41d62f119c03eee75af19a24b7afdb80fe", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334128442978", + "ids": { + "documentId": 58, + "documentVersionId": 58, + "embeddingJobId": 56, + "fileObjectId": 56 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-70d798ac940240a196a56d6652604d63", + "sha256": "5b6caf7ba08fc075fc7748fec448cbf746b6af1f3c6ae4c0548568ee64f7810a", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334129753173", + "ids": { + "documentId": 59, + "documentVersionId": 59, + "embeddingJobId": 57, + "fileObjectId": 57 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-9b6ec2049bf9488f9d4d6e475576a0c2", + "sha256": "6acc5225c8a998e1e9d79e823dc96f1c554330c26e68685da025e17f4337657a", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334131037142", + "ids": { + "documentId": 60, + "documentVersionId": 60, + "embeddingJobId": 58, + "fileObjectId": 58 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-b1c28ab2ab4245cc83d9a1877e3110e6", + "sha256": "2bb8c341cf0b163cbd7891f7c0e0ba1e5195136f958c209a12251bd40f114a32", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334132300936", + "ids": { + "documentId": 61, + "documentVersionId": 61, + "embeddingJobId": 59, + "fileObjectId": 59 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-c2f3833d4d4347e7a370f6ca4022c717", + "sha256": "c61b92f0cb27263620175ac11f2ab02ace5a2949efcf7ce82d2f311cbe925a3f", + "size": 642, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "attempt_count": 0, + "chunk_count": 0, + "document_status": "UPLOADED", + "embedding_count": 0, + "generation": "1792334133812207", + "ids": { + "documentId": 62, + "documentVersionId": 62, + "embeddingJobId": 60, + "fileObjectId": 60 + }, + "job_status": "PENDING", + "relations_match": true, + "request_id": "upload-d2053ef0277749fbbacb44ff38feca0e", + "sha256": "db9acd78e12caeeb5c26775c71672ef41bfea93a45c781a186523abd8f13068a", + "size": 1046, + "state": "RESTORED", + "version_status": "UPLOADED" + }, + { + "request_id": "upload-4157706187474b57a8898d172289f611", + "state": "DB_MISSING" + }, + { + "request_id": "upload-0c1e3ccc285348759c881b404b7d6e5c", + "state": "DB_MISSING" + }, + { + "request_id": "upload-5b9715d9f6174a6d90fb341f48834b28", + "state": "DB_MISSING" + }, + { + "request_id": "upload-fd3c8ddd7e4f49dba2cffecdca1edd91", + "state": "DB_MISSING" + }, + { + "request_id": "upload-0d79ad5c61194016bcf99cf56a4ecc5d", + "state": "DB_MISSING" + } + ], + "repair_ran": false, + "source_access_used": false, + "worker_started": false +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/inputs.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/inputs.json new file mode 100644 index 00000000..2583d8d2 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/inputs.json @@ -0,0 +1,426 @@ +{ + "backup": { + "generation": "1792333408912061", + "id": "20261018T142124", + "sha256": "94ea69548b354b04e1639de2f5568f8b52c981b97eb39a413675c6aa637e333e", + "size": 104355840 + }, + "captured_at_utc": "2026-10-18T14:36:37.339419+00:00", + "checks": { + "backup_verified": true, + "identity_available_offsite": true, + "public_config_preserved": true, + "receiver_healthy": true, + "restore_target_isolated": true, + "rollback_ready": true, + "wal_chain_verified": true + }, + "identity_sha256": "4427d99d5a46a822deafc4effe1f5a901bd0b8e4d9a32e7d58fb851a5e25aaee", + "originals": [ + { + "fileObjectId": 1, + "generation": "1792333394887109", + "sha256": "cb6c0dff45951c630ae0b02694bc954e159100cb2495170f8379171c20637ca8", + "size": 1045 + }, + { + "fileObjectId": 2, + "generation": "1792333396369177", + "sha256": "6fca27eab558e63f007d70172241ee91cd3d3c40b868b7b6e5a821fc64d656bc", + "size": 641 + }, + { + "fileObjectId": 3, + "generation": "1792333397665022", + "sha256": "5d12c58a83aae3d5a60f4d776086df081a2d061b89536e4623b5ef66953b15d2", + "size": 1045 + }, + { + "fileObjectId": 4, + "generation": "1792333398846804", + "sha256": "f281ebd98e5360437810e41ee282d7fbc3aea13859a25bff53494c709fd8e71c", + "size": 641 + }, + { + "fileObjectId": 5, + "generation": "1792333400275899", + "sha256": "8a3cbd1a5c5338d68075c056a729d4752a8e31807309e0caa12eb0937ea64902", + "size": 1045 + }, + { + "fileObjectId": 6, + "generation": "1792333801474838", + "sha256": "c1941811f0d2747ce0736b458119484e6973699b0587eecf070d796e7440a328", + "size": 1045 + }, + { + "fileObjectId": 7, + "generation": "1792333802798444", + "sha256": "ad816c20f4f0bc5be74c279296cedd1d37991dd7f59cc55722ce28e8d92ffc02", + "size": 641 + }, + { + "fileObjectId": 8, + "generation": "1792333804125436", + "sha256": "62181f80ab73059b412276fb9bc28765b242dbb26f36ac7dd7ecd2127584e66f", + "size": 1045 + }, + { + "fileObjectId": 9, + "generation": "1792333805402152", + "sha256": "747645e91e4ee34b21c4e6836d3198433a02a66aa867083becacedbf4971bffb", + "size": 641 + }, + { + "fileObjectId": 10, + "generation": "1792333806788228", + "sha256": "cd898e70e7dc88efbfcf5d639615350e27a8de987cfd801addf4eb75f694254b", + "size": 1045 + }, + { + "fileObjectId": 11, + "generation": "1792333808038458", + "sha256": "649e39bc14c365f99405a4293686f5d7abe6c8330e5d6fe4dff58584229baf89", + "size": 642 + }, + { + "fileObjectId": 12, + "generation": "1792333809341078", + "sha256": "cfff30ee9b1a0737eca1ed09ab5883404d15d6f51771610af51f51365f3a8ec7", + "size": 1045 + }, + { + "fileObjectId": 13, + "generation": "1792333810643638", + "sha256": "1b330520eed5e3659cec0d138a26c4a94dfeb0dae80e1aeb949d2bdcc96d94f3", + "size": 642 + }, + { + "fileObjectId": 14, + "generation": "1792333811904377", + "sha256": "319a91561e5255e8e4fa5855ecf45ffc10d5d4567605bed159ec892c1f198389", + "size": 1046 + }, + { + "fileObjectId": 15, + "generation": "1792333813202648", + "sha256": "cdaba700e161a00efb9f2b5376b7d2a819b4116f746253e55f45a408070a3ec2", + "size": 642 + }, + { + "fileObjectId": 16, + "generation": "1792333814551128", + "sha256": "44ed214162a5fe1fb02221dcc2ec494496c231d35dd6e744fe9ebae8fb0cafe2", + "size": 1046 + }, + { + "fileObjectId": 17, + "generation": "1792333815872591", + "sha256": "21c048da2b95a9077ee98f059ed4fd1dceaca716ed8027465222d18121a36f31", + "size": 642 + }, + { + "fileObjectId": 18, + "generation": "1792333817160338", + "sha256": "6321a7fbd1044015e2499bbec3bf05d0194abec00e2cb28bf2109a6c38743a1d", + "size": 1046 + }, + { + "fileObjectId": 19, + "generation": "1792333818431760", + "sha256": "37f65da35477a4d1ff91d17070ef492d6a890ed606568ca1e9af3d6b11aa3bb2", + "size": 642 + }, + { + "fileObjectId": 20, + "generation": "1792333819804887", + "sha256": "ec2fcb418c91cb8b8d0bf78186efc6f99eefea1efe5c2544bc43460ea24b6959", + "size": 1045 + }, + { + "fileObjectId": 21, + "generation": "1792333821311606", + "sha256": "f50e89d1b76a00f28d1051cb44d1b786d9922a56047c9e2bbab7b2f649208c0d", + "size": 642 + }, + { + "fileObjectId": 22, + "generation": "1792333822701925", + "sha256": "538cfe4f5a5ea844e945afef4d23674988270f0aa1f52f27730f98697b7f5a18", + "size": 1045 + }, + { + "fileObjectId": 23, + "generation": "1792333823957315", + "sha256": "125ad2242835634e68f4c82efa85888600dc6f97bd3474765f658133e8c75a52", + "size": 642 + }, + { + "fileObjectId": 24, + "generation": "1792333825235999", + "sha256": "4dd86e6bb9e74cf9b403e286bf0fe3321d3bc7bc13447e61d70bb72246e1ce8f", + "size": 1046 + }, + { + "fileObjectId": 25, + "generation": "1792333826546721", + "sha256": "c24c6c54039e0d1a121cd0d9702a525bc1c996c6409ed1049b0e36b658087e7a", + "size": 642 + }, + { + "fileObjectId": 26, + "generation": "1792333827749600", + "sha256": "dd4d1b12b85e80e7f165418965bfb590546c3d267df5ea6974ffea1ece4ee92f", + "size": 1046 + }, + { + "fileObjectId": 27, + "generation": "1792333829026930", + "sha256": "7199843049649f2d55f8444f8fd795d2f9620569e4d6752e42e3806c6e10af67", + "size": 642 + }, + { + "fileObjectId": 28, + "generation": "1792333830306441", + "sha256": "6f2ab9f8bb530c07b539c787bc926c0161cb7136ee1fab6383d272861133f7d6", + "size": 1046 + }, + { + "fileObjectId": 29, + "generation": "1792333831574400", + "sha256": "1122671a8947a18acc9b49d318e573d3d4f64299328a9a25a4b3103860590dc7", + "size": 642 + }, + { + "fileObjectId": 30, + "generation": "1792333832876770", + "sha256": "5f82ef3f5fc831e1b38624f982436ebd6dd233cf9b36e0af9c12b5d7a42f8ce6", + "size": 1046 + }, + { + "fileObjectId": 31, + "generation": "1792333834140672", + "sha256": "f904af9279ff864430761562c9f2c207dd1539c18c91098a2a43fe3dd484806c", + "size": 642 + }, + { + "fileObjectId": 32, + "generation": "1792333835301903", + "sha256": "8c4163e1723cf99bada4fccc1956b24b36aeb3ff9aaa2b18a1d5a4355c42a0d3", + "size": 1045 + }, + { + "fileObjectId": 33, + "generation": "1792333836593143", + "sha256": "296abbc16a4f567ea8197c9a138b973da051803e082e5b52d627f9b023453dc0", + "size": 642 + }, + { + "fileObjectId": 34, + "generation": "1792334098000268", + "sha256": "4a1955b23633e09c64c251faf53a7f3db408e65c0d31e4131eac14e2e408b036", + "size": 1046 + }, + { + "fileObjectId": 35, + "generation": "1792334099384913", + "sha256": "d517ea588330b3a1009b2a65decaa649f2eb64041318067f7eca6f41310beb19", + "size": 642 + }, + { + "fileObjectId": 36, + "generation": "1792334100779971", + "sha256": "39f7e102ac80a9ba04150f8e357db6123441bdc7cafe5620dc163dda4661ae6f", + "size": 1046 + }, + { + "fileObjectId": 37, + "generation": "1792334102353834", + "sha256": "2a1bfc66a9c58c43d8d51ff1c5391e58dd582fbc386733c41d7c3989fdce9731", + "size": 642 + }, + { + "fileObjectId": 38, + "generation": "1792334103668017", + "sha256": "0b3353283ad6d05cb497416736e7fdda10b57f9a0ffbfb2d316596365791b4b8", + "size": 1046 + }, + { + "fileObjectId": 39, + "generation": "1792334105072235", + "sha256": "6fa3bf5896c7e74952fcd6935204ebf002cda8e6892bd4b34d19839e9849576d", + "size": 642 + }, + { + "fileObjectId": 40, + "generation": "1792334106429281", + "sha256": "604dda88d3af34bfe5fb2ecbd42e1466b7123367aabf2b6479f5187bbde8b1e5", + "size": 1046 + }, + { + "fileObjectId": 41, + "generation": "1792334107767532", + "sha256": "cb1defde74581e45e5606cfe0f948314533669d751d71149d905ac755a067425", + "size": 642 + }, + { + "fileObjectId": 42, + "generation": "1792334109163978", + "sha256": "cca8e204eaa0a10bb320b2979da370eb64ed2bbc54d9ffe190b3bd0206ba97e5", + "size": 1046 + }, + { + "fileObjectId": 43, + "generation": "1792334110385998", + "sha256": "2561e3f2e810a8a4c7c06b6f0d03578d79f27889387f1ad53fde39f5070b63a5", + "size": 642 + }, + { + "fileObjectId": 44, + "generation": "1792334111853252", + "sha256": "7dcaaa488971cae03bfd16b70021146afe24dc803d51107bdb4925b1613af5e8", + "size": 1046 + }, + { + "fileObjectId": 45, + "generation": "1792334113141879", + "sha256": "bac1098d29cc26b2ef7f606bd8f4942f864358d57c99dce5166ea4db3287933e", + "size": 642 + }, + { + "fileObjectId": 46, + "generation": "1792334114523730", + "sha256": "ed8f4d573635655183f52c0a3a7ad5eab7325225399f46ea41372cc7907cefd3", + "size": 1046 + }, + { + "fileObjectId": 47, + "generation": "1792334115989024", + "sha256": "78d4567ad81362950c53cdaf6310b3c6be4280a500ce8422c4473f0974fa178d", + "size": 642 + }, + { + "fileObjectId": 48, + "generation": "1792334117554101", + "sha256": "c3ef81a8b35b77101fa3c1caba153f8dac4c76b0d6ab56e917bbc9ad88440296", + "size": 1046 + }, + { + "fileObjectId": 49, + "generation": "1792334118985900", + "sha256": "6bd1b2aa0f2dd0e8547740529e00c66252bb39153bac639ed6ab0f1655fe3673", + "size": 642 + }, + { + "fileObjectId": 50, + "generation": "1792334120265082", + "sha256": "5c5f78c20f08e1cc1bfe3d8fdecdd7fe41d887bbd6365ddeb02755131ac7e266", + "size": 1046 + }, + { + "fileObjectId": 51, + "generation": "1792334121557537", + "sha256": "7d2dc24cbce3133caeebe9ea1b3f17d7053f1d73083527721b521416de72449a", + "size": 642 + }, + { + "fileObjectId": 52, + "generation": "1792334123050031", + "sha256": "bab62716f6a42b0766db18babe1e4d649e24be3e9e731e6f2553f16e58ec011e", + "size": 1046 + }, + { + "fileObjectId": 53, + "generation": "1792334124486623", + "sha256": "7c1518f5b75aa22b3581da98e94d958e94a4a8bf146ab650a7a245964401c004", + "size": 642 + }, + { + "fileObjectId": 54, + "generation": "1792334125803407", + "sha256": "71f499d0499d9204f28495bf65fb23854f288a8d744a6dbc38c405bc34b319ba", + "size": 1046 + }, + { + "fileObjectId": 55, + "generation": "1792334127146475", + "sha256": "cbe79087c3e591bebb072292ef7c6d41d62f119c03eee75af19a24b7afdb80fe", + "size": 642 + }, + { + "fileObjectId": 56, + "generation": "1792334128442978", + "sha256": "5b6caf7ba08fc075fc7748fec448cbf746b6af1f3c6ae4c0548568ee64f7810a", + "size": 1046 + }, + { + "fileObjectId": 57, + "generation": "1792334129753173", + "sha256": "6acc5225c8a998e1e9d79e823dc96f1c554330c26e68685da025e17f4337657a", + "size": 642 + }, + { + "fileObjectId": 58, + "generation": "1792334131037142", + "sha256": "2bb8c341cf0b163cbd7891f7c0e0ba1e5195136f958c209a12251bd40f114a32", + "size": 1046 + }, + { + "fileObjectId": 59, + "generation": "1792334132300936", + "sha256": "c61b92f0cb27263620175ac11f2ab02ace5a2949efcf7ce82d2f311cbe925a3f", + "size": 642 + }, + { + "fileObjectId": 60, + "generation": "1792334133812207", + "sha256": "db9acd78e12caeeb5c26775c71672ef41bfea93a45c781a186523abd8f13068a", + "size": 1046 + } + ], + "schema_version": 1, + "wals": [ + { + "generation": "1792333416353658", + "name": "00000006.history", + "sha256": "7dbe25c372b72e1132dca125c614a696011f1aeb3c1dfdac9cc8e19055a6d3c6", + "size": 209 + }, + { + "generation": "1792333419682653", + "name": "00000006000000000000002B", + "sha256": "4861c68f46975ba4736e86b9b4472dd5db779a984b7fa14928b1e92c716ee8a6", + "size": 16777216 + }, + { + "generation": "1792333424481173", + "name": "00000006000000000000002C", + "sha256": "9581d9f666be0b253f9dadc2eb72ad872b174f3fd2da33a9b9c3e74debb9d8d9", + "size": 16777216 + }, + { + "generation": "1792333429992965", + "name": "00000006000000000000002D", + "sha256": "cd5a6c199d04384cebceee15e58d497d5e3fdb05e2a53e5955f35679ee38bd55", + "size": 16777216 + }, + { + "generation": "1792333434426895", + "name": "00000006000000000000002E", + "sha256": "3e2ee29d602a650c28e4c290fe8ef1a73c1c108cab06431d86dd4e6c97afc964", + "size": 16777216 + }, + { + "generation": "1792333839694782", + "name": "00000006000000000000002F", + "sha256": "18d14efba2fe22e9444403cffd5707a84ac1587783d7e28f181d6018b183ba81", + "size": 16777216 + }, + { + "generation": "1792334136908930", + "name": "000000060000000000000030", + "sha256": "459e502dfa8b809cc0e9e758f193f956bb19ffcddc4da19560c2e1f41a7d05d3", + "size": 16777216 + } + ] +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/interrupted-job-state.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/interrupted-job-state.json new file mode 100644 index 00000000..a8b3a106 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/interrupted-job-state.json @@ -0,0 +1,8 @@ +{ + "attempt_count": 1, + "chunk_count": 3, + "document_status": "INDEXING", + "embedding_count": 0, + "job_status": "PROCESSING", + "version_status": "EMBEDDING" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/manifest.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/manifest.json new file mode 100644 index 00000000..5ed00eb5 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/manifest.json @@ -0,0 +1,6 @@ +{ + "created_at_utc": "2026-10-18T14:10:55.558+00:00", + "revision": "3bff887a5f6abdf4155957a658599f8f0366a141", + "run_id": "DR-RESTORE-20261018-FRESHNESS1", + "schema_version": 1 +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/operator-activate-receipt.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/operator-activate-receipt.json new file mode 100644 index 00000000..b670e26d --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/operator-activate-receipt.json @@ -0,0 +1,7 @@ +{ + "at_utc": "2026-10-18T14:39:01.755776+00:00", + "result": { + "activated": true + }, + "started_at_utc": "2026-10-18T14:38:17.281562+00:00" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/operator-cleanup-receipt.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/operator-cleanup-receipt.json new file mode 100644 index 00000000..da79d260 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/operator-cleanup-receipt.json @@ -0,0 +1,9 @@ +{ + "at_utc": "2026-10-18T14:53:51.291194+00:00", + "result": { + "owned_resources_removed": true, + "public_config_preserved": true, + "source_cluster_restored": true + }, + "started_at_utc": "2026-10-18T14:52:18.872190+00:00" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/operator-functional-receipt.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/operator-functional-receipt.json new file mode 100644 index 00000000..10515776 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/operator-functional-receipt.json @@ -0,0 +1,36 @@ +{ + "at_utc": "2026-10-18T14:51:39.757916+00:00", + "result": { + "download": { + "at_utc": "2026-10-18T14:39:09.852552+00:00", + "passed": true, + "samples": 4 + }, + "job_resume": { + "at_utc": "2026-10-18T14:40:21.317855+00:00", + "passed": true, + "samples": 1 + }, + "new_download": { + "at_utc": "2026-10-18T14:51:39.754136+00:00", + "passed": true, + "samples": 1 + }, + "new_indexing": { + "at_utc": "2026-10-18T14:51:37.672099+00:00", + "passed": true, + "samples": 1 + }, + "new_upload": { + "at_utc": "2026-10-18T14:40:28.376043+00:00", + "passed": true, + "samples": 1 + }, + "search": { + "at_utc": "2026-10-18T14:39:25.400051+00:00", + "passed": true, + "samples": 1 + } + }, + "started_at_utc": "2026-10-18T14:39:01.842928+00:00" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/operator-restore-receipt.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/operator-restore-receipt.json new file mode 100644 index 00000000..d8552e54 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/operator-restore-receipt.json @@ -0,0 +1,9 @@ +{ + "at_utc": "2026-10-18T14:37:45.124380+00:00", + "result": { + "inputs_sha256": "6bda360bd8f4d2a125f65a8d145adfea76430dcfd231dd6ddc0284729b006606", + "restored": true, + "source_access_used": false + }, + "started_at_utc": "2026-10-18T14:36:37.431253+00:00" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/pre-cutoff-readiness.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/pre-cutoff-readiness.json new file mode 100644 index 00000000..0b8509fa --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/pre-cutoff-readiness.json @@ -0,0 +1,12 @@ +{ + "at_utc": "2026-10-18T14:36:26.951830+00:00", + "backup_verified": true, + "begin_wal": "00000006000000000000002B", + "bootstrap_frozen_before_cutoff": true, + "contiguous_wal_segments": 6, + "license_environment_present": true, + "receiver": { + "streaming_receivers": 1 + }, + "source_code_hashes_unchanged": true +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/preparation-result.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/preparation-result.json new file mode 100644 index 00000000..73183aba --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/preparation-result.json @@ -0,0 +1,6 @@ +{ + "finished_at_utc": "2026-10-18T14:23:57.595523+00:00", + "formal_recovery_executed": false, + "preparation": "PASS", + "run_id": "DR-RESTORE-20261018-FRESHNESS1" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/queue-1-result.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/queue-1-result.json new file mode 100644 index 00000000..34d1958a --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/queue-1-result.json @@ -0,0 +1,18 @@ +{ + "finished_at_utc": "2026-10-18T14:43:55.814234+00:00", + "location": "restored owned logical database; read only", + "method": "identity-checked SQL aggregate embedding_jobs by status", + "observations": { + "jobs_by_status": { + "INDEXED": 27, + "PENDING": 33, + "PROCESSING": 1 + }, + "new_job_status": "PENDING" + }, + "purpose": "새 인덱싱이 대기하는 동안 복원 작업 대기열이 실제로 소진되는지 관측", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-QUEUE1", + "started_at_utc": "2026-10-18T14:43:45.671043+00:00", + "success_criteria": "대기열 카운터와 새 Job 상태를 읽고 추가 작업 생성·상태 변경하지 않음", + "verdict": "OBSERVED" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/queue-1-spec.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/queue-1-spec.json new file mode 100644 index 00000000..af67539a --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/queue-1-spec.json @@ -0,0 +1,8 @@ +{ + "location": "restored owned logical database; read only", + "method": "identity-checked SQL aggregate embedding_jobs by status", + "purpose": "새 인덱싱이 대기하는 동안 복원 작업 대기열이 실제로 소진되는지 관측", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-QUEUE1", + "started_at_utc": "2026-10-18T14:43:45.671043+00:00", + "success_criteria": "대기열 카운터와 새 Job 상태를 읽고 추가 작업 생성·상태 변경하지 않음" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/recovery-backend-events.jsonl b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/recovery-backend-events.jsonl new file mode 100644 index 00000000..42a62b3a --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/recovery-backend-events.jsonl @@ -0,0 +1 @@ +{"at_utc": "2026-10-18T14:38:57.213966+00:00", "event": "application_ready"} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/replay-boundary-result.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/replay-boundary-result.json new file mode 100644 index 00000000..c6c538ec --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/replay-boundary-result.json @@ -0,0 +1,17 @@ +{ + "finished_at_utc": "2026-10-18T14:39:22.640971+00:00", + "last_frozen_wal": "000000060000000000000030", + "location": "licensed OpenSQL separate restored PGDATA; read only", + "method": "restored identity-checked socket SQL: pg_last_wal_replay_lsn and pg_last_xact_replay_timestamp", + "observations": { + "in_recovery": false, + "last_replay_commit_utc": "2026-10-18T14:34:33.013865+00:00", + "replay_lsn": "0/31000000", + "wal_segment_size": "16MB" + }, + "purpose": "외부에 고정한 마지막 WAL과 실제 복원 서버의 재생 끝 위치 대조", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-REPLAY-BOUNDARY", + "started_at_utc": "2026-10-18T14:39:11.960326+00:00", + "success_criteria": "복원 서버 식별 후 비어 있지 않은 replay LSN 관측", + "verdict": "PASS" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/replay-boundary-spec.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/replay-boundary-spec.json new file mode 100644 index 00000000..03aad3eb --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/replay-boundary-spec.json @@ -0,0 +1,8 @@ +{ + "location": "licensed OpenSQL separate restored PGDATA; read only", + "method": "restored identity-checked socket SQL: pg_last_wal_replay_lsn and pg_last_xact_replay_timestamp", + "purpose": "외부에 고정한 마지막 WAL과 실제 복원 서버의 재생 끝 위치 대조", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-REPLAY-BOUNDARY", + "started_at_utc": "2026-10-18T14:39:11.960326+00:00", + "success_criteria": "복원 서버 식별 후 비어 있지 않은 replay LSN 관측" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/runtime-cleanup-observations.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/runtime-cleanup-observations.json new file mode 100644 index 00000000..14bfbed3 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/runtime-cleanup-observations.json @@ -0,0 +1,34 @@ +{ + "containers_removed": true, + "failures": [], + "finished_at_utc": "2026-10-18T14:53:51.289356+00:00", + "gcs": { + "active_objects_remaining": 0, + "deleted_active_objects": { + "database": 9, + "preserved": 61, + "source": 65 + }, + "immediate_physical_erasure_claimed": false, + "soft_delete_policy_may_retain_deleted_objects": true + }, + "private_runtime_data_removed": true, + "processes_removed": true, + "public_config_unchanged": true, + "shared_trial_buckets_retained_for_next_serial_run": true, + "source_objects_removed": true, + "ssh_keys_removed": true, + "started_at_utc": "2026-10-18T14:52:18.872446+00:00", + "target": { + "guard": { + "completed": true, + "patroni_resumed": 2, + "scratch_stopped": true + }, + "scratch_removed": true, + "scratch_stopped": true, + "source_cluster_restored": true, + "target_never_borrowed": false, + "temporary_key_removed": true + } +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/source-backend-events.jsonl b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/source-backend-events.jsonl new file mode 100644 index 00000000..06bc240b --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/source-backend-events.jsonl @@ -0,0 +1,2 @@ +{"at_utc": "2026-10-18T14:16:21.645747+00:00", "event": "application_ready"} +{"at_utc": "2026-10-18T14:20:55.857456+00:00", "event": "application_ready"} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/summary.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/summary.json new file mode 100644 index 00000000..d9cf2e72 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/summary.json @@ -0,0 +1,975 @@ +{ + "execution": "PASS", + "final_audit": { + "acknowledged": 65, + "activation_allowed": true, + "http_error": 0, + "lost_or_incomplete_acknowledged": 5, + "requests": [ + { + "ack_at_utc": "2026-10-18T14:16:35.002+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0296462e118a428da1b889a1310a9f30", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:17:14.038+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ce6a718bdd0f4ace83465acdb3f9d275", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:17:45.307+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-b1f116f44ea449fb90474ed3ffbde583", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:18:19.856+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-123273586c1a4fa0a37270b2550be058", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:20:09.511+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-6e7a22eea289466da0fbb8865647301b", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:24:39.407+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-fb2dd8207fa647fa9d80ffdf4c348c68", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:24:49.127+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-432fd889599b4f88b0d24d6593996a73", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:00.168+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-4aa4c55b25d14d538fcaff279600fda9", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:11.407+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-c8826fde38704624ae5ca21b04789532", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:22.257+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-57d7dce5ad8b4ee2adcf905c04a6f654", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:34.475+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e4d2871c1e404638bab746d1b07f9797", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:45.185+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e584c125eb7e4a5dbe43164f09613254", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:55.924+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e97831e4ebdf4f0398b54d8845adcd49", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:06.927+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e0ed270f022244c19fb49c554ac6ef5b", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:18.268+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-815f489bc96c48d187abac925f3271f4", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:28.971+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e34da01ee9474549b9002f280a723de1", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:39.772+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-5cd702f34097434ab4544b79573c47fc", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:50.450+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a7c7983377aa4cb38474c2c4b639d200", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:02.777+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-8fe5e921c6a64f2bafb9f5b24e23345d", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:12.730+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a7fbd0d4358842d99a18392ea28d5ada", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:25.101+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-52d582fc9eb64427845662582d8a3bbb", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:35.059+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-5cbd3325ea7e4b97b4db3ade4fc2fa7a", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:45.864+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-8af4ad876cd043f0a906915c5b5c9e76", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:57.382+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a2a79e66098444ce812e0e7f2b55741c", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:07.877+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-f35ec7a4a77b466186a1bb8faa995b8f", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:18.857+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-7b83a8a9b01544af88bbec7e9e5a8b94", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:30.164+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-106767bf29ce4751a114beffed1dcf5d", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:40.871+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-c2ec085d264a49c9ab6f691c6ae4aa9b", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:51.926+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-cef30e0526c24946b07bc80888bd0290", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:03.139+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-496f0a710ecb4981a3a644fdea27a2a5", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:13.966+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-3f79f77ddbd24cf1a9f682abcf928cb6", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:25.157+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-63141b14d77e4900965f43624358e6ce", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:36.265+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-52a198f8bd8a4c1f832134222e5505c5", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:30:51.420+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-d029610fe4234219b1db66f348b7b114", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:30:59.249+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ec97bd5c298f483d9cf98729cea26940", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:06.618+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0b7cf70ddca94790b02c8e8024fb040d", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:13.944+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-df6017955c5447b9bf097221f6a7d8be", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:22.044+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ba75b05d8e5d477abb3d2a1180a95077", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:29.191+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a5be52472400424db8f52ed7a54c2e83", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:36.338+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-eb8fb6803498447a94b89421596acf47", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:43.833+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a13bf9f28e9e4643bdea46f2d4a2bd62", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:51.023+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-813ddd2e5f1c4a5d827384f84a0005aa", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:58.501+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-3c1490dc0cba43a5828e902ad9ccb4bf", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:06.107+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0d42b4eeef9e49958d54034debf32404", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:13.448+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-057d9136fc534bd7a05d4a3f2fe91437", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:21.026+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-64250f659e794f5cb774e58461b97e4e", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:28.380+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-57ac7e89e46c4e7c857a794287fca81a", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:36.479+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-2b25d7a9090f48d088f40a97b846d14f", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:44.145+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-2e705577fed54f4e8ad8f6f9e17fe774", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:51.475+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-72aca0b7e4864f92990fe560dbcc0a69", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:58.844+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ddd360391bc64e6c9d20673ae26a73f3", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:06.483+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-3cc178f1e0904fc39ac08a077e048a50", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:15.806+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-8b600c6bb6114cdc8a084fb3bcc63b91", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:27.131+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-9e764d5ec32145c18c4c1af2d22d9644", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:38.097+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-17641dcaf2874cf1bdb416a174662e89", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:48.947+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-70d798ac940240a196a56d6652604d63", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:00.237+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-9b6ec2049bf9488f9d4d6e475576a0c2", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:11.280+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-b1c28ab2ab4245cc83d9a1877e3110e6", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:21.936+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-c2f3833d4d4347e7a370f6ca4022c717", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:32.975+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-d2053ef0277749fbbacb44ff38feca0e", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:35:47.908+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-4157706187474b57a8898d172289f611", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:35:54.975+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0c1e3ccc285348759c881b404b7d6e5c", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:36:02.057+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-5b9715d9f6174a6d90fb341f48834b28", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:36:09.391+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-fd3c8ddd7e4f49dba2cffecdca1edd91", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:36:16.477+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0d79ad5c61194016bcf99cf56a4ecc5d", + "restored_state": "DB_MISSING" + } + ], + "restored_acknowledged": 60, + "unknown_http": 0 + }, + "functional_checks": { + "download": { + "at_utc": "2026-10-18T14:39:09.852552+00:00", + "from_cutoff_seconds": 152.5, + "passed": true, + "samples": 4 + }, + "job_resume": { + "at_utc": "2026-10-18T14:40:21.317855+00:00", + "from_cutoff_seconds": 223.965, + "passed": true, + "samples": 1 + }, + "new_download": { + "at_utc": "2026-10-18T14:51:39.754136+00:00", + "from_cutoff_seconds": 902.401, + "passed": true, + "samples": 1 + }, + "new_indexing": { + "at_utc": "2026-10-18T14:51:37.672099+00:00", + "from_cutoff_seconds": 900.319, + "passed": true, + "samples": 1 + }, + "new_upload": { + "at_utc": "2026-10-18T14:40:28.376043+00:00", + "from_cutoff_seconds": 231.023, + "passed": true, + "samples": 1 + }, + "search": { + "at_utc": "2026-10-18T14:39:25.400051+00:00", + "from_cutoff_seconds": 168.047, + "passed": true, + "samples": 1 + } + }, + "initial_audit": { + "acknowledged": 65, + "activation_allowed": true, + "http_error": 0, + "lost_or_incomplete_acknowledged": 5, + "requests": [ + { + "ack_at_utc": "2026-10-18T14:16:35.002+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0296462e118a428da1b889a1310a9f30", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:17:14.038+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ce6a718bdd0f4ace83465acdb3f9d275", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:17:45.307+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-b1f116f44ea449fb90474ed3ffbde583", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:18:19.856+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-123273586c1a4fa0a37270b2550be058", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:20:09.511+00:00", + "cohort": "functional", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-6e7a22eea289466da0fbb8865647301b", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:24:39.407+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-fb2dd8207fa647fa9d80ffdf4c348c68", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:24:49.127+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-432fd889599b4f88b0d24d6593996a73", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:00.168+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-4aa4c55b25d14d538fcaff279600fda9", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:11.407+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-c8826fde38704624ae5ca21b04789532", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:22.257+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-57d7dce5ad8b4ee2adcf905c04a6f654", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:34.475+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e4d2871c1e404638bab746d1b07f9797", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:45.185+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e584c125eb7e4a5dbe43164f09613254", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:25:55.924+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e97831e4ebdf4f0398b54d8845adcd49", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:06.927+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e0ed270f022244c19fb49c554ac6ef5b", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:18.268+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-815f489bc96c48d187abac925f3271f4", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:28.971+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-e34da01ee9474549b9002f280a723de1", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:39.772+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-5cd702f34097434ab4544b79573c47fc", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:26:50.450+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a7c7983377aa4cb38474c2c4b639d200", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:02.777+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-8fe5e921c6a64f2bafb9f5b24e23345d", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:12.730+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a7fbd0d4358842d99a18392ea28d5ada", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:25.101+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-52d582fc9eb64427845662582d8a3bbb", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:35.059+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-5cbd3325ea7e4b97b4db3ade4fc2fa7a", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:45.864+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-8af4ad876cd043f0a906915c5b5c9e76", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:27:57.382+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a2a79e66098444ce812e0e7f2b55741c", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:07.877+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-f35ec7a4a77b466186a1bb8faa995b8f", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:18.857+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-7b83a8a9b01544af88bbec7e9e5a8b94", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:30.164+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-106767bf29ce4751a114beffed1dcf5d", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:40.871+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-c2ec085d264a49c9ab6f691c6ae4aa9b", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:28:51.926+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-cef30e0526c24946b07bc80888bd0290", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:03.139+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-496f0a710ecb4981a3a644fdea27a2a5", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:13.966+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-3f79f77ddbd24cf1a9f682abcf928cb6", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:25.157+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-63141b14d77e4900965f43624358e6ce", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:29:36.265+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-52a198f8bd8a4c1f832134222e5505c5", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:30:51.420+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-d029610fe4234219b1db66f348b7b114", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:30:59.249+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ec97bd5c298f483d9cf98729cea26940", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:06.618+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0b7cf70ddca94790b02c8e8024fb040d", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:13.944+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-df6017955c5447b9bf097221f6a7d8be", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:22.044+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ba75b05d8e5d477abb3d2a1180a95077", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:29.191+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a5be52472400424db8f52ed7a54c2e83", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:36.338+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-eb8fb6803498447a94b89421596acf47", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:43.833+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-a13bf9f28e9e4643bdea46f2d4a2bd62", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:51.023+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-813ddd2e5f1c4a5d827384f84a0005aa", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:31:58.501+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-3c1490dc0cba43a5828e902ad9ccb4bf", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:06.107+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0d42b4eeef9e49958d54034debf32404", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:13.448+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-057d9136fc534bd7a05d4a3f2fe91437", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:21.026+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-64250f659e794f5cb774e58461b97e4e", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:28.380+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-57ac7e89e46c4e7c857a794287fca81a", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:36.479+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-2b25d7a9090f48d088f40a97b846d14f", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:44.145+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-2e705577fed54f4e8ad8f6f9e17fe774", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:51.475+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-72aca0b7e4864f92990fe560dbcc0a69", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:32:58.844+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-ddd360391bc64e6c9d20673ae26a73f3", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:06.483+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-3cc178f1e0904fc39ac08a077e048a50", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:15.806+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-8b600c6bb6114cdc8a084fb3bcc63b91", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:27.131+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-9e764d5ec32145c18c4c1af2d22d9644", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:38.097+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-17641dcaf2874cf1bdb416a174662e89", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:33:48.947+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-70d798ac940240a196a56d6652604d63", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:00.237+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-9b6ec2049bf9488f9d4d6e475576a0c2", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:11.280+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-b1c28ab2ab4245cc83d9a1877e3110e6", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:21.936+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-c2f3833d4d4347e7a370f6ca4022c717", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:34:32.975+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-d2053ef0277749fbbacb44ff38feca0e", + "restored_state": "RESTORED" + }, + { + "ack_at_utc": "2026-10-18T14:35:47.908+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-4157706187474b57a8898d172289f611", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:35:54.975+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0c1e3ccc285348759c881b404b7d6e5c", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:36:02.057+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-5b9715d9f6174a6d90fb341f48834b28", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:36:09.391+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-fd3c8ddd7e4f49dba2cffecdca1edd91", + "restored_state": "DB_MISSING" + }, + { + "ack_at_utc": "2026-10-18T14:36:16.477+00:00", + "cohort": "loss", + "http_outcome": "ACKNOWLEDGED", + "request_id": "upload-0d79ad5c61194016bcf99cf56a4ecc5d", + "restored_state": "DB_MISSING" + } + ], + "restored_acknowledged": 60, + "unknown_http": 0 + }, + "reason": null, + "rpo_guarantee": "NOT_EVALUATED", + "run_id": "DR-RESTORE-20261018-FRESHNESS1", + "scope": "isolated_document_download_search_indexing" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-0-result.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-0-result.json new file mode 100644 index 00000000..a941a3cd --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-0-result.json @@ -0,0 +1,11 @@ +{ + "backup_bytes": 104355840, + "completed_wals": 5, + "elapsed_seconds": 67.284, + "finished_at_utc": "2026-10-18T14:23:57.591442+00:00", + "preserved_originals": 5, + "purpose": "완성 WAL·identity·기준 백업과 원본 보존본의 외부 전송 및 바이트 대조", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-TRANSFER-0", + "started_at_utc": "2026-10-18T14:22:50.309279+00:00", + "verdict": "PASS" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-0-spec.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-0-spec.json new file mode 100644 index 00000000..9b1ad145 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-0-spec.json @@ -0,0 +1,5 @@ +{ + "purpose": "완성 WAL 및 원본의 외부 보존", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-TRANSFER-0", + "started_at_utc": "2026-10-18T14:22:50.309167+00:00" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-0-wal-barrier.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-0-wal-barrier.json new file mode 100644 index 00000000..8a8f373a --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-0-wal-barrier.json @@ -0,0 +1,22 @@ +{ + "advance_lsn": "0/2F000090", + "archive_attempts": [ + { + "at_utc": "2026-10-18T14:23:01.683264+00:00", + "elapsed_seconds": 0.14, + "exit_code": 0 + }, + { + "at_utc": "2026-10-18T14:23:02.834416+00:00", + "elapsed_seconds": 0.146, + "exit_code": 0 + } + ], + "at_utc": "2026-10-18T14:23:57.583636+00:00", + "covered_acknowledged_requests": 5, + "generation": "1792333434426895", + "remote_bytes_verified": true, + "required_wal": "00000006000000000000002E", + "sha256": "3e2ee29d602a650c28e4c290fe8ef1a73c1c108cab06431d86dd4e6c97afc964", + "size": 16777216 +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-1-result.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-1-result.json new file mode 100644 index 00000000..2bdeb788 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-1-result.json @@ -0,0 +1,11 @@ +{ + "backup_bytes": 104355840, + "completed_wals": 6, + "elapsed_seconds": 66.737, + "finished_at_utc": "2026-10-18T14:30:43.010845+00:00", + "preserved_originals": 33, + "purpose": "완성 WAL·identity·기준 백업과 원본 보존본의 외부 전송 및 바이트 대조", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-TRANSFER-1", + "started_at_utc": "2026-10-18T14:29:36.275974+00:00", + "verdict": "PASS" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-1-spec.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-1-spec.json new file mode 100644 index 00000000..6fca7377 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-1-spec.json @@ -0,0 +1,5 @@ +{ + "purpose": "완성 WAL 및 원본의 외부 보존", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-TRANSFER-1", + "started_at_utc": "2026-10-18T14:29:36.275540+00:00" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-1-wal-barrier.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-1-wal-barrier.json new file mode 100644 index 00000000..85213269 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-1-wal-barrier.json @@ -0,0 +1,27 @@ +{ + "advance_lsn": "0/30000090", + "archive_attempts": [ + { + "at_utc": "2026-10-18T14:29:46.962446+00:00", + "elapsed_seconds": 0.161, + "exit_code": 0 + }, + { + "at_utc": "2026-10-18T14:29:48.110029+00:00", + "elapsed_seconds": 0.142, + "exit_code": 0 + }, + { + "at_utc": "2026-10-18T14:29:49.266988+00:00", + "elapsed_seconds": 0.152, + "exit_code": 0 + } + ], + "at_utc": "2026-10-18T14:30:43.006289+00:00", + "covered_acknowledged_requests": 33, + "generation": "1792333839694782", + "remote_bytes_verified": true, + "required_wal": "00000006000000000000002F", + "sha256": "18d14efba2fe22e9444403cffd5707a84ac1587783d7e28f181d6018b183ba81", + "size": 16777216 +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-2-result.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-2-result.json new file mode 100644 index 00000000..a6499f59 --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-2-result.json @@ -0,0 +1,11 @@ +{ + "backup_bytes": 104355840, + "completed_wals": 7, + "elapsed_seconds": 67.315, + "finished_at_utc": "2026-10-18T14:35:40.297898+00:00", + "preserved_originals": 60, + "purpose": "완성 WAL·identity·기준 백업과 원본 보존본의 외부 전송 및 바이트 대조", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-TRANSFER-2", + "started_at_utc": "2026-10-18T14:34:32.984484+00:00", + "verdict": "PASS" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-2-spec.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-2-spec.json new file mode 100644 index 00000000..14a00e3a --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-2-spec.json @@ -0,0 +1,5 @@ +{ + "purpose": "완성 WAL 및 원본의 외부 보존", + "run_id": "DR-RESTORE-20261018-FRESHNESS1-TRANSFER-2", + "started_at_utc": "2026-10-18T14:34:32.984299+00:00" +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-2-wal-barrier.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-2-wal-barrier.json new file mode 100644 index 00000000..e524d15d --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/DR-RESTORE-20261018-FRESHNESS1/transfer-2-wal-barrier.json @@ -0,0 +1,22 @@ +{ + "advance_lsn": "0/31000090", + "archive_attempts": [ + { + "at_utc": "2026-10-18T14:34:43.738557+00:00", + "elapsed_seconds": 0.143, + "exit_code": 0 + }, + { + "at_utc": "2026-10-18T14:34:44.892523+00:00", + "elapsed_seconds": 0.153, + "exit_code": 0 + } + ], + "at_utc": "2026-10-18T14:35:40.286094+00:00", + "covered_acknowledged_requests": 60, + "generation": "1792334136908930", + "remote_bytes_verified": true, + "required_wal": "000000060000000000000030", + "sha256": "459e502dfa8b809cc0e9e758f193f956bb19ffcddc4da19560c2e1f41a7d05d3", + "size": 16777216 +} diff --git a/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/export-index.json b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/export-index.json new file mode 100644 index 00000000..834ba97c --- /dev/null +++ b/docs/test-results/wal-offsite-recovery-recheck-20261018/evidence/export-index.json @@ -0,0 +1,332 @@ +{ + "artifacts": [ + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/manifest.json", + "source_sha256": "979f1c8ae7e4979a1edfe57802872f6b58ec82c867cf561a61ae29b7d99100c7", + "export_sha256": "979f1c8ae7e4979a1edfe57802872f6b58ec82c867cf561a61ae29b7d99100c7", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/events.jsonl", + "source_sha256": "5a1523668f658ceeb33e618d7f668348353a360fdaa9ca23a81aeb42e02d9157", + "export_sha256": "5a1523668f658ceeb33e618d7f668348353a360fdaa9ca23a81aeb42e02d9157", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/inputs.json", + "source_sha256": "6bda360bd8f4d2a125f65a8d145adfea76430dcfd231dd6ddc0284729b006606", + "export_sha256": "6bda360bd8f4d2a125f65a8d145adfea76430dcfd231dd6ddc0284729b006606", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/summary.json", + "source_sha256": "7917be9d18923da0c98d4a8f1d1e10725e8bc68ce8ced8db0e6b6d6176c41b65", + "export_sha256": "7917be9d18923da0c98d4a8f1d1e10725e8bc68ce8ced8db0e6b6d6176c41b65", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/formal-spec.json", + "source_sha256": "a78e1da0487ad8f16fac63221b15cbbf2b1f468ee6d3eea163724d30b2403783", + "export_sha256": "61a378c0fbed6c74afae924e898b138c598e096e1aab8e99ace263cc9bb3812b", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/replay-boundary-spec.json", + "source_sha256": "05d1f04e50e15b83a1228921a92cafa4ffa89c52b3bce978164227e6b5199a17", + "export_sha256": "9d2d3d8c2cb815b27a18668f1ba970142dc28f4ad529f729c0d7cd5a701cecce", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/replay-boundary-result.json", + "source_sha256": "a30c075479dfee273b2ec62b9267e2403692c0718b7e04eacb4f2c2e92f288d8", + "export_sha256": "5f1a364b14c1f5c29d8935754ffdfedf032fc3f7c2db18e21338fd12f9549829", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/queue-1-spec.json", + "source_sha256": "f50f284cb1ea56bf440eb8991699dcb0407047995c1ea633bf37850674b870a6", + "export_sha256": "1d58c3cc89a043d564be73a1a134c901813f353561fd3cb4a82208168efca367", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/queue-1-result.json", + "source_sha256": "2873943c713387c26d081ccc248715356aada4955dccf74a0bcb3f3d16198775", + "export_sha256": "cbb15c611065b06e3b566e4f064c73a899a647276a3040891a051948c25bd26b", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/artifact-identity.json", + "source_sha256": "833f8c40e7811030dd2eaa3593b00bb666a1e7367e6532eda17fdab7acfb9153", + "export_sha256": "9310aceeb8a96df5f47677be42937b7c3c1797b3741fda249caa62ddb5641e31", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/preparation-result.json", + "source_sha256": "bb8078c527f6bc84bd7da8c2fd49259a5b4dba8114f85197abb7140a63ed6800", + "export_sha256": "47d88cb453d2656dc59899a2cff238681720be5aff09763962f12cc2d8ae13ed", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/pre-cutoff-readiness.json", + "source_sha256": "1ec1ceee8b35a90e1556806b7fea1de1e9014b0d097bdafbaaff47f8ac5ff62d", + "export_sha256": "4b7ed1211464ed5bfff0c8f1e8ee6926339fcad917689e5fc0fd9c14571f23da", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/initial-snapshot.json", + "source_sha256": "74e98ed51cca1798c6249d89942d14c153ed46ace2aff37ce2946ad2759fff3c", + "export_sha256": "74e98ed51cca1798c6249d89942d14c153ed46ace2aff37ce2946ad2759fff3c", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/initial-audit.json", + "source_sha256": "dcd22a73b4be1794be1ba97cdb6efe5151c056ae3210c2678e14c2f92afd5a44", + "export_sha256": "dcd22a73b4be1794be1ba97cdb6efe5151c056ae3210c2678e14c2f92afd5a44", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/checkpoint-coverage.json", + "source_sha256": "e62afee23efa205e9adf40ef66b314fd705aac5d849f12024c7d420d26cd5aa5", + "export_sha256": "f1dfe4c9e7bd39208e396526900c93b8714fcd507b28b6f8a90056bbbeef7751", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/final-snapshot.json", + "source_sha256": "fc15ac52eeaaec8eaaa072e2bbf7da27899dd569f02e6d45e1aca5283f77b8c4", + "export_sha256": "fc15ac52eeaaec8eaaa072e2bbf7da27899dd569f02e6d45e1aca5283f77b8c4", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/final-audit.json", + "source_sha256": "dcd22a73b4be1794be1ba97cdb6efe5151c056ae3210c2678e14c2f92afd5a44", + "export_sha256": "dcd22a73b4be1794be1ba97cdb6efe5151c056ae3210c2678e14c2f92afd5a44", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/functional-observations.json", + "source_sha256": "84af7201aa5bc4d513e9d6fd5500d8137ade4da595df9733852135cf432f6550", + "export_sha256": "44a1144795fefc62b3af4b7d75c45d39643153b059979302be0875e1d023e3e9", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/operator-restore-receipt.json", + "source_sha256": "4bd1522aae803aa94109b22bd5676a76ba568723379a5c22fa5132b325e86562", + "export_sha256": "52c037463cfd3250318818407d297522f86f4a0c58a7f9f2da590dcefbb24161", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/operator-activate-receipt.json", + "source_sha256": "a9b33411a27f35492a09b722f769e47cb34d9cad07166d73cd576c2f7c8bd099", + "export_sha256": "08a1f5b804c168c69442b6434463e80538891e53aa36d1e67a6ac040d6d2dceb", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/operator-functional-receipt.json", + "source_sha256": "0a9a7633ff17ba4d6aed792a7aa3f2e1ab392ce66abdc88e9fd4dbb1800f2f03", + "export_sha256": "3500501fe1799cf9fdc14d08edc152cdde87058bd3d2e37222e2dab290ec3c0a", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/operator-cleanup-receipt.json", + "source_sha256": "c9895dd14e27b276d99436ebe13d0fb3b5eeb524b5135318e8087f8ec1510612", + "export_sha256": "148617e8d33beaf000dac0e06c3a7efe9ed9908a94fc017655d5fe15489d1af5", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/runtime-cleanup-observations.json", + "source_sha256": "f051f5535eb4af01ffbb4d5bd75a8f88fc75a01e17fc6e1352048fccadbeeb44", + "export_sha256": "116f3c3f74efc0b9a76f91573a95c55046b43bc37c2814bf52f4c57f7dc37a19", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/interrupted-job-state.json", + "source_sha256": "933451ff5422c5a7a85b82784feab3da50dfd8dd3024ea1338b2bf21527dd791", + "export_sha256": "4dd28e2fd04a60d64f27b09d2771ccaee501d841ef7444679b410821fd63222c", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/embedding-events.jsonl", + "source_sha256": "cf70d6f04a5add03f61585d2b356bad37b431fe1ff196b36e3e470011d478781", + "export_sha256": "cf70d6f04a5add03f61585d2b356bad37b431fe1ff196b36e3e470011d478781", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/source-backend-events.jsonl", + "source_sha256": "1c9431eb0234ffc258b5156d12a3c252bb4e35df2e7aac0517d3e60a83c2247b", + "export_sha256": "1c9431eb0234ffc258b5156d12a3c252bb4e35df2e7aac0517d3e60a83c2247b", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/recovery-backend-events.jsonl", + "source_sha256": "111b602e55f89a4d110f31beab2d19d67b425bb13b50d209828c2a13acc5ecb6", + "export_sha256": "111b602e55f89a4d110f31beab2d19d67b425bb13b50d209828c2a13acc5ecb6", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/DATABASE-result.json", + "source_sha256": "bdb5db7d082902c8f061866fe2268922ee21b8723d333f81a85ff8fbc3566fc6", + "export_sha256": "bc8b5133ad76d0ab0d21e33dd7c01fff9703b54a127dbafb775eada94be037b0", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/TOOLING-result.json", + "source_sha256": "9f85cc3172a8b9f225dab73e9321befd908b5dd2abedea5327cfb31bc8f39635", + "export_sha256": "b9f7aa20d9bf40dbb5d8fb1e155f3caa7e15738d45d7aab7d3b8fec91ffb65b2", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/SERVICES-result.json", + "source_sha256": "910f042409924aa8999b3f2b810b2365e1fe193a8956124b979c7d9aed80f364", + "export_sha256": "2d0e512f50dfdb721c2be77bcdfa2c8edbd135f0a4b31c86a0d4b62eb4250f0b", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/CORPUS-result.json", + "source_sha256": "8fe5d85278bdebdd12e6728e044301099e20085b621acc4c60ecd24eba4784e9", + "export_sha256": "8a3efefc75bcb10c3b81539b67fbd38b6083a0c42a8edf1e157252eb32605e75", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/RECEIVER-result.json", + "source_sha256": "1d9e28d0b1d6b2e5cde75c9c633562e811c902a5f01630085403ac0cd4ac5406", + "export_sha256": "61a6a455ef2f229158a746eba061b307e28f83997711ecebfa04123c51a708f5", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/INTERRUPT-result.json", + "source_sha256": "0e18ffeb7ccc50b26717853d3d1f5b711fbf3935826fb389c28440ec13bde06e", + "export_sha256": "b5e2685becb2296d5ccfdadfe3d45107c9ab12af8273bce737d87a097c79a3f1", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/BACKUP-result.json", + "source_sha256": "e5db83184e98e695fbe6caa644042fd4ed13df47e30c45ba409883c0e83be452", + "export_sha256": "83f7dabd16938d3b9c5c8773daa76d431ba785a1a3387465b553b019c550317c", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/BOOTSTRAP-result.json", + "source_sha256": "e2039c9473784bd16b176a0b86bbffac6c880b8e4fb42182a01cbd31ecfff8b4", + "export_sha256": "3872f0fa2e6ccaae9c11a5f7981a392ad19dcd67a4d3aeaefd7bb15542366f29", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/OFFSITE-result.json", + "source_sha256": "f1614a9be85d70ab40ba8fad542121a4b8aaeed42d400b5b19ac9e7deec1f8d3", + "export_sha256": "236d76b3a1630002cb3190a5653fa1ca874546d355d374ead0020e10094d37ac", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/transfer-0-spec.json", + "source_sha256": "5e069e2d3b74cb4d113cc5d58e09056e2e6ddac74abea461c05298b049335850", + "export_sha256": "7db0db522ec2dd8856c92ebe4eb91e91f94a227709a0a3ed2f0c9bdbb8ff7ce8", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/transfer-0-result.json", + "source_sha256": "9f6bb1de3c3127229be2ddf593aab3b6a126cee55f6e9f7aded2eee696920a34", + "export_sha256": "a14ee18a712eef2d86bc4c7aa6d848fb48196964bd359bf378b5002229b914bf", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/transfer-0-wal-barrier.json", + "source_sha256": "24c7ca5d78178f4ead46077ffdacfcd9254189a6e3e6c449aba10aef5a42cf42", + "export_sha256": "cb7ac07728bcf9f303dc399b86a4618891be70f6d44b5ac87586ca73d65184e6", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/transfer-1-spec.json", + "source_sha256": "d58404e7dd4780ecd69ba3e932af9c0b279520c131c901a8b261745523b41405", + "export_sha256": "ff2e8bfc60cd09c654e41bd3d016ad2a52d8ea10f992044e7b3939458744ac17", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/transfer-1-result.json", + "source_sha256": "0e0becc313d8065fa8aca66405b61e8b5394d5736bac3c8800519659ba9c1ce6", + "export_sha256": "6ecb2da3d1c4ec0d76d1a7d078718ebb0c002d3a78671013cb4e9ed64edd69b7", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/transfer-1-wal-barrier.json", + "source_sha256": "1338af7555a72fdc519b811895b07958af13bc3f21e816c6bcbd8bb984d38afb", + "export_sha256": "4424aa7dff20f59ee5c83c6c3b7dcd6f99b59c18a758d5c6a43bfbc713446d2a", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/transfer-2-spec.json", + "source_sha256": "cc6daac85a05a39ff3ef8e6400399d4487b9c40e89bcdb0fd8e0e70fd930c59c", + "export_sha256": "2134c8aed5f080374ca7f81b550cc761a7244b0f7839dbc50ab6ffe7d5bfbc22", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/transfer-2-result.json", + "source_sha256": "cc5411201d9ebf9bf56ef0ca17072269ae119ccc6de56a3575380db863428abc", + "export_sha256": "11d87cc922cdfb685e74487e527d9e16bbdd2b148db9200f23856e4453eeed70", + "redacted": false + }, + { + "artifact": "DR-RESTORE-20261018-FRESHNESS1/transfer-2-wal-barrier.json", + "source_sha256": "9cca03a63305aa4ac6fb01cbb7f61fe67af5185a66b5a5e97b61840cfecbc628", + "export_sha256": "6bec8e4e8d265cafb26c38620d62616cd3f180cd6eaa252d8ed72a779be22371", + "redacted": false + }, + { + "artifact": "metrics.json", + "source_sha256": "728f330812864f8864eb9b1d8eb0aa2a782b461a40b591f2bd39910c4c29899e", + "export_sha256": "1433ce6c319bac506c86d766776b03dab2b1429aa8a00a5df65822270412cd61", + "redacted": false + }, + { + "artifact": "task-cleanup-r1/spec.json", + "source_sha256": "723920a85a24bc1ef3339696d643a03a791d3369255a40016696b260f0ecbe1b", + "export_sha256": "260a24e6f3519dcef389e9f8c4c4e663addf6d390beb28302cc315778727e61c", + "redacted": false + }, + { + "artifact": "task-cleanup-r1/result.json", + "source_sha256": "af148f5108abb846b14dd2678c170a4216c7dee43de018e5d5ba2562a8a397e6", + "export_sha256": "c6087cb72933050446e342980ab3c42c731964c0df6de6fabbefe704e1bee7b7", + "redacted": false + }, + { + "artifact": "local-preservation-r1/spec.json", + "source_sha256": "1fb5e8f9bc5014d79eeeaed2d4c6e52d6ce37c863664d0b649f03601c86981c3", + "export_sha256": "6251cd8b73eb34d0338961eb1a76f92c032df383da8c8f7fee85cded8d60f9a4", + "redacted": false + }, + { + "artifact": "local-preservation-r1/result.json", + "source_sha256": "273eb377b267719701775187c41d6872525a74ba14816bb5cf5824eb822a2bf2", + "export_sha256": "724d5fd405e3532092ac6fecf931ca6bcbd8a48d07461d87bdb9c72be4d327bd", + "redacted": false + } + ], + "checks": [ + { + "pattern": "gs://", + "matches": 0 + }, + { + "pattern": "(?