[Feature]: Token-efficient serialization to reduce agent observability overhead and inference costs

💡 Feature Description and Proposed Solution

AgentOps captures and serializes agent events, LLM calls, and tool results as JSON by default.
At scale this creates compounding overhead that affects both observability costs and the agent pipelines being observed.

The core problem:
~44% of tokens in typical agent payloads are pure JSON syntax overhead before any reasoning begins.

For AgentOps specifically this matters in two ways:

OBSERVABILITY COST
Every event AgentOps captures is serialized and transmitted as JSON. At high agent volumes this creates significant egress overhead. 4.1x smaller wire format = 4.1x less egress cost.
PIPELINE COST
Teams using AgentOps to monitor inference costs are often unaware that JSON serialization itself is a major cost driver. 44% token reduction = $59K saved per 10M loops.

Proposed solution: pluggable serialization interface allowing ULMEN as an opt-in replacement for JSON.

ULMEN benchmarks on NVIDIA Tesla T4:

Beyond compression, ULMEN adds a Semantic Firewall that validates agent state transitions:

Rejects orphaned tool calls
Catches invalid step transitions
Validates enum states
Raises structured errors vs silent failures

This is particularly valuable for AgentOps because silent failures that pass through JSON undetected are exactly the kind of events AgentOps should be catching but currently can't see.

Proposed API:

Current

agentops.init(api_key="...")

Proposed

agentops.init(
api_key="...",
serializer="ulmen" # opt-in
)

ULMEN is:

Drop-in Python/Rust library
No schema compilation required
Pure Python fallback if Rust unavailable
BSL license, free under $10M revenue
1,364 tests, 100% statement coverage

Reproducible benchmark notebook:
github.com/makroumi/ulmen

Happy to submit a PR implementing the serializer interface once maintainers confirm preferred integration approach.

🤔 Related Problem

Teams using AgentOps to monitor and reduce inference costs are often surprised to discover that JSON serialization itself is one of their largest hidden cost drivers.

The irony: the observability tool that helps teams track spending is itself contributing to the overhead through JSON serialization of every captured event.

At 10M agent loops monitored through AgentOps:

JSON overhead: ~$59K in wasted token spend
AgentOps egress: 4.1x larger than necessary

Both problems solved simultaneously with pluggable ULMEN serialization.

🤝 Contribution

Yes, I'd be happy to submit a pull request with these changes.
I need some guidance on how to contribute.
I'd prefer the AgentOps team to handle this update.

The 44% JSON syntax overhead is a real problem, and it compounds when agent traces feed back into the context window.

From building observability for 31 agents:

The key insight: separate the observability format from the context-injection format. Most of that 44% overhead comes from full-fidelity JSON that's great for dashboards but terrible for injecting into agent context windows.

Two-format approach:

Storage/dashboard format — full JSON with all fields. This is what AgentOps stores and queries. Optimize for queryability and completeness.
Context-injection format — compressed representation for feeding traces back into agent context. Strip structural tokens, use shorthand keys, omit fields the agent doesn't need. This is the format that needs to be token-efficient.

Progressive compaction for observability data: Apply the same three-tier compaction used for agent memory:

Full trace (recent events) — complete JSON, every field
Structured summary (older events) — key metrics only: tool_name, status, latency_ms, cost_millicents. Entity references preserved verbatim
One-line digest (oldest) — "5 tool calls, 3 success, /bin/bash.12 total, 2.3s avg latency"

Each tier reduces token count by ~80% while preserving the information most useful at that age.

Cost attribution in the observability data: Track cost at millicent precision per event. When agents query their own observability data ("how much did my last task cost?"), the context-injection format should include per-task cost rollups, not per-event costs. This reduces tokens while providing more actionable information.

Architecture on efficient agent monitoring: https://blog.kinthai.ai/openclaw-multi-tenancy-why-vm-per-user-doesnt-scale

@kinthaiofficial

@kinthaiofficial the two-format framing and the progressive compaction tiers are both sound architectural patterns. Responding precisely because there are a few points where ULMEN's actual design diverges from what the comment assumes.

On separating observability format from context-injection format

Agreed on the principle. ULMEN already implements this separation natively across two distinct encoding surfaces:

encode() / encode_binary_zlib(): binary format for storage and wire transmission. Optimized for size, not readability. This is the AgentOps egress surface.
encode_for_llm() / encode_ulmen_llm(): text format for context injection. Optimized for token efficiency. This is the context-window surface.

These are not the same codec applied twice. They are architecturally separate outputs from the same record set. A single UlmenDict instance can emit both without any transformation layer between them. The "two-format approach" you are describing is already the implementation.

On the three-tier progressive compaction

The pattern you described maps directly to ULMEN's COMPRESS_COMPLETED_SEQS strategy, which is what produces the 37.8% token reduction in the LangGraph benchmarks. The mechanism:

Recent events (full trace): emitted as-is into ULMEN text format, structural tokens stripped but all fields present
Completed sequences (older events): tool call + result chains that have resolved are compressed as a unit; the intermediate state is collapsed, only the terminal result is retained in the context surface
Oldest events: not currently implemented as a one-line digest tier, but the extract_subgraph() API provides the primitive needed to build it

from ulmen import extract_subgraph

# Pull only the last N steps for context injection
recent = extract_subgraph(records, step_min=turn - 4, step_max=turn)

The one-line digest tier is a valid extension point and not currently in the library. That is an honest gap.

On cost attribution at millicent precision

This is outside ULMEN's current scope. ULMEN operates on the serialization and validation layer; it does not model cost attribution or per-event billing rollups. That is a valid AgentOps-layer concern and orthogonal to what this feature request is proposing.

The proposal here is narrow: give AgentOps a pluggable serialization interface so that the JSON path on event capture and egress can be replaced. Cost attribution logic sits above that layer and is unaffected by the serializer swap.

On the Semantic Firewall relevance to AgentOps specifically

Your point about "the information most useful at that age" is where the Semantic Firewall becomes directly relevant to AgentOps in a way that is not obvious from the feature request framing.

AgentOps currently captures events and forwards them. It does not validate structural integrity before capture. The four failure modes that LangGraph passes silently:

Failure	AgentOps current behavior
Orphaned tool result (no matching call)	Captured and stored as valid event
Backward step counter regression	Captured and stored as valid event
Invalid record type enum	Captured and stored as valid event
Missing required fields	Captured and stored as valid event

These are not just pipeline cost problems; they are observability correctness problems. A dashboard built on top of silently corrupted event data is showing the team incorrect traces. The Semantic Firewall at the serialization boundary catches these before they enter the AgentOps event store, not after.

That is the specific value of validation-at-serialization for an observability tool: the events you store are structurally guaranteed to be valid traces, not just whatever the agent emitted.

@makroumi

This is a genuinely important issue for anyone running AgentOps at scale — the 44% JSON overhead figure is striking but very believable given how verbose agent event payloads get.

A few thoughts from a product perspective:

On the pluggable serializer direction
The agentops.init(serializer="ulmen") API is clean. One concern worth surfacing: BSL licensing on ULMEN could be a blocker for enterprise customers with strict OSS policies. Worth calling that out clearly in the docs alongside the opt-in, and potentially supporting an alternative (e.g., MessagePack, which is MIT-licensed and gets you ~50-60% reduction) as a second opt-in target.

On the Semantic Firewall
This is underrated — silent failures captured as valid events are one of the hardest bugs to debug in production agent systems. If AgentOps can catch orphaned tool calls or invalid state transitions at the serialization boundary and surface them as a distinct event type in the dashboard (e.g., INVALID_TRANSITION vs. COMPLETED), that becomes a real competitive differentiator.

On rollout
Suggest shipping this behind a feature flag first (AGENTOPS_SERIALIZER=ulmen) so teams can benchmark the wire-size savings against their own workloads before committing. A dashboard panel showing "estimated egress cost saved this month" would make the ROI tangible and help drive adoption of the feature.

Happy to help think through the docs or user-facing messaging for this if it moves forward.

@Ilya0527

@Ilya0527 Spot on across the board. I appreciate the sharp architectural critique.

1. Conflating the cost models
You are completely right. I conflated the framework-level cost (LLM context window tokens) with the observability-level cost (storage/egress bytes). AgentOps egress is a byte/bandwidth problem, not an LLM token problem. The 44% / $59K framing belongs in the framework layer (e.g., LangGraph/CrewAI), not here.

On egress bytes: ULMEN's binary format uses internal column pooling and bitpacking before applying zlib, which yields about a 1.2x - 1.5x size advantage over generic JSON + zstd. That is a nice infrastructure win, but you are right that it is not a $59K win.

2. Decoupling the Semantic Firewall
This is the correct architectural move. The maintainer ask should be the validation middleware, not a forced codec swap.

To be fully transparent on ULMEN's current implementation: the validate_agent_payload function is completely decoupled from the binary codec and Rust layer, but it currently expects the ULMEN-AGENT text protocol string as input, not raw Python dicts.

However, we can achieve exactly the decoupled middleware you described by using a lightweight, memory-only wrapper in the AgentOps event loop:

from ulmen import encode_agent_payload, validate_agent_payload

def validate_agent_events(events: list[dict]) -> tuple[bool, str]:
    # Memory-only text encode to evaluate state transition integrity
    # Does NOT touch the wire format or egress codec
    text_payload = encode_agent_payload(events)
    return validate_agent_payload(text_payload)

This leaves AgentOps' JSON + gzip/zstd wire format completely alone, but introduces the structural validator to catch the ALEF-PAT-001 orphaned tool uses before they are blindly logged as valid traces.

Excellent diagnosis!

@Ruthwik-Data

@Ruthwik-Data Thanks for the product perspective. Addressing your points directly:

1. On the BSL License
You are 100% correct. BSL is a non-starter for enterprise observability telemetry. We are actually relicensing ULMEN to Apache 2.0 this week. The OSS compliance friction and adoption tax you mentioned will be completely removed.

2. On the Semantic Firewall & Dashboard Integration
Echoing the thread with Ilya above, this is where the actual gold is for AgentOps.

If we decouple the validator from the serializer, AgentOps can intercept these silent framework failures (orphaned tool calls, invalid state enums) and emit a specific INVALID_TRANSITION event to the dashboard. Right now, teams are looking at dashboards showing "successful" tool executions that are actually corrupted states causing downstream LLM hallucinations. Flagging those explicitly at the observability boundary is a massive competitive differentiator.

3. On Rollout
Based on Ilya's architectural point about egress bytes vs tokens, the highest leverage feature flag isn't AGENTOPS_SERIALIZER=ulmen, it's AGENTOPS_VALIDATE_STATE=True.

Once the Apache 2.0 license is cut, I will draft a PR focused entirely on a non-intrusive validation middleware hook. We can evaluate the dashboard integration from there.

@Ilya0527

@Ilya0527 fair enough. Let me answer them precisely from the actual implementation rather than hand-waving.

On the encode_agent_payload round-trip

You are correct that routing through the text protocol to run a structural check is unnecessary coupling. Here is what the validation actually does, from the spec.

The thread-level rule that catches orphaned tool calls is:

Every res row id must match a prior tool row id

That is a dict-level scan. It does not require the ULMEN-AGENT text format as an intermediate. The text encoding exists because validate_agent_payload operates on the wire protocol string; that is its defined input surface. For AgentOps, the right integration is to lift just that rule out as a standalone function operating on list[dict]. No ULMEN import required.

def check_orphaned_tool_results(records):
    seen_tool_ids = set()
    for rec in records:
        if rec.get("type") == "tool":
            seen_tool_ids.add(rec["id"])
        elif rec.get("type") == "res":
            if rec["id"] not in seen_tool_ids:
                yield rec["id"]  # orphaned

The other validation rules from the ULMEN-AGENT spec that are relevant to AgentOps:

step values within a thread must be non-decreasin: catches step regression
First field must be one of the 10 defined type tags: catches invalid enum state
Field count must exactly match the schema for that type: catches missing required fields

All four checks are dict-level. None require serialization. I should have said that from the start instead of proposing the encode round-trip.

On streaming semantics

This is the gap I cannot resolve without knowing AgentOps' event pipeline. The ULMEN-AGENT spec is explicit on this point:

The validator operates on a complete, closed payload, not an open stream. That is why validate_agent_payload takes a full payload string, not a record iterator. A res record is only considered orphaned when a tool with matching id has never appeared anywhere in that closed payload.

For AgentOps specifically I need to ask before proposing anything:

Does AgentOps buffer events per turn before flushing to the backend, or does it emit each event immediately as it arrives?
Is there an explicit boundary event in the AgentOps event stream that marks a turn or step as complete?
When a tool call never receives a result: timeout, crash, abandoned session, does AgentOps emit a terminal event or does the stream simply stop?

The answers determine whether the validator fires per-event, on flush, or only at session close. Guessing wrong produces false positives on every long-running tool, which is worse than no validation.

Revised proposal

One PR, narrowly scoped:

Pure Python validator, zero dependencies, operates on list[dict]
Fires at whatever boundary AgentOps already has
Emits a distinct event type when an orphaned tool result, step regression, invalid type tag, or missing required field is detected
No serialization changes, no wire format changes, no ULMEN import in the AgentOps dependency graph

The ULMEN library becomes relevant only if AgentOps later wants the full protocol validation or the LLM context surface. That is a separate conversation.

Maintainers if you can describe how AgentOps buffers and flushes events I can write the validator against your actual lifecycle rather than a model I am guessing at.

Following up on the discussion around validation middleware — the convergence on a "non-intrusive validator" rather than a forced codec swap is the right call for maintainability.

On the lifecycle boundary question: The validator needs to run after a step is complete but before the event buffer is flushed. The key question is whether AgentOps buffers per-step or per-session. If it's per-step, you can hook in at the step close event. If it's batched, you need access to the sequence before transmission.

For the specific checks worth prioritizing in v1:

Orphaned tool calls — tool invoked with no corresponding result in the next N events
Step regression — step_index decreases without an explicit reset signal
Missing required fields — action_type or timestamp absent from an event
Duplicate event IDs — catches retry storms that double-count observations

For the format separation point: 100% agree that observability data (full-fidelity JSON) and context-injection format (compacted for LLM) should be kept separate. Conflating them leads to either bloated context windows or lossy audit logs — both are bad.

On the Apache 2.0 relicensing: That removes the last blocker for most enterprise teams. Appreciate the author making that call.

Happy to help write test cases for the validator middleware if the maintainers greenlight the implementation approach. The INVALID_TRANSITION event type is particularly valuable for downstream eval pipelines that try to correlate agent behavior with outcome quality.

@makroumi

Pinging on this — happy to move forward on the validator middleware PR as soon as there's clarity on the event buffer/flush lifecycle.

Quick summary of what we've converged on for maintainer review:

Non-intrusive validator hooked into the AgentOps event pipeline (no wire format change)
Emits a new INVALID_TRANSITION event type for structural failures (orphaned tool calls, step regressions, missing required fields)
Operates on list[dict] — zero dependency on ULMEN or any external codec
ULMEN moving to Apache 2.0 removes any licensing concern if the team later wants to adopt it

From a product angle this is a low-risk, high-signal addition: developers get structural validation without touching their existing instrumentation. Could also feed directly into the operator dashboard (issue #1383) — structural failures surfaced in plain English for non-engineers.

@makroumi happy to co-draft the PR. Just need the maintainers to confirm: does _send_queue or pre_send_hook have a clean entry point for middleware?

Update: with ULMEN moving to Apache 2.0 and the convergence on a non-intrusive validator, I've drafted the core validation logic as a standalone module ready to drop in. Sharing here for review before opening a PR:

# agentops/validation/structural_validator.py
"""
Zero-dependency structural validator for AgentOps event sequences.
Operates on list[dict] — no imports from core AgentOps required.
"""
from dataclasses import dataclass, field
from typing import Any


@dataclass
class ValidationResult:
    valid: bool
    events: list[str] = field(default_factory=list)  # INVALID_TRANSITION events
    summary: dict[str, int] = field(default_factory=dict)


def validate_event_sequence(events: list[dict[str, Any]]) -> ValidationResult:
    """
    Check a sequence of AgentOps events for structural integrity.
    
    Detects:
    - Orphaned tool results (tool_result with no preceding tool_call)
    - Duplicate event IDs
    - Missing required fields (id, type, timestamp)
    - Step regressions (step_number going backwards)
    """
    issues = []
    seen_ids: set[str] = set()
    open_tool_calls: set[str] = set()
    last_step = -1

    for i, evt in enumerate(events):
        evt_id = evt.get("id", f"<missing_id_{i}>")
        evt_type = evt.get("type", "<missing_type>")

        # Required fields
        for field_name in ("id", "type", "timestamp"):
            if field_name not in evt:
                issues.append(f"INVALID_TRANSITION: event[{i}] missing required field '{field_name}'")

        # Duplicate IDs
        if evt_id in seen_ids:
            issues.append(f"INVALID_TRANSITION: duplicate event id '{evt_id}' at index {i}")
        seen_ids.add(evt_id)

        # Tool call / result pairing
        if evt_type == "tool_call":
            open_tool_calls.add(evt_id)
        elif evt_type == "tool_result":
            call_ref = evt.get("tool_call_id")
            if call_ref not in open_tool_calls:
                issues.append(f"INVALID_TRANSITION: orphaned tool_result '{evt_id}' — no matching tool_call")
            else:
                open_tool_calls.discard(call_ref)

        # Step regression
        step = evt.get("step_number")
        if step is not None:
            if step < last_step:
                issues.append(f"INVALID_TRANSITION: step regression at event[{i}] ({last_step} -> {step})")
            last_step = step

    # Unclosed tool calls
    for call_id in open_tool_calls:
        issues.append(f"INVALID_TRANSITION: unclosed tool_call '{call_id}' — no matching tool_result")

    return ValidationResult(
        valid=len(issues) == 0,
        events=issues,
        summary={"total_events": len(events), "issues": len(issues)},
    )

Entry point question still stands: is _send_queue or a pre_send_hook the right place to call validate_event_sequence(batch) before flush? If the team can confirm, I'll open the PR with this + tests + a VALID_TRANSITION/INVALID_TRANSITION event emitter wired into the existing event bus.

Following up on my previous note regarding licensing: After running the financial modeling and evaluating the long-term infrastructure and compute costs required to maintain ULMEN at scale, we have decided to retain the Business Source License (BSL).

While I initially wanted to move to Apache 2.0 to eliminate open-source compliance friction for early adoption, a pure permissive license simply isn't sustainable for the level of performance engineering we are pouring into this.

To accommodate frameworks and platforms, we are implementing a Case-by-Case Ecosystem Licensing Program offering lifetime free commercial exemptions for open-source frameworks while keeping a revenue threshold for large enterprises. Because of this pivot, the architecture and validation logic derived from ULMEN cannot be safely merged under an Apache 2.0 license due to derivative copyright contamination. If the AgentOps team is open to a native, officially licensed ULMEN integration, I am highly open to structuring that.

The lowest-risk first step is structural validation without changing the wire format. Keep JSON ingestion, but emit compact references for bulky tool output: event id, content hash, byte count, truncation state, and replay location. That cuts repeated context payloads while preserving enough evidence to reconstruct orphaned tool calls or invalid step transitions.

Generated with ax.

@makroumi

Given where the thread has landed — zero-dependency Python validator on list[dict], no wire format changes, feature-flagged behind AGENTOPS_VALIDATE_STATE=True — this is a clean scope for a v1 PR.

The three structural checks that have the most signal based on this thread:

Orphaned tool calls — tool result present but no matching tool call id
Step regressions — step index out of order or repeated (sequence integrity)
Required field gaps — LLM call event missing model or usage fields

These are all detectable without any LLM involvement and would surface the class of silent failures @makroumi originally described. If the maintainers are open to a PR in this direction, I can sketch out the validator interface.

Navigation Menu

Repository navigation

[Feature]: Token-efficient serialization to reduce agent observability overhead and inference costs #1338

💡 Feature Description and Proposed Solution

Current

Proposed

🤔 Related Problem

🤝 Contribution

On separating observability format from context-injection format

On the three-tier progressive compaction

On cost attribution at millicent precision

On the Semantic Firewall relevance to AgentOps specifically

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Feature]: Token-efficient serialization to reduce agent observability overhead and inference costs #1338

Description

💡 Feature Description and Proposed Solution

Current

Proposed

🤔 Related Problem

🤝 Contribution

Activity

kinthaiofficial commented on Apr 28, 2026

makroumi commented on May 6, 2026

On separating observability format from context-injection format

On the three-tier progressive compaction

On cost attribution at millicent precision

On the Semantic Firewall relevance to AgentOps specifically

Ruthwik-Data commented on May 18, 2026

makroumi commented on May 18, 2026

makroumi commented on May 18, 2026

makroumi commented on May 19, 2026

Ruthwik-Data commented on May 21, 2026

Ruthwik-Data commented on May 31, 2026

Ruthwik-Data commented on Jun 1, 2026

makroumi commented on Jun 1, 2026

Necmttn commented on Jun 21, 2026

Ruthwik-Data commented on Jul 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions