Agent tracing, evaluation, session inspection, cost tracking, and OpenTelemetry instrumentation.
37 projects · Upstream documentation checked 2026-09-30. Curated by agentlist.io.
Understand what your agent did, and whether it helped.
Start with the question you cannot answer today: why a run failed, where the cost went, which tool call caused trouble, or whether a change improved the result. Traces, evaluations, and cost reports answer different questions. Choose the evidence you need before choosing a dashboard.
Tools for inspecting agent behavior, measuring usage, or evaluating outcomes. Tracing backends, instrumentation libraries, local session tools, and evaluation frameworks are distinct categories; one does not automatically replace another.
- How to choose
- Decision table
- Tracing and evaluation platforms
- Instrumentation
- Usage and gateway monitoring
- Evaluation and regression checks
- Local session inspection
- Related awesome lists
- More from Agentlist
- Contributing
- Works with: Does it capture your agent framework, model calls, and tools? Can you instrument custom steps and export standard telemetry?
- Runs where: Where are traces stored and evaluations executed? Which features are available when self-hosted?
- Needs access to: Will prompts, outputs, files, or credentials appear in telemetry? What redaction and access controls can you configure?
- Keeps what: Can you retain and export traces, datasets, evaluation results, and cost records? What are the retention boundaries?
- Human involvement: Can you follow a failed run, review examples, and turn findings into regression checks? Who reviews automated evaluation judgments?
- Main limitation: Which steps or costs remain invisible? A successful request, a low bill, and a useful task outcome are different measurements.
Use these questions with the decision table. Every entry summarizes its linked documentation. These lists were not install-tested, so entries do not repeat that. A capability the linked page does not state is unknown.
Start with your requirement. These examples highlight documented differences; they are not rankings.
| When you need | Documented fit | Key distinction | Unresolved | Source |
|---|---|---|---|---|
| Does adding a client library also choose where traces are stored? | Braintrust stores traces in its data plane. Logfire's SDK can export spans to any OpenTelemetry backend instead. | A hosted product is the system of record. An SDK that only emits spans leaves retention, access, and redaction to the backend you select. | Unknown which Logfire span fields remain when you export to a backend other than the Logfire service. | Braintrust, Logfire |
| If the Braintrust data plane is mine, what does Braintrust still operate? | Braintrust always operates the control plane. SaaS, BYOC, and self-hosted change only the data plane. | SaaS stores data in Braintrust infrastructure. BYOC runs the data plane in your cloud while Braintrust operates it. Self-hosted means you operate that plane. BYOC and self-hosted require Enterprise. The UI stays on the control plane. | Unknown how a non-standard self-hosted layout differs from the published artifacts. That path needs a review. | Braintrust |
| Can I offer Phoenix itself as a hosted or managed service? | Phoenix is the self-hosted trace app. Managed workflows are a separate product, not this repository. | The LICENSE file is Elastic License 2.0, not an OSI license, and it bars providing the software to third parties as a hosted or managed service. | Unknown which Phoenix features are behind the license-key functionality that the same LICENSE text says not to circumvent. | Phoenix |
| Is the whole Langfuse repository under one MIT license? | Langfuse is the trace platform. Only content outside the ee trees is MIT Expat. | The LICENSE file puts ee/, web/src/ee/, and worker/src/ee/ under ee/LICENSE. Calling the whole checkout MIT would overstate that grant. | Unknown which product features live only in the ee trees. The LICENSE file does not state retention. | Langfuse |
| Does Helicone see the prompt, or only a span exported later? | Helicone's gateway receives the chat request. OpenLLMetry is listed there as a separate async logging path. | The quick start sets an OpenAI client base URL to the Helicone gateway, so that request passes through the proxy. Async OpenLLMetry logging is a different integration, not proof of a trace store you run. | Unknown whether the async logging path omits prompt text that the gateway receives, and the README does not state retention. | Helicone, OpenLLMetry |
| Does a local cost report say the agent task succeeded? | ccusage reads local coding-agent usage files and prints token and cost reports. | Daily, weekly, monthly, and session reports are usage measurements. A completed request or a low total is not a judgment that the task helped. | Unknown whether any supported source log contains a task-success field. The README documents tokens, cost, and JSON output, not outcome scores. | ccusage |
| Does an evaluation library keep the production trace? | Inspect scores prompts, tools, and dialog. Langfuse and Phoenix are separate trace stores. | Inspect is a Python evaluation framework. Its README does not document a trace store, so a model-graded score is not a retained production session. | Unknown where Inspect writes eval logs, how long they are kept, and whether Inspect drives a production agent rather than the eval you start. | Inspect, Langfuse, Phoenix |
| Which retention window is stated, and which is not? | Braintrust states Starter and Pro windows. The Pezzo and Inspect READMEs do not state a retention window. | Starter keeps logs and playground data 14 days and cannot extend that. Pro starts at 30 days and can extend it to 180. A missing number is not the same as keeping data forever. | Unknown how a self-hosted Braintrust bucket policy interacts with these windows. The Pezzo and Inspect READMEs do not state where logs are stored. | Braintrust, Pezzo, Inspect |
| Does Latitude's 30-day retention apply when I self-host? | Latitude states 30-day retention in the hosted quick start, not in the self-host section. | The 30-day figure is next to signup for the hosted service. Container-image and Helm self-host instructions do not restate a retention period. | Unknown what a self-hosted Latitude install keeps, and whether the hosted 30-day window can be extended. | Latitude |
| Does AgentsView only read sessions, or can it start the agent? | AgentsView syncs local coding-agent sessions into SQLite. It is not itself the agent. | The usual server reads existing session files and serves a loopback UI. A separate capture run command can start one non-interactive claude -p or codex exec and does not start the daemon. | Unknown how long the local SQLite archive is kept, and which of the many named agents yield only partial transcripts. | AgentsView |
| Does Claude Code Agent Monitor only watch sessions? | Claude Code Agent Monitor reads Claude Code, Cursor, and Codex activity into local SQLite. | Hooks and local transcripts feed an Express server at data/dashboard.db. The README also documents an HTTP API that can spawn claude subprocesses, so it is not only a viewer. | Unknown how long data/dashboard.db is kept, and whether that run API can start Cursor or Codex as well as claude. | Claude Code Agent Monitor |
| Is the Logfire UI part of the open-source repository? | Logfire in this catalog is the Python SDK. The platform UI and backend are not in this repo. | The server application is closed source, the SDKs can export to any OpenTelemetry backend, and self-hosting that platform requires a purchased enterprise license. | Unknown which fields the closed-source UI stores or redacts. The README does not state retention. | Logfire |
- AgentOps - Python SDK and app directory for session traces, replays, and LLM cost tracking. It names CrewAI, AG2 (AutoGen), Agno, LangGraph, the OpenAI Agents SDK, CAMEL, and LangChain via the GitHub description. Sessions show on a dashboard after an API key. The dashboard and API can be self-hosted under MIT. Where a hosted account stores data is not stated. Python SDK and app.
- Langfuse - Platform for LLM traces, prompt versions, datasets, and evaluations, including LLM-as-a-judge, code evaluators, user feedback, and manual labels. Langfuse Cloud and self-hosting via Docker Compose, a VM, or Helm are documented, with Python and JS/TS SDKs. Outside ee/, web/src/ee/, and worker/src/ee/ the license is MIT Expat; those paths are under ee/LICENSE. The LICENSE file does not state retention. Self-hosted or cloud.
- LangSmith - Product for traces, datasets, and evaluation. It can run as cloud, hybrid, or self-hosted, and people can annotate traces. The open repository langchain-ai/langsmith-sdk is the client SDK, listed separately, and is not the server. This entry does not give a retention limit for each deployment mode. Tracing platform.
- LangSmith SDK - Python and JavaScript client for the LangSmith platform, not the server. Set an API key and workspace id, wrap clients such as OpenAI, or mark functions so traces and evaluation calls go to LangSmith, including a native LangChain path. Retention, deployment mode, and where the platform stores prompts are not documented here; they belong to the product, listed separately. Platform SDK.
- Braintrust - Tracing and evaluation product. The control plane is Braintrust-managed SaaS. A customer-hosted data plane can run in your AWS, GCP, or Azure account, operated by Braintrust or by your team. SDKs send events directly to that data plane. The browser UI connects to the same plane. Org-level provider secrets stay on the control plane. Tracing platform.
- Opik - Platform for agent and LLM traces, datasets, experiments, prompt playground runs, and evaluations, with human feedback scores and LLM-as-a-judge checks. The server, web app, and those core pieces are Apache-2.0 and can be self-hosted, and the client can target Opik Cloud. Annotation is in the UI or Python SDK, with a pytest hook for CI. No retention limit is stated. Self-hosted or cloud.
- Phoenix - Self-hosted app for OpenTelemetry traces, datasets, experiments, prompt versions, and evaluations. You run Phoenix yourself, locally or in your container or cloud account. Managed workflows are Arize AX, a separate product, using the same OpenInference instrumentation. LICENSE is Elastic License 2.0, not OSI, and you cannot offer the software as a hosted or managed service. No retention period is stated. Self-hosted platform.
- Weave - Weights & Biases Python toolkit for tracing functions and evaluating language-model apps. A Weights & Biases account is required. weave.init takes a project name, and decorated functions log inputs, outputs, and traces to that service. No self-hosted Weave server is documented in this repository. Older engine code is described as paused while the project focuses on tracing and evaluations. Python toolkit.
- Laminar - Observability platform for agent traces, plain-language behavior signals, annotation, datasets, and evaluations, with SQL over span data. A managed service and a Docker Compose self-host are documented. The SDK is OpenTelemetry-native for the Vercel AI SDK, LangChain, OpenAI, and Anthropic. Self-hosted installs send anonymized product telemetry unless LAMINAR_TELEMETRY_DISABLED is set. How long trace payloads are kept is not stated. Self-hosted or cloud.
- Logfire - Python SDK for an OpenTelemetry observability platform. This repository is the SDK, not the server: the UI and backend are closed source. Spans can go to the Logfire service after logfire auth, or to any OpenTelemetry backend. Self-hosting the platform requires a purchased enterprise license. It covers ordinary Python telemetry plus LLM and agent apps. Retention is not stated. Python SDK.
- LangWatch - Platform for LLM traces, simulation tests, evaluations, prompt management, and an OpenAI-compatible gateway, with session and cost views for Claude Code, Codex, GitHub Copilot, and OpenCode. LangWatch Cloud and a self-host command are documented. Apache 2.0 applies; modules under platform/app/ee/ need a commercial license in production, and the SDKs are MIT. Self-hosted gateway gaps and trace retention are not stated. Self-hosted or cloud.
- Latitude - App for multi-turn sessions, tool calls, and OpenTelemetry traces, with failing traces grouped into signals. TypeScript or Python clients target the hosted service or container-image self-hosting. The hosted quick start states 30-day data retention. Container-image self-hosting does not restate 30 days. It can dispatch Claude Code or Cursor with sample traces to open a pull request. The license is MIT. Self-hosted or cloud.
- OpenLIT - OpenTelemetry app you run with Docker Compose, plus Python and TypeScript SDKs exporting traces and metrics to your collector. Spans cover LLM calls, tools, prompts, tokens, and cost, plus LLM-as-a-judge checks such as hallucination and toxicity. A coding-agent path covers Claude Code, Cursor, and Codex. The quick-start dashboard runs where you start the stack. No retention limit is stated. Self-hosted platform.
- Future AGI - Platform for tracing, evaluations, simulations, guardrails, and an OpenAI-compatible gateway, with managed cloud and self-host paths. The core is Apache 2.0. A standalone install uses Postgres and ClickHouse. This checkout is a nightly release with rough edges. traceAI, listed separately, sends OpenTelemetry to your collector. Standalone data cannot move to the distributed layout later. No retention period is stated. Self-hosted or cloud.
- Failproof - Local hooks and a dashboard for run history, plus policy checks that can block a tool call before it runs. It attaches to Claude Code, Codex, Cursor, Hermes, and OpenClaw, and serves traces on localhost without an account. A hosted fleet view's self-host option is on an enterprise plan. The license is MIT with a Commons Clause limiting commercial resale. Local hooks and dashboard.
- Pezzo - LLMOps app for prompt versions and request observability. Node.js, Python, and LangChain clients are marked for prompt management, observability, and caching. The stack can run locally with Docker Compose on Postgres, ClickHouse, Redis, and SuperTokens, and the source is Apache 2.0. It does not document retention, a review step, or agent instrumentation beyond LangChain. Self-hosted LLMOps app.
- MLflow - Tracking server for ML experiments and LLM and agent traces, broader than agent observability. You can start a local server, enable autolog for clients such as OpenAI, and use OpenTelemetry, evaluations, prompt versions, and an OpenAI-compatible gateway. A setup wizard can also target Databricks. Retention is not stated. Experiment tracking or deployment does not by itself explain an agent trajectory. Tracking server.
- Langtrace - OpenTelemetry app for LLM, vector, and framework traces, with metrics and evaluations. There is a cloud signup and a self-host path. The application in this repository is AGPL-3.0. The separate Python and TypeScript SDKs are Apache-2.0. A self-hosted install does not send telemetry off your servers. No retention period is stated. Tracing and eval app.
- OpenInference - OpenTelemetry conventions and instrumentation plugins for LLM calls, retrieval, and tools, not a trace store. Phoenix and Arize AX accept this format natively, and any OpenTelemetry-compatible backend can receive it, with libraries in more than one language. Spans can carry prompts, retrieved context, and tool calls. Redaction is not documented, and you still choose where those spans are kept. Instrumentation libraries.
- OpenLLMetry - OpenTelemetry instrumentations and Python SDK for LLM providers, vector stores, and agent frameworks, maintained under Apache 2.0. Traceloop.init emits standard OpenTelemetry. Traces can go to Traceloop or other backends, including Datadog, Grafana, Braintrust, Laminar, and an OpenTelemetry collector. A separate JavaScript repository holds that SDK. Redaction and retention depend on the backend you select and are not set here. Instrumentation libraries.
- traceAI - OpenTelemetry instrumentation for LLM calls, prompts, tokens, retrieval, and agent or tool steps. Not a dashboard and not the Future AGI product. Traces go to the OpenTelemetry backend you configure, including Datadog, Grafana, Jaeger, and Future AGI. Packages exist for Python, TypeScript, Java, and C. Captured prompts are visible to that backend's operator, and retention is not set here. OTel instrumentation.
- ccusage - Local CLI for daily, weekly, monthly, and session cost reports from usage files coding-agent CLIs store. Sources include Claude Code, Codex, OpenCode, Hermes Agent, Goose, OpenClaw, Copilot CLI, Gemini CLI, and others, with JSON via --json. GitHub's license field was null on 2026-09-30. It does not call the models. A cost report is not a judgment that the task succeeded. Local CLI.
- Helicone - Proxy and app for LLM request logs, cost, latency, and agent session traces. The quick start points an OpenAI client at the gateway so requests pass through, and a separate path logs asynchronously through OpenLLMetry. Hosted and Docker Compose self-host are documented, with no feature-gap list. The license is Apache-2.0. A logged request is not evidence the task was useful. Proxy and app.
- LiteLLM - Python SDK and proxy that present many model providers through one OpenAI-shaped API, broader than agent tracing. Clients post chat messages through the gateway you run, with provider keys configured there, so the proxy sees those messages. Outside enterprise/ the license is MIT; that directory, if present, has its own license. How long request logs are kept is not stated. Gateway and SDK.
- DeepEval - Python framework for pytest-style checks on LLM apps, RAG pipelines, and agent trajectories, including task completion and tool-call metrics. Metrics can use an LLM-as-a-judge or local models. Hooks cover LangChain, LangGraph, CrewAI, Pydantic AI, and the OpenAI Agents SDK. Confident AI is a separate hosted product for shared reports. A judge score is not a stored trace. Python framework.
- Promptfoo - CLI and library for prompt and model evaluations, CI checks, and red-team scans, not a tracing backend. It is MIT licensed. Local evals keep prompts on your machine. Providers include OpenAI, Anthropic, Bedrock, and Ollama. Team sharing omits the destination. A red-team report is not a live agent session. No sign-off rule for a failing case is stated. CLI and library.
- TruLens - Python toolkit recording per-step inputs, outputs, tokens, and cost as OpenTelemetry spans, scored with LLM judges. Spans export to any OTLP backend, including Jaeger, Grafana Tempo, Datadog, and New Relic. Evaluations run live or later over a dataset. It covers RAG checks and agent steps. No retention is stated, and it does not require a person to approve judge scores. Python toolkit.
- Inspect - Python evaluation framework from the UK AI Security Institute for prompts, tool use, multi-turn dialog, and model-graded scoring. Other packages can supply extensions, and pre-built evaluations can run on a model you choose. Execution is the Python process you start. It is not a tracing backend. Eval log storage, retention, and export format are not documented. Python framework.
- Evidently - Python library for data, ML, and LLM checks, broader than agent traces. You can compute reports and pass/fail test suites, including LLM-as-a-judge text evals, and export JSON, a Python dictionary, or HTML. Results can be viewed in a self-hosted monitoring UI or Evidently Cloud. It also covers tabular drift. Agent-framework instrumentation and a cloud retention limit are not documented. Python library.
- Giskard - Python libraries for agent checks and vulnerability scans that generate adversarial cases from a plain-language description. Version 3 targets multi-turn agents and RAG quality. Checks run against a callable you supply. Optional telemetry sends aggregated analytics, not prompts or outputs, unless enabled. Older v2 tabular code is no longer maintained. Not a tracing backend; no hosted trace store is documented. Python libraries.
- Prompt flow - Microsoft toolkit for building, batch-testing, and evaluating LLM flows, broader than agent tracing. You create a flow locally with the pf CLI, store an API key in a connection, and run interactive tests. You can deploy the flow or use optional Azure AI cloud for team collaboration. Evaluation is added to the flow and CI. Agent-session retention is not documented. Python toolkit.
- DeepTeam - Python red-team framework for LLM apps and agents, built on DeepEval, not a tracing backend. It runs locally, attacks vulnerabilities you select, and scores them with an LLM-as-a-judge that needs a model key. Confident AI is a separate product for storing and sharing reports. Retention here is not documented. A failed case is not a normal user-session trace. Python framework.
- OpenEvals - Python and JavaScript package of ready-made evaluators, including LLM-as-a-judge prompts and checks for strings or structured output. Agent-specific evals live in the separate langchain-ai/agentevals repository. A judge call needs a model client, with LangChain chat models as the default. It does not record traces, retain datasets, or require a person to review the judge comment. Python and JS library.
- Ragas - Python toolkit for evaluating retrieval-augmented generation, which is broader than agent traces. The project is now at vibrantlabsai/ragas. It documents metrics, test-set generation, and a RAG evaluation quick start. Agent-eval templates are still marked coming soon. No trace store is documented, and no retention period is stated. Python library.
- Kitaru - Replay-based server for agent sessions: record or import runs, then re-execute against a later prompt, model, or code change. Imports include Langfuse, LangSmith, Braintrust, Logfire, Phoenix, and MLflow. Adapters cover PydanticAI, LangGraph, OpenAI Agents SDK, and Claude Agent SDK. Replays run on your workers. Local Docker, Podman, and managed cloud are under Apache 2.0. A person calibrates evaluators with judgments. Replay eval server.
- AgentsView - Local desktop and CLI for coding-agent sessions and token cost. First run syncs sessions into local SQLite and serves loopback UI. The archive stays local unless shared. Agents include Claude Code, Codex, Cursor, and others, with Aider opt-in and Amp deprecated. Capture run can start one non-interactive claude -p or codex exec without starting the daemon. No retention is stated. Local desktop and CLI.
- Claude Code Agent Monitor - Local dashboard for Claude Code, Cursor, and Codex activity. Native hooks for Claude Code and Codex, plus local Cursor transcripts, feed an Express server with SQLite at data/dashboard.db. A desktop app, MCP server, JSON export, and HTTP API that can spawn claude are documented. That run API is not documented as spawning Cursor or Codex. No retention window is stated. Local dashboard.
Independent collections for deeper discovery. These are references, not affiliations or endorsements.
- ContextJet-ai/awesome-llm-observability - LLM observability links plus skills from the list's own author. Useful leads, not an independent audit.
- tensorchord/Awesome-LLMOps - Broad LLMOps list. Many rows are training, serving, or gateways rather than agent tracing.
- backblaze-labs/awesome-agent-infrastructure - Cross-cutting infrastructure list. Thin on tracers specifically.
- dyronrh/awesome-agentops-landscape - Comparison tables of agent operations tools. At least one cell has been wrong: Helicone was marked non-open-source while its repository license is Apache-2.0. Use the links, not the cells.
- InftyAI/Awesome-LLMOps - LLMOps tool list, separate from the TensorChord list, and broader than agent tracing.
- Awesome Agent List
- Awesome Personal Assistants
- Awesome Agent Clients
- Awesome Agent Memory
- Awesome Agent Sandboxes
- Awesome Agent Orchestration
Browse agents · Compare agents · GitHub organization
Missing something useful? Read the contribution guide and open an issue or pull request with an official source.
The machine-readable list.json includes a primary-source link and a documentation-check date for every entry. Descriptions are short editorial summaries of that linked documentation. Inclusion does not mean the project was installed, run, security-tested, or endorsed. Hosted services and source code may have different terms.
To update the list, edit list.json, run bun run build, then bun run check. The README is generated; avoid editing it directly.
CC0 applies to this list’s text and data. Linked projects retain their own licenses.