Skip to content

feat(genai-util): implement ExtendedInvocationMetricsRecorder for non-LLM invocation types (tool/agent/embedding/retrieval/rerank) #272

Description

@RichardoMrMu

What problem do you want to solve?

ExtendedInvocationMetricsRecorder.record_extended() in
util/opentelemetry-util-genai/src/opentelemetry/util/genai/extended_metrics.py
declares support for chat, generate_content, embedding, execute_tool,
invoke_agent, create_agent, retrieval, rerank, entry and react-step invocations,
but currently records metrics only for LLMInvocation:

def record_extended(self, span, invocation, *, error_type=None):
    if isinstance(invocation, LLMInvocation):
        self.record(span, invocation, error_type=error_type)
        return
    # TODO: Implement extended metrics recorder

Meanwhile ExtendedTelemetryHandler already calls _record_extended_metrics(...)
from every stop_* / fail_* path (stop_execute_tool, stop_invoke_agent,
stop_embedding, stop_retrieval, stop_rerank, stop_create_agent,
stop_entry, stop_react_step, ...). So spans are emitted for all of these
invocation types, but the duration/token metrics are silently dropped for
everything except LLM.

Concretely: an operator charting gen_ai.client.operation.duration sees only
LLM operations; tool execution, agent invocation, embedding, retrieval and
rerank latency are invisible on the metrics pillar even though their spans
exist. This is also what #124 was asking about (how to collect metrics from a
non-LLM instrumentation).

Describe the solution you'd like

Implement record_extended for the non-LLM invocation types, reusing the
existing duration histogram (and token histogram where token counts apply, e.g.
embeddings) with the correct gen_ai.operation.name per type, so the metrics
pillar matches the span coverage that already exists.

Question before I open a PR

extended_metrics.py carries the comment "This is just an empty implementation
for now, which is a placeholder for enterprise implementation."
Before I invest
in a PR I'd like to confirm: is implementing the community/open-source
record_extended for these invocation types welcome, or is this intentionally
reserved for an enterprise build?
If it's open, I'm happy to implement it with
unit tests covering each invocation type.

Would you like to implement a fix?

Yes, if the maintainers confirm this is open for contribution.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions