Skip to content

feat(mcp): enable model and conversation capture by default #278

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lucasheriques wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(mcp): enable model and conversation capture by default #278

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/tidy-models-report.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'posthog-ruby': minor
---

Enable MCP model capture and conversation ids by default, matching the JavaScript and Python MCP SDKs. Pass `capture_model: false` or `enable_conversation_id: false` to opt out.
6 changes: 3 additions & 3 deletions lib/posthog/mcp/client.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,15 +113,15 @@ def capture_missing_capability(context: nil, parameters: nil, protocol_version:
emit(event)
end

# Inject the `context` argument (and, with `capture_model`, `llm_model`) into
# Inject the `context` and `llm_model` arguments into
# every tool descriptor (Hash with `inputSchema`) so agents state their intent,
# and optionally append the `get_more_tools` virtual tool. Returns a new Array
# of new Hashes. A tool whose schema is composed (oneOf/allOf/anyOf) or a
# `$ref` is passed through untouched.
# `$ref` is passed through untouched. Pass `capture_model: false` to opt out.
#
# @param tools [Array<Hash>] `tools/list` entries
# @return [Array<Hash>]
def prepare_tool_list(tools, context: true, report_missing: false, capture_model: false)
def prepare_tool_list(tools, context: true, report_missing: false, capture_model: true)

@greptile-apps greptile-apps Bot Sep 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Model opt-out is bypassed

Passing capture_model: false only removes llm_model from the advertised schema. prepare_tool_call does not receive or retain that setting, so if an agent still sends llm_model, the method strips the value from the tool arguments and returns it for analytics capture. This makes the documented opt-out ineffective and differs from the wrapped-server path, which gates extraction on capture_model_enabled?. Preserve the setting through call preparation so disabling model capture also prevents extracting and recording the value.

Knowledge Base Used:

Prompt To Fix With AI 
This is a comment left during a code review.
Path: lib/posthog/mcp/client.rb
Line: 124

Comment:
**Model opt-out is bypassed**

Passing `capture_model: false` only removes `llm_model` from the advertised schema. `prepare_tool_call` does not receive or retain that setting, so if an agent still sends `llm_model`, the method strips the value from the tool arguments and returns it for analytics capture. This makes the documented opt-out ineffective and differs from the wrapped-server path, which gates extraction on `capture_model_enabled?`. Preserve the setting through call preparation so disabling model capture also prevents extracting and recording the value.

**Knowledge Base Used:**
- [MCP analytics integration](https://app.greptile.com/posthog-org-19734/-/custom-context/knowledge-base/posthog/posthog-ruby/-/docs/mcp-analytics-integration.md)
- [MCP server lifecycle instrumentation](https://app.greptile.com/posthog-org-19734/-/custom-context/knowledge-base/posthog/posthog-ruby/-/docs/mcp-server-lifecycle-instrumentation.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

options = Options.new(context: context, capture_model: capture_model)
prepared = tools.map do |tool|
next tool unless tool.is_a?(Hash) && (options.context_enabled? || options.capture_model_enabled?)
Expand Down
5 changes: 2 additions & 3 deletions lib/posthog/mcp/instrumentation.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -506,9 +506,8 @@ def warn_stateless_session_not_wired
'Warning: an MCP request arrived over streamable HTTP with no session id, so PostHog generated a ' \
'per-process $session_id that will fragment across requests and pods. In stateless mode the ' \
'client must replay the Mcp-Session-Id header PostHog::MCP mints at initialize; for a custom Rack ' \
'stack add PostHog::MCP::RackMiddleware. Enabling conversation ids ' \
'(PostHog::MCP.instrument(server, enable_conversation_id: true)) also anchors the session without ' \
'any middleware. See https://posthog.com/docs/mcp-analytics/installation#ruby.'
'stack add PostHog::MCP::RackMiddleware. Conversation ids, which are enabled by default, also anchor ' \
'the session without any middleware. See https://posthog.com/docs/mcp-analytics/installation#ruby.'
)
end

Expand Down
8 changes: 4 additions & 4 deletions lib/posthog/mcp/options.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,13 +52,13 @@ class Options
attr_reader :report_missing
# @return [String] Name of the virtual tool. Default `get_more_tools`.
attr_reader :missing_capability_tool_name
# @return [Boolean] Inject `conversation_id` and anchor `$session_id` on it. Default false.
# @return [Boolean] Inject `conversation_id` and anchor `$session_id` on it. Default true.
attr_reader :enable_conversation_id
# @return [Boolean] Emit a sibling `$exception` event for failed calls. Default true.
attr_reader :enable_exception_autocapture
# @return [Boolean, ContextOptions] Inject the required `context` argument. Default true.
attr_reader :context
# @return [Boolean, ModelOptions] Capture `$mcp_llm_model`. Default false.
# @return [Boolean, ModelOptions] Capture `$mcp_llm_model`. Default true.
attr_reader :capture_model
# @return [#call, UserIdentity, Hash, nil] `(request, extra) -> UserIdentity | Hash | nil`, or a static identity.
attr_reader :identify
Expand All @@ -70,8 +70,8 @@ class Options
attr_reader :event_properties

def initialize(logger: nil, report_missing: false, missing_capability_tool_name: nil,
enable_conversation_id: false, enable_exception_autocapture: true, context: true,
capture_model: false, identify: nil, intent_fallback: nil, before_send: nil,
enable_conversation_id: true, enable_exception_autocapture: true, context: true,
capture_model: true, identify: nil, intent_fallback: nil, before_send: nil,
event_properties: nil)
@logger = logger
@report_missing = report_missing == true
Expand Down
9 changes: 5 additions & 4 deletions spec/posthog/mcp/client_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,15 +96,15 @@
expect(client.prepare_tool_call('get_more_tools').is_missing_capability).to be(true)
end

it 'advertises llm_model on every tool and on the virtual one when capture_model is on' do
it 'advertises llm_model on every tool and on the virtual one by default' do
tools = [{ name: 'a', inputSchema: { type: 'object', properties: {} } }]
prepared = client.prepare_tool_list(tools, capture_model: true, report_missing: true)
prepared = client.prepare_tool_list(tools, report_missing: true)
expect(prepared[0][:inputSchema][:properties].keys).to eq(%i[context llm_model])
expect(prepared[1][:name]).to eq('get_more_tools')
expect(prepared[1][:inputSchema][:properties].keys).to eq(%i[context llm_model])
expect(prepared[1][:inputSchema][:required]).to contain_exactly('context', 'llm_model')

without = client.prepare_tool_list(tools, report_missing: true)
without = client.prepare_tool_list(tools, capture_model: false, report_missing: true)
expect(without[0][:inputSchema][:properties].keys).to eq([:context])
expect(without[1][:inputSchema][:properties].keys).to eq([:context])
end
Expand Down Expand Up @@ -144,7 +144,8 @@

it 'strips only the context argument it injected when given the tool schema' do
own = { type: 'object', properties: { context: { type: 'string' } }, required: ['context'] }
expect(client.prepare_tool_list([{ name: 'search', inputSchema: own }])[0][:inputSchema]).to eq(own)
expect(client.prepare_tool_list([{ name: 'search', inputSchema: own }], capture_model: false)[0][:inputSchema])
.to eq(own)
kept = client.prepare_tool_call('search', { context: 'application data' }, input_schema: own)
expect(kept.args).to eq(context: 'application data')
# The tool declares `context`, so its value is the tool's own data: it stays in
Expand Down
16 changes: 10 additions & 6 deletions spec/posthog/mcp/http_transport_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ def parse(response)
let(:transport) { MCP::Server::Transports::StreamableHTTPTransport.new(server, stateless: true, enable_json_response: true) }

it 'mints a session token on initialize, recovers it on replay, and stamps transport identity' do
PostHog::MCP.instrument(server, client)
PostHog::MCP.instrument(server, client, enable_conversation_id: false)
response = transport.call(env_for(initialize_body, 'user-agent' => 'claude-code/2.1.0 (cli)', 'x-anthropic-client' => 'cli'))
expect(response[0]).to eq(200)
token = response[1]['mcp-session-id']
Expand Down Expand Up @@ -107,7 +107,7 @@ def parse(response)
let(:transport) { MCP::Server::Transports::StreamableHTTPTransport.new(server, enable_json_response: true) }

it 'hashes the transport session id deterministically across requests' do
PostHog::MCP.instrument(server, client)
PostHog::MCP.instrument(server, client, enable_conversation_id: false)
response = transport.call(env_for(initialize_body))
session_id = response[1]['mcp-session-id']
expect(PostHog::MCP.decode_session_id(session_id)).to be_nil
Expand All @@ -121,7 +121,9 @@ def parse(response)
end

it 'runs get_more_tools through the real request lifecycle so in-flight entries are released' do
PostHog::MCP.instrument(server, client, report_missing: true)
PostHog::MCP.instrument(
server, client, report_missing: true, capture_model: false, enable_conversation_id: false
)
session_id = transport.call(env_for(initialize_body))[1]['mcp-session-id']
3.times do |i|
response = transport.call(env_for(rpc(i + 2, 'tools/call', { name: 'get_more_tools', arguments: { context: 'csv export' } }),
Expand All @@ -145,7 +147,9 @@ def parse(response)
let(:identify) { ->(_request, extra) { { distinct_id: extra['headers']['user-agent'] } } }

before do
PostHogMcpHttpSpecCaptureTool.analytics = PostHog::MCP.instrument(server, client, identify: identify)
PostHogMcpHttpSpecCaptureTool.analytics = PostHog::MCP.instrument(
server, client, identify: identify, enable_conversation_id: false
)
PostHogMcpHttpSpecCaptureTool.before_capture = nil
end

Expand Down Expand Up @@ -229,7 +233,7 @@ def dispatching_app(status: 200, body: nil)
end

it 'mints from the instrumented server without reading the request body' do
PostHog::MCP.instrument(server, client)
PostHog::MCP.instrument(server, client, enable_conversation_id: false)
env = { 'REQUEST_METHOD' => 'POST', 'rack.input' => PostHogMcpUnreadableInput.new }
app = ->(_e) { [200, { 'content-type' => 'application/json' }, [server.handle_json(initialize_json)]] }
status, headers, = described_class.new(app).call(env)
Expand All @@ -245,7 +249,7 @@ def dispatching_app(status: 200, body: nil)
end

it 'publishes the request headers so a server below it sees the HTTP context' do
PostHog::MCP.instrument(server, client)
PostHog::MCP.instrument(server, client, enable_conversation_id: false)
env = env_for(initialize_json, 'user-agent' => 'claude-code/2.1.0 (cli)', 'x-anthropic-client' => 'cli')
_, headers, = described_class.new(dispatching_app).call(env)
token = headers['mcp-session-id']
Expand Down
32 changes: 17 additions & 15 deletions spec/posthog/mcp/instrument_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ def initialize_request(id = 1, version = '2025-06-18')
seen << data[:method]
handler.call
})
described_class.instrument(server, client)
described_class.instrument(server, client, capture_model: false, enable_conversation_id: false)
server.handle(rpc(1, 'tools/list'))
expect(seen).to eq(['tools/list'])
expect(drain_events(client).map { |e| e[:event] }).to include('$mcp_tools_list')
Expand All @@ -177,7 +177,7 @@ def initialize_request(id = 1, version = '2025-06-18')
before { allow(Kernel).to receive(:warn) }

it 'captures initialize, tools/list and a successful tool call with $lib override' do
described_class.instrument(server, client)
described_class.instrument(server, client, capture_model: false, enable_conversation_id: false)
response = server.handle(initialize_request)
expect(response[:result][:protocolVersion]).to eq('2025-06-18')
list = server.handle(rpc(2, 'tools/list'))
Expand Down Expand Up @@ -217,7 +217,7 @@ def initialize_request(id = 1, version = '2025-06-18')
end

it 'strips the injected context before the tool sees it but keeps a tool-owned context' do
described_class.instrument(server, client)
described_class.instrument(server, client, capture_model: false, enable_conversation_id: false)
result = server.handle(rpc(1, 'tools/call', { name: 'owns_context', arguments: { context: 'mine' } }))
expect(result[:result][:content][0][:text]).to eq('ctx=mine')
list = server.handle(rpc(2, 'tools/list'))
Expand Down Expand Up @@ -247,7 +247,7 @@ def initialize_request(id = 1, version = '2025-06-18')
end

it 'treats isError results as errors and honours enable_exception_autocapture: false' do
described_class.instrument(server, client, enable_exception_autocapture: false)
described_class.instrument(server, client, enable_exception_autocapture: false, enable_conversation_id: false)
server.handle(rpc(1, 'tools/call', { name: 'soft_fail', arguments: {} }))
events = drain_events(client)
expect(events.map { |e| e[:event] }).to eq(['$mcp_initialize', '$mcp_tool_call'])
Expand Down Expand Up @@ -342,9 +342,9 @@ def initialize_request(id = 1, version = '2025-06-18')
calls += 1
{ distinct_id: 'user-1', properties: { name: 'Alice' }, groups: { organization: 'org_123' } }
end
described_class.instrument(server, client, identify: identify, event_properties: lambda { |_r, _e|
{ env: 'production' }
})
event_properties = ->(_request, _extra) { { env: 'production' } }
described_class.instrument(server, client, identify: identify, enable_conversation_id: false,
event_properties: event_properties)
server.handle(initialize_request)
3.times { |i| server.handle(rpc(i + 2, 'tools/call', { name: 'echo', arguments: { message: 'x' } })) }
events = drain_events(client)
Expand Down Expand Up @@ -384,7 +384,9 @@ def initialize_request(id = 1, version = '2025-06-18')
end
shared = MCP::Server.new(name: 'spec-server', version: '9.9.9',
tools: [PostHogMcpSpecParkedTool, PostHogMcpSpecEchoTool])
PostHogMcpSpecParkedTool.analytics = described_class.instrument(shared, client, identify: identify)
PostHogMcpSpecParkedTool.analytics = described_class.instrument(
shared, client, identify: identify, enable_conversation_id: false
)

alice = Thread.new { shared.handle(rpc(1, 'tools/call', { name: 'parked', arguments: { user: 'alice' } })) }
PostHogMcpSpecParkedTool.entered.pop
Expand Down Expand Up @@ -420,7 +422,7 @@ def initialize_request(id = 1, version = '2025-06-18')
payload['properties']['bloat'] = 'x' * 40_000
payload
end
described_class.instrument(server, client, before_send: before_send)
described_class.instrument(server, client, before_send: before_send, enable_conversation_id: false)
server.handle(rpc(1, 'tools/call', { name: 'echo', arguments: { message: 'hi' } }))
events = drain_events(client)
# The hook runs after truncation, so without a second pass the batch would
Expand Down Expand Up @@ -453,13 +455,13 @@ def initialize_request(id = 1, version = '2025-06-18')
describe 'conversation ids, get_more_tools and llm_model' do
before { allow(Kernel).to receive(:warn) }

it 'mints, prompts back, mirrors into structuredContent and anchors the session on an echo' do
described_class.instrument(server, client, enable_conversation_id: true)
it 'mints, prompts back, mirrors into structuredContent and anchors the session on an echo by default' do
described_class.instrument(server, client)
list = server.handle(rpc(1, 'tools/list'))
structured = list[:result][:tools].find { |t| t[:name] == 'structured' }
expect(structured[:inputSchema][:properties]).to have_key(:conversation_id)
expect(structured[:outputSchema][:properties]).to have_key(:_mcp_instructions)
expect(structured[:inputSchema][:required]).to eq(['context'])
expect(structured[:inputSchema][:required]).to contain_exactly('context', 'llm_model')

first = server.handle(rpc(2, 'tools/call', { name: 'echo', arguments: { message: 'hi', context: 'c' } }))
prompt_back = JSON.parse(first[:result][:content][1][:text])
Expand Down Expand Up @@ -524,7 +526,7 @@ def initialize_request(id = 1, version = '2025-06-18')
end

it 'advertises and intercepts get_more_tools as $mcp_missing_capability' do
described_class.instrument(server, client, report_missing: true)
described_class.instrument(server, client, report_missing: true, capture_model: false)
list = server.handle(rpc(1, 'tools/list'))
virtual = list[:result][:tools].last
expect(virtual[:name]).to eq('get_more_tools')
Expand Down Expand Up @@ -585,8 +587,8 @@ def initialize_request(id = 1, version = '2025-06-18')
'$mcp_llm_model_source' => 'self_reported')
end

it 'captures llm_model from the injected argument or client metadata' do
described_class.instrument(server, client, capture_model: true)
it 'captures llm_model from the injected argument or client metadata by default' do
described_class.instrument(server, client)
server.handle(rpc(1, 'tools/call',
{ name: 'echo', arguments: { message: 'hi', llm_model: ' claude-opus-4-8 ' } }))
server.handle(rpc(2, 'tools/call', { name: 'echo', arguments: { message: 'hi', llm_model: 'unknown' } }))
Expand Down
10 changes: 8 additions & 2 deletions spec/posthog/mcp/options_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,20 @@
it 'has JS/Python defaults' do
options = described_class.new
expect(options.report_missing).to be(false)
expect(options.enable_conversation_id).to be(false)
expect(options.enable_conversation_id).to be(true)
expect(options.enable_exception_autocapture).to be(true)
expect(options.context_enabled?).to be(true)
expect(options.context_description).to be_nil
expect(options.capture_model_enabled?).to be(false)
expect(options.capture_model_enabled?).to be(true)
expect(PostHog::MCP::Tools.missing_capability_tool_name(options)).to eq('get_more_tools')
end

it 'allows conversation and model capture to be disabled' do
options = described_class.new(enable_conversation_id: false, capture_model: false)
expect(options.enable_conversation_id).to be(false)
expect(options.capture_model_enabled?).to be(false)
end

it 'normalises hash forms' do
options = described_class.new(context: { description: 'why' }, capture_model: { description: 'which' },
missing_capability_tool_name: 'find_tools')
Expand Down
Loading