Skip to content

perf(core): update log event JSON size cache incrementally on insert and remove - #27099

Draft
gwenaskell wants to merge 2 commits into
masterfrom
yoenn.burban/incremental-json-size-cache
Draft

gwenaskell wants to merge 2 commits into
masterfrom
yoenn.burban/incremental-json-size-cache

Conversation

@gwenaskell

@gwenaskell gwenaskell commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

LogEvent::insert and LogEvent::remove used to invalidate the cached estimated JSON size, so every component that touched a single field paid a full walk of the event at its output.

The cache is now updated from the inserted, replaced and removed values when the target's parent object already exists:

  • replace: + new - old
  • new member: + "key":value, plus a comma if the object was not empty
  • removed member: - "key":value, minus a comma if the object had other members

Inner::clone now keeps the JSON size cache so the update also works after Arc::make_mut.

Fallbacks to invalidation (same behavior as before):

  • missing parent or non-container on the path (insert creates intermediates)
  • array element as target (padding / negative index semantics)
  • remove_prune(.., true)
  • root path
  • any get_mut / value_mut access

size_cache (ByteSizeOf) is still invalidated.

Not covered: remap, since VrlTarget goes through into_parts / from_parts and drops the cache.

References

Vector configuration

N/A

How did you test this PR?

  • Unit tests checking that the cache matches a full recompute after replace / new member / nested (incl. through arrays) / remove / missing-key remove, and that the fallback cases invalidate.
  • Quickcheck property test: random nested inserts/removes (small key pool, indices, sometimes on a shared event) and asserts the cached size equals a full recompute after each one. Checked it fails when the comma rule is broken.

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

@github-actions github-actions Bot added the domain:core Anything related to core crates i.e. vector-core, core-common, etc label Nov 8, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain:core Anything related to core crates i.e. vector-core, core-common, etc

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant