Repository navigation
Go L3 intraprocedural dataflow, app-outermost ids and L2 library calls - #16
Merged
Merged
Conversation
…ojectDir; span-less external params; chi externals determinism
BuildCFG now returns the pruned node set with the edges.
Adds a by-definition CDG reference check over the dataflow and chi corpora.
BuildNodes takes the keys in recording order instead of rebuilding a token.Pos from line:col, which is wrong under //line directives. Nodes also carry their directive-adjusted line:col range for emission.
dataflow.Run builds cfg/cdg/ddg per callable over -j workers; core runs it at -a 3 and v2emit merges the side table into each callable (statement nodes, id/parent, edge lists, k_limit). L1/L2 output is unchanged.
Also cover the unmatched-closure skip path.
GO_CFG_NEXT (_k=kind), GO_CDG and GO_DDG (_k=var|prov) are now live, keyed on body-node global ids. :GoBodyNode gains call_node and var.
The edge dedup was keyed on (signature, callee), so with several func init() in one package only the first-visited file's init kept an edge to a shared callee, and the visit order was random map order. Key the dedup on the declaring file too, and walk files, types and callables in sorted order.
Neo4j only accepts CALL { ... } IN TRANSACTIONS in an auto-commit
transaction; the purge ran inside ExecuteWrite, so every --eager push onto
existing data failed.
Per-app volume, kind breakdowns, DDG distribution, cost, size and consistency checks, with the extraction script.
r.Read(buf[:n]), io.ReadFull(r, hdr[:]), copy(dst[i:], src) and clear(s[:n]) recorded no write: pathOf has no SliceExpr case. Strip slice expressions to the base and weak-define its elements.
L1 records call sites via fset.Position, which applies //line directives; the index rebuilt a token.Pos from that virtual line:col, so sites under a directive lost their callee and edge. Key the index by (file, start, end) as L1 records them; drops the per-site file scan.
A field selector with no recorded selection, an undeclared name, or a member of a failed import dropped the def and turned the LHS into a use. Treat the selector as a field step, give unresolved names a spelling-identified base (weak root.*), and never make an LHS use-only.
The analyzer type-checks targets with its own go/types, so a binary built with an older Go silently degrades on newer modules. Release with go-version: stable, add a CI job on stable, document it, and replace the per-package TooNew warnings with one consolidated warning.
The deferred call node now uses the paths it may write through, so a write after the defer registration (defer w.Flush(); w.buf = ...) reaches it. Document the remaining L3 precision gaps.
External call edges and external_symbols exist; --emit neo4j is L3; ids are can://<app>/go/...; record the L3 Go leaves (defer two-node rule, defer_resume, body kinds, ids from L1, k_limit, span-less @entry/@EXIT). Mark the R19 closure source_file gap fixed.
Clone, prep, tool checks and type-check status per app; the run phase reads the prepared entries, builds the analyzer with EVAL_GO and adds a type-check column. Kubernetes (root and k8s.io/api) joins the manifest.
Rerun the six apps and both Kubernetes modules (root, k8s.io/api) with the post-fix analyzer; measure protobuf-marshaller DDG volume (I5) on k8s-api. cockroach kept as a labelled pre-fix subsection. l3_metrics.py groups DDG by callable name and counts zz_generated files as generated.
x/tools v0.46 cannot read Go 1.27's export data (version 5), so loads that read export data instead of type-checking deps from source fail on Go 1.27. v0.50+ supports it and requires Go 1.26, raising the go.mod minimum.
This was referenced Oct 25, 2026
On malformed source the parser can end a function one past its file, where the FileSet has no token.File; nodeSource dereferenced it and the analyzer panicked at every level. It now returns "" for such nodes.
buildCallSites stopped at a go statement's call, so g() in go f(g()) had no call node, callee or edge. It now tags the launched call is_goroutine and records nested calls as ordinary sites. The L3 mirror (recordedCalls) and the test mirror (gateCallKeys) change in lockstep; at L3 the nested calls are satellites of the go node.
append(dst[:0], x), or append into spare capacity, writes dst's backing array; it now weak-defines dst[*] alongside the LHS def. A per-callable panic also logs its stack at -vv.
…al params DeferWrap shows a deferred named-result write never kills; a recording session shows eagerPurge runs its IN TRANSACTIONS statement through Run, not ExecuteWrite; vlib.G/H show unnamed and blank library params become _<i>.
With --target-files the push carries only the targeted packages, so an app-wide purge first would leave the rest of the app missing from the graph.
Contributor
Author
sinha108
marked this pull request as ready for review
October 25, 2026 15:23
pavuluri
approved these changes
Oct 27, 2026
pavuluri
left a comment
pavuluri
left a comment
Contributor
There was a problem hiding this comment.
The PR is rather large with substantial code changes. The main addition is intraprocedural L3 analysis: for each callable, it produces a control-flow graph (CFG), control-dependence graph (CDG), and data-dependence graph (DDG), in addition to the existing symbol and call-graph information. It also projects those graphs into Neo4j. It includes extensive tests and a real-application evaluation, with good scale results.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds L3 intraprocedural dataflow to codeanalyzer-go at
-a 3, switches ids to the app-outermostcan://<app>/go/…grammar, and makes L2 keep calls to library functions.Motivation and Context
L3 brings the Go analyzer to parity with codeanalyzer-python and codeanalyzer-java. It is also what python-sdk's slicing and taint walk over: every callable now carries statement body nodes plus
cfg/cdg/ddgedges, in bothanalysis.jsonand the Neo4j projection.The id change (
can://go/<app>becomescan://<app>/go) puts the app outermost, as the Python and Java analyzers already do, so a polyglot app's analyses share onecan://<app>prefix.L2 previously resolved calls to library functions and then threw them away. It now keeps them as
call_graphedges into a newapplication.external_symbolsmap.Design:
docs/design/specs/v2-l3-l4-dataflow.md. L4 (interprocedural SDG) is a separate, later PR.How Has This Been Tested?
Unit and gate tests (
go test ./..., run in CI on Go 1.26, thego.modminimum, and on the latest stable Go):x/tools/go/cfg-j 1vs-j Nbyte-identityLive Neo4j 5.26: the
neo4j_itsuite (41 tests) and repeated--eagerpushes, run in a throwaway Docker container.Real-app evaluation, reproducible with
scripts/eval_l3.sh --preparethenscripts/eval_l3.sh --all:k8s.io/api)-j 1,-j 16and a second-j 16docs/design/notes/l3-dataflow-metrics.mdanddocs/design/notes/l3-eval-2026-10.mdk8s.io/api)cfg/cdg/ddgedges-a 2(-j 1)-a 3analysis.json,-a 2→-a 3k8s.io/api)Takeaways:
-a 3in 12.4 s at 7.1 GB.k8s.io/api, 627kddgedges come almost entirely fromgenerated.pb.go, and the largest marshaller has 14.8k edges.ddgis long-tailed. The top 1% of callables hold 18–53% of each app's edges; the largest is generated conversion code at 130k.Bugs found by the evaluation and reviews, fixed here with regression tests:
func init().--eagerpush failed on Neo4j 5.x.--eagercombined with--target-filespurged the whole app from Neo4j and pushed back only the targeted packages. The purge now runs only on a full run.//linedirectives.gostatement arguments, such asg()ingo f(g()), were not recorded.x/toolsv0.46 could not read Go 1.27's compiled package data; upgraded to v0.52.Breaking Changes
Ids:
can://go/<app>/<file>/…becomescan://<app>/go/<file>/…. External symbols arecan://<app>/@external/<import-path>/<sig>.Body nodes: they now carry
id(<callable-id>@<local>) at every level, including-a 1and-a 2, as codeanalyzer-python does.L2 output gains library edges.
call_graphgains edges to library callees, a call node'scalleeis set for them instead ofnull, andapplication.external_symbolsis new. A type conversion is still a call node, but itscalleestaysnull; it is no longer resolved to a<Type>.__new__pseudo-callee.L1 records more:
kindcan bealiasordefinedis_deferred: trues.A().B(), are separate call nodes keyedl:candl:c/2gostatement arguments are now call sitesConsumers that count call nodes, or parse body keys as plain
line:col, are affected.Neo4j:
--emit neo4jnow projects at L3, addingGO_CFG_NEXT,GO_CDGandGO_DDG.:GoBodyNode.call_nodeis the local parent key, matching Python.python-sdk: saved Go
analysis.jsonfixtures need regenerating, and the Go backend's reader must mapcall_nodeback toparent(python-sdk#434).Toolchain:
go.modmoves fromgo 1.25.0togo 1.26.0, becausegolang.org/x/toolsv0.52 requires it.goline prints one clear warning instead of degrading silently.Upgrade notes for all of the above are in the README's "Upgrading" section.
Types of changes
Checklist
Additional context
-a 3and--graph-field-depth <k>(default 3).go/ast;deferis two nodes plus an exit chain (defer_resumeedges), and panic-capable statements getexceptionedges.prov: ["ssa"].go/ssacomparison check.//linedirectives, and one call walker shared by L1 and L3.