Optimize byte entropy and evaluate ARM64 assembly

[dreadl0ck]

Decision

Keep the portable, single-pass entropy optimization in production. Do not route production entropy through assembly or the four-bank Go variant. The ARM64 entropy assembly evaluation is complete: the kernel is correct, but it does not justify production adoption on the measured packet workload.

Retain the actual assembly and comparison harness behind entropyexperiment for reproducibility. Normal builds select only entropy.go and no assembly files. Enabling the tag exposes experimental functions; it does not change the decoder call path.

Changes

• Replace packet entropy's 256 full payload scans and Protobuf's map counting with a shared, allocation-free 256-bin histogram. Reuse it for file entropy.
• Preserve empty-input behavior and byte-based Shannon entropy. Protobuf accumulation becomes deterministic rather than map-iteration ordered, so insignificant floating-point differences are possible.
• Implement a scalar, four-bank ARM64 assembly histogram with exact tails and a matching four-bank portable Go baseline. This is assembly, not a SIMD implementation; neither uses approximate logarithms.
• Add known-value, randomized differential, allocation, fuzz, raw-histogram, misalignment, wide-counter, and inaccessible-page tests.
• Add synthetic and real-PCAP entropy replay benchmarks. Label non-ARM64/purego results as GoFallback, not assembly.
• Remove the speculative 16 KiB dispatcher: no defensible crossover or packet-workload benefit was demonstrated.

Measurements

Apple M5 Max, darwin/arm64, Go 1.27.0. All entropy implementations below allocate zero bytes per call. The shared machine had substantial concurrent load and timing drift; results are descriptive, not statistically significant throughput guarantees.

Portable Optimization Versus Previous Code

Median ns/op from three 1-second samples, 1,500-byte inputs, from the initial implementation run:

Input	Shared histogram	Old packet helper	Old Protobuf helper
Repeated	1,768	5,751	8,397
Text	731	6,651	10,928
Random	1,755	8,853	22,192

These are entropy microbenchmarks, not whole-capture speedups. File entropy already used a histogram; no file-specific speedup is claimed.

Assembly Versus Stronger Go Baseline

Full entropy including zeroing, counting, merging, and exact logs. Medians from five separate 1-second benchmark passes with -cpu=1:

Length	Input	Single-bank Go ns/op	Four-bank Go ns/op	Four-bank ARM64 ns/op
16,384	Repeated	23,396	4,863	4,841
16,384	Text	7,284	5,548	4,982
16,384	Random	7,864	6,992	6,813
1,048,576	Repeated	1,273,320	263,289	306,811
1,048,576	Text	460,716	369,903	317,842
1,048,576	Random	418,705	367,086	343,071

Assembly has some large-buffer wins, but portable four-bank Go captures much of the improvement and wins on 1 MiB repeated data. Large-buffer microbenchmarks alone do not support changing packet entropy.

Real Captured Payloads

The-Ultimate-PCAP-v20200224.pcapng: 21,933 packets, 53,579 Ethernet/IPv4/TCP/UDP layer payload inputs, and zero inputs >= 16 KiB. Payload replay excludes file reading and packet decoding. Median process CPU per complete replay from three samples of ten replays each:

Implementation	CPU/replay
Single-bank Go	101.758 ms
Four-bank Go	122.952 ms
Four-bank ARM64	118.553 ms

Assembly used about 17% more CPU than the production Go baseline in this replay. Process CPU includes runtime work and remains sensitive to frequency/core placement; it is not an isolated hardware-counter measurement.

Differential Verification on a Real Capture

pcaps/The Ultimate PCAP v20250325.pcapng: 49,380 packets, 15,989,140 bytes, mixed per-packet encapsulation (39,236 Ethernet, 9,298 IEEE 802.3br mPackets, 846 Linux cooked). Full default build including DPI, all decoders enabled, --entropy=true, JSON output, file extraction on, one packet worker and one stream worker.

Master (e33ac5f7) versus this branch, comparing every audit record with the per-run header line excluded:

Output	Result
Ethernet, IPv4, IPv6, TCP, UDP, ICMPv4, ICMPv6, ICMPv6Echo, EthernetCTP, EthernetCTPReply	138,578 records byte-identical, PayloadEntropy included
File.json (Entropy, MD5/SHA1/SHA256, analysis flags)	34 records identical after normalizing the output path
Extracted file tree	17 files, identical
61 of 84 audit record files	identical
23 remaining files	differ — see control below

The 23 differing files are pipeline nondeterminism, not this change. Running the same master binary twice produces the same differing set, with larger divergence:

Comparison	Protobuf unique records	DNS unique	Connection unique	Missing files
master vs master (control)	4,221 / 3,980	1,300 / 1,300	845 / 845	Exploit.json absent from one run
master vs branch	4,394 / 4,374	1,294 / 1,294	828 / 828	none

HTTP, DeviceProfile, Host and TLSRecord were identical in both comparisons. Master-versus-master also varied by 241 PayloadEntropy values, more than the 20 seen master-versus-branch. Stream decoder output is not reproducible run to run in this checkout, independent of this PR.

Entropy Cost on That Capture's Real Payloads

Replaying the 95,261 real packet-layer payload inputs (25,991,234 bytes, zero inputs >= 16 KiB) from the same capture. Median process CPU per full replay, five samples of ten replays:

Implementation	CPU/replay	Allocations
Shared histogram (this PR)	117.2 ms	0 B, 0 allocs
Old packet helper (256 scans)	644.8 ms	0 B, 0 allocs
Old Protobuf helper (map)	621.7 ms	440.6 MB, 691,368 allocs
Four-bank Go	149.6 ms	0 B, 0 allocs
Four-bank ARM64 assembly	149.4 ms	0 B, 0 allocs

On this real traffic the merged implementation is about 5.5x faster than the previous packet helper and 5.3x faster than the previous Protobuf helper, while removing 440 MB of allocation churn. Both the assembly and the four-bank Go variant are slower than the merged portable code here, which is the direct evidence for rejecting assembly.

Capture Pipeline

Built three temporary CLI binaries with nodpi,nomagika,noyara,entropyexperiment, selecting each entropy implementation unconditionally for the trial. Production source was restored before committing. The CLI-only benchmark builds used an ignored placeholder embedded frontend, never served or committed.

Replayed the same PCAP concatenated 16 times with mergecap -a: 350,928 packets, approximately 113 MB. One worker, GOMAXPROCS=1, GOGC=100, entropy on, null output, no payload/context storage, TCP reassembly off, resolver databases/DNS off, WebUI off. One warm-up per binary, then five rounds with rotated binary order.

Implementation	Median process CPU	Median processing wall time
Single-bank Go	5.078 s	6.691 s
Four-bank Go	5.941 s	8.762 s
Four-bank ARM64	5.927 s	8.387 s

Wall times were highly noisy under concurrent host load; do not interpret their ratios as reliable speedup estimates. The completed run did not demonstrate an assembly benefit. Null output omits serialization, so this is not an output-throughput benchmark. QUIC/Kerberos UDP decoders still initialize in this checkout despite the packet-only include list.

A separate short HTTP CPU profile of the scalar capture attributed approximately 53% cumulative samples to entropy, including approximately 35% to math.log2. This supports measuring the complete entropy function rather than its counting loop alone; the sample is too short for precise attribution claims.

End-to-end timing of the full DPI-enabled pipeline on the 49,380-packet capture could not resolve a difference. Across five alternating repetitions, median process CPU was 24.11 s for master and 22.54 s for this branch with entropy enabled, but 22.64 s versus 24.44 s with entropy disabled — where the two binaries execute identical code. The sign flips, so these end-to-end numbers are noise on this loaded host and no end-to-end speedup is claimed. Entropy is a small share of a full DPI capture, and packet entropy is off by default.

Reproduce the Committed Experiments

go test -tags entropyexperiment ./internal/entropy
go test -race -tags entropyexperiment ./internal/entropy
go test -tags entropyexperiment,purego ./internal/entropy
go vet -tags entropyexperiment ./internal/entropy
go test -tags entropyexperiment ./internal/entropy -run '^$' -fuzz '^FuzzExperiment$' -fuzztime=10s -parallel=4
go test -tags entropyexperiment ./internal/entropy -run '^$' -bench '^BenchmarkExperiment$' -benchtime=300ms -count=3 -cpu=1
NETCAP_ENTROPY_PCAP='pcaps/The Ultimate PCAP v20250325.pcapng' go test -tags entropyexperiment ./internal/entropy -run '^$' -bench BenchmarkCapturePayloads -benchtime=10x -count=5 -cpu=1

For a CLI trial, temporarily select BytesGo4(data) or BytesARM64(data) in entropy.Bytes, build with nodpi,nomagika,noyara,entropyexperiment, and restore the production function afterward. Use these capture arguments for each binary and a distinct output directory:

GOMAXPROCS=1 GOGC=100 ./net capture --read=ultimate-16x.pcapng \
  --include=Ethernet,IPv4,TCP,UDP --null=true --entropy=true \
  --compress=false --buf=false --payload=false --context=false \
  --reverse-dns=false --local-dns=false --macDB=false --ja4DB=false \
  --serviceDB=false --geoDB=false --dpi=false --reassemble-connections=false \
  --ip4defrag=false --ignore-unknown=true --log-errors=false \
  --workers=1 --pbuf=1000 --quiet=true --time=true --http "" --out=trial-output

Validation and Limits

• Default helper and all three affected decoder packages pass with nodpi.
• Experimental native, purego, race, vet/assembly declarations, fuzzing, and guard-page checks pass. Assembly memory accesses themselves are not race-instrumented.
• Linux AMD64 and 386 test binaries cross-compile; this is portability validation, not native AMD64 performance evaluation.
• Default go list confirms [entropy.go] []: no experimental Go or assembly in normal builds.
• Fresh worktrees need ignored protobuf generation: protoc --gogofaster_out=types/. netcap.proto. CLI builds also need frontend embed assets.
• Use --http "" to disable WebUI; --http= is rejected here. Use HTTP pprof for profiling: the existing --cpuprof startup is incorrectly deferred. Neither unrelated CLI issue is changed by this PR.
• Full local go test -short -race ./... passes with a built frontend bundle and the gitignored PCAP fixtures present. This evaluation concerns the ARM64 entropy kernel, not all possible checksum/parser/SIMD optimizations.
• Two pre-existing defects were observed while verifying and are not addressed here: types/bfd.go:96 calls the value-receiver getString on a nil *BFDAuthHeader, so --csv export segfaults on any BFD packet lacking an auth header; and stream decoder output is not reproducible between identical runs.
• Default packet entropy remains disabled. No default-capture speedup or universal assembly conclusion is claimed.

[github-actions]

Thank you for your contribution to NETCAP!

Before we can accept your pull request, you need to sign our Contributor License Agreement (CLA).

To sign the CLA, please read the Individual CLA and comment below with:

I have read and agree to the NETCAP CLA

If you are contributing on behalf of a company or organization, please contact the maintainers to complete the Entity CLA instead.

---

You can re-trigger this check by commenting recheck on this PR.

---

I have read and agree to the NETCAP CLA

---

_{You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.}