Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# signals

A command for tracing security-relevant evidence stored in Git across repository history. It shows when detected evidence appears, changes, or disappears, with the commit and path behind each change. `signals` walks commit trees with [`github.com/git-pkgs/history`](https://github.com/git-pkgs/history). Local parsers detect workflow risks and permissions, artifact-provenance configuration, CI and security tools, threat surface, secrets, certificates and trust stores, Agent Skills, sensitive ignore policy, REUSE metadata, package managers, manifests, lockfiles, dependency pinning, update configuration, licenses, SBOMs, fuzzers, and binary artifacts. Commit data adds dependency-bot and signature events plus contributor, maintenance, review, and signing metrics. The output contains the evidence and its provenance without calculating a score. The scan reads local Git objects, with forge APIs, release APIs, credential validation, and external services disabled.
A command for tracing security-relevant evidence stored in Git across repository history. It shows when detected evidence appears, changes, or disappears, with the commit and path behind each change. `signals` walks commit trees with [`github.com/git-pkgs/history`](https://github.com/git-pkgs/history). Local parsers detect workflow risks and permissions, artifact-provenance configuration, CI and security tools, threat surface, secrets, certificates and trust stores, stored SARIF results, Agent Skills, sensitive ignore policy, REUSE metadata, package managers, manifests, lockfiles, dependency pinning, update configuration, licenses, SBOMs, fuzzers, and binary artifacts. Commit data adds dependency-bot and signature events plus contributor, maintenance, review, and signing metrics. The output contains the evidence and its provenance without calculating a score. The scan reads local Git objects, with forge APIs, release APIs, credential validation, and external services disabled.

The local-only timeline answers questions about when repository evidence changed. Examples include:

Expand Down Expand Up @@ -154,6 +154,7 @@ The additional detectors use the same persistent signal model. A threat-surface
| Packaging | Literal packaging and publication commands or known publication Actions; presence does not show that a workflow ran |
| Package-Provenance | Explicit npm provenance flags or environment settings, and PyPI publication configured with an OIDC token |
| Runner-Selection | Explicit self-hosted runner labels, runner groups, and dynamic runner expressions in GitHub Actions jobs |
| SARIF | Bounded summaries of committed SARIF tools, result counts and levels, baselines, rules, suppressions, and CWE taxa with names and categories |
| Package-Parse | Package files that were identified but could not be parsed |
| Secret | Findings from the embedded `git-pkgs/secrets` rule corpus; output contains `[redacted]` in place of the credential |
| Sensitive-Ignore | Effective `.gitignore` rules covering a bounded list of environment, credential, private-key, secret, state, and database paths, with the source pattern and line |
Expand All @@ -163,7 +164,7 @@ The additional detectors use the same persistent signal model. A threat-surface

The root commit and missing history boundaries receive a full scan. Later commits reuse their first parent's signal state and update the paths changed by that commit. Cross-file detectors, including Agent Skills, dependency-update configuration, REUSE metadata, and sensitive ignore policy, receive a full commit tree when one of their inputs changes. Agent Skills rerun when a `SKILL.md` or a file inside a known skill package changes. Sensitive ignore policy reruns only when a project `.gitignore` changes.

Parser results for changed files are cached by Git blob OID and path for the whole run. Repository files that cannot contain a certificate, license, SBOM, fuzzer, policy, or SAST configuration need only a 1 KB prefix for binary classification. Commit reading can run concurrently, while signal-state updates remain ordered. `github.com/git-pkgs/languages` keeps a language inventory so Brief can rerun language-gated detection only when needed. `github.com/git-pkgs/roles` removes bundled and derived paths from fuzzing and license detection, and removes test, fixture, example, cache, and build-output paths from certificate detection. Test paths remain eligible for fuzz detection because fuzz targets commonly use them.
Parser results for changed files are cached by Git blob OID and path for the whole run. Repository files that cannot contain a certificate, license, SARIF log, SBOM, fuzzer, policy, or SAST configuration need only a 1 KB prefix for binary classification. SARIF candidates use streaming byte, tool, and distinct-value limits, and bundled, derived, test, and example paths are skipped for SARIF detection. Commit reading can run concurrently, while signal-state updates remain ordered. `github.com/git-pkgs/languages` keeps a language inventory so Brief can rerun language-gated detection only when needed. `github.com/git-pkgs/roles` removes bundled and derived paths from fuzzing and license detection, and removes test, fixture, example, cache, and build-output paths from certificate detection. Test paths remain eligible for fuzz detection because fuzz targets commonly use them.

Brief reads commit trees through `history.TreeFS` and reruns only when Brief's changed-file filter identifies an input to one of the emitted tool signals. Package files are parsed when their paths change, and the parse result supplies dependency constraints, pins, Actions references, container images, lockfile resolution, integrity records, and manifest-declared license values or file references. Results are cached by blob OID and path. Shell commands that download or install packages are not yet included in this local package detector. Package-manager detection uses `managers.AffectedDirectories` to rescan only the affected projects. Secret scanning updates changed paths and caches results by blob OID and path because some rules depend on the filename. REUSE metadata changes trigger a full REUSE scan; ordinary changes update the affected file state and coverage count.

Expand All @@ -179,6 +180,8 @@ The local detectors share parsers and changed-path caches across related signals
- [`github.com/git-pkgs/managers`](https://github.com/git-pkgs/managers) identifies package managers and their manifest or lockfile evidence from commit trees.
- [`github.com/git-pkgs/sbom`](https://github.com/git-pkgs/sbom) identifies and parses SPDX and CycloneDX JSON documents.
- [`github.com/git-pkgs/spdx`](https://github.com/git-pkgs/spdx) validates and normalizes SPDX expressions declared by package manifests.
- [`github.com/git-pkgs/sarif`](https://github.com/git-pkgs/sarif) reads bounded summaries from committed SARIF logs.
- [`github.com/git-pkgs/cwe`](https://github.com/git-pkgs/cwe) adds names and categories to CWE taxa referenced by SARIF results.
- [`github.com/git-pkgs/brief`](https://github.com/git-pkgs/brief) detects CI systems, build tools, security tools, and pre-commit runners from an `fs.FS` commit tree.
- [`github.com/git-pkgs/languages`](https://github.com/git-pkgs/languages) detects source languages from paths and blob content.
- [`github.com/git-pkgs/reuse`](https://github.com/git-pkgs/reuse) resolves SPDX headers, sidecars, REUSE.toml annotations, and DEP5 metadata from an `fs.FS` commit tree.
Expand Down
1 change: 1 addition & 0 deletions detector.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ const (
checkDependencyUpdateTool = "Dependency-Update-Tool"
checkFuzzing = "Fuzzing"
checkLicense = "License"
checkSARIF = "SARIF"
checkSAST = "SAST"
checkSBOM = "SBOM"
checkSecurityPolicy = "Security-Policy"
Expand Down
2 changes: 2 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ require (
actionlint.kjanat.dev v1.17.0
github.com/git-pkgs/brief v0.15.1-0.20261107204040-d1ed5a631f83
github.com/git-pkgs/certs v0.1.1-0.20261104093717-2e1d0e2e0f63
github.com/git-pkgs/cwe v0.1.2-0.20261102114243-2d9276da42a5
github.com/git-pkgs/gitignore v1.3.1-0.20261108194038-d09d5ccc19b9
github.com/git-pkgs/history v0.2.0
github.com/git-pkgs/languages v0.1.0
Expand All @@ -16,6 +17,7 @@ require (
github.com/git-pkgs/manifests v0.13.1-0.20261107212808-69ba49d733ae
github.com/git-pkgs/reuse v0.2.0
github.com/git-pkgs/roles v0.1.3
github.com/git-pkgs/sarif v0.1.4-0.20261108212606-7e62951486f6
github.com/git-pkgs/sbom v0.1.10-0.20261107160107-38436af679c6
github.com/git-pkgs/secrets v0.2.1-0.20261107154440-a977b8886af8
github.com/git-pkgs/skills v0.2.1
Expand Down
4 changes: 4 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ github.com/git-pkgs/citation v0.1.0 h1:0toQBjxVOtaUsxRtcDJL9uZ6SYFb6IgbHgKpTpP0c
github.com/git-pkgs/citation v0.1.0/go.mod h1:4vrS78/jjHeusS+eozjP/d3pX2NogEi1R/7gLVqkgqs=
github.com/git-pkgs/codemeta v0.3.0 h1:zkWndmquCv2tPIHCHb+JXx3eKjmxRoD48f4/3sNAlGw=
github.com/git-pkgs/codemeta v0.3.0/go.mod h1:R1vGXVohTa22vVKaV3/8xVs3+tcgvqc4ziaMikdXc/I=
github.com/git-pkgs/cwe v0.1.2-0.20261102114243-2d9276da42a5 h1:6RiZJiHYlzqQM2SteX96wk4l9eZ6CONI2ji4UT9D8fg=
github.com/git-pkgs/cwe v0.1.2-0.20261102114243-2d9276da42a5/go.mod h1:NYsSggIvPa2LcKe0nrTj7v+dJs8v3BFpPkLKqtBiZOA=
github.com/git-pkgs/gitignore v1.3.1-0.20261108194038-d09d5ccc19b9 h1:+1HyWVLjMCi4Xoq9Gh1oq2tmbNn1Aa52tShtuPGddOo=
github.com/git-pkgs/gitignore v1.3.1-0.20261108194038-d09d5ccc19b9/go.mod h1:zleLGZ4PhQrUXrJSsVQWKoeEllEjJH49+DjyKpzIFPM=
github.com/git-pkgs/history v0.2.0 h1:UCbrp/PX6rfLukUCRnYme8PtyCqKO8Q5wrpDUY0SOTE=
Expand All @@ -66,6 +68,8 @@ github.com/git-pkgs/reuse v0.2.0 h1:zftBZEVsAsiLAN6kb7CqOHfxz7ykX0+QThHFLtSxDKs=
github.com/git-pkgs/reuse v0.2.0/go.mod h1:qYnkGISyubaTfY8/a4qSIoyS8Hmw1urMepveCPKiGYg=
github.com/git-pkgs/roles v0.1.3 h1:BkXoBFg0/4szCvBes8mD+UApRe1jITaZ/4bs+XtRxdw=
github.com/git-pkgs/roles v0.1.3/go.mod h1:USQTDlh22S7jNyAxU5SwtMfXjbX0lyJpVsGLTccyYis=
github.com/git-pkgs/sarif v0.1.4-0.20261108212606-7e62951486f6 h1:9kW358S2DIq/Lf63r3EF9qrxbU3auD/NAX2TjVnbWfo=
github.com/git-pkgs/sarif v0.1.4-0.20261108212606-7e62951486f6/go.mod h1:dxohVtow0t8jXFaJ1g6Lmn2d/POHxXKUukKPQU0/jgs=
github.com/git-pkgs/sbom v0.1.10-0.20261107160107-38436af679c6 h1:2xvJPZQJcYe8eHKStvfNx5TUN0tw3R9LHMdO9UI0nuE=
github.com/git-pkgs/sbom v0.1.10-0.20261107160107-38436af679c6/go.mod h1:KkUYj+Q0RQIuINEj35XNmQCv/LA3xJN7VfZbOMQj6Dk=
github.com/git-pkgs/scan v0.1.1 h1:q5VsnGgd4wpjt/EZ2YnuIpjUZkj1OWzySv3IEwo8wPk=
Expand Down
232 changes: 219 additions & 13 deletions repository_files.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,13 @@ import (
"unicode"

gitcerts "github.com/git-pkgs/certs"
"github.com/git-pkgs/cwe"
githistory "github.com/git-pkgs/history"
gitlanguages "github.com/git-pkgs/languages"
gitlicenses "github.com/git-pkgs/licenses"
"github.com/git-pkgs/magic"
"github.com/git-pkgs/roles"
sarifsummary "github.com/git-pkgs/sarif/summary"
"github.com/git-pkgs/sbom"
)

Expand Down Expand Up @@ -95,6 +97,10 @@ const (
asn1HeaderBytes = 2
asn1MaxLengthBytes = 4
bitsPerByte = 8
sarifMaxBytes = 16 << 20
sarifInvalidName = "invalid"
sarifMaxUniqueValues = 5_000
sarifMaxTools = 100
)

var certificateFileExtensions = map[string]bool{
Expand Down Expand Up @@ -226,19 +232,11 @@ func repositoryFileSignalsFromFS(
})
}

if sbomContentPath(filePath) {
parsed, err := scanRepositoryFile(tree, filePath, func(reader io.Reader) ([]Signal, error) {
signal, ok, inspectErr := sbomSignal(filePath, reader)
if !ok {
return nil, inspectErr
}
return []Signal{signal}, inspectErr
})
if err != nil {
return nil, err
}
signals = append(signals, parsed...)
structured, err := repositoryStructuredSignals(tree, filePath)
if err != nil {
return nil, err
}
signals = append(signals, structured...)

if securityPolicyPath(filePath) {
parsed, err := scanRepositoryFile(tree, filePath, func(reader io.Reader) ([]Signal, error) {
Expand Down Expand Up @@ -274,6 +272,24 @@ func repositoryFileSignalsFromFS(
return signals, nil
}

func repositoryStructuredSignals(tree fs.FS, filePath string) ([]Signal, error) {
if sarifFilePath(filePath) {
return scanRepositoryFile(tree, filePath, func(reader io.Reader) ([]Signal, error) {
return sarifSignals(filePath, reader)
})
}
if !sbomContentPath(filePath) {
return nil, nil
}
return scanRepositoryFile(tree, filePath, func(reader io.Reader) ([]Signal, error) {
signal, ok, err := sbomSignal(filePath, reader)
if !ok {
return nil, err
}
return []Signal{signal}, err
})
}

func readRepositoryFile(tree fs.FS, filePath string, limit int64) ([]byte, error) {
var data []byte
err := scanRepositoryFileError(tree, filePath, func(reader io.Reader) error {
Expand Down Expand Up @@ -657,6 +673,196 @@ func sbomFileName(filePath string) bool {
return false
}

func sarifFilePath(filePath string) bool {
labels, err := roles.Match(filePath)
if err == nil && (labels.BundledOrDerived() || labels.Has(roles.Test) || labels.Has(roles.Example)) {
return false
}
lower := strings.ToLower(path.Base(filePath))
return strings.HasSuffix(lower, ".sarif") || strings.HasSuffix(lower, ".sarif.json")
}

func sarifSignals(filePath string, reader io.Reader) ([]Signal, error) {
inspection, err := sarifsummary.Inspect(reader, sarifsummary.Options{
MaxBytes: sarifMaxBytes,
MaxUniqueValues: sarifMaxUniqueValues,
MaxTools: sarifMaxTools,
})
if errors.Is(err, sarifsummary.ErrInvalid) {
return []Signal{{
Check: checkSARIF,
Kind: "candidate",
Name: sarifInvalidName,
Path: filePath,
}}, nil
}
if err != nil && !errors.Is(err, sarifsummary.ErrLimit) {
return nil, err
}

signals := []Signal{{
Check: checkSARIF,
Kind: "document",
Name: inspection.Version,
Value: "runs=" + strconv.Itoa(inspection.Runs) +
"; results=" + strconv.Itoa(inspection.Results) +
"; occurrences=" + strconv.Itoa(inspection.Occurrences) +
"; suppressed=" + strconv.Itoa(inspection.SuppressedResults),
Path: filePath,
}}
for _, tool := range inspection.Tools {
value := ""
if tool.SemanticVersion != "" {
value = "semantic-version=" + tool.SemanticVersion
}
if tool.Version != "" {
if value != "" {
value += "; "
}
value += "version=" + tool.Version
}
signals = append(signals, Signal{
Check: checkSARIF,
Kind: "tool",
Name: tool.Name,
Value: value,
Path: filePath,
})
}
signals = appendSARIFCounts(signals, filePath, "level", inspection.Levels)
signals = appendSARIFCounts(signals, filePath, "baseline", inspection.BaselineStates)
signals = appendSARIFCounts(signals, filePath, "rule", inspection.Rules)
signals = appendSARIFTaxa(signals, filePath, inspection.Taxa)
signals = appendSARIFCWE(signals, filePath, inspection.Taxa)
signals = appendSARIFCounts(signals, filePath, "suppression", inspection.Suppressions)
if inspection.Truncated || errors.Is(err, sarifsummary.ErrLimit) {
signals = append(signals, Signal{
Check: checkSARIF,
Kind: "coverage",
Name: "truncated",
Path: filePath,
})
}
sortSignals(signals)
return deduplicateSignals(signals), nil
}

func appendSARIFTaxa(signals []Signal, filePath string, taxa []sarifsummary.Taxon) []Signal {
for _, taxon := range taxa {
name := taxon.ID
if name == "" {
name = taxon.GUID
}
switch {
case isCWETaxonomy(taxon.TaxonomyName):
if id, _, ok := cwe.Lookup(name); ok {
name = id
} else {
name = "CWE:" + name
}
case taxon.TaxonomyName != "":
name = taxon.TaxonomyName + ":" + name
case taxon.TaxonomyGUID != "":
name = taxon.TaxonomyGUID + ":" + name
}
signals = append(signals, Signal{
Check: checkSARIF,
Kind: "taxon",
Name: name,
Value: strconv.Itoa(taxon.Count),
Path: filePath,
})
}
return signals
}

func appendSARIFCWE(signals []Signal, filePath string, taxa []sarifsummary.Taxon) []Signal {
type weakness struct {
count int
name string
categoryID string
}
weaknesses := make(map[string]weakness)
categories := make(map[string]int)
categoryNames := make(map[string]string)
for _, taxon := range taxa {
if !isCWETaxonomy(taxon.TaxonomyName) {
continue
}
id, entry, ok := cwe.Lookup(taxon.ID)
if !ok {
continue
}
categoryID := cwe.CategoryID(entry.Category)
current := weaknesses[id]
current.count += taxon.Count
current.name = entry.Name
current.categoryID = categoryID
weaknesses[id] = current
if categoryID != "" {
categories[categoryID] += taxon.Count
categoryNames[categoryID] = entry.Category
}
}

ids := make([]string, 0, len(weaknesses))
for id := range weaknesses {
ids = append(ids, id)
}
sort.Strings(ids)
for _, id := range ids {
item := weaknesses[id]
value := "count=" + strconv.Itoa(item.count) + "; name=" + item.name
if item.categoryID != "" {
value += "; category=" + item.categoryID
}
signals = append(signals, Signal{
Check: checkSARIF,
Kind: "weakness",
Name: id,
Value: value,
Path: filePath,
})
}
ids = ids[:0]
for id := range categories {
ids = append(ids, id)
}
sort.Strings(ids)
for _, id := range ids {
signals = append(signals, Signal{
Check: checkSARIF,
Kind: "weakness-category",
Name: id,
Value: "count=" + strconv.Itoa(categories[id]) + "; name=" + categoryNames[id],
Path: filePath,
})
}
return signals
}

func isCWETaxonomy(name string) bool {
return strings.EqualFold(name, "CWE") || strings.EqualFold(name, "Common Weakness Enumeration")
}

func appendSARIFCounts(signals []Signal, filePath, kind string, counts map[string]int) []Signal {
names := make([]string, 0, len(counts))
for name := range counts {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
signals = append(signals, Signal{
Check: checkSARIF,
Kind: kind,
Name: name,
Value: strconv.Itoa(counts[name]),
Path: filePath,
})
}
return signals
}

func securityPolicyPath(filePath string) bool {
switch strings.ToLower(filePath) {
case "security.md", ".github/security.md", "docs/security.md",
Expand Down Expand Up @@ -887,7 +1093,7 @@ func (d *localDetectors) runRepositoryFiles(
}

delete(current.errors, detectorRepositoryFiles)
checks := []string{checkBinaryArtifacts, checkCertificate, checkFuzzing, checkLicense, checkSAST, checkSBOM, checkSecurityPolicy}
checks := []string{checkBinaryArtifacts, checkCertificate, checkFuzzing, checkLicense, checkSARIF, checkSAST, checkSBOM, checkSecurityPolicy}
if forceFull {
for _, check := range checks {
switch check {
Expand Down
Loading
Loading