Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# signals

A command for tracing security-relevant evidence stored in Git across repository history. It shows when detected evidence appears, changes, or disappears, with the commit and path behind each change. `signals` walks commit trees with [`github.com/git-pkgs/history`](https://github.com/git-pkgs/history). Local parsers detect workflow risks and permissions, artifact-provenance configuration, CI and security tools, threat surface, secrets, certificates and trust stores, Agent Skills, REUSE metadata, package managers, manifests, lockfiles, dependency pinning, update configuration, licenses, SBOMs, fuzzers, and binary artifacts. Commit data adds dependency-bot and signature events plus contributor, maintenance, review, and signing metrics. The output contains the evidence and its provenance without calculating a score. The scan reads local Git objects, with forge APIs, release APIs, credential validation, and external services disabled.
A command for tracing security-relevant evidence stored in Git across repository history. It shows when detected evidence appears, changes, or disappears, with the commit and path behind each change. `signals` walks commit trees with [`github.com/git-pkgs/history`](https://github.com/git-pkgs/history). Local parsers detect workflow risks and permissions, artifact-provenance configuration, CI and security tools, threat surface, secrets, certificates and trust stores, Agent Skills, sensitive ignore policy, REUSE metadata, package managers, manifests, lockfiles, dependency pinning, update configuration, licenses, SBOMs, fuzzers, and binary artifacts. Commit data adds dependency-bot and signature events plus contributor, maintenance, review, and signing metrics. The output contains the evidence and its provenance without calculating a score. The scan reads local Git objects, with forge APIs, release APIs, credential validation, and external services disabled.

The local-only timeline answers questions about when repository evidence changed. Examples include:

Expand Down Expand Up @@ -156,11 +156,12 @@ The additional detectors use the same persistent signal model. A threat-surface
| Runner-Selection | Explicit self-hosted runner labels, runner groups, and dynamic runner expressions in GitHub Actions jobs |
| Package-Parse | Package files that were identified but could not be parsed |
| Secret | Findings from the embedded `git-pkgs/secrets` rule corpus; output contains `[redacted]` in place of the credential |
| Sensitive-Ignore | Effective `.gitignore` rules covering a bounded list of environment, credential, private-key, secret, state, and database paths, with the source pattern and line |
| REUSE | Per-file SPDX license declarations, REUSE.toml or DEP5 metadata, and the number of non-empty covered files with both license and copyright information |

## History scanning

The root commit and missing history boundaries receive a full scan. Later commits reuse their first parent's signal state and update the paths changed by that commit. Cross-file detectors, including Agent Skills, dependency-update configuration, and REUSE metadata, receive a full commit tree when one of their inputs changes. Agent Skills rerun when a `SKILL.md` or a file inside a known skill package changes.
The root commit and missing history boundaries receive a full scan. Later commits reuse their first parent's signal state and update the paths changed by that commit. Cross-file detectors, including Agent Skills, dependency-update configuration, REUSE metadata, and sensitive ignore policy, receive a full commit tree when one of their inputs changes. Agent Skills rerun when a `SKILL.md` or a file inside a known skill package changes. Sensitive ignore policy reruns only when a project `.gitignore` changes.

Parser results for changed files are cached by Git blob OID and path for the whole run. Repository files that cannot contain a certificate, license, SBOM, fuzzer, policy, or SAST configuration need only a 1 KB prefix for binary classification. Commit reading can run concurrently, while signal-state updates remain ordered. `github.com/git-pkgs/languages` keeps a language inventory so Brief can rerun language-gated detection only when needed. `github.com/git-pkgs/roles` removes bundled and derived paths from fuzzing and license detection, and removes test, fixture, example, cache, and build-output paths from certificate detection. Test paths remain eligible for fuzz detection because fuzz targets commonly use them.

Expand All @@ -172,6 +173,7 @@ The local detectors share parsers and changed-path caches across related signals

- [`github.com/git-pkgs/magic`](https://github.com/git-pkgs/magic) detects binary content.
- [`github.com/git-pkgs/certs`](https://github.com/git-pkgs/certs) scans bounded certificate files, trust stores, and small archives.
- [`github.com/git-pkgs/gitignore`](https://github.com/git-pkgs/gitignore) resolves effective ignore rules and returns their source pattern and line.
- [`github.com/git-pkgs/licenses`](https://github.com/git-pkgs/licenses) identifies license text, notices, and SPDX declarations with its embedded ScanCode corpus.
- [`github.com/git-pkgs/manifests`](https://github.com/git-pkgs/manifests) parses package metadata, declared licenses, Actions, Docker, NuGet, and other dependency files.
- [`github.com/git-pkgs/managers`](https://github.com/git-pkgs/managers) identifies package managers and their manifest or lockfile evidence from commit trees.
Expand Down
1 change: 1 addition & 0 deletions detector.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ const (
checkSAST = "SAST"
checkSBOM = "SBOM"
checkSecurityPolicy = "Security-Policy"
checkSensitiveIgnore = "Sensitive-Ignore"
checkTokenPermissions = "Token-Permissions"
signalKindDeclaration = "declaration"
signalKindFileReference = "file-reference"
Expand Down
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ require (
actionlint.kjanat.dev v1.17.0
github.com/git-pkgs/brief v0.15.1-0.20261107204040-d1ed5a631f83
github.com/git-pkgs/certs v0.1.1-0.20261104093717-2e1d0e2e0f63
github.com/git-pkgs/gitignore v1.3.1-0.20261108194038-d09d5ccc19b9
github.com/git-pkgs/history v0.2.0
github.com/git-pkgs/languages v0.1.0
github.com/git-pkgs/licenses v0.10.1-0.20261107154534-00c5bf03ade8
Expand Down
2 changes: 2 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ github.com/git-pkgs/citation v0.1.0 h1:0toQBjxVOtaUsxRtcDJL9uZ6SYFb6IgbHgKpTpP0c
github.com/git-pkgs/citation v0.1.0/go.mod h1:4vrS78/jjHeusS+eozjP/d3pX2NogEi1R/7gLVqkgqs=
github.com/git-pkgs/codemeta v0.3.0 h1:zkWndmquCv2tPIHCHb+JXx3eKjmxRoD48f4/3sNAlGw=
github.com/git-pkgs/codemeta v0.3.0/go.mod h1:R1vGXVohTa22vVKaV3/8xVs3+tcgvqc4ziaMikdXc/I=
github.com/git-pkgs/gitignore v1.3.1-0.20261108194038-d09d5ccc19b9 h1:+1HyWVLjMCi4Xoq9Gh1oq2tmbNn1Aa52tShtuPGddOo=
github.com/git-pkgs/gitignore v1.3.1-0.20261108194038-d09d5ccc19b9/go.mod h1:zleLGZ4PhQrUXrJSsVQWKoeEllEjJH49+DjyKpzIFPM=
github.com/git-pkgs/history v0.2.0 h1:UCbrp/PX6rfLukUCRnYme8PtyCqKO8Q5wrpDUY0SOTE=
github.com/git-pkgs/history v0.2.0/go.mod h1:DaoJasfycHYLtwr8ZTDp7PDnU8WMjAqmadHgJDqAO4A=
github.com/git-pkgs/languages v0.1.0 h1:Y7bwb86k9DgagCJn3bXW6+8lOtwayDD17Np2usEPooY=
Expand Down
31 changes: 30 additions & 1 deletion local.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ const (
detectorBrief = "Brief"
detectorAgentSkills = "Agent-Skills"
detectorRepositoryFiles = "Repository-Files"
detectorSensitiveIgnore = "Sensitive-Ignores"
detectorWorkflows = "Workflows"

maxSecretBlobSize = 1 << 20
Expand All @@ -51,6 +52,7 @@ var localDetectorNames = []string{
detectorPackages,
detectorPackageManagers,
detectorRepositoryFiles,
detectorSensitiveIgnore,
detectorWorkflows,
checkSecret,
checkREUSE,
Expand Down Expand Up @@ -165,6 +167,8 @@ func (d *localDetectors) detect(
packageRun := packageFull || packagesAffected(changes)
repositoryFileFull := forceFull || parent.errors[detectorRepositoryFiles] != ""
repositoryFileRun := repositoryFileFull || len(changes) > 0
sensitiveIgnoreFull := forceFull || parent.errors[detectorSensitiveIgnore] != ""
sensitiveIgnoreRun := sensitiveIgnoreFull || sensitiveIgnoresAffected(changes)
workflowAffected := workflowsAffected(changes)
workflowError := parent.errors[detectorWorkflows]
workflowFull := forceFull || workflowError != "" &&
Expand All @@ -185,7 +189,7 @@ func (d *localDetectors) detect(
reuseRun := reuseFull || parent.reuseActive && len(changes) > 0

var tree fs.FS
if anyDetectorRuns(agentSkillRun, briefRun, dependencyUpdateRun, packageRun, repositoryFileRun, workflowRun, managerRun, reuseRun) {
if anyDetectorRuns(agentSkillRun, briefRun, dependencyUpdateRun, packageRun, repositoryFileRun, sensitiveIgnoreRun, workflowRun, managerRun, reuseRun) {
var err error
tree, err = d.repo.TreeFS(commit.Hash)
if err != nil {
Expand All @@ -198,6 +202,7 @@ func (d *localDetectors) detect(
d.runDependencyUpdateConfigs(tree, dependencyUpdateRun, dependencyUpdateFull, &current, stats)
d.runPackages(tree, changes, packageRun, packageFull, &current, stats)
d.runRepositoryFiles(ctx, tree, changes, repositoryFileRun, repositoryFileFull, &current, stats)
d.runSensitiveIgnores(tree, sensitiveIgnoreRun, &current, stats)
d.runWorkflows(tree, changes, workflowRun, workflowFull, &current, stats)
d.runPackageManagers(
tree,
Expand Down Expand Up @@ -237,6 +242,30 @@ func (d *localDetectors) runAgentSkills(
addSignals(current.signals, signals)
}

func (d *localDetectors) runSensitiveIgnores(
tree fs.FS,
run bool,
current *commitState,
stats *scanStats,
) {
detectorStats := stats.detectors[detectorSensitiveIgnore]
if !run {
detectorStats.skipped++
return
}
recordDetectorRun(detectorStats, true)
started := time.Now()
signals, err := sensitiveIgnoreSignals(tree)
detectorStats.duration += time.Since(started)
if err != nil {
current.errors[detectorSensitiveIgnore] = err.Error()
return
}
delete(current.errors, detectorSensitiveIgnore)
removeCheckSignals(current.signals, checkSensitiveIgnore)
addSignals(current.signals, signals)
}

func anyDetectorRuns(runs ...bool) bool {
for _, run := range runs {
if run {
Expand Down
195 changes: 195 additions & 0 deletions sensitive_ignores.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,195 @@
package main

import (
"io/fs"
"path"
"sort"
"strconv"

"github.com/git-pkgs/gitignore"
githistory "github.com/git-pkgs/history"
"github.com/git-pkgs/roles"
)

const (
maxSensitiveIgnoreScopes = 10_000
maxSensitiveIgnoreFileSize = 1 << 20
sensitiveIgnoreCredential = "credential"
sensitiveIgnoreDatabase = "database"
sensitiveIgnoreEnvironment = "environment"
sensitiveIgnorePrivateKey = "private-key"
sensitiveIgnoreSecret = "secret"
sensitiveIgnoreState = "state"
sensitiveIgnoreCoverage = "coverage"
)

type sensitiveIgnoreCandidate struct {
kind string
path string
}

var sensitiveIgnoreCandidates = []sensitiveIgnoreCandidate{
{kind: sensitiveIgnoreCredential, path: ".aws/credentials"},
{kind: sensitiveIgnoreCredential, path: ".docker/config.json"},
{kind: sensitiveIgnoreCredential, path: ".gem/credentials"},
{kind: sensitiveIgnoreCredential, path: ".netrc"},
{kind: sensitiveIgnoreCredential, path: ".npmrc"},
{kind: sensitiveIgnoreCredential, path: ".pypirc"},
{kind: sensitiveIgnoreCredential, path: "application_default_credentials.json"},
{kind: sensitiveIgnoreCredential, path: "credentials.json"},
{kind: sensitiveIgnoreCredential, path: "credentials.yaml"},
{kind: sensitiveIgnoreCredential, path: "credentials.yml"},
{kind: sensitiveIgnoreDatabase, path: "backup.sql"},
{kind: sensitiveIgnoreDatabase, path: "database.dump"},
{kind: sensitiveIgnoreDatabase, path: "database.sql"},
{kind: sensitiveIgnoreEnvironment, path: ".env"},
{kind: sensitiveIgnoreEnvironment, path: ".env.local"},
{kind: sensitiveIgnoreEnvironment, path: ".env.production"},
{kind: sensitiveIgnoreEnvironment, path: ".envrc"},
{kind: sensitiveIgnorePrivateKey, path: "private.key"},
{kind: sensitiveIgnorePrivateKey, path: "private.pem"},
{kind: sensitiveIgnorePrivateKey, path: "id_rsa"},
{kind: sensitiveIgnorePrivateKey, path: "id_ed25519"},
{kind: sensitiveIgnorePrivateKey, path: "config/master.key"},
{kind: sensitiveIgnorePrivateKey, path: "certificate.p12"},
{kind: sensitiveIgnorePrivateKey, path: "certificate.pfx"},
{kind: sensitiveIgnoreSecret, path: "config/secrets.yml"},
{kind: sensitiveIgnoreSecret, path: "secrets.json"},
{kind: sensitiveIgnoreSecret, path: "secrets.yaml"},
{kind: sensitiveIgnoreSecret, path: "secrets.yml"},
{kind: sensitiveIgnoreState, path: "terraform.tfstate"},
{kind: sensitiveIgnoreState, path: "terraform.tfstate.backup"},
{kind: sensitiveIgnoreState, path: ".terraform/terraform.tfstate"},
}

func sensitiveIgnoreSignals(tree fs.FS) ([]Signal, error) {
matcher, scopeNames, signals, truncatedScopes, err := loadSensitiveIgnorePatterns(tree)
if err != nil {
return nil, err
}
signals = append(signals, matchSensitiveIgnoreCandidates(matcher, scopeNames)...)
for _, patternErr := range matcher.Errors() {
signals = append(signals, Signal{
Check: checkSensitiveIgnore,
Kind: "invalid-pattern",
Name: patternErr.Pattern,
Value: patternErr.Message,
Path: patternErr.Source,
Line: uint(patternErr.Line),
})
}
if truncatedScopes {
signals = append(signals, Signal{
Check: checkSensitiveIgnore,
Kind: sensitiveIgnoreCoverage,
Name: "scopes-truncated",
Value: "limit=" + strconv.Itoa(maxSensitiveIgnoreScopes),
})
}
sortSignals(signals)
return deduplicateSignals(signals), nil
}

func loadSensitiveIgnorePatterns(tree fs.FS) (*gitignore.Matcher, []string, []Signal, bool, error) {
matcher := gitignore.New("")
scopes := map[string]bool{"": true}
var signals []Signal
truncatedScopes := false
err := fs.WalkDir(tree, ".", func(filePath string, entry fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if entry.IsDir() {
if filePath != "." && sensitiveIgnorePathExcluded(filePath) {
return fs.SkipDir
}
return nil
}
if entry.Name() != ".gitignore" || sensitiveIgnorePathExcluded(filePath) {
return nil
}
info, err := entry.Info()
if err != nil {
return err
}
if info.Size() > maxSensitiveIgnoreFileSize {
signals = append(signals, Signal{
Check: checkSensitiveIgnore,
Kind: sensitiveIgnoreCoverage,
Name: "oversized-ignore-file",
Value: strconv.FormatInt(info.Size(), 10),
Path: filePath,
})
return nil
}
data, err := fs.ReadFile(tree, filePath)
if err != nil {
return err
}
dir := path.Dir(filePath)
if dir == "." {
dir = ""
}
matcher.AddPatternsWithSource(data, dir, filePath)
if len(scopes) < maxSensitiveIgnoreScopes {
scopes[dir] = true
} else if !scopes[dir] {
truncatedScopes = true
}
return nil
})
if err != nil {
return nil, nil, nil, false, err
}

scopeNames := make([]string, 0, len(scopes))
for dir := range scopes {
scopeNames = append(scopeNames, dir)
}
sort.Strings(scopeNames)
return matcher, scopeNames, signals, truncatedScopes, nil
}

func matchSensitiveIgnoreCandidates(matcher *gitignore.Matcher, scopeNames []string) []Signal {
seen := make(map[string]bool)
var signals []Signal
for _, dir := range scopeNames {
for _, candidate := range sensitiveIgnoreCandidates {
target := path.Join(dir, candidate.path)
result := matcher.MatchDetail(target)
if !result.Ignored {
continue
}
key := result.Source + "\x00" + strconv.Itoa(result.Line) + "\x00" + candidate.kind
if seen[key] {
continue
}
seen[key] = true
signals = append(signals, Signal{
Check: checkSensitiveIgnore,
Kind: candidate.kind,
Name: target,
Value: result.Pattern,
Path: result.Source,
Line: uint(result.Line),
})
}
}
return signals
}

func sensitiveIgnorePathExcluded(filePath string) bool {
labels, err := roles.Match(filePath)
return err == nil && (labels.BundledOrDerived() || labels.Has(roles.Test) ||
labels.Has(roles.Fixture) || labels.Has(roles.Example) || labels.Has(roles.Cache) ||
labels.Has(roles.BuildOutput))
}

func sensitiveIgnoresAffected(changes []githistory.Change) bool {
for _, change := range changes {
if path.Base(change.Path) == ".gitignore" && !sensitiveIgnorePathExcluded(change.Path) {
return true
}
}
return false
}
Loading
Loading