Testo command line tool

.github/workflows/qa.yml

@@ -49,7 +49,7 @@ jobs:
       - name: Run unit tests without race detector
         if: runner.os == 'Windows'
         run: |
-          go test -v -cover -coverpkg=./... ./...
+          go test -v -cover -coverpkg=.,./testo...,./internal/... ./...
 
       - name: Test examples output
         run: go test -v -tags e2e -count=1 ./examples_test.go

.golangci.yaml

@@ -54,6 +54,10 @@ linters:
           - godot
           - funlen
           - gosec
+      # printing to stdout is the CLI's output
+      - path: cmd/
+        linters:
+          - forbidigo
       - path: internal/
         linters:
           - revive
@@ -79,6 +83,9 @@ linters:
     errcheck:
       exclude-functions:
         - "(*os.File).Close"
+        - "fmt.Fprint"
+        - "fmt.Fprintf"
+        - "fmt.Fprintln"
     gocritic:
       enable-all: true
       disabled-checks:

CHANGELOG.md

@@ -5,6 +5,11 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## Unreleased
+
+## Added
+
+- Auxiliary command line tool for Testo featuring linter, suites explorer and runner.
 ## [1.8.0] - 2026-08-30
 
 ### Added

CLAUDE.md

@@ -0,0 +1,63 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Overview
+
+Testo is a modular testing framework for Go built on top of `testing.T`, with suites, parametrized tests, lifecycle hooks, and an extensive plugin system. It is a zero-dependency library (module `github.com/ozontech/testo`, Go 1.24+).
+
+## Commands
+
+```bash
+make                  # full check: generate, fmt, lint, test, test-examples, tidy
+make test             # go test -race -shuffle=on ./...
+make test-examples    # verify examples output matches golden files (go test -tags e2e ./examples_test.go)
+make fmt              # golangci-lint fmt
+make lint             # golangci-lint run --tests=false
+make coverage         # test coverage report
+make install          # install the ./cmd/testo CLI
+
+# Run a single test
+go test -race -run 'TestName' .
+
+# Regenerate golden output for examples (after changing examples or output format)
+make update-examples-output
+```
+
+Examples under `examples/` have expected output stored in `examples/*/output.txt`; `examples_test.go` (build tag `e2e`) runs each example package and diffs against those files. If you change log/error formatting or the examples, run `make update-examples-output`.
+
+## Architecture
+
+### Package layout
+
+- **Root package `testo`** — the framework core. Key files:
+  - `t.go` — `T` (the `testing.T` wrapper), `TestingT`, `CommonT` interfaces. Users embed `*testo.T` plus plugin pointers into their own `T` struct.
+  - `construct.go` — `construct[T]`: reflective construction of user `T` types, filling embedded plugin fields. This is the core DI mechanism of the framework; plugin instances are deduplicated so all pointers to the same plugin type share one instance.
+  - `runner.go` — `RunSuite`, `RunTest`, `Test`: entry points. A hidden wrapper test named `testo!` (contains `!` so it can't collide with Go identifiers) wraps suite tests so hooks work with parallel tests.
+  - `collector.go` — reflection over suite types: collects `TestXxx` methods and `CasesXxx` parametrization providers, validates naming.
+  - `suite.go` — `Suite[T]` base type and the suite hook interface (`BeforeAll`/`BeforeEach`/`AfterEach`/`AfterAll`).
+  - `plugin.go` — merging of multiple plugin `Spec`s (plans sorted by priority, hooks/overrides chained).
+  - `annotations.go` — `For`/`ForEach`: attach static options to tests via a global registry keyed by function identity.
+  - `options.go`, `flag.go` — plugin options and `-testo.*` command-line flags (e.g. `-testo.strict`, env `TESTO_STRICT`).
+- **`testoplugin`** — public plugin API: `Spec` = `Plan` (filter/reorder/duplicate tests before run) + `Hooks` (BeforeAll/Each/SubEach etc.) + `Overrides` (replace built-in `T` methods like `Log`, `Error`). Plugins implement a `Plugin(parent, options)` method; innermost (deepest embedded) plugins are initialized first.
+- **`testoreflect`** — public read-only reflection API (`TestInfo`, `SuiteInfo`) given to plugins.
+- **`testocache`** — key-value cache persistent between test runs, with namespaces.
+- **`internal/`** — helpers: `parse` (method-name parsing), `testnamer`, `reflectutil`, `pragma` (`DoNotImplement` sealing), `stack`, `syncutil`, `env`.
+- **`cmd/testo`** — optional auxiliary CLI (linter, suites explorer, runner: `testo lint`, `testo run`, `testo suites`, `testo tags`). Experimental.
+- **`vscode-extension/`** — VS Code extension (npm/vsce, own Makefile).
+
+### Lifecycle (see docs/technical-overview.md for full detail)
+
+`RunSuite` runs a root test named after the suite: collects and validates tests → initializes plugins (via `.Plugin(parent, options)`) → `BeforeAll` (plugin hooks, then suite hook) → collects parametrized cases from `CasesXxx` → applies plugin plans → runs each test under the `testo!` wrapper (per-test plugin construction, `BeforeEach`/`AfterEach`; sub-tests via `testo.Run` get `BeforeEachSub`/`AfterEachSub`) → `AfterAll`. Panics in tests and per-test hooks are caught; panics in `BeforeAll`/`AfterAll` are not.
+
+### Plugin DI model
+
+Users compose their `T` as a struct embedding `*testo.T` and plugin pointers. Plugins can depend on other plugins by embedding them; Testo resolves the graph so every reference to a given plugin type points at the same instance. `construct.go` is where this happens — changes there affect the entire framework.
+
+## Conventions
+
+- Zero dependencies in the main module — do not add third-party imports to library code.
+- Public API stability matters: interfaces are sealed with `internal/pragma.DoNotImplement` private methods; follow that pattern for new public interfaces.
+- Every user-visible change must be described in `CHANGELOG.md` under the "Unreleased" section (Keep a Changelog format, SemVer).
+- Lint runs on non-test code only (`--tests=false`); format with `golangci-lint fmt`, not plain gofmt.
+- Per CONTRIBUTING.md, AI-assisted work must be disclosed in PRs (tool + extent).

Makefile

@@ -31,15 +31,15 @@ doc:
 
 # get test coverage
 coverage:
-	go test -coverprofile=coverage.out -coverpkg=./... ./...
+	go test -coverprofile=coverage.out -coverpkg=.,./testo...,./internal/... ./...
 	go tool cover -func coverage.out
 
 # visualize test coverage
 coverage-html: coverage
 	go tool cover -html coverage.out
 
 install:
-	go install ./cmd/testo
+	go install ./cmd/...
 
 update-examples-output:
 	./update-examples-output.sh

README.md

@@ -32,6 +32,7 @@ and [Allure report plugin](https://github.com/ozontech/testo-allure).
 - [Test reflection](https://pkg.go.dev/github.com/ozontech/testo/testoreflect) - deeply inspect a test's meta-information.
 - [Caching](./docs/how-to.md#how-to-use-persistent-cache) - key-value storage persistent between test runs.
 - [Zero dependencies](./go.mod).
+- [Auxiliary command line tool](./cmd/testo) - linter, suites explorer and runner.
 
 ## Why Testo
 
@@ -132,6 +133,25 @@ The extension adds run/debug buttons for individual suite tests, plus snippets.
 
 ![VSCode extension screenshot showing codelens buttons for running and debugging a test](./vscode-extension/example.png)
 
+## Testo command line tool
+
+Testo has an auxiliary command line tool featuring linter, suites explorer and runner.
+
+[See more here](./cmd/testo).
+
+Example:
+
+```bash
+go install github.com/ozontech/testo/cmd/testo
+
+testo lint ./...
+testo run mypkg/Functional.TestFoo
+testo suites -f "{{ .Package }}/{{ .Suite }}" | fzf
+```
+
+> [!NOTE]
+> This command line tool is _completely optional_ and _is not required_ to run tests.
+
 ## Minimum supported Go version
 
 Testo guarantees to support at least **3 latest major** [Go releases](https://go.dev/doc/devel/release).

REVIEW.md

@@ -0,0 +1,113 @@
+# cmd/testo review
+
+Code review of the auxiliary CLI (`cmd/testo`): bugs found, how they were
+reproduced, and improvement vectors. All bugs listed below are fixed.
+
+## Confirmed bugs
+
+### 1. `testo run` could silently run zero tests (false green)
+
+`loadRunners` (`internal/loader/runner.go`) tracked the enclosing test function
+by remembering the last `FuncDecl` whose name passes `parse.IsTest` — without
+checking `Recv == nil` and without resetting on non-test functions. Suite
+methods like `func (Suite) TestFoo(t T)` are `FuncDecl`s too, so a `RunSuite`
+call inside a plain helper was attributed to the previous method or test
+declared in the file.
+
+Reproduced: with a helper function calling `RunSuite`, the tool generated
+`go test -run '^TestFoo$/^Suite$'` (the *suite method* name); `go test`
+reported "no tests to run, PASS" — exit 0.
+
+Fix: only function declarations without a receiver update the current test
+name, and it is reset on non-test functions. A suite whose `RunSuite` caller
+cannot be found now produces a warning instead of a silently wrong `-run`
+pattern.
+
+### 2. `-- [test flags]` was broken without a pattern
+
+`parseFlagSet` (`internal/cli/cli.go`) collected positionals and re-parsed
+them, losing the `--` separator. `testo run -n -- -count=1` died with
+`flag provided but not defined: -count`.
+
+Fix: `--` is detected and everything after it is kept as-is; the final
+re-parse is performed behind a synthetic `--` terminator so flag-like
+positionals are never parsed as the command's own flags.
+
+### 3. One bad package made `suites`/`run` unusable for the whole repo
+
+`loader.Load` deliberately returns `(suites, *LoadError)`, but `cmdsuites`
+and `cmdrun` treated any error as fatal. Reproduced on this repository:
+`testo suites ./...` printed nothing because `examples/06_errors`
+(intentionally malformed) produces diagnostics.
+
+Fix: both commands report diagnostics as warnings on stderr and proceed with
+the suites that did load. Only `lint` treats diagnostics as failure.
+
+### 4. Suites were collected twice via `go list -test` package variants
+
+`packageslite` skipped only the `pkg.test` binary; both `pkg` and
+`pkg [pkg.test]` were type-checked and scanned, so a suite declared in a
+non-test file appeared twice. The default `suites` format hid it via output
+dedup, but `-f '{{ .Package.Path }}/...'` printed both, and `run` did the
+runner scan twice.
+
+Fix: when the test variant of a package is present the base package is
+dropped (the variant is a strict superset of its files), and the variant's
+import path is normalized (the ` [pkg.test]` suffix stripped) after type
+checking. Runner matching compares suites by normalized package path + type
+name rather than type identity, so suites and their callers agree across
+variants.
+
+### 5. Diagnostics printed out of source order
+
+`Load` sorted diagnostics with unstable `slices.SortFunc` comparing file
+names only. Reproduced: lines 23, 28, 35, **16**, 47 reported in that order
+for a single file.
+
+Fix: diagnostics sort by (file name, position).
+
+## Minor bugs
+
+- **Dead/inconsistent code in `asSuite`** (`internal/loader/loader.go`):
+  `if invalidParams { continue }` sat *after* the append as the last statement
+  of the loop body — a no-op. A 1-parameter test with the wrong `T` was still
+  listed while a 2-parameter test with the wrong `T` was skipped. Fixed:
+  a test with any signature diagnostic is not listed.
+- **Auto-derived tags included implicit tags.** `BuildTags` collects every
+  tag in the module, so a `//go:build windows` file added `windows` to
+  `-tags`, pulling mutually exclusive platform files into type checking.
+  `Load` also silently swallowed the `BuildTags` error. Fixed: implicit tags
+  (GOOS/GOARCH values, `go1.*`, `cgo`, `race`, …) are excluded from
+  derivation (but still shown by `testo tags`), and the error propagates.
+- **Panic risk in `asT`** (`internal/loader/loader.go`): four chained
+  unchecked type assertions on the internal layout of `testo.Suite[T]`
+  (`struct{ _ [0]*T }`). Version skew between the installed CLI and the
+  project's testo turned into a panic instead of a non-match. Fixed with
+  checked assertions.
+- **`goList` decoded `Error` but never checked it** (uses `go list -e`), so
+  broken packages surfaced later as an opaque `package %q is not complete`.
+  Fixed: a directly-matched package with files and a list error fails with
+  that error; matched directories excluded by build constraints are skipped.
+- **Empty `-tags ''` argument** was passed to `go test` when no tags were
+  derived, and empty strings survived tag splitting. Fixed.
+
+## Improvement vectors (applied)
+
+- Usage text printed raw `os.Args[0]` — under `go run` that is the full
+  go-build cache path. Now uses `filepath.Base`.
+- `loadRunners` cached runners for every suite type *except* the queried
+  one, and returned map keys in nondeterministic order (also affecting the
+  package argument order of the built `go test` command, i.e. `-n` output).
+  Now scans once, caches everything, and output is sorted.
+- `testo suites` printed nothing for a suite with zero tests. Now the suite
+  line is printed once (with an empty `.Test`).
+- Added regression tests for CLI flag parsing (`internal/cli`).
+
+## Improvement vectors (not applied)
+
+- Golden end-to-end tests for `lint`/`suites`/`run -n` output over
+  `examples/` would lock in exit codes and diagnostic ordering cheaply,
+  matching the framework's own golden-file approach (`examples_test.go`).
+- The `suite.test` regex pattern of `testo run` splits on the *first* `.`,
+  so a suite regex containing `.` (e.g. `Test.+` as a suite-only pattern)
+  cannot be expressed.

cmd/testo/README.md

@@ -0,0 +1,31 @@
+# Testo CTL
+
+Auxiliary command line tool for Testo featuring linter, suites explorer and runner.
+
+> [!WARNING]
+> This tool is experimental, handle with care;
+> may change without warning
+
+## Install
+
+```bash
+go install github.com/ozontech/testo/cmd/testo
+```
+
+## Usage
+
+Run `testo -h` to see available commands:
+
+```txt
+Usage:
+  testo [command]
+
+Available Commands:
+  lint       Run testo linter
+  run        Run testo suites
+  suites     Show testo suites
+  tags       Show project build tags
+  version    Show testo version
+```
+
+Run `testo [command] -h` to show help for the given command.