Skip to content

security-testing.md

Latest commit

 

History

History
172 lines (136 loc) · 8.14 KB

security-testing.md

File metadata and controls

172 lines (136 loc) · 8.14 KB
Raw
type Test Strategy
title Security Testing
description Fail-point injection and layered security regression testing strategy.
tags
bashkit
testing
security

Security Testing with Fail Points

Status

Implemented

Overview

Bashkit uses fail-rs for fault injection security testing of error handling paths and resource limit enforcement. Fail points are disabled by default, zero runtime overhead when disabled.

cargo test --features failpoints security_ -- --test-threads=1

Available Fail Points

Resource Limits (limits.rs)

Fail Point Actions Security Test Purpose
limits::tick_command skip_increment, force_overflow, corrupt_high Test command limit bypass resistance
limits::tick_loop skip_check, reset_counter Test loop limit bypass resistance
limits::push_function skip_check, corrupt_depth Test recursion limit bypass resistance

Filesystem (fs/memory.rs)

Fail Point Actions Security Test Purpose
fs::read_file io_error, permission_denied, corrupt_data Test read failure handling
fs::write_file io_error, disk_full, permission_denied, partial_write Test write failure handling

Interpreter (interpreter/mod.rs)

Fail Point Actions Security Test Purpose
interp::execute_command panic, error, exit_nonzero Test command execution failure handling

yq atomic replacement (builtins/yq.rs)

Fail Point Actions Security Test Purpose
yq::temp_allocate exhausted Source preservation when no sibling temporary can be allocated
yq::temp_chmod error Temporary cleanup and source preservation on mode-copy failure
yq::temp_rename error Temporary cleanup and source preservation before atomic replacement
sed::temp_allocate exhausted sed -i source preservation when no sibling temporary can be allocated
sed::temp_chmod error sed -i temporary cleanup and source preservation on mode-copy failure
sed::temp_rename error sed -i temporary cleanup and source preservation before atomic replacement

The existing fs::write_file actions cover yq temporary-write failures, including the partial-write class. security_yq_inplace_* asserts original bytes and absence of .bashkit-yq-* files after every injected stage.

Usage

In tests: fail::cfg("limits::tick_command", "return(skip_increment)") before, fail::cfg("name", "off") after. Action syntax (return/panic/sleep/pause, probability 10%, count 5*): see fail crate docs.

Security Test Categories

  1. Resource limit bypass: counter corruption, check skipping, overflow/underflow
  2. Filesystem failure: I/O errors, permission denied, disk full, partial writes, data corruption
  3. Interpreter failure: execution errors, panic recovery, unexpected exit codes

od Width Regression Oracle

od_resource_tests compares valid-width layouts with GNU od when available (required in Linux CI). Zero and type-misaligned widths are asserted directly against Bashkit's existing type-size fallback. Some GNU versions abort on -w0, so an invalid-width GNU subprocess cannot serve as a compatibility oracle; see the GNU report. The enormous-width tests invoke only the fixed builtin, after safe bounded regressions established the allocation path.

Function Definition Metadata

function_metadata_security_tests uses small function budgets and at most a few KiB of padded source operands. It covers per-function and aggregate filename charges, exact byte boundaries, failed redefinition atomicity, replacement and unset refunds, filename rollback in subshells/substitutions/pipelines/background jobs/executable VFS scripts, and removal of old metadata on shell-state restore. The interpreter unit test function_metadata_shares_storage_and_drops_with_functions proves definitions from one source share their filename allocation and that rollback/unset leave no orphan entries or retained filename allocation. No host-exhaustion probe is needed. The real-Bash comparison sources both the original and post-unset replacement; Bash versions differ in the filename they assign to a definition inside -c.

Arithmetic Associative-Key Budgets

arithmetic_resource_tests exercises the public Bash::exec path with keys bounded to a few KiB and small live-memory/work budgets. It covers repeated $name and ${name} through ((...)), quoted let, and $((...)), reads, assignment/compound assignment/increment/decrement, recursive key scanning, namerefs, 64 KiB key boundaries, and session recovery after rejection. Finite valid-key cases compare stdout and status with Bash >= 4 (associative arrays are unavailable in macOS's stock Bash 3.2; Linux CI supplies Bash 5.x). The interpreter unit test checks the exact UTF-8 append boundary and a cancellation checkpoint without growing a buffer after rejection. Resource probes run in externally supervised subprocesses; full memory-exhaustion payloads are unnecessary.

Adding New Fail Points

Add fail_point!("module::function", |action| ...) under #[cfg(feature = "failpoints")] at the critical location, document it in this spec, and add tests in tests/security_failpoint_tests.rs.

Best Practices

  1. Always clean up: Call fail::cfg("name", "off") after tests.
  2. Single-threaded tests: Use --test-threads=1 for fail point tests due to global state.
  3. Document actions: List all supported actions in code comments and this spec.
  4. Test both paths: Test that fail points affect behavior AND that normal operation works without them.

Filesystem Certification

The backend-neutral helper in crates/bashkit/tests/support/filesystem_security_conformance.rs certifies binary/null content, unsafe pathname rejection, root-clamped canonical identity, and normalized public error kinds. The integration and RealFs feature suites run the same helper against every production storage adapter. Wrapper cases cover symlink identity, read-only policy, mount boundaries, transactional failures, quota retention, and tar preflight. security_fs_failed_writes_are_atomic runs all fs::write_file failpoint actions and proves prior bytes and usage survive.

JavaScript Security Tests

The JavaScript/TypeScript bindings have a dedicated security test suite at crates/bashkit-js/__test__/security.spec.ts (run: cd crates/bashkit-js && pnpm test). White-box and black-box scenarios across 18 categories:

  1. Resource limit enforcement (TM-DOS)
  2. Output truncation (TM-DOS)
  3. Sandbox escape prevention (TM-ESC)
  4. VFS security, path traversal, file count, nesting, filename limits (TM-DOS, TM-INJ)
  5. Instance isolation (TM-ISO)
  6. Error message safety (TM-INT)
  7. TypeScript wrapper injection prevention (TM-INJ)
  8. Adversarial script inputs, null bytes, deep nesting, expansion bombs
  9. Unicode & encoding attacks (TM-UNI)
  10. Injection via constructor options (TM-INJ)
  11. Concurrency & cancellation (TM-DOS)
  12. Async API security
  13. BashTool metadata safety
  14. Bash feature abuse, traps, special variables, /dev/tcp
  15. Mounted files security
  16. Rapid instance creation/destruction
  17. Edge case inputs
  18. Async factory security

Related Files

  • crates/bashkit/tests/security_failpoint_tests.rs - Fail-point security tests
  • crates/bashkit/tests/threat_model_tests.rs - Threat model tests (51 tests)
  • crates/bashkit/tests/builtin_error_security_tests.rs - Builtin error security tests (39 tests, includes TM-INT-003)
  • crates/bashkit-js/__test__/security.spec.ts - JavaScript security tests (90+ tests, 18 categories)
  • crates/bashkit/src/limits.rs - Resource limit fail points
  • crates/bashkit/src/fs/memory.rs - Filesystem fail points
  • crates/bashkit/src/interpreter/mod.rs - Interpreter fail points, panic catching
  • crates/bashkit/src/builtins/system.rs - Hardcoded system builtins
  • Threat Model - Threat model specification