diff --git a/crates/bashkit/docs/threat-model.md b/crates/bashkit/docs/threat-model.md index 996e5ff5a..2388773f2 100644 --- a/crates/bashkit/docs/threat-model.md +++ b/crates/bashkit/docs/threat-model.md @@ -206,7 +206,7 @@ handling, including discarded output. Stdout cached text shares that budget | jq control normalization amplification (TM-DOS-100) | Literal controls expand sixfold as `\u00XX` | Charge single-pass work and lease live bytes before allocation growth | MITIGATED | | yq structured-data amplification (TM-DOS-101) | Deep/multi-document YAML or JSON, runaway filters, expanded output | Parser depth and 4096-document caps, aggregate budgets, shared jaq work/deadline/output limits, final render cap | MITIGATED | | Archive decoder pre-allocation (TM-DOS-102) | Compressed output grows before memory checks | Validate and charge each decoder chunk before reserve/copy | MITIGATED | -| Post-allocation charging (TM-DOS-103) | Archive/compression buffers or tar file-list names grow before their live-byte check | Owning budget-aware string/vector/byte builders charge before reserve, roll back errors, and release on drop; tar file lists also lease raw reads and charge per-line work | MITIGATED | +| Post-allocation charging (TM-DOS-103) | Archive/compression buffers, tar file-list names, or standard-stream operands grow before their live-byte check | Owning budget-aware string/vector/byte builders charge before reserve, roll back errors, and release on drop; tar file lists also lease raw reads and charge per-line work. Standard-stream captures enforce stdout/stderr ceilings before write/append and combined collection, retain byte/text leases through redirects/hooks, and check work/cancellation/deadlines on every write. Wrapper-owned stdin/copy storage is also admitted | MITIGATED | | Pipeline stderr aggregation (TM-DOS-104) | Many stderr-producing pipeline stages accumulate past the output cap | Stop appending at `max_stderr_bytes` and propagate the truncation flag to the result | MITIGATED | | Zip creation pre-allocation (TM-DOS-106) | Recursive inputs and encoded output allocate before live-memory checks | Collect paths only, lease file sizes before VFS reads, and encode through a budget-aware byte buffer | MITIGATED | | Analysis command-name validation (TM-DOS-107) | A huge comment before thousands of commands makes validation rescan the source per command | Index source character positions once, then binary-search each name | MITIGATED | diff --git a/crates/bashkit/src/fs/memory.rs b/crates/bashkit/src/fs/memory.rs index aff5e93b8..0ffd38e12 100644 --- a/crates/bashkit/src/fs/memory.rs +++ b/crates/bashkit/src/fs/memory.rs @@ -55,7 +55,8 @@ use super::vfs_join; use fail::fail_point; /// Bytes returned by one read of `/dev/zero` (TM-DOS-003: never unbounded). -const DEV_ZERO_READ_BYTES: usize = 1 << 20; +pub(crate) const DEV_ZERO_READ_BYTES: usize = 1 << 20; +pub(crate) const DEV_RANDOM_READ_BYTES: usize = 8192; /// In-memory filesystem implementation. /// @@ -534,8 +535,7 @@ impl InMemoryFs { /// Returns exactly 8192 bytes to prevent unbounded reads while /// supporting common patterns like `od -N8 -tx1 /dev/urandom`. fn generate_random_bytes() -> Result> { - const SIZE: usize = 8192; - let mut buf = vec![0; SIZE]; + let mut buf = vec![0; DEV_RANDOM_READ_BYTES]; // Important decision: use the OS CSPRNG backing getrandom, not // non-CSPRNG hasher seeds, because scripts commonly use these paths // for tokens, salts, nonces, and keys. diff --git a/crates/bashkit/src/fs/mod.rs b/crates/bashkit/src/fs/mod.rs index 5879a71eb..892acab5e 100644 --- a/crates/bashkit/src/fs/mod.rs +++ b/crates/bashkit/src/fs/mod.rs @@ -648,7 +648,7 @@ pub async fn verify_filesystem_requirements(fs: &dyn FileSystem) -> Result<()> { Ok(()) } -/// Resolve every symlink in `path` (realpath), with an ELOOP hop cap. +/// Resolve every symlink to a POSIX VFS path on every host, with an ELOOP hop cap. pub(crate) async fn canonicalize(fs: &dyn FileSystem, path: &Path) -> crate::Result { let mut rest: std::collections::VecDeque = path .components() @@ -665,7 +665,7 @@ pub(crate) async fn canonicalize(fs: &dyn FileSystem, path: &Path) -> crate::Res out.pop(); continue; } - let candidate = out.join(&comp); + let candidate = vfs_join(&out, &comp); match fs.lstat(&candidate).await { Ok(m) if m.file_type.is_symlink() => { hops += 1; @@ -673,7 +673,7 @@ pub(crate) async fn canonicalize(fs: &dyn FileSystem, path: &Path) -> crate::Res return Err(std::io::Error::other("Too many levels of symbolic links").into()); } let target = fs.read_link(&candidate).await?; - if target.is_absolute() { + if target.has_root() { out = PathBuf::from("/"); } let mut parts: Vec = target diff --git a/crates/bashkit/src/fs/std_streams.rs b/crates/bashkit/src/fs/std_streams.rs index 4e61f73a7..8a9cfe77e 100644 --- a/crates/bashkit/src/fs/std_streams.rs +++ b/crates/bashkit/src/fs/std_streams.rs @@ -9,6 +9,13 @@ //! are captured and appended to the builtin's stdout/stderr afterwards. //! Only fds 0-2: `/dev/fd/63` etc. are the shell's process substitutions, //! answered by `ProcSubFs` underneath. +//! +//! Decision: this interception bypasses filesystem quotas. Capture and stdin +//! copies therefore own request-budgeted capacity; capture ceilings apply +//! before writes and collection, even when an outer redirect discards output. +//! Collected byte/text leases survive redirect and hook consumption. Returned +//! stdin Vecs cannot carry leases through FileSystem, so the wrapper retains +//! their conservative charges until its last owner drops. use std::path::{Path, PathBuf}; use std::sync::{Arc, Mutex}; @@ -17,19 +24,86 @@ use async_trait::async_trait; use super::{DirEntry, FileSystem, FileSystemExt, FileType, FsLimits, FsUsage, Metadata}; use crate::error::Result; +use crate::limits::{ + BudgetedBytes, BudgetedVec, ExecutionBudget, ExecutionBudgetLease, ExecutionLimits, + LimitExceeded, +}; +use crate::stream::StreamStorage; use crate::time_compat::{SystemTime, UNIX_EPOCH}; +use crate::{ExecResult, StreamData}; /// Captured writes to `/dev/stdout` and `/dev/stderr`. -#[derive(Default)] pub(crate) struct StdCapture { - pub(crate) stdout: Vec, - pub(crate) stderr: Vec, + stdout: BudgetedBytes, + stderr: BudgetedBytes, +} + +impl StdCapture { + fn new(budget: &ExecutionBudget) -> Result { + Ok(Self { + stdout: BudgetedBytes::new(Some(budget))?, + stderr: BudgetedBytes::new(Some(budget))?, + }) + } + + pub(crate) fn collect( + &mut self, + result: &mut ExecResult, + budget: &ExecutionBudget, + limits: &ExecutionLimits, + ) -> Result<[StreamStorage; 2]> { + let capture = std::mem::replace(self, Self::new(budget)?); + Ok([ + collect_stream( + &mut result.stdout, + capture.stdout, + budget, + limits.max_stdout_bytes, + )?, + collect_stream( + &mut result.stderr, + capture.stderr, + budget, + limits.max_stderr_bytes, + )?, + ]) + } +} + +fn collect_stream( + output: &mut StreamData, + capture: BudgetedBytes, + budget: &ExecutionBudget, + limit: usize, +) -> Result { + if capture.is_empty() { + return Ok([None, None]); + } + let bytes = if output.is_empty() { + capture + } else { + // Keep the capture lease alive while constructing the admitted copy. + budget.consume_work(1 + output.len().div_ceil(1024) as u64)?; + let mut bytes = BudgetedBytes::new(Some(budget))?; + bytes.try_extend_from_slice_with_limit(output.as_bytes(), limit)?; + bytes.try_extend_from_slice_with_limit(&capture, limit)?; + bytes + }; + let (stream, storage) = StreamData::from_budgeted_bytes(bytes, budget)?; + *output = stream; + Ok(storage) } +type StdCaptureHandle = Arc>; + pub(crate) struct StdStreamsFs { inner: Arc, - stdin: Vec, - capture: Arc>, + stdin: BudgetedBytes, + read_storage: Mutex>, + capture: StdCaptureHandle, + budget: ExecutionBudget, + stdout_limit: usize, + stderr_limit: usize, } /// Which standard stream a path names, if any. @@ -52,22 +126,36 @@ impl StdStreamsFs { pub(crate) fn wrap( inner: Arc, stdin: &[u8], - ) -> (Arc, Arc>) { - let capture = Arc::new(Mutex::new(StdCapture::default())); + budget: &ExecutionBudget, + limits: &ExecutionLimits, + ) -> Result<(Arc, StdCaptureHandle)> { + budget.consume_work(1 + stdin.len().div_ceil(1024) as u64)?; + let mut input = BudgetedBytes::new(Some(budget))?; + input.try_extend_from_slice(stdin)?; + let capture = Arc::new(Mutex::new(StdCapture::new(budget)?)); let fs = Arc::new(Self { inner, - stdin: stdin.to_vec(), + stdin: input, + read_storage: Mutex::new(BudgetedVec::new(Some(budget))?), capture: Arc::clone(&capture), + budget: budget.clone(), + stdout_limit: limits.max_stdout_bytes, + stderr_limit: limits.max_stderr_bytes, }); - (fs, capture) + Ok((fs, capture)) } - fn write_stream(&self, fd: u8, content: &[u8]) { + // THREAT[TM-DOS-103]: reserve retained capacity before any stream growth. + fn write_stream(&self, fd: u8, content: &[u8]) -> Result<()> { + self.budget + .consume_work(1 + content.len().div_ceil(1024) as u64)?; let mut capture = self.capture.lock().unwrap_or_else(|e| e.into_inner()); - match fd { - 1 => capture.stdout.extend_from_slice(content), - _ => capture.stderr.extend_from_slice(content), - } + let (buffer, limit) = match fd { + 1 => (&mut capture.stdout, self.stdout_limit), + _ => (&mut capture.stderr, self.stderr_limit), + }; + buffer.try_extend_from_slice_with_limit(content, limit)?; + Ok(()) } fn stream_metadata(&self) -> Metadata { @@ -101,25 +189,33 @@ impl FileSystemExt for StdStreamsFs { impl FileSystem for StdStreamsFs { async fn read_file(&self, path: &Path) -> Result> { match std_stream_fd(path) { - Some(0) => Ok(self.stdin.clone()), + Some(0) => { + self.budget.consume_input(self.stdin.len())?; + self.budget + .consume_work(1 + self.stdin.len().div_ceil(1024) as u64)?; + let mut data = BudgetedBytes::new(Some(&self.budget))?; + data.try_extend_from_slice(&self.stdin)?; + let (data, lease) = data.into_parts(); + if let Some(lease) = lease { + self.read_storage + .lock() + .unwrap_or_else(|e| e.into_inner()) + .try_push(lease)?; + } + Ok(data) + } _ => self.inner.read_file(path).await, } } async fn write_file(&self, path: &Path, content: &[u8]) -> Result<()> { match std_stream_fd(path) { - Some(fd @ (1 | 2)) => { - self.write_stream(fd, content); - Ok(()) - } + Some(fd @ (1 | 2)) => self.write_stream(fd, content), _ => self.inner.write_file(path, content).await, } } async fn append_file(&self, path: &Path, content: &[u8]) -> Result<()> { match std_stream_fd(path) { - Some(fd @ (1 | 2)) => { - self.write_stream(fd, content); - Ok(()) - } + Some(fd @ (1 | 2)) => self.write_stream(fd, content), _ => self.inner.append_file(path, content).await, } } @@ -156,8 +252,47 @@ impl FileSystem for StdStreamsFs { async fn copy(&self, from: &Path, to: &Path) -> Result<()> { match (std_stream_fd(from), std_stream_fd(to)) { (None, None) => self.inner.copy(from, to).await, + (Some(0), _) => { + // Borrow the already admitted stdin instead of making a copy. + self.budget + .consume_work(1 + self.stdin.len().div_ceil(1024) as u64)?; + self.write_file(to, &self.stdin).await + } _ => { - let data = self.read_file(from).await?; + // The wrapper owns this temporary read, so admit the backend's + // reported size before loading it, including discarded sinks. + let source = self + .budget + .run(super::canonicalize(&*self.inner, from)) + .await??; + let metadata = self.budget.run(self.inner.stat(&source)).await??; + // System devices report size zero, but materialize bounded data. + // Resolve aliases and use the backend's canonical read ceilings. + let reported_size = usize::try_from(metadata.size) + .map_err(|_| LimitExceeded::Memory("stream copy size overflow".into()))?; + let device_size = match source.to_str() { + Some("/dev/zero") => super::memory::DEV_ZERO_READ_BYTES, + Some("/dev/random" | "/dev/urandom") => super::memory::DEV_RANDOM_READ_BYTES, + _ => 0, + }; + let size = reported_size.max(device_size); + let limit = match std_stream_fd(to) { + Some(1) => self.stdout_limit, + Some(2) => self.stderr_limit, + _ => usize::MAX, + }; + if size > limit { + return Err( + LimitExceeded::Memory("stream copy size limit exceeded".into()).into(), + ); + } + self.budget.consume_input(size)?; + self.budget.consume_work(1 + size.div_ceil(1024) as u64)?; + let _read_storage = self.budget.lease_bytes(size)?; + let data = self.budget.run(self.inner.read_file(&source)).await??; + let _extra_storage = self + .budget + .lease_bytes(data.capacity().saturating_sub(size))?; self.write_file(to, &data).await } } @@ -185,10 +320,16 @@ mod tests { use super::*; use crate::fs::InMemoryFs; + fn test_budget(limits: &ExecutionLimits) -> ExecutionBudget { + ExecutionBudget::new(limits, Arc::new(std::sync::atomic::AtomicBool::new(false))) + } + #[tokio::test] async fn reads_stdin_and_captures_writes() { let inner: Arc = Arc::new(InMemoryFs::new()); - let (fs, cap) = StdStreamsFs::wrap(Arc::clone(&inner), b"in\n"); + let limits = ExecutionLimits::new(); + let budget = test_budget(&limits); + let (fs, cap) = StdStreamsFs::wrap(Arc::clone(&inner), b"in\n", &budget, &limits).unwrap(); assert_eq!( fs.read_file(Path::new("/dev/stdin")).await.unwrap(), b"in\n" @@ -199,7 +340,7 @@ mod tests { assert!(fs.exists(Path::new("/dev/stderr")).await.unwrap()); let (out, err) = { let cap = cap.lock().unwrap(); - (cap.stdout.clone(), cap.stderr.clone()) + (cap.stdout.to_vec(), cap.stderr.to_vec()) }; assert_eq!((out.as_slice(), err.as_slice()), (&b"o"[..], &b"e"[..])); // Nothing reached the real VFS. @@ -209,10 +350,237 @@ mod tests { #[tokio::test] async fn other_fds_pass_through() { let inner: Arc = Arc::new(InMemoryFs::new()); - let (fs, _) = StdStreamsFs::wrap(Arc::clone(&inner), b""); + let limits = ExecutionLimits::new(); + let budget = test_budget(&limits); + let (fs, _) = StdStreamsFs::wrap(Arc::clone(&inner), b"", &budget, &limits).unwrap(); assert_eq!(std_stream_fd(Path::new("/dev/fd/63")), None); assert_eq!(std_stream_fd(Path::new("/dev/../dev/stdin")), Some(0)); fs.write_file(Path::new("/tmp/x"), b"1").await.unwrap(); assert_eq!(inner.read_file(Path::new("/tmp/x")).await.unwrap(), b"1"); } + #[tokio::test] + async fn aliases_append_and_limit_rejection_are_atomic() { + for (path, alias, stdout) in [ + ("/dev/stdout", "/dev/fd/1", true), + ("/dev/stderr", "/dev/fd/2", false), + ] { + let limits = ExecutionLimits::new() + .max_stdout_bytes(2) + .max_stderr_bytes(2); + let budget = test_budget(&limits); + let (fs, cap) = + StdStreamsFs::wrap(Arc::new(InMemoryFs::new()), b"", &budget, &limits).unwrap(); + fs.write_file(Path::new(path), b"a").await.unwrap(); + fs.append_file(Path::new(alias), b"b").await.unwrap(); + let error = fs.write_file(Path::new(path), b"c").await.unwrap_err(); + assert!(matches!(error, crate::Error::ResourceLimit(_))); + let cap = cap.lock().unwrap(); + let bytes = if stdout { &cap.stdout } else { &cap.stderr }; + assert_eq!(&**bytes, b"ab"); + assert_eq!(budget.live_bytes_for_test(), 2); + } + } + + #[tokio::test] + async fn shared_capacity_is_admitted_before_growth() { + let limits = ExecutionLimits::new().max_live_intermediate_bytes(6); + let budget = test_budget(&limits); + let (fs, cap) = + StdStreamsFs::wrap(Arc::new(InMemoryFs::new()), b"", &budget, &limits).unwrap(); + fs.write_file(Path::new("/dev/stdout"), b"abc") + .await + .unwrap(); + fs.write_file(Path::new("/dev/stderr"), b"def") + .await + .unwrap(); + assert_eq!(budget.live_bytes_for_test(), 6); + let error = fs + .append_file(Path::new("/dev/fd/1"), b"g") + .await + .unwrap_err(); + assert!(error.to_string().contains("live intermediate")); + assert_eq!(&*cap.lock().unwrap().stdout, b"abc"); + assert_eq!(budget.live_bytes_for_test(), 6); + drop((fs, cap)); + assert_eq!(budget.live_bytes_for_test(), 0); + } + + #[tokio::test] + async fn empty_writes_check_work_and_cancellation() { + let limits = ExecutionLimits::new().max_work_units(2); + let budget = test_budget(&limits); + let (fs, _) = + StdStreamsFs::wrap(Arc::new(InMemoryFs::new()), b"", &budget, &limits).unwrap(); + fs.write_file(Path::new("/dev/stdout"), b"").await.unwrap(); + let error = fs + .append_file(Path::new("/dev/stderr"), b"") + .await + .unwrap_err(); + assert!(error.to_string().contains("work units")); + assert_eq!(budget.live_bytes_for_test(), 0); + + let cancelled = Arc::new(std::sync::atomic::AtomicBool::new(false)); + let budget = ExecutionBudget::new(&ExecutionLimits::new(), cancelled.clone()); + let (fs, cap) = StdStreamsFs::wrap( + Arc::new(InMemoryFs::new()), + b"", + &budget, + &ExecutionLimits::new(), + ) + .unwrap(); + cancelled.store(true, std::sync::atomic::Ordering::Relaxed); + assert!(matches!( + fs.write_file(Path::new("/dev/fd/2"), b"a").await, + Err(crate::Error::Cancelled) + )); + assert!(cap.lock().unwrap().stderr.is_empty()); + } + + #[tokio::test] + async fn collection_retains_bytes_and_text_leases() { + let limits = ExecutionLimits::new().max_live_intermediate_bytes(16); + let budget = test_budget(&limits); + let (fs, cap) = + StdStreamsFs::wrap(Arc::new(InMemoryFs::new()), b"", &budget, &limits).unwrap(); + fs.write_file(Path::new("/dev/stdout"), b"hi") + .await + .unwrap(); + let mut result = ExecResult::default(); + let storage = cap + .lock() + .unwrap() + .collect(&mut result, &budget, &limits) + .unwrap(); + drop((fs, cap)); + assert_eq!(result.stdout, "hi"); + assert_eq!(budget.live_bytes_for_test(), 4); + drop(result); + drop(storage); + assert_eq!(budget.live_bytes_for_test(), 0); + } + + #[tokio::test] + async fn collection_admits_lossy_text_and_preserves_utf8_boundaries() { + let limits = ExecutionLimits::new(); + let budget = test_budget(&limits); + let (fs, cap) = + StdStreamsFs::wrap(Arc::new(InMemoryFs::new()), b"", &budget, &limits).unwrap(); + fs.write_file(Path::new("/dev/stdout"), &[0xa9, 0xff]) + .await + .unwrap(); + let mut result = ExecResult::ok_bytes(vec![0xc3]); + let _storage = cap + .lock() + .unwrap() + .collect(&mut result, &budget, &limits) + .unwrap(); + assert_eq!(result.stdout.as_bytes(), &[0xc3, 0xa9, 0xff]); + assert_eq!(result.stdout.text_lossy(), "é�"); + + let limits = ExecutionLimits::new().max_live_intermediate_bytes(3); + let budget = test_budget(&limits); + let (fs, cap) = + StdStreamsFs::wrap(Arc::new(InMemoryFs::new()), b"", &budget, &limits).unwrap(); + fs.write_file(Path::new("/dev/stderr"), &[0xff]) + .await + .unwrap(); + let error = cap + .lock() + .unwrap() + .collect(&mut ExecResult::default(), &budget, &limits) + .unwrap_err(); + assert!(error.to_string().contains("live intermediate")); + assert_eq!(budget.live_bytes_for_test(), 0); + } + + #[tokio::test] + async fn stdin_reads_retain_copy_and_container_capacity() { + let limits = ExecutionLimits::new(); + let budget = test_budget(&limits); + let (fs, cap) = + StdStreamsFs::wrap(Arc::new(InMemoryFs::new()), b"hi", &budget, &limits).unwrap(); + assert_eq!(budget.live_bytes_for_test(), 2); + let first = fs.read_file(Path::new("/dev/stdin")).await.unwrap(); + let second = fs.read_file(Path::new("/dev/fd/0")).await.unwrap(); + assert_eq!( + (first.as_slice(), second.as_slice()), + (b"hi".as_slice(), b"hi".as_slice()) + ); + assert_eq!( + budget.live_bytes_for_test(), + 6 + 2 * std::mem::size_of::() as u64 + ); + drop((first, second, fs, cap)); + assert_eq!(budget.live_bytes_for_test(), 0); + + let limits = ExecutionLimits::new().max_live_intermediate_bytes(1); + let budget = test_budget(&limits); + assert!(StdStreamsFs::wrap(Arc::new(InMemoryFs::new()), b"hi", &budget, &limits).is_err()); + assert_eq!(budget.live_bytes_for_test(), 0); + } + + #[tokio::test] + async fn copies_borrow_stdin_and_admit_temporary_file_reads() { + let limits = ExecutionLimits::new(); + let budget = test_budget(&limits); + let inner: Arc = Arc::new(InMemoryFs::new()); + inner.write_file(Path::new("/tmp/in"), b"ok").await.unwrap(); + let (fs, cap) = StdStreamsFs::wrap(inner.clone(), b"hi", &budget, &limits).unwrap(); + fs.copy(Path::new("/dev/stdin"), Path::new("/dev/stdout")) + .await + .unwrap(); + assert_eq!(budget.live_bytes_for_test(), 4); + fs.copy(Path::new("/dev/fd/0"), Path::new("/tmp/out")) + .await + .unwrap(); + assert_eq!(inner.read_file(Path::new("/tmp/out")).await.unwrap(), b"hi"); + fs.copy(Path::new("/tmp/in"), Path::new("/dev/stderr")) + .await + .unwrap(); + assert_eq!(&*cap.lock().unwrap().stderr, b"ok"); + assert_eq!(budget.live_bytes_for_test(), 6); + fs.copy(Path::new("/tmp/in"), Path::new("/tmp/pass")) + .await + .unwrap(); + assert_eq!( + inner.read_file(Path::new("/tmp/pass")).await.unwrap(), + b"ok" + ); + + let limits = ExecutionLimits::new().max_live_intermediate_bytes(1); + let budget = test_budget(&limits); + let (fs, cap) = StdStreamsFs::wrap(inner, b"", &budget, &limits).unwrap(); + let error = fs + .copy(Path::new("/tmp/in"), Path::new("/dev/fd/2")) + .await + .unwrap_err(); + assert!(error.to_string().contains("live intermediate")); + assert!(cap.lock().unwrap().stderr.is_empty()); + } + #[tokio::test] + async fn windows_containment_stream_copy_admits_device_before_read() { + let limits = ExecutionLimits::new().max_stdout_bytes(0); + let budget = test_budget(&limits); + let inner: Arc = Arc::new(InMemoryFs::new()); + inner + .symlink(Path::new("/dev/urandom"), Path::new("/tmp/device-alias")) + .await + .unwrap(); + let (fs, cap) = StdStreamsFs::wrap(inner, b"", &budget, &limits).unwrap(); + // Metadata is zero; only numeric device capacity is inspected. No + // random bytes are read or allocated under zero destination admission. + for source in ["/dev/urandom", "/tmp/device-alias"] { + let error = fs + .copy(Path::new(source), Path::new("/dev/stdout")) + .await + .unwrap_err(); + assert!(matches!( + error, + crate::Error::ResourceLimit(LimitExceeded::Memory(message)) + if message == "stream copy size limit exceeded" + )); + assert_eq!(budget.live_bytes_for_test(), 0); + assert!(cap.lock().unwrap().stdout.is_empty()); + } + } } diff --git a/crates/bashkit/src/interpreter/mod.rs b/crates/bashkit/src/interpreter/mod.rs index 3c0ef5ea5..2d01f7f18 100644 --- a/crates/bashkit/src/interpreter/mod.rs +++ b/crates/bashkit/src/interpreter/mod.rs @@ -9954,7 +9954,9 @@ impl Interpreter { let (fs, capture) = crate::fs::StdStreamsFs::wrap( fs, stdin.map_or(&[][..], crate::StreamData::as_bytes), - ); + &self.execution_budget, + &self.limits, + )?; (fs, Some(capture)) } else { (fs, None) @@ -10038,20 +10040,17 @@ impl Interpreter { result.stderr = stderr; diagnostic_storage = Some(storage); } - if let Some(capture) = std_capture { - let capture = - std::mem::take(&mut *capture.lock().unwrap_or_else(|e| e.into_inner())); - if !capture.stdout.is_empty() { - result - .stdout - .append(&crate::StreamData::from(capture.stdout)); - } - if !capture.stderr.is_empty() { - result - .stderr - .append(&crate::StreamData::from(capture.stderr)); - } - } + // Retain both materialized representations through redirects and + // hooks. Collection must not replace admission with a late charge. + let std_storage = if let Some(capture) = std_capture { + Some(capture.lock().unwrap_or_else(|e| e.into_inner()).collect( + &mut result, + &self.execution_budget, + &self.limits, + )?) + } else { + None + }; self.execution_budget.consume_work( u64::try_from( result @@ -10068,6 +10067,7 @@ impl Interpreter { let result = self.apply_redirections(result, redirects).await?; let result = self.apply_after_tool(name, result); drop(diagnostic_storage); + drop(std_storage); result }) } diff --git a/crates/bashkit/src/limits.rs b/crates/bashkit/src/limits.rs index 1a29f0586..b24991942 100644 --- a/crates/bashkit/src/limits.rs +++ b/crates/bashkit/src/limits.rs @@ -1191,12 +1191,26 @@ impl BudgetedVec { impl BudgetedVec { pub(crate) fn try_extend_from_slice(&mut self, values: &[T]) -> Result<(), LimitExceeded> { + self.try_extend_from_slice_with_limit(values, usize::MAX) + } + + /// Bound logical length and geometric capacity before allocation. + pub(crate) fn try_extend_from_slice_with_limit( + &mut self, + values: &[T], + limit: usize, + ) -> Result<(), LimitExceeded> { let required = self .inner .len() .checked_add(values.len()) .ok_or_else(|| LimitExceeded::Memory("intermediate buffer size overflow".into()))?; - let capacity = self.growth_capacity(required)?; + if required > limit { + return Err(LimitExceeded::Memory( + "intermediate buffer size limit exceeded".into(), + )); + } + let capacity = self.growth_capacity(required)?.min(limit); self.try_reserve_capacity(capacity)?; self.inner.extend_from_slice(values); Ok(()) @@ -2154,6 +2168,21 @@ mod tests { assert_eq!(&*bytes, b"1234", "failed growth must not mutate"); } + #[test] + fn budgeted_vec_length_cap_also_bounds_geometric_capacity() { + let limits = ExecutionLimits::new().max_live_intermediate_bytes(5); + let budget = ExecutionBudget::new(&limits, Arc::new(AtomicBool::new(false))); + let mut bytes = BudgetedBytes::new(Some(&budget)).unwrap(); + bytes.try_extend_from_slice_with_limit(b"abc", 5).unwrap(); + bytes.try_extend_from_slice_with_limit(b"d", 5).unwrap(); + assert_eq!(budget.live_bytes_for_test(), 5); + assert!(bytes.try_extend_from_slice_with_limit(b"ef", 5).is_err()); + assert_eq!(&*bytes, b"abcd"); + assert_eq!(budget.live_bytes_for_test(), 5); + drop(bytes); + assert_eq!(budget.live_bytes_for_test(), 0); + } + #[test] fn budgeted_vec_rejects_element_size_overflow_without_allocating() { let limits = ExecutionLimits::new().max_live_intermediate_bytes(u64::MAX); diff --git a/crates/bashkit/tests/integration/dev_fd_path_tests.rs b/crates/bashkit/tests/integration/dev_fd_path_tests.rs index 6d94d21f4..621d5bb0b 100644 --- a/crates/bashkit/tests/integration/dev_fd_path_tests.rs +++ b/crates/bashkit/tests/integration/dev_fd_path_tests.rs @@ -154,3 +154,40 @@ async fn literal_dev_stdin_argument_is_untouched() { let (_, r) = run("echo /dev/stdin").await; assert_eq!(r.stdout, "/dev/stdin\n"); } + +// Zero admission is a configuration fixture: only a two-byte ordinary input. +#[tokio::test] +async fn stream_operand_checks_admission_before_redirect() { + let mut bash = Bash::builder() + .concurrent_jobs(false) + .limits(bashkit::ExecutionLimits::new().max_stderr_bytes(0)) + .build(); + let result = bash.exec("printf hi | tee /dev/fd/2 2>/dev/null").await; + assert!(matches!(result, Err(bashkit::Error::ResourceLimit(_)))); + assert!(bash.exec(":").await.unwrap().is_success()); +} + +#[tokio::test] +async fn stream_collection_checks_combined_admission() { + let mut bash = Bash::builder() + .concurrent_jobs(false) + .limits(bashkit::ExecutionLimits::new().max_stdout_bytes(3)) + .build(); + let result = bash.exec("printf hi | tee /dev/stdout >/dev/null").await; + assert!(matches!(result, Err(bashkit::Error::ResourceLimit(_)))); +} + +#[tokio::test] +async fn stream_aliases_and_append_match_bash() { + let script = "printf hi | tee -a /dev/stdout /dev/fd/1 /dev/stderr /dev/fd/2"; + let (_, result) = run(script).await; + assert_eq!(result.stdout, "hihihi"); + assert_eq!(result.stderr, "hihi"); + let oracle = std::process::Command::new("bash") + .args(["-c", script]) + .output() + .unwrap(); + assert_eq!(result.stdout.as_bytes(), oracle.stdout); + assert_eq!(result.stderr.as_bytes(), oracle.stderr); + assert_eq!(Some(result.exit_code), oracle.status.code()); +} diff --git a/docs/security.md b/docs/security.md index af0a47913..eddd013aa 100644 --- a/docs/security.md +++ b/docs/security.md @@ -28,6 +28,16 @@ job slot. Exceeding a limit returns a resource-limit error. Pipelines within these budgets keep their streaming, isolation and `PIPESTATUS` behavior; `concurrent_jobs(false)` uses sequential stages without producer slots. +Builtin file operands for `/dev/stdout`, `/dev/stderr`, `/dev/fd/1` and +`/dev/fd/2` share the request's `max_live_intermediate_bytes` budget. +Capture reserves capacity before each write or append and obeys +`max_stdout_bytes` / `max_stderr_bytes` before growth and combined collection. +Cached text and stdin copies are also admitted; collection retains its storage +charges through redirects and hooks. Every write checks work, cancellation +and deadlines. Refused admission returns a resource-limit error even when +output would be discarded by a redirect. Ordinary aliases and append/copy +semantics remain unchanged within the configured budgets. + `dd conv=sync` checks its padded block length against the 64 MiB copy cap before allocating or zero-filling. Copied payload capacity also counts against `max_live_intermediate_bytes` until output handling completes, including when diff --git a/knowledge/foundations/vfs.md b/knowledge/foundations/vfs.md index 4d1b12b5c..d6fc7cb54 100644 --- a/knowledge/foundations/vfs.md +++ b/knowledge/foundations/vfs.md @@ -224,6 +224,26 @@ same stream (`tee /dev/stdout` prints its copies back to back, as through a pipe). Only fds 0-2: `/dev/fd/63` stays a real file for process substitution. +`StdStreamsFs` receives the shared `ExecutionBudget` and stream ceilings. +Both write and append paths use `BudgetedBytes`: logical size and planned +capacity are admitted before growth; every write, including empty writes, +charges work and checks cancellation/deadlines. Combined builtin/capture +bytes are admitted before collection, and lossy cached text is built directly +into `BudgetedString`; byte/text leases remain held through redirects and +result hooks. A refused stream admission returns a resource-limit error even +when an outer redirect would discard the stream. + +The wrapper's stdin snapshot and returned fd-0 copies reserve retained capacity. +Because `FileSystem::read_file` returns a plain `Vec`, the wrapper retains each +copy's lease (and the lease-vector capacity) conservatively until it drops. +Copy from fd 0 borrows admitted stdin. Copies involving another stream admit +temporary source storage from backend metadata before reading and retain it +through the destination write. Canonical system-device read ceilings apply even +when metadata reports zero; source aliases resolve and normalize back to POSIX +VFS spelling before that admission, including on Windows. +Ordinary copies still delegate to the backend. +Backend metadata/read-size correctness remains the filesystem contract. + #### /dev/zero Handled in `InMemoryFs`: every read returns `DEV_ZERO_READ_BYTES` (1 MiB) of zeros, writes and appends are discarded like `/dev/null`. The directory @@ -370,7 +390,8 @@ their `stat` is an `lstat`, and they refuse to read or write through a link. directory, `..` clamped at `/` (TM-ESC-002). Writes through a dangling link create the target. - Recursive `find` and `grep` share `fs::canonicalize`, resolving links through - `lstat`/`read_link` with the same 40-hop cap. Walkers compare resolved ancestor + `lstat`/`read_link` with the same 40-hop cap. It uses VFS joins and root-aware + symlink targets on every host. Walkers compare resolved ancestor paths to reject directory cycles; the per-lookup cap alone cannot bound fan-out. - At most 40 links per lookup, then "Too many levels of symbolic links" (TM-DOS-011). `exists` is false for dangling and looping links. diff --git a/knowledge/security/security-testing.md b/knowledge/security/security-testing.md index 6577cb5be..984cc1f9b 100644 --- a/knowledge/security/security-testing.md +++ b/knowledge/security/security-testing.md @@ -144,6 +144,23 @@ amplification reproducer is needed for these admission contracts. See [Builtins](../foundations/builtins.md) and TM-DOS-126 in the [Threat Model](threat-model.md). +## Standard-stream capture admission + +`fs::std_streams::tests` uses at most a few bytes per buffer. It checks +shared capacity admission before growth, alias/write/append atomicity, +empty-write work and cancellation, retained collection byte/text leases, +lossy UTF-8 boundaries, stdin copy/container ownership, and temporary-copy +admission with ordinary filesystem pass-through. Zero device-copy admission +checks only a numeric read ceiling and never loads device bytes. Its +`windows_containment_` name also includes it in the existing Windows job; the +assertion requires the pre-read admission error for direct and absolute-symlink +sources. The numeric +`budgeted_vec_length_cap_also_bounds_geometric_capacity` fixture checks a +five-byte ceiling without a large allocation. `dev_fd_path_tests` adds +zero/three-byte admission configurations, session recovery, discarded +redirects, and a two-byte alias/append comparison with real Bash. +No amplified-output or host-exhaustion workload is required. + ## Adding New Fail Points Add `fail_point!("module::function", |action| ...)` under diff --git a/knowledge/security/threat-model.md b/knowledge/security/threat-model.md index 4e2c47c7d..604df625a 100644 --- a/knowledge/security/threat-model.md +++ b/knowledge/security/threat-model.md @@ -1424,7 +1424,7 @@ This section maps former vulnerability IDs to the new threat ID scheme and track | TM-DOS-100 | jq control-character normalization amplification | A jq JSON string consisting of literal controls expands sixfold when each byte becomes `\u00XX`; an unmetered compatibility copy can exhaust memory or CPU before strict parsing | The jq-only normalizer charges input-length work before its single pass, borrows unchanged input, and acquires/grows a shared live-intermediate lease before every allocation growth (`builtins/jq/input.rs`), **MITIGATED** | | TM-DOS-101 | yq structured-data amplification | YAML aliases, YAML/JSON nesting, multi-document floods, filters, and output format expansion are all attacker-controlled | A pre-deserialization lexical pass rejects aliases without misclassifying quoted or block scalar content; real parsers with recursion/depth caps, aggregate budgets, shared jaq work/deadline/output controls, and a final rendered-output cap; integration tests cover alias bombs plus YAML+JSON depth/document/input/output bounds, proptest composes arbitrary YAML+filters, and `yq_fuzz` covers stdin/in-place format paths, **MITIGATED** | | TM-DOS-102 | Archive decoder allocation-before-check | A small gzip/bzip2 stream makes the decoder output buffer allocate beyond live-memory or filesystem quotas before the post-growth size check runs | Decoder chunks validate absolute/ratio bounds and acquire a shared execution-budget lease before `try_reserve_exact` and copy; corrupt, truncated, CRC-invalid, ratio-bomb, and live-budget tests fail closed, **MITIGATED** | -| TM-DOS-103 | Post-allocation resource charging | Archive/compression and other growing buffers could call `reserve`/`extend` before taking a live-byte lease; concurrent `fetch_add` accounting could also wrap or transiently overcommit the ceiling | `BudgetedVec`/`BudgetedBytes` and `BudgetedString` own their `ExecutionBudgetLease`, charge planned capacity before `try_reserve_exact`, roll back on allocation/error, and release on drop. CAS admission rejects overflow and concurrent overcommit. Archive creation, extraction/list output, gzip/bzip2 encoders, decompression, `od`, and every `fmt` output path use the builders. `fmt` charges generated indentation/prefix bytes before copying and optimal-fit search steps against the shared work/cancellation/deadline budget; formatter errors propagate instead of being discarded (`fmt_resource_tests`). Tar `-T`/`--files-from` leases raw list reads before VFS loading, retains budget-owned names/vector capacity, decodes lossy UTF-8 directly into admitted storage, and charges bytes/lines/decoding work; bounded allocation regressions include nonexistent names, repeated lists, stdin, and session recovery. Shell diagnostic names are bounded to 1,024 UTF-8 bytes before copying; prefix rewriting and its cached lossy text use these builders before routing, retain both leases through routing/hooks, and charge work/check cancellation and deadlines during scanning and emission. `diagnostic_prefix_tests` covers small-budget amplification across capture, streaming and redirects, plus recovery. `od` rejects widths above 65,536 bytes before layout arithmetic (including empty input), pads only a single <= 8-byte field on the stack, uses u64 padding products for 32-bit safety, and charges input/field rendering work; `od_resource_tests` cover huge-width rejection, boundary formatting, live/work budgets, and subsequent execution, **MITIGATED** | +| TM-DOS-103 | Post-allocation resource charging | Archive/compression and other growing buffers could call `reserve`/`extend` before taking a live-byte lease; concurrent `fetch_add` accounting could also wrap or transiently overcommit the ceiling | `BudgetedVec`/`BudgetedBytes` and `BudgetedString` own their `ExecutionBudgetLease`, charge planned capacity before `try_reserve_exact`, roll back on allocation/error, and release on drop. CAS admission rejects overflow and concurrent overcommit. Archive creation, extraction/list output, gzip/bzip2 encoders, decompression, `od`, and every `fmt` output path use the builders. `StdStreamsFs` capture uses the same builders with stdout/stderr ceilings before write/append and combined collection, charges work/checks cancellation on every write, and retains admitted byte/text leases through redirects and hooks. Its stdin snapshot/read copies and stream-copy temporary reads are admitted before copying/loading; ordinary filesystem operations retain pass-through semantics (`fs::std_streams::tests`, `dev_fd_path_tests`). `fmt` charges generated indentation/prefix bytes before copying and optimal-fit search steps against the shared work/cancellation/deadline budget; formatter errors propagate instead of being discarded (`fmt_resource_tests`). Tar `-T`/`--files-from` leases raw list reads before VFS loading, retains budget-owned names/vector capacity, decodes lossy UTF-8 directly into admitted storage, and charges bytes/lines/decoding work; bounded allocation regressions include nonexistent names, repeated lists, stdin, and session recovery. Shell diagnostic names are bounded to 1,024 UTF-8 bytes before copying; prefix rewriting and its cached lossy text use these builders before routing, retain both leases through routing/hooks, and charge work/check cancellation and deadlines during scanning and emission. `diagnostic_prefix_tests` covers small-budget amplification across capture, streaming and redirects, plus recovery. `od` rejects widths above 65,536 bytes before layout arithmetic (including empty input), pads only a single <= 8-byte field on the stack, uses u64 padding products for 32-bit safety, and charges input/field rendering work; `od_resource_tests` cover huge-width rejection, boundary formatting, live/work budgets, and subsequent execution, **MITIGATED** | | TM-DOS-104 | Pipeline stderr aggregation exceeds the configured output cap before final truncation | A pipeline with many stderr-producing stages can accumulate their combined diagnostics in memory | The pipeline aggregate stops appending at `max_stderr_bytes` and propagates its truncation signal to the final result, **MITIGATED** | | TM-DOS-106 | Zip creation allocation-before-check | `zip` recursively collected complete file contents and encoded its output before acquiring live-byte leases, allowing transient allocations above the configured intermediate-memory ceiling | Zip traversal collects paths only; each VFS file size is leased before `read_file`, leases remain live with the entries, and archive encoding uses `BudgetedBytes` to charge capacity before growth — **MITIGATED** | | TM-DOS-108 | CLI host-stdin read can stall before execution | A producer keeps the one-shot CLI's stdin pipe open while idle or trickles fewer than 10 MiB, so the pre-execution read never reaches EOF or the byte cap | Read stdin on a dedicated thread and bound the wait with the selected execution timeout; timeout exits with a diagnostic suggesting `--no-stdin`, while the existing byte cap remains enforced — **MITIGATED** | @@ -1460,7 +1460,7 @@ This section maps former vulnerability IDs to the new threat ID scheme and track | Execution timeout (30s) | TM-DOS-023 | `limits.rs` | Yes | | Builtin output pre-allocation caps | TM-DOS-058, TM-DOS-090 | `limits.rs`, `builtins/shuf.rs` | Yes | | Shared request execution budget | TM-DOS-096 | `limits.rs`, `parser/mod.rs`, `interpreter/mod.rs`, high-risk builtins/runtimes | Yes | -| Pre-allocation buffer admission | TM-DOS-103 | `limits.rs`, `builtins/archive.rs` | Yes | +| Pre-allocation buffer admission | TM-DOS-103 | `limits.rs`, `fs/std_streams.rs`, `stream.rs`, `builtins/archive.rs` | Yes | | Persistent custom fd cap | TM-DOS-063 | `limits.rs`, `interpreter/mod.rs` | Yes | | Command hash table entry cap and retained-byte leases | TM-DOS-129 | `builtins/introspect.rs` | Yes | | Array/arith report caps (32 levels, 64 KiB, 256 chars) | TM-DOS-130 | `interpreter/arithmetic.rs`, `interpreter/mod.rs`, `interpreter/expansion.rs` | Yes |