From 19d856ba06a4851ed66195b4802b59670bda3503 Mon Sep 17 00:00:00 2001 From: devdanzin <74280297+devdanzin@users.noreply.github.com> Date: Sat, 5 Sep 2026 19:46:14 -0300 Subject: [PATCH] fleet: keep the log a restart destroys, and stop calling high CPU a kill Two things found while comparing a 12-instance PyPy fleet against a 5-instance one. Neither changes what fusil fuzzes. **`fusil.out` was truncated on every restart.** `fleet-run` redirected with `>`, so each restart destroyed the log of the run that had just died -- which is the only one anybody wants to read. systemd restarts an instance whenever the cgroup OOM killer reaps it (`fusil@.service`: MemoryMax + OOMPolicy=continue + Restart=always), so this is routine, not exceptional: one fleet restarted 10 times in 7.5 h and left a 155 KB log for a 2079-session instance, against 913 KB for an instance that never restarted. The death reason survived only in the remote systemd journal, which does not travel with a copied fleet directory. Now appends; the existing "# fleet-run instance ..." line marks each run boundary inside the file. **`cpu_load_kills` never counted kills.** `CpuProbe.live()` only sets a score, logs, sends `session_rename "cpu_load"` and stops monitoring -- it terminates nothing. The Python fuzzer then sets `watch.cpu.max_score = 0` unless `--record-high-cpu` (`fusil/python/__init__.py:938`), so the usual effect is a name part and nothing else: the session runs to completion and is not discarded. The counter, its schema comment ("sessions the CPU-load watcher killed") and the report's `cpu-kills` column all said otherwise, and that misreading turned a fleet comparison into a claim about discarded work that was simply wrong. The sidecar key becomes `high_cpu_sessions` (schema v3) and the report column `high-cpu`. `cpu_load_kills` is still emitted as an alias and still read on merge, because archived sidecars carry only the old name -- verified by merging 22 real schema-2 sidecars from an existing fleet and getting the same 10002 under both keys. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01WPBSmN87d2BqUnrDbojUT1 --- HANDOFF-tsan.md | 164 ++++++++++++++++++ ...6\371\320\027\244\023\306\372\341\257\027" | Bin 0 -> 171 bytes "Z@\302\212C" | Bin 0 -> 6197 bytes ...tkho\\Scza\\AXo\\BAqHnxyi\\BpSAGZhqEHeUVH" | Bin 0 -> 277 bytes co.txt | 0 fleet/fleet-run | 10 +- fusil/process/cpu_probe.py | 6 +- fusil/python/fleet_report.py | 7 +- fusil/python/session_stats.py | 29 +++- fusil_use_guide.md | 88 ++++++++++ ...\235\302\262\302\207\302\2628\026\302\262" | Bin 0 -> 4890 bytes ro.txt | 0 tests/python/test_fleet_report.py | 2 +- tests/python/test_session_stats.py | 26 ++- tests/python/test_stats_agent.py | 2 +- ...01%\023b\341\204\022\366\254\365\225E\343" | Bin 0 -> 5275 bytes "\302\235\302\201" | Bin 0 -> 171 bytes ...17\302\216\n\002\302\256<\303\210\303\242" | Bin 0 -> 4890 bytes ...4\251\347\216\226\353\204\222\342\260\260" | Bin 0 -> 5746 bytes 19 files changed, 318 insertions(+), 16 deletions(-) create mode 100644 HANDOFF-tsan.md create mode 100644 "OQz\205=\202\326\371\320\027\244\023\306\372\341\257\027" create mode 100644 "Z@\302\212C" create mode 100644 "\\WUFEhjxWgstkho\\Scza\\AXo\\BAqHnxyi\\BpSAGZhqEHeUVH" create mode 100644 co.txt create mode 100644 fusil_use_guide.md create mode 100644 "po\303\260\024e\024\303\214\303\212V\302\245\303\276\303\272M\303\235\302\262\302\207\302\2628\026\302\262" create mode 100644 ro.txt create mode 100644 "\263\023\303\301%\023b\341\204\022\366\254\365\225E\343" create mode 100644 "\302\235\302\201" create mode 100644 "\303\255\"\0010\303\252\023f\020\303\217\302\216\n\002\302\256<\303\210\303\242" create mode 100644 "\345\234\266\341\217\272\356\255\243\355\226\226\356\264\270\345\256\225\345\234\235\340\255\235\357\266\225\347\270\233\354\244\240\341\213\226\341\255\260\346\244\251\347\216\226\353\204\222\342\260\260" diff --git a/HANDOFF-tsan.md b/HANDOFF-tsan.md new file mode 100644 index 0000000..19ee820 --- /dev/null +++ b/HANDOFF-tsan.md @@ -0,0 +1,164 @@ +# Handoff — fusil `--tsan` feature set (for Codex) + +Written 2026-07-18. Purpose: give an incoming agent the **off-repo** context around the +ThreadSanitizer (`--tsan`) data-race fuzzing feature — builds, venvs, the sibling catalog, +fleet infra, campaign state, and gotchas that are **not** discoverable from the fusil repo +alone. Complements CLAUDE.md's in-repo `--tsan` section — read that first for the code-side +overview; this file is the surrounding environment. Uncommitted; delete when done. + +Repo states at handoff: **fusil** `main`@`45061ea`; sibling **cpython-tsan-findings** +`main`@`819e740`. + +--- + +## 1. What `--tsan` is, and where it lives in this repo + +`--tsan` turns fusil into a free-threading **data-race** fuzzer (vs the OOM/crash mode). Instead +of the single-threaded OOM sweep, it emits a **concurrency-stress region**: a few SHARED objects +hammered by many worker threads (barrier-released), so concurrent C-level access to one object's +state trips a TSan race. + +In-repo touch points: +- **`fusil/python/write_python_code.py` → `_write_tsan_stress_region()`** — the emitter. + The op-mix is ops **(a)–(i)**; (h) shared-iterator races and (i) read-while-mutate are the + newest (PR #211) and are what just found the bytes-iterator race. Emitter is **gated behind + `--tsan`** — non-`--tsan` generated output (and its goldens) are unchanged, so don't expect + golden regen for tsan-only edits. +- **`fusil/python/tsan_dedup.py`** — the race-report **parser + dedup engine + `Suppressor`**. + `parse_report(stdout)` → `{signature, sites, framework, kind}`, where `signature` is the + sorted `"file:func | file:func"` site pair. **This is a cross-repo contract** (see §4). +- **`fusil/python/__init__.py`** — the `--tsan*` CLI options (`--tsan-threads`, + `--tsan-iterations`, `--tsan-shared-objects`, `--tsan-suppressions`, `--tsan-dedup-catalog`, + `--tsan-dedup-keep`, `--tsan-dedup-prune`). `setupProject()` makes `--tsan` imply `--no-numpy`. +- **`doc/tsan-mode-plan.md`** — the design doc + phase history (Phase 3.1 = the new iterator ops). +- **Tests**: `tests/python/test_tsan_generation.py` (emitter markers + "emitted script compiles") + and `tests/python/test_tsan_dedup.py` (parser/signature/suppression, pure-Python). +- **`fleet/`** — systemd-based multi-instance runner (`fleet up/down/status/report/finds`, + `fleet check` is `--tsan`-aware). README documents the tsan fleet config. + +fusil applies the whole TSan **env recipe internally** when `--tsan` is passed (see §3) — you +don't set it by hand for a fuzzing run, only for manual repros. + +--- + +## 2. Off-repo: builds & venvs + +**CPython build matrix** — `~/projects/python_build_matrix/builds//python`. Built via +`cd ~/projects/python_build_matrix && JOBS=$(nproc) bash build_all.sh `. clang-21 at +`/usr/local/bin`. Builds relevant to TSan work: +- `debug-ft-nojit-tsan` — **the TSan target** (CPython 3.16.0a0, `--disable-gil + --with-thread-sanitizer`). What the fleets run against. +- `release-ft-nojit`, `release-ft-nojit-asan`, `debug-ft-nojit-asan`, `debug-ft-nojit` — used + for crash-repro / severity confirmation of races (ASan gives UAF/OOB backtraces). +- `release-ft-nojit-o0` — a `-O0` **NDEBUG** FT build made for the #153928 OOB-char demo + (`-O2` register-caches the iterator cursor, hiding the OOB window; NDEBUG turns the UAF into a + real segfault). Handy for demonstrating iterator OOB reads. + +**Venvs** (the ones that actually exist and work): +- `~/venvs/fusil_np_verify/bin/python` — **CPython 3.14, GIL**, has `python-ptrace`. Use for + running the test suite and non-FT verification. The `oom_dedup`/`tsan_dedup` tests are + pure-Python and run here fine. +- `~/venvs/matrix_venvs/fusil_debug-ft-nojit_venv/bin/python` — **CPython 3.16 free-threaded**, + has `python-ptrace` + fusil. This is the **fleet runner** venv (a FT runner is required because + `PYTHON_GIL=0` is inherited by the runner too). +- **`~/venvs/fusil_venv` is GONE** — don't use it. + +`python-ptrace` is a hard import-time dependency of `fusil.application`, so the runtime stack +only imports under a venv that has it (the two above). Pure-engine tests (`tsan_dedup`, +`oom_dedup`, generation) don't need the runtime stack. + +--- + +## 3. Off-repo: the TSan env recipe (needed for MANUAL repros) + +fusil sets these itself under `--tsan`; you only need them to reproduce a race by hand against +`debug-ft-nojit-tsan`: + +```sh +setarch -R bash -c "ulimit -v unlimited; PYTHON_GIL=0 \ + TSAN_OPTIONS='halt_on_error=1:symbolize=1:exitcode=66:history_size=4' \ + DEBUGINFOD_URLS= \ + /python repro.py" +``` +- **`setarch -R`** (ASLR off) and **`ulimit -v unlimited`** (RLIMIT_AS) are mandatory for TSan. +- **`DEBUGINFOD_URLS=`** is load-bearing: without it `llvm-symbolizer` hangs for ~120 s on the + blackholed Ubuntu debuginfod server (any local ASan/TSan symbolizer hang = this). gdb is immune. +- exit **66** = a data race was detected. `--no-numpy` (numpy isn't FT-clean; the plugin injects + `import numpy` which the tsan build can't import). + +--- + +## 4. Off-repo: the sibling catalog `cpython-tsan-findings` (CRITICAL LOCKSTEP) + +`~/projects/cpython-tsan-findings` (published: github.com/devdanzin/cpython-tsan-findings) is the +triage/reporting home for the races. It is **coupled to fusil**: + +- `scripts/ingest.py` **imports fusil's `tsan_dedup.py`** via + `FUSIL_TSAN_DEDUP=/home/danzin/projects/fusil/fusil/python/tsan_dedup.py`. So **the signature + format that `tsan_dedup.parse_report` emits is a hard contract** — if you change how signatures + are built (site pair format, framework filtering, the ACCESS/stanza regexes), you change what + matches `catalog/known_races.tsv` and can silently break dedupe across the whole catalog. There + is a matching-order + normalization subtlety (`gen_known_races._normalize` sorts the pair); + keep them in sync. +- `catalog/known_races.tsv` (119 sigs / 34 races) = the dedupe snapshot fusil reads with + `--tsan-dedup-catalog`. Regenerated from `reports/*/meta.json` by `scripts/gen_known_races.py`. +- `catalog/suppressions.txt` = TSan-style suppressions consumed by `tsan_dedup.Suppressor` + (framework noise, glibc/OpenSSL false positives, subinterpreter machinery). Post-hoc regex over + the signature. +- Triage loop: `FUSIL_TSAN_DEDUP=… python3 scripts/ingest.py '/inst-*/python*/*'` buckets + crash dirs by signature; new signatures get a `reports/TSAN-NNNN-*/meta.json` (+ optional + `repro.py` / `backtrace.txt`), then `gen_known_races.py` regenerates the TSV. Catalog convention: + **commit straight to `main`** (no PR), then push. + +**If Codex edits `tsan_dedup.py`, re-run the catalog's `test_tsan_dedup.py` AND re-ingest a fleet +to confirm known races still match.** The parser has been bug-fixed twice for exactly this class +of regression (PR #207: the `ACCESS` regex missed "Previous atomic write" stanzas → fabricated +`A | A` signatures; framework-FP over-matching). + +--- + +## 5. Off-repo: fleet infra + campaign state + +- Fleets run at **`/home/fusil/runs/fusil-tsan_fleet_NN/inst-*/python*/*`** (systemd, as the + `fusil` user). **Ingest glob must be `inst-*/python*/*`** — a restarted instance gets a fresh + `python-2`/`python-3` project dir; `inst-*/python/*` silently ingests only the pre-restart run. +- Campaign so far: fleets **01–06 triaged**, all converged (each ingests 0 new signatures after + its faces are folded). **37 races cataloged (TSAN-0001..0037)**; umbrella issue + **python/cpython#153852** holds 15 findings. +- **PR #211 (the new iterator ops) just paid off**: fleet-06 (first run with them) surfaced + **TSAN-0037 = the `bytes` iterator race** (`striter_next` non-atomic `it_index` + `it_seq` + double-DECREF), 196/223 dirs — the bytes analog of **python/cpython#153928** (the `str` + iterator, johng's issue, which we reproduced + commented on). We just commented on #153928 with + the bytes `it_seq` UAF + the "whole sequence-iterator family" point. The `list`/`tuple`/`range` + iterators share the exact shape and haven't been fleet-caught yet — likely fleet-07 material. +- **Process-leak history**: the fleet leaked python processes owned by the `fusil` user; fixed in + **PRs #208/#209** (`start_new_session=True` + `killpg`; reap-on-self-exit in + `fusil/process/create.py`). If you touch process/child handling, don't regress this. +- **Policy**: do NOT file FT-**subinterpreter** races upstream (per cpython#143232) — they're + suppressed in `suppressions.txt`, not cataloged as bugs. + +--- + +## 6. Running the tests (gotcha) + +Some test modules do `from python._test_options import …`, so run from the **`tests/` dir** or via +discover so the `python` subpackage resolves: +```sh +cd ~/projects/fusil && ~/venvs/fusil_np_verify/bin/python -m unittest discover -s tests +# or a single module (from tests/): +cd tests && ~/venvs/fusil_np_verify/bin/python -m unittest python.test_tsan_generation +``` +CI runs **both** `ruff check` and `ruff format --check` over `fusil/ tests/ +fuzzers/fusil-python-threaded` — `check` passing does not imply `format --check` passes. + +--- + +## 7. If Codex changes TSan things — quick guardrail checklist + +- Edited `_write_tsan_stress_region`? → `test_tsan_generation.py` must still pass (it compiles the + emitted script + asserts op markers); keep the emitter gated behind `--tsan`. +- Edited `tsan_dedup.py`? → `test_tsan_dedup.py` must pass **and** re-ingest a fleet to confirm + `known_races.tsv` still matches (the signature format is the cross-repo contract, §4). +- Edited process/child handling? → don't regress the #208/#209 process-reaping fix. +- New race worth cataloging? → that's the sibling-repo's job (meta.json + `gen_known_races.py`), + not a fusil change. Ping the maintainer (devdanzin) rather than filing upstream directly. diff --git "a/OQz\205=\202\326\371\320\027\244\023\306\372\341\257\027" "b/OQz\205=\202\326\371\320\027\244\023\306\372\341\257\027" new file mode 100644 index 0000000000000000000000000000000000000000..ade26a5c5537fd33b23d4a1421b805a1f01e7b38 GIT binary patch literal 171 zcmeyplKB^7T_yt%{F1jxD$UF((ap?L$W1ND$WKuK^E2~`^-?QRlkHZrW-O&4F1qzLbX-a`AghV7z`iL-v@GvDn;-(@Btv4k-XU2EN zch|Q1gH?WbeC+vj&v(u}&&h2o8B(Q;-A*K3wH`ApDQ?=DYD!Ktd#1b#%qL7S=2EwJ~aSNT!kY5D1WnrP;b+QAg@^d9ub&-|--^>>D3YByh4GxWPiDhGtRB)2^)X{OqtlUOp5NJuec81os% zvdn&zXs#1$?QjfJB6?#(vNcf`OK5cqW3}mIBBs^>uSDH~M17K|NmWZrj>c41%Vbqu zVnmlTIHT1tiIQcc)#w6>aZR;!qbUkJCxiXw0|d<0x0M=Gkt;svYuAl0R`*_(>04Kq z{9(zK59I`KefTCj&&@ERqW92kEhRu_lA;)v;VKF^?sG^6AGG(;=bu@6s4PX_?E3A+ znx4v3uIOxDLFY`>Iv69CtN26(PU#D?{?VkKk?FbnmiN7CT#(^1SvgWhL)a9Mz@ecH z1^A+yaIh5PP*+L2Z3)!NHh)_3#K1n8av{fV&!@{@1(>4Koe#e9-8S;-=Qs8(-?!>- zGQG0c{J8v~($pwi3ou3JdJWuF$aUWb$z9iHMpBdu7!tEEBtqLph6INN_@eV}z^J}M zX;<3_D`v$bauq!`Q$c2FV%nWbr{QK4N?h+hhU*BP zY?*v`(f3;}%5+E1D{nX-edv|7S|pDKu{w~TtcBJ@N^g`-{S)NpnQtHIi+=N*TvVst zg-*Q%iNg;i0+U_Swz$*I{#mB$H@8*ZK`Rv+p9GLar=Cm1E#bw4?Lp2q*J1fS{iiHOk_dE;U=7JYHpB?AY z5j8uEiD|d)SXsMwK&BgCdv3?Ct^X7|!do#aCSp_=mMi4irga;hDqp-`rnz;GPpY0b z=tcAW7y=>lkrviE`Ga35Q)ZjG+F;+@}wc;v*I5kK<$ z0zB5BRX4c?X-)pThgIkvV9;NlW#AaaaaAwbf!S~=y70tDx4yIKyi7B8TjxM;6q;*U zKIH;TG0JAvSklTr-!99b~2{<`3Yz3Zy?2&xZ1^D8UUJLsB zOL1@&qT^fpPb7Lie@dpm>RCQ?Focj#^oAwq*8A3ALLN>mzq38WguS01b7O;V^C*fTG z@Og{U9fs+~46AZjivs&C(VV!q-YWi9i)V&wZSxHy$_0S0LpWYPTkpfe zLN}xO8{6Brz-q$&?E@_O*Qe{>c0?EFa$N{Iblj zybIOwsl6MXeCzFJWy%H7#Cd^qUt$g-v@9} zU6O0nP$12n7jx8!HpM71XTo^?qe-&Uvm2(4?P`;sr`zq7jE^&mb z3~f+mRSZo3=UJI@VHBnORe%wdeyPB(ik&6!HPU8r!_rBYm|gxyrtSPg)OJ|gQ-1J- zH(7uYHC6{^DoRFmGK%6G>UTfE(y+?<&Hj)~xe!=74o1{lRbXjQL<|Q?iK|)~L2JVk zE8gAx^?Nt$ktr8)>@nZkaWJCRaZ&}0=cy7N*ZpbL>7moTGUdW3D*3AbBdSaU4%Q9l z|C~2M0y{T0935KyzD&7*RSG9a1Lrr^nQuRb1$dE6JRqaMhb#O}?B6zT+7I3j-EWLb ZH}#|Sx77Ynrd-Iet@-Nts{oTo{11{k47&gT literal 0 HcmV?d00001 diff --git "a/\\WUFEhjxWgstkho\\Scza\\AXo\\BAqHnxyi\\BpSAGZhqEHeUVH" "b/\\WUFEhjxWgstkho\\Scza\\AXo\\BAqHnxyi\\BpSAGZhqEHeUVH" new file mode 100644 index 0000000000000000000000000000000000000000..4c041b88caecfcb651962c5cce51a7bae57c1a36 GIT binary patch literal 277 zcmeyplKB^hO>s$4W?s5oCL;p_1CX1WpHiBWYPXUVEC43b*B`33b+~ub4$k-u)X7+v z3Dm*xOWr1_G&83}H#1KmH?<@qKScq|&&(^8Q4JWR8Pq)PE5*4RaeMQQ&5l3DK03= oPtywk5}7%vML^M{tkmR^VvxTOrf=c3coebVt6e&b@!P-v0E7Zr*#H0l literal 0 HcmV?d00001 diff --git a/co.txt b/co.txt new file mode 100644 index 0000000..e69de29 diff --git a/fleet/fleet-run b/fleet/fleet-run index a2f6945..a0af1c4 100755 --- a/fleet/fleet-run +++ b/fleet/fleet-run @@ -41,11 +41,17 @@ export PYTHONPYCACHEPREFIX="${PYTHONPYCACHEPREFIX:-/tmp/fusil-pycache-root}" # `import fusil` works whether or not fusil is pip-installed in RUNNER_PY's environment. export PYTHONPATH="$(dirname "$(dirname "$FUSIL_PY")")${PYTHONPATH:+:$PYTHONPATH}" -# Output sink: a per-run logfile (truncated each start) or discard. +# Output sink: a per-instance logfile (appended across restarts) or discard. The +# "# fleet-run instance ..." line below marks each run boundary inside it. if [ "${LOG:-file}" = "none" ]; then exec >/dev/null 2>&1 else - exec >"$inst_dir/fusil.out" 2>&1 + # APPEND, never truncate: systemd restarts an instance whenever the cgroup OOM killer + # reaps it (fusil@.service: MemoryMax + OOMPolicy=continue + Restart=always), and a + # truncating redirect destroyed the log of the run that just died -- which is exactly the + # one you need to see why. A 12-instance PyPy fleet restarted 10 times in 7.5 h and left + # 155 KB of log for a 2079-session instance, against 913 KB for one that never restarted. + exec >>"$inst_dir/fusil.out" 2>&1 fi echo "# fleet-run instance $i gil=$gil $(date -u +%FT%TZ) dir=$inst_dir" diff --git a/fusil/process/cpu_probe.py b/fusil/process/cpu_probe.py index f4bae7d..ece8fb0 100644 --- a/fusil/process/cpu_probe.py +++ b/fusil/process/cpu_probe.py @@ -54,7 +54,11 @@ def live(self): if duration < self.max_duration: return - # Success + # Success. NOTE: this does NOT kill the process -- it only scores, logs, and tags the + # session name. The Python fuzzer sets max_score to 0 unless --record-high-cpu, so the + # usual effect is a "cpu_load" name part and nothing else: the session runs to + # completion and is not discarded. The stats counter is named accordingly + # (high_cpu_sessions); its "cpu_load_kills" alias is legacy and was always a misnomer. self.score = self.max_score self.error( "CPU load (%.1f%%) bigger than maximum (%.1f%%) during %.1f sec: score=%.1f%%" diff --git a/fusil/python/fleet_report.py b/fusil/python/fleet_report.py index 9d32786..59fac83 100644 --- a/fusil/python/fleet_report.py +++ b/fusil/python/fleet_report.py @@ -234,7 +234,8 @@ def collect_instance(inst_dir: str, *, now=None, systemd=True) -> dict: "kept_crashes": crash_total, "sidecar_crashes": merged.get("crashes", 0), "timeouts": merged.get("timeouts", 0), - "cpu_load_kills": merged.get("cpu_load_kills", 0), + # Either spelling: v3 sidecars carry "high_cpu_sessions", older ones only the alias. + "high_cpu_sessions": merged.get("high_cpu_sessions", merged.get("cpu_load_kills", 0)), "new": new_candidate_count(by_label), "mode": merged.get("mode") or (current or {}).get("mode"), "plugins": merged.get("plugins") or (current or {}).get("plugins") or [], @@ -409,7 +410,7 @@ def render_instance(r: dict) -> str: to_pct = (100.0 * r["timeouts"] / r["sessions"]) if r["sessions"] else 0.0 find = (1000.0 * r["kept_crashes"] / r["sessions"]) if r["sessions"] else 0.0 L.append( - "sessions %-8d (%s) crashes %-5d (%.1f/1k) timeouts %d (%.1f%%) cpu-kills %d" + "sessions %-8d (%s) crashes %-5d (%.1f/1k) timeouts %d (%.1f%%) high-cpu %d" % ( r["sessions"], format_rate(r["sessions"], r["uptime_s"]), @@ -417,7 +418,7 @@ def render_instance(r: dict) -> str: find, r["timeouts"], to_pct, - r["cpu_load_kills"], + r["high_cpu_sessions"], ) ) L.append("-" * 72) diff --git a/fusil/python/session_stats.py b/fusil/python/session_stats.py index 7ce6558..7db7bf2 100644 --- a/fusil/python/session_stats.py +++ b/fusil/python/session_stats.py @@ -10,7 +10,7 @@ Sidecar schema (v1):: { - "schema": 2, + "schema": 3, "mode": "oom", # active fuzzing mode ("oom"/"tsan"/"rustpython"/ # "concurrency-stress"/"normal"/a "+"-joined combo) "plugins": ["cereggii"], # names of the loaded fusil plugins (entry-point discovered) @@ -22,7 +22,13 @@ "sessions": 3092, # sessions finished "crashes": 112, # sessions scored as a success/finding "timeouts": 70, # sessions the target-process timeout fired on - "cpu_load_kills": 5, # sessions the CPU-load watcher killed + "high_cpu_sessions": 5, # sessions that spent >=10s above 75% CPU. NOT kills: + # CpuProbe only scores + renames, and the Python fuzzer + # sets its max_score to 0 unless --record-high-cpu, so + # the session is neither ended nor discarded. + "cpu_load_kills": 5, # legacy alias of the above, still emitted so older readers + # and archived sidecars keep working. Misnamed: it never + # counted kills. "modules": { # per-target-module breakdown "_blake2": {"hits": 15, "crashes": 3, "timeouts": 0}, ... @@ -41,7 +47,9 @@ import time from typing import Callable -SCHEMA_VERSION = 2 # v2 adds "mode" + "plugins" (additive; readers use .get with defaults) +# v2 adds "mode" + "plugins"; v3 renames "cpu_load_kills" -> "high_cpu_sessions" and keeps the +# old key as an alias (additive both ways; readers use .get with defaults and accept either). +SCHEMA_VERSION = 3 class SessionStats: @@ -74,7 +82,7 @@ def __init__( self.sessions = 0 self.crashes = 0 self.timeouts = 0 - self.cpu_load_kills = 0 + self.high_cpu_sessions = 0 # module name -> {"hits": int, "crashes": int, "timeouts": int} self.modules: dict[str, dict[str, int]] = {} # --tsan shared-object composition -> count (Slice B). Empty for non-tsan runs. @@ -96,7 +104,7 @@ def record( if timeout: self.timeouts += 1 if cpu_load: - self.cpu_load_kills += 1 + self.high_cpu_sessions += 1 if tsan_kind: self.tsan_kinds[tsan_kind] = self.tsan_kinds.get(tsan_kind, 0) + 1 # ``module`` can legitimately be None very early (before the first module loads); @@ -126,7 +134,9 @@ def to_dict(self) -> dict: "sessions": self.sessions, "crashes": self.crashes, "timeouts": self.timeouts, - "cpu_load_kills": self.cpu_load_kills, + "high_cpu_sessions": self.high_cpu_sessions, + # Legacy alias: archived sidecars and any older reader still key on this name. + "cpu_load_kills": self.high_cpu_sessions, "modules": self.modules, "tsan_kinds": self.tsan_kinds, } @@ -158,6 +168,7 @@ def merge(dicts: list[dict]) -> dict: "sessions": 0, "crashes": 0, "timeouts": 0, + "high_cpu_sessions": 0, "cpu_load_kills": 0, "modules": {}, "tsan_kinds": {}, @@ -174,8 +185,12 @@ def merge(dicts: list[dict]) -> dict: out["mode"] = d["mode"] if d.get("plugins"): out["plugins"] = list(d["plugins"]) - for key in ("sessions", "crashes", "timeouts", "cpu_load_kills"): + for key in ("sessions", "crashes", "timeouts"): out[key] += int(d.get(key, 0) or 0) + # Accept either spelling: a v3 sidecar carries both, a pre-v3 one only the alias. + high_cpu = d.get("high_cpu_sessions", d.get("cpu_load_kills", 0)) + out["high_cpu_sessions"] += int(high_cpu or 0) + out["cpu_load_kills"] = out["high_cpu_sessions"] for name, bucket in (d.get("modules") or {}).items(): acc = out["modules"].setdefault(name, {"hits": 0, "crashes": 0, "timeouts": 0}) for field in ("hits", "crashes", "timeouts"): diff --git a/fusil_use_guide.md b/fusil_use_guide.md new file mode 100644 index 0000000..4b79126 --- /dev/null +++ b/fusil_use_guide.md @@ -0,0 +1,88 @@ +# fusil Guide + +This guide will help you install and use fusil, both locally and on the cloud. + +## Installing fusil locally + +To install fusil locally, follow these steps: +```bash +git clone https://github.com/devdanzin/fusil +cd fusil +# Activate the virtual environment, then +pip install -e . +``` + +Then you'll need to create a `fusil` user, under which the fuzzer will run. You can do this using the `adduser` command: +```bash +sudo adduser fusil +``` + +Installing fusil on the cloud is similar. On the OracleArm machine, fusil is already installed and the `fusil` user is already created. + +## Using fusil + +fusil must be launched by root user: it will create a new fusil instance and run the fuzzer scripts as the `fusil` user. This is done to ensure that the fuzzer runs with the correct permissions and does not interfere with the system. + +The easiest way to use fusil is to launch as a fleet from a root console: +```bash +sudo su # Switch to root user +export F=/home/opc/projects/fusil/fleet/fleet # Point this to the fleet executable +export FLEET_CONF=/home/opc/fleet.conf # Point this to a fleet configuration file +$F up 3 # Launch the fleet with 3 workers +``` + +A fusil fleet is a collection of workers that run fuzzing jobs in parallel. Configuration sets the fuzzed modules/extensions and many other options. Here's an example configuration file: +```bash +TARGET_PYTHON=/home/danzin/venvs/matrix_venvs/main-debug-gil-nojit-asan/bin/python +GIL_MODES="1" # GIL build -> clean ASan UAF/free-stacks +RUNNER_PY=/home/danzin/venvs/matrix_venvs/fusil_debug-ft-nojit_venv/bin/python # has fusil + ptrace +FUSIL_PY=/home/danzin/projects/fusil/fuzzers/fusil-python-threaded +FLEET_DIR=/home/fusil/runs/fusil-apsw_asan_01 + +FUSIL_FLAGS="--discover-in-target --modules=apsw --test-private \ +--classes-number 25 --methods-number 12 --functions-number 30 --objects-number 20 \ +--child-memory-limit-mb 4096 --crash-drain-ms=3000 \ +-v --timeout 60 --success=12000 --fast --exitcode-score=1" + +INSTANCES=6 +MEM_MAX=6G +NICE=10 +LOG=file +``` + +In the OracleArm machine, there are a couple configuration files available in `/home/opc/`. You can use these as a starting point for your own configuration. Here's a concrete example in `/home/opc/fleet_multidict_oom_foreign.conf`: +```bash +TARGET_PYTHON=/home/opc/venvs/new_3.14_venv/bin/python +GIL_MODES="1" # allocation-failure paths; deterministic for the gdb resolver +RUNNER_PY=/home/opc/venvs/release_cpython_venv/bin/python +FUSIL_PY=/home/opc/projects/fusil/fuzzers/fusil-python-threaded +CATALOG=/home/opc/projects/cpython-oom-findings/catalog/known_sites.tsv # CPython sites; apsw/SQLite + # crashes won't match -> oomNEW +INGEST=/home/opc/projects/cpython-oom-findings/scripts/ingest.py +FLEET_DIR=/home/fusil/runs/fusil-multidict_oomforeign_04 + +# Shallow sweep bounds: foreign allocations fault at LOW start indices, so a deep set_nomemory-style +# sweep just burns cycles. Sequences (--oom-seq) matter more here than depth: a failed allocation +# mid-Connection/Cursor lifecycle is exactly the cross-call stale-state class. +FUSIL_FLAGS="--modules=multidict --packages=multidict --test-private \ +--classes-number 25 --methods-number 10 --functions-number 25 --objects-number 20 \ +--child-memory-limit-mb 4096 \ +-v --timeout 60 --success=12000 --fast --exitcode-score=1 \ +--oom-fuzz --oom-verbose --oom-foreign --oom-dedup-prune --oom-dedup-catalog=$CATALOG \ +--oom-seq --oom-seq-randomize --oom-seq-len 8 --oom-window 16 --oom-max-start 50 " +INSTANCES=6 +MEM_MAX=4G +NICE=10 +LOG=file +``` + +Once the fleet is up, you can monitor it using the `$F status` command: +```bash +$F status +``` +To stop the fleet, use the `$F down` command: +```bash +$F down +``` + +The findings will be written to the `$FLEET_DIR` directory. There are many findings in `/home/fusil/runs/fusil-multidict_oomforeign_04`, most of which are uninteresting. diff --git "a/po\303\260\024e\024\303\214\303\212V\302\245\303\276\303\272M\303\235\302\262\302\207\302\2628\026\302\262" "b/po\303\260\024e\024\303\214\303\212V\302\245\303\276\303\272M\303\235\302\262\302\207\302\2628\026\302\262" new file mode 100644 index 0000000000000000000000000000000000000000..31044231635bc573a008d3ecc76221aea543befd GIT binary patch literal 4890 zcmc&&Yitx%6yDO7_xmBB0;OOp;Oy)IZ9#F?M1uhZ3dpMzCbKhlZ^uq&mYG@FLKG3j zL{Z{1iV{MMMwEy`2qq}n5EjI4{C-iX}nb-nTSs;o+Ao5S0NWSjYpAeTU;z*sTz-z zu_fVHsE7;6MPsZA^E`9Q%|20LZEf-O%9%2 zVIqk{kA!V@Hppo&imI;Kq6mS@A!5jadvzN&;?THW5#0OI(B)|R@JP~Z3*Up_HUvW! zoL@d+&DBL6BKY|ObyrUOG&quR!Jm-$b(IsQiF8|ZePLwxj9Jq@TYp->OTL^nx_tg0 z0-U$mi6|z%nw?!2d^c0@8Fyg^kTHWJikD6imjoUVw0=YJKzY2cNF4=@h}#ASp4B zq=fHV&UKkjVIDr?x=Tn__Z6$8N(5JNp#dMLMn-X{5Jn}FBjYbFnY8kWUj# zJPgr4?-9Ir{Up zL(2yG4Kodf2_uD_i^+T-dkHHQjQ@)uZl(&3aq~(Sas@T$M9mPmzG-ploOw%(jHsAAwdFXF(`wqN>2S_HQu;XM2E)Dw*-jp02aL#!XMZ_{`$nFdb0?;4=xFWQaM- zh5MO$rA@b4T}N&aFf~}Yg*TK7qdB}!;I_a=B^##srDZ$k4BI7OY9!5qmNMyCN<-YO(zO zHOLvjm7Hy}SHRTp(!y_>1!o&KPdu$l2d2c?xfZH!1vR9z;r%=E(3<)Wp%_sk zX$mb}q+1?_r7Q4UjKYHk7_eR8Mp@NaFMSVs7LRW-`h6~7YLF=3O`=R%w0Vy!Aq}3D zFx-=HT`zYvOmn6X=CJ*G;yzUCh2|G)y^u6lxYTJdEOq{kgAH0F+cw!NxcOpEbnN6K z0;Yy{zJ3m_z1E@KE62YVFf~}pK{uu}QQYD4@8pD4@3&o|Lq1W%<4c2K`SQK!x&_n? ztYx#6-2b_r#qj#+v1JIi`mVy41WXN&Hw}j6y*1@LMb!uzP%cR7`0``; z?O0@R-w38g(wy#QYZ?qo74lqERO>&sX#dAUd&{@{dGBumrUtpSQ$T`BMjo@Yr5FnH x@T{1iCq{&)7x`m|8*kdgracVZfC;-VrF^P*WYh6g0;WdNT+ou9Zh4qc=pXDRPQd^G literal 0 HcmV?d00001 diff --git a/ro.txt b/ro.txt new file mode 100644 index 0000000..e69de29 diff --git a/tests/python/test_fleet_report.py b/tests/python/test_fleet_report.py index 1bbee26..0927047 100644 --- a/tests/python/test_fleet_report.py +++ b/tests/python/test_fleet_report.py @@ -27,7 +27,7 @@ def _sidecar(path, **kw): "sessions": 0, "crashes": 0, "timeouts": 0, - "cpu_load_kills": 0, + "high_cpu_sessions": 0, "modules": {}, } data.update(kw) diff --git a/tests/python/test_session_stats.py b/tests/python/test_session_stats.py index 5c7776f..ccf641e 100644 --- a/tests/python/test_session_stats.py +++ b/tests/python/test_session_stats.py @@ -30,10 +30,33 @@ def test_counters_and_per_module(self): self.assertEqual(s.sessions, 4) self.assertEqual(s.crashes, 2) self.assertEqual(s.timeouts, 1) - self.assertEqual(s.cpu_load_kills, 1) + self.assertEqual(s.high_cpu_sessions, 1) self.assertEqual(s.modules["json"], {"hits": 3, "crashes": 2, "timeouts": 0}) self.assertEqual(s.modules["sqlite3"], {"hits": 1, "crashes": 0, "timeouts": 1}) + def test_high_cpu_key_is_emitted_under_both_names(self): + """v3 renamed the counter; the old key stays as an alias. + + `cpu_load_kills` was always a misnomer -- `CpuProbe` only scores and renames the + session, and the Python fuzzer zeroes its score unless --record-high-cpu, so nothing + is killed or discarded. Renaming it outright would break archived sidecars and any + reader still keying on the old name, so both are emitted. + """ + s = self._stats() + s.record("json", cpu_load=True) + d = s.to_dict() + self.assertEqual(d["high_cpu_sessions"], 1) + self.assertEqual(d["cpu_load_kills"], 1) + + def test_merge_reads_pre_v3_sidecars(self): + """A sidecar written before the rename carries only the alias; merge must still count it.""" + legacy = {"sessions": 10, "crashes": 1, "timeouts": 2, "cpu_load_kills": 5} + current = {"sessions": 10, "crashes": 1, "timeouts": 2, "high_cpu_sessions": 3} + out = SessionStats.merge([legacy, current]) + self.assertEqual(out["high_cpu_sessions"], 8) + # the alias mirrors it, so an old reader aggregating a mixed fleet still agrees + self.assertEqual(out["cpu_load_kills"], 8) + def test_tsan_kinds_counted(self): # Slice B: --tsan sessions record their shared-object composition; non-tsan sessions # (tsan_kind=None) never touch the counter. @@ -84,6 +107,7 @@ def test_to_dict_schema(self): "sessions", "crashes", "timeouts", + "high_cpu_sessions", "cpu_load_kills", "modules", ): diff --git a/tests/python/test_stats_agent.py b/tests/python/test_stats_agent.py index e0c5c3a..d6f0c40 100644 --- a/tests/python/test_stats_agent.py +++ b/tests/python/test_stats_agent.py @@ -112,7 +112,7 @@ def test_timeout_and_cpu_load_via_rename(self): agent.on_session_start() # resets rename parts agent.on_session_rename("cpu_load") agent.on_session_done(0.0) - self.assertEqual(agent.stats.cpu_load_kills, 1) + self.assertEqual(agent.stats.high_cpu_sessions, 1) def test_tsan_kind_recorded_from_source(self): # Slice B: StatsAgent reads the --tsan shared-object composition off the PythonSource and diff --git "a/\263\023\303\301%\023b\341\204\022\366\254\365\225E\343" "b/\263\023\303\301%\023b\341\204\022\366\254\365\225E\343" new file mode 100644 index 0000000000000000000000000000000000000000..7dfd7209bd9e2d06868348aaac77aadfcc660ec2 GIT binary patch literal 5275 zcmcIoYitx%6fRQU74Sh&QQ1(SV%VKs`hX(dRm4|-SVa&}9A{^5?~a|$E_bH1)rdw! z^oN+JAj%&VF;-$Cf=G;sK|n#35|tOpEBiI{0!GZj0fajG;dP%_Ora4?5iNzwZ%CKoz z4Nc*u2~}!UMVMI)ifJTV1OgOl>FK0lFtZ%(J+cCw||=!;tM!SuFk_FsQsA7LxMdH0{`Ck~P*XubO! z2hT}ys$ujnY_ofUoW81RScapj5V#&F1|O{D(?`GCF#R-PyPs_yvgLun@od4>Y{cLU z)EhXbmZN$?B~7ozmkvJh(Qd*@#(tp;9riVDCLe&3(Ggw(76df#ArD`07d91R9ws$f zpSC3F%hwKiXhqFh!i16K7w5_qOFpLHa>tz~!q#!?#|=N!`25Trgzdlm)ViIMpl0&5 z=3xr1^(2JL+v1K|BYQTFSi1BGVZy+axB^om@U3G?2$+vAxbE?o)kD>Kt&T#SEl9%$ zZPIB5%0FEv<;W`a#t*8FK-`7VN!-Pfk3s$O&T)@W8>vNF(@^@Yz4cVu(D4+2@p!y&JWHha*B+?+B(yLTXB$s$8~V!4k9)0r?m6BD zy~aE8a3jWmgw_EWZa^Qm9nJMOXv>dnUGqlCBXG`bxT=`WYw8ux^4^}0-CzolIT=?Q z3O0d1zqxF>@At_M$M+C6bNrl%b5|6_vANs;P}e~|C0BGFzObSf0Do&Sp50Pm?wdV- z`FFj(C5%5^HTU&VXWVqU6;%gbc5I1x_`<4w5>@v%(^|5V)M#9G;jCZuU4&IvzgQ7V zo^rL{hT73tUIP#5gwMmH_CjWMIjS$wVTN*OC;Gqswcqx(G3yAMf7kx4e@DDho`w3+ zZGKm-{yaSDFX+fO9w(GOYLc4N!6q4$Ty;}w&C$|U!iry;l8%Ia^qORkg$9T&YEvfF zyAx}uj9Oa4q-CLWZKzB&?F^KCq`mez{F-oQXCif=O48%HfycKJCJgMq9AE6ceDtJ5 zgJC*h!wR*ts4(JF-Hx~uzUmG3_|9Om-~8a@1HbIuNtiIOINc_RmjMm~A*M<W>|~MqU?)>ECcS?KE;_=<@@qZ)uzf`aKG>JjW(^rI7%;-%GdxHTl2eOT z{V;O*Q-ldac5$@tVgjyzAc8q5FUoLkOHmGHkFtTO!w2_#NSH7%%GD?eXC6M$lh6%b zmTm(Ui(*{1++6yL+-uJ}E#5c111&_KLj7&YE>-_*Nn&WV}1Y1kz zSGCRqN@4hAA9yR4e2h#vylYBMGcI4Sd3lq*&({q`<{RNh*K43&U~NZk)4}VO9y}8I z`ME=c38NEAvE*Z9%8@=Lv~OtP9%Z80p|-S@0qJ$mT#{Nc?hC?%(Fv(o@-Z^$sIKE` z*ic}OX6aP!PKlE2j=JWz2opw@YhD5c%EXe7ky*P*pp<5d1PWfa^6-L#jgPH{zjxY= zPAJ8akC7>(L39oKf1biY$LPCw{iSV}oEK-pz}lUMHHA|Rzc<(as>>Fc?`?y-wJ>Bc z-DERuqPko?(@qcBbIs_!aVCtM?ilesQ{j4y|3-P4BhJM;rojQzMFy>p&iQRl=}UwO PBg+@&`XZKmOf>o*nPAmO literal 0 HcmV?d00001 diff --git "a/\302\235\302\201" "b/\302\235\302\201" new file mode 100644 index 0000000000000000000000000000000000000000..c80ad0c2f026b089570e9c801e5d4602c8e43bb8 GIT binary patch literal 171 zcmeyplKB^7T_yt%{F1jxD$UF((ap?L$W1ND$WKuK^E2~`^-?QRlkHZrW-n5EjI4{C-iX}nb-nTSs;o+Ao5S0NWSjYpAeTU;z*sTz-z zu_fVHsE7;6MPsZA^E`9Q%|20LZEf-O%9%2 zVIqk{kA!V@Hppo&imI;Kq6mS@A!5jadvzN&;?THW5#0OI(B)|R@JP~Z3*Up_HUvW! zoL@d+&DBL6BKY|ObyrUOG&quR!Jm-$b(IsQiF8|ZePLwxj9Jq@TYp->OTL^nx_tg0 z0-U$mi6|z%nw?!2d^c0@8Fyg^kTHWJikD6imjoUVw0=YJKzY2cNF4=@h}#ASp4B zq=fHV&UKkjVIDr?x=Tn__Z6$8N(5JNp#dMLMn-X{5Jn}FBjYbFnY8kWUj# zJPgr4?-9Ir{Up zL(2yG4Kodf2_uD_i^+T-dkHHQjQ@)uZl(&3aq~(Sas@T$M9mPmzG-ploOw%(jHsAAwdFXF(`wqN>2S_HQu;XM2E)Dw*-jp02aL#!XMZ_{`$nFdb0?;4=xFWQaM- zh5MO$rA@b4T}N&aFf~}Yg*TK7qdB}!;I_a=B^##srDZ$k4BI7OY9!5qmNMyCN<-YO(zO zHOLvjm7Hy}SHRTp(!y_>1!o&KPdu$l2d2c?xfZH!1vR9z;r%=E(3<)Wp%_sk zX$mb}q+1?_r7Q4UjKYHk7_eR8Mp@NaFMSVs7LRW-`h6~7YLF=3O`=R%w0Vy!Aq}3D zFx-=HT`zYvOmn6X=CJ*G;yzUCh2|G)y^u6lxYTJdEOq{kgAH0F+cw!NxcOpEbnN6K z0;Yy{zJ3m_z1E@KE62YVFf~}pK{uu}QQYD4@8pD4@3&o|Lq1W%<4c2K`SQK!x&_n? ztYx#6-2b_r#qj#+v1JIi`mVy41WXN&Hw}j6y*1@LMb!uzP%cR7`0``; z?O0@R-w38g(wy#QYZ?qo74lqERO>&sX#dAUd&{@{dGBumrUtpSQ$T`BMjo@Yr5FnH x@T{1iCq{&)7x`m|8*kdgracVZfC;-VrF^P*WYh6g0;WdNT+ou9Zh4qc=pXDRPQd^G literal 0 HcmV?d00001 diff --git "a/\345\234\266\341\217\272\356\255\243\355\226\226\356\264\270\345\256\225\345\234\235\340\255\235\357\266\225\347\270\233\354\244\240\341\213\226\341\255\260\346\244\251\347\216\226\353\204\222\342\260\260" "b/\345\234\266\341\217\272\356\255\243\355\226\226\356\264\270\345\256\225\345\234\235\340\255\235\357\266\225\347\270\233\354\244\240\341\213\226\341\255\260\346\244\251\347\216\226\353\204\222\342\260\260" new file mode 100644 index 0000000000000000000000000000000000000000..06d9f1f3064baab9fcd8149da37af354bb9300c8 GIT binary patch literal 5746 zcmc&&eT-Dq6(2r#`C44Jwp1uAic5jaycyUpD83PGD64{ev=%D*`sU5uJGcAxy~%xV zmRS;Q8WU2}M!Qx-5)#ltWux(nO0-r|j7e*$MPRLA(L%ZcjUN>GkYJF|*m>bW$DCgSn95=DklpJ6o1 z?Vl0Db)xM#hnWgV&u>Yz$D5*Yqp3C8m`cQ>dK2)9H_eMTCy1WVjg;aTTz8F3R!=LO zq!k0+(aubwWjSeQd7v0GbSur4j{=@!A%1fyfw0rPBUuc&;y`0_myWx2Syhrg@SIlD zxVt792d?*B70Syzl})JUw!-VN4vI8sXXD=Fn={hj&owUnjecqd-x-3o}AEm zMx}*Cb%!<$yP(2)^Kr-+W98%h5W+A^{41XGy(3YoX`OW1mPGy84+>r`0(-PZbOS*E?AF58$rk`uZGncNr zl1fT^knQSg%$7s#wL6ihJq&lBk+GS92L(icDx{Tr^%u`VUX9F0u`%e613l1x=9@eIa7U6J8+*d}`@!A*I{f4s3&L3L zpU)_X7J%)qaV$OSWNcS+T-{4P92XIW16^A6?ard&!$Mt|hQSEK*X_m}vi?A@%|x|ZvOYtG51*g{wfW7hym%ypt7*j>vrpB5=&2+WM;kHUO!Va9qf{0#jVKU zY0B%nRY(i=mwuDE>wn#A#wFiWsW4CgJcIMz%NrJ^a?Et2%&ISzy5HcL-rM{}gvwpif85lCy%_U&PJ~#%4hsM&-7$au*zVS2eI{Mz@AE;Cq z7;G)VVK>dM_k#^v;(>t^XTG}Z(Y0R?1yS#1hNSC z!OCSP2lg+8@*@lxb81PRy=}oPCuR~$R>Z!~+9nPD@|a46QQ)c{VyGH175K<-4MB)0 zxHhWPh<3T6Cd3bO+WL%JrP!%DB; zlW2sIZEmwCKK8?ppjH))0)OiA=*3jvBSYVG^<)DV-E}!8d;*4s105u5Cx3DJf*KgX z`i7+;_tZew#0QS6R2VX3VraBpkK%_~wu%CFv5Ib3vf$CbzOhfG_z3?53_^tkKEi1H_0LWTHtbH<^t@p5CnqGSFap>zJ~C{0hHH3B zL5;Ii4yyOn@xwD>R_*e-%&&V@DvSWXjE{^zy+lr$oh9(m&E_#?rAZ!lTo}zKJ%4)R zU%z`vrNSui+dya_;4(fk;w{1ILd5WS>?`DX+pLCn-#)xUrNRi}3{k~Y;3K2P1JpEg zzGAZ!-uwKsA(AW}5|Qw$U6|;Gr`^VP+dfyRFk}K{_ASX1Lr$~bQk)dM#YwO48`7M` uf@n1Oq=6q9{Va->W#!#)_DiA-c~+lTe*V>mH>y+^1^%GFgk^l<@&5zVN@gbj literal 0 HcmV?d00001