[yeonwooson]

Summary

Contributor motivation

Adding an host opt-in to allow unsaved editor buffers to be analyzed without changing the module identities used by direct queries and imports.

AI-assisted implementation and rationale

Use the full ModulePath identity to select an explicit in-memory source. Load::load_from_path checks that exact override first and otherwise uses the variant's existing loader. A filesystem-based host opts in per entry by passing ModulePath::filesystem(p) to set_memory; no per-State flag or second constructor is needed.

Removing an override restores normal loading, including the existing error when its saved source is missing. It does not save or delete the file. Memory(p) and FileSystem(p) remain independent, and invalidation stays changed.contains(handle.path()).

Dependency: this revision depends on #5540 and currently includes its NFC changes. The precursor should land first. Existing PR history and the current upstream OpenFiles integration are preserved; this cleanup neither modifies the precursor branch nor rebases or force-pushes this PR.

AI-assisted review follow-up: filesystem contents advance the external-input generation only when loaded without a memory override. Editor override edits do not count as disk changes; removing an override can still advance generation when loading changed disk contents. The CLI snippet reuses its existing module identity, and the memory documentation describes exact source overrides.

The second-pass cleanup removes the two specifically requested redundant tests, flattens the remaining regressions to top-level test_* functions, and shares the filesystem import fixture between State and incremental tests. The existing six-step incremental generation test remains the coverage for filesystem override updates, rechecks, diagnostics and disk fallback. The explicitly optional duplicate-lookup optimization is left unchanged.

Existing native callers continue to supply Memory identities. This does not change native unsaved-import authority or diagnostic scheduling. Import discovery, TSP messages, the browser target/configuration, dependencies and lockfiles are unchanged. No guest, transport, sandbox migration or performance claim is added.

No issue auto-close is requested.

• I am a human, and I fully understand the code I am submitting.

Test Plan

AI-assisted test report: Windows ARM64 using public Rust 1.99.1.

For the three-file test-only cleanup:

• PASS, 127 tests; 0 failed or ignored, including a fresh publication rerun: cargo test -p pyrefly --lib --locked --offline -- test::state:: test::incremental:: test::util:: --test-threads=2
• PASS, all native targets typechecked/linted on the identical final source: cargo clippy -p pyrefly --all-targets --locked --offline --message-format=json
• PASS, including a fresh publication rerun with no source changes: python test.py --mode cargo --no-test --no-tensor-shapes --no-conformance --no-jsonschema --no-extension

The executed test set is exactly the prior affected tests after the two requested deletions and four renames, plus the existing utility test. Unique identity independence, pending removal/rollback, lookup masking and missing-source coverage remain. All prior unrelated State tests and the six-step incremental check closure/assertions are unchanged.

The retained generation regression uses committable transactions and TestSubscriber:

Step	Modules rechecked	Errors in main	Generation delta
Initial analysis	foo, main	0	0
Interface-changing override	foo, main	1	0
Comment-only override edit	foo	1	0
Disk update/invalidation under an active override	none	1	0
Remove override and load disk contents	foo, main	0	1
Remove the absent override again	none	0	0

It was first run against unchanged product code and reproduced the generation bug: the interface-changing override advanced generation to 1 instead of keeping it at 0. The source-origin guard makes the full contract pass.

Broader validation on parent 3cc54c0f00019704e6abf411293536c0328d8cf5, before this test-only cleanup:

• PASS, 599 tests; 0 failed or ignored: cargo test -p pyrefly --lib --locked --offline -- test::state:: embed:: playground:: state::transaction_manager:: lsp::wasm::document::tests:: test::incremental:: test::lsp::definition:: error::suppress:: state::errors:: test::map_int_tuples_binding:: test::tsp:: test::lsp::lsp_interaction::did_change:: test::lsp::lsp_interaction::notebook_sync:: test::lsp::lsp_interaction::io:: --test-threads=2
• PASS: cargo check -p pyrefly --lib --target wasm32-wasip1 --locked --offline
• PASS, actual compilation/linking from pyrefly_wasm: cargo build --lib --target wasm32-unknown-unknown --locked --offline

An out-of-tree Rust consumer was rebuilt for WASIp1 against that parent and executed using the existing public browser_wasi_shim provider:

MODULE_MEMORY_OK: overlays; identity, imports, updates, rollback, disk fallback

The previous Wasm validation used WASI SDK 34 and Node 24.15.0. Wasm builds were not repeated for test cleanup alone: production source, dependencies and target configuration are unchanged. The consumer is not included in this PR. Node core WASI directory enumeration is unsupported on this Windows host; product loading was not changed to hide that host limitation.

Existing unrelated warnings remain; there are no added-line diagnostics. The full native suite, local Linux/macOS execution and a new browser-runtime/packaging test were not run. Meta-internal import/test results and fresh remote CI are separate from these local checks.

[github-actions]

✅ This PR meets all requirements from our pull request checklist.

Optional recommendations:

• If this PR addresses an open issue, please link the issue in the PR description and claim it.

Note: this comment updates automatically after changes to the PR. Updates may take up to half an hour.

[meta-codesync]

This pull request has been imported. If you are a Meta employee, you can view this in D129108088. (Because this pull request was imported automatically, there will not be any future comments.)

[github-actions]

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

[rchen152]

@kinto0 looks like this is part of a series of changes that you've been reviewing?

[kinto0]

awesome job parallelizing work. this is an interesting PR. it seems like it also implements #5005.

it seems special-casey to me. I wonder if there are simpler options.

(idea)
Instead of a per-State flag, can we key MemoryFiles by ModulePath? Then Load checks memory first for any variant and falls back to the variant's normal source. A WASI host overlays a filesystem module by calling set_memory with ModulePath::filesystem(p). No flag, no second constructor, and invalidation stays changed.contains(handle.path()). Suggest an [NFC] precursor that changes the key type, then a small diff for the load change.

[kinto0]

Review automatically exported from Phabricator review in Meta.

[kinto0]

Thanks, this is the shape we wanted. Two requests before we land it:

1. Don't count in-memory loads of filesystem modules as external changes.

In Transaction's load validation (state.rs, where a changed load sets inputs_changed), the check assumes that new contents for a FileSystem module came from disk. With this PR they can come from memory, so an override edit moves StateData::generation. For the TSP that means the snapshot moves twice per edit: once for the edit event and once for a "disk change". Only reads from disk should set it:

             {
+                // Only contents read from disk change what Pyrefly reads from outside the editor.
                 if matches!(
                     module_data.handle.path().details(),
                     ModulePathDetails::FileSystem(_)
-                ) {
+                ) && self
+                    .memory_lookup()
+                    .get(module_data.handle.path())
+                    .is_none()
+                {
                     self.data.inputs_changed.store(true, Ordering::Relaxed);
                 }

Removing the override reloads from disk, so that path still moves the generation, as it should.

2. Add a test for incremental behavior and the generation.

Current tests check contents and errors, but not what gets rechecked or the generation. Something like this in test/incremental.rs: main imports foo, run each step in a committable transaction with a TestSubscriber, and assert (modules rechecked, errors in main, generation moved):

step	rechecked	errors	generation moved
initial check	foo, main	0	no
override FileSystem(foo) with an interface change	foo, main	1	no
override with the same interface (comment-only edit)	foo	1	no
write a new foo.py to disk + invalidate_disk, override still active	nothing	1	no
remove the override	foo, main	0	yes
remove it again	nothing	0	no

Smaller points:

• The doc comment on StateData::memory still says "contents for ModulePath::memory values". Please update it to say that memory contents override the module path's usual source.
• In commands/check.rs, reuse module_path instead of rebuilding it with ModulePath::memory(module_path.as_path().to_path_buf()).
• Pyrefly convention puts tests under pyrefly/lib/test/ and appends them at the end of the file. Please move the state/memory.rs tests there, and move mod module_path_overlay_tests below the existing tests in test/state.rs.

[kinto0]

Thanks, this addresses all of the last round, and the generation test looks good. Second pass, mostly test cleanup:

1. Remove tests that the new incremental test covers. test_filesystem_overrides_preserve_incrementality_and_generation checks the same behavior, plus rechecks and the generation:
   • filesystem_overrides_invalidate_importers_and_return_to_disk runs almost the same steps (override, disk write under the override, remove the override). Please delete it.
   • exact_overrides_and_removal_preserve_each_variants_normal_loader tests the one if in Load::load_from_path against all 6 ModulePath variants. Please delete it.
2. Flatten the test modules. test/state.rs uses top-level #[test] fn test_* functions. Please flatten mod module_path_overlay_tests and mod memory_files_tests into that style. The setup (temp dir, ConfigFile, main/foo handles) is also copied between test/state.rs and test/incremental.rs. Share one helper, or delete the copy with item 1.
3. (nit, optional) The new guard in state.rs looks up the override again after Load::load_from_path already did. This is correct as is. If you like, look it up once and use the result in both places.

[github-actions]

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅