[Engine] DeepSeek 0.1.51: 256K auto RAM store shrinks from 48 GB to 17.8 GB on 2x RTX 3090 #4147

Area

[Engine] (speed, kernels, models, the server and API)

Desktop app, Terminal or source?

Source checkout using serve/server.py with the official prebuilt Linux CUDA13 DeepSeek engine packages. No locally rebuilt or patched native engine.

What happened

After updating from Strata source v0.2.4.2 / DeepSeek engine 0.1.49.2 to v0.2.6 / engine 0.1.51, the automatic expert RAM-store ceiling fell from 48.0 GB to about 17.8 GB on the same two discrete RTX 3090 cards and 64 GB-class RAM system, with a 262144-token context.

The new run retained substantial reclaimable memory headroom: the minimum sampled Linux MemAvailable during validation was about 35.4 GiB. This includes reclaimable file cache; I am not treating it as physically unused RAM, process RSS, or memory that can all safely be allocated.

The new configuration also ran slower in the saved long-context workload. The sizing behavior is locally observed; the degree to which it causes the speed difference has not been isolated by a same-version cache-size A/B.

Expected: automatic sizing should distinguish a discrete-GPU machine from a true unified-memory GPU before applying the additional UMA context reserve. I am not asking it to overcommit RAM or fit the entire CPU expert lane into this 64 GB machine.

Smallest steps or command that reproduce it

These are the conditions under which the startup sizing was observed, not a claim that every dual-GPU machine will reproduce it:

1. Make both discrete RTX 3090 cards visible and use the existing IST-DAS 2.00bpw pack.
2. Start the official 0.1.51 engine through the source Python server at context 262144, leaving expert-store size, mmap mode, prefill chunk, CPU threads and expert count at their generated/automatic settings.
3. Observe the startup RAM-store sizing. This happened on both successful 0.1.51 starts, including a process restart: approximately 17.8 GB and 17.7 GB respectively.

Native invocation recorded by the server, with executable and pack paths replaced by placeholders:

<engine> --serve --pack <PACK_DIR> --max-context 262144 --eos 1 --dspark auto --dspark-conf 0.65 --tp 2 --layer-split auto

There was no explicit --ssd-cache or --mmap-experts setting in the tested model configuration. The native log selected 8192-token prompt chunks and 8 CPU threads. Both versions reported effective TP 2 -> 1 due to this pack's 256-value-block / n_ff 2304 constraint, with layers distributed over both GPUs; this was not a one-card run. DSpark auto did not load the drafter. Expert count was left automatic, with no faster-mode reduction message.

Reproduction status

Reproduced locally: the much smaller automatic RAM-store allocation on this discrete-GPU configuration, and the slower recorded workload. The source explanation below is consistent with the logs and hardware query, but total performance causality is not established.

Strata version

• Before: source v0.2.4.2; official DeepSeek engine 0.1.49.2, CUDA13.
• After: source v0.2.6 (4b9e05a2a806a3fe24f7b2560d98626c37dade35); official DeepSeek engine 0.1.51, declared build source 562a0c2ed7169ff6373f893b162106d55a828297, CUDA13.

Graphics card, memory and system

• 2 x NVIDIA GeForce RTX 3090, 24 GiB VRAM each.
• 64 GB-class installed RAM; approximately 62.7 GiB usable MemTotal.
• AMD Ryzen 7 5700X, Ubuntu 24.04.5 LTS, x86_64.
• NVIDIA driver 595.91.07 at reporting time; the old run did not separately record driver metadata.
• A read-only CUDA driver query returned CU_DEVICE_ATTRIBUTE_INTEGRATED = 0 for both cards.

Model, size and settings

• DeepSeek-V4.1-Flash-1017 GSQ-RCO, ISTA-DASLab IQ2_XXS / IST-DAS-2.00bpw set; same prepared model weights, not requantized.
• Context: 262144 in both comparisons; text-only requests, no image inputs.
• Sequential streaming workload, thinking off, temperature 0, top_p 1, seed 11. Initial long answers had a 256-token output allowance; follow-ups had 128. Short probe/side requests had 32.
• Same frozen initial public-document A/B prompts, with 245754 / 245784 input tokens in each version. Continuations used each run's actual assistant answers. The independent approximately 27K-token side prompt differed slightly between the two runs.
• 15 requests per version: two long conversations, continued turns, a short probe and an independent side conversation. This was a local cache-validation workload, not a standard strata bench run protocol submission.

Logs

Only the relevant numeric startup lines are included; no original log files, prompts, local paths, device identifiers or deployment metadata are attached.

0.1.49.2:

ds4: RAM store: up to 4672 slots of 10.3 MB (48.0 GB), policy 0, 8 reader threads, unbuffered reads
ds4: RAM store: 4416 of the profile's hottest experts read in (34.3 GB) in 10.0 s
ds4: 11152 routed experts on the CPU lane (96.1 GB, in place in the model's files), 8 threads

0.1.51:

ds4: RAM store: 56.9 GB of RAM free after the GPU's weights: store sized 17.9 GB, not 51.1
ds4: RAM store: up to 1737 slots of 10.3 MB (17.8 GB), policy 0, 8 reader threads, unbuffered reads
ds4: RAM store: 1481 of the profile's hottest experts read in (11.5 GB) in 3.3 s
ds4: 11150 routed experts on the CPU lane (96.1 GB, in place in the model's files), 8 threads

The store GB figures are slot-capacity ceilings, not measured RSS. The preload counts are useful evidence, but the printed preload GB values use an estimate based on the first layer's blob size; they are not exact unique quantized payload measurements for a mixed-format pack.

Suspected sizing mechanism

The installed build's post-upload RAM-sizing rule infers UMA from the drop in available RAM rather than the actual integrated-GPU attribute:

const bool uma = avail0_ != ~uint64_t{0} && avail0_ > avail && (double) (avail0_ - avail) / 1e9 >= 4.0;
const double ctx_gb = uma ? 1.1e-4 * std::max(0.0, (double) o.max_context - 16384.0) : 0.0;
const double room = (double) avail / 1e9 - 12.0 - ctx_gb;

At context 262144, the extra reserve is 27.0336 decimal GB. The recorded values match that branch:

• Initial automatic target 51.1 GB implies roughly 63.1 GB available under the initial available - 12 rule.
• After GPU-weight loading, the log says 56.9 GB available: an inferred drop of about 6.2 GB, exceeding the 4 GB threshold.
• 56.9 - 12 - 27.0336 = 17.8664 GB, matching the logged 17.9 GB target before slot rounding.
• Without that UMA-specific extra reserve, the same arithmetic gives about 44.9 GB, not a requirement to use the whole RAM pool.

This appears to apply the UMA-specific reservation to a machine whose two CUDA devices both report integrated = 0. Could this check use the actual memory architecture, or otherwise avoid treating ordinary host-memory changes during discrete-GPU loading as a UMA signal?

Recorded performance impact and limits

Prefill is newly read tokens / native prompt time, excluding reused tokens. Decode is generated tokens / native decode time, not generated tokens / whole-request time. First-content latency is measured at the streaming client.

Conversation KV reuse itself remained functional: both traces had 10/10 switched returns above 95% reuse, minimum approximately 99.985%, and 15/15 controlled identifier checks correct. The new run also restored long contexts after RAM-slot eviction and process restart. This does not measure expert-weight cache hit rate.

Important limitations:

• These are historical measurements on the same hardware, not an interleaved, contemporaneous or engine-only A/B; the Python source server also changed versions.
• Initial expert-profile contents, OS file-cache state, clocks/temperature and all background activity were not frozen as matched variables. The newer deployment also pinned physical-device selection explicitly and used a fresh versioned conversation-cache directory; matched physical GPU ordering was not established.
• The two cold long prompts are identical, but the full continued trace is not asserted bit-identical and the side prompt differs slightly. This is one 15-request session per version, not three independent repeats.
• The saved request histories have null expert hit-rate / RAM-file-byte fields; matched expert-store misses, waits and actual storage I/O were not captured. Missing measurements are not zeros.
• No same-version explicit 42/44/48 GB store experiment, forced-copy experiment, or native-code fix has been tested. There is no claim that increasing the store alone will recover all of the speed difference.

Related search found #3942, which concerns an older RAM-store limit on a different system; this report is specifically about the extra post-upload UMA reserve and the observed allocation reduction on discrete GPUs at 256K. No engine hang was observed in these measured requests.