RAM, VRAM and fit estimates

Three different claims

File size is storage. Memory admission is a coarse estimate against current resources. Successful generation is actual runtime evidence. Keep those claims separate. A file that fits on disk may not fit in RAM, and a plan that passes may still fail to allocate or lack a required kernel.

The current planner uses captured free memory, reserves 1.5 GiB on the first NVIDIA GPU and 6 GiB of host RAM, then estimates placement. These reserves are engineering assumptions, not measured overhead for every model.

Weight placement

For L transformer blocks and g selected GPU blocks, the estimate is approximately:

GPU weight estimate  = complete file bytes × g / L
Host weight estimate = complete file bytes − GPU weight estimate

Real layers are not uniform. Embeddings, output tensors, shared experts and routed experts can skew placement. The complete file also contains metadata and alignment. This arithmetic is useful for an initial gate, not a byte-accurate engine allocation report.

Attention state

For an ordinary full-attention model, a simplified FP16 K/V lower bound for one sequence is:

KV bytes = context × attention layers × KV heads
           × (key dimension + value dimension) × 2 bytes

This excludes graph buffers, compute workspaces, allocator overhead and additional state. Hybrid/recurrent models count their full-attention blocks separately and can require recurrent state beyond this lower bound. Missing metadata leaves the estimate unknown rather than inventing zero overhead.

Why an OOM can still happen

Other processes may allocate memory after the snapshot. Engine workspaces vary with architecture, batch shape, kernels and context. Host mappings may interact with resident memory differently from the simplified split. A model’s encoding can also be unsupported even with abundant memory.

Refresh before a deliberate load when the machine’s workload changes. Reduce GPU layers or context only as an explicit user choice. Moving more computation to RAM can admit a model while making token generation much slower. SSD paging is not treated as an admitted substitute for sufficient RAM.

For MoE models, expert residency is a separate strategy. Lemonade CPU-expert placement uses the complete parsed tensor-role inventory and conservatively budgets the full file in host RAM. The separate Strata planner, under native qualification, budgets a GPU expert cache, a calibrated state/workspace envelope and mapped-file plus pinned-arena host pressure. Neither passing gate is a generation or quality claim.