One memory pool, three observation scopes

One memory pool, three observation scopes An architecture diagram generated by Archify. SSE client · First content + completion · Architecture component SSE client First content + completion llama-server · Process RSS: resident pages · Architecture component llama-server Process RSS: resident pages Q4_K_M model file · 5.03 GB on the SSD · Architecture component Q4_K_M model file 5.03 GB on the SSD Model buffers · Same model in every condition · 16 GiB physical memory shared by CPU, GPU and system Model buffers Same model in every condition F16 KV buffer · Allocation grows with context · 16 GiB physical memory shared by CPU, GPU and system F16 KV buffer Allocation grows with context Compute buffers · Temporary runtime workspace · 16 GiB physical memory shared by CPU, GPU and system Compute buffers Temporary runtime workspace macOS + other apps · System compression / pressure · 16 GiB physical memory shared by CPU, GPU and system macOS + other apps System compression / pressure 127.0.0.1 mmap Metal execution shares physical RAM 16 GiB physical memory shared by CPU, GPU and system

Process scope

  • • RSS and Metal allocations overlap.
  • • Do not add the reported totals.

Runtime scope

  • • Model, KV and compute sizes come from logs.
  • • Boxes show roles, not allocation proportions.

System scope

  • • Compression and swap include other apps.
  • • Swap uses the SSD; it is not extra GPU RAM.