CSCD01 Engineering Large Software Systems
A look into how LLMs are Hosted
Cho Yin Yong / Aleksander Bodurri

Large Language Models

A large language model (LLM) is a set of files that store a neural network's trained weights. Weights are the billions of numbers that training adjusts, and inference multiplies inputs by them to compute the output. Inference requires separate software that loads these files.
config.json
tokenizer.json
model.safetensors

Quantization

Quantization stores each weight in fewer bits, so the model uses less memory and serves more requests at some cost in accuracy.

Throughput and scores from a Qwen3 32B benchmark

Precision Weights per 1B parameters Throughput MMLU Pro HumanEval
16 bit 2 GB 1.0x 70.2% 39.0%
8 bit 1 GB 1.5x 69.6% 39.0%
4 bit 0.5 GB 2.7x 68.7% 31.1%

Full precision is technically 32 bits, but 16 bits loses almost no accuracy, so people loosely call 16 bits full precision.

Open Weight Model Names

Qwen 3.8
Model and version
27B
27 billion parameters
Q4_
4 bit weights
K_
Quantization method
M
Medium mix, key layers at 6 bits
GGUF
File format with weights and metadata

Other common labels are Chat or Instruct for a fine tune and A3B for 3B active parameters.

Dense and MoE Models

A dense model uses every weight at each step. In a mixture of experts (MoE) model, a router picks a few expert subnetworks for each step.

Token

A token is the model's unit of input and output and typically encodes a word or word fragment.

LLM Runtimes

An LLM runtime loads model weights onto hardware and executes inference.

What does this look like?

LLMs are Stateless

A stateless model retains no state between requests, so each request carries the full conversation history.

Single GPU Scaling Limits

So then why aren't the LLMs we use every day incredibly slow?

Request Lifecycle Stages

Prefill processes the prompt and stores its attention keys and values in the KV cache. Decode generates the reply.

Nodes and Pools

A node is a machine with multiple GPUs. Nodes serving the same stage form a pool, and each node groups its concurrent requests into a batch.

Kimi K3

  • Moonshot AI released Kimi K3 as an open weights LLM in July 2026.
  • The model has 2.8 trillion parameters and a 1M token context window.
  • Moonshot serves it through the Kimi chat service.

Request Flow at Kimi

Prefix Caching

A block is a KV cache segment of fixed size. This diagram reduces Kimi's 512 token blocks to 1 token each.

Memory Tiers

A tier is one level of the storage hierarchy, and faster tiers have higher cost and lower capacity.

Kimi's Scheduler

A scheduler assigns each request to a prefill node and a decode node.

KV Cache Placement at Kimi

The cache pool is spare CPU memory and SSD space across Kimi's GPU machines.

Complete Request Path at Kimi