vllm serve TheBloke/Llama-2-7B-Chat-AWQ --quantization awq

Loads the AWQ-quantized Llama-2-7B weights with the matching --quantization awq flag, which vLLM needs because quantized files are not self-describing. AWQ (Activation-aware Weight Quantization) shrinks the model to 4-bit precision for lower VRAM use. The quantization value must match how the repo's weights were produced.