lm-eval run --model hf --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct --tasks mmlu --num_fewshot 5
Evaluate a model on MMLU
Runs the lm-evaluation-harness on the MMLU benchmark with the model loaded via Hugging Face transformers, giving 5 example answers per question (few-shot) before scoring. The pretrained= value is a Hugging Face model ID. Runs are long and download model weights first; expect many minutes to hours depending on GPU.
Looking for more? Search all 7,657 commands — works offline, in English or Spanish, and fixes typos.