lm-eval run --model hf --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct --tasks gsm8k --apply_chat_template
Evaluate a chat-tuned model properly
Runs the GSM8K math benchmark with --apply_chat_template so prompts are formatted as chat turns, matching how instruct-tuned models were trained instead of raw text. Skipping this flag tanks scores on chat models. Add --num_fewshot 8 for the standard GSM8K few-shot setup.
Looking for more? Search all 7,657 commands — works offline, in English or Spanish, and fixes typos.