lm-eval run --model hf --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct --tasks truthfulqa_mc2 --device cuda:0

Category: AI/ML Tooling

lm-eval run --model hf --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct --tasks truthfulqa_mc2 --device cuda:0

Evaluate on TruthfulQA with a device

Scores the model on truthfulqa_mc2, a benchmark measuring whether answers are both true and informative, while pinning inference to GPU 0 with --device cuda:0. Use --device cpu to force CPU execution on machines without CUDA. Results print as a table with acc and other metrics per task.
Looking for more? Search all 7,657 commands — works offline, in English or Spanish, and fixes typos.