mlr --csv uniq -a data.csv

Category: Data Science & Analysis

mlr --csv uniq -a data.csv

Remove duplicate rows, comparing all fields

The uniq verb drops duplicate records; -a compares every field of the record. Use it after concatenating files to de-duplicate a dataset where whole rows repeat. Only the first occurrence of each unique row is kept. For duplicates based on a single key column, use -g instead.
Looking for more? Search all 7,657 commands — works offline, in English or Spanish, and fixes typos.