csvkit vs OpenRefine

csvkit is a free suite of command-line tools for converting, inspecting and transforming CSV data on Linux, macOS and Windows. Its tools handle common file and table tasks: in2csv converts formats such as Excel, JSON and fixed-width files; csvcut selects, removes or reorders columns; and csvlook and csvstat display data and summary statistics. Other commands filter matching rows, sort records, join related data or stack subsets. For database workflows, csvsql can query CSV data with SQL and import it into PostgreSQL, while sql2csv exports query results. csvjson converts CSV to JSON or GeoJSON. Commands can read standard input and write standard output, making them suitable for repeatable pipelines. By default, csvkit infers format from the first 1024 bytes and infers cell types, with options to turn off either behavior. Installation instructions include pip and Homebrew and recommend a virtual environment. The tutorial assumes basic command-line familiarity, but not prior data-processing experience. The project notes that larger files or faster processing may exceed its limits and suggests SQL or other tools in that case.

OpenRefine is a free, open-source tool for cleaning, transforming, and extending messy data. It works on the user's machine rather than sending data to a cloud service, and basic functions can run offline through a small web server on that computer. Facets help explore large datasets and apply changes to filtered views. Clustering uses heuristics to find similar values and address inconsistencies; reconciliation matches values to external databases through reconciliation services. Users can undo and redo operations without a stated history limit. OpenRefine imports formats including CSV, TSV, fixed-width text, JSON, XML, spreadsheets, PC-Axis, MARC, RDF formats, and Wikitext. It can also upload data to Google Sheets and create or update Wikidata statements. Releases are available for Windows, Mac, and Linux. Java must be installed and configured, although the Mac version and some Windows packages include it. Data is not encrypted by default; operating-system full-disk encryption can protect it. The project offers support through its forum, GitHub, community calls, and Gitter chat.