covid19india.org: cleaning data every day
In March 2020, as Covid-19 cases rose, there was no single, clean, machine-readable source of Indian case data. Every state published its numbers in its own way: daily bulletins as PDFs, images, press releases and social media posts, each with its own layout and definitions.
A group of volunteers built covid19india.org to fix that. A small core team read the official bulletins every day and entered the figures into shared spreadsheets. At its peak, more than 300 volunteers contributed. The result was one clean, consistent dataset with an open interface that researchers and journalists widely used.
The group stopped updating the site on 31 October 2021. A successor, incovid19.org, run by IIT Madras, the Centre for Networked Intelligence at IISc, the Indian Statistical Institute and Semantic Web India, continued the data from 1 November 2021.
- Turn many messy sources into one clean table with one set of rules.
- Check each day's load against the totals the sources report.
- Publish the clean table, not the mess.
Chapter 1: Why data arrives messy
Exports are made for systems, not for analysis. Learn the usual problems and you can fix any of them.
- Recognise the common problems in any export
- Diagnose a file in minutes, before touching it
- Decide between a quick fix and a recorded one
This article is adapted from the book. Examples marked “Example” use invented names and numbers; company stories are based on public sources listed in the book. For general learning; use your own judgement for your situation.
