| date | location | location_type | location_code | location_code_type | data_type | value |
|---|---|---|---|---|---|---|
| 2021-01-23 | Alabama | state | 01 | fips_code | cases_total | 439442 |
| 2021-01-23 | Alabama | state | 01 | fips_code | deaths_total | 6657 |
| 2021-01-23 | Alaska | state | 02 | fips_code | cases_total | 52820 |
| 2021-01-23 | Alaska | state | 02 | fips_code | deaths_total | 250 |
| 2021-01-23 | Arizona | state | 04 | fips_code | cases_total | 717593 |
| 2021-01-23 | Arizona | state | 04 | fips_code | deaths_total | 12170 |
Marine Data Science for All, Regardless of Backbone!
The Covid19R Project
R
covid19R
We want our coronavirus data! We want it now! We want it TIDY!
tl;dr Go check out the Covid19R project and it’s documentation - a way for many people to bring their unique data harvesting efforts of Covid-19 data into a single tidy data format, and then be redistributed through a single low-dependency package, covid19R. It’s all one big work-in-progress, but check it out!
OK, the story!
Back in February (remember February), while teaching data science, I was trying to think of ways to spice up the class with more relevant data. We were in the midst of data tidying, when I saw a tweet from Rami Krispin about starting a Covid R data package (whose account as of 2022-11-27 is suspended?)
Cool! A coronavirus data package. I went on to make a neat data tidying lab - well, I think it’s neat - as well as weaving the data into other parts of the course. I also made a few pull requests on the repo to add features….as did Amanda Dobbyn. Rami contacted both of us, and thus a collaboration was born!
Tidy Covid-19 Data
We had been pulling from many different data sources, and found lots of difficulty in bringing them together. I had used rnaturalearth to get at unique place identifiers, but the proliferation of data sets and data aggregation efforts was bewildering. How to get a handle on the wide variety of data types, spatial location types, and more?
So, we began designing a very simple long format data specification and agreed to make our data packages create data in that format. I wrote up a package to harvest the updated New York Times data as an example.
One Package to Rule Them All
This was good - but, how would people get data?
We noticed that with nearly every different data set, we needed a different approach to data manipulation. If we threw all of our efforts into a single package, the dependency tree would rapidly spiral out of control. This seemed not good.
Many small packages, each specializing in unique data sets or collections of data sets seemed good, but…. how to make all of that searchable? Did we really want to make people have to install a bazillion different small packages? This seemed inefficient.
So, we took a cue from the many excellent packages by the good folk at ropensci and created a master package - covid19R. This package wouldn’t do anything, save redistribute data files. Moreover, so it didn’t have to depend on all of those little packages, and thus have all of those same problems as above, we built a separate covid19Rdata package/repo that updates once daily to refresh all of the data from any package that is part of the project.
So, you can get the information about what data is part of this Tidy Covid-19 data project, choose a relevant data set, and grab it, all without racking having to download tons of packages, satisfy weird dependencies, or otherwise pull your hair out!
Here’s what’s part of the project at time of writing
| data_set_name | package_name | function_to_get_data | data_details | data_url | license_url | data_types | location_types | spatial_extent | has_geospatial_info | get_info_passing | refresh_status | last_refresh_update |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| covid19nytimes_states | covid19nytimes | refresh_covid19nytimes_states | Open Source data from the New York Times on distribution of confirmed Covid-19 cases and deaths in the US States. For more, see https://www.nytimes.com/article/coronavirus-county-data-us.html or the readme at https://github.com/nytimes/covid-19-data. | https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-counties.csv | https://github.com/nytimes/covid-19-data/blob/master/LICENSE | cases_total, deaths_total | state | country | FALSE | TRUE | Passed | 2021-01-24 16:57:46 |
| covid19nytimes_counties | covid19nytimes | refresh_covid19nytimes_counties | Open Source data from the New York Times on distribution of confirmed Covid-19 cases and deaths in the US by County. For more, see https://www.nytimes.com/article/coronavirus-county-data-us.html or the readme at https://github.com/nytimes/covid-19-data. | https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-counties.csv | https://github.com/nytimes/covid-19-data/blob/master/LICENSE | cases_total, deaths_total | state | country | FALSE | TRUE | Passed | 2021-01-24 16:58:02 |
| covid19france | covid19france | refresh_covid19france | Open Source data from opencovid19-fr on distribution of confirmed Covid-19 cases and deaths in the US States. For more, see https://github.com/opencovid19-fr/data. | https://raw.githubusercontent.com/opencovid19-fr/data/master/dist/chiffres-cles.csv | https://github.com/opencovid19-fr/data/blob/master/LICENSE | confirmed, dead, icu, hospitalized, recovered, discovered | county, region, country, overseas collectivity | country | FALSE | TRUE | Passed | 2021-01-24 16:58:07 |
| CanadaC19_cases | CanadaC19 | refresh_CanadaC19_cases | Open Source data from multiple public reporting data throughout Canada. For more, see https://github.com/ishaberry/Covid19Canada. | https://raw.githubusercontent.com/ishaberry/Covid19Canada/master/cases.csv | https://github.com/debusklaneml/CanadaC19/blob/master/LICENSE | cases_new | state | state | FALSE | TRUE | Passed | 2021-01-24 16:58:17 |
| covid19us | covid19us | refresh_covid19us | Open Source data from COVID Tracking Project on the distribution of Covid-19 cases and deaths in the US. For more, see https://github.com/opencovid19-fr/data. | https://covidtracking.com/api | https://github.com/aedobbyn/covid19us/blob/master/LICENSE.md | positive, negative, pending, total_test_results_source, total_test_results, hospitalized_currently, hospitalized_cumulative, in_icu_currently, in_icu_cumulative, on_ventilator_currently, on_ventilator_cumulative, recovered, death, hospitalized, total_tests_viral, positive_tests_viral, negative_tests_viral, positive_cases_viral, death_confirmed, death_probable, total_test_encounters_viral, total_tests_people_viral, total_tests_antibody, positive_tests_antibody, negative_tests_antibody, total_tests_people_antibody, positive_tests_people_antibody, negative_tests_people_antibody, total_tests_people_antigen, positive_tests_people_antigen, total_tests_antigen, positive_tests_antigen, positive_increase, negative_increase, total, total_test_results_increase, death_increase, hospitalized_increase, negative_regular_score, negative_score, positive_score | state | country | FALSE | TRUE | Passed | 2021-01-24 16:58:33 |
| VirginiaC19 | VirginiaC19 | refresh_VirginiaC19 | Open Source data from Virginia Department of Health. For more information, see https://www.vdh.virginia.gov/coronavirus/. | https://www.vdh.virginia.gov/content/uploads/sites/182/2020/03/VDH-COVID-19-PublicUseDataset-Cases.csv | https://github.com/debusklaneml/VirginiaC19/blob/master/LICENSE | cases_new,deaths_new,hosp_new | county | state | FALSE | TRUE | Passed | 2021-01-24 16:58:33 |
| covid19tunisia | covid19tunisia | refresh_covid19tunisia | Open Source data on distribution of confirmed Covid-19 cases, recovered ones and deaths in Tunisia. For more, https://github.com/MounaBelaid/covid19datatunisia | https://raw.githubusercontent.com/MounaBelaid/covid19datatunisia/master/dist/data.csv | https://github.com/MounaBelaid/covid19datatunisia/blob/master/LICENSE | cases_new, deaths_new, recovered_new | state | country | FALSE | TRUE | Passed | 2021-01-24 16:58:33 |
| covid19mobility_apple_country | covid19mobility | refresh_covid19mobility_apple_country | Data reflects relative volume of directions requests compared to a baseline volume on January 13th, 2020 for multiple transportation modes aggregated at the country level. | https://www.apple.com/covid19/mobility | https://www.apple.com/covid19/mobility | driving, walking, transit | country | global | FALSE | TRUE | Passed | 2021-01-24 16:58:36 |
| covid19mobility_apple_subregion | covid19mobility | refresh_covid19mobility_apple_subregion | Data reflects relative volume of directions requests compared to a baseline volume on January 13th, 2020 for multiple transportation modes aggregated at the subregion (state) level. | https://www.apple.com/covid19/mobility | https://www.apple.com/covid19/mobility | driving, walking, transit | state | global | FALSE | TRUE | Passed | 2021-01-24 16:58:59 |
| covid19mobility_apple_city | covid19mobility | refresh_covid19mobility_apple_city | Data reflects relative volume of directions requests compared to a baseline volume on January 13th, 2020 for multiple transportation modes aggregated at the city level. | https://www.apple.com/covid19/mobility | https://www.apple.com/covid19/mobility | driving, walking, transit | city | global | TRUE | TRUE | Passed | 2021-01-24 16:59:11 |
| covid19mobility_google_country | covid19mobility | refresh_covid19mobility_google_country | Changes for each day are compared to a baseline value for that day of the week as compared to the 5-week period Jan 3-Feb 6, 2020 for visits to places falling in to certain categories. | https://www.google.com/covid19/mobility/ | https://www.google.com/covid19/mobility/ | retail_and_recreation_percent_change_from_baseline,grocery_and_pharmacy_percent_change_from_baseline,parks_percent_change_from_baseline,transit_stations_percent_change_from_baseline,workplaces_percent_change_from_baseline,residential_percent_change_from_baseline | country | global | FALSE | TRUE | Passed | 2021-01-24 16:59:31 |
| covid19mobility_google_subregions | covid19mobility | refresh_covid19mobility_google_subregions | Changes for each day are compared to a baseline value for that day of the week as compared to the 5-week period Jan 3-Feb 6, 2020 for visits to places falling in to certain categories. Data is aggregated at the state or subdivision level. | https://www.google.com/covid19/mobility/ | https://www.google.com/covid19/mobility/ | retail_and_recreation_percent_change_from_baseline,grocery_and_pharmacy_percent_change_from_baseline,parks_percent_change_from_baseline,transit_stations_percent_change_from_baseline,workplaces_percent_change_from_baseline,residential_percent_change_from_baseline | state | global | FALSE | TRUE | Passed | 2021-01-24 17:00:12 |
| covid19mobility_google_us_counties | covid19mobility | refresh_covid19mobility_google_us_counties | Changes for each day are compared to a baseline value for that day of the week as compared to the 5-week period Jan 3-Feb 6, 2020 for visits to places falling in to certain categories. Data is aggregated at the county level for the USA only. | https://www.google.com/covid19/mobility/ | https://www.google.com/covid19/mobility/ | retail_and_recreation_percent_change_from_baseline,grocery_and_pharmacy_percent_change_from_baseline,parks_percent_change_from_baseline,transit_stations_percent_change_from_baseline,workplaces_percent_change_from_baseline,residential_percent_change_from_baseline | county | country | FALSE | TRUE | Passed | 2021-01-24 17:02:45 |
| coronavirus_jhu | coronavirus | refresh_coronavirus_jhu* | The 2019 Novel Coronavirus COVID-19 (2019-nCoV) Dataset from the Johns Hopkins University Center for Systems Science and Engineering | https://systems.jhu.edu/research/public-health/ncov/ | https://github.com/CSSEGISandData/COVID-19/ | cases_new, recovered_new, deaths_new | country, state | global | TRUE | TRUE | Passed | 2021-01-24 17:02:48 |
What is a Covid19R Project Package?
We have a few requirements for a package to be added to our project. First, file an issue that you’re creating one! Then, take a gander at our documentation on building packages. We’ve provided a template for you to use, if you’re new to package building.
We then ask that you provide at least two functions in your package - a get_info_*() function, where * is the package name, that supplies standard information that we chonk into the table y’all see above. And for each data set a refresh_*() function, where * is the data set name, that pulls in tasty fresh data and reformats it into our tidy Covid-19 data standard using an ever-expanding standardized vocabulary - and you can file issues to keep expanding it.
Our request to the community
We’d love this to become a much larger community - have existing Covid-19 data package developers implement our two simple functions so we can add their data to the project, provide support for people interested in creating data distribution packages - maybe their first package ever - to use our format so that data is good and tidy. We’re very interested in conversation - start one up here for the moment!
Our Hope
What we hope is that, with many datasets going in using common best practices, we’ll be able to facilitate discovery. Data cleaning and joining is sometimes 80-90% of the effort of any data science project. With a standard format, a standardized vocabulary, and the usage of standard geospatial codes, we hope to seriously reduce that time in order to speed up other work.
Won’t you join us?