Giters

beanumber / airlines

An R package providing access to medium airline flight delay data

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Problem setting up local MySQL database to store airlines data

trant22t opened this issue · comments

commented

I tried to set up a local MySQL database to store the airlines data. When I ran:

library(etl)
library(airlines)
db <- src_mysql_cnf(dbname = "airlines")
ontime <- etl("airlines", db = db, dir = "~/dumps/airlines")
ontime %>%
  etl_init() %>%
  etl_update(years = 2016)

R gave me the error message:
Error in download.file(url, method = method, ...) :
cannot download all files
In addition: Warning message:
In download.file(url, method = method, ...) :
URL 'http://tsdata.bts.gov/PREZIP/On_Time_On_Time_Performance_2016_10.zip': status was '404 Not Found'

I'm on a Mac and my R version is 3.3.1.

commented

I hope that there are ways to make the data loading a bit more robust. I worry that this command would download 30 years of data then bomb out because of the missing months in 2016.

I also wonder if the instructions might be modified to start with a less aggressive data load:

ontime %>%
etl_init() %>%
etl_update(years = 1987:2016)

Perhaps showing how to install a single (complete year, e.g., 2015) then add in the other years?

ontime %>%
etl_init() %>%
etl_update(years = 2015)

takes a long time!

ontime %>%
etl_update(years= c(1987:2014, 2016))

Additional note from NJH: the example now has more than 24GB!

There are over 300 months worth of files to download, and they will occupy more than 21 GB in their zipped and unzipped states.

commented

This is a known bug (#54). The problem is that we don't know which months of data are available on the server until we try to download them. I'm happy to field a pull request if you have a way for this function to fail gracefully.

Note that while the error breaks etl_extract(), it occurs after you have downloaded all of the valid data. So you can proceed by running etl_transform() and etl_load() in succession and these should still work.

commented