mezzario / geonames-data-import

Cross-platform app to automate downloading of GeoNames database dumps and importing them to MySQL.

geonames-data-import

Cross-platform console application written in Node.js to automate downloading/unzipping of GeoNames worldwide geographical database dumps and importing them to database of choice (MySQL is supported at the moment).

Demo

(some large files were omitted)

Usage

• Install Node.js and Docker (if you plan to run MySQL as a container).

• Clone git repository:

git clone https://github.com/mezzario/geonames-data-import.git

• From app's folder install node modules:

npm i

• Edit src/config.js to adjust configuration, if needed (see below).

• Start MySQL instance (wait for it to load):

npm run mysql

• Run application in another terminal window:

npm start

To automatically run additional SQL queries after import, please refer to file:

assets/db/<db-type>/post-import.sql

Configuration

Edit src/config.js to adjust app's configuration:

{
  // base URL do download files from
  baseUrl: "http://download.geonames.org/export",

  // list of file paths to download (relative to 'baseUrl')
  dataFilePaths: [...],

  // directory where files will be downloaded
  localDownloadDir: "download",

  // local directory with files missing on remote server
  localDataFilesDir: "assets/data",

  // db queries to create, import or drop data
  localDbAssetsDir: "assets/db",

  // currently selected db type to import data to
  dbType: "mysql",

  // settings for MySQL db
  mysql: {
    // connection params; please see complete list of options here:
    // https://github.com/felixge/node-mysql/#connection-options
    connection: {...},

    // db name to import data to
    databaseName: "geonames",
  },

  // set to true to overwrite already downloaded files
  forceDownloading: false,

  // action to perform if DB already exists:
  // - ClearDbAction.None: error will be raised if db exists
  // - ClearDbAction.Drop: db will be dropped before import
  // - ClearDbAction.Truncate: all db tables will be truncated
  actionIfDbExists: ClearDbAction.None,
}

Credits

• GeoNames-MySQL-DataImport for basic SQL scripts;
• node-status-bar for great status bar on long-running operations.