MaxMind database fetching is broken due to privacy change on MaxMind side

[ncla]

https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/

Requires a license key now.

[Dwight Watson]

Quick fix for those affected, get a licence key and update the update_url in your config.

'maxmind_database' => [
    'class' => \Torann\GeoIP\Services\MaxMindDatabase::class,
    'database_path' => storage_path('app/geoip.mmdb'),
    'update_url' => sprintf('https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key=%s&suffix=tar.gz', env('MAXMIND_LICENSE_KEY')),
    'locales' => ['en'],
],

[Eric Tendian]

@dwightwatson did that work for you? I tried it but since the file that gets downloaded is a .tar.gz, the resulting file is still a tar file, although decompressed by gzip already. Within that tar file is a folder, and within that is the actual .mmdb file.

I had to do the download with a shell script to get it working:

# Download the database tarball
EDITION_ID="GeoLite2-City"
LICENSE_KEY="REPLACE_ME"
curl -fSL "https://download.maxmind.com/app/geoip_download?edition_id=$EDITION_ID&license_key=$LICENSE_KEY&suffix=tar.gz" | \
# Extract only the file we need from it within the only dir in the tarball
tar xzf - --wildcards --no-anchored "*/$EDITION_ID.mmdb" --strip-components 1
# Rename to what we expect (this should be the same as database_path)
mv $EDITION_ID.mmdb geoip.mmdb

[Dwight Watson]

Ah, good catch - I had only tested that it was downloading but didn't actually check the end result.

Looks like just updating the suffix query parameter to gzip doesn't work.

Would it be as simple as calling gzdecode on the downloaded result?

[Eric Tendian]

@dwightwatson unfortunately not, I looked around but it seems the only option for the city DB is .tar.gz, here's what that looks like:

$ tar xf GeoLite2-City_20191224.tar.gz
$ ls GeoLite2-City_20191224
COPYRIGHT.txt  GeoLite2-City.mmdb  LICENSE.txt  README.txt

The file would need to be downloaded, then gzdecode() could work (which it seems this library is already doing), but then after that the file would need to be untarred, and then within the resulting directory copy GeoLite2-City.mmdb to whatever the database_path is.

[Dwight Watson]

Oh my bad, misunderstood which part was missing.

Looks like PharData::extractTo exists but I'm not sure if that's included with a standard PHP install, to the point we could rely on it here.

[Tyler]

I haven't had time to make this into a clean pull request yet, but this is some quick code I hacked out to support the tar.gz. Basically it is just adding a tar_file field to the config that looks for the file "GeoLite2-City.mmdb" and extracts it. You have to change the config's class value as well to map to this overridden MaxMindDatabase service implementation.

https://gist.github.com/tylermann/7322474072fc8dbc036739269d16c000

[Vernon Jacobs]

Hi, Any update on completing this pull request?

[Alex Powell]

@dwightwatson any update on this? #164 looks ready to be merged 🤞

[Dwight Watson]

I'm happy for it go to, just waiting on @Torann to review and release.