MBtiles caching

[Eduard Popov]

Hello.
I host mbtiles with tilelive + mbtiles. When I update .mbtiles file, server returns an old tiles, even if I delete .mbtiles file. I can't find any info, how to prevent caching.
In the docs: file will be cached only if we use new sqlite3.cached.Database(), but you use new sqlite3.Database() (https://github.com/mapbox/node-mbtiles/blob/v0.9.0/lib/mbtiles.js#L59). So I can't understand, why the file is cached and how to prevent it.

[Tom MacWright]

You'll need to debug your setup: node-mbtiles does not cache output, but there's probably some part of your stack that does. Only you will have direct access to your configuration.

[Eduard Popov]

Here is a full reproduce

Setup

node -v # v6.9.1
npm install mbtiles@0.9.0
npm install @mapbox/tilelive@5.12.3

const MBTiles = require('mbtiles');
const tilelive = require('@mapbox/tilelive');
const http = require('http');

MBTiles.registerProtocols(tilelive);

tilelive.load('mbtiles://./sample.mbtiles', function (err, source) {
	if (err) throw err;

	http.createServer(function (req, res) {
		source.getTile(14, 14195, 6012, function (err, tile, headers) {
			if (err) {
				res.end(err.message);
			} else {
				res.end(tile);
			}
		});
	}).listen(1234);
});

Test

curl localhost:1234 # {pbf binary content} - OK
mv sample.mbtiles _sample.mbtiles
curl localhost:1234 # {pbf binary content} - FAIL

{reload server}

curl localhost:1234 # Tile does not exist - OK

[Teemu Sirkiä]

I noticed the same issue. I can remove the .mbtiles file but node-mbtiles is still able to serve the data. If this module does not do any caching, could it be the sqlite3 module?

[Teemu Sirkiä]

The problem is that I have another process which replaces the mbtiles file every now and then. However, the changes do not become visible before the serving process is restarted.

[Teemu Sirkiä]

SQLite documentation states: "The default suggested cache size is -2000, which means the cache size is limited to 2048000 bytes of memory." However, setting this cache size to zero did not change the behavior.

[Teemu Sirkiä]

Another point is that in Unix-systems, the database will still have the open handle for the removed file. So, as long the database is open, SQLite will have access to it although the file is deleted.

[Teemu Sirkiä]

OK, if I truncate the existing mbtiles files to be empty, I can confirm that the data will disapper regardless of the cache size. Updating the mbtiles file requires truncating the existing file and then appending new data to that file.

[Eduard Popov]

My workaround is:

setInterval(function () {
	tilelive.load(mbtResources['traffic'], function (err, source) {
		if (err) throw err;
		mbtSources['traffic'] = source;
	});
}, 60 * 1000);

Remaking traffic data by tippecanoe every minute by another process.