Android/iOS support
MFAshby opened this issue · comments
sqlite is the de-facto on-device database on Android [0], and it's also popular in iOS apps.
litestream might a be really useful and simple library for synchronizing data from a mobile app to a server, without having to change the app's persistence layer.
There's a couple of proprietary libraries [1] [2] in this space that I know of, but they do more than sync, and require using the library as data layer.
Since litestream is already cross-compiled for aarch64, the binary should already work out of a box. Due to Android's sandboxing, it's not usually possible for one app to access the database of another. Litestream would have to run as part of the app the database belongs to. Maybe a library could be created that wraps the existing executable, and invokes it at appropriate times, e.g. using the SyncAdapter [3]
[0] https://developer.android.com/training/data-storage/sqlite
[1] https://objectbox.io
[2] https://realm.io
[3] https://developer.android.com/training/sync-adapters
Just an idea.
If it is a golang android app it should work. litestream can be embedded in the golang project.
I think you'd be better looking at the sqlite changeset API and send database diffs yourself.
Problems I can think of immediately with using litestream on the client include
- possibly having to sync the entire database every time the WAL is truncated
- currently no security on transport layer by design
- no ability to limit to a subset of the database, e.g. just data that is relevant to a specific user
I think you'd probably be better served fetching the data from a server via a REST interface and propagating that yourself on the client. The changeset API allows you to easily send the diff between two databases as a binary blob and apply it to a copy of the original database.
@ralferoo what about this then ?
https://pkg.go.dev/zombiezen.com/go/sqlite#Changegroup
It seems to support the required functionality to use changesets.
Also the WAL in LiteStream does not require syncing the entire DB ?? Its incremental i thought...
I guess @benbjohnson will correct me if I'm wrong, but I think when the WAL file gets too big, the read lock is released to allow sqlite to compact it back into the main database, and I think at that point the entire database is re-sent... I could be wrong though as I haven't yet seen it happening!
@ralferoo thanks for the tip re changeset! I'll look into it.
Your example is the reverse of what I was thinking; my specific use case was syncing fitness tracking data recorded on the phone back to a server.
If there's no transport layer security, that could be problematic!
litestream might a be really useful and simple library for synchronizing data from a mobile app to a server
Litestream is good at replicating a primary database to another location (e.g. S3), however, it's not meant for synchronization. You wouldn't be able to make changes on the mobile app and the server. You would have to pick one. As @ralferoo mentioned, I think the session extension would be a better option as it could handle updates from multiple locations and merge them.
I think when the WAL file gets too big, the read lock is released to allow sqlite to compact it back into the main database
Yes, Litestream works incrementally and it'll manage the checkpointing to copy data from the WAL back into the main database. If Litestream is turned off for some reason while the SQLite database is being updated then you can get in a situation where the WAL is overwritten or truncated and, in that case, Litestream will have to re-snapshot the whole database.
currently no security on transport layer by design
There's no TLS on the live replication over HTTP, however, there's encryption on the replica clients (S3, GCS, MinIO, etc). I'm open to adding TLS to HTTP but it's just a first pass right now so I can add it later.