any plans on an async interface?

[bnewbold]

Hi! I'm curious if there is any intention of adding an async/await interface to this library. The docs say "For blocking usage, use BlockStore", which I read as there might be.

I'm not sure if this would make any sense building on top of rusqlite, but I thought I would ask. An alternative I was looking at was iroh, which seems to have an async blockstore internally, built on RocksDB. But that project has not published any crates yet, and does not seem as mature as this one, at the present moment.

The project i'm currently working on is an implementation of the bluesky atproto "personal data server", which needs a blockstore. I've gotten things working with a blocking API, but it feels like the sort of project/service which would benefit from async networking and I/O.

[Roland Kuhn]

That’s a cool project @bnewbold, I might want to have a look and possibly contribute!

Regarding this block store: sqlite is fundamentally synchronous, so there is nothing to be gained from making this BlockStore use async/await. This doesn’t mean that the rest of the server needs to be synchronous, too. My preferred solution would be using the Actor model, but there is no reasonable implementation in Rust yet (perhaps I’ll have to fix that), so it boils down to:

• spawn a thread with the block store in it, running a synchronous request–response loop
• send requests to that thread via an mpsc channel (flume or async-channel look good)
• send responses to a oneshot channel that was contained in the request message — this gives you .await

[bnewbold]

Ok, thanks for the response. Right now i'm using a simple Mutex around the blockstore.

Here is my work in progress: https://gitlab.com/bnewbold/adenosine

[bnewbold]

I have a couple other questions / feature requests. I can move these elsewhere if that would be better:

• compatibility with distributed sqlite systems like https://fly.io/blog/introducing-litefs/. I don't have any current plans to deploy or work with such a system, but seems like it would be a nice-to-have
• ability to share sqlite database with other application tables. eg, get a rusqlite handle to the underlying database, or construct a block store from an existing handle. acknowledging that the blockstore configuration might have specific needs (WAL, pragmas, etc). probably not a good idea for all applications, but maybe useful for some to keep it simple?
• more sophisticated IPLD graph support, eg graph "walkers". this is probably the sort of thing that should be layered on top of the blockstore, not implemented in it directly... is this something that already exists in the Rust/IPLD ecosystem? eg, graph query by path syntax, or ability to recursively walk graph nodes with some filter/predicate to ignore some sub-graphs.

[Roland Kuhn]

Sharing sqlite is done by opening multiple Connections, so I don’t think there is a necessity to add an API in the block store (this is also done in ipfs-embed to run GC).

Regarding LiteFS: that doesn’t seem to require much in terms of support, but it is not clear to me why this particular block store should care — IPFS is peer-to-peer after all, with mechanisms to move data between nodes, so why should the local block store also move data between nodes?

Concerning graph walkers I agree with you that they should probably live in a layer atop IPLD & block store.