Giters

neosmart / SqliteCache

An ASP.NET Core IDistributedCache provider backed by SQLite

Home Page:https://neosmart.net/blog/2019/sqlite-cache-for-asp-net-core/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Adding bulkinsert support

Rap22tor opened this issue · comments

commented

Hi

I've recently run into a problem where I have to insert a large list of objects/keys into the cache database. Using the existing Set and SetAsync methods, this obviously ran pretty slowly. I now have added a SetBulkAsync method, which reduces the time consumed significantly. In my case, it improved a process that took around ~25 mins to about 25 seconds.
This method takes in a IEnumerable of KeyValuePairs and produces a single Sqlite query. This query is then executed just like in SetAsync. In my program I can now seperate my list into smaller chunks and use SetBulkAsync to insert them pretty quickly.

I hope this is useful for others too. Let me know if you prefer the solution any different, I'll gladly adapt.

Thanks for this library!
Elias

commented

Hello Elias,

Thanks for stopping by and for your feedback! The latest version of this package (5.0.0-preview2) changes the way the connection is opened and shared between the threads to use a shared cache and a write-ahead log, this should substantially improve the performance as compared to the latest non-preview release.

Are you able to compare the performance of the latest preview release to that of the previous release with your changes?

commented

Hi, thanks for the quick reply

I had indeed been using an older version of the package. I updated the times did drop significantly. Though I was still able to measure around a 3x improvement with my method (1 minutes and 7 seconds vs 22 seconds). I used the System.Diagnostics.Stopwatch to measure and the dataset was the same (about 90k objects).
Although my problem would be basically fixed with the new version, could a bulk insert still be of use?

commented

Hello @mqudsi

I wanted to ask briefly if this is still being considered to be in a new version or what the status is on this issue. More testing on our side showed that this bulk insert option would still be of much use. If there will be a update do you have any estimate when it will be available?

commented

Honestly, I haven’t moved on this because I was a bit paralyzed as to whether it would be of use or not, given your earlier statements about the latest release performing well enough. Your update provides the missing information to make a decision :)

I’ll have a look at the PR, but tentatively I don’t see why we can’t add this if you (and/or others) think it would still be beneficial.

commented

I would want the changes.

commented

@gutzi can you provide some information about what you'd use this for/how you were previously bottlenecked without it?

I've just released 5.0-preview3 on nuget that includes this feature, among others. @Rap22tor: you're named in the changelog, let me know if you prefer the credit worded differently.

commented

Awesome, thank you!