Saving to memory instead of disk

[Benjamin Sergeant]

Hi there,

How could the library be extended so that we just download the torrent but don't write it to disk, but to a memory buffer instead ?

Would the idea be to make a 'memory-storage' out of this class/code ?

https://github.com/cenkalti/rain/tree/master/internal/storage

Thanks !

[Cenk Altı]

Yes, that would be great. I think the interface is simple enough to be implemented in memory.

[Cenk Altı]

This may also include some overlapping work for #19 .

[Cenk Altı]

This project might worth checking out: https://github.com/spf13/afero

[Cenk Altı]

Another resource: https://benjamincongdon.me/blog/2021/01/21/A-Tour-of-Go-116s-iofs-package/

[Benjamin Sergeant]

Thanks for all the pointers. I am trying to debug an issue where the filesystem cache is being ‘abused’, and was hoping that writing to memory would help. But I have switched the transport for my program to http and the problem persist so it might be something else. That being said having different storage (or just using afero as the main one) could be a good thing.

…

On Oct 8, 2021, at 8:45 PM, Cenk Altı ***@***.***> wrote: This may also include some overlapping work for #19 <#19> . — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#29 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC2O6UOCRKAWHKZ3WETLLUDUF63F7ANCNFSM5FO2SBQA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[Cenk Altı]

I would like to help on debugging the issue. Can you elaborate how it is being abused? Is it read or write cache that is being abused? What is the access pattern?

I'd like to give some information about the current state of the read cache.

rain/torrent/config.go

Lines 146 to 153 in 4bda7a0

 // Number of bytes to read when a piece is requested by a peer. 
 ReadCacheBlockSize int64 
 // Number of cached bytes for piece read requests. 
 ReadCacheSize int64 
 // Read bytes for a piece part expires after duration. 
 ReadCacheTTL time.Duration 
 // Number of read operations to do in parallel. 
 ParallelReads uint 

When a piece is requested from Rain, it assumes that the peer will request blocks from the same piece again. For that reason, Rain reads more than the actual requested block size. It divides the piece in larger blocks of size ReadCacheBlockSize and read all at once.

Example:
Let's assume that torrent piece size is 256K peer will request blocks in size of 16K.

0123456789ABCDEF

If ReadCacheBlockSize is 128K (default value), from the perspective of read cache, torrent piece is divided to 128K blocks.

01234567  89ABCDEF

In ideal scenario, the peer starts requesting peers from the start in order. When the first block (0) is requested, Rain reads the range between 0 and 7 into memory and caches it. The following requests from 1 to 7 are served directly from cache without going to disk. For the next range between 8 and F, a second read is made from the disk.

[Cenk Altı]

If peer requests pieces and blocks completely random, then the read cache will not be useful. If that is the case, you can try increasing ReadCacheSize.

[Benjamin Sergeant]

I might give this a try, but I noticed that my disk problem was completely unrelated to this library, so feel free to close this ticket, or use it for reference.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.