Giters

0chain / blobber

A storage provider (blobber) interface to the blockchain and consumers of storage.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Improve file download locking mechanism

boddumanohar opened this issue · comments

commented

detailed discussion here:
https://0chain.slack.com/archives/GREHVKQKS/p1667645602453419

sometimes when I try to download I get this error:

0box-sdk        [DEBUG]  2022/11/05 11:52:09.762237 blockdownloadworker.go:206: Lock exists error.
0box-sdk        [DEBUG]  2022/11/05 11:52:09.765869 blockdownloadworker.go:206: Lock exists error.
 0 / 21008 [-----------------------------------------------------------------------------------------------------------------------------------------]   0.00%0box-sdk        [DEBUG]  2022/11/05 11:52:10.763496 blockdownloadworker.go:238: Retrying for Error occurred: lock_exists: lock_exists: lock exists for key: 65dbd6f4c9b5af90da402f78c0826d5301886fce58c0cb11c0bb4172b0a29e13:d87ebdf303f69f38e0d61eeb50579185af4731280761588e42c751deb0fa60f1
0box-sdk        [DEBUG]  2022/11/05 11:52:10.766197 blockdownloadworker.go:238: Retrying for Error occurred: lock_exists: lock_exists: lock exists for key: 65dbd6f4c9b5af90da402f78c0826d5301886fce58c0cb11c0bb4172b0a29e13:d87ebdf303f69f38e0d61eeb50579185af4731280761588e42c751deb0fa60f1
 0 / 4856 [------------------------------------------------------------------------------------------------------------------------------------------]   0.00%0box-sdk        [DEBUG]  2022/11/05 11:52:10.892057 blockdownloadworker.go:206: Lock exists error.
0box-sdk        [DEBUG]  2022/11/05 11:52:10.892061 blockdownloadworker.go:206: Lock exists error.
 0 / 21008 [-----------------------------------------------------------------------------------------------------------------------------------------]   0.00%^C

But lock_exists error is normal to have. At that point in time it was acquired by some key.

when we try to download a file, we acquire a lock here and the logic for lock and unlock can be found here

The lock is key-level locking. This lock does not occupy memory if not used. In unlock operation if respective key-lock is not acquired by any other process then it will delete its entry from the map.

as discussed in the thread, there are a fews to this logic can be improved.

commented

the goal of the ticket was to avoid this error:
lock_exists: lock exists for key: 65dbd6f4c9b5af90da402f78c0826d5301886fce58c0cb11c0bb4172b0a29e13

using the latest gosdk I tried this multiple times. Not able to reproduce the issue. It must have been fixed with the recent fixes.