Multiple readers and writers under concurrent hash map not working.
Jiboxiake opened this issue · comments
Hello, I'm the guy who posted about the experiment on Reddit. https://www.reddit.com/r/cpp/comments/z6bn4n/my_own_concurrent_hash_map_picks/ Currently I'm using the if_countain_unsafe function with a lambda that atomically modifies a field of the "value.
Exception: EXC_BAD_ACCESS (code=1, address=0x0)
The call stack is:
std::equal_to::operator()(const unsigned long long &, const unsigned long long &) const operations.h:413 phmap::EqualTo::operator()(const unsigned long long &, const unsigned long long &) const phmap_base.h:79 phmap::priv::raw_hash_set::EqualElement::operator()<…>(const unsigned long long &, const std::piecewise_construct_t &, std::tuple<…> &&, std::tuple<…> &&) const phmap.h:1877 phmap::priv::memory_internal::DecomposePairImpl<…>(phmap::priv::raw_hash_set<…>::EqualElement<…> &&, std::pair<…>) phmap.h:751 phmap::priv::DecomposePair<…>(phmap::priv::raw_hash_set<…>::EqualElement<…> &&, std::pair<…> &) phmap.h:4122 phmap::priv::NodeHashMapPolicy::apply<…>(phmap::priv::raw_hash_set<…>::EqualElement<…> &&, std::pair<…> &) phmap.h:4357 phmap::priv::hash_policy_traits::apply<…>(phmap::priv::raw_hash_set<…>::EqualElement<…> &&, std::pair<…> &) phmap_base.h:518 phmap::priv::raw_hash_set::find_impl<…>(const unsigned long long &, unsigned long, unsigned long &) phmap.h:1843 phmap::priv::raw_hash_set::find_ptr<…>(const unsigned long long &, unsigned long) phmap.h:1750 phmap::priv::parallel_hash_set::find_ptr<…>(const unsigned long long &, unsigned long, phmap::LockableBaseImpl<…>::DoNothing &) phmap.h:3740 phmap::priv::parallel_hash_set::modify_if_impl<…>(const unsigned long long &, <lambda> &&) phmap.h:3256 phmap::priv::parallel_hash_set::if_contains_unsafe<…>(const unsigned long long &, <lambda> &&) const phmap.h:3238 Parallel_Node_Hashmap_Bench::reader(phmap::parallel_node_hash_map<…> (&)[4]) Parallel_Hashmap_Bench.hpp:48 decltype(static_cast<void (*>(fp)(static_cast<std::__1::reference_wrapper<phmap::parallel_node_hash_map<unsigned long long, Transaction_Table_Entry, phmap::Hash<unsigned long long>, phmap::EqualTo<unsigned long long>, std::__1::allocator<std::__1::pair<unsigned long long const, Transaction_Table_Entry> >, 4ul, phmap::NullMutex> [4]>>(fp0))) std::__1::__invoke<void (*)(phmap::parallel_node_hash_map<unsigned long long, Transaction_Table_Entry, phmap::Hash<unsigned long long>, phmap::EqualTo<unsigned long long>, std::__1::allocator<std::__1::pair<unsigned long long const, Transaction_Table_Entry> >, 4ul, phmap::NullMutex> (&) [4]), std::__1::reference_wrapper<phmap::parallel_node_hash_map<unsigned long long, Transaction_Table_Entry, phmap::Hash<unsigned long long>, phmap::EqualTo<unsigned long long>, std::__1::allocator<std::__1::pair<unsigned long long const, Transaction_Table_Entry> >, 4ul, phmap::NullMutex> [4]> >(void (*&&)(phmap::parallel_node_hash_map<unsigned long long, Transaction_Table_Entry, phmap::Hash<unsigned long long>, phmap::EqualTo<unsigned long long>, std::__1::allocator<std::__1::pair<unsigned long long const, Transaction_Table_Entry> >, 4ul, phmap::NullMutex> (&) [4]), std::__1::reference_wrapper<phmap::parallel_node_hash_map<unsigned long long, Transaction_Table_Entry, phmap::Hash<unsigned long long>, phmap::EqualTo<unsigned long long>, std::__1::allocator<std::__1::pair<unsigned long long const, Transaction_Table_Entry> >, 4ul, phmap::NullMutex> [4]>&&) type_traits:3918 std::__thread_execute<…>(std::tuple<…> &, std::__tuple_indices<…>) thread:287 std::__thread_proxy<…>(void *) thread:298 <unknown> 0x000000019405206c
I have attached my code. I think the code itself is fairly simple and straightforward. Hope to hear back from you some time later. Thanks so much!
Concurrent Hash Map Test.zip
Hi @Jiboxiake , I have updated your benchmark... it now works well. Instead of an array of hash maps, I use a single parallel hashmap.
Concurrent_Hash_Map_Test.zip
Hello sir. Thank you and I will look into that.
Hello Greg, thank you for all the help. I understand it now, but I do have questions about it. As you can see my readers, although use atomic operations on the transaction entries, do not really have data race with other threads. So I'm wondering if such modification can take place without having a write lock protection on it? Also in the future, I will want readers to be able to remove entries from the map. From the programming logic I will ensure when a delete takes place, no concurrent reads to that key-value pair will happen. Do you have suggestions on what I can do to use parallel hashmap to do that? Thanks.
It is possible to do that, but it is error-prone. Using a parallel hashmap with a large number of sub-tables as I did in the example (template parameter = 12, so 4096 sub tables), the contention should be reduced to a minimum. What did you think of the performance as in the example I sent? You can increase NUM_THREADS if you have a processor with many cores.
I will ensure when a delete takes place, no concurrent reads to that key-value pair will happen
Even so, if other entries are added to the map in other threads, this is not safe, because the map can be resized (and moved in memory) when new entries are added.
Hello Greg, thank you for all the help. I see your first point now. I will look into how I want to proceed with my tasks. But I do have a question about the second point. If what I mentioned is not safe, what will be the ideal way to do concurrent deletions and insertions of different key-value pairs in a map? Should it be something like modify_if() and include the erase operation within it? Moreover, I'm wondering if there is a lock upgrade or downgrade protocol. Thanks.
I don't quite understand the question. Why not use the code I suggested, it does concurrent deletions and insertions of different key-value pairs. Is there something that you don't like about it?
Should it be something like modify_if() and include the erase operation within it?
No, probably not a good idea. Why would you want to do that?
Hello thank you! Sorry for the confusion I brought up. But I don't think it involves deletion/erase in the solution you suggested. Is there something I missed? Let's say in my future actual system's implementation, I want the reader who decrements the op_count to 0 to remove the entry. Is simply invoking .erase() safe? Thanks.
yes, erase is safe. You can do it like this:
bool erase = false;
txnTables.modify_if(
txnID, [&erase](typename Map::value_type& pair) { erase = (pair.second.op_count.fetch_sub(1) == 0); });
if (erase)
txnTables.erase(txnID);
Thank you! Generally speaking, will multiple concurrent erases of the same key still be safe? Not trying to be picky but to me making erase outside of modify_if's write lock seems a bit dangerous.
Generally speaking, will multiple concurrent erases of the same key still be safe?
yes, absolutely safe! However is it possible that another thread would insert the same key txnID between the modify_if and the erase call? If that's possible use erase_if instead.
Thank you! You helped me a lot. I will later also do a benchmark on the flat hash map and let you know if I find anything interesting.
However is it possible that another thread would insert the same key txnID between the modify_if and the erase call? If that's possible use erase_if instead.
Hello sir. I have done some experiments and find out Flat Hashmap provides the best performance for my workload. However, I do have one question. Why would the following code not work:
using entry_ptr = std::shared_ptr<Transaction_Table_Entry>;
using Map = phmap::parallel_flat_hash_map<
uint64_t,
entry_ptr ,
phmap::priv::hash_default_hash<uint64_t>,
phmap::priv::hash_default_eq<uint64_t>,
std::allocator<std::pair<const uint64_t, entry_ptr>>,
12,
std::mutex>;
static void reader(Map* (txnTables)[THREAD_NUM]){
std::random_device dev;
std::mt19937 rng(dev());
std::uniform_int_distribution<std::mt19937::result_type> threadDist(0,THREAD_NUM-1);
std::uniform_int_distribution<std::mt19937::result_type> txnIDDist(0,(1ul<<LAZY_UPDATE_WORKLOAD_ORDER)-1);
//Transaction_Table_Entry* entryPtr;
while(!flag.load());
auto start = std::chrono::high_resolution_clock::now();
long opCount=0;
while(true){
int threadID = threadDist(rng);
int local_txnID = txnIDDist(rng);
uint64_t txnID = bwtransaction::generateTxnID(threadID,local_txnID);
entry_ptr ptr;
//entry_ptr ptr =txnTables[threadID]->find(txnID)->second;
txnTables[threadID]->if_contains(txnID,[&ptr](typename Map::value_type& pair){ptr = pair.second;});
if(ptr->op_count.fetch_sub(1)==1){
txnTables[threadID]->erase(txnID);
}
opCount++;
auto stop = std::chrono::high_resolution_clock::now();
auto duration = std::chrono::duration_cast<std::chrono::microseconds>(stop - start);
if(duration.count()>60000000){
break;
}
}
totalOp.fetch_add(opCount);
}
I try to eliminate as many latches as possible but it seems like the pointer would complain. But I'm using a shared pointer so I don't really get how could that memory be freed when I already made a shared pointer to the same object. Thanks.
You are looking at the pointer outside the lock, while another thread can erase it. Do this:
bool shouldErase;
txnTables[threadID]->if_contains(txnID,[&shouldErase](typename Map::value_type& pair){ shouldErase = pair.second->op_count.fetch_sub(1)==1;});
if(shouldErase){
txnTables[threadID]->erase(txnID);
}
Actually yes, you are right, with the shared_ptr assigned inside the if_contains lambda, it should be safe (if op_count is a std::atomic)
Thank you. Haha I made a mistake in my code. I should forget the case that if_contains does not even get executed. My bad.
I'm planning on using the flat hashmap in my project but still haven't decided whether I want to store pointers or objects directly. So I will try out both and compare performance when the full system is set up. But I do have a question. I personally believe this is safe according to my experiments and study. But want to confirm. For object storage, assume I still have atomic fields of each entry. Then I believe it is safe to only grab the read lock if operations only involve reads and updates of those atomic fields. Operations on atomic fields using atomic primitives are always safe, and read locks are enough to protect against conflicting write operations (insert or delete in the same map). So it is safe then. Can I ask what you think about that? Also I will give you proper acknowledgment in the paper of this project. Thanks!
I believe it is safe to only grab the read lock if operations only involve reads and updates of those atomic fields. Operations on atomic fields using atomic primitives are always safe, and read locks are enough to protect against conflicting write operations (insert or delete in the same map). So it is safe then.
Yes, absolutely, this should be fine.
haven't decided whether I want to store pointers or objects directly.
The parallel_node_hash_map will store pointers to objects, so if you want to compare the two options, you can just switch from parallel_flat_hash_map to parallel_node_hash_map without changing anything else.
Also I will give you proper acknowledgment in the paper of this project.
Cool. Please send me a link to the paper :-)
Thank you for the information. Well the paper will still take a while to finish. I have to finish our system first! Also I don't fully get what you are talking about. I think I meant to say keeping pointers inside the map or keeping the actual object. Do you mean I can keep a pointer to an entry in the map in the node hash map? I will take a look. Thanks.
If you use node_hash_map<key, value> it will internally store pointers to value, so it would be somewhat equivalent to flat_hash_map<key, value*>.
I did not know that. Will take a look.
May I ask how do I make a local copy of the pointer? If the map stores pointers internally how can I store a pointer locally after a lookup to the entry in the table? Thanks.
Sorry, you can't. If you need to do that, then you'll have to manage the pointers yourself like you suggested.
I see. Thank you.