hash storage variant
s3inlc opened this issue · comments
Would it be possible to add the feature to select another type of storage for the normal hashes instead of mysql? Because the server gets really at its limits (as long as it's not a high-end server) when using larger hash lists (for example ~1million hashes).
As I have some experience with using Cassandra, this would be my wish-system to have as hash storage option. If you like the idea I would also be willing to help programming on this as I used Cassandra also with PHP and I also could help with a test environment with Cassandra installed.
does it follow sql standard?
From: Sein Coray [mailto:notifications@github.com]
Sent: Friday, February 26, 2016 3:09 PM
To: curlyboi/hashtopus
Subject: [hashtopus] hash storage variant (#38)
Would it be possible to add the feature to select another type of storage for the normal hashes instead of mysql? Because the server gets really at its limits (as long as it's not a high-end server) when using larger hash lists (for example ~1million hashes).
As I have some experience with using Cassandra, this would be my wish-system to have as hash storage option. If you like the idea I would also be willing to help programming on this as I used Cassandra also with PHP and I also could help with a test environment with Cassandra installed.
—
Reply to this email directly or view it on GitHub #38 . https://github.com/notifications/beacon/AHh7hj-nKilFaBz15gPNAOiUv0uLrdSMks5poFPggaJpZM4Hj0Db.gif
Not exactly. It is limited for some requests as Cassandra is more like a key/value store with some additional possibilities to create indexes.
Just to make sure you're understanding me right. I thought of just use Cassandra for the 'hashes' and not for any other tables, so the other tables would just stay MySQL.
Hmm... I am very often joining regular tables with the hash tables so if I would do this,
I would need to transfer the whole database to Cassandra.
From: Sein Coray [mailto:notifications@github.com]
Sent: Friday, February 26, 2016 3:58 PM
To: curlyboi/hashtopus
Cc: cURLy bOi
Subject: Re: [hashtopus] hash storage variant (#38)
Not exactly. It is limited for some requests as Cassandra is more like a key/value store with some additional possibilities to create indexes.
Just to make sure you're understanding me right. I thought of just use Cassandra for the 'hashes' and not for any other tables, so the other tables would just stay MySQL.
—
Reply to this email directly or view it on GitHub #38 (comment) . https://github.com/notifications/beacon/AHh7hgkOMA7wwLay2khs0nE78vXOYPhxks5poF9jgaJpZM4Hj0Db.gif
Too bad, that would not be possible to do all with Cassandra, because for example joins are not possible on Cassandra. I thought perhaps it would be possible to just use it as storage for the hashes because this is the critical part when having large hash lists.
Would it be much work to list me the join statements you are using with the hashtables, or at least give some examples? I will take a look at it, perhaps I will just fork the current repo and play around a little and will look if there could be a workaround for this problem.
Yeah you can definitely look into the source. The SQL is pretty self-explanatory there.
From: Sein Coray [mailto:notifications@github.com]
Sent: Friday, February 26, 2016 4:11 PM
To: curlyboi/hashtopus
Cc: cURLy bOi
Subject: Re: [hashtopus] hash storage variant (#38)
Too bad, that would not be possible to do all with Cassandra, because for example joins are not possible on Cassandra. I thought perhaps it would be possible to just use it as storage for the hashes because this is the critical part when having large hash lists.
Would it be much work to list me the join statements you are using with the hashtables, or at least give some examples? I will take a look at it, perhaps I will just fork the current repo and play around a little and will look if there could be a workaround for this problem.
—
Reply to this email directly or view it on GitHub #38 (comment) . https://github.com/notifications/beacon/AHh7hp9wCAAO2ZTDEiqaQO1g0jSRe8Qcks5poGJugaJpZM4Hj0Db.gif
You could organize hashes in a seperate database and organize them by hex value, that way you'd decrease the table size. (For example you order by aa, ab, ac, ad) That'd take 255 tables max.
Any updates on this?