Failed transactions on reinforce (was #10)

Question

Failed transactions on reinforce (was #10)

coccoinomane opened this issue 2 years ago · comments

As noted by @nikohd12, reinforce transactions often fail resulting in lost gas.

This happens because the two available strategies (HighestMp and HighestBp) aim to rent the cheapest crab with the highest stats, a strategy likely to met fierce competition from other bots & players.

I see two ways to fix this:

Bid higher: Introduce a parameter to bid X more TUS than the renting price floor.
Use the Web endpoint: When you are on the Crabada website and try to rent a unavailable crab, the UI stops you with a FAILED message, without making you spend gas. The bot could do the same.

The two options are not mutually exclusive.
Personally, I lean towards option no. 2, because it minimizes the chance to lose gas, but I don't know what endpoint is used by the UI.

Eugene Teu · Answer 1 · Sun Mar 20 2022 10:35:50 GMT+0800 (China Standard Time)

2 seems good. I think 1 wouldnt solve the failing problem.

btw, I have some friends running the looting script to automatically reinforce. I find that running a reinforce script for each team and staggering their timings (using a cron job) helps w reducing the failed txn (but not much)

GeekFromPH · Answer 2 · Sun Mar 20 2022 10:40:54 GMT+0800 (China Standard Time)

Hi @coccoinomane ,

Thanks for fixing the bug. I would also lean on doing no. 2. However, I also don't know what the endpoint is for Tavern renting.

If I may suggest, we can introduce option 1 while waiting for a solution on option 2?

coccoinomane · Answer 3 · Mon Mar 21 2022 03:52:18 GMT+0800 (China Standard Time)

Hi guys and thank you for your suggestions!

I agree to start with option 1, given how simple it is to implement compared to option 2.

I have pushed a test branch with a minimal implementation of option 1.

I am testing it and so far I have seen no failures, but I will wait for more data before going ahead with the full implementation. Feel free to test the branch yourselves if you want to accelerate the process :-)

Cheers,
Cocco

GeekFromPH · Answer 4 · Mon Mar 21 2022 11:26:43 GMT+0800 (China Standard Time)

Thank you @coccoinomane , I'll test it out for 24 hours and report back.

I took a screenshot of failed transactions when reinforcing and the gap was around 1 minute each for 12 teams, I'll provide details tomorrow when I'm done with the testing of Test Branch

Not related to this issue, but do you think you can also do self-reinforce or reinforcing from inventory?

blanketry · Answer 5 · Mon Mar 21 2022 12:24:01 GMT+0800 (China Standard Time)

@nikohd12 reinforcing from owned in-game-crab would solve the problem. owned crabs in the game, that are not for rent in the tavern and not in a team, have a 1/2 hour cool down and if staggered can be rented by multiple teams at 1/2 hour intervals, but not twice in the same mine. i think this is an overlooked strategy

GeekFromPH · Answer 6 · Mon Mar 21 2022 13:11:05 GMT+0800 (China Standard Time)

Indeed, @blanketry. This strategy is the most efficient, we just need to stagger teams 30 minutes a.k.a manipulate crontab.

We could probably add Crab IDs from the inventory to reinforce and run the reinforce script every 15~30 minutes for maximum efficiency.

If statement would look like:

If Inventory Crab is available, then reinforce with it
Else, reinforce using tavern with maxTus price set and strategy.

cc: @coccoinomane

GeekFromPH · Answer 7 · Mon Mar 21 2022 13:12:39 GMT+0800 (China Standard Time)

Follow up re: Test Higher Reinforce Bid

Prior to deployment:

With the new commit (6 hours of testing):

Thank you as always @coccoinomane! I'll report back again in 24 hours.

coccoinomane · Answer 8 · Mon Mar 21 2022 22:32:15 GMT+0800 (China Standard Time)

Hey guys, thank you for all the suggestions, and welcome @blanketry to the discussion 🙂

I think reinforcing from inventory crabs is a good idea: let's move the discussion in a separate issue > #12

As for the original thread, thank you @nikohd12 for sharing your results before & after the new strategy, they are really helpful. I have seen similar results on my side and I will go ahead fully implementing the option 1.

PS: Thanks @EugeneTeu about the staggering suggestion! Would that help if reinforceDefense and reinforceAttack only affected one team at a time, instead than doing all of the available teams?

Eugene Teu · Answer 9 · Mon Mar 21 2022 22:35:43 GMT+0800 (China Standard Time)

@coccoinomane yeap i went with that here https://github.com/EugeneTeu/crabada.py/blob/single-scripts/bin/mining/reinforceDefenseSingle.py

essentially just duplicated the logic and then allow the user to input the team they wanted to reinforce etc. Seems to be working

GeekFromPH · Answer 10 · Mon Mar 21 2022 22:42:14 GMT+0800 (China Standard Time)

Thank you for this @coccoinomane. I have yet to encounter errors up to this point. Well done!

p.s the bot still gives me good crabs, lower than the max TUS I set. So it’s doing the right thing with little to no failed transactions. 👍

coccoinomane · Answer 11 · Mon Mar 21 2022 22:42:48 GMT+0800 (China Standard Time)

Thanks for sharing @EugeneTeu !
I was going to implement something similar, with a difference: the script will automatically determine the first available team, without the need for the user to specify a team ID.
May I ask whether it is important for you being able to specify the team ID?

Eugene Teu · Answer 12 · Mon Mar 21 2022 22:44:16 GMT+0800 (China Standard Time)

@coccoinomane there may be cases where people have different teams used for mining AND looting so specifying the team id gives them the control in those cases

GeekFromPH · Answer 13 · Mon Mar 21 2022 22:46:39 GMT+0800 (China Standard Time)

@coccoinomane there may be cases where people have different teams used for mining AND looting so specifying the team id gives them the control in those cases

Wouldn’t making a different .env file fix this?

GeekFromPH · Answer 14 · Mon Mar 21 2022 22:47:20 GMT+0800 (China Standard Time)

My current setup is run a cron job to auto settle and reinforce loot from 9am to 5pm and auto mine from 6pm to 8am.

You could also create a separate folder, let’s say:

crabada-miners
crabada-looters

Eugene Teu · Answer 15 · Mon Mar 21 2022 22:48:23 GMT+0800 (China Standard Time)

Wouldn’t making a different .env file fix this?

running via CLI commands is much more convenient.

Eugene Teu · Answer 16 · Mon Mar 21 2022 22:48:51 GMT+0800 (China Standard Time)

My current setup is run a cron job to auto settle and reinforce loot from 9am to 5pm and auto mine from 6pm to 8am.

You could also create a separate folder, let’s say:

crabada-miners crabada-looters

for sure, but wouldnt it mean 2 folders to manage

interesting setup... im just mining all the time

GeekFromPH · Answer 17 · Mon Mar 21 2022 22:49:36 GMT+0800 (China Standard Time)

My current setup is run a cron job to auto settle and reinforce loot from 9am to 5pm and auto mine from 6pm to 8am.
You could also create a separate folder, let’s say:
crabada-miners crabada-looters

for sure, but wouldnt it mean 2 folders to manage

For simplicity, you could make 2 .env files and maintain 1 folder. You can run reinforceAttack and reinforceDefense at the same time.

Eugene Teu · Answer 18 · Mon Mar 21 2022 22:52:40 GMT+0800 (China Standard Time)

My current setup is run a cron job to auto settle and reinforce loot from 9am to 5pm and auto mine from 6pm to 8am.
You could also create a separate folder, let’s say:
crabada-miners crabada-looters

for sure, but wouldnt it mean 2 folders to manage

For simplicity, you could make 2 .env files and maintain 1 folder. You can run reinforceAttack and reinforceDefense at the same time.

i mean if i could just input the team id... i can just change my cron job without touching my folders at all right (at least that was the thinking when i added that script)

@coccoinomane has the final say :P

coccoinomane · Answer 19 · Mon Mar 21 2022 23:17:40 GMT+0800 (China Standard Time)

Thanks guys for the insightful discussion! What if we adopt the following compromise?

Each script applies only to the first available team or mine (no team_id argument).
For each team, you specify in .env whether it is supposed to loot or to mine.

For example, with the following .env configuration, the bot will only send team 1111 mining, while team 2222 will be considered only for looting-related tasks:

USER_1_TEAM_1="1111"
USER_1_TEAM_1_TASK="mine"

USER_1_TEAM_2="2222"
USER_1_TEAM_2_TASK="loot"

This approach has the following advantages:

No need to have multiple .env files or folders.
In the future, if the need arises, we can implement more types of tasks (battle?)

On a side note...

I have not tested nor I necessarily recommend it, but you could manage multiple .env files in the following way:

source .env_1 && python3 -m bin.mining.reinforceDefense <your address>
source .env_2 && python3 -m bin.mining.reinforceDefense <your address>

Just make sure that you export every line in the .env files.

Eugene Teu · Answer 20 · Mon Mar 21 2022 23:19:06 GMT+0800 (China Standard Time)

Thanks guys for the insightful discussion! What if we adopt the following compromise?

Each script applies only to the first available team or mine (no team_id argument).

For each team, you specify in .env whether it is supposed to loot or to mine.

For example, with the following .env configuration, the bot will only send team 1111 mining, while team 2222 will be considered only for looting-related tasks:
USER_1_TEAM_1="1111"
USER_1_TEAM_1_TASK="mine"

USER_1_TEAM_2="2222"
USER_1_TEAM_2_TASK="loot"
This approach has the following advantages:

No need to have multiple .env files or folders.

In the future, if the need arises, we can implement more types of tasks (battle?)

On a side note...

I have not tested nor I necessarily recommend it, but you could manage multiple .env files in the following way:
source .env_1 && python3 -m bin.mining.reinforceDefense <your address>
source .env_2 && python3 -m bin.mining.reinforceDefense <your address>
Just make sure that you export every line in the .env files.

sounds very good to me!

ck0111 · Answer 21 · Mon Mar 21 2022 23:35:21 GMT+0800 (China Standard Time)

As noted by @nikohd12, reinforce transactions often fail resulting in lost gas.

This happens because the two available strategies (HighestMp and HighestBp) aim to rent the cheapest crab with the highest stats, a strategy likely to met fierce competition from other bots & players.

I see two ways to fix this:

Bid higher: Introduce a parameter to bid X more TUS than the renting price floor.

Use the Web endpoint: When you are on the Crabada website and try to rent a unavailable crab, the UI stops you with a FAILED message, without making you spend gas. The bot could do the same.

The two options are not mutually exclusive. Personally, I lean towards option no. 2, because it minimizes the chance to lose gas, but I don't know what endpoint is used by the UI.

Hello Coco, I think Option 2 could be the best idea as it totally prevents tx failures. However, as you mentioned the endpoint for the UI might be hard to find. Also, one thing that I'm worried about is if it will introduce more delay to the bot script(?) performing the rents if it needs to check the endpoint.
Adding a task to specific teams is a great idea as well, it helps a lot when playing with multiple teams that are doing different jobs (loot/mine).

In the meantime, the new reinforce strategy of renting the 4th available crabs is working pretty well. Spending 1TUS more compared to 5 failed txs is +EV. I'm been playing around with it, seems like 4th crab works with HighestBPReinforce and 2nd crab works with HighestMPReinforce.

GeekFromPH · Answer 22 · Tue Mar 22 2022 10:51:38 GMT+0800 (China Standard Time)

Just to give you guys an update, I have only encountered 3 failed transactions in a span of 36 hours for 12 Teams Mining.

Great work @coccoinomane, your solution worked. I think others can try and tweak the listing of crabs to their liking, as @ck0111 mentioned.

I have tried his solution and encountered some failed transactions when setting 2nd crab for HighestMpReinforce. I'm trying to set it to 3rd crab and will report back. This is more of user tweaking, the issue itself was fixed when doing 4th crab logic.

coccoinomane · Answer 23 · Tue Mar 22 2022 13:53:08 GMT+0800 (China Standard Time)

Hi @ck0111 , thank you for sharing your thoughts & feedback!

However, as you mentioned the endpoint for the UI might be hard to find. Also, one thing that I'm worried about is if it will introduce more delay to the bot script(?) performing the rents if it needs to check the endpoint.

That's a good point. By waiting for an extra endpoint, we might lose the window to rent the crab. Will need to think it through.

Adding a task to specific teams is a great idea as well, it helps a lot when playing with multiple teams that are doing different jobs (loot/mine).

The task feature has been shipped to master, feel free to test it :-)

coccoinomane · Answer 24 · Tue Mar 22 2022 14:23:24 GMT+0800 (China Standard Time)

Just to give you guys an update, I have only encountered 3 failed transactions in a span of 36 hours for 12 Teams Mining.

Thanks for sharing your results! To be honest, I wasn't expecting such a big improvement 😅

Before pushing the feature to master, we probably need a user-level parameter to control which crab to rent, because renting the 4th crab is too arbitrary.

I am undecided between an ordering parameter (rent the n-th crab from the cheapest crab) or a TUS parameter (rent the first crab that costs X more TUS than the cheapest crab).

Any thoughts?

GeekFromPH · Answer 25 · Tue Mar 22 2022 14:28:19 GMT+0800 (China Standard Time)

I am undecided between an ordering parameter (rent the n-th crab from the cheapest crab) or a TUS parameter (rent the first crab that costs X more TUS than the cheapest crab).

Any thoughts?

Hmmm. I have the following thoughts:

Inputting a minimum TUS spend. ❌ However, this feels like work cos you have to be updated to the current market trend/price of optimal crabs to reinforce to determine the right TUS price.
Rent the n-th crab from the cheapest crab. This is the current implementation right? Since we are sorting it from cheapest and we have a max TUS price to protect ourselves from inflation, I have a feeling that this could work and that this method is tweak-able.
TUS parameter. ❌ As mentioned, I think this fluctuates to much plus most people put decimals to confuse other people on Tavern.

Imo. Renting the n-th crab from the cheapest will also give you a chance to rent a crab that is 0.01~3 cents higher than the cheapest vs a TUS parameter.

ck0111 · Answer 26 · Tue Mar 22 2022 14:32:26 GMT+0800 (China Standard Time)

Just to give you guys an update, I have only encountered 3 failed transactions in a span of 36 hours for 12 Teams Mining.

Thanks for sharing your results! To be honest, I wasn't expecting such a big improvement 😅

Before pushing the feature to master, we probably need a user-level parameter to control which crab to rent, because renting the 4th crab is too arbitrary.

I am undecided between an ordering parameter (rent the n-th crab from the cheapest crab) or a TUS parameter (rent the first crab that costs X more TUS than the cheapest crab).

Any thoughts?

I like the idea of an ordering parameter. Another additional layer of filter that could be added would be a check for the n-th crab to have the same bp/mp as the 1st crab detected depending if you're looting(bp) or mining(mp) so as to not end up renting a crab that has its stats way off the first crab.

GeekFromPH · Answer 27 · Tue Mar 22 2022 14:37:02 GMT+0800 (China Standard Time)

would be a check for the n-th crab to have the same bp/mp as the 1st crab detected depending if you're looting(bp) or mining(mp) so as to not end up renting a crab that has its stats way off the first crab.

Wouldn't that introduce a bit of a delay to rent thus missing or failing a transaction?

ck0111 · Answer 28 · Tue Mar 22 2022 14:37:35 GMT+0800 (China Standard Time)

would be a check for the n-th crab to have the same bp/mp as the 1st crab detected depending if you're looting(bp) or mining(mp) so as to not end up renting a crab that has its stats way off the first crab.

Wouldn't that introduce a bit of a delay to rent thus missing or failing a transaction?

It could. I guess have to test it out.

coccoinomane · Answer 29 · Tue Mar 22 2022 14:39:01 GMT+0800 (China Standard Time)

Wouldn't that introduce a bit of a delay to rent thus missing or failing a transaction?

It should be quite negligible, as the endpoint already returns the stats of all rentable crabs.

ck0111 · Answer 30 · Tue Mar 22 2022 15:22:41 GMT+0800 (China Standard Time)

Just to give you guys an update, I have only encountered 3 failed transactions in a span of 36 hours for 12 Teams Mining.

Thanks for sharing your results! To be honest, I wasn't expecting such a big improvement 😅
Before pushing the feature to master, we probably need a user-level parameter to control which crab to rent, because renting the 4th crab is too arbitrary.
I am undecided between an ordering parameter (rent the n-th crab from the cheapest crab) or a TUS parameter (rent the first crab that costs X more TUS than the cheapest crab).
Any thoughts?

I like the idea of an ordering parameter. Another additional layer of filter that could be added would be a check for the n-th crab to have the same bp/mp as the 1st crab detected depending if you're looting(bp) or mining(mp) so as to not end up renting a crab that has its stats way off the first crab.

Or like a minimum BP/MP filter

GeekFromPH · Answer 31 · Tue Mar 22 2022 16:05:22 GMT+0800 (China Standard Time)

Or like a minimum BP/MP filter

The cons of this is that we have to monitor the market always to hit maxTUS to our target BP/MP.

ck0111 · Answer 32 · Tue Mar 22 2022 16:26:01 GMT+0800 (China Standard Time)

I'm not sure I'm understanding what you mean. A minimum BP/BP filter just means it won't rent any crabs that goes below that treshold.

coccoinomane · Answer 33 · Tue Mar 22 2022 20:56:58 GMT+0800 (China Standard Time)

Thanks guys for sharing your ideas!

Given the consensus reached for the N-th crab strategy, I will implement it and add the N parameter to .env.

@ck0111, thank you for proposing the target BP/MP idea.
It could be implemented as a tolerance parameter, that is:

if you set bpTolerance=5 you are signalling that you are happy to get a crab with BP as small as the BP of the first crab minus 5;
if you set bpTolerance=0, it means you don't want to compromise, and the strategy will get the highest-N-crab with the same stats as the top one.

It is a bit of a complex implementation though, so I think it's best to freeze it until we see the effects of the simple N-th crab mechanism.

coccoinomane · Answer 34 · Wed Mar 23 2022 02:35:08 GMT+0800 (China Standard Time)

Hello snib-snibbers!

I have pushed to master the new n-th crab mechanism for reinforcing.

To activate the feature you just need to insert a greater-than-one value in the parameter USER_X_TEAM_Y_REINFORCEMENT_TO_PICK. If you do not include the parameter in your .env, it will automatically fallback to 1, which is the old behaviour.

In the end, I've decided to include the parameter at the team level, to give players with multiple teams the possibility to fine-tune it according to the team strategy, e.g. set it differently for looting teams vs mining teams.

As always, I would be very grateful if you could test the feature and let me know of any bug 🙂

Cheers,
Cocco

GeekFromPH · Answer 35 · Wed Mar 23 2022 06:40:07 GMT+0800 (China Standard Time)

Thanks guys for sharing your ideas!

Given the consensus reached for the N-th crab strategy, I will implement it and add the N parameter to .env.

@ck0111, thank you for proposing the target BP/MP idea. It could be implemented as a tolerance parameter, that is:

if you set bpTolerance=5 you are signalling that you are happy to get a crab with BP as small as the BP of the first crab minus 5;

if you set bpTolerance=0, it means you don't want to compromise, and the strategy will get the highest-N-crab with the same stats as the top one.

It is a bit of a complex implementation though, so I think it's best to freeze it until we see the effects of the simple N-th crab mechanism.

Hi @coccoinomane, thanks for the update. Will test it out and play with the n-th crab.

Here are my thoughts regarding tolerance levels. Imo, it is not needed. With huge supplies of Bulk and it’s population, there is no need for this feature. You would almost always want to reinforce Bulk too since it has the highest BP out of the 3 BP crabs.

Same goes for MP, ideally you reinforce craboids and with the current rise of population of Primes, you would always want either 82, 81 or 79 MP crabs to get to 40% MR.

Lastly, with Swimmer network around and reducing gas cost by up to 85% we could tweak n-th crab to a lower number and accept that we lose some transactions if the gas are cheaper by that time.

We should focus on self reinforce while waiting for Swimmer network announcement. Again, in my opinion only.

coccoinomane · Answer 36 · Mon Mar 28 2022 23:09:33 GMT+0800 (China Standard Time)

Thanks @nikohd12 for sharing your thoughts!

I agree that the BP/MP tolerance implementation is not urgent and, if the need arises, it can always be implemented as a separate reinforce strategy.

On a related note, I'll try to make the strategy-making tutorial by next week, so that everybody can easily contribute with their own strategies.

Cheers,
Cocco

yigitest · Answer 37 · Tue Mar 29 2022 17:39:40 GMT+0800 (China Standard Time)

@coccoinomane
Came across this question and tried to implement it. I couldn't be able to detect a transaction that will fail but maybe it helps you to figure it out.

In Web3Client.py / buildContractTransaction()

baseTx = self.buildBaseTransaction()
try:
    result = contractFunction.estimateGas(baseTx, "pending")
    logger.debug(f"Estimate gas result: {result}")
except Exception as err:
    logger.warning(f"Transaction will probably fail!", exc_info=True)

return contractFunction.buildTransaction(baseTx)

without "pending":

result = contractFunction.estimateGas(baseTx)

and without "baseTx" parameters when estimating gas:

try:
    result = contractFunction.estimateGas()
    logger.debug(f"Estimate gas result: {result}")
except Exception as err:
    logger.warning(f"Transaction will probably fail!", exc_info=True)

baseTx = self.buildBaseTransaction()
return contractFunction.buildTransaction(baseTx)

All raised the same exception : execution reverted: GAME:NOT OWNER

coccoinomane · Answer 38 · Tue Mar 29 2022 22:16:48 GMT+0800 (China Standard Time)

Nice find @yigitest ! This might just be the approach used by the Crabada website 🙌

It remains to be seen how much the time required to make the extra check will reduce the chances to actually get the reinforcements.

My guess is that it could still be a good compromise, given the reduced risk of wasting gas.

yigitest · Answer 39 · Tue Mar 29 2022 22:25:10 GMT+0800 (China Standard Time)

Sorry if i was misunderstood. In these examples estimateGas() always raised an exception. Even if the real reinforceDefence transaction completed successful afterwards; or not.

I don't know if i did something wrong, or it is something to do with avax-eth differences?

coccoinomane · Answer 40 · Tue Apr 05 2022 18:19:52 GMT+0800 (China Standard Time)

Sorry if i was misunderstood. In these examples estimateGas() always raised an exception. Even if the real reinforceDefence transaction completed successful afterwards; or not.

I don't know if i did something wrong, or it is something to do with avax-eth differences?

Fixed in #40 🙂