Troubleshooting disappearing Storage Account and TaskHub instance
xsurfer opened this issue · comments
Recently, one of the Storage Accounts and its associated TaskHub instance disappeared from the list of available ones. I do have proper access to the Storage Account.
However, I am able to connect to the TaskHub by using the ConnectionString and TaskHub name.
Unfortunately, I am unable to delete the TaskHub using this method, and it's something I would like to do.
I have already attempted reinstalling the latest extension (version 6.4.0), and I have also tried downgrading it to version 6.3.0, but it didn't resolve the issue.
Here is the link to an image for reference:
Do you have any suggestions on how to troubleshoot this further?
Thank you!
Hi @xsurfer , DfMon-as-VsCode's default algorithm of discovering Task Hubs is as follows:
It tries to 'list keys' of all Storage accounts accessible by you. If successful, it then continues to use those Storage keys for interacting those Storage accounts.
If for whatever reason it is unable to get Storage keys, it falls back to using (your) identity-based access.
To explicitly control which method to use there's this setting:
If you say that a Storage account suddenly "disappeared" from DfMon's list, then I suggest you check if someone (or something) has changed access policies and other settings of that Storage account. A typical change being made is prohibiting Storage keys to be used (note that it might take a while to see the effect of that on your existing connection strings).
You can also play with the above "Task Hubs Discovery Mode" setting, to see if different modes produce different results. Just make sure you restart VsCode after changing it.
Also "Durable Functions Monitor" output channel should give you more insight.
hi @scale-tone, thank you for your reply.
You can also play with the above "Task Hubs Discovery Mode" setting, to see if different modes produce different results. Just make sure you restart VsCode after changing it.
I did but unfortunately it didn't help :(
If you say that a Storage account suddenly "disappeared" from DfMon's list, then I suggest you check if someone (or something) has changed access policies and other settings of that Storage account. A typical change being made is prohibiting Storage keys to be used (note that it might take a while to see the effect of that on your existing connection strings).
I do have same RBAC assignment as for other Storage Accounts that are visible in the extension.
I also checked storage key setting and it is enabled.
Also "Durable Functions Monitor" output channel should give you more insight.
Unfortunately "Durable Functions Monitor" output channel doesn't show any trace at all.
Do you have any other reco/idea on what else to check?
What is the monitor exactly looking for within the storage account (on top of instances and history tables, and work item and control queues)? Maybe someone could have deleted a relevant item by mistake (some colleague of mine delete tables and queues manually so that TaskHub is recreated from scratch when Func App restarts).
Update: I think I found the answer to previous question. It seems that it is looking for Instances and History tables and then extracting TaskHub name from the full table name. https://github.com/microsoft/DurableFunctionsMonitor/blob/784a0e980b3d7c6f992d1685f4b9e9cc125c14b5/durablefunctionsmonitor-vscodeext/src/TaskHubsCollector.ts#L85C1-L98C1
Do you think this may be related to the fact that I've recently upgraded Durable Task library to v3 which is in preview mode?
Thank you!
it is looking for Instances and History tables and then extracting TaskHub name from the full table name
Yes, and the only next thing I can suggest is to manually look for those tables. You can use Azure portal or Azure Storage Explorer for that.
Do you think this may be related to the fact that I've recently upgraded Durable Task library to v3 which is in preview mode?
I don't know, and would appreciate if you check that hypothesis (whether they've out of a sudden changed table name format).
Do you still need help on this, @xsurfer ?
@scale-tone if you would be able to help me further, I'd really appreciate that!
Both tables, Instances and History, are in place (they are not renamed/deleted in the Durable Task v3) along with a new one, Partitions, added in the Durable Task v3.
Said that, everything seems in order, still I'm not able to list the hub stored in one of my Storage Account
@xsurfer , can you tell me how many tables you have there in this storage account in total?
I am aslo facing the same issue. Were you able to find a solution to resolve this?
I have these 2
I also see that the taskhubname is appearing as null:
@mkraghavan , are you saying you have those two (and only two) tables in your storage account and you do not see any Task Hubs in DfMon VsCode?
Do you see any Task Hubs in any other storage account in DfMon VsCode?
Were you able to see those Task Hubs there before?
Are you sure you're signed in into Azure in your VsCode? Occasionaly you need to re-login, also just recently they seem to roll out a change in VsCode, which caused re-logins be required more frequently.
Can you see any other Azure resources on your AZURE tab in VsCode? Again, due to that potential recent change in VsCode it might take several VsCode restarts and login interactions to make you see your Azure resources there again.
Does DfMon's Output Channel show any errors to you?
That screen with "TaskHubName" column being null - where does it come from (cause it is definitely not a part of DfMon)?
@scale-tone
are you saying you have those two (and only two) tables in your storage account and you do not see any Task Hubs in DfMon VsCode? No, now when i checked Azure storage explorer I see these also:
Do you see any Task Hubs in any other storage account in DfMon VsCode? Yes, I have a lot of other storage accounts withTaskhubs and I can see the task hubs in there
Are you sure you're signed in into Azure in your VsCode? All the other storage accounts have the same permissions as this one and I can see all the others. I manually added the storage account in dfmon and it shows something like this:
Can you see any other Azure resources on your AZURE tab in VsCode? Yes, I can
Does DfMon's Output Channel show any errors to you? No
That screen with "TaskHubName" column being null - where does it come from (cause it is definitely not a part of DfMon)?
That is from the azure storage explorer - Instances table:
Thanks for that info, @mkraghavan .
Then my suspicion is that due to some recent change in Azure Tables REST API it started doing batching (sending x-ms-continuation-nexttablename response header) more aggressively.
Just pushed DfMon VsCode extension v6.4.1 with a fix for that. It should automatically land in your VsCode after a couple of restarts.
Can you please check if it resolved the issue for you?
That actually did resolve it for me! Thanks a lot!
I can also confirm the fix resolved the issue for me as well. @scale-tone thank you so much!