Backend fails to start

Question

Backend fails to start

junalmeida opened this issue 2 years ago · comments

Trying to start the backend in VSCODE, I can see the following log:

Azure Functions Core Tools
Core Tools Version:       4.0.4785 Commit hash: N/A  (64-bit)
Function Runtime Version: 4.10.4.19213

Warning: Proxies are not supported in Azure Functions v4. Instead of 'proxies.json', try Azure API Management: https://aka.ms/AAfiueq
[2022-11-25T15:58:10.623Z] Csproj not found in c:\Users\junior\.vscode\extensions\durablefunctionsmonitor.durablefunctionsmonitor-6.0.0\backend directory tree. Skipping user secrets file configuration.
[2022-11-25T15:58:17.990Z] A host error has occurred during startup operation '4bac7748-7552-499c-aa98-c04bfd6e38b8'.
[2022-11-25T15:58:17.990Z] Microsoft.Azure.WebJobs.Extensions.Http: Could not load file or assembly 'System.Net.Http.Formatting, Version=5.2.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified.
ERROR: Value cannot be null. (Parameter 'provider')ERROR:

It does not complete.

Konstantin Lepeshenkov · Answer 1 · Sat Nov 26 2022 21:01:43 GMT+0800 (China Standard Time)

@junalmeida , as a quick workaround, you can always fall back to the previous backend version:

Are you able to run any 4.x Function projects on that machine?

Marcos Junior · Answer 2 · Sat Nov 26 2022 23:07:00 GMT+0800 (China Standard Time)

Yep thanks for the tip, previous net core seems to work
And yes, I can run other net 6 az func v4 projects.

Thanks for looking into this

Konstantin Lepeshenkov · Answer 3 · Mon Nov 28 2022 22:48:46 GMT+0800 (China Standard Time)

@junalmeida , was it really the whole log output that DfMon VsCode extension is showing you?

Normally there should be some more lines in the beginning, like Using Functions host: ****** and more.
Can you post the entire log output you're getting when the backend fails to start?

Marcos Junior · Answer 4 · Mon Nov 28 2022 22:52:05 GMT+0800 (China Standard Time)

Using Functions host: C:\Program Files\Microsoft\Azure Functions Core Tools\func.exe
Attempting to start the backend from c:\Users\junior\.vscode\extensions\durablefunctionsmonitor.durablefunctionsmonitor-6.0.0\backend on http://localhost:37072/a/p/i...
Azure Functions Core Tools
Core Tools Version:       4.0.4785 Commit hash: N/A  (64-bit)
Function Runtime Version: 4.10.4.19213

Warning: Proxies are not supported in Azure Functions v4. Instead of 'proxies.json', try Azure API Management: https://aka.ms/AAfiueq
[2022-11-28T14:50:16.957Z] Csproj not found in c:\Users\junior\.vscode\extensions\durablefunctionsmonitor.durablefunctionsmonitor-6.0.0\backend directory tree. Skipping user secrets file configuration.
[2022-11-28T14:50:19.102Z] A host error has occurred during startup operation '1e1b4f23-ebee-469c-afb3-7be33b6187a8'.
[2022-11-28T14:50:19.103Z] Microsoft.Azure.WebJobs.Extensions.Http: Could not load file or assembly 'System.Net.Http.Formatting, Version=5.2.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified.
ERROR: Value cannot be null. (Parameter 'provider')ERROR:

Konstantin Lepeshenkov · Answer 5 · Tue Nov 29 2022 04:23:32 GMT+0800 (China Standard Time)

In short, looks very much like Azure/azure-functions-core-tools#3115.
As people are suggesting there, the solution is to reinstall both .NET 6 SDK and Aure Functions Core Tools.

What surprises me though is that you're saying that other Function projects run normally on your machine.
Are you running them in Visual Studio? Can you instead try to just func start in Function project's folder?

Also can you please try to install the latest Azure Functions Core Tools ?

(just to mention, I tried to install and use your exact Core Tools version and wasn't able to repro)

Marcos Junior · Answer 6 · Tue Nov 29 2022 22:47:06 GMT+0800 (China Standard Time)

I did reinstall net 6, I've removed previous 6.x SDK versions, and installed the latest one. I've also refreshed Az Func Core Tools, and the vscode extension itself (I've even manually deleted the folder mentioned in the logs).

It didn't work unfortunately. I did this a couple of weeks ago so a new version might be available by then, I'll try upgrade the tools again.

Other projects runs normally, and I run them in VSCode using the very same path as in the following logs:

 *  Executing task: func host start -p 7072 


Azure Functions Core Tools
Core Tools Version:       4.0.4785 Commit hash: N/A  (64-bit)
Function Runtime Version: 4.10.4.19213

[2022-11-29T14:40:21.862Z] Found C:\Projects\***\***\***\****.csproj. Using for user secrets file configuration.

Functions:

        AddNew*****Function: orchestrationTrigger

(I've redacted parts of the log)

Konstantin Lepeshenkov · Answer 7 · Tue Nov 29 2022 23:04:53 GMT+0800 (China Standard Time)

Can you try to go to c:\Users\junior\.vscode\extensions\durablefunctionsmonitor.durablefunctionsmonitor-6.0.0\backend folder, execute the following there:

set AzureWebJobsStorage=my-storage-conn-string
func start --verbose

and post the output (without any sensitive data) here?

Mark Priem · Answer 8 · Sun Dec 04 2022 23:02:57 GMT+0800 (China Standard Time)

Sorry to hijack the thread. I have exactly the same issue. @scale-tone when running the commands you suggested I get the following results:

PS /home/vscode/.vscode-server/extensions/durablefunctionsmonitor.durablefunctionsmonitor-6.0.0/backend> func start --verbose              Can't determine project language from files. Please use one of [--csharp, --javascript, --typescript, --java, --python, --powershell, --custom]
Can't determine project language from files. Please use one of [--csharp, --javascript, --typescript, --java, --python, --powershell, --custom]
Can't determine project language from files. Please use one of [--csharp, --javascript, --typescript, --java, --python, --powershell, --custom]

                  %%%%%%
                 %%%%%%
            @   %%%%%%    @
          @@   %%%%%%      @@
       @@@    %%%%%%%%%%%    @@@
     @@      %%%%%%%%%%        @@
       @@         %%%%       @@
         @@      %%%       @@
           @@    %%      @@
                %%
                %


Azure Functions Core Tools
Core Tools Version:       4.0.4865 Commit hash: N/A  (64-bit)
Function Runtime Version: 4.12.2.19454

Warning: Proxies are not supported in Azure Functions v4. Instead of 'proxies.json', try Azure API Management: https://aka.ms/AAfiueq
Can't determine project language from files. Please use one of [--csharp, --javascript, --typescript, --java, --python, --powershell, --custom]
Can't determine project language from files. Please use one of [--csharp, --javascript, --typescript, --java, --python, --powershell, --custom]
Can't determine project language from files. Please use one of [--csharp, --javascript, --typescript, --java, --python, --powershell, --custom]
[2022-12-04T14:54:55.277Z] Building host: version spec: , startup suppressed: 'False', configuration suppressed: 'False', startup operation id: '3e4a2336-7cf9-47ed-bb81-4770e677a5b6'
[2022-12-04T14:54:55.295Z] Reading host configuration file '/home/vscode/.vscode-server/extensions/durablefunctionsmonitor.durablefunctionsmonitor-6.0.0/backend/host.json'
[2022-12-04T14:54:55.298Z] Host configuration file read:
[2022-12-04T14:54:55.299Z] {
[2022-12-04T14:54:55.300Z]   "version": "2.0",
[2022-12-04T14:54:55.300Z]   "extensions": {
[2022-12-04T14:54:55.300Z]     "http": {
[2022-12-04T14:54:55.301Z]       "routePrefix": ""
[2022-12-04T14:54:55.301Z]     }
[2022-12-04T14:54:55.301Z]   }
[2022-12-04T14:54:55.302Z] }
[2022-12-04T14:54:55.389Z] FUNCTIONS_WORKER_RUNTIME set to powershell. Skipping WorkerConfig for language:node
[2022-12-04T14:54:55.408Z] FUNCTIONS_WORKER_RUNTIME set to powershell. Skipping WorkerConfig for language:java
[2022-12-04T14:54:55.428Z] FUNCTIONS_WORKER_RUNTIME set to powershell. Skipping WorkerConfig for language:python
[2022-12-04T14:54:55.444Z] Loading extensions from /home/vscode/.vscode-server/extensions/durablefunctionsmonitor.durablefunctionsmonitor-6.0.0/backend/bin. BundleConfigured: False, PrecompiledFunctionApp: False, LegacyBundle: False
[2022-12-04T14:54:55.445Z] Script Startup resetting load context with base path: '/home/vscode/.vscode-server/extensions/durablefunctionsmonitor.durablefunctionsmonitor-6.0.0/backend/bin'.
[2022-12-04T14:54:55.458Z] Loading startup extension 'DurableTask'
[2022-12-04T14:54:55.627Z] Loaded extension 'DurableTask' (2.0.0.0)
[2022-12-04T14:54:55.694Z] Reading host configuration file '/home/vscode/.vscode-server/extensions/durablefunctionsmonitor.durablefunctionsmonitor-6.0.0/backend/host.json'
[2022-12-04T14:54:55.694Z] Host configuration file read:
[2022-12-04T14:54:55.694Z] {
[2022-12-04T14:54:55.695Z]   "version": "2.0",
[2022-12-04T14:54:55.695Z]   "extensions": {
[2022-12-04T14:54:55.695Z]     "http": {
[2022-12-04T14:54:55.695Z]       "routePrefix": ""
[2022-12-04T14:54:55.696Z]     }
[2022-12-04T14:54:55.702Z]   }
[2022-12-04T14:54:55.703Z] }
[2022-12-04T14:54:56.685Z] FUNCTIONS_WORKER_RUNTIME set to powershell. Skipping WorkerConfig for language:node
[2022-12-04T14:54:56.685Z] FUNCTIONS_WORKER_RUNTIME set to powershell. Skipping WorkerConfig for language:java
[2022-12-04T14:54:56.687Z] FUNCTIONS_WORKER_RUNTIME set to powershell. Skipping WorkerConfig for language:python
[2022-12-04T14:54:57.087Z] Using the default storage provider: AzureStorage.
[2022-12-04T14:54:57.160Z] A host error has occurred during startup operation '3e4a2336-7cf9-47ed-bb81-4770e677a5b6'.
[2022-12-04T14:54:57.160Z] Microsoft.Azure.WebJobs.Extensions.DurableTask: Unable to resolve the Azure Storage connection named 'Storage'.
[2022-12-04T14:54:57.217Z] Using the default storage provider: AzureStorage.
Value cannot be null. (Parameter 'provider')
[2022-12-04T14:54:57.338Z] Initialization cancellation requested by runtime.
[2022-12-04T14:54:57.350Z] Stopping host...
[2022-12-04T14:54:57.351Z] Host shutdown completed.
PS /home/vscode/.vscode-server/extensions/durablefunctionsmonitor.durablefunctionsmonitor-6.0.0/backend>

Konstantin Lepeshenkov · Answer 9 · Sun Dec 04 2022 23:22:12 GMT+0800 (China Standard Time)

@mpriem , thanks for that, but your above output

Microsoft.Azure.WebJobs.Extensions.DurableTask: Unable to resolve the Azure Storage connection named 'Storage'

indicates that func host was unable to find the Storage connection string in AzureWebJobsStorage env variable. I suspect, it was because my above commands were intended for Windows, while you are running Linux (or MacOS?..).

In Bash the correct way to set that env variable would be:

export AzureWebJobsStorage="my-storage-conn-string"

(note the quotes around).

And in Powershell it would be:

$Env:AzureWebJobsStorage = 'my-storage-conn-string'

Can you try it like that?

Also, I presume, when using the VsCode extension, you're getting a different error. Can you post the logs from there as well?

Mark Priem · Answer 10 · Mon Dec 05 2022 02:28:06 GMT+0800 (China Standard Time)

sorry. I see what is wrong here. I'm using the VSCode extension and it tries to run it inside the same workspace as my PowerShell functions project. I now moved it inside a dotnet based container and the backend starts without issues.

Thanks for getting me on the right path @scale-tone

pjgsk · Answer 11 · Mon Dec 12 2022 21:51:33 GMT+0800 (China Standard Time)

Well I have very same issue.
When I set explicitly what @scale-tone proposed here:

Can you try to go to c:\Users\junior\.vscode\extensions\durablefunctionsmonitor.durablefunctionsmonitor-6.0.0\backend folder, execute the following there:
set AzureWebJobsStorage=my-storage-conn-string
func start --verbose
and post the output (without any sensitive data) here?

it all works well, finds and loads 17 functions and job host starts ok. BUT when I try to open DFM from VCS I get this

@mpriem does it work for you now when triggered from VCS?

Mark Priem · Answer 12 · Mon Dec 12 2022 22:04:33 GMT+0800 (China Standard Time)

That didn't work for me. The problem I had was that the project I'm working on uses the PS runtime and the monitor expects dotnet.
I ended up just running it as a separate container. Works perfectly.

pjgsk · Answer 13 · Mon Dec 12 2022 22:27:50 GMT+0800 (China Standard Time)

Well, I wouldn't ever think the project's runtime has anything to do with the DFM itself.
I am able to run it in separate container and access from the browser but that's not as convenient as just having multiple tabs opened in VCS.

Konstantin Lepeshenkov · Answer 14 · Mon Dec 12 2022 22:37:09 GMT+0800 (China Standard Time)

Thanks @pjgsk , I think, we're close to something here.

Your experiment basically tells us that there's some difference in env variable values between your console window and your VsCode instance (from which DfMon extension inherits them when starting the backend).

May I ask you to do the same func start --verbose exercise in VsCode's Terminal window ? What is the outcome then?

Mark Priem · Answer 15 · Mon Dec 12 2022 22:37:11 GMT+0800 (China Standard Time)

Well in my case I couldn't get it to run. When trying to run it from VSC either one or the other project wouldn't load due to not finding functions for the specific runtime. I did not have issues when running it inside VSC with dotnet projects, just with other runtimes such as PS or Python.

pjgsk · Answer 16 · Mon Dec 12 2022 23:00:43 GMT+0800 (China Standard Time)

I actually solved this. It was using my azure-functions-core-tools from this folder: /Users/*****/.nvm/versions/node/v16.15.0/lib/node_modules/azure-functions-core-tools/bin/func which was v.4.0.4544
So I changed the extension setting durableFunctionsMonitor.customPathToAzureFunctionsHost to point to /usr/local/bin/func which is 4.0.4915 and it started work all ok.

May I ask you to do the same func start --verbose exercise in VsCode's Terminal window ? What is the outcome then?

I already updated my nvm default node version so its not what it was before.

Fabio Perfetti · Answer 17 · Thu Dec 22 2022 01:03:12 GMT+0800 (China Standard Time)

I was facing the same issue reported by @junalmeida.
Removing and installing the latest version of Azure Functions Core Tools (version 4.0.4915) fixed the issue.

Marcos Junior · Answer 18 · Thu Dec 22 2022 01:26:44 GMT+0800 (China Standard Time)

I also updated tooling and it works like @xsurfer pointed out.

Konstantin Lepeshenkov · Answer 19 · Sun Feb 05 2023 18:32:19 GMT+0800 (China Standard Time)

Just confirmed by experimenting on a Codespaces instance.

The issue takes place with Core Tools version up to 4.0.4590 (Functions Runtime version 4.5.2.18383).
Starting from version 4.0.4629 (Runtime version 4.6.1.18388) it disappears.

Didn't need to uninstall anything, just did npm i -g azure-functions-core-tools.

Will add version check to VsCode Ext's code.

Konstantin Lepeshenkov · Answer 20 · Tue Feb 07 2023 03:33:18 GMT+0800 (China Standard Time)

Added runtime version check in v6.1. Should be resolved now.