Giters

microsoft / go

The Microsoft build of the Go toolset

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

AzDO pipeline polling error: "invalid character '<' looking for beginning of value"

dagood opened this issue · comments

commented

While polling a build for completion, a call got this error, making the pipeline fail:

invalid character '<' looking for beginning of value

https://dev.azure.com/dnceng/internal/_build/results?buildId=2261106&view=logs&j=8d802004-fbbb-5f17-b73e-f23de0c1dec8&t=995a157e-87d5-56c6-b42a-d8b689a8a0cd&l=409

In my experience this happens when the API returns HTML instead of the expected JSON. Authentication is an easy way to hit it, but maybe it's rate limiting or some internal error?

At the very least, we should try to change the code to show the whole response when this happens so we can investigate. I'm not sure how deep in the AzDO library this would need to be done. We should also see if there are newer versions of the module available that might address this.

commented
  • #1039 hit something that I think is probably the same issue, an outage response from AzDO while trying to retain the build. Although the HTML tags seem to be omitted from this output text.

https://dev.azure.com/dnceng/internal/_build/results?buildId=2263023&view=logs&j=307c05fb-395e-5cff-ceb4-9869362bab1d&s=f6031391-65b9-5416-93f9-6593f6c32fa6&t=c21f8ad3-d0c9-57f6-6f87-8d3464674117&l=56

Excerpt:

        Azure DevOps Services Unavailable
    
    
        
            
            Azure DevOps Services
        
        
            
            
                Sorry! Our services aren't available right now.
                We're working to restore all services as quickly as possible. Please check back soon.
                To see the latest status on our services, please visit our support page.

I don't see anything related in https://status.dev.azure.com/_1es/_history. The fact that it failed just once across a lot of jobs makes me think we're starting to hit flakiness in a few APIs that used to be more solid.

commented

There was no clear diagnosis at the AzDO level in the IcM ticket (potentially network instability somewhere deeper), and they would like the IcM reopened if we hit it again.