Document that programs may hang after upgrading to 14.0.0+
Zweitze opened this issue · comments
Describe the bug:
#480 describes a situation that causes programs to hang. The cause was found and described as
Try using the async version. This issue has come up before -- it has nothing to do with .NET Framework, but certain subparts of .NET Framework that doesn't like waiting on threads (e.g. WinForms).
And with that #480 was closed, and therefore hiding this information for other users: It took me two full days to find this explanation.
Expected behavior:
Make the information in #480 more public, for instance mention this in the Readme, and/or Changelog at v14.0.0 - the first version with this problem.
Client library version and API version:
Client library version: 14.0.0 and higher
Google Ads API version: V11
.NET version: .NET Framework 4.7.2
Operating system (Linux, Windows, ...) and version (if the bug is platform-specific): Windows 10
Anything else we should know about your project / environment
Visual Studio 2019.
Project type: WinForms (.NET Framework)
#480 is pinned, so I thought that would have been sufficient. Let me add this to the README.md
@AnashOommen Any updates on this? Even when using Async methods, the API calls hangs, we are unable to use ANY version of the API besides V10 which is already getting requests rejected due to sunset.
How can we upgrade to the new libraries? The entire process hangs (both WinForms and Web Forms, 4.72, using latest Google Ads .NET library from NuGet (15.0.0 stable)
Please respond, as this is already causing us major issues in production.
@AnashOommen while the async version still works, breaking all sync methods is not cool.
And we are not using WinForms. Just standard ASP.Net 4.7.2 web app.
Most likely related: #386 (comment)
And using async methods is not even a good workaround for people stuck with .Net 4.7.2 since we don't have async streams and ForEachAsync(). What's the recommended way of iterating through SearchAsync() results in .Net 4.7.2?
All the examples are for synchronous usage. Also, in some cases, for example ListAccessibleCustomers() and ListAccessibleCustomersAsync(), have totally different signature with different params and return values.
@cleardarkz wrote:
@AnashOommen Any updates on this?
We got a similar situation as @cleardarkz describes. It always hangs (no ambiguity) after doing the proper action at ads.google.com.
Even when using Async methods, the API calls hang,
I rename a campaign at Ads.Google.com, and I can do it asynchronously. It works, but I have to wait until finished. That wait hangs.
we are unable to use ANY version of the API besides V10,
which is already getting requests rejected due to sunset.
Same here. We tried v13.0.2 with all versions, v14.0.0 with all versions, v15.0.0 with all versions. Same result in all cases: It hangs (after doing the proper operation at ads.google.com)
How can we upgrade to the new libraries? The entire process hangs (both WinForms and Web Forms, 4.7.2, using the latest Google Ads .NET library from NuGet (15.0.0 stable)
Please respond, as this is already causing us major issues in production.
Our production has stopped as well. That is not a desirable situation.
- @AnashOommen, are you working on this issue right now?
- How long time can it take to find a solution to the problem?
- What should we expect?
- Are there workarounds?
I upgraded my projects from v13.0.2 to v15.0.0, and changed all methods by their Async counterparts, and... it does work.
But it's very hard, took me weeks - but the VS solution contains over 100 projects.
Some pointers for fellow developers:
- Learn everything about async and await keywords. Make sure you understand everything, for instance why you should never call Task.Wait or Task.Result
- When something hangs, it's probably because the threadpool was exhausted. If you have no clue what I mean by this, check the first pointer.
- If you really think a method of the library is broken, try creating a new project and just call this method - you'll learn it does work. If not, you've got a perfect bug report.
@hakimio : Coding for .NET Framework 4.72, for searching I use GoogleAdsServiceClient.SearchStreamAsync(). Before I used GoogleAdsServiceClient.SearchStream(). Note that SearchStreamAsync expects a delegate which is called whenever a bunch of rows is ready. That delegate will be called synchronously.
I might have an update on this:
Try using config.UseGrpcCore = true;. If the code used to work before, there is a very good chance that this will continue to work now. E.g. this code snippet works for me on .NET Framework 4.8, but I haven't tested this exhaustively.
private void button1_Click(object sender, EventArgs e)
{
GoogleAdsConfig config = new GoogleAdsConfig();
config.UseGrpcCore = true;
GoogleAdsClient client = new GoogleAdsClient(config);
// Get the GoogleAdsService.
GoogleAdsServiceClient googleAdsService = client.GetService(Services.V13.GoogleAdsService);
// Create a query that will retrieve all campaigns.
string query = @"SELECT campaign.id, campaign.name, campaign.network_settings.target_content_network FROM " +
"campaign ORDER BY campaign.id";
List<long> Ids = new List<long>();
try
{
// Issue a search request.
googleAdsService.SearchStream(customerId.ToString(), query,
delegate (SearchGoogleAdsStreamResponse resp)
{
foreach (GoogleAdsRow googleAdsRow in resp.Results)
{
Ids.Add(googleAdsRow.Campaign.Id);
}
}
);
MessageBox.Show(Ids.Count.ToString());
}
catch (GoogleAdsException ex)
{
MessageBox.Show($"Failure: {ex.Failure}");
}
}This comes with the following caveats and nuances:
- I cannot guarantee this works 100% of the time, but at least, this will restore the old dependencies and behavior.
- This change has its roots in https://grpc.io/blog/grpc-csharp-future/. tl;dr; is that Google sunset its .NET implementation of grpc, in favor of Microsoft's implementation of grpc. I still maintain a dual dependency on the old Google implementation and the new Microsoft implementation, that's why this workaround is possible.
- The good news is that if this works, you can continue using this in the future. I plan to maintain this dependency long term.
- The bad news is that since the Google implementation of grpc for .NET is sunset, it is unlikely we will get a bug fix if anything breaks on the .NET Framework side. My experience so far is that the code is very stable, so I don't anticipate any issues.
- The additional catch is that Microsoft may not be be interested in fixing this behaviour in their libraries for legacy .NET Framework. I'll ask in either case.
To throw more search terms onto the pile:
Working to upgrade from v10. We are using multiple services so I'm thinking I've got some updating to work on to make everything async.
The functions I was using didn't seem to work anymore after changing to v13. First thing I noticed was GoogleAdsService.Search() no longer working. I found an example of instead using GoogleAdsService.SearchStream() , so I tried that and it worked. Did not to use config.UseGrpcCore = true; . I tried that later when trying to get CustomerService.ListAccessibleCustomers() to work, and finding a Exception thrown: 'System.PlatformNotSupportedException' in Grpc.Net.Client.dll . But that did not work, same exception. So now I'm going to try and replace everything with async methods.
Example I found for SearchStream() was I think this: https://developers.google.com/google-ads/api/docs/reporting/example
search Grpc.Net.Client on google
https://github.com/grpc/grpc-dotnet/blob/456098bcfeb9c796c19ea023b3ab3703181f1089/src/Shared/HttpHandlerFactory.cs#L48
https://learn.microsoft.com/en-us/aspnet/core/grpc/netstandard?view=aspnetcore-7.0
therefore I have to get out of .NET Framework?
the repo says it supports .NET Framework 4.7.2+ (net472), .NET 5.0 (net5.0), and Not .NET 6.0 (net6.0) yet
workarounds?
config.UseGrpcCore = true;
doesn't work. seems to use Grpc.Net.Client anyway.
Turns out my workaround had a small issue. It should be
GoogleAdsConfig config = new GoogleAdsConfig();
config.UseGrpcCore = true;
GoogleAdsClient client = new GoogleAdsClient(config);
I am also trying to upgrade from v11->v12/v13
I am using .NET Framework 4.7.2, we have a website using vb.net and another winforms app using the Google Api
I have made many tests in the last 2 days, below the experiences I made.
At the beginning I got in both projects this forever pending request while testing ListAccessibleCustomers (sync version)
Changing the app type from Windows Form App -> Console App caused that it works in sync versions. I made multiple get/mutate tests - everything is fine. I assume that this is because of the issue Anash mentioned in #480 (comment)
I still cannot find a solution for the website. It is mainly MVC but also having some old webforms pages.
I tried the fix with config.UseGrpcCore = true but it had no impact on the behavior, same error.
Even more: I tested the async version and there is also a pending request...
I copied the test with ListAccessibleCustomersAsync to the console app and works without problems.
Also: I created a new class library project and installed the api package there - works without problems.
Any ideas what dlls/refrences could cause the problems in the web app?
The first part of the app I am trying to fix is using CustomerClientLinkService & CustomerManagerLinkService to connect accounts. Also using GoogleAdsService as one would expect.
I changed the GoogleAdsService queries to use SearchStream() instead, and that works. But I see that when I create the GoogleAdsService I get the: Exception thrown: 'System.PlatformNotSupportedException' in Grpc.Net.Client.dll .
I tried using the previous methods CustomerClientLinkService MutateCustomerClientLink() & CustomerManagerLinkService MutateCustomerManagerLink . They function: I see the accounts being effected in ads.google.com . But I'm not sure if they fail or they are hanging as others are describing, because the code will not continue to execute. I have to stop the code and then comment things out to move along the flow past these functions.
So now I am working on changing to the async methods of these fucntions, as well as CustomerService ListAccessibleCustomers .
@ian9 could you try the local nuget packages in this attachment? Does that improve the situation?
for-release.zip
@AnashOommen
I'm not good at packages so don't take my word on this.
I did the steps for the above zip file, with the instructions found here, and I'm not seeing a difference: #491
I see the zipped packages are using Microsoft.Extensions 3.1.9 . But I am using 6.0.X currently. Do I need to downgrade these packages to give the zipped packages what they need? Or do the 6.0.X libraries also have 3.1.9 stuff in them.
I am messing around with these packages because I am worried that visual studio is using the nuget.org version of the packages even though I ask otherwise.
edit: when I install the packages, when I look in the bin, the dll's have the same dates as the publish dates of the nuget.org versions. Does that mean it is giving me the nuget.org versions?
Possible... let me bump the versions a bit and give you new packages...
Bumped versions, restored Microsoft.Extensions.* dependencies... LMK how it goes.
I have installed the the packages @AnashOommen provided but the Search() request is still pending...
I think the package installation succeeded at my machine, the Google.Ads.GoogleAds.dll modification date is 17.03, so it is newer than the original one, also Microsoft.Extensions version is differenr - 5.x (in the origin package it was upgraded to 6.x)
config.UseGrpcCore = true;
Are you saying the above fix 'should' work for synchronous calls?
I have tried googleAdsServiceClient.Search (V13) in .NET 4.7.2 and .NET 4.8 with Google.Ads.GoogleAds v15.0.0 and I still have the hang.
My code was working fine in Google.Ads.GoogleAds v13.0.2
We solved the mystery of the hanging Google Ads API calls (well, in our case)! Our Windows forms app is running .NET 4.7.2 and Google.Ads.GoogleAds v15.0.0 and we also have a MySql/Maria DB database.
We are limited to one DB connection to the database, that is a Maria DB thing. When checking for a free second MySql/Maria DB connection, the gRPC routines that Google uses start to hang indefinitely (in our case); all the GAQL calls still work, but MutateCampaigns, etc sync/async got into a hanging wait.
The solution for us: No checking for a free db connection. Instead, use the one we already had. Hope this helps anyone out there.
@ReneBrandon @jensmalmgren I still haven't quite figured out why 13.0.2 used to work and 15.0.0 doesn't, except that the Grpc dependencies changed afterwards. The base code hasn't been touched in almost 2+ years. I have a few colleagues looking into it.
Anyways... here's another attempt to try fix the issue. This time, I have included two additional projects -- A .NET5 WinForms application and a .NET472 WinForms application. Both should work with the attached dependencies. In both cases, open Form1.cs, update your credentials and customerId. Then try running the button1_click method. You can lower the budget to 500 to trigger an error, and 500000 to trigger a successful call, for instance.
Let me know how it goes.
Appreciate all the help, and thanks for your patience.
distro.zip
@AnashOommen
Yes! This moved the issue forward and instead of a hang, an UNREGONIZED_FIELD error for 'Metrics.cost_micros' was thrown.
I had a quick google to confirm this had not been depreciated, before I figured the query string is now case sensitive, so I edited from 'Metrics.cost_micros' to 'metrics.cost_micros' and all is well again.
Perhaps had I not had this unreported error waiting for me, I wouldn't have experienced the hangs.
So to confirm, I have this working on .NET 4.8. with 15.0.2.
Many thanks for your prompt help with this.
I just executed a test on my side and I could also make first succesful api calls get/mutate (sync) with .Net 4.7.2 and 15.0.2 :)
Can I assume that this 15.0.2 will become an official nuget package version soon?
Thanks a lot for the adaptations!
Yes! I tried 15.0.2, and I can confirm it works. Created campaigns, ad groups, and assets - all of it. No hangs. We continue testing/production but so far we are really happy to be back in business.
https://github.com/googleads/google-ads-dotnet/releases/tag/v15.0.1 is live now (My version was a little bit higher on the local build). It contains the fix. Pls take the library and give it a try.
Hello, I've been having a similar issue with GoogleAdsClient in a VS 2019 .Net 4.72 API project. Specifically when using UploadClickConversions. I've updated to Google.Ads.GoogleAds 15.0.1 but it did not fix it. My code is nearly identical to the example examples/Remarketing/UploadOfflineConversion.cs
When the API gets to UploadClickConversionsResponse response = conversionUploadService.UploadClickConversions( the service hangs and never continues.
Interestingly, the same code in a console app works fine.
Same behaviour for me: the official 15.0.1 is not a fix for us.
Right now only the unofficial 15.0.2 is a solution.
As the sunset date of the V11 is still 29.03 I think that we will have to go to production with this unofficial package...
I do not feel very comfortable with it but we are running out of time soon.
@AnashOommen - any plans to release 15.0.2 as official package? is the V11 sunset date still valid?
@VoxelSpace does the unofficial 15.0.2 attached on this bug work for you? If so, pls use it. The two builds use slightly different approaches, and I'm fine picking whichever approach that eventually works for the community.
Could you also get me a demo project that illustrates the issue?
@amuszalik feel free to use 15.0.2 unofficial build if that works for you. I don't want to rush into making yet another official release until I figure out what's going wrong with @VoxelSpace issue.
If you are running short on time, pls email googleadsapi-support@google.com and mention in the email to route to me (Anash). Point to #488 and share your developer MCC account ID. I'll see what can be done to buy more time.
@AnashOommen yes, 15.0.2 worked for me! Looking forward to the official release. Thanks very much.
-edit: Also, I'd like to add to the request to extend the sunset deadline, since we've been unable to prepare an upgrade solution for our production environment.
#488 (comment) Worked for me too! I downloaded and add update the references to 15.0.2. Thanks!
Can someone share a demo WinForm application that shows v15.0.1 hanging when making an API call?
@AnashOommen will 15.0.2 be officially released soon? v11 API is set to sunset tomorrow
v15.0.2 is live now.
@AnashOommen the official 15.0.2 version is not working. I am getting the same hanging at the conversionUploadService.UploadClickConversions step I was before.
The unofficial 15.0.2 patch you provided in this thread does still work.
I'm having the same experience as VoxelSpace. I can use the zipped copies of 15.0.2 within our complicated app and things work: the entire auth flow of querying accounts and then connecting accounts (CustomerClientLinkService & CustomerManagerLinkService). But if I use the nuget version, the issue is not fixed. Same Exception thrown: 'System.PlatformNotSupportedException' in Grpc.Net.Client.dll .
This is using .NET Framework 4.7.2 .
@AnashOommen
Hi
I have the same freeze problem on the official 15.0.2.
The unofficial 15.0.2 is working.
Our development is on:
.NET version: .NET Framework 4.7.2
Operating system: Windows 10
Visual Studio 2022
VB
Project type: MVC , Web Forms, WinForms.
I have provided for you sample projects to reproduce the error in MVC.
To create it I have used clean standard visual studio new project template for MVC platform in 4.7.2.
Then I installed the official 15.0.0.2 release in WebApplicationmvcFreeze solution.
Then I installed the unofficial 15.0.0.2 release in WebApplicationmvcWorking solution.
In both samples I have used config.UseGrpcCore = true;
freeze effect
WebApplicationmvcFreeze.zip
working sample
WebApplicationmvcWorking.zip
Both projects run the test method on load in HomeController Index Action:
`
Dim searchQuery = "SELECT campaign.id FROM campaign"
Dim ret As New List(Of GoogleAdsRow)
Dim config As GoogleAdsConfig = New GoogleAdsConfig With {
.UseGrpcCore = True
}
Dim GoogleAdsClient As New GoogleAdsClient(config)
GoogleAdsClient.Config.LoginCustomerId = 111
GoogleAdsClient.Config.OAuth2RefreshToken = "test"
Dim GCustomerId As Long = 222
Dim googleAdsService As GoogleAdsServiceClient = GoogleAdsClient.GetService(Google.Ads.GoogleAds.Services.V12.GoogleAdsService)
Dim searchPagedResponse As PagedEnumerable(Of SearchGoogleAdsResponse, GoogleAdsRow) = googleAdsService.Search(GCustomerId.ToString(), searchQuery)
For Each row In searchPagedResponse 'freeze
Dim ix = row.AdGroupAd
Next
Dim res = True
`
WebApplicationmvcFreeze project stops on line For Each row In searchPagedResponse
WebApplicationmvcWorking on line For Each row In searchPagedResponse returns exception RpcException: Status(StatusCode="Unavailable". Which is desired result in our test case.
Could you solve official release package and let us know, that the fix will be included in the long term release politics?
We are forced to use the unofficial 15.0.0.2 version on live systems because of the upcoming sunset.
That is far away from desirable situation for us.
Same here, spent hours until I found this thread. My code base has been working for over 2 years, similar to others. I tried the official 15.0.2 today. code is still hanging. Tried both setting useGrPC=true or not. Same for each, code hangs. using WinForms also. No other changes have been made to the code base. Just upgrade google ads library, code freezes.
I made another local branch not contaminated by the zipped packages and tried the ' config.UseGrpcCore = true; ' with 15.0.2 and got the same situation that others are saying: issue persists.
What I do notice is that my local branch that works by using the zipped packages is using a copy of Google.Ads.Gax.2.1.2 and Google.Ads.GoogleAds.Core.2.1.2 . The 2.1.2 versions do not show up when using nuget, I only see up to 2.1.0 . Do the 2.1.2 versions contain the fix?
Yep, unfortunately we missed a version bump on Google.Ads.Gax when making yesterday's release. WIP...
Please try v15.0.3 from Nuget. We
- Bumped the versions of
Google.Ads.Gax => 2.2.0, andGoogle.Ads.GoogleAds.Core => 2.2.0 - Bumped the dependency of
Google.Ads.GoogleAds v15.0.3to pick up the newGoogle.Ads.GaxandGoogle.Ads.GoogleAds.Corelibrary versions.
v15.0.3 from nuget fixed the issue for me.
v15.0.3 from nuget seems to be working for me now too. Thank you.