Geographic_Report using (SearchGoogleAdsStreamResponse)
nitheeshls opened this issue · comments
In continue to issue #511
Code used for getting geographic report
case "geographic_view":
googleAdsService.SearchStream(googleAPIDetails.CustomerId.ToString(), query,
delegate (SearchGoogleAdsStreamResponse resp)
{
if (resp.FieldMask != null)
{
CsvFile csvFile = new CsvFile();
// Set the header for the CSV file.
csvFile.Headers.AddRange(resp.FieldMask.Paths);
foreach (GoogleAdsRow googleAdsRow in resp.Results)
{
rows.Add(googleAdsRow);
}
foreach (GoogleAdsRow googleAdsRow in rows)
{
csvFile.Records.Add(new string[]
{
googleAdsRow.GeographicView.LocationType.ToString(),
googleAdsRow.Campaign.Id.ToString(),
googleAdsRow.Campaign.Name.ToString(),
googleAdsRow.Campaign.ResourceName.ToString(),
googleAdsRow.AdGroup.Name.ToString(),
googleAdsRow.AdGroup.Id.ToString(),
googleAdsRow.Segments.Date.ToString(),
googleAdsRow.Segments.Device.ToString(),
googleAdsRow.Segments.GeoTargetMetro.ToString(),
googleAdsRow.Segments.GeoTargetPostalCode.ToString(),
googleAdsRow.Segments.GeoTargetRegion.ToString(),
googleAdsRow.Metrics.AllConversions.ToString(),
googleAdsRow.Metrics.AllConversionsValue.ToString(),
googleAdsRow.Metrics.Clicks.ToString(),
googleAdsRow.Metrics.CostMicros.ToString(),
googleAdsRow.Metrics.Impressions.ToString(),
googleAdsRow.Metrics.Interactions.ToString(),
googleAdsRow.Metrics.VideoViews.ToString(),
googleAdsRow.Metrics.ViewThroughConversions.ToString(),
});
}
csvFile.Write(fileDirectory + fileName);
}
As response from the above search stream is taking too long for getting past 30 days file. I'm encountering the error:
geographic_view || One or more errors occurred. (Status(StatusCode="Unknown", Detail="Stream removed", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1683871756.281000000","description":"Error received from peer ipv4:172.253.122.95:443","file":"......\src\core\lib\surface\call.cc","file_line":1070,"grpc_message":"Stream removed","grpc_status":2}"))
Also I checked issue #488 and tried with 15.0.2 and 15.0.3 same issues.
But any how its working in my local for atleast 30 days sometimes if the connection remains stable for 30 to 40 min, but the same is not working inside azure function apps where this code is configured with in 12 or 15 min its throwing the above exception.
As Business is facing lot of issues bcz of this issue, Need your help..
Out of all these reports we are extracting only problem with geographic_view report
Tried this as well still facing the same issues
Can you please paste the GAQL query you are using? I'm wondering if there are some improvements that can be made in terms of segmentation.
@Raibaz Thanks for Helping!!
This is the code i'm using
shortFileName = Path.GetFileNameWithoutExtension(file);
string newFormat = DateTime.ParseExact(startDate, "yyyy'-'MM'-'dd", CultureInfo.InvariantCulture).ToString("MM'-'dd'-'yyyy");
fileName = "Google_" + shortFileName + "" + newFormat + "" + DateTime.Now.ToString("HH_mm_ss") + ".csv";
List allFields = File.ReadAllLines(file).ToList();
var fieldswithComma = string.Join(",", allFields);
if (googleAPIDetails.EnableLogs == 1)
{
GoogleUtility.LogToDatabase(googleAPIDetails.ServerName, googleAPIDetails.DBName, googleAPIDetails.BatchId, shortFileName, fileName, "csv", log);
}
string query = @"SELECT " + fieldswithComma + " FROM " + shortFileName + " WHERE segments.date BETWEEN '" + startDate + "' AND '" + endDate + "'";
GoogleAdsServiceClient googleAdsService = googleAdsClient.GetService(Services.V13.GoogleAdsService);
try
{
List<GoogleAdsRow> rows = new List<GoogleAdsRow>();
switch (shortFileName)
{
case "geographic_view":
googleAdsService.SearchStream(googleAPIDetails.CustomerId.ToString(), query,
delegate (SearchGoogleAdsStreamResponse resp)
{
if (resp.FieldMask != null)
{
CsvFile csvFile = new CsvFile();
QUERY
SELECT
geographic_view.location_type,
campaign.id,
campaign.name,
campaign.resource_name,
ad_group.name,
ad_group.id,
segments.date,
segments.device,
segments.geo_target_metro,
segments.geo_target_postal_code,
segments.geo_target_region,
metrics.all_conversions,
metrics.all_conversions_value,
metrics.clicks,
metrics.cost_micros,
metrics.impressions,
metrics.interactions,
metrics.video_views,
metrics.view_through_conversions
FROM geographic_view WHERE segments.date BETWEEN
"2023-04-01" AND "2023-04-30"
Can you please help me with the above issue. As it is heavily impacting the business needs!!
Today I got specific error message, may be this will help you to identify the exact issue
One or more errors occurred. (Status(StatusCode="Unknown", Detail="Stream removed", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1684842236.419000000","description":"Error received from peer ipv4:172.253.62.95:443","file":"......\src\core\lib\surface\call.cc","file_line":1070,"grpc_message":"Stream removed","grpc_status":2}"))
2023-05-23T11:52:17Z [Error] Unable to update the data to database: System.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the login process. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)
---> System.ComponentModel.Win32Exception (10054): An existing connection was forcibly closed by the remote host.
at System.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e)
at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions) at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at ElancoCORE_MINT_AdsAPI.Utils.GoogleUtility.UpdateLog(String serverName, String dbName, String batchId, String fileName, String status, String comment, ILogger log) in C:\Users\02709994\Downloads\rel-1\1\ElancoCORE-MINT-AdsAPI\Utils\GoogleUtility.cs:line 562
ClientConnectionId:a3722a53-721d-4017-9754-49f5a1b84f14
Error Number:10054,State:0,Class:20
Since you are running this in an azure function with a shorter timeout, have you considered using paged search rather than searchStream?
I'm wondering whether retrieving one page at a time in subsequent azure function calls rather than trying to retrieve all rows at once in a single function call may be helpful for your scenario.
Also, do you have an approximate idea of how many rows your GAQL query is supposed to be returning?
Even tried out with "functionTimeout:-1" still same result.
And the same code is working fine with 13.0.2 version with an execution time of 3-4 mins . The moment we updated it to 15 and later versions this issue is popping up.
And the rows retrieved with above query is around 8,90,000
@Raibaz @AnashOommen
The process cannot access the file 'C:\home\site\wwwroot\GoogleReports\Google_geographic_view_05-25-2023_12_20_12.csv' because it is being used by another process.
Error
Unable to update the data to database: System.Data.SqlClient.SqlException (0x80131904): Incorrect syntax near ''. at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource1 completion, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite, String methodName) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at ElancoCORE_MINT_AdsAPI.Utils.GoogleUtility.UpdateLog(String serverName, String dbName, String batchId, String fileName, String status, String comment, ILogger log) in D:\a\1\s\ElancoCORE-MINT-AdsAPI\Utils\GoogleUtility.cs:line 559 ClientConnectionId:2d50069a-8614-4ecc-a700-2eb1f9ef330d Error Number:102,State:1,Class:15 ClientConnectionId before routing:dc4723fa-907f-4c38-a96a-a0d99bb001ef Routing Destination:e38c47c1813d.tr38588.eastus1-a.worker.database.windows.net,11014
At initial run for single day the code seems to be hanged
And again I ran same one more time, Thsi time got this error( I re-tried after 10 min of initial run).
Need help/resolution as soon as possible
Plzz.. at least respond for this issue. I'm receiving many escalations on this. I don't know what to answer clients.
Other than authors, If anyone found any solution for this please help me with this one. Any help would be appreciated. Thankss!!
In the last error message you pasted, the line
Unable to update the data to database: System.Data.SqlClient.SqlException (0x80131904): Incorrect syntax near ''.
Seems to indicate that there may be a syntax error in the SQL query you use to save to database the data you retrieve from the Google Ads API.
As an additional consideration that may improve your timeout issues: are you retrieving data from a GAQL query and storing it to database in the same azure function call?
Splitting the two processes in two different functions may be helpful, if that's the case.
No that error popped when we trigger twice at similar time otherwise only GRPC exceptions
Have you tried setting a custom timeout for your Google Ads API request as explained here?
No I will try that once.
But these issues came up when updated to V13 from V11 services.
@Raibaz, hanks for Helping!!
As per your suggestion I have tried out paged search as well, but when trying to retrieve past 10 days data, Its keep on running/hanging for hours.
But working fine for single day
code
-->
int pageSize = 10000;
SearchGoogleAdsRequest request = new SearchGoogleAdsRequest
{
CustomerId = googleAPIDetails.CustomerId.ToString(),
Query = query_geographic_view,
PageSize = pageSize
};
PagedEnumerable<SearchGoogleAdsResponse, GoogleAdsRow> response = googleAdsService.Search(request);
foreach (SearchGoogleAdsResponse page in response.AsRawResponses())
{
// Check if the page has a field mask.
if (page.FieldMask != null)
{
// Process the current page of results.
foreach (GoogleAdsRow googleAdsRow in page.Results)
{
rows.Add(googleAdsRow);
}
// Generate the CSV file using the field mask.
CsvFile csvFile = new CsvFile();
csvFile.Headers.AddRange(page.FieldMask.Paths);
foreach (GoogleAdsRow googleAdsRow in rows)
{
// Add the data to the CSV file.
csvFile.Records.Add(new string[]
{
googleAdsRow.GeographicView.LocationType.ToString(),
googleAdsRow.Campaign.Id.ToString(),
googleAdsRow.Campaign.Name.ToString(),
googleAdsRow.Campaign.ResourceName.ToString(),
googleAdsRow.AdGroup.Name.ToString(),
googleAdsRow.AdGroup.Id.ToString(),
googleAdsRow.Segments.Date.ToString(),
googleAdsRow.Segments.Device.ToString(),
googleAdsRow.Segments.GeoTargetMetro.ToString(),
googleAdsRow.Segments.GeoTargetPostalCode.ToString(),
googleAdsRow.Segments.GeoTargetRegion.ToString(),
googleAdsRow.Metrics.AllConversions.ToString(),
googleAdsRow.Metrics.AllConversionsValue.ToString(),
googleAdsRow.Metrics.Clicks.ToString(),
googleAdsRow.Metrics.CostMicros.ToString(),
googleAdsRow.Metrics.Impressions.ToString(),
googleAdsRow.Metrics.Interactions.ToString(),
googleAdsRow.Metrics.VideoViews.ToString(),
googleAdsRow.Metrics.ViewThroughConversions.ToString()
});
}
csvFile.Write(fileDirectory + fileName);
}
else
{
// Handle the case when there is no data for the submitted report request parameters.
nullReport = true;
log.LogError("Unable to generate Google " + shortFileName + " report: There is no data for the submitted report request parameters.");
// Other error handling code...
}
}
if (!nullReport)
{
byte[] fileByte = File.ReadAllBytes(fileDirectory + fileName);
await UploadToAzureStorageAsync(googleAPIDetails.ComConnectionString, googleAPIDetails.ComContainerName, fileName, fileByte, googleAPIDetails.ComFolderPath + "google_" + shortFileName + "/", log);
}
I'm wondering if this issue is only for me or for all the other users also...
Please help me on this. Thanks!!
Any other work around solution. Please suggest
@nitheeshls this seems like an API issue rather than a client library issue. Could you send an email to googleadsapi-support@google.com with the logs? If you can capture the request ID also, it would be most helpful.
Just mention that your queries have performance issues, and were redirected here, so escalate to L2.
Closing the stale issue, pls reopen if the issue persists.