goccy / bigquery-emulator

BigQuery emulator server implemented in Go

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Unable to use UploadCsvAsync

jeroen-corsius-choreograph opened this issue · comments

What happened?

Trying to upload a CSV file fails.

System.Net.Http.HttpRequestException: IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Pa...

System.Net.Http.HttpRequestException
IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName') (0.0.0.0:9050)
   at Google.Cloud.BigQuery.V2.BigQueryClientImpl.UploadDataAsync(JobConfigurationLoad loadConfiguration, Stream input, String contentType, JobCreationOptions options, CancellationToken cancellationToken)
   at Google.Cloud.BigQuery.V2.BigQueryClientImpl.UploadCsvAsync(TableReference tableReference, TableSchema schema, Stream input, UploadCsvOptions options, CancellationToken cancellationToken)
   at Plugin.Google.Test.Integration.Export.ExampleTest.Example() in C:\Projects\badger\services\Plugin.Google.Test\Integration\Export\TemptTest.cs:line 37
   at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass46_0.<<InvokeTestMethodAsync>b__1>d.MoveNext() in /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 253
--- End of stack trace from previous location ---
   at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 48
   at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in /_/src/xunit.core/Sdk/ExceptionAggregator.cs:line 90

System.ArgumentException
IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName')
   at System.Net.Dns.GetHostEntryOrAddressesCoreAsync(String hostName, Boolean justReturnParsedIp, Boolean throwOnIIPAny, Boolean justAddresses, AddressFamily family, CancellationToken cancellationToken)
   at System.Net.Dns.GetHostAddressesAsync(String hostNameOrAddress, AddressFamily family, CancellationToken cancellationToken)
   at System.Net.Sockets.SocketAsyncEventArgs.DnsConnectAsync(DnsEndPoint endPoint, SocketType socketType, ProtocolType protocolType)
   at System.Net.Sockets.Socket.ConnectAsync(SocketAsyncEventArgs e, Boolean userSocket, Boolean saeaCancelable)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)

What did you expect to happen?

The action succeeding

How can we reproduce it (as minimally and precisely as possible)?

using System.Net;
using System.Text;
using Google.Cloud.BigQuery.V2;
using Testcontainers.BigQuery;
using Xunit;

namespace Plugin.Google.Test.Integration.Export;

public class ExampleTest {
  [Fact]
  public async Task Example() {
    var projectName = "some_project";
    var datasetName = "some_dataset";
    var tableName = "some_table";

    // Create/start container
    var bigQueryContainer = new BigQueryBuilder().WithProject(projectName).WithReuse(true).Build();
    await bigQueryContainer.StartAsync();
    
    // Create client
    var bigQueryClient = await new BigQueryClientBuilder {
      BaseUri = bigQueryContainer.GetEmulatorEndpoint(),
      ProjectId = projectName,
    }.BuildAsync();

    // Create dataset
    await bigQueryClient.CreateDatasetAsync(datasetName);
    
    // Create table
    var createTableQuery = $"CREATE TABLE IF NOT EXISTS {datasetName}.{tableName} (id INT64, name STRING);";
    await bigQueryClient.ExecuteQueryAsync(createTableQuery, null);
    
    // Create CSV memory stream
    var csvData = "1,John\n2,Jane\n3,Michael";
    var csvMemoryStream = new MemoryStream(Encoding.UTF8.GetBytes(csvData));
    
    // Upload CSV
    var uploadCsvJob = await bigQueryClient.UploadCsvAsync(datasetName, tableName, null, csvMemoryStream);
    await uploadCsvJob.PollUntilCompletedAsync();
  }
}

Anything else we need to know?

NuGet package version: 3.8.0

Workaround:
If I replace the "Create client" part of the provided example with the code below, I'm able to upload CSV files:

    // Create client
    var mappedPublicPort = bigQueryContainer.GetMappedPublicPort(BigQueryBuilder.BigQueryPort);
    var exposedContainerAddress = $"localhost:{mappedPublicPort}";
    var bigQueryClient = await new BigQueryClientBuilder {
      BaseUri = bigQueryContainer.GetEmulatorEndpoint(),
      ProjectId = projectName,
      HttpClientFactory = HttpClientFactory.ForProxy(new WebProxy(exposedContainerAddress)),
    }.BuildAsync();

I'm sorry, I've created the issue on the wrong repository (should have been: testcontainers/testcontainers-dotnet#1175).