storage: when using an emulator, it is not possible to use the same Client object for both uploading and other operations
justinruggles opened this issue · comments
Client
Storage
Environment
MacOS 10.14.6
Go Environment
$ go version
go version go1.14.1 darwin/amd64
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/REDACTED/Library/Caches/go-build"
GOENV="/Users/REDACTED/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY="REDACTED"
GONOSUMDB="REDACTED"
GOOS="darwin"
GOPATH="/Users/REDACTED/go"
GOPRIVATE="REDACTED"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="REDACTED/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/03/wp97sc8538b75zs4q619t5x00000gn/T/go-build425541068=/tmp/go-build -gno-record-gcc-switches -fno-common"
Code
package main
import (
"context"
"encoding/json"
"flag"
"io"
"log"
"os"
"path/filepath"
"cloud.google.com/go/storage"
"google.golang.org/api/option"
)
const TestBucket = "test-bucket"
const TestProject = "test-project"
func uploadFile(ctx context.Context, filename string, project string, bucket string, object string) error {
os.Setenv("STORAGE_EMULATOR_HOST", "localhost:8080")
defer os.Unsetenv("STORAGE_EMULATOR_HOST")
client, clientErr := storage.NewClient(ctx,
option.WithEndpoint("http://localhost:8080/storage/v1/"))
if clientErr != nil {
return clientErr
}
file, openErr := os.Open(filename)
if openErr != nil {
return openErr
}
defer file.Close()
obj := client.Bucket(bucket).Object(object)
w := obj.NewWriter(ctx)
_, copyErr := io.Copy(w, file)
if copyErr != nil {
return copyErr
}
closeErr := w.Close()
if closeErr != nil {
return closeErr
}
log.Printf("upload successful")
obj = client.Bucket(bucket).Object(object)
attrs, attrsErr := obj.Attrs(ctx)
if attrsErr != nil {
return attrsErr
}
objectJSON, _ := json.MarshalIndent(attrs, "", " ")
log.Printf("%s", string(objectJSON))
return nil
}
func main() {
flag.Parse()
filename := flag.Arg(0)
objName := filepath.Base(filename)
ctx := context.Background()
err := uploadFile(ctx, filename, TestProject, TestBucket, objName)
if err != nil {
log.Println(err)
os.Exit(1)
}
}
GCS emulator request logs (pretty-printed for ease of reading)
{
"time": "2020-06-17T21:20:19.012239Z",
"severity": "INFO",
"httpRequest": {
"status": 200,
"requestMethod": "POST",
"requestUrl": "/upload/storage/v1/b/test-bucket/o?alt=json&name=test.gif&prettyPrint=false&projection=full&uploadType=multipart",
"userAgent": "google-api-go-client/0.5"
},
"httpHeaders": {
"Accept-Encoding": "gzip",
"Content-Type": "multipart/related; boundary=e4160c25b64069901887390168eabbda53b5192962a4c418be63c16f2d70",
"X-Goog-Api-Client": "gl-go/1.14.1 gccl/20200417"
},
"httpQuery": {
"alt": "json",
"name": "test.gif",
"prettyPrint": "false",
"projection": "full",
"uploadType": "multipart"
},
"logging.googleapis.com/trace": "65698c2757a78a529c576f3212c1ab1d",
"logging.googleapis.com/trace_sampled": true,
"message": "200 OK"
}
{
"time": "2020-06-17T21:20:19.012702Z",
"severity": "ERROR",
"httpRequest": {
"status": 404,
"requestMethod": "GET",
"requestUrl": "/b/test-bucket/o/test.gif?alt=json&prettyPrint=false&projection=full",
"userAgent": "google-api-go-client/0.5"
},
"httpHeaders": {
"Accept-Encoding": "gzip",
"X-Goog-Api-Client": "gl-go/1.14.1 gccl/20200417"
},
"httpQuery": {
"alt": "json",
"prettyPrint": "false",
"projection": "full"
},
"logging.googleapis.com/trace": "9bbe57ce2a19f3c273f0b78dfbba215a",
"logging.googleapis.com/trace_sampled": true,
"message": "code=404, message=Not Found"
}
Expected behavior
When performing client operations using a GCS emulator, one should be able to use a single Client object for both uploading and general operations even though the endpoint prefixes differ.
Actual behavior
When using a GCS emulator from a client application, one must set the STORAGE_EMULATOR_HOST
environment variable and use option.WithEndpoint
when creating the Client
with storage.NewClient
. For general operations, the endpoint specified must include the path prefix /storage/v1
(e.g. http://localhost:8080/storage/v1
) if the emulator uses the same prefix as the public JSON API. Doing this results in the correct value being set for BasePath in the underlying raw client. However, when performing an upload operation, the BasePath is overridden to remove the path component (
google-cloud-go/storage/writer.go
Lines 128 to 130 in 89ef506
/upload/storage/v1
) when making the API call. The result is that any general operations performed after uploading using the same Client object will fail because they no longer use the full /storage/v1
path.
To work around this, I attempted to make the emulator serve the endpoints without the prefix, but then the upload operations failed because the prefix is always added to those requests in the client.
Additional Context
I'm using a GCS emulator written for my employer that is not yet publicly available.
@justinruggles thanks for the detailed report! Could you let me know what versions of cloud.google.com/go/storage and google.golang.org/api you are using in your application?
cloud.google.com/go/storage v1.9.0
google.golang.org/api v0.26.0
I was able to work around the issue by using a http.Client
(via option.WithHTTPClient
) with a custom http.RoundTripper
that overrides the URL Scheme and Host, rather than using STORAGE_EMULATOR_HOST
and option.WithEndpoint
. But it's not a great solution. Ideally it would be nice to only set STORAGE_EMULATOR_HOST
and not worry about manually setting the endpoint.
Here's an example should anyone come across this issue:
type roundTripper url.URL
func (rt roundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
req.Host = rt.Host
req.URL.Host = rt.Host
req.URL.Scheme = rt.Scheme
return http.DefaultTransport.RoundTrip(req)
}
func main() {
u, _ := url.Parse("http://localhost:8000/")
hClient := &http.Client{Transport: roundTripper(*u)}
gClient, err := storage.NewClient(context.Background(), option.WithHTTPClient(hClient))
}
Looks like those three lines in storage/writer.go
are not needed at all. Service.BasePath
is not modified elsewhere and is read only by googleapi.ResolveRelative()
function which is called by generated doRequest()
s. Upload requests paths are handled as absolute paths, so there is no need to "fix" that field.
Or maybe I am missing something?
I've ran into this issue while trying to use this emulator in http mode (i.e. w/o https). If I run the client without STORAGE_EMULATOR_HOST
env var, it tries to use https for some operations and fails. If I run the client with STORAGE_EMULATOR_HOST
- BasePath
gets overwritten and requests get sent with incorrect URL paths. And if I run the emulator in https mode, I get certificate errors :)
I ended up locally commenting out the if
statement mentioned above in order to get some work done. Can we please get some attention to this bug?
Apologies for the delay on this-- I'm working on a resolution!
@tritone Hi! May be helpfull for your works
firebase/firebase-tools#3556
PS Can you tell me when the emulator will officially support go sdk?
NB: when running under go -race
, this actually causes a read/write data race entirely within Google code client if you try to use STORAGE_EMULATOR_HOST
.
Hi everyone, this issue has been fixed and will be in the next release. You should then be able to use a Storage Client for all operations, including upload.
Ideally it would be nice to only set
STORAGE_EMULATOR_HOST
and not worry about manually setting the endpoint.
Additionally, @justinruggles, we implemented this suggestion. Thank you for bringing our attention to this issue!
@BrennaEpp was this #4608 ?
#4608 fixes the original upload issue, correct.
#4616 adds @justinruggles's suggestion on allowing users to set STORAGE_EMULATOR_HOST
only without having to set the endpoint.