Concurrent writes to the same S3 key could leave invalid (empty) objects

[Igor Bondarenko]

Dependencies

{:deps {com.cognitect.aws/api       {:mvn/version "0.8.641"}
        com.cognitect.aws/endpoints {:mvn/version "1.1.12.398"}
        com.cognitect.aws/s3        {:mvn/version "825.2.1250.0"}}}

Description with failing test case

Concurrent writes to the same S3 key could leave invalid (empty) objects without reporting any errors to the caller. Seems to happen only if body passed to :PutObject is an InputStream.

Here's how to reproduce:

(require '[clojure.java.io :as io]
         '[cognitect.aws.client.api :as aws])
(import '[java.io File])

(def b "test-bucket")
(def k "s3zero.test")
(def c (aws/client {:api :s3}))

(def f (File/createTempFile "s3zero" ""))
(spit f "TEST")

(defn put [f]
  (aws/invoke c {:op :PutObject :request {:Bucket b :Key k :Body (-> f io/input-stream)}}))

(defn run [max-n]
  (println "run" max-n)
  (let [barrier (java.util.concurrent.CyclicBarrier. 3)
        one     (future (.await barrier) (put f))
        two     (future (.await barrier) (put f))
        three   (future (.await barrier) (put f))
        res     [@one @two @three]
        etags   (into #{} (map :ETag) res)]
    (cond
      (> (count etags) 1) [:fail res]
      (zero? max-n)       [:ok res]
      :else               (recur (dec max-n)))))

(println (run 10))
(println (aws/invoke c {:op :GetObject :request {:Bucket b :Key k}}))

Run it and you'll get something like this (re-run a few times if it didn't fail, but usually it does):

clj -Sdeps '{:deps {com.cognitect.aws/api {:mvn/version "0.8.641"} com.cognitect.aws/endpoints {:mvn/version "1.1.12.398"} com.cognitect.aws/s3 {:mvn/version "825.2.1250.0"}}}' -M test.clj
run 10
run 9
run 8
run 7
[:fail [{:ETag "033bd94b1168d7e4f0d644c3c95e35bf", :ServerSideEncryption AES256} {:ETag "033bd94b1168d7e4f0d644c3c95e35bf", :ServerSideEncryption AES256} {:ETag "d41d8cd98f00b204e9800998ecf8427e", :ServerSideEncryption AES256}]]
{:LastModified #inst "2023-02-12T15:22:38.000-00:00", :ETag "d41d8cd98f00b204e9800998ecf8427e", :Metadata {}, :ServerSideEncryption AES256, :ContentLength 0, :ContentType application/octet-stream, :AcceptRanges bytes, :Body nil}

Note the different ETag in the last response and :ContentLength 0, :Body nil in the :GetObject response.

Passing byte array as a :Body seems to fix the issue. Note .readAllBytes to convert input-stream to the byte array first.

(defn put [f]
  (aws/invoke c {:op :PutObject :request {:Bucket b :Key k :Body (-> f io/input-stream .readAllBytes)}}))

Re-run:

clj -Sdeps '{:deps {com.cognitect.aws/api {:mvn/version "0.8.641"} com.cognitect.aws/endpoints {:mvn/version "1.1.12.398"} com.cognitect.aws/s3 {:mvn/version "825.2.1250.0"}}}' -M test.clj
run 10
run 9
run 8
run 7
run 6
run 5
run 4
run 3
run 2
run 1
run 0
[:ok [{:ETag "033bd94b1168d7e4f0d644c3c95e35bf", :ServerSideEncryption AES256} {:ETag "033bd94b1168d7e4f0d644c3c95e35bf", :ServerSideEncryption AES256} {:ETag "033bd94b1168d7e4f0d644c3c95e35bf", :ServerSideEncryption AES256}]]
{:LastModified #inst "2023-02-12T15:22:16.000-00:00", :ETag "033bd94b1168d7e4f0d644c3c95e35bf", :Metadata {}, :ServerSideEncryption AES256, :ContentLength 4, :ContentType application/octet-stream, :AcceptRanges bytes, :Body #object[java.io.BufferedInputStream 0x48b9ee43 java.io.BufferedInputStream@48b9ee43]}

Object now has proper :ContentLength and body is there.

[David Chelimsky]

Thanks for the repro!

As I assume you've surmised, the problem comes from aws-api reading the InputStream into a byte[] before io/input-stream has consumed all the bytes in f. There is no error for AWS to report because it's valid to :PutObject with an empty value, so there's nothing for us to do there.

We could try to ensure we read all the bytes, but I modified the code in aws-api to use .readAllBytes and this problem persisted, so I'm not quite sure how we could do it.

[Igor Bondarenko]

Thanks for prompt response!

Yeah, that was my thinking but I didn't manage to reproduce it without actually writing to S3, so something more intricate might be at play here.

One of the things I wanted to look at is how underlying http client handles the body but it seems like it's code isn't public?..

Anyway, revisiting it again I think I know what the problem is. See, util/->bbuf is called inside the send-request, which could be retried and will try to read the same input stream again, resulting in empty ByteBuffer.

[David Chelimsky]

One of the things I wanted to look at is how underlying http client handles the body but it seems like it's code isn't public?

It is, but not in the way everybody has grown accustomed: the source is in plain text in the jar. It's just not conveniently posted on a site like github.

Anyway, revisiting it again I think I know what the problem is. See, util/->bbuf is called inside the send-request, which could be retried and will try to read the same input stream again, resulting in empty ByteBuffer.

Brilliant! I didn't think of that as I was perusing. I'll look into getting that fixed.

[David Chelimsky]

Fixed in b6c861d. I'll close this when we release. Should be some time this week or early next.

[David Chelimsky]

Released in v 0.8.652