b/convert sometimes fails to copy the last byte array

Question

b/convert sometimes fails to copy the last byte array

malcolmsparks opened this issue 9 years ago · comments

Let coll be a collection of (large) byte arrays.

What I'm seeing is (b/convert coll (class (byte-array 0))) occasionally returns a combined byte-array but without the last byte-array in coll. I'm seeing this only sometimes. My test suite will pass, then it will fail, then it will pass again. It seems to fail about 25% of the time.

I have identified the problem to be b/convert because when I replace the call to b/convert with this logic, the bug disappears. This logic is my own hurried attempt to System/arraycopy all the bytes together, which is what I want b/convert to do for me.

(let [rs (reductions + (map count coll))
       result (byte-array (last rs))]

    (reduce (fn [_ [source offset]]
              (System/arraycopy source 0 result offset (count source)))
            nil
            (map vector coll (cons 0 rs)))
    result)

I'm sorry to raise an issue without providing the means to reliably replicate the problem. Feel free to close it, but I'm raising it more as a note in case anyone else has an idea. From reasoning about the code, is there anything that might account for the behavior I'm seeing?

Malcolm Sparks · Answer 1 · Tue Nov 10 2015 07:46:08 GMT+0800 (China Standard Time)

Is there a way of telling the conversion path that a collection of byte-arrays to a single byte-array would take? It feels like there's something occasionally not being flushed.

Zach Tellman · Answer 2 · Tue Nov 10 2015 12:51:40 GMT+0800 (China Standard Time)

What version are you using, and is the input a sequence or vector of byte arrays?

Malcolm Sparks · Answer 3 · Tue Nov 10 2015 22:50:30 GMT+0800 (China Standard Time)

Hi - version is 0.2.0, I'm using a sequence but could easily be a vector.

Malcolm Sparks · Answer 4 · Wed Nov 11 2015 08:42:24 GMT+0800 (China Standard Time)

More info: The problem never occurs when a vector is used. If I use a sequence, however, it fails occasionally/erratically. Here are the results of my test-suite.

FFPPPFFPPPPPPFPPPPPPPPPPF

where F=FAIL, P=PASS.

Zach Tellman · Answer 5 · Wed Nov 11 2015 08:57:35 GMT+0800 (China Standard Time)

Can you test with the latest 0.2.1 beta? If it still fails, can you provide a reproducing case for me to run?

Malcolm Sparks · Answer 6 · Wed Nov 11 2015 09:36:14 GMT+0800 (China Standard Time)

Testing with 0.2.1-alpha1 seems to cause occasionally hangs on my test-suite. I've tried a few different combinations, looks like occasionally the thread is blocking. I'll investigate further the exact nature of these hangs, and try to distill a better failing test.

Zach Tellman · Answer 7 · Tue Nov 17 2015 15:46:24 GMT+0800 (China Standard Time)

So I've been trying to reproduce this on my end, using some variant of

(bs/to-byte-array (repeatedly 1e3 #(byte-array 1e5)))

I'm not able to, but I did notice that it hung when I ran this inside the byte-streams namespace, since byte-array is shadowed there. An error was logged, but it was otherwise stuck. This obviously shouldn't happen, and I've made a fix such that it logs and actually returns, but maybe this is why it was hanging for you? Let me know.

Malcolm Sparks · Answer 8 · Tue Nov 17 2015 21:07:05 GMT+0800 (China Standard Time)

I believe I've reduced the problem down to a simple example here:
https://github.com/malcolmsparks/bs-bug/blob/master/src/bs_bug/core.clj

If you clone the repo

git clone git@github.com:malcolmsparks/bs-bug.git

This might not be the actual issue described above, or might be closely related to it.

If you try the bs_bug/core ns, you'll see the problem I'm having. It could be
that this is normal expected behavior from byte-streams and I'm assuming
this is a bug when it isn't.

The wider context is that I have to use this ->closing-stream function in
my tests that use ring-mock, because the request :body is an input stream.
This causes my tests to hang, but the actual code is working fine with real
browsers. It's a big of a snag, but isn't a huge issue, but would be nice
to understand why it's hanging.

Zach Tellman · Answer 9 · Thu Nov 19 2015 04:03:01 GMT+0800 (China Standard Time)

This looks like a bug in the stream lifecycle handling. I'll take a closer look. Thanks for the reproducing case.

Zach Tellman · Answer 10 · Tue Nov 24 2015 11:23:48 GMT+0800 (China Standard Time)

Okay, this was a little stranger than I expected. The problems were two-fold: manifold.stream/reduce doesn't return an exception when given a non-stream input, and the closeable-seq implementation in byte-streams assumed everything it closed over was clojure.lang.IPending. Both of these are fixed, in [byte-streams "0.2.1-alpha2"] and [manifold "0.1.2-alpha1"], respectively. You'll need to wrap get-buffers in s/->source for your failing case to work, though.

Malcolm Sparks · Answer 11 · Fri Nov 27 2015 18:31:21 GMT+0800 (China Standard Time)

Hi Zach, I've updated to both library versions and this has been working
well. Thanks, Malcolm

Malcolm Sparks
Director

Email: malcolm@juxt.pro
Web: https://juxt.pro

JUXT LTD.
Software Consulting, Delivery, Training

On 24 November 2015 at 03:23, Zach Tellman notifications@github.com wrote:

Okay, this was a little stranger than I expected. The problems were
two-fold: manifold.stream/reduce doesn't return an exception when given a
non-stream input, and the closeable-seq implementation in byte-streams
assumed everything it closed over was clojure.lang.IPending. Both of
these are fixed, in [byte-streams "0.2.1-alpha2"] and [manifold
"0.1.2-alpha1"], respectively. You'll need to wrap get-buffers in
s/->source for your failing case to work, though.

—
Reply to this email directly or view it on GitHub
#20 (comment)
.