Exceptions don't throw when cp fn called within core-async loop

[Dhruv Bhatia]

Hi there,

I've hit a strange issue where Exceptions don't throw when claypoole is called within a core-async go-loop.

Note how Example B doesn't throw the hit Exception to the REPL, whereas Example A does:

; EXAMPLE A
(defonce my-pool (cp/threadpool 3))
(defonce my-chan (async/chan 3))

(defn my-parallel-fn [job]
  (println "Do something with" job)
  (throw (Exception. (str "Some Exception: " job))))

(cp/upmap my-pool my-parallel-fn ["My Job"])

; repl throws the expected Exception -
;> Do something with My Job
;> java.lang.Exception: Some Exception: My Job

; EXAMPLE B
(defonce my-pool (cp/threadpool 3))
(defonce my-chan (async/chan 3))

(defn my-parallel-fn [job]
  (println "Do something with" job)
  (throw (Exception. (str "Some Exception: " job))))

(async/go-loop []
  (when-let [job (async/<! my-chan)]
    (cp/upmap my-pool my-parallel-fn [job])
    (recur)))

(async/put! my-chan "My Job")

; does not throw the expected exception to repl -
;> Do something with My Job

Help appeciated!

[Leon Barrett]

The problem is that upmap is creating a parallel sequence that will be evaluated in the background, but you're never accessing it, so you never see an exception. It's just like creating a future that throws an exception but never dereferencing it--you'll never see the exception thrown. It works in the REPL because the REPL forces the sequence, thereby getting the exception.

I tested this code, and I think if you put doall around the upmap, you'll see the exception you expect. Also, if you're just calling my-parallel-fn for the side effects, you might try cp/prun!.

Does that help?

[Dhruv Bhatia]

Thanks @leon-barrett - appreciate your guidance!

I had tried wrapping the upmap call with doall, and that does throw the Exception as per your explanation - however that also causes my-parallel-fn to be called in sequence rather than parallel. For example, if my-parallel-fn is a long-running fn, the entire return sequence appears to conduct each coll item one-at-a-time rather than concurrently, resulting in the doall'd version taking longer than than the standard version. prun! appears to have the same effect.

I suspect this issue is more to do with my understanding (or lack thereof) of parallelism in Clojure, rather than a claypoole specific issue, so feel free to close if you feel this is off-topic!

[Leon Barrett]

I'm not sure when you want to get the exception, and you can't be sure whether your long-running my-parallel-fn will throw an exception or not until it is complete. If you want that result in the main thread, that means you need to dereference it at some point to see if it's thrown an exception. Guessing a bit what you want, here are some options:

1. Have my-parallel-fn catch errors itself and handle them appropriately, maybe putting errors on a channel or something. That's simple and uses threads effectively.
2. Put the cp/upmap or cp/future result on a channel and dereference it later.
3. Use core.async's parallelism, e.g. pipeline. This is a great option since you're already working with core.async.

[Dhruv Bhatia]

Thanks once again for the help.
I've managed to achieve what I was after using core-async's pipeline, as per your recommendation. 👍