Accumulate Weird Behaviour

[David Byrne]

I'm still relatively new to TDF, so this behaviour could just be something beyond my understanding (of either TDF or maybe even Rust), but I am trying to use the accumulate operator on a stream, and I've run into some seemingly odd behaviour. When running the following code:

     worker.dataflow(|scope| {
            scope.input_from(&mut input)
                .broadcast() // everyone sends their local totals to each other
                .accumulate( // everyone calculates the new global means
                    vec!(),
                    move |totals: &mut Vec<Point>, 
                    locals: timely_communication::message::RefOrMut<Vec<Vec<(Point, usize)>>>| {
                    // |totals, locals| {
                        let mut sums = vec!();
                        for local in locals.iter() {
                            println!("Worker {} sums = {:?}", index2, sums);
                            for (i, pair) in local.iter().enumerate() {
                                println!("{:?} on iter {}", index2, i);
                                if sums.len() <= i {
                                    println!("{:?} adding {:?}", index2, pair);
                                    sums.push((pair.0.clone(), pair.1));
                                } else {
                                    sums[i].0.add(&pair.0);
                                    sums[i].1 += pair.1;
                                }
                            }
                        }
                        for sum in sums.iter() {
                            totals.push(sum.0.scalar_div(&(sum.1 as f64)));
                        }
                    }
                )
                .inspect(move |v| println!("worker {} sees {:?}", index, v))
                .probe_with(&mut probe);
        });

        for i in 0..10 {
            println!("worker {} sending round {}", index, i);
            input.send(vec!(
                // these pairs will be the "sum" and "count" of values
                // associated with each mean (in this case, two)
                (Point::new(i as f64+5.0, i as f64), index + 1), // first mean
                (Point::new(-i as f64-5.0, i as f64), index + 1) // second mean
            ));
            input.advance_to(input.epoch() + 1);
            while probe.less_than(input.time()) {
                worker.step();
            }
        }

(here's a link to the actual code without print statements and whatnot) I get output along the lines of this:

worker 0 sending round 0
worker 1 sending round 0
Worker 0 sums = []
0 on iter 0
0 adding ((5.00,0.00), 1)
0 on iter 1
0 adding ((-5.00,0.00), 1)
Worker 1 sums = []
1 on iter 0
1 adding ((5.00,0.00), 1)
1 on iter 1
1 adding ((-5.00,0.00), 1)
Worker 1 sums = [((5.00,0.00), 1), ((-5.00,0.00), 1)]
1 on iter 0
1 on iter 1
Worker 0 sums = []
0 on iter 0
0 adding ((5.00,0.00), 2)
0 on iter 1
0 adding ((-5.00,0.00), 2)
worker 0 sees [(5.00,0.00), (-5.00,0.00), (2.50,0.00), (-2.50,0.00)]
worker 1 sees [(1.67,0.00), (-1.67,0.00)]
worker 0 sending round 1
worker 1 sending round 1

What I'm expecting is that the "sums" vector maintains the inserted values from the previous "local" in the "locals" iterator. However, based on the output it seems (at random) the sums vector is getting wonky, since "len" must be reporting a length that is apparently incorrect given the output of printing the debugged vec.

Honestly, the longer I think and type this out, the weirder the whole situation seems. I'm chalking this up to be that either I don't understand accumulate or broadcast properly and that I'm inducing some sort of race condition. Any help would be appreciated.

[David Byrne]

Never mind! After looking at the source for accumulate I realised why. My sums variable would leave scope if the closure would be called before all workers send their data, then be re-created when the next worker in that round would send data. I suppose I'm still struggling to intuitively understand what's going on when the code actually runs. Closing because this was just a case of bad design. For anyone curious, I have updated like so:

            .accumulate( // everyone calculates the new global means
                    vec!(),
                    move |totals: &mut Vec<(Point, usize)>, 
                    locals: timely_communication::message::RefOrMut<Vec<(Point, usize, usize)>>| {
                        // for local in locals.iter() {
                        //     println!("Worker {} sums = {:?}", index2, sums);
                        for pair in locals.iter() {
                            let i = pair.2;
                            // println!("{:?} on mean {} sees sum.len() = {}", index2, i, totals.len());
                            if totals.len() <= i {
                                // println!("{:?} adding {:?}", index2, pair);
                                totals.push((pair.0.clone(), pair.1));
                            } else {
                                totals[i].0.add(&pair.0);
                                totals[i].1 += pair.1;
                            }
                        }
                    }
                )
                .map(|point_sums| {
                    point_sums
                    .into_iter()
                    .map(|point_sum| point_sum.0.scalar_div(&(point_sum.1 as f64)))
                    .collect::<Vec<Point>>()
                })