How to keep original order when using pool?
dalianzhu opened this issue · comments
Excuse me, I want to process each element concurrently, but keep the original order in the final operation, what should I do?
thanks for your help.
error code:
s := []int{1, 2, 3, 4, 5, 9, 10, 11}
<-rxgo.Create([]rxgo.Producer{func(ctx context.Context, next chan<- rxgo.Item) {
for _, item := range s {
next <- rxgo.Of(item)
}
}}).
Map(func(c context.Context, i interface{}) (interface{}, error) { // map 1
tp := i.(int)
// Simulate a time-consuming operation
time.Sleep(time.Millisecond * time.Duration(rand.Intn(10)))
return tp, nil
}, rxgo.WithPool(10), rxgo.WithBufferedChannel(10)).
DoOnNext(func(i interface{}) {
fmt.Printf("%v\n", i) // i want to print 1,2,3,4,5,9,10,11
})
have you tried rxgo.Serialize? https://github.com/ReactiveX/RxGo/blob/master/doc/options.md#serialize
@gungdr Yes, but using Serialize requires a continuous ID.
s := []int{1, 2, 3, 4, 5, 9, 10, 11}
<-rxgo.Create([]rxgo.Producer{func(ctx context.Context, next chan<- rxgo.Item) {
for _, item := range s {
next <- rxgo.Of(item)
}
}}).
Map(func(c context.Context, i interface{}) (interface{}, error) { // map 1
tp := i.(int)
// Simulate a time-consuming operation
time.Sleep(time.Millisecond * time.Duration(rand.Intn(10)))
return tp, nil
}, rxgo.WithPool(10), rxgo.WithBufferedChannel(10), rxgo.Serialize(func(i interface{}) int {
return i.(int)
})).
DoOnNext(func(i interface{}) {
fmt.Printf("%v\n", i) // i want to print 1,2,3,4,5,9,10,11
})
The results are 1 2 3 4 5 （9, 10, 11 is missing）
If I try to assign a continuous ID to each data, it will be very difficult to ensure that it is continuous, because there are many exceptions to be handled.
I have the same problem, seems like the observable suffers from premature death using rxgo.Serialize. @teivah any idea why this happens?
The role of rxgo.Serialize is to guarantee order in the value, thus it needs complete continuity in the indexes. It has no way to guess which indexes it's supposed to drop.
You can solve your problem by separating index from value. Here i used a richer Item type for this.
package main
import (
"context"
"fmt"
"math/rand"
"time"
"github.com/reactivex/rxgo/v2"
)
type Item struct {
Idx int
Value int
}
func main() {
s := []int{1, 2, 3, 4, 5, 9, 10, 11}
<-rxgo.Create([]rxgo.Producer{func(ctx context.Context, next chan<- rxgo.Item) {
for i, item := range s {
next <- rxgo.Of(Item{Idx: i, Value: item})
}
}}).
Map(func(c context.Context, i interface{}) (interface{}, error) { // map 1
tp := i.(Item)
// Simulate a time-consuming operation
time.Sleep(time.Millisecond * time.Duration(rand.Intn(10)))
return tp, nil
}, rxgo.WithPool(10), rxgo.WithBufferedChannel(10), rxgo.Serialize(func(i interface{}) int {
return i.(Item).Idx
})).
DoOnNext(func(i interface{}) {
fmt.Printf("%v\n", i.(Item).Value) // i want to print 1,2,3,4,5,9,10,11
})
}If I try to assign a continuous ID to each data, it will be very difficult to ensure that it is continuous, because there are many exceptions to be handled.
After re-reading, i think that what you want to say is that the Serialize is broken if you return an error. That is also true in my example. Here's a (not ideal) workaround where you never leverage the error mechanism, instead you track errors in the custom item.
package main
import (
"context"
"errors"
"fmt"
"github.com/reactivex/rxgo/v2"
)
type Item struct {
Idx int
Value string
Error error
}
func main() {
<-rxgo.
Create(
[]rxgo.Producer{func(ctx context.Context, next chan<- rxgo.Item) {
// simulate item production
strings := []string{"hello", "error", "world"}
for i, s := range strings {
next <- rxgo.Of(Item{Idx: i, Value: s})
}
}},
).
Map(
func(c context.Context, i interface{}) (interface{}, error) {
item := i.(Item)
// simulate error cases
if item.Value == "error" {
item.Error = errors.New("oops")
}
return item, nil
},
rxgo.WithPool(10),
rxgo.WithBufferedChannel(10),
rxgo.Serialize(
func(i interface{}) int {
return i.(Item).Idx
},
),
).
DoOnNext(
func(i interface{}) {
item := i.(Item)
// handle error cases
if item.Error != nil {
return
}
fmt.Println(i)
},
)
}