Concurrent map writes of search attributes due to potential racing condition
longquanzheng opened this issue · comments
Describe the bug
Upseart search attributes could got error of concurrent map writes
To Reproduce
Is the issue reproducible?
- Yes, but not stable
The same test got pass in Temporal, but sometimes fail with Cadence.
Steps to reproduce the behavior:
Uncomment the test for Cadence Run the integ test from iWF
Expected behavior
The integ test should succeed.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
runtime.throw({0x103003d8f?, 0x1400084d7e8?})
/usr/local/go-dists/1.18.3/go/src/runtime/panic.go:992 +0x50 fp=0x14000c4ae90 sp=0x14000c4ae60 pc=0x1021b2500
runtime.mapassign_faststr(0x102fe0d6b?, 0x103fadc00?, {0x103001b1f, 0x14})
/usr/local/go-dists/1.18.3/go/src/runtime/map_faststr.go:212 +0x3bc fp=0x14000c4af00 sp=0x14000c4ae90 pc=0x10219016c
go.uber.org/cadence/internal.mergeSearchAttributes(...)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_event_handlers.go:354
go.uber.org/cadence/internal.(*workflowEnvironmentImpl).updateWorkflowInfoWithSearchAttributes(...)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_event_handlers.go:339
go.uber.org/cadence/internal.(*workflowEnvironmentImpl).UpsertSearchAttributes(0x1400091c900, 0x102a26c8c?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_event_handlers.go:334 +0x258 fp=0x14000c4aff0 sp=0x14000c4af00 pc=0x1029e4f38
go.uber.org/cadence/internal.(*workflowEnvironmentInterceptor).UpsertSearchAttributes(0x14000917980, {0x1032f8e20?, 0x103562ea0?}, 0x10344b7a0?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/workflow.go:1347 +0x94 fp=0x14000c4b030 sp=0x14000c4aff0 pc=0x102a26d84
go.uber.org/cadence/internal.UpsertSearchAttributes({0x10357a788, 0x140005e7f40}, 0x140005e7f40?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/workflow.go:1340 +0x78 fp=0x14000c4b070 sp=0x14000c4b030 pc=0x102a26ca8
go.uber.org/cadence/workflow.UpsertSearchAttributes(...)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/workflow/workflow.go:452
github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).UpsertSearchAttributes(0x10335ccc0?, {0x1035656b8?, 0x140005f1570?}, 0x14?)
/Users/qlong/indeed/iwf/iwf-server/service/interpreter/cadence/workflowProvider.go:28 +0x54 fp=0x14000c4b0c0 sp=0x14000c4b070 pc=0x102a37cb4
github.com/indeedeng/iwf/service/interpreter.(*stateExecutingManager).updateSearchAttribute(0x1400092e0c0)
/Users/qlong/indeed/iwf/iwf-server/service/interpreter/stateExecutingManager.go:60 +0x19c fp=0x14000c4b1c0 sp=0x14000c4b0c0 pc=0x10269d64c
github.com/indeedeng/iwf/service/interpreter.(*stateExecutingManager).completeStates(0x1400092e0c0, {{0x14000297918?, 0x140004f6750?}, 0x140005c5e90?, 0x140005c5ea0?})
/Users/qlong/indeed/iwf/iwf-server/service/interpreter/stateExecutingManager.go:46 +0xb0 fp=0x14000c4b200 sp=0x14000c4b1c0 pc=0x10269d450
github.com/indeedeng/iwf/service/interpreter.InterpreterImpl.func3.1()
/Users/qlong/indeed/iwf/iwf-server/service/interpreter/workflowImpl.go:83 +0x40 fp=0x14000c4b240 sp=0x14000c4b200 pc=0x10269ea40
runtime.deferCallSave(0x14000c4b2e0, 0x14000c4be68?)
/usr/local/go-dists/1.18.3/go/src/runtime/panic.go:750 +0x8c fp=0x14000c4b250 sp=0x14000c4b240 pc=0x1021b1d3c
runtime.runOpenDeferFrame(0x0?, 0x1400073a0a0)
/usr/local/go-dists/1.18.3/go/src/runtime/panic.go:723 +0x1b8 fp=0x14000c4b2a0 sp=0x14000c4b250 pc=0x1021b1b68
runtime.Goexit()
/usr/local/go-dists/1.18.3/go/src/runtime/panic.go:482 +0x188 fp=0x14000c4b320 sp=0x14000c4b2a0 pc=0x1021b11f8
go.uber.org/cadence/internal.(*coroutineState).exit.func1({0x14000c4b368?, 0x102a0fbe0?}, 0x14000746d90?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:850 +0x20 fp=0x14000c4b330 sp=0x14000c4b320 pc=0x102a11040
go.uber.org/cadence/internal.(*coroutineState).initialYield(0x140005f7a80, 0x3, {0x1400027e9e0, 0x19})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:797 +0xbc fp=0x14000c4b370 sp=0x14000c4b330 pc=0x102a10c0c
go.uber.org/cadence/internal.(*coroutineState).yield(...)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:806
go.uber.org/cadence/internal.(*channelImpl).Receive(0x1400072dc20, {0x10357a980, 0x1400092e270}, {0x0, 0x0})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:621 +0x214 fp=0x14000c4b400 sp=0x14000c4b370 pc=0x102a0fbf4
go.uber.org/cadence/internal.(*decodeFutureImpl).Get(0x14000738408, {0x10357a980, 0x1400092e270}, {0x1032af200?, 0x14000011578})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:1282 +0x4c fp=0x14000c4b4a0 sp=0x14000c4b400 pc=0x102a148cc
github.com/indeedeng/iwf/service/interpreter/cadence.(*futureImpl).Get(0x140005f1960, {0x1035656b8?, 0x140005f1740?}, {0x1032af200, 0x14000011578})
/Users/qlong/indeed/iwf/iwf-server/service/interpreter/cadence/workflowProvider.go:130 +0x7c fp=0x14000c4b4e0 sp=0x14000c4b4a0 pc=0x102a386ec
github.com/indeedeng/iwf/service/interpreter.executeState({0x1035656b8?, 0x140005f1700?}, {0x1035813e0, 0x104035540}, {{0x14000297918?, 0x140004e2700?}, 0x140005c5e90?, 0x140005c5ea0?}, {{0x1400032b020, 0x15}, ...}, ...)
/Users/qlong/indeed/iwf/iwf-server/service/interpreter/workflowImpl.go:210 +0x4b0 fp=0x14000c4bce0 sp=0x14000c4b4e0 pc=0x10269f770
github.com/indeedeng/iwf/service/interpreter.InterpreterImpl.func3({0x1035656b8, 0x140005f1700})
/Users/qlong/indeed/iwf/iwf-server/service/interpreter/workflowImpl.go:90 +0x24c fp=0x14000c4bee0 sp=0x14000c4bce0 pc=0x10269e59c
github.com/indeedeng/iwf/service/interpreter/cadence.workflowProvider.GoNamed.func1({0x10357a980?, 0x1400092e1e0})
/Users/qlong/indeed/iwf/iwf-server/service/interpreter/cadence/workflowProvider.go:80 +0x8c fp=0x14000c4bf20 sp=0x14000c4bee0 pc=0x102a3839c
go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0x1400077ae20?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:884 +0xc4 fp=0x14000c4bfb0 sp=0x14000c4bf20 pc=0x102a117b4
go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func2()
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:885 +0x34 fp=0x14000c4bfd0 sp=0x14000c4bfb0 pc=0x102a116b4
runtime.goexit()
/usr/local/go-dists/1.18.3/go/src/runtime/asm_arm64.s:1263 +0x4 fp=0x14000c4bfd0 sp=0x14000c4bfd0 pc=0x1021e6074
created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:875 +0x314
Yeah, i meet this problem recently ...
fatal error: concurrent map read and map write
/go/src/gitlab.momenta.works/hdmap-workflow/cadence-system-worker/internal/workflowimpl/workflow_interpreter.go:429 +0x25d
gitlab.momenta.works/hdmap-workflow/cadence-system-worker/internal/workflowimpl.(*ExecutionVisitor).visitActivityInvocation.func1()
go.uber.org/cadence/internal.(*workflowEnvironmentImpl).UpsertSearchAttributes(0xc0165d2120, 0xc00d6db8f0?)
/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/workflow.go:1347 +0x90
go.uber.org/cadence/internal.UpsertSearchAttributes({0x1190758, 0xc000bc61b0}, 0xfef115?)
/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_decision_state_machine.go:951 +0x89
go.uber.org/cadence/internal.(*decisionsHelper).addDecision(0xc007b885d0, {0x1199208, 0xc0144f9b20})
It seems like using go routine execute upsert attributes would cause this problem.
I've to use explicit lock to avoid this concurrent problem
more repro:
/usr/bin/make -f /Users/qlong/indeed/iwf/iwf-server/Makefile -C /Users/qlong/indeed/iwf/iwf-server integTests
go test -v ./integ
=== RUN TestAnyCommandCloseWorkflowTemporal
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] POST /api/v1/workflowState/start --> github.com/indeedeng/iwf/integ/workflow/common.WorkflowHandler.ApiV1WorkflowStateStart-fm (3 handlers)
[GIN-debug] POST /api/v1/workflowState/decide --> github.com/indeedeng/iwf/integ/workflow/common.WorkflowHandler.ApiV1WorkflowStateDecide-fm (3 handlers)
2022/12/29 09:22:27 INFO No logger configured for temporal client. Created default one.
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] GET / --> github.com/indeedeng/iwf/service/api.(*handler).index-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/start --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowStart-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/signal --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowSignal-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/dataobjects/get --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGetDataObjects-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/searchattributes/get --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGetSearchAttributes-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/get --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGet-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/getWithWait --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGetWithWait-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/search --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowSearch-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/reset --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowReset-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/stop --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowStop-fm (3 handlers)
2022/12/29 09:22:27 INFO Started Worker Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@
2022/12/29 09:22:27 received API request {any_command_close1672334547 any_command_close 10 http://localhost:9714 S1 <nil> <nil> <nil>}
2022/12/29 09:22:27 Started workflow WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8
[GIN] 2022/12/29 - 09:22:27 | 200 | 8.913833ms | ::1 | POST "/api/v1/workflow/start"
2022/12/29 09:22:27 received API request {any_command_close1672334547 <nil> test-signal-name2 0x140003aa410}
[GIN] 2022/12/29 - 09:22:27 | 200 | 7.184583ms | ::1 | POST "/api/v1/workflow/signal"
2022/12/29 09:22:27 received API request {any_command_close1672334547 <nil> <nil> <nil>}
2022/12/29 09:22:27 DEBUG ExecuteActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 Attempt 1 ActivityID 11 ActivityType StateStart
2022/12/29 09:22:27 INFO StateStartActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ ActivityID 11 ActivityType StateStart Attempt 1 WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 input {http://localhost:9714 {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S1-1} any_command_close S1 0x1400042c920 [] []}}
2022/12/29 09:22:27 received state start request, {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S1-1} any_command_close S1 0x140003ab1d0 [] []}
[GIN] 2022/12/29 - 09:22:27 | 200 | 155µs | ::1 | POST "/api/v1/workflowState/start"
2022/12/29 09:22:27 DEBUG ExecuteActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 Attempt 1 ActivityID 17 ActivityType StateDecide
2022/12/29 09:22:27 INFO StateDecideActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ ActivityID 17 ActivityType StateDecide Attempt 1 WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 input {http://localhost:9714 {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S1-1} any_command_close S1 0x140008d07a0 [] [] [] 0x1400041db80}}
2022/12/29 09:22:27 received state decide request, {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S1-1} any_command_close S1 0x1400042d3d0 [] [] [] 0x140004f53b0}
[GIN] 2022/12/29 - 09:22:27 | 200 | 90.708µs | ::1 | POST "/api/v1/workflowState/decide"
2022/12/29 09:22:27 DEBUG ExecuteActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 Attempt 1 ActivityID 25 ActivityType StateStart
2022/12/29 09:22:27 INFO StateStartActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ ActivityID 25 ActivityType StateStart Attempt 1 WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 input {http://localhost:9714 {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S2-1} any_command_close S2 <nil> [] []}}
2022/12/29 09:22:27 received state start request, {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S2-1} any_command_close S2 <nil> [] []}
[GIN] 2022/12/29 - 09:22:27 | 200 | 43.208µs | ::1 | POST "/api/v1/workflowState/start"
2022/12/29 09:22:27 DEBUG ExecuteActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 Attempt 1 ActivityID 31 ActivityType StateDecide
2022/12/29 09:22:27 INFO StateDecideActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ ActivityID 31 ActivityType StateDecide Attempt 1 WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 input {http://localhost:9714 {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S2-1} any_command_close S2 <nil> [] [] [] 0x1400041df40}}
2022/12/29 09:22:27 received state decide request, {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S2-1} any_command_close S2 <nil> [] [] [] 0x14000136f50}
[GIN] 2022/12/29 - 09:22:27 | 200 | 69.208µs | ::1 | POST "/api/v1/workflowState/decide"
[GIN] 2022/12/29 - 09:22:27 | 200 | 103.351375ms | ::1 | POST "/api/v1/workflow/getWithWait"
2022/12/29 09:22:27 INFO Stopped Worker Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@
--- PASS: TestAnyCommandCloseWorkflowTemporal (0.21s)
=== RUN TestAnyCommandCloseWorkflowCadence
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] POST /api/v1/workflowState/start --> github.com/indeedeng/iwf/integ/workflow/common.WorkflowHandler.ApiV1WorkflowStateStart-fm (3 handlers)
[GIN-debug] POST /api/v1/workflowState/decide --> github.com/indeedeng/iwf/integ/workflow/common.WorkflowHandler.ApiV1WorkflowStateDecide-fm (3 handlers)
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] GET / --> github.com/indeedeng/iwf/service/api.(*handler).index-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/start --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowStart-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/signal --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowSignal-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/dataobjects/get --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGetDataObjects-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/searchattributes/get --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGetSearchAttributes-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/get --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGet-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/getWithWait --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGetWithWait-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/search --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowSearch-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/reset --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowReset-fm (3 handlers)
[GIN-debug] POST /api/v1/workflow/stop --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowStop-fm (3 handlers)
{"level":"info","ts":"2022-12-29T09:22:27.363-0800","caller":"internal/internal_worker.go:223","msg":"No logger configured for cadence worker. Created default one."}
{"level":"info","ts":"2022-12-29T09:22:27.363-0800","caller":"internal/internal_worker.go:227","msg":"No metrics scope configured for cadence worker. Use NoopScope as default."}
{"level":"info","ts":"2022-12-29T09:22:27.434-0800","caller":"internal/internal_worker.go:834","msg":"Started Workflow Worker","Domain":"default","TaskList":"Interpreter_DEFAULT","WorkerID":"6745@IT-USA-VX8131@Interpreter_DEFAULT@2f8d29d1-e39a-433e-a7a2-f00411651f29"}
{"level":"info","ts":"2022-12-29T09:22:27.448-0800","caller":"internal/internal_worker.go:859","msg":"Started Activity Worker","Domain":"default","TaskList":"Interpreter_DEFAULT","WorkerID":"6745@IT-USA-VX8131@Interpreter_DEFAULT@2f8d29d1-e39a-433e-a7a2-f00411651f29"}
2022/12/29 09:22:27 received API request {any_command_close1672334547 any_command_close 10 http://localhost:9714 S1 <nil> <nil> <nil>}
2022/12/29 09:22:27 Started workflow WorkflowID any_command_close1672334547 RunID eaf1acf2-9109-4d32-9e28-db1a9c40737b
[GIN] 2022/12/29 - 09:22:27 | 200 | 21.851791ms | ::1 | POST "/api/v1/workflow/start"
2022/12/29 09:22:27 received API request {any_command_close1672334547 <nil> test-signal-name2 0x140007d4f90}
[GIN] 2022/12/29 - 09:22:27 | 200 | 20.215042ms | ::1 | POST "/api/v1/workflow/signal"
2022/12/29 09:22:27 received API request {any_command_close1672334547 <nil> <nil> <nil>}
fatal error: concurrent map writes
goroutine 268 [running]:
runtime.throw({0x1050c9ae5?, 0x14000b8a768?})
/usr/local/go-dists/1.18.3/go/src/runtime/panic.go:992 +0x50 fp=0x1400049ec90 sp=0x1400049ec60 pc=0x10424cbb0
runtime.mapassign_faststr(0x10542fc00?, 0x14000ce06b0?, {0x1050c7518, 0x14})
/usr/local/go-dists/1.18.3/go/src/runtime/map_faststr.go:212 +0x3bc fp=0x1400049ed00 sp=0x1400049ec90 pc=0x10422829c
go.uber.org/cadence/internal.mergeSearchAttributes(...)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_event_handlers.go:354
go.uber.org/cadence/internal.(*workflowEnvironmentImpl).updateWorkflowInfoWithSearchAttributes(...)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_event_handlers.go:339
go.uber.org/cadence/internal.(*workflowEnvironmentImpl).UpsertSearchAttributes(0x1400087d560, 0x1400049ee58?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_event_handlers.go:334 +0x258 fp=0x1400049edf0 sp=0x1400049ed00 pc=0x104d72998
go.uber.org/cadence/internal.(*workflowEnvironmentImpl).GetVersion(0x1400087d560, {0x1050b0fc9, 0x6}, 0x1055f75a0?, 0x1)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_event_handlers.go:623 +0x148 fp=0x1400049efa0 sp=0x1400049edf0 pc=0x104d74a38
go.uber.org/cadence/internal.(*workflowEnvironmentInterceptor).GetVersion(0x1400049f058?, {0x1053c9b00?, 0x10563eea8?}, {0x1050b0fc9?, 0x1400049f038?}, 0x104e86684?, 0x105490940?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/workflow.go:1614 +0x40 fp=0x1400049efe0 sp=0x1400049efa0 pc=0x104db58a0
go.uber.org/cadence/internal.GetVersion({0x105656c48, 0x14000e17ae0}, {0x1050b0fc9, 0x6}, 0x78?, 0x105554a80?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/workflow.go:1610 +0x90 fp=0x1400049f040 sp=0x1400049efe0 pc=0x104db5800
go.uber.org/cadence/workflow.GetVersion(...)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/workflow/workflow.go:344
github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GetVersion(0x1400049f0d8?, {0x1056416c8?, 0x14000e21a90?}, {0x1050b0fc9, 0x6}, 0x0?, 0x1400049f108?)
/Users/qlong/indeed/iwf/iwf-server/service/interpreter/cadence/workflowProvider.go:161 +0x70 fp=0x1400049f090 sp=0x1400049f040 pc=0x104e866a0
github.com/indeedeng/iwf/service/interpreter.(*globalVersioner).isAfterVersionOfUsingGlobalVersioning(...)
/Users/qlong/indeed/iwf/iwf-server/service/interpreter/globalVersionProvider.go:21
github.com/indeedeng/iwf/service/interpreter.InterpreterImpl({0x1056416c8?, 0x14000e21a90}, {0x10565dbe8, 0x106133940}, {{0x140003bceb8, 0x11}, {0x140003bced0, 0x15}, {0x14000ce0688, 0x2}, ...})
/Users/qlong/indeed/iwf/iwf-server/service/interpreter/workflowImpl.go:12 +0xc8 fp=0x1400049f3a0 sp=0x1400049f090 pc=0x104e80638
github.com/indeedeng/iwf/service/interpreter/cadence.Interpreter({0x105656c48?, 0x14000e17ae0}, {{0x140003bceb8, 0x11}, {0x140003bced0, 0x15}, {0x14000ce0688, 0x2}, {0x0, 0x0}, ...})
/Users/qlong/indeed/iwf/iwf-server/service/interpreter/cadence/workflow.go:10 +0xb0 fp=0x1400049f450 sp=0x1400049f3a0 pc=0x104e85790
runtime.call256(0x140001ebcb0, 0x10563b0b8, 0x14000cd9100, 0x78, 0x78, 0x88, 0x1400049fa80)
/usr/local/go-dists/1.18.3/go/src/runtime/asm_arm64.s:515 +0x84 fp=0x1400049f560 sp=0x1400049f450 pc=0x10427ea34
runtime.reflectcall(0x105554a80?, 0x14000cd9000?, 0x4?, 0x50c5041?, 0x1?, 0x12?, 0x105554a80?)
<autogenerated>:1 +0x34 fp=0x1400049f5a0 sp=0x1400049f560 pc=0x104282394
reflect.Value.call({0x1054210e0?, 0x10563b0b8?, 0x1400049fc88?}, {0x1050a7a2d, 0x4}, {0x14000bc9680, 0x2, 0x104d686dc?})
/usr/local/go-dists/1.18.3/go/src/reflect/value.go:556 +0x5e4 fp=0x1400049fc10 sp=0x1400049f5a0 pc=0x1042e15c4
reflect.Value.Call({0x1054210e0?, 0x10563b0b8?, 0x1400049fcd8?}, {0x14000bc9680, 0x2, 0x2})
/usr/local/go-dists/1.18.3/go/src/reflect/value.go:339 +0x98 fp=0x1400049fc90 sp=0x1400049fc10 pc=0x1042e0dd8
go.uber.org/cadence/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0x14000bc92c0?, {0x105656c48?, 0x14000e17ae0?}, {0x14000e21a50?, 0x14000d2e120?}, {0x14000e21a70?, 0x1?, 0x14000003c28?})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/workflow.go:681 +0x200 fp=0x1400049fdd0 sp=0x1400049fc90 pc=0x104db0330
go.uber.org/cadence/internal.(*workflowExecutor).Execute(0x14000ca89e0, {0x105656c48, 0x14000e17ae0}, {0x14000d2e120, 0x85, 0x90})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker.go:700 +0x3b8 fp=0x1400049feb0 sp=0x1400049fdd0 pc=0x104d96168
go.uber.org/cadence/internal.(*syncWorkflowDefinition).Execute.func1({0x105656e40, 0x14000bc9590})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:467 +0xe4 fp=0x1400049ff20 sp=0x1400049feb0 pc=0x104d9ce04
go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0x14000cc1060?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:884 +0xc4 fp=0x1400049ffb0 sp=0x1400049ff20 pc=0x104d9f214
go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func2()
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:885 +0x34 fp=0x1400049ffd0 sp=0x1400049ffb0 pc=0x104d9f114
runtime.goexit()
/usr/local/go-dists/1.18.3/go/src/runtime/asm_arm64.s:1263 +0x4 fp=0x1400049ffd0 sp=0x1400049ffd0 pc=0x104280884
created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:875 +0x314
goroutine 1 [chan receive]:
testing.(*T).Run(0x14000582820, {0x1050e1465?, 0x9b7f7eca3f5f?}, 0x10563b028)
/usr/local/go-dists/1.18.3/go/src/testing/testing.go:1487 +0x314
testing.runTests.func1(0x0?)
/usr/local/go-dists/1.18.3/go/src/testing/testing.go:1839 +0x74
testing.tRunner(0x14000582820, 0x140004fbcb8)
/usr/local/go-dists/1.18.3/go/src/testing/testing.go:1439 +0x110
testing.runTests(0x14000147f40?, {0x1060ed340, 0x10, 0x10}, {0xfa00000000000000?, 0x104222190?, 0x106102da0?})
/usr/local/go-dists/1.18.3/go/src/testing/testing.go:1837 +0x3f0
testing.(*M).Run(0x14000147f40)
/usr/local/go-dists/1.18.3/go/src/testing/testing.go:1719 +0x500
main.main()
_testmain.go:77 +0x1d0
goroutine 59 [select]:
net/http.(*persistConn).roundTrip(0x140007dd680, 0x1400095b340)
/usr/local/go-dists/1.18.3/go/src/net/http/transport.go:2620 +0x810
net/http.(*Transport).roundTrip(0x1060ece20, 0x14000257c00)
/usr/local/go-dists/1.18.3/go/src/net/http/transport.go:594 +0x6bc
net/http.(*Transport).RoundTrip(0x14000c82b00?, 0x105644c48?)
/usr/local/go-dists/1.18.3/go/src/net/http/roundtrip.go:17 +0x20
net/http.send(0x14000257c00, {0x105644c48, 0x1060ece20}, {0x1044e4d10?, 0x8?, 0x0?})
/usr/local/go-dists/1.18.3/go/src/net/http/client.go:252 +0x4fc
net/http.(*Client).send(0x1061023e0, 0x14000257c00, {0x140005b1428?, 0x104221d20?, 0x0?})
/usr/local/go-dists/1.18.3/go/src/net/http/client.go:176 +0x9c
net/http.(*Client).do(0x1061023e0, 0x14000257c00)
/usr/local/go-dists/1.18.3/go/src/net/http/client.go:725 +0x6ec
net/http.(*Client).Do(...)
/usr/local/go-dists/1.18.3/go/src/net/http/client.go:593
github.com/indeedeng/iwf/gen/iwfidl.(*APIClient).callAPI(0x1400016b140, 0x105655208?)
/Users/qlong/indeed/iwf/iwf-server/gen/iwfidl/client.go:171 +0x9c
github.com/indeedeng/iwf/gen/iwfidl.(*DefaultApiService).ApiV1WorkflowGetWithWaitPostExecute(0x1400016b148, {{0x105655208, 0x14000128008}, {0x10565eca8, 0x1400016b148}, 0x14000961170})
/Users/qlong/indeed/iwf/iwf-server/gen/iwfidl/api_default.go:459 +0x314
github.com/indeedeng/iwf/gen/iwfidl.ApiApiV1WorkflowGetWithWaitPostRequest.Execute(...)
/Users/qlong/indeed/iwf/iwf-server/gen/iwfidl/api_default.go:398
github.com/indeedeng/iwf/integ.doTestAnyCommandCloseWorkflow(0x1400040bba0, {0x1050b30c0, 0x7})
/Users/qlong/indeed/iwf/iwf-server/integ/any_command_close_test.go:79 +0x6d0
github.com/indeedeng/iwf/integ.TestAnyCommandCloseWorkflowCadence(0x1400040bba0)
/Users/qlong/indeed/iwf/iwf-server/integ/any_command_close_test.go:29 +0x68
testing.tRunner(0x1400040bba0, 0x10563b028)
/usr/local/go-dists/1.18.3/go/src/testing/testing.go:1439 +0x110
created by testing.(*T).Run
/usr/local/go-dists/1.18.3/go/src/testing/testing.go:1486 +0x300
goroutine 94 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0x14000b84900)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:324 +0x70
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:339
google.golang.org/grpc.(*csAttempt).recvMsg(0x1400050cd80, {0x10537e980?, 0x14000b88258}, 0x104915610?)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:1011 +0x7c
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x140005f9768?)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:865 +0x2c
google.golang.org/grpc.(*clientStream).withRetry(0x14000b846c0, 0x140005f9798, 0x140005f9768)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:724 +0x1c8
google.golang.org/grpc.(*clientStream).RecvMsg(0x14000b846c0, {0x10537e980?, 0x14000b88258?})
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:864 +0xbc
google.golang.org/grpc.invoke({0x105655278?, 0x140001eb230?}, {0x14000044f40?, 0x140007d54b0?}, {0x1053a8bc0, 0x14000b882a0}, {0x10537e980, 0x14000b88258}, 0x0?, {0x140007e51c0?, ...})
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:73 +0xb8
google.golang.org/grpc.(*ClientConn).Invoke(0x14000866000?, {0x105655278?, 0x140001eb230?}, {0x14000044f40?, 0x3c?}, {0x1053a8bc0?, 0x14000b882a0?}, {0x10537e980?, 0x14000b88258?}, {0x140007d54b0?, ...})
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:37 +0x1a8
go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0x14000850ea0, {0x105655240, 0x140007d9560}, 0x1400050ccc0, 0x1050b5b05?, 0x14000124c00, {0x1?, 0x10426c2c4?, 0x106102da0?})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x368
go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0x14000850ea0, {0x105655240, 0x140007d9560}, 0x1400050ccc0)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0xc0
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240?, 0x140007d9560?}, 0x1050a82fd?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
go.uber.org/yarpc/internal/observability.(*Middleware).Call(0x14000844a80, {0x105655240, 0x140007d9560}, 0x1400050ccc0, {0x105658fb0, 0x140001eb110})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0xf8
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x140007d9560}, 0x104759b78?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x105580d00?, {0x105655240, 0x140007d9560}, 0x1400050ccc0, {0x105658fb0?, 0x140001eb0e0})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xbc
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x1, 0x1}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x140007d9560}, 0x140001eb0b0?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x105658eb0?, 0x14000851020?}, {0x12d78cb18?, 0x14000851020?}}, {0x105655240, 0x140007d9560}, 0x140003a8600?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x90
go.uber.org/yarpc/encoding/protobuf.(*client).Call(0x140008514a0, {0x105655240?, 0x140007d9560?}, {0x1050d348a?, 0x1010140005faee8?}, {0x10564d008?, 0x140002caa10?}, 0x10563b460, {0x140007d95c0?, 0x7?, ...})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0xb8
github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkflowAPIYARPCCaller).GetWorkflowExecutionHistory(0x140007e6c80?, {0x105655240?, 0x140007d9560?}, 0x10426c94c?, {0x140007d95c0?, 0x104222190?, 0x140001eb020?})
/Users/qlong/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220505064515-7bb6b0808383/go/proto/api/v1/service_workflow.pb.yarpc.go:505 +0x64
go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.GetWorkflowExecutionHistory({{0x1056581f0, 0x1400086a260}, {0x10565d0c0, 0x1400086a280}, {0x10565d038, 0x1400086a2a0}, {0x10565ae50, 0x1400086a2c0}}, {0x105655240, 0x140007d9560}, ...)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/compatibility/adapter.go:86 +0x88
go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).GetWorkflowExecutionHistory(0x140008516e0, {0x105655240, 0x140007d9560}, 0x140005fb0e8?, {0x140007d95c0, 0x4, 0x6})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/common/metrics/service_wrapper.go:163 +0x80
go.uber.org/cadence/internal.(*workflowClient).GetWorkflowHistory.func1.1()
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow_client.go:524 +0xcc
go.uber.org/cadence/internal/common/backoff.Retry({0x1056551d0, 0x140005d0380}, 0x140005fb2c8, {0x105643488, 0x140001948d0}, 0x10563bfc0)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/common/backoff/retry.go:98 +0x8c
go.uber.org/cadence/internal.(*workflowClient).GetWorkflowHistory.func1({0x0, 0x0, 0x0})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow_client.go:515 +0x334
go.uber.org/cadence/internal.(*historyEventIteratorImpl).HasNext(0x140007d9500)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow_client.go:1071 +0x8c
go.uber.org/cadence/internal.(*workflowRunImpl).Get(0x140007d9440, {0x1056551d0, 0x140005d0380}, {0x105382dc0?, 0x14000b881f8})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow_client.go:1124 +0x74
github.com/indeedeng/iwf/service/api/cadence.(*cadenceClient).GetWorkflowResult(0x0?, {0x1056551d0, 0x140005d0380}, {0x105382dc0, 0x14000b881f8}, {0x140003420a0?, 0x0?}, {0x0?, 0x0?})
/Users/qlong/indeed/iwf/iwf-server/service/api/cadence/client.go:177 +0x70
github.com/indeedeng/iwf/service/api.(*serviceImpl).doApiV1WorkflowGetPost(0x140004711c0, {0x1056551d0, 0x140005d0380}, {{0x140003420a0, 0x1b}, 0x0, 0x0, 0x0}, 0x1)
/Users/qlong/indeed/iwf/iwf-server/service/api/service.go:138 +0x160
github.com/indeedeng/iwf/service/api.(*serviceImpl).ApiV1WorkflowGetWithWaitPost(0x140005fb600?, {0x1056551d0?, 0x140005d0380?}, {{0x140003420a0, 0x1b}, 0x0, 0x0, 0x0})
/Users/qlong/indeed/iwf/iwf-server/service/api/service.go:126 +0x44
github.com/indeedeng/iwf/service/api.(*handler).doApiV1WorkflowGetPost(0x1400086a340, 0x140005dc200, 0x1)
/Users/qlong/indeed/iwf/iwf-server/service/api/handler.go:158 +0x1f8
github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGetWithWait(0x140005fb6a8?, 0x1042294d0?)
/Users/qlong/indeed/iwf/iwf-server/service/api/handler.go:144 +0x24
github.com/gin-gonic/gin.(*Context).Next(...)
/Users/qlong/go/pkg/mod/github.com/gin-gonic/gin@v1.8.1/context.go:173
github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0x140005dc200)
/Users/qlong/go/pkg/mod/github.com/gin-gonic/gin@v1.8.1/recovery.go:101 +0x80
github.com/gin-gonic/gin.(*Context).Next(...)
/Users/qlong/go/pkg/mod/github.com/gin-gonic/gin@v1.8.1/context.go:173
github.com/gin-gonic/gin.LoggerWithConfig.func1(0x140005dc200)
/Users/qlong/go/pkg/mod/github.com/gin-gonic/gin@v1.8.1/logger.go:240 +0xb0
github.com/gin-gonic/gin.(*Context).Next(...)
/Users/qlong/go/pkg/mod/github.com/gin-gonic/gin@v1.8.1/context.go:173
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0x1400086e340, 0x140005dc200)
/Users/qlong/go/pkg/mod/github.com/gin-gonic/gin@v1.8.1/gin.go:616 +0x568
github.com/gin-gonic/gin.(*Engine).ServeHTTP(0x1400086e340, {0x105654118?, 0x140005d80e0}, 0x140005dc100)
/Users/qlong/go/pkg/mod/github.com/gin-gonic/gin@v1.8.1/gin.go:572 +0x1d8
net/http.serverHandler.ServeHTTP({0x105649508?}, {0x105654118, 0x140005d80e0}, 0x140005dc100)
/usr/local/go-dists/1.18.3/go/src/net/http/server.go:2916 +0x3fc
net/http.(*conn).serve(0x140003fe820, {0x105655278, 0x14000a58c60})
/usr/local/go-dists/1.18.3/go/src/net/http/server.go:1966 +0x56c
created by net/http.(*Server).Serve
/usr/local/go-dists/1.18.3/go/src/net/http/server.go:3071 +0x450
goroutine 147 [IO wait]:
internal/poll.runtime_pollWait(0x12d53d800, 0x72)
/usr/local/go-dists/1.18.3/go/src/runtime/netpoll.go:302 +0xa4
internal/poll.(*pollDesc).wait(0x1400095e280?, 0x14000960851?, 0x0)
/usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:83 +0x2c
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0x1400095e280, {0x14000960851, 0x1, 0x1})
/usr/local/go-dists/1.18.3/go/src/internal/poll/fd_unix.go:167 +0x1e4
net.(*netFD).Read(0x1400095e280, {0x14000960851?, 0x14000860fa8?, 0x104d99d2c?})
/usr/local/go-dists/1.18.3/go/src/net/fd_posix.go:55 +0x2c
net.(*conn).Read(0x1400018aa38, {0x14000960851?, 0x100000001?, 0x104d99ef0?})
/usr/local/go-dists/1.18.3/go/src/net/net.go:183 +0x38
net/http.(*connReader).backgroundRead(0x14000960840)
/usr/local/go-dists/1.18.3/go/src/net/http/server.go:672 +0x48
created by net/http.(*connReader).startBackgroundRead
/usr/local/go-dists/1.18.3/go/src/net/http/server.go:668 +0xd0
goroutine 299 [select]:
go.uber.org/cadence/internal.(*locallyDispatchedActivityTunnel).getTask(0x140004b1728)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:193 +0x60
go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).pollLocallyDispatchedActivity(0x14000874080, {0x1043160f8?, 0x140008440d8?})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1122 +0x2c
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetrics(0x14000874080, {0x105655240?, 0x14000a62f60?}, 0x14000878418?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1019 +0x3c
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetricsFunc.func1({0x105655240?, 0x14000a62f60?})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1013 +0x3c
go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:256 +0x50
created by go.uber.org/cadence/internal.(*basePoller).doPoll
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:255 +0x18c
goroutine 104 [select]:
go.uber.org/cadence/internal.(*basePoller).doPoll(0x14000874080, {0xd0?}, 0x14000799278)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:261 +0x1c4
go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).PollTask(0x14000874080)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1113 +0x8c
go.uber.org/cadence/internal.(*baseWorker).pollTask(0x14000852540)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:263 +0x90
go.uber.org/cadence/internal.(*baseWorker).runPoller(0x14000852540)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:227 +0x8c
created by go.uber.org/cadence/internal.(*baseWorker).Start
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:190 +0x70
goroutine 337 [select]:
go.uber.org/cadence/internal.(*locallyDispatchedActivityTunnel).getTask(0x140004b1728)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:193 +0x60
go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).pollLocallyDispatchedActivity(0x14000874080, {0x1043160f8?, 0x14000844018?})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1122 +0x2c
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetrics(0x14000874080, {0x105655240?, 0x1400060e360?}, 0x14000878238?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1019 +0x3c
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetricsFunc.func1({0x105655240?, 0x1400060e360?})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1013 +0x3c
go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:256 +0x50
created by go.uber.org/cadence/internal.(*basePoller).doPoll
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:255 +0x18c
goroutine 119 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0x14000a49c20)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:324 +0x70
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:339
google.golang.org/grpc.(*csAttempt).recvMsg(0x140009772c0, {0x10537e980?, 0x140007999e0}, 0x104915610?)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:1011 +0x7c
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x140004a8488?)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:865 +0x2c
google.golang.org/grpc.(*clientStream).withRetry(0x14000a499e0, 0x140004a84b8, 0x140004a8488)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:724 +0x1c8
google.golang.org/grpc.(*clientStream).RecvMsg(0x14000a499e0, {0x10537e980?, 0x140007999e0?})
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:864 +0xbc
google.golang.org/grpc.invoke({0x105655278?, 0x14000a58ff0?}, {0x1400047cc40?, 0x140008d1a40?}, {0x1053a8bc0, 0x14000799a28}, {0x10537e980, 0x140007999e0}, 0x0?, {0x14000a5a540?, ...})
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:73 +0xb8
google.golang.org/grpc.(*ClientConn).Invoke(0x14000866000?, {0x105655278?, 0x14000a58ff0?}, {0x1400047cc40?, 0x32?}, {0x1053a8bc0?, 0x14000799a28?}, {0x10537e980?, 0x140007999e0?}, {0x140008d1a40?, ...})
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:37 +0x1a8
go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0x14000850ea0, {0x105655240, 0x14000899a40}, 0x14000977200, 0x1050b5b05?, 0x140001cc968, {0x1?, 0x18?, 0x106102da0?})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x368
go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0x14000850ea0, {0x105655240, 0x14000899a40}, 0x14000977200)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0xc0
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240?, 0x14000899a40?}, 0x1050a82fd?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
go.uber.org/yarpc/internal/observability.(*Middleware).Call(0x14000844a80, {0x105655240, 0x14000899a40}, 0x14000977200, {0x105658fb0, 0x14000a58ed0})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0xf8
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x14000899a40}, 0x104759b78?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x105580d00?, {0x105655240, 0x14000899a40}, 0x14000977200, {0x105658fb0?, 0x14000a58ea0})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xbc
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x1, 0x1}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x14000899a40}, 0x14000a58e70?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x105658eb0?, 0x14000851020?}, {0x12d78cb18?, 0x14000851020?}}, {0x105655240, 0x14000899a40}, 0x12d552708?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x90
go.uber.org/yarpc/encoding/protobuf.(*client).Call(0x14000851500, {0x105655240?, 0x14000899a40?}, {0x1050c5f4c?, 0x14000903c08?}, {0x10564dde8?, 0x14000a54d20?}, 0x10563b3d0, {0x14000899b00?, 0x14000a58d80?, ...})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0xb8
github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkerAPIYARPCCaller).PollForActivityTask(0x1400009b900?, {0x105655240?, 0x14000899a40?}, 0x10426c94c?, {0x14000899b00?, 0x104222190?, 0x14000a58d50?})
/Users/qlong/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220505064515-7bb6b0808383/go/proto/api/v1/service_worker.pb.yarpc.go:397 +0x64
go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.PollForActivityTask({{0x1056581f0, 0x1400086a260}, {0x10565d0c0, 0x1400086a280}, {0x10565d038, 0x1400086a2a0}, {0x10565ae50, 0x1400086a2c0}}, {0x105655240, 0x14000899a40}, ...)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/compatibility/adapter.go:121 +0x88
go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).PollForActivityTask(0x14000851b60, {0x105655240, 0x14000899a40}, 0x4?, {0x14000899b00, 0x4, 0x6})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/common/metrics/service_wrapper.go:212 +0x80
go.uber.org/cadence/internal.(*activityTaskPoller).poll(0x1400080de80, {0x105655240, 0x14000899a40})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:991 +0x348
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetrics(0x1400080de80, {0x105655240?, 0x14000899a40?}, 0x14000055e00?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1019 +0x3c
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetricsFunc.func1({0x105655240?, 0x14000899a40?})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1013 +0x3c
go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:256 +0x50
created by go.uber.org/cadence/internal.(*basePoller).doPoll
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:255 +0x18c
goroutine 102 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x140007e6370, 0x1)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/controlbuf.go:408 +0x15c
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x140007d8840)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/controlbuf.go:535 +0x78
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/http2_client.go:419 +0x68
created by google.golang.org/grpc/internal/transport.newHTTP2Client
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/http2_client.go:417 +0x16e0
goroutine 105 [select]:
go.uber.org/cadence/internal.(*basePoller).doPoll(0x14000874080, {0xe0?}, 0x14000bfdde8)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:261 +0x1c4
go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).PollTask(0x14000874080)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1113 +0x8c
go.uber.org/cadence/internal.(*baseWorker).pollTask(0x14000852540)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:263 +0x90
go.uber.org/cadence/internal.(*baseWorker).runPoller(0x14000852540)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:227 +0x8c
created by go.uber.org/cadence/internal.(*baseWorker).Start
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:190 +0x70
goroutine 96 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0x140009639e0)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:324 +0x70
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:339
google.golang.org/grpc.(*csAttempt).recvMsg(0x1400079f500, {0x10537e980?, 0x140005a9680}, 0x104915610?)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:1011 +0x7c
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x140009164e8?)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:865 +0x2c
google.golang.org/grpc.(*clientStream).withRetry(0x140009637a0, 0x14000916518, 0x140009164e8)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:724 +0x1c8
google.golang.org/grpc.(*clientStream).RecvMsg(0x140009637a0, {0x10537e980?, 0x140005a9680?})
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:864 +0xbc
google.golang.org/grpc.invoke({0x105655278?, 0x14000961b90?}, {0x14000c00440?, 0x14000c80710?}, {0x1053a8bc0, 0x140005a96c8}, {0x10537e980, 0x140005a9680}, 0x0?, {0x1400095b780?, ...})
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:73 +0xb8
google.golang.org/grpc.(*ClientConn).Invoke(0x14000866000?, {0x105655278?, 0x14000961b90?}, {0x14000c00440?, 0x32?}, {0x1053a8bc0?, 0x140005a96c8?}, {0x10537e980?, 0x140005a9680?}, {0x14000c80710?, ...})
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:37 +0x1a8
go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0x14000850ea0, {0x105655240, 0x1400095ccc0}, 0x1400079f440, 0x0?, 0x1400018aaf0, {0x14000864348?, 0x18?, 0x106102da0?})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x368
go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0x14000850ea0, {0x105655240, 0x1400095ccc0}, 0x1400079f440)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0xc0
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240?, 0x1400095ccc0?}, 0x1050a82fd?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
go.uber.org/yarpc/internal/observability.(*Middleware).Call(0x14000844a80, {0x105655240, 0x1400095ccc0}, 0x1400079f440, {0x105658fb0, 0x14000961aa0})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0xf8
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x1400095ccc0}, 0x104759b78?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x105580d00?, {0x105655240, 0x1400095ccc0}, 0x1400079f440, {0x105658fb0?, 0x14000961a70})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xbc
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x1, 0x1}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x1400095ccc0}, 0x14000961a40?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x105658eb0?, 0x14000851020?}, {0x12d78cb18?, 0x14000851020?}}, {0x105655240, 0x1400095ccc0}, 0x0?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x90
go.uber.org/yarpc/encoding/protobuf.(*client).Call(0x14000851500, {0x105655240?, 0x1400095ccc0?}, {0x1050c5f5f?, 0x14000682d58?}, {0x10564de48?, 0x1400095cde0?}, 0x10563b3d8, {0x1400095cd80?, 0x13?, ...})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0xb8
github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkerAPIYARPCCaller).PollForDecisionTask(0x1400018fce0?, {0x105655240?, 0x1400095ccc0?}, 0x10426c94c?, {0x1400095cd80?, 0x104222190?, 0x140009619b0?})
/Users/qlong/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220505064515-7bb6b0808383/go/proto/api/v1/service_worker.pb.yarpc.go:361 +0x64
go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.PollForDecisionTask({{0x1056581f0, 0x1400086a260}, {0x10565d0c0, 0x1400086a280}, {0x10565d038, 0x1400086a2a0}, {0x10565ae50, 0x1400086a2c0}}, {0x105655240, 0x1400095ccc0}, ...)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/compatibility/adapter.go:126 +0x88
go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).PollForDecisionTask(0x14000851b60, {0x105655240, 0x1400095ccc0}, 0x0?, {0x1400095cd80, 0x4, 0x6})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/common/metrics/service_wrapper.go:219 +0x80
go.uber.org/cadence/internal.(*workflowTaskPoller).poll(0x140008209c0, {0x105655240, 0x1400095ccc0})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:767 +0x20c
go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:256 +0x50
created by go.uber.org/cadence/internal.(*basePoller).doPoll
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:255 +0x18c
goroutine 101 [IO wait]:
internal/poll.runtime_pollWait(0x12d53d710, 0x72)
/usr/local/go-dists/1.18.3/go/src/runtime/netpoll.go:302 +0xa4
internal/poll.(*pollDesc).wait(0x14000a56580?, 0x140007e8000?, 0x0)
/usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:83 +0x2c
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0x14000a56580, {0x140007e8000, 0x8000, 0x8000})
/usr/local/go-dists/1.18.3/go/src/internal/poll/fd_unix.go:167 +0x1e4
net.(*netFD).Read(0x14000a56580, {0x140007e8000?, 0x1400008ebf8?, 0x1046b74c8?})
/usr/local/go-dists/1.18.3/go/src/net/fd_posix.go:55 +0x2c
net.(*conn).Read(0x14000124780, {0x140007e8000?, 0x1400008ec58?, 0x10426c2c4?})
/usr/local/go-dists/1.18.3/go/src/net/net.go:183 +0x38
bufio.(*Reader).Read(0x140007d87e0, {0x140007ba200, 0x9, 0x14000bc8900?})
/usr/local/go-dists/1.18.3/go/src/bufio/bufio.go:236 +0x1e4
io.ReadAtLeast({0x105641168, 0x140007d87e0}, {0x140007ba200, 0x9, 0x9}, 0x9)
/usr/local/go-dists/1.18.3/go/src/io/io.go:331 +0xa4
io.ReadFull(...)
/usr/local/go-dists/1.18.3/go/src/io/io.go:350
golang.org/x/net/http2.readFrameHeader({0x140007ba200?, 0x9?, 0x1400008ed58?}, {0x105641168?, 0x140007d87e0?})
/Users/qlong/go/pkg/mod/golang.org/x/net@v0.1.0/http2/frame.go:237 +0x5c
golang.org/x/net/http2.(*Framer).ReadFrame(0x140007ba1c0)
/Users/qlong/go/pkg/mod/golang.org/x/net@v0.1.0/http2/frame.go:498 +0x84
google.golang.org/grpc/internal/transport.(*http2Client).reader(0x14000947860)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/http2_client.go:1516 +0x2e8
created by google.golang.org/grpc/internal/transport.newHTTP2Client
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/http2_client.go:369 +0x1204
goroutine 163 [chan receive]:
go.uber.org/cadence/internal.(*coroutineState).call(...)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:839
go.uber.org/cadence/internal.(*dispatcherImpl).ExecuteUntilAllBlocked(0x140007e4f00)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:923 +0x1fc
go.uber.org/cadence/internal.executeDispatcher({0x105656c48, 0x14000e17ae0}, {0x105656cf0, 0x140007e4f00})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:540 +0x7c
go.uber.org/cadence/internal.(*syncWorkflowDefinition).OnDecisionTaskStarted(0x1400087d560?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:514 +0x34
go.uber.org/cadence/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0x14000e0ae70, 0x14000d32180, 0xd8?, 0x1)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_event_handlers.go:820 +0x7a0
go.uber.org/cadence/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0x1400050ce40, 0x14000bc8ea0)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_handlers.go:895 +0x15ec
go.uber.org/cadence/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0x14000816b00, 0x14000bc8ea0, 0x14000bc8ed0)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_handlers.go:787 +0x320
go.uber.org/cadence/internal.(*workflowTaskPoller).processWorkflowTask(0x140008209c0, 0x140009618c0)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:341 +0x260
go.uber.org/cadence/internal.(*workflowTaskPoller).ProcessTask(0x140008209c0, {0x10538a0c0?, 0x140009618c0?})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:314 +0x74
go.uber.org/cadence/internal.(*baseWorker).processTask(0x14000852620, {0x105389b80?, 0x14000c805a0?})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:324 +0xe0
created by go.uber.org/cadence/internal.(*baseWorker).runTaskDispatcher
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:253 +0x98
goroutine 112 [IO wait]:
internal/poll.runtime_pollWait(0x12d53d260, 0x72)
/usr/local/go-dists/1.18.3/go/src/runtime/netpoll.go:302 +0xa4
internal/poll.(*pollDesc).wait(0x140001ec080?, 0x140007ab000?, 0x0)
/usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:83 +0x2c
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0x140001ec080, {0x140007ab000, 0x1000, 0x1000})
/usr/local/go-dists/1.18.3/go/src/internal/poll/fd_unix.go:167 +0x1e4
net.(*netFD).Read(0x140001ec080, {0x140007ab000?, 0x14000596c58?, 0x10421a668?})
/usr/local/go-dists/1.18.3/go/src/net/fd_posix.go:55 +0x2c
net.(*conn).Read(0x14000124bb8, {0x140007ab000?, 0x0?, 0x0?})
/usr/local/go-dists/1.18.3/go/src/net/net.go:183 +0x38
net/http.(*persistConn).Read(0x140007dd680, {0x140007ab000?, 0x10421a710?, 0x14000956960?})
/usr/local/go-dists/1.18.3/go/src/net/http/transport.go:1929 +0x54
bufio.(*Reader).fill(0x140007d8fc0)
/usr/local/go-dists/1.18.3/go/src/bufio/bufio.go:106 +0x100
bufio.(*Reader).Peek(0x140007d8fc0, 0x1)
/usr/local/go-dists/1.18.3/go/src/bufio/bufio.go:144 +0x64
net/http.(*persistConn).readLoop(0x140007dd680)
/usr/local/go-dists/1.18.3/go/src/net/http/transport.go:2093 +0x148
created by net/http.(*Transport).dialConn
/usr/local/go-dists/1.18.3/go/src/net/http/transport.go:1750 +0x11f4
goroutine 113 [select]:
net/http.(*persistConn).writeLoop(0x140007dd680)
/usr/local/go-dists/1.18.3/go/src/net/http/transport.go:2392 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go-dists/1.18.3/go/src/net/http/transport.go:1751 +0x1240
goroutine 92 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0x14000962120)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:324 +0x70
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:339
google.golang.org/grpc.(*csAttempt).recvMsg(0x1400079f080, {0x10537e980?, 0x140005a9020}, 0x104915610?)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:1011 +0x7c
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x1400076e4e8?)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:865 +0x2c
google.golang.org/grpc.(*clientStream).withRetry(0x1400090be60, 0x1400076e518, 0x1400076e4e8)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:724 +0x1c8
google.golang.org/grpc.(*clientStream).RecvMsg(0x1400090be60, {0x10537e980?, 0x140005a9020?})
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:864 +0xbc
google.golang.org/grpc.invoke({0x105655278?, 0x14000960210?}, {0x14000122740?, 0x1400042db50?}, {0x1053a8bc0, 0x140005a9068}, {0x10537e980, 0x140005a9020}, 0x0?, {0x1400095ab40?, ...})
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:73 +0xb8
google.golang.org/grpc.(*ClientConn).Invoke(0x14000866000?, {0x105655278?, 0x14000960210?}, {0x14000122740?, 0x32?}, {0x1053a8bc0?, 0x140005a9068?}, {0x10537e980?, 0x140005a9020?}, {0x1400042db50?, ...})
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:37 +0x1a8
go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0x14000850ea0, {0x105655240, 0x1400095c3c0}, 0x1400079efc0, 0x1050b5b05?, 0x1400018aa08, {0x1?, 0x18?, 0x106102da0?})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x368
go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0x14000850ea0, {0x105655240, 0x1400095c3c0}, 0x1400079efc0)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0xc0
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240?, 0x1400095c3c0?}, 0x1050a82fd?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
go.uber.org/yarpc/internal/observability.(*Middleware).Call(0x14000844a80, {0x105655240, 0x1400095c3c0}, 0x1400079efc0, {0x105658fb0, 0x140009600f0})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0xf8
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x1400095c3c0}, 0x104759b78?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x105580d00?, {0x105655240, 0x1400095c3c0}, 0x1400079efc0, {0x105658fb0?, 0x140009600c0})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xbc
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x1, 0x1}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x1400095c3c0}, 0x14000960090?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x105658eb0?, 0x14000851020?}, {0x12d78cb18?, 0x14000851020?}}, {0x105655240, 0x1400095c3c0}, 0x1?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x90
go.uber.org/yarpc/encoding/protobuf.(*client).Call(0x14000851500, {0x105655240?, 0x1400095c3c0?}, {0x1050c5f5f?, 0x1400008bc68?}, {0x10564de48?, 0x1400095c4e0?}, 0x10563b3d8, {0x1400095c480?, 0x32?, ...})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0xb8
github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkerAPIYARPCCaller).PollForDecisionTask(0x1400018f860?, {0x105655240?, 0x1400095c3c0?}, 0x10426c94c?, {0x1400095c480?, 0x104222190?, 0x14000919fb0?})
/Users/qlong/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220505064515-7bb6b0808383/go/proto/api/v1/service_worker.pb.yarpc.go:361 +0x64
go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.PollForDecisionTask({{0x1056581f0, 0x1400086a260}, {0x10565d0c0, 0x1400086a280}, {0x10565d038, 0x1400086a2a0}, {0x10565ae50, 0x1400086a2c0}}, {0x105655240, 0x1400095c3c0}, ...)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/compatibility/adapter.go:126 +0x88
go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).PollForDecisionTask(0x14000851b60, {0x105655240, 0x1400095c3c0}, 0x1042b5fbc?, {0x1400095c480, 0x4, 0x6})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/common/metrics/service_wrapper.go:219 +0x80
go.uber.org/cadence/internal.(*workflowTaskPoller).poll(0x140008209c0, {0x105655240, 0x1400095c3c0})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:767 +0x20c
go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:256 +0x50
created by go.uber.org/cadence/internal.(*basePoller).doPoll
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:255 +0x18c
goroutine 106 [select]:
go.uber.org/cadence/internal.(*baseWorker).runTaskDispatcher(0x14000852540)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:241 +0xdc
created by go.uber.org/cadence/internal.(*baseWorker).Start
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:194 +0x130
goroutine 60 [IO wait]:
internal/poll.runtime_pollWait(0x12d53d350, 0x72)
/usr/local/go-dists/1.18.3/go/src/runtime/netpoll.go:302 +0xa4
internal/poll.(*pollDesc).wait(0x140000b3d80?, 0x105512080?, 0x0)
/usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:83 +0x2c
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0x140000b3d80)
/usr/local/go-dists/1.18.3/go/src/internal/poll/fd_unix.go:614 +0x1d0
net.(*netFD).accept(0x140000b3d80)
/usr/local/go-dists/1.18.3/go/src/net/fd_unix.go:172 +0x28
net.(*TCPListener).accept(0x140001c9ae8)
/usr/local/go-dists/1.18.3/go/src/net/tcpsock_posix.go:139 +0x2c
net.(*TCPListener).Accept(0x140001c9ae8)
/usr/local/go-dists/1.18.3/go/src/net/tcpsock.go:288 +0x30
net/http.(*Server).Serve(0x140008521c0, {0x105653f08, 0x140001c9ae8})
/usr/local/go-dists/1.18.3/go/src/net/http/server.go:3039 +0x318
net/http.(*Server).ListenAndServe(0x140008521c0)
/usr/local/go-dists/1.18.3/go/src/net/http/server.go:2968 +0x88
github.com/indeedeng/iwf/integ.startWorkflowWorker.func1()
/Users/qlong/indeed/iwf/iwf-server/integ/util.go:42 +0x24
created by github.com/indeedeng/iwf/integ.startWorkflowWorker
/Users/qlong/indeed/iwf/iwf-server/integ/util.go:41 +0x20c
goroutine 132 [IO wait]:
internal/poll.runtime_pollWait(0x12d53d530, 0x72)
/usr/local/go-dists/1.18.3/go/src/runtime/netpoll.go:302 +0xa4
internal/poll.(*pollDesc).wait(0x14000a56600?, 0x4?, 0x0)
/usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:83 +0x2c
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0x14000a56600)
/usr/local/go-dists/1.18.3/go/src/internal/poll/fd_unix.go:614 +0x1d0
net.(*netFD).accept(0x14000a56600)
/usr/local/go-dists/1.18.3/go/src/net/fd_unix.go:172 +0x28
net.(*TCPListener).accept(0x14000799950)
/usr/local/go-dists/1.18.3/go/src/net/tcpsock_posix.go:139 +0x2c
net.(*TCPListener).Accept(0x14000799950)
/usr/local/go-dists/1.18.3/go/src/net/tcpsock.go:288 +0x30
net/http.(*Server).Serve(0x14000852380, {0x105653f08, 0x14000799950})
/usr/local/go-dists/1.18.3/go/src/net/http/server.go:3039 +0x318
net/http.(*Server).ListenAndServe(0x14000852380)
/usr/local/go-dists/1.18.3/go/src/net/http/server.go:2968 +0x88
github.com/indeedeng/iwf/integ.startIwfService.func3()
/Users/qlong/indeed/iwf/iwf-server/integ/util.go:84 +0x24
created by github.com/indeedeng/iwf/integ.startIwfService
/Users/qlong/indeed/iwf/iwf-server/integ/util.go:83 +0x4ec
goroutine 65 [select]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0x140003cd600)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/balancer_conn_wrappers.go:112 +0x68
created by google.golang.org/grpc.newCCBalancerWrapper
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/balancer_conn_wrappers.go:73 +0x230
goroutine 103 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0x140007dd440)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:324 +0x70
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:339
google.golang.org/grpc.(*csAttempt).recvMsg(0x1400050c9c0, {0x10537e980?, 0x140001c9bc0}, 0x104915610?)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:1011 +0x7c
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x14000a52488?)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:865 +0x2c
google.golang.org/grpc.(*clientStream).withRetry(0x140007dd200, 0x14000a524b8, 0x14000a52488)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:724 +0x1c8
google.golang.org/grpc.(*clientStream).RecvMsg(0x140007dd200, {0x10537e980?, 0x140001c9bc0?})
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:864 +0xbc
google.golang.org/grpc.invoke({0x105655278?, 0x140007df6b0?}, {0x14000044b40?, 0x140007d4b00?}, {0x1053a8bc0, 0x140001c9c08}, {0x10537e980, 0x140001c9bc0}, 0x0?, {0x140007e45c0?, ...})
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:73 +0xb8
google.golang.org/grpc.(*ClientConn).Invoke(0x14000866000?, {0x105655278?, 0x140007df6b0?}, {0x14000044b40?, 0x32?}, {0x1053a8bc0?, 0x140001c9c08?}, {0x10537e980?, 0x140001c9bc0?}, {0x140007d4b00?, ...})
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:37 +0x1a8
go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0x14000850ea0, {0x105655240, 0x140007d8960}, 0x1400050c900, 0x0?, 0x14000124988, {0x14000865868?, 0x18?, 0x106102da0?})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x368
go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0x14000850ea0, {0x105655240, 0x140007d8960}, 0x1400050c900)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0xc0
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240?, 0x140007d8960?}, 0x1050a82fd?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
go.uber.org/yarpc/internal/observability.(*Middleware).Call(0x14000844a80, {0x105655240, 0x140007d8960}, 0x1400050c900, {0x105658fb0, 0x140007df5c0})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0xf8
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x140007d8960}, 0x104759b78?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x105580d00?, {0x105655240, 0x140007d8960}, 0x1400050c900, {0x105658fb0?, 0x140007df590})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xbc
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x1, 0x1}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x140007d8960}, 0x140007df560?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x105658eb0?, 0x14000851020?}, {0x12d78cb18?, 0x14000851020?}}, {0x105655240, 0x140007d8960}, 0x14000907a18?)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x90
go.uber.org/yarpc/encoding/protobuf.(*client).Call(0x14000851500, {0x105655240?, 0x140007d8960?}, {0x1050c5f4c?, 0x14000682d58?}, {0x10564dde8?, 0x140007e6500?}, 0x10563b3d0, {0x140007d8a20?, 0x140007df470?, ...})
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0xb8
github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkerAPIYARPCCaller).PollForActivityTask(0x1400016afa0?, {0x105655240?, 0x140007d8960?}, 0x10426c94c?, {0x140007d8a20?, 0x104222190?, 0x140007df440?})
/Users/qlong/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220505064515-7bb6b0808383/go/proto/api/v1/service_worker.pb.yarpc.go:397 +0x64
go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.PollForActivityTask({{0x1056581f0, 0x1400086a260}, {0x10565d0c0, 0x1400086a280}, {0x10565d038, 0x1400086a2a0}, {0x10565ae50, 0x1400086a2c0}}, {0x105655240, 0x140007d8960}, ...)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/compatibility/adapter.go:121 +0x88
go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).PollForActivityTask(0x14000851b60, {0x105655240, 0x140007d8960}, 0x0?, {0x140007d8a20, 0x4, 0x6})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/common/metrics/service_wrapper.go:212 +0x80
go.uber.org/cadence/internal.(*activityTaskPoller).poll(0x1400080de80, {0x105655240, 0x140007d8960})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:991 +0x348
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetrics(0x1400080de80, {0x105655240?, 0x140007d8960?}, 0x14000055e00?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1019 +0x3c
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetricsFunc.func1({0x105655240?, 0x140007d8960?})
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1013 +0x3c
go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:256 +0x50
created by go.uber.org/cadence/internal.(*basePoller).doPoll
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:255 +0x18c
goroutine 131 [select]:
google.golang.org/grpc.(*ClientConn).WaitForStateChange(0x14000866000, {0x1056551d0, 0x140003cd700}, 0x2)
/Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/clientconn.go:515 +0x98
go.uber.org/yarpc/transport/grpc.(*grpcPeer).monitorConnectionStatus(0x140003cd740)
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/peer.go:82 +0x84
created by go.uber.org/yarpc/transport/grpc.(*Transport).newPeer
/Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/peer.go:69 +0x52c
goroutine 133 [select]:
go.uber.org/cadence/internal.(*localActivityTunnel).getTask(0x0?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:167 +0x5c
go.uber.org/cadence/internal.(*localActivityTaskPoller).PollTask(0x14000471340?)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:540 +0x24
go.uber.org/cadence/internal.(*baseWorker).pollTask(0x14000852700)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:263 +0x90
go.uber.org/cadence/internal.(*baseWorker).runPoller(0x14000852700)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:227 +0x8c
created by go.uber.org/cadence/internal.(*baseWorker).Start
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:190 +0x70
goroutine 134 [select]:
go.uber.org/cadence/internal.(*baseWorker).runTaskDispatcher(0x14000852700)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:241 +0xdc
created by go.uber.org/cadence/internal.(*baseWorker).Start
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:194 +0x130
goroutine 135 [select]:
go.uber.org/cadence/internal.(*basePoller).doPoll(0x140008209c0, {0xa0?}, 0x1400042d9b0)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:261 +0x1c4
go.uber.org/cadence/internal.(*workflowTaskPoller).PollTask(0x140008209c0)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:298 +0x70
go.uber.org/cadence/internal.(*baseWorker).pollTask(0x14000852620)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:263 +0x90
go.uber.org/cadence/internal.(*baseWorker).runPoller(0x14000852620)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:227 +0x8c
created by go.uber.org/cadence/internal.(*baseWorker).Start
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:190 +0x70
goroutine 136 [select]:
go.uber.org/cadence/internal.(*basePoller).doPoll(0x140008209c0, {0xa0?}, 0x14000c805b0)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:261 +0x1c4
go.uber.org/cadence/internal.(*workflowTaskPoller).PollTask(0x140008209c0)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:298 +0x70
go.uber.org/cadence/internal.(*baseWorker).pollTask(0x14000852620)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:263 +0x90
go.uber.org/cadence/internal.(*baseWorker).runPoller(0x14000852620)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:227 +0x8c
created by go.uber.org/cadence/internal.(*baseWorker).Start
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:190 +0x70
goroutine 137 [select]:
go.uber.org/cadence/internal.(*baseWorker).runTaskDispatcher(0x14000852620)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:241 +0xdc
created by go.uber.org/cadence/internal.(*baseWorker).Start
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:194 +0x130
goroutine 138 [select]:
go.uber.org/cadence/internal.(*basePoller).doPoll(0x1400080de80, {0xf0?}, 0x14000799968)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:261 +0x1c4
go.uber.org/cadence/internal.(*activityTaskPoller).PollTask(0x1400080de80)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1042 +0x88
go.uber.org/cadence/internal.(*baseWorker).pollTask(0x14000852460)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:263 +0x90
go.uber.org/cadence/internal.(*baseWorker).runPoller(0x14000852460)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:227 +0x8c
created by go.uber.org/cadence/internal.(*baseWorker).Start
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:190 +0x70
goroutine 139 [select]:
go.uber.org/cadence/internal.(*basePoller).doPoll(0x1400080de80, {0xb0?}, 0x140001c9b78)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:261 +0x1c4
go.uber.org/cadence/internal.(*activityTaskPoller).PollTask(0x1400080de80)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1042 +0x88
go.uber.org/cadence/internal.(*baseWorker).pollTask(0x14000852460)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:263 +0x90
go.uber.org/cadence/internal.(*baseWorker).runPoller(0x14000852460)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:227 +0x8c
created by go.uber.org/cadence/internal.(*baseWorker).Start
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:190 +0x70
goroutine 140 [select]:
go.uber.org/cadence/internal.(*baseWorker).runTaskDispatcher(0x14000852460)
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:241 +0xdc
created by go.uber.org/cadence/internal.(*baseWorker).Start
/Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:194 +0x130
FAIL github.com/indeedeng/iwf/integ 1.891s
FAIL
make: *** [integTests] Error 1
Process finished with exit code 2
I got more clues about how this happens:
- When workflow workers are having issues with some errors -- likely some workflow task timeout/fail and cause retry
- When using defer in workflow code
Workaround seems working after removing the defer function: #1198