Explore state exploration in turmoil
mcches opened this issue · comments
State exploration entails navigating all (or some portion of) the possible states a program can enter during its execution. Model checkers exist (TLA+, P, etc.), but they require building a model that is separate from the actual implementation.
turmoil provides an interesting opportunity where we are running all of the real code, but with a simulated network. The network provides a place to both view states and control state transitions. Can we expose the right APIs to make state exploration possible?
Note that this approach differs from fuzzing the network, which is already possible today.
One idea for this is to track states externally, but provide enough hooks to support it. The advantage here is that consumers of turmoil have the ability to understand the encoded bytes that are written and read via the networking types, as they define the protocols.
Currently, Sim::run() runs an event loop that drives all the hosts and the network forward. We could break the logic out into a public step API that allows callers to control this. Instead of running a simulation to completion, the test needs manage its own event loop.
Within the loop we'd:
- step (run a single iteration on each host) and hold all packets
- introspect the network to identify possible states
- pick a state, track it, and proceed by releasing messages for delivery
- repeat until we hit a terminal path
We'll also need a top level loop to traverse all the possible paths through the graph. Each time we hit a terminal path we can rebuild the simulation (note things are deterministic) and run a new path.
Some thoughts from the original repo:
We could break the logic out into a public step API that allows callers to control this
This would be very useful to us but for a different reason. We need to run a custom piece of logic after every tick in order to coordinate some tasks that are not managed by tokio. Being able to manage our own even loop would allow us to do that easily.
We could break the logic out into a public step API that allows callers to control this
This would be very useful to us but for a different reason. We need to run a custom piece of logic after every tick in order to coordinate some tasks that are not managed by tokio. Being able to manage our own even loop would allow us to do that easily.
Great to hear that this would be useful! I think the initial changes would make your use case very easy to support.
#76 adds "step" capabilities.
#77 adds the ability to hold messages using patterns.
Once 0.5 is released we should have the building blocks in place to start exploring.
I'm closing this issue as exposing the above linked hooks make it possible to investigate and choose different paths throughout the execution of the sim.