Shoes Spec, Initial Version
noahgibbs opened this issue · comments
This is finally near enough I can put it up as an issue usefully.
For awhile we've needed to distinguish between "this is a bug in Scarpe" and "this is something Shoes would allow but we don't." Sometimes that's because we're not building it yet (hover/motion/animate events) and sometimes because of limits in our underlying display library (multi-App multi-window). But either way, we've had to just mentally keep track of "this is a bug in Scarpe for fixing soon" and "this is just different from Shoes."
A Shoes Spec, which Scarpe won't necessarily rate 100% on, is a way to separate those two things. It's okay to have Shoes tests that Scarpe-Webview fails, like for multi-window.
One practical requirement of a Shoes Spec is that we need a way to load a Shoes app, then run a test against it. This is a bit like CatsCradle, but not quite (CatsCradle was an important step toward a Shoes Spec!)
Some important steps along the way:
- Having a custom bespoke assertion library for the spec is silly. We shouldn't make people reimplement it. I'm going with Minitest for our assertions. I have Minitest importer/exporter libs written. I may put them up in a PR before the rest is ready. I'm hoping I can switch CatsCradle over to using them.
- An implementation harness for each library. I'm working on the Scarpe-Webview one very actively right now.
- A suite of tests that can run against the implementations. I have only the barest beginnings of this.
- A portable testing language, allowing a portable test to specify something like "click at point X and then move to (x1, y1), then (x2, y2)" and other higher-level things like "click the OK button" or "select this radio button." This is going to be ongoing work in the spec and the display services, which will need to implement it.
- a certain amount of selecting and scoring tests, reporting how well an implementation does, etc. This will be straightforward and fall out over time.
PR #399 is a start on this.
PR #435 helps this one.
The scarpe_wasm branch of the shoes-spec repo has a good start on this -- it's loading the apps in scarpe-wasm and running Capybara, but I still need to implement the ShoesSpec test API methods in Capybara. That's next.
Okay. Plenty of room to improve this, but the bare beginning is there, running the same two tests for Webview and Wasm.