using circleci.test with ultra

Question

using circleci.test with ultra

mt3593 opened this issue 6 years ago · comments

Hi

I would like to use ultra ontop of https://github.com/circleci/circleci.test. But the two don't seem to play nicely together.

I think this is due to the fact circleci.test runs outside of the normal lein test work flow.

Digging into circleci.test a bit more I can see that they rely on reporters to be created that would enable ultra to work with circleci.test. Something that conforms to the protocol TestReporter.

I'm very new to the tooling side of things so i'm a little out of my depth, I'm going to have a play with getting a reporter for ultra setup.

If you have any time could you confirm my above thoughts? And would you be interested in adding such a reporter to this library? Or do you see this a separate?

Mark Tinsley · Answer 1 · Mon Sep 10 2018 21:03:58 GMT+0800 (China Standard Time)

Also opened an issue on circleci.test circleci/circleci.test#30

David Jarvis · Answer 2 · Mon Sep 10 2018 21:23:16 GMT+0800 (China Standard Time)

Hey there.

Can you elaborate a bit on exactly what you mean when you say they don't play nicely together? Something like "I tried to do X and expected Y, but Z happened." These are both somewhat complex projects and there are a number of ways I'd anticipate them to interfere with each other but I'd like a clearer idea of what you want to happen before we start making changes. It may be that there are configuration options that give you what you want without having to make a change.

Mark Tinsley · Answer 3 · Tue Sep 11 2018 02:46:40 GMT+0800 (China Standard Time)

Hey @venantius

Thanks for getting back to me, so I use circle-ci.test for a couple of things in our projects: outputting to junit format to enable our CI/CD pipeline (we use circleci) to understand the output of the tests. We also use it to setup global fixtures.
But I prefer to output ultra gives to tests, especially the output of tests using the diff code in ultra.

@gordonsyme came back on the issue I raised in circleci.test and gave some excellent advice on how the two libraries could work together. From this I've just hobbled together a simple POC https://github.com/mt3593/circleci-ultra-reporter

Is this something you would be interested in adding? Or do you see this as outside of the library?

David Jarvis · Answer 4 · Tue Sep 11 2018 05:30:09 GMT+0800 (China Standard Time)

I think global test fixtures are a feature that probably will never conflict with Ultra and could probably be implemented independent of either library. So I'm going to set that aside for now.

Support for outputting with junit format is something I'd be interested in supporting within Ultra. It might be worth opening a separate issue for that. I don't think I want to support circleci-test in order to support junit, though. That brings on a bit more than I want in order to support the thing that is an obvious value add, and would require some pretty heavy refactoring in order to end up in a clean state.

So I think what I'd recommend would be making a bridge piece of software instead. Something that takes Ultra's test reporting format but uses circleci.test's reporter format. You can then just use circleci.test for testing and disable any potential conflict with Ultra by setting the :test key in your Ultra config to false.

Does that sound reasonable? Is there anything I'm missing?

Mark Tinsley · Answer 5 · Tue Sep 11 2018 14:20:26 GMT+0800 (China Standard Time)

No that sounds fine and makes sense, the reporter is exactly what I've done in the POC above. I'll brush up the POC and work with that. Thanks for all your help, I'll close the issue now.