Write system-tests

Question

Write system-tests

JustinBeckwith opened this issue 6 years ago · comments

It's way to easy to break this module and not know it. All the unit tests are currently passing, but #185 was pretty blatantly broken. We need better system/integration tests that exercise full paths of the code.

Stephen · Answer 1 · Thu Jul 05 2018 22:50:12 GMT+0800 (China Standard Time)

This repo is especially tricky, because it has no awareness of the libraries that use it. We have improved, in that we have nightly CI on those libraries, however.

But, to improve, we could try the following, so we don't have to wait on those nightly builds to catch failures:

Write a system-test script (Mocha or otherwise) that will:

git clone a googleapis/nodejs-* lib
link @google-cloud/common via the master branch
run the googleapis/nodejs-* lib's system tests

If that sounds like what you had in mind, what library/libraries should we use as guinea pigs? Maybe two stables of each library type, Storage (REST) and Datastore (gapic)?

F. Hinkelmann · Answer 2 · Thu Aug 23 2018 00:08:03 GMT+0800 (China Standard Time)

Instead of adding system-tests to common, I'd rather we change our process of publishing. Can we add a requirement like "publish only if all client libraries run system tests successfully?"

Justin Beckwith · Answer 3 · Thu Aug 23 2018 00:17:42 GMT+0800 (China Standard Time)

I love that idea! I think it would be pretty manual to set up though.

F. Hinkelmann · Answer 4 · Thu Aug 23 2018 17:46:05 GMT+0800 (China Standard Time)

Yes - but better than maintaining system tests.

Chris Arnesen · Answer 5 · Thu Sep 20 2018 00:34:55 GMT+0800 (China Standard Time)

I was surprised that this repo doesn't have any tests that actually connects to Google Cloud. I'm assuming that's what we mean by "system tests". I agree that would be a good idea.

Separately it would be good to programmatically test whether the client libraries are able to consume this library as expected. These "integration tests" could be accomplished by having the clients pull in common@latest and then run their system tests. A simpler, somewhat-smaller-scope integration test though would be to have the clients pull in common@latest and then run their unit tests.

As for cadence, at a minimum it would be important to know whether a change was going to break integration tests before publishing because that would influence the major/minor/patch calculus.

sofisl · Answer 6 · Fri Feb 03 2023 04:14:39 GMT+0800 (China Standard Time)

Seems like there is a basic set of system tests now. I think if anyone is interested in adding system tests, they're welcome to open a PR.