Dependent Tests, skip some subset of tests if the parent test fails

Question

Dependent Tests, skip some subset of tests if the parent test fails

TJM opened this issue 4 years ago · comments

Perhaps I am just not searching for the right terms, so I am going to try to explain what I am trying to accomplish...

I am currently adding tests to github.com/TJM/go-trello and I have hit a conditions multiple times where tests need to depend on the previous test. For example:

Create Board
Set Board to red
Set Board Description
(etc)

If "Create Board" fails, the other two need to "skip" or "fail fast" or something, but currently it just blows up and I get the go equivalent of a null pointer exception.

I was thinking of adding an assertion at the top, which should cause the test to fail at least (better than the crash I was getting before), but the failure state would look better if it showed them as dependent tests that were skipped :-/

// NOTE: "client" is setup in "init()" currently, but could probably be moved to a g.Before()?

		g.It("should create a board", func() {
			board, err = client.CreateBoard(testBoardName)
			Expect(err).To(BeNil())
			Expect(board).NotTo(BeNil())
			Expect(board.Name).To(Equal(testBoardName))
		})

		g.It("should get a board by ID", func() {
			Expect(board).NotTo(BeNil()) // Create Board needs to have succeeded
			board, err = client.Board(board.ID)
			Expect(err).To(BeNil())
			Expect(board).NotTo(BeNil())
			Expect(board.Name).To(Equal(testBoardName))
		})

		g.It("should change the board background to red", func() {
			Expect(board).NotTo(BeNil()) // Create Board needs to have succeeded
			err = board.SetBackground("red")
			Expect(err).To(BeNil())
			Expect(board.Prefs.Background).To(Equal("red"))
		})

		g.It("should change the description to something", func() {
			Expect(board).NotTo(BeNil()) // Create Board needs to have succeeded
			err = board.SetDescription("something")
			Expect(err).To(BeNil())
			Expect(board.Desc).To(Equal("something"))
		})

Marcos Nils · Answer 1 · Tue Oct 27 2020 02:03:11 GMT+0800 (China Standard Time)

Dependent tests is a practice that's not usually encouraged since the idea for tests is that they could potentially run in any order.

What you can do instead is to use the before and beforeEach blocks to perform certain operations that will get executed before every test, like creating a new board for example.

Marcos Nils · Answer 2 · Tue Oct 27 2020 02:05:29 GMT+0800 (China Standard Time)

		g.It("should create a board", func() {
			board, err = client.CreateBoard(testBoardName)
			Expect(err).To(BeNil())
			Expect(board).NotTo(BeNil())
			Expect(board.Name).To(Equal(testBoardName))
		})

		g.It("should get a board by ID", func() {
			Expect(board).NotTo(BeNil()) // Create Board needs to have succeeded
			board, err = client.Board(board.ID)
			Expect(err).To(BeNil())
			Expect(board).NotTo(BeNil())
			Expect(board.Name).To(Equal(testBoardName))
		})

		g.It("should change the board background to red", func() {
			Expect(board).NotTo(BeNil()) // Create Board needs to have succeeded
			err = board.SetBackground("red")
			Expect(err).To(BeNil())
			Expect(board.Prefs.Background).To(Equal("red"))
		})

		g.It("should change the description to something", func() {
			Expect(board).NotTo(BeNil()) // Create Board needs to have succeeded
			err = board.SetDescription("something")
			Expect(err).To(BeNil())
			Expect(board.Desc).To(Equal("something"))
		})

^ With this you can do something like

g.Describe("Board")
   g.BeforeEach(func() {
         // Create new board here before each test
   })
   g.AfterEach(func() {
         // Cleanup created board.
   })
   g.It("should be created with params xxx") // Test that creates a new board
   g.It("should be fetched by id") // Fetch the board created in the beforeEach function
.....

Makes sense?

If creating and destroying the board before each it block is too much you can use before and after to do it once in the same Describe block and re-use it across all the it cases.

Tommy McNeely · Answer 3 · Tue Oct 27 2020 03:13:58 GMT+0800 (China Standard Time)

Hmm, thanks for the suggestion...

Re: BeforeEach/AfterEach - I am already having issues (429) with hitting my limits on API calls while running tests, so I think I will try to do my best to work with one set of prerequisites for the entire "type" instead of creating/deleting them foreach.

Re: Before/After - I am already attempting to do that to create the prerequisites for the deeper tests... for example: https://github.com/TJM/go-trello/blob/master/card_test.go#L39-L54

... but I don't think it really likes the assertions I put into the before ;)

The "structure" is something like:

trello.Client:
  member:
  board:
    - list:
      - card:
        - checklist:
            - item

... so, for example, in order to test "card" (SetName, SetDescription, AddMember, etc), you need to have a working client, board, and card. I have each of those in the "before" ... maybe it just comes down to having the tests in the wrong place, but it seems crazy to not test checklist items while you have just created a checklist, rather than to tear it all down and build it in another file.

Tommy McNeely · Answer 4 · Fri Oct 30 2020 03:41:16 GMT+0800 (China Standard Time)

It would be nice to have the ability to indicate dependent tests in order to "skip" them when their prerequisite test fails. However, I re-arranged my tests so that the g.Before() lays out all the pre-requisites for that test. I guess that comes down to a "code organization" thing.. I am "new" to go... obviously. :)

The previous developer (that I forked from) chose to put "CreateBoard" into the "board.go" instead of the "client.go" even though it starts with func (c *Client) ... perhaps that should be in "client.go" anyhow? (shrug)... I moved the test to client_test.go, and added the board prereq to g.Before... but along those same lines func (c *Client) Board(boardID string) (board *Board, err error) is also arguably a "client" operation, but I need a "boardID" to be able to run it, so I left that inside the board_test.go, where I should presumably have a working "board" object (board.ID).

Thanks,
Tommy

Tommy McNeely · Answer 5 · Sat Oct 31 2020 05:34:44 GMT+0800 (China Standard Time)

OK, new question... How do you "fail" the g.Before()?

I have the following error:

--- FAIL: TestMembership (1.31s)
panic: Asserts should be written inside an It() block. [recovered]
	panic: Asserts should be written inside an It() block.

goroutine 35 [running]:
testing.tRunner.func1.1(0x132f280, 0x140b280)
	/usr/local/Cellar/go/1.15.2/libexec/src/testing/testing.go:1076 +0x30d
testing.tRunner.func1(0xc000184600)
	/usr/local/Cellar/go/1.15.2/libexec/src/testing/testing.go:1079 +0x41a
panic(0x132f280, 0x140b280)
	/usr/local/Cellar/go/1.15.2/libexec/src/runtime/panic.go:969 +0x175
github.com/franela/goblin.(*G).errorCommon(0xc00019c4e0, 0xc00022e500, 0x13e, 0x1)
	/Users/tmcneely/go/pkg/mod/github.com/franela/goblin@v0.0.0-20201006155558-6240afcb2eb7/goblin.go:356 +0x176
github.com/franela/goblin.(*G).Fail(0xc00019c4e0, 0x132f280, 0xc00029a3b0)
	/Users/tmcneely/go/pkg/mod/github.com/franela/goblin@v0.0.0-20201006155558-6240afcb2eb7/goblin.go:375 +0x99
github.com/TJM/go-trello.TestMembership.func1(0xc00022e3c0, 0x13e, 0xc0004dcfb8, 0x1, 0x1)
	/Users/tmcneely/go/src/github.com/TJM/go-trello/membership_test.go:30 +0x58

... so it did "sorta" work, in that all the other tests didn't try to run, but as I said before, it doesn't like assertions in the g.Before... what is the proper way to "fail" or can we make it so that g.Before can have assertions?