How to debug a second test run ?

Some tests may only fail the second time you run them, but the integrated debugging in GoLand will only run them once by default.

How can we use it to debug them?

The problem with interdependent tests

In an ideal code, unit tests are well isolated from each other, and their dependencies are initialized correctly at each iteration.

But in the real world, this will not always be the case, and a test can perfectly succeed on first run, not alerting to an underlying issue, until a more curious tester runs the test suite in a different order (with go test -shuffle) or several times (with go test -count=2), allowing them to reveal the problem.

One reason this can happen could be use of a global variable by an imported package, the value of which will persist from one test to another, causing different behavior depending on the order of the tests, or the fact that the sequence is executed a second time.

This causes situations where a normal execution will succeed, but another will fail for the same sequence:

• go test → hit
• go test -count=2 → success on the first pass, failure on the second
• go test -shuffle=1 → success sometimes, failure other times

At this point, how can we debug the situation with the Delve debugger built into GoLand?

Default “run configurations” for debugging

To debug a test in GoLand, just click on the green triangle located next to the name of the test of interest in the editor, and choose the option to debug as following picture:

This will create a default run configuration, and start running in debug mode.

The problem is that this execution will only take place once, which will not trigger the problem.

First attempt: mimicking go test -count=2

How would we solve this on the command line? In the example above, we would use for example a syntax like:

go test -count=2 -run=TestCountNumbers .

Looking at the run configuration GoLand just created for on our first try, we do find a place to add arguments to go test:

Alas, when running, the test is still only executed once.

What is going on ? The explanation appears by unfolding the <4 go setup calls> to see how GoLand actually runs the tests in this setup:

In these configurations, as shown in the previous screenshot, GoLand runs the tests a little differently from how we would run them on the CLI:

• it starts by building a test executable with these options:
  • -c to produce a test binary instead of running it on the fly
  • -o <path> to put it in a private location
• then, it runs go tool test2json on the Delve launcher, passing the program built in the first step, along with some flags intercepted by the (self-generated or manual) TestMain:
  • -test.v to display all tests
  • -test.paniconexit0 to trigger panic if a test executes os.Exit(0)
  • -test.run ^\QTestCountNumbers\E$ to run only the chosen test
  • … but nothing else: the -count=2 argument is not part of the operation.

When running that configuration instead of using debugging, the mechanism is the same, save for the fact that GoLand runs the test binary from go tool test2json itself, instead of having Delve run it.

The solution

Let us consider how GoLand transforms the test arguments:

• it produces the test binary, potentially using the arguments of go test that might apply to this step, of which -count is not one
• it runs go tool test2json, separating the flags and arguments intended for the test binary from those meant for go tool itself, using the -- delimiter

If we re-examine the test configuration, below the field for the Go tool arguments meant for go test, we find another field holding the arguments meant for the program itself.

This allows us to pass the count flag from go test as a test.count flag of the test program, similar to how GoLand added test.v and others:

And now, we do get a double run of our problematic test:

All that remains to do is actually debugging it. Should not be too hard!

Credits to Daniil Maslov for suggesting use of that field.