Some tests may only fail the second time you run them, but the integrated debugging in GoLand will only run them once by default.
How can we use it to debug them?
The problem with interdependent tests
In an ideal code, unit tests are well isolated from each other, and their dependencies are initialized correctly at each iteration.
But in the real world, this will not always be the case,
and a test can perfectly succeed on first run, not alerting to an underlying issue,
until a more curious tester runs the test suite in a different order
go test -shuffle) or several times (with
go test -count=2),
allowing them to reveal the problem.
One reason this can happen could be use of a global variable by an imported package, the value of which will persist from one test to another, causing different behavior depending on the order of the tests, or the fact that the sequence is executed a second time.
This causes situations where a normal execution will succeed, but another will fail for the same sequence:
go test→ hit
go test -count=2→ success on the first pass, failure on the second
go test -shuffle=1→ success sometimes, failure other times
At this point, how can we debug the situation with the Delve debugger built into GoLand?
Default “run configurations” for debugging
To debug a test in GoLand, just click on the green triangle located next to the name of the test of interest in the editor, and choose the option to debug as following picture:
This will create a default run configuration, and start running in debug mode.
The problem is that this execution will only take place once, which will not trigger the problem.
First attempt: mimicking go test -count=2
How would we solve this on the command line? In the example above, we would use for example a syntax like:
go test -count=2 -run=TestCountNumbers .
Looking at the run configuration GoLand just created for on our first try,
we do find a place to add arguments to
Alas, when running, the test is still only executed once.
What is going on ? The explanation appears by unfolding the <4 go setup calls> to see how GoLand actually runs the tests in this setup:
In these configurations, as shown in the previous screenshot, GoLand runs the tests a little differently from how we would run them on the CLI:
- it starts by building a test executable with these options:
-cto produce a test binary instead of running it on the fly
-o <path>to put it in a private location
- then, it runs
go tool test2jsonon the Delve launcher, passing the program built in the first step, along with some flags intercepted by the (self-generated or manual)
-test.vto display all tests
panicif a test executes
-test.run ^\QTestCountNumbers\E$to run only the chosen test
- … but nothing else: the
-count=2argument is not part of the operation.
When running that configuration instead of using debugging, the mechanism is the same,
save for the fact that GoLand runs the test binary from
go tool test2json itself,
instead of having Delve run it.
Let us consider how GoLand transforms the test arguments:
- it produces the test binary, potentially using the arguments of
go testthat might apply to this step, of which
-countis not one
- it runs
go tool test2json, separating the flags and arguments intended for the test binary from those meant for
go toolitself, using the
If we re-examine the test configuration, below the field for the
Go tool arguments meant for
go test, we find another field holding the arguments
meant for the program itself.
This allows us to pass the
count flag from
go test as a
test.count flag of the test program, similar to how GoLand added
test.v and others:
And now, we do get a double run of our problematic test:
All that remains to do is actually debugging it. Should not be too hard!
Credits to Daniil Maslov for suggesting use of that field.