Putting Tests in a Separate Git Repo
Posted June 2, 2022 ‐ 2 min read
In the debate on whether to put everything in the so-called mono-repo, system tests are usually not considered a separate codebase. My take on the mono-repo vs multiple-repo debate is "split histories on a stable API boundary", and I think that for system tests this split becomes valid too.
With the emerging CI/CD standard stacks that encourage us to put the tests with the code being tested, it seems to me that the tendency to split the tests from the main code gets even weaker, where it promotes a defaulted mono-repo design that unites main code and test code together to the same repository.
In this post I present arguments mostly against putting system tests in the same Git repository.
API compatibility
If a system presents APIs to external entities (i.e. your customers), it means that your system tests should be able to do feature testing using these APIs in a way that would allow new tests to work on old code. Tests relying on features not supported by the old code can be auto-disabled.
The system tests are just another customer that insists on not upgrading across this stable API boundary. The high-level tests are aware of this API layer, and therefore can be versioned separately.
Git Bisect
If you are using Git bisect to pinpoint the change that introduced a bug, then
the test code that reproduces it may be quite new, and the Git history in the
older versions where the bug was introduces does not include that test code at
all. Therefore, it is only natural that the test lives in a separate repo,
otherwise you would need to do weird git checkout maneuvers to get the testee
and tester working on the same run.
Exception: tightly integrated unit tests
One may go too far saying let's put all tests in a separate repo. However, I think that line should be drawn carefully and checked for whether it works in practice. For the tests that are tightly bound to the code being tested, for instance unit tests that depend on internal interfaces or symbols, it may be harder to apply this principle.