Include testing framework in git repo?

Question

I'm slowly making git and unit testing parts of my regular workflow. I'd appreciate advice on whether unit testing frameworks (not the tests themselves, but the actual framework) should be included in a project's git repo.

On the one hand, it seems like it should not, as frameworks are a) readily available elsewhere and b)not unique to a project (I'm talking about popular frameworks such as qunit for JS or simpletest in PHP). In that case, perhaps a simple readme in the repo indicating the version of the framework and where to get it would suffice.

On the other hand, the tests ARE part of the project (and part of the repo), and do need the framework to run, so for the sake of completeness, perhaps the framework should be included as well?

Thanks.

Answer 1

I almost always include the framework (a sane one should be a few libraries and executables, so not that large) - the source and tests are versioned, and If I ever go back and get version 1.2 of my code from 2001, I want the tests to run when I make that hotfix - not to fail because I'm using the test framework that I changed to in 2006.

Without the test framework in source control, I add one more thing I have to match versions of. Heck, I'd version the compiler and language libraries in there as well if it were feasible.

If you have a giant test tool/platform that can't easily be put in alongside (or can't legally be put in alongside), make sure you include a doc of some sort that tells what tool, what version, and possibly how to get it alongside the tests.

Answer 2

Reading the answers to this question is interesting, as it shows a very broad range of interpretations of many different aspects of software development. I am normally of the belief that dependencies of any kind should not be stored in the VCS (by VCS, I refer to tools like git, hg, svn, cvs, and not larger systems that incorporate such tools, like Xcode) because that is not the job of the VCS. Dependency tracking is best left to a packaging tool. (It seems that many people use the VCS as a primitive packaging tool, which IMHO is a huge mistake.)

I think you need to clarify what you mean by 'test framework'. For me, most of my test suites are invoked by the framework provided by automake. I certainly do not include automake in my git repository, but there is no issue with version differences in the future because all of the test framework generated by automake is included in the distribution tarball. So really the question is pushed back one more level. If you are using only a VCS to track your releases, then you have a problem because you really do need to have the test framework available when you check out an old version of the project. If the VCS is the only storage mechanism for retrieving old releases (ie, you aren't using any sort archiving of releases in tarballs), then it seems you are forced to put the framework in the VCS. But putting external software in your VCS is, frankly, stupid. The only reason to do that is if you are using the VCS as your distribution tool. So I suppose my answer is: don't put the framework in your VCS, but do put the framework in your distributions. If your VCS is your distribution tool, then there is no proper way to deal with the test framework.