What’s the difference between ScalaTest and Scala Specs unit test frameworks?

Question

Specs and ScalaTest are both good tools with happy users, but they differ in several ways. You will probably want to pick one as your main testing tool in Scala, but need not give up the other because you can use pieces of both. If you like ScalaTest's FeatureSpec syntax and specs' Mockito syntax, for example, you can put both jar files in your classpath and use both at the same time. Here I'll try and capture the main design philosophy differences I've noticed between specs and ScalaTest.

Probably the main philosophical difference between the tools is that specs is designed for Behavior-Driven Development (BDD), whereas ScalaTest is more general. ScalaTest provides traits that you can mix together to get the behavior you prefer in your test classes, including BDD, and you can also easily define your own behavior if you want something different.

ScalaTest supports BDD through its Spec, FeatureSpec, WordSpec, FlatSpec, and GivenWhenThen traits, and also has traits that you can mix in to get a nice matcher syntax. If you like "should", you mix in ShouldMatchers. If you like "must", you mix in MustMatchers. But if you like BDD but don't like matcher syntax, you can just use one of ScalaTest's Spec traits without mixing in a matchers trait. Specs has a Specification class that you extend, and you must use the word "must" in your matcher expressions. A big philosophical difference that is evident here is that ScalaTest gives you a lot more choices. To make this space of choice easier to navigate, I provide a decision tree here:

http://www.scalatest.org/quick_start

The matcher syntax is also different between ScalaTest and specs. In ScalaTest I tried to see how far I could go with operator notation, and ended up with matcher expressions that read very much like English sentences, with spaces between the words. Specs matcher syntax runs words together more with camel case.

Specs has more matchers than ScalaTest, and that I think reflects a difference in design attitude. I actually cut probably 2/3 of the matcher syntax I built and considered for release. I will add more matchers in future releases, but wanted to be sure I knew users actually wanted something before I added it. However ScalaTest's matchers includes a dynamic property matcher syntax takes up some of that slack. For example in Specs you can write on a java.io.File:

file must beDirectory

This will invoke the isDirectory and make sure it is true. ScalaTest does not have any special matchers for java.io.Files currently, but in ScalaTest, you could just use a dynamic check like this:

file must be a ('directory)

Anytime you pass a symbol in after be, it will use reflection to look for (in this case) a method or field named directory or a method named isDirectory. There's also a way to make this static, by defining a BePropertyMatcher (which requires only 2 or 3 lines of code usually). So basically in ScalaTest I try to provide more functionality with less API.

Another general design attitude difference between specs and ScalaTest involves implicit conversions. By default you get only one implicit conversion when you use ScalaTest, which is the one that puts the === operator on everything. (If you need to, you can "turn off" this implicit conversion with one line of code. The only reason you would need to do that is if you were trying to test something that has its own === operator, and you get a conflict.) ScalaTest defines many other implicit conversions, but to use them you need to explicitly "invite" them into your code by mixing in a trait or doing an import. When you extend class Specification in specs I think you pretty much get dozens of implicit conversions by default. I'm not sure how much that will matter in practice, but I figure people will want to test code that uses their own implicits, and sometimes there may be a conflict between the test framework's implicits and those of the production code. When that happens I think it may be easier to work around the problem in ScalaTest than specs.

Another difference in design attitude that I've noticed is comfort with operators. One goal I had was that any programmer looking at someone else's test code that uses ScalaTest would be able to guess what the meaning was without looking anything up in the ScalaTest documentation. I wanted ScalaTest client code to be drop dead obvious. One way that goal manifested itself is that ScalaTest is very conservative about operators. I only define five operators in ScalaTest:

• ===, which means equals
• >, which means greater than
• <, less than
• >=, greater than or equal
• <=, less than or equal.

That's it. So these things pretty much look like what mean. If you see in someone else's code:

result should be <= 7

My hope is that you won't need to run to the API documentation to guess what that <= means. By contrast, specs is much freer with operators. Nothing wrong with that, but it is a difference. Operators can make code more concise, but the tradeoff is you may have to run to the documentation when you find things like ->-, >>, |, |>, !, or ^^^ (which all have special meanings in Specs) in your colleague's test code.

One other philosophical difference is that I do try and make it just slightly easier in ScalaTest to use a functional style when you need to share a fixture, whereas Specs by default continues the tradition of the setUp and tearDown approach popularized by JUnit, in which you reassign vars before each test. However if you want to test that way, it is also very easy in ScalaTest. You just need to mix in the BeforeAndAfter trait.

For more insight into ScalaTest, you can watch the "Get Higher with ScalaTest" presentation I gave at the 2009 Devoxx conference here:

http://parleys.com/play/514892260364bc17fc56bde3/chapter0/about

The main differences are (mostly from a specs point of view :-) ):

• ScalaTest provides more "testing styles" than specs (you can visit each bullet point on the quick start page to get a detailed view on each style)

• ScalaTest and specs have a different set of matchers. You can compare them here for ScalaTest and here for specs. On that side of things, specs has a lot of small features that you may like when writing your specification: xml matchers, matchers composition (an easy way to reuse matchers by transforming them), precise failures, detailed differences for long strings, ...

• Mockito has been given a nice BDD support in specs: Mockito

• specs has DataTables which allow to group a lot of small example in a sort of table (if you can stand operators being used as the table delimiters)

• In specs, you can define examples which are nested as libidum and automatically cleaned-up at every level

This is certainly a very partial and biased comparison and many other differences exist (and the libraries are still evolving, ...).

At the end of the day I think that it really depends on your testing/specifying style. If it's simple (simple specification structure, setups, expectations, ...) then both libraries will appear very similar. Otherwise, both have their take on how things should be done. As a last example of this you can have a look at tagging: in ScalaTest and in specs.

I hope this helps.

As far as I know, barring a few highly specialized features, it comes down to personal preference according to the style.

IDE support may be another point

I've been trying to get Specs to work with Eclipse through JUnit, and I found the official solution to be a bit "hacky". Specs setup: http://code.google.com/p/specs/wiki/RunningSpecs#Run_your_specification_with_JUnit4_in_Eclipse

ScalaTest's integration (also through JUnit) with seems a bit less hacky. Still, I haven't got any of them to work as well as JUnit and Java.

ScalaTest setup: http://groups.google.com/group/scalatest-users/web/running-scalatest-from-eclipse