Explanation of the differences between testing tools in PlayFramework 2 (WithApplication, WithServer, WithBrowser, InMemory etc...)

Question

I am new to web application development, and even more so with Play Framework. My goal is to ensure my application is well tested, following Test Driven Development principles.

Play provides in its docs several means of testing a Play application, and often times I have difficulty in deciding which kinds of tests I should do, and which ones I can do without.

1) testing controllers vs WithApplication vs WithServer

• option 1 is to test controllers as plain unit test
• option 2 is to test the route using WithApplication and FakeRequest (knowing that the route calls the controller function, this approach feels more complete than option 1)
• option 3 is to use WithServer with WS to make a request and await a response (this feels very similar to option 2, except it's using a real server)

Is testing with option 3 just a redundancy over testing with option 2? Can one be discarded in favor of the other?

2) in memory DB vs real DB

• the in-memory DB (H2) does not seem to support some Postgres functionalities
• testing against in-memory DB does not reflect a connection to a real database

Following the reasons above, I feel like testing with in-memory DB can result in uncaught bugs. Now, I understand that using a real DB is no longer called unit testing, as there are external dependencies. But is unit testing really something we want in this case?

3) WithBrowser (Selenium)

The advantages of this approach are clear, and likely irreplaceable (right?)

Seems like i am missing something when it comes to testing web applications, and clarification would be greatly appreciated.

Answer 1

WithApplication is for testing with a Play application. It's not strictly needed for testing routing/invoking controllers etc, they can all be tested without a running application (except for when they can't - some things rely on global state, but this is something that we are gradually fixing in Play). WithApplication I think is useful for when you want to test all your components working together. By using WithApplication, you let Play instantiate and wire everything together for you, which may be a lot easier than setting it up manually yourself in your tests.

WithServer has a number of interesting use cases. For one, it's more thorough integration testing than WithApplication, if you invoke a controller with a fake request, a lot of short cuts are taken, whereas invoking a controller with a real request over the wire doesn't take any shortcuts. Another interesting use case is testing HTTP client code - you may want to make sure that your HTTP client actually makes HTTP requests that make sense, so you setup some mock controllers with a mock router, and run them with WithServer. Finally, WithServer may be useful if you want to test an actual client to a REST API that you've written, talking to the actual service.

Whether you use an in memory database or a real database for testing is a question of hot debate, and Play is not opinionated here, it gives you the necessary tools for doing both. Some people like to use database abstractions tools, and keep their database access database agnostic. The motivations for this can be wide and varied, and certainly one that comes to play is so that unit testing can be done with in memory databases. Testing with in memory databases offers a lot of advantages, you can instantiate a new database for every test, ensuring test isolation - this is the biggest problem I've seen with running tests against a real database. You can also run your tests in parallel, they are usually faster, and they can run on any platform without any infrastructure setup. Of course, testing against a different database to production does open the possibility for bugs to slip through - but then, anything short of testing every permutation of every possible input and output opens the possibility for bugs to slip through, so all testing is imperfect at best, and a balance has to be achieved between test coverage and convenience of writing and maintainability of tests. So, for some, the advantages of testing against an in memory database outweighs the disadvantages. And then of course, there's people that like to take advantage of database specific features, for these, in memory database testing will be impossible. It's not hard to write test code against a real database in Play, I've done it a lot.