Error vs Fatal in tests

Question

I am developing a JSON web service using Go-Json-Rest. I am writing tests.

...
recorded = test.RunRequest(t, &api.Handler,
    test.MakeSimpleRequest("POST", "http://localhost/api/products",
        product))
recorded.CodeIs(201)
recorded.ContentTypeIsJson()

var newProduct Product
err := recorded.DecodeJsonPayload(&newProduct)
if err != nil {
    t.Fatal(err)
}
...

I am using Fatal as I am coming from Python world where an assert would immediately stop test case method execution. And this make sense: why trying to decode the data, if it's not JSON?

But recorded.CodeIs(201), recorded.ContentTypeIsJson() and other tests I've seen use Error which doesn't stop test execution.

What should I use in tests? Error or Fatal?

Answer 1

I think you use Error until continuing to run the test can't possibly give you any more information useful in debugging, then you use Fatal. And if you're not sure (like if you're writing a factored-out method like CodeIs to be used in the context of lots of different tests), go for Error, since you're generally not doing harm by continuing to run the test.

By that criteria, it makes sense for you to Fatal at failed JSON decoding after which, as you say, nothing interesting is going to happen. And it's understandable that CodeIs and ContentTypeIsJson use Error because they're methods that are going to be used across different tests.

A different example might better illustrate why to use Error until you know nothing else interesting will happen: say you want to sanity-check several different things about the JSON response, and any subset them could be wrong. (Like, your product API could return price using the wrong type, or it could fail to return empty descriptions when you don't want that, or...) Using Error instead of Fatal for each check means your test will always run them all and report which ones failed.