Unit Testing Third Party ORM

Question

I've read a few threads on SO about usefulness of unit-testing various applications. The opinions can range from "test everything all the time" to "unit tests are useless", and everything in between ("test where it makes sense"). I tend to lean towards the middle one.

That leads me to my question. I am trying to decide if it would be beneficial or practical to have some basic unit-tests testing 3rd party ORM as suggested in this SO post: link text

some baseline tests may be useful as insurance against future breaking changes, depending on how you are using the tool. For example, instead of mocking up the entire n-tier chain (I'm not a fan of mocking when it is not necessary), just use the ORM tool to create, read, update, and delete a typical object/record, and verify the operation using direct SQL statements on the (test) database. That way if the 3rd-party vendor later updates something that breaks the basic functionality you'll know about it, and new developers to your project can easily see how to use the ORM tool from the unit test examples.

My main reservations following this advise is that it would require way too much setup, would be a headache to maintain, and over all it would not be practical in our environment. Here's the summary of some points to consider:

1. The ORM we're using requires static datasource object(s) to be created and registered with its Data Access Layer and associated with authenticated user. This would require a lot of test setup, and probably would be problematic on the build server where no user is logged on.
2. ORM vendor has a pretty good track record of releasing new updates and not breaking basic functionality. Furthermore whenever it's time to update ORM to the latest version, I would imagine that application wouldn't go straight to production, but would be thoroughly regression tested anyway.
3. Maintaining test db for Unit testing is kind of problematic in this environment. Test db gets wiped out after each major release and replaced with db backup from staging with sensative data obfuscated. I would imagine in order to have a test db for ORM unit testing, we would need to run some scripts/code that would set the database in a "test" state. Again too much setup and maintenance.
4. And finally ORM documentation/help for new developers. I can see how something like that could be useful. But ORM vendor provides pretty good documentation/help with demo apps. So writing unit tests on top of that doesn't seem to be worth all the efforts.

So, is it worth to go through all these troubles just to make sure that ORM does what it supposed to do (which is CRUD)? Shouldn't it be a responsibility of the vendor anyway?

Answer 1

You said it yourself. Test where it makes sense. If it will make you "feel" better to test the 3rd party ORM, then do it. But, ultimately, you're putting your trust in their tool. What are you going to do if the ORM suddenly stops working? Have you written enough code against it that you can't easily rip it out? You'd wait for them to fix it, probably.

Basically, you have to treat 3rd party tools as the proverbial black boxes and let them do what you bought them to do. That was the reason you paid the money you did, right? To keep from having to write that yourself.

Answer 2

In this particular case, I wouldn't bother. I think you are correct in assuming a bad ROI here.

And yes, I consider it the responsibility of the vendor. I expect (and assume) their stuff works. That's how I treat my vendors.