Beginning Automated Testing [closed]

Question

I'm ashamed to admit it, but I've never done any automated testing of my code. Usually I'll write a function or method, then do some manual testing and continue on. I realize this is very bad practice so I'm trying to change my ways. Problem is, I'm a bit lost on how to actually start. I'm writing a small Python web application using Flask that is connected to a database and I'd like to use this as a chance to force myself into better development practices.

I'm guessing I need to populate my database with some test data (probably just writing scripts to generate it), then proceed to write tests for each function in my program.

For example: I have a function that pulls all of a users comments from the system. Would I have to write a test for a specific user and then check to make sure the list of returned comment IDs matched an array I created from a manual database query? Is this unit testing?

Just having a hard time understanding it all. Even pointing me towards some newb friendly docs would be very appreciated.

Answer 1

So, just to start with, I would like to mention the purpose of automated testing. The purpose is not to find bugs, but rather to catch regressions. It does have the benefit of catching bugs sometimes, but in general it's a fair amount of work to write and maintain an automated test suite. If you were just out to find bugs, it's far cheaper to just test manually. The advantage of having automated testing is that you can run your suite when you make substantial changes and be more confident you haven't broken the world. This regression testing is the key to why it's worthwhile to write automated tests.

I'm going to describe the methods of testing and then general testing strategies. I'm also just going to explain everything, please don't be offended if you already know half of what I say.

That being said, there are three broad levels of testing:

Unit Testing

Unit testing is testing on the smallest scale. It tests individual functions and/or classes. There are many libraries out there that can be used for unit testing, for all sorts of different languages. One example for python is PyUnit.

You generally accomplish unit testing by writing lots and lots of tiny-scoped tests. For example, if you have written your own container class (unlikely in python but it's a simple example), you would have unit tests for adding items, removing items, looking up items, etc.

Component testing

Component testing is testing your program as individual components. This is generally accomplished by creating mocks of the other components. It's also generally done through the use of a unit testing library, just with larger tests. Where you might have 10's or hundreds of unit tests for each class (largely dependent on the size of the class), you'll have a handful of component tests for each component of your system.

As an example, you mention a database. You could component test by writing a mocked database that returns canned data, and testing the parts that interface with the database using the mock. This is more difficult that unit testing because testability starts to become a concern in the design. Things come up like how are you going to mock a database if the database is hardcoded into your program. It's still relatively easy to do though, it just requires some forethought and planning.

System testing

System testing is the hardest to write tests for, because you have to have integration points to drive your system. On a personal project you can probably skip system testing, especially if you've done a thorough job with unit and component testing.

System testing involves going through scenarios on the whole system. The difficulty is vastly increased, especially for a UI application, because you have to have a way to simulate user input. With a text based system, like compilers, you just have to generate text, but with a UI you have to either drive input or have test hooks into the input callbacks. There is no one size fits all solution like there is for unit testing.

Testing Advice

Real world testing is completely different than what I thought testing was before I got into the industry. When I was in school I wasted a lot of time doing unproductive things that I thought were productive when I was "testing." That being said, here's my advice:

1. Bucketize things, and spread your effort across buckets. Look at various aspect, such as stress, performance, border cases, regular input, localization, UI, and anything else you can come up with a bucket for. Then decide what's worth pursuing and what isn't. Localization likely won't be a priority for a hobby project.
2. Don't test things just to test things. Before you decide to write a test, decide if it's worth writing. For example, if you find out that your website crashes if you give a 500,000 character string as input, would you fix that? I know I wouldn't unless it was a trivial fix. So that testcase isn't worth writing.
3. Do think about the program under test as a piece of code. A lot of people subscribe to the black box view, where you aren't supposed to assume anything about the implementation details, but I think that's unproductive. Write tests that are mean to the implementation, and also that are mean to any future implementations. This goes back to the purpose of automated tests, not to find bugs but to catch regressions later.
4. Think about the border cases. If you are writing a container class like I mentioned previously, you should be thinking about empty containers, containers of max size (or if there is a max size), containers with one element, etc. There should be a few tests for the normal cases and a few more for the border cases. Be pragmatic here though, keep point 2 in mind.

Further reading:

• PyUnit tutorial
• Probably the best testing book in existence