How can you ensure secure coding with Test Driven Development?

Question

I've been coming up to speed on the latest trend that is Test Driven Development (TDD). Most of the development I do is in C or C++. It strikes me that there is a very obvious conflict between common TDD practices and common secure coding practices. At it's heart, TDD tells you that you shouldn't write new code for something that you don't have a failing test for. To me, that means that I shouldn't write secure code unless I have unit tests to see if my code is secure.

That brings up two issues:

1. How can I effectively write unit tests to test for Buffer Overflows, Stack Overruns, Heap Overruns, Array Index Errors, Format String Bugs, ANSI vs Unicode vs MBCS string size mistmatches, an Safe String Handling (from Howard and LeBlanc's "Writing Secure Code")?

2. At what point in the standard TDD practice should these tests be included since much of security is non-functional.

Surprisingly, I have found very little research discussing TDD and security. Most of what I come across are TDD papers that mention at a very high level that TDD will "make your code more secure."

I'm looking for any direct answers to the issues above, any research that pertains to this (I looked already and didn't find much), or any place that TDD guru's live so I can go knock on their door (virtually) and see if they have any good answers.

Thanks!

EDIT:

The topic of Fuzzing has come up, which I think is a great approach to this problem (in general). This raises the questions: Does Fuzzing fit into TDD? Where in the TDD process does fuzzing fit?

Parameterized Unit Testing (possibly automated) has also crossed my mind. This might be a way to get fuzzing-like results earlier into the testing process. I'm not sure exactly where that fits into TDD either.

EDIT 2:

Thank you all for your answers thus far. At this point, I am extremely interested in how we can leverage parameterized tests to serve as pseudo fuzzers for our functions. But, how do we determine what tests to write for testing security? And how can we be sure that we adequately cover the attack space?

It is a well known problem in software security that if you protect against 5 attack scenarios, the attacker will just look for, and use, a 6th attack. It is a very difficult cat-and-mouse game. Does TDD give us any advantage against this?

Answer 1

Yes, TDD is a tool/technique that can help to ensure secure coding.

But as with all things in this industry: assume it's a silver bullet, and you'll shoot yourself in the foot.

Unknown Threats

As you indicated in Edit 2: "you protect against 5 attack scenarios, the attacker will just look for, and use, a 6th attack". TDD is not going to protect you from unknown threats. By its very nature, you have to know what you want to test in order to write the test in the first place.

So suppose threat number 6 is discovered (hopefuly not due to breach, but rather internally due to another tool/technique that attempts to find potential attack vectors).

TDD will help as follows:

• Tests can be written to verify the threat.
• A solution can be implemented to block the threat, and quickly be confirmed to be working.
• More importantly, provided all other tests still pass, you can quickly verify that:
  • All other security measures still behave correctly.
  • All other functionality still behaves correctly.
• Basically TDD assists in allowing a quick turnaround time from when a threat is discovered to when a solution becomes available.
• TDD also provides a high degree of confidence that the new version behaves correctly.

Testable Code

I have read that TDD is often misinterpreted as a Testing Methodology, when in fact it is more of a Design Methodology. TDD improves the design of your code, making it more testable.

Specialised Testing

An important feature of test cases is their ability to run without side-effects. Meaning you can run tests in any order, any number of times, and they should never fail. As a result, a number of other aspects of a system become easier to test purely as a result of the testability. For example: Performance, Memory Utilisation.

This testing is usually implemented by way of running special checks of an entire test suite - without directly impacting the suite itself.

A similar security testing module could overlay a test suite and look for known security concerns such as secure data left in memory, buffer overruns or any new attack vector that becomes known. Such an overlay would have a degree of confidence, because it has been checked for all known functionality of the system.

Improved Design

On of the key design improvements arising as a side-effect of TDD is explicit dependencies. Many systems suffer under the weight of implicit or derived dependencies. And these would make testing virtually impossible. As a result TDD designs tend to be more modular in the right places. From a security perspective this allows you to do things like:

• Test components that receive network data without having to actually send it over the network.
• One can easily mock-out objects to behave in unexpected / 'unrealistic' ways as might occur in attack scenarios.
• Test components in isolation.
• Or with any desired mix of production components.

Unit Testing

One thing that should be noted is that TDD favours highly localised (unit testing). As a result you could easily test that:

• SecureZeroMemory() would correctly erase a password from RAM.
• Or that GetSafeSQLParam() would correctly guard against SQL injection.

However, it becomes more difficult to verify that all developers have used the correct method in every place that it's required.
A test to verify a new SQL related feature would confirm that the feature works - it would work just as well with both the 'safe' and 'unsafe' versions of GetSQLParam.

It is for this reason you should not neglect other tools/techniques that can be used to "ensure secure coding".

• Coding Standards
• Code Reviews
• Testing