How do you evaluate reliability in software?

Question

We are currently setting up the evaluation criteria for a trade study we will be conducting.

One of the criterion we selected is reliability (and/or robustness - are these the same?).

How do you assess that software is reliable without being able to afford much time evaluating it?

Edit: Along the lines of the response given by KenG, to narrow the focus of the question: You can choose among 50 existing software solutions. You need to assess how reliable they are, without being able to test them (at least initially). What tangible metrics or other can you use to evaluate said reliability?

Answer 1

Reliability and robustness are two different attributes of a sytem:

Reliability

The IEEE defines it as ". . . the ability of a system or component to perform its required functions under stated conditions for a specified period of time."

Robustness

is robust if it continues to operate despite abnormalities in input, calculations, etc.

So a reliable system performs its functions as it was designed to within constraints; A robust system continues to operate if the unexpected/unanticipated occurs.

If you have access to any history of the software you're evaluating, some idea of reliability can be inferred from reported defects, number of 'patch' releases over time, even churn in the code base.

Does the product have automated test processes? Test coverage can be another indication of confidence.

Some projects using agile methods may not fit these criteria well - frequent releases and a lot of refactoring are expected

Check with current users of the software/product for real world information.

Answer 2

It depends on what type of software you're evaluating. A website's main (and maybe only) criteria for reliability might be its uptime. NASA will have a whole different definition for reliability of its software. Your definition will probably be somewhere in between.

If you don't have a lot of time to evaluate reliability, it is absolutely critical that you automate your measurement process. You can use continuous integration tools to make sure that you only ever have to manually find a bug once.

I recommend that you or someone in your company read Continuous Integration: Improving Software Quality and Reducing Risk. I think it will help lead you to your own definition of software reliability.