How to do fuzzing testing with Selenium

Question

I'm new to Selenium, and also fuzz testing. I see that Selenium IDE only allows the fixed test cases. But then fuzz testing seems to be helpful.

So what's behind a fuzz testing, what kind of tests does Selenium offer, is this a black box or white box testing.

Any help would be appreciated.

Answer 1

For a short answer:

• Selenium is mostly about black-box testing, but you could do some whiter testing also with Selenium.
• Selenium RC gives you much more freedom to do fuzz testing than Selenium IDE.

For a long answer, see below:

In this post I would try to explain the concept of randomly testing your web application using Selenium RC.

Normally speaking, a black-box testing technique like Selenium gives you a good freedom to

• (1) Enter any value to a certain field
• (2) Choose any field to test in a certain HTML form
• (3) Choose any execution order/step to test a certain set of fields.

Basically you

• use (1) to test a specific field in your HTML form (did you choose a good maximum length for a field), your JavaScript handling of that field's value (e.g. turning "t" into today's date, turning "+1" into tomorrow's date), and your back end Database's handling of that variable (VARCHAR length, conversion of numerical string into numerical value, ...).
• use (2) to test ALL possible fields
• use (3) to test the interaction of the fields with each other: is there a JavaScript alert popped up if the username field was not entered before the password field, is there a database (e.g. Oracle) trigger "popped up" when certain condition is not met.

Note that testing EVERYTHING (all states of your program, constructed by possible combinations of all variables) is not possible even in theory (e.g.: consider testing your small function used to parse a string, then how many possible values does a string have ?). Therefore, in reality, given a limited resource (time, money, people) you want to test only the "most crucial" execution paths of your web application. A path is called more "crucial" if it has more of the properties: (a) is executed frequently, (b) a deviation from specification causes serious loss.

Unfortunately, it is hard to know which execution cases are crucial, unless you have recorded all use cases of your application and select the most frequent ones, which is a very time consuming process. Furthermore even some bugs at the least executed use case could cause a lot of trouble if it is a security hole (e.g. someone steals all customers' password given a tiny bug in an URL handling of some PHP page).

That is why you need to randomly scan the testing space (i.e. the space of values used in those use cases), with the hope to run-something-and-scan-everything. This is called fuzz testing.

Using Selenium RC you could easily do all the phases (1), (2) and (3): testing any value in any field under any execution step by doing some programming in a supported language like Java, PHP, CSharp, Ruby, Perl, Python.

Following is the steps to do all these phases (1), (2) and (3):

• Create list of your HTML fields so that you could easily iterate through them. If your HTML fields are not structurized enough (legacy reason), think of adding a new attribute that contains a specific id, e.g. selenium-id to your HTML element, to (1) simplify XPath formation, (2) speed up XPath resolution and (3) to avoid translation hassle. While choosing the value for these newly added selenium-id, you are free to help iterating while fuzzing by (a) using consecutive numbers, (b) using names that forms a consistency.
• Create a random variable to control the step, say rand_step
• Create a random variable to control the field, say rand_field
• Eventually, create a random variable to control the value entered into a certain field, say rand_value.
• Now, inside your fuzzing algorithm, iterate first through the values of rand_step, then with each such iteration, iterate through rand_field, then finally iterate through rand_value.

That said, fuzz testing helps to scan your whole application's use case values space after a limited execution time. It is said that "a plague of new vulnerabilities emerge that affected popular client-side applications including Microsoft Internet Explorer, Microsoft Word and Microsoft Excel; a large portion of these vulnerabilities were discovered through fuzzing"

But fuzz testing does not come without drawback. One if which is the ability to reproduce a test case given all those randomness. But you could easily overcome this limitation by either doing one of the following:

• Generating the test cases before hand in a batch file to be used in a certain period of time, and apply this file gradually
• Generating the test cases on the fly, together with logging down those cases
• Logging down only the failed cases.