TestNG: Which approach is better? Login once and run all tests (@BeforeSuite) or Login before every test (@BeforeTest)

Question

APPROACH#1: Login before each test:

PROS:

1. Every test is a different session and if one test causes browser crash, only that test fails and it does not affect other tests.
2. Parallel tests possible.

CONS:

1. If login fails (because the AUT was down or the login functionality has a bug), we still call it before each and every test. But this is a minor problem as this can be fixed by creating a test that tests login functionality and can abort the entire suite if it fails.

APRROACH#2: Login once and run all tests:

PROS:

1. You do not have to deal with so many browser windows because you will have just one open

CONS:

1. If a test causes a browser crash, can we recover before the next test starts?
2. Can we run more than 1 test in parallel?

Although both the approaches have their pros and cons, I'm still not sure which is the best way to go. Is it really a waste of resources to login before each test as a separate session? There is no DB connection involved. The only potential problem I see with APPROACH#1 is that since every test is a Java process, there is a chance of overloading the system with too many simultaneous java processes.

Answer 1

The answer is, You decide.

But, here is my opinion:

@BeforeSuite is too risky and not realistic as the UI tests are the most flakiest in testing world. There are thousand different reasons the login process can fail. Specially if you run those tests automatically from CI server or through some other automated process your full test suit will not execute.

@BeforeTest is too redundant I believe even though sometimes you need to just to refresh the environment and go back to the known state. Except for those scenario I will not do this either.

Upto this point and as per my knowledge @BeforeClass is the best approach. Since, this will run just once per test class and log in once and execute all the tests. You can use @BeforeTest in conjunction to this just to reset the environment to a known state so that other tests know where to start from

Answer 2

If "a waste of resources" is your cause for concern, explore reducing the number of integration tests you need to run (e.g. by creating more unit / functional tests). You can configure the number of threads TestNG uses, and how parallel your tests are, so the extra sessions shouldn't crash your test process.

Integration tests are supposed to be expensive in order to actually model the whole system. If users need to log in, your test should be logging in. If you try to fake or modify the behavior for testing you only make your tests less useful; and seeing as they're expensive, you want to make them as useful as possible. You also always want to make your tests independent of each other where possible; once one test fails, having thirty other tests fail for the same reason is just noise that makes reporting and responding to failure more difficult.

Therefore what you should do is identify tests that are sure to fail if other tests fail (e.g. all tests that need to log in will fail if your login test fails) and avoid running them. You can do this with dependencies. I would suggest defining groups for your entry-point tests, like logging in, and using dependsOnGroups to ensure all tests that have to log in only run once that's verified.