KIF: How to auto-run/stress test an iOS app to find the cause of a rare UI bug?

Question

Note: I added kif to the title just for search indexing puposes, considering that most of the answer turned out to discuss it

I'm looking for something like selenium for iOS, basically a test-automation/unit test framework that can run a certain UI scenario many many times until it crashes, which would help me narrow down the cause of a UI bug that happens very rarely and randomly.

(and by the way, I've NSLogged every single line of code of datasource/table interaction and spent hours analyzing the potential cause.. but found nothing conclusive.. again this bug very rarely happens).

I looked at some of the unit testing frameworks in iOS, but they seem to be so many. I'm not sure which to pick. Also my reference to selenium is based on conjecture, as I've worked with QA folks who've used Selenium in large web projects in the past (and i'm assuming that there must be something similar for iOS).

Now that I'm a one man team working on an iOS project, I'm gonna have to put a QA hat on and figure this bug out.

I'm facing a classic bug that happens when there is a discrepancy between the actual number of rows inserted in a UITableView and the number of rows that the datasource delegate returns. This is the error message:

*** Assertion failure in -[UITableView
 _endCellAnimationsWithContext:] Exception in insertRows: Invalid
 update: invalid number of rows in section 0.

The number of rows contained in an existing section after the update (2) must be equal to
 the number of rows contained in that section before the update (2),
 plus or minus the number of rows inserted or deleted from that section
 (1 inserted, 0 deleted) and plus or minus the number of rows moved
 into or out of that section (0 moved in, 0 moved out).

I click on a UITableViewCell that takes me into another UITableView. Sometimes it works

and sometimes (very rarely) it doesn't (with the above error):

Answer 1

update:.. i've added example code about KIF 2.0 at the bottom after divider.. for those who are more interested in KIF than the specific problem i'm facing:

After some research and experimenting.. I've narrowed down my options to two test-automation libraries: Frank and KIF. I ultimately decided to use KIF while borrowing cucumber's Gherkin syntax to describe my unit tests.

The reason I chose KIF (rather than Frank) was that KIF is 100% obj-c based, rather than using ruby as well as was the case with Frank. So setting up is simpler, and it was more applicable to my narrow test case requirement. That being said, I admit Frank would be more useful if my application was more complicated (ie using intput from multiple servers etc). You can see the last quarter of this excellent presentation to learn more about the pros and cons of KIF, Frank and other automation-testing frameworks including Apple's own UI Automation.

After using KIF, I found the bug causing the error above, and I could reproduce it using KIF 100% of the time! The reason why it happened so rarely was because it happened only when I tapped through the screens really fast.. and since KIF automates the steps.. it does them at an incredibly fast speed.. which exposed the bug :).

So following will be a sample of the code I used for testing.. this is just to give you a quick feel of what KIF (and Gherkin) can do for you:

in one file I specify the scenarios I want to run:

- (void)initializeScenarios;
{
    [self addScenario:[KIFTestScenario scenarioToCompleteSignInAndLoadInbox]];
    [self addScenario:[KIFTestScenario scenarioToFillAttachmentsWithData]];
    [self addScenario:[KIFTestScenario scenarioToViewAndLoadFileBucket]];
    [self addScenario:[KIFTestScenario scenarioToViewAndLoadFileBucketSubView]];
}

each scenario maps to steps (to understand more about the gherkin syntax -and behavioral driven development, which is based on test driver development, I strongly recommend to read this excellent book about cucumber):

/* @given the application is at a fresh state
   @and   the user already has an imap email account with a valid username/pwd

   @then  the user can successfully log in
   @and   the inbox view will be loaded
   @and   the inbox will get loaded with the latest batch of emails in the user inbox
 */
+ (id)scenarioToCompleteSignInAndLoadInbox
{
    KIFTestScenario *scenario = 
      [KIFTestScenario scenarioWithDescription:@"Test that a user 
                                                 can successfully log in."];
    [scenario addStepsFromArray:[KIFTestStep stepsCompleteSignInAndLoadInbox]];

    return scenario;
}


/* @given that the user is already signed in
   @and   the user has already downloaded their folders 

   @then  the user can click on the folders view
   @and   the user can click on the 'attachments' remote folder
   @and   the latest batch from the 'attachments' remote folder will download
 */
+ (id)scenarioToFillAttachmentsWithData {
    KIFTestScenario* scenario = 
      [KIFTestScenario scenarioWithDescription:@"Test that we can view the 
                                                 attachments folder and fill 
                                                 it with data."];
    [scenario addStepsFromArray:[KIFTestStep stepsToFillAttachmentsWithData]];
    return scenario;

}

/* @given that the user is already signed in
   @and   the user has already downloaded their folders
   @and   the user has already downloaded attachments

   @then  the user can click on inbox menu button
   @and   the user can click on folder list menu button
   @and   the user can click on the file bucket icon (on the account list view)
   @and   the data for the file bucket is fetched from the dbase
   @and   the file bucket view displayes the attachments
 */
+ (id)scenarioToViewAndLoadFileBucket {
    KIFTestScenario *scenario = 
       [KIFTestScenario scenarioWithDescription:@"Test that a user can successfully 
                                                  view and load 
                                                  file bucket parent view"];
    [scenario addStepsFromArray:[KIFTestStep stepsToViewAndLoadFileBucketPage]];

    return scenario;
}

/* @given that the user is already signed in
   @and   the user has already downloaded their folders
   @and   the user has already downloaded attachments
   @and   the user has already opened file bucket view 

   @then  the user can click on a random row in the file bucket view table
   @and   the subview will retrieve data from the dbase pertaining to that row
   @and   the subview will display the data in the uitableview
 */
+ (id)scenarioToViewAndLoadFileBucketSubView {
    KIFTestScenario *scenario = 
       [KIFTestScenario scenarioWithDescription:@"Test that a user can successfully
                                                  view and load filet
                                                  bucket sub view"];
    [scenario addStepsFromArray:[KIFTestStep stepsToViewAndLoadFileBucketSubPage]];
    return scenario;   
}

and steps are defined using KIF's UI automation methods (this is just one example):

// this step assumes there is an attachment folder that contains emails with attachments
+ (NSArray *)stepsToFillAttachmentsWithData {

    NSMutableArray* steps = [@[] mutableCopy];

    [steps addObject:
        [KIFTestStep stepToTapViewWithAccessibilityLabel:@"InboxMenuButton"]];

    NSIndexPath* indexPath = 
        [NSIndexPath indexPathForRow:remoteAttachmentFolderNumber inSection:0];
    KIFTestStep* tapAttachmentRowStep = 
        [KIFTestStep stepToTapRowInTableViewWithAccessibilityLabel:
                                     @"attachments" atIndexPath:indexPath];

    [steps addObject:[KIFTestStep stepToWaitForNotificationName:
         (NSString *)kBeganSyncingOlderEmails object:nil           
                          whileExecutingStep:tapAttachmentRowStep]];

    [steps addObject:tapAttachmentRowStep];

    [steps addObject:
        [KIFTestStep stepToWaitForViewWithAccessibilityLabel:@"attachments"]];

    KIFTestStep *fillingInboxStep = 
        [KIFTestStep stepToWaitForNotificationName:
                                 (NSString *)kOldMailBatchDelivered object:nil];

    [fillingInboxStep setTimeout:kSpecialTimeoutForLongTests];
    [steps addObject:fillingInboxStep];

    return steps;
}

---

KIF 2.0 sample code: KIF 2.0 uses Xcode 5's all new test navigator.. which is a huge improvement than what KIF 1.0 was doing.. now your tests feel a lot more organic and natural than the past.. (ie it goes in real time.. rather than creating scenarios that run in the future etc).. you even get to test each one with a play button etc.. you should try it out.

here are some examples (again using gherkin syntax):

#import <KIF/KIF.h>
#import "KIFUITestActor+EXAdditions.h"
#import "KIFUITestActor+UserRegistration.h"

@interface LoginTests : KIFTestCase

@end
@implementation LoginTests

- (void)testReset {
    [tester flushDbase];
    [tester reset];
}

/* @given that the app is in a fresh clean state
 @and   that no one has ever registered with the server

 @then  the user can register their themselves with the server
 @and   immediately start with the rider's map
 @and   their location on the map shows
 */

- (void)testRegistration
{
    [tester flushDbase];
    [tester reset];
    [tester singleUserRegistration];
    [tester showUserCurrentLocationOnMap];
}

/* @given that the user has already registered with the server
   @and the user is not currently logged in

 @then  the user can login using their user name and password
 @and   immediately start with the rider's map
 @and   their location on the map shows
 */
- (void)testSuccessfulLogin
{
    [tester reset];
    [tester login];
    [tester showUserCurrentLocationOnMap];
}

/* @given that the user has already registered 
   @and that the user is already logged in before app launch

 @then the user starts on the map view with the location visible
 @and the button prompts them to set pick up location
 */
- (void)testStartOfApplication {
    [tester showUserCurrentLocationOnMap];
    [tester showsPickUpButton];
}
@end

here is the implementation of some of the test cases in the category files:

- (void)reset
{
    [self runBlock:^KIFTestStepResult(NSError **error) {
        BOOL successfulReset = YES;

        // Do the actual reset for your app. Set successfulReset = NO if it fails.
        AppDelegate* appDelegate = [[UIApplication sharedApplication] delegate];
        [appDelegate resetApp];

        KIFTestCondition(successfulReset, error, @"Failed to reset some part of the application.");

        return KIFTestStepResultSuccess;
    }];
}

- (void)flushDbase {
    [self runBlock:^KIFTestStepResult(NSError **error){
        NSURL *url = [NSURL URLWithString:@"http://randomdomain.com/flush_db"];
        NSURLRequest *request = [NSURLRequest requestWithURL:url];
        NSError *connectionError = nil;

        BOOL databaseFlushSucceeded = YES;

        NSURLResponse *response;
        NSData *resultData = [NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&connectionError];
        if (!resultData) {
            databaseFlushSucceeded = NO;
            KIFTestCondition(databaseFlushSucceeded, error, @"failed to connect to server!");
        }

        if (connectionError) {
            databaseFlushSucceeded = NO;
            KIFTestCondition(databaseFlushSucceeded, error, [NSString stringWithFormat:@"connection failed. Error: %@", [connectionError localizedDescription]]);
        }

        return KIFTestStepResultSuccess;
    }];
}


- (void)navigateToLoginPage
{
    [self tapViewWithAccessibilityLabel:@"login email"];
}

- (void)returnToLoggedOutHomeScreen
{
    [self tapViewWithAccessibilityLabel:@"Logout"];
    [self tapViewWithAccessibilityLabel:@"Logout"]; // Dismiss alert.
}