How can I control mobile screen from browser?

Question

Update
References: perfecto mobile and device anywhere

---

I am working on a cloud based mobile testing solution which should support both iOS and Android. It requires connected mobile devices to be handled from browser.

I have learnt that a mobile device has an agent sort of program (which requires device to be rooted) installed which

• streams mobile screen to the browser
• and inject events

Is there any other approach to accomplish this possibly without rooting/jail breaking the device?

This question might look very broad but I have been trying hard to figure out to move in right direction.

For iOS, I am taking clue from this SOF question.

Any pointer is appreciated.

---

Update 1:
This question is close to what I am looking for.

Update 2: I have found Android Screen Library for Android devices and have tested it on couple of devices. It doesn't requires device to be rooted but requires a service restart from Command Line on every device reboot and couldn't get it work on Lollipop..

Update 3: While Android Screen Library helps in capture the screen without rooting but it doesn't help in injecting events. Even screen capture seems buggy - sometime captures with black patch and doesn't work on Lollipop!

Update 4: References: perfecto mobile and device anywhere It seems they are using ADB to handle many things like app install/uninstall, sending events via adb shell input tap x y. Can anybody please shed some light?

Update 5: I have come across this SO Post from Adrian Taylor, an ex-RealVnC engineer. This is the most detailed explanation. Though Android Lollipop has MediaProjection APIs but it seems to store the screenshots as MP4 files on sdcard. Also, as per google dashboard - Aug 2015 update Lollipop is still around 15% of Android installation base, So Kitkat has to be considered for any solution.

Update 6: I have found libvncserver, wondering if it will do the job. I'll test and post the result.

Thanks

Answer 1

Use WebKit, this will get you pretty much as close as you can get and still MORE than likely pass through the approval process with Apple

Here's a simple example, this basically makes a selector on the web page, sends the string value to me on the app, you process that value and send back the result and then you post as the headline of the web page.

<h2 id="headline">loading...</h2>
<select id="selector">
    <option value="systemVersion" selected>iOS Version</option>
    <option value="systemName">System Name</option>
    <option value="name">Device Name</option>
    <option value="model">Device Model</option>
    <option value="userInterfaceIdiom">User Interface</option>
    <option value="identifierForVendor">Vendor ID</option>
</select>

Here's the javascript:

var headline = $("#headline");
var selection = $("#selector");

function set_headline (text) {
    headline.text(text);
}

function call_native () {
    var prop = selection.val();
    set_headline("asked for " + prop + "...");
    window.webkit.messageHandlers.observe.postMessage(prop);
}

setTimeout(call_native, 1000);
selection.on("change", call_native);

On app end, you set up the following:

//configure webView and place webview on screen

[controller addScriptMessageHandler:self name:@"observe"];
configuration.userContentController = controller;
NSURL *jsbin = [NSURL URLWithString:k_JSBIN_URL];

_webView = [[WKWebView alloc] initWithFrame:self.view.frame configuration:configuration];
[_webView loadRequest:[NSURLRequest requestWithURL:jsbin]];
[self.view addSubview:_webView];

handle webView events:

-(void)userContentController:(WKUserContentController *)userContentController didReceiveScriptMessage:(WKScriptMessage *)message
{
// Check to make sure the name is correct
    if ([message.name isEqualToString:@"observe"]) {
        // Log the message received
        NSLog(@"Received event %@", message.body);

        // Then pull something from the device using the message body
        NSString *version = [[UIDevice currentDevice] valueForKey:message.body];

        // Execute some JavaScript using the result
        NSString *exec_template = @"set_headline(\"received: %@\");";
        NSString *exec = [NSString stringWithFormat:exec_template, version];
        [_webView evaluateJavaScript:exec completionHandler:nil];
    }
}