How to access html content of AccessibilityNodeInfo of a WebView element using Accessibility Service in Android?

Question

I'm trying to access the textual content of another app which is probably built using a non-native(js+html) based framework.

Hence, I figured I'll try to access the data from an accessibility node corresponding to a WebView element. However, I'm unable to grab textual/html data using the usual methods since methods like getText() work only if it is a native android element such as a TextView, Button etc.

public class MyAccessibilityService extends AccessibilityService {

@Override
public void onAccessibilityEvent(AccessibilityEvent accessibilityEvent) {
    AccessibilityNodeInfo accessibilityNodeInfo = accessibilityEvent.getSource();
    if (accessibilityNodeInfo == null) {
        return;
    }
    int childCount = accessibilityNodeInfo.getChildCount();
    for (int i = 0; i < childCount; i++) {
        AccessibilityNodeInfo accessibilityNodeInfoChild = accessibilityNodeInfo.getChild(i);
        myRecursiveFunc(accessibilityNodeInfoChild);
    }
}

@Override
public void onInterrupt() {

}


private void myRecursiveFunc(AccessibilityNodeInfo accessibilityNodeInfoChild) {
    if (accessibilityNodeInfoChild.getChildCount() > 0) {
        for (int j = 0; j < accessibilityNodeInfoChild.getChildCount(); j++) {
            AccessibilityNodeInfo child = accessibilityNodeInfoChild.getChild(j);
            if (child != null) {
                myRecursiveFunc(child);
            }
        }
    } else {
        if ("android.webkit.WebView".equals(accessibilityNodeInfoChild.getClassName())) {

            //===========This is a WebView's AccessibilityNodeInfo !!!!
            //===========How to get HTML data from nodeinfo object here ??

        }
    }
}

}

Answer 1

Does your service have FLAG_REQUEST_ENHANCED_WEB_ACCESSIBILITY?

final AccessibilityServiceInfo info = getServiceInfo();
info.flags |= AccessibilityServiceInfo.FLAG_REQUEST_ENHANCED_WEB_ACCESSIBILITY;
setServiceInfo(info);

Android's default AccessibilityService (TalkBack) requires this to be able to screen read the text in WebViews. From examining the source, it appears that there's some script injection and some slightly hackiness in order to be able to get the textual content of the HTML elements

You should check out their source:

TalkBackService

WebInterfaceUtils

ProcessorWebContent