Android unable read window content on few devices using accessibility service

Question

My requirement: Reading the text from pop up, dialog etc for particular app.

I have implemented an accessibility service and I am receiving proper events and data as per my requirement. However while testing I realized that on some devices instead of using a AlertDialog or Dialog they have used an activity(themed as a dialog). Hence in my accessibility event I receive only the activity title, is there a way I could find the text displayed by this particular pop up activity?

I have searched pretty hard but could not get much help on the topic nor did the documentation be of any good in this concern. There is not much in the code of accessibility service but if you still need I will post it later.

Thanks

Answer 1

Use accessiblityNodeInfo to get information even in the case of when phone is returning it will fetch the ussd response also it will dismiss dialog when there is option to enter multiple choices.

First of all in case of [pohne] event class name is returned as ussdalertactivty so i used only "alert" for identification of alert dialog of ussd response.

public void onAccessibilityEvent(AccessibilityEvent event) {
    if (event.getPackageName().toString().equals("com.android.phone")
                        && event.getClassName().toString().toLowerCase()
                                .contains("alert")) {

                    AccessibilityNodeInfo source = event.getSource();

                    if (source != null) {
                    String pcnResponse = fetchResponse(source);
                    }
    }

Now i have made a function called fetchResponse which will return the response from pcn as string and will also dismiss the dialog so need to do performGlobalAction(GLOBAL_ACTION_BACK).

private String fetchResponse(AccessibilityNodeInfo accessibilityNodeInfo) {

        String fetchedResponse = "";
        if (accessibilityNodeInfo != null) {
            for (int i = 0; i < accessibilityNodeInfo.getChildCount(); i++) {
                AccessibilityNodeInfo child = accessibilityNodeInfo.getChild(i);
                if (child != null) {

                    CharSequence text = child.getText();

                    if (text != null
                            && child.getClassName().equals(
                                    Button.class.getName())) {

                        // dismiss dialog by performing action click in normal
                        // cases
                        if((text.toString().toLowerCase().equals("ok") || text
                                        .toString().toLowerCase()
                                        .equals("cancel"))) {

                            child.performAction(AccessibilityNodeInfo.ACTION_CLICK);
                            return fetchedResponse;

                        }

                    } else if (text != null
                            && child.getClassName().equals(
                                    TextView.class.getName())) {

                        // response of normal cases
                        if (text.toString().length() > 10) {
                            fetchedResponse = text.toString();
                        }

                    } else if (child.getClassName().equals(
                            ScrollView.class.getName())) {

                        // when response comes as phone then response can be
                        // retrived from subchild
                        for (int j = 0; j < child.getChildCount(); j++) {

                            AccessibilityNodeInfo subChild = child.getChild(j);
                            CharSequence subText = subChild.getText();

                            if (subText != null
                                    && subChild.getClassName().equals(
                                            TextView.class.getName())) {

                                // response of different cases
                                if (subText.toString().length() > 10) {
                                    fetchedResponse = subText.toString();
                                }

                            }

                            else if (subText != null
                                    && subChild.getClassName().equals(
                                            Button.class.getName())) {

                                // dismiss dialog by performing action click in
                                // different
                                // cases
                                if ((subText.toString().toLowerCase()
                                                .equals("ok") || subText
                                                .toString().toLowerCase()
                                                .equals("cancel"))) {
                                    subChild.performAction(AccessibilityNodeInfo.ACTION_CLICK);
                                    return fetchedResponse;
                                }

                            }
                        }
                    }
                }
            }
        }
        return fetchedResponse;
    }

Hope this will solve the issue irrespective of the devices.

Answer 2

This is the code I use and it works for me:

public class USSDService extends AccessibilityService {

public static String TAG = USSDService.class.getSimpleName();

@Override
public void onAccessibilityEvent(AccessibilityEvent event) {
    Log.d(TAG, "onAccessibilityEvent");

    AccessibilityNodeInfo source = event.getSource();
    if(event.getEventType() == AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED && !String.valueOf(event.getClassName()).contains("AlertDialog")) {
        return;
    }
    if(event.getEventType() == AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED && (source == null || !source.getClassName().equals("android.widget.TextView"))) {
        return;
    }
    if(event.getEventType() == AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED && TextUtils.isEmpty(source.getText())) {
        return;
    }

    List<CharSequence> eventText;

    if(event.getEventType() == AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED) {
        eventText = event.getText();
    } else {
        eventText = Collections.singletonList(source.getText());
    }

    String text = processUSSDText(eventText);

    if( TextUtils.isEmpty(text) ) return;

    // Close dialog
    performGlobalAction(GLOBAL_ACTION_BACK); // This works on 4.1+ only

    Log.d(TAG, text);
    // Handle USSD response here

}

private String processUSSDText(List<CharSequence> eventText) {
    for (CharSequence s : eventText) {
        String text = String.valueOf(s);
        // Return text if text is the expected ussd response
        if( true ) {
            return text;
        }
    }
    return null;
}

@Override
public void onInterrupt() {
}

@Override
protected void onServiceConnected() {
    super.onServiceConnected();
    Log.d(TAG, "onServiceConnected");
    AccessibilityServiceInfo info = new AccessibilityServiceInfo();
    info.flags = AccessibilityServiceInfo.DEFAULT;
    info.packageNames = new String[]{"com.android.phone"};
    info.eventTypes = AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED | AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED;
    info.feedbackType = AccessibilityServiceInfo.FEEDBACK_GENERIC;
    setServiceInfo(info);
}
}

Declare it in Android manifest

<service android:name=".USSDService"
android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
<intent-filter>
    <action android:name="android.accessibilityservice.AccessibilityService" />
</intent-filter>
<meta-data android:name="android.accessibilityservice"
    android:resource="@xml/ussd_service" />

Create a xml file that describes the accessibility service called ussd_service

<?xml version="1.0" encoding="utf-8"?>
<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
android:accessibilityEventTypes="typeWindowStateChanged|typeWindowContentChanged"
android:accessibilityFeedbackType="feedbackGeneric"
android:accessibilityFlags="flagDefault"
android:canRetrieveWindowContent="true"
android:description="@string/accessibility_service_description"
android:notificationTimeout="0"
android:packageNames="com.android.phone" />

Answer 3

you should check whether there are any sub child for child nodes.

private void clickPerform(AccessibilityNodeInfo nodeInfo)
 {
   if(nodeInfo != null)
 {

            for (int i = 0; i < nodeInfo.getChildCount(); i++) {
                AccessibilityNodeInfo childNode = nodeInfo.getChild(i);
                Log.e("test", "clickPerform: "+childNode );
                if (childNode != null) {
                    for (int j = 0; j <= childNode.getChildCount(); j++) {
                        AccessibilityNodeInfo subChild = childNode.getChild(i);

                        if (String.valueOf(subChild.getText()).toLowerCase().equals("ok")) {
                            subChild.performAction(AccessibilityNodeInfo.ACTION_CLICK);
                        } else {
                            Log.e("t2", "clickPerform: ");
                        }
                    }
                }

                }
            }
}