How to use Accessibility Services for "Taking Action for Users"?

Question

Background

Back a few years ago, I asked how TeamViewer allows the user to control the device without normal interaction with the device. I was told it's a special "backdoor" that manufacturers allow specifically for this app, and only possible using root priviledge for other apps.

Seeing that an app like "Airplane Mode Shortcut" allows to toggle airplane mode, by automatic navigation to its screen and toggling the switch, it made me realize this situation has changed.

The problem

It is said in the docs:

Starting with Android 4.0 (API Level 14), accessibility services can act on behalf of users, including changing the input focus and selecting (activating) user interface elements. In Android 4.1 (API Level 16) the range of actions has been expanded to include scrolling lists and interacting with text fields. Accessibility services can also take global actions, such as navigating to the Home screen, pressing the Back button, opening the notifications screen and recent applications list. Android 4.1 also includes a new type of focus, Accessibilty Focus, which makes all visible elements selectable by an accessibility service.

These new capabilities make it possible for developers of accessibility services to create alternative navigation modes such as gesture navigation, and give users with disabilities improved control of their Android devices.

But there is no more information about how to use it. Only samples I've found are at the bottom, but those are very old and a part of the apiDemos bundle.

The question

How do I make a service that can query, focus, click, enter text, and perform other UI related operations?

Answer 1

By implementing AccessibilityService (https://developer.android.com/training/accessibility/service.html) you get access to that features.

You can either inspect or perform action on the element lastly interacted by user or inspect whole application which currently active.

Intercept user events by implementing onAccessibilityEvent(AccessibilityEvent event), here you can retrieve virtual view (representing original view) with event.getSource() and then inspect it with getClassName() or getText() or anything you find in the documentation.

Inspect whole application by calling getRootInActiveWindow() and iterate throught tree of virtaul views with getRootInActiveWindow().getChild(index).

Both getRootInActiveWindow() and event.getSource() return AccessibilityNodeInfo, on which you can invoke performAction(action) and do something like Click, Set Text, etc..

Example: Play Store

Search for 'facebook' app and open it's page on play store, once you opened the play store app.

    @Override
    public void onAccessibilityEvent(final AccessibilityEvent event) {

        AccessibilityNodeInfo rootInActiveWindow = getRootInActiveWindow();
        //Inspect app elements if ready
        if (rootInActiveWindow != null) {
            //Search bar is covered with textview which need to be clicked
            List<AccessibilityNodeInfo> searchBarIdle = rootInActiveWindow.findAccessibilityNodeInfosByViewId("com.android.vending:id/search_box_idle_text");
            if (searchBarIdle.size() > 0) {
                AccessibilityNodeInfo searchBar = searchBarIdle.get(0);
                searchBar.performAction(AccessibilityNodeInfo.ACTION_CLICK);
            }
            //Check is search bar is visible
            List<AccessibilityNodeInfo> searchBars = rootInActiveWindow.findAccessibilityNodeInfosByViewId("com.android.vending:id/search_box_text_input");
            if (searchBars.size() > 0) {
                AccessibilityNodeInfo searchBar = searchBars.get(0);
                //Check is searchbar have the required text, if not set the text
                if (searchBar.getText() == null || !searchBar.getText().toString().equalsIgnoreCase("facebook")) {
                    Bundle args = new Bundle();
                    args.putString(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE, "facebook");
                    searchBar.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, args);
                } else {
                    //There is no way to press Enter to perform search, so find corresponding suggestion and click
                    List<AccessibilityNodeInfo> searchSuggestions = rootInActiveWindow.findAccessibilityNodeInfosByViewId("com.android.vending:id/suggest_text");
                    for (AccessibilityNodeInfo suggestion : searchSuggestions) {
                        if(suggestion.getText().toString().equals("Facebook")) {
                            //We found textview, but its not clickable, so we should perform the click on the parent
                            AccessibilityNodeInfo clickableParent = suggestion.getParent();
                            clickableParent.performAction(AccessibilityNodeInfo.ACTION_CLICK);
                        }
                    }
                }
            }


        }
   }

---

EDIT: full code below:

MyAccessibilityService

public class MyAccessibilityService extends AccessibilityService {

    @Override
    public void onCreate() {
        super.onCreate();
        Log.d("MyAccessibilityService", "onCreate");
    }

    @Override
    public void onAccessibilityEvent(final AccessibilityEvent event) {
        Log.d("MyAccessibilityService", "onAccessibilityEvent");
        AccessibilityNodeInfo rootInActiveWindow = getRootInActiveWindow();
        //Inspect app elements if ready
        if (rootInActiveWindow != null) {
            //Search bar is covered with textview which need to be clicked
            List<AccessibilityNodeInfo> searchBarIdle = rootInActiveWindow.findAccessibilityNodeInfosByViewId("com.android.vending:id/search_box_idle_text");
            if (searchBarIdle.size() > 0) {
                AccessibilityNodeInfo searchBar = searchBarIdle.get(0);
                searchBar.performAction(AccessibilityNodeInfo.ACTION_CLICK);
            }
            //Check is search bar is visible
            List<AccessibilityNodeInfo> searchBars = rootInActiveWindow.findAccessibilityNodeInfosByViewId("com.android.vending:id/search_box_text_input");
            if (searchBars.size() > 0) {
                AccessibilityNodeInfo searchBar = searchBars.get(0);
                //Check is searchbar have the required text, if not set the text
                if (searchBar.getText() == null || !searchBar.getText().toString().equalsIgnoreCase("facebook")) {
                    Bundle args = new Bundle();
                    args.putString(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE, "facebook");
                    searchBar.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, args);
                } else {
                    //There is no way to press Enter to perform search, so find corresponding suggestion and click
                    List<AccessibilityNodeInfo> searchSuggestions = rootInActiveWindow.findAccessibilityNodeInfosByViewId("com.android.vending:id/suggest_text");
                    for (AccessibilityNodeInfo suggestion : searchSuggestions) {
                        if (suggestion.getText().toString().equals("Facebook")) {
                            //We found textview, but its not clickable, so we should perform the click on the parent
                            AccessibilityNodeInfo clickableParent = suggestion.getParent();
                            clickableParent.performAction(AccessibilityNodeInfo.ACTION_CLICK);
                        }
                    }
                }
            }
        }
    }

    @Override
    public void onInterrupt() {
    }
}

AndroidManifest.xml

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
          package="com.example.findfacebookapp">

    <application
        android:allowBackup="true"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:supportsRtl="true"
        android:theme="@style/AppTheme">
        <activity android:name=".MainActivity">
            <intent-filter>
                <action android:name="android.intent.action.MAIN"/>

                <category android:name="android.intent.category.LAUNCHER"/>
            </intent-filter>
        </activity>
        <service
            android:name=".MyAccessibilityService"
            android:label="@string/accessibility_service_label"
            android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
            <intent-filter>
                <action android:name="android.accessibilityservice.AccessibilityService"/>
            </intent-filter>

            <meta-data
                android:name="android.accessibilityservice"
                android:resource="@xml/accessibility_service_config"/>
        </service>
    </application>

</manifest>

res/xml/accessibility_service_config.xml

<?xml version="1.0" encoding="utf-8"?>
<accessibility-service
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:accessibilityEventTypes="typeAllMask"
    android:accessibilityFeedbackType="feedbackAllMask"
    android:accessibilityFlags="flagDefault"
    android:canRequestEnhancedWebAccessibility="true"
    android:canRetrieveWindowContent="true"
    android:description="@string/app_name"
    android:notificationTimeout="100"/>

MainActivity

public class MainActivity extends AppCompatActivity {

    public void onEnableAccClick(View view) {
        startActivityForResult(new Intent(Settings.ACTION_ACCESSIBILITY_SETTINGS), 1);
    }

}