How to listen to taps and get views using Accessibility in Android?

Question

I want to implement an Accessibility service in my Android app which can do the following things:

=>Get the onscreen taps/clicks across any apps.

=>Get the view which was tapped/clicked.

Initially I thought it would not be possible to do so because of security reasons but while doing some research I came across an app (Native clipboard) which could do following things:

=>Detect taps on EditTexts from any app

=>Add value (string) to those EditTexts.

I also saw Google's talkback which speaks whatever you tap on. For it to speak, it needs to access the view (to get the text) across apps.

These apps obviously makes use of 'Accessibility services' to do so, but I would like to know how can I implement that?

I mostly find tutorials or stuffs for things I need to achieve but I am struggling finding some for implementing Accessibility service for my app. I did visit the Android official documentation which is too technical for a newbie like me. (I initially prefer to learn from Youtube, SO and tutorial websites). It will also be great if you can pin point me to some other tutorials which covers these things.

Answer 1

Accessibility services are pretty poorly documented, but I have created some accessibility service boilerplate code, that sets up a starter project and logs the base callbacks. Here is a bit of code that I think you care about given your specific questions. The scaffolding, project set up and such I leave for the repo.

Below is the onAccessibilityEvent callback. This is where you will listen for different types of a events, and the most convenient place to grab onto screen content for most scenarios. Though, as an accessibility service you also don't have to wait for events. You could just as easily spawn an AsynTask and grab on to it on an interval of some kind.

public void onAccessibilityEvent(AccessibilityEvent event) {
    CLog.d(event.toString());

    switch (event.getEventType()) {
        //On Gesture events print out the entire view heirarchy!

        case AccessibilityEvent.TYPE_GESTURE_DETECTION_START:
            CLog.d(A11yNodeInfo.wrap(getRootInActiveWindow()).toViewHeirarchy());

        case AccessibilityEvent.TYPE_VIEW_CLICKED:
            CLog.d(event.getSource().toString());

        default: {
            //The event has different types, for you, you want to look for "action clicked"
            if (event.getSource() != null) {
                CLog.d(A11yNodeInfo.wrap(event.getSource()).toViewHeirarchy());
            }
        }
    }
}

I will point out one bit of configuration for this, because it is super important. Accessibility services are best configured through an XML file connected to your service through the Manifest file. The contents of this file are:

<?xml version="1.0" encoding="utf-8"?>
<accessibility-service 
xmlns:android="http://schemas.android.com/apk/res/android"
android:description="@string/accessibility_service_description"
android:accessibilityEventTypes="typeAllMask"
android:accessibilityFlags="flagReportViewIds"
android:canRetrieveWindowContent="true"
android:canRequestTouchExplorationMode="true"
android:accessibilityFeedbackType="feedbackSpoken"
android:notificationTimeout="100"
android:settingsActivity="com.moba11y.basicaccessibilityservice.SettingsActivity"
/>

For you the important bits are canRetrieveWindowContent="true" and accessibilityEventTypes="typeAllMask". A design bit I like, is ensuring that your grabbing onto the minimal set of event types you want. Different Accessibility Events report broadly different results. For example, many events return "null" from getSource(). This forces you to add a lot of filters for this, or risk null pointer exceptions. It's quite annoying.

The last bit you need is Accessibility Actions. This is what allows you to simulate clicks, long clicks AND add text to an editable text view. Below is code that allows you to do this.

public void onAccessibilityEvent(AccessibilityEvent event) {
    AccessibilityNodeInfo source = event.getSource();
    if (source != null & event.getClassName().equals("android.widget.EditText")) {
        Bundle arguments = new Bundle();
        arguments.putCharSequence(
            AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE,"some value");
        source.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, arguments);
    }
}

https://github.com/chriscm2006/Android-Accessibility-Service-Boilerplate