Android Accessibility Service - Screen updates not consistent

Question

I have an Android AccessibilityService deployed to a Samsung Note 4 running Android 5.0.1.

I'm using WhatsApp as a test bed, but this applies to any Application and is more a question around how the Accessibility Service fires off events.

The event 2048 (TYPE_WINDOW_CONTENT_CHANGED) is not consistent fired by Android. If I send messages to my WhatsApp with it in focus and the screen on 75% of the time this event is fired and at times is simply doesn't.

Is there a reason for this? Are Accessibility events not dependendable..?

Additionally, it appears that the event 4096 (TYPE_VIEW_SCROLLED) does get fired consistently when the user scrolls or when new correspondence appear in WhatsApps' chat window however, there doesn't seem to be anyway to determine what the current scroll position of the device is? AccessibilityEvent.getSource() provides access to some meta-data for the list (in this case android:id/list) however there is no information available as to the scroll position of this list or its children elements. The child list is relative to what is displayed on the screen and the boundsToScreen/Parent values are the same regardless if you're looking at the bottom of the list or the middle or top. Is there any clues to help me determine the scroll position from the AccessibilityEventNodeInfo instance I am presented with?

Lastly, when the 2048 (TYPE_WINDOW_CONTENT_CHANGED) event does fire, there are times when the new elements are not actually available from the AccessibiltyEvent.getSource() (even when you iterate up to the root element vi a while loop using getParent() and then scan down again). It appears that the event is taking a snapshot of the screen before the change has been applied to the UI. A thread.sleep does not help - as it appears the AccessibilityEventNodeInfo is more of a snapshot than live access to the UI? Any way around this?

Answer 1

Is there a reason for this? Are Accessibility events not dependendable..?

I find it more likely that you are misunderstanding when a TYPE_WINDOW_CONTENT_CHANGED event is fired, than that it is not fired or caught consistently. For example, this particular event does not fire on screen refreshes, only on new window content. An app developer may choose, instead of launching a new Activity, to re-draw content over their activity. In this case, from the users point of view the window content has changed, however, on the backend of the application all that his happened is new views have been drawn.

A thread.sleep does not help - as it appears the AccessibilityEventNodeInfo is more of a snapshot than live access to the UI? Any way around this?

This is also the explanation for why elements are missing from the event source. You are getting the event before dynamic elements are drawn. So, a new Activity is launched, and initialized with new content, however, the application probably goes into a waiting pattern for some type of network/REST resources. Before these resources come in the event is fired, then shortly thereafter new content is drawn. So, it appears to your eyes that your getting incomplete content, but what's really happening is you're getting the complete content as of the time the event was triggered. You're thread.sleep method works just fine. However, after you sleep, you can't inspect the value of event.getSource(), sleeping won't change what was passed to this function. Instead, you want to sleep, and then crawl through the entire view heirarchy to find the info you want. Sleep and then use

getRootInActiveWindow();

in place of

event.getSource();

Is there any clues to help me determine the scroll position from the AccessibilityEventNodeInfo instance I am presented with?

Yes and no. No, there is no way from an accessibility service to detect the location of the scroll bar. However, you can detect the approximate position of the scroll event by doing the following:

private float getScrollPosition(AccessibilityEvent event) {
    final AccessibilityRecordCompat record = new AccessibilityRecordCompat(event);
    final int itemCount = event.getItemCount();
    final int fromIndex = event.getFromIndex();

    // First, attempt to use (fromIndex / itemCount).
    if ((fromIndex >= 0) && (itemCount > 0)) {
        return (fromIndex / (float) itemCount);
    }

    final int scrollY = record.getScrollY();
    final int maxScrollY = record.getMaxScrollY();

    // Next, attempt to use (scrollY / maxScrollY). This will fail if the
    // getMaxScrollX() method is not available.
    if ((scrollY >= 0) && (maxScrollY > 0)) {
        return (scrollY / (float) maxScrollY);
    }

    // Finally, attempt to use (scrollY / itemCount).
    // TODO(alanv): Hack from previous versions -- is it still needed?
    if ((scrollY >= 0) && (itemCount > 0) && (scrollY <= itemCount)) {
        return (scrollY / (float) itemCount);
    }

    return 0.5f;
}

This is direct from Google's TalkBack code. You can find it in ScrollFormatter.java in the EyesFree project. It's not an ideal solution. If the event happens to come from a large layout (as happens frequently with web pages in chrome), you can end up with the same result for a large portion of scrolling. However, the APIs don't support anything more precise than this, so this is a necessary hack if you want to know you approximate scroll location. I think significant improvements could be made to this method, even with available APIs, though it would take some work.