How can I use AccessibilityService.getWindows() to obtain a traversable AccessibilityNodeInfo?

Question

I am writing an AccessibilityService for Android and, up to API level 20, I have been using the AccessibilityEvent.getSource() method to obtain a traversable AccessibilityNodeInfo when onAccessibilityEvent(AccessibilityEvent event) is triggered. Although the resulting AccessibilityNodeInfo does not always reflect the content of the screen, it is still possible to work with it.

Starting on API level 21, the new AccessibilityService.getWindows() is supposed to be not only able to better represent the view hierarchy (i.e., Z-ordering is respected), but it is also supposed to be able to expose a node that includes all views in the current input method (IME). I would like to take advantage of this but I have not been able to do so and I do not know what exactly I am doing wrong. Incidentally, I have been unable to find any more detailed information on how to do this, other than the very minimal java docs.

I have already done the following:

• Configured service to retrieve window content (android:canRetrieveWindowContent="true")
• Added flagRetrieveInteractiveWindows to service flags

My code is as follows:

@Override
public void onAccessibilityEvent(AccessibilityEvent event) {
            ArrayList<AccessibilityNodeInfo> nodes = getNodesFromWindows();
                switch (event_type) {
                case AccessibilityEvent.TYPE_VIEW_ACCESSIBILITY_FOCUSED:
                case AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED:
                case AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED:
                //case AccessibilityEvent.TYPE_VIEW_TEXT_CHANGED:
                case AccessibilityEvent.TYPE_VIEW_FOCUSED:
                case AccessibilityEvent.TYPE_VIEW_SELECTED:
                case AccessibilityEvent.TYPE_VIEW_SCROLLED:
                //case AccessibilityEvent.TYPE_VIEW_CLICKED:
                    updateTargetLeafs(nodes);
                }
}

where getNodesFromWindows() does the following:

private ArrayList<AccessibilityNodeInfo> getNodesFromWindows() {
    List<AccessibilityWindowInfo> windows = getWindows();
    ArrayList<AccessibilityNodeInfo> nodes =
            new ArrayList<AccessibilityNodeInfo>();
    if (windows.size() > 0) {
        for (AccessibilityWindowInfo window : windows) {
            nodes.add(window.getRoot());
        }
    }
    return nodes;
}

after this, updateTargetLeafs() collects all clickable, enabled and visible nodes into a separate AccessibilityNodeInfo ArrayList so I can index and access them at will (see below). When using AccessibilityEvent.getSource() on API Level 20 and below, the size of this array is always close to the number of views on the screen, but when I use AccessibilityService.getWindows() the size is almost always 1 (sometimes 0), and the bounds of the only AccessibilityNodeInfo in the list are always outside of the screen.

EDIT: Add the code for iterating through all nodes children (where mNodes is the output of getNodesFromWindows()):

        ...
        ArrayList<AccessibilityNodeInfo> theseleafs =
                    new ArrayList<AccessibilityNodeInfo>();
        AccessibilityNodeInfo thisnode;
        Queue<AccessibilityNodeInfo> q =
                new LinkedList<AccessibilityNodeInfo>();
        for (AccessibilityNodeInfo n : mNodes) {
            q.add(n);
        }
        while (!q.isEmpty()) {
            thisnode = q.poll();
            if (shouldIncludeNode(thisnode)) {
                //Add only if it fulfills all requirements!
                theseleafs.add(thisnode);
            }
            for (i=0; i<thisnode.getChildCount(); ++i) {
                AccessibilityNodeInfo n = thisnode.getChild(i);
                if (n != null) q.add(n); // Add only if not null!
            }
        };
        LogD("MyTag", theseleafs.size() + " leafs in this node!");
        ...

Odd, I know, but what am I doing wrong?

Answer 1

You can get access of windows content using getChild() method. In your onAccessibilityEvent(AccessibilityEvent event) you can do like below.

@Override
    public void onAccessibilityEvent(AccessibilityEvent event) {
          AccessibilityNodeInfo mSource = event.getSource();
          int child = mSource.getChildCount();
          // iterate through all child of parent view
          for (int i=0; i<child; i++){
            AccessibilityNodeInfo childNodeView = mParent.getChild(i);
            // Do something with this window content
          }
}