Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Android Clickable On screen text

I was curious, I seen this app the other day that allowed it to open other apps and set certain functions up for you automatically. I have came to realize that it must be using an on screen click function of some sort, but I can't seem to find any documentation for something like this. For example if we know the on screen text from the other app is "Ready", is there a way to read that text and maybe do something like:

protected void processText(String text)
{
  if (text.contains("Ready"))
      // click the ready text
}
like image 756
Jayce Avatar asked Jul 04 '17 00:07

Jayce


People also ask

How do you click on TextView?

To click on a piece of the text (not the whole TextView ), you can use Html or Linkify (both create links that open urls, though, not a callback in the app).

How to set part of the Android textview as clickable?

This example demonstrates how do I set the part of the Android textView as clickable. Step 1 − Create a new project in Android Studio, go to File ⇒ New Project and fill all required details to create a new project.

How to open a stack activity from a textview?

then use a link with custom scheme and host in the text of the TextView as (in String.xml): and "activate" the link with setMovementMethod () (in onCreate () for activities or onCreateView () for fragments): This will open the stack activity with a tap on the "stack" word.

Does onclick set the clickable attribute in Lollipop?

funnily enough, using setOnClickListener sets the clickable attribute, but using the onClick attribute apparently does not. It seems like onClick does set the clickable attribute in Android 5.0 Lollipop (API 21). Maybe they considered it a bug that this didn't happen in older versions?

Is there a way to add links to text in textview?

I tested this solution works fine. To click on a piece of the text (not the whole TextView ), you can use Html or Linkify (both create links that open urls, though, not a callback in the app). TextView textView = ... textView.setText (R.string.links); Linkify.addLinks (textView, Linkify.ALL);


Video Answer


2 Answers

I have done this using AccessibilityService. It will only work fine on API level >= 16 though.

You need to extend AccessibilityService. For instance this class will get text of USSD responses and dismiss the dialog.

// ....
public class UssdAccessibilityService extends AccessibilityService {
    public UssdAccessibilityService() {
    }

    @TargetApi(16)
    @Override
    public void onAccessibilityEvent(AccessibilityEvent event) {
        if (!"com.android.phone".equalsIgnoreCase((String)event.getPackageName())){
            // In this example we are only interested in events comming
            // from "com.android.phone" package
            event.recycle();
            return;
        }

        String className = (String)event.getClassName();
        if (className == null || (!className.contains("AlertDialog") && !className.contains("AlertDialog"))){
            // Class is not an USSD dialog
            event.recycle();
            return;
        }

        AccessibilityNodeInfo source = event.getSource();
        if (source == null) {
            // getSource() is annotated @Nullable, so we do this to be
            // safe just in case
            event.recycle();
            return;
        }

        AccessibilityNodeInfo acceptButton = null;
        String ussdText = null;

        int childCount = source.getChildCount();
        for (int i = 0; i < childCount; i++){
            AccessibilityNodeInfo current = source.getChild(i);
            if (current == null)
                continue;

            String currentText = (String)current.getText();
            if (current.isClickable()){
                // In the case of USSD dialogs, there is only one clickable.
                // May be necessary to do more robust search in other scenarios
                acceptButton = current;
                continue;
            }

            ussdText = currentText;

            current.recycle();
        }

        if (ussdText!= null) {
            if (acceptButton != null)
                acceptButton.performAction(AccessibilityNodeInfo.ACTION_CLICK);

        }

        source.recycle();
        event.recycle();
    }

    // ....
}

You must declare the accessibility service in the manifest under <application>

<service
        android:name=".UssdAccessibilityService"
        android:enabled="true"
        android:label="Read USSD codes and dismiss"
        android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
        <intent-filter>
            <action android:name="android.accessibilityservice.AccessibilityService" />
        </intent-filter>

        <meta-data
            android:name="android.accessibilityservice"
            android:resource="@xml/accessibility_service_config" />
</service>

Under res/xml create accessibility_service_config.xml

<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
    android:description="@string/accessibility_service_description"
    android:packageNames="com.android.phone,com.ats.android.activationcodebot"
    android:accessibilityEventTypes="typeWindowStateChanged|typeWindowContentChanged"
    android:accessibilityFlags="flagDefault"
    android:accessibilityFeedbackType="feedbackSpoken"
    android:notificationTimeout="100"
    android:canRetrieveWindowContent="true"
    />

Of course you have to adapt this code to your own needs.

Finally you will have to enable the accessibility service manually on Settings > Accessibility in Android (or ask the user to do it).

Read more ... Developing an Accessibility Service

like image 84
Atscub Avatar answered Oct 29 '22 18:10

Atscub


To be able to do that from another app you need to capture the screen and determine the text position using text recognition service.

when another application is active, then your application should be active to be able capture the screen. So you can only use android service which always works background.

However to capture ScreenShot for your activity you have to need a View of your activity, and which one is not present in your service so you have to make a TimerTask which will call your activity at every hour and your activity responding it to with current appear view and you can capture the ScreenShot from that.

Or If you want to take a ScreenShot of your current device screen (any application) then you have to root permission, and read framebuffer for that which will give raw data of current screen then convert it to bitmap or any picture file you can do it in your service.

Briefly, It may possible for rooted phones. Otherwise, You can't take screenshot of other apps if you are non rooted. Allowed only to take screenshot of your app.

To be able analyze the screen whether there is any "Ready" text or not. Your application should be lived/actived to be able to capture of the screen.

like image 41
ziLk Avatar answered Oct 29 '22 20:10

ziLk