how can I extract text contents from GUI apps in linux?

Question

I want to extract text contents from GUI apps,here are 2 examples::

example 1:

Suppose I opened firefox, and input url : www.google.com

how can I extract the string "www.google.com" from firefox using my own app ?

example 2:

open calculator(using gcalctool),then input 1+1

How can I extract the string "1+1" of calculator from my own program?

in brief ,what I want is to find out whether there is a way to extract the text contents from any widget of an GUI application

Thanks

Answer 1

I don't think there's a generic way to do this, at least not a very elegant one.

Some inelegant ideas:

You might be able to modify the X window system or even some toolkit framework to extract what is being displayed in specific window elements as text.

You could take a screenshot and use an OCR library to convert the pixels back into text for the interesting areas.

You could recompile the apps of interest to add some kind of mechanism for asking them questions.

You could use something like xtest to inject events highlighting the region of interest and copying it to the clipboard.

Answer 2

I believe firefox and gcalctool are for examples only and you just want to know in general how to pass output of one application to other application.

There are many ways to do that on Linux, like:

piping

application1 | application2

btw here is the Firefox command line manual if you want to start firefox on Ubuntu with a URL. eg:

firefox "$url"

where $url is a variable whose value can be www.mozilla.org

Answer 3

That sounds difficult. Supposing you're running X11, you can very easily grab a window picture ( see "man xwd"); however there is no easy way to get to the text unless it's selected and therefore copied to the clipboard.

Alternatively, if you only want to capture user input, this is quite easy to do, too, by activation the X11 record extension: put this in your /etc/X11/xorg.conf:

Section "Module"
        Load  "record"
        #Load other modules you need ...
 EndSection

though it may prove difficult to use too, see example code for Xorg/X11 record extension fails