How to simulate python key presses in Google Collab notebook?

Question

I have developed a Collab notebook in Python that intends to simulate keyboard keys' presses in order to play a game which it will monitor as a part of reinforcement learning. I have tried using several recommendations (like pyautogui, Quartz etc) but I am not even able to import these libraries.

For instance, when I installed pyautogui and Xlib and tried importing pyautogui, it gave KeyError: 'DISPLAY'. However, pyautogui works fine on my local machine when run from the terminal.

Similarly, after installing (!pip install pynput) pynput to Collab notebook, when I tried importing it, it showed DisplayNameError: Bad display name "", of which I have no clue.

I even tried the getkey module, however from getkey import getkey, keys but to no luck.

Is there any way to get this working? Also, can anybody please explain why is it that these modules work fine on my local hardware but not on Google Collab? Also, please suggest any alternative to these modules that can work fine on Collab?

Many thanks.

Answer 1

When you use such a module on your local machine, it connects with hardware to get your mouse/keyboard/monitor events.

However, Google Colab is run on a machine instance in Google Cloud. Python there cannot gain access to your keyboard/monitor. You interact with Colab through the browser only.

An alternative is to use JavaScript to get these events and pass it to Python. I make an example how to connect from JS to PY as a gist here.

https://gist.github.com/korakot/d10a43490f3da17d4915cdc1f200b180

It's very difficult to do, I warn you. You must implement passing actions back and forth between JS and PY.

I suggest you just do it on you local machine instead. You can use Local Runtime, if you like Colab.

https://research.google.com/colaboratory/local-runtimes.html