Sending keypress events to a window

Build with Robocorp RPA Framework
#1

Hi,
I’d like to ask for some tips for my first RPA Framework / Robocorp Lab robot.
During the holidays, I started working on a hobby project: the robot is supposed to play a simple realtime game on Windows 10 using keyboard controls. The game runs in a single window, which is pretty typical: it has a titlebar, menu and the game graphics surface. So, just a typical windows SDL game.

I’m able to launch the game (Open Using Run Dialog) and activate the window (Switch To Application), but nothing happens when the robot is supposed to press the ‘e’ key to start the game. I’ve tried all the usual keywords like Send Keys To Input, Send Keys and Press Keys, but nothing seems to work. A few key observations:
-If I let the robot launch and activate the game and then press the key manually, it works fine.
-If I launch and activate another application like calc.exe with the same method, the keypresses are registered just fine although they’re sent by the robot.

Without reading any of the RPA Framework source code yet, I’m wondering if the keypresses might not be sent on a sufficiently low level - perhaps the application expects to receive input events and those keywords just don’t work on that level? Is there some additional lib that is usually used for low-level keyboard emulation? Or am I missing some best practice recommendation about sending keys with RPA Framework?

Thanks and happy new year,
-Jukka

#2

Hi @jukka.salovaara! Welcome to the forum :slight_smile:

The Desktop.Windows library is using pywinauto to send keys. Games might be implemented with various techs which might then be difficult to interact with this library at least.

You could try the cross-platform library Desktop, because it is using different method of sending keys to the window.

*** Settings ***
Library       RPA.Desktop

*** Tasks ***
Using my game
    Open application   game.exe
    Press Keys      w
    Press Keys      s
    Press Keys     enter

See more from the library documentation

#3

Thanks for the tips! I believe I tried every possible typing-related keyword from RPA Framework without results. However, today I decided to try PyAutoGUI, which installs nicely with pip. I got it to work by sending keydown & keyup events, like this:
pyautogui.keyDown(‘e’)
pyautogui.keyUp(‘e’)
Curiously, the press function (pyautogui.press(‘e’)) would not work, although it’s just supposed to wrap keyDown() and keyUp(). But in any case, this seems to be a relatively clean and easy way to generate some keypress events. And as one might guess, it’s important to activate the right window first when using this approach.
It’s also convenient to have this level of control when it comes to controlling a game - now I can use these functions to create a wrapper function that also accepts a ‘duration’ argument, so that a button can be help down for e.g. 2 seconds - just sleep for a while between the down & up events.
-Jukka