Trigger robot using the Raspberrypi Pico

This approach of using a microcontroller for automation is a proof of concept. I had lots of fun developing. In this tutorial, I show you how the Raspberrypi Pico can be used as an USB trigger to run a Robocorp robot or any other automation.

My aim was to use the Raspberrypi Pico as my daily productivity booster, wherein the script would open up and log me into all applications (about 8) and accounts I use when I insert the Pico into the USB slot. Typically, this approach of automating my daily chore saves me about 3 minutes daily and a whole lot of clicks and navigation. Wake up, switch the pc on, let the Pico prepare my day!

Before we begin, here are the applications and software needed to move forward:

  1. Thonny :white_check_mark:
  2. Rcc :white_check_mark:

Here is an official walkthrough to setup the Raspberrypi Pico
As you go through the walkthrough, instead of using MicroPython firmware, download the CircuitPython firmware from Pico Download and continue with the setup process. In my case, I use CircuitPython 7.2.5

Note : Since the MicroPython firmware does not support Human Interface Device (HID), we need to flash the CircuitPython firmware to the Pico. Once this is flashed into your Pico, open up Thonny and choose → Run → Select interpreter and choose CircuitPython (generic).

After selecting CircuitPython (generic) as the interpreter, you will see the following shell output.

Good! However, when we check for the HID libraries with the help("modules") command, you will not find the required HID drivers in the default version of CircuitPython firmware. CircuitPython includes all MicroPython libraries of many custom libraries. However, we need to add the HID driver to the lib folder of the Pico ourselves.

Thankfully, Adafruit already has done the heavy-lifting by releasing a HID driver.
Releases · adafruit/Adafruit_CircuitPython_HID · GitHub.

Since we downloded CircuitPython 7.2.5 during our initial setup, lets download the corresponding version of adafruit_hid driver.

Download the zip fileUnzip the fileCopy the file in the lib folder

Copy adafruit_hid to the Pico (It will be visible as a drive in your PC) and done :white_check_mark:

Recap : We now have installed Thonny, CircuitPython and HID library from Adafruit.

After the above setps, your Raspberrypi Pico can send keys to your computer, just like any other HID (mouse, keyboard, joystick). Great, but wait, keyboards come in various layouts and languages, how do we tell Pico which keyboard language and layout to choose?

That is where another library comes in. Cheers to Neradoc for sharing these. Circuitpython_Keyboard_Layouts/PICODUCKY.md at main · Neradoc/Circuitpython_Keyboard_Layouts · GitHub

Similar to the adafruit_hid driver, Download Circuitpython_Keyboard_Layouts zip fileUnzipNavigate to the lib folderCopy all or specific languages and keyboard layoutsPaste them in the lib folder of your Pico.

To check if all the drivers and libraries are installed, we run the following command in the Thonny shell (ensure you stop the Pico once so that it reboots). You do this by clicking on the Stop button once in Thonny.

import usb_hid
from adafruit_hid.keyboard import Keyboard
from adafruit_hid.keycode import Keycode
import time
from keyboard_layout_win_da import KeyboardLayout
from keycode_win_da import Keycode
kbd = Keyboard(usb_hid.devices)
layout = KeyboardLayout(kbd)
help(Keycode)

The output shows us all the mapped keycodes available for the Danish/Norwegian language and keyboard layout. In this tutorial, I will use the Danish :denmark: keyboard layout, which is the closest I can get to a Norwegian (NOB) layout of windows :norway:


Let the scripting begin!

import usb_hid
import time
from adafruit_hid.keyboard import Keyboard
from adafruit_hid.keycode import Keycode
from keyboard_layout_win_da import KeyboardLayout
from keycode_win_da import Keycode

# Keyboardlayouts : https://github.com/Neradoc/Circuitpython_Keyboard_Layouts/blob/main/PICODUCKY.md
# Set up a keyboard device and layout
kbd = Keyboard(usb_hid.devices)
layout = KeyboardLayout(kbd)

# Lets populate program dictionary to run
programs = {"RobocorpRobot": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe rcc run --space assist --robot C:\\Users\\%username%\\example-robotsparebin-starter-main\\robot.yaml" }

# Some keys need to be specified within a keycode dict. I do this to make the next steps easier. 
keycodeDict = {"\\":46,
               "-": 56,
               "ø": 51,
               "æ": 52,
               "å": 47,
               ":": Keycode.SEMICOLON,
               ".": Keycode.PERIOD,
               " ": Keycode.SPACEBAR}

The first helper function maps keycodes to letters and numbers.

def letter_to_int(index):
    """
    Returns the letter from the given index
    """
    alphabet = list('    abcdefghijklmnopqrstuvwxyz1234567890')
    return alphabet[index]

Why does the alphabet list have empty spaces in the first 4 items? That is because the keycode we want to use start from an index 4. Key A is found at keycode number 4. When in doubt, help(Keycode) command and Danish - Virtual Keys - Keyboard Layout Info can save you a lot of time.

In a for loop, we add all required alphabets and digits to the keycodeDict dictionary.

for i in range(4,40):
    keycodeDict[letter_to_int(i)] = i  # helper to map keycodeDict for all items in alphabet
    

The main helper function, which sends keys to the host machine.

def WriteText(text):
    """
    Send keys depending on the text input. 
    Special keys need SHIFT, rest of the keycodes found in keycodeDict
    """
    for i in text:
        if i==":":
           kbd.send(Keycode.SHIFT, 55)
        elif i=="(":
           kbd.send(Keycode.SHIFT, 37)
        elif i==")":
           kbd.send(Keycode.SHIFT, 38)
        elif i=="/":
            kbd.send(Keycode.SHIFT, 36)
        elif i=="_":
            kbd.send(Keycode.SHIFT, 56)
        elif i=="%":
            kbd.send(Keycode.SHIFT, 34)
        else:
            key = keycodeDict[i.lower()]
            kbd.send(key)

The code which performs the automation

def RunCommand(CommandText):
    """
    Run the automation when the trigger is active. 
    """
    kbd.send(Keycode.GUI, Keycode.R)  # Send windows+r
    time.sleep(0.2)  # wait for RUN window
   
    WriteText(CommandText)  # write the command 
    time.sleep(2)  # wait for type into to finish
    kbd.send(Keycode.TAB)  # Send Tab
    kbd.send(Keycode.RETURN)  # Send enter

Remember we are running this on a microcontroller. It does not understand that we need to run the above code only once when it is connected to the host machine. There is no pre and post event checks as the controller / firmware in its current form has no way to parse the GUI applications on screen. All we can exploit is static delays in-between sending keys. This is also the biggest drawback of this whole approach.

We need a while loop to ensure that the automation starts only when the microcontroller is activated (using a boolean check) and once done, it stops the loop i.e., run only once.

started = False
while started==False:
    time.sleep(10)  # I have some other automation. This delay can be lower. 
    RunCommand(programs["RobocorpRobot"])
    started = True

When we insert the Pico in the USB slot, the 10 second countdown kicks in and the chosen automation / Robocorp robot runs.

2 Likes

This is so cool! :exploding_head: Super-detailed and interesting tutorial - thanks for sharing this, @Jeevith!

1 Like

@jani Thank you. Interfacing hardware to automation was fun.

If someone has a better way to support different keyboard layouts, this approach can be easily ported. For now, let say you want to use a Finnish keyboard language and layout the keycodeDict will need to be changed. The key codes change in each language and layout :frowning:

Great article @Jeevith Would be good also as a Medium article for example.

About keycodes, can’t you get correct code for any keyboard layout with ord("%") ?

1 Like

My suggestion probably does not work in your use case. I did not look into details :slight_smile:

Hi @mika,

That was new to me ord("%")

So, I did some tests:

  1. Looks like the % has same keys which makes sense
    image

  2. However the keycode does something weird when I try the special characters å æ ø
    image

    Keycode 47 returns å
    While Keycode returned from ord(“å”) is 229 and returns nothing back when used with keyboard.send
    image

For completeness
image

I myself have not dived fully into the rabbithole of Keyboard layouts and how scancodes, keycodes and virtualkeys function. Thank you for a nice find though.