How to read message from Messages app

I am doing a form automation and the process is like this:

  1. Input a phone number in a web form, this will bring you to a new form input where you need a code that is sent via SMS
  2. Receive a code on SMS
  3. Use the code received in SMS to confirm step 1

I have done number 1, i.e automated the input of phone number, I have managed to open the Messages app on my Macbook. I try to do a image based UI.locator to get the code from the SMS. I have installed the rpaframework-recognition package. But I do not think I have done the right thing in order to add it to the environment in JupyterLabs/Robocorp Labs. I get this: ValueError: Keyword requires OCR features
Where do I find info on how to add this? Is it inside the RPA package?

Or is it some other strategy that will help me get this code from the Messages app?

EDIT: this is the Messages app in MacBook:
And the code I need is (in the last message) 330cl
Is it better to do a OCR (or other get text) of the blue part, where you know the position of the last message?

Hi @tm1

did you do it like descriped here (under installation):

You can also use tesseract instead. Here is an example from the portal:

Sorry, I do not know the Messages app and I don’t have a Mac. I assume in Windows you could use other ways to get this code if it is not an image but I do not know how it is with a Mac…maybe somebody else can help you here.

1 Like

Thank you.
How to I add the rpaframework-recognition so I can use it in Robocorp Labs?
I recon I have to import something in the Settings?
Library “the thing that make rpaframework-recognition” available in Keyword contexts

Hi @tm1, I just downloaded the example robot linked by @RoboHeart and tested it on Mac. It seems to work fine. Have you double checked that your conda.yaml contains tesseract and rpa framework with cv as in the example conda.yaml

  # Define conda channels here.
  - conda-forge

  - python=3.7.5
  - tesseract=4.1.1
  - pip=20.1
  - pip:
      - rpaframework[cv]==9.5.0

After that it should be enough the import RPA.Desktop library to use OCR:

*** Settings ***
Library    RPA.Desktop

*** Keywords ***
Test OCR
    Click    ocr:New
1 Like

Thank you. It brought me a bit forward.
What I try to do now is to use a image from part of the Messages app where the new code will appear.nte_code

then move a bit to the right, and then try to read the text/code.
This give me: ElementNotFound: No matches found for: image:nte_code.png
Any suggestions?

*** Keywords ***
Open the Message app
Run Process open -a Messages
Sleep 5 seconds
${label_region}= Find Element image:nte_code.png
${value_region}= Move Region ${label_region} 100 0
${text}= Read Text ${value_region}

Your code looks good, couple of ideas:

  1. How did you create the nte_code.png? Did you do it in Lab’s UI Locators -tool and did you validate it there?
  2. When you run the code, can you visually verify that the window is unobstructed and area in the image is visible?
1 Like

So what I have done now is getting the code from the Message app. I used an image from part of the Messages app where the code will arrive. I then can read part of the message with the code in it.

I created a Python function where I clean the string with the code. And get a clean code as return.
Now I am struggling with variables(!).
I define a variable in the Variabels section:
*** Variables ***
${CLEAN_CODE} blank

I try to set the variable with the value return from the Python function. I get the correct string back from the function, the problem is to set the variable.
But the way I do it will not work…

*** Keywords ***
Open the Message app
Run Process open -a Messages
Sleep 5 seconds
${label_region}= Find Element image:nte_code.png
${value_region}= Move Region ${label_region} 100 0
${code}= Read Text ${value_region}
${CLEAN_CODE}= Get Code ${code}

The use of variables seems correct - just verify that you have 2+ spaces between variable and keyword e.g.

${CLEAN_CODE}=    Get Code    ${code}

…and you don’t need to define the variable ${CLEAN_CODE} in Variables -section.

1 Like

So the variable context is shared between different keywords?

I have one Keyword for getting the code, another for input into the web form

Normally variable scope is only the current task/keyword and values are passed as keyword arguments and return values. However, you can make variable global with Set Global Variable

1 Like

…but you are correct that variables created in Variables -section are automatically placed to Suite -scope, so they are available in every keyword. It is an another way to make Suite -level variables (in addition to using Set Suite Variable)

I split the process into more keywords than necessary.

The variables issue:
The variables in the Variables section, am I allowed to update them? Or is it really constants?

Tried to read about how to return a variable from a keyword, and how to pass it in.

And the easy solution: gather all the things that have to do with the code inside one keyword. And this work perfect.
The process is now automated
from going to the web site,
consent the cookie by clicking a button
click the button in order to go to the form
input the phone number in the textbox
check the checkbox for privacy,
open the Messages app
read the last message where the code is inside
do a Python function where the code is extracted from the message string and returned
take the code string and input in the textbox on the web site
click the button for sending the code
Thanks for all the help! It is a bit quirky with all these spaces to be aware of but a lot of fun doing progress.

Now I want to learn on how to schedule this prosess :slight_smile:

1 Like

Great progress!

Now I want to learn on how to schedule this prosess :slight_smile:

Welcome to :grinning_face_with_smiling_eyes:

  1. Login/register to cloud
  2. Link Lab to cloud
  3. Upload robot from lab to cloud
  4. Create process with the uploaded robot and proper schedule
1 Like

Sound great.
And work as expected from Cloud. I can start the process from Cloud and run it on my local environment.
The only thing I miss is to be able to experiment with the scheduling features from the free tier.