Cannot find element with RPA.Desktop

weiwei.cai3 · February 18, 2021, 7:57am

Hi, experts,

I’m new with RPA. I’m trying write my own automation test case with an outlook plugin. The process should be:

Launch Outlook.
Click plugin icon in Outlook Ribbon to log in. The text for the icon is “Log In”.

Below is my python script:
import time
from RPA.Desktop import Desktop
desktop = Desktop()

def lanuch():
    desktop.open_application("my application path and executive file here")
    time.sleep(10)
    desktop.click('ocr:"Log In"')
    time.sleep(10)

But when I run with robotframework, it shows:

TimeoutException: No matches found for: ocr:“Log In”

Troubleshooted:
1.Verifyed the application window was opened and the “Log In” icon also displayed on windows.
2. Tried “desktop.wait_for_element(‘ocr:“Log In”’)”, also cannot find this icon.

Ask for help:
Are there any steps I’m missing here? How to implement the python script find element via control’s text?

Thanks very much!

weiwei.cai3 · February 22, 2021, 1:11am

I still stuck on click through control text, can someone help to check this? Thanks in advanced.

jani · February 22, 2021, 8:13am

Hi, @weiwei.cai3! Can you share your conda.yaml configuration file? I would like to see what dependencies you have there.

I’m thinking if any of these might affect the OCR:

Advanced computer-vision features such as image template matching and OCR require an additional library called rpaframework-recognition.

The dependency can be either added separately or through additional extras with rpaframework[cv]. If installing recognition through pip instead of conda, the OCR feature also requires tesseract.

I don’t have access to my Windows computer and Outlook, so can not test this myself right now.

weiwei.cai3 · February 22, 2021, 8:29am

Thanks for your reply.
Where can I find the conda.yaml configuration file? I create my sample test case with Robot Framework and edit the python file and .robot file with notepad. There is no conda.yaml configuration file in my test suit folder.
rpaframework-recognition and tesseract libraries are installed through pip on my machine.

In addition, I tried use ‘Click ocr:“Archive”’ to click one of my inbox folder, it works fine. But the ocr:“Log In” for my outlook plugin doesn’t work.

Best Regards,
Weiwei

weiwei.cai3 · February 22, 2021, 8:29am

Below is the screenshot of my plugin location in Outlook desktop application.

Teppo · February 22, 2021, 8:35am

Hi

OCR locators are unreliable especially with buttons. In this case the OCR might not work, because text if divided into two lines. Could you use Windows locators or images instead?

There is also a dedicated library for handling Outlook tasks in Windows: RPA.Outlook.Application. I don’t know if it would help you.

mika · February 22, 2021, 8:43am

Just quick comment, could you try to put newline into string you are finding ocr:"Log\nIn" because texts are in separate lines ?

weiwei.cai3 · February 22, 2021, 8:44am

Hi Teppo,

Thanks for your suggestion, I checked the RPA.Outlook.Application library, I don’t found click button related keywords.

weiwei.cai3 · February 22, 2021, 8:45am

Hi mika,

I tried ocr:“Log\nIn”, but still no luck.
Thanks.

Teppo · February 22, 2021, 9:01am

I don’t think OCR can work with this. You will have to fallback to windows locators or image locators.

mika · February 22, 2021, 11:41am

@weiwei.cai3
In addition to what @Teppo mentioned. Check if application has shortcut for the action you want perform. They are usually the easiest way to navigate places in the app.

https://support.microsoft.com/en-us/topic/keyboard-shortcuts-for-outlook-3cdeb221-7ae5-4c1d-8c1d-9e63216c1efd?ui=en-us&rs=en-us&ad=us

mika · February 22, 2021, 11:45am

As that “Log In” is in the Home “ribbon”. Maybe pressing keys ALT+H and then L might work ?

mika · February 22, 2021, 11:48am

Actually… press that ALT+H while in the Outlook application and it should show you the shortcut key what to press next.

weiwei.cai3 · February 23, 2021, 12:52am

Hi mika,

Thanks for your suggestion. I can use shortcut key to navigate to the UI. Since I’m new for RPA.Desktop, I’m not sure if I use the correct syntax or if I miss something. But now I can confirm the syntax and usage is correct.

Thanks for all guys to help me check this issue. And I have reported this issue on github, before this issue resolved, I will use screenshot or shortcut to navigate to UI: ocr:"Log In" doesn't work if the Log in text in two lines · Issue #141 · robocorp/rpaframework · GitHub

Thanks again.