Scrape Windows Desktop Application

Hi there, I have been working on automating a legacy windows desktop application where I need to extract a grid contents. The grid does not have any UI elements so we could not extract it with the native methods like Get Text. I went ahead and used Tesseract OCR and the results were not reliable at all.

In past, I automated this application and did the same with something called a scraper with UiPath. I am curious if Robocorp also has something like that? Basically my use case is to extract the line items from the grid, do some checking and then select one from them and move further.

Any help highly appreciated!

Here is some more information.

The screen which I am trying to extract data from:

Keyword I am using:

${region}=  Define Region  ${left}    ${top}    ${right}    ${bottom}
${text_}=    Read Text    ${region}   

Output:
Me Chane Des Gage Sh Sot setrecet Se con ul

FREKGHT ROA FF OWA WA om 10000 WA
FREIGHT INV 2356 fF OWA WA om 25000 WA
TARE In 20365 fF WA_WA 00 28000 WA
FREIGHT NV SS FF WAT WA oo) Tata00 WA
VENDED SERVCES O61 vRNA 009 2000 WA
FREIGHT IN 8856 vf 4 OWA 000 635000 WA
VENDED SERVCES TARFF vf ¥ oWA 00 58600 WA
FRIGHT 55 vo WA 00 10000 WA
FREGHT INV S54 vo WA 000 5950500 WA
FREIGHT IN 8856 vf YONA 000 535000 WA
TARE I e7ss7as vo WA oo 10000 WA
Inv ssrA UP vf YONA om S300 WA
FREIGHT IN 524587 vf WA om 25000 WA
WaYFAR DoNoT USE wa) Da) Dee

I have been doing the same with another tool which provides a utility to test scrape by giving it the coordinates and then provide a connector to use it in the automation. The screenshot below is the result of that utility.

Hi Himanshu and thank you for your interest in your dev tools and willingness to find a working solution for your case! (with discoveries detailed in the forum)

For Windows automation, as you maybe already tried, we recommend RPA.Windows where if you’re in luck with such UI patterns followed by the app developer, you’ll be able to obtain data with keywords like:

  • Get Text
  • Get Value
  • You can extend the library yourself and play with the obtained WindowsElement object returned by Get Element in case you find something useful in there which is not implemented/wrapped by us already.

I’m not familiar with such UiPath scrapper tool, but we do support in our VSCode extension both Image and Windows locators capture tools.

For more recording options and locators crafting, search for How to inspect header under documentation, as there are 3 ways: Windows Accessibility Insights tool, our recorder or even the Print Tree keyword.


If none of these work, then maybe you can tweak Tesseract to return you more accurate results by further pre-processing the captured screenshot before getting the text out, otherwise maybe you can chain the tool above with our Robocorp automation in order to get the text out.

Another way would be to use a 3rd-party service through RPA.DocumentAI library | Robocorp documentation and get with the help of it the fields inside the table.

Try also playing with the Desktop resolution and scaling (and even accessibility support for fonts and zoom) in such a way you’ll get bigger widgets with easier to understand text.