Get Elements returns coordinates of the element with negative values

Hey,

I want to click element by coordinates and for that I want every time to take the exact coordinates using Get Elements. Now this is returning me the coordinates for bottom and top with negative values and the click can not happen. Any idea why this is happening? I am trying that on Desktop application.

Thank you,

Lora Lambreva

Could you maybe provide more information about error? Maybe library used, code snip that causes issue or perhaps log?

1 Like

Here it is the screenshot with the information that I got from RPA.Windows.Get Element and Accessibility Insights for Windows. In order to click the element I need the coordinates like they are displayed in Accessibility Insights for Windows (Black screen in the attached file). My thoughts are that the click is not working because of the negative values which are interpreted by RPA as 0 for ycenter.

Thank you,
Lora

Forgot to mentions that it would probably good to know what version of rpaframework is used.

As @linkraivo said, maybe is time upgrading to the latest rpaframework==17.1.1 just to be sure this isn’t a past bug which might be solved already. (here’s a conda.yaml example also supporting OCR with image-based locators, just strip the [cv] and tesseract deps since you don’t need them)

Meanwhile I created an Issue for it in order to debug it closely and have a potential fix if the problem is still present. Can you let us know what app (if public) are you trying to automate so we can replicate? Also, can you try with a different display resolution (fitting your entire screen) and a scaling of 100% just to be sure that’s not a display problem?

I couldn’t get negative coordinates with a simple Calculator test. Can you try running that task (Get Elements Coords) as well to see if you get elements with negative coordinates this time on your side? Thank you!

Do you have more than one display? And if so, how are those positioned in relation to each other? If there are more than one display, things might get complicated.

Other thing is that those screenshots are showing different values themselves. Are you sure that there are not more than one element matching to your regexp matcher? Is regexp matcher actually needed, or would name:HWHG locator do?

I was not able to reproduce it on different application I guess this is related to the application that I am using but it is not public. Here it is a screenshot of it. I have a guessing that is something connected to multiple frames used.

seems like origin for outer-window is top-left-corner (which is standard) and origin for inner window is bottom-left-corner (at the inner-side of outer-window). reproduction with an other application was not possible. a prototype with outer window and inner pane led to correct coordinates from RPA. So our guess is, that the problem occurs in the special case of outer window and inner window. Using the Keywords

Control Window             regex:SDS...*

Control Child Window      regex:Auftrag...*

did not help to get positive coordinates.
Please note (other coordinates in the example are not matching, because the window was moved after execution). This issue is not related to multiple displays, negative coordinates are also returned, when using a single display. I am running the recent version rpaframework==17.1.1

Just to have more info, can you provide to us even more display settings details, like?

  1. DPI size
  2. Scaling percentage
  3. Resolution
  4. Any other particular settings not usually met on other systems

But like you said, it seems app specific and how these coordinates get relative to either the left-top corner or right-bottom one, thing which we couldn’t replicate so far.

As an experiment, try controlling the inner window with something like: Control Window regex:SDS...* > regex:Auftrag...* which automatically uses as parent for the 2nd locator the 1st one and see if anything changes.
If nothing changes, as a workaround, try avoiding absolute coordinates entirely and rely on clicking with offsets, like: Click <locator> offset:<x>,<y> where these two coords are relative to the center of the identified element. And if this is not an option, then correcting negative coordinates (when detected) based on the window size & position relative to the right-bottom corner will do the trick (as you discovered already – ofc, there’s more work involved, but I hope you’ll find this tradeoff reasonable).

→ Technical comment on the intrinsics of the library when it comes to relative coordinates based clicking.

The given suggestions are not working for us but we manage to find a workaround for sorting with in-build functionality of application under test.

Thank you for your time,
Lora

1 Like