Website robot detection

Build with Robocorp
#1

I’m trying to scrape some data from a public website. I could load the website without any problem. But when I tried to click on the “Next Page” button, the website detects a robot and goes into a captcha challenge.

I tried loading next page URL directly with the Go To command. Same issue.

But the interesting thing is if I load the second page with the Open Available Browser, then it works. It’s not detected as a robot. I could scrape one page, close and then open the next page. But I wonder if there is a better way to avoid being detected as a robot?

#2

The robot detection is in place probably for a reason, so tread carefully. Check the terms of service of the target site. :slight_smile:

Anyway, one option might be to try and slow the robot down a bit. It could be that the detection is based on how fast the user interacts with the page (too fast = robot).

#3

Do they have API? API might be better suited to robot use. So before scraping, try to see if there is better ways to do it.

And as Jani said, there is bot detection for reason.

Other thing could be to just contact that site owners and ask how to do what you want but with robot.

#4

Thanks for the reply. Slowing down didn’t work. They probably have more sophisticated robot detection. But somehow, opening the the browser and reading all the data in it didn’t cause any problem. Just moving from one page to another. That’s strange. I solve it buy closing the browser and reopening it to another page. That worked.

1 Like