|
Title: Scraping bots on this forums Post by: stingers on January 18, 2018, 07:45:33 PM I am collecting some data, a lot of which can be collected from this forums.
How many HTTP requests per minute can i send at max? I read a post about this last year, it was allowed until then. Confirming status as for Jan 2018. My kind little bot will do nothing more than scrape this forum for required data. Title: Re: Scraping bots on this forums Post by: InvoKing on January 18, 2018, 07:48:48 PM AFAIK bots are blocked by cloudflare which was implemented lately. That's why http://www.bctalkaccountpricer.info for example stopped working... Other members can confirm this.
Title: Re: Scraping bots on this forums Post by: ibminer on January 18, 2018, 09:48:44 PM You want to keep it at least 1 second per request, and you may want to send a friendly PM to theymos to avoid any potential issues/blacklisting. And it's better to request data while not being logged in.
Title: Re: Scraping bots on this forums Post by: stingers on January 19, 2018, 03:02:58 AM You want to keep it at least 1 second per request, and you may want to send a friendly PM to theymos to avoid any potential issues/blacklisting. And it's better to request data while not being logged in. Yes, I won't be logging in. Sending requests from my google cloud so it won't even be my IP. Thanks for the heads-up |