Scraping bots on this forums

stingers (OP)

Legendary

Offline

Activity: 1184
Merit: 1013

Scraping bots on this forums

January 18, 2018, 07:45:33 PM

I am collecting some data, a lot of which can be collected from this forums.
How many HTTP requests per minute can i send at max?
I read a post about this last year, it was allowed until then. Confirming status as for Jan 2018.

My kind little bot will do nothing more than scrape this forum for required data.

InvoKing

Legendary

Offline

Activity: 2142
Merit: 1065

✋(▀Ĺ̯ ▀-͠ )

Re: Scraping bots on this forums

January 18, 2018, 07:48:48 PM

AFAIK bots are blocked by cloudflare which was implemented lately. That's why http://www.bctalkaccountpricer.info for example stopped working... Other members can confirm this.

PSPD:law and order enforcement!
Press Section Police Department!

ibminer

Legendary

Offline

Activity: 1825
Merit: 2793

Goonies never say die.

Re: Scraping bots on this forums

January 18, 2018, 09:48:44 PM

You want to keep it at least 1 second per request, and you may want to send a friendly PM to theymos to avoid any potential issues/blacklisting. And it's better to request data while not being logged in.

| _ )| _ \ |_ _| | _ \ :•: 2024 Halvening / Halving Playing Cards - Featuring Bitcointalk Users
| _ \| _/ | | | _/ :•: Most Merit | Most Activity | Most Trusted | Most Recognized | Flag Log | Most Merited Threads | DT Change Log | Most Deleted by Mods
|__/|_| |__| |_| :•: ıllıllı Sheeple ~ The System ~ Free World ~ Don't look down (NSFW) ~ Hook ~ Brainwashed ıllıllı

stingers (OP)

Legendary

Offline

Activity: 1184
Merit: 1013

Re: Scraping bots on this forums

January 19, 2018, 03:02:58 AM

Quote from: ibminer on January 18, 2018, 09:48:44 PM

Yes, I won't be logging in. Sending requests from my google cloud so it won't even be my IP.
Thanks for the heads-up

Pages: [1]

Bitcoin Forum > Other > Meta > Scraping bots on this forums

« previous topic next topic »