Bitcoin Forum

I would like to confirm the policy surrounding scraping content via automated means. Is this something that is permissible? The stipulation is no more than one (1) page will be accessed every one (1) second.

The rules are the same as for humans. But keep in mind:
- No one is allowed to access the site more often than once per second on average. (Somewhat higher burst accesses are OK.)
- Every post must be on-topic. Any bot response to a topic is almost certainly off-topic. Changetip's behavior of responding to user commands publicly would not be allowed, for example.
- If someone complains about an unsolicited PM you send them, then you're probably going to be banned.

Those IPs are not blocked currently. But your other abusive IPs were blocked. Just your quotefast requests (which are only part of what your crawler does) were occurring at an average frequency of 7.6 requests per second in the most recent access logging period. Your requests constituted 3.4% of all forum requests in this period. This is entirely unacceptable and of course resulted in those IPs being banned.

The maximum allowed bot request frequency is 1 request per second. Those IPs are now accessing pages at an average of 2.5 requests per second combined. If you continue exceeding the allowed request limit, we will continue banning your IPs.

LoyceV (https://bitcointalk.org/index.php?action=profile;u=459836) is scraping forum all the time. You can try to use his data for your purpose.
Few of his threads:

https://bitcointalk.org/index.php?topic=5202231.0
https://bitcointalk.org/index.php?topic=5153695.0
https://bitcointalk.org/index.php?topic=5134444.0

And this his page with bunch of scrapped data (profiles, posts).
http://loyce.club/archive/

Also, just to add to this as well, if LoyceV's data works for you, and w/ his permission, you're able to use it, that would be recommended/best.

The last thing the forum needs IMO is 20+ people scraping the forum for the same data but for different purposes

@anaconda46: can you share what you're looking for? For the right purpose, I could make my data accessible in compressed file format so you don't have to download a couple million files.

<…> As long as there are no "are you a human" tests <...>

I am doing some research.

There aren’t (yet). Cloudflare seems to handle things pretty well, giving you an error if you try to load pages at a rate faster than 1 page per second. Other than that, I have never encountered any issues to date (although I scrape a limited amount during the time span of 12 consecutive hours or so, and every now and then go for a while longer).

Depending on the content you're scraping it might be better to contact third parties that are already collecting the data. I think between them they've got most data covered. Some of these might have an API instead of scraping also.

Where can I find these APIs? What data does it cover?

You can run run this code to keep your script running 24/7 on a server:
def scrape_function():
    try:
        #8 spaces
        some_code_to_scrape
    except:
        pass
some_variable = True
while some_variable == True:
    scrape_function()

Where can I find these APIs? What data does it cover?

Be careful w/ that code btw. If using, you're going to want to put a delay on the loop to ensure you're not unintentionally DOSing the site.