There is nothing wrong with doing this. There are at least 2 big projects that scrape pretty much every aspect of the forum and make the data public (trust, posts, merit, users, etc...). And they are both maintained by trusted members.
http://loyce.club/http://bpip.org/Just respect the 1 request per second limit and the rules and you will be fine.
The rules are the same as for humans. But keep in mind:
- No one is allowed to access the site more often than once per second on average. (Somewhat higher burst accesses are OK.)
- Every post must be on-topic. Any bot response to a topic is almost certainly off-topic. Changetip's behavior of responding to user commands publicly would not be allowed, for example.
- If someone complains about an unsolicited PM you send them, then you're probably going to be banned.
Those IPs are not blocked currently. But your other abusive IPs were blocked. Just your quotefast requests (which are only part of what your crawler does) were occurring at an average frequency of 7.6 requests per second in the most recent access logging period. Your requests constituted 3.4% of all forum requests in this period. This is entirely unacceptable and of course resulted in those IPs being banned.
The maximum allowed bot request frequency is 1 request per second. Those IPs are now accessing pages at an average of 2.5 requests per second combined. If you continue exceeding the allowed request limit, we will continue banning your IPs.
