Querying bitcointalk from script

[shahzadafzal]

I tried to google but could not find an answer.

If I'm querying bitcointalk from a script what's the minimum time I should leave between each call? To avoid any IP ban or tagged as DDoS?

I have been doing this for quite some time and I always keep e.g. 2~2.5 second delay between each request. But this seems very slow now can I decrease it to 1 sec or less?

[LoyceV]

can I decrease it to 1 sec

Yes.

or less?

No. You'll get rate limited if you do that (although a short burst period is allowed).

Depending on what data you're looking for, someone may have it already. If it's something I have, a data dump can save you months of scraping.

[TryNinja]

From theymos himself:

Yes, but:

 - All of the posting rules still apply to bots.
 - You are not allowed to send more than one HTTP request to the forum per second on average.

I also have some data if that helps: https://docs.ninjastic.space

And can also give you a data dump on request ().

[BitcoinGirl.Club]

[...]

Shit Bot was supper fast 🤣. I was you in mind while I was writing the response to OP suggesting to keep it longer than 1 second. Long ago I read somewhere that a safe zone is to have 1 second delay between queries.

Edit: Now TryNinja too LOL

[PX-Z]

I bet @Tryninja, @mprep and @LoyceV and other forum tech geeks can answer you about this, mentioning them might be a big help.

After some research, although i don't know exactly about the script you're trying to make, but here's an answer from @Tryninja for the telegram bot he made for probably scrapping data from bitcointalk, i don't know actually.

-What's the frequency of the scrapping? giammangiato's is 5 minutes for merits and 1 for quotes; and it sometimes takes a lot longer to do so.

Mentions/quotes every 5 seconds.
Merits every 25 seconds with a delay per each user. edit: increased to 30 seconds + 5 seconds  per user.

I tested and it's quite faster than the one from giammangiato (noticed 1 min VS 30 min merit notification with few users, besides the complete merit information). Not that it matters a lot (no competition here), but it's still nice.

Edit: Wew, lol. Seems like it take a lot for me to type this reply, while all the mentioned guys already made a reply.

[shahzadafzal]

Depending on what data you're looking for, someone may have it already. If it's something I have, a data dump can save you months of scraping.

Yes of course LoyceV your dumps are the first ones I will always refer to and in fact I'm using one right now from here related to usernames https://bitcointalk.org/index.php?topic=5202231.0

Nothing complex just small query for this a small quiz by cygan https://bitcointalk.org/index.php?topic=5420570.msg61422530#msg61422530

I also have some data if that helps: https://docs.ninjastic.space

And can also give you a data dump on request ().

Yes I have seen this too and thank you for that.

Actually you, LoyceV and DdmrDdmr too and I know you guys have very useful data dumps will always be appreciated.


For me I don't do much but for my own searches mostly, I also have data of some the users e.g Satoshi, Hal Finney, theymos and of course some of my favorite posters on WO (i don't want to name..) and I use it to query and refer to old posts for a quick search.

For example this or when recently LoyceV said that he did not use word vibrant so I thought of confirming myself and at first I found he did use it but... later on manual inspection I found that it is a quoted word but quoted in a wrong way, wrong in a sense that bots will not know, and here too, that's where the AI fails

Shit Bot was supper fast 🤣. I was you in mind while I was writing the response to OP suggesting to keep it longer than 1 second. Long ago I read somewhere that a safe zone is to have 1 second delay between queries.

You are clearly confusing God with Bots.

[PrimeNumber7]

[...]

Shit Bot was supper fast 🤣. I was you in mind while I was writing the response to OP suggesting to keep it longer than 1 second. Long ago I read somewhere that a safe zone is to have 1 second delay between queries.

Edit: Now TryNinja too LOL

In general, unless you have permission otherwise, a good rule of thumb is to limit your queries to a website to 1 per second. You generally will not get permission to make more frequent queries unless you are paying for some kind of API access.

[BitcoinGirl.Club]

Shit Bot was supper fast 🤣. [...]

You are clearly confusing God with Bots.

I am not, Bot knows about it 😉

You generally will not get permission to make more frequent queries unless you are paying for some kind of API access.

Is it something to do with the IP where the forum is hosted or the system admin can create any rule for their own interest? I don't see theymos have any intention to make money from selling data.

[FatFork]

Nothing complex just small query for this a small quiz by cygan https://bitcointalk.org/index.php?topic=5420570.msg61422530#msg61422530

Isn't that like cheating? Just saying... 

[shahzadafzal]

Nothing complex just small query for this a small quiz by cygan https://bitcointalk.org/index.php?topic=5420570.msg61422530#msg61422530

Isn't that like cheating? Just saying... 

Well not really... because when I saw 3 answers to that question only then I thought of checking how many legendary users there are fitting this criteria  _ _ _ t _ _ _ _ and found out there are quite few.

[cygan]

Nothing complex just small query for this a small quiz by cygan https://bitcointalk.org/index.php?topic=5420570.msg61422530#msg61422530

Isn't that like cheating? Just saying...  

you are welcome to join us today for the next round and then prove your skills and quickness again

Nothing complex just small query for this a small quiz by cygan https://bitcointalk.org/index.php?topic=5420570.msg61422530#msg61422530

Isn't that like cheating? Just saying...  

Well not really... because when I saw 3 answers to that question only then I thought of checking how many legendary users there are fitting this criteria  _ _ _ t _ _ _ _ and found out there are quite few.

this is how it looks