Bitcoin Forum

I tried to google but could not find an answer.

If I'm querying bitcointalk from a script what's the minimum time I should leave between each call? To avoid any IP ban or tagged as DDoS?

I have been doing this for quite some time and I always keep e.g. 2~2.5 second delay between each request. But this seems very slow now can I decrease it to 1 sec or less?

Yes.

No. You'll get rate limited if you do that (although a short burst period is allowed).

---

---

Depending on what data you're looking for, someone may have it already. If it's something I have, a data dump can save you months of scraping.

From theymos himself:


I also have some data if that helps: https://docs.ninjastic.space

And can also give you a data dump on request (;D).

Shit Bot was supper fast 🤣. I was you in mind while I was writing the response to OP suggesting to keep it longer than 1 second. Long ago I read somewhere that a safe zone is to have 1 second delay between queries.

Edit: Now TryNinja too LOL

I bet @Tryninja, @mprep and @LoyceV and other forum tech geeks can answer you about this, mentioning them might be a big help.

After some research, although i don't know exactly about the script you're trying to make, but here's an answer from @Tryninja for the telegram bot he made for probably scrapping data from bitcointalk, i don't know actually.

Edit: Wew, lol. Seems like it take a lot for me to type this reply, while all the mentioned guys already made a reply.

Yes of course LoyceV your dumps are the first ones I will always refer to and in fact I'm using one right now from here related to usernames https://bitcointalk.org/index.php?topic=5202231.0

Nothing complex just small query for this a small quiz by cygan https://bitcointalk.org/index.php?topic=5420570.msg61422530#msg61422530

---

Yes I have seen this too and thank you for that.

Actually you, LoyceV and DdmrDdmr too and I know you guys have very useful data dumps will always be appreciated.

---

For me I don't do much but for my own searches mostly, I also have data of some the users e.g Satoshi, Hal Finney, theymos and of course some of my favorite posters on WO (i don't want to name..) and I use it to query and refer to old posts for a quick search.

For example this  (https://bitcointalk.org/index.php?topic=178336.msg59833649#msg59833649) or when recently LoyceV said that he did not use word vibrant (https://bitcointalk.org/index.php?topic=5426875.msg61412812#msg61412812) so I thought of confirming myself and at first I found he did use it (https://bitcointalk.org/index.php?topic=2818350.msg58356358#msg58356358) but... later on manual inspection I found that it is a quoted word but quoted in a wrong way, wrong in a sense that bots will not know, and here (https://bitcointalk.org/index.php?topic=5393956.msg59818312#msg59818312) too, that's where the AI fails :)

---

You are clearly confusing God with Bots.

In general, unless you have permission otherwise, a good rule of thumb is to limit your queries to a website to 1 per second. You generally will not get permission to make more frequent queries unless you are paying for some kind of API access.

I am not, Bot knows about it 😉

Is it something to do with the IP where the forum is hosted or the system admin can create any rule for their own interest? I don't see theymos have any intention to make money from selling data.

Isn't that like cheating? Just saying...  ;D

Well not really... because when I saw 3 answers to that question only then I thought of checking how many legendary users there are fitting this criteria  _ _ _ t _ _ _ _ and found out there are quite few.

you are welcome to join us today for the next round and then prove your skills and quickness again ;D

---

this is how it looks :)