Bitcoin Forum
July 07, 2024, 10:25:39 PM
Welcome,
Guest
. Please
login
or
register
.
News
: Latest Bitcoin Core release:
27.0
[
Torrent
]
Home
Help
Search
Login
Register
More
Bitcoin Forum
>
Other
>
Meta
>
Scraping bots on this forums
Pages: [
1
]
« previous topic
next topic »
Print
Author
Topic: Scraping bots on this forums (Read 139 times)
stingers
(OP)
Legendary
Offline
Activity: 1184
Merit: 1013
Scraping bots on this forums
January 18, 2018, 07:45:33 PM
#1
I am collecting some data, a lot of which can be collected from this forums.
How many HTTP requests per minute can i send at max?
I read a post about this last year, it was allowed until then. Confirming status as for Jan 2018.
My kind little bot will do nothing more than scrape this forum for required data.
InvoKing
Legendary
Offline
Activity: 2142
Merit: 1065
✋(▀Ĺ̯ ▀-͠ )
Re: Scraping bots on this forums
January 18, 2018, 07:48:48 PM
#2
AFAIK bots are blocked by cloudflare which was implemented lately. That's why
http://www.bctalkaccountpricer.info
for example stopped working... Other members can confirm this.
P
S
P
D
:
l
a
w
a
n
d
o
r
d
e
r
e
n
f
o
r
c
e
m
e
n
t
!
P
r
e
s
s
S
e
c
t
i
o
n
P
o
l
i
c
e
D
e
p
a
r
t
m
e
n
t
!
ibminer
Legendary
Offline
Activity: 1840
Merit: 2861
Goonies never say die.
⇾
Re: Scraping bots on this forums
January 18, 2018, 09:48:44 PM
#3
You want to keep it at least 1 second per request, and you may want to send a friendly PM to theymos to avoid any potential issues/blacklisting. And it's better to request data while not being logged in.
| _ )| _ \
|_ _| | _ \
:•:
2024 Halvening / Halving Playing Cards - Featuring Bitcointalk Users
| _ \| _/ | | | _/
:•:
Most Merit
|
Most Activity
|
Most Trusted
|
Most Recognized
|
Flag Log
|
Most Merited Threads
|
DT Change Log
|
Most Deleted by Mods
|__/|_| |__|
|_|
:•:
ı
l
l
ı
l
l
ı
Sheeple
~
The System
~
Free World
~
Don't look down (NSFW)
~
Hook
~
Brainwashed
ı
l
l
ı
l
l
ı
stingers
(OP)
Legendary
Offline
Activity: 1184
Merit: 1013
Re: Scraping bots on this forums
January 19, 2018, 03:02:58 AM
#4
Quote from: ibminer on January 18, 2018, 09:48:44 PM
You want to keep it at least 1 second per request, and you may want to send a friendly PM to theymos to avoid any potential issues/blacklisting. And it's better to request data while not being logged in.
Yes, I won't be logging in. Sending requests from my google cloud so it won't even be my IP.
Thanks for the heads-up
Pages: [
1
]
Print
Bitcoin Forum
>
Other
>
Meta
>
Scraping bots on this forums
« previous topic
next topic »
Jump to:
Please select a destination:
-----------------------------
Bitcoin
-----------------------------
=> Bitcoin Discussion
===> Legal
===> Press
===> Meetups
===> Important Announcements
=> Development & Technical Discussion
===> Wallet software
=====> Electrum
=====> Bitcoin Wallet for Android
=====> BitcoinJ
=====> Armory
=====> Mycelium
=====> Hardware wallets
=> Mining
===> Mining support
===> Pools
===> Mining software (miners)
===> Hardware
=====> Group buys
===> Mining speculation
=> Bitcoin Technical Support
=> Project Development
-----------------------------
Economy
-----------------------------
=> Economics
===> Speculation
=> Marketplace
===> Goods
=====> Computer hardware
=====> Digital goods
=======> Invites & Accounts
=====> Collectibles
===> Services
===> Currency exchange
===> Gambling
=====> Games and rounds
=====> Investor-based games
=====> Gambling discussion
===> Lending
=====> Long-term offers
===> Securities
===> Auctions
===> Service Announcements
=====> Micro Earnings
===> Service Discussion
=====> Web Wallets
=====> Exchanges
=> Trading Discussion
===> Scam Accusations
===> Reputation
-----------------------------
Other
-----------------------------
=> Meta
===> New forum software
===> Bitcoin Wiki
=> Politics & Society
=> Beginners & Help
=> Off-topic
=> Serious discussion
===> Ivory Tower
=> Archival
===> Корзина
===> CPU/GPU Bitcoin mining hardware
===> Chinese students
===> Obsolete (buying)
===> Obsolete (selling)
===> MultiBit
-----------------------------
Alternate cryptocurrencies
-----------------------------
=> Altcoin Discussion
=> Announcements (Altcoins)
===> Tokens (Altcoins)
=> Mining (Altcoins)
===> Pools (Altcoins)
=> Marketplace (Altcoins)
===> Service Announcements (Altcoins)
===> Service Discussion (Altcoins)
===> Bounties (Altcoins)
=> Speculation (Altcoins)
-----------------------------
Local
-----------------------------
=> العربية (Arabic)
===> العملات البديلة (Altcoins)
=====> النقاشات
===> إستفسارات و أسئلة المبتدئين
===> التعدين
===> النقاشات الأخرى
===> منصات التبادل
=> Bahasa Indonesia (Indonesian)
===> Marketplace (Bahasa Indonesia)
===> Mining (Bahasa Indonesia)
===> Altcoins (Bahasa Indonesia)
===> Trading dan Spekulasi
===> Ekonomi, Politik, dan Budaya
===> Topik Lainnya
=> Español (Spanish)
===> Mercado y Economía
=====> Servicios
=====> Trading y especulación
===> Hardware y Minería
===> Esquina Libre
===> Mercadillo
=====> Mexico
=====> Argentina
=====> España
=====> Centroamerica y Caribe
===> Primeros pasos y ayuda
===> Altcoins (criptomonedas alternativas)
=====> Minería de altcoins
=====> Servicios
=====> Tokens (Español)
=> 中文 (Chinese)
===> 跳蚤市场
===> 山寨币
===> 媒体
===> 挖矿
===> 离题万里
=> Hrvatski (Croatian)
===> Trgovina
===> Altcoins (Hrvatski)
=====> Announcements (Hrvatski)
===> Off-topic (Hrvatski)
=> Deutsch (German)
===> Anfänger und Hilfe
===> Mining (Deutsch)
===> Trading und Spekulation
===> Projektentwicklung
===> Off-Topic (Deutsch)
===> Treffen
===> Presse
===> Altcoins (Deutsch)
=====> Announcements (Deutsch)
===> Marktplatz
=====> Auktionen
=====> Suche
=====> Biete
=> Ελληνικά (Greek)
===> Αγορά
===> Mining Discussion (Ελληνικά)
===> Altcoins (Ελληνικά)
=====> Altcoin Announcements (Ελληνικά)
=====> Altcoin Mining (Ελληνικά)
=> עברית (Hebrew)
=> Français
===> Actualité et News
===> Débutants
===> Discussions générales et utilisation du Bitcoin
===> Mining et Hardware
===> Économie et spéculation
===> Place de marché
=====> Échanges
=====> Produits et services
=====> Petites annonces
===> Le Bitcoin et la loi
===> Wiki, documentation et traduction
===> Développement et technique
===> Vos sites et projets
===> Hors-sujet
===> Altcoins (Français)
=====> Annonces
=> India
===> Mining (India)
===> Marketplace (India)
===> Regional Languages (India)
===> Press & News from India
===> Alt Coins (India)
===> Buyer/ Seller Reputations (India)
===> Off-Topic (India)
=> Italiano (Italian)
===> Guide (Italiano)
===> Progetti
===> Discussioni avanzate e sviluppo
===> Trading, analisi e speculazione
===> Mercato
=====> Mercato valute
=====> Beni
=====> Servizi
=====> Esercizi commerciali
=====> Hardware/Mining (Italiano)
=====> Gambling (Italiano)
===> Accuse scam/truffe
===> Mining (Italiano)
===> Alt-Currencies (Italiano)
=====> Annunci
===> Raduni/Meeting (Italiano)
===> Crittografia e decentralizzazione
===> Off-Topic (Italiano)
=> 日本語 (Japanese)
===> アルトコイン
=> Nederlands (Dutch)
===> Markt
===> Gokken/lotterijen
===> Mining (Nederlands)
===> Beurzen
===> Alt Coins (Nederlands)
===> Off-topic (Nederlands)
===> Meetings (Nederlands)
=> Nigeria (Naija)
===> Politics and society (Naija)
===> Off-topic (Naija)
=> 한국어 (Korean)
===> 대체코인 Alt Coins (한국어)
=> Pilipinas
===> Altcoins (Pilipinas)
=====> Altcoin Announcements (Pilipinas)
===> Pamilihan
===> Others (Pilipinas)
=> Polski
===> Tablica ogłoszeń
===> Alternatywne kryptowaluty
=====> Nowe kryptowaluty i tokeny
=====> Tablica ogłoszeń (altcoiny)
=> Português (Portuguese)
===> Primeiros Passos (Iniciantes)
===> Economia & Mercado
===> Mineração em Geral
===> Desenvolvimento & Discussões Técnicas
===> Criptomoedas Alternativas
===> Brasil
===> Portugal
=> Русский (Russian)
===> Новички
===> Бизнес
=====> Барахолка
=====> Обменники
===> Идеи
===> Кодеры
===> Майнеры
===> Политика
===> Трейдеры
===> Альтернативные криптовалюты
=====> Токены
=====> Бayнти и aиpдpoпы
===> Хайпы
===> Работа
===> Разное
===> Oбcyждeниe Bitcoin
=====> Новости
=====> Юристы
=> Română (Romanian)
===> Anunturi importante
===> Offtopic
===> Market
=====> Discutii Servicii
===> Minerit
===> Tutoriale
===> Bine ai venit!
===> Presa
===> Altcoins (Monede Alternative)
=====> Anunturi Monede Alternative
=> Skandinavisk
=> Türkçe (Turkish)
===> Bitcoin Haberleri
===> Pazar Alanı
===> Madencilik
===> Ekonomi
===> Servisler
=====> Fonlar
===> Proje Geliştirme
===> Alternatif Kripto-Paralar
=====> Madencilik (Alternatif Kripto-Paralar)
=====> Duyurular (Alternatif Kripto-Paralar)
===> Konu Dışı
===> Yeni Başlayanlar & Yardım
===> Buluşmalar
=> Other languages/locations
Loading...