TryNinja
Legendary
Offline
Activity: 2968
Merit: 7379
|
|
February 22, 2022, 03:04:46 PM Merited by JayJuanGee (1) |
|
So that I can make a better Bitcointalk search engine. I can't stand the built-in one on Simple Machines Forum - the search is slow (sometimes they take over a minute for results to load - if they even do) and the results are never specific enough.
That's exactly what I use it for ( https://ninjastic.space/search). Maybe it doesn't suit your needs?
|
|
|
|
NotATether
Legendary
Offline
Activity: 1736
Merit: 7273
In memory of o_e_l_e_o
|
|
February 22, 2022, 05:55:37 PM |
|
So that I can make a better Bitcointalk search engine. I can't stand the built-in one on Simple Machines Forum - the search is slow (sometimes they take over a minute for results to load - if they even do) and the results are never specific enough.
That's exactly what I use it for ( https://ninjastic.space/search). Maybe it doesn't suit your needs? No, unfortunately. The search is sometimes not specific just like Bitcointalk search. Take for example "public key addition" https://ninjastic.space/search?content=public%20key%20addition there is only 1 result on the page with the exact words in order. For this example I was expecting some of my posts about public key addition to appear.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3444
Merit: 17390
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
February 22, 2022, 06:07:00 PM |
|
I get 2,376 results on that link. If I add double quotes, I get 8 results. For this example I was expecting some of my posts about public key addition to appear. If I search my archive on your posts, I get only one match for "public key addition": https://loyce.club/archive/posts/5933/59330710.htmlThat's your post above. If I would download your post history, I may find it again if you edited those posts later. The problem is we can't know which posts are edited, without checking all posts again. And downloading all posts takes about half a year.
|
|
|
|
NotATether
Legendary
Offline
Activity: 1736
Merit: 7273
In memory of o_e_l_e_o
|
|
February 23, 2022, 01:28:18 AM |
|
The problem is we can't know which posts are edited, without checking all posts again. And downloading all posts takes about half a year.
Just for clarification, that time is assuming there's a 1-second sleep between requests, right? (Theymos said he'd ban IP addresses that stress the server out with more frequent requests so that's what I have in my scraper)
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3444
Merit: 17390
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
February 23, 2022, 08:01:47 AM |
|
Just for clarification, that time is assuming there's a 1-second sleep between requests, right? Yes. (Theymos said he'd ban IP addresses that stress the server out with more frequent requests so that's what I have in my scraper) You'll get a warning page if you go to fast. I've also seen bans for bots that use multiple IPs so don't go there.
|
|
|
|
NotATether
Legendary
Offline
Activity: 1736
Merit: 7273
In memory of o_e_l_e_o
|
|
February 23, 2022, 10:50:41 AM |
|
(Theymos said he'd ban IP addresses that stress the server out with more frequent requests so that's what I have in my scraper) You'll get a warning page if you go to fast. I've also seen bans for bots that use multiple IPs so don't go there. Yeah, I figured already that multiple IPs would be unwelcome since it basically circumvents the 1sec restriction.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3444
Merit: 17390
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
April 23, 2022, 08:01:59 AM |
|
I've censored the following posts in my archive because of doxing ( as requested here): The first post has been removed from Bitcointalk, the other 3 posts have been edited by theymos. I didn't bother highlighting the changed, I just wiped them.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3444
Merit: 17390
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
June 21, 2022, 08:01:10 AM |
|
I missed posts 60409954-60410008, my scraper couldn't catch them because of Cloudflare. The nearest ones I have are 60409953 and 60410009. That leaves a 15 minute gap.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3444
Merit: 17390
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
December 11, 2022, 04:01:29 PM Last edit: December 29, 2022, 09:48:13 PM by LoyceV |
|
Full disclosure (I blame Cloudflare): I miss all posts between posts 61426048 and 61427300. That's 1251 missing posts in 5 hours and 40 minutes. For 8 hours and 24 minutes, my scraper wasn't working. Anything between posts 61513760 and 61515614 (1853 posts) is missing
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3444
Merit: 17390
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
December 31, 2022, 07:37:17 AM |
|
I've reduced my scraping frequency for new posts (to reduce unnecessary server load). I created my posts archive when there were many more posts per day, and the recent page only shows the last 10 so I have to be quick. If I start missing more posts, that's either because it's spam and MindlessElectron deleted it before I scraped them, or because there were many posts in a short interval. I'll keep an eye on this.
|
|
|
|
Mitchell
Staff
Legendary
Offline
Activity: 4046
Merit: 2240
Verified awesomeness ✔
|
|
December 31, 2022, 12:55:54 PM |
|
Full disclosure (I blame Cloudflare): I miss all posts between posts 61426048 and 61427300. That's 1251 missing posts in 5 hours and 40 minutes. For 8 hours and 24 minutes, my scraper wasn't working. Anything between posts 61513760 and 61515614 (1853 posts) is missing Yeah, CloudFlare is a bit of a bitch, but not much you can do.
|
| | | . Duelbits | | | ▄████▄▄ ▄█████████▄ ▄█████████████▄ ▄██████████████████▄ ▄████▄▄▄█████████▄▄▄███▄ ▄████▐▀▄▄▀▌██▄█▄██▐▀▄▄▀▌███ ██████▀▀▀▀████▀███▀▀▀▀█████ ▐████████████■▄▄▄■██████████▀ ▐██████████████████████████▀ ██████████████████████████▀ ▀███████████████████████▀ ▀███████████████████▀ ▀███████████████▀ | | | | | . ▄ ▄▄▀▀▀▀▄▄ ▄▀▀▄ █ █ ▀▄ █ ▄█▄ ▀▄ █ ▄▀ ▀▄ ▀█▀ ▄▀ ▀█▄▄▄▀▀ ▀ ▄▀ ▄▀ ▄▀
Live Games | | ▄▄▀▀▀▀▀▀▀▄▄ ▄▀ ▄▄▀▀▀▀▀▄▄ ▀▄ ▄▀ █ ▄ █ ▄ █ ▀▄ █ █ ▀ ▀ █ █ ▄▄▄ █ ▀▀▀▀▀▀▀▀▀▀▀▀▀ █ █ █ █▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀█ █▄█ █ ▀▀█ ▀▀█ ▀▀█ █ █▄█
Slots | | . ▄▀▀▀▀▀▀▀▀▀▀▀▀▀▄ █ ▄▄ █ ▄▀▀▀▀▀▀▀▀▀▀▀▀▀▄ █ █ ▄▄ █ █ █ █ █ █ ▄▀▀▄▀▀▄ █ █ █ ▀▄ ▄▀ █ █
Blackjack | | | | █▀▀▀▀▀█▄▄▄ ▀████▄▄ ██████▄ ▄▄▄▄▄▄▄▄█▀ ▀▀█ ████████▄ █ █████████▄ █ ██████████▄ ▄██ █████████▀▀▀█▄▄████ ▀▀███▀▀ ████ █ ███ █ █▀ ▄█████▄▄▄ ▄▄▀▀ ███████▀▀▀ | | | | | | | | | | [ Đ ][ Ł ] AVAILABLE NOW | |
Advertisements are not endorsed by me.
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3444
Merit: 17390
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
December 31, 2022, 02:17:35 PM |
|
Yeah, CloudFlare is a bit of a bitch, but not much you can do. My scraper got whitelisted, but don't know yet if that solves all problems.
|
|
|
|
Mitchell
Staff
Legendary
Offline
Activity: 4046
Merit: 2240
Verified awesomeness ✔
|
My bot is whitelisted as well, but when CloudFlare goes to its highest security level, the whitelist doesn't matter, as it will always show a Javascript challenge.
|
| | | . Duelbits | | | ▄████▄▄ ▄█████████▄ ▄█████████████▄ ▄██████████████████▄ ▄████▄▄▄█████████▄▄▄███▄ ▄████▐▀▄▄▀▌██▄█▄██▐▀▄▄▀▌███ ██████▀▀▀▀████▀███▀▀▀▀█████ ▐████████████■▄▄▄■██████████▀ ▐██████████████████████████▀ ██████████████████████████▀ ▀███████████████████████▀ ▀███████████████████▀ ▀███████████████▀ | | | | | . ▄ ▄▄▀▀▀▀▄▄ ▄▀▀▄ █ █ ▀▄ █ ▄█▄ ▀▄ █ ▄▀ ▀▄ ▀█▀ ▄▀ ▀█▄▄▄▀▀ ▀ ▄▀ ▄▀ ▄▀
Live Games | | ▄▄▀▀▀▀▀▀▀▄▄ ▄▀ ▄▄▀▀▀▀▀▄▄ ▀▄ ▄▀ █ ▄ █ ▄ █ ▀▄ █ █ ▀ ▀ █ █ ▄▄▄ █ ▀▀▀▀▀▀▀▀▀▀▀▀▀ █ █ █ █▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀█ █▄█ █ ▀▀█ ▀▀█ ▀▀█ █ █▄█
Slots | | . ▄▀▀▀▀▀▀▀▀▀▀▀▀▀▄ █ ▄▄ █ ▄▀▀▀▀▀▀▀▀▀▀▀▀▀▄ █ █ ▄▄ █ █ █ █ █ █ ▄▀▀▄▀▀▄ █ █ █ ▀▄ ▄▀ █ █
Blackjack | | | | █▀▀▀▀▀█▄▄▄ ▀████▄▄ ██████▄ ▄▄▄▄▄▄▄▄█▀ ▀▀█ ████████▄ █ █████████▄ █ ██████████▄ ▄██ █████████▀▀▀█▄▄████ ▀▀███▀▀ ████ █ ███ █ █▀ ▄█████▄▄▄ ▄▄▀▀ ███████▀▀▀ | | | | | | | | | | [ Đ ][ Ł ] AVAILABLE NOW | |
Advertisements are not endorsed by me.
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3444
Merit: 17390
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
December 31, 2022, 03:11:58 PM |
|
My bot is whitelisted as well, but when CloudFlare goes to its highest security level, the whitelist doesn't matter, as it will always show a Javascript challenge. I was afraid of that: In this case it may not have mattered. Cloudflare was having a real bad morning today. Got the calls from a few clients that they could not get to their own sites from the own offices that are in the IP whitelist. I'm just flabbergasted that Cloudflare doesn't allow a real whitelist. Makes you wonder: are they trying to force Tor users to use a less secure connection? What could they possibly gain from that? Wait, never mind: Cloudflare is very probably an NSA honeypot
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3444
Merit: 17390
Thick-Skinned Gang Leader and Golden Feather 2021
|
DDOS It looks like there's a DDOS going on, which started yesterday. I started DROPping ports, but I forgot to exclude Bitcointalk (it was quite late), so I broke my scraper. For 7 hours and 2 minutes, I didn't download any posts between posts 62480523 and 62481467. That's 944 missing posts, which also mean my notifications failed for those.
|
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3444
Merit: 17390
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
July 09, 2023, 03:45:53 AM |
|
Cloudflare on Bitcointalk is now blocking me from scraping (again). Unfortunately, I can't do anything about that. I noticed it this morning, when my browser gave me a Cloudflare page before showing the forum.
|
|
|
|
Don Pedro Dinero
Legendary
Offline
Activity: 1442
Merit: 1682
|
|
September 22, 2023, 03:28:30 PM |
|
Ever wanted to see who's lying when a post has been edited or deleted? I may be able to help!
I have a question about this. I am sure this post has been edited, and does not appear as such on either bitcointalk.org or loyce.club: I'm not saying that my campaign manager lied or anything like that, I'm just curious because I was checking to see if the escrow had been filled in and when I didn't see it in later posts, I see that the balance has been updated in that post but without it being edited. At first I thought about asking it in the campaign thread but it would seem a bit off topic there. Could it be because of the API, which automatically updates the balance at source of the data, and that's why the post does not appear as edited on the forum? After publishing the post I see that the image in question does appear as invalid but in the preview appeared correctly: And now after posting this I see that the image in the quote has changed and appears as correct. Crazy.
|
| Duelbits | ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ | | TRY OUR UNIQUE GAMES! ◥ DICE ◥ MINES ◥ PLINKO ◥ DUEL POKER ◥ DICE DUELS | | | | █▀▀ █ █ █ █ █ █ █ █ █ █ █ █▄▄ | ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ | ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ | ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ | ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ | ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ ███ ▀▀▀ | | ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ KENONEW ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ | ▀▀█ █ █ █ █ █ █ █ █ █ █ █ ▄▄█ | | 10,000x MULTIPLIER | | ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ | | ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ |
[/tabl
|
|
|
AB de Royse777
Legendary
Offline
Activity: 2618
Merit: 4067
Hire Bitcointalk Camp. Manager @ r7promotions.com
|
|
September 22, 2023, 05:02:16 PM |
|
@Don Pedro Dinero, you are dealing with API calls :-D
|
..Stake.com.. | | | ▄████████████████████████████████████▄ ██ ▄▄▄▄▄▄▄▄▄▄ ▄▄▄▄▄▄▄▄▄▄ ██ ▄████▄ ██ ▀▀▀▀▀▀▀▀▀▀ ██████████ ▀▀▀▀▀▀▀▀▀▀ ██ ██████ ██ ██████████ ██ ██ ██████████ ██ ▀██▀ ██ ██ ██ ██████ ██ ██ ██ ██ ██ ██ ██████ ██ █████ ███ ██████ ██ ████▄ ██ ██ █████ ███ ████ ████ █████ ███ ████████ ██ ████ ████ ██████████ ████ ████ ████▀ ██ ██████████ ▄▄▄▄▄▄▄▄▄▄ ██████████ ██ ██ ▀▀▀▀▀▀▀▀▀▀ ██ ▀█████████▀ ▄████████████▄ ▀█████████▀ ▄▄▄▄▄▄▄▄▄▄▄▄███ ██ ██ ███▄▄▄▄▄▄▄▄▄▄▄▄ ██████████████████████████████████████████ | | | | | | ▄▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▄ █ ▄▀▄ █▀▀█▀▄▄ █ █▀█ █ ▐ ▐▌ █ ▄██▄ █ ▌ █ █ ▄██████▄ █ ▌ ▐▌ █ ██████████ █ ▐ █ █ ▐██████████▌ █ ▐ ▐▌ █ ▀▀██████▀▀ █ ▌ █ █ ▄▄▄██▄▄▄ █ ▌▐▌ █ █▐ █ █ █▐▐▌ █ █▐█ ▀▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▀█ | | | | | | ▄▄█████████▄▄ ▄██▀▀▀▀█████▀▀▀▀██▄ ▄█▀ ▐█▌ ▀█▄ ██ ▐█▌ ██ ████▄ ▄█████▄ ▄████ ████████▄███████████▄████████ ███▀ █████████████ ▀███ ██ ███████████ ██ ▀█▄ █████████ ▄█▀ ▀█▄ ▄██▀▀▀▀▀▀▀██▄ ▄▄▄█▀ ▀███████ ███████▀ ▀█████▄ ▄█████▀ ▀▀▀███▄▄▄███▀▀▀ | | | ..PLAY NOW.. |
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3444
Merit: 17390
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
September 22, 2023, 05:20:56 PM |
|
Could it be because of the API, which automatically updates the balance at source of the data, and that's why the post does not appear as edited on the forum? Correct: [img]https://bitcoindata.science/api/addressbalance.php?address=bc1qfqd0mqhws95d5q2rv6cgeqyr7cyxd4slltg7ze07wt42x0xg7q5q9l55a0[/img] The image needs to be generated when Bitcointalk's image proxy asks for it, and that doesn't always work (or only after a while).
|
|
|
|
|