60M posts! View unedited/deleted posts (search per post, per user or per topic)

[TryNinja]

So that I can make a better Bitcointalk search engine. I can't stand the built-in one on Simple Machines Forum - the search is slow (sometimes they take over a minute for results to load - if they even do) and the results are never specific enough.

That's exactly what I use it for (https://ninjastic.space/search). Maybe it doesn't suit your needs?

[1714518183]

1714518183

[1714518183]

1714518183

[1714518183]

1714518183

[NotATether]

So that I can make a better Bitcointalk search engine. I can't stand the built-in one on Simple Machines Forum - the search is slow (sometimes they take over a minute for results to load - if they even do) and the results are never specific enough.

That's exactly what I use it for (https://ninjastic.space/search). Maybe it doesn't suit your needs?

No, unfortunately. The search is sometimes not specific just like Bitcointalk search. Take for example "public key addition" https://ninjastic.space/search?content=public%20key%20addition there is only 1 result on the page with the exact words in order. For this example I was expecting some of my posts about public key addition to appear.

[LoyceV]

Take for example "public key addition" https://ninjastic.space/search?content=public%20key%20addition there is only 1 result on the page with the exact words in order.

I get 2,376 results on that link. If I add double quotes, I get 8 results.

For this example I was expecting some of my posts about public key addition to appear.

If I search my archive on your posts, I get only one match for "public key addition":
https://loyce.club/archive/posts/5933/59330710.html
That's your post above. If I would download your post history, I may find it again if you edited those posts later.

The problem is we can't know which posts are edited, without checking all posts again. And downloading all posts takes about half a year.

[NotATether]

The problem is we can't know which posts are edited, without checking all posts again. And downloading all posts takes about half a year.

Just for clarification, that time is assuming there's a 1-second sleep between requests, right? (Theymos said he'd ban IP addresses that stress the server out with more frequent requests so that's what I have in my scraper)

[LoyceV]

Just for clarification, that time is assuming there's a 1-second sleep between requests, right?

Yes.

(Theymos said he'd ban IP addresses that stress the server out with more frequent requests so that's what I have in my scraper)

You'll get a warning page if you go to fast. I've also seen bans for bots that use multiple IPs so don't go there.

[NotATether]

(Theymos said he'd ban IP addresses that stress the server out with more frequent requests so that's what I have in my scraper)

You'll get a warning page if you go to fast. I've also seen bans for bots that use multiple IPs so don't go there.

Yeah, I figured already that multiple IPs would be unwelcome since it basically circumvents the 1sec restriction.

[LoyceV]

I've censored the following posts in my archive because of doxing (as requested here):

• post 59828825
• post 59899827
• post 59906257
• post 59906329

The first post has been removed from Bitcointalk, the other 3 posts have been edited by theymos. I didn't bother highlighting the changed, I just wiped them.

[LoyceV]

I missed posts 60409954-60410008, my scraper couldn't catch them because of Cloudflare.
The nearest ones I have are 60409953 and 60410009. That leaves a 15 minute gap.

[LoyceV]

Full disclosure (I blame Cloudflare):

I miss all posts between posts 61426048 and 61427300. That's 1251 missing posts in 5 hours and 40 minutes.

For 8 hours and 24 minutes, my scraper wasn't working. Anything between posts 61513760 and 61515614 (1853 posts) is missing

[LoyceV]

I've reduced my scraping frequency for new posts (to reduce unnecessary server load). I created my posts archive when there were many more posts per day, and the recent page only shows the last 10 so I have to be quick.
If I start missing more posts, that's either because it's spam and MindlessElectron deleted it before I scraped them, or because there were many posts in a short interval. I'll keep an eye on this.

[Mitchell]

Full disclosure (I blame Cloudflare):

I miss all posts between posts 61426048 and 61427300. That's 1251 missing posts in 5 hours and 40 minutes.

For 8 hours and 24 minutes, my scraper wasn't working. Anything between posts 61513760 and 61515614 (1853 posts) is missing

Yeah, CloudFlare is a bit of a bitch, but not much you can do.

[LoyceV]

Yeah, CloudFlare is a bit of a bitch, but not much you can do.

My scraper got whitelisted, but don't know yet if that solves all problems.

[Mitchell]

My scraper got whitelisted, but don't know yet if that solves all problems.

My bot is whitelisted as well, but when CloudFlare goes to its highest security level, the whitelist doesn't matter, as it will always show a Javascript challenge.

[LoyceV]

My bot is whitelisted as well, but when CloudFlare goes to its highest security level, the whitelist doesn't matter, as it will always show a Javascript challenge.

I was afraid of that:

In this case it may not have mattered. Cloudflare was having a real bad morning today. Got the calls from a few clients that they could not get to their own sites from the own offices that are in the IP whitelist.

I'm just flabbergasted that Cloudflare doesn't allow a real whitelist. Makes you wonder: are they trying to force Tor users to use a less secure connection? What could they possibly gain from that? Wait, never mind:

Cloudflare is very probably an NSA honeypot

[LoyceV]

DDOS

It looks like there's a DDOS going on, which started yesterday. I started DROPping ports, but I forgot to exclude Bitcointalk (it was quite late), so I broke my scraper.
For 7 hours and 2 minutes, I didn't download any posts between posts 62480523 and 62481467. That's 944 missing posts, which also mean my notifications failed for those.

[LoyceV]

The DDOS got worse before migigating it, and I missed more posts. Once in a while, my server managed to get some data again, and I'll only list the bigger gaps. I haven't archived any posts between the following msgIDs:
62490527-62490570
62490579-62490594
62490606-62490612
62490641-62490690
62490699-62490726
62490748-62490777
62490802-62490831
62490964-62491018
62491027-62491089
62491098-62491150
62491159-62491277
62491286-62491552
62491561-62494873

In total, I missed 4172 posts in the 62490000-category.

Long-term, I want to setup a backup scraper so I won't lose posts again. That's going to be complicated to process for members and topics, considering I still don't use a database. So that's why it's a long-term plan

[LoyceV]

Cloudflare on Bitcointalk is now blocking me from scraping (again). Unfortunately, I can't do anything about that.
I noticed it this morning, when my browser gave me a Cloudflare page before showing the forum.

[Don Pedro Dinero]

Ever wanted to see who's lying when a post has been edited or deleted? I may be able to help!

I have a question about this. I am sure this post has been edited, and does not appear as such on either bitcointalk.org or loyce.club:

<...>

Week#8 payment sent: fecf5189175f804322440a38e72e1ad4d3050704d4568d076986a2ca03a92c2a

Current balance: 0.11616218 BTC (The balance is enough to cover more than 1 week)

Once there are refill, I will update.

<...>

I'm not saying that my campaign manager lied or anything like that, I'm just curious because I was checking to see if the escrow had been filled in and when I didn't see it in later posts, I see that the balance has been updated in that post but without it being edited. At first I thought about asking it in the campaign thread but it would seem a bit off topic there. Could it be because of the API, which automatically updates the balance at source of the data, and that's why the post does not appear as edited on the forum?

After publishing the post I see that the image in question does appear as invalid but in the preview appeared correctly:



And now after posting this I see that the image in the quote has changed and appears as correct. Crazy.

[AB de Royse777]

@Don Pedro Dinero, you are dealing with API calls :-D

[LoyceV]

Could it be because of the API, which automatically updates the balance at source of the data, and that's why the post does not appear as edited on the forum?

Correct:

Code:

[img]https://bitcoindata.science/api/addressbalance.php?address=bc1qfqd0mqhws95d5q2rv6cgeqyr7cyxd4slltg7ze07wt42x0xg7q5q9l55a0[/img]

The image needs to be generated when Bitcointalk's image proxy asks for it, and that doesn't always work (or only after a while).