60M posts! View unedited/deleted posts (search per post, per user or per topic)

[johhnyUA]

Hello, how can i see deleted posts which belongs to specific user?

I wanna try to find really epic joke which was deleted by mods. Thx for answer.

[1714542457]

1714542457

[1714542457]

1714542457

[LoyceV]

Hello, how can i see deleted posts which belongs to specific user?

Yes. See http://loyce.club/archive/members/
It still gets only manual updates, I'm running an update now. This will probably take a few hours.

[FontSeli]

Hello, how can i see deleted posts which belongs to specific user?

Yes. See http://loyce.club/archive/members/
It still gets only manual updates, I'm running an update now. This will probably take a few hours.

Hello. I can't use your service. After clicking on the link gives 404 error. Is this a temporary problem?

[LoyceV]

Hello. I can't use your service. After clicking on the link gives 404 error. Is this a temporary problem?

Yes. I'm still working on restoring my data on another server.
I have a temporary fix for this temporary problem: add aws. in front of loyce.club:

Code:

http://aws.loyce.club/archive/posts/

I don't make it clickable, because the link will expire the moment I get all my data back online. So feel free to view it, but please don't post links to the temporary location.

Once done, archived posts will be available in a matter of seconds (instead of a minute in the old situation).

[nutildah]

Hey Loyce, are you aware that BPIP is currently redirecting to your home page? Kind of weird... I know its off-topic here but I thought you should know.

[LoyceMobile]

Hey Loyce, are you aware that BPIP is currently redirecting to your home page? Kind of weird... I know its off-topic here but I thought you should know.

I didn't know until you told me. Vod got hit by a HP bug that wiped his drives, guess this is a temporary solution until he restores s backup.

[FontSeli]

I have a temporary fix for this temporary problem: add aws. in front of loyce.club:

Code:

http://aws.loyce.club/archive/posts/

It works! Thank you for your help and for your great services!

[logfiles]

Hi LoyceV
I am trying to look for archived posts of a particular user from http://loyce.club/archive/members because his account was nuked but the Profile ID doesn't show up

The last update was on 2020-01-04 16:27 but on http://loyce.club/archive/posts the last update was just today, 2020-01-11 12:31. Could it be an irregularity or a technical error preventing an update on the members link.

[TECSHARE]

Bump (I don't want to delete my previous bump, because it's merited).

I think you keep merit even if a post is deleted? Some one correct me if I am wrong.

[TryNinja]

Bump (I don't want to delete my previous bump, because it's merited).

I think you keep merit even if a post is deleted? Some one correct me if I am wrong.

You do. He’s not worried about that.

He probably wants to keep the reference for which post earned him his merit. Otherwise, it says it was for post “Deleted”.

[LoyceV]

I am trying to look for archived posts of a particular user from http://loyce.club/archive/members because his account was nuked but the Profile ID doesn't show up

The last update was on 2020-01-04 16:27 but on http://loyce.club/archive/posts the last update was just today, 2020-01-11 12:31. Could it be an irregularity or a technical error preventing an update on the members link.

I still haven't fully tested this part yet, so updates are only manually started. I'm running an update now, it shouldn't take too long to process a week's worth of posts.
Update: see http://loyce.club/archive/members/274/2743460.html

I think you keep merit even if a post is deleted? Some one correct me if I am wrong.

Correct. And TryNinja is also right, I just don't like my "Merit earned for deleted posts"-counter to go up.

[LoyceV]

I've been thinking about expanding my archived posts to all posts that haven't been deleted yet. It would be useful for a case like this. It requires scraping a couple million pages, and storing 50+ million posts. I can limit the number of files on the server by storing 10 or 100 posts per page. Would this be useful?

[LoyceV]

I've been thinking about expanding my archived posts to all posts that haven't been deleted yet.

An update: I have started this project! Measured in scraping time, it's the biggest project I ever started. In the past 9 days, I've scraped about 4% of all data, so I expect to complete this around August.
There's also a chance I'll run out of disk space because of the millions of large posts made by bounty spammers, but I'll deal with that when it happens.

Sneak preview: http://loyce.club/archive/oldposts/
How to use:

• Find the msgID you need. Let's use 28228
• Remove the last 5 digits from the msgID to get the directory name (if there are 5 or less digits, use 0): 0
• Replace the last 2 digits of the msgID by xx, and add .html (if there are 2 or less digits, use 0xx): 282xx.html
• Add "#msg" and the msgID: #msg28228
• Put everything together and go to http://loyce.club/archive/oldposts/0/282xx.html#msg28228

Limitations

• Currently, the first 2.1 million posts are available.
• I'll scrape the first 5.21 million topics and all posts in there.
• That means I'll archive 53.36 million posts, this partially overlaps with my scraper for new posts.
• This is a one-time thing, I won't update it with newer posts (I scrape unedited versions for those).
• The time "scraped on" is Amsterdam time.

If no username is mentioned, it's either "Anonymous" or "random". I forgot those exist when I started scraping, and it's not important enough to start over.

[LoyceV]

Something failed in the above scrape, the Wall Observer thread stopped scraping after page 2628. I thought I had the "last page detection" working, but there still seems to be a flaw.

[Vod]

LoyceV, are you saving the raw data, or just converting it? 

[nutildah]

I've been thinking about expanding my archived posts to all posts that haven't been deleted yet. It would be useful for a case like this. It requires scraping a couple million pages, and storing 50+ million posts. I can limit the number of files on the server by storing 10 or 100 posts per page. Would this be useful?

Yes, I'm all for it!

[LoyceV]

LoyceV, are you saving the raw data, or just converting it?  

I'm only saving the raw post (one line from the raw HTML), and my own header like this one (post number, link to post, link to my archive, by username and scraping time).

Yes, I'm all for it!

It's started already

[LoyceV]

Sneak preview: http://loyce.club/archive/oldposts/

I now have the first 6.1 million Bitcointalk posts archived. Data processing took longer than expected, but it's published now. See link above.

[LoyceV]

Update: I finally got to work on a topic-view. It'll be published at http://loyce.club/archive/topics/, and it's currently crunching data on 1.9 million posts. I'm downloading the thread titles that I don't have yet, and that part takes a lot of time.

This viewer should make it much easier to find back all posts made in a certain thread. Obviously this is also limited to posts made after I started scraping.

[dkbit98]

If it is not a secret, how much data space is needed for all that millions of posts?
And is there a way to use some compression?