[overview] LoyceV's useful data on Bitcointalk

[LoyceV]

If you are going to dload all again, this may be a sign from the crypto gods to use a database.  

A database would be very good for some of my projects, but far out of my league and I'm far too deep in to make easy changes now.
But the missing data is related to data dumps (for which I "only" have 800 GB in my backup, currently downloading the rest at 100 kB/s), Bitcoin addresses (I only have bits and pieces of the page design, and currently can't download an update) and this huge file (which I can recreate after I downloaded everything else).

I already have a new VPS for the second project (3 cores, 100 GB SSD, 16 TB bandwidth), and I'm currently looking for another VPS for the rest (2 TB HDD, 16 TB bandwidth).

[joker_josue]

Update:
Last week, my (donated and dedicated) server disappeared, and I haven't received any response from my sponsor. So I assume it's gone, which messes up a lot of my large data projects. And because it's so large, I don't even have complete backups of everything (but I can download most of it again). I'm currently working on at least keeping my weekly Trust and Merit data processing going, so I don't end up with a missing week.

What a boring situation, I hope you can recover everything.

Have you already found another server?

[LoyceV]

Have you already found another server?

Nope, I'm still looking. I found a good deal, but it's IPv6 only, and I don't think that makes adding a domain name very easy.

I forgot to add: the good news is that my forum archive, post scraping and notifications are still running (on AWS).

[joker_josue]

Have you already found another server?

Nope, I'm still looking. I found a good deal, but it's IPv6 only, and I don't think that makes adding a domain name very easy.

Are you looking for a managed VPS server or not?

I've never used unmanaged, but it has better prices. But, it is more complex and you need to have deeper knowledge of servers.
Even so, one day I want to buy a month or two, from an unmanaged one, to test.

[LoyceV]

Are you looking for a managed VPS server or not?

Since I left shared hosting (years ago), I've always used unmanaged servers.

Even so, one day I want to buy a month or two, from an unmanaged one, to test.

You can even get a VPS for $0.001333 per hour, after depositing one dollar. I'm not using nor recommending that host for anything serious, but it's the cheapest I know to get a server.

[ccwbtc]

I can't reply to your PM right now, I have to wait an hour.

I'll respond then with a way to significantly speed up this process.

[LoyceV]

I'll respond then with a way to significantly speed up this process.

An API key would work, but from what I've seen, they're quite expensive. I don't really want to setup many different servers to hammer them (at 100 kB/s each). But it's not really a problem, I'm waiting for a reply from a potential host, and once I have one, I also need to upload 800+ GB from my local backups.

[joker_josue]

Are you looking for a managed VPS server or not?

Since I left shared hosting (years ago), I've always used unmanaged servers.

Even so, one day I want to buy a month or two, from an unmanaged one, to test.

You can even get a VPS for $0.001333 per hour, after depositing one dollar. I'm not using nor recommending that host for anything serious, but it's the cheapest I know to get a server.

To test it seems like a good solution.

As I'm not used to directly managing the server, I've always opted for managed options, to provide support for what I can't resolve. Even though sometimes they don't seem to have much desire to help... but that's it.

I had to explore and try to understand how to make everything work correctly, maintaining good levels of security and protection.

[LoyceV]

As I'm not used to directly managing the server, I've always opted for managed options, to provide support for what I can't resolve. Even though sometimes they don't seem to have much desire to help... but that's it.

There's even management software, but usually I just install Apache2 and make some manual settings for subdomains.

I ordered another VPS today. It's still going to take a few days go get things settled, but after that I should be able to automate most scripts again. I ran them locally last in the past days, and I don't want to keep doing that.

[LoyceV]

I ordered another VPS today. It's still going to take a few days go get things settled, but after that I should be able to automate most scripts again. I ran them locally last in the past days, and I don't want to keep doing that.

So... I have this new server, with 8 cores, but I suspect I only got quarter-cores, and my Merit data analysis from last week is still running today! The new one started already while the first one isn't done yet. I kinda miss my dedicated Xeon.

[LoyceV]

I scraped this data for a project, so here's a data dump in case someone can use this:
BoardIDs and their names: https://loyce.club/other/boards.txt (not all boardIDs are public)
CSV with boardID, topicID and title: https://loyce.club/other/board_topicID_title.txt (102 MB)

[Shishir99]

I don't know which one is your official thread for Loyce.club website, but I am commenting here because I did not find the one if there are any ANN for that.

I am happy that you designed your landing page, which looks good now.
Since you use Claude now, could you please move your ass a little and add pagination to your website data and make it human-readable?  Or do you plan to keep it for the robots or people who want to scrap? Nah, I mean, sometimes I open your website, and it loads a long page, and I end up closing my browser, even with 16 gigs of RAM. I have no fucking idea how I can access those pages, bro.

I don't know if you would trust me with the data, but give me the credentials, and I would try to do something. Just believe me that I won't wipe out your server 

[LoyceV]

I am happy that you designed your landing page, which looks good now.

That's what people keep saying I really couldn't care less about it myself

Since you use Claude now

No I don't.

could you please move your ass a little and add pagination to your website data and make it human-readable?

No
Well, it depends. Files like merit.html (603 MB) are just relics from when I started this. By now it's far too much to show all 1.25 million Merit transactions on one "human readable" page. And you should probably not try to view all posts in the Wall Observer thread on one page. But if you really, REALLY want to, it's still a great way to use CTRL-F, and with pagination, that becomes terrible. I sometimes search for something on Bitcointalk page after page, only because it's not possible to have 1000 posts on one page and search only once.

Or do you plan to keep it for the robots or people who want to scrap?

Again: it depends on the page.

Nah, I mean, sometimes I open your website, and it loads a long page, and I end up closing my browser, even with 16 gigs of RAM. I have no fucking idea how I can access those pages, bro.

Can you give an example of a page? It's more of a browser limitation than a RAM problem. Try a different browser.

I don't know if you would trust me with the data

I can answer this: I don't trust anyone with my data

give me the credentials, and I would try to do something.

I don't think you realize how many millions of pages I have created

[joker_josue]

By any chance, can you compile some files that contain the following information:

user_id
username
rank
activity
merit
reg_date

post_id
user_id
timestamp
board_id
topic_id

Perhaps (I think so) you already have these files.

I was thinking about data from the last year. But for you, the last year, or the entire time, is almost the same. 


Another thing, do you happen to have data on who is quoting whom?

[LoyceV]

user_id
username
earned merit

I have these, see https://loyce.club/usernames/ and https://loyce.club/Merit/userID_sent_received.txt

rank
activity
reg_date

I didn't keep track of this. I think BPIP has all of this.

post_id
user_id
timestamp (when I scraped it, a few seconds after posting)
topic_id

From my posts archive, I can find the above. To add board_id, you can combine it with https://loyce.club/other/board_topicID_title.txt (large file, which needs an update for this).

Perhaps (I think so) you already have these files.
I was thinking about data from the last year. But for you, the last year, or the entire time, is almost the same.  

I have most of it, but not all. And the result will be one very large file.

Another thing, do you happen to have data on who is quoting whom?

I have all the (unedited) posts, but quotes and nested quotes are a pain to deal with. It's spread out over tens of millions of files.

[joker_josue]

post_id
user_id
timestamp (when I scraped it, a few seconds after posting)
topic_id

From my posts archive, I can find the above.

In that case, were you able to compile only that information? I don't need the content of the post.

[LoyceV]

In that case, were you able to compile only that information? I don't need the content of the post.

It's running, starting from post 52,000,000 (about 7 years ago). That's 15 million files, at about 60 per second. It's going to take a few days. Or more, when the server starts throttling.
Remind me in a few days if I forgot about this

Sample:

Code:

userID,topicID,postID,scrapetime
2623379,1082909,52000000,Tue Jul 30 03:11:19 CEST 2019
198837,5170152,52000001,Tue Jul 30 03:11:19 CEST 2019
144969,5056363,52000002,Tue Jul 30 03:11:19 CEST 2019
1751710,5165081,52000003,Tue Jul 30 03:11:19 CEST 2019
1102440,4453455,52000004,Tue Jul 30 03:11:19 CEST 2019
554165,5169648,52000005,Tue Jul 30 03:10:03 CEST 2019
958698,2680309,52000006,Tue Jul 30 03:10:03 CEST 2019
2653916,5168675,52000007,Tue Jul 30 03:10:03 CEST 2019
1853667,5169567,52000008,Tue Jul 30 03:10:37 CEST 2019
2337647,1097017,52000009,Tue Jul 30 03:10:37 CEST 2019
1955691,5158332,52000010,Tue Jul 30 03:11:48 CEST 2019
2649358,5165196,52000011,Tue Jul 30 03:11:48 CEST 2019
2618256,5164148,52000012,Tue Jul 30 03:12:17 CEST 2019
396256,5132445,52000013,Tue Jul 30 03:12:17 CEST 2019
1113183,4971628,52000014,Tue Jul 30 03:12:17 CEST 2019
1190631,5170027,52000015,Tue Jul 30 03:12:17 CEST 2019
33309,5056363,52000016,Tue Jul 30 03:12:30 CEST 2019
2225303,5166033,52000017,Tue Jul 30 03:13:19 CEST 2019
1229230,5166162,52000018,Tue Jul 30 03:13:19 CEST 2019
107836,1365991,52000019,Tue Jul 30 03:13:19 CEST 2019
266384,5132445,52000020,Tue Jul 30 03:13:34 CEST 2019
17849,5132445,52000021,Tue Jul 30 03:13:34 CEST 2019
1098615,4971628,52000022,Tue Jul 30 03:13:46 CEST 2019
1634977,5131793,52000023,Tue Jul 30 03:16:17 CEST 2019
450671,5056363,52000024,Tue Jul 30 03:16:17 CEST 2019
1985308,3223136,52000025,Tue Jul 30 03:16:17 CEST 2019
1841562,5137778,52000026,Tue Jul 30 03:16:17 CEST 2019
1853667,5159865,52000027,Tue Jul 30 03:15:06 CEST 2019
1113183,4971628,52000028,Tue Jul 30 03:15:19 CEST 2019
32278,5127573,52000029,Tue Jul 30 03:15:31 CEST 2019
34775,5132445,52000030,Tue Jul 30 03:15:42 CEST 2019
585444,5166451,52000031,Tue Jul 30 03:15:59 CEST 2019
101604,5163998,52000032,Tue Jul 30 03:16:17 CEST 2019
514199,5132445,52000033,Tue Jul 30 03:16:34 CEST 2019
874254,1535816,52000034,Tue Jul 30 03:16:55 CEST 2019
252510,178336,52000035,Tue Jul 30 03:17:49 CEST 2019
144969,5127573,52000036,Tue Jul 30 03:17:49 CEST 2019
874254,1535816,52000037,Tue Jul 30 03:18:03 CEST 2019
47138,5056363,52000038,Tue Jul 30 03:21:14 CEST 2019
1046012,5158332,52000039,Tue Jul 30 03:21:14 CEST 2019
1005212,4730287,52000040,Tue Jul 30 03:21:14 CEST 2019
55704,5132445,52000041,Tue Jul 30 03:21:14 CEST 2019
980501,5169734,52000042,Tue Jul 30 03:21:14 CEST 2019
33309,5127573,52000043,Tue Jul 30 03:20:08 CEST 2019
452049,5170152,52000044,Tue Jul 30 03:20:08 CEST 2019
1390155,5163998,52000045,Tue Jul 30 03:21:05 CEST 2019
450671,5127573,52000046,Tue Jul 30 03:21:05 CEST 2019
1109095,5164591,52000047,Tue Jul 30 03:21:05 CEST 2019
402414,5168190,52000048,Tue Jul 30 03:21:35 CEST 2019
2445060,5165196,52000049,Tue Jul 30 03:21:35 CEST 2019

[joker_josue]

In that case, were you able to compile only that information? I don't need the content of the post.

It's running, starting from post 52,000,000 (about 7 years ago). That's 15 million files, at about 60 per second. It's going to take a few days. Or more, when the server starts throttling.
Remind me in a few days if I forgot about this

I don't need that much time. For what I'm planning to do, 1 year would be enough. 

Thank you. I'll wait without any problem. Big hug.

[TryNinja]

Another thing, do you happen to have data on who is quoting whom?

Something like this?

I think it's easier for me to get this type of data, since I have the posts indexed in an elasticsearch database.

[joker_josue]

Another thing, do you happen to have data on who is quoting whom?

Something like this?

I think it's easier for me to get this type of data, since I have the posts indexed in an elasticsearch database.

Yes. That could also be interesting for the analysis I'm considering.

I was thinking about a different type of data for you, which might be interesting for the analysis I'm considering.
Number of characters per post. But I'd rather not have that right away... because later there will be too many variables to use.  
I prefer not to exaggerate without knowing if it will actually make sense.