Additional data dumps?

[TheBeardedBaby]

It's not a necrobump.
Can we have the modlog and seclog dumps instead everyone to scrape the data from the server?

[100bitcoin]

It's not a necrobump.
Can we have the modlog and seclog dumps instead everyone to scrape the data from the server?

Can one still able to scrape data from the server? I thought theymos prohibited it since bitcointalk.to started to scrape the whole forum.

[TheBeardedBaby]

Can one still able to scrape data from the server? I thought theymos prohibited it since bitcointalk.to started to scrape the whole forum.

Both LoyceV and Vod are doing it, also i've seen other users too, so I think there is no any prohibition, yet.
I think if those dumps are available for download directly from the forum, more people can benefit out of it and there will be less traffic to the server.

[LoyceV]

Scraping works, but the current modlog covers only a limited time. It's not possible to get a complete overview of all banned users. From various sources, I have a list of 170k banned users now, but it's far from complete.

[TheBeardedBaby]

Scraping works, but the current modlog covers only a limited time. It's not possible to get a complete overview of all banned users. From various sources, I have a list of 170k banned users now, but it's far from complete.

Is there any place to have the modlog in raw format available? Even for the limited time. I want to check some things

[LoyceV]

Is there any place to have the modlog in raw format available? Even for the limited time. I want to check some things

Do you mean older versions? I used archive.li and archive.org.

[notocactus]

UID -> name, merit, potential activity, posts
 post ID -> topic ID, time, UID
 topic ID -> board ID, first post ID
 board ID -> board name

The new year, so I bump it to ask for additional data dump granted by theymos.

Besides these formats above, I ask for this one (for merit data):

Code:

time amount msg user_from user_to boardid
1516831941  1 2818066.msg28853325 35 877396 24

I already collected the boardid, so if the merit data has only one additional variable for board's ID (boardid), it will eliminate the need to scrap data (from LoyceV's help) each 6 months. Although I don't know the others need such variable in data dumps or not.

For some sorts of analyses like these:

• Time series plots of daily merits over boards
• Merit distributions over boards

[notocactus]

From the reply of admin yesterday, I think now it is a very good time to think of a consistent format for the forum's data dumps. Each dataset has different variables inside, but I think all of them should be connected with only common variable (at least one variable) - userid.

Username, no matter it is username or display name or both will result in differences when connecting different datasets dumped by the forum.

For additional data dumps, it is not the priority and I am not in a position to ask for it too much, but for current data formats, a small adjustment: from username to userid will be good.

LoyceV asked for this change too: https://bitcointalk.org/index.php?topic=5104467.msg53551686#msg53551686