Request: Merit history downloadable as raw data

[EcuaMobi]

I already posted this on the official Merit thread. However it just got a couple of comments and then got buried.
So I'm re-posting it here. I really think this would be useful:


I'd like to see a link to download raw data regarding the latest merit activity.
Basically this https://bitcointalk.org/index.php?action=merit;stats=recent but as txt, json, csv or similar format including name/id of sender, name/id of receiver and post ID.

When the DT was provided as text some interesting processing was possible.
With merit raw data we could easily write scripts to find suspicious activity, to show it graphically, advanced stats, among other things.

[1713560964]

1713560964

[TheQuin]

Bumping this as I think it's a good idea. I'd also like to see more stats rather than just the top 50 for each category.

[aso118]

This is a very good idea. People can use the raw data to generate statistics they need.
This would be especially useful in early days of implementation of the system, to detect cases of blatant abuse.

[EcuaMobi]

This would be especially useful in early days of implementation of the system, to detect cases of blatant abuse.

Yes, definitely! And also in the long-term to have better stats, to both keep us informed and to help improving it.
I don't think it would be too difficult to provide the data, and it will allow all sorts of development.

[pixie85]

I know that admins don't have time to analyse this and would appreciate help from the community. We wouldn't have to rely on people spotting and reporting possible cases of merit trading and could do it ourselves. I've seen some posts where people were getting 50 merit from a single account all at once which is at the very least suspicious. A list of merit activity would allow us to clearly see what's going on all around the forum. I'd expect most merit trading to be happening in the local boards and if you delete the merited post all visible trace is lost.

[EcuaMobi]

you want Real-time data or just txt file "manually updated"

I want raw data updated every specif amount of time, every day, week, whatever makes sense, so that third party scripts can download it regularly without overloading bitcointalk's server and knowing no data is lost.

it’s easy to make .txt file from pages like https://bitcointalk.org/index.php?action=merit;stats=topusersat.

I'm more interested in this, actually: https://bitcointalk.org/index.php?action=merit;stats=recent
I wouldn't say easy, but yes it's possible. There are a few problems about doing that though:

• The script must override the logging in and parse the HTML. This can change at any time and the script would stop working
• Because there's no specif amount of time after which the data is updated, the script must download the data regularly, very often. This can cause problems with the anti-DDos system
• If the script downloads data from here, it doesn't know immediately who received the Merit. It would need to open every post to see who the author is, causing a lot of traffic and, again, having problems with the DDoS system

So it would be much more difficult, not very reliable, can stop working at any time and causes much more load to bitcointalk.
Providing raw data (like this) is by far the best solution.

I'm willing to write a Merit script and generate some stats, but I'm really not willing to do so when it represents so much work and knowing it will stop working at any time.

you download it from here https://file.io/LOKibO

What is this supposed to be? I just see this:

{"success":false,"error":404,"message":"Not Found"}

[Husires]

• The script must override the logging in and parse the HTML. This can change at any time and the script would stop working

you need a "key" from theymos to override the logging in.

What is this supposed to be? I just see this:

I just converted the page to a file like this

[EcuaMobi]

• The script must override the logging in and parse the HTML. This can change at any time and the script would stop working

you need a "key" from theymos to override the logging in.

What is this supposed to be? I just see this:

I just converted the page to a file like this

Yes, I know it's possible, even for the page I want (https://bitcointalk.org/index.php?action=merit;stats=recent).
But it's unreliable as I explained in my previous post.

(And you're not really being helpful by just posting the result of the conversion. It could have been done manually for all I know)

[pablito1989]

I think this is a good suggestion.
Personally I don't like the new Merit system but... If it's supposed to continue, we should be sure no one will make shady things

[botany]

The only (possible) downside to this suggestion is that it will probably make unmasking of the merit sources easier.
Theymos didn't want to publish the list, because it could result in them being pestered for merits...

[EcuaMobi]

The only (possible) downside to this suggestion is that it will probably make unmasking of the merit sources easier.
Theymos didn't want to publish the list, because it could result in them being pestered for merits...

Theymos also said:

If you pay attention, you'll get a pretty clear idea of who the active sources are

and that is true with or without third-party scripts. Mainly, you'll just have to check this in a few months and you'll have a good idea of the list.

So I don't think this suggestion will make a difference on that matter. It will help identifying other things much harder to spot only with the currently available tools, such as abuse and advanced stats.

[Quickseller]

The merit system is still very new, users are still leaning and experimenting with it, and there is the strong potential it will be tweaked in the coming months to address issues discovered.

I believe any effort into tracking stats about the merit economy this early into its life will probably be of little use, and will contain mostly outlier data. After a few months, especially after most of the merit initially given to various users is spent, stats about merit will be more useful.

[EcuaMobi]

The merit system is still very new, users are still leaning and experimenting with it, and there is the strong potential it will be tweaked in the coming months to address issues discovered.

I believe any effort into tracking stats about the merit economy this early into its life will probably be of little use, and will contain mostly outlier data. After a few months, especially after most of the merit initially given to various users is spent, stats about merit will be more useful.

Yes, the system is new and it could change. More sources may be added, people could learn to use it better.
But I don't think the basics will change. It will still be "User A merits users B's post X" and "User A has granted a total of Y merit points to user B", so most stats would still work.

Stats will definitely be more useful in the future, but that doesn't mean it won't be useful at the moment, or that developing a script now will be useless.
Maybe the analysis with the current little available data would be of little help, but it is required to start grabbing the data now so it's useful soon.

[theymos]

Here you go: https://bitcointalk.org/merit.txt.xz

Similar to trust.txt.xz, it'll be updated weekly. It will show only the last 120 days of data; someone else should archive the old ones if you want them.

I am especially interested in analyses of this data which could point to sub-communities where the initial sMerit is exhausted and new sources are necessary, and people who might be good merit sources.

Edit: Note that for a little while I had user_to and user_from as names, but I decided to change it to IDs.

[EcuaMobi]

Here you go: https://bitcointalk.org/merit.txt.xz
Similar to trust.txt.xz, it'll be updated weekly. It will show only the last 120 days of data; someone else should archive the old ones if you want them.

Great, thanks. I'll start downloading it to my server and writing some scripts

I am especially interested in analyses of this data which could point to sub-communities where the initial sMerit is exhausted and new sources are necessary, and people who might be good merit sources.

I'll try to do something like that. However it seems knowing what board/sub-community every msg belongs to would require crawling the forum a lot.


For the moment I'll aggregate total merits sent from one user to another and direct exchange of merits between 2 users; and I'll do my best about finding small groups of users sending merits to each other.

[sncc]

Here you go: https://bitcointalk.org/merit.txt.xz
Similar to trust.txt.xz, it'll be updated weekly. It will show only the last 120 days of data; someone else should archive the old ones if you want them.

Great, thanks. I'll start downloading it to my server and writing some scripts

I am especially interested in analyses of this data which could point to sub-communities where the initial sMerit is exhausted and new sources are necessary, and people who might be good merit sources.

I'll try to do something like that. However it seems knowing what board/sub-community every msg belongs to would require crawling the forum a lot.


For the moment I'll aggregate total merits sent from one user to another and direct exchange of merits between 2 users; and I'll do my best about finding small groups of users sending merits to each other.

Nice, I posted my quick research here:

Merit stats & all transactions more than 40 Merits
https://bitcointalk.org/index.php?topic=3046077.0

where I created a histogram to try to see tendencies and a clickable table for all transactions more than 40 Merits.
From the link you can directly check user profile, merit history, and the thread where the large transaction performed.

In total 85524 sMerits have been sent, which is much smaller than 11975 sMerits created by 57 merit sources per month.
It seems that we need more merit sources.

[Saint-loup]

Another good thread made with these datas

"Merit Stat from theymos data"
https://bitcointalk.org/index.php?topic=3082289.0

[tbct_mt2]

Here you go: https://bitcointalk.org/merit.txt.xz

Similar to trust.txt.xz, it'll be updated weekly. It will show only the last 120 days of data; someone else should archive the old ones if you want them.

I am especially interested in analyses of this data which could point to sub-communities where the initial sMerit is exhausted and new sources are necessary, and people who might be good merit sources.

Edit: Note that for a little while I had user_to and user_from as names, but I decided to change it to IDs.

Hi Theymos,

Would you mind giving me link to download full data of the forum related to merit (receivers, senders and their ranks, boards in the forum, etc.). The current ones lack those essential variables, and I don't have skills to get them from the forum raw dataset you gave.
Thank you very much for your help.

[TheQuin]

Here you go: https://bitcointalk.org/merit.txt.xz

Similar to trust.txt.xz, it'll be updated weekly. It will show only the last 120 days of data; someone else should archive the old ones if you want them.

I am especially interested in analyses of this data which could point to sub-communities where the initial sMerit is exhausted and new sources are necessary, and people who might be good merit sources.

Edit: Note that for a little while I had user_to and user_from as names, but I decided to change it to IDs.

Hi Theymos,

Would you mind giving me link to download full data of the forum related to merit (receivers, senders and their ranks, boards in the forum, etc.).
Thank you very much for your help.

That is the link in the post you quoted. You have to manipulate the data yourself and crawl the forum for the data as to ranks and boards.

LoyceV has already started to do some work on that. https://bitcointalk.org/index.php?topic=3078328

[tbct_mt2]

That is the link in the post you quoted. You have to manipulate the data yourself and crawl the forum for the data as to ranks and boards.

LoyceV has already started to do some work on that. https://bitcointalk.org/index.php?topic=3078328

Thanks, TheQuin. I am not developers, so I don't have skills to get raw data from website/ forum like this one. I can do good statistical analysis and nice, informative plots for the forum users to get general trend of merit system operation over the last two months. But I can only do it if I get data.
It is likely that I can't get it. Anyway, thanks again for your thread.