Bitcoin Forum

Other => Meta => Topic started by: EcuaMobi on January 30, 2018, 08:49:06 PM



Title: Request: Merit history downloadable as raw data
Post by: EcuaMobi on January 30, 2018, 08:49:06 PM
I already posted this on the official Merit thread (https://bitcointalk.org/index.php?topic=2818350.msg28935113#msg28935113). However it just got a couple of comments and then got buried.
So I'm re-posting it here. I really think this would be useful:



I'd like to see a link to download raw data regarding the latest merit activity.
Basically this https://bitcointalk.org/index.php?action=merit;stats=recent but as txt, json, csv or similar format including name/id of sender, name/id of receiver and post ID.

When the DT was provided as text (https://bitcointalk.org/index.php?topic=1175364.msg12385896#msg12385896) some interesting processing was possible.
With merit raw data we could easily write scripts to find suspicious activity, to show it graphically, advanced stats, among other things.


Title: Re: Request: Merit history downloadable as raw data
Post by: TheQuin on January 31, 2018, 08:18:39 AM
Bumping this as I think it's a good idea. I'd also like to see more stats rather than just the top 50 for each category.


Title: Re: Request: Merit history downloadable as raw data
Post by: aso118 on February 01, 2018, 06:19:03 AM
This is a very good idea. People can use the raw data to generate statistics they need.
This would be especially useful in early days of implementation of the system, to detect cases of blatant abuse.


Title: Re: Request: Merit history downloadable as raw data
Post by: EcuaMobi on February 02, 2018, 03:59:11 PM
This would be especially useful in early days of implementation of the system, to detect cases of blatant abuse.

Yes, definitely! And also in the long-term to have better stats, to both keep us informed and to help improving it.
I don't think it would be too difficult to provide the data, and it will allow all sorts of development.


Title: Re: Request: Merit history downloadable as raw data
Post by: pixie85 on February 02, 2018, 04:35:32 PM
I know that admins don't have time to analyse this and would appreciate help from the community. We wouldn't have to rely on people spotting and reporting possible cases of merit trading and could do it ourselves. I've seen some posts where people were getting 50 merit from a single account all at once which is at the very least suspicious. A list of merit activity would allow us to clearly see what's going on all around the forum. I'd expect most merit trading to be happening in the local boards and if you delete the merited post all visible trace is lost.


Title: Re: Request: Merit history downloadable as raw data
Post by: EcuaMobi on February 02, 2018, 06:10:04 PM
you want Real-time data or just txt file "manually updated"
I want raw data updated every specif amount of time, every day, week, whatever makes sense, so that third party scripts can download it regularly without overloading bitcointalk's server and knowing no data is lost.

itís easy to make .txt file from pages like https://bitcointalk.org/index.php?action=merit;stats=topusersat (https://bitcointalk.org/index.php?action=merit;stats=topusersat).
I'm more interested in this, actually: https://bitcointalk.org/index.php?action=merit;stats=recent
I wouldn't say easy, but yes it's possible. There are a few problems about doing that though:
  • The script must override the logging in and parse the HTML. This can change at any time and the script would stop working
  • Because there's no specif amount of time after which the data is updated, the script must download the data regularly, very often. This can cause problems with the anti-DDos system
  • If the script downloads data from here, it doesn't know immediately who received the Merit. It would need to open every post to see who the author is, causing a lot of traffic and, again, having problems with the DDoS system
So it would be much more difficult, not very reliable, can stop working at any time and causes much more load to bitcointalk.
Providing raw data (like this (https://bitcointalk.org/index.php?topic=1175364.msg12385896#msg12385896)) is by far the best solution.

I'm willing to write a Merit script and generate some stats, but I'm really not willing to do so when it represents so much work and knowing it will stop working at any time.

you download it from here https://file.io/LOKibO (https://file.io/LOKibO)
What is this supposed to be? I just see this:
Quote
{"success":false,"error":404,"message":"Not Found"}


Title: Re: Request: Merit history downloadable as raw data
Post by: Husires on February 02, 2018, 06:52:17 PM
  • The script must override the logging in and parse the HTML. This can change at any time and the script would stop working
you need a "key" from theymos to override the logging in.
What is this supposed to be? I just see this:
I just converted the page to a file like this


Title: Re: Request: Merit history downloadable as raw data
Post by: EcuaMobi on February 02, 2018, 06:54:47 PM
  • The script must override the logging in and parse the HTML. This can change at any time and the script would stop working
you need a "key" from theymos to override the logging in.
What is this supposed to be? I just see this:
I just converted the page to a file like this
Yes, I know it's possible, even for the page I want (https://bitcointalk.org/index.php?action=merit;stats=recent).
But it's unreliable as I explained in my previous post.

(And you're not really being helpful by just posting the result of the conversion. It could have been done manually for all I know)


Title: Re: Request: Merit history downloadable as raw data
Post by: pablito1989 on February 02, 2018, 07:13:10 PM
I think this is a good suggestion.
Personally I don't like the new Merit system but... If it's supposed to continue, we should be sure no one will make shady things


Title: Re: Request: Merit history downloadable as raw data
Post by: botany on February 03, 2018, 11:24:40 PM
The only (possible) downside to this suggestion is that it will probably make unmasking of the merit sources easier.
Theymos didn't want to publish the list, because it could result in them being pestered for merits...


Title: Re: Request: Merit history downloadable as raw data
Post by: EcuaMobi on February 04, 2018, 05:02:27 PM
The only (possible) downside to this suggestion is that it will probably make unmasking of the merit sources easier.
Theymos didn't want to publish the list, because it could result in them being pestered for merits...

Theymos also said:
If you pay attention, you'll get a pretty clear idea of who the active sources are
and that is true with or without third-party scripts. Mainly, you'll just have to check this (https://bitcointalk.org/index.php?action=merit;stats=topsendat) in a few months and you'll have a good idea of the list.

So I don't think this suggestion will make a difference on that matter. It will help identifying other things much harder to spot only with the currently available tools, such as abuse and advanced stats.


Title: Re: Request: Merit history downloadable as raw data
Post by: Quickseller on February 04, 2018, 06:50:50 PM
The merit system is still very new, users are still leaning and experimenting with it, and there is the strong potential it will be tweaked in the coming months to address issues discovered.

I believe any effort into tracking stats about the merit economy this early into its life will probably be of little use, and will contain mostly outlier data. After a few months, especially after most of the merit initially given to various users is spent, stats about merit will be more useful.


Title: Re: Request: Merit history downloadable as raw data
Post by: EcuaMobi on February 04, 2018, 09:58:26 PM
The merit system is still very new, users are still leaning and experimenting with it, and there is the strong potential it will be tweaked in the coming months to address issues discovered.

I believe any effort into tracking stats about the merit economy this early into its life will probably be of little use, and will contain mostly outlier data. After a few months, especially after most of the merit initially given to various users is spent, stats about merit will be more useful.
Yes, the system is new and it could change. More sources may be added, people could learn to use it better.
But I don't think the basics will change. It will still be "User A merits users B's post X" and "User A has granted a total of Y merit points to user B", so most stats would still work.

Stats will definitely be more useful in the future, but that doesn't mean it won't be useful at the moment, or that developing a script now will be useless.
Maybe the analysis with the current little available data would be of little help, but it is required to start grabbing the data now so it's useful soon.


Title: Re: Request: Merit history downloadable as raw data
Post by: theymos on March 01, 2018, 11:37:16 AM
Here you go: https://bitcointalk.org/merit.txt.xz

Similar to trust.txt.xz, it'll be updated weekly. It will show only the last 120 days of data; someone else should archive the old ones if you want them.

I am especially interested in analyses of this data which could point to sub-communities where the initial sMerit is exhausted and new sources are necessary, and people who might be good merit sources.

Edit: Note that for a little while I had user_to and user_from as names, but I decided to change it to IDs.


Title: Re: Request: Merit history downloadable as raw data
Post by: EcuaMobi on March 01, 2018, 02:56:12 PM
Here you go: https://bitcointalk.org/merit.txt.xz
Similar to trust.txt.xz, it'll be updated weekly. It will show only the last 120 days of data; someone else should archive the old ones if you want them.
Great, thanks. I'll start downloading it to my server and writing some scripts

I am especially interested in analyses of this data which could point to sub-communities where the initial sMerit is exhausted and new sources are necessary, and people who might be good merit sources.
I'll try to do something like that. However it seems knowing what board/sub-community every msg belongs to would require crawling the forum a lot.


For the moment I'll aggregate total merits sent from one user to another and direct exchange of merits between 2 users; and I'll do my best about finding small groups of users sending merits to each other.


Title: Re: Request: Merit history downloadable as raw data
Post by: sncc on March 01, 2018, 05:09:53 PM
Here you go: https://bitcointalk.org/merit.txt.xz
Similar to trust.txt.xz, it'll be updated weekly. It will show only the last 120 days of data; someone else should archive the old ones if you want them.
Great, thanks. I'll start downloading it to my server and writing some scripts

I am especially interested in analyses of this data which could point to sub-communities where the initial sMerit is exhausted and new sources are necessary, and people who might be good merit sources.
I'll try to do something like that. However it seems knowing what board/sub-community every msg belongs to would require crawling the forum a lot.


For the moment I'll aggregate total merits sent from one user to another and direct exchange of merits between 2 users; and I'll do my best about finding small groups of users sending merits to each other.

Nice, I posted my quick research here:

Merit stats & all transactions more than 40 Merits
https://bitcointalk.org/index.php?topic=3046077.0

where I created a histogram to try to see tendencies and a clickable table for all transactions more than 40 Merits.
From the link you can directly check user profile, merit history, and the thread where the large transaction performed.

In total 85524 sMerits have been sent, which is much smaller than 11975 sMerits created by 57 merit sources per month.
It seems that we need more merit sources.


Title: Re: Request: Merit history downloadable as raw data
Post by: Saint-loup on March 07, 2018, 04:25:56 PM
Another good thread made with these datas

"Merit Stat from theymos data"
https://bitcointalk.org/index.php?topic=3082289.0


Title: Re: Request: Merit history downloadable as raw data
Post by: tbct_mt2 on March 08, 2018, 04:10:58 PM
Here you go: https://bitcointalk.org/merit.txt.xz

Similar to trust.txt.xz, it'll be updated weekly. It will show only the last 120 days of data; someone else should archive the old ones if you want them.

I am especially interested in analyses of this data which could point to sub-communities where the initial sMerit is exhausted and new sources are necessary, and people who might be good merit sources.

Edit: Note that for a little while I had user_to and user_from as names, but I decided to change it to IDs.
Hi Theymos,

Would you mind giving me link to download full data of the forum related to merit (receivers, senders and their ranks, boards in the forum, etc.). The current ones lack those essential variables, and I don't have skills to get them from the forum raw dataset you gave.
Thank you very much for your help.


Title: Re: Request: Merit history downloadable as raw data
Post by: TheQuin on March 08, 2018, 04:14:27 PM
Here you go: https://bitcointalk.org/merit.txt.xz

Similar to trust.txt.xz, it'll be updated weekly. It will show only the last 120 days of data; someone else should archive the old ones if you want them.

I am especially interested in analyses of this data which could point to sub-communities where the initial sMerit is exhausted and new sources are necessary, and people who might be good merit sources.

Edit: Note that for a little while I had user_to and user_from as names, but I decided to change it to IDs.
Hi Theymos,

Would you mind giving me link to download full data of the forum related to merit (receivers, senders and their ranks, boards in the forum, etc.).
Thank you very much for your help.

That is the link in the post you quoted. You have to manipulate the data yourself and crawl the forum for the data as to ranks and boards.

LoyceV has already started to do some work on that. https://bitcointalk.org/index.php?topic=3078328



Title: Re: Request: Merit history downloadable as raw data
Post by: tbct_mt2 on March 08, 2018, 04:24:18 PM
That is the link in the post you quoted. You have to manipulate the data yourself and crawl the forum for the data as to ranks and boards.

LoyceV has already started to do some work on that. https://bitcointalk.org/index.php?topic=3078328


Thanks, TheQuin. I am not developers, so I don't have skills to get raw data from website/ forum like this one. I can do good statistical analysis and nice, informative plots for the forum users to get general trend of merit system operation over the last two months. But I can only do it if I get data.
It is likely that I can't get it. Anyway, thanks again for your thread.


Title: Re: Request: Merit history downloadable as raw data
Post by: LoyceV on March 13, 2018, 02:33:24 PM
LoyceV has already started to do some work on that. https://bitcointalk.org/index.php?topic=3078328
My progress is very slow at the moment (still recovering from being ill).

I have one request for theymos though: would it be possible to get a one-time complete overview over all initial Merit at the introduction on January 25th? Having such a baseline would make it much easier to analyze who ranked up after the Merit system was introduced.


Title: Re: Request: Merit history downloadable as raw data
Post by: treatWy on March 13, 2018, 02:55:55 PM
LoyceV has already started to do some work on that. https://bitcointalk.org/index.php?topic=3078328
My progress is very slow at the moment (still recovering from being ill).

I have one request for theymos though: would it be possible to get a one-time complete overview over all initial Merit at the introduction at January 25th? Having such a baseline would make it much easier to analyze who ranked up after the Merit system was introduced.

Be better soon LoyceV.

I agree. It looks like exciting to see what the progress of merit to entire members?
It could clear up the doubt of merit effectivity nor it could clear that merit is not effective so far?


Title: Re: Request: Merit history downloadable as raw data
Post by: Lunarics on March 23, 2018, 04:15:38 AM
Hi, have anyone in this form of a base or in a similar

https://preview.ibb.co/k0sZHx/2018_03_23_11_01_56.png (https://ibb.co/gtykPc)

or html ... .. in any format the main thing that was name post


Title: Re: Request: Merit history downloadable as raw data
Post by: Vod on March 23, 2018, 04:20:17 AM
I (BPIP) have collected all merit data in a database, and I can send you a copy in the format you'd like if you send an email to vod@martinlawrence.ca

(I may retract this offer if I get too many requests)


Title: Re: Request: Merit history downloadable as raw data
Post by: Lunarics on March 23, 2018, 04:41:54 AM
I (BPIP) have collected all merit data in a database, and I can send you a copy in the format you'd like if you send an email to vod@martinlawrence.ca

(I may retract this offer if I get too many requests)

Sending an email, thank


Title: Re: Request: Merit history downloadable as raw data
Post by: TheQuin on March 23, 2018, 06:28:25 AM
I (BPIP) have collected all merit data in a database, and I can send you a copy in the format you'd like if you send an email to vod@martinlawrence.ca

(I may retract this offer if I get too many requests)

Sending an email, thank

There's also this from LoyceV

New: theymos' data in human readable format (https://loycev.000webhostapp.com/merit_human_readable_2018-03-18.html) (14 MB).



Title: Re: Request: Merit history downloadable as raw data
Post by: LoyceV on March 23, 2018, 07:24:10 AM
There's also this from LoyceV
New: theymos' data in human readable format (https://loycev.000webhostapp.com/merit_human_readable_2018-03-18.html) (14 MB).
I'll update it with post titles if Cloudflare once turned off again. At the moment, I can't download from the forum using command line:
Code:
ERROR 503: Service Temporarily Unavailable.


Title: Re: Request: Merit history downloadable as raw data
Post by: Lunarics on March 23, 2018, 08:54:56 AM
I (BPIP) have collected all merit data in a database, and I can send you a copy in the format you'd like if you send an email to vod@martinlawrence.ca

(I may retract this offer if I get too many requests)

Sending an email, thank

There's also this from LoyceV

New: theymos' data in human readable format (https://loycev.000webhostapp.com/merit_human_readable_2018-03-18.html) (14 MB).

Not suitable,  Today at 04:46:08 AM: 1 from NLNico for   need full name post     Re: BetKing.io - Poker, Dice, 50 BTC Jackpot, 20 currencies


Title: Re: Request: Merit history downloadable as raw data
Post by: EcuaMobi on March 23, 2018, 02:35:04 PM
There's also this from LoyceV
New: theymos' data in human readable format (https://loycev.000webhostapp.com/merit_human_readable_2018-03-18.html) (14 MB).
I'll update it with post titles if Cloudflare once turned off again. At the moment, I can't download from the forum using command line:
Code:
ERROR 503: Service Temporarily Unavailable.
Have you tried using this?
https://github.com/Anorov/cloudflare-scrape

It works fine for me (but I didn't try today, I'm not sure if you mean an additional issue)


Title: Re: Request: Merit history downloadable as raw data
Post by: Lunarics on March 26, 2018, 03:54:27 AM
there is no base in this form
Today at 04:46:08 AM: 1 from NLNico for   need full name post >>>>   Re: BetKing.io - Poker, Dice, 50 BTC Jackpot, 20 currencies ????