Bitcoin Forum
June 27, 2019, 01:19:31 AM *
News: Latest Bitcoin Core release: 0.18.0 [Torrent] (New!)
 
   Home   Help Search Login Register More  
Pages: [1] 2 »  All
  Print  
Author Topic: Merit: data visualization storytelling - Full interactive data  (Read 717 times)
Piggy
Hero Member
*****
Offline Offline

Activity: 770
Merit: 1356



View Profile WWW
April 13, 2018, 05:42:34 AM
Last edit: July 08, 2018, 08:13:35 AM by Piggy
Merited by TMAN (10), Foxpup (6), malevolent (5), qwk (5), dbshck (5), suchmoon (5), LoyceV (4), bill gator (1), bobq (1), criz2fer (1), DdmrDdmr (1), theyoungmillionaire (1)
 #1

Hello,

I collected the data regarding all the Merits given since it was introduced (updated till 6th of April) an put it into a graph which can be navigated, this can give a pretty good idea how and where the Merits are getting distributed across the whole forum.

Starting from the "Bitcoin Forum" section, you can drill down till the thread where the Merits have been given and at the same time see the number of merit collected by each section/subsection of the forum.

It took some time to put it together, but i think it was worth Smiley
Enjoy!

https://albertoit.github.io/Visualization/index.html



Bonus pic:





How to use it:
- Mouse over on any parth of the graph will show the name of the section and the Merits which has been given under that section.
- Click on each section to zoom into that section, each subsection/threads will be shown directly in the immediate outer ring strating from the center.
- Click on the center to go back to the previous section.

NEW GAME FORMAT
JACKPOT UP TO $60000+
Guess The Symbols Of a Real Ethereum Hash
PLAY NOW
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1561598371
Hero Member
*
Offline Offline

Posts: 1561598371

View Profile Personal Message (Offline)

Ignore
1561598371
Reply with quote  #2

1561598371
Report to moderator
1561598371
Hero Member
*
Offline Offline

Posts: 1561598371

View Profile Personal Message (Offline)

Ignore
1561598371
Reply with quote  #2

1561598371
Report to moderator
1561598371
Hero Member
*
Offline Offline

Posts: 1561598371

View Profile Personal Message (Offline)

Ignore
1561598371
Reply with quote  #2

1561598371
Report to moderator
LoyceV
Legendary
*
Offline Offline

Activity: 1526
Merit: 4159


Self-made Legendary outside Meta!


View Profile WWW
April 13, 2018, 06:26:49 AM
 #2

Well done! It clearly shows the Local sections (often said to not receive merit) do receive a lot of merit. What surprises me, is that the Altcoin boards receive almost twice as much as the Bitcoin boards!

It took some time to put it together, but i think it was worth Smiley
Would this be something you can keep up to date once in a while?

Update:
Quote
I collected the data regarding all the Merits given since it was introduced (updated till 6th of April)
Only after I've merited your post, I realized I disagree to your data:
Total: 125281
How did you get to 52,778 in total?

edwardceng
Member
**
Offline Offline

Activity: 266
Merit: 42


View Profile
April 13, 2018, 06:59:44 AM
 #3

Starting from the "Bitcoin Forum" section, you can drill down till the thread where the Merits have been given and at the same time see the number of merit collected by each section/subsection of the forum.

what methods are used to obtain data from each local board?

do you calculate the Merit manually from each local moderator?
I'm curious because I have ever counted Merits on my local board using the manual method with the sMerit count of each moderator.

Piggy
Hero Member
*****
Offline Offline

Activity: 770
Merit: 1356



View Profile WWW
April 13, 2018, 07:53:50 AM
Last edit: April 13, 2018, 09:31:34 AM by Piggy
 #4

Well done! It clearly shows the Local sections (often said to not receive merit) do receive a lot of merit. What surprises me, is that the Altcoin boards receive almost twice as much as the Bitcoin boards!

It took some time to put it together, but i think it was worth Smiley
Would this be something you can keep up to date once in a while?

Update:
Quote
I collected the data regarding all the Merits given since it was introduced (updated till 6th of April)
Only after I've merited your post, I realized I disagree to your data:
Total: 125281
How did you get to 52,778 in total?

Many of the threads/comments that received merits were actually deleted or not accessible so some of that data is not actually appearing in the graph.

What you see there is there is what you can actually verify/see at the moment in the forum.

The data was collected using the raw file https://bitcointalk.org/merit.txt.xz

Im planning to keep it updated.

LoyceV
Legendary
*
Offline Offline

Activity: 1526
Merit: 4159


Self-made Legendary outside Meta!


View Profile WWW
April 13, 2018, 10:10:48 AM
 #5

Many of the threads/comments that received merits were actually deleted or not accessible so some of that data is not actually appearing in the graph.
That does make sense, but it would mean almost 58% of all merits were sent to posts that have now been deleted.
I've just (manually) checked a sample of posts with high amounts of merit from theymos' data in human readable format (17 MB), and from that I'd say less than 10% went to deleted posts.

If an entire thread is deleted, obviously you can't see the board anymore. But did you included the board for deleted posts where the thread still exists?

Quote
Im planning to keep it updated.
Very nice!

Piggy
Hero Member
*****
Offline Offline

Activity: 770
Merit: 1356



View Profile WWW
April 13, 2018, 10:36:44 AM
 #6

Many of the threads/comments that received merits were actually deleted or not accessible so some of that data is not actually appearing in the graph.
That does make sense, but it would mean almost 58% of all merits were sent to posts that have now been deleted.
I've just (manually) checked a sample of posts with high amounts of merit from theymos' data in human readable format (17 MB), and from that I'd say less than 10% went to deleted posts.

If an entire thread is deleted, obviously you can't see the board anymore. But did you included the board for deleted posts where the thread still exists?

Quote
Im planning to keep it updated.
Very nice!

The rule i used was to discard comments from the counting regardless if the thread was still there.

It is strange indeed that so many comments with merits were actually removed, but i had specifically put a flag for it so im pretty confident that what is missing is for that reason.

I will pick few cases and check actually by hand if there is more to it that i missed.

In any case i believe the graph has enought data to give a good representation of where the merits are going.

DdmrDdmr
Hero Member
*****
Offline Offline

Activity: 532
Merit: 2343

There are lies, damned lies and statistics. MTwain


View Profile WWW
April 13, 2018, 10:38:17 AM
 #7

Hello,

I collected the data regarding all the Merits given since it was introduced (updated till 6th of April) an put it into a graph which can be navigated, this can give a pretty good idea how and where the Merits are getting distributed across the whole forum.
<...>

I think there's something wrong with the data:

Up until the 23/03/2018, my calculations were as follows (much in line with with Zentdex's "Where the merit pours" distribution).

This is what I calculated:




In your Data for example, the Local Board adds up to 19.003, which is half of my calculation (and I calculated it 2 weeks before so there should be more, not less in your graph...).

Piggy
Hero Member
*****
Offline Offline

Activity: 770
Merit: 1356



View Profile WWW
April 13, 2018, 11:14:26 AM
 #8

From what i can see we are talking abou threads missing as if you have a link to a comment missing the thread still shows up. Is when the thread is missing that you get nothing.

I had previously double checked the number i see in the graph for a given thread with what i see going in a thread manually and they seems to be the same.

I should run it again and check the total numbers against the threads i found to be missing and see if it match up with the total merits(125281 in the raw file i used).

athanz88
Sr. Member
****
Offline Offline

Activity: 476
Merit: 355


View Profile
April 13, 2018, 11:16:59 AM
 #9

Wow, this is next level. I made data about merit too before for my own topic but this is way out of my level, it is an awesome job. I hope you can continue to update this merits data.

Regarding to the difference between the number of merit from other thread, i guess there must be some archive to the posts that got meritted by people, so we can know the quality of the posts and increasing the accuracy of data gathering like this piggy does, and it also helps to spot merit cheaters.
DdmrDdmr
Hero Member
*****
Offline Offline

Activity: 532
Merit: 2343

There are lies, damned lies and statistics. MTwain


View Profile WWW
April 13, 2018, 11:28:10 AM
 #10

From what i can see we are talking abou threads missing as if you have a link to a comment missing the thread still shows up. Is when the thread is missing that you get nothing.

I had previously double checked the number i see in the graph for a given thread with what i see going in a thread manually and they seems to be the same.

I should run it again and check the total numbers against the threads i found to be missing and see if it match up with the total merits(125281 in the raw file i used).

Deleted posts with a Merit Tx involved are in the 4% to 6% area (of sMerit with no matching thread), depending on the dataset used in the past. That would give you a max around 7.500 sMerits not asociable to a forum section with the current dataset, but the remaining should be accounted for (unless there's been a massive deletion of threads these past two weeks, which I doubt it).

What is your count of merit awarded to deleted threads ?

LoyceV
Legendary
*
Offline Offline

Activity: 1526
Merit: 4159


Self-made Legendary outside Meta!


View Profile WWW
April 13, 2018, 11:38:13 AM
 #11

The rule i used was to discard comments from the counting regardless if the thread was still there.
Did you pause for one second between scraping posts? If you scrape too fast, you'll get empty pages, which could explain you missed about half the merited posts.

Piggy
Hero Member
*****
Offline Offline

Activity: 770
Merit: 1356



View Profile WWW
April 13, 2018, 11:45:01 AM
 #12

From what i can see we are talking abou threads missing as if you have a link to a comment missing the thread still shows up. Is when the thread is missing that you get nothing.

I had previously double checked the number i see in the graph for a given thread with what i see going in a thread manually and they seems to be the same.

I should run it again and check the total numbers against the threads i found to be missing and see if it match up with the total merits(125281 in the raw file i used).

Deleted posts with a Merit Tx involved are in the 4% to 6% area (of sMerit with no matching thread), depending on the dataset used in the past. That would give you a max around 7.500 sMerits not asociable to a forum section with the current dataset, but the remaining should be accounted for (unless there's been a massive deletion of threads these past two weeks, which I doubt it).

What is your count of merit awarded to deleted threads ?

My bad,I actually found the problem every amount of merits given for a comment is just counted as 1, instead of the real amount of merits given.  Embarrassed

Ill be updating the graph soon  Grin
 

DdmrDdmr
Hero Member
*****
Offline Offline

Activity: 532
Merit: 2343

There are lies, damned lies and statistics. MTwain


View Profile WWW
April 13, 2018, 11:48:27 AM
 #13

My bad,I actually found the problem every amount of merits given for a comment is just counted as 1, instead of the real amount of merits given.  Embarrassed

Ill be updating the graph soon  Grin
 

Good. Sum for Count and ready to roll...

Piggy
Hero Member
*****
Offline Offline

Activity: 770
Merit: 1356



View Profile WWW
April 14, 2018, 10:28:53 AM
 #14

Updated till 13th of April, now the total is correct 

Have fun Wink

Piggy
Hero Member
*****
Offline Offline

Activity: 770
Merit: 1356



View Profile WWW
April 23, 2018, 06:53:41 AM
 #15

Updated to 20th of April

Quote
Total amount of merits so far was 135823 what is missing from the graph are merits belonging to deleted threads!

Total amount of merits browsable in the graph is 129923

Piggy
Hero Member
*****
Offline Offline

Activity: 770
Merit: 1356



View Profile WWW
April 27, 2018, 05:10:03 PM
 #16


Updated to 27th of April

Quote
Total amount of merits so far was 140.662 what is missing from the graph are merits belonging to deleted threads!

Total amount of merits browsable in the graph is 134.713


bobq
Sr. Member
****
Offline Offline

Activity: 924
Merit: 302



View Profile
April 27, 2018, 06:53:27 PM
 #17


Updated to 27th of April

Quote
Total amount of merits so far was 140.662 what is missing from the graph are merits belonging to deleted threads!

Total amount of merits browsable in the graph is 134.713



Congratulations for having done such a good job. But in the end I didn't understand which is the percentage of merited posts which have been deleted (most likely because of the deletion of the entire thread). In this case, by the way, does the person keep the merit of a deleted post?

Piggy
Hero Member
*****
Offline Offline

Activity: 770
Merit: 1356



View Profile WWW
April 27, 2018, 07:02:39 PM
 #18


Updated to 27th of April

Quote
Total amount of merits so far was 140.662 what is missing from the graph are merits belonging to deleted threads!

Total amount of merits browsable in the graph is 134.713



Congratulations for having done such a good job. But in the end I didn't understand which is the percentage of merited posts which have been deleted (most likely because of the deletion of the entire thread). In this case, by the way, does the person keep the merit of a deleted post?

Thank you.

Deleted threads containing merits account for ~6k merits, about ~4.2% of the total.
The people that got the merits in those thread get to keep them so it doesn't affect anybody, is just not possible for us to see where and which post was merited.

Piggy
Hero Member
*****
Offline Offline

Activity: 770
Merit: 1356



View Profile WWW
April 30, 2018, 11:07:21 AM
 #19

Just added the % and a new category containing the merits belonging to deleted threads, so you can have the all figure


kokoska
Member
**
Offline Offline

Activity: 658
Merit: 17


View Profile
May 03, 2018, 07:59:56 PM
 #20

That's cool!  Shocked

But where is the English distribution sMerit?
I see only the location of the whole world, but there is no basic English distribution sMerit.

△ M!R△CLE TELE   ▌  BRINGING MAGIC TO THE TELECOM INDUSTRY  ▐   JOIN US NOW!
▐▐   40% Biweekly Rewards     ▬▬▬   Calls at €0.2   ▬▬▬     Traffic from €0.01 worldwide   ▌▌
▬▬▬▬▬▬   ANN  Lightpaper  Bounty  Facebook  Twitter  Telegram   ▬▬▬▬▬▬
Pages: [1] 2 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!