Merit: data visualization storytelling - Full interactive data

[Piggy]

Hello,

I collected the data regarding all the Merits given since it was introduced (updated till 6th of April) an put it into a graph which can be navigated, this can give a pretty good idea how and where the Merits are getting distributed across the whole forum.

Starting from the "Bitcoin Forum" section, you can drill down till the thread where the Merits have been given and at the same time see the number of merit collected by each section/subsection of the forum.

It took some time to put it together, but i think it was worth
Enjoy!

https://albertoit.github.io/Visualization/index.html



Bonus pic:





How to use it:
- Mouse over on any parth of the graph will show the name of the section and the Merits which has been given under that section.
- Click on each section to zoom into that section, each subsection/threads will be shown directly in the immediate outer ring strating from the center.
- Click on the center to go back to the previous section.

[LoyceV]

Well done! It clearly shows the Local sections (often said to not receive merit) do receive a lot of merit. What surprises me, is that the Altcoin boards receive almost twice as much as the Bitcoin boards!

It took some time to put it together, but i think it was worth

Would this be something you can keep up to date once in a while?

Update:

I collected the data regarding all the Merits given since it was introduced (updated till 6th of April)

Only after I've merited your post, I realized I disagree to your data:

Total: 125281

How did you get to 52,778 in total?

[edwardceng]

Starting from the "Bitcoin Forum" section, you can drill down till the thread where the Merits have been given and at the same time see the number of merit collected by each section/subsection of the forum.

what methods are used to obtain data from each local board?

do you calculate the Merit manually from each local moderator?
I'm curious because I have ever counted Merits on my local board using the manual method with the sMerit count of each moderator.

[Piggy]

Well done! It clearly shows the Local sections (often said to not receive merit) do receive a lot of merit. What surprises me, is that the Altcoin boards receive almost twice as much as the Bitcoin boards!

It took some time to put it together, but i think it was worth

Would this be something you can keep up to date once in a while?

Update:

I collected the data regarding all the Merits given since it was introduced (updated till 6th of April)

Only after I've merited your post, I realized I disagree to your data:

Total: 125281

How did you get to 52,778 in total?

Many of the threads/comments that received merits were actually deleted or not accessible so some of that data is not actually appearing in the graph.

What you see there is there is what you can actually verify/see at the moment in the forum.

The data was collected using the raw file https://bitcointalk.org/merit.txt.xz

Im planning to keep it updated.

[LoyceV]

Many of the threads/comments that received merits were actually deleted or not accessible so some of that data is not actually appearing in the graph.

That does make sense, but it would mean almost 58% of all merits were sent to posts that have now been deleted.
I've just (manually) checked a sample of posts with high amounts of merit from theymos' data in human readable format (17 MB), and from that I'd say less than 10% went to deleted posts.

If an entire thread is deleted, obviously you can't see the board anymore. But did you included the board for deleted posts where the thread still exists?

Im planning to keep it updated.

Very nice!

[Piggy]

Many of the threads/comments that received merits were actually deleted or not accessible so some of that data is not actually appearing in the graph.

That does make sense, but it would mean almost 58% of all merits were sent to posts that have now been deleted.
I've just (manually) checked a sample of posts with high amounts of merit from theymos' data in human readable format (17 MB), and from that I'd say less than 10% went to deleted posts.

If an entire thread is deleted, obviously you can't see the board anymore. But did you included the board for deleted posts where the thread still exists?

Im planning to keep it updated.

Very nice!

The rule i used was to discard comments from the counting regardless if the thread was still there.

It is strange indeed that so many comments with merits were actually removed, but i had specifically put a flag for it so im pretty confident that what is missing is for that reason.

I will pick few cases and check actually by hand if there is more to it that i missed.

In any case i believe the graph has enought data to give a good representation of where the merits are going.

[DdmrDdmr]

Hello,

I collected the data regarding all the Merits given since it was introduced (updated till 6th of April) an put it into a graph which can be navigated, this can give a pretty good idea how and where the Merits are getting distributed across the whole forum.
<...>

I think there's something wrong with the data:

Up until the 23/03/2018, my calculations were as follows (much in line with with Zentdex's "Where the merit pours" distribution).

This is what I calculated:




In your Data for example, the Local Board adds up to 19.003, which is half of my calculation (and I calculated it 2 weeks before so there should be more, not less in your graph...).

[Piggy]

From what i can see we are talking abou threads missing as if you have a link to a comment missing the thread still shows up. Is when the thread is missing that you get nothing.

I had previously double checked the number i see in the graph for a given thread with what i see going in a thread manually and they seems to be the same.

I should run it again and check the total numbers against the threads i found to be missing and see if it match up with the total merits(125281 in the raw file i used).

[athanz88]

Wow, this is next level. I made data about merit too before for my own topic but this is way out of my level, it is an awesome job. I hope you can continue to update this merits data.

Regarding to the difference between the number of merit from other thread, i guess there must be some archive to the posts that got meritted by people, so we can know the quality of the posts and increasing the accuracy of data gathering like this piggy does, and it also helps to spot merit cheaters.

[DdmrDdmr]

From what i can see we are talking abou threads missing as if you have a link to a comment missing the thread still shows up. Is when the thread is missing that you get nothing.

I had previously double checked the number i see in the graph for a given thread with what i see going in a thread manually and they seems to be the same.

I should run it again and check the total numbers against the threads i found to be missing and see if it match up with the total merits(125281 in the raw file i used).

Deleted posts with a Merit Tx involved are in the 4% to 6% area (of sMerit with no matching thread), depending on the dataset used in the past. That would give you a max around 7.500 sMerits not asociable to a forum section with the current dataset, but the remaining should be accounted for (unless there's been a massive deletion of threads these past two weeks, which I doubt it).

What is your count of merit awarded to deleted threads ?

[LoyceV]

The rule i used was to discard comments from the counting regardless if the thread was still there.

Did you pause for one second between scraping posts? If you scrape too fast, you'll get empty pages, which could explain you missed about half the merited posts.

[Piggy]

From what i can see we are talking abou threads missing as if you have a link to a comment missing the thread still shows up. Is when the thread is missing that you get nothing.

I had previously double checked the number i see in the graph for a given thread with what i see going in a thread manually and they seems to be the same.

I should run it again and check the total numbers against the threads i found to be missing and see if it match up with the total merits(125281 in the raw file i used).

Deleted posts with a Merit Tx involved are in the 4% to 6% area (of sMerit with no matching thread), depending on the dataset used in the past. That would give you a max around 7.500 sMerits not asociable to a forum section with the current dataset, but the remaining should be accounted for (unless there's been a massive deletion of threads these past two weeks, which I doubt it).

What is your count of merit awarded to deleted threads ?

My bad,I actually found the problem every amount of merits given for a comment is just counted as 1, instead of the real amount of merits given.  

Ill be updating the graph soon  
 

[DdmrDdmr]

My bad,I actually found the problem every amount of merits given for a comment is just counted as 1, instead of the real amount of merits given.  

Ill be updating the graph soon  
 

Good. Sum for Count and ready to roll...

[Piggy]

Updated till 13th of April, now the total is correct 

Have fun

[Piggy]

Updated to 20th of April

Total amount of merits so far was 135823 what is missing from the graph are merits belonging to deleted threads!

Total amount of merits browsable in the graph is 129923

[Piggy]

Updated to 27th of April

Total amount of merits so far was 140.662 what is missing from the graph are merits belonging to deleted threads!

Total amount of merits browsable in the graph is 134.713

[bobq]

Updated to 27th of April

Total amount of merits so far was 140.662 what is missing from the graph are merits belonging to deleted threads!

Total amount of merits browsable in the graph is 134.713

Congratulations for having done such a good job. But in the end I didn't understand which is the percentage of merited posts which have been deleted (most likely because of the deletion of the entire thread). In this case, by the way, does the person keep the merit of a deleted post?

[Piggy]

Updated to 27th of April

Total amount of merits so far was 140.662 what is missing from the graph are merits belonging to deleted threads!

Total amount of merits browsable in the graph is 134.713

Congratulations for having done such a good job. But in the end I didn't understand which is the percentage of merited posts which have been deleted (most likely because of the deletion of the entire thread). In this case, by the way, does the person keep the merit of a deleted post?

Thank you.

Deleted threads containing merits account for ~6k merits, about ~4.2% of the total.
The people that got the merits in those thread get to keep them so it doesn't affect anybody, is just not possible for us to see where and which post was merited.

[Piggy]

Just added the % and a new category containing the merits belonging to deleted threads, so you can have the all figure

[kokoska]

That's cool! 

But where is the English distribution sMerit?
I see only the location of the whole world, but there is no basic English distribution sMerit.