Bitcointalk Merit Dashboard

[DdmrDdmr]

Merit Dashboard

The dashboard gives you access to anyone’s complete merit history in the TX tab, surpassing the 120 day limit.
Link:  Merit Dashboard.    



Google Sheet Tabular sumary for all merited accounts (updated weekly):
(user_id, name, datecreated, rank, posts, activity, merit, MeritReceived, startedwith, meritsent, probableInitialRank, url)
https://docs.google.com/spreadsheets/d/1mD6cdsQNJ8dauTL-XLi8I0oDG5WBl0z0NeKzXTByNyw/edit?usp=sharing


Data as of 27/09/2024:

Updated the Merit Dashboard to reflect the most recent sMerit available data:

Total sMerit: 1.890.882
Total TXs:    1.006.069
From Users:     25.845
To Users:         48.645
minDate:     2018-01-24 22:12:21
maxDate:    2024-09-27 02:47:20

New: added Matrix Tab (last Tab) to reflect and update this analysis: Merited on multiple Subsections and the Correlation Matrix

Merit Race per board (only adding Merits on that board) - selectable users..


-> Does not reorder positions, and frames are months.

Just added a quick tab to the Dashboard (called "last time merited") that allows us to see when we were last merited by someone, and/or when we last merited a given profile.



How it works:

1)   Type in your username on the top right box, and press enter.

2)   The top left list on will display the date of the last time you merited each profile. The default order is by Username, but you can click on the date header to sort by date.

3)   The bottom left list displays the date when you last merited each profile (same sort order comments as above).

4)   The Received Merits chart displays the overall received merits received by the UserName provided in step 1. Likewise with the Sent Merits chart.

5)   If you click on a given name on the "Last Time Received" list, the "Received Merit" chart will show you how many merits per month the selected profile has sent you. Click the name from the list to deselect (and erase the applied filter).

6)   Likewise, do the same in the "Last Time Sent" list to see how many merits per month you sent to a given profile on the list. Click again to deselect.

https://public.tableau.com/views/BitcointalkMeritDashboard/LastTimeMerited?:display_count=y&:origin=viz_share_link

Note: I wanted to have both lists and charts filter when you select a name from any of the lists at the same time. So far, it needs to be done separately on each list.

Merit per post on the Merit Dashboard

I’ve added the above table, with all the full 14 months of data to the Merit Dashboard (last 2 Tabs). There, you can play around with the filters to:

-   Delimit the comparison to a set of specific boards.
-   Delimit the months (i.e. exclude abnormal values).

The output is something like this:


https://public.tableau.com/views/BitcointalkMeritDashboard/MeritperPost?:display_count=y&:origin=viz_share_link


https://public.tableau.com/views/BitcointalkMeritDashboard/MeritperPostIIChart?:display_count=y&:origin=viz_share_link

Full description here: Merit per post per board II

Added the lists to the Merit Dashboard.  (Last tab, or https://public.tableau.com/shared/J8MWCR5H2?:display_count=y&:origin=viz_share_link). The data included is as of 07/02/2019. I’ve still got to see what update frequency to use for this tab (initially, I’ll try to make it weekly, alongside all other tabs but the one on the number of posts which is always monthly). The screen looks like this:



There are 3 sections on the screen:

a) Who ranked whom – Ranking:

It shows an ordered list of those who have ranked most people on the forum. In order to search within the list, either scroll around, or use the UserFromName filter on the right hand side of the screen (once filtered though, we’ll lose sight of the proper rank position, as it will be limited to a single entry).


b) Who ranked me:

It displays the list of profiles that ranked me up, the date, TX, merits and post. The post name is clickable, and the menu that pops-up lets you go to the post itself on Bitcointalk.


c) Who I ranked:

It displays the list of profiles that I’ve ranked-up, the date, TX, merits and post. The post name is clickable, and the menu that pops-up lets you go to the post itself on Bitcointalk.

You can change the user here in the UserName box (and press enter) in order to change the view to that of another user (applies to both sections "b" and "c" simultaneously).

Google Sheet Tabular sumary for all merited accounts (updated weekly):
(user_id, name, datecreated, rank, posts, activity, merit, MeritReceived, startedwith, meritsent, probableInitialRank, url)
https://docs.google.com/spreadsheets/d/1mD6cdsQNJ8dauTL-XLi8I0oDG5WBl0z0NeKzXTByNyw/edit?usp=sharing
-Created a couple of Tabs to show a Wordcloud with who merited you and whom you merited.
-Created Tab to view the total number of Merit Senders and Receivers forum wide (see sMerit participants – a worrying descending trend (!/?)).
-Created Tab to view Merit Streaks (see Received Merit - Top Streaks (merited days in a row) - Updated).
-Created Tab to compare post evolution on local boards (see Analysis – Net Post evolution in all Local Boards).


Note: If anyone wants to know how many merits they earned per day (as opposed to per month) and/or when was the last time so and so merited someone or vice-versa (or see the merits received per month from such and such), that is achievable through the Personal Summary Tab (by clicking on meriters/merited names and or changing the view to day or week instead on month). This is pretty powerful, although not trivial to use. If anyone want’s to give it a go and encounters issues, tell me and I’ll detail the process.


Edit 11/10/2019:
Added a couple of Tabs at the end of the Merit Dashboard to display:

Who merited you (size is proportional to the amount of sMerits they sent you):

https://public.tableau.com/shared/5QX4S5TCQ?:display_count=yes&:origin=viz_share_link

Whom you merited:


https://public.tableau.com/shared/5QX4S5TCQ?:display_count=yes&:origin=viz_share_link

Edit 02/10/2019:
I’ve created a new Tab called Merit Senders /Receivers (fourth from the left) which lets us see most of the information shown in sMerit participants – a worrying descending trend (!/?), but from a monthly perspective:



The novel elements are nFromUsers (Senders) which represents the total number of distinct users that sent sMerit on a given month, and nToUsers (Receivers) which is the total number of distinct recipients of sMerit in a given month.

-   If we apply no filters, we get the general forum-wide overview.
-   We can filter by UserFromName (or Id), to see the amount of distinct people we merit per month.
-   Likewise, filtering by UserToName (or Id) will display the number of distinct people that have merited that profile.
-   Similarly, applying the Section/Subsection filter allows us to see the number of meriters and merited by that concept (i.e. local board).


Edit 09/11/2018:
I added by request a filter on the Ranked-up Tab (called "subsection (choose one)"). This filter was non-trivial, and on this tab has the semantics of filtering the ranked-up users to those with at least 1 received sMerit in the selected Subsection.
This is very useful for local boards, as for example filtering by subsection “French” will show all users that have ranked-up needing merit, and that have received at least 1 sMerit in the French Local board. It is very likely that these users are therefore French (or affinities), and thus is a way of seeing who has ranked-up on a local board.

Obviously there can be false positives, created by multilingual people who post and get merited on more than one local board. Also locals who only get merited on English board sections only will not be detected as a local of a given language.

Edit 11/10/2018:
Tableau has had recent updates, which after a bit of tweaking, now allow the Merit Dashboard to be visible on multiple devices with better fitting. I’ve arranged things around so that a generic phone device can now be used to view the Merit Dashboard (specially in portrait mode). I may still make arrangements in the following days, since one or two horizontal scrolls are not working yet, and a few other minor things need to be adjusted.

Since the Dashboard often filters when you select a chart bar or a table row, it may not be that easy to get used to it on a mobile, while scrolling around and not accidently activating a filter. I personally prefer a larger screen though, but I think this version is a decent starting point for mobile visualization of the Dashboard.

Edit 09/10/2018:
Added a new tab to the Merit Dashboard called "Merited User Summary". The tab allows you to get a quick visual summary for your profile (or anyone else’s), summarizing the core indicators, summary of sMerits received and sent per month, implied Ranks, and board section/subsections, as well as the users you merit and that have merited you.



It’s interesting to click on the images and use them as filters. For example:

-   Clicking on a specific Received or Sent month on the charts, adjusts all sections to the data for that month (click again to deselect or use the arrows situated on the bottom right of the screen).

-   Clicking on a Specific subsection acts likewise.

-   Clicking on a specific From User allows us to see a summary of when that user sent us sMerit, adjusting the month and subsections accordingly. To User works in the same way.


Edit 04/07/2018:
Added a Tab called Reciprocal sMerit which, if used properly, could help to detect Merit Abuse (see my post on this thread: https://bitcointalk.org/index.php?topic=4428616.msg41487786#msg41487786). The Dashboard has now grown now quite a bit since the initial version...

The dashboard gives you access to anyone’s complete merit history in the TX tab, surpassing the 120 day limit.
Added this week: Post summary (distribution of sMerit per TX and TXs per post).
Added Recently: De-ranked Users, Potential Merit Sources, User Summary, Section/Subsection, Rankings and WordCloud.


I’ve been considering updating the data of some of my previous analysis. Since the process is rather manual, I thought it would be nice to bring it together into a common Merit Dashboard, in order to keep it all in one place, and to ease the updating process on a more regular basis if necessary. I’ve started off by including what I’ve considered the most relevant information, using a set of tabs to hold the information for a given conceptual vision of the data.
 
The resulting dashboard is a work in progress, since I will add more features and datasets whenever possible. To mention that I do prefer the colours on my previous analytical posts (black background on the graphs), but for now the dashboard will have a white background since I’m concentrated on content and a little bit less on aesthetics.

The dashboard requires no installation, but it does result to java script (so it must be enabled in order for it to work).

Note: Screen shots show up with certain date information based on my local settings. If this works correctly, the actual Merit Dashboard should work based on your local setings... (dates and number formats, not captions).

The dashboard has the following Tabs:


1.   Global Summary:




Shows a summary of how merit is awarded, broken down by:

-   Week (Monday to Sunday- on previous analysis I may have used Sunday to Saturday since it’s the default in the software I use for data crunching).
-   Day of the Week.
-   Hour (UTC – This is not too important, but some merit awarding varies by date depending for example if the data is UTC, UTC-6 or UTC+6).
-   Weekly Ratios I’ve used before, including total aggregates at the top of the table.

Since the data originated in the public merit.txt file is published on Friday mornings, the data to the most recent week is partial (lacking most of Friday, as well as Saturday and Sunday). The rest of the weeks are complete.

We can see a clear weekly downtrend again since the end of April onwards after a light recovery in the last fortnight of April.

We already knew that Thursday is the most merited day and that UTC 8-20h are the most merited hours of the day.

New weekly users that receive Merit are around the 25%-27% area (so every week roughly one quarter of merited users are merited for the first time -> see %ToNew in the Weekly Ratios).



2.   From/To Ranks



Shows which ranks merit which ranks both in absolute value (number of awarded merits) and relative values (% of awarded merits per “from rank” to each “to rank”).

This is similar to Which Ranks send sMerit to which Ranks - and who ranked up, but now updated.

Comparing current data to the analysis above stated, we can see for example that:
-   Legendaries keep roughly the same awarding proportions overall as they had around end of march, being the most favourable movement increase for Jr. Members (4,37% -> 6,18%).
-   Heroes keep nearly the exact proportions as end of March.
-   Sr. Member have dropped their awarding rate to Sr. Members (23,15% -> 20,56%) and have moved the drop to Members (23,02%->24,46%) and Jr. Members (6,57%-> 7,28%).
-   Full members have dropped a bit on their ratio of awarding full members (31,74%->29,52%), moving the difference to Members (26,69% -> 29,57%).
-   Members awarding to Full Members has moved from 20,26% to 17,15%,  being Members the most favoured increase (42,92%-> 47,07%).
-   Jr. Members now favour less Full Members than before (19,48% -> 14,01%), but increase the Newbies rank (4,22% -> 8,62%).

All in all there are slight changes, but not too significant.

Note that the rank is derived from the current user profile, and not at the time of actual awarding which may have been different (we cannot access the rank at time of awarding, so this is the next best thing we can use).


3.   Received Merit



Shows the overall received Merit breaking it down by the Ranks that send it, month, specific date and user names along with the aggregate quantity they’ve sent.
By default, the data shown is the overall merit, but you can filter by user name/s to delimit the view to a specific user set  (i.e. filter by your own username to see the merit you received).
 

4.   Sent Merit



Shows the overall sent Merit breaking it down by the Ranks the merit has been sent to, month, specific date and user names along with the aggregate quantity they’ve received.
By default, the data shown is the overall merit, but you can filter by user name/s to delimit the view to a specific user set (i.e. filter by your own username to see the merit you sent).


5.   Ranked-up



This gives us the amount of users that have ranked-up using the new merit system, indicating their starting rank and current rank. This is similar to what I’ve done before numerically in posts. To see a specific set it is easier to filter by (probable) initial rank and current rank.

I’ve omitted including combinations relative to ranks that are not amongst the conventional ranks that need merit to rank-up. There are even some de-ranked cases (omitted), probably caused by purchasing an account and deleting the posts.


6. Ranking-up pipeline



This gives us a global vision of how many users are depending on what (activity, merit, both) in order to rank-up. This only considers users that have been awarded Merit (in at least 1 Merit TX), not those that had a Merit airdrop and have not received any further Merit.

This is similar to Stats on the Rank pipeline - How many are we on the way to ranking up? extension

Comparing the above mentioned analysis (created during mid-April 2018) to current dashboard data, we can see that:
-   We’ve gone from 135 to 219 Heroes that could have ranked-up, but lack Merit.
-   We’ve gone from 297 to 444 Sr. Members that could have ranked-up, but lack Merit.
-   We’ve gone from 1.146 to 1.919 Full Members have not ranked-up due to lack of Merit (by contrast, those lacking both activity and merit have decreased 1.458 -> 911).
-   We’ve gone from 2.032 to 2.711 Members that could have ranked-up, but lack Merit.
-   We’ve gone from 1.914 to 2.460 Members that could have ranked-up, but lack Merit.

In summary, merit is slowing down the ranking process (as we already knew), and candidates awaiting “just” to earn merits in order to rank-up are greater by number.


7.   TXs.




A list of all the Merit Transactions. The idea here is to filter by User From Name or User To name to see all Tx history for a given user/s. This can also be done on the Merit Network.


8.   Merit Network



This should contain and embedded link to Our very own sMerit Network Picture - enhanced with access to all sMerit TXs.

Unfortunately, I can see the embedded webpage fine in development, but not once I view the published Merit Dashboard. Until I find a solution (if there is one, which I’m not too positive about), I’ve placed a button that takes us there.


Dashboard Usage:



Using the dashboard is pretty straight forward. It may be good to mention that:

If you scroll down you’ll find a very useful toolbar that allows you to:
-   (<-) Undo if you apply a filter to the data and cannot revert easily to the previous unfiltered situation.
-   (-> )Redo.
-   (rectangle with a downward arrow) Save as image or pdf.
-   (rectangle) Full Screen.
-   (ESC) Exit Full Screen.

Also, when applying a filter (for example in the Received Merit Tab), you can type in the name and press the plus sign to add the name to the filter (multiple names are allowed), or use the magnifier to search for all name that contain the typed string (and then select the desired names).
To delete a filter you can hover over the filter value until a cross sign appears to delete the item, or use the undo arrow on the toolbar mentioned above.

[InvoKing]

Nice stats, would be better if you used English (Google fault) rather than Spanish but it isn't that hard to figure it out tho.
+3 virtual smerit, ok <3 vod

[DdmrDdmr]

Nice stats, would be better if you used English (Google fault) rather than Spanish but it isn't that hard to figure it out tho.
+3 virtual smerit, ok <3 vod

The actual dashboard should show dates and numbers based on your locale. It's just on the screenshots that show up based on mine. I've just set another device to english UK and english USA and it does the change correctly...

[MainIbem]

This is a commendable effort you put in.

What is the merit network all about? How do you get over 120 merits you gave to persons as outlined in your link post, https://fusiontables.google.com/DataSource?docid=1NLEPIaT4K4pcBf4pE3tiH4w_3zoZFUIWy7bEfn83#rows:id=1

The information you presented is very upbuilding.

[DdmrDdmr]

<...>
What is the merit network all about? How do you get over 120 merits you gave to persons as outlined in your link post, https://fusiontables.google.com/DataSource?docid=1NLEPIaT4K4pcBf4pE3tiH4w_3zoZFUIWy7bEfn83#rows:id=1
<...>

The Merit Network is initially a graphical representation of all the Forum Users that one has interacted with either sending them sMerit, or receiving it from them. You can parametrize the user you want to query about. It basically shows the merit log information that is found on the forum page, but in a nicer manner and without the 120 day historical limit.

The limitation is that it needs to be updated by me, and that is not a short process. Also, it cannot be done with any granularity of less than a week, since official merit transaction file is released weekly.
I’ve recently enhanced it to include the related numerical values as well as the complete overall Merit Transaction file.

The 120 sMerits I’ve awarded to other users come from 240 Merits I’ve been awarded by other forum members for certain posts (a bit more now)...

[Vod]

The limitation is that it needs to be updated by me, and that is not a short process.

Maybe you and I can enter into a data transfer service agreement, once I've completed enhancements to BPIP.

I have the data, but feel no need to make all these pretty graphs when you have already done an awesome job!

[coinlocket$]

Amazing this is the final thread for the merit, but I have a request 

Is possible to set hyperlinks here?

[DdmrDdmr]

Maybe you and I can enter into a data transfer service agreement, once I've completed enhancements to BPIP.

Perhaps... let’s see how thing playout and evolve...
 

<...>
Is possible to set hyperlinks here?

coinlocket$, I added the hyperlink to the user profile from where you suggested, and also in the TX Tab. When you click on the user, a small pop-up menu appears and allows you to click the hyperlink to go to the Bitcointalk profile (or message in the TX tab).

This is another of those features that on the development environment works one way, but on exploitation requires a different solution (in development I can just click and go to the profile, with no prior menu selection). It not that much of an issue so I’m happy with what I’ve added.

[sncc]

Nice work, it allows us to check the merit history in detail without 120 days limitation.  I see "Set Merit" in the tab, maybe a typo of "Sent Merit"?

[DdmrDdmr]

Nice work, it allows us to check the merit history in detail without 120 days limitation.  I see "Set Merit" in the tab, maybe a typo of "Sent Merit"?

Cheers. I corrected the typo you spotted. The Merit Dashboard uses the same dataset as the merit network so as to stay synchronized between the two applications, both surpassing the 120 day TX limit of our user profile on the forum.

I’ve still got a few tweaks to do on the current tabs. For example, I’m now allowing to click on one of the graphs on tabs where user names are detailed, in order to filter the username list in a seamless manner (click again to undo).
I’ve done that so far on the Ranked-up and Ranking-up Pipeline Tabs, and need to tweak something alike on the Received and Sent Merit tabs.

[suchmoon]

Great job as usual.

Thursday is sticking out because the merit system was introduced on a Thursday and we sent a gazillion merits on that first day.

Maybe the day-of-week stats should consider only full weeks? E.g. start on the first Sunday (or Monday) after the system was introduced and also exclude the last incomplete week.

[DdmrDdmr]

<…>
Thursday is sticking out because the merit system was introduced on a Thursday and we sent a gazillion merits on that first day.
Maybe the day-of-week stats should consider only full weeks? E.g. start on the first Sunday (or Monday) after the system was introduced and also exclude the last incomplete week.

Yes, Thursday is a bit of a bugger as it biases the data as you say. Perhaps the ideal solution would be not to pre-filter data out (so as not to lose the total aggregates), but to include some filters that allows us to narrow down the data (for example, a date interval). That should deal with both of your suggestions in one go.

The dashboard, although everything is based on the same set of data, has some external pre-rolled-up data on some tabs (Global Summary Tab is one of them). That makes it quicker for the dashboard, but limits filtering options. That means I’ll have to change the way things are done internally in order to be able to introduce granular filters by dates, going against the basic merit TXs, and not a rolled-up derived set like I do now.

I’ll add it to my backlog of tasks to do on the dashboard…

EDIT:

Just broke the "thursday is the best day to post" mith!:

I've added a date calendar filter that can now be applied on the first tab (I've a minor issue still with the time which I have to reset to 00:00:00 on the first date of the selection range). If we filter from the 29/01/2018 00:00:00 .. 27/05/2018 23:59:59 (complete weeks, skipping the first and last partial weeks) we now get:

Monday: 15,15%
Tuesday: 15,47%
Wednesday: 15,68%
Thursday: 15,31%
Friday: 14,02%
Saturday: 12,15%
Sunday: 12,21%

We can now see that Thursday is not a thrasher at all ...

[DdmrDdmr]

Merit Dashboard

Edit 08/06/2018:

Updated Dashboard to reflect the most recent sMerit available data:
Total sMerit: 167.409       
Total Txs:      75.332
From Users:  14.548
To Users:      16.303
minDate:      2018-01-24 22:12:21   
maxDate:     2018-06-08 02:48:03.000

Link:  BitcoinTalk Merit Dashboard   

[zentdex]

Freaking awesome job! And now I see stats that are so dynamic that enable other users to interact with it.
I even spent almost half an hour exploring, filtering, searching, sorting and extracting meaningful information.
Those cool merit network diagrams complete it all. I have fun dragging around the nodes. Thanks for creating this.

[DdmrDdmr]

<...>And now I see stats that are so dynamic that enable other users to interact with it.
I even spent almost half an hour exploring, filtering, searching, sorting and extracting meaningful information.
Those cool merit network diagrams complete it all. I have fun dragging around the nodes. <...>

Thankyou zentdex for spending some time using the Merit Dashboard and Network. As you say, the dynamics of the tools bare the true value, going from a statical views to dynamical view on the data. The filter features allow us to focus on specific sets of data and derive potential useful information from it, being seeing our own personal records just a minor part to it all. Export to pdf allows us to capture snapshots of data and keep it as a reference for comparison reasons overtime.

My initial idea was to keep building on it, adding features from previous analysis into it.

I don’t believe we’ll find many forums with this level of transparency on some of their key operative features, and even less with public dashboards built around them. Well, we could say that Facebook surpasses this level of transparency without “meaning to” …

[DdmrDdmr]

Merit Dashboard

Edit 15/06/2018:

Updated Dashboard to reflect the most recent sMerit available data:
Total sMerit: 171.654    
Total Txs:       77.387
From Users:  14.796
To Users:      16.583
minDate:      2018-01-24 22:12:21   
maxDate:     2018-06-15 02:42:15


The dashboard gives you access to anyone’s complete merit history in the TX tab, surpassing the 120 day limit.

Added Tabs this week: User Summary, Section/Subsection, Rankings and WordCloud.

[bitart]

...
6. Ranking-up pipeline
...
Comparing the above mentioned analysis (created during mid-April 2018) to current dashboard data, we can see that:
-   We’ve gone from 135 to 219 Heroes that could have ranked-up, but lack Merit.
...

How do you determine if a Hero has reached the activity points needed to rank up?
If I understand well, it's not possible to determine if a Hero is not ranking up only because of the lack of Merits or because of the lack of both activity and the Merits, if the Hero has already reached the 775 activity points.

EDIT:
just checked the other topic, and saw that you use the 775 activity point to calculate with for every Hero

Is it possible to check somewhere that a Hero has reached his/her personal limit of the activity points? (and ony waiting for the Merits to come?)

[DdmrDdmr]

<...>
Is it possible to check somewhere that a Hero has reached his/her personal limit of the activity points? (and ony waiting for the Merits to come?)

Yes, I used the lower Activity limit as you’ve seen. On the dashboard, the Ranking-up pipeline shows us what you’ve asked. If you click on the top left chart on the 247 number associated to "1. Could have ranked up to legendary, lacking merit" item then the user list on the right gives you their user names. From there you can click on any user and follow the "go to user profile" to see the bitcointalk user profile.
The bottom graph tells us how far these users are from their objective broken down by groups of 20%.

[lobcmt2]

Highly appreciate dedication of  DdmrDdmr to create such informative topic on merit distribution in the forum.
Nevertheless, it might be better if you can give additional analysis on merit distribution on different member rank, for example mean +/- standard deviation, and median (interquartile range) of merits (received, sent).
With unnormal distributions, median and interquartile range should be used instead of mean and standard deviations. Presenting uncorrect statistics might  lead to misunderstandings, mis-interpretation of merit distributions on different groups of member in the forum, and should be avoided.
Additionally, outliers should be stated clearly and truncation approach should be used in some specific cases to present more correct statistics for the whole sample. (For instance, cases like nullius, iasenko, etc. should be truncated  ). After using truncation approach for outliers, mean and median of sample will be almost the same.

[bitart]

Yes, I used the lower Activity limit as you’ve seen.
...

Thanks. So it's not possible to check if the personal legendary activity limit for a Hero has already been reached, if the Hero don't have 1000 Merits. (If the Hero's current activity is somewhere between 775 and 1030.)