Piggy (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/lamp.gif) |
June 27, 2018, 07:31:57 AM Last edit: July 11, 2018, 10:18:22 AM by Piggy Merited by Vod (16), redsn0w (15), malevolent (12), Foxpup (9), TMAN (9), mprep (8), LoyceV (6), vlad230 (6), suchmoon (5), Welsh (5), coinlocket$ (4), jamyr (4), DdmrDdmr (3), Lakai01 (3), qwk (2), fronti (2), Jet Cash (2), Tytanowy Janusz (2), Joel_Jantsen (2), bones261 (2), theyoungmillionaire (2), babo (1), LeGaulois (1), Coin-1 (1), fillippone (1), TheBeardedBaby (1), bL4nkcode (1), krishnapramod (1), r1s2g3 (1), cryptovigi (1), spirali (1) |
|
UPDATES: This is a tool that allow you to run full SQL queries over the merit data and is based on SQLite. If you are familiar with SQL you know the possibilities are basically endless or limited to your SQL knowledge. My previous similar tool ( here) is perhaps more user friendly, but has some limitation: speed and research flexibility. So this is why i made this. You can find more information about SQL syntax in here if you are interested to learn, i may also give a hand if somebody want to get out some particular information out, even though my SQL skills are a bit rusty. Table structure:MeritData( "Date" TEXT, "Merit" Integer, "Msg" TEXT, "FromID" Integer, "ToID" Integer, "Board" TEXT, "SubBoard" TEXT, "TitleThread" TEXT )
UserData( UserId Integer PRIMARY KEY, UserName TEXT, Rank TEXT, Trust TEXT, Location TEXT ) How to use it: Just type the query in the box and press Exceute, F5 or Ctrl-Enter to execute it. Link:https://albertoit.github.io/Merit-Explorer-SQL/![](https://ip.bitcointalk.org/?u=https%3A%2F%2Fi.imgur.com%2Fz3NmTvL.png&t=665&c=S2zVvv7PzhKMvQ) For those of you who are not familiar with SQL, beside the fact that you will get any result extremely fast, this is what it means:[/b] We can easily get the Top Receiver for any local section simply using this: SELECT MAX(result.total) as "Total Merit", result.toid as "Top merit receiver", result.SubBoard as "Local board" FROM ( SELECT toid, SubBoard, SUM(Merit) AS total FROM meritdata as m WHERE m.Board in (SELECT Board FROM meritdata Where Board like "Local%" GROUP BY Board) GROUP BY toid, Board ORDER BY Board,total Desc) AS result GROUP BY SubBoard ORDER BY result.total Desc or for any board: SELECT MAX(result.total) as "Total Merit", result.toID as "Top merit receiver", result.SubBoard as "Board" FROM ( SELECT toid, SubBoard, SUM(Merit) AS total FROM meritdata as m WHERE m.SubBoard in (SELECT SubBoard FROM meritdata GROUP BY SubBoard) GROUP BY toid, SubBoard ORDER BY SubBoard,total Desc) AS result GROUP BY SubBoard ORDER BY result.total Desc; Similarly we can find out the Top Giver: SELECT MAX(result.total) as "Total Merit", result.fromid as "Top merit giver", result.SubBoard as "Local board" FROM ( SELECT fromid, SubBoard, SUM(Merit) AS total FROM meritdata as m WHERE m.Board in (SELECT Board FROM meritdata Where Board like "Local%" GROUP BY Board) GROUP BY fromid, Board ORDER BY Board,total Desc) AS result GROUP BY SubBoard ORDER BY result.total Desc; or for any any board: SELECT MAX(result.total) as "Total Merit", result.fromID as "Top merit giver", result.SubBoard as "Board" FROM ( SELECT fromid, SubBoard, SUM(Merit) AS total FROM meritdata as m WHERE m.SubBoard in (SELECT SubBoard FROM meritdata GROUP BY SubBoard) GROUP BY fromID, SubBoard ORDER BY SubBoard,total Desc) AS result GROUP BY SubBoard ORDER BY result.total Desc; Find out the total merit awarded so far: SELECT Sum(Merit) FROM MeritData; Want the full history for a particular user? SELECT * FROM MeritData WHERE toID=35 OR fromID=35 ORDER BY fromid,toid;
|
|
|
|
mdayonliner
Copper Member
Sr. Member
![*](https://bitcointalk.org/Themes/custom1/images/star.gif) ![*](https://bitcointalk.org/Themes/custom1/images/star.gif) ![*](https://bitcointalk.org/Themes/custom1/images/star.gif)
Offline
Activity: 630
Merit: 420
We are Bitcoin!
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 27, 2018, 07:52:06 AM |
|
I was interested in SELECT all query. And... This things are getting really interesting for me. May be I want to start learning how to scrape data ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif) Good job mate. PS: It took long time to pull-up the entire table (Letting you know only - hoping you can do something)
|
Be happy be at peace. Looking forward to BTC at $1M
|
|
|
DdmrDdmr
Legendary
Offline
Activity: 2436
Merit: 10989
There are lies, damned lies and statistics. MTwain
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 27, 2018, 08:03:13 AM |
|
Nicely done there Piggy, i'm sure you'll build on it furthermore. Perhaps you could transform de unix date field into common date field for viewers who cannot do it themselves.
|
|
|
|
lukeburchill
Member
![*](https://bitcointalk.org/Themes/custom1/images/star.gif)
Offline
Activity: 78
Merit: 17
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 27, 2018, 08:12:57 AM |
|
Table structure:MeritData( "Date" Float, "Merit" Integer, "Msg" TEXT, "FromID" Float, "ToID" Float, "Board" TEXT, "SubBoard" TEXT ) I have been creating your script in XAMPP, but have errors then I add a bit script "create table" and removes the symbol (""). and done created a table in DB create table MeritData( Date Float, Merit Integer, Msg TEXT, FromID Float, ToID Float, Board TEXT, SubBoard TEXT )
|
|
|
|
babo
Legendary
Offline
Activity: 3724
Merit: 4378
The hacker spirit breaks any spell
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 27, 2018, 08:21:33 AM |
|
if you didnt block some operations, do it fast block
- INSERT - UPDATE - DELETE
for avoid trolls and vandals
|
|
|
|
Piggy (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 27, 2018, 08:34:31 AM |
|
All the changes are "local", the database cannot be modified really as is hosted in Github and there are no write permissions in there. If you make changes and then refresh the page the original data will be back. Nicely done there Piggy, i'm sure you'll build on it furthermore. Perhaps you could transform de unix date field into common date field for viewers who cannot do it themselves.
Yes can be further improved for sure. I'll improve the data quality so can be used easily and more effectively. create table MeritData( Date Float, Merit Integer, Msg TEXT, FromID Float, ToID Float, Board TEXT, SubBoard TEXT )
Yes, that above was just to show the field in the table I was interested in SELECT all query. And... This things are getting really interesting for me. May be I want to start learning how to scrape data ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif) Good job mate. PS: It took long time to pull-up the entire table (Letting you know only - hoping you can do something) Not much can be done there, prefer to keeping it "unchecked", everything run on local so no harm is done.
|
|
|
|
babo
Legendary
Offline
Activity: 3724
Merit: 4378
The hacker spirit breaks any spell
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 27, 2018, 11:58:43 AM |
|
little advice use a different repo (and not one repo for all)
i sent you a pm
|
|
|
|
funsponge
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 27, 2018, 05:06:25 PM |
|
This could spit out some interesting data what about gathering a certain rank like "hero members" and spitting out the usernames of the ones which have received at least 5 merits what sort of command would need to be put in?
|
|
|
|
jackg
Copper Member
Legendary
Offline
Activity: 2856
Merit: 3071
https://bit.ly/387FXHi lightning theory
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 27, 2018, 06:01:14 PM |
|
if you didnt block some operations, do it fast block
- INSERT - UPDATE - DELETE
for avoid trolls and vandals
I mean, considering vod's database previously got hacked, I'd be surprised if those commands weren't already blocked. I was interested in SELECT all query. And... This things are getting really interesting for me. May be I want to start learning how to scrape data ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif) Good job mate. PS: It took long time to pull-up the entire table (Letting you know only - hoping you can do something) I'd suggest against doing this unless you want a copy of the database. Unless all commands run with more than a few seconds/minutes lag then don't do anything op.
|
|
|
|
Piggy (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 27, 2018, 06:03:27 PM |
|
This could spit out some interesting data what about gathering a certain rank like "hero members" and spitting out the usernames of the ones which have received at least 5 merits what sort of command would need to be put in?
I will have a look tomorrow and add the rank information per user in another table, perhaps LoyceV has already it available, in that case would be quite fast to include that in. After that should not be a problem giving an answer to that question. At the moment since there is no rank you can run this for all the users: Link because the forum is blocking sql in the message: https://pastebin.com/3dqMdeK0
|
|
|
|
funsponge
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 27, 2018, 06:32:15 PM |
|
This could spit out some interesting data what about gathering a certain rank like "hero members" and spitting out the usernames of the ones which have received at least 5 merits what sort of command would need to be put in?
I will have a look tomorrow and add the rank information per user in another table, perhaps LoyceV has already it available, in that case would be quite fast to include that in. After that should not be a problem giving an answer to that question. At the moment since there is no rank you can run this for all the users: Link because the forum is blocking sql in the message: https://pastebin.com/3dqMdeK0Nice one I just tried the tool and its incredibly fast so good work on that. Keep on bringing us these sort of things please as I'm sure everyone appreciates these data dumps and tools.
|
|
|
|
mdayonliner
Copper Member
Sr. Member
![*](https://bitcointalk.org/Themes/custom1/images/star.gif) ![*](https://bitcointalk.org/Themes/custom1/images/star.gif) ![*](https://bitcointalk.org/Themes/custom1/images/star.gif)
Offline
Activity: 630
Merit: 420
We are Bitcoin!
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 27, 2018, 11:38:48 PM |
|
I'd suggest against doing this unless you want a copy of the database.
I actually wanted to see all data of the table ![Tongue](https://bitcointalk.org/Smileys/default/tongue.gif)
|
Be happy be at peace. Looking forward to BTC at $1M
|
|
|
Piggy (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 28, 2018, 10:31:45 AM |
|
One thing i forgot to add, since the output is shown on a web page, is possible to give html formatting to it, this means we can add direct links for the users profile page and to the post merited, here is an example: SELECT date, '<a href=https://bitcointalk.org/index.php?action=profile;u=' || fromid || '>' || fromid || '</a>' as Sender, '<a href=https://bitcointalk.org/index.php?action=profile;u=' || toid || '>' || toid || '</a>' as Receiver, merit, '<a href=https://bitcointalk.org/index.php?topic=' || msg || '#' || substr(msg,instr(msg,'.')+1) || '>Link Merited post</a>' as MeritedPost FROM meritdata WHERE merit >= 50 LIMIT 10;
|
|
|
|
Piggy (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 29, 2018, 11:20:24 AM |
|
Updated the data in general (29/06/2018): New link: https://albertoit.github.io/Merit-Explorer-SQL/Added User data: 23147 users, with name and rank, that have either sent or received any merits Table: TABLE UserData( UserId Integer PRIMARY KEY, UserName TEXT, Rank TEXT ); Updated the merit data table, added readable format date and Title of the thread where the comment was merited TABLE MeritData( "Date" TEXT, "Merit" Integer, "Msg" TEXT, "FromID" Integer, "ToID" Integer, "Board" TEXT, "SubBoard" TEXT, "TitleThread" TEXT ) Few new example based on the update in this link (forum preventing posting SQL for security reason): https://pastebin.com/PRrQ81Ld- Show the user information instead of the user id:
- Selecting data by a particular time frame is quite easy (all merit transaction happened the 1/06/2018):
- Select by a time frame (from 2018 to 15/06/2018):
- You can also use hour minutes (and seconds) to check a 30 minutes interval:
- How many merits sent for each rank?
- How may merits sent from one rank to another?
|
|
|
|
TheBeardedBaby
Legendary
Offline
Activity: 2240
Merit: 3150
₿uy / $ell ..oeleo ;(
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 29, 2018, 02:36:29 PM |
|
I can't wait to get my PC back online to play a little with your database. I just want to list all the users with 9 , 99, 249,... etc. merit and some other fun stats.
|
|
|
|
Piggy (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 29, 2018, 06:37:25 PM |
|
I can't wait to get my PC back online to play a little with your database. I just want to list all the users with 9 , 99, 249,... etc. merit and some other fun stats.
Yes that is the idea, if somebody has some particular method or theory to show something interesting within the data, it can be done quite easily, your imagination is the limit ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif) Just want to remind, If anybody need some help with the SQL i can advise.
|
|
|
|
Makkara
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 30, 2018, 06:53:53 PM |
|
Very nice work. ![Cool](https://bitcointalk.org/Smileys/default/cool.gif) One suggestion: Why not at this point adding the text of the comment merited itself, that could be good info as could also be used to search for particular words in the message itself. Offcourse could make the db much heavier, but peraphs is worth investigating and make some test?
|
|
|
|
Piggy (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 01, 2018, 02:07:11 PM |
|
I should make some calculation on paper, but i believe the database would become quite large and the main problem i see is the waiting time to download it. Otherwise there should not be big problems speed wise, if the tables will be properly indexed.
What im planning to do next is some sort of scripting system ( that would be the equivalent of store procedures, which are missing in SQLite ), nothing very sophisticated, just some injecting of javascript where you can define the sql, execute it and manipulate the results to get the information you want in some complex case.
|
|
|
|
Piggy (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 06, 2018, 07:44:31 AM |
|
Updated with the latest data from today (6th of July), added Trust and Location fields to the user table: UserData( UserId Integer PRIMARY KEY, UserName TEXT, Rank TEXT, Trust TEXT, Location TEXT )
|
|
|
|
Piggy (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 08, 2018, 05:35:20 AM Last edit: July 11, 2018, 05:10:46 AM by Piggy |
|
Im working on the scripting system at the moment and it should be ready by the next week, with it is going to be easier to extract data without comin up with very long sql queries, like this one i'm about to show you now.
I was actually looking into an interesting question, what is the minimum amount of smerits generated just by sending them to somebody and still unspent ? We need for each user to calculate the merits received, half that minus the merit sent. We are going to disregard remaining airdropped smerits or smerits given to merit sources. This is why i refer to it as minimum amount of smerits unspent, in reality is higher but this still can be a good indicator for future sustainability. Select sum(res) FROM ( SELECT fromid, table2.received, Sum(MERIT) as given, (received/2 - Sum(MERIT)) as res FROM MeritData INNER JOIN ( SELECT toid, Sum(MERIT) as received FROM MeritData WHERE toid in (SELECT userid FROM userdata ) GROUP BY toid ) as table2 ON table2.toid = fromid WHERE fromID in (SELECT userid FROM userdata ) GROUP BY fromid Having received/2 - Sum(MERIT) >=0)
Btw the answer is 11846 This Sql above doesn't work as intended unfortunately, must have got something wrong ![Cry](https://bitcointalk.org/Smileys/default/cry.gif)
|
|
|
|
|