Announcing BPIP - The Bitcointalk Public Information Project!

[suchmoon]

I have moved all the profiles into one table.  Made a lot of things easier.  However, it is increased CPU processing significantly.

Still can't reconcile merit records.   My profile says I am at 1860, but only have the sum of 855 in the merit log.

I have tested with LoyceV's full merit set, and mine as well.  Same results.  :/

Piggy's SQL tool shows 859:



That's accurate because it doesn't include LoyceV's last merit:



Maybe you can try Piggy's data: https://bitcointalk.org/index.php?topic=5046627

Edit: never mind, I checked LoyceV's data and it adds up to 859 excluding that last merit so it's identical to Piggy's data.

[ibminer]

So it looks like these are the ones missing from the merit log for Vod?  

2018-11-20T20:08:24        11/20/18    20:08:24    372243 (sirazimuth)   30747    1    Flat Earth    1009045.msg48082280 (deleted?)
2018-11-20T05:03:07        11/20/18    5:03:07    81292 (Stunna)    30747    4    How recognized are you in the bitcointalk world?   4799468.msg43295993

[LoyceV]

Still can't reconcile merit records.   My profile says I am at 1860, but only have the sum of 855 in the merit log.

I have tested with LoyceV's full merit set, and mine as well.  Same results.  :/

For convincence, since I haven't received any new Merit^** since theymos' last data dump, I'll compare data on LoyceV

https://bitcointalk.org/index.php?action=profile;u=459836 : 2280 Merit
My own Merit analysis: 1280 Merit received
https://bpip.org/profile.aspx?p=loycev : Current Merit 2280; Merit Received 1270

http://loycevsbasement.privatedns.org/Merit/history/459836.html : my full data on all Merit transactions
https://bpip.org/smerit.aspx?to=LoyceV : Vod's full data on received Merit transactions

From what I can find, BPIP misses these lines:
Tue Nov 20 13:32:53 2018: 10 from suchmoon (history)for [overview] Recover Bitcoin from any old storage format
Mon Sep 17 21:12:04 2018: 1 from suchmoon (history)for Re: Community generated suggestions to improve the forum (+ eventual voting on them)
When I add up all the Merit shown on https://bpip.org/smerit.aspx?to=LoyceV , I get 1269.

My conclusion: BPIP misses 2 merit transactions, and somewhere 1 Merit was counted twice.

^**Update: I received Merit between the moment I started typing this post, and posting it. Ignore the values currently shown on BPIP and my profile for my analysis of the differences

Update (2): If it helps, the post from September 17 was merited by suchmoon twice:
Wed Sep 19 18:40:20 2018: 3 from suchmoon (history)for Re: Community generated suggestions to improve the forum (+ eventual voting on them)

[suchmoon]

^**Update: I received Merit between the moment I started typing this post, and posting it. Ignore the values currently shown on BPIP and my profile for my analysis of the differences

LOL sorry for messing it up.

So there is some issue with Nov 20 merits. I checked my own history and BPIP is missing the merits I received from TMAN on Nov 20.

[asche]

^**Update: I received Merit between the moment I started typing this post, and posting it. Ignore the values currently shown on BPIP and my profile for my analysis of the differences

LOL sorry for messing it up.

So there is some issue with Nov 20 merits. I checked my own history and BPIP is missing the merits I received from TMAN on Nov 20.

You appear to be right:

11/21/2018 10:43:33 AM   tactac   asche   1   Re: How to reveal spammers who pretending to be translators of native speakers
11/18/2018 1:32:00 PM   Saint-loup   asche   1   Le forum fran?ais et le merit

I'm missing 5 merits from november 20th as well.

[Vod]

LoyceV - thank you for all your help on this.

I believe merit has been fixed now. 

As a thank you, I give you this reward.  You can easily parse it daily by using the d variable for the metric date.

https://bpip.org/metalog.aspx?d=2018-11-25

[LoyceV]

As a thank you, I give you this reward.  You can easily parse it daily by using the d variable for the metric date.

https://bpip.org/metalog.aspx?d=2018-11-25

I like it And I only have one question left: when did you start keeping track (ie. when do I start scraping data)?

[asche]

I like it And I only have one question left: when did you start keeping track (ie. when do I start scraping data)?

I was actually looking for this yesterday. Awesome Vod thanks.

[weblife]

I am happy and proud to announce the official launch of the Bitcointalk Public Information Project!

This project aims to track and record all forum activity of all users.

We currently track the following:
- Full Merit History (updated every 60 seconds)
- Security/Mod log (updated every 60 seconds)
- All other profile info is updated on a dynamic schedule based on your recent activity.

Some features are not yet added, such as the ability to sort searches.  Additional pages such as Merit History will be added soon.

THIS PROJECT HAS JUST STARTED.  In the following weeks I will be adding real time tracking for Default Trust membership.  I also plan to let users add profiles to custom lists.  This will be useful if you want to (example) track profiles to not send merit to, or if you run a bounty campaign. 

If you can think of anything else you would like to see tracked, please comment here or click on the SPONSOR link to contact me.

https://bpip.org

Very nice Project that sounds good,

[Vod]

As a thank you, I give you this reward.  You can easily parse it daily by using the d variable for the metric date.

https://bpip.org/metalog.aspx?d=2018-11-25

I like it And I only have one question left: when did you start keeping track (ie. when do I start scraping data)?

I can send you a username of all ids that were archived before today (2018-11-26).  I just need a bit of time to verify everything.

Starting 2018-11-27, you can access your first full day of data.  The previous day will not change.

Time zone is default forum time zone - UTC.

[LoyceV]

I can send you a username of all ids that were archived before today (2018-11-26).  I just need a bit of time to verify everything.

I can scrape it, but was looking for the start date. I found it already: https://bpip.org/metalog.aspx?d=2018-08-03 seems to be the oldest version. I was hoping you would have much older data from modlog.

Starting 2018-11-27, you can access your first full day of data.  The previous day will not change.

Time zone is default forum time zone - UTC.

My plan is to create a list banned.html and update it daily from the VPS. To make my life much easier: can you add the userID to the daily list? I could scrape the names from BPIP again, but some usernames with weird characters make that unreliable.

[Vod]

I can send you a username of all ids that were archived before today (2018-11-26).  I just need a bit of time to verify everything.

I can scrape it, but was looking for the start date. I found it already: https://bpip.org/metalog.aspx?d=2018-08-03 seems to be the oldest version. I was hoping you would have much older data from modlog.

Starting 2018-11-27, you can access your first full day of data.  The previous day will not change.

Time zone is default forum time zone - UTC.

My plan is to create a list banned.html and update it daily from the VPS. To make my life much easier: can you add the userID to the daily list? I could scrape the names from BPIP again, but some usernames with weird characters make that unreliable.

I do have much older data from the modlog - but historical entries do not have a corresponding record in the meta log.  I can fix this.

However, rather than you reparsing thousands of days worth of people being archived, I would prefer just to send you the list.

I use the same profile ID that bitcointalk uses.  The name they have for their ID will be the same as mine. 

[LoyceV]

I do have much older data from the modlog - but historical entries do not have a corresponding record in the meta log.  I can fix this.

However, rather than you reparsing thousands of days worth of people being archived, I would prefer just to send you the list.

If you can just send me the raw modlogs, I'll get the banned accounts from there. That's probably the least amount of work for you.

I use the same profile ID that bitcointalk uses.  The name they have for their ID will be the same as mine.  

I know, but BPIP doesn't show the profile IDs on metalog. That makes it much easier to link them to my big lists of forum abusers.

[Vod]

I do have much older data from the modlog - but historical entries do not have a corresponding record in the meta log.  I can fix this.

However, rather than you reparsing thousands of days worth of people being archived, I would prefer just to send you the list.

If you can just send me the raw modlogs, I'll get the banned accounts from there. That's probably the least amount of work for you.

I use the same profile ID that bitcointalk uses.  The name they have for their ID will be the same as mine.  

I know, but BPIP doesn't show the profile IDs on metalog. That makes it much easier to link them to my big lists of forum abusers.

I'm just processing all the "wake up" records in the seclog I missed.  Once the database is up to date, I will update the page and give you what you need. 

[eternalgloom]

I've noticed that there are merit abusers showing up in the rankings on BPIP.
Would there be some way to remove them?

Sure they might technically have those merits, but I personally think they pollute the rankings.

From 'Most Merited' rankings:





https://bitcointalk.org/index.php?action=profile;u=911155
https://bitcointalk.org/index.php?action=profile;u=2189580

They've been banned from Bitcointalk, so no reason to leave them in the stats, right?

[Vod]

I've noticed that there are merit abusers showing up in the rankings on BPIP.
Would there be some way to remove them?

Sure they might technically have those merits, but I personally think they pollute the rankings.

From 'Most Merited' rankings:





https://bitcointalk.org/index.php?action=profile;u=911155
https://bitcointalk.org/index.php?action=profile;u=2189580

They've been banned from Bitcointalk, so no reason to leave them in the stats, right?

I don't want to start moderating the lists.   They will drop off the top lists in time.

You can also view only the active members by adding &a=1 to the report.
https://bpip.org/report.aspx?r=mostmerited&a=1

[Vod]

On another note, I had trust tracking all ready to go, but this new system resulted in thousands of changes in mere seconds.  I need to rework how I am going to do this.

[eternalgloom]

@vod

Yeah, a couple of minutes after writing that reply, I was actually thinking that there's actually good reason to keep the list objective.
In the end, I actually agree with you to keep the list as is -- unmoderated.

I'll just make a mental note to myself in the future, to rank myself a place higher on the 'most merited' list

[Vod]

New report launched:

Most trust sent - order by positive, negative, total or attitude (positive - negative)

https://bpip.org/r/trustsent.aspx

https://bitcointalk.org/index.php?topic=5096116.0

[Veleor]

There is a topic on the forum "List of Autobans".

After checking data there and at BPIP.org I've discovered that many banned members haven't correct status in the 'Security/Moderator Log' column, e.g.:
https://bpip.org/profile.aspx?p=Liang_Kubur
https://bpip.org/profile.aspx?p=Bitcoin0916
https://bpip.org/profile.aspx?p=Vivace92
https://bpip.org/profile.aspx?p=President79
https://bpip.org/profile.aspx?p=dEjAvU97
...