[BPIP] Bitcointalk Public Information Project [Back in Action]

[Bitcoin_Arena]

I don't know if I have not been so observant in the past but lately there seem to be so many profiles just woken up showing as "inactive" or "archived" several minutes or hours later after posting

Like this user: https://bpip.org/Profile?id=1286231



Is it normal?

[1714175670]

1714175670

[1714175670]

1714175670

[1714175670]

1714175670

[1714175670]

1714175670

[1714175670]

1714175670

[LoyceV]

Is it normal?

Does this mean BPIP doesn't scrape all new posts? If that's the case, I can easily share a list of all users that posted "today". I could provide historic data too: I have this data starting May 23, 2019 already (I needed it to keep remove unbanned users from the list of banned users).

[ibminer]

~
Is it normal?

Does this mean BPIP doesn't scrape all new posts? If that's the case, I can easily share a list of all users that posted "today". I could provide historic data too:
~

Thanks, and no not normal. As of yesterday the recent posts job had been locked for around ~26 days.. so the job hadn't been running on the parsers, picking up those posts, and updating the status of those users. I cleared the lock so it's been running correctly now for new posts, but I may need some time to figure out a way to either re-process posts through the parsers in that timeframe, or find some way to manually update. If any users continue to post, it should at least update the status on their next post.

[LoyceV]

or find some way to manually update.

If it helps: loyce.club/other/tmp/ibminer/ (scheduled to be deleted in 30 days) lists all userIDs who posted on the past 29-ish days. Note that the files contain duplicates if users posted more than once.

[LoyceV]

I saw this post:

he is in 85th place (of 3529391 members) among the top spammers in terms of the number of deleted posts by moderators.

And indeed, bpip.org/Profile?p=MFahad shows:

Code:

Posts deleted by moderators: 372 (Rank: 85th)

However, bpip.org/Report?r=postsdeleted shows:

Code:

85 	MFahad 	164

I checked a few other profiles in the list, and for them, the data is consistent. How did MFahad go from 164 to 372?

[ibminer]

~
I checked a few other profiles in the list, and for them, the data is consistent. How did MFahad go from 164 to 372?

Thanks, I'm seeing inconsistencies for many users as I spot checked, not just MFahad. Some are off by a few (Fredomago, cvm, Alex-007, etc.), some are off by larger amounts (joshy23, Adriano2010, JollyGood, etc), and a lot seem to match up correctly.

Trying to wrap my head around this one as I haven't really looked at this job (and related procedures) recently in the past, but there does appear to be a problem here which will need some troubleshooting which I can't do at the moment, I'll try to take a closer look as soon as I can.

[Timelord2067]

https://bpip.org/metalog.aspx?d=2023-04-25

Meta Log Report
(2023 - 04 - 25)

When were the first entries recorded by BPIP.org ?

[Bitcoin_Arena]

When were the first entries recorded by BPIP.org ?

Depends on the information you are looking for, but as far as I know. The log kept getting modified over time

The earliest records you will find are of most merited posts daily, and then Changed to Archived status, mostly because data would be displayed on the site even when Vod was still part of the project(I see some info as far as 2018)

[Timelord2067]

Hey thanks for that. I had a feeling it was going to be a long shot.


I see a couple of people have been discussing the disparigy between bpip and other sites' information (in the above example numbers of posts deleted).  I recently had someone suggest I'd only had 33 posts deleted, but the thread was locked before I could point out both loyce and ninja's sites have my post counts hundreds of posts higher than what bpip says with the Forum's user profile page only showing the number of posts that *haven't been* deleted.

The other two sites are much closer to the count, but probably don't include those posts of mine that were deleted in the first four or five years after registration.

[LoyceV]

I recently had someone suggest I'd only had 33 posts deleted

I've seen BPIP overlook deleted posts from modlog, so this counter could be too low.

both loyce and ninja's sites have my post counts hundreds of posts higher than what bpip says

BPIP only counts posts deleted by Mods, I scrape all posts including the ones you delete by yourself later on. Judging by my own data, I have deleted about 2000 of my own posts (mostly bumps).

The other two sites are much closer to the count, but probably don't include those posts of mine that were deleted in the first four or five years after registration.

Correct.

[Avirunes]

I've some questions regarding BPIP security/moderator log. Does it shows the full list of security and modlog till date? or is there a point of date? I am checking a profile and just trying to make sure of something.

Also is there a way where I can see full merit history of a profile?

[LoyceV]

I am checking a profile and just trying to make sure of something.

If it helps: LoyceV's seclog log.

Also is there a way where I can see full merit history of a profile?

There's loyce.club/Merit/history/175302.html.

[Avirunes]

I am checking a profile and just trying to make sure of something.

If it helps: LoyceV's seclog log.

Also is there a way where I can see full merit history of a profile?

There's loyce.club/Merit/history/175302.html.

Ah LoyceV's tools. That will for sure do. Thanks a lot 

[ibminer]

Deleted posts from mods value & rank:
To provide some follow-up on my prior post:
Looking into the counts & ranks closer here for this particular report, I was able to narrow down what was happening, but still need to fix it.

As it stands now:
a) The profile page is getting its value/count from a table/field which is being updated by parsers.
b) The report page is getting its value/count from a table/field which is not being used or updated by parsers.
c) Both pages are pulling their rank from the same table/field, which unfortunately has not been part of the stored procedure which calculates and updates ranks.

Because of this, the visible result is:
The report page for deleted posts by mods hasn't been updated at all (in terms of value/rank), for some time they've been static.
You can compare an archive from 2020 with a recent archive from the end of March and see the value & ranks haven't changed.

For the profile page, the value has been changing, as can be shown checking the archives of someone like JollyGood (current & archive from Feb 2021 and June 2022).
But the rank has remained static on the profile page as well, again due to the current table not being included in the rank calculations.

All this said, there is a chance for potential small gaps in capturing the modlog/seclog in the past, if there were parser issues or downtime occurring, which occasionally happened more in the earlier days of development then it would now, but I'd expect the count shown on the profile page is close to what it should be, based on what has been collected from the parsing since 2018.

I've some questions regarding BPIP security/moderator log. Does it shows the full list of security and modlog till date? or is there a point of date? I am checking a profile and just trying to make sure of something.

Here is what I'm seeing:
Seclog on BPIP goes back to March 16th, 2018.
Modlog on BPIP has ~1.3M saved records with an accurate date/time which ranges from August 3rd, 2018 until today. The modlog also has 177,836 records with a (default) unix timestamp from 1970. Based on the ID's of these records without a timestamp, they were all created (assumingly in order) before the range of time above. I didn't go back and look but the lack of a timestamp on these were likely related to a bug which was ultimately fixed in August of 2018, when it started capturing the date/time correctly.

Also is there a way where I can see full merit history of a profile?

Currently, you'd only be able to see:
Full sent: https://bpip.org/smerit.aspx?from=Avirunes
Full received: https://bpip.org/smerit.aspx?to=Avirunes

But it might not be a bad idea to have a view showing them merged together for a profile, which shouldn't be too difficult. I'll add this to my TODO list.

[LoyceV]

Timelord2067 posted this in my topic, but it belongs here:

It looks as though BPIP hasn't updated in the last nearly a dozen days. theymos updated the DT lottery some fourteen hours ago by the looks of it and at least one of the organisers (of bpip) is now inactive if the website's Trust Log is anything to go by. 🤔

The last entry in BPIP's DT Change Log is from May 20.

[ibminer]

The last entry in BPIP's DT Change Log is from May 20.

Long story short, one of the tables involved with the TrustLog has max'd out its identity column.. because, for some reason, the table appears to have started out with a high seed value which has caused it to max out after only ~1.3M 11.3M rows, when it should be able to handle ~2.1B rows as an int. Possibly at some point SM had to wipe this table and the seed started off high at that point, but I'm not 100% sure.

Once I feel comfortable I won't screw anything else up here by resetting these IDs, I'll need to reseed the table, which should get this flowing again.

edit: After re-counting the rows in the latest re-seeded table, I realized I miscounted digits when calculating the original table, which was actually ~11.3M rows, not 1.3M like was in this post, just updating to keep things accurate.

[ibminer]

OK, /TrustLog should be flowing again and the ranks have been updated for most posts deleted.
To summarize the changes recently:

-Added 2023 option to Most Earned Merit report (by time)
-Adjusted column header for Most Earned Merit report from "Times Merited" to "Merit Received" to clear up confusion.
-Modified UpdateRanks SP to fix deleted posts report & ranks not updating
-Re-seeded one of the trust tables to get the /TrustLog operational again
Enjoy the music.

[Bitcoin_Arena]

There is a rise in the number of accounts still carrying the inactive badge even after they have woken up and posted like
- https://bitcointalk.org/index.php?action=profile;u=2576340
- https://bitcointalk.org/index.php?action=profile;u=3427812

Perhaps this is the effect of the recent Cloudflare chaos. Just asking the team to check and see if there is a problem.

[ibminer]

Perhaps this is the effect of the recent Cloudflare chaos. Just asking the team to check and see if there is a problem.

Thanks for the alert, there does seem to be a problem here. I'm not caught up on the Cloudflare chaos, but both parsers are currently unable to login to the forum at the moment. I'm not entirely sure why yet, and (unfortunately) have a limited amount of time tonight to look.. but we'll try to figure out ASAP.

Also to note, this is why the BPIP logo has stopped spinning on the site, it stops spinning if the database is up but parsing is not happening for some reason.. and it turns red if the database itself is down.

Yesterday I did the statistics of most-merited users among users of the Russian locale and found one error in BPIP:
~

Haven't had time to address this or figure out why it is inaccurate, but there does seem to be a discrepancy here.

Sorry to both - out of sMerit, I'll get you guys later!


edit: Getting a little caught up now, smh.. there appears to be a forum-wide issue with parsers at the moment apparently being caused by CloudFlare.  

edit #2: Other parsers like TryNinja & LoyceV were whitelisted yesterday... message has been sent to theymos.

[ibminer]

^ BPIP's parsers are good again. Thank you theymos!