[BPIP] Bitcointalk Public Information Project [Back in Action]

[joker_josue]

Hello, I just discovered this website and have some questions, I'll appreciate if you answer me. P.S. I am not a developer but would love to know a thing.

1. As I have searched, bitcointalk doesn't have an official API, so how do you get data from the website? Are you scrapping?
2. If you are scrapping, doesn't it slow down the website? Doesn't it increase their server load? If yes, then why do admins allow you to do so?
3. What programming languages and frameworks (if any) are used on this project?

Doesn't it increase their server load?

Every page load (slightly) increases server load.

If yes, then why do admins allow you to do so?

Probably because of this:

the forum's mission to be as free as possible.

It's rate limited though: a maximum of 1 page per second on average.

Scraping programs normally use an interval of at least 2 seconds by default.

For this type of statistics sites, I take into account that there is no need for an update every second, it is enough to define the collection of information from minute to minute for most cases. And there are still cases where some information has already been collected by other users and therefore can be used without having to scrape again.

In the midst of all this, the only thing that influences is the statistics of the number of views/pages viewed, many of which are done by bots. But the forum is already at a level that this information turns out to be of little relevance. As long as the numbers of bots remain minimally acceptable.

[1714235668]

1714235668

[1714235668]

1714235668

[1714235668]

1714235668

[1714235668]

1714235668

[Little Mouse]

I don't know if this has been suggested or not. I was looking for threads with the most merits on some boards through the help of BPIP extension but at some point, I got bored as I can't sort it. I'm not also aware of any other working method. However, as the screenshot says, all the option marked has clickable link behind the text so that one can sort according to asc/desc order. It makes it easier to find out the most for example viewed topics on that board.
If the "merit" button next to "Views" can also have the same feature, I think it would be very much helpful. Of course, if the "Merit" button is from BPIP extension  

[suchmoon]

If the "merit" button next to "Views" can also have the same feature, I think it would be very much helpful.

The problem is that sorting is done by the Bitcointalk server, and our extension works entirely inside the user's browser. So there is no easy way to implement sorting by merits. We could sort the current page but that doesn't help and might even be misleading, since the threads on other pages might have more merits. A more robust option would be to keep a list of threads on our (BPIP) server or perhaps use TryNinja's site to grab all threads for the board, but then it's complicated to display the result with all the other columns (i.e. last post etc).

[GazetaBitcoin]

I noticed recently three inconsistencies on bpip.org. I'm not sure if these were ever discussed, but I don't remember seeing these issues ever raised before. If they were mentioned earlier I apologize.

1. For some reason, while looking at my profile, I see a Sent/Received merits ratio of 1.08. This is impossible, for two reasons: (1) I am not a merit source and I did not receive any airdropped merit, therefore I can't send more than 50% of the merits I received, thus my max ratio should be 0.5; (2) the merits shown are correct -- 1604 sent and 3335 received -- and, if we divide 1604 by 3335 we obtain the score of 0.48. However, the "S/R Ratio" shows 1.08.



2. If I click on "Refresh Profile" a small window pops for a short time, showing "Most merited posts in the last 24 hours". However, I don't think that the posts shown there are also correct. For example, let's look at the fifth post written -- the one made by Hellena Yu, which appears to have received 5 merits. It's hard to believe that this is the fifth most merited posts within last 24 hours.



3. Still at "Most merited posts in the last 24 hours" window, I also observed that the merits shown there are incorrect. For example, Hellena Yu's post appears as having received 5 merits while, in fact, the post has 13 merits. Ddmr's post appears as having 10 merits while, in fact, the post has 29 merits and so on.

I hope this helps.

[ibminer]

I hope this helps.

Thank you, if I had any sMerit, I'd send some.  

So I may have initially misinterpreted what this report was pulling a while back when creating the "info" box on it. It looks like this report is actually calculating the number of times a post was merited, which is apparently why the column was named "Times Merited", not total merit received, so the wording here may be confusing. Currently the Helena Yu post "Cybersecurity subforum" is showing 6 on the report page (link is for last 48 hours, since I'm a day behind on your post   ) - this report would offer the same data that this popup window pulls from, in a longer form. 6 because there are 6 users who have sent merit to it. Not sure if this deserves a new report or if we should change this one to actually count the total merit received?

It doesn't appear to be unique users though, as it is showing '19' on this post, but I'm only counting 18 people.. so I'm guessing the same person merit'd 2 times which is giving the 19.
edit: ^ Actually, there are more than 1 users who have given merit more than once so I think the inconsistency here may have been me not refreshing the page but the parser picking up another merit'er... or the merit happened before 48 hours.. so I do think it is supposed to be unique users, but I haven't delved back into the code to check on this one yet, just my observation.

[suchmoon]

Not sure if this deserves a new report or if we should change this one to actually count the total merit received?

I think it should be the actual total merit, or maybe both.

I'd say we can start by changing the current value to be merits (not senders), that should be a database-only change. Then we can decide if we want a column for unique senders.

I'm still looking into the S/R issue.

[GazetaBitcoin]

Thank you, if I had any sMerit, I'd send some.  

Hehe, thank you anyway

It looks like this report is actually calculating the number of times a post was merited, which is apparently why the column was named "Times Merited", not total merit received, so the wording here may be confusing.

Oh I understand now... it figures.

Not sure if this deserves a new report or if we should change this one to actually count the total merit received?

I think this is up to you and suchmoon... After all, you are the ones working on this website and only you know how much work such change would need Either way, at least we know now what those numbers represent.

And regarding the other two issues I raised (the ones from 1. and 2.) do you have any news?


Excepting that, I observed one more inconsistency, as it follows: sometimes when I click on "Refresh Profile" button, the "profile's numbers" update on the profile page, but not also through the site. By "profile numbers" I mean, for example, the number of earned merits -- so this number is updated. However, if I rank up a few positions on the ladder of "Most merit" or "Most earned merit", this rank up is not reflected on the website, nor on the upper part of the profile page (the one with badges). I will try explain this step by step, in case it's easier to understand.

- step 1: I know that I have 3411 merits and I am the 57th user with most merits; let's suppose that the user from 56th position has 3412 merits. At same time, I am also the 45th user with most earned merits. Let's suppose that the one from 44th place has 3413 merits. This information is properly shown on the profile page:



- step 2: I receive 3 merits. At this point, I know (in my mind) that I have more merits than the user which was earlier on 56th place on Most Merit ladder and also than the one from 44th place from Most Earned Merit ladder
- step 3: I click on "Refresh Profile" and the new number of merits is now shown on my bpip profile (under "Merit" and "Earned" sections)

Result: although the correct number of merits is shown on bpip profile page, my new achieved positions on both ladders (Most Merit and Most Earned Merit) -- meaning 56th most merited and 44th user with most earned merits -- are not updated. The badges from the upper part of the page are also not updated and they still show that I am on 57th place (instead of 56th) on Most Merit ladder and 45th on Most Earned Merit ladder (instead of 44th). If I access both ladders (https://bpip.org/Report?r=mostmerit and https://bpip.org/Report?r=earnedmerit), I am also shown as the 57th / 45th (and with the number of merits not refreshed), instead of 56th / 44th.

It's like the refreshed number of merits is reflected only on the bpip profile page and does not propagate through the rest of the website. I am not sure if same happens also with the other badges / ladders.

Of course, after a while the refreshed information is propagated through the entire site. I just wanted to emphasize that it does not happen after "Refresh Profile" button is pressed.

[gunhell16]

If here in our country the record of people born is recorded in the census, when they were born, day, date, month, year, place, and who the parents are.

Here in the forum, this BPIP is like a census, where you can see the detailed information of all the members here in this forum platform when it was done in this forum, when it was active and when it was not active and we will also know if there is any improvement in their accounts via merit within 90 days, 120 days or 150 days. And we will also see who are the top merits and so on.

This is a good place if we have any members who want to investigate here as well.

[ibminer]

And regarding the other two issues I raised (the ones from 1. and 2.) do you have any news?

#2 would be seen as the same issue as #3, the popup window just shows a smaller set of the data, so both arewere showing number of meriters, not the total merit received. -- But the DB was modified last night, so both the popup window and mostemerit.aspx reports should now be displaying the total merit received (and column name/header text will be updated in the near future)
#1:

~I'm still looking into the S/R issue.

It's like the refreshed number of merits is reflected only on the bpip profile page and does not propagate through the rest of the website. I am not sure if same happens also with the other badges / ladders.

The general stats on a profile page will get updated instantly when a refresh profile happens, as this updates the main profile table in the DB... but ranks are done through a procedure at regular intervals to all top users, and it would include all badges getting updated at once. For resource/performance purposes, allowing a refresh profile button to recalculate the ranks of thousands of users might not be worth it.

[LoyceV]

While on the subject of small improvements:

Shouldn't the Least Trusted counter go deeper than -2^21? There's a new contestant (currently #5) who will very soon have the exact same value as the Top 4.
I know it's nitpicking, but may the worst of them "win"!

[ibminer]

I'm still looking into the S/R issue.

Took a closer look at this myself and there is not really an "issue" per se, unless it makes more sense to change it, but similar to the Most Merited report, currently the ratios are calculated using unique send/receivers and not the sum of merit:


So in the case of GazetaBitcoin, currently it ends up being:
(277 times merit was sent to unique/distinct users) / (231 times merit was sent from unique/distinct users) = which is where the 1.20 (rounded) ratio comes in.

I know it's nitpicking, but may the worst of them "win"!

Part of me sees what you are saying, and part of me feels like there shouldn't be a 'winner' in the sense that currently they all end up at the same dead-end of shame. I feel like some of these guys might take some sort of pride in being worse then another, and I'd rather not cater to it, but that's just my opinion.

[JayJuanGee]

I know it's nitpicking, but may the worst of them "win"!

Part of me sees what you are saying, and part of me feels like there shouldn't be a 'winner' in the sense that currently they all end up at the same dead-end of shame. I feel like some of these guys might take some sort of pride in being worse then another, and I'd rather not cater to it, but that's just my opinion.

Interesting concept ibminer.

In accordance with your preference not to rank (which there is already a kind of ranking), maybe instead list the top 50 or something like that by alphabetical order or some other criteria that just puts "them" in the group of "bad boys (or girls)" but does not subdivide within the group... I just noticed that I had using the word "them."  hahahahahaha..

I am being a bit judgmental depending on how BIG you might make the group or what criteria that you might chose to use.  All of us might have a streak of "bad boy/girl" contained therein, whether we acknowledge it or not.

[ibminer]

In accordance with your preference not to rank (which there is already a kind of ranking), maybe instead list the top 50 or something like that by alphabetical order or some other criteria that just puts "them" in the group of "bad boys (or girls)" but does not subdivide within the group... I just noticed that I had using the word "them."  hahahahahaha..

I'm not saying I have a strong preference to not rank the top least trusted, I just don't mind the negative ranks stopping at a dead-end.. which already sort of creates a group at the dead-end of currently 3 accounts.

Nobody is perfect, but you know you don't have to be a bad boy or girl to be cool, right JJG?  

Regardless, based on how BPIP stands now, the bad list would probably be deemed as the top 100 least trusted, based on their feedback score (which can be sorted on the "Trusted" column on the main page).

[joker_josue]

I would like to know if you can get me a list of all users who were active in 2022.
Or else a link where you have that information.

[ibminer]

I would like to know if you can get me a list of all users who were active in 2022.
Or else a link where you have that information.

Are you trying to go by the 'last active' date of a profile, or if an account made a post in 2022?  It might be more accurate with the latter for BPIP, since a parser wouldn't check inactive profiles for the 'last active' date (unless a manual refresh is done), it is ultimately looking for an account to make a post... so there could have been profiles that were inactive but logged in at some point in 2022, but if they didn't make a post, BPIP likely wouldn't have updated the 'last active' field unless manual refreshes were being done on specific accounts. I may be able to query for a list of every user who made a post in 2022 but I'm not sure if that's what you are looking for or not.

[joker_josue]

Are you trying to go by the 'last active' date of a profile, or if an account made a post in 2022?  It might be more accurate with the latter for BPIP, since a parser wouldn't check inactive profiles for the 'last active' date (unless a manual refresh is done), it is ultimately looking for an account to make a post... so there could have been profiles that were inactive but logged in at some point in 2022, but if they didn't make a post, BPIP likely wouldn't have updated the 'last active' field unless manual refreshes were being done on specific accounts. I may be able to query for a list of every user who made a post in 2022 but I'm not sure if that's what you are looking for or not.

Yeah, sometimes it's hard to define what an active user is, one that regularly visits the forum or one that posts.

What I would like to get is a list of users who made at least one post in 2022 and received merits. In short, as long as I can get a list of users who have made at least one post, that's enough for me. Think you can help?

[LoyceV]

What I would like to get is a list of users who made at least one post in 2022 ~. In short, as long as I can get a list of users who have made at least one post, that's enough for me.

I can do that
I have data on all users, and it's easy to check which files have been edited (meaning: they made a post) since January 1st, 2022 (Amsterdam time, this includes users active in 2023): see this file. If you don't want to include banned users, see this file.
If you ever want an update: remind me of this post:

Code:

find -type f -newermt '2022-01-01' | cut -d'/' -f3 | cut -d'.' -f1 | sort -n > ~/loyce.club/other/tmp/joker_josue.txt
comm -23 <(sort ~/loyce.club/other/tmp/joker_josue.txt) <(sort ~/loyce.club/bans/banned.txt) | sort -n > ~/loyce.club/other/tmp/joker_josue_not_banned.txt

[ibminer]

I can do that
I have data on all users, and it's easy to check which files have been edited (meaning: they made a post) since January 1st, 2022 (Amsterdam time, this includes users active in 2023): see this file. If you don't want to include banned users, see this file.
If you ever want an update: remind me of this post:

Code:

find -type f -newermt '2022-01-01' | cut -d'/' -f3 | cut -d'.' -f1 | sort -n > ~/loyce.club/other/tmp/joker_josue.txt
comm -23 <(sort ~/loyce.club/other/tmp/joker_josue.txt) <(sort ~/loyce.club/bans/banned.txt) | sort -n > ~/loyce.club/other/tmp/joker_josue_not_banned.txt

Thank you Sir!

I ended up running some queries anyway, numbers didn't match up with your data because it looks like there are users in your list who have not made a post, and BPIP's list would not include anyone in 2023. I kept the queries simple but this is what I'm seeing:

LoyceV's
https://loyce.club/other/tmp/joker_josue.txt -- (55,937 results)
some examples of users with no posts (28452, 14624, 392646, 28877)

BPIP
https://bpip.org/active_in_2022.txt -- (51,791 results)
https://bpip.org/active_in_2022_and_received_merit_in_2022.txt -- (5,300 results)

[LoyceV]

some examples of users with no posts (28452, 14624, 392646, 28877)

I think you made a mistake on AverageGlabella, who made many posts (and is listed in your list).
The other 3 users posted spam/malware (1, 2, 3), and MindlessElectron must have gotten to them before your scraper saw them. They're also banned, so it doesn't matter.

[TryNinja]

I also want to join the party…

I got 52593 users by checking distinct users from all posts on my database >= 2022-01-01 and 2023-01-01 <

Idk why it’s a lot lower than LoyceV’s.

https://public.ninjastic.space/users-2022.txt