[BPIP] Bitcointalk Public Information Project [Back in Action]

[ibminer]

We just have to request Theymos to active it back instead to scrape all the data again. If it's done trough the forum software it will be using less resources if for example use a data scraper only for this. The servers will be still on load anyway.

I'm not 100% sure what theymos' stance on it is, the question has been asked a few times in meta over the years but I've assumed, based on the reason it was disabled, that it is too dangerous to re-enable due to potential abuse. He may rather have parsers hitting public pages (which are restricted to 1 parse per second) than enabling a direct DB query that might take a good bit of resources in one click, allowing abusers to repeatedly hit the largest user stats page over and over again.

Although, CloudFlare's caching/protections might help with that these days.. not sure.

[1713587445]

1713587445

[1713587445]

1713587445

[1713587445]

1713587445

[1713587445]

1713587445

[1713587445]

1713587445

[1713587445]

1713587445

[suchmoon]

We just have to request Theymos to active it back instead to scrape all the data again. If it's done trough the forum software it will be using less resources if for example use a data scraper only for this. The servers will be still on load anyway.

Do you (or anyone) have a screenshot of the stats page or some other source as to what used to be on that page?

Or perhaps to approach it from a different angle: what info would you like to get?

We might already have some or most of it so it may not require much additional scraping, or we could show at least what we got.


BTW I fixed another DT2 bug courtesy of OmegaStarScream :wink:

https://bitcointalk.org/index.php?topic=5221970.msg53740521#msg53740521

[LoyceV]

Do you (or anyone) have a screenshot of the stats page or some other source as to what used to be on that page?

I remember the number of posts (and percentage) per board, number of poll-votes and a graph showing the distribution of activity per hour of the day

BTW I fixed another DT2 bug courtesy of OmegaStarScream :wink:

Do you have any method to get a userID with a username through an automated process?

[TheBeardedBaby]

I don't remember well, because the stats were disabled just a few weeks after i registered my accounts but i found this, someone has to confirm that its correct:
P. S. Talked to a guy who confirm that this is how it looled like before disabling this feature.


The source of the image : https://steemit.com/deeponion/@niuniu/bitcointalk-user-id

[suchmoon]

I don't remember well, because the stats were disabled just a few weeks after i registered my accounts but i found this, someone has to confirm that its correct:

I wouldn't have poll data (requires major scraping) or time spent online (impossible). I can get post and topic counts - totals, by time of day, by board - with some caveats, e.g. deleted/moved posts would not be accounted for. Not sure what the board by activity percentage is supposed to mean.

[Last of the V8s]

Not sure what the board by activity percentage is supposed to mean.

it's the number your post in a board divided by number of all posts in a board.
so if I have 1000 posts in meta and there are 100000 meta posts in total, it'll show as 0.1%.

hth

[Kakmakr]

I don't remember well, because the stats were disabled just a few weeks after i registered my accounts but i found this, someone has to confirm that its correct:



The source of the image : https://steemit.com/deeponion/@niuniu/bitcointalk-user-id

If I remember correctly, it also showed a graph of the areas where the users posted the most of his or her posts. So, you were quickly able to see where the user spends most of his or her time on the forum. This information highlighted if someone was posting too much in one section of the forum and what he or she is concentrating on with their posting habits.  

At one glance, you were able to summarize the kind of poster, without having to dive into their post history and having to go through 100's of posts to see a trend in his or her posting habits.

A Signature campaign manager can also quickly access the applicants post stats to see which users are posting in the sections that the advertiser wants them to post in.  

[Vod]

I wouldn't have poll data (requires major scraping) or time spent online (impossible).

Not really impossible with your extension.   You could add an option, or a button, that allowed your ext to read the total time online.  Of course, you have to be careful how you develop and announce it, so you don't conjur up fears of privacy invasion. 

Of course, I fully understand if you want to maintain a strict "no parsing" policy in the extension.

[Vod]

https://bitcointalk.org/index.php?topic=4746201.msg52576329#msg52576329

The site runs so fast now!  Suchmoon knows his stuff!

Finally figured out why it was running so slow as I was rebuilding my servers after the crash.   Turns out I had a test DNS server (Old PC - non-hp) I forgot to disable.   Thing is, I had mentioned to IB I thought it was a DNS issue right at the start, but priorities shifted and I never pursued it. 


(I think a double post is warranted since the topics are so different)

[ibminer]

There's a new "Support" page with a couple FAQs in place of "Changes" & a small bug fix to correct spaces/special characters on dt1changes.aspx (a more robust version of this page is coming soon.)

Finally figured out why it was running so slow as I was rebuilding my servers after the crash.   Turns out I had a test DNS server (Old PC - non-hp) I forgot to disable.   Thing is, I had mentioned to IB I thought it was a DNS issue right at the start, but priorities shifted and I never pursued it. 

DNS could cause some problems in an AD environment.. and like I mentioned back then, AD itself causes a bit of overhead on a web server. There's been several performance increases on the back-end, and front-end, since that prior "live" site before December, but the site is definitely much snappier now.

[Timelord2067]

Do you (or anyone) have a screenshot of the stats page or some other source as to what used to be on that page?

I remember the number of posts (and percentage) per board, number of poll-votes and a graph showing the distribution of activity per hour of the day

The first two Known alts threads have various screen shots (as well as links to archived pages) - the Mk III thread doesn't have any given the elapsed time after the feature was disabled.  A very handy tool - some users would be well and truly Butt Hurt ^TM they weren't an alt and the proof would be in the pudding as it were when the stats page would show (for example) both users were online in the same two hour period of the day over many weeks of activity...

Perhaps if you tried the SMF link at bottom of page and viewed their version or, find other forums that use the SMF system then you can glean an idea of the format used.


It would be an interesting exercise to compare users who aren't related and hadn't expected such a feature to be revived...

[suchmoon]

Heard the rumors about a new BPIP firefox extension, if you guys need anyone to test it, let me know

BPIP extension for Firefox (and Tor Browser) is ready for testing. Please check your PMs.

Chrome version should be coming soon but Google's approval process is quite stupid so not sure when.

[TryNinja]

BPIP extension for Firefox (and Tor Browser) is ready for testing. Please check your PMs.

Can I get a PM too?

[TheBeardedBaby]

BPIP extension for Firefox (and Tor Browser) is ready for testing. Please check your PMs.

Can I get a PM too?

Is definitely a nice looking piece of extension, with room for improvement and i will play with it tomorrow, and will come with comments here or on PM, it's up to suchmoon.
Its quite late here.

[suchmoon]

BPIP extension for Firefox (and Tor Browser) is ready for testing. Please check your PMs.

Can I get a PM too?

Done.

here or on PM, it's up to suchmoon.

Please PM for now, I don't want to clutter this thread. Eventually I think we'll have a separate thread for the extension.

[Timelord2067]

Am happy to test out the Firefox extension too.

Early days but am loving what I am seeing so far!

[examplens]

I am also interested to test Firefox extension.
thanks

I accidentally find this:
Here are two accounts with murat username Murat and murat. Only on BPIP, not on forum search.
I know that user iwantapony (or something like that) changed username to Murat
When a search on bpip, first is there everything regular there is showed this user Hero rank, but when you click on his username going to some newbie account details.

[suchmoon]

I accidentally find this:
Here are two accounts with murat username Murat and murat. Only on BPIP, not on forum search.
I know that user iwantapony (or something like that) changed username to Murat
When a search on bpip, first is there everything regular there is showed this user Hero rank, but when you click on his username going to some newbie account details.

Interesting. The lowercase one is this user apparently: https://bitcointalk.org/index.php?action=profile;u=252546 and it seems to have been renamed to "murat_" (with underscore at the end) to make way for Murat. BPIP doesn't automatically refresh archived (inactive for a long time or banned) profiles unless they wake up but clicking "Refresh Profile" manually should fix this.

[OmegaStarScream]

Interesting. The lowercase one is this user apparently: https://bitcointalk.org/index.php?action=profile;u=252546 and it seems to have been renamed to "murat_" (with underscore at the end) to make way for Murat. BPIP doesn't automatically refresh archived (inactive for a long time or banned) profiles unless they wake up but clicking "Refresh Profile" manually should fix this.

Wouldn't it be better if you access the profiles (on the search list) through the userID[1] instead of the usernames[2]? That should fix the problem as well.
I've also noticed that the search could be optimized a bit (unless this is done on purpose). If I'm not mistaken:

BPIP: WHERE userName LIKE 'StarScream%' (Find all names that starts with StarScream)
Forums: WHERE userName LIKE '%StarScream%' (Find any name that has 'StarScream' in any position)

I find that using the second way is better.
Also, I have a question regarding the banned accounts, are you getting them from the modlog? does that mean that any account that has been banned before BPIP won't show as 'banned'?

[1] https://bpip.org/profile.aspx?id=815246
[2] https://bpip.org/profile.aspx?p=murat

[suchmoon]

Interesting. The lowercase one is this user apparently: https://bitcointalk.org/index.php?action=profile;u=252546 and it seems to have been renamed to "murat_" (with underscore at the end) to make way for Murat. BPIP doesn't automatically refresh archived (inactive for a long time or banned) profiles unless they wake up but clicking "Refresh Profile" manually should fix this.

Wouldn't it be better if you access the profiles (on the search list) through the userID[1] instead of the usernames[2]? That should fix the problem as well.

We have no way of knowing which usernames have changed unless we re-fetch all ~3 million profiles. That's the main problem in this case. The link in the search results works fine, just the username is the old one. Once it gets updated, the link will have the new username and will still work.

I've also noticed that the search could be optimized a bit (unless this is done on purpose). If I'm not mistaken:

BPIP: WHERE userName LIKE 'StarScream%' (Find all names that starts with StarScream)
Forums: WHERE userName LIKE '%StarScream%' (Find any name that has 'StarScream' in any position)

I find that using the second way is better.

Unfortunately the cheap version of SQL Server that we use would do a full table scan on the '%StarScream%' query. We have some long-term plans to address this.

Also, I have a question regarding the banned accounts, are you getting them from the modlog? does that mean that any account that has been banned before BPIP won't show as 'banned'?

That's correct.