[BPIP] Bitcointalk Public Information Project [Back in Action]

[Aveatrex]

I can add a username to the file if it helps you. It would be UTF8.

Yes please, it would make things much easier especially now that I need to get additional info from other sources users are referenced by username.

That makes sense, I just checked some users that couldn't be found and all of them have 0 feedback. Which means that if an user can't be found in this csv it either means that the user has Black trust OR it's an incorrect username/bitcointalk profile url (I'm thinking of a scenario where a bounty manager have a list of users and a bounty hunter provides an incorrect url or username) so gotta differentiate the two cases. To do that I'll do additional checking using theymos trust dump or Loyce's viewer.

Thanks!

plus more-frequent-than-weekly updates. At least as far as the DT ratings are concerned. LoyceV obviously maintains non-DT trust lists, which we don't.

That's the biggest advantage of bpip imho. So for getting DT ratings gonna use bpip and to confirm 0 feedback users gonna use LoyceV's trust list. You guys service complement each other

[LoyceV]

Damn I forgot that to view profiles you need to be logged in

Although I've had some troubles with it, it works quite easily by feeding your scraper your browser cookie. My scraper hasn't been bothered by Cloudflare in a long time.

Hmm, maybe I should at first use this CSV to check the user's trust, if he's not in it, I will search for him in LoyceV's Custom trust list

I only check/publish Trust scores for users with or on a custom Trust list (and at least one post). I (usually) update the Trust scores on Thursday, and publish them on Saturday in the latest version of my Trust list viewer.

That's the biggest advantage of bpip imho. So for getting DT ratings gonna use bpip and to confirm 0 feedback users gonna use LoyceV's trust list. You guys service complement each other

I didn't realize until now that it's relatively easy to get DT Trust scores for all users, and I'm not sure if BPIP uses it already: by checking the sent feedback for all DT-members, you know exactly who has received DT-feedback. It's probably less than 1000 pages to scrape for an update, but it's more work to set it up. I'm currently scraping 13651 profiles per week for an update, so this can be improved.

[suchmoon]

Yes please, it would make things much easier especially now that I need to get additional info from other sources users are referenced by username.

I have added a column for user name, added column headers, and set it up to run around midnight UTC (give or take a few minutes depending on other tasks in the same batch).

https://bpip.org/all_trust.csv

I didn't realize until now that it's relatively easy to get DT Trust scores for all users, and I'm not sure if BPIP uses it already: by checking the sent feedback for all DT-members, you know exactly who has received DT-feedback. It's probably less than 1000 pages to scrape for an update, but it's more work to set it up. I'm currently scraping 13651 profiles per week for an update, so this can be improved.

IIRC we parse from the receiving end. Your idea seems faster.

[examplens]

I'm not sure you changed anything here, but it seems merit count again does not work well.
For example, user @suchmoon has earned "Merit Score   5497" but in merit information table stay "Has received 4495 ". There is a difference of 2 latest earned merit. I don't count merit assigned at the beginning.

[suchmoon]

I'm not sure you changed anything here, but it seems merit count again does not work well.
For example, user @suchmoon has earned "Merit Score   5497" but in merit information table stay "Has received 4495 ". There is a difference of 2 latest earned merit. I don't count merit assigned at the beginning.

Good catch, thank you. I had a parser crash yesterday and didn't clean up the job queue properly so the merit update got stuck. Should be updated within the next few minutes.

Under normal circumstances there can be a slight discrepancy between sent merits and total merits because those two pieces of information are scraped from two different places. It shouldn't last more than a minute or so.

[LoyceV]

Users with a newly created custom Trust list (such as protrader786) don't get a working link to my Trust list viewer yet.
Every week, I create a file user_ids.txt for this, but I don't think it gets processed.

[ibminer]

but I don't think it gets processed.

I've still been manually processing this file.. need to get back with suchmoon on adding a task to the server to do this for me but I just hadn't gotten to the recent update yet. It's good now. Thanks for the reminder!

[Vod]

What happened to the flags updates?

If I were Suchmoon, I would have locked everything down for the Sep 11 19th anniversary.   After all, we expect another attack on the 20th, but this year would have surprised us.  He was just being safe. 

[suchmoon]

What happened to the flags updates?

I've pinged ibminer, flag parsing is his baby. Something must've gone broken but I'm sure he'll pick it up and fill in missing flags.

If I were Suchmoon, I would have locked everything down for the Sep 11 19th anniversary.   After all, we expect another attack on the 20th, but this year would have surprised us.  He was just being safe. 

We won't let terrorism define us

Or to be more accurate, we need to set up monitoring to alert us when stuff like that breaks but that's item #214 on the TODO list.

[LoyceV]

we need to set up monitoring to alert us when stuff like that breaks

The monitoring is done by the users, in this case Ratimov. If it breaks and nobody notices, it's not needed

[TheBeardedBaby]

we need to set up monitoring to alert us when stuff like that breaks

The monitoring is done by the users, in this case Ratimov. If it breaks and nobody notices, it's not needed

True, I'm keeping an eye on the site as I'm almost every day checking things so if something fails I will let you guys know as i did it many times before

Out of curiosity, how many unique views the site has per day?

[ibminer]

Flaglog is running again, it has added new flags and is currently about 85% of the way through the weekly re-parsing of all flags... no big errors occurred here, just the parser losing connection to the DB and me not realizing it.  

The monitoring is done by the users, in this case Ratimov. If it breaks and nobody notices, it's not needed

Well, most of the reports pull into the profile page as general information/stats, so just because someone doesn't use the flaglog itself, they could be looking at the number of flags against a user... which will look normal in a situation where the parser has stopped parsing flags, so it might be good to at least have some sort of general alert that a particular parser has stopped, although, I agree it is probably not the highest priority right now... I'll try to check the parser log more often!

[Vod]

Or to be more accurate, we need to set up monitoring to alert us when stuff like that breaks but that's item #214 on the TODO list.

You use AWS services, right?
https://aws.amazon.com/cloudwatch/

[NotATether]

Or to be more accurate, we need to set up monitoring to alert us when stuff like that breaks but that's item #214 on the TODO list.

You use AWS services, right?
https://aws.amazon.com/cloudwatch/

The problem here is BPIP is running inside rented servers on AWS, and not across the über hyper-scaled AWS platform (correct me if I'm wrong). The Cloudwatch event somehow has to determine whether BPIP updating is broken and you can't do that without creating a middleware inside the server that collects BPIP diagnostics and send them somewhere like S3 where cloudwatch can access them.

From my development experience, large tasks like this are as laborious and boring to implement, especially if you're not getting paid to do it, as they are necessary and useful.

[suchmoon]

The problem here is all of BPIP's updating is happening within programs inside rented servers (I guess?). The Cloudwatch event somehow has to determine whether BPIP updating is broken and you can't do that without creating a middleware inside the server that collects BPIP diagnostics and send them somewhere like S3 where cloudwatch can access them.

From my development experience, large tasks like this are as laborious and boring to implement, especially if you're not getting paid to do it, as they are necessary and useful.

AFAIK CloudWatch has agents to gather info from places like database servers and log files although to be honest I'd prefer to stay off Amazon-specific stuff in case they go full evil and hike the prices up and force us to leave. So for monitoring I was thinking to use a third-party uptime service that could check (among other things) web server's heartbeat and basic status (OK/ERROR) of every important module. That would require just a simple web page so perhaps easier to set up than CloudWatch.

[NotATether]

AFAIK CloudWatch has agents to gather info from places like database servers and log files although to be honest I'd prefer to stay off Amazon-specific stuff in case they go full evil and hike the prices up and force us to leave. So for monitoring I was thinking to use a third-party uptime service that could check (among other things) web server's heartbeat and basic status (OK/ERROR) of every important module. That would require just a simple web page so perhaps easier to set up than CloudWatch.

I'm guessing that BPIP is running on ASP.Net Core, from inspecting the request headers sent by it. If that's true then there's a way to directly wire heartbeats to it: https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks?view=aspnetcore-3.1, though I admit that ASP.Net is not my strong suite so there isn't much else I can recommend.

You could also throw in some functionality that sends an email whenever a module goes down, so the error statuses don't go unnoticed.

[ibminer]

CloudWatch does seem like a bit more than is necessary, although I'm not familiar with it and have never used it. It probably makes sense to do what suchmoon is saying here, if I'm understanding correctly. We are already storing each job and a timestamp when it was last run, so maybe one page like 'modcheck?mod=flagparse' which checks the DB for timestamps on that particular job, and returns OK/Error based on the result?

I guess we could do the same thing with the logic we are using in the logo as well (like when DB is offline, the logo turns red, if the DB is up and the parsing has stopped, the logo is not spinning). We could just check the DB timestamps and if anything hasn't last run in... lets say 20 minutes to give buffer for any long-running jobs.. the logo spins but turns yellow, giving the indication something inside the parser is not running. Could get more specific here in the length of time and each job, but just a general thought. 

[Vod]

Active members was hovering above 90k for a few days, but it just dropped below.  :/

When I was your age, we didn't have these fancy websites to warn us things were failing.   Sometimes they'd fail right when we were using 'em...

[ibminer]

You make me feel so young.
After the physical and emotional stress I've endured so far in 2020, I'll estimate I will have aged 15+ years by 2021. lol

So what are the odds the active members drop is related to COVID-19?... or, maybe just someone's account farm went offline or something?   Where's the poll?? lol 

.. BPIP should really get some historical tracking going with those stats ..

[Vod]

You make me feel so young.

Hmmm, didn't you say that right before crawling into bed naked with a young girl???


Yep.   :/