Aveatrex
|
|
August 31, 2020, 01:40:15 PM Last edit: September 01, 2020, 04:46:25 AM by Aveatrex |
|
I can add a username to the file if it helps you. It would be UTF8.
Yes please, it would make things much easier especially now that I need to get additional info from other sources users are referenced by username. That makes sense, I just checked some users that couldn't be found and all of them have 0 feedback. Which means that if an user can't be found in this csv it either means that the user has Black trust OR it's an incorrect username/bitcointalk profile url (I'm thinking of a scenario where a bounty manager have a list of users and a bounty hunter provides an incorrect url or username) so gotta differentiate the two cases. To do that I'll do additional checking using theymos trust dump or Loyce's viewer.
Thanks!
plus more-frequent-than-weekly updates. At least as far as the DT ratings are concerned. LoyceV obviously maintains non-DT trust lists, which we don't. That's the biggest advantage of bpip imho. So for getting DT ratings gonna use bpip and to confirm 0 feedback users gonna use LoyceV's trust list. You guys service complement each other
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3318
Merit: 16691
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
August 31, 2020, 06:20:31 PM |
|
Damn I forgot that to view profiles you need to be logged in Although I've had some troubles with it, it works quite easily by feeding your scraper your browser cookie. My scraper hasn't been bothered by Cloudflare in a long time. Hmm, maybe I should at first use this CSV to check the user's trust, if he's not in it, I will search for him in LoyceV's Custom trust list I only check/publish Trust scores for users with or on a custom Trust list (and at least one post). I (usually) update the Trust scores on Thursday, and publish them on Saturday in the latest version of my Trust list viewer. That's the biggest advantage of bpip imho. So for getting DT ratings gonna use bpip and to confirm 0 feedback users gonna use LoyceV's trust list. You guys service complement each other I didn't realize until now that it's relatively easy to get DT Trust scores for all users, and I'm not sure if BPIP uses it already: by checking the sent feedback for all DT-members, you know exactly who has received DT-feedback. It's probably less than 1000 pages to scrape for an update, but it's more work to set it up. I'm currently scraping 13651 profiles per week for an update, so this can be improved.
|
|
|
|
suchmoon
Legendary
Offline
Activity: 3682
Merit: 8923
https://bpip.org
|
|
September 01, 2020, 12:02:17 AM |
|
Yes please, it would make things much easier especially now that I need to get additional info from other sources users are referenced by username.
I have added a column for user name, added column headers, and set it up to run around midnight UTC (give or take a few minutes depending on other tasks in the same batch). https://bpip.org/all_trust.csvI didn't realize until now that it's relatively easy to get DT Trust scores for all users, and I'm not sure if BPIP uses it already: by checking the sent feedback for all DT-members, you know exactly who has received DT-feedback. It's probably less than 1000 pages to scrape for an update, but it's more work to set it up. I'm currently scraping 13651 profiles per week for an update, so this can be improved. IIRC we parse from the receiving end. Your idea seems faster.
|
|
|
|
examplens
Legendary
Online
Activity: 3290
Merit: 3217
Crypto Swap Exchange
|
|
September 01, 2020, 11:41:20 PM |
|
I'm not sure you changed anything here, but it seems merit count again does not work well. For example, user @suchmoon has earned "Merit Score 5497" but in merit information table stay "Has received 4495 ". There is a difference of 2 latest earned merit. I don't count merit assigned at the beginning.
|
|
|
|
suchmoon
Legendary
Offline
Activity: 3682
Merit: 8923
https://bpip.org
|
|
September 02, 2020, 12:04:18 AM |
|
I'm not sure you changed anything here, but it seems merit count again does not work well. For example, user @suchmoon has earned "Merit Score 5497" but in merit information table stay "Has received 4495 ". There is a difference of 2 latest earned merit. I don't count merit assigned at the beginning.
Good catch, thank you. I had a parser crash yesterday and didn't clean up the job queue properly so the merit update got stuck. Should be updated within the next few minutes. Under normal circumstances there can be a slight discrepancy between sent merits and total merits because those two pieces of information are scraped from two different places. It shouldn't last more than a minute or so.
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3318
Merit: 16691
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
September 12, 2020, 08:17:26 AM |
|
Users with a newly created custom Trust list ( such as protrader786) don't get a working link to my Trust list viewer yet. Every week, I create a file user_ids.txt for this, but I don't think it gets processed.
|
|
|
|
ibminer (OP)
Legendary
Offline
Activity: 1824
Merit: 2793
Goonies never say die.
|
|
September 12, 2020, 01:37:24 PM |
|
but I don't think it gets processed.
I've still been manually processing this file.. need to get back with suchmoon on adding a task to the server to do this for me but I just hadn't gotten to the recent update yet. It's good now. Thanks for the reminder!
|
|
|
|
Vod
Legendary
Offline
Activity: 3710
Merit: 3084
Licking my boob since 1970
|
|
September 14, 2020, 03:02:53 AM |
|
What happened to the flags updates?
If I were Suchmoon, I would have locked everything down for the Sep 11 19th anniversary. After all, we expect another attack on the 20th, but this year would have surprised us. He was just being safe.
|
https://nastyscam.com - landing page up https://vod.fan - advanced image hosting - coming soonish! OGNasty has early onset dementia; keep this in mind when discussing his past actions.
|
|
|
suchmoon
Legendary
Offline
Activity: 3682
Merit: 8923
https://bpip.org
|
|
September 14, 2020, 04:58:54 AM |
|
What happened to the flags updates?
I've pinged ibminer, flag parsing is his baby. Something must've gone broken but I'm sure he'll pick it up and fill in missing flags. If I were Suchmoon, I would have locked everything down for the Sep 11 19th anniversary. After all, we expect another attack on the 20th, but this year would have surprised us. He was just being safe. We won't let terrorism define us Or to be more accurate, we need to set up monitoring to alert us when stuff like that breaks but that's item #214 on the TODO list.
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3318
Merit: 16691
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
September 14, 2020, 05:47:08 AM |
|
we need to set up monitoring to alert us when stuff like that breaks The monitoring is done by the users, in this case Ratimov. If it breaks and nobody notices, it's not needed
|
|
|
|
TheBeardedBaby
Legendary
Offline
Activity: 2184
Merit: 3134
₿uy / $ell
|
|
September 14, 2020, 08:50:00 AM |
|
we need to set up monitoring to alert us when stuff like that breaks The monitoring is done by the users, in this case Ratimov. If it breaks and nobody notices, it's not needed True, I'm keeping an eye on the site as I'm almost every day checking things so if something fails I will let you guys know as i did it many times before Out of curiosity, how many unique views the site has per day?
|
|
|
|
ibminer (OP)
Legendary
Offline
Activity: 1824
Merit: 2793
Goonies never say die.
|
|
September 14, 2020, 10:09:46 AM Merited by JayJuanGee (1) |
|
Flaglog is running again, it has added new flags and is currently about 85% of the way through the weekly re-parsing of all flags... no big errors occurred here, just the parser losing connection to the DB and me not realizing it. The monitoring is done by the users, in this case Ratimov. If it breaks and nobody notices, it's not needed Well, most of the reports pull into the profile page as general information/stats, so just because someone doesn't use the flaglog itself, they could be looking at the number of flags against a user... which will look normal in a situation where the parser has stopped parsing flags, so it might be good to at least have some sort of general alert that a particular parser has stopped, although, I agree it is probably not the highest priority right now... I'll try to check the parser log more often!
|
|
|
|
Vod
Legendary
Offline
Activity: 3710
Merit: 3084
Licking my boob since 1970
|
|
September 14, 2020, 11:49:40 PM |
|
Or to be more accurate, we need to set up monitoring to alert us when stuff like that breaks but that's item #214 on the TODO list.
You use AWS services, right? https://aws.amazon.com/cloudwatch/
|
https://nastyscam.com - landing page up https://vod.fan - advanced image hosting - coming soonish! OGNasty has early onset dementia; keep this in mind when discussing his past actions.
|
|
|
NotATether
Legendary
Offline
Activity: 1610
Merit: 6761
bitcoincleanup.com / bitmixlist.org
|
|
September 15, 2020, 01:37:36 AM |
|
Or to be more accurate, we need to set up monitoring to alert us when stuff like that breaks but that's item #214 on the TODO list.
You use AWS services, right? https://aws.amazon.com/cloudwatch/The problem here is BPIP is running inside rented servers on AWS, and not across the über hyper-scaled AWS platform (correct me if I'm wrong). The Cloudwatch event somehow has to determine whether BPIP updating is broken and you can't do that without creating a middleware inside the server that collects BPIP diagnostics and send them somewhere like S3 where cloudwatch can access them. From my development experience, large tasks like this are as laborious and boring to implement, especially if you're not getting paid to do it, as they are necessary and useful.
|
. .BLACKJACK ♠ FUN. | | | ███▄██████ ██████████████▀ ████████████ █████████████████ ████████████████▄▄ ░█████████████▀░▀▀ ██████████████████ ░██████████████ █████████████████▄ ░██████████████▀ ████████████ ███████████████░██ ██████████ | | CRYPTO CASINO & SPORTS BETTING | | │ | | │ | ▄▄███████▄▄ ▄███████████████▄ ███████████████████ █████████████████████ ███████████████████████ █████████████████████████ █████████████████████████ █████████████████████████ ███████████████████████ █████████████████████ ███████████████████ ▀███████████████▀ ███████████████████ | | .
|
|
|
|
suchmoon
Legendary
Offline
Activity: 3682
Merit: 8923
https://bpip.org
|
|
September 15, 2020, 01:47:08 AM |
|
The problem here is all of BPIP's updating is happening within programs inside rented servers (I guess?). The Cloudwatch event somehow has to determine whether BPIP updating is broken and you can't do that without creating a middleware inside the server that collects BPIP diagnostics and send them somewhere like S3 where cloudwatch can access them.
From my development experience, large tasks like this are as laborious and boring to implement, especially if you're not getting paid to do it, as they are necessary and useful.
AFAIK CloudWatch has agents to gather info from places like database servers and log files although to be honest I'd prefer to stay off Amazon-specific stuff in case they go full evil and hike the prices up and force us to leave. So for monitoring I was thinking to use a third-party uptime service that could check (among other things) web server's heartbeat and basic status (OK/ERROR) of every important module. That would require just a simple web page so perhaps easier to set up than CloudWatch.
|
|
|
|
NotATether
Legendary
Offline
Activity: 1610
Merit: 6761
bitcoincleanup.com / bitmixlist.org
|
|
September 15, 2020, 10:06:20 AM |
|
AFAIK CloudWatch has agents to gather info from places like database servers and log files although to be honest I'd prefer to stay off Amazon-specific stuff in case they go full evil and hike the prices up and force us to leave. So for monitoring I was thinking to use a third-party uptime service that could check (among other things) web server's heartbeat and basic status (OK/ERROR) of every important module. That would require just a simple web page so perhaps easier to set up than CloudWatch.
I'm guessing that BPIP is running on ASP.Net Core, from inspecting the request headers sent by it. If that's true then there's a way to directly wire heartbeats to it: https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks?view=aspnetcore-3.1, though I admit that ASP.Net is not my strong suite so there isn't much else I can recommend. You could also throw in some functionality that sends an email whenever a module goes down, so the error statuses don't go unnoticed.
|
. .BLACKJACK ♠ FUN. | | | ███▄██████ ██████████████▀ ████████████ █████████████████ ████████████████▄▄ ░█████████████▀░▀▀ ██████████████████ ░██████████████ █████████████████▄ ░██████████████▀ ████████████ ███████████████░██ ██████████ | | CRYPTO CASINO & SPORTS BETTING | | │ | | │ | ▄▄███████▄▄ ▄███████████████▄ ███████████████████ █████████████████████ ███████████████████████ █████████████████████████ █████████████████████████ █████████████████████████ ███████████████████████ █████████████████████ ███████████████████ ▀███████████████▀ ███████████████████ | | .
|
|
|
|
ibminer (OP)
Legendary
Offline
Activity: 1824
Merit: 2793
Goonies never say die.
|
|
September 15, 2020, 10:55:02 AM |
|
CloudWatch does seem like a bit more than is necessary, although I'm not familiar with it and have never used it. It probably makes sense to do what suchmoon is saying here, if I'm understanding correctly. We are already storing each job and a timestamp when it was last run, so maybe one page like 'modcheck?mod=flagparse' which checks the DB for timestamps on that particular job, and returns OK/Error based on the result?
I guess we could do the same thing with the logic we are using in the logo as well (like when DB is offline, the logo turns red, if the DB is up and the parsing has stopped, the logo is not spinning). We could just check the DB timestamps and if anything hasn't last run in... lets say 20 minutes to give buffer for any long-running jobs.. the logo spins but turns yellow, giving the indication something inside the parser is not running. Could get more specific here in the length of time and each job, but just a general thought.
|
|
|
|
Vod
Legendary
Offline
Activity: 3710
Merit: 3084
Licking my boob since 1970
|
|
September 16, 2020, 02:13:37 AM |
|
Active members was hovering above 90k for a few days, but it just dropped below. :/
When I was your age, we didn't have these fancy websites to warn us things were failing. Sometimes they'd fail right when we were using 'em...
|
https://nastyscam.com - landing page up https://vod.fan - advanced image hosting - coming soonish! OGNasty has early onset dementia; keep this in mind when discussing his past actions.
|
|
|
ibminer (OP)
Legendary
Offline
Activity: 1824
Merit: 2793
Goonies never say die.
|
|
September 16, 2020, 11:14:14 AM |
|
You make me feel so young. After the physical and emotional stress I've endured so far in 2020, I'll estimate I will have aged 15+ years by 2021. lol So what are the odds the active members drop is related to COVID-19?... or, maybe just someone's account farm went offline or something? Where's the poll?? lol .. BPIP should really get some historical tracking going with those stats ..
|
|
|
|
Vod
Legendary
Offline
Activity: 3710
Merit: 3084
Licking my boob since 1970
|
|
September 17, 2020, 08:51:22 PM |
|
You make me feel so young.
Hmmm, didn't you say that right before crawling into bed naked with a young girl??? Yep. :/
|
https://nastyscam.com - landing page up https://vod.fan - advanced image hosting - coming soonish! OGNasty has early onset dementia; keep this in mind when discussing his past actions.
|
|
|
|