suchmoon
Legendary
Offline
Activity: 3766
Merit: 9007
https://bpip.org
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 25, 2020, 11:15:22 PM |
|
I know... it looks like it's just some autogenerated doc for the node.js crap. So basically what you see on the website. I don't think it has "get all new and updated trust lists since <timestamp>" or anything like that. Hopefully there will still be at least the weekly downloadable files. As long as the forum displays the last update, you can always recursively parse any top level category as needed.
Possibly. But on a busy forum like Bitcointalk you'd expect a lot of boards have activity so multiple pages would need to be reparsed every time. A patrol/recent posts page/API would be better and I don't think EpochTalk has that. Maybe putting all boards on a watchlist would solve it.
|
|
|
|
Timelord2067
Legendary
Offline
Activity: 3766
Merit: 2235
💲🏎️💨🚓
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 26, 2020, 07:35:02 AM |
|
Hi guys, just a quick heads up, the link https://bpip.org/report.aspx?r=mostmerited returns a page: https://bpip.org/?noreportname Regards,
|
|
|
|
TheBeardedBaby
Legendary
Offline
Activity: 2240
Merit: 3150
₿uy / $ell ..oeleo ;(
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 26, 2020, 07:41:17 AM |
|
Hi guys, just a quick heads up, the link https://bpip.org/report.aspx?r=mostmerited returns a page: https://bpip.org/?noreportname Regards, Just change the : https://bpip.org/report.aspx?r=mostmerited to: https://bpip.org/report.aspx?r=earnedmerit and it will be fine. I think there was an update, now I get also reward for the "Most earned merit" ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif) Well I lost 2 positions due to inactivity the last few months but.. still in top 25 ![Tongue](https://bitcointalk.org/Smileys/default/tongue.gif) Edit: wasn't mostmerit but earnedmerit, corrected now.
|
|
|
|
Timelord2067
Legendary
Offline
Activity: 3766
Merit: 2235
💲🏎️💨🚓
|
![](https://bitcointalk.org/Themes/custom1/images/post/lamp.gif) |
August 26, 2020, 07:47:13 AM |
|
Just change the : https://bpip.org/report.aspx?r=mostmerited to: https://bpip.org/report.aspx?r=earnedmerit and it will be fine. I think there was an update, now I get also reward for the "Most earned merit" ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif) Well I lost 2 positions due to inactivity the last few months but.. still in top 25 ![Tongue](https://bitcointalk.org/Smileys/default/tongue.gif) Most merited is not the same as earned merit - the link is contained in the first shield of the five shields on each users' page information. Hopefully it's just a glitch during the fortnightly update. (have gone back from 39th to 37th with my last fortnight's posts, so about the same as before. )
Are your house building works done yet?
|
|
|
|
TheBeardedBaby
Legendary
Offline
Activity: 2240
Merit: 3150
₿uy / $ell ..oeleo ;(
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 26, 2020, 08:07:21 AM |
|
Just change the : https://bpip.org/report.aspx?r=mostmerited to: https://bpip.org/report.aspx?r=earnedmerit and it will be fine. I think there was an update, now I get also reward for the "Most earned merit" ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif) Well I lost 2 positions due to inactivity the last few months but.. still in top 25 ![Tongue](https://bitcointalk.org/Smileys/default/tongue.gif) Most merited is not the same as earned merit - the link is contained in the first shield of the five shields on each users' page information. Hopefully it's just a glitch during the fortnightly update. (have gone back from 39th to 37th with my last fortnight's posts, so about the same as before. )
Are your house building works done yet? Then mostmerited = mostmerit https://bpip.org/report.aspx?r=mostmerit Are your house building works done yet? Renovating. 2 floors done, now is the basement this winter. Build some small things in the garden as well, almost finished with shed for wood. I'm doing all alone so it takes time.
|
|
|
|
Aveatrex
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 30, 2020, 09:59:12 AM |
|
Hey BPIP team @ibminer @Vod @suchmoon,
I'm currently developing a mini-software for the community, is web scraping your website allowed via automated requests? I prefer to ask before as I don't know if you can afford that with your hosting especially that right now, the website's response seem to be slow
Let me know if you need additional information on the intended usage
|
|
|
|
suchmoon
Legendary
Offline
Activity: 3766
Merit: 9007
https://bpip.org
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 30, 2020, 12:09:27 PM |
|
Hey BPIP team @ibminer @Vod @suchmoon,
I'm currently developing a mini-software for the community, is web scraping your website allowed via automated requests? I prefer to ask before as I don't know if you can afford that with your hosting especially that right now, the website's response seem to be slow
Let me know if you need additional information on the intended usage
Let me know what you want to scrape (via PM if you don't want to make it public) and we might have a better solution for it.
|
|
|
|
Aveatrex
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 30, 2020, 01:21:40 PM |
|
Hey BPIP team @ibminer @Vod @suchmoon,
I'm currently developing a mini-software for the community, is web scraping your website allowed via automated requests? I prefer to ask before as I don't know if you can afford that with your hosting especially that right now, the website's response seem to be slow
Let me know if you need additional information on the intended usage
Let me know what you want to scrape (via PM if you don't want to make it public) and we might have a better solution for it. I'm scraping the trust of a set list of users from an excel file that the user selects and saves the results on the same excel file. The problem of this is that it's slow, is heavy on the server and triggers false positives from 1-2 anti-virus that thinks the software is performing a ddos attack or is part of some bot net. Is it possible to have a compressed file of the trust of each user updated like each 24 hours, or is that too much to ask for?
|
|
|
|
suchmoon
Legendary
Offline
Activity: 3766
Merit: 9007
https://bpip.org
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 30, 2020, 01:31:09 PM Last edit: November 30, 2020, 06:16:26 AM by suchmoon |
|
I'm scraping the trust of a set list of users from an excel file that the user selects and saves the results on the same excel file. The problem of this is that it's slow, is heavy on the server and triggers false positives from 1-2 anti-virus that thinks the software is performing a ddos attack or is part of some bot net. Is it possible to have a compressed file of the trust of each user updated like each 24 hours, or is that too much to ask for?
What do you mean by trust - is it this part? Edited 2020-11-30 to fix a broken image
|
|
|
|
hosseinimr93
Legendary
Offline
Activity: 2492
Merit: 5558
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 30, 2020, 01:57:48 PM Last edit: August 30, 2020, 07:58:14 PM by hosseinimr93 |
|
...Is it possible to have a compressed file of the trust of each user updated like each 24 hours, or is that too much to ask for?
I don't think you need to scrape BPIP. You can use the following file provided by theymos to extract all trusts and distrusts. It is updated once per week. trust.txt.xzYou can also use the tool created by LoyceV to see trust scores of all users in a single page. Custom Trust list viewer, ( Topic on bitcointalk)
|
|
|
|
Aveatrex
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 30, 2020, 02:18:25 PM |
|
What do you mean by trust - is it this part
No, the part just under that, if it's Red, Black or Green. Thank you for your suggestion I'll take a look at it Edit: I'm curious, if theymos trust dump updates every week, how bpip updates the trust of users on demand?
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3402
Merit: 17238
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 30, 2020, 02:24:12 PM |
|
No, the part just under that, if it's Red, Black or Green. I also have Trust score images, but it depends on how many you need. Edit: I'm curious, if theymos trust dump updates every week, how blip updates the trust of users on demand? Theymos only dumps Trust lists (only for users who have at least one post), not Trust (feedback) scores. BPIP has to scrape each profile for each update.
|
|
|
|
suchmoon
Legendary
Offline
Activity: 3766
Merit: 9007
https://bpip.org
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 30, 2020, 02:33:09 PM |
|
Edit: I'm curious, if theymos trust dump updates every week, how bpip updates the trust of users on demand?
We continuously scrape every user profile, prioritizing those who are most active. It's technically possible that some trust ratings (colors) aren't updated quickly if e.g. a rarely active DT member posts a rating for another rarely active user, so keep that in mind. I'll see how I can make this easier for you.
|
|
|
|
Aveatrex
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 31, 2020, 01:24:37 AM |
|
No, the part just under that, if it's Red, Black or Green. I also have Trust score images, but it depends on how many you need. Well if I'm going to release the software publicly, I can't predict how many users an user X will include in his excel file so that won't work. Technically, I can use and scrape this page instead of bpip it would be much easier but the downside is that it updates every week which is a bit too long to my taste. We continuously scrape every user profile, prioritizing those who are most active.
Now that you say it, I wonder if i shouldn't make the bot do the same? i.e scrape directly bitcointalk profiles instead of using bpip as a middleman? That for sure wouldn't solve the false postives detected by 2 out of 68 anti viruses (using virus total) since it will still be making automated requests ![Cry](https://bitcointalk.org/Smileys/default/cry.gif) but is a solution to be considered
|
|
|
|
suchmoon
Legendary
Offline
Activity: 3766
Merit: 9007
https://bpip.org
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 31, 2020, 01:48:45 AM |
|
Now that you say it, I wonder if i shouldn't make the bot do the same? i.e scrape directly bitcointalk profiles instead of using bpip as a middleman? That for sure wouldn't solve the false postives detected by 2 out of 68 anti viruses (using virus total) since it will still be making automated requests ![Cry](https://bitcointalk.org/Smileys/default/cry.gif) but is a solution to be considered You could do that but it's quite a bit of hassle. Your bot will need a login, possibly deal with CloudFlare, and of course scrape multiple pages. Here's a file of all trust ratings that we have on BPIP - let me know if it works for you: https://bpip.org/all_trust.csvSince you mentioned Excel I created it in a CSV format, which you should be able to import into a spreadsheet. If you prefer a different format, like JSON or XML - let me know. The columns are: user_id, positive_score, negative_score, neutral_score, trust_color, dt_status, dt1_strength, dt2_strength, flag_count If this is good I can set it up to be updated once a day.
|
|
|
|
Vod
Legendary
Offline
Activity: 3794
Merit: 3115
Licking my boob since 1970
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 31, 2020, 04:04:37 AM |
|
Hey BPIP team @ibminer @Vod @suchmoon Hey Aveatrex https://bpip.org/about.aspxI assume you included me because I created it. thx! But I'm not part of the official team anymore. ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif)
|
|
|
|
Aveatrex
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 31, 2020, 08:33:51 AM |
|
Your bot will need a login, possibly deal with CloudFlare
Damn I forgot that to view profiles you need to be logged in Since you mentioned Excel I created it in a CSV format, which you should be able to import into a spreadsheet. If you prefer a different format, like JSON or XML - let me know. The columns are: user_id, positive_score, negative_score, neutral_score, trust_color, dt_status, dt1_strength, dt2_strength, flag_count If this is good I can set it up to be updated once a day. That format works for me! It would be great if you also added a "username" column so I can make it cross-compatible wether the user included bitcointalk profiles urls or usernames in the excel file.
Now, there is only ~27k users, is that what you meant by It's technically possible that some trust ratings (colors) aren't updated quickly if e.g. a rarely active DT member posts a rating for another rarely active user, so keep that in mind.
because if it's the case from my tests 27k is definitely not enough , like half the users I tested are not found in it Hmm, maybe I should at first use this CSV to check the user's trust, if he's not in it, I will search for him in LoyceV's Custom trust list
I assume you included me because I created it. thx! But I'm not part of the official team anymore. ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif) Yes that's why I included you,I wasn't aware you left the team sorry!
|
|
|
|
suchmoon
Legendary
Offline
Activity: 3766
Merit: 9007
https://bpip.org
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 31, 2020, 10:54:33 AM |
|
because if it's the case from my tests 27k is definitely not enough , like half the users I tested are not found in it Can you give me some examples of users that you can't find? Do those users have DT ratings? The list includes only users who have at least one (negative, positive, or neutral) rating from a DT member, and/or have at least one DT-supported flag, and/or are DT1/2 members themselves. If a user is not in the list then it should mean they have none of the above (i.e. trust color "black") but if you could get me some examples I can find out what's going on.
|
|
|
|
Aveatrex
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 31, 2020, 11:52:19 AM |
|
Can you give me some examples of users that you can't find? Do those users have DT ratings?
The list includes only users who have at least one (negative, positive, or neutral) rating from a DT member, and/or have at least one DT-supported flag, and/or are DT1/2 members themselves.
If a user is not in the list then it should mean they have none of the above (i.e. trust color "black") but if you could get me some examples I can find out what's going on.
That makes sense, I just checked some users that couldn't be found and all of them have 0 feedback. Which means that if an user can't be found in this csv it either means that the user has Black trust OR it's an incorrect username/bitcointalk profile url (I'm thinking of a scenario where a bounty manager have a list of users and a bounty hunter provides an incorrect url or username) so gotta differentiate the two cases. To do that I'll do additional checking using theymos trust dump or Loyce's viewer. Thanks!
|
|
|
|
suchmoon
Legendary
Offline
Activity: 3766
Merit: 9007
https://bpip.org
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 31, 2020, 12:05:28 PM |
|
That makes sense, I just checked some users that couldn't be found and all of them have 0 feedback. Which means that if an user can't be found in this csv it either means that the user has Black trust OR it's an incorrect username/bitcointalk profile url (I'm thinking of a scenario where a bounty manager have a list of users and a bounty hunter provides an incorrect url or username) so gotta differentiate the two cases. To do that I'll do additional checking using theymos trust dump or Loyce's viewer.
Thanks!
I can add a username to the file if it helps you. It would be UTF8. I don't want to discourage you from double-checking with other sources, or having a backup if BPIP has an issue, but I think we have the same data that's in the trust dump or LoyceV's list, plus more-frequent-than-weekly updates. At least as far as the DT ratings are concerned. LoyceV obviously maintains non-DT trust lists, which we don't.
|
|
|
|
|