[BPIP] Bitcointalk Public Information Project [Back in Action]

[suchmoon]

https://epochtalk.org/reference/api-reference.html#get-recent-messages
That is only per user, but they do have a framework.

I know... it looks like it's just some autogenerated doc for the node.js crap. So basically what you see on the website. I don't think it has "get all new and updated trust lists since <timestamp>" or anything like that. Hopefully there will still be at least the weekly downloadable files.

As long as the forum displays the last update, you can always recursively parse any top level category as needed.

Possibly. But on a busy forum like Bitcointalk you'd expect a lot of boards have activity so multiple pages would need to be reparsed every time. A patrol/recent posts page/API would be better and I don't think EpochTalk has that. Maybe putting all boards on a watchlist would solve it.

[1715265081]

1715265081

[1715265081]

1715265081

[1715265081]

1715265081

[1715265081]

1715265081

[Timelord2067]

Hi guys,

just a quick heads up, the link

Code:

https://bpip.org/report.aspx?r=mostmerited

returns a page:

Code:

https://bpip.org/?noreportname

Regards,

[TheBeardedBaby]

Hi guys,

just a quick heads up, the link

Code:

https://bpip.org/report.aspx?r=mostmerited

returns a page:

Code:

https://bpip.org/?noreportname

Regards,

Just change the :

Code:

https://bpip.org/report.aspx?r=mostmerited

to:

Code:

https://bpip.org/report.aspx?r=earnedmerit

and it will be fine. I think there was an update, now I get also reward for the "Most earned merit" Well I lost 2 positions due to inactivity the last few months but.. still in top 25

Edit: wasn't mostmerit but earnedmerit, corrected now.

[Timelord2067]

Just change the :

Code:

https://bpip.org/report.aspx?r=mostmerited

to:

Code:

https://bpip.org/report.aspx?r=earnedmerit

and it will be fine. I think there was an update, now I get also reward for the "Most earned merit" Well I lost 2 positions due to inactivity the last few months but.. still in top 25

Most merited is not the same as earned merit - the link is contained in the first shield of the five shields on each users' page information.

Hopefully it's just a glitch during the fortnightly update. (have gone back from 39^th to 37^th with my last fortnight's posts, so about the same as before. )


Are your house building works done yet?

[TheBeardedBaby]

Just change the :

Code:

https://bpip.org/report.aspx?r=mostmerited

to:

Code:

https://bpip.org/report.aspx?r=earnedmerit

and it will be fine. I think there was an update, now I get also reward for the "Most earned merit" Well I lost 2 positions due to inactivity the last few months but.. still in top 25

Most merited is not the same as earned merit - the link is contained in the first shield of the five shields on each users' page information.

Hopefully it's just a glitch during the fortnightly update. (have gone back from 39^th to 37^th with my last fortnight's posts, so about the same as before. )


Are your house building works done yet?

Then mostmerited = mostmerit

Code:

https://bpip.org/report.aspx?r=mostmerit

Are your house building works done yet?

Renovating. 2 floors done, now is the basement this winter. Build some small things in the garden as well, almost finished with shed for wood. I'm doing all alone so it takes time.

[Aveatrex]

Hey BPIP team @ibminer @Vod @suchmoon,

I'm currently developing a mini-software for the community, is web scraping your website allowed via automated requests? I prefer to ask before as I don't know if you can afford that with your hosting especially that right now, the website's response seem to be slow


Let me know if you need additional information on the intended usage

[suchmoon]

Hey BPIP team @ibminer @Vod @suchmoon,

I'm currently developing a mini-software for the community, is web scraping your website allowed via automated requests? I prefer to ask before as I don't know if you can afford that with your hosting especially that right now, the website's response seem to be slow

Let me know if you need additional information on the intended usage

Let me know what you want to scrape (via PM if you don't want to make it public) and we might have a better solution for it.

[Aveatrex]

Hey BPIP team @ibminer @Vod @suchmoon,

I'm currently developing a mini-software for the community, is web scraping your website allowed via automated requests? I prefer to ask before as I don't know if you can afford that with your hosting especially that right now, the website's response seem to be slow

Let me know if you need additional information on the intended usage

Let me know what you want to scrape (via PM if you don't want to make it public) and we might have a better solution for it.

I'm scraping the trust of a set list of users from an excel file that the user selects and saves the results on the same excel file. The problem of this is that it's slow, is heavy on the server and triggers false positives from 1-2 anti-virus that thinks the software is performing a ddos attack or is part of some bot net. Is it possible to have a compressed file of the trust of each user updated like each 24 hours, or is that too much to ask for?

[suchmoon]

I'm scraping the trust of a set list of users from an excel file that the user selects and saves the results on the same excel file. The problem of this is that it's slow, is heavy on the server and triggers false positives from 1-2 anti-virus that thinks the software is performing a ddos attack or is part of some bot net. Is it possible to have a compressed file of the trust of each user updated like each 24 hours, or is that too much to ask for?

What do you mean by trust - is it this part?


Edited 2020-11-30 to fix a broken image

[hosseinimr93]

...Is it possible to have a compressed file of the trust of each user updated like each 24 hours, or is that too much to ask for?

I don't think you need to scrape BPIP.
You can use the following file provided by theymos to extract all trusts and distrusts. It is updated once per week.
trust.txt.xz

You can also use the tool created by LoyceV to see trust scores of all users in a single page.
Custom Trust list viewer, (Topic on bitcointalk)

[Aveatrex]

What do you mean by trust - is it this part

No, the part just under that, if it's Red, Black or Green.

You can also use the tool created by LoyceV to see trust scores of all users in a single page.
Custom Trust list viewer, (Topic on bitcointalk)

Thank you for your suggestion I'll take a look at it

Edit: I'm curious, if theymos trust dump updates every week, how bpip updates the trust of users on demand?

[LoyceV]

No, the part just under that, if it's Red, Black or Green.

I also have Trust score images, but it depends on how many you need.

Edit: I'm curious, if theymos trust dump updates every week, how blip updates the trust of users on demand?

Theymos only dumps Trust lists (only for users who have at least one post), not Trust (feedback) scores. BPIP has to scrape each profile for each update.

[suchmoon]

Edit: I'm curious, if theymos trust dump updates every week, how bpip updates the trust of users on demand?

We continuously scrape every user profile, prioritizing those who are most active. It's technically possible that some trust ratings (colors) aren't updated quickly if e.g. a rarely active DT member posts a rating for another rarely active user, so keep that in mind.

I'll see how I can make this easier for you.

[Aveatrex]

No, the part just under that, if it's Red, Black or Green.

I also have Trust score images, but it depends on how many you need.

Well if I'm going to release the software publicly, I can't predict how many users an user X will include in his excel file so that won't work.

Technically, I can use and scrape this page instead of bpip it would be much easier but the downside is that it updates every week which is a bit too long to my taste.

We continuously scrape every user profile, prioritizing those who are most active.

Now that you say it, I wonder if i shouldn't make the bot do the same? i.e scrape directly bitcointalk profiles instead of using bpip as a middleman? That for sure wouldn't solve the false postives detected by 2 out of 68 anti viruses (using virus total) since it will still be making automated requests   but is a solution to be considered

[suchmoon]

Now that you say it, I wonder if i shouldn't make the bot do the same? i.e scrape directly bitcointalk profiles instead of using bpip as a middleman? That for sure wouldn't solve the false postives detected by 2 out of 68 anti viruses (using virus total) since it will still be making automated requests   but is a solution to be considered

You could do that but it's quite a bit of hassle. Your bot will need a login, possibly deal with CloudFlare, and of course scrape multiple pages.

Here's a file of all trust ratings that we have on BPIP - let me know if it works for you: https://bpip.org/all_trust.csv

Since you mentioned Excel I created it in a CSV format, which you should be able to import into a spreadsheet. If you prefer a different format, like JSON or XML - let me know. The columns are:

Code:

user_id, positive_score, negative_score, neutral_score, trust_color, dt_status, dt1_strength, dt2_strength, flag_count

If this is good I can set it up to be updated once a day.

[Vod]

Hey BPIP team @ibminer @Vod @suchmoon

Hey Aveatrex

https://bpip.org/about.aspx

I assume you included me because I created it.  thx!   But I'm not part of the official team anymore. 

[Aveatrex]

Your bot will need a login, possibly deal with CloudFlare

Damn I forgot that to view profiles you need to be logged in

Since you mentioned Excel I created it in a CSV format, which you should be able to import into a spreadsheet. If you prefer a different format, like JSON or XML - let me know. The columns are:

Code:

user_id, positive_score, negative_score, neutral_score, trust_color, dt_status, dt1_strength, dt2_strength, flag_count

If this is good I can set it up to be updated once a day.

That format works for me! It would be great if you also added a "username" column so I can make it cross-compatible wether the user included bitcointalk profiles urls or usernames in the excel file.


Now, there is only ~27k users, is that what you meant by

It's technically possible that some trust ratings (colors) aren't updated quickly if e.g. a rarely active DT member posts a rating for another rarely active user, so keep that in mind.

because if it's the case from my tests 27k is definitely not enough , like half the users I tested are not found in it

Hmm, maybe I should at first use this CSV to check the user's trust, if he's not in it, I will search for him in LoyceV's Custom trust list

I assume you included me because I created it.  thx!   But I'm not part of the official team anymore.  

Yes that's why I included you,I wasn't aware you left the team sorry!

[suchmoon]

because if it's the case from my tests 27k is definitely not enough , like half the users I tested are not found in it

Can you give me some examples of users that you can't find? Do those users have DT ratings?

The list includes only users who have at least one (negative, positive, or neutral) rating from a DT member, and/or have at least one DT-supported flag, and/or are DT1/2 members themselves.

If a user is not in the list then it should mean they have none of the above (i.e. trust color "black") but if you could get me some examples I can find out what's going on.

[Aveatrex]

Can you give me some examples of users that you can't find? Do those users have DT ratings?

The list includes only users who have at least one (negative, positive, or neutral) rating from a DT member, and/or have at least one DT-supported flag, and/or are DT1/2 members themselves.

If a user is not in the list then it should mean they have none of the above (i.e. trust color "black") but if you could get me some examples I can find out what's going on.

That makes sense, I just checked some users that couldn't be found and all of them have 0 feedback. Which means that if an user can't be found in this csv it either means that the user has Black trust OR it's an incorrect username/bitcointalk profile url (I'm thinking of a scenario where a bounty manager have a list of users and a bounty hunter provides an incorrect url or username) so gotta differentiate the two cases. To do that I'll do additional checking using theymos trust dump or Loyce's viewer.

Thanks!

[suchmoon]

That makes sense, I just checked some users that couldn't be found and all of them have 0 feedback. Which means that if an user can't be found in this csv it either means that the user has Black trust OR it's an incorrect username/bitcointalk profile url (I'm thinking of a scenario where a bounty manager have a list of users and a bounty hunter provides an incorrect url or username) so gotta differentiate the two cases. To do that I'll do additional checking using theymos trust dump or Loyce's viewer.

Thanks!

I can add a username to the file if it helps you. It would be UTF8.

I don't want to discourage you from double-checking with other sources, or having a backup if BPIP has an issue, but I think we have the same data that's in the trust dump or LoyceV's list, plus more-frequent-than-weekly updates. At least as far as the DT ratings are concerned. LoyceV obviously maintains non-DT trust lists, which we don't.