[TOOL] ⭐Bitcointalk BRDb score

[*Ace*]

I’ll most likely remove the badge; personally, it was meant to give proper recognition to those who try to keep this forum active every day. I’m sure that acting as a source of merit is no small undertaking and requires consistency.
I don’t have a list of users who are sources of merit, but I could get hold of one, and I don’t think I’ll be sharing it 
Just so I don’t give too much away.
Next week I’ll be heading back to my Batcave to try and sort out the tool once and for all – or break it forever xD

[fillippone]

I’ll most likely remove the badge; personally, it was meant to give proper recognition to those who try to keep this forum active every day. I’m sure that acting as a source of merit is no small undertaking and requires consistency.
I don’t have a list of users who are sources of merit, but I could get hold of one, and I don’t think I’ll be sharing it 
Just so I don’t give too much away.
Next week I’ll be heading back to my Batcave to try and sort out the tool once and for all – or break it forever xD

The point is your algorithm is broken, as you cannot mtracecmerit sources like this.
Even with a generosity index in the positive territory you have a sufficient condition, not a necessary one!

[*Ace*]

I’ll most likely remove the badge; personally, it was meant to give proper recognition to those who try to keep this forum active every day. I’m sure that acting as a source of merit is no small undertaking and requires consistency.
I don’t have a list of users who are sources of merit, but I could get hold of one, and I don’t think I’ll be sharing it 
Just so I don’t give too much away.
Next week I’ll be heading back to my Batcave to try and sort out the tool once and for all – or break it forever xD

The point is your algorithm is broken, as you cannot mtracecmerit sources like this.
Even with a generosity index in the positive territory you have a sufficient condition, not a necessary one!

Unfortunately, you’re absolutely right – the algorithm is faulty. As soon as I get back from holiday, I’ll either sort it out or scrap the whole thing.
In the meantime, I’ll remove this badge – it’s not really necessary, even though it looked nice, but if it causes an error, it becomes a problem.

[*Ace*]

I have a question for those more experienced than me.
Given that I no longer use Bitlist as my data source, but scraping directly from Bitcointalk for thousands and thousands of users would also take a long time, what would be the best way to obtain the necessary data for all users in the shortest possible time?
If I’m not mistaken, the forum allows one request per second; exceeding this limit results in a ban.
Thanks in advance to anyone who can give me a hand with this.

[LoyceV]

what would be the best way to obtain the necessary data for all users in the shortest possible time?

Based on what you need, you're going to have to be selective. You can't update everything all the time. BPIP for instance scrapes profiles on a regular basis, depending on how active the user is.

If I’m not mistaken, the forum allows one request per second; exceeding this limit results in a ban.

You'll get "rate limited", not banned. After a few seconds it should work again. Unless you really push it, in that case Cloudflare bans you for an hour.

[*Ace*]

what would be the best way to obtain the necessary data for all users in the shortest possible time?

Based on what you need, you're going to have to be selective. You can't update everything all the time. BPIP for instance scrapes profiles on a regular basis, depending on how active the user is.

If I’m not mistaken, the forum allows one request per second; exceeding this limit results in a ban.

You'll get "rate limited", not banned. After a few seconds it should work again. Unless you really push it, in that case Cloudflare bans you for an hour.

That’s exactly my problem – being too selective.
What occurred to me was to use the ‘recent posts’ section to scrape profiles based on their most recent posts, without making too many requests on the forum.
However, this wouldn’t update the scores of profiles that haven’t posted for a long time.
I’m not sure if this idea would work well

[TryNinja]

That’s exactly my problem – being too selective.
What occurred to me was to use the ‘recent posts’ section to scrape profiles based on their most recent posts, without making too many requests on the forum.
However, this wouldn’t update the scores of profiles that haven’t posted for a long time.
I’m not sure if this idea would work well

You can have a list on your database with the last activity of an user.

Check recent posts page -> loops posts -> sees TryNinja posted just now -> marks TryNinja with "last post: [date]"

Then you can have a worker process that checks your list once every day. In your list you know that TryNinja posted on [date], if that was 6 month ago, you lower his score or whatever.

[*Ace*]

That’s exactly my problem – being too selective.
What occurred to me was to use the ‘recent posts’ section to scrape profiles based on their most recent posts, without making too many requests on the forum.
However, this wouldn’t update the scores of profiles that haven’t posted for a long time.
I’m not sure if this idea would work well

You can have a list on your database with the last activity of an user.

Check recent posts page -> loops posts -> sees TryNinja posted just now -> marks TryNinja with "last post: [date]"

Then you can have a worker process that checks your list once every day. In your list you know that TryNinja posted on [date], if that was 6 month ago, you lower his score or whatever.

Thank you so much for the advice; I’ll do as you suggested – it seems like a good compromise.
I’ll run continuous loops (which I actually already do for both new posts and merit points), but I’ll tag them by date; after that, I’m not sure whether to check the profile data or not. I’ll have a think about it over the next few days.