[TOOL] ⭐Bitcointalk BRDb score

[DYING_S0UL]

OK, I’ve sorted out some more bugs tonight.
The post and merit data should now be captured correctly by the scraper, and should update automatically whenever a new post is written or a new merit is sent or received. Please let me know if everything is working as it should.

My score is still missing btw, however I was able to see others data just a moment ago. But after a quick refresh it's all gone now - " BRDb pending". I remember seeing fillippone's 8.2 and yours 5.2 something. But now it's not there.

Do we need to manually update the userscript again? I haven't recently.

Well, of course it depends on your hardware. But generally speaking I guess no, unless you have a monster in your studio.
I would recommend some services like ollama.com.

I have a decent specs, R9 7900X and a 5060 TI 16GB, with 32GB RAM, and 2TB....and I have been using ollama for some times now. Will this GLM supported in Ollama?

[*Ace*]

OK, I’ve sorted out some more bugs tonight.
The post and merit data should now be captured correctly by the scraper, and should update automatically whenever a new post is written or a new merit is sent or received. Please let me know if everything is working as it should.

My score is still missing btw, however I was able to see others data just a moment ago. But after a quick refresh it's all gone now - " BRDb pending". I remember seeing fillippone's 8.2 and yours 5.2 something. But now it's not there.

Do we need to manually update the userscript again? I haven't recently.

Well, of course it depends on your hardware. But generally speaking I guess no, unless you have a monster in your studio.
I would recommend some services like ollama.com.

I have a decent specs, R9 7900X and a 5060 TI 16GB, with 32GB RAM, and 2TB....and I have been using ollama for some times now. Will this GLM supported in Ollama?

No, buddy, as I said earlier, the tool is down. Cloudflare has limited my operations. I'm rewriting the new BRDb on my VPS. I've already done a few things, but I'm going to sleep now. I'll make it public as soon as possible. In the meantime, I'm attaching just a couple of screenshots.

[LoyceV]

The scraper writes to the database but also reads from it; for each user, it has to read the number of posts, number of merit , the number of posts over the last 120 days, the number of merit points over the last 120 days, the registration date, the last login date (if available), check for new posts, check for new merit – multiply all that by thousands of users and several times a day, and you’ll see that 2.5 billion is actually quite low.

It sounds like you need to start caching data if you're reading 30,000 "data points" per second.

for each user, it has to read the number of posts, ~ over the last 120 days

Take me as an example: if you're checking this, does that mean you read 1 entry, or about 1000 individual entries for each post?

[dkbit98]

I saw BRDb pending information yesterday, and now I see it's temporary down again.
I will turn of usersrvipt until system is back up again, hopefully soon.

No, buddy, as I said earlier, the tool is down. Cloudflare has limited my operations. I'm rewriting the new BRDb on my VPS. I've already done a few things, but I'm going to sleep now. I'll make it public as soon as possible. In the meantime, I'm attaching just a couple of screenshots.

It is nice to see my username near the top, I don't know what I did to deserve that place 

[Lillominato89]

Well, I reckon rewriting the tool from scratch is a bit of a faff, but if it’s the only option, I support your decision.
Personally, I would have tried to tweak what you’d already done! The layout was much better.
We’ll wait for you to release the new version before giving our own opinion! Good luck, mate

[fillippone]

It’s difficult, when you have an (almost) functioning product to stop and refactor it from scratch to assess the underlying tech debt affecting the proper functioning of the tool.
So yeah, this is the long, yet correct way.

[DYING_S0UL]

I saw BRDb pending information yesterday, and now I see it's temporary down again.
I will turn of usersrvipt until system is back up again, hopefully soon.

The scores are back again (but I'm not sure whether it's the latest updated or browser caching, most probably old score). It's viewable from the profile information.

Right now you can only see that part (yours is 7.2), any further to that, you see Cloudflare's limit error "D1_ERROR: Your account has exceeded D1's free tier daily row read limit."

Heck, this is gonna take so much time building from scratch, I'm just imagining how much token it'll cost him. Hopefully he has premium plans! Otherwise he is cooked. The last time I tried building something, I depleted 1 weeks worth of token in a day! LOL.

[*Ace*]

The scraper writes to the database but also reads from it; for each user, it has to read the number of posts, number of merit , the number of posts over the last 120 days, the number of merit points over the last 120 days, the registration date, the last login date (if available), check for new posts, check for new merit – multiply all that by thousands of users and several times a day, and you’ll see that 2.5 billion is actually quite low.

It sounds like you need to start caching data if you're reading 30,000 "data points" per second.

for each user, it has to read the number of posts, ~ over the last 120 days

Take me as an example: if you're checking this, does that mean you read 1 entry, or about 1000 individual entries for each post?

The problem is not just what the scraper reads, but when the userscripts call the database. The score appears next to each user's username, and at least 20 users appear on each thread page. So all those calculations are done instantly for 20 users generating that many reads in the database rows. Definitely yes! I need cached data.
I've now moved everything to my server. I've rewritten almost the entire tool from scratch, removing some things I consider to be of little use.
The new BRDb is already online, I just need to buy a domain to make it available to everyone. I hope this time I don't have too many bugs to fix. 

Just to be fair, I'll let you know that I've scraped your loycev.club site to get the list of Merit transactions, but also posts to determine the 120-day writing periods, but I also noticed that your last update was in August, so there is a discrepancy on the real value of the merit transactions

[LoyceV]

I also noticed that your last update was in August, so there is a discrepancy on the real value of the merit transactions

Did you use merit.html (610 MB)? That file takes over a week to update now that my server got slower again.
It's better to use merit.all.txt (58 MB), this file is updated each Friday, shortly after theymos' weekly data dump.

[*Ace*]

I also noticed that your last update was in August, so there is a discrepancy on the real value of the merit transactions

Did you use merit.html (610 MB)? That file takes over a week to update now that my server got slower again.
It's better to use merit.all.txt (58 MB), this file is updated each Friday, shortly after theymos' weekly data dump.

No, I didn't use either of the two files you mentioned. I used this page for every user, https://loyce.club/Merit/history/459836.html I think it was the best compromise to get the data per user , but I saw that my last Merit sent dated back to August, but in reality I had also sent them in September

[Gragebox]

The problem is not just what the scraper reads, but when the userscripts call the database. The score appears next to each user's username, and at least 20 users appear on each thread page. So all those calculations are done instantly for 20 users generating that many reads in the database rows. Definitely yes! I need cached data.
I've now moved everything to my server. I've rewritten almost the entire tool from scratch, removing some things I consider to be of little use.
The new BRDb is already online, I just need to buy a domain to make it available to everyone. I hope this time I don't have too many bugs to fix. 

Just to be fair, I'll let you know that I've scraped your loycev.club site to get the list of Merit transactions, but also posts to determine the 120-day writing periods, but I also noticed that your last update was in August, so there is a discrepancy on the real value of the merit transactions

That is very reasonable indeed, running all the queries in every thread page for 20+ users may have unnecessary DB load, so, if the data is fetched to your own DB and the displayed data is cached, then it must be a much better solution.

Also, writing the tool over again can be a chance to trim up the features that are not necessarily used so that the system become lighter. Hope that this time BRDb could be more stable and then you don't spend a long time finding bugs.

And thanks for the clear explanation of data scraped from loycev.club. The difference of the Merit transactions are quite logical if last update was in August and the up-to-date data resource will help in a more reliable calculations.

Eager to view the new version after yourdomain name and publish it for everyone to see, good luck with that.

[LoyceV]

No, I didn't use either of the two files you mentioned. I used this page for every user, https://loyce.club/Merit/history/459836.html I think it was the best compromise to get the data per user

All Merit data in those files is derived from merit.all.txt. You could have saved yourself tens of thousands of downloads, and get more recent data.

[*Ace*]

No, I didn't use either of the two files you mentioned. I used this page for every user, https://loyce.club/Merit/history/459836.html I think it was the best compromise to get the data per user

All Merit data in those files is derived from merit.all.txt. You could have saved yourself tens of thousands of downloads, and get more recent data.

I thought, and I also mistakenly believe, that having them already divided by users would have been simpler and quicker

But you're right, I would have saved myself a lot of data downloads. I'll keep this in mind if I need Merit data again later.
First of all, thank you for making them available to everyone.

[*Ace*]

No, I didn't use either of the two files you mentioned. I used this page for every user, https://loyce.club/Merit/history/459836.html I think it was the best compromise to get the data per user

All Merit data in those files is derived from merit.all.txt. You could have saved yourself tens of thousands of downloads, and get more recent data.

I checked your file and finally decided to use it, the structure was the same, only the post id title was missing.So I was making requests to get this missing data, but now I see that I'm getting error 429. What is this? Has my IP been banned?

I hope it's just temporary

[LoyceV]

So I was making requests to get this missing data, but now I see that I'm getting error 429. What is this? Has my IP been banned?

Did Bitcointalk give this error? It looks like you're being rate limited: try to stay under 1 request per second (bursts allowed).

If you want the titles for all Merited posts that I've collected over the years: titles.txt (scheduled to be deleted in 7 days). These are post titles, not topic titles. And some may have been edited (or posts deleted).

[*Ace*]

So I was making requests to get this missing data, but now I see that I'm getting error 429. What is this? Has my IP been banned?

Did Bitcointalk give this error? It looks like you're being rate limited: try to stay under 1 request per second (bursts allowed).

If you want the titles for all Merited posts that I've collected over the years: titles.txt (scheduled to be deleted in 7 days). These are post titles, not topic titles. And some may have been edited (or posts deleted).

I've temporarily suspended scraping activities. I'm not receiving any data because my IP address has likely been blocked. I sent a PM to theymos, hoping they'll read it , And that takes me off the bad list. I'll try to bend the forum once an hour to see if I get an HTTP 200 status. I tried reducing the requests to 2 seconds but nothing happens

[*Ace*]

It looks like the ban has finally been lifted. The scraper is back to collecting data, but I had to extend the time. Between scraping posts and Merit and updating profiles, the entire cycle takes 5 minutes.  Is it too much? Too little?
I have no idea, but it works for now since there are only a few users in the database.

[Lillominato89]

It looks like the ban has finally been lifted. The scraper is back to collecting data, but I had to extend the time. Between scraping posts and Merit and updating profiles, the entire cycle takes 5 minutes.  Is it too much? Too little?
I have no idea, but it works for now since there are only a few users in the database.

Great, I haven’t been following this discussion for a few days and now I see there are some important updates.
I can’t wait to see the new tool in action as soon as you’ve secured a domain.
When you get a chance, please do post the updates you’ve made over the last few days – at least that way we can see the progress.

[The Sceptical Chymist]

Well here's another very interesting thread that somehow escaped my mosquito-capacity attention.....thing.  And it escaped it for months; that's inexcusable, because aside from the very tricky problem of trying to treat post quality with math, I think you did a damn fine job on this project *Ace*. 

I also congratulate you on jumping up through the ranks so fast and thought I might give you a little boost since I think you deserve it.  I wasn't familiar with your username and thought you might have been an older member who'd come back from pasture, but nope.  You're just a good writer and did a nice thing here for the people who go nutty over stats (and there's nothing wrong with that so far as I can see).  I shall keep your page bookmarked alongside bitlist.co and the other forum-related sites I visit from time to time.

Cool beans, sir.  Cool beans.

[fillippone]

Well here's another very interesting thread that somehow escaped my mosquito-capacity attention.....thing.  And it escaped it for months; that's inexcusable, because aside from the very tricky problem of trying to treat post quality with math, I think you did a damn fine job on this project *Ace*.  

I also congratulate you on jumping up through the ranks so fast and thought I might give you a little boost since I think you deserve it.  I wasn't familiar with your username and thought you might have been an older member who'd come back from pasture, but nope.  You're just a good writer and did a nice thing here for the people who go nutty over stats (and there's nothing wrong with that so far as I can see).  I shall keep your page bookmarked alongside bitlist.co and the other forum-related sites I visit from time to time.

Cool beans, sir.  Cool beans.

I think this project is a good addition, and it repurposes all the stats we already have, trying to assess the quality of a poster.
original idea, I guess, was from LoyceV, and I later suggested an implementation to *Ace*, whom I already knew as a good coder.
Not an AI-driven coder, a real one.