[TelegramBot] Merit watcher and Mention Notification Bot

[Nellayar]

First time to try your bot OP, and damn it is really good! Now, I can see the thread that mentioned my name. It is really helpful to lessen my time in looking for the thread that mentioned or quoting my reply. Let me share my first trial and the one that mentioned my name.

[1715644069]

1715644069

[1715644069]

1715644069

[giammangiato]

First time to try your bot OP, and damn it is really good! Now, I can see the thread that mentioned my name. It is really helpful to lessen my time in looking for the thread that mentioned or quoting my reply. Let me share my first trial and the one that mentioned my name.

Thank you for you nice words!

fixed problem
Today i fix a problem, when a merit is received, another task go to grab user, merit quantity and thread

the problem is the time (fast) and bitcointalk not respond me
i add a delay and i hope this work
@fillippone tell me if now receive information about received merits


this week
now i'm working on notify about trust change

[fillippone]

fixed problem
Today i fix a problem, when a merit is received, another task go to grab user, merit quantity and thread

the problem is the time (fast) and bitcointalk not respond me
i add a delay and i hope this work
@fillippone tell me if now receive information about received merits

Yes! I received the improved notification.
I would substitute the link in the notification with a "Merit Page" link, and I would make the title of the post clickable linking to the actual thread, so In case it would be possible to easily visit the merited post.
 
 

[Lillominato89]

Thanks @giammangiato, this update is really useful. Great job

[giammangiato]

Hello
I had to increase the delay
unfortunately if I call the forum too many times in a short time, this does not answer me (to avoid I think too much load)
hopefully it works, let me know if you still have these problems, I increase the delay even more

thanks

[fillippone]

Hello
I had to increase the delay
unfortunately if I call the forum too many times in a short time, this does not answer me (to avoid I think too much load)
hopefully it works, let me know if you still have these problems, I increase the delay even more

thanks

Thank you.
Still, it is broken. It notified your merit, but I cannot se who sent it, and for which thread. Also I don't know how many users your BOT has, but I think both TryNinja and Piggy figured out how to deal with a greater number of active users, a I never had this problem with their bot!
Come on, don't get demoralised, and keep up the good work!

[LoyceV]

I had to increase the delay
unfortunately if I call the forum too many times in a short time, this does not answer me (to avoid I think too much load)
hopefully it works, let me know if you still have these problems, I increase the delay even

You're allowed one request per second on average, although faster bursts are allowed shortly. You can test this by quickly opening many forum-links in new tabs:

Too fast / overloaded (503)

Most likely, you are accessing the forum too quickly. You are only allowed to access at a rate of one pageload per second on average (somewhat higher bursts).

It is also possible that the server is overloaded or temporarily down.

[Csmiami]

Not sure why, but I've gotten 2 duplicated notifications on merit today. First one was 1 merit from 1 user in one thread; and hours later (now), 1 person 1 merit on the same thread.

[Bttzed03]

Didn't receive notifications for @mention and quote. The second was probably because it was inserted later (no notifcation from Tryninja's as well) but I'm not sure why the first one was missed by the bot. 

[TryNinja]

Didn't receive notifications for @mention and quote. The second was probably because it was inserted later (no notifcation from Tryninja's as well) but I'm not sure why the first one was missed by the bot.  

Yes, the quote was added later (in the second). The first one had the mention since the beginning, so you should have been notified. Don't want to hijack OP's thread for issues related to my bot so please PM me if you didn't.

edit: I also got two merit notifications from OP's bot. I have it enabled as a "backup" in case mine ever fails.

[giammangiato]

Strange.


Duplicates are probably when I was testing, I'm sorry!

I'm trying to understand why it doesn't do scraping correctly, I increased the delay to 15 seconds between user and user and I increased the scraping pass for the merits to 10 minutes

[fillippone]

Strange.


Duplicates are probably when I was testing, I'm sorry!

I'm trying to understand why it doesn't do scraping correctly, I increased the delay to 15 seconds between user and user and I increased the scraping pass for the merits to 10 minutes

Hey Giammangiato.
Please be aware that merit notification is broken. No information displayed on the thread where I got merit(s) from or who sent those:

"(UP) MERITS 3296 your merit page https://bitcointalk.org/index.php?action=merit;u=1852120 You received merit(s) from"

I think this need an urgent fix!

[babo]

I've got the same problem.
I think you have to increase the delays again, try to increase the delays in the cronjob that you have set rather than putting sleeps.

In my experience, probabily the problem is here. Put 10 minutes and 5 minutes for the two cronjob.
Before it worked because you had few users, now you have more.

[giammangiato]

There are too many users, sorry. I had to bring the cronjob to 30 minutes hoping it will work, so you should get notifications every minute 0 and 30.

To understand if there are merits I make one call per user and if he has a change of amount, he makes another call again.

Alternative, is roll every minute a single user. Try in this way.

[Csmiami]

---

Seems to be working fine now; no duplicated notification and I can see how much merit, from who, and where it's been received

Keep up the great work!

[giammangiato]

I don't understand, I've increased the time a lot (every 30 minutes) but still can't complete
I am going crazy

I make another attempt, I leave the task every 30 minutes and decrease the delay from 15 sec to 5
do some tests, unfortunately I can't do anything else

update
i write random sleep from 5 to 10 seconds for every actions

[fillippone]

For me, merit notification has worked correctly for 24 hours last time you fixed it (on June 4th).
Then it has stopped working correctly since then.
I hope you can fined a good fix for this, because your work shouldn.'t be wasted!

[giammangiato]

@fillippone I sent you a merit, let me know if it got there right.
I still try to edit the delays after your feedback.
I have no other way unfortunately, the forum refuses me the call, maybe I make too many forum calls?

[fillippone]

@fillippone I sent you a merit, let me know if it got there right.
I still try to edit the delays after your feedback.
I have no other way unfortunately, the forum refuses me the call, maybe I make too many forum calls?

I received your merit and it was correctly notified.
I don't know anything about technical aspects, but I know @tryninja bot is working with 99 users according to his dashboard.

Maybe you'll find some interesting informations there.

[TryNinja]

@fillippone I sent you a merit, let me know if it got there right.
I still try to edit the delays after your feedback.
I have no other way unfortunately, the forum refuses me the call, maybe I make too many forum calls?

You are probably making too many requests in a short period of time.

I don't know how your bot works, so I can't help a lot, but mine has a scrapper checking the Recent Merits page every 5 seconds and scrapping every merit < 20 minutes old in there. I also scrape all new posts simultaneously, so it has all the information on the merited post (author being the most important). However, many posts can't be find in my DB since they can be older than the bot itself or something wrong may have happened (e.g connection issues), so if this is the case, the bot goes to the merited post and scrapes it, saving it in the database. If the author of the merited post is subbed to the bot, the bot saves the merit data to the database, otherwise, it just ignores it.

While all of this happens, another instance, which only controls the Telegram Bot logic, pulls all unnotified merits from the database (notified === false) and loops through them. If the author is subbed and has merit notifications turned on, the bot sends the notification and sets notified = true.

You need to set a delay between every request to the forum. I'm assuming what you do is check the user's profile for changes in the merit count, and if noticed an increase, goes to his merit page and scrappes the merit data. If that's the case, you need to set a delay between each time the bot checks an user profile and a delay before going to his merit page when noticed a change.

Ex:
- Checks user 1 profile; no merit change;
- Waits 5 seconds.
- Checks user 2 profile; noticed the merit changed;
- Waits 2 seconds
- Go to user 2 merit page; scrapes and sends notification.
- Waits 5 seconds.
- Checks user 3 profile; [...]

If that's not how you do it, you need to see where the bot is failing the requests, which error it's showing, and set a delay if the issue is the forum's rate-limit.