[TELEGRAM] Yet Another BitcoinTalk Notification BOT (merits, mentions, topics,+)

[OmegaStarScream]

Is there any reason behind scrapping (10 months old) posts only? Is it the storage you're worried about or just the time it might take to scrap everything? If the latter, I might be able to help.

[1714152116]

1714152116

[1714152116]

1714152116

[1714152116]

1714152116

[TryNinja]

Is there any reason behind scrapping (10 months old) posts only? Is it the storage you're worried about or just the time it might take to scrap everything? If the latter, I might be able to help.

Every post before May 2020 was provided to me by Loyce. But he only had/gave me 10 months worth of posts. I could scrape more myself - while also fixing old posts - , but time is indeed an issue. I would appreciate if there is any way you could help with that.

[LoyceV]

Every post before May 2020 was provided to me by Loyce. But he only had/gave me 10 months worth of posts.

I've now scraped 90% of the topics, I expect to be done in a couple of weeks. How are you on disk space?
Note: I didn't keep the original post date, but it's quite easy to get an estimate.
Note2: I almost forgot: when I'm done scraping, I have to re-do some local boards (with weird layout). This will take a bit longer.

[OmegaStarScream]

Every post before May 2020 was provided to me by Loyce. But he only had/gave me 10 months worth of posts. I could scrape more myself - while also fixing old posts - , but time is indeed an issue. I would appreciate if there is any way you could help with that.

I've now scraped 90% of the topics, I expect to be done in a couple of weeks. How are you on disk space?
Note: I didn't keep the original post date, but it's quite easy to get an estimate.
Note2: I almost forgot: when I'm done scraping, I have to re-do some local boards (with weird layout).

Not sure how you guys are doing it, but I was thinking of writing a small .NET Core console app to run it on multiple servers (different IPs). That's what I did when I made this list. Vultr allows up to 5 instances, but I believe that could be increased on request.

[TryNinja]

I've now scraped 90% of the topics, I expect to be done in a couple of weeks. How are you on disk space?
Note: I didn't keep the original post date, but it's quite easy to get an estimate.
Note2: I almost forgot: when I'm done scraping, I have to re-do some local boards (with weird layout).

Honestly, not that good.

My VPS plan only comes with 45 GB of storage and I'm already using 45% of it. The main issue is that the database indexes (that lets people search the database without needing to wait 5+ minutes) are using half of the storage. So, it basically doubles the size of the database. I sent an email to my VPS asking for their prices on additional space and they charge $0.09/month per extra GB.

edit: I was thinking about maybe getting a second VPS with more storage to host my files/database. E.g: $3.30/month for one with 256 Gb (up to 3 TB for $5) VS my current one where an extra 211 gb would cost another ~$19/month.

Not sure how you guys are doing it, but I was thinking of writing a small .NET Core console app to run it on multiple servers (different IPs). That's what I did when I made this list. Vultr allows up to 5 instances, but I believe that could be increased on request.

Is theymos ok with that? I could do something similar but with different proxies. That sounds like breaking his 1 request/sec rule, though.

[LoyceV]

Not sure how you guys are doing it, but I was thinking of writing a small .NET Core console app to run it on multiple servers (different IPs).

As far as I know, that's not allowed. I download 1 page, then wait a second. I'm already pushing the limits a bit by adding this to my "normal" scraping, but that's less than 1 page per second. I don't want to cause more load than I'm doing already.

[OmegaStarScream]

Is theymos ok with that? I could do something similar but with different proxies. That sounds like breaking his 1 request/sec rule, though.

As far as I know, that's not allowed. I download 1 page, then wait a second. I'm already pushing the limits a bit by adding this to my "normal" scraping, but that's less than 1 page per second. I don't want to cause more load than I'm doing already.

I did the same actually, I had a loop, and I limited the requests to one per second. But from my understanding, the 1 request/sec rule is for the IP addresses and not accounts? If you do many requests, your IP address will get blocked automatically after a while so even if you wanted to break it, you won't be able to but I'll P.M theymos and get a confirmation about this before starting anything. Thanks for bringing this to my attention.

[dre1982]

Just setup the bit. Sounds interesting. If you need a Dutch translation, just send me a dm.

[dkbit98]

@TryNinja

One thing I noticed is that some topics don't have title showing in telegram bot.
For example, let's take my topic : [LEARN] Phishing Quizzes - Beginners & Experts

This is info I get in telegram when I receive merit for that:
~Unknown Title~


Link is still correct, only thing that is missing is correct title name.

[TryNinja]

Link is still correct, only thing that is missing is correct title name.

This is an old post I scrapped from Loyce's files. Most of them have no title because he didn't included it on the file. However, I have made so the bot scrappes the post again and updates the database with the correct title when someone receives a merit for it. With the recent updates I have made (fixing most posts author uid), something had broken up. It should be fixed now. Thanks.

edit: Pretty much all posts have been updated with their author UID and a lot of them got their title fixed. This means that almost all users in the /addresses page will come with their UID when you copy the table bbcode.

[witcher_sense]

Update: Deleted posts notification! Suggested by @LoyceV here.

You can now choose to get a notification when a post you made vanishes because its parent thread got deleted. The notification will also have a link to a copy/archive of your original post (only the unedited original version, since I can't re-scrape posts to keep them updated).

- You will get a notification if you post in the thread X and the same thread gets deleted by a mod later.
- You will NOT get a notification if your post was manually deleted by a mod. The forum already PM's you about this.

Everybody has this option disabled by default. To enable it, just go to "Notifications" in the menu, and click "Enable Deleted Posts". If you can't see this option, run /menu and check again.

As explained by Loyce in his suggestion post, you will not get a notification if the author of the thread gets nuked OR if your post gets deleted by an user in his self-moderated thread. This is ONLY for when a mod manually deletes the whole thread and your post gets deleted in the process, which isn't notified by the forum.

Example:

Also worth mentioning: it only works for posts made in the last ~10 months (for now).

I will organize the new strings later for those who want to help translating them to your language.

If you find any bugs, please let me know.

Hi TryNinja! Thanks for the another great feature, but how exactly is it supposed to work? Recently, a thread from newbie has been deleted, so have my four posts with it. However, I haven't recieved a notification (deleted posts option is on).

As far as I know...

My posts weren't deleted manually by creator or moderator.

The creator of the said topic wasn't nuked https://bitcointalk.org/index.php?action=profile;u=2679380

[TryNinja]

Hi TryNinja! Thanks for the another great feature, but how exactly is it supposed to work? Recently, a thread from newbie has been deleted, so have my four posts with it. However, I haven't recieved a notification (deleted posts option is on).

Do you know when did this happen? I checked the modlog and I can't find anything about this thread being removed. The bot scrappes the modlog page and checks for entries about topics that got removed ("Remove topic: X in topic #Y by member #Z"). If this happened before I implemented the feature, he can't know it got deleted. I checked the database and I couldn't find it.

[winterlemon912]

Good to see that there is a notification with the use of the telegram because sometimes I do not have time to check on my profile what are the merits I have and what are the notifications came from my account it is very helpful to me. I'm looking for more updates on it thank you.

[witcher_sense]

Do you know when did this happen? I checked the modlog and I can't find anything about this thread being removed. The bot scrappes the modlog page and checks for entries about topics that got removed ("Remove topic: X in topic #Y by member #Z"). If this happened before I implemented the feature, he can't know it got deleted. I checked the database and I couldn't find it.

I am bit perplexed. The thread is new, it was created yesterday or so and got deleted several hours ago. I believe that feature was already implemented by that time.

The very last post is http://loyce.club/archive/posts/5470/54707466.html

How long does it take usually to scrap a modlog event and send a notification to particular user? Just noticed that my post count hasn't changed, it is supposed to be 4 posts less after removing of thread, isn't it?

[LoyceV]

I am bit perplexed.

I'd say open a new topic (in Meta), to find out what happened to this topic. It looks like it was deleted without showing up in modlog. I didn't know that's possible, but it sure seems like it now.

[OmegaStarScream]

I am bit perplexed.

I'd say open a new topic (in Meta), to find out what happened to this topic. It looks like it was deleted without showing up in modlog. I didn't know that's possible, but it sure seems like it now.

That topic has been moved to the "Trashcan" board (restorable), and not completely removed, that's why it's not showing in the modlog.

[LoyceV]

That topic has been moved to the "Trashcan" board (restorable), and not completely removed, that's why it's not showing in the modlog.

I thought all topics could be restored after deletion. Can you share the reason for using either trashcan or total delete?

This trashcan messes up notifications.

[OmegaStarScream]

Can you share the reason for using either trashcan or total delete?

We usually use the trashcan board when we want to leave a redirect message, like this one or, if we want a topic to be easily restorable (when/if needed).

I thought all topics could be restored after deletion.

I meant to say "easily" restorable, because a removed topic can only be restored by admins, while the ones moved to the trashcan can be restored by global mods as well.

Staff members like myself, can see the board and its content, but cannot restore topics from there.

[TheBeardedBaby]

Topic tracking is great but can we have a Section tracking??? Like for instance, if you want to buy some nice physical coin, then you can keep an eye on what people sell and you get notification for every new topic in Collectables

It's possible. I will take a look on how I can do this. Thanks!

Great!! Thank you

[TryNinja]

Topic tracking is great but can we have a Section tracking??? Like for instance, if you want to buy some nice physical coin, then you can keep an eye on what people sell and you get notification for every new topic in Collectables

It's possible. I will take a look on how I can do this. Thanks!