processWatchdog - Must have tool for everybody who runs bots

[tesla80]

Hello everyone,

Keeping your trading, alert, or analysis bots online and running optimally is crucial—downtime or unexpected failures can cost you dearly. I've developed a project that might be very beneficial for anyone running such bots. It's called Process Watchdog, a Linux utility designed to monitor your processes and automatically restart any that crash or stop sending a heartbeat. It also works with non-heartbeat sending processes.

With Process Watchdog, you get a flexible, real-time monitoring solution that logs key statistics, making it easier to track performance and diagnose issues. Additionally, simple file commands let you manually control process restarts, stops, or even a full system reboot if needed, ensuring minimal downtime.

I believe this tool will enhance your bot uptime and provide a reliable safeguard against performance hiccups.

Check it out and feel free to contribute or ask for features:

https://github.com/diffstorm/processWatchdog

Happy coding!

[ABCbits]

Hello everyone,

Keeping your trading, alert, or analysis bots online and running optimally is crucial—downtime or unexpected failures can cost you dearly. I've developed a project that might be very beneficial for anyone running such bots. It's called Process Watchdog, a Linux utility designed to monitor your processes and automatically restart any that crash or stop sending a heartbeat. It also works with non-heartbeat sending processes.

With Process Watchdog, you get a flexible, real-time monitoring solution that logs key statistics, making it easier to track performance and diagnose issues. Additionally, simple file commands let you manually control process restarts, stops, or even a full system reboot if needed, ensuring minimal downtime.

I believe this tool will enhance your bot uptime and provide a reliable safeguard against performance hiccups.

Check it out and feel free to contribute or ask for features:

https://github.com/diffstorm/processWatchdog

Happy coding!

I have few thoughts and questions.
1. What makes it different from tool such as systemd service? With systemd service, i can configure any script/executable to auto-start on boot and restart if it detect crash.
2. I know i'm being extremely skeptical, but how can i know you (tesla80) actually own account https://github.com/diffstorm?

[tesla80]

1. What makes it different from tool such as systemd service? With systemd service, i can configure any script/executable to auto-start on boot and restart if it detect crash.

Answer is simple.

While systemd can restart crashed processes, Process Watchdog goes beyond basic service management, making it ideal for trading, alert, and analysis bots:

• Heartbeat Monitoring – Unlike systemd, which only checks if a process is running, Process Watchdog ensures the bot is actually functioning by requiring it to send periodic heartbeats. If the process hangs but doesn’t crash, systemd won’t detect it—Process Watchdog will.
• Process-Aware Restarting – Some trading bots or scripts don’t fail completely but instead freeze or become unresponsive. Process Watchdog can restart them even when systemd wouldn’t.
• Lightweight & Easy to Use – Process Watchdog doesn’t require systemd configuration files or deep Linux knowledge. It works via simple file-based control, making it easier to integrate into custom setups.
• Statistics Logging – It logs process starts, crashes, and heartbeats, allowing you to track stability over time—useful for debugging and optimizing bot performance.
• Manual Control Options – You can send simple file-based commands to start, stop, restart, or even reboot the system as needed, adding flexibility beyond systemd.

systemd is good for auto-restarting, but Process Watchdog ensures your bots stay functional, not just running.
As it is a custom design system, it can answer any need, but systemd cannot.

I skip your second question as it does not matter who is who.

[mcdouglasx]

1. What makes it different from tool such as systemd service? With systemd service, i can configure any script/executable to auto-start on boot and restart if it detect crash.

Answer is simple.

While systemd can restart crashed processes, Process Watchdog goes beyond basic service management, making it ideal for trading, alert, and analysis bots:

• Heartbeat Monitoring – Unlike systemd, which only checks if a process is running, Process Watchdog ensures the bot is actually functioning by requiring it to send periodic heartbeats. If the process hangs but doesn’t crash, systemd won’t detect it—Process Watchdog will.
• Process-Aware Restarting – Some trading bots or scripts don’t fail completely but instead freeze or become unresponsive. Process Watchdog can restart them even when systemd wouldn’t.
• Lightweight & Easy to Use – Process Watchdog doesn’t require systemd configuration files or deep Linux knowledge. It works via simple file-based control, making it easier to integrate into custom setups.
• Statistics Logging – It logs process starts, crashes, and heartbeats, allowing you to track stability over time—useful for debugging and optimizing bot performance.
• Manual Control Options – You can send simple file-based commands to start, stop, restart, or even reboot the system as needed, adding flexibility beyond systemd.

systemd is good for auto-restarting, but Process Watchdog ensures your bots stay functional, not just running.
As it is a custom design system, it can answer any need, but systemd cannot.

I skip your second question as it does not matter who is who.

While you are correct in what you say, it is worth noting that this is designed for specific applications, which limits its applicability. Furthermore, if someone knows how to handle systemd, they can replicate some features of Process Watchdog. In the end, whether it is better or not depends on the situation and how the bot to be used is implemented, as it is not an integral solution.

[tesla80]

While you are correct in what you say, it is worth noting that this is designed for specific applications, which limits its applicability. Furthermore, if someone knows how to handle systemd, they can replicate some features of Process Watchdog. In the end, whether it is better or not depends on the situation and how the bot to be used is implemented, as it is not an integral solution.

In terms of process management, there is no case that systemd can handle but processWatchdog cannot. It is not for specific applications, it does not limit its applicability at all.
You can use systemd to restart crashing applications and you do not get any stats on it. Process Watchdog mainly focuses on keeping processes alive.
systemd is always an option of course and it was there for ages, it is harder to configure and use.

[mcdouglasx]

While you are correct in what you say, it is worth noting that this is designed for specific applications, which limits its applicability. Furthermore, if someone knows how to handle systemd, they can replicate some features of Process Watchdog. In the end, whether it is better or not depends on the situation and how the bot to be used is implemented, as it is not an integral solution.

In terms of process management, there is no case that systemd can handle but processWatchdog cannot. It is not for specific applications, it does not limit its applicability at all.
You can use systemd to restart crashing applications and you do not get any stats on it. Process Watchdog mainly focuses on keeping processes alive.
systemd is always an option of course and it was there for ages, it is harder to configure and use.

It is true that PW is simpler and more straightforward in its purpose, but stating that it does not limit applicability at all is debatable, as it depends on the specific needs of the user. systemd excels in managing service dependencies, enabling the configuration of timers, sockets, startup targets, and more. The robustness of systemd makes it preferable when the environment demands advanced functionalities and can also be configured to obtain statistics about bots with tools like journalctl, Prometheus, and Nagios. What I’m trying to convey is that depending on the requirements, one can be equally useful or a better choice than the other. It’s like saying Python is better than C++ just because it’s easier to use.

[tesla80]

Sure, processWatchdog is easy to use process manager with stats and heartbeats. It is not direct competitor of systemd

[ABCbits]

1. What makes it different from tool such as systemd service? With systemd service, i can configure any script/executable to auto-start on boot and restart if it detect crash.

Answer is simple.

While systemd can restart crashed processes, Process Watchdog goes beyond basic service management, making it ideal for trading, alert, and analysis bots:

• Heartbeat Monitoring – Unlike systemd, which only checks if a process is running, Process Watchdog ensures the bot is actually functioning by requiring it to send periodic heartbeats. If the process hangs but doesn’t crash, systemd won’t detect it—Process Watchdog will.
• Process-Aware Restarting – Some trading bots or scripts don’t fail completely but instead freeze or become unresponsive. Process Watchdog can restart them even when systemd wouldn’t.

FWIW, systemd actually also have those feature. But it require additional configuration and limited capability. See https://superuser.com/a/689209.

• Lightweight & Easy to Use – Process Watchdog doesn’t require systemd configuration files or deep Linux knowledge. It works via simple file-based control, making it easier to integrate into custom setups.
• Statistics Logging – It logs process starts, crashes, and heartbeats, allowing you to track stability over time—useful for debugging and optimizing bot performance.
• Manual Control Options – You can send simple file-based commands to start, stop, restart, or even reboot the system as needed, adding flexibility beyond systemd.

Good point. Based on my experience with systemd, it's definitely more complicated. And while systemd have logs (i don't remember what exactly is being logged), it can get tricky to filter and view it.

[NotATether]

Have you tested this monitor on a trading bot such as Gunbot? And how does it perform in this scenario?

Does it come with a web interface?

[tesla80]

Have you tested this monitor on a trading bot such as Gunbot? And how does it perform in this scenario?

Does it come with a web interface?

I haven't, but you can run Gunbot easily with processWatchdog, all the same process. Just you need to disable heartbeat monitoring by setting heartbeat_interval = 0, so processWatchdog wont expect any heartbeat messages from Gunbot. Please feel free to share if you run Gunbot with processWatchdog.

It does not have a web interface yet, a log file and stats file is what you can check if all good or not.

[tesla80]

🚀 processWatchdog just got even better!

Enjoy improved stability, enhanced efficiency, and exciting new features like periodic reboot. Don’t miss out—check it out now : https://github.com/diffstorm/processWatchdog

[tesla80]

CPU and Memory stats are added. Now it reports the max/min/avg/current CPU/RAM usages of every process.

https://github.com/diffstorm/processWatchdog

[tesla80]

processWatchdog v1.4.0 released!

New security and safety improvements, it is now more failsafe and secure!

https://github.com/diffstorm/processWatchdog

[Vod]

processWatchdog v1.4.0 released!

New security and safety improvements, it is not more failsafe and secure!

https://github.com/diffstorm/processWatchdog

Sorry to bump this thread from last year, but why do you monitor irrelevant things such as "max/min/avg/current CPU/RAM usages of every process"?   Maybe a software author would be able to recognize their parameters, but how do I know my bot is not just looping or crashing, still using resources?  You need to monitor the end result of the process.  Why not have the bot cache-bust the latest record in your database; if the timestamp is off, something is wrong.  That's what I did with my system indicators for mysql, redis and CDN.

[tesla80]

Sorry to bump this thread from last year, but why do you monitor irrelevant things such as "max/min/avg/current CPU/RAM usages of every process"?   Maybe a software author would be able to recognize their parameters, but how do I know my bot is not just looping or crashing, still using resources?  You need to monitor the end result of the process.  Why not have the bot cache-bust the latest record in your database; if the timestamp is off, something is wrong.  That's what I did with my system indicators for mysql, redis and CDN.

Thanks for your advice. CPU/RAM monitoring only shows a process is running, not that it's making progress. I added the Progress Signal feature.
Instead of database timestamps, your bot just appends `+` to its heartbeat when work completes.

The watchdog detects stagnation instantly, needs no infrastructure, works for any app type, and solves exactly the problem you described: distinguishing "running" from "making progress".

[batonidze]

Nice direction with the Progress Signal idea. That solves the classic "process is alive but logic is dead" problem much better than plain CPU/RAM checks.

A feature that helps a lot in production bot fleets is anti-flap control:
- exponential backoff on repeated restarts
- circuit-breaker mode after N failures in T minutes
- optional cooldown + manual ack before resume

Do you already have something like this in processWatchdog, or is it on the roadmap? For MM/arbitrage bots this can prevent bad loops from burning API limits and fees.

[tesla80]

Nice direction with the Progress Signal idea. That solves the classic "process is alive but logic is dead" problem much better than plain CPU/RAM checks.

A feature that helps a lot in production bot fleets is anti-flap control:
- exponential backoff on repeated restarts
- circuit-breaker mode after N failures in T minutes
- optional cooldown + manual ack before resume

Do you already have something like this in processWatchdog, or is it on the roadmap? For MM/arbitrage bots this can prevent bad loops from burning API limits and fees.

Yes, There are three layers of protection against a process repeatedly failing:

- Exponential backoff - configurable `max_retries` and `base_delay`, so repeated failures don't cause an immediate restart loop. The delay grows exponentially and is capped at 1 hour.
- Cooldown - after repeated restart/failure activity, the process can be put into a cooldown period before another automatic attempt. It can also be configured to require manual acknowledgement after N (or first) cooldowns.
- Circuit breaker - failures are counted within a configurable time window. Once the threshold is reached, the circuit opens and restarts are blocked. After the window expires it enters a half-open state and allows a single trial restart. If that succeeds it closes again; otherwise it opens again.

So for something like an arbitrage/trading bot, you can configure it so a broken deployment doesn't endlessly restart, hammer an API, or burn fees.