How to deal with kernel updates

[seoincorporation]

This is a question I had since a long time ago, if I'm running a Linux distro with my Bitcoin Node each time I update de system and it has a Kernel update (which is really common) I have to reboot the system. Which means to stop the Bitcoin Core.

I would like to know how the community deal with kernel updates.

[1714867200]

1714867200

[1714867200]

1714867200

[DaveF]

You reboot, not a big deal. Unless you are running a mining pool a few minutes of downtime is not a big deal.
Even if you are running a lightning node a few minutes of being down here or there is not a big deal.

Any reason you are worried about it?

-Dave

[Foxpup]

Which means to stop the Bitcoin Core.

Then restart it, either manually or with a script to start it automatically on boot. Bitcoin Core will automatically resynchronise with the network and download all blocks and transactions it missed during the time it was offline; you do not need to be online continuously in order to receive transactions.

If for any reason you do need to be online continuously, the solution is the same as any other network service: run multiple servers with automatic failover (and don't upgrade them all at once).

[NotATether]

Red Hat distros have a particular Live Patch program that installs the kernel updates on the running kernel so that a reboot is not required. I do not believe it's available for other distros though, since Red Hat builds their own kernels.

Code:

yum install kpatch
yum kpatch auto

[stanner.austin]

For centos there is kernelcare package free & paid both available suitable for apply patch without reboot.

[ABCbits]

Just reboot it, but make sure Bitcoin Core exit cleanly. Init system such as systemd kill any process if it took longer than 90 second.

Red Hat distros have a particular Live Patch program that installs the kernel updates on the running kernel so that a reboot is not required.

IIRC it meant to fix critical CVE without rebooting and Red Hat recommend you to make a reboot during planned downtime.

I do not believe it's available for other distros though, since Red Hat builds their own kernels.

Code:

yum install kpatch
yum kpatch auto

Any distro based on Red Hat should have such feature. If you're Arch user, you could install and configure kpatch manually.

[bot0111]

I do not believe it's available for other distros though, since Red Hat builds their own kernels.

There are other programs for almost all other distros too. Some of them may require a license tough.
For more details:
https://blog.kernelcare.com/avoid-death-taxes-and-linux-server-reboots-kernel-updates-3-different-ways

[NotATether]

For centos there is kernelcare package free & paid both available suitable for apply patch without reboot.

Just reboot it, but make sure Bitcoin Core exit cleanly. Init system such as systemd kill any process if it took longer than 90 second.

Red Hat distros have a particular Live Patch program that installs the kernel updates on the running kernel so that a reboot is not required.

IIRC it meant to fix critical CVE without rebooting and Red Hat recommend you to make a reboot during planned downtime.

I do not believe it's available for other distros though, since Red Hat builds their own kernels.

Code:

yum install kpatch
yum kpatch auto

Any distro based on Red Hat should have such feature. If you're Arch user, you could install and configure kpatch manually.

I do not believe it's available for other distros though, since Red Hat builds their own kernels.

There are other programs for almost all other distros too. Some of them may require a license tough.
For more details:
https://blog.kernelcare.com/avoid-death-taxes-and-linux-server-reboots-kernel-updates-3-different-ways

kpatch & friends should be sufficient if you're just concerned about applying kernel security updates without reboots. And generally, if you're leaving a node and supporting software running for an extended amount of time, chances are, you don't want feature updates since they could break your workflow.

[seoincorporation]

Any reason you are worried about it?

Sometimes a reboot is a complex task, is not about stopping and starting bitcoin core, as a developer I'm running other services like SQL, Apache, JSNode, and Other services, and if you think about the fact that we have a kernel update really often, it would be nice to avoid restarting the workstation.

...

Code:

yum install kpatch
yum kpatch auto

My server is running Fedora, i will try kpatch, that should do the magic. And is important to mention that kpatch works in the next linux distros:

    Fedora, RHEL, CentOS
    Oracle Linux 7
    Ubuntu
    Debian 9 (Stretch)
    Debian 8 (Jessie)
    Debian 7 (Lenny)
    Gentoo

Source: https://github.com/dynup/kpatch

[dkbit98]

This is a question I had since a long time ago, if I'm running a Linux distro with my Bitcoin Node each time I update de system and it has a Kernel update (which is really common) I have to reboot the system. Which means to stop the Bitcoin Core.

You don't need to update Linux OS all the time, except it the case of some important security update, so you can disable it and do manual updates.
There is nothing wrong in restarting your computer and node sometimes, do some cleanup if needed and than quickly sync up your node after that.
I don't know what Linux OS you are suing but I prefer stable version that is rock solid and just works, unlike rolling release system that are often updated with latest software releases.

[Accardo]

From the things you listed that runs on your server you can consider a cluster network where one system will reboot without affecting a running program on the other systems. You can also consider using a Virtual IP to create a firewall or cluster router to make sure that they are no single point of failure. Also check this thread on stack

https://unix.stackexchange.com/questions/345561/how-linux-servers-update-their-kernel-without-rebooting

[seoincorporation]

From the things you listed that runs on your server you can consider a cluster network where one system will reboot without affecting a running program on the other systems. You can also consider using a Virtual IP to create a firewall or cluster router to make sure that they are no single point of failure. Also check this thread on stack

https://unix.stackexchange.com/questions/345561/how-linux-servers-update-their-kernel-without-rebooting

This solution is like having a clon of the server this way we can direct the IP to the second server while we update the other one. And this is a good solution if we have a public service, but in my case, I run the server for development, and isn't public at all. So, I don't think this is the best solution for me.

I was reading more about the topic and rebooting the system is the best way to deal with updates. The problem with the live patch for updates is not only the kernel, if I update any of the services I'm running (like SQL or Apache) I will have to restart the service to use the new version. That's why is important to restart after the updates.

[PrimeNumber7]

From the things you listed that runs on your server you can consider a cluster network where one system will reboot without affecting a running program on the other systems. You can also consider using a Virtual IP to create a firewall or cluster router to make sure that they are no single point of failure. Also check this thread on stack

https://unix.stackexchange.com/questions/345561/how-linux-servers-update-their-kernel-without-rebooting

This solution is like having a clon of the server this way we can direct the IP to the second server while we update the other one. And this is a good solution if we have a public service, but in my case, I run the server for development, and isn't public at all. So, I don't think this is the best solution for me.

I was reading more about the topic and rebooting the system is the best way to deal with updates. The problem with the live patch for updates is not only the kernel, if I update any of the services I'm running (like SQL or Apache) I will have to restart the service to use the new version. That's why is important to restart after the updates.

If your server is only used for development, as others have mentioned, your best bet is to just exit bitcoin core cleanly and restart. You can create a script to do this after hours at a time when the lowest number of developers are going to be using your server, and provide those who use your server advance notification of the downtime.

[n0nce]

Any reason you are worried about it?

Sometimes a reboot is a complex task, is not about stopping and starting bitcoin core, as a developer I'm running other services like SQL, Apache, JSNode, and Other services, and if you think about the fact that we have a kernel update really often, it would be nice to avoid restarting the workstation.

Honestly, no matter if you're a developer or not, if a reboot is such a big issue, your server isn't setup correctly. In my opinion, you should setup your stuff in a way that you can reboot (or boot e.g. after a power loss) at any time without issues.

I assume you have multiple tmux or screen sessions in the background, each running some important scripts. All of this type of stuff should be set up as a service so it's restarted automatically after booting, but also after a dropped internet connection or a power loss. Anything else will be pretty risky and unreliable in the long run.

[seoincorporation]

...
Honestly, no matter if you're a developer or not, if a reboot is such a big issue, your server isn't setup correctly. In my opinion, you should setup your stuff in a way that you can reboot (or boot e.g. after a power loss) at any time without issues.

I assume you have multiple tmux or screen sessions in the background, each running some important scripts. All of this type of stuff should be set up as a service so it's restarted automatically after booting, but also after a dropped internet connection or a power loss. Anything else will be pretty risky and unreliable in the long run.

You are totally right, the reboot isn't a big deal if our server is configured the right way.

A couple of days ago I configured my services, that way they auto-start when I reboot the machine, and now the only process that I have to start manually is JS-Node and the coin core. But all the other services are configured full auto now.

[n0nce]

Now the only process that I have to start manually is JS-Node and the coin core. But all the other services are configured full auto now.

Cool! I don't know what JS-Node is, but for Bitcoin Core if you need assistance, there's a section on it in my full node guide. Should be possible to copy the service file with minor modifications (username, path).