[Guide] C-Lightning backup guide

[n0nce]

Edit (08/22/2022): The guide has been updated for Core Lightning v0.11.2. It switched from backup plugin to new built-in backup functionality.

For completeness sake, since I already have a topic where I walk through my main node's setup, I am creating this separate guide for backing it up.
I generally follow the official guides, with a few tweaks here and there where I think makes sense. This one here, is based on https://lightning.readthedocs.io/BACKUP.html#sqlite3-wallet-main-backup-and-remote-nfs-mount.

It's worth noting depending on your hardware, it can be trivial to setup the system from the get-go to run on a RAID-0 array, which would mean a constantly up-to-date mirror of your boot drive, but this would be out of the scope of this little guide.

Part 1: hsm_secret backup.
The hsm_secret needs to be backed up once, e.g. on paper, best even multiple copies. Think of it as a seed word backup. It secures the on-chain balance, but we will need Part 2 to always have an up-to-date backup of our channels' states.

[1] Switch to unprivileged bitcoin user and extract the hsm_secret. By using this command, it will open in the text editor and not be saved to your history for better security. It is essential to now write down the hex data on a non-digital medium and afterwards closing vim without saving the file. Do so by opening vim's command prompt (colon), followed by q!, so the whole command will look like :q!↵.

Code:

su - bitcoin
cd ~/.lightning/bitcoin
xxd hsm_secret | vim -

Part 2: Automatic channel state backups

[1] Choose a location for your backup. It makes 0 sense to use a location on the same drive, so it either has to be an external drive (USB / LAN share) or a remote network share. Setting this up is again outside the scope of this guide. Let's assume the location is /mnt/share/lnbackup.

[2] Stop the node and go back into bitcoin user.

Code:

exit
sudo service lightningd stop
su - bitcoin

[3] Open up the Lightningd configuration file.

Code:

nano /home/bitcoin/.lightning/bitcoin/lightningd.conf

[4] Add the following line at the very bottom.

Code:

wallet=sqlite3:///home/bitcoin/.lightning/bitcoin/lightningd.sqlite3:/mnt/share/lnbackup/lightningd.sqlite3

make sure to set the correct path for your (network) share and call it like the original file, i.e. lightningd.sqlite3.

[5] Now we can start up C-Lightning again!

Code:

exit
sudo service lightningd start

[6] If you get no errors starting up the service or anything, you're good! For good measure, you could send yourself some sats from a mobile wallet, restart the service with sudo service lightningd restart and even restore the node from the backup, then check if everything's fine.

[xyris]

Hi, thanks, this is really helpful. I'm a (non-technical) Raspiblitz user and I don't have enough know-how to do a c-lightning backup just with the available (too technical) documentation I could find online.

Unfortunately I stumbled over this command:

Code:

pip install -r backup/requirements.tx

where I got this error:

Code:

ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'

Indeed, there is no file "requirements.txt" anywhere, I checked in directories backup and plugins

What is going wrong here? Please help, thanks!

[n0nce]

~

Oh, indeed, thanks for bringing this up! It seems the file has disappeared in the last two commits (after almost a year of silence): https://github.com/lightningd/plugins/commit/42f88ce24ab0469597bed3e90754a880ac9f956d

Now somehow, they switched to poetry instead of using Python's built-in pip. I generally prefer to stick with the popular, well-known and 'tried and tested' tools. You could certainly check-out the repo at an earlier point in time (before the two commits of last week) or try doing it with poetry.

I will update my guide as soon as possible, but I can't promise that it'll happen today.

[BlackHatCoiner]

This guide must be outdated. Once I run the backup-cli command, I get:

Code:

Traceback (most recent call last):
  File "backup/backup-cli", line 2, in <module>
    from backends import get_backend
  File "/home/bitcoin/lightning/plugins/backup/backends.py", line 5, in <module>
    from backend import Backend
  File "/home/bitcoin/lightning/plugins/backup/backend.py", line 7, in <module>
    from tqdm import tqdm
ModuleNotFoundError: No module named 'tqdm'

[n0nce]

This guide must be outdated. Once I run the backup-cli command, I get:

Code:

Traceback (most recent call last):
  File "backup/backup-cli", line 2, in <module>
    from backends import get_backend
  File "/home/bitcoin/lightning/plugins/backup/backends.py", line 5, in <module>
    from backend import Backend
  File "/home/bitcoin/lightning/plugins/backup/backend.py", line 7, in <module>
    from tqdm import tqdm
ModuleNotFoundError: No module named 'tqdm'

It is! I will add a red note to the original post, right now and update the guide as soon as possible. Sorry for the inconvenience!

[n0nce]

This guide must be outdated. Once I run the backup-cli command, I get:

Code:

Traceback (most recent call last):
  File "backup/backup-cli", line 2, in <module>
    from backends import get_backend
  File "/home/bitcoin/lightning/plugins/backup/backends.py", line 5, in <module>
    from backend import Backend
  File "/home/bitcoin/lightning/plugins/backup/backend.py", line 7, in <module>
    from tqdm import tqdm
ModuleNotFoundError: No module named 'tqdm'

It is! I will add a red note to the original post, right now and update the guide as soon as possible. Sorry for the inconvenience!

I just checked their repository and found out the installation guide is also outdated[1]. The guide mention you use pip to install requirements, but today they use Poetry[2] which is different library package manager.

[1] https://github.com/lightningd/plugins#additional-dependencies
[2] https://github.com/lightningd/plugins/blob/master/backup/poetry.lock

Actually, within the backup plugin's README file, it tells you to just use poetry install. And to then invoke ./backup-cli using poetry run ./backup-cli ....

But I remember that I did minimal modification to my original install / workflow which is described here, to get it working again. It could be as simple as: pip install tqdm after Step 2. @BlackHatCoiner, this is worth trying..

I will check my history file once I find the time and have SSH access to my node (the two are not always synchronous..  ).

[ABCbits]

--snip--

I just checked their repository and found out the installation guide is also outdated[1]. The guide mention you use pip to install requirements, but today they use Poetry[2] which is different library package manager.

[1] https://github.com/lightningd/plugins#additional-dependencies
[2] https://github.com/lightningd/plugins/blob/master/backup/poetry.lock

Actually, within the backup plugin's README file, it tells you to just use poetry install. And to then invoke ./backup-cli using poetry run ./backup-cli ....

Thanks for the correction, i didn't notice they put README.md file on their directory. Even so, i still find wording on main README.md file means you can install dependency for all plugin using pip

Additionally, some Python plugins come with a requirements.txt which can be used to install the plugin's dependencies using the pip tools:

But I remember that I did minimal modification to my original install / workflow which is described here, to get it working again. It could be as simple as: pip install tqdm after Step 2. @BlackHatCoiner, this is worth trying..

I'd advice to install exact version of tqdm (pip install tqdm==4.62.3) on poerty.lock file to avoid possible package conflict.

[n0nce]

Indeed, there is no file "requirements.txt" anywhere, I checked in directories backup and plugins

What is going wrong here? Please help, thanks!

This guide must be outdated. Once I run the backup-cli command, I get:

Code:

Traceback (most recent call last):
  File "backup/backup-cli", line 2, in <module>
    from backends import get_backend
  File "/home/bitcoin/lightning/plugins/backup/backends.py", line 5, in <module>
    from backend import Backend
  File "/home/bitcoin/lightning/plugins/backup/backend.py", line 7, in <module>
    from tqdm import tqdm
ModuleNotFoundError: No module named 'tqdm'

But I remember that I did minimal modification to my original install / workflow which is described here, to get it working again. It could be as simple as: pip install tqdm after Step 2. @BlackHatCoiner, this is worth trying..

I'd advice to install exact version of tqdm (pip install tqdm==4.62.3) on poerty.lock file to avoid possible package conflict.

Hey everybody! Guide fixed. At last. Replaced the backup plugin entirely in favor of the new built-in backup feature; this way we skip all this poetry packet manager stuff and the whole setup process is much cleaner and simpler to do.
I did spin up a whole new VM in which I installed Core Lightning as per my own install guide as well as following the steps I described here for backing up, so it should work, but if anyone's got any issues don't hesitate to ask.

[Jason Brendon]

[6] If you get no errors starting up the service or anything, you're good! For good measure, you could send yourself some sats from a mobile wallet, restart the service with sudo service lightningd restart and even restore the node from the backup, then check if everything's fine.

Question:

1. Is restoring the lightning node from the same device or even an entirely different device going to cause my existing channels to close forcibly?
2. How to restore a c-lightning node? Do i simply remove the hsm_secret file and that is called a restore?


Thanks guys.

[n0nce]

[6] If you get no errors starting up the service or anything, you're good! For good measure, you could send yourself some sats from a mobile wallet, restart the service with sudo service lightningd restart and even restore the node from the backup, then check if everything's fine.

Question:

1. Is restoring the lightning node from the same device or even an entirely different device going to cause my existing channels to close forcibly?

No, that's the idea of the backups. While your old / broken node is offline, the channel will just be offline. The counterparty could force-close, if they don't believe you're going to come back online, though.
If they don't, all the channels should be back just as they were before.

2. How to restore a c-lightning node? Do i simply remove the hsm_secret file and that is called a restore?
Thanks guys.

As the guide says:

To recover, simply get all the backup database files. Note that SQLITE3 will sometimes create a -journal or -wal file, which is necessary to ensure correct recovery of the backup; you need to copy those too, with corresponding renames if you use a different filename for the backup database, e.g. if you named the backup backup.sqlite3 and when you recover you find backup.sqlite3 and backup.sqlite3-journal files, you rename backup.sqlite3 to lightningd.sqlite3 and backup.sqlite3-journal to lightningd.sqlite3-journal. Note that the -journal or -wal file may or may not exist, but if they do, you must recover them as well (there can be an -shm file as well in WAL mode, but it is unnecessary; it is only used by SQLITE3 as a hack for portable shared memory, and contains no useful data; SQLITE3 will ignore its contents always). It is recommended that you use the same filename for both main and backup databases (just on different directories), and put the backup in its own directory, so that you can just recover all the files in that directory without worrying about missing any needed files or correctly renaming.

[Jason Brendon]

[6] If you get no errors starting up the service or anything, you're good! For good measure, you could send yourself some sats from a mobile wallet, restart the service with sudo service lightningd restart and even restore the node from the backup, then check if everything's fine.

Thanks mate. I mean how can i test a restore? By what? by deleting the hsm_secret, will i then be able to restore the node? Thanks man.

[n0nce]

[6] If you get no errors starting up the service or anything, you're good! For good measure, you could send yourself some sats from a mobile wallet, restart the service with sudo service lightningd restart and even restore the node from the backup, then check if everything's fine.

Thanks mate. I mean how can i test a restore? By what? by deleting the hsm_secret, will i then be able to restore the node? Thanks man.

Yes; first of all make sure you don't have a lot of funds on it.
Then, shut the node down and delete everything. You can even wipe the machine.

Or spin up a fresh Virtual Machine, or use a different device - you get the idea. The 'real' node must be offline for good, though.
Then restore your backup as explained in the guide and check that your channels are still there.

Now you know you have a good backup.
Again: it's not just about hsm_secret, if you don't have the channel state database, the channels are gone.

[BlackHatCoiner]

From lightning.readthedocs.io:

Recovery of the hsm_secret is sufficient to recover any onchain funds. Recovery of the hsm_secret is necessary, but insufficient, to recover any in-channel funds. To recover in-channel funds, you need to use one or more of the other backup strategies below.

To backup both off-chain and on-chain you need both the hsm_secret and the lightningd.sqlite3.

I'd advice Jason Brendon to test recovering a Lightning node (on-chain & off-chain funds) with fake money first (tBTC). That way, you don't need to be afraid of losing anything. That's why testnet exists. If you don't have tBTC, acquire here: https://bitcointalk.org/index.php?topic=5359454.msg57912622#msg57912622

[Jason Brendon]

From lightning.readthedocs.io:

Recovery of the hsm_secret is sufficient to recover any onchain funds. Recovery of the hsm_secret is necessary, but insufficient, to recover any in-channel funds. To recover in-channel funds, you need to use one or more of the other backup strategies below.

To backup both off-chain and on-chain you need both the hsm_secret and the lightningd.sqlite3.

I'd advice Jason Brendon to test recovering a Lightning node (on-chain & off-chain funds) with fake money first (tBTC). That way, you don't need to be afraid of losing anything. That's why testnet exists. If you don't have tBTC, acquire here: https://bitcointalk.org/index.php?topic=5359454.msg57912622#msg57912622

Well, thanks, buddy. yes tBtTC, I see veterans do experiments with testnet all the time. Me haven't tried anything like that. because i am afraid of ending up messing things up....

[BlackHatCoiner]

Well, thanks, buddy. yes tBtTC, I see veterans do experiments with testnet all the time. Me haven't tried anything like that. because i am afraid of ending up messing things up....

No problem. Just ask some in that faucet-thread.

[Jason Brendon]

hi op,

one question that often stays on top of my mind is that:

is the sq file constantly writing like every minute or second?
let's say I have just 5 channels open to the others. and let's say in the next 2 days, I am not receiving nor sending anything. So, the channel status stays the same, right?

my question is that: if i take the old sq file (2 days ago) and th hsm secret, will i be able to restore the node and not be punished for being caught cheating? (the sq file is 2 days old.)


need people with a confirmed answer. Thanks a lot!

[BlackHatCoiner]

[...]

The lightningd.sqlite3 file is updated whenever there is a change in the state of the channel (such as if you send a payment, receive a payment, or route a payment). However, if you're trying to backup your wallet, this way is not recommended by the developers. It is nowhere stated that copying the database file elsewhere is safe. The backup page describes that if you want to backup your wallet, use either the --wallet parameter properly, or the backup plugin.

[n0nce]

let's say I have just 5 channels open to the others. and let's say in the next 2 days, I am not receiving nor sending anything. So, the channel status stays the same, right?

Not entirely correct. Even if you are not sending / receiving, someone else may use your node to route their payment, thus changing your channel states.
The recommended backup methods all update the backup automatically whenever a channel state happens.

Your manual method would not work, if you only update backups whenever you make payments. You'd have to keep track of every payment going through your node, at which point you're simply coding the backup plugin all over again.

[Jason Brendon]

let's say I have just 5 channels open to the others. and let's say in the next 2 days, I am not receiving nor sending anything. So, the channel status stays the same, right?

Not entirely correct. Even if you are not sending / receiving, someone else may use your node to route their payment, thus changing your channel states.
The recommended backup methods all update the backup automatically whenever a channel state happens.

Your manual method would not work, if you only update backups whenever you make payments. You'd have to keep track of every payment going through your node, at which point you're simply coding the backup plugin all over again.

great. thank you two for the explanation.
One other question, considering that i don't have many channels open. Is there an intensive writing to the disk? the reason i am asking is i am considering buying a tf card or a usb. If it requires constant and high density writing, i need to buy a good tf card. If not, i will just buy a cheaper one..

[bnbstorm]

This seems interesting. I would try to use it for my next project. Is this guide still up to date?