The Lightning Network node experience

[Rath_]

Happy to answer more questions -- oh, and feel free to open a channel to me. [...] I do require a minimum of 5M sats for any incoming channels.

I am actually thinking about opening a channel to your node. Is there any particular reason why you reject channels smaller than 5 million satoshi? Do you want to avoid frequent rebalancing this way?

[1715195198]

1715195198

[1715195198]

1715195198

[1715195198]

1715195198

[1715195198]

1715195198

[1715195198]

1715195198

[TheJuice]

Happy to answer more questions -- oh, and feel free to open a channel to me. [...] I do require a minimum of 5M sats for any incoming channels.

I am actually thinking about opening a channel to your node. Is there any particular reason why you reject channels smaller than 5 million satoshi? Do you want to avoid frequent rebalancing this way?

There are 4 major reason:

1) It's part of the game to get higher rankings. A lot of the rankings take into consideration average channel size. When I first started out I had the limit at 1M sats and have been slowly increasing. Higher rankings help with #4 below.
2) I've found that smaller channels almost never generate significant transactions -- I suspect this is both b/c smaller nodes open smaller channels and it's harder to keep sats on both sides of a small channel.
3) My rebalance script targets 2M sats on each side; just makes the scripting easy with no small channels.
4) Once you get a rather large node you start having "gravity", meaning people open channels to you randomly. This is great b/c your node grows without having to invest more capital. I found most times people open the smallest possible channel -- so by now using a 5M sat make people who want to connect open a larger channel than they might have otherwise. The other reason is closing channels can be expensive (esp force closed ones), so I don't want small nodes who may shut down in a month to open to me. I want long term node partners.

If BTC crosses 100k, I'll probably drop my min channel size to 4M sats.

I have considered if I should let people connect for less who want to use my node as a connection for their lightning node wallet (as opposed to another routing node) -- maybe one of these days i'll do that.

[Quickseller]

3) My rebalance script targets 2M sats on each side; just makes the scripting easy with no small channels.

What does your rebalance script do? Does it adjust fees once one side of a channel reaches a certain point? Or does it send a invoice to yourself that is paid to a channel with a high balance to a channel with a low balance?

[TheJuice]

3) My rebalance script targets 2M sats on each side; just makes the scripting easy with no small channels.

What does your rebalance script do? Does it adjust fees once one side of a channel reaches a certain point? Or does it send a invoice to yourself that is paid to a channel with a high balance to a channel with a low balance?

I have 2 -- one dynamically adjust fees based on the proportion of the sats on each side (and some other parameters). The rebalance script looks to get atleast 2M sats outgoing on each channel -- however it will only rebalance if the expected fees from future transactions are higher (by 3x) than the fee to rebalance.

[Rath_]

Yesterday, I performed a triangle swap with two other bitcointalk members. Each of us opened a 0.05 BTC channel to the other person. This way, we gained additional 0.10 BTC capacity, which means that this swap has doubled my node's capacity. I paid 222 sat for the opening transaction. We dropped the fees to 0 and TheJuice's script took care of the rebalancing. On the same day, I routed 6 transactions from the channel I opened and I got paid 222.76 satoshi. The channel paid itself off in just a few hours.

Note that on some days (ex. today) I routed a few small payments (< 1 sat). It's impossible to tell it from the graph at this point.

[Quickseller]

3) My rebalance script targets 2M sats on each side; just makes the scripting easy with no small channels.

What does your rebalance script do? Does it adjust fees once one side of a channel reaches a certain point? Or does it send a invoice to yourself that is paid to a channel with a high balance to a channel with a low balance?

I have 2 -- one dynamically adjust fees based on the proportion of the sats on each side (and some other parameters). The rebalance script looks to get atleast 2M sats outgoing on each channel -- however it will only rebalance if the expected fees from future transactions are higher (by 3x) than the fee to rebalance.

Very interesting. How does it anticipate the expected future fees? (what assumptions does it make)

I suspect that when LN has an increased adoption rate, the most successful LN node operators will automate their channel balances such that they are balanced (no pun intended), and that there is a low risk channels will be closed due to an unbalanced channel.

[DaveF]

For those who have not followed my other posts I have been running a few nodes in a box: https://bitcointalk.org/index.php?topic=5364113 today the fan in one of them failed: https://bitcointalk.org/index.php?topic=5364113.msg58485606#msg58485606

Since it's an RPi the fan is more or less not needed. BUT since it was making a really annoying noise I decided to close all the channels and shut it down. Well, since 2 of the other ends were offline I now waiting for them to close. Whatever, not a big deal. But, if I did have a lot of BTC locked up in a channel that I needed I could see that being a real annoyance. I know on an intellectual level that this could happen and it's just a wait it out. But it's still something people should think about. And since this is about the experience of running a LN node. Well, I guess dealing with things like this is worth a mention.

-Dave

[n0nce]

For those who have not followed my other posts I have been running a few nodes in a box: https://bitcointalk.org/index.php?topic=5364113 today the fan in one of them failed: https://bitcointalk.org/index.php?topic=5364113.msg58485606#msg58485606

Since it's an RPi the fan is more or less not needed. BUT since it was making a really annoying noise I decided to close all the channels and shut it down. Well, since 2 of the other ends were offline I now waiting for them to close. Whatever, not a big deal. But, if I did have a lot of BTC locked up in a channel that I needed I could see that being a real annoyance. I know on an intellectual level that this could happen and it's just a wait it out. But it's still something people should think about. And since this is about the experience of running a LN node. Well, I guess dealing with things like this is worth a mention.

-Dave

That's good to mention! I have had to force close channels before and indeed, sometimes it can be quite inconvenient and even a bit worrisome when you don't see the funds coming back for days the first time you do a force close. A buddy recently did his first force close and he was quite irritated as well. Definitely worth to consider, maybe even reducing lock time during channel creation if you anticipate the possibility of needing a quick force close in the future.

It can also be argued that it's a feature, not a bug, but yeah I see how it can be disturbing especially for large amounts of BTC.

[Rath_]

Definitely worth to consider, maybe even reducing lock time during channel creation if you anticipate the possibility of needing a quick force close in the future.

It's worth noting that you can make others wait longer to claim the funds than yourself. Currently, my node makes other people wait 432 blocks (~3 days), but I still accept channels with up to 2016 blocks (~2 weeks) delay. I don't mind waiting that long. Most people don't change the default value of 144 blocks anyway.

If you want to reject channels with excessive timelocks change either bitcoin.maxlocaldelay (LND) or max-blocktime-watch (c-lightning). If you want to force the other party to wait longer if they broadcast their commitment transaction change bitcoin.defaultremotedelay or watchtime-blocks (c-lightning). Note that those parameters do not affect existing channels.

[TheJuice]

Yesterday, I performed a triangle swap with two other bitcointalk members. Each of us opened a 0.05 BTC channel to the other person. This way, we gained additional 0.10 BTC capacity, which means that this swap has doubled my node's capacity. I paid 222 sat for the opening transaction. We dropped the fees to 0 and TheJuice's script took care of the rebalancing. On the same day, I routed 6 transactions from the channel I opened and I got paid 222.76 satoshi. The channel paid itself off in just a few hours.

Note that on some days (ex. today) I routed a few small payments (< 1 sat). It's impossible to tell it from the graph at this point.

Glad it's working out! I forwarded 288 payments yesterday so sometimes I miss individual ones! Keep growing that node!

3) My rebalance script targets 2M sats on each side; just makes the scripting easy with no small channels.

What does your rebalance script do? Does it adjust fees once one side of a channel reaches a certain point? Or does it send a invoice to yourself that is paid to a channel with a high balance to a channel with a low balance?

I have 2 -- one dynamically adjust fees based on the proportion of the sats on each side (and some other parameters). The rebalance script looks to get atleast 2M sats outgoing on each channel -- however it will only rebalance if the expected fees from future transactions are higher (by 3x) than the fee to rebalance.

Very interesting. How does it anticipate the expected future fees? (what assumptions does it make)

I suspect that when LN has an increased adoption rate, the most successful LN node operators will automate their channel balances such that they are balanced (no pun intended), and that there is a low risk channels will be closed due to an unbalanced channel.

Let's say I have a channel to node A. I set that fee to say 100 sats per 1M sats. If that channel is empty the script is confident that if it was repleted it would make 100 sats per 1M again. So it looks for a rebalance path that would be less than 40 sats/1M (I playing around with the cutoffs, but thats the general approach).

[Rath_]

I have been getting a lot of "Local_failed" routing attempts in RTL since the triangle swap. lightning-cli listforwards returns WIRE_TEMPORARY_CHANNEL_FAILURE for all of these recent transactions so I decided to investigate further. I turned on debug logging and here's what I found:

Code:

03562bdcf00fe0cf44e8a491a8c9b26f31c4e45c9a88cdfd6a2f0f2550a304c73e-channeld-chan#85: rcvd_revoke_and_ack: HTLC REMOTE 108 = RCVD_ADD_ACK_REVOCATION/SENT_ADD_ACK_REVOCATION 
[...]
037659a0ac8eb3b8d0a720114efc861d3a940382dcfa1403746b4f8f6b2e8810ba-channeld-chan#29: Adding HTLC 1126 amount=249816476msat cltv=711259 gave CHANNEL_ERR_CHANNEL_CAPACITY_EXCEEDED

This way, you can learn which channel failed to route the payment and how large it was. RTL shows only the "IN" channel.

Glad it's working out! I forwarded 288 payments yesterday so sometimes I miss individual ones! Keep growing that node!

Thanks! I won't be able to come close to your node's capacity, but I will do whatever I can to help the network.

[Rath_]

November was so far the best month. My node routed 59 transactions and collected 548.46 satoshi in fees. I earned the most from payments incoming from the channels which were opened this month.

[Rath_]

I decided to scrap my home server so I had to move my Bitcoin and Lightning nodes back to a Raspberry Pi. The migration process was painless. Here's what I did:

1) I made sure that my old VM won't boot anymore. You really don't want to accidentally run two instances at the same time or an outdated database.
2) I copied my hsm_secret to .lightning/bitcoin folder on my Raspberry Pi.
3) I ran ./backup-cli restore file:///mnt/external/location ~/.lightning/bitcoin/lightningd.sqlite3 to restore my channel backup.
4) I had to remove my backup file and initialize the backup plugin again: ./backup-cli init --lightning-dir ~/.lightning/bitcoin file:///mnt/external/location/file.bkp

My node is up and running with the same Tor address and channels. It is worth mentioning that LND docs explicitly state that migration between different operating systems and system architectures might lead to data corruption. I guess that's another legitimate reason to choose c-lightning over LND.

[n0nce]

~

Interesting. When I backed my C-Lightning node up and restored it recently, I just had to back up hsm_secret and the database. But I just copied the files after shutting down the service. No tools necessary. Worked flawlessly Just copied them to the new installation after it was done, and before starting it up; that's it.

[Rath_]

Interesting. When I backed my C-Lightning node up and restored it recently, I just had to back up hsm_secret and the database. But I just copied the files after shutting down the service. No tools necessary. Worked flawlessly Just copied them to the new installation after it was done, and before starting it up; that's it.

I could have done the same, but I wanted to test out the backup plugin in case I ever need to use it due to drive failure. Hopefully, this never happens.

[n0nce]

Interesting. When I backed my C-Lightning node up and restored it recently, I just had to back up hsm_secret and the database. But I just copied the files after shutting down the service. No tools necessary. Worked flawlessly Just copied them to the new installation after it was done, and before starting it up; that's it.

I could have done the same, but I wanted to test out the backup plugin in case I ever need to use it due to drive failure. Hopefully, this never happens.

Is that plugin just an alias for rsync? Or how is it more sophisticated than that? I'll look it up in the code later; I guess you can schedule it more simply to run periodically, right.

[Rath_]

Is that plugin just an alias for rsync? Or how is it more sophisticated than that? I'll look it up in the code later; I guess you can schedule it more simply to run periodically, right.

Still, if lightningd updates the database while rsync is running, it could result in a corrupted backup. The backup plugin uses a special hook which prevents the database from being modified.

[n0nce]

Is that plugin just an alias for rsync? Or how is it more sophisticated than that? I'll look it up in the code later; I guess you can schedule it more simply to run periodically, right.

Still, if lightningd updates the database while rsync is running, it could result in a corrupted backup. The backup plugin uses a special hook which prevents the database from being modified.

Oh, that's cool! Otherwise I'd have written a 3 line shell script that simply stops lightningd service, copies the stuff and starts the service up again. Should be equivalent in the end.

[Rath_]

December might look kind of underwhelming to you given my previous stats. I quickly ran out of liquidity this month and I didn't have much time to think about rebalancing my channels. I might end up letting the rebalance plugin run in the background to avoid this kind of situation again.

[JayJuanGee]

December might look kind of underwhelming to you given my previous stats. I quickly ran out of liquidity this month and I didn't have much time to think about rebalancing my channels. I might end up letting the rebalance plugin run in the background to avoid this kind of situation again.

I am a little confused about the point that you are making Rath_.

Of course, I don't run a node, so I am attempting to live somewhat vicariously through some of these lightning thread topics... at least this node one.  So maybe your point would be more clear for someone who is actually more interactive with some kind of personal node running.

You say this month, but we are January.

Of course your attached chart seems to show December information.. presumably the whole month.... in which it looks like you had a lot of activity in the first two days and then on the 7th but other than that you did not have a lot of activity.  Can you elaborate upon the meaning/interpretation a bit more?