The Lightning Network FAQ

[fillippone]

Just to make you aware that in latest Bitcoin Optech Newsletter #66, we have full disclosure of LN vulnerabilities we have all been warned about during the last weeks:

Full disclosure of fixed vulnerabilities affecting multiple LN implementations: several weeks ago, the developers of C-Lightning, Eclair, and LND announced the previous discovery of an undisclosed issue in each of their implementations, which they had each fixed in a recent release. At that time, they strongly encouraged their users to upgrade (a message which Optech relayed) and promised a full disclosure in the future, which they have now done with an email by vulnerability discoverer and C-Lightning developer Rusty Russell and a blog post by LND developers Olaoluwa Osuntokun and Conner Fromknecht.

Briefly, the issue appears to have been that the implementations did not confirm that channel open transactions paid the correct script, amount, or both. Because of this, the implementations would accept payments within the channel which they would later be unable to get confirmed onchain, allowing them to be defrauded. As of this writing, Optech is not aware of any reports that this issue was exploited prior to the warning last month. Now that the issue has been disclosed, a PR has been opened to update the specification to note that this check is needed.

[Wind_FURY]

Another use-case that illustrates that the future of Bitcoin payments is off-chain, https://medium.com/lightnite/lightnite-a-bitcoin-integrated-battle-royale-game-2ec9f3746b0e?sk=2145c0ea5da25a65c6a2d7a54fa23be1

The game's online store will start a more liquid marketplace for online items, finding more price-efficiency for those items.

[d5000]

I have a question regarding LN which may deserve an own thread, but first (to avoid cluttering the Development forum) I ask it here.

Is it possible to use LN for offline payments?

The goal is to find a way for "pre-arranged" offline transactions in regions where internet connection is unstable/unreliable (people have asked me in the Spanish subforum with Venezuela in mind).

I had the following scheme in mind:

The situation

Alice wants to sell Bob a good, or sell BTC for fiat, but the trade will be carried out in an area without or with unreliable internet access. With on-chain transactions, there is a possible technique using timelocks and a secret to release the amount in question, but it requires a confirmed transaction on the blockchain (see here and for a similar example in another context, this proposal for payments of digital goods). This is not ideal because transaction fees would be paid even if the trade isn't carried out.

LN-based solution?

In a payment channel, we could arrange a "conditional payment" in the following way:

- Bob and Alice are connected by a payment channel.
- Bob (the buyer) sends Alice a signed commitment transaction, with a penalty like in a standard LN transaction, but a difference: the amount of the trade made conditional, requiring a secret to be spent (like in atomic swaps or the technique I just outlined). The transaction includes the hash of the secret.
- Bob and Alice meet in the place without Internet access.
-- If the trade takes place, Alice receives the secret from Bob, and she can verify it using the hash, with a simple electronic device not requiring internet access. She can now in theory release the funds including the amount of the trade with an on-chain transaction closing the channel. But instead, she tells the LN software the secret, and her balance is updated.
-- If the trade does not take place, Alice doesn't know the secret and thus cannot release the funds reserved for the trade, and thus she must use the last commitment transaction to close the channel, with the state of the channel before the trade.

I think with a simple payment channel this setup could work, if I've not overseen something. But is it also possible to route it via additional LN nodes? I don't know if the requirement of an additional secret would be possible to be "included" in the HTLCs necessary to route the payment through other LN nodes.

The solution would have a crucial advantage to the on-chain solution: it would not require any spendings of transaction fees, and in the case it's not carried out, not even LN fees would have to be paid.

[Carlton Banks]

- Bob and Alice are connected by a payment channel.

ok

- Bob (the buyer) sends Alice a signed commitment transaction, with a penalty like in a standard LN transaction, but a difference: the amount of the trade made conditional, requiring a secret to be spent (like in atomic swaps or the technique I just outlined). The transaction includes the hash of the secret.
- Bob and Alice meet in the place without Internet access.

hmmm, not seeing the necessity of this

if alice and bob have a payment channel open between one another, there's no need at all for additional commitment exchange, they can perform the zero hop routing operation in an ad-hoc network i.e. out of band at their meeting spot. If both Alice and Bob live in the same area, and both lost internet at the same time, then the exisiting protocol can work just as well offline as online (private channels are essentially doing exactly this. We typically assume they do it connected to the wider internet, but it's not strictly necessary)


People with intermittent/zero internet access could always try setting up a smaller private network of payment channels to circulate currency locally, but agreeing on what channels are valid is a difficult problem if they lose access to the internet. Freezing the blockchain state to before the time at which internet access was lost limits lightning usage to devices that were in those precise geographical regions when the outage hit, and potentially different (but contiguous) areas went out at different times. The shorter period of time for which access to the blockchain is curtailed, the more viable a local channel network is.

Still though, huge amounts of trade are often local only, so depending on how well connected channels are in a given locality (reflecting high proportion of entirely local trade), it could work pretty well. The correlation of areas affected by internet outages with micro-economies could well be pretty high in practice, i.e. if a whole town loses internet, the whole town may well also happen to do 60-80% of trade internally. Is it not the case that mesh networking is more likely to exist in places with poor access to the corporate owned internet?

[fillippone]

I have a question regarding LN which may deserve an own thread, but first (to avoid cluttering the Development forum) I ask it here.

Is it possible to use LN for offline payments?

I don't know if it is related to your question, but this actually just hit the news:

A form of offline payments is being developed for Bitcoin’s Lightning Network

Lightning Rod, a new offline-style payments protocol for the Bitcoin Lightning Network, will enable users to make Lightning payments to a payee without their mobile nodes being online at the same time.

The functioning is quite simple in principle:

The new protocol works by routing a Lightning network payment through an intermediate node. This intermediate node is always online and can securely forward a payment without needing that the payer and the payee maintain an active connection to the network.

It's not clear (to me, at least) if this solution is fully trustless. If it weren't it would be little more than a gimmick. Otherwise, another step in the right direction of a more flexible and user-friendly UX for the Lightning Network.

[d5000]

if alice and bob have a payment channel open between one another, there's no need at all for additional commitment exchange, they can perform the zero hop routing operation in an ad-hoc network i.e. out of band at their meeting spot.

Good idea, thanks! I will report this idea to the Spanish forum. It is surely appliable for many cases, above all in urban areas, but also in some rural use cases (e.g. a person buying essential goods at the same grocery with BTC like it seems to be the case in countries like Venezuela).

The question remains if the setup I outlined is compatible with the Lightning network, i.e. if there is a way to pay offline when there is no direct payment channel between both.

A form of offline payments is being developed for Bitcoin’s Lightning Network
[...]

The new protocol works by routing a Lightning network payment through an intermediate node. This intermediate node is always online and can securely forward a payment without needing that the payer and the payee maintain an active connection to the network.

Thanks, that sounds interesting, too, although I would need to know more details to know if it's applicable to the "offline payment" use case. The "forwarding", however, could indicate it's based on a similar principle like that I had in mind.

I have found the Lightning Rod protocol here at Github, but I must re-read it to fully understand it.

[Carlton Banks]

The question remains if the setup I outlined is compatible with the Lightning network, i.e. if there is a way to pay offline when there is no direct payment channel between both.

so here would be a way to make it work...

set up that private Lightning network, and keep it very exclusive to people within a given area. The crucial part would be: don't use it, except in outage situations. Use uptime of the general internet to store BTC in the private network, but use the regular Lightning network for everyday payments while internet remains up. Maybe a good idea would be to physically assemble people at a certain time every week to top up the liquidity in the private network, that way any disputes about which channel balances on the private local network are valid can be worked out there and then if internet dies coincidently while the actual meeting takes place.

Then, when internet goes down, this group of people have a pool of money that can reliably be exchanged in a zero hop fashion, as in my first example. And, if they can set up a meshnet, then they can achieve your suggestion: use multi-hop routing within the local private channel network to increase the liquidity

All of the above is possible today with some patching of Lightning software to switch between the mainnet and the private-net, and configuring devices and network routers to be capable of switching to mesh and/or adhoc modes Plus, the organizational task of setting and topping up the private channel network. Channel factories may serve as a fully packaged solution to that, but that's future stuff.

[samuel-sd]

Hi everybody, I set up a lightning node (CentOS + bitcoind + lnd + some scripts) and I have some question about configuring LND.
1. The port, which LND uses to listen to incoming connection is 9735. Is it tcp only port or tcp + udp?
2. The same question about watchtowers' port (9911)
3. How to securely backup offchain data? If something goes wrong with my node's software or hardware, how can I make sure if I have actual offchain data? I know that I will be punished for using outdated data even if I don't mean it. I found the channel.backup file but it's stored localy, what would I do if my ssd died?
4. I read about the watchtowers client, that it is supposed to "connect to backup encrypted justice transactions". I don't know what justice transactions are. Are they my offchain data or they are something else? Is watchtowers client help me to backup offchain data?

[Rath_]

1. The port, which LND uses to listen to incoming connection is 9735. Is it tcp only port or tcp + udp?

TCP should be enough just like for a Bitcoin full node.

2. The same question about watchtowers' port (9911)

Same as above.

3. How to securely beckup offchain data? If something goes wrong with my node's software or hardware, how can I make sure if I have actual offchain data? I know that I will be punished for using outdated data even if I don't mean it. I found the channel.backup file but it's stored localy, what would I do if my ssd died?

There is no safe method of channel backup at the moment. It's much easier for private channels since they don't participate in payment routing. If you lose a backup and don't want to try using the old one, the only "solution" is to wait for the other party to close the channel.

4. I read about the watchtowers client, that it is supposed to "connect to backup encrypted justice transactions". I don't know what justice transactions are. Are they my offchain data or they are something else? Is watchtowers client help me to backup offchain data?

Watchtowers do not store any backups of your channel. They simply monitor the network in case the other party of your channel attempted to cheat by broadcasting an old state of the channel. They store and publish justice transactions when needed.

Every time a Lightning Network transaction is settled, a commitment transaction (a transaction which settles the current balance of the channel) is signed and the revocation key for the previous transaction is shared. The commitment transaction also states what happens if the other party revokes the transaction or if they don't do it in x blocks (x=144 by default). A justice transaction revokes the outdated commitment transaction.

[samuel-sd]

Thanks for the answer with the explanation!

There is no safe method of channel backup at the moment.

I googled about possibilities to organize an offchain backup. The developers say:

After version v0.6-beta of lnd, the daemon now ships with a new feature called Static Channel Backups (SCBs). We call these static as they only need to be obtained once: when the channel is created. From there on, a backup is good until the channel is closed. The backup contains all the information we need to initiate the Data Loss Protection (DLP) feature in the protocol, which ultimately leads to us recovering the funds from the channel on-chain. This is a foolproof safe backup mechanism.

We say safe, as care has been taken to ensure that there are no foot guns in this method of backing up channels, vs doing things like rsyncing or copying the channel.db file periodically. Those methods can be dangerous as one never knows if they have the latest state of a channel or not. Instead, we aim to provide a simple, safe method to allow users to recover the settled funds in their channels in the case of partial or complete data loss. The backups themselves are encrypted using a key derived from the user's seed, this way we protect privacy of the users channels in the back up state, and ensure that a random node can't attempt to import another user's channels.

Thus, we can assume that the backup file has always actual data about our channels. So, we just need to put it in a safe place. I am thinking of setting up an NFS server somewhere in a cloud. That way I can mount a remote directory on my LND node and backup the offchain data to it.
What do you think about this solution?

[Rath_]

-snip

I totally forgot about this method, sorry! Since SCBs make use of a feature built into the protocol, they should be reliable. Personally, I have never had a chance to restore my node from such a backup. Once the DLP is initiated, the remote node sends the latest commitment transaction and forcefully closes the channel. The only problem with your proposal is that your NFS backup might become invalid anytime unless you stop participating in the payment routing. How often would you backup the file? Every time it was modified?

[samuel-sd]

Once the DLP is initiated, the remote node sends the latest commitment transaction and forcefully closes the channel.

I'm sorry, what is DLP?

The only problem with your proposal is that your NFS backup might become invalid anytime unless you stop participating in the payment routing. How often would you backup the file? Every time it was modified?

You did not get the idea; let me clarify.
I mounted a remote directory to my local directory /mnt/nfsstorage. So, everything I put in there ends up on my NFS server (in amazon cloud). Then I added the backup file path in the lnd.conf file (backupfilepath=/mnt/nfsstorage/lndbackup/channel.backup). The lnd service keeps channel.backup updated constantly. Thus, I do not need to copy anything; the backup file will be updated by lnd directly on the cloud server.

The only problem I might have is with connection to the cloud server. However, the NFS protocol allows to automatically reconnect to a server if the previous connection was interrupted (that's what manual says .
I installed the NFS server yesterday, It seems to be working fine.

PS I would have gladly put everything on a cloud if I had not needed to store the entire blockchain there. Keeping 300Gb on a cloud server is quite expensive.

[Rath_]

I'm sorry, what is DLP?

Data Loss Protection, it was mentioned in the LND v0.6 changelog which you quoted.

PS I would have gladly put everything on a cloud if I had not needed to store the entire blockchain there. Keeping 300Gb on a cloud server is quite expensive.

Actually, you don't have to run a full Bitcoin node if you are using LND! LND supports neutrino which lets you interact with the blockchain using third-party nodes. Keep in mind that it's still experimental but it's been vastly improved since it was released. You might give it a try. Thanks for clarifying your previous message.

[samuel-sd]

Actually, you don't have to run a full Bitcoin node if you are using LND! LND supports neutrino which lets you interact with the blockchain using third-party nodes. Keep in mind that it's still experimental but it's been vastly improved since it was released. You might give it a try. Thanks for clarifying your previous message.

I thought about it, I really did, but

lnd.conf

[neutrino]
; Connect only to the specified peers at startup. This creates a persistent
; connection to a target peer. This is recommended as there aren't many
; neutrino compliant full nodes on the test network yet.
; neutrino.connect=


So, LND needs a full node and I can specify the only one of them. What happens to my LND node if that neutrino node stops working? It would be much more convenient if I did not need to specify a node. I wish it could find a compliant full node on the network itself and connect to it. Like torrent clients do.

One more question about tor nodes. I often see in logs that my node fails to connect to nodes with tor addresses (*.onion). Do I need to choose between Tor and the regular Internet or I can install tor and connect to both of them?

[rdbase]

How was it even possible to lose bitcoin using this network?
This user reported they lost 4btc using lightning. https://bitcointalk.org/index.php?topic=5195729.msg52855065
Wasnt it created inorder to not being susceptible for such things to happen?

[d5000]

set up that private Lightning network, and keep it very exclusive to people within a given area. The crucial part would be: don't use it, except in outage situations. Use uptime of the general internet to store BTC in the private network, but use the regular Lightning network for everyday payments while internet remains up.

Thanks for the suggestion. If a mesh network can be established in the region, then this can be a good solution, but it would also be more suitable for urban areas (in most places of Venezuela, however, it should work). As far as I understand the LN architecture, that "private network" would use the same channels (i.e. the same multisig channel opening transactions) than the "real LN", only that the "communication layer" would change, so no additional on-chain TX would be necessary which is the important feature we are seeking ... (correct me if I'm wrong)

@rdbase: Seems to be the same case like the one Khaos77 mentioned, the user simply broadcasted the wrong transaction to close the channel. I would strongly favour to not discuss this topic here as it's the user's fault (it's possible however that the LN software could have been designed to prevent this).

[Carlton Banks]

How was it even possible to lose bitcoin using this network?

they didn't know the risks, and screwed up trying to close a channel using an outdated transaction (i.e. an old channel state)

This user reported they lost 4btc using lightning. https://bitcointalk.org/index.php?topic=5195729.msg52855065

yep, that was too risky considering they didn't close their channel competently. Every time you send a large amount of money anywhere, on any system, you should be triple-sure you know what you're doing.

Wasnt it created inorder to not being susceptible for such things to happen?

nope.

You're probably thinking of the eltoo form of Lightning, which avoids this problem. You cannot make that mistake using eltoo, but it's a long way off (new opcode required, and the details are still being debated, in an already long debate)

[Carlton Banks]

As far as I understand the LN architecture, that "private network" would use the same channels (i.e. the same multisig channel opening transactions) than the "real LN", only that the "communication layer" would change, so no additional on-chain TX would be necessary which is the important feature we are seeking ... (correct me if I'm wrong)

yes, you commit the channel states onchain, but do not connect to nodes on the "main" Lightning network (and so you don't announce your channels on the main network either). Of course, the opening tx's are can only be created using the public internet, but then your alternative Lightning network can be set up on a completely private network (and that comes with all the liquidity and trust trade-offs as mentioned before)

Connecting to the main Lightning network today is like a trip back to the 1990's; you look up a list of LN node IP addresses on a listing site, connect to one, it tells you the current (entire) channel map, then you start making channels at your discretion using the current map. There's absolutely nothing stopping you getting together with a bunch of other people and starting an alternative LN network from scratch, you just need LN nodes and each others IP addresses.

[samuel-sd]

Hi, I have been working on a project where I need to run lots of lnd services on a single machine (physical or virtual, does not matter). In theory, it seems easy, I created a directory structure for each node with its own conf file, tcp ports and so on. It was not difficult and I started 16 nodes for a test on a single VM machine. (CentOS 7)
After starting them, each node begins to update network graph (file channel.db in /data/graph/mainnet); this process is quite long and CPU consuming. After the graph is updated, an lnd service is supposed to go into idle mode with low cpu usage. In reality, this happens but not for a long time. At first, CPU usage stays low but only for a couple of hours, then all services one by one started loading CPU heavily with no reason. The lnd services will not stop doing that until I restart them, then everything happens all over again. One interesting thing is that the bitcoind service does not use almost any CPU resourses; it means that lnd services do not use bitcoind, when they load CPU. I could not find anything unusual in the lnd logs, everything seems normal.
I don't understand what's going on and ho to fix it. My host's hardware should be enought for the purpose (Core i3-3.8Ghz/16Gb RAM/SSD) Maybe somenody has already come accross with this issue. I fould the similar issue on the lnd's github page but there is no solution there.
https://github.com/lightningnetwork/lnd/issues/3370
I am interested to listen to everyone who has any thoughts about it or solutions how to fix it.

[Carlton Banks]

After the graph is updated, an lnd service is supposed to go into idle mode with low cpu usage. In reality, this happens but not for a long time. At first, CPU usage stays low but only for a couple of hours, then all services one by one started loading CPU heavily with no reason. The lnd services will not stop doing that until I restart them, then everything happens all over again. One interesting thing is that the bitcoind service does not use almost any CPU resourses; it means that lnd services do not use bitcoind, when they load CPU. I could not find anything unusual in the lnd logs, everything seems normal.

as you've seen, this is a known bug in lnd, for which there is currently no fix. I don't know if this is affecting arm64 builds (RasPi nodes are arm64), but if so this is pretty bad news, as most people are opt for lnd because the daemon is provided as a pre-built package (LL have been good in that respect, there are alot of OS's and platforms supported with builds and prebuilt binaries, don't know if that's because golang makes that easy, I suspect so)

try using c-lightning as the fix lnd has other resource usage issues right now (uses alot of RAM), if you're running multiple nodes I would guess resource usage per-daemon could be important to you. FWIW, I'm running c-lightning on a Raspberry Pi 3B+ (1GB RAM) with a handful of channels, so no problems on low resource devices. That's not to say that c-lightning is perfect (they've been wrangling bugs too), but resource usage has not been an issue. You'll need to sync the channel map as with lnd, but maybe you'll appreciate that you can use PostgreSql with the db seeing as you're doing something that sounds demanding