The Lightning Network FAQ

[n0nce]

Easiest way to confirm that ports are the issue would be to shut down the main Core Lightning node and then start the lightningmobile one.

Tried it. The issue wasn't resolved. The mobile node couldn't startup.

From the log file and the behaviour of the SSH session where you start up the node, it appears to me that it does start up, just that lightning-cli has issues connecting to the RPC interface.
One way to make sure the node is running would be to have someone attempt opening a channel with your node or sending it a payment, just to make sure it's online.

[vv181]

it appears to me that it does start up, just that lightning-cli has issues connecting to the RPC interface.

I hardly understand C^[1], but I doubt it finished up the start-up process until it shows the output I mentioned above. @BCH could try to confirm it by deleting the lightning-rpc file and restarting the lightningd, if the file didn't get initialised I think it stuck on the gossip initialisation things.

@BlackHatCoiner, it is weird that on the first hand the node is working successfully but now isn't. Looking through the GitHub issues, issue #4810 which lead to #4822 is the closer that you got since it has a similar log that stuck at "REPLY WIRE_GOSSIPD_INIT_REPLY with 0 fds". But I don't think it has something to do with yours since those issues are relating to BTCPay Server or -plugindir, and also your main node is working fine.

I don't know where else to debug, but if I were you, see if the gossip_store file is safe to delete. If it is, I might just simply delete it, and try to run lightningd again. #4810 also has some way to trace/debug it further in case you are interested.

[BlackHatCoiner]

Which file contains the keys that grant me access to the channel? I have opened one private channel with my main node, worth of 300,000 sats. I've sent 150,000 sats, to make it balance, but I now want to empty it and retry, from a brand new node. Is it lightningd.sqlite3?

I have renamed ./lightningmobile/bitcoin to ./lightningmobile/bitcoin2, ran lightningd again, and it created a new bitcoin directory. The "Server started with public key x" appeared, but it currently has 0 channels. So, the access to the channel must be at bitcoin2. Note that the lightning-rpc isn't located at ./lightningmobile, but at ./lightningmobile/bitcoin for some reason I don't understand.

---------------
I'm far away from home and do everything with SSH via Tor. It goes so slowly...

[vv181]

Is it lightningd.sqlite3?

Yes. If the database file has other files that end with a -journal or -wal files, copy those too.

[BlackHatCoiner]

I renamed the new lightningd.sqlite3 to lightningd.sqlite4, copied the old lightningd.sqlite3 (which grants me access to the channel) to the bitcoin directory.

Like that:

I run the lightningd, debug.log stucks at the "gossipd: REPLY WIRE_GOSSIPD_INIT_REPLY" message again. Specifically:

Code:

2022-07-23T12:23:33.265Z DEBUG   lightningd: Opened log file /home/bitcoin/.lightningmobile/debug.log
2022-07-23T12:23:33.265Z DEBUG   lightningd: Opened log file /home/bitcoin/.lightningmobile/debug.log
2022-07-23T12:23:33.273Z DEBUG   plugin-manager: started(31731) /usr/local/bin/../libexec/c-lightning/plugins/autoclean
2022-07-23T12:23:33.283Z DEBUG   plugin-manager: started(31732) /usr/local/bin/../libexec/c-lightning/plugins/bcli
2022-07-23T12:23:33.291Z DEBUG   plugin-manager: started(31733) /usr/local/bin/../libexec/c-lightning/plugins/fetchinvoice
2022-07-23T12:23:33.300Z DEBUG   plugin-manager: started(31734) /usr/local/bin/../libexec/c-lightning/plugins/funder
2022-07-23T12:23:33.322Z DEBUG   plugin-manager: started(31735) /usr/local/bin/../libexec/c-lightning/plugins/topology
2022-07-23T12:23:33.351Z DEBUG   plugin-manager: started(31736) /usr/local/bin/../libexec/c-lightning/plugins/keysend
2022-07-23T12:23:33.372Z DEBUG   plugin-manager: started(31737) /usr/local/bin/../libexec/c-lightning/plugins/offers
2022-07-23T12:23:33.392Z DEBUG   plugin-manager: started(31738) /usr/local/bin/../libexec/c-lightning/plugins/pay
2022-07-23T12:23:33.417Z DEBUG   plugin-manager: started(31739) /usr/local/bin/../libexec/c-lightning/plugins/txprepare
2022-07-23T12:23:33.442Z DEBUG   plugin-manager: started(31740) /usr/local/bin/../libexec/c-lightning/plugins/spenderp
2022-07-23T12:23:33.580Z DEBUG   lightningd: testing /usr/local/libexec/c-lightning/lightning_channeld
2022-07-23T12:23:33.600Z DEBUG   lightningd: testing /usr/local/libexec/c-lightning/lightning_closingd
2022-07-23T12:23:33.622Z DEBUG   lightningd: testing /usr/local/libexec/c-lightning/lightning_connectd
2022-07-23T12:23:33.642Z DEBUG   lightningd: testing /usr/local/libexec/c-lightning/lightning_gossipd
2022-07-23T12:23:33.663Z DEBUG   lightningd: testing /usr/local/libexec/c-lightning/lightning_hsmd
2022-07-23T12:23:33.684Z DEBUG   lightningd: testing /usr/local/libexec/c-lightning/lightning_onchaind
2022-07-23T12:23:33.705Z DEBUG   lightningd: testing /usr/local/libexec/c-lightning/lightning_openingd
2022-07-23T12:23:33.726Z DEBUG   hsmd: pid 31748, msgfd 39
2022-07-23T12:23:34.010Z DEBUG   connectd: pid 31749, msgfd 43
2022-07-23T12:23:34.011Z DEBUG   hsmd: new_client: 0
2022-07-23T12:23:34.104Z DEBUG   connectd: Proxy address: 127.0.0.1:9050
2022-07-23T12:23:34.104Z DEBUG   connectd: Created IPv4 listener on port 9835
2022-07-23T12:23:34.104Z DEBUG   connectd: REPLY WIRE_CONNECTD_INIT_REPLY with 0 fds
2022-07-23T12:23:34.109Z DEBUG   gossipd: pid 31750, msgfd 42
2022-07-23T12:23:34.110Z DEBUG   hsmd: new_client: 0
2022-07-23T12:23:34.203Z DEBUG   gossipd: gossip_store_compact_offline: 0 deleted, 0 copied
2022-07-23T12:23:34.203Z DEBUG   gossipd: total store load time: 0 msec
2022-07-23T12:23:34.204Z DEBUG   gossipd: gossip_store: Read 0/0/0/0 cannounce/cupdate/nannounce/cdelete from store (0 deleted) in 1 bytes
2022-07-23T12:23:34.204Z DEBUG   gossipd: seeker: state = STARTING_UP New seeker
2022-07-23T12:23:34.204Z DEBUG   gossipd: REPLY WIRE_GOSSIPD_INIT_REPLY with 0 fds

[n0nce]

I run the lightningd, debug.log stucks at the "gossipd: REPLY WIRE_GOSSIPD_INIT_REPLY" message again. Specifically:

Code:

2022-07-23T12:23:33.265Z DEBUG   lightningd: Opened log file /home/bitcoin/.lightningmobile/debug.log
2022-07-23T12:23:33.265Z DEBUG   lightningd: Opened log file /home/bitcoin/.lightningmobile/debug.log
2022-07-23T12:23:33.273Z DEBUG   plugin-manager: started(31731) /usr/local/bin/../libexec/c-lightning/plugins/autoclean
2022-07-23T12:23:33.283Z DEBUG   plugin-manager: started(31732) /usr/local/bin/../libexec/c-lightning/plugins/bcli
2022-07-23T12:23:33.291Z DEBUG   plugin-manager: started(31733) /usr/local/bin/../libexec/c-lightning/plugins/fetchinvoice
2022-07-23T12:23:33.300Z DEBUG   plugin-manager: started(31734) /usr/local/bin/../libexec/c-lightning/plugins/funder
2022-07-23T12:23:33.322Z DEBUG   plugin-manager: started(31735) /usr/local/bin/../libexec/c-lightning/plugins/topology
2022-07-23T12:23:33.351Z DEBUG   plugin-manager: started(31736) /usr/local/bin/../libexec/c-lightning/plugins/keysend
2022-07-23T12:23:33.372Z DEBUG   plugin-manager: started(31737) /usr/local/bin/../libexec/c-lightning/plugins/offers
2022-07-23T12:23:33.392Z DEBUG   plugin-manager: started(31738) /usr/local/bin/../libexec/c-lightning/plugins/pay
2022-07-23T12:23:33.417Z DEBUG   plugin-manager: started(31739) /usr/local/bin/../libexec/c-lightning/plugins/txprepare
2022-07-23T12:23:33.442Z DEBUG   plugin-manager: started(31740) /usr/local/bin/../libexec/c-lightning/plugins/spenderp
2022-07-23T12:23:33.580Z DEBUG   lightningd: testing /usr/local/libexec/c-lightning/lightning_channeld
2022-07-23T12:23:33.600Z DEBUG   lightningd: testing /usr/local/libexec/c-lightning/lightning_closingd
2022-07-23T12:23:33.622Z DEBUG   lightningd: testing /usr/local/libexec/c-lightning/lightning_connectd
2022-07-23T12:23:33.642Z DEBUG   lightningd: testing /usr/local/libexec/c-lightning/lightning_gossipd
2022-07-23T12:23:33.663Z DEBUG   lightningd: testing /usr/local/libexec/c-lightning/lightning_hsmd
2022-07-23T12:23:33.684Z DEBUG   lightningd: testing /usr/local/libexec/c-lightning/lightning_onchaind
2022-07-23T12:23:33.705Z DEBUG   lightningd: testing /usr/local/libexec/c-lightning/lightning_openingd
2022-07-23T12:23:33.726Z DEBUG   hsmd: pid 31748, msgfd 39
2022-07-23T12:23:34.010Z DEBUG   connectd: pid 31749, msgfd 43
2022-07-23T12:23:34.011Z DEBUG   hsmd: new_client: 0
2022-07-23T12:23:34.104Z DEBUG   connectd: Proxy address: 127.0.0.1:9050
2022-07-23T12:23:34.104Z DEBUG   connectd: Created IPv4 listener on port 9835
2022-07-23T12:23:34.104Z DEBUG   connectd: REPLY WIRE_CONNECTD_INIT_REPLY with 0 fds
2022-07-23T12:23:34.109Z DEBUG   gossipd: pid 31750, msgfd 42
2022-07-23T12:23:34.110Z DEBUG   hsmd: new_client: 0
2022-07-23T12:23:34.203Z DEBUG   gossipd: gossip_store_compact_offline: 0 deleted, 0 copied
2022-07-23T12:23:34.203Z DEBUG   gossipd: total store load time: 0 msec
2022-07-23T12:23:34.204Z DEBUG   gossipd: gossip_store: Read 0/0/0/0 cannounce/cupdate/nannounce/cdelete from store (0 deleted) in 1 bytes
2022-07-23T12:23:34.204Z DEBUG   gossipd: seeker: state = STARTING_UP New seeker
2022-07-23T12:23:34.204Z DEBUG   gossipd: REPLY WIRE_GOSSIPD_INIT_REPLY with 0 fds

Alright, I just compared with my node's startup sequence and there are a few notable differences.
Three dots stand for a bunch of lines which I truncated, empty on one side means that that's something only the other user has in his log file.

What stands out at first is that your node somehow creates an IPv4 listener, but still tries to use Tor somehow. It's possible that this is the normal behaviour if you followed Core Lightning's Tor setup guide, though. I'm not sure since I have my own way of doing that. That could be something you may want to try. Basically manually creating a Tor service and tunneling the Lightning traffic through it.

Next up, I notice your connectd_init_done isn't executed, as well as much later in the process, the 'Server started with public key' message is missing. Though that's an INFO message, of which you have none. I'm not sure how that's possible, since debug should be a higher log level than info.

Code:

BlackHatCoiner														n0nce

DEBUG   connectd: Proxy address: 127.0.0.1:9050										DEBUG   connectd: Proxy address: 127.0.0.1:9050
DEBUG   connectd: Created IPv4 listener on port 9835									DEBUG   connectd: Created listener on 127.0.0.1:9735
DEBUG   connectd: REPLY WIRE_CONNECTD_INIT_REPLY with 0 fds								DEBUG   connectd: REPLY WIRE_CONNECTD_INIT_REPLY with 0 fds
						 									DEBUG   connectd: connectd_init_done
						 									INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
						 									DEBUG   lightningd: All Bitcoin plugin commands registered
															...
DEBUG   gossipd: pid 31750, msgfd 43											DEBUG   gossipd: pid 18646, msgfd 46
															...
DEBUG   hsmd: new_client: 0												DEBUG   hsmd: new_client: 0								
															...
DEBUG   gossipd: gossip_store_compact_offline: 0 deleted, 0 copied							DEBUG   gossipd: gossip_store_compact_offline: 350 deleted, 329864 copied
															...
DEBUG   gossipd: total store load time: 0 msec										
DEBUG   gossipd: gossip_store: Read 0/0/0/0 cannounce/cupdate/nannounce/cdelete from store (0 deleted) in 1 bytes	
DEBUG   gossipd: seeker: state = STARTING_UP New seeker									
															DEBUG   connectd: REPLY WIRE_CONNECTD_ACTIVATE_REPLY with 0 fds
															INFO    lightningd: --------------------------------------------------
															INFO    lightningd: Server started with public key [redacted]
DEBUG   gossipd: REPLY WIRE_GOSSIPD_INIT_REPLY with 0 fds								DEBUG   gossipd: REPLY WIRE_GOSSIPD_GET_ADDRS_REPLY with 0 fds

[vv181]

~

AFAIK, Raspibolt bitcoin user management doesn't make root accessible. Which ideally the lightningd should be exited due to a permission error, but that's not in your case.

Maybe, changing the permission to the default one could give any luck. Otherwise, I don't know where else to debug, n0nce post should point out where it's wrong, I got a similar start-up sequence with it.

Code:

DEBUG   connectd: connectd_init_done
INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
DEBUG   lightningd: All Bitcoin plugin commands registered

The default file permission:

Code:

chown bitcoin:bitcoin /home/bitcoin/.lightningmobile/lightningd.sqlite3
chmod 644 /home/bitcoin/.lightningmobile/lightningd.sqlite3

[BlackHatCoiner]

Raspibolt bitcoin user management doesn't make root accessible.

I've used to run chmod 777 to give all permissions. Bad practice? 

Maybe, changing the permission to the default one could give any luck.

Tried, but no lightning at the end of the tunnel.

What stands out at first is that your node somehow creates an IPv4 listener, but still tries to use Tor somehow.

At the time it was working, I had run a getinfo, and seen it listened on both 9735 (from Tor) and 9835 (from IPv4). I don't know if that's relevant.


Okay, so no sense of what has gone wrong. Plan C: Take my money back and hit the self-destruction button. If I force-close the channel, will I get access to the funds with no lightning daemon running?

[LoyceV]

Raspibolt bitcoin user management doesn't make root accessible.

I've used to run chmod 777 to give all permissions. Bad practice?  

Quite bad
I usually go for 755, I'd never give other users write-access to files owned by root. And of course I restrict user's home directories as much as possible (usually 700 or occasionally 750).

[ABCbits]

Raspibolt bitcoin user management doesn't make root accessible.

I've used to run chmod 777 to give all permissions. Bad practice?  

I'd say it's horrible practice. It could break application which have very strict permission check, although i'm not sure any LN implementation has such strict permission check. I remember my acquaintance broke their OS by chmod 777 /.

And of course I restrict user's home directories as much as possible (usually 700 or occasionally 750).

Do you ever see any application broke when you use 700? Usually i'd use 750.

[LoyceV]

Do you ever see any application broke when you use 700? Usually i'd use 750.

I must admit I don't usually use 700 for executables, but mostly for files. For most programs 755 is totally fine, and I haven't used 700 often enough to see if it breaks things, but I wouldn't do it to system files. Several services run from a different user, and that can't happen if they can't access the files they need.

[psycodad]

<snip>

And of course I restrict user's home directories as much as possible (usually 700 or occasionally 750).

Do you ever see any application broke when you use 700? Usually i'd use 750.

In most standard situation it makes no difference as it's the users own group owning his home directory. It starts to become interesting when your primary group has more than one member or you chown the group ownership of your home directory to some group with more than yourself as member. In the vast majority of standard setups both settings have an identical effect.

One thing to note is that omitting all access from world/other (the third byte) breaks certain things, like ssh key-auth or mail forwarding. So I've seen 751 or 711 as recommended alternative to 750/700 [1][2].

HTH

[1] https://unix.stackexchange.com/questions/315799/default-permissions-on-linux-home-directories
[2] https://unix.stackexchange.com/questions/95897/permissions-755-on-home-user

[n0nce]

Okay, so no sense of what has gone wrong. Plan C: Take my money back and hit the self-destruction button. If I force-close the channel, will I get access to the funds with no lightning daemon running?

I haven't had to restore a node yet, but force-closing a channel (from the other node of course), will send the funds to your node's on-chain wallet.
You can restore that using hsm-secret.

Here is more information on Core Lightning backups:
https://lightning.readthedocs.io/BACKUP.html

[BlackHatCoiner]

I've restored my hsm_secret to a brand new lightning node. I had previously force-closed my personal channel, so hsm_secret is enough to access the on-chain funds. Okay, so what's the next step? I run listfunds with the old hsm_secret and get nothing:

Code:

bitcoin@raspibolt:~/.lightningmobile2/bitcoin $ lightning-cli --lightning-dir=/home/bitcoin/.lightningmobile2 --conf=/home/bitcoin/.lightningmobile2/lightningd.conf listfunds
{
   "outputs": [],
   "channels": []
}

My main node has received the 150,000 sats.


BTW: It turns out that the sqlite3 file was the problem. I tried to use that old one, instead of the new, automatically created sqlite3. With the new, daemon starts normally. With the old, it's when I see it stuck.

[dkbit98]

Is anyone collecting links for all the open source payment processors that are supporting Lightning payments?
I know BTCPay server is probably the best option if you want to accept payments in Bitcoin and LN, but it can he heavier and more complicated to set up.
One alternative option I found before was Cypherpunkpay.org, but recently I found one more open source option called SatSale.
Code has MIT license, it's lightweight and written in Python, and it can accept on-chain and Lightning payments without any middleman or third party.

SatSale is still in early development but I like that we are starting to see more open source options, that will only improve overall ecosystem.
There is a demo page you can check to see how everything works:
https://try.satsale.org/


https://github.com/SatSale/SatSale

[n0nce]

I've restored my hsm_secret to a brand new lightning node. I had previously force-closed my personal channel, so hsm_secret is enough to access the on-chain funds. Okay, so what's the next step? I run listfunds with the old hsm_secret and get nothing:

Code:

bitcoin@raspibolt:~/.lightningmobile2/bitcoin $ lightning-cli --lightning-dir=/home/bitcoin/.lightningmobile2 --conf=/home/bitcoin/.lightningmobile2/lightningd.conf listfunds
{
   "outputs": [],
   "channels": []
}

My main node has received the 150,000 sats.


BTW: It turns out that the sqlite3 file was the problem. I tried to use that old one, instead of the new, automatically created sqlite3. With the new, daemon starts normally. With the old, it's when I see it stuck.

Still nothing? If I understand correctly, you had a single channel with your other node.
Your other (working) node initiated the force close and already received its part of the channel balance on-chain, confirmed?

It's normal that a force close takes some time to be completed, but if the working node already got the funds, I'm pretty certain the other one should get them at the same time, too. Maybe Rath_ can confirm.
By the way: get your node back up, Rath_! It went offline again a few days ago..

[darkv0rt3x]

Raspibolt bitcoin user management doesn't make root accessible.

I've used to run chmod 777 to give all permissions. Bad practice? 

Maybe, changing the permission to the default one could give any luck.

Tried, but no lightning at the end of the tunnel.

What stands out at first is that your node somehow creates an IPv4 listener, but still tries to use Tor somehow.

At the time it was working, I had run a getinfo, and seen it listened on both 9735 (from Tor) and 9835 (from IPv4). I don't know if that's relevant.


Okay, so no sense of what has gone wrong. Plan C: Take my money back and hit the self-destruction button. If I force-close the channel, will I get access to the funds with no lightning daemon running?

Hi. Can you wait until maybe after my lunch? I think we can try a fee other things before you go Mission Impossible with the destruct íon button. xD
Like, we can try to run only your problematic node with your main nide shut down and in the default port. Or we can try to check which ports are being used when you have your 2 nodes running to see what traffic is going where with nmap.

Another note is that this message

Code:

2022-07-17T18:36:04.268Z DEBUG   gossipd: REPLY WIRE_GOSSIPD_INIT_REPLY with 0 fds

is not bad. If I'm not mistaken, fds is a file descriptor and in C, a return value of zero is usually sign that things are ok, zero errors.

In Core Lightning github, there is a list with a description of a bunch of these messages. @_Rath knows is almost for sure out of his memory. I have to look for it. Maybe we can find the description for that message and we get an hint what what it really means to make sure there is no problem there.

[BlackHatCoiner]

Still nothing?

Yep.

Your other (working) node initiated the force close and already received its part of the channel balance on-chain, confirmed?

Yes. I've received the force-closing channel's funds from my main node, and so have from my mobile's node. It's just that I can't access them. Here's the multi-sig address to see yourself, no privacy concern: bc1q8lyz0239h864putwe9ue9zug75tmvg5mx8ug7uctxkyfe4jv2fgskrflw8.

As you can see, I've spent the 150,000 sats, since they've returned to my main node's wallet, but the other 149,695 remain untouched.


I made the question in stackexchange: https://bitcoin.stackexchange.com/questions/114709/how-do-i-gain-access-to-the-on-chain-funds-with-hsm-secret

[n0nce]

~snip~

Let's take a step back: are you sure that the restore of hsm_secret worked out?

You could compare hashes of the hsm_secret you had backed up and the one that is running on your node right now just to make sure.
The Core Lightning backup guide explains a few steps that you could potentially have missed and that might have caused the file not to be picked up (e.g. due to wrong permissions).
https://lightning.readthedocs.io/BACKUP.html#backing-up-your-c-lightning-node

Code:

sha256sum -b /home/bitcoin/.lightning/bitcoin/hsm_secret
sha256sum -b /home/bitcoin/hsm_backup/hsm_secret

Something like this and compare the two.

[darkv0rt3x]

So, while @BlackHatCoiner doesn't come back, I'll take the discussion somewhere else.

Today I wanted to test a loop I have with 4 other nodes, a so called pentagon. 4 + me.

So, with my LN node, I used getroute to see if I could get a route only using the peers of this pentagon, by sending a payment and providing the starting node as an extra parameter and kinda force the route to use the most possible peers in this pentagon, ideally all of them (I think there are at least 2 peers with other channels to other peers of this pentagon which could possibly shorten the number of hops used).

Let's call nodes A, B, C, D and E, where D is my node. So, my goal would be to get a route from D -> E -> A -> B -> C. This loop was opened with 1million sats.
So, I used the following command to try to get the route:

Code:

lightning-cli -k getroute id=node_C fromid=node_E msatoshi=700000000msat riskfactor=0

The result was:

Code:

{
   "route": []
}

I tried to lower the amount in 100k sats steps down to 100k sats, and also tried 500 sats, 100 sats, 1 sat, and they all returned empty route arrays.
I also tried to raise the max hops number up to 5000 just for the sake of confirmation (default is 20 hops according to docs) but still no route!
If I don't specify the fromid parameter, I can find routes but none are my loop peers!
I'm not sure I can conclude that node E is a bad routing node because it is quite a big node and with many channels!

You guys have any thoughts on this or want to make a similar experience and share results?
Like, I have a loop and its route is not found/selected to route any payments! This is kinda disappointing!