c-lightning node issues

[darkv0rt3x]

Hello.

I was about to ask a few questions, in the FAQ thread, about my c-lightning node but I think a new thread is better. I'll keep the questions here instead of spamming the FAQ thread.

I installed my c-lightning implementation node from Elements Project in my Debian Buster.
I have configured it to run behind tor and I usually change the default ports whenever I can.

The issue is that I can connect to other nodes but shortly after I get disconnected. This in less than a minute.
When I run the command (to connect to a friend's node):

Code:

lightning-cli connect 02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2

I get the following output:

2020-10-13T22:51:54.003Z DEBUG   connectd: Now try LN connect out for host bf66hmp43bjegdkma6yuyjiseebfxoa6kxjdvbg55i36jc5b3anlgrqd.onion
2020-10-13T22:51:54.003Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-connectd: Connected out, starting crypto
2020-10-13T22:51:54.912Z DEBUG   hsmd: Client: Received message 1 from client
2020-10-13T22:51:54.912Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-connectd: Connect OUT
2020-10-13T22:51:54.912Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-connectd: peer_out WIRE_INIT
2020-10-13T22:51:55.693Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: seeker: chosen as startup peer
2020-10-13T22:51:55.693Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-connectd: peer_in WIRE_INIT
2020-10-13T22:51:55.693Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: seeker: starting gossip
2020-10-13T22:51:55.697Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-openingd-chan#1: pid 410, msgfd 39
2020-10-13T22:51:55.697Z DEBUG   hsmd: Client: Received message 10 from client
2020-10-13T22:51:55.697Z DEBUG   hsmd: Client: Received message 9 from client
2020-10-13T22:51:55.697Z DEBUG   hsmd: new_client: 1
2020-10-13T22:51:55.714Z DEBUG   hsmd: Client: Received message 18 from client
2020-10-13T22:51:55.715Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-openingd-chan#1: Handed peer, entering loop
2020-10-13T22:51:55.715Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-openingd-chan#1: peer_out WIRE_GOSSIP_TIMESTAMP_FILTER

shortly after, and some gossip later, I get this:

2020-10-13T22:53:47.624Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-openingd-chan#1: peer_out WIRE_ERROR
2020-10-13T22:53:47.624Z INFO    02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-openingd-chan#1: Peer connection lost
2020-10-13T22:53:47.624Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-openingd-chan#1: Status closed, but not exited. Killing
2020-10-13T22:53:47.626Z INFO    02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-chan#1: Owning subdaemon openingd died (9)
2020-10-13T22:53:47.626Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: reply_channel_range 645080+7537 (of 504500+148117) 5157 scids

and the command lightning-cli getinfo reports 0 peers connected.

Is this a problem on my side?

[1715012063]

1715012063

[jackg]

Have you tried leaving it on for a few minutes afterwards to see if it reconnects? And have you checked their node stays online (by asking them).

Do you have a channel open with them also, is there a reason your retrying to connect? Unless c-lightning implements a client that doesn't need to sync the blockchain you shouldn't need to connect until you have a channel open...

[darkv0rt3x]

Have you tried leaving it on for a few minutes afterwards to see if it reconnects? And have you checked their node stays online (by asking them).

Do you have a channel open with them also, is there a reason your retrying to connect? Unless c-lightning implements a client that doesn't need to sync the blockchain you shouldn't need to connect until you have a channel open...

Yes, his node is online and I can see it listed on 1ml.com site.
There is some gossip going on all the time but I'm new to the network and I don't know how to interpret the messages that comes out to the terminal.

I don't have an open channel with him, no. But by what you're telling me, our nodes doesn't need to stay connected until there is an open channel between us, is that it?
The important is that I connect to him and let him know "I'm here" Advertise me to the nodes you're connected to! I'l do the same." After this, the nodes an drop the connection and everything is still fine. Is this how it is supposed to be?

Also, something else we tried yesterday is that I gave him my node ID and he tried to connect to me but he got an error message saying that his 'myNode' interface could not find my node.

[mocacinno]

Since you're running a hidden service, i can't try to connect to your node as a test... The main thing i wanted to say is that Christian Decker is usually really fast and friendly when you bring such issues to his attention. He has always helped me whenever I had some issues/found some "bugs".
However, he doesn't monitor bitcointalk that closely. Bitcointalk is a great place to ask for assistance, but in c-lightning's case, posting your issue here might lead to a serious delay in support time...

Try opening an issue on github, and you'll probably get help much faster:
https://github.com/ElementsProject/lightning

[jackg]

Have you tried leaving it on for a few minutes afterwards to see if it reconnects? And have you checked their node stays online (by asking them).

Do you have a channel open with them also, is there a reason your retrying to connect? Unless c-lightning implements a client that doesn't need to sync the blockchain you shouldn't need to connect until you have a channel open...

Yes, his node is online and I can see it listed on 1ml.com site.
There is some gossip going on all the time but I'm new to the network and I don't know how to interpret the messages that comes out to the terminal.

I don't have an open channel with him, no. But by what you're telling me, our nodes doesn't need to stay connected until there is an open channel between us, is that it?
The important is that I connect to him and let him know "I'm here" Advertise me to the nodes you're connected to! I'l do the same." After this, the nodes an drop the connection and everything is still fine. Is this how it is supposed to be?

Also, something else we tried yesterday is that I gave him my node ID and he tried to connect to me but he got an error message saying that his 'myNode' interface could not find my node.

Yeah. When a node starts up on the lightning network (from the implementations I've seen) if you're connected - either by the way you're doing it or by having a channel) the node you connect to will send a list of other nodes and potentially routes but your node will also compute routes between them.

You could try opening a channel with him? If you both trust each other afaik the worst that can happen anyway is that one of you messes up and the other has to split the funds back to the other as they were when you made your last transaction (but this event in itself is extremely unlikely).

[darkv0rt3x]

Since you're running a hidden service, i can't try to connect to your node as a test... The main thing i wanted to say is that Christian Decker is usually really fast and friendly when you bring such issues to his attention. He has always helped me whenever I had some issues/found some "bugs".
However, he doesn't monitor bitcointalk that closely. Bitcointalk is a great place to ask for assistance, but in c-lightning's case, posting your issue here might lead to a serious delay in support time...

Try opening an issue on github, and you'll probably get help much faster:
https://github.com/ElementsProject/lightning

Well, but probably, if there is something wrong with my c-lightning node, is configuration related. Not that it's some c-lightning bug or so. I'm still very new to the network, so, there are many details that I'm not sure they are supposed to be like they are, such as this I just said in my OP.

I'm not sure, after what @jackg said, if my node is supposed to be connected all the time to another node when there are no open channels between both of us (me and my firend's node).

Another question I would like to ask, is if my node is supposed to be listed on these nodes list sites once it is connected (or supposedly connected) behind Tor network or if it is supposed to not be listed. I think it is not supposed to be listed there, but I just wanted some confirmation.

I'm not at home so I think I can't show here my node ID. However, I'll try to access my home computer via ssh and if I can get in, I can place here my node ID.

[darkv0rt3x]

Have you tried leaving it on for a few minutes afterwards to see if it reconnects? And have you checked their node stays online (by asking them).

Do you have a channel open with them also, is there a reason your retrying to connect? Unless c-lightning implements a client that doesn't need to sync the blockchain you shouldn't need to connect until you have a channel open...

Yes, his node is online and I can see it listed on 1ml.com site.
There is some gossip going on all the time but I'm new to the network and I don't know how to interpret the messages that comes out to the terminal.

I don't have an open channel with him, no. But by what you're telling me, our nodes doesn't need to stay connected until there is an open channel between us, is that it?
The important is that I connect to him and let him know "I'm here" Advertise me to the nodes you're connected to! I'l do the same." After this, the nodes an drop the connection and everything is still fine. Is this how it is supposed to be?

Also, something else we tried yesterday is that I gave him my node ID and he tried to connect to me but he got an error message saying that his 'myNode' interface could not find my node.

Yeah. When a node starts up on the lightning network (from the implementations I've seen) if you're connected - either by the way you're doing it or by having a channel) the node you connect to will send a list of other nodes and potentially routes but your node will also compute routes between them.

You could try opening a channel with him? If you both trust each other afaik the worst that can happen anyway is that one of you messes up and the other has to split the funds back to the other as they were when you made your last transaction (but this event in itself is extremely unlikely).

Ok, so I think my outgoing connections are just fine.
So, incoming connections might be a problem.
I can try to open a channel with him to see if he can connect with me. But as I said above, I gave him my node ID and his 'myNode' interface said he couldn't find my node. I also tried to ask at some other site for an inbound channel and they requested me to provide them my node ID and when I clicked to request the channel, I also got a message saying my node couldn't be found.

I ask another small question. When I give my node ID to others, is it mandatory that I provide the connection port too or the ID is enough be itself? I ask this because I can connect to other nodes just using their ID. No port is needed.

I should also mention that I'm not using default ports on my bitcoin node and if I conclude that all is fine with my c-lightning node, I'll probably try to change c-lightning node default ports too. Not sure if this might be a problem too, for incoming connections.

[jackg]

Ports are just extra identifiers for software. There's normally a protocol label or another piece of information within the packet/stack frame you transmit between each of you so the port probably isn't required. And if its a dot onion service, I doubt you're using the domain for anything else?

And I think it takes some time to get listed on the site with nodes, not sure how long or if you manually have to input it though.

[darkv0rt3x]

Ports are just extra identifiers for software. There's normally a protocol label or another piece of information within the packet/stack frame you transmit between each of you so the port probably isn't required. And if its a dot onion service, I doubt you're using the domain for anything else?

And I think it takes some time to get listed on the site with nodes, not sure how long or if you manually have to input it though.

No, I'm not using the .onion address for anything else. I have one created for the Bitcoin node and another one for the c-lightning node.

Hum, didn't know about that of waiting some tome for the node to be listed there. My node is more than 2 weeks now despite the fact that I haven't it online 24/7. At the beginning, and while experimenting and testing configurations, it was off and on all the time. The last 2 or 3 days it was most of the time online.

But will it be listed even if it's behind Tor? Well, my friend told me his node is also behind Tor and it shows up in that site. I thought it wasn't supposed that nodes behind Tor to be publicly listed...?

[jackg]

But will it be listed even if it's behind Tor? Well, my friend told me his node is also behind Tor and it shows up in that site. I thought it wasn't supposed that nodes behind Tor to be publicly listed...?

Tor hides YOU it doesn't hide the node from other peuole seeing it.

Your friends node advertised his onion to whoever he connects to afaik so they can build up potential routes through him.
I think another user here mentioned about not being listed though so it might just be random gør the site to load.

Did the channel open?

[darosior]

Hi,

Yeah as mentioned above opening an issue is generally preferable as it gets more eyeballs than here (if a thread here is noticed at all).

A few questions:
- What version are you running ?
- Did you try other peers as well ?
- What implementation is your friend running ? What version ?
- Could you provide slightly more logs (specifically the lines from `openingd` just before `peer_out WIRE_ERROR`) ?

Is this a problem on my side?

Yes. I can connect to your friend's node just fine using both v0.9.0-1 and v0.9.1 .

[darkv0rt3x]

But will it be listed even if it's behind Tor? Well, my friend told me his node is also behind Tor and it shows up in that site. I thought it wasn't supposed that nodes behind Tor to be publicly listed...?

Tor hides YOU it doesn't hide the node from other peuole seeing it.

Your friends node advertised his onion to whoever he connects to afaik so they can build up potential routes through him.
I think another user here mentioned about not being listed though so it might just be random gør the site to load.

Did the channel open?

Hum, ok. I honestly thought the node wouldn't be visible unless I provided my .onion address.

I'm at home but my friend is not online, apparently.

Hi,

Yeah as mentioned above opening an issue is generally preferable as it gets more eyeballs than here (if a thread here is noticed at all).

A few questions:
- What version are you running ? c-lightning v0.9.1-59-gd5cb0d8
- Did you try other peers as well ? Yes, I tried with a couple of nodes in 1ml.com site. I didn't notice if I get the same error as I stated in my OP but I know that when I connecte to those nodes before I go bed, and when in the next morning I come to see the output of lightning-cli getinfo and check for connected peers, it's zero'ed. So, all connections get, eventually lost!
- What implementation is your friend running ? What version ? Can't know right now. He's not online atm. I'll update here when I can reach him. I only know he's running LND in a raspberry and using myNode interface. (Hope LND and c-lightning implementations can coexist and communicate between each other).
- Could you provide slightly more logs (specifically the lines from `openingd` just before `peer_out WIRE_ERROR`) ? Sure. Will do outside the quote.

Is this a problem on my side?

Yes. I can connect to your friend's node just fine using both v0.9.0-1 and v0.9.1 . I can also connect to him. Problem is that I get disconnected shortly after and he cannot connect to my node.

So, right after I issue the lightning-cli connect command, I get this:

2020-10-14T18:35:48.800Z DEBUG   connectd: Now try LN connect out for host bf66hmp43bjegdkma6yuyjiseebfxoa6kxjdvbg55i36jc5b3anlgrqd.onion
2020-10-14T18:35:48.800Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-connectd: Connected out, starting crypto
2020-10-14T18:35:49.266Z DEBUG   hsmd: Client: Received message 1 from client
2020-10-14T18:35:49.267Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-connectd: Connect OUT
2020-10-14T18:35:49.267Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-connectd: peer_out WIRE_INIT
2020-10-14T18:35:49.878Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: seeker: starting gossip
2020-10-14T18:35:49.879Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-connectd: peer_in WIRE_INIT
2020-10-14T18:35:49.884Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-openingd-chan#3: pid 25466, msgfd 39
2020-10-14T18:35:49.884Z DEBUG   hsmd: Client: Received message 10 from client
2020-10-14T18:35:49.884Z DEBUG   hsmd: Client: Received message 9 from client
2020-10-14T18:35:49.884Z DEBUG   hsmd: new_client: 3
2020-10-14T18:35:49.900Z DEBUG   hsmd: Client: Received message 18 from client
2020-10-14T18:35:49.900Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-openingd-chan#3: Handed peer, entering loop
2020-10-14T18:35:49.900Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-openingd-chan#3: peer_out WIRE_GOSSIP_TIMESTAMP_FILTER
2020-10-14T18:35:55.140Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 652486x469x0/1
2020-10-14T18:35:55.141Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: Received node_announcement for node 02995ebaa827b658e6715371d0fc9b7d342c8b1f93650b92c7276ce5c02abd48cc
2020-10-14T18:35:55.142Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: Received node_announcement for node 03eaedf407e398fd32a68c3665f561029b7dd705c5143e1d42c9bc270d8327ef28
2020-10-14T18:35:55.142Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: Received node_announcement for node 03d51355cacc5c563354e9d208c79773aabbb8955e2d2836eefea2c2b04fda9e59
2020-10-14T18:35:55.142Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: Received node_announcement for node 021b330d8a3fb3884cde30a904051f2a5891c647f42e7f9631e2760ba62ef7e514
2020-10-14T18:36:00.288Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: Received node_announcement for node 038fed639d43b40621da90b309c40cc376e6d9623586d7002c362064ac73d70a8b
2020-10-14T18:36:03.927Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: seeker: state = PROBING_SCIDS Seeking scids 504500 - 652620
2020-10-14T18:36:03.927Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: sending query_channel_range for blocks 504500+148121
2020-10-14T18:36:04.842Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: reply_channel_range 504500+58958 (of 504500+148121) 8000 scids
2020-10-14T18:36:04.914Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: reply_channel_range 563457+25249 (of 504500+148121) 8000 scids
2020-10-14T18:36:05.057Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: reply_channel_range 588705+35565 (of 504500+148121) 8000 scids
2020-10-14T18:36:05.219Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: reply_channel_range 624269+20963 (of 504500+148121) 8000 scids
2020-10-14T18:36:05.599Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-openingd-chan#3: peer_out WIRE_ERROR
2020-10-14T18:36:05.599Z INFO    02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-openingd-chan#3: Peer connection lost
2020-10-14T18:36:05.599Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-openingd-chan#3: Status closed, but not exited. Killing
2020-10-14T18:36:05.599Z INFO    02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-chan#3: Owning subdaemon openingd died (9)
2020-10-14T18:36:05.600Z DEBUG   02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-gossipd: reply_channel_range 645231+7390 (of 504500+148121) 5102 scids

[darosior]

Ok, that's a known issue with LND, but i thought it was patched. Thanks.


The encountered error is due to a protocol violation by the node your friend is running. At some point either the LND implemention was sending some gossip messages in the wrong order (a channel update before a channel announcement).

c-lightning v0.9.1-59-gd5cb0d8

If possible, please downgrade to the last release (git checkout v0.9.1). You are currently running master.

Hope LND and c-lightning implementations can coexist and communicate between each other

They are! That's why there is a protocol 

[darkv0rt3x]

Ok, that's a known issue with LND, but i thought it was patched. Thanks.


The encountered error is due to a protocol violation by the node your friend is running. At some point either the LND implemention was sending some gossip messages in the wrong order (a channel update before a channel announcement).

c-lightning v0.9.1-59-gd5cb0d8

If possible, please downgrade to the last release (git checkout v0.9.1). You are currently running master.

Hope LND and c-lightning implementations can coexist and communicate between each other

They are! That's why there is a protocol 

Ohhhh, I never noticed this:

Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 652486x469x0/1

I know @darosior is already taking care of opening an issue about this problem in Github. Many thanks @darosior!