How usable is CLN currently (vs LND)?

[Jodmapht]

Hi,

I have installed CLN on my (RPI) node as it seemed to have a better performance and to have some neat features vs LND, but I seem to be constantly hitting road blocks with it, starting with being only limited to Zeus and the defunct Zap as  mobile wallets. RTL works well with it, but there is no support for ThunderHub, which seems to be better, and no access to Loop. My understanding of the issue is that CLN and LND are only compatible on the Lightning side and do not share a common RPC API for the client side, and many projects and services are developed by the LND team. So without Loop, how do users of CLN rebalance channels? I will be mostly receiving sats with my LN node... I am trying to see if I should keep trying to stick with CLN or give up and switch to LND before I start putting this into "production".

Thanks!

[1714880537]

1714880537

[1714880537]

1714880537

[1714880537]

1714880537

[HeRetiK]

I've been running a minor CLN node for a while and it definitely feels like the grass is greener on the LND side of things. At least without digging really deep into the guts of it.

The rebalancing I've been doing via commandline, though there might be a simpler way by now. Haven't followed the progress that closely lately.

[DaveF]

Take a look at: https://www.lightningnode.info/comparison
And at https://www.reddit.com/r/lightningnetwork/comments/u2zknw/clightning_vs_lnd/
There are also a few other discussions in general on the r/lightningnetwork subreddit about this topic.
Deep in the lightning discussion posts here there are also some comparisons.

With that being said, I have found clightning development a lot slower then LND but it is happening and things are being added. Just not as fast as LND
This probably has a bit to do with for a few years LND was the only option for the prebuilt node images, so you tended to see more work going towards that.

-Dave

[Jodmapht]

I've been running a minor CLN node for a while and it definitely feels like the grass is greener on the LND side of things. At least without digging really deep into the guts of it.

The rebalancing I've been doing via commandline, though there might be a simpler way by now. Haven't followed the progress that closely lately.

Thanks. When you say you have been rebalancing via commandline, how do you do that? For now I expect to mostly only receive LN payments and not have to send any myself, so I would be overall using it in a very unbalanced way...

Edit: Based on what I read from the links posted by DaveF, that would be PeerSwap, correct? Thanks!

[Jodmapht]

Take a look at: https://www.lightningnode.info/comparison
And at https://www.reddit.com/r/lightningnetwork/comments/u2zknw/clightning_vs_lnd/
There are also a few other discussions in general on the r/lightningnetwork subreddit about this topic.
Deep in the lightning discussion posts here there are also some comparisons.

With that being said, I have found clightning development a lot slower then LND but it is happening and things are being added. Just not as fast as LND
This probably has a bit to do with for a few years LND was the only option for the prebuilt node images, so you tended to see more work going towards that.

-Dave

Thanks for these links, they were very useful! They make me want to keep trying to stick with CLN. I did not know about Boltz.exchange. And also PeerSwap,

[HeRetiK]

I've been running a minor CLN node for a while and it definitely feels like the grass is greener on the LND side of things. At least without digging really deep into the guts of it.

The rebalancing I've been doing via commandline, though there might be a simpler way by now. Haven't followed the progress that closely lately.

Thanks. When you say you have been rebalancing via commandline, how do you do that? For now I expect to mostly only receive LN payments and not have to send any myself, so I would be overall using it in a very unbalanced way...

Edit: Based on what I read from the links posted by DaveF, that would be PeerSwap, correct? Thanks!

Actually I've been using the rebalance plugin, rebalancing individual channels manually:
https://github.com/lightningd/plugins/tree/master/rebalance

Running a larger node with good liquidity will definitely require some automation though and the LND ecosystem seems to have a lower learning curve in that regard.

[BlackHatCoiner]

CLN is like the lightweight, bug-safer LND. If you're running a node on a Pi, you'll notice difference in performance. While CLN does have plugins, they don't come pre-installed as in LND, because it focuses on efficiency. Stuff such as rebalance and backup-plugin must have a better recognition and development from the LND community though.

In terms of users, I think LND is more used.

[Jodmapht]

I've been running a minor CLN node for a while and it definitely feels like the grass is greener on the LND side of things. At least without digging really deep into the guts of it.

The rebalancing I've been doing via commandline, though there might be a simpler way by now. Haven't followed the progress that closely lately.

Thanks. When you say you have been rebalancing via commandline, how do you do that? For now I expect to mostly only receive LN payments and not have to send any myself, so I would be overall using it in a very unbalanced way...

Edit: Based on what I read from the links posted by DaveF, that would be PeerSwap, correct? Thanks!

Actually I've been using the rebalance plugin, rebalancing individual channels manually:
https://github.com/lightningd/plugins/tree/master/rebalance

Running a larger node with good liquidity will definitely require some automation though and the LND ecosystem seems to have a lower learning curve in that regard.

Thanks. My understanding though is that for what I describe, I need a swap and not a rebalancing between LN channels?

[Jodmapht]

CLN is like the lightweight, bug-safer LND. If you're running a node on a Pi, you'll notice difference in performance. While CLN does have plugins, they don't come pre-installed as in LND, because it focuses on efficiency. Stuff such as rebalance and backup-plugin must have a better recognition and development from the LND community though.

In terms of users, I think LND is more used.

Thanks! I opened a single (public) channel on CLN using RTL. It has been using 40%+ of the RPI's CPU and bugging down the SSD for over 1 hour. I am not sure what it could be doing given that there can't be anything going on in this channel since my node is not connected to any other channel and I am not doing any transaction? What am I missing?

[DaveF]

Thanks! I opened a single (public) channel on CLN using RTL. It has been using 40%+ of the RPI's CPU and bugging down the SSD for over 1 hour. I am not sure what it could be doing given that there can't be anything going on in this channel since my node is not connected to any other channel and I am not doing any transaction? What am I missing?

If you run top what process is grabbing the CPU?

Also, are you sure it's not thermal throttling? Can you run vcgencmd to check?
https://elinux.org/RPI_vcgencmd_usage

It's easy to get 40% load when the processor is running at 25% speed.

Last year I built a bunch of nodes in a box: https://bitcointalk.org/index.php?topic=5364113 and over time have had a few fan issues. Only once did it cause it to thermal throttle but when it did it caused what looked like a CPU spike but it was just running a such a slow speed that you could do anything.

-Dave

[Jodmapht]

Thanks! I opened a single (public) channel on CLN using RTL. It has been using 40%+ of the RPI's CPU and bugging down the SSD for over 1 hour. I am not sure what it could be doing given that there can't be anything going on in this channel since my node is not connected to any other channel and I am not doing any transaction? What am I missing?

If you run top what process is grabbing the CPU?

I see lightningd using between 40% and 100%, and bitcoind using about 15%.

https://elinux.org/RPI_vcgencmd_usage[/url]

It's easy to get 40% load when the processor is running at 25% speed.

Last year I built a bunch of nodes in a box: https://bitcointalk.org/index.php?topic=5364113 and over time have had a few fan issues. Only once did it cause it to thermal throttle but when it did it caused what looked like a CPU spike but it was just running a such a slow speed that you could do anything.

-Dave

It does not seem to be thermal throttling. It uses passive cooling through a case that is basically a big aluminum heat sink in thermal contact with all the heat-generating chips. I have never seen CPU temp above 45C I think. I looked many times today and the highest I saw was 42C. I can give a try with vcgencmd.

[Jodmapht]

No throttling (RPI4 with 8 GB of RAM, using UASP+Trim for SSD):

Code:

> for src in arm core h264 isp v3d uart pwm emmc pixel vec hdmi dpi ; do \
> echo -e "$src:\t$(vcgencmd measure_clock $src)" ; \
> done
arm:    frequency(48)=1500345728
core:   frequency(1)=500000992
h264:   frequency(28)=0
isp:    frequency(45)=0
v3d:    frequency(46)=499987808
uart:   frequency(22)=48001464
pwm:    frequency(25)=0
emmc:   frequency(50)=250000496
pixel:  frequency(29)=81000000
vec:    frequency(10)=0
hdmi:   frequency(0)=0
dpi:    frequency(4)=0

> vcgencmd measure_temp
temp=40.4'C

> top
Tasks: 178 total,   1 running, 177 sleeping,   0 stopped,   0 zombie
%Cpu(s): 17.8 us, 10.4 sy,  0.0 ni, 69.6 id,  [b]1.5 wa[/b],  0.0 hi,  0.8 si,  0.0 st
MiB Mem :   7862.9 total,     99.2 free,   2454.8 used,   5308.9 buff/cache
MiB Swap:   7872.0 total,   7872.0 free,      0.0 used.   5283.3 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                     
 417629 cln       20   0   98872  94388   3332 S  [b]38.3[/b]   1.2 105:12.34 [b]lightningd[/b]                                                                                                                                  
 415815 bitcoin   20   0 4350500 890128 113516 S  [b]15.2[/b]  11.1 104:25.51 [b]bitcoind[/b]

> iostat 1 1000

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          16.91    0.00    1.64    0.15    0.00   81.30

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
sda              22.92      4863.75       633.34       604.88 3876127573  504738369  482052560


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          16.24    0.00   15.74    1.52    0.00   66.50

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
sda             172.00     13656.00      7860.00     17028.00      13656       7860      17028

[DaveF]

When possible when posting things like that use the

Code:

 tags it makes it easier to read.

[code]
No throttling (RPI4 with 8 GB of RAM, using UASP+Trim for SSD):
> for src in arm core h264 isp v3d uart pwm emmc pixel vec hdmi dpi ; do \
> echo -e "$src:\t$(vcgencmd measure_clock $src)" ; \
> done
arm:    frequency(48)=1500345728
core:   frequency(1)=500000992
h264:   frequency(28)=0
isp:    frequency(45)=0
v3d:    frequency(46)=499987808
uart:   frequency(22)=48001464
pwm:    frequency(25)=0
emmc:   frequency(50)=250000496
pixel:  frequency(29)=81000000
vec:    frequency(10)=0
hdmi:   frequency(0)=0
dpi:    frequency(4)=0

> vcgencmd measure_temp
temp=40.4'C

> top
Tasks: 178 total,   1 running, 177 sleeping,   0 stopped,   0 zombie
%Cpu(s): 17.8 us, 10.4 sy,  0.0 ni, 69.6 id,  [b]1.5 wa[/b],  0.0 hi,  0.8 si,  0.0 st
MiB Mem :   7862.9 total,     99.2 free,   2454.8 used,   5308.9 buff/cache
MiB Swap:   7872.0 total,   7872.0 free,      0.0 used.   5283.3 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                     
 417629 cln       20   0   98872  94388   3332 S  [b]38.3[/b]   1.2 105:12.34 [b]lightningd[/b]                                                                                                                                  
 415815 bitcoin   20   0 4350500 890128 113516 S  [b]15.2[/b]  11.1 104:25.51 [b]bitcoind[/b]

> iostat 1 1000

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          16.91    0.00    1.64    0.15    0.00   81.30

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
sda              22.92      4863.75       633.34       604.88 3876127573  504738369  482052560


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          16.24    0.00   15.74    1.52    0.00   66.50

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
sda             172.00     13656.00      7860.00     17028.00      13656       7860      17028

Something is definitely not right. You should not be using that much CPU for either of them.
Stupid as it may be, can you reboot and see if it clears up. Could be something else hung, but that is a guess.

If you look at 1ml.com do you see the channel open?

I'm exhausted, but I'm sure someone else can chime in with other ideas.

-Dave[/code]

[Jodmapht]

Something is definitely not right. You should not be using that much CPU for either of them.
Stupid as it may be, can you reboot and see if it clears up. Could be something else hung, but that is a guess.

If you look at 1ml.com do you see the channel open?

I'm exhausted, but I'm sure someone else can chime in with other ideas.

-Dave

Code:

top - 00:31:48 up 11 min,  6 users,  load average: 1.04, 0.68, 0.48
Tasks: 175 total,   1 running, 174 sleeping,   0 stopped,   0 zombie
%Cpu(s): 29.0 us, 12.4 sy,  0.0 ni, 57.1 id,  0.9 wa,  0.0 hi,  0.7 si,  0.0 st
MiB Mem :   7862.9 total,     87.0 free,   1717.7 used,   6058.2 buff/cache
MiB Swap:   7872.0 total,   7872.0 free,      0.0 used.   6020.7 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                     
   1633 cln       20   0   71608  69236   5460 S  49.0   0.9   0:22.59 lightningd                                                                                                                                  
   1671 cln       20   0   20196  18856   3232 S  34.4   0.2   0:13.67 lightning_gossi                                                                                                                             
    747 bitcoin   20   0 3342684 633720 230944 S  13.2   7.9   1:36.80 bitcoind                                                                                                                                    
    712 debian-+  20   0   53320  49208  18376 S  10.6   0.6   0:41.84 tor

Thanks. This is strange, CPU usage was normal for about 5 minutes after the reboot, then it started again. Initially lightning_gossipd uses some CPU as well, then it goes away and lightningd becomes the main consumer.
Yes I see my channel as active on 1ml.com                    

[HeRetiK]

Actually I've been using the rebalance plugin, rebalancing individual channels manually:
https://github.com/lightningd/plugins/tree/master/rebalance

Running a larger node with good liquidity will definitely require some automation though and the LND ecosystem seems to have a lower learning curve in that regard.

Thanks. My understanding though is that for what I describe, I need a swap and not a rebalancing between LN channels?

Right, the method I describe is only for balancing between LN channels. To balance channels by moving coins on- and off-chain, similar to Loop's service, you'll need a swap.

[Jodmapht]

Something is definitely not right. You should not be using that much CPU for either of them.
Stupid as it may be, can you reboot and see if it clears up. Could be something else hung, but that is a guess.

If you look at 1ml.com do you see the channel open?

I'm exhausted, but I'm sure someone else can chime in with other ideas.

-Dave

Code:

top - 00:31:48 up 11 min,  6 users,  load average: 1.04, 0.68, 0.48
Tasks: 175 total,   1 running, 174 sleeping,   0 stopped,   0 zombie
%Cpu(s): 29.0 us, 12.4 sy,  0.0 ni, 57.1 id,  0.9 wa,  0.0 hi,  0.7 si,  0.0 st
MiB Mem :   7862.9 total,     87.0 free,   1717.7 used,   6058.2 buff/cache
MiB Swap:   7872.0 total,   7872.0 free,      0.0 used.   6020.7 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                     
   1633 cln       20   0   71608  69236   5460 S  49.0   0.9   0:22.59 lightningd                                                                                                                                  
   1671 cln       20   0   20196  18856   3232 S  34.4   0.2   0:13.67 lightning_gossi                                                                                                                             
    747 bitcoin   20   0 3342684 633720 230944 S  13.2   7.9   1:36.80 bitcoind                                                                                                                                    
    712 debian-+  20   0   53320  49208  18376 S  10.6   0.6   0:41.84 tor

Thanks. This is strange, CPU usage was normal for about 5 minutes after the reboot, then it started again. Initially lightning_gossipd uses some CPU as well, then it goes away and lightningd becomes the main consumer.
Yes I see my channel as active on 1ml.com                    

So when I look at the debugging information, I see thousands of lines like these

Code:

Updated pending announce with update 759871x572x1/0
Received channel_announcement for channel 759871x1337x0

and lots of lines like these as well

Code:

Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 766980x708x1/1
Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 765702x318x0/0

[DaveF]

Found this for an older version but from reading the issue may still exist.

https://github.com/lightningnetwork/lnd/issues/6837

https://github.com/Bitcoinite/RaspiBolt/commit/cc5374ff073c499f912d0321673283c9ab15d81b

Although there are a lot of helpful people here, looking at github to see if others have a similar issue can save you some time.
I personally don't contribute a lot to things there, but I do read a lot to find answers and comment if it works.


Also, see what db.bolt.auto-compact= is set to.
IIRC you need to set it to 1 to compact / clean the db.

-Dave

[Jodmapht]

Found this for an older version but from reading the issue may still exist.

https://github.com/lightningnetwork/lnd/issues/6837

https://github.com/Bitcoinite/RaspiBolt/commit/cc5374ff073c499f912d0321673283c9ab15d81b

Although there are a lot of helpful people here, looking at github to see if others have a similar issue can save you some time.
I personally don't contribute a lot to things there, but I do read a lot to find answers and comment if it works.


Also, see what db.bolt.auto-compact= is set to.
IIRC you need to set it to 1 to compact / clean the db.

-Dave

These seem to all apply to LND and not CLN though, correct? I can open an issue on the CLN's repository.

[Jodmapht]

Ok, so it looks like this could be all normal and due to the fact that my LN node is new. It is supposed to calm down...

[DaveF]

These seem to all apply to LND and not CLN though, correct? I can open an issue on the CLN's repository.

Sorry yeah, I am remote into my desktop that is logged into the forum and the last part "There might be something similar in CLightning" did not make it.

Ok, so it looks like this could be all normal and due to the fact that my LN node is new. It is supposed to calm down...

At least for the CLN part, I'm not sure why core is using anything. Mine only has activity when blocks are found and even then not much.

How are you powering the RPi? I have seen some 5V adapters cause issues.
This is one of my favorites.... 5% over voltage with no load, add a load and I can get it to 5.5V / 10% over voltage, then bad things^tm happen.



-Dave