The Lightning Network node experience

[Rath_]

Note: This thread is self-moderated to prevent it from being derailed. Please, keep the discussion on Lightning implementations and your experience of running a node. Any related questions are welcome.

I have been running my current c-lightning node for over 4 months now so I would like to share some data with you. I would appreciate if other node operators could share their experience, thoughts and anything else like plugins, fee strategy, third-party software.

My node's URI is 0273da0a525390c36857841e208f1d289275c76ebfa7ecfde697c6cbf4f235b4f5@fjpmf23cd2c6sqp7e4wzexa6ixkttl3jabivldakwq733tw634ymjbid.onion:9735

My node has successfully routed 143 payments (until the 31st October) with ~0.09 BTC capacity and 5 active channels. Here's a breakdown of my earnings from payment routing into each month:

July: 539.54 sat/17 forwards



August: 114.64 sat/34 forwards



September: 161.75 sat/28 forwards



October: 75.56 sat/64 forwards



The largest payment I have ever routed was 960,320 satoshi and I got paid 77.825 satoshi for it. The highest fee I have ever charged was 91.476 satoshi for a 952,385 satoshi payment. 23 forwards were free of charge. As you can see, I made the highest profit in July. That's probably because I was adjusting my fee policy for each channel manually every few days unlike the other months - I adjusted my fee policy only at the beginning and at the end of each month. I haven't used any fee auto-adjusting plugins yet.

While it's true that earnings from my Lightning node do not cover the electricity cost of my server (which I also use for other projects anyway), I have already earned back the fees I paid to open my channels. Note that some of my channels were not opened by me which means that I did not have to pay anything for them. Also, I saved some money by opening more than one channel in a single transaction and opening dual-funded channels so that I would not have to worry about using paid third-party services to balance my channels.

You might be surprised by the number of failed forwards - 389 payments. Why are there so many of them? More than 95% of them failed at some further point in the route. It's difficult to tell how many of them were legitimate payments and not channel/route probing attempts. I used some website a couple of time to probe routes from/to my node and each time I would get another three failed payments.

[DaveF]

Quoting myself from earlier today:

...I have seen a major uptick in transactions routing through my nodes over the last few weeks. Not a major amount, but where my 3 nodes would do 5 a month I have been averaging about 7 or 8 a week since mid October. Anyone else seeing this?

-Dave

So you seem to be both routing and charging a lot more on average then me.
Most of my fees are 1 or 2 sat. I have it set to minimum.

Are you seeing any bursts like this?:



I posted about it back in January and still see these kind of transactions in bursts, then they stop again.
https://bitcointalk.org/index.php?topic=5158920.msg56082955#msg56082955

-Dave

[Rath_]

Most of my fees are 1 or 2 sat. I have it set to minimum.

I have recently set my fees to 0/0 for some of my channels because they become totally unbalanced. I usually keep my fee rates around 15-30 ppm with one exception - the channel with bfx-lnd1 node. I routed quite a few transactions through it with a fee rate > 90 ppm. All of my channels have 0 msat base fee now.

Are you seeing any bursts like this?:

No, I hardly ever route payments lower than 10k satoshi. There is usually an interval of a few hours between them.

[DaveF]

No, I hardly ever route payments lower than 10k satoshi. There is usually an interval of a few hours between them.

I think the largest I ever routed was in the low 30k range. Most of mine seem to be in the 1500 sat or lower range.
On the other hand I have under 150 failed routing attempts combined since the beginning of the year.

Guess it's just what other nodes we are connected to and how much is in the channels.

That or nobody loves me :-(

Will be interesting to see what other people have seen with their nodes.

-Dave

[ajas]

I am running a lightning node (LND) for a long time.
But I didn't look too much into details of its operation
but I actually would like to do that if it does not take too much effort.

Are there any tools available to diagnose the performance of an LND node in some detail ?
(usage history of channels etc.)
Probably there are. Can anybody provide any info on that ?

Thanks in advance, A.

[n0nce]

Guess it's just what other nodes we are connected to and how much is in the channels.

Yeah, it depends a ton about who you're connecting to and also the size of those channels. To maximize routing and profits, some people actually manually monitor their channels. Some recommendations I picked up:

* Don't create channel between 2 already connected nodes. (e.g. check via LN explorers)
* If a channel isn't routing anything for a long time, close it.
* If a channel starts to get used much, but only in one direction, rebalance it so it continues being used.
* Look for nodes with much usage, but no 1-hop connection and connect them through you.
* Adapt fees up / down in the channels that are used much, like sometimes your fees are just too high; sometimes users keep paying if you increase them since you're the only route to some location.

This is just off the top of my mind, not sure if I can find the original 'list' anymore. We may just create a new thread for that. I don't feel qualified for it myself though, since I was never a big routing node myself.

With regards to 'LN node experience', I will post in here in the future when I find the time to write down all my experiences and make a concise but informative post of it.

[Rath_]

Probably there are. Can anybody provide any info on that ?

The easiest and the fastest way for you would be to use the lncli fwdinghistory command. If you have some spare time, you could install either ThunderHub or RTL. I haven't used the former one since it's LND exclusive but it should be as good as RTL or even better for channel management. It will be much easier for you to interpret all the data if you use either of them. The graphs I posted were created by RTL.

[Rath_]

Whoops, I missed n0nce's reply.

* Don't create channel between 2 already connected nodes. (e.g. check via LN explorers)

That's exactly what I had been avoiding until I made a mistake. I opened a channel to both Bitfinex and Nicehash. I had been routing fairly large payments between those two channels until Bitfinex bumped their fee rate from 1 ppm to 100 ppm. While I was trying to figure it out, I learnt that Nicehash and Bitfinex have had a channel for quite some time. It looks like my node was routing payments between them whenever their channel was unbalanced, which happened quite often. I still keep those two channels open because I keep routing payments through them from/to other channels.

* If a channel starts to get used much, but only in one direction, rebalance it so it continues being used.

It might sound easy, but it's a huge pain in practice. I mostly use ZigZag.io and Boltz for non-dual-funded channels. When I was running LND node, I tried Loop but it failed 90% of the time. Since I have a couple of active channels, I might try circular rebalance soon. Also, it makes sense to wait a couple of days and see if the channel will balance itself.

[Quickseller]

Most of my fees are 1 or 2 sat. I have it set to minimum.

I have recently set my fees to 0/0 for some of my channels because they become totally unbalanced.

When your channel becomes unbalanced, you can reduce the fees on the unbalanced side so that you will receive transactions on the side that has a low balance.

You can also use dynamic pricing so that as one side gets low, the fee rate will change to encourage transactions to be routed to that side of the channel.

[Rath_]

You can also use dynamic pricing so that as one side gets low, the fee rate will change to encourage transactions to be routed to that side of the channel.

Sure, that's what I have been doing as well. Unfortunately, those channels remained silent as I periodically dropped the fees so I decided to make their use free of charge. One channel suddenly started routing transactions again in both directions even after I raised the fees. I will try rebalancing the other channel through a circular payment for a reasonable fee. It became unbalanced just a few hours after it was opened so I might earn back the cost of rebalancing and make some profit easily.

[Quickseller]

You can also use dynamic pricing so that as one side gets low, the fee rate will change to encourage transactions to be routed to that side of the channel.

Sure, that's what I have been doing as well. Unfortunately, those channels remained silent as I periodically dropped the fees so I decided to make their use free of charge. One channel suddenly started routing transactions again in both directions even after I raised the fees. I will try rebalancing the other channel through a circular payment for a reasonable fee. It became unbalanced just a few hours after it was opened so I might earn back the cost of rebalancing and make some profit easily.

Thats interesting. I wonder if some people are closing channels (or not closing channels) based on fees on various routes. If there is competition offering free routes, it might not be worth it to tie up bitcoin, but once a channel is closed for likely low future fees, they might not open a new channel due to the associated initial costs of doing so (tx fees).

It would be interesting to see how much someone might be able to earn running a LN node if they were trying to maximize their profit.

[n0nce]

~

Interesting, thanks for you insight! So it might actually be worth to connect 'already connected' peers after all, cool cool. More experimentation possibilities

It would be interesting to see how much someone might be able to earn running a LN node if they were trying to maximize their profit.

There are some writeups / articles online and also YouTube videos where people show their dashboard etc., but from what I've seen so far, to be profitable you need to put in a lot of manual work. We all know time is money & time is limited; so all that work for a few satoshi is usually not worth it. The 'maximum profits' I've seen so far, usually required a lot of BTC locked up + work under normal hourly wage.

[Quickseller]

I don’t see any reason why adjusting fees couldn’t be automated depending on the specific circumstances.

If a lot of BTC is required to earn from running a LN node, perhaps a reasonable interest amount should be charged against earnings to account for what could have been earned by lending your bitcoin to whatever reputable DeFi company.

[darkv0rt3x]

Hi there...

Here is my contribution. My noded URI is in my signature.
My node has 7 channels open, with a total of 4,500,222 sats of outbound capacity and 6,286,444 sats of inbound capacity. I'm running a bare metal Debian BullsEye, Bitcoin Core v0.21, C-Lightning v0.10.2 and RTL v0.10.1-beta. I have a few other non-related services running in this device (RockPro64, Samsung NVMe drive and 1TB external USB drive) but I don't think they are relevant for this matter.

My node is not very active as apparently I'm using some high fees that are probably putting me kind of isolated in the network. I'll lower the fees as soon as I can to see if there is some more traffic through it.
My node has been running for over a year but it only started forwarding payments successfully since January this year. Here is my data.


January, 2021


February, 2021


March, 2021


April, 2021


May, 2021


June, 2021


July, 2021


August, 2021


September, 2021


October, 2021


So, as we can see, I'm using high fees for sure. Not much activity these 2 last months and there's nothing in November so far!
I remember I used a plugin to auto-adjust fees and I made the mistake of making it auto-start whenever the node was restarted which made the plugin to adjust fees too often and make fees too low too.

[Rath_]

Thanks @darkv0rt3x for sharing your data. As I have already told you, I spent some time manually adjusting the fees for each channel every few days and it looks like I found the right values.

November looks really promising for my node. I am also about to open two new channels beside the one with n0nce.

[darkv0rt3x]

No problem @_Rath.

I still have tons of failed transactions with no apparent reason. I have 5 pages of failed transactions at November 9th. It's 48 failed attempts, of which only 6 were LOCAL_FAILED. All the others are simply FAILED.
I have enough liquidity in the outgoing channels of these attempts, so I can't understand why, even the LOCAL_FAILED ones, failed.



There is no outgoing channel for these LOCAL_FAILED. Have no idea why.

Then, the other attempts all failed further down in the route. What I question in my mind is if when a payment fails, for a given amount and route, the next attempts will try to use exactly the same route or will it try new routes? If it tries the same route, what could be the reasons for the failures? And if it tries a different route for each attempt, is it normal to fail like 10 or 12 times before it finally gives up on trying to route this payment?

[Rath_]

There is no outgoing channel for these LOCAL_FAILED. Have no idea why.

You need to use lightning-cli listforwards command to see the exact error. Here you can find a list of errors with explanations.

[...] is it normal to fail like 10 or 12 times before it finally gives up on trying to route this payment?

Maybe the payment was split into multiple chunks and some of them failed? It would make sense to route one part through you again if it worked in some previous attempt.

[darkv0rt3x]

There is no outgoing channel for these LOCAL_FAILED. Have no idea why.

You need to use lightning-cli listforwards command to see the exact error. Here you can find a list of errors with explanations.

[...] is it normal to fail like 10 or 12 times before it finally gives up on trying to route this payment?

Maybe the payment was split into multiple chunks and some of them failed? It would make sense to route one part through you again if it worked in some previous attempt.

Yes, I know about the listforwards command outputting the exact reasons, but not for all failed payments. I also know about the description of each error. However, at least me, cannot understand by only reading the descriptions in that link. So, it's almost useless to me to read them. :p

I'm not sure we can check if a payment was split or not. What it seems to me is that, at least for some payments, there are several attempts and as the values changes by small amounts, I assume it tried different routes, hence, different fees.

[TheJuice]

I've been meaning to do a larger write up of my experience once I hit 1M sats in fees.

I run the node Prince, https://1ml.com/node/02644f80b5d32ed9a9888690571159692a17d7ead7db2df5124a8e2a72a8447d30

I am currently ranked 8th on the BOS rankings (https://fulmo.org/bos-score.html) and 12th on lightning terminal rankings. I have 8-9 BTC on my node with about 110 channels (I have more incoming that outcoming liquidity, so that 8BTC is not my investment).

Top line numbers: Over the past 5 months have routed over 6700 payments worth over 1.7 billion sats earning me about 900k sats in routing fees; I've spent about 100k sats in on-chain fees and rebalancing fees. For those doing the math, that comes out to about 0.5% APR on my locked-up capital. Although my volume is steadily increasing and I may be able to get closer to 1% in the next year.

I have everything automated including fees and re-balancing.
-- I use lnd-charge for fees: https://github.com/accumulator/charge-lnd
-- I use c-otto's script for automatic re-balancing: https://c-otto.de/

Happy to answer more questions -- oh, and feel free to open a channel to me. The best way to get started is to open liquidity triangles with people (ie A->B; B->C; C->A). Happy to be a part of a couple here if there is interest. I do require a minimum of 5M sats for any incoming channels.

[ndalliard]

my node also runs for about 4 months and has 17 channels. it forwarded 616 payments which collected 978 sats. i've set the basefee to 1 sat and no percentage fee for all channels. i played around with the fees in the beginning (thats why i earned more sats than the number of forwards)

i only added the summary plugin to the node for now

some weeks ago i wanted to rebalance a channel with a trusted peer (all of the balance is on my side, pushing over 50% and the peer sending the amount onchain back to me), but the keysend to the lnd node didn't work. i've sent christian decker an email with the logs and he said he will look into it to debug it, but i never heard back... it still isn't working

apart from that i didn't have any bigger problems, but i am also not using my lightning node for a lot of my own economic activity at the moment (only some small donations)