Lightning Network Observer

[fillippone]

LN Finally crossed the  4,000 BTC threshold for the first time.



Lately, the growth has been slow, but steady.
People are starting to get the details of the implementation, rather than raw growth metrics.
Anecdotal evidence tells me channel size comes second to channels management (liquidity balances , routing fees etc).

[1714277112]

1714277112

[1714277112]

1714277112

[franky1]

LN Finally crossed the  4,000 BTC threshold for the first time.

for the first time.. (treat next sentence as humour)
so you are predicting it will DROP and it might hit 4000  a second time, seems your not confident in future growth

[fillippone]

LN Finally crossed the  4,000 BTC threshold for the first time.

for the first time.. (treat next sentence as humour)
so you are predicting it will DROP and it might hit 4000  a second time, seems your not confident in future growth

When I checked the news and opened 1ml.con myself, the capacity was actually below 4,000. It was at 3,999.53.
So yeah, I am not certain of future growth, extrapolating is going to zero, or it will cross 4,000 again (multiple times).
Treat the previous sentence as humor.

[cAPSLOCK]

Ugh... massive hardware failure.  Two nodes down.  Might need a little advice and moral support for what to do...

Going to try to recover the LND channels first.



LOL.

LND:
Seed words (verified them by checking the zpub with an azeed tool)
Static Channel Backups, including a recent one since the last change

I am assuming with this set of things I will see the blockchain download (gonna take a moment) and then all my channels will attempt to close, and hopefully funds start to make their way back on chain.

CL:
Have seed words, and hsm_secret (99% sure these are the right ones)
Have a very recent lightningd.sqlite3.backup file from right before the drive died.

Just hoping with the above nothing is incorrect, and I am not sure what recovery looks like on this node.

I will do the LND node first... and then the CL node.

If I have channel(s) with you, PLEASE do not force close them yet.  Some folks think if the channel DB is good, and the hsm_secret is right I might be able to bring the CL node back up.  I doubt this, but time will tell...

*sigh*

[n0nce]

~

Yuck! Good luck with the recoveries!
As for LND, I have no idea; for Core Lightning did you use its backup plugin? If not, I'd prefer to prevent running the risk of losing one (or more) channel's full balance and rather ask counterparties to force close them (or wait until that happens). If you are 100% certain that the backup doesn't differ from the latest channel state, you can restore it safely and prevent any losses. But honestly at current mempool sizes, force closing channels isn't too expensive and it's the safer route, in my opinion.

[cAPSLOCK]

~

Yuck! Good luck with the recoveries!
As for LND, I have no idea; for Core Lightning did you use its backup plugin? If not, I'd prefer to prevent running the risk of losing one (or more) channel's full balance and rather ask counterparties to force close them (or wait until that happens). If you are 100% certain that the backup doesn't differ from the latest channel state, you can restore it safely and prevent any losses. But honestly at current mempool sizes, force closing channels isn't too expensive and it's the safer route, in my opinion.

I am not 100% certain as to the integrity of my CL backups.  I have them.  I have the hsm_secret and a database backup.  But I will need to finish with the LND recovery first.  And that will require the node be brought back up.  Looks like that will be end of the week.



To be honest, I also trust my channel partners (a few of you are right here) and am entirely comfortable with them force closing the channels as they see fit.  I also realize I am vulnerable now, and those partners could decide to broadcast a channel state I may not be able to argue with.  I am hoping they will not.  But honestly my main hope is I can still produce the private keys to the addresses the close transactions will send my funds to.

We will see.

And I see the logic you are saying about the channels being force closed before I try my backup.  I really only have three risks right now.

1.  I have the wrong hsm_secret  (no idea)
2.  The backup script, which runs at shutdown, was not able to do so in a way the completed the database backup. (No way to tell if it is good until I get far enough that it is decrypted by the hsm_secret, I presume)
3.  My channel partners decide to take advantage of me.

[DooMAD]

LN Finally crossed the  4,000 BTC threshold for the first time.

for the first time.. (treat next sentence as humour)
so you are predicting it will DROP and it might hit 4000  a second time, seems your not confident in future growth

Total capacity is something that will fluctuate naturally.  But the overall trend is clearly upwards.

Perhaps you could do with a little less confidence and a little more modesty, given how completely and utterly wrong you are half the time you say anything.  You act like a know-it-all but you're mostly just an obnoxious windbag with a petty grudge.  Current capacity is currently 4012 BTC, but there's nothing written in stone which says it'll never drop below 4000 again. 

[cAPSLOCK]

One nice bit of info for our CL friends here... if you ever get in trouble this is what the respiblitz devs are suggesting.  I assume if you are using the backup plugin the sequence would be similar.

https://gist.github.com/openoms/3516cd8f393d69d52f858c3d47c9e469

[MargielaMadman]

Is there an IOS app that can use Lightening Network? I know of CashApp. Any others?

[n0nce]

To be honest, I also trust my channel partners (a few of you are right here) and am entirely comfortable with them force closing the channels as they see fit.  I also realize I am vulnerable now, and those partners could decide to broadcast a channel state I may not be able to argue with.  I am hoping they will not.

It's not even necessarily about their honesty; it's that you're at risk of accidentally cheating. And force closing may also happen without their notice since there are certain automatic timeouts, so don't be mad if someone happens to force close your channel. They probably didn't do it manually / on purpose. And as I pointed out, force closes will be better / safer for you (at the cost of some mining fees) since you won't be able to 'accidentally cheat' by publishing old channel states when attempting recovery.

But honestly my main hope is I can still produce the private keys to the addresses the close transactions will send my funds to.

Don't you have the hsm_secret? If so, your on-chain addresses are safe, no worries!

One nice bit of info for our CL friends here... if you ever get in trouble this is what the respiblitz devs are suggesting.  I assume if you are using the backup plugin the sequence would be similar.

https://gist.github.com/openoms/3516cd8f393d69d52f858c3d47c9e469

Yeah it looks like this caters exactly to users of Core Lightning's backup plugin!

Is there an IOS app that can use Lightening Network? I know of CashApp. Any others?

I'd recommend actual non-custodial node applications.
[1] https://breez.technology/
[2] https://phoenix.acinq.co/
[3] https://muun.com/

[cAPSLOCK]

Update to my drama:

I was able to restore the database so I must have had the right hsm_secret ?  I guess I have to wait now for the node to finish syncing to see if we have the keys to anything...  Or maybe the node is just working?  That would be odd.

My understanding is the sqlite database backup would not restore with the wrong hsm_secret.

Anyway... the CL node might actually SURVIVE this torture.  Not 100% sure about the LND node.  At the same time I still feel like I have a long way to go.  All the TOR addresses look different now, and I have no idea what the node ID(s) is.  As long as I have the keys to force closed channels...

The HARDEST POT to watch boil:

[cAPSLOCK]

I want to add this.

I have only run LND and CL nodes.

I put a LOT more work, time and value into LND.  Through the last 7 months I have run the CL node, I am feeling a LITTLE like it is a more well rounded smartly designed product.

The plugin architecture is a major strength.

I put WAY MORE work into securing backups for the LND node... but when the rubber hit the road?  It is shaping up to look like the CL node will fair better.  And that is with truly reckless behaviour on my part.  In retrospect I was not careful at ALL about how I performed the setup and maintenance.  I think the reason for this was I intended for the thing to be an experiment with a few 100k sats on it just to say I was running it.  And I ended up putting nearly the same amount on ~13 channels with it as >40 on the LND node.

I am no genius when it comes to this stuff... as can be seen by my most recent issues... but perhaps hast gives me the credit to say CL is the more idiot proof as well...  In fact, if I keep running a node after this ordeal I might just stick with the Blockstream build...

All that said, I have recovered nothing yet. LOL.  I reserve the right to take it all back.  I might think just the opposite in the end.

[n0nce]

I was able to restore the database so I must have had the right hsm_secret ?  I guess I have to wait now for the node to finish syncing to see if we have the keys to anything...  Or maybe the node is just working?  That would be odd.

My understanding is the sqlite database backup would not restore with the wrong hsm_secret.

I'm not 100% sure as I haven't yet had to restore a node myself, so I can't comment on that for sure but I'd tend to agree with your assessment.

The HARDEST POT to watch boil:

What is this UI, by the way? I've never seen it myself so far.

I put a LOT more work, time and value into LND.  Through the last 7 months I have run the CL node, I am feeling a LITTLE like it is a more well rounded smartly designed product.

Interesting! I have very limited experience with LND, but the theory of using the backup plugin and restoring a backup with is indeed pretty straight forward. I'm looking forward to seeing how it turns out in practice!
Did you set it up to save the backups on an external drive?

Like shown here?

[5] Initial channel state backup

Code:

cd ~/plugins
backup/backup-cli init --lightning-dir /home/bitcoin/.lightning/bitcoin/ file:///mnt/share/lnbackup/backup.bkp

[cAPSLOCK]

I was able to restore the database so I must have had the right hsm_secret ?  I guess I have to wait now for the node to finish syncing to see if we have the keys to anything...  Or maybe the node is just working?  That would be odd.

My understanding is the sqlite database backup would not restore with the wrong hsm_secret.

I'm not 100% sure as I haven't yet had to restore a node myself, so I can't comment on that for sure but I'd tend to agree with your assessment.

The HARDEST POT to watch boil:

What is this UI, by the way? I've never seen it myself so far.

I put a LOT more work, time and value into LND.  Through the last 7 months I have run the CL node, I am feeling a LITTLE like it is a more well rounded smartly designed product.

Interesting! I have very limited experience with LND, but the theory of using the backup plugin and restoring a backup with is indeed pretty straight forward. I'm looking forward to seeing how it turns out in practice!
Did you set it up to save the backups on an external drive?

Like shown here?

[5] Initial channel state backup

Code:

cd ~/plugins
backup/backup-cli init --lightning-dir /home/bitcoin/.lightning/bitcoin/ file:///mnt/share/lnbackup/backup.bkp

That UI is the Raspiblitz UI.  Well part of it.  It is just monitoring the amount the chain-sync is behind.  To be honest I have no idea what the "lightning progress" is though.  It actually sort of worries me... 68% of what? lol.

As to the backup, yes I wrote a script that put them on an external usb drive (flash drive), but I made a mistake in my calculations about space and rotation, and the thing had not been saving backups for a while.  As well as the fact that I was not really doing them frequently enough. I set it up on a cronjob... My error handling was also kinda ziltch.  But again, thanks to Raspiblitz the node seems to (fairly frequently?) make backups regularly to the SD card as well. And the one on the SD card was within minutes of when it was shut down.  So I am hopeful I will not see many justice transactions  I have not yet decided whether I just want to ensure I can get to the addresses and let the nodes close over time... or risk it to just turn the thing back on.  I almost wish I could set some switch that would query all channel partners for a current balance, and just accept it if I am within X% or Xsats.

I have become fond of Raspiblitz and Rootzroll and Openoms.  Using it reminds me of starting to use Slackware linux after having built a disaster of a machine doing LFS.  There is still plenty of room to noodle and configure, but LOTS of the tricky bits, and helpful things are built into the system... interesting project.  I have decided I will donate a percentage of everything I recover at the end of this madness...  If I get it all, that donation might actually sting a little haha.

[n0nce]

That UI is the Raspiblitz UI.  Well part of it.  It is just monitoring the amount the chain-sync is behind.  To be honest I have no idea what the "lightning progress" is though.  It actually sort of worries me... 68% of what? lol.

I see! Yeah; that's odd. I wouldn't know what it tries to sync; Lightning startup from 0 is usually pretty fast anyway. Maybe it has to do with how many of your channel funding transactions are synced into the blockchain or something; but since you're at only around 50% of the blockchain download, it doesn't seem viable. I'm sure Raspiblitz have good docs about this.

As to the backup, yes I wrote a script that put them on an external usb drive (flash drive), but I made a mistake in my calculations about space and rotation, and the thing had not been saving backups for a while.  As well as the fact that I was not really doing them frequently enough. My error handling was also kinda ziltch.  But again, thanks to Raspiblitz the node seems to (fairly frequently?) make backups regularly to the SD card as well. And the one on the SD card was within minutes of when it was shut down.  So I am hopeful I will not see many justice transactions

Oooh I see, custom stuff. Alright. I'd always test such scripts before 'deployment', especially when it comes to something financially important such as Lightning backups. Or stick to existing tools.
But Raspiblitz backup sounds good; if it's within minutes of shutting down and you don't have a ton of activity in your channels, you could go for gold and restore those backups.
Does Raspiblitz backup everything by default? Or is it a plugin as well?

By the way; regarding channel activity, here a few impressions of fellow node operators from the forum. We have lots of days without any activity at all. So if you're anything like us, there's not going to be much to worry about.

Here are the stats for January. One of my largest channels, which I opened with some other bitcointalk user, was offline for at least half a month. Some of the routed payments were free of charge as I wanted to rebalance two of my channels this way.

I should say I haven't been very active changing fees to meet the new balances after a couple of forwardings. Maybe that is also holding my node a bit back. My node is not very big either. I think only around 0.06BTC of incoming liquidity, so I think it's kinda reasonable that my routing is not that good!

New record! 647.73 satoshi in fees collected from 60 transactions in February. A few days ago, I opened a dual-funded channel with @n0nce and it turned out to be the right call. We routed 41 transactions in a single day!

March doesn't look good in comparison to other months. My Internet connection was unstable for over half a month and my node was down for a couple of days. I managed to fix it around the 26th and it took about 3 days before I started seeing routing attempts again.



On a positive note, my node routed 6 transactions today and I earned 351.58 satoshi. April looks promising!

Hi. Once more I leave my results. Not much activity here either. Last couple of weeks I saw 2 channels closed. One from a Ring of Fire (without any apparent reason nor warning, ) and another from a damaged channel database from LightningWatch node.

And from what I can see, I might be using very low fees in my outgoing channels. I might have to go for another run changing some fees.

Here are my March stats..
I need to rebalance often, so I'll have a look into adding another channel soon that might help 'route the other way round' more & keep channels more balanced.



@darkv0rt3x you seem to collect a lot of fees! What are your fee rates?

I have not yet decided whether I just want to ensure I can get to the addresses and let the nodes close over time... or risk it to just turn the thing back on.  I almost wish I could set some switch that would query all channel partners for a current balance, and just accept it if I am within X% or Xsats.

This sounds like a very useful feature! For instance, letting the node start up (without broadcasting channel states or other functionalities); just for getting what the peer believes to be the latest channel state; then checking the backup and if they match, really start up (exchanging channel states and generally running the whole protocol again).

I have become fond of Raspiblitz and Rootzroll and Openoms.  Using it reminds me of starting to use Slackware linux after having built a disaster of a machine doing LFS.  There is still plenty of room to noodle and configure, but LOTS of the tricky bits, and helpful things are built into the system... interesting project.  I have decided I will donate a percentage of everything I recover at the end of this madness...  If I get it all, that donation might actually sting a little haha.

I'm not a big fan of SBC nodes due to their weak CPUs and my impatience when compiling software; but if I were to spin up another node on a SBC, I'd look into Raspiblitz with Core Lightning!

[cAPSLOCK]

Oooh I see, custom stuff. Alright. I'd always test such scripts before 'deployment', especially when it comes to something financially important such as Lightning backups. Or stick to existing tools.
But Raspiblitz backup sounds good; if it's within minutes of shutting down and you don't have a ton of activity in your channels, you could go for gold and restore those backups.
Does Raspiblitz backup everything by default? Or is it a plugin as well?

It uses the plugin framework and the backup plugin, but it is active by default.  And looking at the logs I would think it backs up frequently

By the way; regarding channel activity, here a few impressions of fellow node operators from the forum. We have lots of days without any activity at all. So if you're anything like us, there's not going to be much to worry about.

My LND node was doing hundreds of routes a day a couple months back...  It was quite active.  But in recent times it has been much more quiet, and some days just a few transactions...  Lucky in retrospect.  The CL node never got anywhere near as active as the LND node.  Another reason I am going to use the static channel backups method on it.  Not that I had a choice... the rescue files I had were woefully outdated.  It's sad as the node has been up since 2019, and is a BOS ranked node... or WAS.  Now it will just be, I hope a nice pile of UTXOs I have the keys for.  

I'm not a big fan of SBC nodes due to their weak CPUs and my impatience when compiling software; but if I were to spin up another node on a SBC, I'd look into Raspiblitz with Core Lightning!

Understand.  I am split on them.  I like the idea that the node uses "night light" level electricity.  But it is still a bit Janky IMHO with an external drive hanging off the end of it...  I would prefer one of the stouter SBCs with a teeny custom case.  Or upgrade to some of the little thin-client+ type teeny form factor Dell machines.  Could also do an old laptop... but that feels even jankier than the RPi.  In a way.  Probably the best bang for the buck would be to grab a corporate cast off machine off ebay for <100$.  Some fairly nice equipment can be had for low figures...

I appreciate the good conversation, n0nce, whilst I work to see how much money I have set fire to in my foolishness.. heh.  I guess the sagging Bitcoin price has the dubious good effect of making that POTENTIAL value smaller each day!

[franky1]

New record! 647.73 satoshi in fees collected from 60 transactions in February. A few days ago, I opened a dual-funded channel with @n0nce and it turned out to be the right call. We routed 41 transactions in a single day!

gotta love it.. the 'promise' of MICRO payments(microfees) under 100,000th of normal unit
becomes millipayments/milli fee's, under 1,000th of normal unit
and then centi-payments and centi fee's, under 100th of normal unit

seems you LN fangirls wanted to promise fee's of under 1 sat(complete route, meaning on average less than 0.05sat(to allow for a 20 hop route to still be under 1 sat)
where you have all been describing the mill and micro buzzwords ALOT suggesting 1000ths and 100th thousands of the times smaller. and all the promises of allow 'millions of payments.

check it out. do the math, run some scenario's
raths quote feb6th 148sat /25 events = ~6sat for his hop in a route
darkvortex 207sat /7 events = ~30sat for his hop in a route
raths quote feb 28th 647sat /60 events = ~11sat for his hop in a route
raths quote april 66sat /14 events = ~5sat for his hop in a route
darkvortex april 271sat /13 events = ~21sat for his hop in a route
n0nce april 44/21 events = ~2sat for his hop..

now lets take the average
rath averages 30sat/hop
darkvortex averages 26sat/hop
n0nce is 2sat

and lets show a possible route
user->rath->darkvortex->n0nce->destination
          30sat   26sat          2sat

so if a user wanted to pay destination using the average fee of known hops on just a THREE HOP payment..
that user is having to pay 58sat

seems their promise of millisat payments and micropayments has gone and now its into centi-payments. where fee's are above 10% of a a regular bitcoin transaction

imagine if a user had to pay someone not 3 hops away. but 15 hops away..  yep suddenly its not so cheap as promised and just a LN is half the fee of bitcoin
especially if n0nce starts doing the same average fees as rath/darkvortex that amount will go up more if the new averages are more than a few sats each hop

yet simple maths of simple examples today suggest your faked promises were just that and you all already have switched from bait to trap

still laugh when i see more and more LN promises fail..

oh its been months now and although people show stats of their successes. they are too AFRAID to show the payment fails % at differing amounts..
shame on you for your utopian fantasies. but hey. its your network and if your the only ones using it, it can only bite yourselves in the ass in the end. enjoy that experience
....
p.s i have been a LN observer, so very much on topic. i know your desperately want to get me banned or posts deleted to hide the issues. but hey a observer and a moral user actually wants to know the bad aswell as the good, to get a risk assessment observation, they dont want to observe the dream hyped up utopian promise trying to sell them riches of profiting from fee's. they want to observe the reality that if they themselves wanted to spend their own money. how much it actually would cost on average..

[n0nce]

It uses the plugin framework and the backup plugin, but it is active by default.  And looking at the logs I would think it backs up frequently

That sounds like a job well done by the Raspiblitz guys!

I'm not a big fan of SBC nodes due to their weak CPUs and my impatience when compiling software; but if I were to spin up another node on a SBC, I'd look into Raspiblitz with Core Lightning!

Understand.  I am split on them.  I like the idea that the node uses "night light" level electricity.  But it is still a bit Janky IMHO with an external drive hanging off the end of it...  I would prefer one of the stouter SBCs with a teeny custom case.  Or upgrade to some of the little thin-client+ type teeny form factor Dell machines.  Could also do an old laptop... but that feels even jankier than the RPi.  In a way.  Probably the best bang for the buck would be to grab a corporate cast off machine off ebay for <100$.  Some fairly nice equipment can be had for low figures...

I like that idea, too, but I don't know how many years of uptime will be required to break even with a cheap, used laptop. As you said, under $100 is possible; see [Guide] How to run a Bitcoin Core full node for under 50 bucks!. It's not beautiful, compact, or power-saving though.
If I were to build another SBC node, I'd probably go for the latest-gen Raspberry Pi (better hardware and software support than Orange Pi..) and use the Argon One m.2 case with an m.2 SSD. They aren't that expensive anymore these days and the device will actually boot off the SSD. I hate to see a machine that is supposed to be reliable with a high uptime rely on a microSD card as a boot drive (Orange Pi can't boot off external drives - I'll elaborate in my Apollo BTC review - sorry for off-topic).

I appreciate the good conversation, n0nce, whilst I work to see how much money I have set fire to in my foolishness.. heh.  I guess the sagging Bitcoin price has the dubious good effect of making that POTENTIAL value smaller each day!

Likewise! Fingers remain crossed!

[cAPSLOCK]

-snip whining-

Oh dear... who peed in your cornflakes?

I see lots of things wrong with your statements, and am not going to take the time to explain much of it as I doubt you would really be interested. 

As to failures routing payments...  I have not personally seen tons of that.  I do see them.  And more with personal nodes, or neutrino implementations.  Custodial lightning works basically perfectly.  And before you say "But that's not the right kind", I know... but what it does do is prove a well run node can not only manage payments well for a user, but for a whole country.  Therefore it is possible that node software, and light noncustodial wallets are going to improve over time, and be even more reliable.

As to fees... There is no real fee market yet.  People charge all different kinds of fees.  I will say that the VAST MAJORITY of my payments cost between 0-30 sats.  Only a handful ever go anywhere near 100, and only in special circumstances do I see them higher than that.  Particularly when opening a "turbo channel" or the like.

Anyway your point that lightning is far from perfect is true.  But I certainly do not see it as the failure you seem to... and I only see it getting better.

[cAPSLOCK]

Thanks for the help watching my pot boil... looks like by lunch tomorrow I should be looking at a full blockchain.




I am tempted to make sure lnd and lightningd are NOT running before I hit the sack tonight.  I think I would rather turn those on then just see them pop on randomly and trigger a cascade of justice transactions.  I know the result could be the same... but I want to :

1.  Perform whatever on-chain analysis I can first.
2.  Make a solid eyes-open decision.
3.  Be the master of my fate... if I am gonna lose value I want to have done it by my own act of will!

At this point I still do not know if either node will come back.  I have noodled around with the hsm_secret for ages looking at addresses derived from it, and they all come up blank.  I have even used the Xprivs I get with hsm_tool etc.  I just hope that CL uses some wacky derivation or salted something-or-another and I am just looking at the wrong addresses.  I hope what I believe is right.  That the database backup was encrypted with hsm_secret, and since it was able to be restored then I have the right one.

Same thing with the LND node.  I have a wallet in electrum based on a AZEED based seed that shows the on chain balance at the time the node crashed (by the way LND does seed words and derivation TOTALLY NOT the same way as Bip39 so if you need to know about it let me know... took me a while to figure it out) .  And since then SEVERAL channels have been closed unilaterally by my partners, yet no value has ended up in that wallet.  I am hoping that the node has to do something to complete those multisig contracts?  Or the funds are still not unlocked?  The LND node, sadly has no chance of being restored... I just want to see all the funds go back.

God help me.  (Seriously God... I owe ya already, but I am asking again)