Important Lighting Network reading- for everyone!

[pebwindkraft]

...
Network routing is one of those arcane specialties filled with scary-smart people who know all the minutiae of complex systems which most developers are barely aware exist.  (How does the Internet really work?  How do all those little packets know where to go?  Magic!)  I am not in that specialty, and I’m not familiar with its research literature

I'm also not a specialist, but understand enough on the basics. So no, not magic, and I bet you know as well :-)
IP protocol is well defined, with routing information (or destination) in the header of each packet. In the very early days a router would ask his neighbor, do you have a route for network x? (neighbor discovery protocol). Someone to whom I was connected would answer accordingly (or not, then "PATH NOT FOUND"). And the question of routing optimization started quite early. There was this ATM network model at the same time (goal to unify networks with telephony and IP in 53 byte packets!)... Remember good old CISCO 27xx or 29xx models with configs for RIPv1 and v2, and then open shortest path first (OSPF)? These were the internal routing protocols. Then there was the bigger networks in the late 80s and they needed a professional base layer (border gateway protocol, BGP and IGRP/EGRP, or EGP). And of course a DNS. btw: why is DNSsec not used today?

... what first occured to me was that Spanning Tree Protocol might somehow be applicable. Of course, that’s not an Internet routing protocol; but it is the standard staple for organizing the network topology on LANs.

Spanning tree was the protection layer in bridges, to prevent loops in larger networks, with the "routing" of MAC addresses (bridge = layer 2 device). Yup, no routing at layer 4 here ...

The general question is:  Given a global set of nodes which form and remove links between each other unpredictably, how does each node organize its own view of potential routes and choose optimal paths?

Lookup tables?
I took a look into the routing in Lightning - you come to deal with SPINX, HMACs, and of course onion routings. I think the difficulty layer of lightning routing stems from the fact, that the node only knows the predecessor and the successor of a route. Nothing else. Not the origin, not the final destination, not the amount (but then, how to know, that the channel supports the requested value transfer?), and for sure in a way bullet proof, that the node cannot benefit from forwarding the package to a different target. There I find this table in the data structure of 20 entries @65 bytes (hops_data), which makes me think on what it is used for? Is there a max of 20 hops?
Also the flare white paper (http://bitfury.com/content/5-white-papers-research/whitepaper_flare_an_approach_to_routing_in_lightning_network_7_7_2016.pdf) says:

Hence, it is in the interest of the sender to optimize fees and make the final decision on which route to choose to the recipient (otherwise the sender gets potentially unpredictable expenses, as other nodes are not incentivized to optimize for the cheapest path).

and

Source routing leads to the requirement that the sender node should be able to collect information on fees and available channel capacity to pick the best route (as well as knowing which nodes are currently online). Thus, an overlay mechanism should exist to enable requesting information about a channel from any of its owners.

Interesting, that would mean no dynamic routing?

I don't understand (yet) how routing achieves a dynamic management, when it is pre-defined from the beginning. And this would mean, it does not require IP routing specialists? There's work to be done!

[TheQuin]

Remember good old CISCO 27xx or 29xx models with configs for RIPv1 and v2, and then open shortest path first (OSPF)? These were the internal routing protocols. Then there was the bigger networks in the late 80s and they needed a professional base layer (border gateway protocol, BGP and IGRP/EGRP, or EGP). And of course a DNS. btw: why is DNSsec not used today?

Surely you mean the 2500 Series? (Ooops showing my age again.) This is a good demonstration of how easy it is solve scaling issues at layer 3. When the limitations of RIP became apparent it was easy to simply add additional routing protocols to solve the problems. For example, EIGRP is a Cisco proprietary protocol that can be used in areas of the network using exclusively Cisco equipment and then BGP handles connecting to other areas that may not be.

I see so many debates about the possible limitations of Lightning Network that miss this. It's not an end product set in stone, it will constantly evolve to address any issues that arise. That's not so easy to do in blockchain where much was set in stone in the genesis block.

Spanning tree was the protection layer in bridges, to prevent loops in larger networks, with the "routing" of MAC addresses (bridge = layer 2 device). Yup, no routing at layer 4 here ...

Pedantic Note: STP allowed layer 2 topologies to be designed with redundancy built in by temporarily blocking loops and then opening them when a failure elsewhere eliminated the loop.

[Kakmakr]

While you guys dissect the technical aspect of this document, I would just like to thank the author for a very enlightened piece of art. The main point of this article in my humble opinion, was to highlight the fact that we are still in the earliest phase of this experiment. We are building the "Internet of Money", piece by piece and if second layer applications are necessary to do that, then so be it.

The point is, Block size increases can only take us to a specific level, before it becomes destructive to the network. We have to rely on second layer applications to take us to the next level.

[nullius]

Thanks for the networking discussion, folks.  I want to underscore this twice:

I see so many debates about the possible limitations of Lightning Network that miss this. It's not an end product set in stone, it will constantly evolve to address any issues that arise. That's not so easy to do in blockchain where much was set in stone in the genesis block.

So—how would you take all this networking knowledge, and apply it to routing and network topology in Lightning?  More to the point (and my original question), are the Lightning devs doing so—and if so, how?

(Aside, or perhaps not:  Shifting analogies around and down to the link layer, per OP’s article, we no longer have only a broadcast network as with old Ethernet hubs.  There is a reason I thought of Spanning Tree first.)

I see plenty of speculation about what LN will look like, topology-wise.  Yet much of that depends not only on what potential links are available, but also on how nodes use them.  I don’t see how a simple look-up table would suffice.  If you are connected to A, B, and C, and you want to reach Z, that’s not an easy problem.  It’s not easy in the first instance; and however it’s answered now, I expect that could be fertile ground for optimization in the future.

[TheQuin]

More to the point (and my original question), are the Lightning devs doing so—and if so, how?

I'm afraid I can't answer this bit as I don't know.

So—how would you take all this networking knowledge, and apply it to routing and network topology in Lightning?  
......

I see plenty of speculation about what LN will look like, topology-wise.  Yet much of that depends not only on what potential links are available, but also on how nodes use them.  I don’t see how a simple look-up table would suffice.  If you are connected to A, B, and C, and you want to reach Z, that’s not an easy problem.  It’s not easy in the first instance; and however it’s answered now, I expect that could be fertile ground for optimization in the future.

The lesson I would take from IP routing protocols would the introduction of 'areas' that allowed changes that occur in certain places need only be propagated within that area and have no effect on the network topology as a whole. The IP routing table in Internet routers is a simple lookup table, but these protocols keep the updating of that table to a minimum.

[pebwindkraft]

If I understood the docs correctly, the big difference between routing in IP networks and in Lightning is the used model: in Lightning I see source routing used, so the sender first creates the route, packs it in layers (hence onion), and then sends the package. In IP networks there is ROUTERS all over the network, that do this job. So in the IP world you can send a package with a destination address (encapsulated in the header field) to the next router, and he has these lookup tables, and forwards accordingly.
In a lightning node would have to ask "the network" first, then create the route (based on fees), and then send the package over this pre-defined route. So the node has it's own "lookup table". It might be called differently though. I still have an issue to understand the "ask the network", so leave it open for discussion.

There was some text from the devs on the mailing list:
https://lists.linuxfoundation.org/pipermail/lightning-dev/2015-December/000384.html

Surely you mean the 2500 Series? (Ooops showing my age again.)

yes, and yes  , me too!

[Angel35]

Man, I want to say thanks for such info, because I couldn't find something usefull about that. It's really interesting and I think it has the future.

[TrumpD]

Very good read, and the analogies helped, as most of us aren't deep inclined. Maybe if a short comic-like/graphic video was created, we can get a larger pool of people to understand this upcoming feature of bitcoin, and re-affirm their belief that the future is good for bitcoin. I'm really looking forward to the Launch of Bitcoin's Lightening Network, and the Raiden Network of Ethereum.

[ChiBitCTy]

NP guys!  But the smart mo fo who typed this bad boy up deserves all credit. I've been looking for articles like this that teach the non-coder/programmer (as badly as I'd die to be) and they aren't easy to find.

[bitmover]

I think it’s important that everyone who supports bitcoin be on the same page about how the Lightning Network works and why bitcoin’s future is still as bright as ever. This short read was written up by a buddy in a slack channel and I thought it was important to pass along. It’s easy reading for anyone! 10 minutes of your time and if you’re not fully educated on the Lighting Network basics..you will be!

https://medium.com/@melik_87377/lightning-network-enables-unicast-transactions-in-bitcoin-lightning-is-bitcoins-tcp-ip-stack-8ec1d42c14f5

I was researching about lightning network and found this topic. Very nice article.

I made a summary and translated it, and posted in my local board. Thanks for sharing.

https://bitcointalk.org/index.php?topic=3179641.msg32927586

[BEX]

The only problem lies in the fact that the channels are only defined between two persons. In order to reach a person with whom you do not share a channel, you will need to use intermediate nodes or “hubs” (Bob in our example). However, these nodes could be considered by the regulatory agencies as “money transmitters” and as such be subject to requirements such as the minimum capitalization, KYC checks (…) which cannot be filled by anyone.
More here...https://www.blockchains-expert.com/en/lightning-network/