Re: How (and why) to use the Relay Network

[o_solo_miner]

  looks like that the public.eu.relay.mattcorallo.com Node is down, as I got this message:

[2016-03-10 12:39:25.858+00] 00000000000000000149a9d6ccc79d07f460bbe888bc220e9ed643b3ba0226ec sent, size 998129 with 882427 bytes on the wire
[2016-03-10 12:39:25.868+00] public.02.relay.mattcorallo.com Disconnect: failed to read message header (Broken pipe)


I manualy changed to US-WEST and that works.
Strange, as it went down after the blockfound message.

Anybody else with problems on eu?

[Matt Corallo]

Yea, eu went down for a bit...should be good now.

[o_solo_miner]

Thank you matt
now it works ok.

I was just suprised, never got an error before.

[-ck]

Hey Matt. Any update on what the plans are for your invaluable network? Thanks.

[Matt Corallo]

I was just suprised, never got an error before.

Servers go down sometimes .

Hey Matt. Any update on what the plans are for your invaluable network? Thanks.

Until other things that are much higher priority (ie the current drama) is dealt with, I will keep restarting servers when they go down but no other changes are gonna happen...I'll revisit things in a month or three.

[MeatPopsicle]

If worst comes to worst could you throw together a short docu on how the server components fit together? Trying to piece together how and with what each of them talk to by digging through the C code is less than optimal.

[mmeijeri]

I didn't want to wade through 19 pages of text and I know this has been answered before, but why would it be a bad idea to integrate this into the P2P network on a bilateral basis? Wouldn't it still help a bit?

[kano]

I didn't want to wade through 19 pages of text and I know this has been answered before, but why would it be a bad idea to integrate this into the P2P network on a bilateral basis? Wouldn't it still help a bit?

Yeah I asked that already ... the problem is that at least some of the bitcoin devs are morons.
https://github.com/bitcoin/bitcoin/issues/7049

... especially when you get the part where gmaxwell considers exceptionally useful code black magic

[Matt Corallo]

I didn't want to wade through 19 pages of text and I know this has been answered before, but why would it be a bad idea to integrate this into the P2P network on a bilateral basis? Wouldn't it still help a bit?

See https://github.com/TheBlueMatt/bips/blob/master/bip-TODO.mediawiki for something similar which is more targeted at the P2P nature of Bitcoin. See-also https://github.com/TheBlueMatt/bitcoin/tree/udp-wip for something which is intended to eventually replace the Relay Network - a whitelisted-peers-only secondary relay channel that is easy for miners to use between their own nodes (ie its just an additional flag) and will also be much faster than the relay network today in its worst-case (and probably average) relay times.

[Matt Corallo]

For those paying attention...The relay network donation address was recently nearly entirely depleted to pay for a new set of servers...The network is going to switch entirely to a new infrastructure based on bitcoind with UDP-FEC (ie no more delays from TCP lag) which allows for cut-through routing by rumoring UDP-FEC packets around the entire network before receiving the entire block. You can find the source for this patch at https://github.com/TheBlueMatt/bitcoin/tree/udp-wip...The API to it is being refactored to enable the cut-through routing optimizations, and note that it includes some LGPL components in the binary, so its probably all secretly under LGPL, but IANAL, please dont sue me.

[Matt Corallo]

So I'm sure everyone has seen, by now, my blog post on the future of the relay network. I do have a test-network based on FIBRE which has shown incredibly good performance, but I'm not quite sure if I will allow public connections to it (it currently only connects to random nodes on the public P2P network and to the original relay network) or allow others to use FIBRE to take up the mantle of running high-speed relay networks. For now, the original relay network continues to run and the FIBRE network running side-by-side is only making it faster!
If you want any help running your own FIBRE-based network, I'm more than happy to help out!

[kano]

I block all UDP and run no UDP services coz it's been the majority cause of DDoS that occurred since I started the pool ...
... since there is no 'network security' applied to UDP on the internet.

[Matt Corallo]

I block all UDP and run no UDP services coz it's been the majority cause of DDoS that occurred since I started the pool ...
... since there is no 'network security' applied to UDP on the internet.

If you're talking about running your own FIBRE-based network: No need to enable incoming UDP on pool servers to run FIBRE. TCP/Compact Block relay to servers only a few ms away is plenty fast, so you could just as easily put a relay network server on a separate server (and probably should, given that its still beta and based on Bitcoin Core master just before segwit was merged).

If you're talking about connecting to someone else's: I never really envisioned it as running over UDP between miners and the first-hop server. While this is definitely an option (and FIBRE has an "untrusted" mode for this), there is little advantage to spending 10ms calculating FEC data over just eating the 10ms RTT for a dropped packet or two if the servers are nearby. FIBRE is great for long-haul links, but Compact Blocks work just as well if you're talking about short-hops.

General note: FIBRE's UDP does have a shared-key HMAC in each packet, so that packets are easily and trivially authenticated.

[kano]

I block all UDP and run no UDP services coz it's been the majority cause of DDoS that occurred since I started the pool ...
... since there is no 'network security' applied to UDP on the internet.

If you're talking about running your own FIBRE-based network: No need to enable incoming UDP on pool servers to run FIBRE. TCP/Compact Block relay to servers only a few ms away is plenty fast, so you could just as easily put a relay network server on a separate server ...

Yep good solution.
One collection of 'satellites' I have are sub 1ms (around 1/3ms)

(and probably should, given that its still beta and based on Bitcoin Core master just before segwit was merged).

Well looks like I'll be behind the times for a while if it's been merged, coz yeah not really interested in adding code to centralise bitcoin and give away mining fees to those centralising it so they can take those fees.

[Matt Corallo]

If anyone is looking to set up a FIBRE-based network, I'm happy to share the slow-as-fuck bash script which generates the slow-to-render pretty graphs on test network stats page.

[-ck]

This has been misbehaving for the last week or so. Any chance it could get a kick?

[Matt Corallo]

Hi folks,

I'd like to turn off the relay network on December 1st. For that to happen, I'd like to first have a public relay network of nodes based on FIBRE available ASAP.

In order to make the network much more reliable/lower-latency than the original relay network, I'd like to use real servers with good,
dedicated, bandwidth. I speced out a network which would cost about $1550/month and would have public peering nodes in Hong Kong, Tokyo, Seattle, Washington DC, Beijing, and one of London/Amsterdam/Frankfurt. Additionally, there would be (at least) one node in Siberia to utilize the TEA Cable route.

A few folks have already agreed to chip in to make this a reality, but I'd love to see a few more so that we can make it incredibly cheap for
everyone involved. If you'd like to help, I'm looking for people willing to make commitments to cover a fair share of the above costs, and would list the sponsors on bitcoinrelaynetwork.org (which would be updated with new info).

[kano]

Well ... a few things.

I take it that the current relay network has been crappy for a while coz you decided you weren't getting enough sponsorship?
i.e. when the sponsorship isn't enough in your (unstated) opinion you just let it go to crap?
(No I've never sponsored it - but I do connect to every relay ... even the long dead hk one - our ckpool block relay is faster than using the relay to distribute to other relay nodes so I connect to all nodes to get our blocks out faster)

Unless you have the top pools connected (i.e. every pool over a few % of the network) it's value is marginal so I'd think you should actually list the pools that are sponsoring and are connected - and update it when that changes - since that's not a large list.
I'd also suggest you whitelist access to most of it. That way you know who is connected and know it is still useful.
e.g. f2pool wasn't on the relay until long after the relay existed, and I've no idea if all of the 3 bitmain pools use it. Bitfury's high orphan rate at times makes me wonder if they have always used it.

Well I'm certainly no where near 7% of the network but I'll put forward ~$100 in BTC a month towards it (as long as the biggest pools are on it)
N.B. none of my servers but the single main back end cost me even half of that So I guess this network is either using an expensive provider or high end servers on every node

[Matt Corallo]

I take it that the current relay network has been crappy for a while coz you decided you weren't getting enough sponsorship?
i.e. when the sponsorship isn't enough in your (unstated) opinion you just let it go to crap?
(No I've never sponsored it - but I do connect to every relay ... even the long dead hk one - our ckpool block relay is faster than using the relay to distribute to other relay nodes so I connect to all nodes to get our blocks out faster)

Yea, the HK one died after the host decided to wipe the drive on the server when the NIC failed....yes, you read that right. I didn't bother to re-install since a) I dont actually remember how to reinstall that stuff, and havent had time to look it up (FIBRE is so much easier...) and b) I want to turn the thing off for SegWit (though getting a public FIBRE network up has taken longer than expected :/.

Unless you have the top pools connected (i.e. every pool over a few % of the network) it's value is marginal so I'd think you should actually list the pools that are sponsoring and are connected - and update it when that changes - since that's not a large list.
I'd also suggest you whitelist access to most of it. That way you know who is connected and know it is still useful.
e.g. f2pool wasn't on the relay until long after the relay existed, and I've no idea if all of the 3 bitmain pools use it. Bitfury's high orphan rate at times makes me wonder if they have always used it.

Indeed, one of the things I'm planning on doing is listing those sponsoring on the site by putting their logo up. I assume if folks are paying 2-300$/month for something they will use it. Indeed, it was an uphill battle to get people to use it initially, but its much better these days. Bitfury still has some insanity occasionally, but I'm working with them to solve it.

Well I'm certainly no where near 7% of the network but I'll put forward ~$100 in BTC a month towards it (as long as the biggest pools are on it)
N.B. none of my servers but the single main back end cost me even half of that So I guess this network is either using an expensive provider or high end servers on every node

Yea, they're all dedicated servers since the random lag introduced by VPSes was incredibly annoying. Sadly, for dedicated servers, unlike virtual ones, it can be hard to get them in good locations in some places (APAC especially), and its also very nice to have them all on one network, so they're all on Softlayer. They're a bit more expensive than neccessary, but they have really, really good latency between any two points on their network (with the exception of JPY<->EU as they dont have TEA bandwidth), as well as solid peering in Tokyo/HK with PCCW/Pacnet, which is important for China.

[newIndia]

@Matt I think it is good if u please update OP with FIBRE related information.