Re: How (and why) to use the Relay Network

[Matt Corallo]

I rearranged my bitcoinds (10 days ago) and ended up with the relay using the external IP
(will fix it next time I restart bitcoind, don't like doing that too often)
Anyway, as a result it got itself banned - seems the relay had a lot of bad transactions 6 hours ago

2015-04-07 16:01:35 Misbehaving: a.b.c.d:33466 (0 -> 10)
2015-04-07 16:25:06 Misbehaving: a.b.c.d:33466 (90 -> 100) BAN THRESHOLD EXCEEDED

That's the first message in the 10 days, then only 24 minutes to hit the threshold

Can you paste the log contents around those lines (ie why it got banned?)

[kano]

I rearranged my bitcoinds (10 days ago) and ended up with the relay using the external IP
(will fix it next time I restart bitcoind, don't like doing that too often)
Anyway, as a result it got itself banned - seems the relay had a lot of bad transactions 6 hours ago

2015-04-07 16:01:35 Misbehaving: a.b.c.d:33466 (0 -> 10)
2015-04-07 16:25:06 Misbehaving: a.b.c.d:33466 (90 -> 100) BAN THRESHOLD EXCEEDED

That's the first message in the 10 days, then only 24 minutes to hit the threshold

Can you paste the log contents around those lines (ie why it got banned?)

Looks like it was these:

Code:

2015-04-07 16:23:53 ERROR: CheckTransaction() : vin empty
2015-04-07 16:23:53 ERROR: AcceptToMemoryPool: : CheckTransaction failed

I'll PM you more of the log

[kano]

Had a lot of trouble with the relay over the past day.
Anyone else seeing problems with us-west?

And like at the moment, I seem to often be the only one sending data to the relay, not receiving anything from the relay.
... it suddenly comes back to life with a truck load of Receives for a very short time, but again now back to only me Sending

[twebit]

Had a lot of trouble with the relay over the past day.
Anyone else seeing problems with us-west?

And like at the moment, I seem to often be the only one sending data to the relay, not receiving anything from the relay.
... it suddenly comes back to life with a truck load of Receives for a very short time, but again now back to only me Sending

For me it was sometimes off. Other times saw an inflood of transactions.
Sorry to say I shut it down and moved to pool mining again but I have one question...

Why? if this is a relay, do we/I have to connect to what it says on the home page of this post. Rather connect to anyone? place or thing?
I don't understand much but can't there be other relays in the bunch other than the initial connect to "java -jar ./RelayNodeClient.jar public.us-east.relay.mattcorallo.com 127.0.0.1:8333" ? reliability seems to be not so much currently. No dis on Matt. But why is he the only one to initiate the relay "string" for lack of better word.
I've seen it where it goes down and you can't connect. Connect to another?
Just saying
Thanks

[kano]

Well I connected to east also and ... all the transactions sent to west came back on east
Each time one went out west another would come in on east the same size.
So yeah seems like there's some problems with the relay.

I do still see runs of receives (like just now as I write this after stopping east again) but they never last long.

Edit: heh - well it's finally acting OK now for the last 5 minutes

Lasted about an hour and a half or so.
But back to me only sending again (nothing received)

Edit: and it's back working OK again.

[kano]

Looks like the relay is messed up at the moment.
It seems to be resending the last 1000+ blocks again ...

Its not really causing any problems but seeing so many ignored blocks arrive off the relay is a little disconcerting ...

[kano]

Another problem with the relay today.

Invalid blocks were coming through for a while which actually resulted in drastically slowing down block acceptance.
I run multiple relays and multiple bitcoinds on multiple nodes so I was able to see that only the relay connected bitcoinds had the problem.

Here's a few (debug.log UTC)
The UpdateTip in each case is when the bitcoind 'later' got it from the net, not via the relay

Code:

2015-06-02 03:55:40 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-02 03:55:40 ERROR: ProcessNewBlock : CheckBlock FAILED
2015-06-02 03:55:40 Misbehaving: 127.0.0.1:37844 (600 -> 700)
2015-06-02 03:55:42 UpdateTip: new best=000000000000000016f785f6c56856755d3fc89ca3ef005064ca10400872c421  height=359029  log2_work=82.873403  tx=70633458  date=2015-06-02 03:54:37 progress=0.999999  cache=5511

2015-06-02 03:56:40 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-02 03:56:40 ERROR: ProcessNewBlock : CheckBlock FAILED
2015-06-02 03:56:40 Misbehaving: 127.0.0.1:46865 (500 -> 600)
2015-06-02 03:57:21 UpdateTip: new best=00000000000000000b3f90e0f60046423db6b2cac055f41290ae29f966433fc9  height=359030  log2_work=82.873437  tx=70633604  date=2015-06-02 03:56:13 progress=0.999999  cache=6156

2015-06-02 04:08:39 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-02 04:08:39 ERROR: ProcessNewBlock : CheckBlock FAILED
2015-06-02 04:08:39 Misbehaving: 127.0.0.1:46865 (600 -> 700)
2015-06-02 04:08:42 CreateNewBlock(): total size 600611
2015-06-02 04:08:42 UpdateTip: new best=00000000000000000fa58e5a02b8676291bb491192a738a6d800f4bef296f3a8  height=359031  log2_work=82.87347  tx=70634550  date=2015-06-02 04:14:25 progress=1.000006  cache=8546

In the above case I was running both east and west relay connections (as you can see they came from both)

On the same node there's another bitcoind not connected to the relay and it was getting the same blocks at roughly the same time, but without errors.

However, on another node somewhere else on the planet also connected to the relay, I didn't get all of them, but only the last one:

Code:

2015-06-02 04:08:39 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-02 04:08:39 ERROR: ProcessNewBlock : CheckBlock FAILED
2015-06-02 04:08:39 Misbehaving: 127.0.0.1:45711 (0 -> 100) BAN THRESHOLD EXCEEDED
2015-06-02 04:08:39 Warning: not banning local peer 127.0.0.1:45711!
2015-06-02 04:08:39 UpdateTip: new best=00000000000000000fa58e5a02b8676291bb491192a738a6d800f4bef296f3a8  height=359031  log2_work=82.87347  tx=70634550  date=2015-06-02 04:14:25 progress=1.000006  cache=8074
2015-06-02 04:08:40 receive version message: /RelayNetworkClient:42/: version 70000, blocks=0, us=[::]:0, peer=4901, peeraddr=127.0.0.1:33197

Then it stopped after block 359031

[Matt Corallo]

Sorry for this thread being completely dead over the past few weeks. I was working hard on this and then got distracted with Sidechain Elements Alpha release and the blocksize debate. In any case, I pushed a series of updates yesterday and today which include:

• Fixed a bug on the server where servers were not properly filtering transactions, allowing at best DoSing of nodes (some people saw this as 100% bitcoin node usage) and at worst, getting relay clients banned from local bitcoin nodes.
• Some simple improvements that speed things up as much as 50%
• Fixed bug that might have allowed for duplicate block sends
• Update clients to send their local mempool on first connect/reconect, making the first block(s) sent also compress well
• Removed the old Java/Python clients, they're slow and dumb - use the C++ ones

Note that these updates include some great client-side changes, though things should still work fine if you dont.

This still leaves the merkle hash tree mismatches, which I've been aware of for a while but are quite hard to track down :/ (oh well, if we miss a few blocks, we miss a few blocks, this is a backup network :p).

Also put up a website at http://bitcoinrelaynetwork.org/, which should have the latest information. I want to put up a status flag there, but thats for later.

[Matt Corallo]

Oh, and I believe the merkle tree issues should (finally) be fixed now as well

[kano]

Sorry for this thread being completely dead over the past few weeks. I was working hard on this and then got distracted with Sidechain Elements Alpha release and the blocksize debate. In any case, I pushed a series of updates yesterday and today which include:

• Fixed a bug on the server where servers were not properly filtering transactions, allowing at best DoSing of nodes (some people saw this as 100% bitcoin node usage) and at worst, getting relay clients banned from local bitcoin nodes.
• Some simple improvements that speed things up as much as 50%
• Fixed bug that might have allowed for duplicate block sends
• Update clients to send their local mempool on first connect/reconect, making the first block(s) sent also compress well
• Removed the old Java/Python clients, they're slow and dumb - use the C++ ones

Note that these updates include some great client-side changes, though things should still work fine if you dont.

This still leaves the merkle hash tree mismatches, which I've been aware of for a while but are quite hard to track down :/ (oh well, if we miss a few blocks, we miss a few blocks, this is a backup network :p).

Also put up a website at http://bitcoinrelaynetwork.org/, which should have the latest information. I want to put up a status flag there, but thats for later.

Thanks for this.
Updating a few nodes to see how many of my bitcoin problems are relay and how many are 0.10.x exploit network problems
(I run bitcoinds without the relay for comparison)

[Matt Corallo]

Actually, just pushed yet another big performance win. Things should be back down to much quicker-than-latency compression runs.

[Matt Corallo]

Oh, and a big memory usage reduction win, too, though clients dont care too much about that .

[loshia]

Oh, and a big memory usage reduction win, too, though clients dont care too much about that .

I do not think so
I am a sawp hater
Thanks Matt great job!

[Matt Corallo]

I do not think so
I am a sawp hater
Thanks Matt great job!

Well the client-side stuff wont even use more than a few 10s of MB, so I dont think thats an issue for anyone?

[loshia]

I do not think so
I am a sawp hater
Thanks Matt great job!

Well the client-side stuff wont even use more than a few 10s of MB, so I dont think thats an issue for anyone?

Honestly speaking I did not watch old client mem usage at all. My mem usage issues were not because of your client. What I wanted to say is that we do care about mem usage in general.

[kano]

I do not think so
I am a sawp hater
Thanks Matt great job!

Well the client-side stuff wont even use more than a few 10s of MB, so I dont think thats an issue for anyone?

Honestly speaking I did not watch old client mem usage at all. My mem usage issues were not because of your client. What I wanted to say is that we do care about mem usage in general.

Get more ram
Relay uses about 10Mb of ram at the moment ... nothing compared to anything else running on my pool.

[loshia]

I do not think so
I am a sawp hater
Thanks Matt great job!

Well the client-side stuff wont even use more than a few 10s of MB, so I dont think thats an issue for anyone?

Honestly speaking I did not watch old client mem usage at all. My mem usage issues were not because of your client. What I wanted to say is that we do care about mem usage in general.

Get more ram
Relay uses about 10Mb of ram at the moment ... nothing compared to anything else running on my pool.

I know:)
once again i was speaking in general

for the record

             total       used       free     shared    buffers     cached
Mem:      16431628    6109668   10321960          0     393708    3062792
-/+ buffers/cache:    2653168   13778460
Swap:      9959420          0    9959420

Now Ontopic new relay client with log filtered out



Thu Jun 18 08:28:53 EEST 2015: Starting relaynetworkclient
Connected to bitcoind with version 70002
Finished connect handshake with bitcoind
Closing relay socket, failed to read message header (104: Connection reset by peer)

Connected to relay node with protocol version the blocksize
[2015-06-18 05:56:02+00] 000000000000000004fd86600f6a12e5186ec18f11d7502afba3b7f7e00cae1a Block recv'd, size 932103 with 5222 bytes on the wire
[2015-06-18 05:58:00+00] 00000000000000000bb1ed574ab76212bc3fd86fad043e48a804b5baab873114 Block recv'd, size 924435 with 1573 bytes on the wire
[2015-06-18 06:03:14+00] 000000000000000010e5b0cdfc902b4b7f700177dd400f411f85bb861837fe42 Block recv'd, size 213069 with 827 bytes on the wire
[2015-06-18 06:08:58+00] 00000000000000000d7ba82013c1bd3354330dfab5406e3b5b31d779bae7306d Block recv'd, size 298966 with 1170 bytes on the wire
[2015-06-18 06:22:16+00] 000000000000000008a705191b01905215e1c37de8c4fd5f3c8aec8f4a98fa3f Block recv'd, size 438564 with 1899 bytes on the wire
[2015-06-18 06:24:21+00] 00000000000000001023e903ecbc7d4308c2c0659c92c5bb8d67e95341b81990 Block recv'd, size 118202 with 561 bytes on the wire
[2015-06-18 06:24:47+00] 0000000000000000121fe92d40e84a77c5e1a3e2aa0ef202f2c150e82c7fd1ce Block sent, size 43943 with 35187 bytes on the wire
[2015-06-18 06:24:52+00] 0000000000000000121fe92d40e84a77c5e1a3e2aa0ef202f2c150e82c7fd1ce Block recv'd, size 43919 with 4917 bytes on the wire
[2015-06-18 06:54:13+00] 000000000000000014e4aca7174b5f4468836a7569147b1b6ed787b842869e05 Block recv'd, size 749195 with 3347 bytes on the wire
[2015-06-18 07:05:45+00] 00000000000000000dc26e1e68c1c04e34db2faeaa2771b0156738d463e074bf Block sent, size 707268 with 603451 bytes on the wire
[2015-06-18 07:05:46+00] 00000000000000000dc26e1e68c1c04e34db2faeaa2771b0156738d463e074bf Block recv'd, size 707244 with 1880 bytes on the wire
[2015-06-18 07:09:56+00] 0000000000000000148e854a12e5eb378e5adaf6f8f5e416963e6e78bd0b6e1a Block recv'd, size 201317 with 952 bytes on the wire
[2015-06-18 07:10:20+00] 00000000000000000d1e68b1aabb79565f8307b97655d2a21c6f99b0d209a989 Block sent, size 26317 with 24792 bytes on the wire
[2015-06-18 07:10:39+00] 00000000000000000d1e68b1aabb79565f8307b97655d2a21c6f99b0d209a989 Block recv'd, size 26293 with 229 bytes on the wire
[2015-06-18 07:30:18+00] 0000000000000000119247bbeea1fa3ae3697482e9d1ec294e7d151e5927c583 Block recv'd, size 741555 with 2971 bytes on the wire
[2015-06-18 07:32:34+00] 000000000000000007802e4a62af5b844a93271cb6ab6d14285a4fa34c3b23fe Block recv'd, size 115723 with 395 bytes on the wire

And i am faster with two blocks only  
There is no match between well connected bitcoin deamon and mat realy

Thanks once again Matt!

[Matt Corallo]

Posted some anonymized stats for those interested at http://bitcoinrelaynetwork.org/stats.html. I had rotated the logs not too long ago because they changed format, so they only go back a few days, but I'll update them every now and again (or when pinged) and they should show more history.

[xyzzy099]

I am trying to upgrade to the latest client on github, but it is not working for me for reasons I don't understand.  The old client still seems to work, but when I try to start the new one, I get this:

Code:

relaynetworkclient.exe public.us-east.relay.mattcorallo.com 127.0.0.1 8333
Closing relay socket, failed to read message header (0: )
127.0.0.1 Disconnect: failed to read message header (No error)
127.0.0.1 Disconnect: failed to read message header (No error)
Closing relay socket, failed to read message header (0: )
127.0.0.1 Disconnect: failed to read message header (No error)
Closing relay socket, failed to read message header (0: )
127.0.0.1 Disconnect: failed to read message header (No error)
Closing relay socket, failed to read message header (0: )

Any ideas what the problem might be?

[Matt Corallo]

Any ideas what the problem might be?

The problem is that Windows is an absolutely terrible OS...but it should work now.