Geebus hadn't found the issue yet when I posted about it.
But what Geebus found is also very interesting... Out of the last 50 blocks that the pool has found, 4 of them have been invalid.That's an 8% invalid rate over the past 50 blocks!
In the mean time (during basically the same time period), Slush's pool has only had 2 invalid blocks out of the past 500 (0.4%). And Tycho's pool pays 48.5BTC for each block found even if it is eventually found to be invalid.
If they are working on a block and there are 2 other pools that are 5x and 8x faster than them, then that wouldn't exactly surprise me. One of the other pools comes ripping through and finds the solution right before bitcoinpool doesn't seem that far fetched.
That's not how it works.
Invalid blocks are a function of how far away (from an average node distance standpoint) the pool's bitcoind is from the rest of the bitcoin network. A high number of invalid block submissions implies that the bitcoind instance is relatively closer to an edge of the network than the center.
The pool is working on a block with a prevhash pointing to the previous found block. But if the pool's bitcoind doesn't have the latest information from the blockchain, then it will be working against the wrong prevhash. The further away the pool is from the center of the network, the longer it will take (on average) for the pool to get blockchain updates.
It's all about how many other nodes in the network you are connected to. This is why it's a P2P network, kinda like torrents. The more peers you have, the faster you get information.
Currently bitcoin is set to 8 max outbound connections, and 125 inbound connections - MAX_OUTBOUND_CONNECTIONS.
Let's explain with the code itself.
static const int MAX_OUTBOUND_CONNECTIONS = 8;
This means the maximum number of OUTBOUND connections (from you to someone else) is 8. If you want your bitcoind to connect to more than 8 nodes in the network (which isn't that much), then change this number to something higher than 8. I use 1000.
else if (nInbound >= GetArg("-maxconnections", 125) - MAX_OUTBOUND_CONNECTIONS)
This means the max number of inbound connections you can have is the "-maxconnections" option (which defaults to 125) minus the MAX_OUTBOUND_CONNECTIONS (which is 8 by default). By default, you can have 8 outbound (from you to others) connections and 117 Inbound (from others to you) connections. Inbound connections don't work if you're behind a firewall and haven't setup port forwarding or access to port 8333.
int nMaxOutboundConnections = MAX_OUTBOUND_CONNECTIONS;
nMaxOutboundConnections = min(nMaxOutboundConnections, (int)GetArg("-maxconnections", 125));
This is choosing the smaller number between MAX_OUTBOUND_CONNECTIONS and the "-maxconnections" command line argument, and uses that number for the maximum number of outbound connections. This would be 8.
So here's the changes I made for the colo server.
static const int MAX_OUTBOUND_CONNECTIONS = 1000;
else if (nInbound >= GetArg("-maxconnections", 1500) - MAX_OUTBOUND_CONNECTIONS)
I also removed line 1133 entirely.
Right now the bitcoind on the server has 256 connections to others nodes in the bitcoin network. I intentionally kept this at 256 connections. 1 of those connections is to a server in a colo that was modified to have 1000 max connections. The colo server currently has 923 connections, and an uptime of 53 days.
I would say we are very, very well connected to the bitcoin network. If nobody else is doing this, then we might be the node with the most connections in the bitcoin network.
It's probably not a good idea for everyone to do this, but since I run a pool, I want to be as well connected to the bitcoin network as possible.
You'll probably be interested to know that slush thinks this is a good idea too. I mentioned it to him several months ago.
I'm responding to your notice in Coop mining thread:
BTW, my modified Linux clients can maintain 1024 active connections (yes, I modified the code...452 connections at the moment) to other users bitcoind processes
Is this patch available? Or how much tricky it is? I didn't read bitcoin code too much as I'm not C programmer, but it looks like rising connection limit constant should be enough. Am I right?
Rising this limit should improve pool performance a little, because new blocks aren't distributed inside bitcoin network effective enough.
I'm not submitting an official patch because if everyone did this the network would get very, very active with the same information being broadcast through out the entire bitcoin network several hundred times.
So back to the point, we just got unlucky with our blocks. Someone else got a better fit answer for the block so the pool didn't get paid, and that's why everyone else didn't get paid. We had code in place to catch when this happens, but we found the typo which caused it to not execute properly. As a result invalid blocks were being treated as valid, which is also the reason everyone's historical earnings were off. We have gone back through and marked the invalid blocks as invalid, and fixed the historical earnings at the same time.
Gotta love them bugs.