Is it possible to have a decentralized pool, or is a central authority required to ensure miners are honest?
It is possible to have a decentralized pool, but IMO building it would be a waste of time, since it will be too expensive to be a full Bitcoin node not too far in the future, and participants in a decentralized pool must be full nodes. A decentralized pool also requires a great deal of bandwidth itself, since all peers must understand the complete state of the pool (as far as I can tell).
One possible design:
- Each miner broadcasts all of the low-difficulty shares they win, which is used to calculate proper ratios for every participant.
- Each miner works on its own block. The coinbase transaction pays out according to ratios that it calculates. Each miner chooses which transactions to include according to its own rules.
- Miners broadcast their block headers, coinbase transactions, and Merkle branches for their coinbase transactions to the entire pool. The pool doesn't need to know which other transactions they include.
- When you receive a header+coinbase, you examine the payout ratios, and if you agree with them, you add the person who sent it to your own coinbase transactions for payout.