Merged Mining is NOT ready and should be stopped until it is

[shads]

Merged mining is only a few blocks away now and from what I can see the bitcoin side of the equation is grossly underprepared.  I hope I'm wrong but I don't think this is going to go well.  The problem as I see it is that the merged mining community has prepared itself but not given the bitcoin community the tools it needs to adapt.  Let me explain.

Documentation is woefully lacking.  Basically limited to a wiki page a some scattered discussion threads on the namecoin forum.  Miners don't need to do anything but bitcoin pools that wish to adopt merged mining need to implement the spec one way or another.  Currently the only way to do this is to use vinced's patched bitcoind and namecoind along with the python merged-mining-proxy.  This proxy has to sit between the poolserver (e.g. pushpool, poolserverj or custom) and the bitcoind.  This represents both a potential bottleneck and an extra point of failure and to my knowledge has never been tested on a load greater than about 50GH.  The alternative is for pools to implement the merged mining spec themselves.  This is easier said than done.  The spec is NOT documented anywhere.  The limit of the documentation that I can find is this:

Parent blockchain (AKA bitcoind)

In order to verify that a client has done hashing on the namecoin blockchain it is nessecary to add a proof that the work has been done on the bitcoin blockchain to the namecoin blockchain. In order to make it possible to have this proof created by the bitcoin blockchain the bitcoind needs a patch. This patch makes it possible for the bitcoind to cryptograhically sign, that there there was work submitted to the namecoin blockchain as well.

Status: Patch ready for testing. Need to ask if it gets implemented to stock bitcoind
[edit] Auxiliary Blockchain (AKA namecoind)

The namecoind now needs to accept the cryptographically signed proof from the bitcoind.

Status: Patch ready for testing. Proposed for block 24000 (but still not agreed upon).
[edit] merged-mine-proxy

This is the daemon all miners connect to. The daemon itself knows how to connect to the bitcoind and the namecoind checking if shares are valid for one of its downstream blockchains. Furthermore it requests getworks from the parent (which must be patched, see above) and delivers it to the miners.

Status: Ready for testing. Implemented using python.

There is also an article in the bitcoin wiki by Mike Hearn that predates the MM implementation by many months but it at best it explains the general principles using quite a different example and none of the specifics.

For a protocol that involves talking to multiple chains, building merkle trees of block hashes in a barely defined order with extra fields that are defined but not actually used in the implementation, extracting merkle branches, parsing undocumented contructs of headers, parts of coinbase transactions and merkle branches and somehow using this to validate aux blocks where the parent block may not even exist... This documentation woefully inadequate, bordering on pathetic.

If someone wants to make an alternative implementation they have no choice but to pore over the source code in two different languages.  I've done this, it took me days to get a grip on it and if weren't for ArtForz's considerable help it would have taken a lot longer.  Even having done that there are a number of ambiguous issues that leave implementation decisions to be guessed at.  Luke-jr has submitted a partial solution but has also pointed out that it's partial because there is no documentation for the spec.  I have attempted to seek clarifications on the spec myself both on the namecoin forum and their IRC channels only to be told that only one person really knows how it works and they are hardly ever online.  The few answers I was able to get simply confirmed that source code can be relied upon as a guide only, there are details of the spec that are ambiguous in the reference implementation and cannot be definatively determined using it as the only reference.

So the bottom line is that every pool that wants to adopt merged mining currently has no choice but the use the untested merged-mining-proxy black box.  So the MM spec is not documented and frankly I think it's grossly irresponsible of the people pushing MM to have let it get this far without doing so well in advance of the cutover block. Without hours/days of poring over the code (in 2 different languages) it's essentially a black box. The provided solution creates a number of problems and the sensible thing to do would be for pools to simply refuse to touch it until that situation is addressed.  

Unfortunately merged mining has created a situation where adopting the change is very difficult to resist. Once one pool adopts it, it puts enormous pressure on the others to follow as the miners will flock to the pool that offers free extra Xcoins for the same mining effort. End result is we have a mass of pools struggling to cope with new xcoind patches and an unproven python point of potential failure/bottleneck and no one available to support it except the merged mining developers who seem to be online for about 10 mins/week.

At best the provided tools (merged-mining-proxy) should be considered a reference implementation of a spec that doesn't exist. I predict there will be grief for pools and miners over the next few weeks/months.  The best case outcome I can think of is that the extra NMCs mined tank the namecoin market and miners lose interest.

If there are problems and 1/2 pools roll back it still doesn't create any incentive for the MM crowd to step up and do the job properly because their motive is increase hash power on the NMC network and they will have acheived that whatever the consequences to bitcoin mining might be...

IMHO merged mining should be stopped until this situation is properly addressed.

[urstroyer]

I totally agree with shads.

I won't take a huge risk on my pool and for all existing miner, if we don't know about any side-effects right now.

If any pool will take the risk with existing software right now, please consider:

- merged mining right now comes with higher risk for your whole backend
-> is the software ready for highloads?
-> what about stale rates?
-> are your pool member ready to accept necessary downtimes and outages?

Maybe i'll regret it, but right now, i'am happy not beeing an early adaptor, even if some people see advantages of merged mining right now and are willing to change a pool.

It's my responsibilty as a pool operator to offer a solid and stable pool service and if i would rush into merged mining right now, i cannot guarantee for it.

[shads]

The key problem though is that you and other more cautious pool ops will come under a lot of pressure to adopt.  There will be pools that do straight away and it's a compelling argument for a miner.  Mine BTC only or switch pools and get bonus NMC at no extra cost.

The usual scenario would be if a new tool is crap no one will use it until it's not crap.  The producer of the tool is strongly incentivized to fix it fast.  In this case there's a strong incentive to use it even if it is crap.  The producer of the tool has no incentive to fix it until they damn well feel like it.

[EhVedadoOAnonimato]

Would somebody care to explain me why people chose to implement Namecoin with an independent chain, instead of a dependent chain?

By dependent chain I mean a chain in which, to produce a block, all you need to do it sign it with the generation address of the bitcoin block you just mined.
This way there would be no need for "merged mining" at all. The only disadvantage I see is that, in the best case, a dependent change can only grow in the same pace as bitcoin's chain. Grow in the number of blocks I mean. And, well, for a naming service, that doesn't look a problem to me.

Another design choice of namecoin I don't understand is the existence of coins. Couldn't bitcoin themselves be used for transferring names, by using contracts?

I clearly don't know much about namecoin and thus don't understand why such choices were made. Probably there was a good reason.... is such reason written somewhere?

Thanks

[Luke-Jr]

Just to be clear, my merged mining implementation AIUI will require two very minor changes to bitcoind, and will not put the Bitcoin mining operation at risk. While the coinbase component (part of Coinbaser, ready for pulling on GitHub) is complete and obviously simple, the part that is lacking is resubmitting new shares to the side MM managing daemon. This is also a fairly trivial change, but without specs to build the external MM manager and test it, I am not comfortable with calling this piece "done".

[twobits]

Merged mining is only a few blocks away now and from what I can see the bitcoin side of the equation is grossly underprepared.  I hope I'm wrong but I don't think this is going to go well.  The problem as I see it is that the merged mining community has prepared itself but not given the bitcoin community the tools it needs to adapt.  Let me explain.

The thing is,  it  is not like it has come out of the blue,  it has taken many months to get here.  People had been complaining about how long it has taken to get here.  The bitcoin community does not need to be prepared.  They can just continue to ignore it if they want to.

Documentation is woefully lacking.  Basically limited to a wiki page a some scattered discussion threads on the namecoin forum.  Miners don't need to do anything but bitcoin pools that wish to adopt merged mining need to implement the spec one way or another.  Currently the only way to do this is to use vinced's patched bitcoind and namecoind along with the python merged-mining-proxy.  This proxy has to sit between the poolserver (e.g. pushpool, poolserverj or custom) and the bitcoind.  This represents both a potential bottleneck and an extra point of failure and to my knowledge has never been tested on a load greater than about 50GH.  The alternative is for pools to implement the merged mining spec themselves.  This is easier said than done.  The spec is NOT documented anywhere.

Yep,  lack of a single place to find most information and much of anything to read that is not code is a real pain. 

Vinced's patched *coind stuff is not the only way to do this.  sacarlson's multicoin-exp has supported merged mining since June now, and he used to post hoping to get others helping to learn and test things out with it.   He has even set up a number of test chains for merged mining.  Almost no one joined in.  People have chosen mostly to ignore mm.

I do agree the documentation should be a lot better.   It is very fragile it seems, and you do need to read code to tweek it at all.  The thing is though it is an OSS project, and they need to do things to get people involved as they are always short man power.   I am tearing my hair out trying to figure out why merged mining works with these two chains, but fails when I add this third chain, but not another third chain etc..     I really wish it was easier to set up.   It is not going to get to be so though with the current man power on it.   

Status: Ready for testing. Implemented using python.

Ready for testing.. but then how many actually have help them with the testing?

So the bottom line is that every pool that wants to adopt merged mining currently has no choice but the use the untested merged-mining-proxy black box.  So the MM spec is not documented and frankly I think it's grossly irresponsible of the people pushing MM to have let it get this far without doing so well in advance of the cutover block. Without hours/days of poring over the code (in 2 different languages) it's essentially a black box. The provided solution creates a number of problems and the sensible thing to do would be for pools to simply refuse to touch it until that situation is addressed.  

Unfortunately merged mining has created a situation where adopting the change is very difficult to resist. Once one pool adopts it, it puts enormous pressure on the others to follow as the miners will flock to the pool that offers free extra Xcoins for the same mining effort. End result is we have a mass of pools struggling to cope with new xcoind patches and an unproven python point of potential failure/bottleneck and no one available to support it except the merged mining developers who seem to be online for about 10 mins/week.

Yep, that's it...  once it is active,  people will take an interest finally, and help developed the needed infrastructure if they want it.  Since pools will probably need it to stay competitive they will probably be a major source of the manpower or bounties to finally get what is needed done after all these months.

If there are problems and 1/2 pools roll back it still doesn't create any incentive for the MM crowd to step up and do the job properly because their motive is increase hash power on the NMC network and they will have acheived that whatever the consequences to bitcoin mining might be...

IMHO merged mining should be stopped until this situation is properly addressed.

Thing is,  we have already had months to work on things, and no one has till it is almost here.  I don't see a delay changing that.  People need real coins to mm or they won't , as the lack of interest in the multicoin-exp test chains shows.

[Luke-Jr]

Thing is,  we have already had months to work on things, and no one has till it is almost here.

No. We don't get to start working until there is proper protocol documentation in place.

[shads]

Thing is,  we have already had months to work on things, and no one has till it is almost here.

No. We don't get to start working until there is proper protocol documentation in place.

^^ this

It far from a trivial process to construct a set of mm blocks and even less so to validate them.  Even one small ambiguity in the process means two different implementations quite likely won't accept each others blocks.

[Gabi]

Well so far it seems to work...

The only risk i see is that you can risk to lose your work on that pool but as long as you accept that risk, i think it's fine

[Luke-Jr]

Well so far it seems to work...

Really? What big pools support it? Where is the data showing there's been no loss of Bitcoin blocks/hashpower? I've in fact heard reports to the contrary...

The only risk i see is that you can risk to lose your work on that pool but as long as you accept that risk, i think it's fine

Due to the panic-rush of it going live before being documented, I have deployed my alternate implementation to Eligius. Unlike most pools (ie, those using the proxy), Eligius does not have this risk.

[Gabi]

Who cares about "big pools"? I only care about my profit. It's not hard to check what my profit should be and then check if i really have that profit. So far, i have that profit.

So again, who cares about "big pools"? As far as i gain bitcoin and namecoin, it is working for me.

[makomk]

Merged solo mining doesn't seem to be too bad, but pooled is another matter entirely. For example, I can't figure out how you're meant to tell if the aux chain part of a share is stale or not...

[iopq]

Well so far it seems to work...

Really? What big pools support it?

slush, apparently

[Luke-Jr]

Well so far it seems to work...

Really? What big pools support it?

slush, apparently

He also uses a custom implementation.

[DeathAndTaxes]

Well so far it seems to work...

Really? What big pools support it?

slush, apparently

Eligius also (well I guess that is a medium pool).

[Luke-Jr]

Well so far it seems to work...

Really? What big pools support it?

slush, apparently

Eligius also (well I guess that is a medium pool).

I meant other pools :p

(obviously Eligius is running my custom implementation)

[sadpandatech]

Luke, did you get the msg about the much needed update to namecoind? http://dot-bit.org/forum/viewtopic.php?p=2182#p2182

  That goes for anyone else that may be solo mining or mining thru namecoind at some of the smaller pools as well.....

[Luke-Jr]

Luke, did you get the msg about the much needed update to namecoind? http://dot-bit.org/forum/viewtopic.php?p=2182#p2182

  That goes for anyone else that may be solo mining or mining thru namecoind at some of the smaller pools as well.....

I reported it. :p

[sadpandatech]

Luke, did you get the msg about the much needed update to namecoind? http://dot-bit.org/forum/viewtopic.php?p=2182#p2182

  That goes for anyone else that may be solo mining or mining thru namecoind at some of the smaller pools as well.....

I reported it. :p

  LMAO. sorry was getting spam happy with it. Sadly, 2 of the larger ones, Slush and Davinci are both away atm I believe. :/

[Luke-Jr]

Luke, did you get the msg about the much needed update to namecoind? http://dot-bit.org/forum/viewtopic.php?p=2182#p2182

  That goes for anyone else that may be solo mining or mining thru namecoind at some of the smaller pools as well.....

I reported it. :p

  LMAO. sorry was getting spam happy with it. Sadly, 2 of the larger ones, Slush and Davinci are both away atm I believe. :/

I know Slush has already upgraded at least.