That is categorically not the case with merged mining.  Due to market forces as discussed above if you fail to adopt there is a cost.  This is particularly the case for pools.  If they adopt to avoid the cost of not doing so (i.e. not because they want to) there is also a cost.  Stability and significant performance hit.  They may not want the benefits but they can't afford not to have them.

Unfortunately you don't have a choice.  I'm not against the concept of merged mining, just the appalling way it was implemented and foisted on the bitcoin chain without proper documentation and testing.  All the destabilization issues we've seen were totally predictable.

The reason there is no choice is because of the 'moar profitz!' mentality.  This will ultimately be proven a false proposition imho but in the short term as some of the replies in this thread clearly demonstrate along with the rapid uptake, if you dangle more profits in front of miners they will take them.  At the end of the day there is no more money in the combined bitcoin/namecoin economies so what little value is left in namecoin after the market has fully digested the change will come off the top of bitcoin.  So being a non-merged miner still means you're missing out.

There's no point having the discussion 'should we support merged mining or not'.  It's happening whether we like it, whether are ready for it or not.  

Sorry I meant how does one get onto this list: https://github.com/Unthinkingbit/charity/blob/master/bitcoindonationinformation.html

I'm only putting my hand up for modifying poolserverj to support multiple aux chains which is prerequisite for pools using it to support devcoin merged mining.  I'm not a C/C++ programmer so I can't really help much with the client.

use the link in the above post

You may want to direct yr grievance elsewhere.  I warned about the problems merged mining was going to cause before it went live.

however I will point out that the drop in hashrate has fuck all to do with merged mining.  A major part of the reason 'other' is so big atm is because btcguild has had a major restructure and doesn't seem to be recognized by btcwatch anymore.  They account for at least half of the 'other' category.

psj is currently recording NMC only shares as our_result=yes reason=partial-stale.  This is an interim solution, there's a discussion on the psj thread atm to decide on the best strategy for recording this properly so pools have complete information for calculating shares that can account for more than one aux chain.

And I reiterate it's possible I'm wrong but what you say makes perfect sense in the context.  More to the point efficiency is being far more negatively affected by cgminer nodes and by non-cgminer nodes.  When pools rectify their accounting it will be the cgminer nodes that take the brunt of it since they are currently getting more than their 'fair shares'...  Pools will have to do this for the same reason many are feeling compelled to introduce merged mining... i.e. they are at a competitive disadvantage if they don't.  I say this without prejedice because I hope the situation for other pools will change quickly but poolserverj has uncovered this problem and is the first to deal with it properly so for the time being poolserverj-mm based pools offer miners better value for their hashes overall.  Once miners realize this the pressure will be on for other pools to sort it out.

It depends how the pool handles them.  As far as I can tell for most pools that use MMP the problem will be hidden so the miner is unlikely to see the stales even though they're occurring.  When psj-mm edition got to the part about handling these (because psj-mm has it's own native implementation and doesn't use merged-mining-proxy) we started seeing mass stales and began investigating.  Of course psj got the blame initially because it appeared other pools were working fine (even I thought it was something wrong with psj).  But after getting all the per chain block tracking working it became pretty clear what was going on...

I know for a fact that NMCBit is wearing the losses at the moment.  They are using PSJ + MMP.  Psj is NOT compatible with MMP, the problem being that psj doesn't know when to switch to new blocks because MMP doesn't send the right signals.  He is copping a LOT of stales but you are not seeing them and because the pool is PPS it's him that's wearing cost, he's not passing it on to miners.  When he switches to psj-mm edition this will probably change a little as you'll actually start seeing yr own real stales.  To clarify atm nmcbit is suffering two kinds of stales, the normal kind that is miner related (which miners should be paying for) and the bonus stales from the PSJ MMP incompatibility.  He's taking the losses for both right now.  With psj-mm the second kind won't happen but the first kind will be transferred back to the miners where it belongs.

I can't comment on slush as he uses his own custom pool software.  I suspect he's probably eating the aux chain stales as well.  Even if he's sending LP when the aux chains update and dealing with the double LP problem he still won't be able to force cgminer clients to switch.

It's probably worth noting that a proportional pool won't actually incur obvious costs from this problem.  Unless they go looking the problem will likely only become apparent after a while when they calculate pool luck for the alt chains.

c_k: yes and I don't see why it would cripple cgminer.  As I understand it (I'm no expert on cgminer) it lowers stales two ways, by prefetching work and by monitoring other pools to see if they move to a new block before the one yr mining does.

I'm not proposing to disable prev_block_hash checks, just to accept that when a LP comes in (as long it's a proper longpoll response with a full payload) that it should start working on that work and replace it's cached work.  Prefetching still happens, it isn't nerfed at all, it just happens a little more often.  Monitoring other pools still gives potential advantages, it's just limited to detecting new blocks on the parent chain (bitcoin).  

So no functionality is lost as far as I can see.  Some of the benefits of cgminer won't work as well for aux chain changes but there'll be no loss of functionality for the main chain so the net effect is +ve and still in excess of what other mining sw does.

kano: I'm not here to debate the merits of merged mining.   I'm not a fan either.  But the market has spoken and miners want it so if cgminer is crippled for merged-mining compared to other mining s/w (and I'd call stales several times higher competitively crippled) it's going to rapidly lose users.  I'm just presenting a solution to yet another problem MM has caused.  It's not really even in my interests do so.  Most of the miner specific issues I've had to deal with for poolserverj are issues with cgminer trying to be too clever so quite frankly it would make my life easier if conman gave MM the finger and refused to deal with it.  Still it would be a shame to lose such a well engineered tool to obsolescence.

yes it will... I would personally suggest that miners who want this fixed pool together and put up a bounty.  I've looked through cgminer code and I'd guess the work involved in building it is in the same order as poolserverj (i.e. months).  I enjoyed writing poolserverj but one of the least fun things about it is supporting new systems like merged-mining which you don't have any interest in.  I'm fairly sure I wouldn't have bothered supporting MM if it weren't for tempting bounties waved in front of me.

I'm not sure if conman feel terribly inclined to make changes that accomodate merged mining but I've come across a serious issue that is cgminer specific so perhaps in this case he may make an exception.  If some of my assumptions here are wrong my apologies but I've observed it's behaviour and looked through the source code and I'm fairly sure I've interpreted it correctly...

The problem is summarised here:
https://bitcointalk.org/index.php?topic=33142.msg593497#msg593497


and a bit more detail here:
https://bitcointalk.org/index.php?topic=33142.msg594007#msg594007

The end result is that cgminer refuses to acknowlege that the block has changed as it doesn't accept LP as an authoritive indicator that it should discard it's local work and use the new work.  This means, particularly in the case of PPS pools, that pools are likely to penalize cgminer users by either not awarding the share at all or only giving them a partial credit.  

Depending on which policy they choose this could end up costing the miner either a significant chunk of their namecoin rewards or possibly even a large chunk of bitcoin rewards.  I'll go through the scenario and the possible effects:

pool rejects partial shares:
What happens is this:  The BTC and NMC blocks change but at slightly different times.  If the BTC block changes first cgminer get LP and acts on it as it sees a new prevblockhash.  A few seconds later the NMC block changes.  This doesn't change the prevblockhash so cgminer carries on with the work it already has.  When it submits a share the pool sees that it's only valid for one of the blockchains and rejects it as stale.  This continues for as long as cgminer would normally hold onto the same work (up to 60 seconds) or until the BTC block changes again.  In between BTC block you'd usually exepct several NMC blocks as the difficulty is lower.  Se even if cgminer has given up the work after 60 seconds, as soon as another NMC comes along the same situation occurs.  The miner is working on work that is only valid for one chain.

This is not theoretical it's really happening and it's being hugely under reported on most pools (see next para for explanation of that), I've seen it in testing and on production pools.  Overall stales are a little higher they were pre-merged mining which is partly to do with more frequent block changes and partly due to some design clashes between merged mining and longpolling.  But cgminer stales are typically many times higher than for other miners.  The 'dumb' miners (i.e. the one's just accept new work from the LP and don't check if it's a new block) work fine because they trust that the pool is right when it says 'new block'.

Many merged mining pools which aren't using poolserverj probably have this problem also but it's invisible to them.  Merged-mining-proxy does not do any sort of LP and unless the pool ops have made some fairly invasive changes to pushpool it won't be internally aware of changing block on different chains either.   I suspect they've just been accepting shares they shouldn't have.  When they all start to realise what's going on expect some policy changes from these pools.

The fix seems fairly simple on the surface but the devil is always in the details...  I think conman is not keen on blindly accepting LPs as some pools send out bullshit LPs occasionaly (even poolserverj does if the block doesn't change after 10mins).  But it needs to be an option.  Perhaps a command line switch that miners can use only if they are merged mining.  All it needs to do is replace the work it's currently hashing with the contents of the longpoll (regardless of whether prevblockhash is different) and clear it's cache.  There's no cost to doing this except perhaps a few getworks to fill up it's cache again.

registering my interest... poolserverj currently only supports one aux chain and will some small mods + a whole lot of testing to enable multiple aux chains. 

BTW how do you get on the devcoin developers list?

If anyone has some spare time to play it would be really helpful if someone could download this version and run it for a while then send me the log:
http://poolserverj.org/dist/mm-mini-binary.09.debug.tar.gz

It looks like stales have been dramatically reduced but for some reason cgminer is sending back a few works that are registering as unknown.  The above version is identical to .08 but has a bit of extra logging in it that should help track down the source of the problem.

edit: updated post to point to .09 which has a minor fix for issue where in some cases partial-stales were treated as full shares.

If anyone's downloaded .08 and found an odd looking file in there:  bitcoin-poolserverj-0.0.2-SNAPSHOT.jar

That's actually poolserverj.jar which I forgot to rename.  It should be in the bin directory and *no where else*.  I've uploaded a new version with the correct name. 

I have to agree.  Writing shares to DB is already one the biggest bottlenecks in psj.   Doubling or tripling the number of rows written is definately not ideal, likewise with using a seperate table with a many-to-one relationship.

I'm wondering if perhaps a floating point share value column should be added.  There would need to be an interface so the pool could dynamically update the value of each chain.  The value field is then the sum of the solved chain's values.  The disadvantage of this is that values are changed frequently then it's retrospectively difficult to go back and try to work which chains each share solved for.

The remaining options are a single consolidated column like bitfield flags or a boolean column per chain.  I'm not sure what factorial planet davinci was on but the way you normally do it is allocate each bit to a flag.  so 1 = bitcoin, 2=namecoin, 4=scamcoin.  Then each is easy to extract with a simple logical &.

If you don't want to record which chains are solved you have a simple choice between accept or reject partials.  Accepting them provides the miner no incentive to fix any latency problems and hurts the network overall, not mention costing PPS operators a pretty penny.  Reject is going to fill yr inboxes with complaints (which I personally think you should be redirecting to the namecoin forum).

Simplecoin uses merged mining which wont work with scrypt based chains.


though if they ran a separate instance of psj there is now scrypt support built into it since .07

if you mean a single share from yr own pool... btc diff / nmc diff ~= number of nmc only shares per btc+nmc share

If you mean across the whole network that's a bit more complicated since you've got to account for non-merged miners.

your suggestion 1 is how the current implementation works.  A normal accept share will have our_result=true, reason=null.  A partial-share will have our_result=true, reason=partial-stale

re: 2.... Perhaps a bit mask would be an option.  Though this is limited and any combo value field tends to make efficient db queries difficult to achieve.

Ive just been looking through the cgminer source code and it looks to me like it does use prev_block_hash as an indicator of whether the block has changed.  I'm happy to stand corrected but if that's the case it's a fairly serious problem.  It means whenever an aux chain finds a new block but the parent chain doesn't that cgminer will ignore the new work (even if it gets it via longpoll).  I don't know how many other miners do this but pool ops should start looking into this.

With 'partial-stale' shares being accepted it effectively means the miner will get credited for both chains while only working on one.  Unless shares are actually assigned a 'credit value per chain' the choice is between crediting these partial shares or disallowing them unless the share is valid for all chains.  The 'credit value per chain' option will require and extra database column which will break pushpool compatibility completely (it had to happen sometime) and require all pool ops to make some mods in how they are calculating worker shares.

Need feedback from pool ops on how to deal with this.  This is a particularly significant issue for PPS pools where you'd end up paying out a full share where only part of the work is valid.

Thanks for the feedback... keep it coming...

ok I've just uploaded the latest revision .08.  Please note the base distribution was updated at .07 to apply .08 you will need to start with .07 and apply the contents of .08 mini-binary over the top.

This contains a number of changes that I hope will address most of the 4 issues I highlighted above...

- support for partially stale shares.  Where a single chain has advanced to the next block old work may still be valid for the other blocks.  This patch checks that the work is valid for at least one block giving three possible outcomes.  accepted, accepted partial-stale, stale

- longpoll passthru.  To address the double longpoll issue. There is a fundamental design clash between longpolling and merged mining.  Basically it works like this:  When there is a double block change (i.e. btc and nmc solved by one solution) the daemons won't see the new block at exactly the same time.  From what I've seen a delay of around a 1-2 seconds is not unusual.  What happens then is that one chain advances to the next block.  PSJ checks the other chains to see if they've updated, sees they haven't so starts sending out LPs.  Before the miner receives the LP and establishes a new LP connection the second chain advances so PSJ starts sending out another batch of LP's.  Those miners who haven't got their new LP connection registered in time miss out so continue working on the old block for as long as they would normally (probably about a minute).

This patch addresses this by setting a longpoll passthru period.  Where two block changes happen within a specified period (10 seconds).  After the 2nd block change and until 10 seconds after the 1st block change has passed, any longpolls received will have a short expiry of 1 second after which they'll return new work to the worker.  This will give any slow miners a few seconds to get their longpoll in and learn that there's a 2nd new block to work on. The reason for the 1 second delay is to prevent longpoll spam.  Most (but not all) miners will immediately send another LP request as soon as they get a response.  With 0 delay this sets up an LP spam loop which thrashes the server and causes high CPU usage and low response times.  They will still spam 1/sec until the passthru period is over which is why this is called a workaround rather than a fix.  There's really no way to fix this issue completely except to ditch either longpolling or merged mining alltogether.