what sort of rates were you seeing previously?

I was just about to ask about this.  Is  anyone else seeing higher stale rates?  From what I've heard it's only cgminer clients that are getting it.  Any info people can report back will be helpful.

There's a numbers of issues at play here I think...

1/ There is a fundamental design clash between longpolling and merged mining.  Basically it works like this:  When there is a double block change (i.e. btc and nmc solved by one solution) the daemons won't see the new block at exactly the same time.  From what I've seen a delay of around a 1-2 seconds is not unusual.  What happens then is that one chain advances to the next block.  PSJ checks the other chains to see if they've updated, sees they haven't so starts sending out LPs.  Before the miner receives the LP and established a new LP connection the second chain advances so PSJ starts sending out another batch of LP's.  Those miners who haven't got their new LP connection registered in time miss out. 

2/ cgminer may be a bit too clever for it's own good.  It does it's own checks on whether work is valid and I presume it uses prev_block_hash to check or maybe the X-Blocknum header.  If an NMC block is found but it isn't a BTC solution then the pseudo block number will advance but the X-Blocknum header (which is for BTC block in psj) and the prev_block_hash won't change.  So cgminer may think it's the same block and carry on using it's cached work.

3/ The rules for accepting a 'partially stale' share are not defined.  If NMC block advances the previous work can still produce a valid BTC block although it will be rejected by the NMC daemon.  The default behaviour is if it's not the current block (block in psj is now defined as a pseudoblocknum which is sum of all chain's blocknums) reject it as stale.  This may have to change but I need some consensus from pool ops on the what the correct behaviour should be.  I'm not going to code every option under the sun coz I'm little bit sick of merged mining TBH.

4/ The code for detecting and switching to new blocks is still pretty raw.  It's possible the 'fireBlockChange' sequence is happening more often that it should which would put valid shares into the stale list.  I don't know if this is the case but the solution 3/ contains the fix for this issue if it exists...

Yes thts the one.  I just search main maven repo & it came up first.

If I'd realised how little had to be changed I would have done this weeks ago but that's how it goes...

The most recent version of psj poolserverj-0.3.0.07.MM.tar.gz now has Scrypt support.  You'll have to remove all the merged mining config options from the properties file and set POWalgorithm=scrypt

Fair warning, this is completely untested as I don't have a scryptBrix/coin client.

If I'd realised how little had to be changed I would have done this weeks ago but that's how it goes...

The most recent version of psj poolserverj-0.3.0.07.MM.tar.gz now has scrypt support.  You'll have to remove all the merged mining config options from the properties file and set POWalgorithm=scrypt

Fair warning, this is completely untested as I don't have a scryptBrix/coin client.

You're missing a class that is in the 'core' snapshot file.  Did you copy the snapshot files into lib/lib_non-maven?

MM edition .06 is now on the downloads page.  This is first version of MM that I'm reasonably happy with.

It should now be sending longpolls if either block changes and the server is now actively monitoring the state of all block chains..  Issues with high CPU load caused by PSJ rejecting valid work should also be resolved.

If you are going to try it I'd appreciate if you could set the following:
debug=true
trace=true
traceTargets=blockmon

If anything odd happens this should give me some useful info to play with.

MM edition .06 is now on the downloads page.  This is first version of MM that I'm reasonably happy with.

It should now be sending longpolls if either block changes and the server is now actively monitoring the state of all block chains..  Issues with high CPU load caused by PSJ rejecting valid work should also be resolved.

If you are going to try it I'd appreciate if you could set the following:
debug=true
trace=true
traceTargets=blockmon

If anything odd happens this should give me some useful info to play with.

Thanks for the info Urstroyer...  I think this is the indicator I needed:


Somewhere in the chain it's discarding valid work and getting more from the daemon.  Not because of duplicates in this case.  One of the validation checks it does is to ensure the work is from the current block before sending out.  It seems having different sets of block numbers is causing a bit confusion.

I'm actually nearly ready to release the fix that should make longpolling work for both chains and I've replaced the blocknum field (which was just a long int) which a new BlockNumbers class that tracks blocks for all chains so that should be the end of it with a bit of luck.

When you see this happen can you try checking http://localhost:8997/?method=getsourcestats and report back is Duplicate rate or Reject rate are above zero?

If not can you set debug=true trace=true traceTargets=merged and throw the output onto pastebin?

Well before the advent of merged mining PSJ had exceptional longpoll performance but as you can see from the last few posts in this thread there's a few issues to be ironed out....

There is one good reason why you'd want to have timestamps set on the psj side rather than DB side.  Because psj caches shares and bulk writes them there can be a delay between when they came in and when the DB sees them.  PSJ timestamps the share as soon as it's received and uses this timestamp when writing to the db.  So if accurate share times are important to you that's something to consider.

I'm not really familiar with mongo or no-sql.  If they have JDBC drivers then adding support would be fairly trivial.  However it won't happen until mm is stabilised.  Dropping mysql support isn't likely sits it's most commonly used.

Having poolserver insert a timestamp directly should also be fairly trivial.  The internal representation is the same as a unix timestamp GMT but in millis instead of seconds.  If you're comfortable building from source it would only be a couple of lines needed modding in DefaultPreparedStatementSharesDBFlushEngine.  If you really want crazy performance and your share writes don't update existing rows have a look at the bulkloader engines in the source.

I presume what yr actually after is just an integer column?  If so have you tried just changing the column type to see if it works?  The code that sets is this:
stmt.setTimestamp(7, new Timestamp(entry.createTime));

And I have a feeling if the target column type is a BIGINT it will probably just convert it.

If not the change you'd need to make would be something like:
stmt.setLong(7, entry.createTime / 1000);

This is a normal/expected exception.  It happens when psj tries to send a longpoll response but the client has silently dropped the connection.  In this case psj will recycle the work for the next LP connection in the queue.

The reason yr seeing it now and may not have before is that while psj-mm is in alpha I'm dumping a lot more events to the log so I can see better what's going on inside.  This particularly exception happens all the time with pre-mm version but isn't logged.

Detailed information about the risks of using paypal for bitcoin transaction can be found here

-ve luck could be caused by a pool accepting shares that shouldn't have been.  e.g. duplicates or if there is a delay between new block on the network and the pool detecting it and marking all new shares issued before that time as stale.

conversely +ve luck could be caused by pool rejecting shares that are valid (but still potentially submitting to bitcoin network).  e.g. pools share cache has expired the work so the share is counted as unknown.   Or blkmond detecting a new block before bitcoind is ready to serve work from the new block so longpolls go out with 'new' work from previous block but when they come back as a share it's rejected as stale due to prevblockhash not matching.

I haven't tried this but I have a suggestion.  Could you make the email sent to the customer include an image of a QR code.  There is at least one client (multibit) that allows you to simply drag a QR code image into a receiver box which automagically populates the receiver address and amount.  Now I think of it the payment page itself could display the code and bypass email altogether.  With properly designed page and a bit of ajax you could have a workflow like this:

checkout complete, QRCode displayed
user drags code to multibit and clicks 'send'
payment is sent and payment page wait for transaction
tx received, page does ajax update to indicate payment received.
user is happy

ok posted as .05.  You'll need the full .02 package then overlay the files in .05

No I haven't posted it up yet.  Give me a bit. I'll designate it .05 so look out for that one

should have checked before I posted.... it works! 

not yet... I'm waiting to prove it with an nmc block on testnet.

ok chill out for a minute Davinci...

Firstly you knew all along I didn't not have them means to fully test it.  A full test means mining a real block and it's well known by many including you that I only have 50MH at my disposal so there was never any chance that I could a release a fully tested version within 24 hrs.  It was at best a guess and hope that the I'd gotten every minor detail of the protocol right.  What I did release was a couple of thousand lines code with one wrong line (see below). That was 1/2 written over the previous few weeks and 1/2 written in a 12 hour coding marathon...

Secondly lets not forget that after this flurry of activity and urgency it took you several days before you even got around to testing it.  Unfortunately you seem to have a habit of coming online just before I go to bed so we haven't exactly had good communications.  You should probably be aware I've been working non-stop on this for the past 5 days.  You carry on as though I've abandoned the project whenever you can't get hold of me for a couple of days.  If I'm not answering on IRC it chances are I'm alseep or I've got head buried so deep in some code that I don't want to break my concentration.  3 other people have been helping me with various test/debug processes when I haven't been able to get a hold of you so you don't worry about idle hours...

I don't know why your miners haven't been able to connect... Other people have been able to.  I'm sorry about the message last night then immediately disappearing.  That was extenuating circumstance as some clown got drunk and drove into a power pole in my street and we lost power for several hours.

The good news is after rolling up sleeves and hacking at the namecoind code yesterday to dump some useful trace info and see what's going on from that end I think I've found the problem.  And as these things often turn out the be it was a single line of code.  I still need to prove it but all the tests I've done so far indicate it will work when a real nmc solution is submitted.

As to your offer, whilst I am a big fan of passive income your comment 'until it's sold to someone else' makes that a little vague.  It could be sold tomorrow for all I know.  So I will accept the other offer of the original bounty.  I feel this is reasonable considering 99% of the work was completed in the original timeframe and that it was impossible to complete the last 1% without outside assistance.