https://bitcointalk.org/index.php?topic=1937549.0

Sorry wasn't quite sure what the best forum was to get this seen by the people that might be interested.

Hi All,

I've been absent a while... Some might remember me. I wrote PoolServerJ which was the engine behind many of the early mining pools including BTC Guild.  I left this bitcoiny world rather suddenly several years ago due some rather dramatic personal circumstances which I never explained publicly. At the time I had big plans for a complete rewrite of PSJ. Eliminating huge quantities of code that existed for the sake of extreme optimisation required to offset inefficiencies of the basic rpc getwork model of mining. Optimisation that became largely irellevant after internal work generation was implemented in PSJ. I had grand plans of pushing a new mining protocol to deal with the inefficiencies of getwork. I never got that far... A year after I disappeared the stratum protocol was born to fill that exact need and PSJ became obsolete.

After a long time out of the crypto scene I've had my interest (obsession?) reignited by a few well meaning folk reminding what it once meant to me... So I'm looking for the right place to apply my skills and make a contribution again... Writing pool server engines is what i did best so it's the obvious place to start... PoolServerJ is dead. 80% of it's code is no longer relevant. Mega optimised code is also no longer a requirement for pools since many of the bottlenecks that existed back then no longer do. So I'm in no way tied to java like I was once. There are several good mining pool frameworks around just waiting for improvement. Although I'm more python friendly than I am to nodejs I'm quite liking node-stratum-pool as a starting point. Although I'm not particularly bound to any language nor any preference over native code vs scripted or JIT based.  But I figure if I've written a pool from scratch I've earned the right to extend on another pool engine authors work

So my question to pool operators is simple to ask though perhaps not so simple to answer... What are the features that you want or need from a pool engine that you haven't got today?

I'm not promising any magic new pool engine. Just exploring ideas to see what's needed and work out if/how I could deliver it....

MineZ.zone

Features:

• PPLNT
• zero fee for 1st month then 1% - first 100 miners get lifetime 0% fees
• DDoS Protected
• Pool URL : http://minez.zone/

Please, give us a try. We'll aim to provide an agile and efficient mining pool.  Our previous experience at the nasty end of pool engine development means you've got people who know mining, pools and crypto currencies inside and out looking after your infrastructure.  In time we hope to grow into a (crypto) household name.

To connect for ZenCash:

Username: your zencash wallet address
Password: anything
URL (difficulty 0.05): stratum+tcp://ny1.minez.zone:3033

Please see our "Getting Started" page (http://minez.zone/getting_started) for details and pre-configured downloads of miners tested and known to work with ZenCash.

I want to purchase some goods from a US merchant.  The good are not illegal but they are on the paypal shitlist so he can't use them.  His current options are western union, moneygram etc...

If I'm going to pay $20 for a WU money order and deal with the delay of sending it to him and him having to cash it, I'd much rather pay a premium to cover his currency risk and pay in bitcoins.  But it seems a good opportunity to upsell him in accepting bitcoins as a normal payment method.

Now this guy has never heard of bitcoins and probably has little interest in them beyond the ability to accept foreign payments quickly.  So what I imagine he would want from a solution would something like this:

He provides a bitcoin address for payment, the solution calculates bitcoin value based on exchange rate + small risk premium, once bitcoin is received it's immediately put on market and converted to USD.

Is there a merchant solution that does all this?  Particularly the last part as his key interest will be fast cross border payments into USD.

[Please do this!]

I'm pretty excited about this release.  It contains some very significant new features and improves overall performance by several 100%.  The merged mining branch of poolserverj has been in alpha for some time and with the help and testing from several pool ops we've been able to stabilise it and iron out most of the MM specific glitches.

The WorkMaker feature represents a fundamental shift the way poolserverj operates.  The rpc bottleneck is gone forever so your bitcoin daemons can have a little rest.

I'll go through the major features one by one and full changelog is at the end of this post.  But first...

Please do this!

The config options have changed significantly.   I highly recommend you start fresh with the sample properties file and transfer your settings over.  Trying to do it the other way is going to be a very error prone process.  I also recommend you take the time to read the comments in properties file in detail.  They are the defacto poolserverj documentation.

Please also make sure you read the 'Default Donation' section of this post.  This can be disabled but please be aware that it's there by default.

Recommended minor patch

I highly recommend making this very small patch to you daemons.  This simply prevents your debug.log being spammed.

In the file rpc.cpp or bitcoinrpc.cpp search for this line (there may be extra strMethod's in there so search for "ThreadRPCServer method="):

Code:

if (strMethod != "getwork")
        printf("ThreadRPCServer method=%s\n", strMethod.c_str());

and change it to:

Code:

if (strMethod != "getwork" && strMethod != "getworkaux" && strMethod != "getauxblock"
               && strMethod != "buildmerkletree" && strMethod != "getblocknumber" && strMethod != "getmemorypool")
       printf("ThreadRPCServer method=%s\n", strMethod.c_str());

 
Now onto the new features...

Merged Mining Support

Poolserverj now has a complete native merged mining implementation.  It handles all the functions of merged-mining-proxy internally and performs all the additional functions that previously required a merged-mining version of bitcoind.   This means that requirements for merged minings are much simpler:

    * No merged-mining-proxy required
    * A stock version of bitcoind that includes the getmemorypool patch (bitcoin 0.5.0 includes this)
    * namecoind merged mining version.
    * Optionally if you want to take advantage of coinbasing on the namecoin chain then you can apply the getmemorypool patch to namecoind (which is very simpe to do)

There are a few gotchas with merged mining we've all discovered over the past few weeks which poolserverj handles:

Partial Stales

When a single chain (e.g. namecoin) finds a new block but the other chain doesn't it's possible for a share to be stale for one but not the other.  Poolserverj detects this and sets our_result=1 and reason=partial_stale.  Optionally you can also configure a BOOLEAN database column for each chain that will be marked with 1,0 so you can calculate share credits on a per chain basis.  This is particularly a problem with cgminer clients (fixed in the latest source code though) who do not respect longpoll unless prev_block_hash has changed.  This will only change when a bitcoin block is found so cgminer client can get partial-stales for the namecoin chain quite frequently.

Longpoll Passthrough

To address the double longpoll issue.  There is a fundamental design clash between longpolling and merged mining.  Basically it works like this:  When there is a double block change (i.e. btc and nmc solved by one solution) the daemons won't see the new block at exactly the same time.  From what I've seen a delay of around a 1-2 seconds is not unusual.  What happens then is that one chain advances to the next block.  PSJ checks the other chains to see if they've updated, sees they haven't so starts sending out LPs.  Before the miner receives the LP and establishes a new LP connection the second chain advances so PSJ starts sending out another batch of LP's.  Those miners who haven't got their new LP connection registered before the second LP miss out so continue working on the old block for as long as they would normally (probably about a minute). This patch addresses this by setting a longpoll passthru period.  Where two block changes happen within a specified period (10 seconds).  After the 2nd block change and until 10 seconds after the 1st block change has passed, any longpolls received will have a short expiry of 1 second after which they'll return new work to the worker.  This will give any slow miners a few seconds to get their longpoll in and learn that there's a 2nd new block to work on.  The reason for the 1 second delay is to prevent longpoll spam.  Most miners will immediately send another LP request as soon as they get a response.  With 0 delay this sets up an LP spam loop.  They will still spam 1/sec for up to 10 seconds which is why this is called a workaround rather than a fix.  There's really no way to fix this issue properly except to ditch either longpolling or merged mining alltogether.

Experimental SCrypt Chain Support

Currently only litecoin is supported but additional chains can be added relatively easily.  All that's required to add new chain support is to define the chain in the source and add a few constants.  This hasn't been tested in the field yet.

Database Fault Tolerance

PoolServerj is now highly tolerant to database failures.  Connections are retried if they fail.

Workers will continue to be served from the cache in this case and given the default cache expiry of 60 minutes (unless you explicitly flush the worker) this means the only worker impact is that changes to the worker from the front end will not be propagated and new workers will not be able to connect until the DB connection is restored.

Shares will be serialized to disk in batches when the DB is not available.  When it comes back online any shares on disk will then start being sent to the database.  Shares are stored in batches as separate files so it is perfectly feasible to take these files and give them to a different poolserverj server to upload.

The end result is your database can go offline for a significant period and poolserverj should happily carry on working with no data loss.  The only impact will be that workers that have not connected for more than 1 hour will not be able to authenticate.

I have tested this by taking mysql down for an hour with a stress test client submitting about 400 shares /sec.  When the database was brought back online it took only a couple of minutes to flush all shares to the database.

WorkMaker

Note that all the variations of WorkMaker and Coinbasing have been tested and proven to work on testnet for both bitcoin and namecoin testnet.

My biggest bugbear with merged mining was the additional overhead it put onto the server to keep track of everything.  But... what merged mining taketh away WorkMaker giveth back 10 fold

WorkMaker is internal work generation.  No more getwork rpc calls to bitcoin daemons which means you can use a stock standard version of bitcoind (as long it's a version that supports the getmemorypool rpc call).

Aside from enabling coinbasing functionality (which I'll discuss below) it offers huge performance benefits.

You may ask "aren't you just moving the CPU load from bitcoind to poolserverj?".  Partially... There are two major ways this is a win though:

   1. You eliminate RPC/network latency overhead which is significant for what should be a microsecond operation.
   2. The bitcoin implementation uses a very inefficient algorithm for generating work.  The majority of the CPU load comes from hashing.  The default implementation requires ~ 2 * nTransactions hashes to generate a work.  The poolserverj implementation requires log2(nTransactions).  For an average block with 50 transactions this means 100 hashes vs 6.  For a large block with say 200 transactions this means 400 hashes vs 8.

Performance

Of course I did some benchmarking to prove the point so here's the numbers...

Raw generation tested by altering poolserverj to consume the work internally so that as soon as it's generated it will try to generate another one...

    * 0.3.0 with JK patched bitcoind daemon: ~2000 works/sec
    * WorkMaker: 24000 works/sec

Frontside getwork capacity - using stress test client with 50 concurrent thread continuously issuing getwork requests.  This measures the throughput including RPC overhead:

    * 0.3.0 with JK patched bitcoind daemon: ~1000 works/sec
    * WorkMaker: ~4000 works/sec

The highest frontside getwork rate I've seen in a production environment with 0.3.0 was on one of BTC Guild's servers: 4500 works/sec so it's probably reasonable to guess that this server would be capable of ~15000/sec.

This is only the first iteration and there numerous ways this can be further optimised which will happen in the future.

Coinbasing

So aside from performance what else does workmaker do for you?  Because it generates the coinbase transaction internally (similar to luke-jr's coinbaser patch) we have a few options to play with.

Firstly you set the payout address in the properties file.  This does not have be associated with the bitcoind you are connected to.  It could be an offline secure wallet address if you want.  Or if you run multiple instances of poolserverj on different servers you can ensure all coinbase rewards go to a single wallet regardless of which server generated them.

Coinbase message string: There is an option to set a short coinbase message string.  I have hardcoded this to be limited to 20 bytes as I don't want to encourage spam in the blockchain.  You may want to use some sort of pool identifier or even a private UID.

Coinbasing can also work on namecoin or other aux chains but it requires the getmemorypool patch to be applied.  This is a very simple patch to apply, even I was able to do it first go and I'm biggest numpty around when it comes to c++.

Coinbase Donations

It is now possible to set an automatic donation to any address in the coinbase transaction.  This can be calculated using 4 different methods:

   1. an absolute value in bitcoins (or fractions)
   2. a percentage of total block reward
   3. a percentage of total block reward excluding transaction fees
   4. a percentage of transaction fees only

Note that I said ANY address.  If you are donating and you use open source software from other developers in your pool please consider sending them a tip as well.  You can set as many donation targets as you like.

Donations will work on aux chains as well if you set the chain to localCoinbasing=true but you MUST have the getmemorypool patch applied to the aux chain daemon.

Why did I put this feature in?

As many of you know I don't run a pool and I don't even own a mining rig.  Donations for development work are my sole source of coins.  This is good for poolserverj users because I am not distracted by the day to day stuff of running a pool and I can concentrate on developing the software.  In full time hours I could probably measure the time I've spent on poolserverj in months.  I never really expected donations but when a few started coming in it was rather nice.  And I noticed it gave me a lot more motivation to keep improving the code.  This is a simple no hassle way you can help keep me and other open source developers interested and motivated.

If you choose not to use this feature I have no problem with that.  There's many reasons people may not (0 fee pools for example) and I'm not going to give any preferential support based on whether people do or don't donate, in fact there no real way I can tell where donations are coming from that I can see.  I think the record has shown I've always been happy in the past to give support and advice without the expectation of donations.

Default Donation

The sample properties file is setup with a default donation.  You can remove this simply by commenting out those lines.  I realise this may be controversial, the reason I've chosen to do it this way is simply so that people have to make a conscious choice to NOT donate.  This feature obviously interests me more than other people and I'm sure that if it wasn't the default option many people that may have been happy to donate would not simply because it never crosses their mind to look at the feature.  This way everyone who uses it has to stop and think about it for a moment.

The first time you start this version of poolserverj you will be prompted with a warning that this default donation exists.  If the file 'donation.ack' exists in the tmp directory you will not see the prompt and poolserverj will start normally.

If anyone genuinely feels they missed the message and donated unwittingly contact me with the blocks involved and as long as I can verify the blocks belong to your pool I'll be happy to send the coins back to the same address that the main coinbase reward was paid to.

 
Future Development

I'll be maintaining the 0.3.x branch for a short time until the 0.4.x branch is considered stable at which time it will be merged.

Non-merged mining is quite possible with 0.4.0 but it hasn't been extensively tested.

A couple of the next major developments I plan to work on include:

    * Enable poolserverj to listen on multiple ports
    * Binary work protocol to replace frontside RPC.  This can reduce bandwidth usage by 85% and will remove the need for longpolling and all it's associated problems.

Full Changelog

From 0.3.0.FINAL to now:

[0.4.0rc1 WorkMaker]

Major Features:

Full merged mining support including longpoll for all aux chains.
Support for SCrypt chains (litecoin, tbx, fbx)
WorkMaker internal work generation (more than 10x faster than rpc with JK patched daemons)
Coinbasing to any payout address both for parent chain and aux chains (aux chain daemon must have getmemorypool patch applied to use this feature)
Donations via coinbase transaction.  PLEASE READ THE SAMPLE CONFIG AS THERE IS A DEFAULT DONATION WHICH YOU CAN REMOVE.
DB fault tolerance, shares serialized to disk if DB connection is lost and sent to server when connection is reestablished.  This means along with Worker caching
you can switch your db off without losing any shares.  The only impact will be that new workers will not be able to authenticate and higher than
normal DB load when you switch it back on.

Detailed changes:

- added useragent as an optional column for share logging.
- added support for SCrypt as a proof of work hashing algorithm.
- Update to generate merged mining auxPoW internally.  This completely removes the dependency on bitcoind-mm version and allows us to use stock bitcoind which is nifty since it contains getmemorypool and we need that.  A point to note: hashes in the 2 merkle branches contained in auxpow are reversed compared to a transaction hash in the main block merkle tree.
- added db.connectionOptions property to allow users to add arbitrary connect paramenters to connection URL.
- Update to share logging to retry failed connections.  If the connection is still failed then shares are serialized to disk. The ShareLogger thread periodically checks for shares on the disk and resubmits them to the database.  This means shares can survive across a poolserverj restart if the DB is failed for that long.  THIS IS AN API BREAKING CHANGE.  Any db.engine.shareLogging plugins that inherit from DefaultPreparedStatementSharesDBFlushEngine will need to have their method signatures updated.
- Integrate local block generation for aux chains allowing coinbasing in the aux blocks
- cleaner thread now checks for dead longpoll connections, can only detect connections that have not been silently dropped by the miner.
- added timestamps to logging
- fix: missing block check interval property
- fix: with a single worksource the fetcher was blocking until a new block check was issued and forced all blocks to be marked in sync.
- block checker change sleep() to wait() so if can be woken up with notify
- Refactored all JsonRpc specific code out of WorkSource, ShareSubmitter and BlockChainTracker.  bitcoind side of server is now complete abstracted from protocol and transport.
- add case sensitive worker name option (caseSensitiveWorkerNames=false)
- add AnyWorkerFetchEngine that bypasses DB lookup and returns a worker with the requested name.  This is for pools that do no have miner accounts and use a payout address as username.  Password is set to empty String.
- add blackhole db share flush engine which does nothing.  A quick work around for disabling writing shares to db.
- hack to allow properties to return empty String  instead of null - use value '~#EMPTY_STRING#~'
- rebuild block tracker to handle multiple chains with different block numbers.
- adjust longpolling to account for additional chains.  LP now fires when any chain finds a new block but waits until a getblocknumber has come back from each chain first to try and prevent double LPs.  If a daemon is down it will timeout and fire the longpoll anyway after 1 second.
- allow subtractive traceTargets.  i.e. 'traceTargets=all,-merged' will show all traceTargets except 'merged'
- fixes for block sync cases where sync tracking loses the plot and never fires block change.
- support for partially stale shares.  Where a single chain has advanced to the next block old work may still be valid for the other blocks.  This patch checks that the work is valid for at least one block giving three possible outcomes.  accepted, accepted partial-stale, stale
- longpoll passthru.  To address the double longpoll issue.  There is a fundamental design clash between longpolling and merged mining.  Basically it works like this:  When there is a double block change (i.e. btc and nmc solved by one solution) the daemons won't see the new block at exactly the same time.  From what I've seen a delay of around a 1-2 seconds is not unusual.  What happens then is that one chain advances to the next block.  PSJ checks the other chains to see if they've updated, sees they haven't so starts sending out LPs.  Before the miner receives the LP and establishes a new LP connection the second chain advances so PSJ starts sending out another batch of LP's.  Those miners who haven't got their new LP connection registered in time miss out so continue working on the old block for as long as they would normally (probably about a minute). This patch addresses this by setting a longpoll passthru period.  Where two block changes happen within a specified period (10 seconds).  After the 2nd block change and until 10 seconds after the 1st block change has passed, any longpolls received will have a short expiry of 1 second after which they'll return new work to the worker.  This will give any slow miners a few seconds to get their longpoll in and learn that there's a 2nd new block to work on.  The reason for the 1 second delay is to prevent longpoll spam.  Most miners will immediately send another LP request as soon as they get a response.  With 0 delay this sets up an LP spam loop.  They will still spam 1/sec which is why this is called a workaround rather than a fix.  There's realy way to fix this issue except to ditch either longpolling or merged mining alltogether.
- added trace message to indicate longpoll passthru has started/stopped
- add user-agent to request meta-data for future logging to database
- fix for restoreWorkMap.  Now that workmaps are not cycled per block the blocknumber is no longer relevent when loading the map from disk.  Only work age matters and old works will be cleaned out on the next Cleaner cycle so we simply accept everything from the file into the workmap.
- when retrieving work from cache add validate check to ensure work is not expired and continue polling until one is found that is valid or cache is empty.
- add retry for worker db fetch.  If connection fails close and reopen connection then retry query before throwing an exception.
- add litecoin support
- extend fireBlockChange to run in lite mode when block hasn't changed but you want to trigger a cache flush and longpoll (e.g. if a high value transaction come in.)
- rebuild default share logger to use column mappings so all columns can now be optional.
- added components needed to trigger a longpoll when new transaction included in the block increase expected fees by more than a user defined threshold.
- enable workmaker for non-merged mining config
- added forced acknowledgement of default coinbase donations on first startup

[0.3.1]
- security hotfixes for two bugs that allowed duplicate shares to be submitted in specific circumstances.

If I'd realised how little had to be changed I would have done this weeks ago but that's how it goes...

The most recent version of psj poolserverj-0.3.0.07.MM.tar.gz now has scrypt support.  You'll have to remove all the merged mining config options from the properties file and set POWalgorithm=scrypt

Fair warning, this is completely untested as I don't have a scryptBrix/coin client.

Merged mining is only a few blocks away now and from what I can see the bitcoin side of the equation is grossly underprepared.  I hope I'm wrong but I don't think this is going to go well.  The problem as I see it is that the merged mining community has prepared itself but not given the bitcoin community the tools it needs to adapt.  Let me explain.

Documentation is woefully lacking.  Basically limited to a wiki page a some scattered discussion threads on the namecoin forum.  Miners don't need to do anything but bitcoin pools that wish to adopt merged mining need to implement the spec one way or another.  Currently the only way to do this is to use vinced's patched bitcoind and namecoind along with the python merged-mining-proxy.  This proxy has to sit between the poolserver (e.g. pushpool, poolserverj or custom) and the bitcoind.  This represents both a potential bottleneck and an extra point of failure and to my knowledge has never been tested on a load greater than about 50GH.  The alternative is for pools to implement the merged mining spec themselves.  This is easier said than done.  The spec is NOT documented anywhere.  The limit of the documentation that I can find is this:

Parent blockchain (AKA bitcoind)

In order to verify that a client has done hashing on the namecoin blockchain it is nessecary to add a proof that the work has been done on the bitcoin blockchain to the namecoin blockchain. In order to make it possible to have this proof created by the bitcoin blockchain the bitcoind needs a patch. This patch makes it possible for the bitcoind to cryptograhically sign, that there there was work submitted to the namecoin blockchain as well.

Status: Patch ready for testing. Need to ask if it gets implemented to stock bitcoind
[edit] Auxiliary Blockchain (AKA namecoind)

The namecoind now needs to accept the cryptographically signed proof from the bitcoind.

Status: Patch ready for testing. Proposed for block 24000 (but still not agreed upon).
[edit] merged-mine-proxy

This is the daemon all miners connect to. The daemon itself knows how to connect to the bitcoind and the namecoind checking if shares are valid for one of its downstream blockchains. Furthermore it requests getworks from the parent (which must be patched, see above) and delivers it to the miners.

Status: Ready for testing. Implemented using python.

There is also an article in the bitcoin wiki by Mike Hearn that predates the MM implementation by many months but it at best it explains the general principles using quite a different example and none of the specifics.

For a protocol that involves talking to multiple chains, building merkle trees of block hashes in a barely defined order with extra fields that are defined but not actually used in the implementation, extracting merkle branches, parsing undocumented contructs of headers, parts of coinbase transactions and merkle branches and somehow using this to validate aux blocks where the parent block may not even exist... This documentation woefully inadequate, bordering on pathetic.

If someone wants to make an alternative implementation they have no choice but to pore over the source code in two different languages.  I've done this, it took me days to get a grip on it and if weren't for ArtForz's considerable help it would have taken a lot longer.  Even having done that there are a number of ambiguous issues that leave implementation decisions to be guessed at.  Luke-jr has submitted a partial solution but has also pointed out that it's partial because there is no documentation for the spec.  I have attempted to seek clarifications on the spec myself both on the namecoin forum and their IRC channels only to be told that only one person really knows how it works and they are hardly ever online.  The few answers I was able to get simply confirmed that source code can be relied upon as a guide only, there are details of the spec that are ambiguous in the reference implementation and cannot be definatively determined using it as the only reference.

So the bottom line is that every pool that wants to adopt merged mining currently has no choice but the use the untested merged-mining-proxy black box.  So the MM spec is not documented and frankly I think it's grossly irresponsible of the people pushing MM to have let it get this far without doing so well in advance of the cutover block. Without hours/days of poring over the code (in 2 different languages) it's essentially a black box. The provided solution creates a number of problems and the sensible thing to do would be for pools to simply refuse to touch it until that situation is addressed.  

Unfortunately merged mining has created a situation where adopting the change is very difficult to resist. Once one pool adopts it, it puts enormous pressure on the others to follow as the miners will flock to the pool that offers free extra Xcoins for the same mining effort. End result is we have a mass of pools struggling to cope with new xcoind patches and an unproven python point of potential failure/bottleneck and no one available to support it except the merged mining developers who seem to be online for about 10 mins/week.

At best the provided tools (merged-mining-proxy) should be considered a reference implementation of a spec that doesn't exist. I predict there will be grief for pools and miners over the next few weeks/months.  The best case outcome I can think of is that the extra NMCs mined tank the namecoin market and miners lose interest.

If there are problems and 1/2 pools roll back it still doesn't create any incentive for the MM crowd to step up and do the job properly because their motive is increase hash power on the NMC network and they will have acheived that whatever the consequences to bitcoin mining might be...

IMHO merged mining should be stopped until this situation is properly addressed.

I didn't have a high opinion of the solidcoin founder CoinHunter aka RealSolid after I read his propaganda piece here:
http://solidcoin.info/solidcoin-ready-for-bitcoin-collapse.php

But what I just read on IRC made my jaw drop.  It does need some context so I've pasted the whole conversation here:
http://pastebin.com/ywxakYP3

but to give a taste here are some snippets.  As I read it he's basically admitting that he knows there's no factual basis for some of the libelous claims in his article and in fact he hasn't explicitly made those claims but he clearly understands that most people will skim it and end up with a certain impression.  I certainly did.  I read this article once and did a double take. I asked myself 'did that seriously just claim that Gavin Andreson is in league with the FBI?'  On carefully rereading it I noted that it didn't explicitly say that but IMHO it was carefully crafted to leave that impression... The IRC conversation confirms this:

<noagendamarket> bron libertarian news said it not solidcoin
<bron> lol?  solidcoin has plenty of positives; look at the uptake of the difficulty adjustment algorithm alone; it's already been copied by 2 chains
<bron> noagendamarket: no, solidcoin said it:  http://solidcoin.info/solidcoin-ready-for-bitcoin-collapse.php
<noagendamarket> no they quoted from a news article
<RealSolid> i only said "implications"
<noagendamarket> http://www.libertariannews.org/2011/08/30/bitcoin-fbi-admits-to-engaging-in-infiltration-disruption-and-dismantling-of-competing-currencies/
<RealSolid> its not my fault people cant read
<bron> noagendamarket: the news article deals with fake physical US currency.  counterfeits.
<RealSolid> i took advantage of the fact humans prefer to get the TONE of the article instead of reading
<bron> noagendamarket: *coin is not counterfeit US currency.

<bron> again, facts point to indictments against those who proilferate fake US currency.  *coin is not.
<RealSolid> who cares what the facts are

<RealSolid> i dont like bitcoin, so that article came out implicating bitcoin in the fbi
<RealSolid> so it fit with the tone i was going for

<bron> <bron> the article doesnt make the factual connection
<RealSolid> i dont have to prove beyond a reasonable doubt in an article to post something factual another site reported
<RealSolid> what dont you get about that
<RealSolid> i can say "another site says there is aliens on earth"
<RealSolid> and its true and factual
<RealSolid> thats exactly what i did there

<RealSolid> if you cant read english drbrontosaurus and are more affected by "tone"
<drbrontosaurus> and all it really says is you have ur bases covered for that scenario
<RealSolid> then you have played into my trap

<RealSolid> a lot of people just cant read well
<drbrontosaurus> and was throughly unimpressed
<drbrontosaurus> libertarians give a bad name to fuck-off-itarians like my self
<RealSolid> there are no lies on that page, but the tone of it certainly leaves the reader with an impression
<RealSolid> and that is what i wanted to achieve

</credibility>

I've just finished writing up the results of some extensive performance tests of poolserverj against pushpoold.

Write up can be found here along with
- test methodology
- test setup
- all the tools necessary to reproduce the tests in your own environment

For the TLDR crowd a quick summary:
- pushpoold beat poolserverj on one test that tested the raw rate it could deliver getworks in a low concurrency scenario (with no share submissions)
- In all the tests that use all functions of the server poolserverj outperformed by factors ranging from 200-400%
- on the test setup pushpoold's best performance for receiving shares equated to a 589 GHash/s pool
- on the test setup poolserverj's best performance for receiving shares equated to a 2154 GHash/s pool
- poolserverj has been reverted to alpha status due to some issue identified during these tests which are explained in the 0.2.2 changelog.txt

Feel free to rip holes in my testing methodology.  As the author of poolserverj I can never credibly claim lack of bias which is why I've provided all the test/benchmark tools for other people to do their own tests.

[Support for performance issues]

Support thread for PoolServerJ.  Ask you questions here.

In case you're wondering what PoolServerJ is http://forum.bitcoin.org/index.php?topic=33142.0

If you are using a bitcoind v0.3.24 or below ensure you have the 4diff patch. There is a known bug that will bring poolserverj to a grinding halt if it occurs.  It is fixed in this patch and it also contains some massive performance improvements that will benefit poolserverj.  See this post for an explanation of how it affects poolserverj and why the fact that it doesn't slow down pushpool is not necessarily a good thing.

Support for performance issues

If you're having performance issues (high CPU load, slow response to workers etc) your problem is probably fixable.  Poolserverj has been proven to handle very high loads in production environments with minimal cpu usage.  There are a number of things that can cause these symptoms though.

If you want me to help you troubleshoot these sort of issues please help me out by providing the following info:

• Poolserverj version
• A summary of your hardware specs.
• What patches are applied to your bitcoind (and if it's an alt chain daemon make sure you tell me that
• How you've got yr pool components splits among servers.  i.e. are poolserverj, bitcoind, frontend, database on the same server?  Multiple bitcoinds?  On which servers.  What type of database.
• A copy of your poolserverj properties file (Please remember to remove your passwords).
• A sample of your log output (error and stdout) with debug=true.  About 10 mins worth is good but try to include a block change in that.
  If your load is not particularly high then also set trace=true and traceTargets=all.  Note: the only data this should contain that might be considered sensitive is worker login names and possibly ip addresses (if trace=true).  If you're concerned about this then scrub the log file first but please use some sort of regex find/replace with a marker so I can see where these events actually happened.
• The output from running the script found in this post.  This will dump two 2sec snapshots of server stats to log directory. This should be from the same time window as the log sample.

If you can't find me on IRC then PM for an email address to send this to.

[Pushpoold Vs PoolServerJ performance test results now published]

Pushpoold Vs PoolServerJ performance test results now published

As a necessary component to a larger project I've had to write a complete pool server in java.  The rest of the project is still in progress but the pool server is ready to release into beta.  The key problem I needed to solve and part of the reason I rolled my own rather than just using pushpoold is scalability.  Pushpoold is great at what it does but the difference I needed was to get around the bitcoind bottleneck.  PoolServerJ solves that problem rather neatly by caching work and also allowing the pool operator to run as many bitcoin daemons as they want to and feed them all into the poolserver.

A few of the features of PoolServerJ:

    * Native merged mining support.  Will work with standard bitcoind 0.5
    * SCrypt blockchain support (e.g litecoin)
    * Customisable coinbase transaction (pay block rewards to any wallet perhaps an offline secure wallet)
    * Internal work generation (more than 10x faster than rpc getwork)
    * Work caching - caches work from bitcoind or internal WorkMaker to handle short term spikes in requests
    * Database resilience.  Server can keep running if database goes down and shares will be sent to database when connection is reestablished
    * DoS resilience with QoS support to ensure workers who've submitted valid work are serviced with priority
    * Capable of pulling work from multiple bitcoin daemons to get around the getwork bottleneck and also to provide some redundancy
    * Notify of block change via HTTP to a user configurable URL to support event triggered share processing
    * Cached database handling (optional) to reduce round trips.
    * Supports Mysql, Postgresql, sqlite3 (JDBC based so others can be added easily - currently only tested on mysql)
    * Runs as a Windows service
    * Longpolling support
    * Integrated block monitoring using all available bitcoin daemon work sources
    * Dump shares to Database, log file or stdout
    * Safe restart - shares issued to client are dumped to file on shutdown and reloaded on startup so your miners won't get stale shares if you restart a live server
    * Only dependency is a Java 6+ JDK.

Performance

Extensive performance testing against pushpoold has shown PoolServerJ capable of more than 4 times the capacity for share submits and 3 times the capacity for getwork requests.  This is the limit of what could tested on limited available hardware however it's architecture allows for significant scaling and theoretically these limits could be significantly exceeded.

update: Since the release of 0.4.0 WorkMaker Edition getwork capacity has improved by 4 times and share submit capacity by 2 times over the original performance test results.

Please give it a go, feel free to send me any feedback, rip into it if you like.

You can find it here

The webserver may be a bit slow, I'm testing out EC2 micros to see how they handle basic web server duties so sorry about that.  Will have a test pool up and running soon (as soon as I finish the front end). <-- update: this never happened and probably won't... I'm too busy working on the software and there's plenty of pools using it now so I've got enough guinea pigs

Not sure what the official channel for bug reporting is so here seems a good place.

I'm developing a pool server in java and during testing under load (max 5 concurrent requests, about 1 request / 30ms) the bitcoin server that's feeding it getwork requests keeps crashing with the following message:



followed by:



It's reproducible every time I run the same simple code that generates a lot of requests but usually stays up for about 1-2 mins of load before the crash.

My bitcoin.conf file is standard except for uncommenting the json-rpc user and password fields.  Bitcoin version is: 0.3.23-beta.  Downloaded as windows installer from main website.

1/ Why adjust difficulty so infrequently?

2/ Why diminishing rewards on such a staggered basis?

On point 1:  The 2 week cycle exposes the network to a number of attacks.  It's safety is based on the assumption that no one party could obtain near majority control easily but this is really not the case and won't be until BTC has grown an awful lot more.  A simple attack vector is to throw masses of power in at the beginning of a block, greatly accelerating the rate of generation and the difficulty on the next block.  As soon the difficulty adjustment happens withdraw all that processing power.  Rate of coin generation drops dramatically transaction processing slows down, miners start dropping out, less coins come on the market so prices go up and said owner of mass processing power can trickle out his coins from the last block to keep prices steady, towards the end of the difficulty cycle they can come back in and take an even bigger slice or wait until the next cycle and clean up on the lowered difficulty. .  Why not a shorter cycle?  In theory enough power could put the network in a stagnant mode for a long time until the next difficulty change. 

On point 2:  There are a lot of unproven assumptions about the incentive of transaction fees as block rewards drops.  No one really knows if they'll be enough to keep driving mining.  Let's be clear, the system doesn't just need miners, it needs a LOT of miners to maintain it's integrity and prevent the kind of manipulation outlined in point 1.  If you're going the change the reward why do it in such spaced out and dramatic fashion?  Once every year or two a 50% cut.  That's going to create shocks to the market every time it happens.  Why not a more graduated approach?  In fact why drop the reward at all?...  Even if the goal is a deflationary currency (and the jury is still way off calling whether that's better than a non-inflationary or deflationary currency) a constant block reward still achieves that goal.  Each block currently adds 50 BTC to the total money supply.  Each 50BTC is a smaller proportion of the total money supply than the last one. 

It's inevitable.  The widely held assumption is that the mining rate will stabilize around about the point where profits = energy costs + a nominal amount for new hardware investment.  But that assumes a level playing field.  Botnet operators will eventually build functionality into their user agents to tap into GPU's if they're available, when they do they'll have the power of a botnet behind them without power costs.  Botnets just became that much more profitable.

What are the implications of the block chain being under the effective control of botnet operators?  Well at least we can be confident they're unlikely to work together, competition in the black markets is probably more fierce than in legal markets.

I've read about all the payment methods and the apparent 30% gain that can be made by pool hopping between PPS pools but I just don't get it.  Where does the advantage come from?

I presume a 'round' is the number of blocks between the pool's last winning block and next winning block.  So for argument sake let's say we have 4 equal sized pools.  Average round for each pool consists of 1000 shares and I'm a miner who can generate 10% of total shares per round if I stick with one pool for the whole round.

for an average round I generate 50 shares with pool 1 and 50 with pool 2.  Pool1 wins so I get 50/1000 * 50 = 2.5btc, same for pool2
If I stick with Pool1 I get 5 btc but expect to win block only half as often.
For a large number number of 1000 round shares I expect to win on average 1/2 with each pool, say 100 rounds
50 * 2.5btc = 125btc
50 * 2.5btc = 125btc
total = 250btc

For a short round say 600 shares I generate 50 shares with pool1 and 10 with pool2
If pool 1 wins I get 50/600 * 50 = 4.16btc
If pool 2 wins I get 10/600 * 50 = 0.83btc
For a large number number of 600 round shares I expect to win on average 1/2 with each pool, say 100 rounds
50 * 4.16btc = 208btc
50 * 0.83btc = 41.5btc
total = ~250btc

For a long round say 2000 shares I generate 50 shares with pool1, 50 shares with pool2, another 50 shares with pool1, another 50 shares with pool2
If pool 1 wins I get 100/2000 * 50 = 2.5btc
If pool 2 wins I get 100/2000 * 50 = 2.5btc
For a large number number of 600 round shares I expect to win on average 1/2 with each pool, say 100 rounds
50 * 2.5btc = 125btc
50 * 2.5btc = 125btc
total = 250btc

What am I missing?

So what are the extra bytes?

I've tried feeding the getwork data field into the BitCoinJ Block class for parsing but I get rubbish results:

This is the JSON result:

Code:

{
    "error": null,
    "id": 2,
    "result": {
        "data": "000000013aef333a9788611a61cee16e0d555b989015425446f0776c00000478000000009d0221ae1fe47f5963e1b868dbe6388070888400ffcce91c7b60e4f157ed70f44e047b5d1a0c2a1200000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000",
        "hash1": "00000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000010000",
        "midstate": "b5160a8f1c6ade7fc3cac93d31f5f34a5d53faa48b04fa549126c7d6cd4fe791",
        "target": "0000000000000000000000000000000000000000000000122a0c000000000000"
    }
}

This is what I get from parsing it with BitCoinJ:

Code:

size: 128
difficulty: 304745498
nonce: 0
time: 1568343118
My system time: 1308916582
version: 16777216
hash: fa336f7e89b73b44ca1ec41b597c03f92c1e9e68a06789d10517ed1fe9cd9c38

The only thing that seems correct is the nonce.

I've tried reversing the bytes but still rubbish.  Should I be offsetting perhaps?

Building a multi source proxy in java, I haven't really got to the guts of verifying blockheaders yet but it's important to the design of the project to know if it matters.

i.e. say I'm taking getworks from multiple sources, when that getwork is solved and ready to be returned does it matter if I return it to a different bitcoin server or is the request somehow keyed to the bitcoin server that generated it?

Hacker steals x thousand BTC.
Hacker tries to withdraw and hits limit $1k USD limit.
Hacker starts dumping BTC to crash price.
Number of bitcoins he can withdraw equivalent to under $1k USD increases dramatically.
Hacker withdraws all remaining BTC.

Easy question, which JSON-RPC spec does bitcoin use?

I've searched a lot but not really found an explanation of what it is or how it works.

I'm keen to get to work on a java implementation of a pool server.  It sounds to me like it's some sort of persistent HTTP connection that allows the server to push a message back to the client sometime after the getwork response has been sent to inform them that the block is finished and they need to get some fresh work to do. Is that the basic idea?

If so is there any doco or spec for it so I can make sure my pool server implements it correctly?