Thoughts and questions on BTC Pools and merged mining

[simplecoin]

Do I need to recompile bitcoind?  Looks like probably yes.

Yes

Do I need to compile namecoind?  Is it a standard namecoind?  Yes and yes?  I dunno

its standard namecoind

The proxy script, what needs to be done with it, other than to make it executable and run it in place of bitcoind and namecoind?  
Can the proxy script even handle a high load or is it going to fail miserably when you dump 100 or 200 GHs at it?  

have a look at PoolServJ which replaces pushpool, lp and proxy

btw.
thats the reason i've started this thread. i want all pool owners to know how merged mining works - so its up to them to get it to work. and i hope some miners do read this and will prefer merged mining pools (like i do)

As this nears completion, it would be benificial for pool operators if there was:
A bitcoind diff for the merged client (since most of us run custom clients)
A merged-mining poolserverj binary
An easily modifiable namecoind source to apply joel catz fixes (or a premodified one).

[1713277950]

1713277950

[1713277950]

1713277950

[1713277950]

1713277950

[1713277950]

1713277950

[1713277950]

1713277950

[Luke-Jr]

More important than any of that is DOCUMENTATION. On the protocol, not the patches assuming I want to risk the main pool operation.

[simplecoin]

More important than any of that is DOCUMENTATION. On the protocol, not the patches assuming I want to risk the main pool operation.

Agreed.

Although, I would run the pool on a separate machine until I was 1000% sure it wouldn't interfere with the normal pool.

[Inaba]

For the big pools, poolserverj is not really an option.  We (not that I'm including myself in the big pools) have our getwork servers, either custom or pushpool highly modified to conform to our database and methods.  Switching over to poolserverj would be a major undertaking and rewrite of the poolserverj software.

Not something that couldn't be done, but it's not something that's going to happen very quickly.  It needs to work with current implementations to gain traction, not foisted off on yet another getwork server.  I don't relish the idea of :

a) Running ANYTHING java based, having to use Java based junk in the enterprise, I can tell you I have never once seen a java based program that was worth anything in high availability/heavy load environments.  Couple that with the fact that without some severe tweaks, the memory limits of the java VM leave a LOT to be desired.  I'll stick with native code, thanks.  That said, it's usually the interface that is complete FAIL with java crap, so maybe Poolserverj doesn't suffer from this, since there's no need for an interface to speak of.  But there are so many other problems with java scalability wise that it's just too unstable and problematic to build a business off of.
b) Rewriting yet MORE code to conform to my standards

[kano]

To put this discussion in the right place ...

The description above by DeathAndTaxes is completely clueless (you missed completely what "create hash" is)

Funny you say then then link to a document which says the same thing.

This is of course a guess - but it would appear that namecoin is putting extra information in the coinbase - i.e. increasing the bitcoin block size.
(I'm guessing this due to a discussion with luke-jr)

No need to guess.  That isn't how it works at all. Funny you "know" what I said was wrong yet you haven't even figured out how it works (still making wrong guesses).

Edit: Well I guess I now have the link to get the official code at least ... from DavinciJ15 : http://dot-bit.org/Merged_Mining
(can't remember where I got it last time I compiled and ran it)

Lets take a look at that.

"Merged mining works like this, you have two totally separate block chains, they are not related in any way nor does either contain any data from the other. When you mine you generate hashes that may be the solution to the current block, this is very very improbable per hash, its like a lottery where everyone generates tickets until someone finds the winning one. Normally you make tickets and check them against the Bitcoin block chain to see if they are the solution. With merged mining you create a ticket and check it against both the Bitcoin block chain and the Namecoin block chain, Bitcoin and Namecoin know nothing about each other, they are two totally different lotteries with different winning numbers, you just sent a copy of your ticket to both. Since you are sending the same ticket to two lotteries you increase your chances of winning one or the other. No Bitcoin data goes into Namecoin no Namecoin data into Bitcoin they remain totally separate, you simply run both the Namecoin and Bitcoin clients on the same machine and submit hashes to both networks, if your hash is the solution to the Namecoin block you get Namecoins if you hash is the solution to the Bitcoin block you get Bitcoins, its exactly like if you where mining on just one network, except you submit the same work twice."

Single hash checked against both bitcoin and namecoin targets.  Hmm sounds exactly like my "wrong" answer above.

I didn't link to that document, I referred to where I could gain the source.
That document is wrong.

As I said you completely missed what "create hash" is.
"create hash" is hashing the contents of the block that goes into the block chain and then checking if it is under the difficulty requirement.
If both are true, you then have a block.

Since you are hashing the contents of the block, you cannot even compare that to some other different block.
It's not possible.

What is possible, is to add extra data to the block, that some other block chain can check against and decide it will allow it to be a block.

The hash of the bitcoin block hashes the following information:
version
previous_hash
time
difficulty
nonce
the miner's extranonce
the transactions in the block (this includes the generation block which has the coinbase)

i.e. the whole block

The above information is mandatory for the block chain to be secure.
It ensures that no one can change the block chain somewhere in the middle.

Thus the bitcoin hash is the hash of the complete bitcoin block contents.
It is not the hash of ANYTHING more or less.

For a hash for namecoin to be securely usable in both bitcoin and namecoin, the bitcoin block would have to contain something extra added that namecoin can use.
If this were not true, then namecoin would have to rely completely on the contents of the bitcoin block chain to secure itself i.e. namecoin would become dependent on having a copy of the whole bitcoin block chain to be able work.

P.S. I still haven't read the namecoin code yet

[flower1024]

P.S. I still haven't read the namecoin code yet

please do that before posting about merged mining again.

the bitcoin block gets 33bytes namecoin data in his coinbase (without a bitcoin block size change).

there are several people who did run this on testnet (and you can do this right now).

[kano]

P.S. I still haven't read the namecoin code yet

please do that before posting about merged mining again.

the bitcoin block gets 33bytes namecoin data in his coinbase (without a bitcoin block size change).

there are several people who did run this on testnet (and you can do this right now).

You CANNOT say "gets 33bytes namecoin data in his coinbase" and "without a bitcoin block size change"

Seriously? You don't see the fallacy of that statement?

Are you telling me that the real bitcoin block chain is going to reject all merged mining bitcoin blocks and only namecoin will accept the namecoin blocks?

If you modify anything in the block (including the coinbase) after calculating the hash, but before putting the block into the real bitcoin block chain, the block will be rejected.

I know this not only by reading but by fact, due to having modified most of the different alternate coin software myself and adding a block into each block chain that says "Kano": I0coin block 29930, SolidCoin V1 block 33533, GeistG block 3676, IXcoin block 21128.
Did it for fun by reading the code and working out how the coinbase works, but more for the sake of understanding how things work.

You see, it has to work with the real bitcoin block chain - doesn't matter if you run a hacked bitcoin, the real bitcoin must accept the bitcoin blocks.

Thus your statement must be wrong (and again, the front page of that site must be wrong)

And Yes I will be reading the code when I have time to go through it (as I said) since no one knows or at least wants to admit what the code actually does (or makes statements that cannot be true)

[flower1024]

P.S. I still haven't read the namecoin code yet

please do that before posting about merged mining again.

the bitcoin block gets 33bytes namecoin data in his coinbase (without a bitcoin block size change).

there are several people who did run this on testnet (and you can do this right now).

You CANNOT say "gets 33bytes namecoin data in his coinbase" and "without a bitcoin block size change"

Seriously? You don't see the fallacy of that statement?

Are you telling me that the real bitcoin block chain is going to reject all merged mining bitcoin blocks and only namecoin will accept the namecoin blocks?

If you modify anything in the block (including the coinbase) after calculating the hash, but before putting the block into the real bitcoin block chain, the block will be rejected.

I know this not only by reading but by fact, due to having modified most of the different alternate coin software myself and adding a block into each block chain that says "Kano": I0coin block 29930, SolidCoin V1 block 33533, GeistG block 3676, IXcoin block 21128.
Did it for fun by reading the code and working out how the coinbase works, but more for the sake of understanding how things work.

You see, it has to work with the real bitcoin block chain - doesn't matter if you run a hacked bitcoin, the real bitcoin must accept the bitcoin blocks.

Thus your statement must be wrong (and again, the front page of that site must be wrong)

And Yes I will be reading the code when I have time to go through it (as I said) since no one knows or at least wants to admit what the code actually does (or makes statements that cannot be true)

it works with default bitcoin client and the default chain (its not an alternate chain). only reason you need a bitcoin patch is if you want to mine solo (or if you are a pool op)

it's the pool owner's bitcoin who decides what to place in the coinbase.

luke-jr (owner of eligius) did a test and placed prayers in it - without a block size change; without a modified client. is this impossible too?

[kano]

it works with default bitcoin client and the default chain (its not an alternate chain). only reason you need a bitcoin patch is if you want to mine solo (or if you are a pool op)

it's the pool owner's bitcoin who decides what to place in the coinbase.

luke-jr (owner of eligius) did a test and placed prayers in it - without a block size change; without a modified client. is this impossible too?

luke-jr did increase the block size.
You didn't understand that?
That was part of the complaint about what he was doing.
He is current just putting "Eligius" in there (and stopped with the prayers) and thus increasing each block size by 3 bytes
The normal coinbase is 8 bytes.

My blocks (that say Kano) in the other 4 chains have increase each single block that I got by 6 bytes.
Yes I changed the bitcoin code for my solo mining - how on earth else do you think that is possible?
It has nothing to do with the mining software.
(and also has nothing to do with normal pool software - only the server that creates the answers for the getwork requests)

[flower1024]

you may be right with that (i just rechecked the thread but did not found anything regard size). do you have a link for that?

anyway: i do not think 33 additional bytes per block is a problem.

and you just confirmed that its possible to set custom data (eg namecoin blockdata) inside the bitcoin block without breaking things for the default bitcoin client.

[kano]

you may be right with that (i just rechecked the thread but did not found anything regard size). do you have a link for that?

anyway: i do not think 33 additional bytes per block is a problem.

and you just confirmed that its possible to set custom data (eg namecoin blockdata) inside the bitcoin block without breaking things for the default bitcoin client.

Which is what I said after I realised how the two chains where being mined with the same hash - and I said that coz it is the only way to actually do it - and said that coz I KNOW it is the only way to actually do that.
I didn't think back in my first post in the cgminer thread that they were putting extra data in the bitcoin block chain ... why?
COZ EVERYONE KEEPS TELLING ME THAT THEY DON'T DO THAT.
However, they ARE putting extra data in the bitcoin block chain.
Thus the web page is wrong.
Again this is all a simple understanding of how bitcoin works - nothing to do with reading how namcoin merged mining works.
If namecoin is generating bitcoin blocks, those bitcoin blocks MUST adhere to the bitcoin block chain rules.

Any additional bytes per blocks are an issue since the size of the bitcoin block chain is already an issue.
... and if every block has these extra bytes (as the namecoin people would hope that everyone does merged mining) then it is even more of an issue.
Saying it is 33 bytes doesn't help. I'll look at the code and determine what it really is.

[Luke-Jr]

That was part of the complaint about what he was doing.

There was no legitimate complaints, only trolls.

[shads]

have a look at PoolServJ which replaces pushpool, lp and proxy

To clarify, poolserverj does not support merged mining natively yet.  It will work behind a merged-mining-proxy instance.  I've been getting my head around it all this last week and plan to make a native implementation tomorrow.  It should be ready by the changeover block unless there's a sudden upsurge in hashrate but it will be alpha.

[shads]

As this nears completion, it would be benificial for pool operators if there was:
A bitcoind diff for the merged client (since most of us run custom clients)
A merged-mining poolserverj binary
An easily modifiable namecoind source to apply joel catz fixes (or a premodified one).

pooserverj will only have one binary.  wether it uses merged mining or not will be configurable via properties.  There will be no additional overhead if not using merged mining.

Yes a JK 4diff for the mm version of bitcoind is needed however I don't think it's really necessary for namecoind or other aux daemons.  They do not deal with a torrent of rpc requests like the bitcoind does.  Just some aux block updates every few seconds and submission of valid shares which is also rare.

[shads]

For the big pools, poolserverj is not really an option.

Really?  Is that why the 2nd biggest pool (and the largest that uses pushpool) is changing over to it?  If you have concerns about it's capacity ask Eleuthria how many pushpool instances he replaced with a single poolserverj instance in a production stress test before it choked.

 We (not that I'm including myself in the big pools) have our getwork servers, either custom or pushpool highly modified to conform to our database and methods.  Switching over to poolserverj would be a major undertaking and rewrite of the poolserverj software.

Again suggest you ask Eleuthria how long it took him to make the necessary mods.  You don't even have to rebuild the binary, it's designed with plugin architecture so you can just override the commonly customised parts (like worker auth, db interaction etc) into a separate jar...

Not something that couldn't be done, but it's not something that's going to happen very quickly.  It needs to work with current implementations to gain traction, not foisted off on yet another getwork server.  I don't relish the idea of :

a) Running ANYTHING java based, having to use Java based junk in the enterprise, I can tell you I have never once seen a java based program that was worth anything in high availability/heavy load environments.

Leaving aside the obvious java hate here (I've learned it's not worth having that argument).  poolserverj was designed from the ground up for large pools.  pushpool was not.  That's not an indictment on the pushpool author, it's simply a consequence of the fact that the network was a lot smaller when pushpool was designed than when poolserverj was.   

  Couple that with the fact that without some severe tweaks, the memory limits of the java VM leave a LOT to be desired.

umm... it's ONE command line switch... -Xmx128m or whatever you want to limit the heap size to.    psj does not use memcached, it keeps all it's caches internal so of course the process is going to use more than pushpool.  It also uses the general principle of preferring performance over limiting memory footprint where there's a choice.  That's a design choice because it's meant to handle high loads.

[shads]

Saying it is 33 bytes doesn't help. I'll look at the code and determine what it really is.

It's the merkle root of all the aux chains blockheaders + number of aux chains.

[[Tycho]]

Running ANYTHING java based, having to use Java based junk in the enterprise, I can tell you I have never once seen a java based program that was worth anything in high availability/heavy load environments.  Couple that with the fact that without some severe tweaks, the memory limits of the java VM leave a LOT to be desired.  I'll stick with native code, thanks.  That said, it's usually the interface that is complete FAIL with java crap
...
But there are so many other problems with java scalability wise that it's just too unstable and problematic to build a business off of

Of course I'm not using poolserverj, and I like native code too, but you are a bit wrong in this part :)

[Inaba]

Really?  Is that why the 2nd biggest pool (and the largest that uses pushpool) is changing over to it?  If you have concerns about it's capacity ask Eleuthria how many pushpool instances he replaced with a single poolserverj instance in a production stress test before it choked.

So he's "changing over to it" and has not actually changed over to it?  See my next comment in further regards to this:

Again suggest you ask Eleuthria how long it took him to make the necessary mods.  You don't even have to rebuild the binary, it's designed with plugin architecture so you can just override the commonly customised parts (like worker auth, db interaction etc) into a separate jar...

In my case, and I'm not speaking for other pool authors, I do a lot of work on the getwork server that most other pools do on their backend.  This is mostly because it's exceptionally inefficient to try to do a scored pool on the back end.  BTC Guild is a proportional pool, so it's not really a surprise that it's substantially easier to switch a proportional pushpool over to another bit of software, since there's literally no changes *needed* in pushpool.  I know Eleuthria has done some modifications in Pushpool, but from speaking to him (granted, this was a couple months ago) his changes are not anywhere near as extensive as mine.

Leaving aside the obvious java hate here (I've learned it's not worth having that argument).  poolserverj was designed from the ground up for large pools.  pushpool was not.  That's not an indictment on the pushpool author, it's simply a consequence of the fact that the network was a lot smaller when pushpool was designed than when poolserverj was.  

My java "hate" comes from countless experiences with Java based products in high capacity enterprise environments (Specifically in the wireless and banking industries).  I have yet to run across a single piece of software that is java based that is a) stable, b) not a resource hog (while still offer sub-par performance) and c) expandable.

I will admit that I have not tried anything with 64 bit Java, since most of the enterprise stuff that's running java based junk is 32 bit, but I'm not talking about 128m of memory limits.. I'm talking about high capacity applications that needs > 8 GB.  This likely doesn't apply to psj, though, so it's probably a moot point. I am just illustrating the limitations of java and why it's not a good choice for applications that are expected to grow or need stable, low latency operation.

Again, I'm not saying psj suffers from any of this, but I've yet to see a java application that doesn't suck and/or have some idiotic limitation because of the exceptionally craptactular JVM.  When a program crashes, I just love having to suss out whether it was the JVM that had a problem or the actual program itself... yay, double my debugging work!

Regardless, I'll take a look at it and see what would be required to convert my setup over to it.  I just have visions of going through that work and then finding myself (once again) up against an nigh impenetrable wall of asinine java design architecture.

[kano]

Having played with pushpool and understanding the best simple optimisations to it, I'll have to have a look at poolserverJ now for my home test network.

Java - one of the best oo languages around ... In My Opinion.

Used Java on and off for many years.
Memory limitations are the only issues I've had that are in any way a true issue.
All other issues are simply just bad programmers.
Now if memory is an issue ... try compiling your Java code instead of 'bytecoding' it

My favourite argument in favour of Java is based on most people's lack of understanding of programming
Java code is expected to be faster than most other code.
Why? Simple to understand reason: optimisation.
You write a program today, compile it then release it.
Result: today's optimisation is the fastest it will get.
Java: release is the bytecode that is compiled at runtime, thus as optimisation gets better over time so does you Java bytecode's performance.

Code stability is the programmers fault and no more.
Running out of memory? Well I gotta say, if you have absolutely no idea of how much memory your program requires then you need to understand your code better ...
Debugging is probably the easiest language of all due to tracebacks being available with way more information than most common languages and easy to see/read

[urstroyer]

Yes a JK 4diff for the mm version of bitcoind is needed however I don't think it's really necessary for namecoind or other aux daemons.  They do not deal with a torrent of rpc requests like the bitcoind does.  Just some aux block updates every few seconds and submission of valid shares which is also rare.

I would really interested in a JK 4diff patch for vinced mm bitcoind version in order to test the poolserverj implementation. Is there any solution out there?