[XPM] [ANN] Primecoin High Performance | HP14 released!

[Lyddite]

FYI primecoin.org home page now has a link to the hp client download. Hope the miner icon I add is likeable

I think this helps new users to find it easier. Once again let's give Mikael a big round of applause for his hard work, dedication and selfless contribution!

I like it!

Any reason to not only use the hp client? Even if not mining?

[cabin]

Looking at the code, unless there is a bug in the algorithm that is say rejecting 10.5 length chains at the moment, it *should* behave linearly. Diff 9.999 should be same as 10, but but from the charts it is obviously not.

I think the distribution of the Fermat's remainder is skewed. Sunny mentioned that briefly in his paper too. The skewed distribution is probably caused by the repeated exponentiation done during the Fermat's test.

I wonder if this is because all the current miners are optimized to sieve for 9 chains at the moment? I wonder if the the difficulty would look more linear if miners switch to sieving for 10 chains now that most 9 chains won't solve a block. I would love for a -sievelength param to be added to the build that would sieve for chains of a set length. I would do it myself and experiment but I am having some difficulty building the project on windows, and I suspect it is at least as hard as bitcoin to build.

I can build the ypool miner easily and from those experiments the sieve length is quite effective at optimizing for chains of a desired length.

I'm a bit doubtful that switching the target difficulty to 10 would be beneficial at this point. It might become beneficial when difficulty is around 9.9 but I haven't really investigated that.

Also it's good to remember that optimal mining parameters are probably not the same for ypool and solo mining. Ypool miners are concerned with maximizing the value of their shares. Previously that system was exploited using shorter chains. Now that the system was changed radically it could be possible to exploit it with longer chains.

The building procedure for the stock Primecoin client should be the same as for Bitcoin. For my version you also need libgmp on top of that.

Ah ok, if the distribution is heavily skewed, and the number of miners was increasing rapidly, that could explain some of it. Well, if the parameter is added I would try it out now even if it is too early and see what happens.

I did eventually get it to build on mingw64 with the gmp 5.1.0 source. For fun I collected stats on the decimal part of the distribution of all 5+ chains, separated into 1% bins. It looks fairly flat:

Code:

66 54 53 55 52 56 44 70 68 53 53 57 54 33 53 46 61 68 66 67 55 69 59 58 39 60 73 58 51 61 56 56 53 56 66 50 72 76 58 57 55 55 61 55 62 67 53 58 46 44 59 60 53 62 39 46 47 55 55 48 69 77 47 56 75 52 65 74 41 49 67 53 74 47 49 63 55 57 47 61 59 63 45 59 54 55 61 62 48 61 55 62 63 55 70 62 50 81 55 64

Would I be correct in saying the chance to find a prime in a set that is sieved is about 0.095 and the chance of finding a prime just randomly is  1/ ln(2^255) = 0.00568 ?

[mikaelh]

Looking at the code, unless there is a bug in the algorithm that is say rejecting 10.5 length chains at the moment, it *should* behave linearly. Diff 9.999 should be same as 10, but but from the charts it is obviously not.

I think the distribution of the Fermat's remainder is skewed. Sunny mentioned that briefly in his paper too. The skewed distribution is probably caused by the repeated exponentiation done during the Fermat's test.

I wonder if this is because all the current miners are optimized to sieve for 9 chains at the moment? I wonder if the the difficulty would look more linear if miners switch to sieving for 10 chains now that most 9 chains won't solve a block. I would love for a -sievelength param to be added to the build that would sieve for chains of a set length. I would do it myself and experiment but I am having some difficulty building the project on windows, and I suspect it is at least as hard as bitcoin to build.

I can build the ypool miner easily and from those experiments the sieve length is quite effective at optimizing for chains of a desired length.

I'm a bit doubtful that switching the target difficulty to 10 would be beneficial at this point. It might become beneficial when difficulty is around 9.9 but I haven't really investigated that.

Also it's good to remember that optimal mining parameters are probably not the same for ypool and solo mining. Ypool miners are concerned with maximizing the value of their shares. Previously that system was exploited using shorter chains. Now that the system was changed radically it could be possible to exploit it with longer chains.

The building procedure for the stock Primecoin client should be the same as for Bitcoin. For my version you also need libgmp on top of that.

Ah ok, if the distribution is heavily skewed, and the number of miners was increasing rapidly, that could explain some of it. Well, if the parameter is added I would try it out now even if it is too early and see what happens.

I did eventually get it to build on mingw64 with the gmp 5.1.0 source. For fun I collected stats on the decimal part of the distribution of all 5+ chains, separated into 1% bins. It looks fairly flat:

Code:

66 54 53 55 52 56 44 70 68 53 53 57 54 33 53 46 61 68 66 67 55 69 59 58 39 60 73 58 51 61 56 56 53 56 66 50 72 76 58 57 55 55 61 55 62 67 53 58 46 44 59 60 53 62 39 46 47 55 55 48 69 77 47 56 75 52 65 74 41 49 67 53 74 47 49 63 55 57 47 61 59 63 45 59 54 55 61 62 48 61 55 62 63 55 70 62 50 81 55 64

Would I be correct in saying the chance to find a prime in a set that is sieved is about 0.095 and the chance of finding a prime just randomly is  1/ ln(2^255) = 0.00568 ?

Yeah, I actually did some testing myself too and observed similar results. The distribution of the remainder is indeed not (significantly) skewed. So I was wrong there. I originally thought that the difficulty curve looked logarithmic which led to me guessing that the distribution would be skewed. I looked at the difficulty curve again and it's not logarithmic. The amount of chains that pass the Fermat's remainder test is simply inversely proportional to the decimal part of the difficulty. That means 20% of 9-chains pass at difficulty 9.8 and only 10% pass at difficulty 9.9.

The probability of finding a prime around 2^255 is indeed about 1 / ln(2^255). Sieving improves the odds significantly. According to my miner's statistics, about 4.3% of candidates are primes on mainnet (7% on testnet).

And SlyWax's earlier formula block rate estimate based on difficulty was indeed correct. I was wrong there too. But please remember that chains/day is not an accurate estimate of actual performance.

[cabin]

Looking at the code, unless there is a bug in the algorithm that is say rejecting 10.5 length chains at the moment, it *should* behave linearly. Diff 9.999 should be same as 10, but but from the charts it is obviously not.

I think the distribution of the Fermat's remainder is skewed. Sunny mentioned that briefly in his paper too. The skewed distribution is probably caused by the repeated exponentiation done during the Fermat's test.

I wonder if this is because all the current miners are optimized to sieve for 9 chains at the moment? I wonder if the the difficulty would look more linear if miners switch to sieving for 10 chains now that most 9 chains won't solve a block. I would love for a -sievelength param to be added to the build that would sieve for chains of a set length. I would do it myself and experiment but I am having some difficulty building the project on windows, and I suspect it is at least as hard as bitcoin to build.

I can build the ypool miner easily and from those experiments the sieve length is quite effective at optimizing for chains of a desired length.

I'm a bit doubtful that switching the target difficulty to 10 would be beneficial at this point. It might become beneficial when difficulty is around 9.9 but I haven't really investigated that.

Also it's good to remember that optimal mining parameters are probably not the same for ypool and solo mining. Ypool miners are concerned with maximizing the value of their shares. Previously that system was exploited using shorter chains. Now that the system was changed radically it could be possible to exploit it with longer chains.

The building procedure for the stock Primecoin client should be the same as for Bitcoin. For my version you also need libgmp on top of that.

Ah ok, if the distribution is heavily skewed, and the number of miners was increasing rapidly, that could explain some of it. Well, if the parameter is added I would try it out now even if it is too early and see what happens.

I did eventually get it to build on mingw64 with the gmp 5.1.0 source. For fun I collected stats on the decimal part of the distribution of all 5+ chains, separated into 1% bins. It looks fairly flat:

Code:

66 54 53 55 52 56 44 70 68 53 53 57 54 33 53 46 61 68 66 67 55 69 59 58 39 60 73 58 51 61 56 56 53 56 66 50 72 76 58 57 55 55 61 55 62 67 53 58 46 44 59 60 53 62 39 46 47 55 55 48 69 77 47 56 75 52 65 74 41 49 67 53 74 47 49 63 55 57 47 61 59 63 45 59 54 55 61 62 48 61 55 62 63 55 70 62 50 81 55 64

Would I be correct in saying the chance to find a prime in a set that is sieved is about 0.095 and the chance of finding a prime just randomly is  1/ ln(2^255) = 0.00568 ?

Yeah, I actually did some testing myself too and observed similar results. The distribution of the remainder is indeed not (significantly) skewed. So I was wrong there. I originally thought that the difficulty curve looked logarithmic which led to me guessing that the distribution would be skewed. I looked at the difficulty curve again and it's not logarithmic. The amount of chains that pass the Fermat's remainder test is simply inversely proportional to the decimal part of the difficulty. That means 20% of 9-chains pass at difficulty 9.8 and only 10% pass at difficulty 9.9.

The probability of finding a prime around 2^255 is indeed about 1 / ln(2^255). Sieving improves the odds significantly. According to my miner's statistics, about 4.3% of candidates are primes on mainnet (7% on testnet).

And SlyWax's earlier formula block rate estimate based on difficulty was indeed correct. I was wrong there too. But please remember that chains/day is not an accurate estimate of actual performance.

Ah ok, so EstimateCandidatePrimeProbability() returns around 9%, but in reality it is more like 4% when you actually divide primes found by the number of tests. Interesting. My calculations also showed it would be a terrible idea to start search for 10 chains already.

Edit: actually the number of tests isn't a good stat for this.. it is the number of chains tested and is incremented even when the sieve is empty and no prime tests are done. I ran some numbers on real tests at each chain length, and it looks like it will always the same, around 0.094 of candidates are prime. I'll run it for longer though to get better results for the longer chains.

0.09442  0.094065  0.094624  0.094806  0.048345  0.099099  0.18182  0  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND

This would also make a good block estimator:
double nineFraction = (currently around 0.17)
double blocksPerDay = 24*(d7ChainsPerHour*0.094^2*nineFraction + d7ChainsPerHour*0.094^3);

[wibtc]

hp10 ubuntu x64 build has the following error everyday. Anybody has the same problem?

primecoind: checkqueue.h:171: CCheckQueueControl<T>::CCheckQueueControl(CCheckQueue<T>*) [with T = CScriptCheck]: Assertion `pqueue->nTotal == pqueue->nIdle' failed.

[mhps]

This would also make a good block estimator:
double nineFraction = (currently around 0.17)
double blocksPerDay = 24*(d7ChainsPerHour*0.094^2*nineFraction + d7ChainsPerHour*0.094^3);

Perhaps either ignore the high order terms (the method I use) or sum up to infinity which gives in a cleaner form of 24*d7ChainsPerHour*0.094^2/(1-nineFraction)

However my observation is that slower machines nest an unproportional less number of chains that are longer than 7.

[SlyWax]

hp10 ubuntu x64 build has the following error everyday. Anybody has the same problem?

primecoind: checkqueue.h:171: CCheckQueueControl<T>::CCheckQueueControl(CCheckQueue<T>*) [with T = CScriptCheck]: Assertion `pqueue->nTotal == pqueue->nIdle' failed.

Yes same error, same OS.

[patapato]

For fun I collected stats on the decimal part of the distribution of all 5+ chains, separated into 1% bins. It looks fairly flat:

Code:

66 54 53 55 52 56 44 70 68 53 53 57 54 33 53 46 61 68 66 67 55 69 59 58 39 60 73 58 51 61 56 56 53 56 66 50 72 76 58 57 55 55 61 55 62 67 53 58 46 44 59 60 53 62 39 46 47 55 55 48 69 77 47 56 75 52 65 74 41 49 67 53 74 47 49 63 55 57 47 61 59 63 45 59 54 55 61 62 48 61 55 62 63 55 70 62 50 81 55 64

mean is 57.54
median is 56
standard deviation (sigma) is 8.94
skewness is -25.13 = third moment / sigma^3 = sum( ((sample - mean)/sigma)^3 ) / total samples
non-parametric skewness is 0.17 = (mean - median) / sigma

I don't know if it is much or few skewness for our problem, just I hope that it could help to think.

[mikaelh]

Ah ok, so EstimateCandidatePrimeProbability() returns around 9%, but in reality it is more like 4% when you actually divide primes found by the number of tests. Interesting. My calculations also showed it would be a terrible idea to start search for 10 chains already.

Edit: actually the number of tests isn't a good stat for this.. it is the number of chains tested and is incremented even when the sieve is empty and no prime tests are done. I ran some numbers on real tests at each chain length, and it looks like it will always the same, around 0.094 of candidates are prime. I'll run it for longer though to get better results for the longer chains.

0.09442  0.094065  0.094624  0.094806  0.048345  0.099099  0.18182  0  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND  -1.#IND

This would also make a good block estimator:
double nineFraction = (currently around 0.17)
double blocksPerDay = 24*(d7ChainsPerHour*0.094^2*nineFraction + d7ChainsPerHour*0.094^3);

Good work. There is indeed a small bug with the number of tests. It's not very significant since every sieve should contain at least dozens of candidates.

I tried skipping the BiTwin candidates and then I'm getting the following stats on testnet:

Code:

2013-08-31 10:25:06 primemeter  51090434 prime/h 514555696 test/h 4380 5-chains/h 11031.065686 chain/d

So the probability is about 0.0992. The estimated probability is also around 0.1 which matches with the stats. The theoretical probability goes down slightly for the subsequent numbers in a chain.

[cabin]

So the probability is about 0.0992. The estimated probability is also around 0.1 which matches with the stats. The theoretical probability goes down slightly for the subsequent numbers in a chain.

That all makes sense. But I do see one weird thing.. even after 10 hours, the probability of finding the 5th prime in a chain remains abnormally low. For now I'll assume it is a bug in my code.. but it is weird.

chains 1 to 10:

Code:

0.094342  0.094123  0.093681  0.093306  0.055778  0.079268  0.15385  0  -1.#IND  -1.#IND

the code that generates this is around line 650 here:
https://github.com/rsweny/primecoin/blob/master/src/prime.cpp

I suspect there are thread issues with this but I'm not sure why it would only skew the 5's.

[mikaelh]

So the probability is about 0.0992. The estimated probability is also around 0.1 which matches with the stats. The theoretical probability goes down slightly for the subsequent numbers in a chain.

That all makes sense. But I do see one weird thing.. even after 10 hours, the probability of finding the 5th prime in a chain remains abnormally low. For now I'll assume it is a bug in my code.. but it is weird.

chains 1 to 10:

Code:

0.094342  0.094123  0.093681  0.093306  0.055778  0.079268  0.15385  0  -1.#IND  -1.#IND

the code that generates this is around line 650 here:
https://github.com/rsweny/primecoin/blob/master/src/prime.cpp

I suspect there are thread issues with this but I'm not sure why it would only skew the 5's.

I'm guessing you're seeing the effect of BiTwin chains in your fifth bin. The test for the tenth prime in a BiTwin will succeed with a much lower probability. The tenth prime of a BiTwin chain corresponds to the fifth prime of a 2CC chain. And that's why you are seeing a lower rate of primes in the fifth bin.

[Krusher33]

hp10 ubuntu x64 build has the following error everyday. Anybody has the same problem?

primecoind: checkqueue.h:171: CCheckQueueControl<T>::CCheckQueueControl(CCheckQueue<T>*) [with T = CScriptCheck]: Assertion `pqueue->nTotal == pqueue->nIdle' failed.

I'm using Ubuntu 13.04 64-bit and haven't seen this yet.

[ReCat]

Well, I've had a primecoin client running on my personally-owned rackserver that I have in a datacenter(In washington), and it's been a proper 24 hours now. The client hasn't synchronized. It's Win Server 08R2, as well as a precompiled build.

Do I need to manually add some peers, or something? Although I also noticed that it was giving that error that everyone else here reports. So I restarted it and it seems to be doing better now. (few hours to sync)

Also, I port forwarded the port 9911 for PrimeCoin. Is there anything else I can do to make my client be of more use to the network? I heard something about super nodes but I'm not sure how that works.

Cheers.

[mechs]

Well, I've had a primecoin client running on my personally-owned rackserver that I have in a datacenter(In washington), and it's been a proper 24 hours now. The client hasn't synchronized. It's Win Server 08R2, as well as a precompiled build.

Do I need to manually add some peers, or something? Although I also noticed that it was giving that error that everyone else here reports. So I restarted it and it seems to be doing better now. (few hours to sync)

Also, I port forwarded the port 9911 for PrimeCoin. Is there anything else I can do to make my client be of more use to the network? I heard something about super nodes but I'm not sure how that works.

Cheers.

The Linux versions seem to be more stable.

[crendore]

The Linux versions seem to be more stable.

+1 don't know why people even bother with windows when it comes to rack mounted servers.  Shows a true level of ignorance of what a server actually does.  "oh well i use windows on my desktop, so i better use windows on my server". lol.

[mechs]

The Linux versions seem to be more stable.

+1 don't know why people even bother with windows when it comes to rack mounted servers.  Shows a true level of ignorance of what a server actually does.  "oh well i use windows on my desktop, so i better use windows on my server". lol.

I think their may be a place for a Windows server for certain applications such as web hosting.  But I agree, Linux is usually superior

[ReCat]

The Linux versions seem to be more stable.

+1 don't know why people even bother with windows when it comes to rack mounted servers.  Shows a true level of ignorance of what a server actually does.  "oh well i use windows on my desktop, so i better use windows on my server". lol.

We are running Windows Server 2008 R2 because we need to run some legacy Windows Server applications. There is a large amount of software that is windows-only. If it were up to me, I would have chosen linux.

Anyways. That was a one-off bug. Right after writing the post, client synced fully, and I am now mining properly.

[Sy]

So - how are all the amazon and vps user doing? Still profitable?

[Trillium]

So - how are all the amazon and vps user doing? Still profitable?

Not likely at current exchange rates and difficulty..

[cryptrol]

After the gpu miner apparent failure, it seems that difficulty is kinda stagnated.

Code:

"blocks" : 147833, "difficulty" : 9.85310471,
"blocks" : 147834, "difficulty" : 9.85307074,
"blocks" : 147837, "difficulty" : 9.85310775,
"blocks" : 147838, "difficulty" : 9.85310000,
"blocks" : 147839, "difficulty" : 9.85312480,
"blocks" : 147841, "difficulty" : 9.85307729,
"blocks" : 147843, "difficulty" : 9.85294372,

Even getting lower.