Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: konradp on March 27, 2014, 01:06:17 PM



Title: Understanding difficulty
Post by: konradp on March 27, 2014, 01:06:17 PM
Hi,

After reading https://en.bitcoin.it/wiki/Difficulty I understand that difficulty is max diff/current diff.

But I don't understand two things:

1. Why does this even exist? Why not use the target, a pool sends a target (lower than network target) and miners try to find a hash for it. I understand that difficulty is something easier to understand for humans, but why *software* works basing on difficulty?

2. Current difficulty is about 5 006 937 017, then why a pool I'm using sends "set_difficulty,240" (stratum), besides on many pools user can set a difficulty between 1 and a few hundred. What is this?



Title: Re: Understanding difficulty
Post by: Altoidnerd on March 27, 2014, 01:19:31 PM
1. Why does this even exist? Why not use the target, a pool sends a target

Because inverses and fractions make people happy.  

Now diff goes like ~ 1/target.

For long times, network_hashrate/diff ~ constant rather than the product network_harshrate*target.  Fraction good, product bad.

...
 then why a pool I'm using sends "set_difficulty,240" (stratum), besides on many pools user can set a difficulty between 1 and a few hundred. What is this?



Sharediff is different... let a true expert explain better but basically, if you have a super fast miner, you can ask it to only send shares that are twice as good, but only half as often...its a server optimization.

If my miner has a sharediff set to 10, I (should) get 10x the credit every time I submit a share.  The rules vary by pool.


Title: Re: Understanding difficulty
Post by: konradp on March 27, 2014, 01:33:50 PM
Thanks.

Quote
If my miner has a sharediff set to 10, I (should) get 10x the credit every time I submit a share.  The rules vary by pool.

OK, I get it, but the range (e.g. sharediff from 1 to e.g. 100) is up to the pool creator yes? Or is there maybe a consensus/well established rules about it?

Edit: or maybe there is no reason to set the maximum of it?


Title: Re: Understanding difficulty
Post by: Altoidnerd on March 27, 2014, 01:39:08 PM
The way the pools work is quite mysterious to me.  Slush invented pools, but he's a busy guy.  I think I know someone who will know the answer, I'll try to loop him in.


Title: Re: Understanding difficulty
Post by: litecoin_messiah on March 27, 2014, 01:47:07 PM
Thanks.

Quote
If my miner has a sharediff set to 10, I (should) get 10x the credit every time I submit a share.  The rules vary by pool.

OK, I get it, but the range (e.g. sharediff from 1 to e.g. 100) is up to the pool creator yes? Or is there maybe a consensus/well established rules about it?

Edit: or maybe there is no reason to set the maximum of it?


65536 is the average number of hashes to be computed in order to find a 1 share at mining difficulty 1 for a bitcoin based coin.

When the pool sets the share-diff to something higher, for example at share diff 10, your miner has to hash 10x more to find that share on average.

So 655360 hashes at share diff 10 and 65536 hashes at share diff 1.


Pools with vardiff or anything similar adjust the share diff to something like 512, because miners with 100's of khashs will have to make more connections to the stratum server at a lower mining difficulty resulting a higher risk of shares being lost. If they send shares at a higher difficulty they will be submitting work less often and have less stales.

Dunno why i'm explaining this or even if it was necessary, i haven't slept so excuse me  :-*


Title: Re: Understanding difficulty
Post by: konradp on March 27, 2014, 01:57:41 PM
Thanks.

Quote
If my miner has a sharediff set to 10, I (should) get 10x the credit every time I submit a share.  The rules vary by pool.

OK, I get it, but the range (e.g. sharediff from 1 to e.g. 100) is up to the pool creator yes? Or is there maybe a consensus/well established rules about it?

Edit: or maybe there is no reason to set the maximum of it?


65536 is the average number of hashes to be computed in order to find a 1 share at mining difficulty 1 for a bitcoin based coin.

When the pool sets the share-diff to something higher, for example at share diff 10, your miner has to hash 10x more to find that share on average.

So 655360 hashes at share diff 10 and 65536 hashes at share diff 1.


Pools with vardiff or anything similar adjust the share diff to something like 512, because miners with 100's of khashs will have to make more connections to the stratum server at a lower mining difficulty resulting a higher risk of shares being lost. If they send shares at a higher difficulty they will be submitting work less often and have less stales.

Dunno why i'm explaining this or even if it was necessary, i haven't slept so excuse me  :-*


Thanks, but now I'm completely lost :(

Why 65536? Why this *constant*? Doesn't it depend on current network difficulty?
And - why sending sending shares less often results in lower risk of shares being lost?


Title: Re: Understanding difficulty
Post by: litecoin_messiah on March 27, 2014, 02:10:14 PM
Thanks.

Quote
If my miner has a sharediff set to 10, I (should) get 10x the credit every time I submit a share.  The rules vary by pool.

OK, I get it, but the range (e.g. sharediff from 1 to e.g. 100) is up to the pool creator yes? Or is there maybe a consensus/well established rules about it?

Edit: or maybe there is no reason to set the maximum of it?


65536 is the average number of hashes to be computed in order to find a 1 share at mining difficulty 1 for a bitcoin based coin.

When the pool sets the share-diff to something higher, for example at share diff 10, your miner has to hash 10x more to find that share on average.

So 655360 hashes at share diff 10 and 65536 hashes at share diff 1.


Pools with vardiff or anything similar adjust the share diff to something like 512, because miners with 100's of khashs will have to make more connections to the stratum server at a lower mining difficulty resulting a higher risk of shares being lost. If they send shares at a higher difficulty they will be submitting work less often and have less stales.

Dunno why i'm explaining this or even if it was necessary, i haven't slept so excuse me  :-*


Thanks, but now I'm completely lost :(

Why 65536? Why this *constant*? Doesn't it depend on current network difficulty?
And - why sending sending shares less often results in lower risk of shares being lost?

Network latency and stuff like that cause disruptions in connections now and then.

dunno i'm very tired.


Title: Re: Understanding difficulty
Post by: konradp on March 27, 2014, 02:22:53 PM
Quote
dunno i'm very tired

But will you be back tomorrow to this question?  ;D


Title: Re: Understanding difficulty
Post by: Altoidnerd on March 27, 2014, 03:05:20 PM

Why 65536? Why this *constant*?

https://www.google.com/#q=2%5E16&safe=off (https://www.google.com/#q=2%5E16&safe=off)

If you haven't already, you should try mining.  That's the best way to learn.


Title: Re: Understanding difficulty
Post by: konradp on March 28, 2014, 07:24:54 AM
Thanks, but it doesn't really help me that 65536==2^16 ...

I am mining, but this doesn't help either.


Title: Re: Understanding difficulty
Post by: Altoidnerd on March 28, 2014, 07:51:46 AM
Thanks, but it doesn't really help me that 65536==2^16 ...

I am mining, but this doesn't help either.

https://en.bitcoin.it/wiki/Difficulty#What_network_hash_rate_results_in_a_given_difficulty.3F


Title: Re: Understanding difficulty
Post by: Jamestty on March 28, 2014, 07:52:49 AM
Thanks, but it doesn't really help me that 65536==2^16 ...

I am mining, but this doesn't help either.
I also mining, for I have not much use!


Title: Re: Understanding difficulty
Post by: Altoidnerd on March 28, 2014, 09:35:42 AM
The linked article explains the origin of the number. 

https://en.bitcoin.it/wiki/Difficulty#What_network_hash_rate_results_in_a_given_difficulty.3F


Title: Re: Understanding difficulty
Post by: Rannasha on March 28, 2014, 10:18:29 AM
Thanks.

Quote
If my miner has a sharediff set to 10, I (should) get 10x the credit every time I submit a share.  The rules vary by pool.

OK, I get it, but the range (e.g. sharediff from 1 to e.g. 100) is up to the pool creator yes? Or is there maybe a consensus/well established rules about it?

Edit: or maybe there is no reason to set the maximum of it?


Pools can pick whatever they want for the share difficulty, as that is something between the pool and the miner. The idea is to give the miner a job that is significantly less difficult than finding a Bitcoin block, so that it can do this job many times before the pool finds a block. The pool then keeps track of how often each miner has performed this job in order to determine payouts.

The higher the share-difficulty, the longer it takes between shares being found by the miner. This means that there is more variability in payouts. On the other hand, every time a share is found, the miner has to communicate with the pool and the pool has to verify the share. If the share difficulty is too low, fast miners will spam the pool-server with new shares, possibly overloading it. That's why some pools adjust the share-difficulty based on the speed of the miner. Fast miners have to generate higher difficulty shares than slow miners. Of course, the higher difficulty shares count more towards the payout in the end.

Not long ago, I wrote a not-too-technical blogpost about the what, why and how of pool-mining. It may help you understand it a bit better. See: http://bitcoin.rannasha.net/?p=29


Title: Re: Understanding difficulty
Post by: konradp on March 28, 2014, 03:31:56 PM
Quote
Pools can pick whatever they want for the share difficulty, as that is something between the pool and the miner

But there must be some kind of a standard, after all, miners must behave in the same way after receiving difficulty level.


Title: Re: Understanding difficulty
Post by: Rannasha on March 28, 2014, 04:01:07 PM
Quote
Pools can pick whatever they want for the share difficulty, as that is something between the pool and the miner

But there must be some kind of a standard, after all, miners must behave in the same way after receiving difficulty level.

Why?

The Bitcoin network difficulty changes regularly as well. If anything, miners should be able to handle any difficulty value. The difficulty doesn't change the computation that is performed, it only affects the chance whether the outcome of the computation is good enough. A miner repeats the same computation with slightly different input until it finds an outcome that is good enough and beats the target.


Title: Re: Understanding difficulty
Post by: Altoidnerd on March 28, 2014, 04:06:24 PM
Quote


But there must be some kind of a standard ..

The network cares if a pool finds a block.  The network doesn't care what a pool does internally.  A pool is free to use any method imaginable to produce blocks.  You are free to start mining blocks mentally if you wish.

Quote
after all, miners must behave in the same way after receiving difficulty level.

Forget the miner sharediff thing - there is a "standard" so that the pools agree on what the next difficulty shall be. Paraphrased, it is "readjust the target so that had the target been so for the previous the previous 2016 blocks, there would have been a 600 seconds block generation time.

If unsure, ask http://blockexplorer.com/q/nextretarget  http://blockexplorer.com/q/estimate

The notion of sharediff is a mining pools option for how they operate.  Slush's pool finds your optimal diff setting for you.  (You can prolly tell I am a Slush fan).


Title: Re: Understanding difficulty
Post by: cp1 on March 28, 2014, 04:10:40 PM
When a pool finds a block they need to divide the reward up between all the miners who helped.  So they need a way to track who helped and how much they helped.  Someone with 1GH/sec shouldn't get as much of a reward as someone with 1TH/sec.

They could have you submit each hash you calculate.  The 1GH/sec would then be sending 1 billion hashes to the pool per second, and the 1TH/sec miner would be sending 1 trillion hashes to the pool per second.  Each hash is 16 bytes, so this would be 16 terabytes of data per second sending back and forth to the pool.  Good luck with that.  Even if they only have you send the nonce, that's a crazy amount of traffic.

So instead you could just send "shares", which is a hash that meets the lowest difficulty (1).  Of course these aren't valid solutions, because the actual difficulty is much much higher than that, but they represent that you did some work, because it actually takes many hashes to luck out and hit a difficulty 1 solution.  So this cuts down on the traffic with the server.  If you have a fast miner, even this is too much traffic, so you can only send in difficulty 2 shares, difficulty 100 shares, etc.  As long as the pool knows you're only sending in these types of shares they can calculate how much hashing power you have, and what % of the reward you should get.


Title: Re: Understanding difficulty
Post by: Altoidnerd on March 28, 2014, 04:16:43 PM
When a pool finds a block they need to divide the reward up between all the miners who helped.  So they need a way to track who helped and how much they helped.  Someone with 1GH/sec shouldn't get as much of a reward as someone with 1TH/sec.

They could have you submit each hash you calculate.  The 1GH/sec would then be sending 1 billion hashes to the pool per second, and the 1TH/sec miner would be sending 1 trillion hashes to the pool per second.  Each hash is 16 bytes, so this would be 16 terabytes of data per second sending back and forth to the pool.  Good luck with that.  Even if they only have you send the nonce, that's a crazy amount of traffic.

So instead you could just send "shares", which is a hash that meets the lowest difficulty (1).  Of course these aren't valid solutions, because the actual difficulty is much much higher than that, but they represent that you did some work, because it actually takes many hashes to luck out and hit a difficulty 1 solution.  So this cuts down on the traffic with the server.  If you have a fast miner, even this is too much traffic, so you can only send in difficulty 2 shares, difficulty 100 shares, etc.  As long as the pool knows you're only sending in these types of shares they can calculate how much hashing power you have, and what % of the reward you should get.

This is a great explanation.  +1

What are the displayed fractions for difficulty of shares found (running for example bfgminer or cgminer)? 

"bla bla, Accepted, 4/1

bla bla, Accepted, 1.5k/1 ... etc"

It's some metric but what exactly?


Title: Re: Understanding difficulty
Post by: cp1 on March 28, 2014, 04:29:51 PM
4/1 means it found a difficulty 4 share, but it only need a difficulty 1 share.  It counts as a difficulty 1 share still when you submit it.


Title: Re: Understanding difficulty
Post by: Altoidnerd on March 28, 2014, 05:16:08 PM
4/1 means it found a difficulty 4 share, but it only need a difficulty 1 share.  It counts as a difficulty 1 share still when you submit it.

Does the mining software determine the sharediff of an Accepted share, or does the pool respond?

How is the fraction x/y arrived at for a given submission?  Is it a measure of the leading zeros?


Title: Re: Understanding difficulty
Post by: konradp on April 01, 2014, 06:16:12 AM
Thank you all.


Title: Re: Understanding difficulty
Post by: -ck on April 01, 2014, 06:21:17 AM
Share diff is chosen by the pool to find an optimal spread between bandwidth to the pool and variance to the miner. The choice of value chosen by the pool is arbitrary but most target diff to work out to somewhere around 20 shares submitted by the miner per minute. Difficulty of the share submitted is paid only based on the share difficulty asked for by the pool, though it can be much much higher (and ideally higher than the network difficulty in order to solve a block for the pool). Difficulty is internally determined in cgminer by checking the ratio of the share hash to diff1 where diff 1 is 26959535291011309493156476344723991336010898738574164086137773096960. Imagine every share's hash is just one massive number.

The 65536 constant mentioned earlier has nothing to do with bitcoin. Ignore it.


Title: Re: Understanding difficulty
Post by: whaSasd on April 01, 2014, 03:02:38 PM
If difficulty grows by 20% do my earnings tank by 20%? Any reputable site that predicts difficulty? There are a few but they estimate different values.


Title: Re: Understanding difficulty
Post by: -ck on April 01, 2014, 10:04:04 PM
If difficulty grows by 20% do my earnings tank by 20%? Any reputable site that predicts difficulty? There are a few but they estimate different values.
Yes and no respectively.


Title: Re: Understanding difficulty
Post by: tacotime on April 01, 2014, 10:43:22 PM
http://bitcoin.stackexchange.com/questions/2924/how-to-calculate-new-bits-value

is the explanation I like to give.

Bitcoin has difficulty 1 set as eight leading zeroes for a share (0x00000000FFFF0000000000000000000000000000000000000000000000000000 in expanded form).  Then you get the numerical difficulty we're familiar with if you divide this by the current target (nBits) in expanded form.

Difficulty and nBits (target) are interconvertible; note that pools often use 256-bit expanded form: 0x00000000FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF being diff 1.


Title: Re: Understanding difficulty
Post by: -ck on April 02, 2014, 12:23:50 AM
Difficulty and nBits (target) are interconvertible; note that pools often use 256-bit expanded form: 0x00000000FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF being diff 1.
This was only true of the getwork mining era. All stratum pools (should) use true diff 1 as their base now.


Title: Re: Understanding difficulty
Post by: DeathAndTaxes on April 02, 2014, 05:39:16 AM
65536 is the average number of hashes to be computed in order to find a 1 share at mining difficulty 1 for a bitcoin based coin.

Not sure where you got 65536 from but it isn't correct.  It takes 2^32 hashes on average to find one which meets the target for difficulty 1. 


Title: Re: Understanding difficulty
Post by: -ck on April 02, 2014, 10:26:30 AM
65536 is the average number of hashes to be computed in order to find a 1 share at mining difficulty 1 for a bitcoin based coin.

Not sure where you got 65536 from but it isn't correct.  It takes 2^32 hashes on average to find one which meets the target for difficulty 1. 
Look at his name, that will give you the clue where the 65536 comes from. It's an scrypt mining quirk.