Price drives difficulty

[organofcorti]

After several discussions on this board and others, I decided to actually analyse "Price drives difficulty" and determine if there really is a significant correlation between the two, and not just a post-bubble correlation either, but one that is valid for the whole MTGOX trading history.

2. Conclusions

• "Price" does indeed "drive difficulty",  and an analysis of correlation coefficients tells us that the "price" is the log of the volume weighted average USDBTC price from two difficulty periods previously, and difficulty is actually the log of D.
• More simply, log(D, lag=2) = log(BTCUSD vwap)*a + b
• A simple linear model shows a clear relationship between the two, but has no significant predictive value.
• A more complex model is slightly better at predicting D, but is still not useful as a lone accurate predictor of D, and also suffers from the possibility of "overfitting".
• It is likely that other contributors to D make a simple D predictor based on price impossible. However a simple predictor as developed above maybe useful in predicting a general direction in the change of difficulty and the strength of the change (barring a very volatile market).
• None of this will be valid in approximately three weeks, when after block number 210000 the block reward halves to 25 btc, and be even less valid with the onset of ASIC hashpower. However the idea behind the analysis stands and may be used once the dust settles.

Details are here:

http://organofcorti.blogspot.com/2012/11/101-price-drives-difficulty.html

[bcpokey]

Interesting model, I will look it over more when I'm not running out the door. I think it's most interesting that your model vs. real world has a tendency for the modelling to over-estimate large swings as events occur. Perhaps a dampening term would make it even more impressively accurate. Makes sense too as miners are reluctant to switch off even in the face of sour events, as well as being hesitant to make huge risks en masse simply because things are going well for a while.

Kudos however for the work.

[bitboyben]

Some body send this guy some BTC!

[organofcorti]

Interesting model, I will look it over more when I'm not running out the door. I think it's most interesting that your model vs. real world has a tendency for the modelling to over-estimate large swings as events occur. Perhaps a dampening term would make it even more impressively accurate. Makes sense too as miners are reluctant to switch off even in the face of sour events, as well as being hesitant to make huge risks en masse simply because things are going well for a while.

Kudos however for the work.

I wrote something similar in the blog. I think miners respond less well to market volatility and mining inertia tends to dampen big swings in hashrate - when things are going well there's still a time cost involved in purchasing new equipment, during which a miner might cancel the order, and when USDBTC is dropping miners, as you say, are reluctant to turn off.

I introduced a dampening term of -log(BTCUSD^3*const) to the third model which reduced the swings a little. I can certainly reduce them further, but since I'm not sure how to model miner sentiment I don't really have anywhere to start. Without a model for how and why miner sentiment might dampen the D response, any improvement in matching the curves could be overfitting. I did find a few models that match actual D quite well, but they probably are overfit and have little predictive value.

Some body send this guy some BTC!

I am organofcorti and I approve this message.

[Jack1Rip1BurnIt]

Awesome graph. Can't wait to see what you can make in the future after the upcoming changes have set in real good.

[Fcx35x10]

nice graph i'm watching this thread

[niko]

Ideally, we should be looking not at the difficulty, but at the total USD value of mining hardware, based on network hashrate and historical price for H/s. It's tricky to unmix the contributions from different technologies (FPGA vs GPU), but could be estimated. Ideally the value of electricity should be added too, but then we'll have to assume some ROI period that miners are comfortable with.
I expect the correlation between this "value" of mining hardware+electricity and the BTC exchange rate to be much stronger than difficulty/exchange rate.

[molecular]

nice!

in fact: really nice!

Now I have a place to point all the "we should mine more so price rises" idiocy. (Not that there was a lack before).

Can you make that image about 30-50% of it's original size?

[bitboyben]

Ideally, we should be looking not at the difficulty, but at the total USD value of mining hardware, based on network hashrate and historical price for H/s. It's tricky to unmix the contributions from different technologies (FPGA vs GPU), but could be estimated. Ideally the value of electricity should be added too, but then we'll have to assume some ROI period that miners are comfortable with.
I expect the correlation between this "value" of mining hardware+electricity and the BTC exchange rate to be much stronger than difficulty/exchange rate.

I thoroughly agree that this could be very useful. And what is going to happen with ASICs coming online is that there will be easy to follow stats to compare dollar value invested in hardware to total hash rate. We already have a good idea of where both will be once they get shipped out and we just need the shipments to happen to get an even more accurate idea.

As for miner profit sentiment there is a poll for that as well and may be useful to some degree.

[organofcorti]

Ideally, we should be looking not at the difficulty, but at the total USD value of mining hardware, based on network hashrate and historical price for H/s. It's tricky to unmix the contributions from different technologies (FPGA vs GPU), but could be estimated. Ideally the value of electricity should be added too, but then we'll have to assume some ROI period that miners are comfortable with.
I expect the correlation between this "value" of mining hardware+electricity and the BTC exchange rate to be much stronger than difficulty/exchange rate.

I think you missed two important part of this thread: the fact it's about predicting difficulty and the fact that difficulty can be modelled as occurring due to changes in price.

I think your post is implying that exchange rate might be a function of the total US$cost of mining. In that case difficulty would be a function of the total cost of mining. This may be true (and I'd be interested to see a thread on it) but it's not what i'm considering in the blog post. Also, I could be wrong but you seem to imply that difficulty could affect exchange rate?  I've tried to dispel the myth that difficulty affects exchange rate in any significant way so I hope I'm wrong.

[molecular]

Ideally, we should be looking not at the difficulty, but at the total USD value of mining hardware, based on network hashrate and historical price for H/s. It's tricky to unmix the contributions from different technologies (FPGA vs GPU), but could be estimated. Ideally the value of electricity should be added too, but then we'll have to assume some ROI period that miners are comfortable with.
I expect the correlation between this "value" of mining hardware+electricity and the BTC exchange rate to be much stronger than difficulty/exchange rate.

I think you missed two important part of this thread: the fact it's about predicting difficulty and the fact that difficulty can be modelled as occurring due to changes in price.

I think your post is implying that exchange rate might be a function of the total US$cost of mining. In that case difficulty would be a function of the total cost of mining. This may be true (and I'd be interested to see a thread on it) but it's not what i'm considering in the blog post. Also, I could be wrong but you seem to imply that difficulty could affect exchange rate?  I've tried to dispel the myth that difficulty affects exchange rate in any significant way so I hope I'm wrong.

before everything is repeated here. Here's a recent example of the issue being discussed. The OP was confused about the issue and asked wether there was a correlation. After some explaining he said

So at this point I can see that price and difficulty are in fact highly correlated, and changes in difficulty are primarily caused by changes in price.

EDIT: As far as I remember that thread also goes into possible factors that might constitute an influence from difficulty on the price, but I think it was argued that these effects were much weaker than the effect of price on difficulty.

[organofcorti]

before everything is repeated here. Here's a recent example of the issue being discussed. The OP was confused about the issue and asked wether there was a correlation. After some explaining he said

So at this point I can see that price and difficulty are in fact highly correlated, and changes in difficulty are primarily caused by changes in price.

EDIT: As far as I remember that thread also goes into possible factors that might constitute an influence from difficulty on the price, but I think it was argued that these effects were much weaker than the effect of price on difficulty.

Looks like I was late to the party - I missed that thread. Thanks for posting the link though molecular - lots of interesting points there.

[molecular]

Looks like I was late to the party - I missed that thread. Thanks for posting the link though molecular - lots of interesting points there.

Here's another relevant thread, an old one (maybe someone wants to necropost).

Not exactly about the topic, but goes into some detail about the correlation. It's about deciding wether it's more profitable to buy or mine:

https://bitcointalk.org/index.php?topic=7427.0

[organofcorti]

And another thread here: http://forum.bitcoin.org/?topic=4339.0

[axilla]

great work sir, you are a scholar and a gentlemen.

[bitboyben]

I thought his idea would be great to measure potential difficulty. I guess I was already of topic too.

[niko]

Ideally, we should be looking not at the difficulty, but at the total USD value of mining hardware, based on network hashrate and historical price for H/s. It's tricky to unmix the contributions from different technologies (FPGA vs GPU), but could be estimated. Ideally the value of electricity should be added too, but then we'll have to assume some ROI period that miners are comfortable with.
I expect the correlation between this "value" of mining hardware+electricity and the BTC exchange rate to be much stronger than difficulty/exchange rate.

I think you missed two important part of this thread: the fact it's about predicting difficulty and the fact that difficulty can be modelled as occurring due to changes in price.

I think your post is implying that exchange rate might be a function of the total US$cost of mining. In that case difficulty would be a function of the total cost of mining. This may be true (and I'd be interested to see a thread on it) but it's not what i'm considering in the blog post. Also, I could be wrong but you seem to imply that difficulty could affect exchange rate?  I've tried to dispel the myth that difficulty affects exchange rate in any significant way so I hope I'm wrong.

Sounds like you misunderstood my post, and I can't blame you as it was not very clear. After looking at all the data in several different ways, I arrived at the conclusion that, so far, the USD/BTC exchange rate has been driving the network hashrate. We all seem to agree on that. Mining appears more lucrative, more people start investing into mining.

The point I was trying to make is that, instead of looking at the network hashrate (or difficulty), we could look into total USD value of mining equipment. The USD price of a GHash/s has been evolving over time, going from CPUs to GPUs to FPGAs. Also, for each of these technologies, there were constant improvements of the efficiency of the mining software. All in all, we should try and correlate the USD/BTC exchange rate with the (network hashrate / price of a GHash/s worth of mining equipment).

[Beans]

The price of btc does not control difficulty. Profits drive difficulty, and it looks similar to the price because they are similar.

[molecular]

The price of btc does not control difficulty. Profits drive difficulty, and it looks similar to the price because they are similar.

meh. In the same manner you could say: Profits don't drive difficulty, the submission of valid block to the network does.

[bcpokey]

Ideally, we should be looking not at the difficulty, but at the total USD value of mining hardware, based on network hashrate and historical price for H/s. It's tricky to unmix the contributions from different technologies (FPGA vs GPU), but could be estimated. Ideally the value of electricity should be added too, but then we'll have to assume some ROI period that miners are comfortable with.
I expect the correlation between this "value" of mining hardware+electricity and the BTC exchange rate to be much stronger than difficulty/exchange rate.

I think you missed two important part of this thread: the fact it's about predicting difficulty and the fact that difficulty can be modelled as occurring due to changes in price.

I think your post is implying that exchange rate might be a function of the total US$cost of mining. In that case difficulty would be a function of the total cost of mining. This may be true (and I'd be interested to see a thread on it) but it's not what i'm considering in the blog post. Also, I could be wrong but you seem to imply that difficulty could affect exchange rate?  I've tried to dispel the myth that difficulty affects exchange rate in any significant way so I hope I'm wrong.

Sounds like you misunderstood my post, and I can't blame you as it was not very clear. After looking at all the data in several different ways, I arrived at the conclusion that, so far, the USD/BTC exchange rate has been driving the network hashrate. We all seem to agree on that. Mining appears more lucrative, more people start investing into mining.

The point I was trying to make is that, instead of looking at the network hashrate (or difficulty), we could look into total USD value of mining equipment. The USD price of a GHash/s has been evolving over time, going from CPUs to GPUs to FPGAs. Also, for each of these technologies, there were constant improvements of the efficiency of the mining software. All in all, we should try and correlate the USD/BTC exchange rate with the (network hashrate / price of a GHash/s worth of mining equipment).

Well let me ask you this, to what end exactly? Say I work out a strong correlative effect between the value of mining hardware on the market, and the price USBTC. Now I know how much mining hardware value is, what does that do for me?

The reason for most of these models (at least from my perspective, perhaps they do not agree) is that people are figuring the profitability under different scenarios for mining (and whether to buy in, or drop out). Knowing what the total hashrate will be does not help, except that from that I can calculate difficulty, via an additional step. So why not cut out the middle man and just model difficulty directly?