I think difficulty do affect price. People will try many difference approach to get BTC. If the difficulty is too high, then it is not very practical to setup mining rigs, those who want to invest in BTC will simply buy them, this will lift the price of BTC. So it is balanced by itself
Hmm, let's run this through a couple scenarios for this. (I'm just kludging them together, feel free to shoot at the principles, but the specifics are made up to create outsized issues that my brain can get a handle on)
Environment:ASIC is out and humming, technology is stable and no preorders are on the horizon that will change things.
Bitcoin mining with current equipment over 12 months results in a ~100% profit according to common current projections.
50% of mining income is used to buy new hardware to keep up with other miners.
Event: Bitcoin loses 50% value
Result: Miners are in survival mode, those who cannot afford the outlay have to put equipment purchases on hold, and bleed devalued bitcoins paying for electricity. difficulty slows or drops as some gear is deemed too unprofitable to afford to run and is powered down.
Event: Bitcoin gains 100% value
Result: Miners are in bonanza mode, profit taking and large hardware purchases will be popular, difficulty will shoot up with this new gear until some equilibrium is reached.
Event: Protocol change renders 50% of mining equipment useless (I know it's out there, but I could not think of another big forcing)
Result: 50% of miners are dancing in the street as their BTC profits go up 100%, other 50% want to kill them. Some of the excess profits will go towards new hardware, as will some investment by the failed miners replacing obsolete hardware, some will drop out entirely. Price may be effected by 20-minute blocktimes for up to 2 weeks, but it would only qualify as a panic, not a factually based reaction. After the On the whole the same amount of BTC is generated, and sold to others who want to buy or are selling hardware.
Event: New Big Fish 1 and New Big Fish 2 start fighting over BitCoin, increasing difficulty by 100% over 3 months
Result: Massive increase in difficulty because of governments or megacorps doing speculative mining with millions of dollars in custom fabbed hardware. Tinfoil hats hold a party, others are annoyed that "The Man" is making his own bitcoins instead of buying ours. Value of Bitcoin shoots up due to difficulty conflict, but only because of the perceived legitimacy, not because the difficulty is high. Many miners shut down inefficient capacity, but some keep some running for ideological reasons.
That's my take on those 4 scenarios, in my scorebook it's pretty obvious that 1 and 2 would happen, and it demonstrates the implied causation between price and difficulty. 3 and 4 are a bit more of a stretch, but in each case I found that the root cause of the price and the difficulty change was largely the same, with the differences in all 4 scenarios being transitory until mining capacity matches price, not the other way around.
I think you are right that there is a short term force applied on the price if there is a large difficulty swing, but I think it's noise compared with the actual event, or the phase of the moon
http://deepblue.lib.umich.edu/bitstream/2027.42/36301/2/b2092645.0001.001.pdf.