I think most people's greatest fear is the governments/established organizations' retaliation.
This was my main fear for a while until I read a post (I forget which one exactly) suggesting the following:
Assume that Bitcoin has the massive potential that we'd all hope for. Which government will want to be the first to try and implement some sort of ban? It could prove to be massively detrimental to that particular country in the event that Bitcoin survives (which it probably would). Whichever country implemented a ban would have deterred adoption amongst its own citizens and therefore missed out on the potentially massive growth of the Bitcoin economy as a whole.
I'd be interested to hear other peoples thoughts.
