There is a lot of good discussion back and forth on this thread and some good points too.
To my thinking there is just one main hurdle above all others:
Legal - Bernard von Nothaus was convicted of counterfeiting US money because he issued pure silver dollars that superficially 'looked' like US circulated coins. (How you can counterfeit something that's worth more than the real thing is beyond me.) What it boiled down to was competition from private currencies against the US Dollar from within the US. Once there is a large enough user base OR some random criminal is caught in some Bitcoin sting then you will see the media criticism ramp up and the attacks will begin against Bitcoin.
The current administration in Washington has made repeated statements, quite recently, that the President needs the ability to "shutdown" the internet. The President also has wide emergency powers. A President could be persuaded, perhaps by Treasury or Fed officials, that this Bitcoin is a threat to the economy. The President could declare it illegal and direct ISP's to filter, block and report all Bitcoin traffic.
Granted, this is an extreme scenario -- these are extreme times however. The fact remains: until we have a legal, competitive currency structure any virtual currency will face hurdles to mainstream acceptance.
Do you think bitcoin being illegal would destroy it entirely? People do plenty of illegal things and there are plenty of people with the need for bitcoin. I guess it would hinder its acceptance into mass use though...