The only thing that scares me with my bitcoins is when it becomes impossible to cash out meanwhile nobody (merchants) accepts bitcoin as payment. But I doubt it would happen because the silk road will always be there to accept bitcoins.
Yes that would be scary, all your hard earned btc unable to cash out, just the thought of it scares me. My greatest fear would still be being hacked. That would break my heart when all my btc would be just gone.