If I just trade one crypto for another, do I have to worry about taxes? or do I just worry when trading back to Fiat? I'm a US citizen if that helps.
Generally the taxable event occurs when you close a position. So if you are selling bitcoins it won't matter if the exhange is to dollars or another virtual currency, you have a gain (or loss) on the original position.
Disclaimer: I'm not a tax accountant.
Some more info:
-
http://en.bitcoin.it/wiki/Tax_compliance