I'm feeling the same way. I wish I'd bought some Bitcoins back when they were fresh. LTC may or may not take off like BTC has, but it couldn't hurt to have a few of them just in case.
Fundamentally, crypto-currencies should be valued by their user base, so Litecoin would only take off with an increase in user base. Though, we may see a significant increase in user base with none ASIC miners moving to Litecoin.
Not only will the asics cause an increase, but with all the hype about BTC lately, more and more people are piling in to it. Because of how hard it is to mine BTC right now, people will look for alternatives, which will cause another increase. Also, if MtGOX accepts LTC, yet another increase will happen. I tried to get into bitcoin back when they were $0.05 per coin, I never knew they would take off like they did now. BTC didn't start in 2011, it started earlier. 2011 is when it started raising in price, and the last few months is when it really took off. The reason I didn't get into it back then is because the proccess was a pain. Now it's a simple 2 step download and install on both the wallet and the miner and your ready.