If I've got a memory card slot in my smartphone and I've stored/kept everything (almost everything related to crypto) and even installed my wallets in that memory card (by selecting it as default).
Holy shit.. that's a really bad idea.
Any data stored on the SD card can be accessed by any application on your mobile.
What you basically did is to circumvent security measurements of android which encapsulate storage from applications from each other.
On the internal memory, an application can only access the data in their own folder. But every application has access to the external memory.
You should never store sensitive information (which a wallet application definitely is) on an SD card. Always store it on the internal memory.
One final question:
While destroying it, what should I take care of?
I mean, I don't want to get blasted off my phone while trying to destroy it. Also, which part should be mainly destroyed to end the possibility of fetching any sort of data from that piece if it ever gets to anybody's hands?
Well.. theoretically.. wipe the (encrypted) mobile and fill it with junk data. That's the easiest way.
Then you are already safe, and don't need too much effort to destroy it physically.
All you need to destroy are the memory chips.