The main thing is a lack of standardisation, the protocol uses a mixture of big endian, little endian, fixed length integers and variable integers. Signatures are encoded using DER, public keys are not. Compressed and uncompressed keys can both be used so every Bitcoin address has a twin depending on which encoding is chosen. Bitcoin addresses support error detection but not error correction and pubKeyHashes are 160 bits in length where everything else is 256 bits in length. The transaction scripting and signing methods means transactions are malleable. OP_CHECKMULTISIG requires an extra null public key to work properly. Transaction sequence numbers don't work.
There are more problems than this but those are the main ones I can think off the top of my head. Also almost all altcoins (except maybe NXT) carry over these annoying inconsistencies instead of fixing them!
Couldn't have said it any better. Working with the raw protocol is a royal pain. And it's not going to be fixed anytime soon, since any change needs to keep compatibility.
I just heard that Matt Whitlock was/is working on building an alternative-but-compatible Bitcoin core.
