A couple of questions:
When generating public / private key pairs, can any number (satisfying the requisite size constraints) be a private key that has a corresponding public key, or is there some property of private keys that rule out some values? Hope that makes sense.
Assuming that I've converted from Hex to decimal properly, a private key can be any value at all between 1 and 115,792,089,237,316,195,423,570,985,008,687,907,852,837,564,279,074,904,382,605,163,141,518,161,494,337
(Expressed as a Hexadecimal value, that would be: 0xFFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFE BAAE DCE6 AF48 A03B BFD2 5E8C D036 4141)
Also, given the size of the blockchain, are there any ideas about keeping the size manageable as bitcoin becomes more mainstream?
Yes, there are pruning processes that are being investigated and tested.
Also I suppose another difficulty as I understand it is unspent transaction outputs that existing in old blocks, meaning these also cannot be got rid of.
Correct. To verify transactions, the UTXO will have to be stored.