Bytes.

- Is it possible to calculate tx size from its hex ?

Yes. There are two hex digits to a byte. Count the number of hex digits, divide by two, and that's the size in bytes. Note if you have a program that is processing transactions, it already has the raw data, and hence already knows how many bytes there are.

- Is there any reference implementation on calculating tx fee from size ?

Bitcoin Core.