Bytes.
- Is it possible to calculate tx size from its hex ?
Yes. There are two hex digits to a byte. Count the number of hex digits, divide by two, and that's the size in bytes. Note if you have a program that is processing transactions, it already has the raw data, and hence already knows how many bytes there are.
- Is there any reference implementation on calculating tx fee from size ?
Bitcoin Core.