I think this could likely be gamed unless rounding were done randomly (and then it is difficult to audit) and it would not average out.
Yeah, you're right, completely forgot about this. It can totally be gamed, just sell your coins in small chunks to avoid the fee altogether, guess we're back at "the correct answer is to count properly with higher precision"
Theoretically, even counting at an arbitrarily large, but finite, precision can be gamed this way. Even with 8 digits of precision, buying satoshis one by one at a 100 Currency/BTC spot and a .5% commission rate would lead to 5E-11 BTC fees being rounded to 0 (given a 1E-8 precision).
Even with a minimum trade amount you'd have some small rounding artefacts on fees due to small partially filled order remainders being executed against, so you have to take an extra step.
So, to summarize, the most correct course of action is to
- Count with an arbitrary precision, preferably very small,
- Impose a minimum order size that is a very large multiple of the precision
- Remove bids from the book when their remaining volume is smaller than (precision /(2*fee))
- Remove asks from the book when their remaining volume is smaller than (execution_price * precision / (2 * fee))
Numerical example for the bid side - Spot is 500 USD/BTC
- Precision is 10^-8
- Fee : .5% (0.005)
(The reasoning is similar for asks but I can't be fucked right now)
Who knew so much fun could come from working with finite-precision numbers