Do you communicate the exchange service fee/rake when signing them up? Sounds like you have been doing is for a little while and this has been very informal.
Cheers
the merchant pays NO FEE's
if the merchant is selling a $20 product he WILL get $20.
the commission comes from the customer side not the merchant side and as bitcoin varies between $497-$520 as long as the customer see's a bitcoin price that roughly equates to the something around that area and does not equate to lets say $590 today, then the customer would not really complain if they have to pay $520ish.
problems do arrise if you start mentioning which exchange you use, which price point and what percentages, and if it was just 20c innaccuracy they would moan and argue non-stop. so its normally best to just keep the bitcoin value within the 'acceptable variance' and not be quoting too much specifics as it just leads to arguments.
after all i know many businesses that quote the most expensive exchange but actually do business on the cheapest exchange. where its simpler to not quote any exchange and just have the prices in the 'fair' market range of all the exchanges.
now then how you would set up your own calculator will all depends on how you wish to obtain the FIAT to pay the merchant.
1) if you personally or a group of FIAT investors have lots of FIAT and they want bitcoin you can pretty much name your own value of bitcoin to ensure the merchant gets the $20 for a $20 product transaction. (within fair volatility range) and as you point out with your comments to the bitcoin co-op cavirtex price was $520, yet btc-e was $497-$510 within a couple of hours of your post which means an acceptable 1%-4% is acceptable by the CUSTOMER, due to their knowledge of volatility.
2) if your getting fiat by selling the bitcoin on an exchange to recoup a FIAT reserve for future transactions. not only are you going to want a couple percent commission to pay your own bills but you probably end up wanting an extra couple % ontop to personally hoard as coins. so this model usually is more about pushing the limits of acceptable pricing for customers and also finding the right balance of getting the merchant the fiat they need and running a profitable business yourself.
i would advise you to think more along the private investment side as oppose to using exchanges to swap bitcoins. establish a good rapport with respected people on the bitcoin-otc/localbitcoins. even find some FIAT investors in your area that way you fully control the value and exchange, rather than using an 'centralized exchange'. also by not selling the coin on an 'centralized exchange' your not dumping coins to cause a price drop. and as i said not having to push the limits of the fair volatility pricing to squeeze in some commission and hoarding %