Can you try running a traceroute to the server and see if you are dropping packets anywhere? Something like ping plotter would be really helpful in diagnosing it as well.
But starting with a post of your traceroute might reveal some clues.
And here's the mtr report:
HOST: wogaut Loss% Snt Last Avg Best Wrst StDev
1.|-- 192.168.1.254 0.0% 50 1.6 1.6 1.5 2.4 0.2
2.|-- 99-63-172-2.lightspeed.mi 80.0% 50 23.3 23.5 22.2 25.5 1.0
3.|-- 71.144.192.62 92.0% 50 36.5 26.6 22.6 36.5 6.6
4.|-- 71.144.192.126 90.0% 50 23.0 22.9 22.6 23.4 0.3
5.|-- 71.144.192.120 98.0% 50 23.5 23.5 23.5 23.5 0.0
6.|-- 12.83.79.141 0.0% 50 22.6 22.5 21.7 23.9 0.4
7.|-- cgcil03jt.ip.att.net 0.0% 50 24.6 32.8 23.9 116.7 20.9
8.|-- 192.205.37.178 0.0% 50 25.4 25.3 24.5 28.1 0.6
| `|-- 192.205.37.174
| |-- 154.54.12.85
9.|-- te0-4-0-1.ccr21.ord01.atl 0.0% 50 25.9 26.2 24.6 39.6 2.5
10.|-- te0-1-0-3.ccr21.mci01.atl 0.0% 50 49.8 49.8 48.6 58.6 1.4
11.|-- te0-2-0-0.mpd22.mci01.atl 0.0% 50 50.1 49.9 48.9 53.4 0.7
12.|-- everest.demarc.cogentco.c 0.0% 50 50.0 49.7 48.5 61.6 1.8
13.|-- 69.30.209.1 0.0% 50 49.8 48.1 47.1 50.3 1.0
14.|-- us1.eclipsemc.com 0.0% 50 47.2 47.7 46.7 49.8 0.8
IMO the loss values reported above are no reason for concern, I would think they are a result of rate limiting, since there's 0% loss down the chain.