MTgox API Poll Failing [Not Solved ... ?]

[rizzn]

I had to create a polling script that mirrored the Ticker script from MTGOX (http://mtgox.com/code/data/ticker.php) since Google's app scripting doesn't allow you to set your USERAGENT, which for some reason MTGOX requires.

For whatever reason, some time over the weekend, this script just hangs and eventually returns a blank page. No errors are being produced in my PHP log. Anyone have any clues as to what's going on? This happening to anyone else?

Here's my code:

<?php
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($ch, CURLOPT_USERAGENT, "google spreadsheet bridge");
//    curl_setopt($ch, CURLOPT_URL, "http://rizzn.com");
//   curl_setopt($ch, CURLOPT_URL, "https://mtgox.com/code/data/ticker.php");
    curl_setopt($ch, CURLOPT_URL, "https://mtgox.com/code/data/ticker.php");
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    $output = curl_exec($ch);
    curl_close($ch);
    echo $output;
?>

UPDATE: as soon as I hit post, I went and refreshed my screen, and it's now returning data. very strange.
UPDATE 2: it appears to be working intermittently now. wtf, over?
UPDATE 3: Alright - switch that back to totally failing again. BTW, i haven't modified my code at all during this.

[1714172298]

1714172298

[1714172298]

1714172298

[1714172298]

1714172298

[1714172298]

1714172298

[rizzn]

Bumpity bump. Problems still happening. Haven't had much of a chance to look at it over the last week, but it's still going on.

[MagicalTux]

Are you sure you're keeping the data in cache to avoid querying too much the API ?

For information our DDoS protection service (Prolexic) drops requests without user agent, and will block ips which present weird behaviour.

[Oldminer]

Hey Mr Tux, I use SierraCharts and bitcoinity.org market trading software but notice sometimes I'll get no data for like several minutes at a time then it all comes though all of a sudden and starts flowing again. Whats up with that?

[rizzn]

Are you sure you're keeping the data in cache to avoid querying too much the API ?

For information our DDoS protection service (Prolexic) drops requests without user agent, and will block ips which present weird behaviour.

I'm currently not caching any data (my script is still just a few lines long, my php skills rusty, and I haven't even written the caching function yet).

Is it possible to whitelist my server for a while, at least until I get my script written?

Also, a rough idea on what rate limits should be set to so I don't aggravate your DDoS protection would be cool

[MagicalTux]

Are you sure you're keeping the data in cache to avoid querying too much the API ?

For information our DDoS protection service (Prolexic) drops requests without user agent, and will block ips which present weird behaviour.

I'm currently not caching any data (my script is still just a few lines long, my php skills rusty, and I haven't even written the caching function yet).

Is it possible to whitelist my server for a while, at least until I get my script written?

Also, a rough idea on what rate limits should be set to so I don't aggravate your DDoS protection would be cool

I have no idea of the applied limits nor any way to whitelist anything. The blocking is handled by Prolexic (our DDoS protection provider) and I don't have direct control over their system.

[rizzn]

Are you sure you're keeping the data in cache to avoid querying too much the API ?

For information our DDoS protection service (Prolexic) drops requests without user agent, and will block ips which present weird behaviour.

I'm currently not caching any data (my script is still just a few lines long, my php skills rusty, and I haven't even written the caching function yet).

Is it possible to whitelist my server for a while, at least until I get my script written?

Also, a rough idea on what rate limits should be set to so I don't aggravate your DDoS protection would be cool

I have no idea of the applied limits nor any way to whitelist anything. The blocking is handled by Prolexic (our DDoS protection provider) and I don't have direct control over their system.

That might be something I'd check up on then, MT. No disrespect, but their system is woefully over-reactive and over-sensitive. Did you know that, if Prolexic truly is the reason my scripts continually get blocked, that they actually blocked _Google_ about a week ago?

The whole reason this development effort started for me was because I created a Google Spreadsheet and needed to update it with live ticker data from MTgox. It worked for about a day (after I figured out how to send the USERAGENT in Google's scripting language), and then stopped working. I then created a php script to mirror the ticker data on one of my servers, which worked for about six hours, and was then blocked.

I started working last night around 6 PM on a different server, was able to download live ticker data and trade depth from the API for a few hours, and then suddenly the API wouldn't send me data any longer.

I'm not overtaxing the server - a few hits in a minute as I'm troubleshooting my script shouldn't trigger a DDoS warning, particularly when it's an API, and particularly when I'm not consistently hitting it multiple times a minute throughout the entire hour.

This is, without exaggeration, making your API completely unusable for me. I would strongly suggest speaking to Prolexic to come up with a way to rectify this issue, because I can't be the only one experiencing this problem.

[bitcoinTrader]

I am also building a script which will be polling mtgox api every 40 seconds.

Dont know if that is acceptable.

Have sent messages to Magical Tux and raised a ticket with support.
No reply yet.

[MagicalTux]

Prolexic's system is based not on fixed polling delays or similar, but on a (supposedly) intelligent system that catches abusive users by analyzing server load and how an average request looks like.

I could try to ask them to unblock you but I'd need a bit more informations (actual query look, source ip, possibly a traceroute, etc)

[SlipperySlope]

I shutdown my recording of Mt Gox trades yesterday because I am turning my attention back to unrelated research.   I had planned to create a bot for market making but found that Mt Gox trading is already too efficient for my simple algorithms to profit.

But for about 10 days I have been polling Mt Gox every 30 seconds for ticker, trades and depth with my Java based API client.  There are occasional periods of up to five minutes or so when Mt Gox times out, and there are also occasional SQL errors returned.  Otherwise I have been satisfied with the API performance.

I recorded about 90,000 trades just fine.

[bitcoinTrader]

Then it should work fine for me too I guess.
BTW, I too am using java client