Bitcoin Forum
December 06, 2016, 07:56:54 PM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: MTgox API Poll Failing [Not Solved ... ?]  (Read 1588 times)
rizzn
Member
**
Offline Offline

Activity: 106



View Profile WWW
July 05, 2011, 09:19:14 PM
 #1

I had to create a polling script that mirrored the Ticker script from MTGOX (http://mtgox.com/code/data/ticker.php) since Google's app scripting doesn't allow you to set your USERAGENT, which for some reason MTGOX requires.

For whatever reason, some time over the weekend, this script just hangs and eventually returns a blank page. No errors are being produced in my PHP log. Anyone have any clues as to what's going on? This happening to anyone else?

Here's my code:

<?php
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($ch, CURLOPT_USERAGENT, "google spreadsheet bridge");
//    curl_setopt($ch, CURLOPT_URL, "http://rizzn.com");
//   curl_setopt($ch, CURLOPT_URL, "https://mtgox.com/code/data/ticker.php");
    curl_setopt($ch, CURLOPT_URL, "https://mtgox.com/code/data/ticker.php");
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    $output = curl_exec($ch);
    curl_close($ch);
    echo $output;
?>


UPDATE: as soon as I hit post, I went and refreshed my screen, and it's now returning data. very strange.
UPDATE 2: it appears to be working intermittently now. wtf, over?
UPDATE 3: Alright - switch that back to totally failing again. BTW, i haven't modified my code at all during this.

founding editor for the SiliconANGLE network.
Bitrated user: rizzn.
1481054214
Hero Member
*
Offline Offline

Posts: 1481054214

View Profile Personal Message (Offline)

Ignore
1481054214
Reply with quote  #2

1481054214
Report to moderator
1481054214
Hero Member
*
Offline Offline

Posts: 1481054214

View Profile Personal Message (Offline)

Ignore
1481054214
Reply with quote  #2

1481054214
Report to moderator
1481054214
Hero Member
*
Offline Offline

Posts: 1481054214

View Profile Personal Message (Offline)

Ignore
1481054214
Reply with quote  #2

1481054214
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481054214
Hero Member
*
Offline Offline

Posts: 1481054214

View Profile Personal Message (Offline)

Ignore
1481054214
Reply with quote  #2

1481054214
Report to moderator
1481054214
Hero Member
*
Offline Offline

Posts: 1481054214

View Profile Personal Message (Offline)

Ignore
1481054214
Reply with quote  #2

1481054214
Report to moderator
1481054214
Hero Member
*
Offline Offline

Posts: 1481054214

View Profile Personal Message (Offline)

Ignore
1481054214
Reply with quote  #2

1481054214
Report to moderator
rizzn
Member
**
Offline Offline

Activity: 106



View Profile WWW
July 12, 2011, 02:39:17 AM
 #2

Bumpity bump. Problems still happening. Haven't had much of a chance to look at it over the last week, but it's still going on.

founding editor for the SiliconANGLE network.
Bitrated user: rizzn.
MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 617


Working on new MtGox features


View Profile WWW
July 12, 2011, 04:26:03 AM
 #3

Are you sure you're keeping the data in cache to avoid querying too much the API ?

For information our DDoS protection service (Prolexic) drops requests without user agent, and will block ips which present weird behaviour.

Oldminer
Legendary
*
Offline Offline

Activity: 1022



View Profile
July 12, 2011, 04:36:21 AM
 #4

Hey Mr Tux, I use SierraCharts and bitcoinity.org market trading software but notice sometimes I'll get no data for like several minutes at a time then it all comes though all of a sudden and starts flowing again. Whats up with that?

If you like my post please feel free to give me some positive rep https://bitcointalk.org/index.php?action=trust;u=18639
Tip me BTC: 1FBmoYijXVizfYk25CpiN8Eds9J6YiRDaX
rizzn
Member
**
Offline Offline

Activity: 106



View Profile WWW
July 12, 2011, 04:52:19 AM
 #5

Are you sure you're keeping the data in cache to avoid querying too much the API ?

For information our DDoS protection service (Prolexic) drops requests without user agent, and will block ips which present weird behaviour.

I'm currently not caching any data (my script is still just a few lines long, my php skills rusty, and I haven't even written the caching function yet).

Is it possible to whitelist my server for a while, at least until I get my script written?

Also, a rough idea on what rate limits should be set to so I don't aggravate your DDoS protection would be cool

founding editor for the SiliconANGLE network.
Bitrated user: rizzn.
MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 617


Working on new MtGox features


View Profile WWW
July 12, 2011, 12:37:02 PM
 #6

Are you sure you're keeping the data in cache to avoid querying too much the API ?

For information our DDoS protection service (Prolexic) drops requests without user agent, and will block ips which present weird behaviour.

I'm currently not caching any data (my script is still just a few lines long, my php skills rusty, and I haven't even written the caching function yet).

Is it possible to whitelist my server for a while, at least until I get my script written?

Also, a rough idea on what rate limits should be set to so I don't aggravate your DDoS protection would be cool

I have no idea of the applied limits nor any way to whitelist anything. The blocking is handled by Prolexic (our DDoS protection provider) and I don't have direct control over their system.

rizzn
Member
**
Offline Offline

Activity: 106



View Profile WWW
July 12, 2011, 03:43:14 PM
 #7

Are you sure you're keeping the data in cache to avoid querying too much the API ?

For information our DDoS protection service (Prolexic) drops requests without user agent, and will block ips which present weird behaviour.

I'm currently not caching any data (my script is still just a few lines long, my php skills rusty, and I haven't even written the caching function yet).

Is it possible to whitelist my server for a while, at least until I get my script written?

Also, a rough idea on what rate limits should be set to so I don't aggravate your DDoS protection would be cool

I have no idea of the applied limits nor any way to whitelist anything. The blocking is handled by Prolexic (our DDoS protection provider) and I don't have direct control over their system.

That might be something I'd check up on then, MT. No disrespect, but their system is woefully over-reactive and over-sensitive. Did you know that, if Prolexic truly is the reason my scripts continually get blocked, that they actually blocked _Google_ about a week ago?

The whole reason this development effort started for me was because I created a Google Spreadsheet and needed to update it with live ticker data from MTgox. It worked for about a day (after I figured out how to send the USERAGENT in Google's scripting language), and then stopped working. I then created a php script to mirror the ticker data on one of my servers, which worked for about six hours, and was then blocked.

I started working last night around 6 PM on a different server, was able to download live ticker data and trade depth from the API for a few hours, and then suddenly the API wouldn't send me data any longer.

I'm not overtaxing the server - a few hits in a minute as I'm troubleshooting my script shouldn't trigger a DDoS warning, particularly when it's an API, and particularly when I'm not consistently hitting it multiple times a minute throughout the entire hour.

This is, without exaggeration, making your API completely unusable for me. I would strongly suggest speaking to Prolexic to come up with a way to rectify this issue, because I can't be the only one experiencing this problem.

founding editor for the SiliconANGLE network.
Bitrated user: rizzn.
bitcoinTrader
Sr. Member
****
Offline Offline

Activity: 364


View Profile
July 13, 2011, 05:39:09 AM
 #8

I am also building a script which will be polling mtgox api every 40 seconds.

Dont know if that is acceptable.

Have sent messages to Magical Tux and raised a ticket with support.
No reply yet.

MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 617


Working on new MtGox features


View Profile WWW
July 13, 2011, 02:24:06 PM
 #9

Prolexic's system is based not on fixed polling delays or similar, but on a (supposedly) intelligent system that catches abusive users by analyzing server load and how an average request looks like.

I could try to ask them to unblock you but I'd need a bit more informations (actual query look, source ip, possibly a traceroute, etc)

SlipperySlope
Hero Member
*****
Offline Offline

Activity: 686

Stephen Reed


View Profile
July 13, 2011, 02:30:12 PM
 #10

I shutdown my recording of Mt Gox trades yesterday because I am turning my attention back to unrelated research.   I had planned to create a bot for market making but found that Mt Gox trading is already too efficient for my simple algorithms to profit.

But for about 10 days I have been polling Mt Gox every 30 seconds for ticker, trades and depth with my Java based API client.  There are occasional periods of up to five minutes or so when Mt Gox times out, and there are also occasional SQL errors returned.  Otherwise I have been satisfied with the API performance.

I recorded about 90,000 trades just fine.
bitcoinTrader
Sr. Member
****
Offline Offline

Activity: 364


View Profile
July 13, 2011, 06:01:10 PM
 #11

Then it should work fine for me too I guess.
BTW, I too am using java client  Smiley

Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!