API fail part II: BTC-E faceplant

[eldentyrell]

This is ridiculous.

All of the normal exchanges give you some sort of signing key in Base64 format.  You turn those Base64 bytes into bytes, use them as a signing key, you're done.

A while back Bitfloor screwed up by mistakenly using the ASCII codes for the Base64-encoded string instead of decoding it.  A mistake, but understandable.

Now Btc-E has taken the fail to a whole new level: they give you the key as a 256-bit number, but you don't use that as the key.  No, that would be too easy.  Instead you must convert that 256-bit number into a big-endian hexadecimal number, then render that hex number in ASCII, then use the bytes of that ASCII as your key.  I am not making this up, folks.  I have implemented it that way and it works, and moreover doing it any other way doesn't work.  By the way this also means they aren't using 93% of the keyspace.

Did I mention that none of this is explained, or even mentioned anywhere in BTC-E's woefully inadequate API documentation?

In recognition of this clusterfuck, I hereby revoke the "worst API award" from CryptoXChange and bestow it upon BTC-E.

[MPOE-PR]

This has been my informative read of the day. Thanks.

[daybyter]

Thanks a lot for your info! So I'll have to change my implementation as well...

Btw: bitparking change it's trade format yesterday. No more type field....

I think we should make an opensource trade lib as a reference implementation.

Edit 1: Regarding the api docs: the given examples worked at some point. But since the api has changed at least 2 times til then, they will most obviously fail now. Just to make clear, that it's not then fault of the source code authors...

Edit 2: it seems you are using an old version of my sources. I could see it from the TradeApp getCurrentGMTTimeMicros() method. I fixed quite some bugs since then, and added new features. Although the order hierachy is still suboptimal (withdraw orders still have an unnecessary price in them as an example).

[crazy_rabbit]

I think we should make an opensource trade lib as a reference implementation.

+1

[ElectricMucus]

I think we should make an opensource trade lib as a reference implementation.

+1

That should have been done like 2 years ago.

[daybyter]

I have some Java sources, that I'm willing to open. They are incomplete and there are some issues, but maybe good enough as a start...

[ElectricMucus]

I have some Java sources, that I'm willing to open. They are incomplete and there are some issues, but maybe good enough as a start...

I think the important part is documentation before implementation. But it might be a start, at least for those who are comfortable with java...

[TangibleCryptography]

I think we should make an opensource trade lib as a reference implementation.

+1

+1

Plus if multiple exchanges adopted it imagine how awesome it would be to add support for a 2nd, 3rd, 4th exchange to your service/app by simply adding another url & key.

And to the OP.... that is just insane. 

[2112]

I think we should make an opensource trade lib as a reference implementation.

I just wanted to ask one question to all people here thinking about the exchange API standarization:

Why don't use an existing standard like eg. FIX protocol?

http://en.wikipedia.org/wiki/Financial_Information_eXchange

Theoretically it alrady has everything that's required.

So what are the roadblocks? Are they technical or ideological? Maybe a software architecture mismatch? Is FIX simply too difficult? Or is FIX considered tainted for some reason?

Thanks in advance for any light you may shine on this question.

[MPOE-PR]

Why don't use an existing standard like eg. FIX protocol?

http://en.wikipedia.org/wiki/Financial_Information_eXchange

Theoretically it alrady has everything that's required.

MPEx sort-of started from there, but changes kept being brought to the point what we use now is barely reminiscent.

[2112]

MPEx sort-of started from there, but changes kept being brought to the point what we use now is barely reminiscent.

Can you describe those changes in a paragraph or two? Was there something missing? I'm actualy more interested in the motivation for changes than the actual software changes.

I'd like to understand motives, if possible. Technical details I would consider secondary.

[MPOE-PR]

MPEx sort-of started from there, but changes kept being brought to the point what we use now is barely reminiscent.

Can you describe those changes in a paragraph or two? Was there something missing? I'm actualy more interested in the motivation for changes than the actual software changes.

I'd like to understand motives, if possible. Technical details I would consider secondary.

Well let's start with an example comparison:

8=FIX.4.2 | 9=178 | 35=8 | 49=PHLX | 56=PERS | 52=20071123-05:30:00.000 | 11=ATOMNOCCC9990900 | 20=3 | 150=E | 39=E | 55=MSFT | 167=CS | 54=1 | 38=15 | 40=2 | 44=15 | 58=PHLX EQUITY TESTING | 59=0 | 47=C | 32=0 | 31=0 | 151=15 | 14=0 | 6=0 | 10=128 |

vs

DIVIDEND|B.MPCD.A|35000000000|350000

FIX uses predefined tags, which then receive values. This is great if a majority of messages will use only a small fraction of available tags, because it saves some space. However, if most messages use most tags it quickly becomes redundant. In a more FIX-ish approach the above MPEx string would have read

1=DIVIDEND|2=B.MPCD.A|3=35000000000|6=350000]

or something like that. Since the vast majority of messages would refer a type of action (1) and a MPSIC (2), most would include a value (3) and few use more than 4-5 fields, the simplified scheme seems preferable.

The FIX style would also allow for the mixing of tags, something MPEx does not tolerate. The advantage of something like that seems not worth the mention however (again on account of the short messages).

FIX uses checksums and some braindamaged signature implementation. We've dispensed with all that, relying instead on GPG clearsign and armored encrypt, which actually seem a much better solution.

You can check out the FAQ for more details on the MPEx protocol.

[shtylman]

I think we should make an opensource trade lib as a reference implementation.

I just wanted to ask one question to all people here thinking about the exchange API standarization:

Why don't use an existing standard like eg. FIX protocol?

http://en.wikipedia.org/wiki/Financial_Information_eXchange

Theoretically it alrady has everything that's required.

So what are the roadblocks? Are they technical or ideological? Maybe a software architecture mismatch? Is FIX simply too difficult? Or is FIX considered tainted for some reason?

Thanks in advance for any light you may shine on this question.

BitFloor has had a FIX gateway for months. I would link you to our docs but the server is offline because the data center lost power -> hosted in NYC and the storm has done a serious number of the power situation.

Should be back online soon.

[markm]

Marketcetera uses FIX, I am trying to get that working on my machine now, last time I tried I did not succeed but this time I have at least some parts of it working.

For end-user GUI it uses http://www.marketcetera.org/confluence/display/PN/Photon which is the part I am having a hard time with. Apparently it works out of the box on Windows but on Linux its ability to use more than one kind of browser's underlying libraries for rendering causes conflicts so the one they provide for Linux crashes and so far so has every one I have tried to compile for myself.

I am glad to hear that FIX does have some kind of signing, even if "braindead", as I was worried that real world financial institutions didn't seem to be concerned about ever being able to prove who ordered what...

-MarkM-

[2112]

as I was worried that real world financial institutions didn't seem to be concerned about ever being able to prove who ordered what...

For the traditional finserv houses this rarely is a problem. In their very competitve world the main problem is when customer wasn't able to place a trade when s/he wanted to. Besides, whatever authentication was used at the protocol level it will never be a single line of defense. There are always some other form of order-stream monitoring tools running behind.

For small-timers the trades are reversible and they are insured.

For big-timers there are VPNs and dedicated connections.

And pretty much everyone, big or small, can place an order over the analog phone line, including wireless where there still is analog wireless available.

Edit: also, FIX (and others) are connection-oriented giving additional measure of monitoring and safety.

[eldentyrell]

So what are the roadblocks? Are they technical or ideological? Maybe a software architecture mismatch? Is FIX simply too difficult? Or is FIX considered tainted for some reason?

I thought the same thing!  Then I started learning about FIX.  Unfortunately, it's a disaster.

Worst case of design-by-committee I've ever seen.  More of a trainwreck than all the XML standards rolled into one.

I think the meta-problem here is that FIX was created by the big Wall Street firms, and for them hiring ten programmers instead of one to implement some needlessly-overcomplicated protocol is no big deal.  In fact, the IT managers kinda like this because it increases the size of their fiefdom.  So, sadly, pre-bitcoin the organizations in a position to standardize this kind of thing also had an incentive to make it as complicated as possible.

In principle I'm in favor of standardization, but if an API is designed well it should really only take you 2-4 hours to write a plugin for it.  I guess that's part of the reason why I'm so prone to rant about crappy APIs.  If the exchange does a good job, the fact that there's no standard isn't really that much of a big deal.

[eldentyrell]

And pretty much everyone, big or small, can place an order over the analog phone line, including wireless where there still is analog wireless available.

I was absolutely floored when I found out that $100 million MBS trades are done by voice telephone call.

Now, the call is recorded, but still… no crypto, nothing.  Yow.

[MPOE-PR]

Worst case of design-by-committee I've ever seen.  More of a trainwreck than all the XML standards rolled into one.

This is true, but as they say, "it just works". At the cost of who knows how many broken fingers by now.

Anyway, given the interest Mr. P wrote out the specification a little, for this thing provisionally called BTC-UXP. All comments more than welcome.

I was absolutely floored when I found out that $100 million MBS trades are done by voice telephone call.

This is the power of "people know people". If you found out just how much of today's money, politics and so forth moves on a simple nod and a handshake....

[ElectricMucus]

Since every part of Bitcoin and largely the web uses JSON it would be better suited.

Just establish some protocol on how to format the messages.

[2112]

Since every part of Bitcoin and largely the web uses JSON it would be better suited.

Just establish some protocol on how to format the messages.

I'm inclinded to agree with this. But just not any JSON. JSON as used in Stratum and Stratum Mining. Mr. slush had made a good compromise between the ease of implementation and the reliability and scalability(power efficiency) requirements.

So Stratum Trading protocol would be a great short-term choice.

It is also a good long-term choice if your goal is to stay in the Bitcoin wading pool forever. But if you want to sail the seven seas sometime later you'll have to learn the protocols that the sea-faring navigators real traders are using.