API fail part II: BTC-E faceplant

[eldentyrell]

This is ridiculous.

All of the normal exchanges give you some sort of signing key in Base64 format.  You turn those Base64 bytes into bytes, use them as a signing key, you're done.

A while back Bitfloor screwed up by mistakenly using the ASCII codes for the Base64-encoded string instead of decoding it.  A mistake, but understandable.

Now Btc-E has taken the fail to a whole new level: they give you the key as a 256-bit number, but you don't use that as the key.  No, that would be too easy.  Instead you must convert that 256-bit number into a big-endian hexadecimal number, then render that hex number in ASCII, then use the bytes of that ASCII as your key.  I am not making this up, folks.  I have implemented it that way and it works, and moreover doing it any other way doesn't work.  By the way this also means they aren't using 93% of the keyspace.

Did I mention that none of this is explained, or even mentioned anywhere in BTC-E's woefully inadequate API documentation?

In recognition of this clusterfuck, I hereby revoke the "worst API award" from CryptoXChange and bestow it upon BTC-E.

[1715031017]

1715031017

[MPOE-PR]

This has been my informative read of the day. Thanks.

[daybyter]

Thanks a lot for your info! So I'll have to change my implementation as well...

Btw: bitparking change it's trade format yesterday. No more type field....

I think we should make an opensource trade lib as a reference implementation.

Edit 1: Regarding the api docs: the given examples worked at some point. But since the api has changed at least 2 times til then, they will most obviously fail now. Just to make clear, that it's not then fault of the source code authors...

Edit 2: it seems you are using an old version of my sources. I could see it from the TradeApp getCurrentGMTTimeMicros() method. I fixed quite some bugs since then, and added new features. Although the order hierachy is still suboptimal (withdraw orders still have an unnecessary price in them as an example).

[crazy_rabbit]

I think we should make an opensource trade lib as a reference implementation.

+1

[ElectricMucus]

I think we should make an opensource trade lib as a reference implementation.

+1

That should have been done like 2 years ago.

[daybyter]

I have some Java sources, that I'm willing to open. They are incomplete and there are some issues, but maybe good enough as a start...

[ElectricMucus]

I have some Java sources, that I'm willing to open. They are incomplete and there are some issues, but maybe good enough as a start...

I think the important part is documentation before implementation. But it might be a start, at least for those who are comfortable with java...

[TangibleCryptography]

I think we should make an opensource trade lib as a reference implementation.

+1

+1

Plus if multiple exchanges adopted it imagine how awesome it would be to add support for a 2nd, 3rd, 4th exchange to your service/app by simply adding another url & key.

And to the OP.... that is just insane. 

[2112]

I think we should make an opensource trade lib as a reference implementation.

I just wanted to ask one question to all people here thinking about the exchange API standarization:

Why don't use an existing standard like eg. FIX protocol?

http://en.wikipedia.org/wiki/Financial_Information_eXchange

Theoretically it alrady has everything that's required.

So what are the roadblocks? Are they technical or ideological? Maybe a software architecture mismatch? Is FIX simply too difficult? Or is FIX considered tainted for some reason?

Thanks in advance for any light you may shine on this question.

[MPOE-PR]

Why don't use an existing standard like eg. FIX protocol?

http://en.wikipedia.org/wiki/Financial_Information_eXchange

Theoretically it alrady has everything that's required.

MPEx sort-of started from there, but changes kept being brought to the point what we use now is barely reminiscent.

[2112]

MPEx sort-of started from there, but changes kept being brought to the point what we use now is barely reminiscent.

Can you describe those changes in a paragraph or two? Was there something missing? I'm actualy more interested in the motivation for changes than the actual software changes.

I'd like to understand motives, if possible. Technical details I would consider secondary.

[MPOE-PR]

MPEx sort-of started from there, but changes kept being brought to the point what we use now is barely reminiscent.

Can you describe those changes in a paragraph or two? Was there something missing? I'm actualy more interested in the motivation for changes than the actual software changes.

I'd like to understand motives, if possible. Technical details I would consider secondary.

Well let's start with an example comparison:

8=FIX.4.2 | 9=178 | 35=8 | 49=PHLX | 56=PERS | 52=20071123-05:30:00.000 | 11=ATOMNOCCC9990900 | 20=3 | 150=E | 39=E | 55=MSFT | 167=CS | 54=1 | 38=15 | 40=2 | 44=15 | 58=PHLX EQUITY TESTING | 59=0 | 47=C | 32=0 | 31=0 | 151=15 | 14=0 | 6=0 | 10=128 |

vs

DIVIDEND|B.MPCD.A|35000000000|350000

FIX uses predefined tags, which then receive values. This is great if a majority of messages will use only a small fraction of available tags, because it saves some space. However, if most messages use most tags it quickly becomes redundant. In a more FIX-ish approach the above MPEx string would have read

1=DIVIDEND|2=B.MPCD.A|3=35000000000|6=350000]

or something like that. Since the vast majority of messages would refer a type of action (1) and a MPSIC (2), most would include a value (3) and few use more than 4-5 fields, the simplified scheme seems preferable.

The FIX style would also allow for the mixing of tags, something MPEx does not tolerate. The advantage of something like that seems not worth the mention however (again on account of the short messages).

FIX uses checksums and some braindamaged signature implementation. We've dispensed with all that, relying instead on GPG clearsign and armored encrypt, which actually seem a much better solution.

You can check out the FAQ for more details on the MPEx protocol.

[shtylman]

I think we should make an opensource trade lib as a reference implementation.

I just wanted to ask one question to all people here thinking about the exchange API standarization:

Why don't use an existing standard like eg. FIX protocol?

http://en.wikipedia.org/wiki/Financial_Information_eXchange

Theoretically it alrady has everything that's required.

So what are the roadblocks? Are they technical or ideological? Maybe a software architecture mismatch? Is FIX simply too difficult? Or is FIX considered tainted for some reason?

Thanks in advance for any light you may shine on this question.

BitFloor has had a FIX gateway for months. I would link you to our docs but the server is offline because the data center lost power -> hosted in NYC and the storm has done a serious number of the power situation.

Should be back online soon.

[markm]

Marketcetera uses FIX, I am trying to get that working on my machine now, last time I tried I did not succeed but this time I have at least some parts of it working.

For end-user GUI it uses http://www.marketcetera.org/confluence/display/PN/Photon which is the part I am having a hard time with. Apparently it works out of the box on Windows but on Linux its ability to use more than one kind of browser's underlying libraries for rendering causes conflicts so the one they provide for Linux crashes and so far so has every one I have tried to compile for myself.

I am glad to hear that FIX does have some kind of signing, even if "braindead", as I was worried that real world financial institutions didn't seem to be concerned about ever being able to prove who ordered what...

-MarkM-

[2112]

as I was worried that real world financial institutions didn't seem to be concerned about ever being able to prove who ordered what...

For the traditional finserv houses this rarely is a problem. In their very competitve world the main problem is when customer wasn't able to place a trade when s/he wanted to. Besides, whatever authentication was used at the protocol level it will never be a single line of defense. There are always some other form of order-stream monitoring tools running behind.

For small-timers the trades are reversible and they are insured.

For big-timers there are VPNs and dedicated connections.

And pretty much everyone, big or small, can place an order over the analog phone line, including wireless where there still is analog wireless available.

Edit: also, FIX (and others) are connection-oriented giving additional measure of monitoring and safety.

[eldentyrell]

So what are the roadblocks? Are they technical or ideological? Maybe a software architecture mismatch? Is FIX simply too difficult? Or is FIX considered tainted for some reason?

I thought the same thing!  Then I started learning about FIX.  Unfortunately, it's a disaster.

Worst case of design-by-committee I've ever seen.  More of a trainwreck than all the XML standards rolled into one.

I think the meta-problem here is that FIX was created by the big Wall Street firms, and for them hiring ten programmers instead of one to implement some needlessly-overcomplicated protocol is no big deal.  In fact, the IT managers kinda like this because it increases the size of their fiefdom.  So, sadly, pre-bitcoin the organizations in a position to standardize this kind of thing also had an incentive to make it as complicated as possible.

In principle I'm in favor of standardization, but if an API is designed well it should really only take you 2-4 hours to write a plugin for it.  I guess that's part of the reason why I'm so prone to rant about crappy APIs.  If the exchange does a good job, the fact that there's no standard isn't really that much of a big deal.

[eldentyrell]

And pretty much everyone, big or small, can place an order over the analog phone line, including wireless where there still is analog wireless available.

I was absolutely floored when I found out that $100 million MBS trades are done by voice telephone call.

Now, the call is recorded, but still… no crypto, nothing.  Yow.

[MPOE-PR]

Worst case of design-by-committee I've ever seen.  More of a trainwreck than all the XML standards rolled into one.

This is true, but as they say, "it just works". At the cost of who knows how many broken fingers by now.

Anyway, given the interest Mr. P wrote out the specification a little, for this thing provisionally called BTC-UXP. All comments more than welcome.

I was absolutely floored when I found out that $100 million MBS trades are done by voice telephone call.

This is the power of "people know people". If you found out just how much of today's money, politics and so forth moves on a simple nod and a handshake....

[ElectricMucus]

Since every part of Bitcoin and largely the web uses JSON it would be better suited.

Just establish some protocol on how to format the messages.

[2112]

Since every part of Bitcoin and largely the web uses JSON it would be better suited.

Just establish some protocol on how to format the messages.

I'm inclinded to agree with this. But just not any JSON. JSON as used in Stratum and Stratum Mining. Mr. slush had made a good compromise between the ease of implementation and the reliability and scalability(power efficiency) requirements.

So Stratum Trading protocol would be a great short-term choice.

It is also a good long-term choice if your goal is to stay in the Bitcoin wading pool forever. But if you want to sail the seven seas sometime later you'll have to learn the protocols that the sea-faring navigators real traders are using.

[2112]

I was absolutely floored when I found out that $100 million MBS trades are done by voice telephone call.

Now, the call is recorded, but still… no crypto, nothing.  Yow.

Just for the historical explanation purposes:

For the telephone trading the primary line of defense is ANI:

http://en.wikipedia.org/wiki/Automatic_number_identification

Straight ANI is often strenghtened by either quick comparison with real time billing information or various SS7 tracing/call maintenance features.

http://en.wikipedia.org/wiki/Signalling_System_No._7

While pikers trade with touch-tones the whales get service of the dedicated VIP staff. It is a form of biometric authentication, people employed in high-value account service have extraordinary auditory  memory (or visual memory, for the in-person service).

[2112]

I thought the same thing!  Then I started learning about FIX.  Unfortunately, it's a disaster.

Worst case of design-by-committee I've ever seen.  More of a trainwreck than all the XML standards rolled into one.

I think the meta-problem here is that FIX was created by the big Wall Street firms, and for them hiring ten programmers instead of one to implement some needlessly-overcomplicated protocol is no big deal.  In fact, the IT managers kinda like this because it increases the size of their fiefdom.  So, sadly, pre-bitcoin the organizations in a position to standardize this kind of thing also had an incentive to make it as complicated as possible.

In principle I'm in favor of standardization, but if an API is designed well it should really only take you 2-4 hours to write a plugin for it.  I guess that's part of the reason why I'm so prone to rant about crappy APIs.  If the exchange does a good job, the fact that there's no standard isn't really that much of a big deal.

I think you forgot to consider "quality of design" versus "quality of implementation" dichotomy. You also completely neglected "cost of initial implementation" versus "cost of ongoing maintenance". FIX maybe like a camel: a horse designed by a commitee. But it has some of the best and most reliable implementations.

I wish that hiring one smart programmer would always guarantee a good solution. But I'm emphatically absolutely positively sure that this isn't a case. Your LiMP design is nice, lean and mean. But for comparison take a bitfury's proposal of mining protocol, where he had made completely newbie mistakes:

https://bitcointalk.org/index.php?topic=84791.0

I actually have an extensive experience in deploying and maintaining SOA (Service Oriented Architecture). We've been in a good position to be able to charge per-request and per-error, and our TOS contract has special provision to deal with people who believe that "if an API is designed well it should really only take you 2-4 hours to write a plugin for it." Both salespeople and tech support people are trained to recognize them as problem customers and charge/treat them appropriately.

I posted about this earlier in slush's thread about Stratum:

https://bitcointalk.org/index.php?topic=55842.msg664479#msg664479

Anyway, I was going to post links to a quite well designed trading protocol that is easier to implement than FIX. But Interactive Brokers are now require to have a properly funded production acount before they let you play with their API with a "paper trading" account. You may still want to review the freely available demos and documentation.

I just post one quote from the "IB gateway" (technically the better term would be "IB proxy") I find crucial:

Must remain running to maintain access to IB trading system.      Yes      Yes     

[2112]

Anyway, given the interest Mr. P wrote out the specification a little, for this thing provisionally called BTC-UXP. All comments more than welcome.

I did a quick look&see. It doesn't seem to have the protection against placing duplicate orders in case of transport failure/timeout. At the minimum all the imperative verb calls should have an OrderID argument that needs to be unique.

There may be some sort of replay attack made out the above flaw, but I don't have a motivation to delve deeper.

[MPOE-PR]

It is a form of biometric authentication, people employed in high-value account service have extraordinary auditory  memory (or visual memory, for the in-person service).

In other words, a wizard does it?

Anyway, given the interest Mr. P wrote out the specification a little, for this thing provisionally called BTC-UXP. All comments more than welcome.

I did a quick look&see. It doesn't seem to have the protection against placing duplicate orders in case of transport failure/timeout. At the minimum all the imperative verb calls should have an OrderID argument that needs to be unique.

There may be some sort of replay attack made out the above flaw, but I don't have a motivation to delve deeper.

That part's intentionally left blank.

If the exchange implements by-connection security (cookie based/https website logins are of this sort) then all orders are valid even if duplicate. If the exchange implements stateless by-payload security (such as the GPG scheme MPEx uses) then the exchange should also enforce unique-payload (either through hashing or some other method).

In either case these are considerations of exchange security, NOT of communication protocol. At least that's the thinking.

[2112]

If the exchange implements stateless by-payload security (such as the GPG scheme MPEx uses) then the exchange should also enforce unique-payload (either through hashing or some other method).

Hopefully to actually take the duplicate order you require at least a different IV (initialization vector) in the GPG armor and it is not sufficient to
simply reformat the armored message.

I personally don't like the houses where accepting duplicate(or otherwise confused) orders are part of the business plan. It always ends in litigation when somebody's cat steps on the keyboard and cleans the account.

With the volume and frequency so low I'm inclined to think that the connection-less protocol is not a problem. It will be if the frequency of trades rises.

In other words, a wizard does it?

I should've probably written "freakish" not just "extraordinary". It is more like a Rainman than a wizard.

[MPOE-PR]

MPEx uses a double layer: to talk to it you first clearsign then the resulting pgp message you encrypt (per FAQ). In order to duplicate you would have to re-sign the order somehow.

I personally don't like the houses where accepting duplicate(or otherwise confused) orders are part of the business plan. It always ends in litigation when somebody's cat steps on the keyboard and cleans the account.

Mr. P is of the same persuasion, hence MPEx working the way it does. Still, by-connection is probably a perfectly legitimate way to do things, in some cases (which almost never are the cases where it's applied in BTCWorld, but anyway).

With the volume and frequency so low I'm inclined to think that the connection-less protocol is not a problem. It will be if the frequency of trades rises.

Care to elaborate? Seems exactly the opposite would be true.

[2112]

Care to elaborate? Seems exactly the opposite would be true.

With connection-less trading a sudden trading spike is indistinguishable from a DDoS attack.

Because setting up and tearing down TCP/IP connection involves several small packets you will learn that your networking infrastructure has two limits: bps (bits per second) and pps (packets per second) and you will be governed by the pps limit (while staying far away from bps limit) during the most important periods of the trading. The connection failure rates will greatly increase, this will be visible on your side as storms of inquisitive verbs (STAT, STATJSON, etc.) The quote/status dissemination during an active trading is just so much more efficient with the stream-oriented protocols that there isn't much room for discussion when comparing it with RPC-style polling.

In addition to the above "ideal world" failure mode, in a "real world" case when somebody actually tries to DDoS you you can mount the defenses much cheaper and much more efficiently if you'll require keeping up the connection while trading.

Socially, I find this RPC-trading phenomenon puzzling. The very same people who obsessively tune their gaming rigs for minimum lag suddenly are fine with the lag values several orders of magnitude higher when trading for real money instead of playing for toy money.

[MPOE-PR]

Care to elaborate? Seems exactly the opposite would be true.

With connection-less trading a sudden trading spike is indistinguishable from a DDoS attack.

Because setting up and tearing down TCP/IP connection involves several small packets you will learn that your networking infrastructure has two limits: bps (bits per second) and pps (packets per second) and you will be governed by the pps limit (while staying far away from bps limit) during the most important periods of the trading. The connection failure rates will greatly increase, this will be visible on your side as storms of inquisitive verbs (STAT, STATJSON, etc.) The quote/status dissemination during an active trading is just so much more efficient with the stream-oriented protocols that there isn't much room for discussion when comparing it with RPC-style polling.

In addition to the above "ideal world" failure mode, in a "real world" case when somebody actually tries to DDoS you you can mount the defenses much cheaper and much more efficiently if you'll require keeping up the connection while trading.

Socially, I find this RPC-trading phenomenon puzzling. The very same people who obsessively tune their gaming rigs for minimum lag suddenly are fine with the lag values several orders of magnitude higher when trading for real money instead of playing for toy money.

Well, I think the problem is the vast pile of conjecture you're fighting with.

[J-Norm]

The best way to create an API is to FIRST write the documentation. Then once you see that what you are documenting makes sense you then implement each feature that is documented.

That way not only do you have a fully documented API but you also see if what you are designing actually makes sense.

If you program it first and then document it you will run into all sorts of design issues and end up lacking documentation.

[MPOE-PR]

The best way to create an API is to FIRST write the documentation. Then once you see that what you are documenting makes sense you then implement each feature that is documented.

That way not only do you have a fully documented API but you also see if what you are designing actually makes sense.

If you program it first and then document it you will run into all sorts of design issues and end up lacking documentation.

In a for-profit venture that's a little more complicated than in the FOSS world:

0. You write up your goals/principles. This is where a committee should be involved.
I. You write the spec. This is something that should be done by one single person.
II. You throw the spec around. This should be a general effort. Even the cleaning lady is allowed to throw items at the spec (provided she satisfies all applicable requirements for keeping the secret).
III. You implement the spec. This is something that should be done by a team (which is a different beast from a committee).
IV. You revise the complaints of the people implementing to try and ascertain whether it's them being lazy or them having a point. This is a committee + autarch job (as in, some points will have to be decided by means of a large stick or other implement with˙ʎɥʍ s,ʇɐɥʇ 'noʎ ʞɔnɟ ǝsnɐɔǝq etched over the business end).
V. You open a beta. Revise etcetera.
VI. You come out of beta.
VII. The general public starts to take an interest at a higher abstraction level than the simple usage of your product.
VIII. You release some version of the spec for public consumption.

We're on step VIII atm.

[daybyter]

Well from my viewpoint (as a coder implementing some API for trading):

- I usually work as a software engineer modeling applications with tools. But...

- I guess I don't have the same knowledge, that most of you seem to have about trading API's, so for me it's a bit 'learning by doing'.

- My trading app is growing slowly, and I learn step by step what a trading app needs, so my trading lib is growing with my app. That process has to do with trading sites and their restrictions. Some only deliver limited data. Others update their data only after x seconds and so on. So a perfect trade app has to consider those limitations and deal with them in some way.

Learning those problems and finding workaround leads me to some iterative development process, redesigning my code again and again. So even if I had written the specs by 100% before doing any coding, I'd still have to rework my code from time to time as I learn about those trade sites.

But seeing stuff growing and working more and more is fun and keeps me motivated to further the development. So it's ok by me...

[ElectricMucus]

I'm gonna bump this.

Been attempting to get this working for myself, but haven't had any luck either.
If anybody who got this working is available for questioning this would be greatly appreciated.

[daybyter]

I can use some of the btc-e API yet. But I'm still struggling with some features. Would be cool if I could discuss deposit and withdrawal with someone.

[kelpy]

Hi,

What's the object HttpUtils.httpPost ?
Where can i found it?!

Code:

HttpUtils.httpPost( "https://" + DOMAIN + "/tapi", headerLines, postData);

http://pastebin.com/jyd9tACF

I just try with HttpURLConnection with no result.

Thanks in advance.

[daybyter]

http://pastebin.com/rBKRQvrG

[kelpy]

http://pastebin.com/rBKRQvrG

Thanks

[ElectricMucus]

In case anybody here is interested in implementing the interface yourself (that is not copying other peoples code  )

You have to copy the UTF-8 runes of the 'Secret' into a byte array. !Not! convert the hexadecimal string into a number.