The block chain API hell

[xenoky]

Hi,

i am developing a bitcoin-based service.
I am using blockchain api to ask various information, to be specific i am calling this endpoints:

https://blockchain.info/q/addressbalance/{BTCADDRESS}
https://blockchain.info/address/{BTCADDRESS}
https://blockchain.info/tx-index/{TXINDEX}
https://blockchain.info/pushtx

The problem is that i want a failover solution because ddos happens on blockchain.info (like now)

The only endpoint that i found has a compatible alternative is the first:
https://blockexplorer/q/addressbalance/{BTCADDRESS}
Also in this case there is a small problem, blockchain returns value in satoshi, blockeplorer returns it in bitcoins. But this is just the smaller problem...

Others endpoints have some equivalent services but with completely different json format on blockr, biteasy, abe, apicoin...

Or i am missing something or i strongly believe standard base API for querying bitcoin blockchain would give great benefit to bitcoin-based services

[gweedo]

Hi I am owner of APICoin.io,

Apicoin would work well as a failover, but we are beta so I recommend actually hold off on using us, currently as a failover or in any production level system.

I am also looking into standardization blockexplorer's api (https://apicoin.io/api/q/doc/) which I feel is the best way to do these quick calls and I follow block explorers lead on those calls.

Thanks for considering us, sorry we can't be more of help but we just launched less than a month ago.

[xenoky]

Thanks for the efforts, i made a donation of 30mBTC to your signature address and if you or anyboady is interested i promise as much* for the implementation of every of the four endpoints i wrote in the first post.

* total bounty is 120mBTC at today (18 apr 2014) USD price, bounty will be converted in fiat value when claimed. Expecting bitcoin rally in the next month
 

[mriou]

This may help too:

http://www.blockcypher.com/

We're aiming for close to no downtime and very low latency. The documentation is over there:

http://dev.blockcypher.com/

We're early on but have a solid infrastructure backing this up already. It should cover your 3 first endpoints, pushing a transaction is coming next week (04/21-23ish). Any feedback really appreciated too.

[xenoky]

This may help too:

http://www.blockcypher.com/

We're aiming for close to no downtime and very low latency. The documentation is over there:

http://dev.blockcypher.com/

We're early on but have a solid infrastructure backing this up already. It should cover your 3 first endpoints, pushing a transaction is coming next week (04/21-23ish). Any feedback really appreciated too.

Hi mriou,

thanks for the message. I tried your api and i would explain better my view.
Apart from the pushtx i could take all the information from your site instead/as alternative of blockchain but... There are differences.
Some may think the difference (http://goo.gl/dtJOCB, http://goo.gl/zIglsO) could by worked around with coding and that's true but it will be a lot easier if the request and response are identical for a subset of properties and apart from the domain and the first part of the url.

Look at this ideal pseudo code from an api client

Code:

Cache cache=new Cache();
HttpClient client=new HttpClient();
String apiUrl1="abc", apiUrl2="def";

function getBalance(String address) {
       String key=  "/q/balance/" + address;
       String balance=cache.get(key);
       if(balance==null) {
             try {
                  balance= client.get( apiUrl1 + key);
             } catch exception {
                  balance= client.get( apiUrl2 + key);
             }
             if(balance!=null)
                  cache.put(key , balance);
       }
       return balance;
}

This code has the problem even with the facts that apicoin is returning balance in btc and blockchain in satoshi with the same url (apart the domain name)
Things getting more complicated if object returned is a json that is marshalled into objects.

If we have to deal with ddos, standard API subset could really help

[xenoky]

Trying to drive standardization process from a github repo:

https://github.com/RCasatta/blockchain-common-api

feedback appreciated

[mriou]

Hey, sorry for the late reply, actually missed yours.

First, we've done some more work trying to get closer to the blockexplorer "standard" (what blockchain.info more or less uses). There's still a little more work left but it's definitely better in terms of JSON format. There's also a few things in the format we will never support just because the design is not scalable. For example:

$ curl http://blockchain.info/rawblock/299994

That's a 310k response. It's silly.

Second, regarding your proposal, I definitely think agreeing on a JSON schema is interesting. Endpoint addresses though... not so much. As long as there's a simple URL mapping, services should be free to pick URLs that match their own API schemes (I think).

[mriou]

I'd love to check you link and see what you guys have done but I don't seem to be very lucky at getting your site to load.

[gweedo]

I'd love to check you link and see what you guys have done but I don't seem to be very lucky at getting your site to load.

I can assure you we are very much up and loading extremely fast tonight. This image was taken from an external web screen shot service so I know it is up for other people. Also we have monitors checking in from Russia, Germany, Spain, NJ, IL,  and California, which say the site has not been down tonight. Also some of our bigger mission critical clients actually are directly connected to me via email, and their systems have not notified us.

If you want you can pm me your ip address, and we will look it up. We have had a lot of suspicious activity in our logs lately and have automatically had those ips ban from the service.

Please don't spread FUD about us.

[Abdussamad]

Others endpoints have some equivalent services but with completely different json format on blockr, biteasy, abe, apicoin...

Or i am missing something or i strongly believe standard base API for querying bitcoin blockchain would give great benefit to bitcoin-based services

Use object oriented programming. Create a base class with common elements and extend it for each specific API. This way you can use any of them without repeating code.

[xenoky]

Sorry for late reply, i missed yours...

First of all, thanks for the efforts, to me it's important for all the ecosystem.

Hey, sorry for the late reply, actually missed yours.

First, we've done some more work trying to get closer to the blockexplorer "standard" (what blockchain.info more or less uses). There's still a little more work left but it's definitely better in terms of JSON format. There's also a few things in the format we will never support just because the design is not scalable. For example:

$ curl http://blockchain.info/rawblock/299994

That's a 310k response. It's silly.

Second, regarding your proposal, I definitely think agreeing on a JSON schema is interesting. Endpoint addresses though... not so much. As long as there's a simple URL mapping, services should be free to pick URLs that match their own API schemes (I think).

Almost totally agree.
If the design has objective drawbacks it's right to point it out and not follow.
Agree that JSON format is more important than the endpoint addresses, anyway i cannot see drawback but just small advantages on choosing really the same. Developers could easily recognize already used endpoints from other API providers, and also if you are caching retrieved data by key, no need of remapping URLS to a unique scheme.

First off wrong api you are working on, we are standardizing block explorer api which is not meant to be a powerhouse of apis, but rather an easy to switch in time of need or attack to get low level data.

Why are you saying it's the wrong API? i personally need this endpoints! 
Joke apart, i don't think /q/ endpoints are enough even for a small bitcoin related services wich need transactions information.
Really a good start tough.

Use object oriented programming. Create a base class with common elements and extend it for each specific API. This way you can use any of them without repeating code.

Currently we have very minimal differences, one is that blockchain returns satoshis, and I don't believe that is the better choice over a full 8 decimals. Modern languages can read in strings and convert them easily to big decimals or even big integers.

Here you are missing my point, i and many others have enough capacity to parse different API formats. But this is avoidable work and less error prone with better standards

[xenoky]

First off wrong api you are working on, we are standardizing block explorer api which is not meant to be a powerhouse of apis, but rather an easy to switch in time of need or attack to get low level data.

Why are you saying it's the wrong API? i personally need this endpoints! 
Joke apart, i don't think /q/ endpoints are enough even for a small bitcoin related services wich need transactions information.
Really a good start tough.

Why should we implement such high level api calls, when we each can do it our own way on our services? Having very low calls would be the better choice, it allows for innovations at the high level. Also you are very much wrong, I have a client we just worked with to switch over their api calls to us and they were using it, it actually works fairly well if you just need to get balances and can respect the rate limit.

you were right if with low calls you can extract the same information...
I read your api doc and i can't find a way to retrieve the same information i get on https://blockchain.info/address/{BTCADDRESS}
happy to be wrong if it's not the case

Currently we have very minimal differences, one is that blockchain returns satoshis, and I don't believe that is the better choice over a full 8 decimals. Modern languages can read in strings and convert them easily to big decimals or even big integers.

Here you are missing my point, i and many others have enough capacity to parse different API formats. But this is avoidable work and less error prone with better standards

Exactly that is what we are trying to standardize... We need people to defend their implements as why it is the best...

Here are the alternatives I use
1. address, index, balance, etc: blockr.io
2. pushtx: eligius pool (Google will have enough info).

Standardization is a long way. Lets first get some real alternative up. Currently there are very few.

agree with both of you,
alternatives could help see which API are better, this is just to connect api developers and their users to see which are and why.

[gweedo]

First off wrong api you are working on, we are standardizing block explorer api which is not meant to be a powerhouse of apis, but rather an easy to switch in time of need or attack to get low level data.

Why are you saying it's the wrong API? i personally need this endpoints!  
Joke apart, i don't think /q/ endpoints are enough even for a small bitcoin related services wich need transactions information.
Really a good start tough.

Why should we implement such high level api calls, when we each can do it our own way on our services? Having very low calls would be the better choice, it allows for innovations at the high level. Also you are very much wrong, I have a client we just worked with to switch over their api calls to us and they were using it, it actually works fairly well if you just need to get balances and can respect the rate limit.

you were right if with low calls you can extract the same information...
I read your api doc and i can't find a way to retrieve the same information i get on https://blockchain.info/address/{BTCADDRESS}
happy to be wrong if it's not the case

Can you reiterate what you are trying to standardize? I am lost, because in your op sounds like you are trying to standardize the block explorer api.

Yes we have the same information, it is broken down differently due to the fact that users of my service, don't need the same information. Remember blockchain.info is taking that same page without the api call and giving it to the people that need it in a json format.

https://apicoin.io/api/v1/doc/#callbackscript
https://apicoin.io/api/v1/doc/#getaddress
https://apicoin.io/api/v1/doc/#getunspenttxforaddress

All three would get you basically the same information, but since most of our clients don't need all that information at once, for scalable we broke it down to 3 different calls.

[xenoky]

First off wrong api you are working on, we are standardizing block explorer api which is not meant to be a powerhouse of apis, but rather an easy to switch in time of need or attack to get low level data.

Why are you saying it's the wrong API? i personally need this endpoints!  
Joke apart, i don't think /q/ endpoints are enough even for a small bitcoin related services wich need transactions information.
Really a good start tough.

Why should we implement such high level api calls, when we each can do it our own way on our services? Having very low calls would be the better choice, it allows for innovations at the high level. Also you are very much wrong, I have a client we just worked with to switch over their api calls to us and they were using it, it actually works fairly well if you just need to get balances and can respect the rate limit.

you were right if with low calls you can extract the same information...
I read your api doc and i can't find a way to retrieve the same information i get on https://blockchain.info/address/{BTCADDRESS}
happy to be wrong if it's not the case

Can you reiterate what you are trying to standardize? I am lost, because in your op sounds like you are trying to standardize the block explorer api.

Yes we have the same information, it is broken down differently due to the fact that users of my service, don't need the same information. Remember blockchain.info is taking that same page without the api call and giving it to the people that need it in a json format.

https://apicoin.io/api/v1/doc/#callbackscript
https://apicoin.io/api/v1/doc/#getaddress
https://apicoin.io/api/v1/doc/#getunspenttxforaddress

All three would get you basically the same information, but since most of our clients don't need all that information at once, for scalable we broke it down to 3 different calls.

I would like to standardize what is needed to the most for a better infrastracture, sharing my personal experience as API caller. If i am the only one or just a few with this needings, then, i am agree that there is no need to standardize.

In your methods personally i miss, information about spent transaction for address. For unspent, getunspenttxforaddress is missing senders and receivers address.
I am not developing another block explorer, but the information i need is almost the same.

[gweedo]

I would like to standardize what is needed to the most for a better infrastracture, sharing my personal experience as API caller. If i am the only one or just a few with this needings, then, i am agree that there is no need to standardize.

In your methods personally i miss, information about spent transaction for address. For unspent, getunspenttxforaddress is missing senders and receivers address.
I am not developing another block explorer, but the information i need is almost the same.

Well then the next question do we want innovation or just the same api calls being used over and over? I personally would want a low level api call that can just do simple things, and let the higher level apis go where they must.

None of our clients have requested seeing spent transactions, so it has moved down our list of what to get done, but I will probably get it out soon As for unspent transactions again our clients use this to build transactions so they don't need that information, so it doesn't make sense to add it.

[gweedo]

In your methods personally i miss, information about spent transaction for address. For unspent, getunspenttxforaddress is missing senders and receivers address.
I am not developing another block explorer, but the information i need is almost the same.

I added the all transactions api call to our list. https://apicoin.io/api/v1/doc/#transactionsbyaddress

[xenoky]

i am developing the web site in my signature, the site needs aggregated blockchain data, to be not dependant on a single service i am now using many API providers.
This post is for sharing my experience and give my personal opinion (that by definition could be wrong)

I need 8 different methods, every one is implemented in at least two providers.
I am using the following API providers:

• http://blockchain.com
  
  • pros: has all of the 8 http methods needed available through their api
  • cons: the mean response time is not really good in comparison to the others, it has big variance especially in complex methods. The /address/ endpoint has block height for transaction but not block timestamp, so i need to perform extra http requests for complete information
• http://helloblock.io
  
  • pros: the best json format, the only one that has a single method with the same info as https://blockchain.info/address (it is better, has block time) 
  • cons: slow response time, it hasn't push transaction and websocket (this is probably ready, but "docs coming soon" is written in their reference so i can't use)
• http://biteasy.com
  
  • pros: one of the fastest, has websocket for new transactions
  • cons: with more than 4 request per seconds you get errors, because of the type of work i would have preferred something like 200 per minute. A method with the same information as https://blockchain.info/address/ is missing
• http://blockr.io
  
  • pros: fast, probably the best documentation, has push transaction
  • cons: if you are using json-to-object mapping, the format returned is not as good as helloblock because you need to build a different response object for every different request. A single method with the same information as https://blockchain.info/address/ is missing, the analogue http://btc.blockr.io/api/v1/address/txs/ requires to make another http call for every transaction
• http://blockexplorer.com
  
  • pros: it is open-source
  • cons: useful for simpler methods, slow and not reliable

Others API are missing because i am not using it for now: http://apicoin.io and http://chain.com for example

One feature for websocket that i would really like is filtering by regular expressions. I've to manage a lot of address so i can't ask to listen every of them.
This addresses are built with a initial string in common, for example "1G1".
Should be great if i can ask websockets to have updates just on address that match the provided regular expression, for example: "1G[1-9].*"


Many thanks to every API developers

[gweedo]

You should really give us a try https://apicoin.io, because a lot of the cons you put down for our services, is what we have build upon to make our service the best. We are actually the market leaders in the PaaS (platform as a service) api niche. We don't promote who uses us due to the fact we value privacy more than spike in traffic for a day.

[xenoky]

You should really give us a try https://apicoin.io, because a lot of the cons you put down for our services, is what we have build upon to make our service the best. We are actually the market leaders in the PaaS (platform as a service) api niche. We don't promote who uses us due to the fact we value privacy more than spike in traffic for a day.

It may be a small thing but i didn't till now because i need to setup a key for apicoin.io
Consider offer free test drive with limitations based on the ip like biteasy
Just an advice, i am adding apicoin to the list of providers even without not authenticated access

[gweedo]

You should really give us a try https://apicoin.io, because a lot of the cons you put down for our services, is what we have build upon to make our service the best. We are actually the market leaders in the PaaS (platform as a service) api niche. We don't promote who uses us due to the fact we value privacy more than spike in traffic for a day.

It may be a small thing but i didn't till now because i need to setup a key for apicoin.io
Consider offer free test drive with limitations based on the ip like biteasy
Just an advice, i am adding apicoin to the list of providers even without not authenticated access

We will consider it and run test to see how it does, but the api keys are industry standard, we want to be at that level and beyond it.

It is very simple to implement them, if you are using php, send me an email at admin@apicoin.io, I helped someone the other day to get it working. We are here to help, any questions or coding help shoot us an email, we never want to hear people wanted to try us but couldn't get over a certain aspect, because we can learn from that and hopefully make that barrier to entry to use us non-existence.