Quite some changes have been applied since our last update.
The most significant probably being us dropping the original NXT based API server and replacing it with a completely new API implementation and server that is now compatible with the
https://www.openapis.org/ standard. The compatibility to the openapi standard was achieved by annotating all java API code and integrating the swagger/jersey libs into HEAT. Now when you run HEAT you can ask it for its API spec (which is an auto-generated JSON document) and which is understood by all tooling providers who adhere to the openapi standard.
There are two main purposes to being openapi compatible.
- provide detailed API documentation to 3rd party developers
- provide fully operational client libs in virtually any programming language available
To get an idea of whats that like please checkout an early draft of HEAT API documentation
http://alpha.heatledger.com/api/ and our interactive HEAT playground which allows you to try out the full API straight from your browser which is here
http://alpha.heatledger.com/interactive/.
The client libs are basically fully typed language specific wrappers around the HEAT API, we'll be hosting those on our developer section of our website and when downloaded allows any developer to use HEAT from their own project in their language of choice. As said the number of supported languages is very large, to name a few (.NET, Java, C++, Python, Node.JS etc etc) have look here to see the number of supported languages
https://github.com/swagger-api/swagger-codegen/tree/master/modules/swagger-codegen/src/main/resourcesAnother large addition was replacing the asset exchange with our own implementation, the new asset exchange supports asset to asset and heat to asset orders, basically allowing anyone to create any trading pair they'd wish. A lot of thought had to be put into this to make it operate both at high speeds and to use very little memory (we succeeded in doing this). Not having a database in the HEAT core was not making things easier since it required us building a custom indexing solution so order matching can happen as fast as possible.
The problem with indexing (pre-sorting orders) was solved by making a hybrid data structure that only keeps the order ids in RAM, these ids are 8 byte identifiers and are kept in plain java arrays, while offloading all other data to the off-heap mmap disk based storage only requesting that on demand.
To give an impression of memory cost required imagine a moment in the future where we have 1,000,000 open orders on the blockchain. This would cost us 8MB of RAM which I believe is not that much. Even further down the road we should still be able to support 100 million open orders since this costs just 100 times more RAM or approx 800MB, still perfectly fine and still possible to run from most desktop PC's.
Our new asset exchange now also comes with the concept of instant order matching which is done by introducing the concept of an unconfirmed order balance. What this means is that instead of just matching orders when the new block comes in (this is how nxt does it) we now match orders when they come in as unconfirmed transactions. This basically allows for instant trades and updates of orders. Since such functionality makes little sense if we dont match unconfirmed orders against other unconfirmed orders and dont take into account that unconfirmed orders could be cancelled by unconfirmed order cancellations. There really was no use in re-using the nxt based asset exchange a rewrite was the only option here.
If HEAT would have been running in a controlled private chain environment with only dedicated and known miners the above exchange functionality would have been sufficient to fully rely on unconfirmed transactions only and present those to outside users as basically instantly confirmed. On the public HEAT chain this of course is not the case and there always remains the chance of one node having to re-adjust its internal state which was based of unconfirmed orders and having to bring it in line with the new block he/she just received with perhaps a slightly different ordering of orders which leads to different trades etc.
To bring this level of instant confirmation to the public chain I'm leaning strongly to creating a very different p2p networking and interop model, one that is smarter and one which uses strong cryptocraphic identities for all nodes, also one where each node on the network is in direct and constant contact to each other node on the network. Luckily this is possible and could be achieved when we switch from the jetty networking libs to
http://akka.io/.
We already had the concept of replication in HEAT testnet and even in FIMK way before that, for anyone new to this replication is basically offering a powerful (detailed and indexed) SQL database model which allows you to inspect the blockchain contents. But make it optional to the consensus mechanism. This means that you could run a HEAT node and only use it for mining or POP participation and not bother with any of database inserts/indexing overhead (if you look here
http://alpha.heatledger.com/api/ you'll notice the mention of "only available if replicator is enabled"). For HEAT testnet we had the replication logic in an external Scala app which updated a Mysql database which in turn holds the detailed blockchain data.
By rewriting the replication logic as a light weight (and making it support multiple databases) addition to the HEAT core and integrating it with the new API we believe we have a better maintainable solution. Also adding bundle based solutions has become much simpler and more maintainable. Bundles being binary messages attached to transactions and which are interpreted by the replication layer and applied to the database. One example of this is the HEAT keystore (basically the same functionality as ALIASES in nxt) the big difference here being that in the case with aliases all nodes around the world would have to carry all aliases and have them indexed and ready and that the consensus mechanism would not even function without everyone having access to all aliases ever created. (same goes for things like digital goods store). What we did with the bundle technology was basically introduce a light weight form of transaction that still offers the same security and decentralized aspects as an alias transaction would in nxt. But without the overhead cost of not being able to prune those away and forcing all nodes to keep those aliases around for ever. Keystore is an extremely simple example of this, all it supports are simple key+value combos (see
http://alpha.heatledger.com/interactive/#!/KeyStore/keyStorePut and
http://alpha.heatledger.com/api/#keystoreget for its API) but the same mechanism applies for much more complex and possibly multi-step inserts and updates giving us nothing less than the basis for a fully blockchain based database or basically any business backend.
To get an idea of how easy it is to use the replicator/bundle functionality have a look at the full KeyStore implementaion here
com.heatledger.replicate.KeyStoreReplicator
