Short dev update here. Just to give you an idea of whats going on behind the scenes.
As we know Svante has been out on a trip this week taking some well deserved rest from the past hectic year, yet still combining that with meetings with potential future HEAT partners and customers.
We'll hopefully be hearing from our refreshed HEAT CEO soon.
Meanwhile while Svante was enjoying his trip, we have been working on other parts.
As mentioned before, in networking whether thats inter-machine (peer 2 peer) but also user to machine (HEAT API - broadcast transactions and lookup of blockchain state [most recent trade, order, etc]) probably the biggest bottle neck is the encoding and decoding of messages. In HEAT and in some form at least every other cyptocurrency these messages are almost exclusively in JSON, which is very flexible but also very costly to decode and encode.
So there is this dilemma, JSON is flexible and simple to use but bloated and slow to process. And on the other hand you have binary data which is the most optimal/minimal representation of data and the absolute fastest to both encode and decode, but its not flexible at all. Binary data consists of a list of bytes (8 fields each consisting of a 1 or a 0), each field in your message following the last one.
Such binary data is super efficient and can speed up processing and transfer costs hundred folds depending on the exact data of course.
However once we hard code our client, heat-sdk, microservice message or heat server to handle this one specific data structure in this highly optimized list of bytes, well thats where it ends. Dont try and update the structure (add one field, or change a fields type) since thats a nightmare. Also you would have to write hard to both write and maintain code for each separate app that handles these complicated binary data, on top you have to hand code how to both read and write that data.
For us it became clear what we need here. Which is of course the flexibility of JSON but as optimized as pure binary data.
This is especially important when we want to start using structured messages to send to microservices, but having to store them on chain.
So that and other things simply have to be done efficiently, while interoperability is almost as important.
Well we've made progress, we're happy to yesterday have committed both the heat server implementation and with heat-sdk the universal impl which runs on your server, browser and mobile and which gives us a cross language, cross device binary encoding solution with the flexibility that JSON gives you. The schemas you need to share among two end points in order to read and write each others structured binary messages is actually JSON itself.
To get an idea of how we define these data structures you can already look in the development branch on the public heat-sdk repo,
https://github.com/Heat-Ledger-Ltd/heat-sdk/blob/avro/src/types.ts.
The whole commit for the full thing can be seen here
https://github.com/Heat-Ledger-Ltd/heat-sdk/commit/6808cafdd2d6f0eb6dca161f474ee5a4e7d44316?diff=unified.
The server code update also involved a RPC solution where you can communicate with heat server over a websocket but using a binary transport.
As mentioned on the bitcointalk thread before we did a test where we used a binary always open websocket connection and a set of pre-generated binary representations of transactions to be broadcasted and we found that we could already do 2000 transactions per second that way. That finally showed our custom storage backend being put to good use.
Well now we've completed the server code and formalized the mechanism and added support for server, mobile and browser cross device but compatible binary message transport, all through our heat-sdk.
We are writing tests for heat-sdk and plan to release what will be version 0.10 already so we can get to the long awaited 2000tps public test which will run from your browser.
After that microservices are up for finalization where this schema based binary protocol will play a key role in how we abstract both microservice invocations and the automatic generation of UI/forms to interact with microservices.
--
Our binary schemas are based on
https://en.wikipedia.org/wiki/Apache_Avro, while on the server side we use the official java libs. In heat-sdk we' ve rolled our own implementation in part by extracting just the encode/decode parts from avsc js lib while making that one run in both the browser and in nodejs.
