HEAT Technical Progress

Yeah, what about rewarding us with a piece of info or an update for being so civilized and patient? Please?
 

End of November is coming really quick

That's true. Working on it!

Quite some changes have been applied since our last update.

The most significant probably being us dropping the original NXT based API server and replacing it with a completely new API implementation and server that is now compatible with the https://www.openapis.org/ standard. The compatibility to the openapi standard was achieved  by annotating all java API code and integrating the swagger/jersey libs into HEAT. Now when you run HEAT you can ask it for its API spec (which is an auto-generated JSON document) and which is understood by all tooling providers who adhere to the openapi standard.

There are two main purposes to being openapi compatible.

• provide detailed API documentation to 3rd party developers
• provide fully operational client libs in virtually any programming language available

To get an idea of whats that like please checkout an early draft of HEAT API documentation http://alpha.heatledger.com/api/ and our interactive HEAT playground which allows you to try out the full API straight from your browser which is here http://alpha.heatledger.com/interactive/.

The client libs are basically fully typed language specific wrappers around the HEAT API, we'll be hosting those on our developer section of our website and when downloaded allows any developer to use HEAT from their own project in their language of choice. As said the number of supported languages is very large, to name a few (.NET, Java, C++, Python, Node.JS etc etc) have look here to see the number of supported languages https://github.com/swagger-api/swagger-codegen/tree/master/modules/swagger-codegen/src/main/resources

Another large addition was replacing the asset exchange with our own implementation, the new asset exchange supports asset to asset and heat to asset orders, basically allowing anyone to create any trading pair they'd wish. A lot of thought had to be put into this to make it operate both at high speeds and to use very little memory (we succeeded in doing this). Not having a database in the HEAT core was not making things easier since it required us building a custom indexing solution so order matching can happen as fast as possible.

The problem with indexing (pre-sorting orders) was solved by making a hybrid data structure that only keeps the order ids in RAM, these ids are 8 byte identifiers and are kept in plain java arrays, while offloading all other data to the off-heap mmap disk based storage only requesting that on demand.
To give an impression of memory cost required imagine a moment in the future where we have 1,000,000 open orders on the blockchain. This would cost us 8MB of RAM which I believe is not that much. Even further down the road we should still be able to support 100 million open orders since this costs just 100 times more RAM or approx 800MB, still perfectly fine and still possible to run from most desktop PC's.

Our new asset exchange now also comes with the concept of instant order matching which is done by introducing the concept of an unconfirmed order balance. What this means is that instead of just matching orders when the new block comes in (this is how nxt does it) we now match orders when they come in as unconfirmed transactions. This basically allows for instant trades and updates of orders. Since such functionality makes little sense if we dont match unconfirmed orders against other unconfirmed orders and dont take into account that unconfirmed orders could be cancelled by unconfirmed order cancellations. There really was no use in re-using the nxt based asset exchange a rewrite was the only option here.

If HEAT would have been running in a controlled private chain environment with only dedicated and known miners the above exchange functionality would have been sufficient to fully rely on unconfirmed transactions only and present those to outside users as basically instantly confirmed. On the public HEAT chain this of course is not the case and there always remains the chance of one node having to re-adjust its internal state which was based of unconfirmed orders and having to bring it in line with the new block he/she just received with perhaps a slightly different ordering of orders which leads to different trades etc.
To bring this level of instant confirmation to the public chain I'm leaning strongly to creating a very different p2p networking and interop model, one that is smarter and one which uses strong cryptocraphic identities for all nodes, also one where each node on the network is in direct and constant contact to each other node on the network. Luckily this is possible and could be achieved when we switch from the jetty networking libs to http://akka.io/.

We already had the concept of replication in HEAT testnet and even in FIMK way before that, for anyone new to this replication is basically offering a powerful (detailed and indexed) SQL database model which allows you to inspect the blockchain contents. But make it optional to the consensus mechanism. This means that you could run a HEAT node and only use it for mining or POP participation and not bother with any of database inserts/indexing overhead (if you look here http://alpha.heatledger.com/api/ you'll notice the mention of "only available if replicator is enabled"). For HEAT testnet we had the replication logic in an external Scala app which updated a Mysql database which in turn holds the detailed blockchain data.

By rewriting the replication logic as a light weight (and making it support multiple databases) addition to the HEAT core and integrating it with the new API we believe we have a better maintainable solution. Also adding bundle based solutions has become much simpler and more maintainable. Bundles being binary messages attached to transactions and which are interpreted by the replication layer and applied to the database. One example of this is the HEAT keystore (basically the same functionality as ALIASES in nxt) the big difference here being that in the case with aliases all nodes around the world would have to carry all aliases and have them indexed and ready and that the consensus mechanism would not even function without everyone having access to all aliases ever created. (same goes for things like digital goods store). What we did with the bundle technology was basically introduce a light weight form of transaction that still offers the same security and decentralized aspects as an alias transaction would in nxt. But without the overhead cost of not being able to prune those away and forcing all nodes to keep those aliases around for ever. Keystore is an extremely simple example of this, all it supports are simple key+value combos (see http://alpha.heatledger.com/interactive/#!/KeyStore/keyStorePut and http://alpha.heatledger.com/api/#keystoreget for its API) but the same mechanism applies for much more complex and possibly multi-step inserts and updates giving us nothing less than the basis for a fully blockchain based database or basically any business backend.

To get an idea of how easy it is to use the replicator/bundle functionality have a look at the full KeyStore implementaion here com.heatledger.replicate.KeyStoreReplicator

Thank you for the hard work and update. Although much of the tech escapes me it seems like we are on the verge of something big. Really big. I'm happy for all the ppl that have a chance to watch this unfold.

1. Core status?

Core now also consists of asset-to-asset exchange, this was added recently on top of payments which were already functional.
Testing is done with a technique i call mirrorring, its very similar to how payments used to be tested. The tests for payments were not sufficient for testing orders, unconfirmed order matching and also supporting rollbacks and testing blockchain reorgs.
 
For this the test code really had to support some way that mimics the way the storage backend supports rollbacks to previous versions AND support transactional operations where either all or none of the changes during a storage transaction were commited to the data storage.
The mirroring mode can be switched on through the app configuration and has separate logic that keeps a one to one clone of the blockchain but in memory, all balances of accounts and assets and orders plus unconfirmed balances are kept track of. But with separate code that does all the calculations itself and does not rely on the code that does this for the main heat balances.

This way when you start heat mirrored and you do a transaction, the same changes to the sender and recipient account balance and unconfirmed balance that happen in the heat storage also are applied to the in memory balances. This supports rollbacks and any possible restores in case an error occurs and a transactional storage operation must be unwound. If you add to the mix that after each operation you compare the heat balance storage with the in memory clone of the balances - you can basically confirm that at least the heat balance updating logic and the code that does the same thing in the mirroring parts are actually in sync.

What remains now are two things. A. Find a way to throw as much and as different as possible transactions at heat while running in mirrored mode (the more difficult the better since you'd ensure all possible code paths are handled). B. Make sure to manually go over the code that does the mirroring analyse it as best as possible and make sure its sound. That last part is actually much easier as compared to manually going over the code that does the same balance handling for in the heat parts. That's because the mirrored code is just a fraction of the size of the code for heat balance tracking - only the most essential parts are there.

When changes are made to heat core code - new transactions added - new asset classes introduced - pop rewards get enabled etc.. We now have a built in diagnostics mechanism, not very much unlike ethereum uses several clients written in several languages that also perform their self-diagnostics.

To assist in sending as much and as different as possible transactions to heat we added the concept of apps which are javascript/nodejs/npm modules you can reference on the command line and have heat load those in the java js vm. Helper classes are injected into those scripts that allow you to easily send transactions, generate blocks, etc. Works really easy. Its not the same as microservices which are also scripts but meant to run and respond to events on the live network. Apps are run instead of the main blockchain, you can assign their own blockchain and configurations through their package.json file. Purpose of apps are either running tests or doing analisys of the blockchain, app scripts run from start to finish and shutdown. Microsservices keep running in the background for ever.

2. Asset exchange status?

The replication parts (detailed optional database model - kept in real-time sync) have been completed and operational. This involves all trade history and all order history which is not needed for consensus and thus has no part in heat core. These replicated parts are now also accessible from the api, use them to list trades and orders and search for them etc.


3. Email account identifiers

To support human readable account identifiers instead of forcing users to remember and exchange numerical account ids we have completed the account identifier parts. What we wanted is a mechanism where users can safely use their own private email addresses as their account identifiers - yet not have a trace of that actual email address on the blockchain. This comes down to user a registering id "", yet not leaving a trace on the blockchain. Yet when some other user sends a payment to "" the payment will go to the correct account. If you wish so you can register your account identifier to be public on the blockchain. It is also possible for another account to assign an identifier for your account - but only after you provide a signed messages, signed with your private key which the assigning account has to provide.

The storage of account ids happens not as actual textual identifiers but as 8 byte longs, very similar to how we generate numeric account ids from the much larger public keys. The range of unsigned 8 byte longs is 18,000,000,000,000,000,000 or 18 billion billion. Which we think is more than enough to support most users unique names yet have have very little to no false positives.

To be able to support account auto completion in the client the replication layer has functionality to store all non-private identifiers and make them searchable. Assigning or announcing names for accounts happens transparantly very similar to announcing public keys and can be done at no extra cost when sending a transaction.

.

the following minimum information should be provided: 1) The real identity of the asset issuer, 2) Some proof of asset backing (for example a bitcoin account address, or similar). In addition, users/investors should be able to write comments about the asset and give a score (1 to 5).

Nice ideas, some of such additional features may indeed be on an external web site operated by third parties. The HEAT core and wallet however are barebones what comes to qualifying and sorting assets. Decentralized markets inherently allow releasing any kind of assets, being quality agnostic.

I also think that assets should be automatically be deleted if there is no trading activity, for example in one year, to reduce spam.

This kind of measures need to be carefully considered as they may be technically not compatible with a decentralized system. The third party evaluation sites ie. service aggregators are the place that can always do such sorting.

Spam is likely to become a larger problem on the HEAT platform compared to NXT.

I'm supposing by spam you mean lower quality content on blockchain. That's quite different from spam - the blockchain is capable of containing such content and it's not the task of the core to apply quality criteria. This will be the task of the service aggregators.

If there are N assets, then I assume that the number of possible trading pairs will grow proportional with N^2?

Probably only asset issuers can allow pricing against and valuation in asset, exact procedures on that aren't yet in place.

Excellent updates ! Happy New Years ! 2017 will be a big year for everything crypto & HEAT will be officially born

We've now completed basic AE testing and are now proceeding with the AE UI and implementation of the BTC gateway. The current testnet will likely be scrapped without much further warning within the coming week (as notified in the welcome message of every new account) so please save any important data you have there such as private messages.

The complexity of the BTC gateway and how quickly we're able to implement that, and to what extent, defines the exact launch date this month. We're aware of the pressure to launch ASAP. The building blocks that are the very core of HEAT need to be in place for successful launch, in order to provide the needed liquidity for trading. Hang on!

We've now completed basic AE testing and are now proceeding with the AE UI and implementation of the BTC gateway. The current testnet will likely be scrapped without much further warning within the coming week (as notified in the welcome message of every new account) so please save any important data you have there such as private messages.

The complexity of the BTC gateway and how quickly we're able to implement that, and to what extent, defines the exact launch date this month. We're aware of the pressure to launch ASAP. The building blocks that are the very core of HEAT need to be in place for successful launch, in order to provide the needed liquidity for trading. Hang on!

For the average user, does anyone need to do anything special for this transition?
Other than if they want to back up personal messages?

For the average user, does anyone need to do anything special for this transition?
Other than if they want to back up personal messages?

Nope.