Why im asking this is because im wondering about the best way to go with global optimisation policy.

Is it best to focus on udp asynchronous io, keep the code simple, because anyway as you said blockchain is mostly sequential and there is no real room for core scaling and parallelisation.

Or is it best to focus on something that can have true robust handling of parallele processing , even if it doesn't seem all that useful in general use case.

Ive been watching the code for a few month already, im more on blackcoin than bitcoin, but the two are still similar, and I dont see that much place where parallelisation can make huge diff.

Outside of the ibd, which still not a minor issue imo because it's a big road block toward centralisation of big blockchain, because it take too much resources and time to keep running a full node at home. Specially for casual users.

And beyond this, there is the question of "blockchain processing speed" in general.

Also when dealing with many nodes, maybe there can be room for core scaling, but from my experience the best to handle message based io is asynchronous interrupt based io, it's much better than threading a blocking io or using non blocking sockets. And I dont see a very good way to scale the processing that much.

Well clearly checking block headers hashes from get header request could be scaled, computing tx hashes from a block too.

Tx processing it make sense if many tx has to be processed by batch, which involve increased latency in the processing, even with higher throughput, which im not sure is better vs lower latency in the processing to have tx processed as it come with lowest latency even if processing 100 tx one/one will take more time than waiting to have the 100 tx to process them in a batch.

For ibd clearly it make sense, because it doesn't matter if 2 year old blocks are processed with zero latency, so synching by batch to me anyway seem to give only benefit.

Cause the question to me is for eg I have 10M/s downlink, there is virtually infinite number of node that have the data, it should take 1h to synch. Why it takes 3 day ??

And even going in more advanced thinking of batch processing from multiple node instead of logic of syncing one block / one block  im pretty sure there are things to do.

We are not trading robots either

We are not like ascendency

Already we already have a working code base

https://github.com/iadix/purenode

So you can see there is a real application framework with memory allocation with reference pointer ( mean safe memory ) https://github.com/iadix/purenode/blob/master/libcon/base/mem_base.c

dynamic data tree https://github.com/iadix/purenode/blob/master/libbase/include/tree.h

code for sockets, http server,https://github.com/iadix/purenode/blob/master/libbase/include/http.h

module

https://github.com/iadix/purenode/tree/master/mod_maker



and already big part of coin logic implemented.

https://github.com/iadix/purenode/blob/master/block_adx/block.c

https://github.com/iadix/purenode/blob/master/stake_pos3/kernel.c


Staking & transaction signing in the browser.

https://github.com/iadix/purenode/blob/master/export/web/keys.html ( need to open it via the node, it will not work opening it like this from the browser )


The ico date we gave at first is i guess too early, so we are still in stage of early annouce, and many things can still be worked on, and we are also still working on organising the escrow properly.

But what I can tell in term of distributed application is that the raytracing is a good way to test the framework, because it can have server side rendering either with software sse multi thread, or with opengl if it's present on the node host, or it can transmit vectorial scene data in json or binary float array, to be renderered on the client via html5 glsl, the node is only computing current keyframes and matrixes of the scene object server side.

Like this it's easy to see how dynamic data can be distributed across several nodes, and either renderered in a client browser in html5, or rendered on the node server side and the browser download a png.

Raytracing is specially interesting because it scale very easily, there is very little data to be shared, and each fragment of the image can be computed separatly by different node, to test maybe clustered raytracing or such. And it can give relatively good visual result for simple object with reflections refraction and shadow, without needing lot of data. Only bitmap textures  need to be stored off chain, but it's not big problem. And raytracing can make use of texture generator who need very little initial data.

Associated with chain based scene data and real time interaction via html5 js application can give good idea of what blockchain can do in term of real time performance when it's not tied to block reward logic.

But for the moment we are still focused on organising things for the ico, but still many things to see, and I guess the site design could be improved too

Html5 is not about css

Html5 is about real time interaction on dynamic data in the browser.

What matter on the raytracing page is :

The browser update scene data from the node live to a glsl texture from binary float data

Full scene with rotation matrixes updated live in json

The camera and light position can be controlled in the browser in javasript

Materials can be edited live in the browser

Works on all recent browser on windows and android ( at least )



Latter object position and material can be changed on the blockchain using in browser key signature and rpc api.



The layout is not great but for the moment I have others things to do than settings up font size and site colors if that's what your html guru is teaching you, you should get another one


But latter, symphony or joomla plugin can be worked on to stick on a joomla to have better template system to look more html5 guru ^^

One potential feature I dont talk about is related to easily up dating as all the coin logic is in cross platform binary modules, it could be done in sort to automatically update modules of all nodes when certain block or signal is encountered, even if it can pose security issues, it would need some kind of central authority to sign modules that are to be used after a certain block, or would require users to manually check modules signature against accepted module emitter, but if a way to work around the central authority module emitter is found, it could be used to ease hard fork procedure. Not than we plan to do hard fork every two day, but in case it can be useful feature.

Then only the txins checking need to be done sequentially, can maybe try to find sequences of tx without child,  not sure what would be the average, but this could be done without checking the scripts in a first pass, and then do the scaled script execution by batch. Script data is not influenced by the out come of previous transaction is it ?

 

The only thing that is really sequential is the txins checking, and as far as i know, a tx in a block cant have input from a tx in the same block, so on a per block basis that shouldn't be a problem ?

Computing block headers hashes and tx hashes doesn't require previous block validation.


 if all the new blocks to be batched are already in memory, the script data is also available , either from the previously stored blocks, or the new ones to be batch checked.

As long as all the data from the tx are in memory, or stored on hard drive ( validated) , the sig scripts can be checked in parallele no ? Outside of the initial data from the tx, which are available in memory and constant,they are self contained, the outscrig signature is constant as well as the txins data, so shouldn't be a pb for scaling.

The only thing is that it mean blocks have to be rejected by batches too, if one block in the batch have bad tx. But maybe not all the work spent on valid blocks can be wasted.

Blocks with bad headers hashes would be rejected in the first pass. Maybe last valid block can be returned.


After either being optimistic or pessimistic on ratio of bad block, can do certain check in firsts passes to potentially eliminate bad blocks quicker not to waste batch of long operations.

Maybe you can check block explorer code to get the logic, they pretty much do what you are talking about

Increasing block size wont scale anything. You get scaling when two processes can work in the same time without waiting on the other.

With blockchain it would mean two or more blocks could be mined in the same time without waiting or depending on each other.

Increasing block size or decreasing block time increases throughtput but doesn't increase scaling.

 The only one valid next block concept prevent true scaling , as all miners/stakers need to synch on the next block, and cant "advance" The blockchain each on their side.

To have true scaling it would require to distribute tx across miners for that they can each mine a different block in the same time. But with current blockchain principle that cannot happen as only one of the two block would get accepted.

Having a double chain system would allow cheap scaling.

To me the only thing that truly prevent mass scaling is the competition for block reward. Without this, many secure scaled system can be thought of.

The logic of coin emission control via block reward make it difficult to scale easily, as the total coin supply need to be controlled on the whole network, if a consistent emission curve is to be maintained.

I guess when cap is reached and emission curve flaten, scaling become easier.

I dont know if accurate profiling data are available somewhere from a recent source tree , with the compiler version, boost version, options etc to have accurate informations on this.

If there is no such test done already, I May get into it


Im a programmer not a trader, im not too much into speculation, I prefer Turing


And with threading and parallelisation with batch processing, if it scale well, it can divide total processing time , and db batch write is supposed to decrease a lot total processing time on 100 or 1000 writes.

Things like computing block headers hashes , tx hashes should scale very well.

Processing sig scripts a priori i dont see why it wouldn't scale well. As far as i can tell scripts are self contained, they dont make reference to data outside of themselves. As long as public keys and sig and script data is loaded in memory from the tx, I dont see why the script checking wouldnt scale.

If algorithm can be made for open cl for hash computation & sig check on 1000 block batch that would make substantial difference not sure how the total script op code execution would fit with open cl, but for simple p2sh it could be worth it, à good proportion of transactions are p2sh or regular multi sig without complex script.


Merkkle root computation can't be scaled easily I think, the check on the txins can probably be scaled per block.

Bittrex is one of the option we are reviewing along side with the others persons mentionned in this thread

Trading and speculation profit is not the only use of blockchain either

My friend was supposed to answer something yesterday , but apparently didn't

We are currently in discussion with several persons, including a known member of this forum to find the best solution for the escrow, it takes time to do it correctly, and probably the ico will be delayed of a week or two, the time we set up everything properly,  we are reviewing differents options for promotion and escrow at the momen, we will give more information when all is set up

We know trust and proper escrow are important and we are reviewing the different options available

To me it still look like some sort of batch processing could speed up ibd.

Like processing blocks by batch of 100

Pass 1 , threaded check of blocks headers hashes , and computation of tx hashes

Pass 2 non threaded check of merkkle root

Pass 3 check of tx in/out , potentially threaded on per block basis

Pass 4 threaded check of all transactions sig scripts

Pass 5 batch write of the blocks


I have in the idea it could improve synch time no ? Maybe im missing dependency issue, or something, or it already been thought of and tested, or it could miss something in the check, but to me it looks it could improve synch time quite signifciantly.

From what I can see, it doesn't seem there is threading for intra block transaction checking, or that it try to use avx or sse , there are Indeed internal function for 256 bits arithmetics, but it looks it's not especially made to exploit simd or parallelisation.

I will probably do some testing at some point, also to evaluate performance of storing engine , and see what take the more time with the block processing, and experiment with udp.

It doesn't look like it try to exploit db batch writing either which could probably have significant impact on synchronisation speed.


I saw a blog earlier they said they made test using Zlib or other to compress block and they were saying 20 to 30% gain.


https://bitco.in/forum/threads/buip010-passed-xtreme-thinblocks.774/


Datastream compression: Testing various compression libraries such as LZO-1x and Zlib have shown it is possible to further reduce block and transaction sizes by 20 to 30% without affecting response times which could also be applied to thinblocks.

Also maybe having udp version of certain request could improve performance, as it's still in the over all message based protocol rather than stream based, and data integrity is already verified in the applications layer, not sure the impact it would have.

Where can I find more information on this ?

Another thing imo which would increase bitcoin adoption is possibility to have on chain meta data associated with addresses, or group of adress, in sort that a sense of legal or moral entity could emerge from the blockchain and the possibility to specialize account like PayPal with merchent and customer type of address or account, to know a bit more of who is what and have more information on what the transaction represent in a higher level, to make things more transparent and "humanly readable".

Detachment.

The way i see it, the developpement world is divided between two "factions" which are what I would call "hackers" people who are into network protocol, crypto, system administration , unix etc, and the other faction is domestic software like vidéo game and desk top application.

To me bitcoin clearly come the hacker faction, so it has well designed protocol and crypto, security, but it lacks completly ergonomy, good system integration, clear interface, and things that make intuitive and safe looking to average computer users, instead of being presented as wall of hash with cryptic nov lang, with software that is not always bug free, and isnt really packaged like you would expect of traditional application, no real install procédure or explanation on anything, need to edit config file by hand, and it's not very ergonomic all together.

Scaling is also another issue, but I think lack of ergonomy and clarity in the applications , often dispatched between miner, wallets, masternode, and things are not really tighly packaged together with clear manual and minimum of garantee In the functioning, is the main thing that hinder wider adoption by "the mass".