[ANN][XEL] Elastic Project - The Decentralized Supercomputer

[Evil-Knievel]

Bump to Version 0.3.5

A thread safety issue was fixed, where the VM got crippled when two work packages were verified at the same time.
This occasionally caused to become blacklisted by others.

Fix: https://github.com/OrdinaryDude/elastic-core/commit/3a2e2e02c14c85fdae5e0486301f6e12fdef22cf

[1713906364]

1713906364

[1713906364]

1713906364

[1713906364]

1713906364

[hagie]

Bump to Version 0.3.5

A thread safety issue was fixed, where the VM got crippled when two work packages were verified at the same time.
This occasionally caused to become blacklisted by others.

Fix: https://github.com/OrdinaryDude/elastic-core/commit/3a2e2e02c14c85fdae5e0486301f6e12fdef22cf

Updated my public accessible node https://elastic.cryptnodes.site/

regards

[unvoid]

After update (wait >10min) and del nxt_test_db can't download full blockchain.

Code:

2016-09-19 18:39:16 FINE: Got 1 confirmations
2016-09-19 18:39:16 INFO: Downloaded 443 blocks in 7 s, 187 per s, 0 min left
2016-09-19 18:39:16 INFO: Finished blockchain download
2016-09-19 18:39:16 FINE: Cannot find any archive peers
2016-09-19 18:39:16 FINE: Remaining 12 pruned transactions
2016-09-19 18:39:19 INFO: Blockchain download in progress
2016-09-19 18:39:20 FINE: Blacklisting 52.28.123.208 because of: nxt.BlockchainProcessor$TransactionNotAcceptedException: Transaction is a duplicate, transaction 7598634667181063828 {"senderPublicKey":"964be53db0cc61dd465fa765b03651c1bb06c7ce6b738be739ee1f143ea1910f","attachment":{"version.WorkIdentifierCancellationRequest":1,"id":"190164203855439272"},"subtype":5,"amountNQT":0,"signature":"ac9f26ac4832382c25bec0605683220b1852d0ff423018eb36a3e23daa30f30c8f407cafc0075d5579242358c8e044376d094db715699feb5d4fd6d4044bed49","feeNQT":100000000,"ecBlockHeight":0,"type":3,"deadline":1440,"version":1,"timestamp":88850158,"ecBlockId":"445516706271790606"}
nxt.BlockchainProcessor$TransactionNotAcceptedException: Transaction is a duplicate, transaction 7598634667181063828 {"senderPublicKey":"964be53db0cc61dd465fa765b03651c1bb06c7ce6b738be739ee1f143ea1910f","attachment":{"version.WorkIdentifierCancellationRequest":1,"id":"190164203855439272"},"subtype":5,"amountNQT":0,"signature":"ac9f26ac4832382c25bec0605683220b1852d0ff423018eb36a3e23daa30f30c8f407cafc0075d5579242358c8e044376d094db715699feb5d4fd6d4044bed49","feeNQT":100000000,"ecBlockHeight":0,"type":3,"deadline":1440,"version":1,"timestamp":88850158,"ecBlockId":"445516706271790606"}
        at nxt.BlockchainProcessorImpl.validateTransactions(BlockchainProcessorImpl.java:1443)
        at nxt.BlockchainProcessorImpl.pushBlock(BlockchainProcessorImpl.java:1329)
        at nxt.BlockchainProcessorImpl.access$1200(BlockchainProcessorImpl.java:38)
        at nxt.BlockchainProcessorImpl$1.downloadBlockchain(BlockchainProcessorImpl.java:550)
        at nxt.BlockchainProcessorImpl$1.downloadPeer(BlockchainProcessorImpl.java:270)
        at nxt.BlockchainProcessorImpl$1.run(BlockchainProcessorImpl.java:175)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2016-09-19 18:39:20 FINE: Will process a fork of 718 blocks, mine is 0
2016-09-19 18:39:20 FINE: Rollback from block 7694898814333207672 at height 3319 to 7694898814333207672 at 3319
2016-09-19 18:39:20 FINE: Didn't accept any blocks, pushing back my previous blocks
2016-09-19 18:39:20 INFO: Finished blockchain download
2016-09-19 18:39:42 FINE: 6 block generators found
2016-09-19 18:39:48 INFO: Blockchain download in progress
2016-09-19 18:39:49 FINE: Blacklisting 108.61.177.169 because of: nxt.BlockchainProcessor$TransactionNotAcceptedException: Transaction is a duplicate, transaction 7598634667181063828 {"senderPublicKey":"964be53db0cc61dd465fa765b03651c1bb06c7ce6b738be739ee1f143ea1910f","attachment":{"version.WorkIdentifierCancellationRequest":1,"id":"190164203855439272"},"subtype":5,"amountNQT":0,"signature":"ac9f26ac4832382c25bec0605683220b1852d0ff423018eb36a3e23daa30f30c8f407cafc0075d5579242358c8e044376d094db715699feb5d4fd6d4044bed49","feeNQT":100000000,"ecBlockHeight":0,"type":3,"deadline":1440,"version":1,"timestamp":88850158,"ecBlockId":"445516706271790606"}
nxt.BlockchainProcessor$TransactionNotAcceptedException: Transaction is a duplicate, transaction 7598634667181063828 {"senderPublicKey":"964be53db0cc61dd465fa765b03651c1bb06c7ce6b738be739ee1f143ea1910f","attachment":{"version.WorkIdentifierCancellationRequest":1,"id":"190164203855439272"},"subtype":5,"amountNQT":0,"signature":"ac9f26ac4832382c25bec0605683220b1852d0ff423018eb36a3e23daa30f30c8f407cafc0075d5579242358c8e044376d094db715699feb5d4fd6d4044bed49","feeNQT":100000000,"ecBlockHeight":0,"type":3,"deadline":1440,"version":1,"timestamp":88850158,"ecBlockId":"445516706271790606"}
        at nxt.BlockchainProcessorImpl.validateTransactions(BlockchainProcessorImpl.java:1443)
        at nxt.BlockchainProcessorImpl.pushBlock(BlockchainProcessorImpl.java:1329)
        at nxt.BlockchainProcessorImpl.access$1200(BlockchainProcessorImpl.java:38)
        at nxt.BlockchainProcessorImpl$1.downloadBlockchain(BlockchainProcessorImpl.java:550)
        at nxt.BlockchainProcessorImpl$1.downloadPeer(BlockchainProcessorImpl.java:270)
        at nxt.BlockchainProcessorImpl$1.run(BlockchainProcessorImpl.java:175)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2016-09-19 18:39:49 FINE: Will process a fork of 718 blocks, mine is 0
2016-09-19 18:39:49 FINE: Rollback from block 7694898814333207672 at height 3319 to 7694898814333207672 at 3319
2016-09-19 18:39:49 FINE: Didn't accept any blocks, pushing back my previous blocks
2016-09-19 18:39:49 INFO: Finished blockchain download
2016-09-19 18:39:57 FINE: Unblacklisting 108.61.177.169
2016-09-19 18:39:57 FINE: Unblacklisting 52.28.123.208
2016-09-19 18:39:57 FINE: Unblacklisting 52.57.31.137
2016-09-19 18:40:01 FINE: Accepted new transaction 16121508601869408022

Stopping at 3319 block.

[Evil-Knievel]

Stopping at 3319 block.

Fixed by blacklisting a number of hard coded tx's on the testnet.

Code:

boolean isDuplicate(Transaction transaction, Map<TransactionType, Map<String, Integer>> duplicates) {
				
				// Whitelist few specific, broken TX on the testnet (TODO, FIXME, TOREMOVE)
				List<Long> valid = new ArrayList<Long>();
				valid.add(Convert.parseUnsignedLong("7598634667181063828"));
				valid.add(Convert.parseUnsignedLong("2649304177138953307"));
				valid.add(Convert.parseUnsignedLong("10947343728076427070"));
				valid.add(Convert.parseUnsignedLong("16510862602090746736"));
				valid.add(Convert.parseUnsignedLong("12295851544522530488"));
				valid.add(Convert.parseUnsignedLong("4910487200763544778"));
				if(valid.contains(transaction.getId()))
					return false;
}

They are a relict from the time when there was no semantic "duplicate tx" check in place (a check that finds duplicate work cancellations with different tx ids for example).
The hard coded fix should avoid the crash ... we have to take this whitelist out when we restart the net some time in the future.

[Evil-Knievel]

Fed up with the infinitely long list of work?
Then use the new pagination feature 

[Evil-Knievel]

In the difficulty plot above, by the way, you see the problem of the current difficulty retargeting approach -> it's oscillating!
We have to fix that!

[hagie]

Fed up with the infinitely long list of work?
Then use the new pagination feature 

Should I see my own miner in the "minign overview" ? In my case it's empty.

Regards

[Evil-Knievel]

hagie, this feature is not yet developed. Doing Starting that today, but yes ... you will see your own miner there.

[yusyus]

Evil-Knievel,
have your team Twitter account for Elastic? to track news about XEL development. tnx

[Evil-Knievel]

Evil-Knievel,
have your team Twitter account for Elastic? to track news about XEL development. tnx

Actually, I think everyone is doing his own stuff here!

One is doing the client, one is doing the website, another one is making a block explorer. We all ("we all is elastic itself, as people come and people go") have the idea behind elastic in mind, and transform our own interpretation of it into code    This means that there is no central authority telling us what to do, and there is also no central authority which gathers weekly status reports to post them on a PR platform.

But this is the fun thing, about Elastic! You can do whatever you want as long as the community accepts it (because if the community does not accept it, it would just ignore it and substitute the specific part with something else)!
So if you feel the need of a twitter page, you can always make one and start collecting updates which you publish there   There is nothing that would stop you from that!

[Redawn]

Evil-Knievel,
have your team Twitter account for Elastic? to track news about XEL development. tnx

there is a reddit page r/XEL

[tomkat]

Evil-Knievel,
have your team Twitter account for Elastic? to track news about XEL development. tnx

Actually, I think everyone is doing his own stuff here!

One is doing the client, one is doing the website, another one is making a block explorer. We all ("we all is elastic itself, as people come and people go") have the idea behind elastic in mind, and transform our own interpretation of it into code    This means that there is no central authority telling us what to do, and there is also no central authority which gathers weekly status reports to post them on a PR platform.

But this is the fun thing, about Elastic! You can do whatever you want as long as the community accepts it (because if the community does not accept it, it would just ignore it and substitute the specific part with something else)!
So if you feel the need of a twitter page, you can always make one and start collecting updates which you publish there   There is nothing that would stop you from that!

Would be good to see Lannister in action.
I mean a webpage, update of this thread, a promo campaign with signatures, etc.

[provenceday]

But we really need to think about the POW difficulty scaling.

Maybe we should organize a contest or so! The current scheme (inspired by the Dark Gravity Wave) obviously "sucks".
When you look at the past blocks, we are *faaaaar* from having a steady #10 tx per block, its more kind of oscillating, even though a miner is working constantly without much fluctuation in its computation power.

pls refer to AntiGravityWave (every block), that may inspire you.
https://github.com/viacoin/viacoin/tree/master/src

[provenceday]

Nice to see some great progress.

[Evil-Knievel]

But we really need to think about the POW difficulty scaling.

Maybe we should organize a contest or so! The current scheme (inspired by the Dark Gravity Wave) obviously "sucks".
When you look at the past blocks, we are *faaaaar* from having a steady #10 tx per block, its more kind of oscillating, even though a miner is working constantly without much fluctuation in its computation power.

pls refer to AntiGravityWave (every block), that may inspire you.
https://github.com/viacoin/viacoin/tree/master/src

The biggest problem is that we may have many many miners but no POW for several blocks when there is no work online. When then a new work package is submitted, the difficulty is low again causing many POW submissions to be broadcasted in bursts until the difficulty adapts again.

[HunterMinerCrafter]

Is the bounty program officially started yet?  What are the criteria?

I have a (growing) laundry list of problems that I see in the implementation...

Let's start with four "easy" bugs in the language parser and vm.  I'd like to see these cleaned up to make other analysis easier.

Starting from most simple:

 1) Conditionals must always have "else" statement.
   The parser allows:
  

Code:

if(true) {};

  but the worstWeight and interpreter will throw ArrayIndexOutOfBounds.
   This works:

Code:

if(true) {} else {};

  so you need to check the size of conditional statements' children nodes in the vm!

 2) Expressions allow mixed bool/int in parser but fail to cast in vm.
   For example:
  

Code:

if(m[0]) {} else {};

  throws a "ClassCastException: java.lang.Integer cannot be cast to java.lang.Boolean" in interpreter and:
  

Code:

m[0] = (true || false);

  fails similarly in trying to cast bool to int.
   Either you need to separate bool/int valued expressions in the grammar, or handle casting!
   I'm assuming the intent was to mimic C style int/bool implicit conversion, but this conflicts with the arguably silly Java semantics underlying the VM.
  
 3) Expressions are not IDs...
   However they are treated as such by the parser, which confuses the interpreter.
  

Code:

(1==0) = 2;

is a nonsense statement allowed by the parser, but resulting (for good reason) in a "ClassCastException: ElasticPL.ASTEQNode cannot be cast to ElasticPL.ASTId" in the interpeter.  This seems to be a general problem in the grammar, which does not correctly distinguish Expressions, PrimitiveExpressions, and IDs.  Statements such as this should be rejected in parse.
  
 4) Parser allows for arbitrarily large but "0 WCET" programs.
   There seem to be a few problems with the WCET, but this one is by far the "simplest" that I've seen so far.  For example, the statement:
  

Code:

   input 8;
   if(true) {
     repeat(2147483647) {
       SHA256 3 3;
     };
   } else {};
   verify true;
  

 is obviously a "very expensive" program, well above the statement limit constant, which implies that it should be rejected.  However, the WCET is calculated as 0, and I doubt anyone will ever find the bounty.  (Worse is what happens to the peer nodes if someone does "find" some work/bounty inputs and publishes a PoW/PoB, turning all peer nodes instantly into nothing more than hot/noisy sha mining rig running on a forked chain, hehe...)

(EDIT: Bug #4 is particularly problematic in that with "seemingly" no execution cost, all instances of the reference miner will immediately jump onto this work as the "most profitable."  Once you've knocked out all competition like this, you can proceed to claim all legitimate work/bounties for yourself.)

These bugs can be trivially used to disable peer nodes and/or miners.  I have more complete examples ready for each if you'd like to see a more detailed confirmation of any.

Let me know when you are ready for more...?

Also, I notice that you see now "some of" the problems with your PoW difficulty mechanism.  One or two of our problems "for later" are related to this.  One thing which I can say for sure is that the difficulty calculation will need to be made entirely per work order job, as some programs will execute at 200k evals/sec/cpu and some will be 2k evals/sec/cpu and (as things stand right now) some might be 0.0002 evals/sec/cpu. ;-)  (I don't have a "workable solution" for how to fix this PoW calculation aspects of your design.)

These are all the least "interesting" (and least fun) of the problems.  There seem to be several more, which I'll present in increasing order of "interestingness" assuming that they stick around, but let's see how handling these few easy issues goes first...

(Hey, at least it's not quite as badly broken as when your plan was "Python or Lua" eh?)

(Disclosure: I hold 0 XEL tokens on testnet or genesis roster.)

[HunterMinerCrafter]

P.S.
Can you put grammar source file to github?

Also, something else that I noticed - the JAR files in the github are not bit-identical to those distributed for the libraries elsewhere.  This could indicate malware or other tampering, and as such they should probably not be used by anyone at all.  (Delete them and download each jar from maven or whatnot if you plan to test this stuff!  Better yet rebuild them from source yourself ofc!!)

It would be great if a maven project file were made so that these binaries didn't have to be distributed with quite such a questionable chain of trust behind them? 

[hagie]

(Disclosure: I hold 0 XEL tokens on testnet or genesis roster.)

post your xel and I'll send some testnet tokens - or use the EK's public node to take some.

Regards

[coralreefer]

One thing which I can say for sure is that the difficulty calculation will need to be made entirely per work order job, as some programs will execute at 200k evals/sec/cpu and some will be 2k evals/sec/cpu and (as things stand right now) some might be 0.0002 evals/sec/cpu.

I tend agree with this assessment that the POW should be per work id.

My understanding is that the owner of the slow eval/sec work is supposed to provide larger POW payouts to entice miners to mine their work; however, I could see this having the opposite affect because of what HunterMinerCrafter wrote.  Regardless of how much the POW payout is of an extremely slow eval/sec work item is, with only 10 POW payouts per block (for all work ids combined) there would be a good chance that even if you found a POW result for the slow eval/sec work you wouldn't get paid as the fast eval/sec work is going to get evaluated 10x, 100x, etc times more often than this slow work and would likely consume most of the 10 POW payouts.

The other issue I see with this disparity between slow & fast work is that it seems very likely that the fast work will end up paying out all their POW rewards, but the slow difficult work may only pay out a small fraction of it due to this disparity between slow and fast work.  It would be great if we could figure out a way that the POW rewards get paid if there are miners working on that work id.

[HunterMinerCrafter]

post your xel and I'll send some testnet tokens - or use the EK's public node to take some.

Regards

If I come to need some, I can let you know.

Right now, testnet tokens are just "free" anyway, you can basically mine proof shares without actually doing the job's work... so if I need some coins I'll just "take" them from the network! 

(But that is another bug, for another day...)