[ANN][XEL] Elastic Project - The Decentralized Supercomputer

[unvoid]

@EK we miss you

Arrived back home,
started to work (like properly working) now! Let me find out what's exactly happening and then I will prove a fix including a detailed explanation.

wb boos  

Now, it will be 12h daily for elastic again  

At the moment, my local node looks fine! Could someone point me into the right direction if there is some issue right now?
If not, I will find the issue anyway, it will just take longer. Will try to stress test the whole thing with a couple machines I think!

Welcome back @EK! Hope you had good time in Greece

You're back and my node isn't blacklisted anymore

But today I set nxt.myAddress to my external IP address and it seems that ban problem went away. Could this be related to the problem? If my node is announcing with different IP address I could be banned after forging too many blocks. What do you think?

Aaaaand it's gone. I'm on a fork again.

[1714214559]

1714214559

[1714214559]

1714214559

[1714214559]

1714214559

[1714214559]

1714214559

[hagie]

hagie, yes forging is based on coins! If you give me your address I can send you some more so your chances will increase.

Hi and thanks,

another problem I encontered is the "Could not verify signature (server side)." I can't register an account and I can't create any work (get the same error).

my address is : XEL-48LM-K27E-ZXWE-FFFF2 and you can check the node under https://elastic.cryptonodes.site

the only changes I made is in the nxt.conf :

Code:

nxt.myAddress=elastic.cryptnodes.site
nxt.myPlatform=linuxDebian8
nxt.apiServerHost=0.0.0.0
nxt.allowedBotHosts=*
nxt.adminPassword=wonttellhere

regards

[unvoid]

And finally I have nice exception in my logs! This should tell you a lot more @EK. Looks like it's my node that is banning.

Give me a min I'll upload log to some file hosting.

Also please take a look at my pull request to elastic-miner.

EDIT: My nxt.log http://s000.tinyupload.com/index.php?file_id=38360208295476988967

[Evil-Knievel]

Welcome back @EK! Hope you had good time in Greece

You're back and my node isn't blacklisted anymore

But today I set nxt.myAddress to my external IP address and it seems that ban problem went away. Could this be related to the problem? If my node is announcing with different IP address I could be banned after forging too many blocks. What do you think?

Aaaaand it's gone. I'm on a fork again.

Really? Maybe we can reproduce that?
Actually, most importantly, did your node blacklist all others, or did others blacklist your node? And if so ... why?
I think what we are experiencing is the very problem that we try to identify for days.

Do you have bitmessage? Or should I contact Lannister? I really think you deserve a couple BTC for your hard work and I am sure he will be more than happy to give you some.

EDIT: The pull request was pulled  Thanks

[Evil-Knievel]

And finally I have nice exception in my logs! This should tell you a lot more @EK. Looks like it's my node that is banning.

Give me a min I'll upload log to some file hosting.

What would be really cool is your nxt_test_db/ folder. Then we can set up your chain state on a different machine and check whats wrong. Maybe it's related to the 2 blocks height fork from a few blocks ago? At least your node does not accept transactions which are accepted by others.

I feeel we are very close. No worries, nxt_test_db/ does only include the public chain, no private stuff.

[unvoid]

And finally I have nice exception in my logs! This should tell you a lot more @EK. Looks like it's my node that is banning.

Give me a min I'll upload log to some file hosting.

What would be really cool is your nxt_test_db/ folder. Then we can set up your chain state on a different machine and check whats wrong.

nxt_test_db: http://s000.tinyupload.com/index.php?file_id=00301526502546875780
nxt.log: http://s000.tinyupload.com/index.php?file_id=38360208295476988967

[unvoid]

Do you have bitmessage? Or should I contact Lannister? I really think you deserve a couple BTC for your hard work and I am sure he will be more than happy to give you some.

I'll set up bitmessage today and contact @Lannister! Thanks.

[unvoid]

My miner isn't working for two days already (maybe you launched yours?). So it's strange that there are some PoW submissions since I think I was the only one who had working miner.

[Evil-Knievel]

Just tell him you participated is the not-yet-officially-existing bug bounty program and the consensus in the thread was 2 BTC per testnet crash ;and what he thinks about it :-)

Regarding your chain state! This is exactly it! This is the problem we have been seeing in the last days! Perfect, I think I will need a hour or two but I think we will identify whats going on soon ...

[Evil-Knievel]

My miner isn't working for two days already (maybe you launched yours?). So it's strange that there are some PoW submissions since I think I was the only one who had working miner.

Well I have just started a miner half an hour ago! This was the time when your node started blacklisting. At the same time there was a 2-block-height fork!
I suspect it must have something with a closed work, that was rolled back and not correctly reopened. Others then submit POW submissions, and your node thinks that the work is still closed while on the (new) fork it should be open. Something like this I think!

[unvoid]

@EK can you send me your bitmessage address? I'll send you passphrase to problem account. Every time I login to it I'm banned (or my node is banning) in 5-100 blocks so you'll have closer look at the problem.

[Evil-Knievel]

@EK can you send me your bitmessage address? I'll send you passphrase to problem account. Every time I login to it I'm banned (or my node is banning) in 5-100 blocks so you'll have closer look at the problem.

I can't remember it, I would have to start the other machine first.
But I have downloaded your nxt_test_chain, logged in with my account and started blacklisting as well. I think its not connected to a specific account but more to the blockchain in nxt_test_chain itself.

I am inspecting now what is different from a working chain. I'll be back to you soon!

[hagie]

Hi,

forging works here but I just forged 15+ blocks in a row - Did I cause a fork or are the other forging nodes down ?

Regards

[BigBoom3599]

Hi,

forging works here but I just forged 15+ blocks in a row - Did I cause a fork or are the other forging nodes down ?

Regards

I think the network is split up big time, I've just forged every block for the last 1-2 hours   

[BigBoom3599]

I've thrown up a VPS to hopefully stabilize the network a little bit more, as all the others you can access it here http://188.166.162.17:6876/index.html,

[hagie]

A restart of the node helps - now the blocks are distributed again ..

regards

[unvoid]

Hi,

forging works here but I just forged 15+ blocks in a row - Did I cause a fork or are the other forging nodes down ?

Regards

I think the network is split up big time, I've just forged every block for the last 1-2 hours   

@EK is working on it. I think that even if you delete db and resync sooner or later you'll be on a fork again. We need to wait to see what great mind of @EK came up with.

[Evil-Knievel]

The problem was that sometimes a block (which includes a work creation transaction) did not propagate quick enough while unconfirmed POW/Bounty submissions were already received from the mempool of different nodes. Since the node did not know the work yet, it assumed that the POW submission is malicious/malformed and blacklisted the node.

Suggested fix for blacklisting issue:
https://github.com/OrdinaryDude/elastic-reference-client/commit/df3177877687b79ef4a63bdb85c55c3afbd9619c

Problem:
This of course opens up a wide surface for mempool flooding attacks with made-up work submissions (where the software can not yet decide if they are malicious or not). NXT suffers from the same problem. We will have to find a solution for that here.

Important:
IF PROBLEMS PERSIST: You need to "rm -rf nxt_text_db/" and redownload the blockchain entirely since forks may have screwed up a lot in the last days.

Pending:
I still suspect one little issue in the block reorganization routine, the "flooding problem" from above as well as block reorganization is the next point on the agenda.

[tomkat]

The problem was that sometimes a block (which includes a work creation transaction) did not propagate quick enough while unconfirmed POW/Bounty submissions were already received from the mempool of different nodes. Since the node did not know the work yet, it assumed that the POW submission is malicious/malformed and blacklisted the node.

Suggested fix for blacklisting issue:
https://github.com/OrdinaryDude/elastic-reference-client/commit/df3177877687b79ef4a63bdb85c55c3afbd9619c

Problem:
This of course opens up a wide surface for mempool flooding attacks with made-up work submissions (where the software can not yet decide if they are malicious or not). NXT suffers from the same problem. We will have to find a solution for that here.

Important:
IF PROBLEMS PERSIST: You need to "rm -rf nxt_text_db/" and redownload the blockchain entirely since forks may have screwed up a lot in the last days.

Pending:
I still suspect one little issue in the block reorganization routine, the "flooding problem" from above as well as block reorganization is the next point on the agenda.

If submitting work cost something, then the submission cost should be enough to prevent flooding, no...?

[Evil-Knievel]

If submitting work cost something, then the submission cost should be enough to prevent flooding, no...?

Submitting work costs, but submitting a POW solutions or a bounty submission is a zero fee transaction, so even accounts with no XEL can start solving work right away.
I mean we could require a fee and then refund it once the tx confirms, but that would make it more difficult for new users to start earning XEL.

I will think a moment about this. Maybe we should work with the NXT devs here; I am almost sure that they suffer from the same issue. The difference is that their TX cost something, but as malicious TX will never confirm and eventually will be rolled back, attacking NXT does not cost the attacker anything after all. I am pretty sure you could DOS the NXT chain easily. We need a solution here.