Bitcoin Forum
May 02, 2024, 03:18:22 PM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 2 3 [4] 5 6 7 8 9 10 »  All
  Print  
Author Topic: Nxt source code analysis (QA)  (Read 14053 times)
ImmortAlex (OP)
Hero Member
*****
Offline Offline

Activity: 784
Merit: 501


View Profile
January 04, 2014, 04:08:43 PM
 #61

One more thought about memory usage.
I see a lot of getBytes().length calls on Transaction and Block. Especially in Transaction.compareTo() this can create a lot of temporaty byte arrays in short time. But length can be easily and much faster calculated without creating array. I think, even for any Attachment.
Activity + Trust + Earned Merit == The Most Recognized Users on Bitcointalk
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
Jean-Luc
Sr. Member
****
Offline Offline

Activity: 392
Merit: 250



View Profile WWW
January 04, 2014, 06:19:48 PM
 #62

Here are a few more thoughts:

(1) This is a minor thing, but i don't think you're checking for overflow in any of the request parameters:
Code:
			int timestamp = ((Long)transactionData.get("timestamp")).intValue();
short deadline = ((Long)transactionData.get("deadline")).shortValue();
byte[] senderPublicKey = convert((String)transactionData.get("senderPublicKey"));
long recipient = (new BigInteger((String)transactionData.get("recipient"))).longValue();
int amount = ((Long)transactionData.get("amount")).intValue();

You're casting the value to a Long and then to an integer, but not checking if truncation occurred. [I would probably move the validation to the Transaction or a transaction factory function. You can also move the validation of positive and valid amounts and fees here].
I have added a few more checks for the values of Transaction.amount and Transaction.fee in 0.5.0. I agree the validation checks are all over the place and need to be cleaned up.
Quote
(2) You're locking around every call to setBalance / setUnconfirmedBalance, which makes it easy to call the functions incorrectly if you forget to call them from within a lock. IMO, it would make sense to move some of that locking into the functions themselves so they can't be called incorrectly. [you can also change these functions to incrementXyz, since that's what you appear to be doing.
This was already on my list of things to fix, and I have fixed in in 0.5.0, but thanks for noticing.

Quote
(3) Instead of a giant doGet function, imo, it would make sense to consider a chain of responsibility where you move each potential action into a separate object
I know, just no time to do that yet.

lead Nxt developer, gpg key id: 0x811D6940E1E4240C
Nxt blockchain platform | Ardor blockchain platform | Ignis ICO
Jean-Luc
Sr. Member
****
Offline Offline

Activity: 392
Merit: 250



View Profile WWW
January 04, 2014, 06:21:51 PM
 #63

One more thought about memory usage.
I see a lot of getBytes().length calls on Transaction and Block. Especially in Transaction.compareTo() this can create a lot of temporaty byte arrays in short time. But length can be easily and much faster calculated without creating array. I think, even for any Attachment.
Not fixed yet, but already I have a //TODO: cache? on top of those getBytes() methods. And indeed need to do it for caching the length.

lead Nxt developer, gpg key id: 0x811D6940E1E4240C
Nxt blockchain platform | Ardor blockchain platform | Ignis ICO
ImmortAlex (OP)
Hero Member
*****
Offline Offline

Activity: 784
Merit: 501


View Profile
January 05, 2014, 04:47:52 AM
 #64

Okey, let's talk about one more TODO in you list Smiley
The blocks, transactions and accounts cache, and how much it will cost to kill our RPi nodes.

As we know since the beginning, blocks.nxt and transactions.nxt is just a result of standard ObjectInputStream work.
At startup they are fully loaded to memory and are there till the end.
So, NRS keeps full blockchain in memory all the time.

Let's do rough calculations. Class Block:
Code:
        int version;
        int timestamp;
        long previousBlock;
        int numberOfTransactions;
        int totalAmount, totalFee;
        int payloadLength;
        byte[] payloadHash;
        byte[] generatorPublicKey;
        byte[] generationSignature;
        byte[] blockSignature;
        byte[] previousBlockHash;
        int index;
        long[] transactions;
        long baseTarget;
        int height;
        long nextBlock;
        BigInteger cumulativeDifficulty;
This is from 0.5.0 actually (sorry for decompilation), but there's almost no difference with 0.4.7.
With 32-bit pointers and no alignment this gives us about 320 bytes for block without transactions, plus 8 byte for each transaction. 64-bit JRE with memory alignment can easily expand this to ~400 bytes. I can't tell you exact numbers, because it heavily depends on JRE implementation, operating system and so on.
Right now we have about 33000 block, which occupies about 11Mb of memory. I think, I should speak "at least 11Mb". Without accounting of long[] transactions...

Okey, take a look at class Transaction:
Code:
        byte type, subtype;
        int timestamp;
        short deadline;
        byte[] senderPublicKey;
        long recipient;
        int amount, fee;
        long referencedTransaction;
        byte[] signature;
        Attachment attachment;

        int index;
        long block;
        int height;
156 bytes without attachement (f.e. alias). Up to 200 in real world. 65000 transactions for now. More than 10Mb of memory. Plus 65000*8 = 0.5Mb of links from blocks to transactions (or we can assume that size of transaction is 164 bytes).

Actual things is even worse because blocks and transactions is not linear lists, but hash maps.
Actual things can be better, because of generatorPublicKey and senderPublicKey can be shared between blocks, transactions and accounts.

I'm bit tired and don't want to calculate size of accounts - it's too small comparing to blocks and transactions.

Next step is to calculate cost of attack.
Attacker can send 255 transactions in block. Let's use simple ordinary payment, 1 NXT amout, 1 NXT fee, 510 NXT for full block. 320+156*255 = 40100 bytes of memory.
According to guide, JRE in RPi can use up to 430Mb for heap. Okey, let's assume, that 401Mb is available for blocks and transactions. 401M / 40100 = 10000. Yes, just ten thousand blocks, and you RPi is dead. Or 5M NXT. F*ckin five million coins.

This is very rough calculations. Very. My intuition tell me, that things are worse actually. Cheeper. Faster.
And then, after RPi, this attack will kill cheap 2Gb VPS's. And then - 32-bit JRE's, which actually hardly can get 2Gb of heap.
And before that we will hear the song of death from our HDD/SSD, tired to save that .nxt files everytime you get new fat block.

Solution?
I think devs know it and has in TODO list Smiley We need to use some embedded DB and cache in memory only what we need right now: couple of last blocks.
msin
Legendary
*
Offline Offline

Activity: 1470
Merit: 1004


View Profile
January 05, 2014, 05:58:48 AM
 #65

Thanks ImmortAlex, please keep up the excellent analysis.
ImmortAlex (OP)
Hero Member
*****
Offline Offline

Activity: 784
Merit: 501


View Profile
January 05, 2014, 06:17:29 AM
Last edit: January 05, 2014, 06:38:23 AM by ImmortAlex
 #66

Kind of Post Scriptum to previous analysis.

1. Java and it's unpredictable object's size, nice article in russian (sorry, guys Smiley ).
2. For those who don't look deeper: 10000 blocks is just 10 days in current rate (AFAIK, we're still not getting desired 1440 blocks per day). Ten days to kill big part of our decentralized network. And ten days more to kill the rest.
3. There will be short-time paradise for miners Smiley 255 coins every minute. Wow.
4. I believe that if such kind of attack will start, devs fix the problem in one or two day, so it is actually not a reason to sell you coins. Just some funsy math. Do you remember how fast C-f-B redo NRS from self-written http code to Jetty? Moving blockchain to DBMS is the same, more or less.
mczarnek
Hero Member
*****
Offline Offline

Activity: 527
Merit: 500


View Profile
January 05, 2014, 07:50:54 AM
Last edit: January 05, 2014, 08:11:54 AM by mczarnek
 #67

About what I figured.. though I am surprised that it's all in one file and you are saying that only one person can work on it at a time.

It may take more time right now but I say fix any big bugs so people don't lose their money, then first thing you should do is refactor the code, split it up, and allow multiple people to work on it.

I read as much as I could about the idea behind Nextcoin and if you guys pull this off, most of the people reading this will make a lot of money off of it, a few years from now.. this fixes all the major problems with Bitcoin and I think it's genius.  Seriously this is by far the best idea for a crypto I've seen yet.

But it needs a lot of work to get there.. and in order to do so, I feel like you are going to need a team and to subversion it.

BitSend ◢◤Clients | Source
www.bitsend.info
█▄
█████▄
████████▄
███████████▄
██████████████
███████████▀
████████▀
█████▀
█▀












Segwit | Core 0.14 | Masternodes
XEVAN | DK3 | Electrum soon
Bitcore - BTX/BTC -Project












BSD -USDT | Bittrex | C.Gather | S.Exchange
Cryptopia | NovaExchange | Livecoin
Litebit.eu | Faucet | Bitsend Airdrop













████
 ████
  ████
   ████
    ████
     ████
      ████
       ████
        ████
       ████
      ████
     ████
    ████
   ████
  ████
 ████
████

████
 ████
  ████
   ████
    ████
     ████
      ████
       ████
        ████
       ████
      ████
     ████
    ████
   ████
  ████
 ████
████
Meizirkki
Hero Member
*****
Offline Offline

Activity: 616
Merit: 500



View Profile
January 05, 2014, 08:00:24 AM
 #68

ImmortAlex deserves a flood of donations for his work Smiley
Come-from-Beyond
Legendary
*
Offline Offline

Activity: 2142
Merit: 1009

Newbie


View Profile
January 05, 2014, 08:06:05 AM
 #69

ImmortAlex deserves a flood of donations for his work Smiley

Agree. 10K sent to the account in his signature.
mczarnek
Hero Member
*****
Offline Offline

Activity: 527
Merit: 500


View Profile
January 05, 2014, 08:29:55 AM
 #70

Good thinking, thanks for your work ImmortAlex!

Also maybe not the place for it, but a suggestion, what if say 1% of all the transaction fees went into a specific wallet to be used for paying people to develop the code?

Seems to me like it'd be nice to support the developers, and when we have our own Nextcoin foundation similar to the Bitcoin foundation, we could have automatic funds to be always developing and improving the currency, so Nextcoin always has some funds to improve itself.  And there are so many directions this idea could be taken!  Nextcoin is going to be amazing some day.

Something you'd have to have people vote on of course..  or maybe a better solution that I all around like better, put an optional radio button in every forger's wallet that says donate O-0% O-1%(default?) O-2% O-3% O-4% of all my NXT forged to the 'Nextcoin Foundation'.

Short term, I think it'd bring some peace of mind to people that these problems will be fixed and you'll be able to pay people to fix them and have a stronger development team behind the project.  Because I'll be honest the code scares me a little seeing how much work is needed to really compete against other cryptos.

Though I suspect the current developers are largely initial investors, who do still have a bunch of NXT of their own at this point to provide them with some incentive Smiley

BitSend ◢◤Clients | Source
www.bitsend.info
█▄
█████▄
████████▄
███████████▄
██████████████
███████████▀
████████▀
█████▀
█▀












Segwit | Core 0.14 | Masternodes
XEVAN | DK3 | Electrum soon
Bitcore - BTX/BTC -Project












BSD -USDT | Bittrex | C.Gather | S.Exchange
Cryptopia | NovaExchange | Livecoin
Litebit.eu | Faucet | Bitsend Airdrop













████
 ████
  ████
   ████
    ████
     ████
      ████
       ████
        ████
       ████
      ████
     ████
    ████
   ████
  ████
 ████
████

████
 ████
  ████
   ████
    ████
     ████
      ████
       ████
        ████
       ████
      ████
     ████
    ████
   ████
  ████
 ████
████
ImmortAlex (OP)
Hero Member
*****
Offline Offline

Activity: 784
Merit: 501


View Profile
January 05, 2014, 08:58:21 AM
 #71

ImmortAlex deserves a flood of donations for his work Smiley

Agree. 10K sent to the account in his signature.
Cпacибo бoльшoe! Пpигoдитcя.
ImmortAlex (OP)
Hero Member
*****
Offline Offline

Activity: 784
Merit: 501


View Profile
January 05, 2014, 09:04:20 AM
 #72

Also maybe not the place for it, but a suggestion, what if say 1% of all the transaction fees went into a specific wallet to be used for paying people to develop the code?
Developers interested theirself to develop Nxt software and rise interest to Nxt because it rises cost ot their existing Nxt funds.
I don't think that any additional fee is needed. Donations are welcome as usual, but no more.
FrictionlessCoin
Legendary
*
Offline Offline

Activity: 868
Merit: 1000


Cryptotalk.org - Get paid for every post!


View Profile
January 05, 2014, 12:43:01 PM
 #73

For people who are completely dismayed with the poor quality of the NXT code base and believe they can do better,  then please join this new project:  https://bitcointalk.org/index.php?topic=398461.msg4305172

Your input will be welcome.

 
                                . ██████████.
                              .████████████████.
                           .██████████████████████.
                        -█████████████████████████████
                     .██████████████████████████████████.
                  -█████████████████████████████████████████
               -███████████████████████████████████████████████
           .-█████████████████████████████████████████████████████.
        .████████████████████████████████████████████████████████████
       .██████████████████████████████████████████████████████████████.
       .██████████████████████████████████████████████████████████████.
       ..████████████████████████████████████████████████████████████..
       .   .██████████████████████████████████████████████████████.
       .      .████████████████████████████████████████████████.

       .       .██████████████████████████████████████████████
       .    ██████████████████████████████████████████████████████
       .█████████████████████████████████████████████████████████████.
        .███████████████████████████████████████████████████████████
           .█████████████████████████████████████████████████████
              .████████████████████████████████████████████████
                   ████████████████████████████████████████
                      ██████████████████████████████████
                          ██████████████████████████
                             ████████████████████
                               ████████████████
                                   █████████
.CryptoTalk.org.|.MAKE POSTS AND EARN BTC!.🏆
FrictionlessCoin
Legendary
*
Offline Offline

Activity: 868
Merit: 1000


Cryptotalk.org - Get paid for every post!


View Profile
January 05, 2014, 12:50:13 PM
 #74

Kind of Post Scriptum to previous analysis.

1. Java and it's unpredictable object's size, nice article in russian (sorry, guys Smiley ).
2. For those who don't look deeper: 10000 blocks is just 10 days in current rate (AFAIK, we're still not getting desired 1440 blocks per day). Ten days to kill big part of our decentralized network. And ten days more to kill the rest.
3. There will be short-time paradise for miners Smiley 255 coins every minute. Wow.
4. I believe that if such kind of attack will start, devs fix the problem in one or two day, so it is actually not a reason to sell you coins. Just some funsy math. Do you remember how fast C-f-B redo NRS from self-written http code to Jetty? Moving blockchain to DBMS is the same, more or less.

That's why bitcoind uses leveldb.  It would be joke to keep the entire block chain in memory.

So you are saying this NXT keeps the entire block chain in memory?  

Pathetic.

I mean, they likely haven't done any analysis of what kind of database to use as a backend.   There is no way that they'll be able to fix this problem in time for folks to realize that they can spam the network and blow up every JVM that sits on this network!

Well... clock is ticking....  waiting on the dbms implementation....

Also,  realize that they use Java serialization to store the transactions in file....  

So when these JVMs go 'out of memory error'... they can't save the transactions file.... and even if they do... that'll blow up!

HILARIOUS!

DO I GET THE PRIZE FOR SEEING THE FATAL FLAW!




 
                                . ██████████.
                              .████████████████.
                           .██████████████████████.
                        -█████████████████████████████
                     .██████████████████████████████████.
                  -█████████████████████████████████████████
               -███████████████████████████████████████████████
           .-█████████████████████████████████████████████████████.
        .████████████████████████████████████████████████████████████
       .██████████████████████████████████████████████████████████████.
       .██████████████████████████████████████████████████████████████.
       ..████████████████████████████████████████████████████████████..
       .   .██████████████████████████████████████████████████████.
       .      .████████████████████████████████████████████████.

       .       .██████████████████████████████████████████████
       .    ██████████████████████████████████████████████████████
       .█████████████████████████████████████████████████████████████.
        .███████████████████████████████████████████████████████████
           .█████████████████████████████████████████████████████
              .████████████████████████████████████████████████
                   ████████████████████████████████████████
                      ██████████████████████████████████
                          ██████████████████████████
                             ████████████████████
                               ████████████████
                                   █████████
.CryptoTalk.org.|.MAKE POSTS AND EARN BTC!.🏆
FrictionlessCoin
Legendary
*
Offline Offline

Activity: 868
Merit: 1000


Cryptotalk.org - Get paid for every post!


View Profile
January 05, 2014, 12:56:35 PM
Last edit: January 05, 2014, 01:10:21 PM by FrictionlessCoin
 #75

Next step is to calculate cost of attack.
Attacker can send 255 transactions in block. Let's use simple ordinary payment, 1 NXT amout, 1 NXT fee, 510 NXT for full block. 320+156*255 = 40100 bytes of memory.
According to guide, JRE in RPi can use up to 430Mb for heap. Okey, let's assume, that 401Mb is available for blocks and transactions. 401M / 40100 = 10000. Yes, just ten thousand blocks, and you RPi is dead. Or 5M NXT. F*ckin five million coins.


You can just use one NXT and bounce it around the network!

In fact, since everything is all HTTP based.  Someone can write a script.

Create 1000 accounts
Loop 10,000 times
    Loop through each account
        Send 1 IXC to each account




 
                                . ██████████.
                              .████████████████.
                           .██████████████████████.
                        -█████████████████████████████
                     .██████████████████████████████████.
                  -█████████████████████████████████████████
               -███████████████████████████████████████████████
           .-█████████████████████████████████████████████████████.
        .████████████████████████████████████████████████████████████
       .██████████████████████████████████████████████████████████████.
       .██████████████████████████████████████████████████████████████.
       ..████████████████████████████████████████████████████████████..
       .   .██████████████████████████████████████████████████████.
       .      .████████████████████████████████████████████████.

       .       .██████████████████████████████████████████████
       .    ██████████████████████████████████████████████████████
       .█████████████████████████████████████████████████████████████.
        .███████████████████████████████████████████████████████████
           .█████████████████████████████████████████████████████
              .████████████████████████████████████████████████
                   ████████████████████████████████████████
                      ██████████████████████████████████
                          ██████████████████████████
                             ████████████████████
                               ████████████████
                                   █████████
.CryptoTalk.org.|.MAKE POSTS AND EARN BTC!.🏆
Come-from-Beyond
Legendary
*
Offline Offline

Activity: 2142
Merit: 1009

Newbie


View Profile
January 05, 2014, 01:04:30 PM
 #76

DO I GET THE PRIZE FOR SEEING THE FATAL FLAW!

Sorry, no. All data is supposed to be kept in virtual memory and be accessible via memory mapped files. This is what "blockchain.nrs" for.

Bitcoin already got a very serious problem coz of using 3rd party data storage software (leveldb). U should read more than u write. Wink
Come-from-Beyond
Legendary
*
Offline Offline

Activity: 2142
Merit: 1009

Newbie


View Profile
January 05, 2014, 01:18:29 PM
 #77

Your ignorance completely surprises me.

If you allocate 1 GB for your JVM instance,  then 1GB is all you have.   Now that data can indeed go into virtual memory (i.e. disk),  but it still is confined to 1GB.




U can map terabytes into this 1 GB and OS will does its best to feed the data as fast as possible.
FrictionlessCoin
Legendary
*
Offline Offline

Activity: 868
Merit: 1000


Cryptotalk.org - Get paid for every post!


View Profile
January 05, 2014, 01:20:24 PM
 #78

DO I GET THE PRIZE FOR SEEING THE FATAL FLAW!

Sorry, no. All data is supposed to be kept in virtual memory and be accessible via memory mapped files. This is what "blockchain.nrs" for.

Bitcoin already got a very serious problem coz of using 3rd party data storage software (leveldb). U should read more than u write. Wink

Still not convinced you have a handle on your memory problem just because you are using Java nio.   Did you happen to have coded a memory caching algorithm so that your java code knows when to take and replace what's in your memory mapped file?  I doubt it,  that would be akin to writing your own object oriented database.

 
                                . ██████████.
                              .████████████████.
                           .██████████████████████.
                        -█████████████████████████████
                     .██████████████████████████████████.
                  -█████████████████████████████████████████
               -███████████████████████████████████████████████
           .-█████████████████████████████████████████████████████.
        .████████████████████████████████████████████████████████████
       .██████████████████████████████████████████████████████████████.
       .██████████████████████████████████████████████████████████████.
       ..████████████████████████████████████████████████████████████..
       .   .██████████████████████████████████████████████████████.
       .      .████████████████████████████████████████████████.

       .       .██████████████████████████████████████████████
       .    ██████████████████████████████████████████████████████
       .█████████████████████████████████████████████████████████████.
        .███████████████████████████████████████████████████████████
           .█████████████████████████████████████████████████████
              .████████████████████████████████████████████████
                   ████████████████████████████████████████
                      ██████████████████████████████████
                          ██████████████████████████
                             ████████████████████
                               ████████████████
                                   █████████
.CryptoTalk.org.|.MAKE POSTS AND EARN BTC!.🏆
Come-from-Beyond
Legendary
*
Offline Offline

Activity: 2142
Merit: 1009

Newbie


View Profile
January 05, 2014, 01:20:56 PM
 #79

Create 1000 accounts
Loop 10,000 times
    Loop through each account
        Send 1 IXC to each account

U mean NXT I believe.

PS: Aye, let's make miners happy!
FrictionlessCoin
Legendary
*
Offline Offline

Activity: 868
Merit: 1000


Cryptotalk.org - Get paid for every post!


View Profile
January 05, 2014, 01:22:01 PM
 #80

Your ignorance completely surprises me.

If you allocate 1 GB for your JVM instance,  then 1GB is all you have.   Now that data can indeed go into virtual memory (i.e. disk),  but it still is confined to 1GB.




U can map terabytes into this 1 GB and OS will does its best to feed the data as fast as possible.

You can feed the data into the memory mapped file,  but can you pull it out?

Duh!!! Triple face palm!

 
                                . ██████████.
                              .████████████████.
                           .██████████████████████.
                        -█████████████████████████████
                     .██████████████████████████████████.
                  -█████████████████████████████████████████
               -███████████████████████████████████████████████
           .-█████████████████████████████████████████████████████.
        .████████████████████████████████████████████████████████████
       .██████████████████████████████████████████████████████████████.
       .██████████████████████████████████████████████████████████████.
       ..████████████████████████████████████████████████████████████..
       .   .██████████████████████████████████████████████████████.
       .      .████████████████████████████████████████████████.

       .       .██████████████████████████████████████████████
       .    ██████████████████████████████████████████████████████
       .█████████████████████████████████████████████████████████████.
        .███████████████████████████████████████████████████████████
           .█████████████████████████████████████████████████████
              .████████████████████████████████████████████████
                   ████████████████████████████████████████
                      ██████████████████████████████████
                          ██████████████████████████
                             ████████████████████
                               ████████████████
                                   █████████
.CryptoTalk.org.|.MAKE POSTS AND EARN BTC!.🏆
Pages: « 1 2 3 [4] 5 6 7 8 9 10 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!