NXT :: descendant of Bitcoin - Updated Information

[ChuckOne]

Does this mean that we are actually generating checkpoint files on all the nodes all the time? If so, how much work would it be to have newly installed nodes query for the most recent db files and start with that, instead of redownloading the entire blockchain. Of course, must be peer verified checksums, etc.

I don't understand what you mean by checkpoint file. The rest is a bad idea.

I think he means that nxt.h2.db could be considered a snapshot file.

[mcjavar]

Alias assignment being undone? I didn´t touch the client when this message came.

It is not because of something you did. The block that had to be popped off contained an alias assignment transaction. When a block is popped off, an attempt is made to undo all transactions in it. But this type of transaction cannot be undone, and in this case a full rescan of the blockchain is needed.

Thank you for clarification. How can a block "popp off"?

[greyw00lf]

du says: 257480 for current nxt_db folder

What? Mine is 111M only.

Err:

Code:

pi@raspberrypi ~/nxt $ du -h nxt_db/
252M	nxt_db/

Wrong usage of du?

I have:

Code:

pi@rpi ~/nxt $ du -h nxt_db/
246M	nxt_db/

But i always copied the folder from one version to the next... maybe that's why?

[greyw00lf]

Have you overclocked your Raspi?

No, i don't know if i will do that, i don't have anything to cool it (no heat sink...) and my pi is in an ordinary plastic-case.

[jl777]

Does this mean that we are actually generating checkpoint files on all the nodes all the time? If so, how much work would it be to have newly installed nodes query for the most recent db files and start with that, instead of redownloading the entire blockchain. Of course, must be peer verified checksums, etc.

I don't understand what you mean by checkpoint file. The rest is a bad idea.

It might be a bad idea, but the problem of how will small machines be able to keep up with 1000TPS volumes is unsolved.
checkpoint file would be "entire state of NXT blockchain" at specific intervals. I am assuming you are not storing the entire blockchain now that we have DB,but maybe that is incorrect?

Almost all NXT functions only need the most recent 1440 blocks, so if each machine kept those 1440 blocks locally, they can purge them after that. blockchain FIFO.

As the blockchain grows, it will take longer and longer for initial install. By having the entire state of NXT every hour, or even every day, in a decentralized peer validated form, would allow starting from the most recent hour (or day) and then start processing blocks from that point.

These two are totally independent technically, but combine to allow small raspis to continue forging and being full nodes
If checkpoint files are a bad idea, I can live with that. Is blockchain FIFO a bad idea too?

[Jean-Luc]

Code:

pi@raspberrypi ~/nxt $ du -h nxt_db/
252M	nxt_db/

Wrong usage of du?

No. I guess the database needs vacuuming once in a while. I will do that at shutdown, it means stopping the server will take slightly longer.

[greyw00lf]

Thanks! I am reluctant about the overclocking, but I run it with -856xmx ...

What does -856xmx do?

It is the memory available to the jvm, I quoted it wrong:

ps -ax
 2040 ?        SNl   61:45 /usr/bin/java -Xms128m -Xmx856m -jar start.jar STOP.PORT=7873 STOP.KEY=0815


-Xms128m is the minimum that is reserved at the start, and -Xmx856m is the maximum it can have, in megabytes. I often had crashes with a maximum smaller 450MB, but with giving it almost all the memory, it runs quite good.

Except that dl'ing the whole blockcahin is quite some venture.

i use -Xms320m -Xmx480m
how can you give it 856m? Don't you have a pi with 512MB? Or are you using swap? Or an other device?

[Jean-Luc]

Thank you for clarification. How can a block "popp off"?

With a little help from a stronger block that kicks him out

Only one block can be at the end of the chain, but multiple nodes try to generate blocks at the same time. The better one wins, if the other was attached first it gets popped off.

[Jean-Luc]

I am assuming you are not storing the entire blockchain now that we have DB,but maybe that is incorrect?

Almost all NXT functions only need the most recent 1440 blocks, so if each machine kept those 1440 blocks locally, they can purge them after that. blockchain FIFO.

If checkpoint files are a bad idea, I can live with that. Is blockchain FIFO a bad idea too?

Of course I store the whole blockchain, this is what the db contains.

You seem to be talking about blockchain trimming or pruning. This is in the plans, but not that soon.

Downloading the whole db will take longer than just the blockchain.

[mcjavar]

Thank you for clarification. How can a block "popp off"?

With a little help from a stronger block that kicks him out

Only one block can be at the end of the chain, but multiple nodes try to generate blocks at the same time. The better one wins, if the other was attached first it gets popped off.

Thank you!
Who decides (and how) which block is "better"?

[ChuckOne]

Complexity of signing is O(1).

That is plain wrong. The more bits you have to sign the longer signing takes. The question is: how long in terms of the input length.

We would have O(n) if Sign(32_bytes) == Sign(64_bytes) / 2. Actually we have Sign(32_bytes) == Sign(3200_bytes) / 2, so it's more like O(1). Right?

Sorry, still wrong.

It looks more like O(log n) but don't think so, as the whole input has to be read. That still takes O(n).

[ChuckOne]

Code:

pi@raspberrypi ~/nxt $ du -h nxt_db/
252M	nxt_db/

Wrong usage of du?

No. I guess the database needs vacuuming once in a while. I will do that at shutdown, it means stopping the server will take slightly longer.

As long as there is feedback in the nxt.log. No problem.

Would be great if you could add something like a progess counter to scanning, too. Maybe:
10%
20%
30%
...

Just to see, that it's still working and not running for nothing.

[ChuckOne]

Thank you for clarification. How can a block "popp off"?

With a little help from a stronger block that kicks him out

Only one block can be at the end of the chain, but multiple nodes try to generate blocks at the same time. The better one wins, if the other was attached first it gets popped off.

Thank you!
Who decides (and how) which block is "better"?

That should be part of the advanced consensus. On part of it: stick to the branch on which you got a higher balance.

[Jean-Luc]

Who decides (and how) which block is "better"?

Each node decides for itself. Eventually they have to agree. The block with higher cumulative difficulty is considered better.

[mcjavar]

Who decides (and how) which block is "better"?

Each node decides for itself. Eventually they have to agree. The block with higher cumulative difficulty is considered better.

Thanks!

[ChuckOne]

Who decides (and how) which block is "better"?

Each node decides for itself. Eventually they have to agree. The block with higher cumulative difficulty is considered better.

cumulative difficulty == target?

[Come-from-Beyond]

Complexity of signing is O(1).

That is plain wrong. The more bits you have to sign the longer signing takes. The question is: how long in terms of the input length.

We would have O(n) if Sign(32_bytes) == Sign(64_bytes) / 2. Actually we have Sign(32_bytes) == Sign(3200_bytes) / 2, so it's more like O(1). Right?

Sorry, still wrong.

It looks more like O(log n) but don't think so, as the whole input has to be read. That still takes O(n).

If we count time to read the input then we should count header processing time too. What about a compromise at O(log log n)?

[ChuckOne]

Complexity of signing is O(1).

That is plain wrong. The more bits you have to sign the longer signing takes. The question is: how long in terms of the input length.

We would have O(n) if Sign(32_bytes) == Sign(64_bytes) / 2. Actually we have Sign(32_bytes) == Sign(3200_bytes) / 2, so it's more like O(1). Right?

Sorry, still wrong.

It looks more like O(log n) but don't think so, as the whole input has to be read. That still takes O(n).

If we count time to read the input then we should count header processing time too. What about a compromise at O(log log n)?

Well. No. It's like a mathematical proof.

E.g. if you want to XOR all byte of the input onto the same variable, you need O(n) because you are reading the input. As simple as that.

When making compromises, let's just assume Qubic is safe. I think it's rather 66% safe as instead of 51%, agreed? I mean 66% is even easier to remember.

[l8orre]

du says: 257480 for current nxt_db folder

What? Mine is 111M only.

Err:

Code:

pi@raspberrypi ~/nxt $ du -h nxt_db/
252M	nxt_db/

Wrong usage of du?

Mine says 114M - still on 0.7.4

du -h

114M   ./nxt_db

[jl777]

I am assuming you are not storing the entire blockchain now that we have DB,but maybe that is incorrect?

Almost all NXT functions only need the most recent 1440 blocks, so if each machine kept those 1440 blocks locally, they can purge them after that. blockchain FIFO.

If checkpoint files are a bad idea, I can live with that. Is blockchain FIFO a bad idea too?

Of course I store the whole blockchain, this is what the db contains.

You seem to be talking about blockchain trimming or pruning. This is in the plans, but not that soon.

Downloading the whole db will take longer than just the blockchain.

I am sorry about not knowing the obvious, still pretty new to crypto, past 10 weeks have been a steep learning curve.

It sounds like you dont have anything against blockchain FIFO. If it is in a DB, I thought it would just be a matter of a single DB call to delete all entries older than fifosize. this is not critical now, but I need to know about how long it will take to get a solution to the HDD storage problem that will hit us when we get to 1000TPS. Many nodes will want to just be able to be current, but not need to provide the entire blockchain to the peers. Just a parameter in web.xml to control fifo size. Anyway just planning for how we get from here to 1000TPS.

It sounds like there is no convenient "state of entire NXT network", so any checkpointing will be a lot of work. Not a problem now either, but after a few months of 1000TPS, no node with less than 1mbps will ever be able to catch up. Again, just planning for future growth.

James