If I'm lucky, it should be done tonight. Otherwise, I'd say sometimes tomorrow

I doubt that will help. These usually top at 32GB of SSD, and most of that is occupied by your OS files. You'll have to wait for HDD opts. If all goes well I'll have something ready for early next week.

Yeah supernode will be very large, it averages 90GB on my machine. That's partly due to LMDB. The Fullnode DB itself sits at around 50GB. The optimized version should approach a 1:1 ratio with Core's blocks folder.

Fullnode will be slow on HDD until I'm done with the current round of optimizations. Ignore supernode on HDDs. There will be a testing release when the changes are solid. Don't toy with the current version if you don't have a SSD.

Oh yeah, I didnt put in a mecanism to update wallet filters from individual wallet status change. Good catch.

Is the wallet filter set to All Wallets?

There are no DB changes in this testing release, there won't be any difference.

Obviously we're not gonna release a DB in a poor state.

LMDB doesn't keep much of its data sequentially as opposed to LevelDB, and that's demolishing HDD speeds. I'm working on some optimizations for HDDs.

Nothing more on the dashboard. If you were running this on Linux, you'd see a report in the terminal for each 2500 block milestone. We'll adjust the progress bar granularity eventually.

HDD is getting murdered I see. I'll test building on a HDD and optimize fullnode for that. Supernode is hopeless on a HDD to begin with.

What's your RAM and CPU usage at again?

There is a wealth of solutions and yours certainly is an acceptable one. Thank you for your concern. However, the issue is about segregation of duties, not implementation. This simply is not a part of the codebase I deal with right now, and every time I ran across this bug in the past few months, I failed to pass it on to the right person.

CircusPeanut is keeping track of everything being reported in this thread and assigning tasks to the proper personnel. This may sound selfish, but I deal with the DB issues, not the rest. Maybe 90% of the DB code was changed this time around and I am solely focusing on getting that part solid. Also I don't usually answer unless the issue is dire or I have at least some sort of a fix worked in already. However I do read all the posts in this thread.

I'm already working on a couple bugs I found on my own a few days ago, although I don't expect any of you will run into those. I'm also paying attention to DB building speed with the different machines you guys run on. From that I will decide if there is need to make anything faster. Keep in mind that I expect most private users will be running fullnode, not supernode, so give that DB type some love too =)

Links have been down for a while now. We're aware of the snafu but have been lazy about it I guess =(

It's not helping that the OS is constantly moving stuff in and out of the swap, it eats disk I/O for no reason. You can try without.

You can halt the scan process at any time, even by killing the process. The DB is resilient to segfaults.

As I said LMDB is a mapped DB. File mapping is entirely controlled by the OS. Depending on how much resources your other active processes require, the OS will throttle down the amount of physical RAM it makes available to hold the mapped file. This in return will crush the write speed (which needs to resolve the page on which the key is or will be before it can write in the data).

My scans take usually an extra 30-40% speed if I play old steam games in the meantime.

To give you an idea, I use a laptop with an i7 haswell, 32GB DDR3 and 3 micro SSDs in RAID0 and no swapping file. It takes me about 1h30 to copy the blocks over and 3h30 to scan supernode (last time I did it was around block 335k). About 45 minutes to scan fullnode, but that's cause the processing is performed in a single thread. Fullnode scans could be lightning fast if I multithreaded that part, as they require very little writing past the block copying. However, that again was not a priority.

I can't wait for DDR4 in laptops so that I can move to 128GB RAM

The DB creation works in 2 phases:

1) Building, which is simply copying the blockchain as a whole over to the DB. This is the phase you are in, which takes very little resources. This code is single threaded and was barely modified in this release (only to support headers first Core). It could be made faster but this isn't a priority in any way.

2) Scanning the transaction history: this part requires heavy IO and processing, and has been largely modified. It will eat all the RAM on your system and I/O capacity of your drives, take several hours/days, and runs on 2 to 4 threads (possible more to come).

You are still building the DB (copying block data over), not scanning (actually parsing the transaction data). When scanning supernode, you will be using over 2GB RAM at least.

Hi,

I'm responsible for the DB code. For those who care, it takes me 3h30 to build supernode, which approximates 90GB with the current blockchain size.

If you have less than 16GB of RAM and a HDD, do not expect supernode to build under a week. Your machine can't handle it. This DB type tracks ALL transactions in the blockchain, it's meant for heavy duty servers. You should at least try and see how long it will take your system to build fullnode before thinking of going with supernode.

The obvious bottleneck is I/O. Moreover, LMDB maps its whole dataset in RAM, so a machine with little RAM and a swapping file on a HDD will get crushed.

Yeah it's fully implemented and set by default. We still use cryptopp only


Translation tags (tr()) have introduced some bugs, we'll get on top of it