As with most transactional DB, there is only 1 write transaction active at any given time, so you can't have several threads writing to the same DB concurrently. In this aspect, multi-threading the build code seems pointless. Because of this, I simply grand-fathered the original build code into the backend overhaul. It relied on mirroring the full content of the blockchain, and that made storage bandwidth the only significant factor in build speed, so I ignored most optimizations in this area. Now that it skips the blockchain copy in fullnode, a meaningful portion of the building phase is taken by block parsing and processing, so there is enough performance to gain to justify the effort of multi-threading this part.


That's the same mentality that brings large IT companies to state that tablets and mobile devices will replace PCs in every home. For their use pattern, they don't need that kind of oomph, and in a way I believe no one but gamers and professionals needed that kind of oomph until large scale P2P networks appeared.

Now with Bitcoin things are quite different. With the blockchain eventually moving into big data territory, not only are big PCs relevant, but the implementation performance matters too. The faster the software, the lower the barrier to entry. If anything, my intention with these changes is for the code to scale with whatever hardware it is thrown at, so that a 10yo computer still has a chance at syncing a wallet against a full node, and that I won't have to rework everything from the ground up 10 years from now either.


My current machine is 6 core cpu (12 execution threads with HT), 32GB ddr4, SATA SSD RAID0 and fiber internet (~250Mb/s). And I'm thinking I should go with 64GB RAM soon. I had thought of buying a dual CPU board and go with a couple Haswell v3 Xeons but I backed off. Can't blow all my bitcoins on a PC =P.

For info, it takes 280~300 sec to scan the DB with ~5k addresses, about 80sec to build, and another ~100 sec for the header reading and what not (that could be reduced to 15sec with some effort).


That would be njaard (I try to avoid referring to people by their name on public forums unless they make a point of doing so). He was part of the original effort in remodeling the backend, which was deployed in 0.93. Since then he was given other responsibilities and I'm the only left with the task of completing this todo list. It still has some large entries to deal with, like blocks over P2P, lite node, full granularity coin selection, Trezor.

Andy is a Python dev. You can see some of his recent work in the hotfix release (0.93.2)

Keep in mind that most the effort was in the scanning code. The building code is still single threaded so a lot of speed can be squeezed there as well (in due time).


Who's Nick o.o"

Looks like we are going with this for the testing release + a couple minor bugs I'm still working on.

Thanks for your invaluable help guys, in particular Carlton Banks =). I guess etotheipi will create a brand new topic for the testing builds, sometimes this week.

It's the same as it was before. We want to implement Trezor support, but we need to finish the new wallets first (our current wallet format is not compatible with Trezor).

I plan on adding at least a command line arg to limit the amount of available cores for the scan. The current setting is to squeeze as much CPU time as available.

Aight, time to re-enable the thread toggling. Bracing myself =D

Forgot to initialize the atomic flag used to signal termination (I'm an idiot). Unlike to most std templates, atomic objects are constructed with an unknown state (which actually makes sense). Obviously I knew all that and I forgot to clear the flag in the ctor...

If this proves to be stable, I'll reactivate the thread toggling (which relies heavily on this flag).

Let's see what the latest push can do

Disabled thread toggling, grab 874cf1e and let's see how far it gets.

You were experiencing that bug before I even added the thread adjustment part. There is something wrong with the code loop through the pull thread queues I'm thinking.

Give it a couple more spins see if it always fails at the same block, please.

There is a backport version of Qt 4.8.6, that should get rid of the Gtk issue I think.

EDIT: actually give it a minute before running this again, pushing some extra verbose.

EDIT2: should be commit 0008c05

I'm told the Gtk-Critical thing is an issue with the Qt version that ships in wheezy (4.8.2, dates from 2012 or something). I use Debian wheezy for my Linux tests and I see this error all the time, without any issue, so I don't believe it is related to BDM snafus.

I've just pushed a small change to squash a dead lock and get rid of BDM not ready false positive errors. This should make it easier to identify the current state of the scanning bug.

This means you are through your original issue. I would like you to run it a few times to determine whether you are completely past that point or this was a fluke.

As for the error, you are on the same thing as Roy, although the nastier form. From the looks of it, it didn't even try to scan anything! Do you have wallets registered?

Here's a tentative fix for Roy's issue. I added some minor changes around the part Carlton is stuck on but I don't expect this is it yet. Pull away.

That one is somewhat nastier. It triggers if the the DB cannot find the hash of the top scanned block, which probably means the scan halted before it failed to commit any blocks at all (and returned an empty top scanned block hash). It could be the concurrency issue triggering early on or something unrelated, although I expect it's the former.

This is a marker that the multi threaded block pulling code is still somehow messy, i.e. the scan fails to pull data for a block and halts. The fixing part detects this, goes over all blocks past that point, realizes there is nothing wrong with the data (or there is and it tries to fix it) and resumes scanning from the highest committed block, on and on.

Also the "attempting fix" code is something I have to go over, it's the stuff from 0.93 that is not really on point with the 0.94 changes.

Thanks, that's enough information for now

The usual conclusion in this case is: there always was an issue with this code, and changes in its vicinity revealed it. Otherwise it was probably occurring at low frequency, passing itself as a scan segfault of sorts.

I'd like you to run it, the main fix in this version is supposed to cover your issue.

And that bit is what's disturbing me considering this part uses the same code across version. Again, this is pretty old code, so I'm puzzled as to why it starts failing now. Anyways, I think I got a clue as to what's going on.


That's pretty fancy. It doesn't really simplify much. It looks like it's meant to speed up interlocked operations that don't overlap in memory, and it's only in some Haswell CPUs so far. I suspect it's the kind of extra assembly optimization that I will personally never have to use, as either compilers or OSes will upgrade their locks with HLE. The way it looks like, there is little reason not to standardize it for every lock operation.