Starting preliminary 0.94 testing - "Headless fullnode"

[goatpig]

It seems to me like the thread count readjustment triggers the BDM not ready's. If it dodges the bug, it adjusts about 3 times during the first 160,000 blocks, then the threads totally smother the CPU cores. Sometimes it doesn't even handle the zeroth adjust, such as the run that generated the output in the previous post.

You were experiencing that bug before I even added the thread adjustment part. There is something wrong with the code loop through the pull thread queues I'm thinking.

[1715227474]

1715227474

[1715227474]

1715227474

[1715227474]

1715227474

[1715227474]

1715227474

[1715227474]

1715227474

[1715227474]

1715227474

[goatpig]

Disabled thread toggling, grab 874cf1e and let's see how far it gets.

[Carlton Banks]

There is a backport version of Qt 4.8.6, that should get rid of the Gtk issue I think.

Not too sure about that theory, just checked and I'm running Qt core 4.8.6 right now, although I was using wheezy with standard Qt a few weeks ago when I first tested pre-0.94.

Disabled thread toggling, grab 874cf1e and let's see how far it gets.

Will give it a try.

[Carlton Banks]

Not far:

Code:

-INFO  - 1434841472: (BlockUtils.cpp:1403) Wrote blocks to DB in 78.0345s
-WARN  - 1434841472: (lmdb_wrapper.cpp:659) Clearing history
-WARN  - 1434841472: (BlockUtils.cpp:1118) Scanning from 0 to 361823
-INFO  - 1434841472: (BlockWriteBatcher.cpp:1675) height 0 exceeds scan range, ending scan
-WARN  - 1434841472: (BlockWriteBatcher.cpp:544) --- feedSleep: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:547) --- workers: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:550) --- writeStxo: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:552) --- writeStxo_grabMutex: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:555) --- waitingOnSerThread: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:557) --- waitForDataToWrite: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:561) --- checkForCollisions: 0.001303 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:563) --- getExistingKeys: 0.001481 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:565) --- getNewKeys: 0.002327 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:567) --- getSSHHeadersLock: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:569) --- computeDBKeys: 0.004153 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:571) --- getSshHeaders: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:573) --- finalizeSerialization: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:578) --- serializeBatch: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:580) --- updateSSH: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:582) --- serializeSubSsh: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:584) --- waitOnSSHser: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:588) --- waitOnWriteThread: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:591) --- putSSH: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:593) --- putSTX: 0 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:596) --- getnextfeed: 8e-06 s
-WARN  - 1434841472: (BlockWriteBatcher.cpp:598) --- inControlThread: 0.000224 s
-INFO  - 1434841472: (BlockUtils.cpp:1443) checking scan integrity
-WARN  - 1434841472: (BlockUtils.cpp:1448) Top scanned block does not match top block header
-ERROR - 1434841472: (BlockUtils.cpp:1478) shouldnt get to this line
-INFO  - 1434841472: (BlockDataViewer.cpp:155) Enabling zero-conf tracking 
-ERROR - 1434841472: (BDM_mainthread.cpp:429) BDM thread failed: BDM is not ready!

Same result for several trials.

[goatpig]

Let's see what the latest push can do

[Carlton Banks]

Previous commit did produce one successful scan, but it mostly generated the deadlock that happens when/just before the scan begins. Scan completed through.

19c1a72e7f seems to reliably avoid the scanning deadlock, or at least it's done so twice consecutively so far. Continuing testing, trying to get a few more data points. Scanning completes.

[goatpig]

Previous commit did produce one successful scan, but it mostly generated the deadlock that happens when/just before the scan begins. Scan completed through.

Forgot to initialize the atomic flag used to signal termination (I'm an idiot). Unlike to most std templates, atomic objects are constructed with an unknown state (which actually makes sense). Obviously I knew all that and I forgot to clear the flag in the ctor...

If this proves to be stable, I'll reactivate the thread toggling (which relies heavily on this flag).

[Carlton Banks]

Current push is stable enough to build and scan Db 4 times completely today, using extensive/varied wallets. Zero failures so far.

[goatpig]

Current push is stable enough to build and scan Db 4 times completely today, using extensive/varied wallets. Zero failures so far.

Aight, time to re-enable the thread toggling. Bracing myself =D

[Carlton Banks]

Current push is stable enough to build and scan Db 4 times completely today, using extensive/varied wallets. Zero failures so far.

Aight, time to re-enable the thread toggling. Bracing myself =D

My CPU heatsink is doing the same

[goatpig]

Current push is stable enough to build and scan Db 4 times completely today, using extensive/varied wallets. Zero failures so far.

Aight, time to re-enable the thread toggling. Bracing myself =D

My CPU heatsink is doing the same

I plan on adding at least a command line arg to limit the amount of available cores for the scan. The current setting is to squeeze as much CPU time as available.

[Carlton Banks]

Looks good. Made it through 3 full build/scans cleanly. Thread toggler only adjusts once, whereas it typically did so at least twice with the commit it was last enabled for (the thread toggler seemed to provoke the crashes before). CPU still gets tortured. It handles being interrupted during the scan OK too; it picked up where it left off fine, did 2 thread adjustments (the only time that happened) and finished the job. If this isn't good enough for the testing release, then it's probably pretty close.

[goatpig]

If this isn't good enough for the testing release, then it's probably pretty close.

Looks like we are going with this for the testing release + a couple minor bugs I'm still working on.

Thanks for your invaluable help guys, in particular Carlton Banks =). I guess etotheipi will create a brand new topic for the testing builds, sometimes this week.

[Carlton Banks]

DB build performance seems now to be outpacing the increase to the blockchain size. With my 4 year old quad core machine, OS and blockchain running from an SSD, I'm getting ~45 minutes to set up Armory, and that's including transaction scanning for several wallets that pretty much covers every bitcoin transaction I've made since 2012. I dare not try, but I suspect that 93.x might take 2-3 times longer to do that job.

Great stuff, goatpig (and I think Nick was involved too?). Thanks.

[goatpig]

DB build performance seems now to be outpacing the increase to the blockchain size. With my 4 year old quad core machine, OS and blockchain running from an SSD, I'm getting ~45 minutes to set up Armory, and that's including transaction scanning for several wallets that pretty much covers every bitcoin transaction I've made since 2012. I dare not try, but I suspect that 93.x might take 2-3 times longer to do that job.

Keep in mind that most the effort was in the scanning code. The building code is still single threaded so a lot of speed can be squeezed there as well (in due time).

Great stuff, goatpig (and I think Nick was involved too?). Thanks.

Who's Nick o.o"

[Carlton Banks]

DB build performance seems now to be outpacing the increase to the blockchain size. With my 4 year old quad core machine, OS and blockchain running from an SSD, I'm getting ~45 minutes to set up Armory, and that's including transaction scanning for several wallets that pretty much covers every bitcoin transaction I've made since 2012. I dare not try, but I suspect that 93.x might take 2-3 times longer to do that job.

Keep in mind that most the effort was in the scanning code. The building code is still single threaded so a lot of speed can be squeezed there as well (in due time).

That sounds impressive, I just assumed the building workload didn't translate well to a multi-threaded design.

You know those people that say stuff like "3 GHz? 8 GB RAM? You can't even use that spec for anything! More processing than anyone could use!". Well, I'm imagining a machine with, say, an 8 core CPU, 32 GB RAM, a PCI express SSD with NVMe, plus a 100 Mbit internet connection. That's probably a mid level dev machine here and now, and I think it could setup Armory 0.94 and Core 0.10 in about 2 hours, going flat out. And the performance of both those pieces of software are set to improve further in the future? I'm slightly amazed that so much efficiency can be gained (and that we have hardware suited to the task )

Great stuff, goatpig (and I think Nick was involved too?). Thanks.

Who's Nick o.o"

Lol, I have clearly forgotten the name. You have another dev who is a Linux type, Andy?

[goatpig]

That sounds impressive, I just assumed the building workload didn't translate well to a multi-threaded design.

As with most transactional DB, there is only 1 write transaction active at any given time, so you can't have several threads writing to the same DB concurrently. In this aspect, multi-threading the build code seems pointless. Because of this, I simply grand-fathered the original build code into the backend overhaul. It relied on mirroring the full content of the blockchain, and that made storage bandwidth the only significant factor in build speed, so I ignored most optimizations in this area. Now that it skips the blockchain copy in fullnode, a meaningful portion of the building phase is taken by block parsing and processing, so there is enough performance to gain to justify the effort of multi-threading this part.

You know those people that say stuff like "3 GHz? 8 GB RAM? You can't even use that spec for anything! More processing than anyone could use!".

That's the same mentality that brings large IT companies to state that tablets and mobile devices will replace PCs in every home. For their use pattern, they don't need that kind of oomph, and in a way I believe no one but gamers and professionals needed that kind of oomph until large scale P2P networks appeared.

Now with Bitcoin things are quite different. With the blockchain eventually moving into big data territory, not only are big PCs relevant, but the implementation performance matters too. The faster the software, the lower the barrier to entry. If anything, my intention with these changes is for the code to scale with whatever hardware it is thrown at, so that a 10yo computer still has a chance at syncing a wallet against a full node, and that I won't have to rework everything from the ground up 10 years from now either.

Well, I'm imagining a machine with, say, an 8 core CPU, 32 GB RAM, a PCI express SSD with NVMe, plus a 100 Mbit internet connection. That's probably a mid level dev machine here and now, and I think it could setup Armory 0.94 and Core 0.10 in about 2 hours, going flat out. And the performance of both those pieces of software are set to improve further in the future? I'm slightly amazed that so much efficiency can be gained (and that we have hardware suited to the task Cheesy)

My current machine is 6 core cpu (12 execution threads with HT), 32GB ddr4, SATA SSD RAID0 and fiber internet (~250Mb/s). And I'm thinking I should go with 64GB RAM soon. I had thought of buying a dual CPU board and go with a couple Haswell v3 Xeons but I backed off. Can't blow all my bitcoins on a PC =P.

For info, it takes 280~300 sec to scan the DB with ~5k addresses, about 80sec to build, and another ~100 sec for the header reading and what not (that could be reduced to 15sec with some effort).

Lol, I have clearly forgotten the name. You have another dev who is a Linux type, Andy?

That would be njaard (I try to avoid referring to people by their name on public forums unless they make a point of doing so). He was part of the original effort in remodeling the backend, which was deployed in 0.93. Since then he was given other responsibilities and I'm the only left with the task of completing this todo list. It still has some large entries to deal with, like blocks over P2P, lite node, full granularity coin selection, Trezor.

Andy is a Python dev. You can see some of his recent work in the hotfix release (0.93.2)

[AussieHash]

My current machine is 6 core cpu (12 execution threads with HT), 32GB ddr4, SATA SSD RAID0 and fiber internet (~250Mb/s). And I'm thinking I should go with 64GB RAM soon. I had thought of buying a dual CPU board and go with a couple Haswell v3 Xeons but I backed off. Can't blow all my bitcoins on a PC =P.

For info, it takes 280~300 sec to scan the DB with ~5k addresses, about 80sec to build, and another ~100 sec for the header reading and what not (that could be reduced to 15sec with some effort).

Perhaps your FAQ should list minimum and recommended hardware specs.

I am running my full node on a $35 ARM single board computer with 1 GB ram.

The letstalkbitcoin guys say they used to use armory, and created a new tipping address for every episode, but stopped using it because it became too slow.

[Carlton Banks]

My current machine is 6 core cpu (12 execution threads with HT), 32GB ddr4, SATA SSD RAID0 and fiber internet (~250Mb/s). And I'm thinking I should go with 64GB RAM soon. I had thought of buying a dual CPU board and go with a couple Haswell v3 Xeons but I backed off. Can't blow all my bitcoins on a PC =P.

For info, it takes 280~300 sec to scan the DB with ~5k addresses, about 80sec to build, and another ~100 sec for the header reading and what not (that could be reduced to 15sec with some effort).

Perhaps your FAQ should list minimum and recommended hardware specs.

I am running my full node on a $35 ARM single board computer with 1 GB ram.

The letstalkbitcoin guys say they used to use armory, and created a new tipping address for every episode, but stopped using it because it became too slow.

You'll find this version will be comparatively faster on anything with more than one core, I think that includes most ARM SoC's these days. And because the database for Armory drops down to 120 MB, it's much less concerning to think about the future disk space.

[goatpig]

My current machine is 6 core cpu (12 execution threads with HT), 32GB ddr4, SATA SSD RAID0 and fiber internet (~250Mb/s). And I'm thinking I should go with 64GB RAM soon. I had thought of buying a dual CPU board and go with a couple Haswell v3 Xeons but I backed off. Can't blow all my bitcoins on a PC =P.

For info, it takes 280~300 sec to scan the DB with ~5k addresses, about 80sec to build, and another ~100 sec for the header reading and what not (that could be reduced to 15sec with some effort).

Perhaps your FAQ should list minimum and recommended hardware specs.

I am running my full node on a $35 ARM single board computer with 1 GB ram.

The letstalkbitcoin guys say they used to use armory, and created a new tipping address for every episode, but stopped using it because it became too slow.

You can't compare the resource cost of initial synchronization with that of maintenance. I expect it would take your board over 2 weeks to sync the blockchain from scratch, yet it can process new blocks at an acceptable pace.

You probably had to sync the chain on an actual computer then move the DB and data over to your RPi (or whatever the board is). Same goes with Armory, maintenance cost is a fraction of what it takes to build the original database. Considering 0.94 hardware requirement, I expect that board of yours can run a fully sync'ed armory daemon just fine in fullnode.

As for the letstalkbitcoin guys, they were probably using an old version that had no scalability. Pre 0.9x kept all data in RAM, pre 0.93 was single threaded, relied on LevelDB (what legos are to structural steel) and had serious scalability issues which made the backend come to a dead stop once it reached critical mass. As for 0.93, it scales with RAM (lots of it). 0.94 is the first version to reduce hardware requirements (120MB disk space from 30GB+,  <2GB RAM from 8+) while increasing performance AND increasing scalability.

This is finally approaching good functionality and stability. It is also trivial to toggle RAM and CPU usage in this version, so I will most likely add command args to tune that for expert users.