Bitcoin Forum
May 30, 2016, 01:04:23 AM *
News: Latest stable version of Bitcoin Core: 0.12.1 [Torrent]
 
  Home Help Search Donate Login Register  
  Show Posts
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 [26] 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 ... 99 »
501  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 25, 2015, 12:08:07 AM
My current machine is 6 core cpu (12 execution threads with HT), 32GB ddr4, SATA SSD RAID0 and fiber internet (~250Mb/s). And I'm thinking I should go with 64GB RAM soon. I had thought of buying a dual CPU board and go with a couple Haswell v3 Xeons but I backed off. Can't blow all my bitcoins on a PC =P.

For info, it takes 280~300 sec to scan the DB with ~5k addresses, about 80sec to build, and another ~100 sec for the header reading and what not (that could be reduced to 15sec with some effort).

Perhaps your FAQ should list minimum and recommended hardware specs.

I am running my full node on a $35 ARM single board computer with 1 GB ram.

The letstalkbitcoin guys say they used to use armory, and created a new tipping address for every episode, but stopped using it because it became too slow.

You can't compare the resource cost of initial synchronization with that of maintenance. I expect it would take your board over 2 weeks to sync the blockchain from scratch, yet it can process new blocks at an acceptable pace.

You probably had to sync the chain on an actual computer then move the DB and data over to your RPi (or whatever the board is). Same goes with Armory, maintenance cost is a fraction of what it takes to build the original database. Considering 0.94 hardware requirement, I expect that board of yours can run a fully sync'ed armory daemon just fine in fullnode.

As for the letstalkbitcoin guys, they were probably using an old version that had no scalability. Pre 0.9x kept all data in RAM, pre 0.93 was single threaded, relied on LevelDB (what legos are to structural steel) and had serious scalability issues which made the backend come to a dead stop once it reached critical mass. As for 0.93, it scales with RAM (lots of it). 0.94 is the first version to reduce hardware requirements (120MB disk space from 30GB+,  <2GB RAM from 8+) while increasing performance AND increasing scalability.

This is finally approaching good functionality and stability. It is also trivial to toggle RAM and CPU usage in this version, so I will most likely add command args to tune that for expert users.
502  Bitcoin / Armory / Re: Armory system settings?? on: June 24, 2015, 03:18:32 AM
Use 0.92.x for x86 machines. 0.94 will bring back x86 compatibility.
503  Bitcoin / Armory / Re: Bug Report: BlockDataManager Warning on: June 24, 2015, 03:17:35 AM
Use a fresh DB folder and see if that happens again.
504  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 24, 2015, 03:16:36 AM
That sounds impressive, I just assumed the building workload didn't translate well to a multi-threaded design.

As with most transactional DB, there is only 1 write transaction active at any given time, so you can't have several threads writing to the same DB concurrently. In this aspect, multi-threading the build code seems pointless. Because of this, I simply grand-fathered the original build code into the backend overhaul. It relied on mirroring the full content of the blockchain, and that made storage bandwidth the only significant factor in build speed, so I ignored most optimizations in this area. Now that it skips the blockchain copy in fullnode, a meaningful portion of the building phase is taken by block parsing and processing, so there is enough performance to gain to justify the effort of multi-threading this part.

Quote
You know those people that say stuff like "3 GHz? 8 GB RAM? You can't even use that spec for anything! More processing than anyone could use!".

That's the same mentality that brings large IT companies to state that tablets and mobile devices will replace PCs in every home. For their use pattern, they don't need that kind of oomph, and in a way I believe no one but gamers and professionals needed that kind of oomph until large scale P2P networks appeared.

Now with Bitcoin things are quite different. With the blockchain eventually moving into big data territory, not only are big PCs relevant, but the implementation performance matters too. The faster the software, the lower the barrier to entry. If anything, my intention with these changes is for the code to scale with whatever hardware it is thrown at, so that a 10yo computer still has a chance at syncing a wallet against a full node, and that I won't have to rework everything from the ground up 10 years from now either.

Quote
Well, I'm imagining a machine with, say, an 8 core CPU, 32 GB RAM, a PCI express SSD with NVMe, plus a 100 Mbit internet connection. That's probably a mid level dev machine here and now, and I think it could setup Armory 0.94 and Core 0.10 in about 2 hours, going flat out. And the performance of both those pieces of software are set to improve further in the future? I'm slightly amazed that so much efficiency can be gained (and that we have hardware suited to the task Cheesy)

My current machine is 6 core cpu (12 execution threads with HT), 32GB ddr4, SATA SSD RAID0 and fiber internet (~250Mb/s). And I'm thinking I should go with 64GB RAM soon. I had thought of buying a dual CPU board and go with a couple Haswell v3 Xeons but I backed off. Can't blow all my bitcoins on a PC =P.

For info, it takes 280~300 sec to scan the DB with ~5k addresses, about 80sec to build, and another ~100 sec for the header reading and what not (that could be reduced to 15sec with some effort).

Quote
Lol, I have clearly forgotten the name. You have another dev who is a Linux type, Andy?

That would be njaard (I try to avoid referring to people by their name on public forums unless they make a point of doing so). He was part of the original effort in remodeling the backend, which was deployed in 0.93. Since then he was given other responsibilities and I'm the only left with the task of completing this todo list. It still has some large entries to deal with, like blocks over P2P, lite node, full granularity coin selection, Trezor.

Andy is a Python dev. You can see some of his recent work in the hotfix release (0.93.2)
505  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 23, 2015, 10:58:09 PM
DB build performance seems now to be outpacing the increase to the blockchain size. With my 4 year old quad core machine, OS and blockchain running from an SSD, I'm getting ~45 minutes to set up Armory, and that's including transaction scanning for several wallets that pretty much covers every bitcoin transaction I've made since 2012. I dare not try, but I suspect that 93.x might take 2-3 times longer to do that job.

Keep in mind that most the effort was in the scanning code. The building code is still single threaded so a lot of speed can be squeezed there as well (in due time).

Great stuff, goatpig (and I think Nick was involved too?). Thanks.

Who's Nick o.o"
506  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 23, 2015, 09:49:30 PM
If this isn't good enough for the testing release, then it's probably pretty close.

Looks like we are going with this for the testing release + a couple minor bugs I'm still working on.

Thanks for your invaluable help guys, in particular Carlton Banks =). I guess etotheipi will create a brand new topic for the testing builds, sometimes this week.
507  Bitcoin / Armory / Re: Will the Armory Wallet be made compatible with the Trezor hardware wallet? on: June 23, 2015, 04:38:22 AM
It's the same as it was before. We want to implement Trezor support, but we need to finish the new wallets first (our current wallet format is not compatible with Trezor).
508  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 22, 2015, 06:20:10 PM
Current push is stable enough to build and scan Db 4 times completely today, using extensive/varied wallets. Zero failures so far.

Aight, time to re-enable the thread toggling. Bracing myself =D

My CPU heatsink is doing the same Cheesy

I plan on adding at least a command line arg to limit the amount of available cores for the scan. The current setting is to squeeze as much CPU time as available.
509  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 22, 2015, 06:09:24 PM
Current push is stable enough to build and scan Db 4 times completely today, using extensive/varied wallets. Zero failures so far.

Aight, time to re-enable the thread toggling. Bracing myself =D
510  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 22, 2015, 03:19:28 PM
Previous commit did produce one successful scan, but it mostly generated the deadlock that happens when/just before the scan begins. Scan completed through.

Forgot to initialize the atomic flag used to signal termination (I'm an idiot). Unlike to most std templates, atomic objects are constructed with an unknown state (which actually makes sense). Obviously I knew all that and I forgot to clear the flag in the ctor...

If this proves to be stable, I'll reactivate the thread toggling (which relies heavily on this flag).
511  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 22, 2015, 01:12:21 AM
Let's see what the latest push can do
512  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 20, 2015, 10:12:23 PM
Disabled thread toggling, grab 874cf1e and let's see how far it gets.
513  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 20, 2015, 10:05:24 PM
It seems to me like the thread count readjustment triggers the BDM not ready's. If it dodges the bug, it adjusts about 3 times during the first 160,000 blocks, then the threads totally smother the CPU cores. Sometimes it doesn't even handle the zeroth adjust, such as the run that generated the output in the previous post.

You were experiencing that bug before I even added the thread adjustment part. There is something wrong with the code loop through the pull thread queues I'm thinking.
514  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 20, 2015, 06:06:18 PM
Give it a couple more spins see if it always fails at the same block, please.

There is a backport version of Qt 4.8.6, that should get rid of the Gtk issue I think.

EDIT: actually give it a minute before running this again, pushing some extra verbose.

EDIT2: should be commit 0008c05
515  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 20, 2015, 05:15:59 PM
I'm told the Gtk-Critical thing is an issue with the Qt version that ships in wheezy (4.8.2, dates from 2012 or something). I use Debian wheezy for my Linux tests and I see this error all the time, without any issue, so I don't believe it is related to BDM snafus.

I've just pushed a small change to squash a dead lock and get rid of BDM not ready false positive errors. This should make it easier to identify the current state of the scanning bug.
516  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 20, 2015, 02:37:58 PM
Still failing, but getting further through. "BDM not ready" appears as a warning dialog box, OK'ing that shuts Armory. Still a deterministic fault, so far, so part of the same issue maybe?

This means you are through your original issue. I would like you to run it a few times to determine whether you are completely past that point or this was a fluke.

As for the error, you are on the same thing as Roy, although the nastier form. From the looks of it, it didn't even try to scan anything! Do you have wallets registered?
517  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 20, 2015, 02:40:07 AM
Here's a tentative fix for Roy's issue. I added some minor changes around the part Carlton is stuck on but I don't expect this is it yet. Pull away.
518  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 19, 2015, 11:01:58 PM
Also the "attempting fix" code is something I have to go over, it's the stuff from 0.93 that is not really on point with the 0.94 changes.

Ah.  Hence the 'should not get here' error?

That one is somewhat nastier. It triggers if the the DB cannot find the hash of the top scanned block, which probably means the scan halted before it failed to commit any blocks at all (and returned an empty top scanned block hash). It could be the concurrency issue triggering early on or something unrelated, although I expect it's the former.
519  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 19, 2015, 10:28:25 PM
Had eleven instances of the 'top scanned block' issue in the logs

This is a marker that the multi threaded block pulling code is still somehow messy, i.e. the scan fails to pull data for a block and halts. The fixing part detects this, goes over all blocks past that point, realizes there is nothing wrong with the data (or there is and it tries to fix it) and resumes scanning from the highest committed block, on and on.

Also the "attempting fix" code is something I have to go over, it's the stuff from 0.93 that is not really on point with the 0.94 changes.
520  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 19, 2015, 09:58:23 PM
Thanks, that's enough information for now
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 [26] 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 ... 99 »
Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!