|
Title: Starting preliminary 0.94 testing - "Headless fullnode" Post by: etotheipi on May 13, 2015, 10:54:19 PM So, preliminarily 0.94 has stabilized, but it hasn't seen much testing outside "the lab." I would like to invite folks who can build, to checkout the ffreeze branch (feature-freeze) and try it out while we continue our internal testing.
https://github.com/etotheipi/BitcoinArmory/tree/ffreeze ++ In 0.94 the Armory home dir databases folder should now be less than 150 MB! This is what we refer to as "headless fullnode" -- All database formats have changed. You must rebuild your databases! (and we don't have a clean way to help the user transition from the GUI). Therefore, to use/test 0.94, please change your datadir, dbdir, or delete the databases folder in the Armory home directory. Though, you might want to just move the directory for the moment, in case you have to go back. Luckily, since the new version uses so much less space, it's not a big deal to create another directory for it! It's also is a heck of a lot faster to start up Armory the first time once Core is sync'd, since we're not building a 30+ GB database anymore. It's called "headless fullnode" because it's the same as previous "fullnode" version of Armory, but without maintaining it's own copy of the blockchain (of course, if you run supernode it will maintain your 2x-3x sized DB, but that's not the default Armory mode). Along with this, there's a ton of bug fixes and stability enhancements. We're looking forward to the database design finally settling down and stabilizing. PLEASE checkout the ffreeze branch and help test! Once we get a little bit more testing we'll do a semi-official testing release with a proper bug bounty! Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: doug_armory on May 14, 2015, 02:38:37 PM FWIW, I've been using 0.94 test builds on OS X for awhile now. They're definitely not perfect but they've been pretty good overall so far. It sure is nice to have a much smaller, faster DB. All hail goatpig. :) (Of course, my old one is backed up just in case. Make backups, lest ye tempt fate.) If you're a power user, you probably wouldn't want this as your daily driver. If you're curious or just use Armory once in awhile, I think it's okay to build this and use it for non-mission-critical tasks.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: TierNolan on May 14, 2015, 03:29:36 PM What does headless full node mean? It just stores the UTXO data or just recent blocks?
[Edit] On an unrelated topic, I tried to scan the QR code in the README and it doesn't scan with the bitcoin smartphone app. It is like an extra "?" is added to the end. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 14, 2015, 06:21:03 PM Whole thing crashes when scanning tx's. CPU/Memory usage goes vertical-ish.
Possibly the source of offence: Code: (python:3377): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed Logs are otherwise uneventful Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: etotheipi on May 14, 2015, 07:00:08 PM Whole thing crashes when scanning tx's. CPU/Memory usage goes vertical-ish. Possibly the source of offence: Code: (python:3377): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed Logs are otherwise uneventful Did you pull the latest ffreeze? There was a problem with thread synchronization a couple days ago, and we all experienced the exact same issue. As of yesterday (when I posted this thread) that was fixed and no one else had the issue. Did you wipe the database? Is the version number 0.93.99.1? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 14, 2015, 07:16:22 PM Did you pull the latest ffreeze? I did: Code: git clone git://github.com/etotheipi/BitcoinArmory.git Did you wipe the database? Yes Is the version number 0.93.99.1? Yes Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 14, 2015, 07:18:32 PM Whole thing crashes when scanning tx's. CPU/Memory usage goes vertical-ish. Possibly the source of offence: Code: (python:3377): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed Logs are otherwise uneventful How far did it get in block height? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 14, 2015, 08:09:13 PM There is a pattern to it. It's very crash prone from blocks 0 > ~200,000. Then it stayed up until top block, minus one seg fault. Doesn't correspond to any pattern in the wallet file/s I'm trying it out with, or I don't think so anyway (I originally though it might be coinbase tx's that were the problem, but I get this with any wallet)
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 14, 2015, 08:32:35 PM Also, SDM mode produces this:
Code: Traceback (most recent call last): and then crashes. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 14, 2015, 10:02:29 PM Also, SDM mode produces this: Code: Traceback (most recent call last): and then crashes. That's the torrent manager failing to find your /blocks folder. That suggests it's looking at an empty folder. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 14, 2015, 10:57:30 PM Can't get the SDM code to find satoshi datadir no matter what I do. Works fine without it though.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: etotheipi on May 14, 2015, 11:08:34 PM This reminds me that we should probably remove all torrent-related code from Armory for 0.94. It is no longer needed with Core 0.10+ (headers first), and it was actually designed such that you could remove the torrent code directory and Armory would bypass trying to use it. We wanted to wait for sufficient people to have upgraded to Core 0.10 and Armory 0.93+, and I think that's likely to be the case now (we couldn't upgrade the Core links in the secure downloader, without inadvertantly leading to 0.92 users installing Core 0.10 which doesn't work).
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 14, 2015, 11:35:04 PM Ok, any wallet that needs it's tx hints recording (plus whatever else) is somewhere close to the problem. If I create an empty wallet and rebuild the Db, nice smooth scanning, except for a couple of seg faults
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: justusranvier on May 15, 2015, 04:23:47 PM It's also is a heck of a lot faster to start up Armory the first time once Core is sync'd, since we're not building a 30+ GB database anymore. It's called "headless fullnode" because it's the same as previous "fullnode" version of Armory, but without maintaining it's own copy of the blockchain (of course, if you run supernode it will maintain your 2x-3x sized DB, but that's not the default Armory mode). Will it ever be possible to run Armory without requiring filesystem-level access to the block files?Seems like a change such as this makes such an operating mode more difficult. I run bitcoind and Armory in separate virtual machines, and finding a way to share those files in a way that gets all the permissions right and satisfies LevelDB's locking requirements is non-trivial. It would be great to have an option for Armory to maintain its own database using blocks it obtains via its peer connection to the node, rather than via direct filesystem access, without requiring a supernode. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 15, 2015, 06:05:11 PM It's also is a heck of a lot faster to start up Armory the first time once Core is sync'd, since we're not building a 30+ GB database anymore. It's called "headless fullnode" because it's the same as previous "fullnode" version of Armory, but without maintaining it's own copy of the blockchain (of course, if you run supernode it will maintain your 2x-3x sized DB, but that's not the default Armory mode). Will it ever be possible to run Armory without requiring filesystem-level access to the block files?Seems like a change such as this makes such an operating mode more difficult. I run bitcoind and Armory in separate virtual machines, and finding a way to share those files in a way that gets all the permissions right and satisfies LevelDB's locking requirements is non-trivial. It would be great to have an option for Armory to maintain its own database using blocks it obtains via its peer connection to the node, rather than via direct filesystem access, without requiring a supernode. We do plan on implementing blocks over p2p Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 15, 2015, 07:38:07 PM Carlton Banks: would you mind building ffreeze in debug mode, running it in gdb and posting the backtrace when it crashes?
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 15, 2015, 08:07:22 PM Carlton Banks: would you mind building ffreeze in debug mode, running it in gdb and posting the backtrace when it crashes? I don't totally know what you mean, but yes in principle. What's gdb? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 15, 2015, 08:42:55 PM Carlton Banks: would you mind building ffreeze in debug mode, running it in gdb and posting the backtrace when it crashes? I don't totally know what you mean, but yes in principle. What's gdb? https://www.gnu.org/software/gdb/ Essentially, linux debug mode. First install gdb. On Debian based systems, sudo apt-get install gdb will do it. I'm not familiar with rpm but I expect there are gdb builds available. With gdb installed, type these commands in the top code folder: Quote make clean make DEBUG=1 <- build with debug symbols gdb python <- start python in debug mode run ./ArmoryQt.py <- The command line argument goes after run ... <- code is now running, wait till it segfaults backtrace <- after it segfaults, you receive control back, type in backtrace and paste the result back in this thread Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: btchris on May 15, 2015, 09:56:21 PM Code: First-chance exception at 0x000007FEDFD63450 (_CppBlockUtils.pyd) in python.exe: 0xC0000005: Access violation reading location 0x0000000088067A8D. On Win 7 64-bit, I'm also getting an access violation while scanning txs. Running the latest in the ffreeze branch (578c072). Here's a stack trace (via VS 2013) from a debug build: Code: _CppBlockUtils.pyd!memcpy() Line 356 Unknown I think there's something going on near GrabThreadData::pullBlockAtIter(); here's a locals dump after switching to that frame: Code: - bdr {ptr_=0x0000000088067a8d "" nBytes_=155478 } BinaryDataRefFirst off, fnum, offset, and size look good. I verified that they point to the beginning of block 258860, and that the block file doesn't appear corrupted. At the very least, the merkle root is correct. Here are two things that pop out at me though. pb.fmp_.current_->filemap_ and pb.fmp_.current_->mapsize_ are clearly invalid, perhaps something is overwriting them? IIUC, bdr.ptr_ is calculated from filemap_ and offset, and so bdr.ptr_ is invalid, and it's what eventually gets passed to memcpy() which causes the access violation. Another thing that looks strange is that I thought (I could be wrong) that bdr.ptr_ == filemap_ + offset, but in the locals above it's not (or multiple things are getting overwritten?). I'm not quite sure where to go from here.... whatever happened, it may be too late at this point in the debug to find. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 15, 2015, 10:52:03 PM First install gdb. On Debian based systems, sudo apt-get install gdb will do it. I'm not familiar with rpm but I expect there are gdb builds available. With gdb installed, type these commands in the top code folder: Quote make clean make DEBUG=1 <- build with debug symbols gdb python <- start python in debug mode run ./ArmoryQt.py <- The command line argument goes after run ... <- code is now running, wait till it segfaults backtrace <- after it segfaults, you receive control back, type in backtrace and paste the result back in this thread Ok, in the gdb environment and using a build with the debug level you specified, 93.99.1 scans the wallet without a single complaint. That seems pretty weird. Didn't re-checkout, didn't re-clone, didn't change a thing other than follow the above instructions. It looks like the ffreeze branch is up to the same commit as yesterday anyway, so checking out or re-cloning couldn't cause a difference. Don't understand it at all. Also tried outside of gdb, also the same. Still waiting for the scan to complete in this trial, but it seems just as stable as when using gdb Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 15, 2015, 11:56:45 PM btchris: unless you consistantly get the same error, this probably the aftermath of a buffer overrun in another thread
Carlton Banks: That's usually symptomatic of a threading issue. Throw off the timing a little and the issue doesn't appear anymore. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 16, 2015, 01:11:26 AM Carlton Banks: That's usually symptomatic of a threading issue. Throw off the timing a little and the issue doesn't appear anymore. I thought it couldn't possibly have been a threading problem, as yesterday the behaviour was totally consistent across multiple trials. And today it isn't, so I guess it must be threading related (someone might think you knew something about this code :D). That's pretty curious though, several consecutive trials that generate basically the same result, and then several more consecutive trials that produce completely different result, also all the same as each other, and all without any material change to the build. In addition, not one GTK+ CRITICAL assertion fails appeared on the console today. To clarify the pattern of crashing yesterday: scanning from blocks 0 - 200,000, Armory crashed without even sending segfault messages to console, and it only got through 2-5 thousand blocks before the crashes. That's a lot of restarts to get it to scan the first 200,000 blocks. So I've got no segfault tracebacks to post, as I've not had even one. Will carry on testing debug build using gdb in hope it will enter the same state again. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 16, 2015, 10:51:28 AM Welcome to my world =P
Threading issues are tricky. Consider no one has ran into trouble in-house, and our first 2 testers get something right away. You could try to force the thread count per task manually see how it reacts. In BlockWriteBatcher.cpp, around line 1540, add the lines in bold: Quote else { //otherwise, give workers 90% of the processing power int32_t writers = (totalThreadCount_ * 10) / 100; if (writers < 1) writers = 1; int32_t workers = totalThreadCount_ - writers; if (workers < 1) workers = 1; workers = 1; writers = 1; nWorkers_.store(workers); nWriters_.store(writers); } This controls the amount of workers (reading through transactions and extracting history) and writers (here you would force them both to 1 to test low performance stability). Also, around line 385: Quote { //Grab the SSHheader static mutex. This makes sure only this scan is //creating new SSH keys. If several scanning threads were to take place, //it could possibly result in key collision, as scan threads are not aware //of each others' state unique_lock<mutex> addressingLock(SSHheaders::keyAddressingMutex_); uint32_t nThreads = 3; if (int(endBlock) - int(startBlock) < 100) nThreads = 1; nThreads = 1; prepareSshToModify(scf); This controls the amount of "grab" threads, which pull raw data from the blockchain and serialize it into PulledBlock objects used by the workers. Be nice and don't comment on my naming sense, unlike some of my colleagues T__T. This stuff needs consolidated but I'll get to that once the thread toggler is done. As you can see, it runs on static thread counts currently, but it's meant to adjust resource per task dynamically (to squeeze every last drop of processing power regardless of the machine, and eventually give the user live control over how many cpu cores it can use). For max performance in Fullnode, you want a grab thread per worker and only 1 writer. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 16, 2015, 07:13:17 PM Code: [New Thread 0x7fffd7fff700 (LWP 3900)] Not typical of the first crashes, but there's probably something useful in here. There's two separate trials from the same tx scan; bitcoind crashed in the first trial, and the tx scan just carried on. I wasn't too sure about that, and thought that at least starting bitcoind again was a good idea. It wasn't/it didn't work out (SDM considered bitcoind reconnected only once Armory went into it's shutdown routine. Where it would normally throw up a balloon notifier to indicate bitcoind disconnection, instead a re-establishment notifier appeared, the shutdown routine then crashed). So there's definitely some information there to help with debugging the Db building when unexpected/disruptive things happen. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: btchris on May 16, 2015, 10:37:03 PM Not typical of the first crashes, but there's probably something useful in here. That stack trace looks exactly the same as mine on Windows (some of the line #s are off, but I think that's just a symptom of using different debuggers). Even the most recent calls are showing a similar out-of-bounds pointer being passed. FYI I've had one additional crash, and the stack trace was the same (with pb.fmp_.current_->filemap_ and pb.fmp_.current_->mapsize_ in GrabThreadData::pullBlockAtIter() corrupted). Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 16, 2015, 11:42:07 PM Not typical of the first crashes, but there's probably something useful in here. That stack trace looks exactly the same as mine on Windows (some of the line #s are off, but I think that's just a symptom of using different debuggers). Even the most recent calls are showing a similar out-of-bounds pointer being passed. FYI I've had one additional crash, and the stack trace was the same (with pb.fmp_.current_->filemap_ and pb.fmp_.current_->mapsize_ in GrabThreadData::pullBlockAtIter() corrupted). Ah, maybe this has nothing to do with those unexpected things happening in that trial (bitcoind crashing, SDM crashing etc). Debugging threaded applications really does seem as difficult as is made out. How people use other complex or subtle programming constructs in combination with threading, I do not know. It's tempting to say I'd like a repeat of the failure patterns I got Thursday, those trials were very unstable. Armory mostly crashed after less than 30 seconds, then whatever caused that pattern shifted half through the scan, and then completed without complaint. But is that behaviour actually a whole other category of bug, that just so happens to be triggering the pointer scope bug as it's own natural consequence? It's difficult to tell. Could the debugging environment be invoked at the hypervisor level? I'm sure I've heard this is possible, whereby a VM has it's entire state recorded during software tests, which can then be "played back" should any given run yield bugs that are difficult to find after the fact. Another little observation is: we're running to stand still, in performance terms. At the time 0.93 was finalised, just a few months ago, I think it took around the same length of time to build and scan as it does with this pre-0.94 build. Obviously we have more blocks/transactions to process/scan now, and goatpig says there's a little more performance to tweak out of this design. Could some of the work (reading the headers or Db writing?) be done concurrently with the blockchain sync? I suspect it would be easier once Armory is actually it's own node. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: btchris on May 17, 2015, 12:01:08 AM Another little observation is: we're running to stand still, in performance terms. For me anyways, 0.94 allows me to place the Armory DB on an SSD whereas before both it and the Bitcoin DB were on an HDD. Practically speaking, this has hugely improved the initial "reading blocks" and "building database" steps, down to about 14 minutes. The tx scanning phase is only slightly faster IIRC (possibly because the Bitcoin DB block files are still on HDD). In short, when 0.94 works, it's brilliant! Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 17, 2015, 12:32:35 AM Another little observation is: we're running to stand still, in performance terms. For me anyways, 0.94 allows me to place the Armory DB on an SSD whereas before both it and the Bitcoin DB were on an HDD. Practically speaking, this has hugely improved the initial "reading blocks" and "building database" steps, down to about 14 minutes. The tx scanning phase is only slightly faster IIRC (possibly because the Bitcoin DB block files are still on HDD). In short, when 0.94 works, it's brilliant! You're reminding me how spoilt I am, I can't imagine a computer without enough SSD space to hold a blockchain or two 8). I get the same performance you do for the reading headers/blocks stage, and then about 40 minutes to scan; surprise me and tell me that the scan takes hours from a mechanical disk (are there not cryptography operations taking place in every iteration of that scan? The CPU is bound to be a significant limiting factor). I'm pretty sure the bottleneck with my machine when it comes to the scan is either RAM or CPU (as both, plus swap, get basically maxed out) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: btchris on May 17, 2015, 12:41:53 AM You're reminding me how spoilt I am Well, your namesake certainly was ;D about 40 minutes to scan I haven't timed it exactly, but it's somewhere around there even with my HDD. I'll pay closer attention next time I do a scan. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: zombieslayer9099 on May 17, 2015, 09:01:15 PM My installation is running just fine
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Searinox on May 18, 2015, 10:24:38 AM Is there a downloadable Win x64 binary?
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: doug_armory on May 18, 2015, 03:44:00 PM Is there a downloadable Win x64 binary? Not for now. This is for people who are able to compile code we supply. We'll offer test binaries eventually. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 18, 2015, 09:21:45 PM Code: (gdb) run ArmoryQt.py A pair of trials that crashed today. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 18, 2015, 10:51:35 PM Code: Program received signal SIGSEGV, Segmentation fault. backtrace from a subsequent run (no GTK+ CRITICAL entries in the log, or anything that appears unusual, like the threads all finishing one after another from the post above this one). Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: njaard on May 18, 2015, 11:43:59 PM Thank you Carlton Banks,
SIGKILL is often caused by Linux's out-of-memory killer. Linux is saying "I'm out of memory, I'm gonna kill an application that I think is to blame", and in this case it guesses Armory, which is probably the correct one. Running dmesg should show that the "OOM" event. Maybe you can confirm? The second crash is definitely our fault, but it's hard to tell what's causing it. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 19, 2015, 11:40:54 AM Carlton Banks, btcchris: something new to try, disable the block data prefetch thread.
In BlockWriteBatcher.h#416, comment out these 2 lines: Quote LoadedBlockData(const LoadedBlockData&) = delete; BFA_PREFETCH getPrefetchMode(void) { /*if (startBlock_ < endBlock_ && endBlock_ - startBlock_ > 100) return PREFETCH_FORWARD;*/ return PREFETCH_NONE; } Regarding speed, in fullnode the bottleneck is your CPU. On my Debian VM with 2 workers I scan fullnode in 40min. On my host with 9 workers I scan it in 5 (i7 5820). Once the thread toggler is ready it should run a lot better than going with some estimated value. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 19, 2015, 12:01:50 PM Carlton Banks, btcchris: something new to try, disable the block data prefetch thread. In BlockWriteBatcher.h#416, comment out these 2 lines: Quote LoadedBlockData(const LoadedBlockData&) = delete; BFA_PREFETCH getPrefetchMode(void) { /*if (startBlock_ < endBlock_ && endBlock_ - startBlock_ > 100) return PREFETCH_FORWARD;*/ return PREFETCH_NONE; } Regarding speed, in fullnode the bottleneck is your CPU. On my Debian VM with 2 workers I scan fullnode in 40min. On my host with 9 workers I scan it in 5 (i7 5820). Once the thread toggler is ready it should run a lot better than going with some estimated value. I don't know whether you're asking me to break the terms of the license by doing that. Can I get some assurance that I'm protected from legal action by your employers? I have zero interest in reading the license terms myself, it is, in my opinion, abhorrent to use litigation to enforce so-called freedom. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 19, 2015, 12:52:43 PM I don't know whether you're asking me to break the terms of the license by doing that. Can I get some assurance that I'm protected from legal action by your employers? I have zero interest in reading the license terms myself, it is, in my opinion, abhorrent to use litigation to enforce so-called freedom. I don't know how the legal stuff goes. I can give you my personal guarantee but I don't know what that is worth either, legally speaking (I doubt I'm in a position to do that). I'm going to push some stuff today and I'll disable the prefetch thread along the way, how about you wait for that? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 19, 2015, 01:26:31 PM I don't know whether you're asking me to break the terms of the license by doing that. Can I get some assurance that I'm protected from legal action by your employers? I have zero interest in reading the license terms myself, it is, in my opinion, abhorrent to use litigation to enforce so-called freedom. I don't know how the legal stuff goes. I can give you my personal guarantee but I don't know what that is worth either, legally speaking (I doubt I'm in a position to do that). I'm going to push some stuff today and I'll disable the prefetch thread along the way, how about you wait for that? That's fine, as long as I perceive no risk to me or my affects, then I'm as happy doing that as I've ever been (you're publishing to github after all). Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 19, 2015, 01:49:58 PM That's fine, as long as I perceive no risk to me or my affects, then I'm as happy doing that as I've ever been (you're publishing to github after all). My understanding of the AGPL license is that if you were to modify Armory source and distribute binaries (or run a commercial activity) based on those changes without publishing the altered code publicly, you would be in infringement. I don't think this affects changes for personal and non commercial use. Regardless, if you were to fork Armory publicly (say on a public github repo), you could do as many changes as you like. That's just my interpretation however. From my perspective there is no license infringement in doing those changes under these conditions, and you have my guarantee I won't be coming after you over this. Again, I'm no lawyer and these words only engage me. I don't have the credentials to speak for the business on this matter. If anything the responsibility is mine since I asked you to do these changes. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 19, 2015, 03:43:30 PM That's fine, as long as I perceive no risk to me or my affects, then I'm as happy doing that as I've ever been (you're publishing to github after all). My understanding of the AGPL license is that if you were to modify Armory source and distribute binaries (or run a commercial activity) based on those changes without publishing the altered code publicly, you would be in infringement. I don't think this affects changes for personal and non commercial use. Regardless, if you were to fork Armory publicly (say on a public github repo), you could do as many changes as you like. That's just my interpretation however. From my perspective there is no license infringement in doing those changes under these conditions, and you have my guarantee I won't be coming after you over this. Again, I'm no lawyer and these words only engage me. I don't have the credentials to speak for the business on this matter. If anything the responsibility is mine since I asked you to do these changes. I have next to zero expectation that ATI would do something like that, but stranger things have happened. I'm just not going to use my own valuable time in order to observe something I believe is wrong. So, testing: yes. Reading/acknowledging conditions in "freedom" licenses: no. "I hereby state the conditions under which you can be free". It's absurd, and fell out of relevance as soon as the first digitally signed commits were available for repo software. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: picobit on May 19, 2015, 05:06:48 PM The GPL states that you can use and modify the code in any way you like, with the sole exception that if you distribute the original or modified code you must do so under the same license. Other open source the licenses do not contain that restriction. The AGPL place the extra restriction on your use of the code that if you make a publicly facing server using modified code, you must make those modifications publicly available. In principle, that is a good idea, you then know what you get as a service. But it does restrict using other peoples codes for your own business purposes. Fortunately, Armory is not publicly facing, it has Bitcoin Core between itself and the world. Otherwise, the license would indeed prevent you from trying out such a modification! >:(
While I agree that trying to protect freedom by enforcing restrictions to said freedom is a bit absurd, it sort of makes sense under the current copyright laws, if you do not want others to make a little change to your code and then claiming it as their own - possibly taking away your own freedom to use it. Unfortunately, the open source license situation is by now completely out of control, with dozens of slightly incompatible licenses out there. Trying to create something using parts from several projects is a nightmare. It became obviously absurd once the GNU Public License stopped being compatible with itself! (version 2 and 3 are incompatible!). </rant> Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 19, 2015, 05:40:12 PM While I agree that trying to protect freedom by enforcing restrictions to said freedom is a bit absurd, it sort of makes sense under the current copyright laws, if you do not want others to make a little change to your code and then claiming it as their own - possibly taking away your own freedom to use it. You can prove you wrote the code without licensing it, and that was possible pretty much as of the advent of PGP (which was less than 10 years after Stallman's original publication of GPL). Copyright as a basic concept makes no sense for openly distributed code, so there is literally no reason to license the code at all. If someone tries to copyright your code after the fact, then the truth will be plain for all to see. Unfortunately, the open source license situation is by now completely out of control, with dozens of slightly incompatible licenses out there. Trying to create something using parts from several projects is a nightmare. It became obviously absurd once the GNU Public License stopped being compatible with itself! (version 2 and 3 are incompatible!). </rant> Glad to hear about how inconvenient this "licensed freedom" stuff is in reality (whoever would have guessed? lol). If I was a lawyer looking at that previous sentence, I would be slavering: "Delicious conflict! Don't stop, I'm almost there..." Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: picobit on May 19, 2015, 06:31:58 PM Quote Glad to hear about how inconvenient this "licensed freedom" stuff is in reality (whoever would have guessed? lol). I used to be a strong supporter of open source licenses; and I still firmly support the principle. But I must admit that they are showing signs of failure. :( BTW, copyright is not something you "claim". Copyright is something you have automatically if you write something, so if you do *not* release your software under some sort of license then you can certainly sue anybody who uses it in their own code later. Open source licenses are probably a necessary evil; the original idea being to "subvert" copyright law to do the opposite of what the law intends: to allow you to reuse the code instead of forbid it. I think that GPL and friends to a large degree have succeeded in that, but their own success (and the resulting proliferation of licenses) hold the seed to failure. I guess we are neither 100% agreeing, nor 100% disagreeing, but certainly approaching 100% off-topic... :) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 19, 2015, 07:35:56 PM All I would add is that people re-using your code or stealing it wholesale doesn't matter.
Someone passing off others ideas as their own only behave that way because they're not sufficiently creative to do it themself. They'll always end up with a magpie's nest of other people's work, so it's all self correcting. If they want new ideas, they will have to go again to those people who actually produce original stuff, you can't fool anyone behaving like that. OTOH, keeping your work proprietary and close to your chest encourages any non-contributors who are privy to the code to manipulate the situation. Proving attribution of unpublished, unsigned work in that case would be far more difficult. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: btchris on May 19, 2015, 08:18:45 PM Fortunately, Armory is not publicly facing, it has Bitcoin Core between itself and the world. Unless you choose to use, say, an armoryd instance to power your blockchain explorer or web store (these are not my words...). Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: jl2012 on May 20, 2015, 03:30:58 AM I'm building a supernode database with 0.93.99.1. Do I need to do it again when 0.94 is finalized and officially released?
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 20, 2015, 10:19:42 AM I'm building a supernode database with 0.93.99.1. Do I need to do it again when 0.94 is finalized and officially released? Unless something goes stupid, I don't intent to modify the format of the DB ever again. I don't expect I'll have to change it to fix the current bug Carlton Banks and btchris are experiencing. On the other hand I'd like some feedback on stability, since a lot of changes were to improve supernode stability. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: jl2012 on May 20, 2015, 12:44:40 PM I'm building a supernode database with 0.93.99.1. Do I need to do it again when 0.94 is finalized and officially released? Unless something goes stupid, I don't intent to modify the format of the DB ever again. I don't expect I'll have to change it to fix the current bug Carlton Banks and btchris are experiencing. On the other hand I'd like some feedback on stability, since a lot of changes were to improve supernode stability. On a VM with 4 cores and 29GB RAM, it took 8.4 hours to build to block 270000, about 1/3 of the total blockchain Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: jl2012 on May 20, 2015, 02:48:58 PM I'm building a supernode database with 0.93.99.1. Do I need to do it again when 0.94 is finalized and officially released? Unless something goes stupid, I don't intent to modify the format of the DB ever again. I don't expect I'll have to change it to fix the current bug Carlton Banks and btchris are experiencing. On the other hand I'd like some feedback on stability, since a lot of changes were to improve supernode stability. On a VM with 4 cores and 29GB RAM, it took 8.4 hours to build to block 270000, about 1/3 of the total blockchain It died around block 270000 with "segmentation fault". Nothing special in log EDIT: I restarted and it shows "Scanning from 270396 to 357270" Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 20, 2015, 03:03:44 PM It died around block 270000 with "segmentation fault". Nothing special in log EDIT: I restarted and it shows "Scanning from 270396 to 357270" Well then I guess both fullnode and supernode suffer from the same symptoms. As for the resume height, that looks correct to me. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on May 20, 2015, 03:27:00 PM It died around block 270000 with "segmentation fault". Nothing special in log EDIT: I restarted and it shows "Scanning from 270396 to 357270" Well then I guess both fullnode and supernode suffer from the same symptoms. As for the resume height, that looks correct to me. Would it help if I try building a Supernode Db? I'm retiring the 0.9.x bitcoin blockchain from my system disk, so I'll have the space and the patience to let it build. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: jl2012 on May 20, 2015, 03:34:34 PM It died around block 270000 with "segmentation fault". Nothing special in log EDIT: I restarted and it shows "Scanning from 270396 to 357270" Well then I guess both fullnode and supernode suffer from the same symptoms. As for the resume height, that looks correct to me. How do I know if it's running properly? Wait until it finishes 272500? EDIT: It passed 272500. Can I assume my database is not corrupted? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 20, 2015, 04:20:15 PM EDIT: It passed 272500. Can I assume my database is not corrupted? Yes Would it help if I try building a Supernode Db? I'm retiring the 0.9.x bitcoin blockchain from my system disk, so I'll have the space and the patience to let it build. I got some changes coming in soon, I'd rather you wait on those Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: jl2012 on May 21, 2015, 03:32:15 AM It died around block 270000 with "segmentation fault". Nothing special in log EDIT: I restarted and it shows "Scanning from 270396 to 357270" Well then I guess both fullnode and supernode suffer from the same symptoms. As for the resume height, that looks correct to me. How do I know if it's running properly? Wait until it finishes 272500? EDIT: It passed 272500. Can I assume my database is not corrupted? The armoryd suddenly runs very slowly and makes my VM very unresponsive It took 2089s to complete 265000-267500 3848s for 267500-270000 everything was fine up to this point it died with "segmentation fault" at 270396 and I restarted It took 3291s to complete 270396-272500 11847s to complete 272500-275000 the VM becomes very unresponsive. I renice the armoryd to 19 but seems not very helpful. Few minutes after I renice it, it died showing "Killed" but I don't think I did anything to kill it manually. The VM becomes normal after armoryd is dead. I know later blocks will take longer to process but I don't think it would suddenly take 3x longer. EDIT: after restarting armoryd it took only 3717s to complete 272867-275000. So there must be some problem before Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: jl2012 on May 21, 2015, 02:34:23 PM It became very slow again and I restarted the VM. However, it is redoing ~4000 completed blocks
-WARN - 1432179524: (BlockUtils.cpp:1071) Scanning from 272867 to 357353 -WARN - 1432183241: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 275000 -WARN - 1432190441: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 277500 -WARN - 1432212681: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 280000 (I killed it here with Control-C) Log file opened at 1432217549: /home/xxx/.armory/armorycpplog.txt -INFO - 1432217554: (BlockUtils.cpp:850) blkfile dir: /home/xxx/.bitcoin/blocks -INFO - 1432217554: (BlockUtils.cpp:851) lmdb dir: /home/xxx/.armory/databases -INFO - 1432217554: (lmdb_wrapper.cpp:439) Opening databases... -INFO - 1432217554: (BlockUtils.cpp:1181) Executing: doInitialSyncOnLoad -INFO - 1432217554: (BlockUtils.cpp:1253) Total number of blk*.dat files: 272 -INFO - 1432217554: (BlockUtils.cpp:1254) Total blockchain bytes: 36,422,961,275 -INFO - 1432217554: (BlockUtils.cpp:1628) Reading headers from db -INFO - 1432217567: (BlockUtils.cpp:1654) Found 357398 headers in db -DEBUG - 1432217567: (Blockchain.cpp:214) Organizing chain w/ rebuild -INFO - 1432217574: (BlockUtils.cpp:1295) Left off at file 271, offset 65665089 -INFO - 1432217574: (BlockUtils.cpp:1298) Reading headers and building chain... -INFO - 1432217574: (BlockUtils.cpp:1299) Starting at block file 271 offset 65665089 -INFO - 1432217574: (BlockUtils.cpp:1301) Block height 357353 -DEBUG - 1432217574: (Blockchain.cpp:214) Organizing chain w/ rebuild -INFO - 1432217575: (BlockUtils.cpp:1337) Looking for first unrecognized block -INFO - 1432217575: (BlockUtils.cpp:1489) Loading block data... file 271 offset 65665081 -ERROR - 1432217575: (BlockUtils.cpp:516) Next block header found at offset 65665089 -INFO - 1432217888: (BlockUtils.cpp:544) Reading raw blocks finished at file 271 offset 87382506 -INFO - 1432217888: (BlockUtils.cpp:1354) Wrote blocks to DB in 7.70157s -INFO - 1432217888: (BlockUtils.cpp:1371) Checking dupIDs from 276324 onward -WARN - 1432217900: (BlockWriteBatcher.cpp:1556) Starting with: -WARN - 1432217900: (BlockWriteBatcher.cpp:1557) 1 workers -WARN - 1432217900: (BlockWriteBatcher.cpp:1558) 1 writers -WARN - 1432217900: (BlockUtils.cpp:1071) Scanning from 276325 to 357427 Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 21, 2015, 05:35:32 PM It became very slow again and I restarted the VM. However, it is redoing ~4000 completed blocks -WARN - 1432179524: (BlockUtils.cpp:1071) Scanning from 272867 to 357353 -WARN - 1432183241: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 275000 -WARN - 1432190441: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 277500 -WARN - 1432212681: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 280000 (I killed it here with Control-C) Log file opened at 1432217549: /home/xxx/.armory/armorycpplog.txt -INFO - 1432217554: (BlockUtils.cpp:850) blkfile dir: /home/xxx/.bitcoin/blocks -INFO - 1432217554: (BlockUtils.cpp:851) lmdb dir: /home/xxx/.armory/databases -INFO - 1432217554: (lmdb_wrapper.cpp:439) Opening databases... -INFO - 1432217554: (BlockUtils.cpp:1181) Executing: doInitialSyncOnLoad -INFO - 1432217554: (BlockUtils.cpp:1253) Total number of blk*.dat files: 272 -INFO - 1432217554: (BlockUtils.cpp:1254) Total blockchain bytes: 36,422,961,275 -INFO - 1432217554: (BlockUtils.cpp:1628) Reading headers from db -INFO - 1432217567: (BlockUtils.cpp:1654) Found 357398 headers in db -DEBUG - 1432217567: (Blockchain.cpp:214) Organizing chain w/ rebuild -INFO - 1432217574: (BlockUtils.cpp:1295) Left off at file 271, offset 65665089 -INFO - 1432217574: (BlockUtils.cpp:1298) Reading headers and building chain... -INFO - 1432217574: (BlockUtils.cpp:1299) Starting at block file 271 offset 65665089 -INFO - 1432217574: (BlockUtils.cpp:1301) Block height 357353 -DEBUG - 1432217574: (Blockchain.cpp:214) Organizing chain w/ rebuild -INFO - 1432217575: (BlockUtils.cpp:1337) Looking for first unrecognized block -INFO - 1432217575: (BlockUtils.cpp:1489) Loading block data... file 271 offset 65665081 -ERROR - 1432217575: (BlockUtils.cpp:516) Next block header found at offset 65665089 -INFO - 1432217888: (BlockUtils.cpp:544) Reading raw blocks finished at file 271 offset 87382506 -INFO - 1432217888: (BlockUtils.cpp:1354) Wrote blocks to DB in 7.70157s -INFO - 1432217888: (BlockUtils.cpp:1371) Checking dupIDs from 276324 onward -WARN - 1432217900: (BlockWriteBatcher.cpp:1556) Starting with: -WARN - 1432217900: (BlockWriteBatcher.cpp:1557) 1 workers -WARN - 1432217900: (BlockWriteBatcher.cpp:1558) 1 writers -WARN - 1432217900: (BlockUtils.cpp:1071) Scanning from 276325 to 357427 The wording is off. It signals the highest buffered block, not the highest processed and written block (cause I was lazy and didn't change this kinda stuff) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: jl2012 on May 21, 2015, 05:56:55 PM It became very slow again and I restarted the VM. However, it is redoing ~4000 completed blocks -WARN - 1432179524: (BlockUtils.cpp:1071) Scanning from 272867 to 357353 -WARN - 1432183241: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 275000 -WARN - 1432190441: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 277500 -WARN - 1432212681: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 280000 (I killed it here with Control-C) Log file opened at 1432217549: /home/xxx/.armory/armorycpplog.txt -INFO - 1432217554: (BlockUtils.cpp:850) blkfile dir: /home/xxx/.bitcoin/blocks -INFO - 1432217554: (BlockUtils.cpp:851) lmdb dir: /home/xxx/.armory/databases -INFO - 1432217554: (lmdb_wrapper.cpp:439) Opening databases... -INFO - 1432217554: (BlockUtils.cpp:1181) Executing: doInitialSyncOnLoad -INFO - 1432217554: (BlockUtils.cpp:1253) Total number of blk*.dat files: 272 -INFO - 1432217554: (BlockUtils.cpp:1254) Total blockchain bytes: 36,422,961,275 -INFO - 1432217554: (BlockUtils.cpp:1628) Reading headers from db -INFO - 1432217567: (BlockUtils.cpp:1654) Found 357398 headers in db -DEBUG - 1432217567: (Blockchain.cpp:214) Organizing chain w/ rebuild -INFO - 1432217574: (BlockUtils.cpp:1295) Left off at file 271, offset 65665089 -INFO - 1432217574: (BlockUtils.cpp:1298) Reading headers and building chain... -INFO - 1432217574: (BlockUtils.cpp:1299) Starting at block file 271 offset 65665089 -INFO - 1432217574: (BlockUtils.cpp:1301) Block height 357353 -DEBUG - 1432217574: (Blockchain.cpp:214) Organizing chain w/ rebuild -INFO - 1432217575: (BlockUtils.cpp:1337) Looking for first unrecognized block -INFO - 1432217575: (BlockUtils.cpp:1489) Loading block data... file 271 offset 65665081 -ERROR - 1432217575: (BlockUtils.cpp:516) Next block header found at offset 65665089 -INFO - 1432217888: (BlockUtils.cpp:544) Reading raw blocks finished at file 271 offset 87382506 -INFO - 1432217888: (BlockUtils.cpp:1354) Wrote blocks to DB in 7.70157s -INFO - 1432217888: (BlockUtils.cpp:1371) Checking dupIDs from 276324 onward -WARN - 1432217900: (BlockWriteBatcher.cpp:1556) Starting with: -WARN - 1432217900: (BlockWriteBatcher.cpp:1557) 1 workers -WARN - 1432217900: (BlockWriteBatcher.cpp:1558) 1 writers -WARN - 1432217900: (BlockUtils.cpp:1071) Scanning from 276325 to 357427 The wording is off. It signals the highest buffered block, not the highest processed and written block (cause I was lazy and didn't change this kinda stuff) So it took 8 hours to buffer ~5000 blocks without processing? How could I know the actual progress? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 21, 2015, 06:14:48 PM It became very slow again and I restarted the VM. However, it is redoing ~4000 completed blocks -WARN - 1432179524: (BlockUtils.cpp:1071) Scanning from 272867 to 357353 -WARN - 1432183241: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 275000 -WARN - 1432190441: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 277500 -WARN - 1432212681: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 280000 (I killed it here with Control-C) Log file opened at 1432217549: /home/xxx/.armory/armorycpplog.txt -INFO - 1432217554: (BlockUtils.cpp:850) blkfile dir: /home/xxx/.bitcoin/blocks -INFO - 1432217554: (BlockUtils.cpp:851) lmdb dir: /home/xxx/.armory/databases -INFO - 1432217554: (lmdb_wrapper.cpp:439) Opening databases... -INFO - 1432217554: (BlockUtils.cpp:1181) Executing: doInitialSyncOnLoad -INFO - 1432217554: (BlockUtils.cpp:1253) Total number of blk*.dat files: 272 -INFO - 1432217554: (BlockUtils.cpp:1254) Total blockchain bytes: 36,422,961,275 -INFO - 1432217554: (BlockUtils.cpp:1628) Reading headers from db -INFO - 1432217567: (BlockUtils.cpp:1654) Found 357398 headers in db -DEBUG - 1432217567: (Blockchain.cpp:214) Organizing chain w/ rebuild -INFO - 1432217574: (BlockUtils.cpp:1295) Left off at file 271, offset 65665089 -INFO - 1432217574: (BlockUtils.cpp:1298) Reading headers and building chain... -INFO - 1432217574: (BlockUtils.cpp:1299) Starting at block file 271 offset 65665089 -INFO - 1432217574: (BlockUtils.cpp:1301) Block height 357353 -DEBUG - 1432217574: (Blockchain.cpp:214) Organizing chain w/ rebuild -INFO - 1432217575: (BlockUtils.cpp:1337) Looking for first unrecognized block -INFO - 1432217575: (BlockUtils.cpp:1489) Loading block data... file 271 offset 65665081 -ERROR - 1432217575: (BlockUtils.cpp:516) Next block header found at offset 65665089 -INFO - 1432217888: (BlockUtils.cpp:544) Reading raw blocks finished at file 271 offset 87382506 -INFO - 1432217888: (BlockUtils.cpp:1354) Wrote blocks to DB in 7.70157s -INFO - 1432217888: (BlockUtils.cpp:1371) Checking dupIDs from 276324 onward -WARN - 1432217900: (BlockWriteBatcher.cpp:1556) Starting with: -WARN - 1432217900: (BlockWriteBatcher.cpp:1557) 1 workers -WARN - 1432217900: (BlockWriteBatcher.cpp:1558) 1 writers -WARN - 1432217900: (BlockUtils.cpp:1071) Scanning from 276325 to 357427 The wording is off. It signals the highest buffered block, not the highest processed and written block (cause I was lazy and didn't change this kinda stuff) So it took 8 hours to buffer ~5000 blocks without processing? How could I know the actual progress? Buffering, processing and writing data are all different tasks that take place in parallel. A batch of blocks is buffered, processed and written, then the batch is cleaned up. Only after a batch has been written will the DB resume past that point. The buffering threads have a bottleneck (only 3 buffers in the waiting queue at most). The processing threads have no bottleneck (and usually out pace the writing threads in supernode) which is why the scanning is getting really slow (RAM gets filled with too much data to allow the writers to run properly). As for why this happens, it's cause I forgot to turn the processing threads bottleneck back on when I was doing some tests o.o"! As for progress, I'll improve the log message with the upcoming fixes. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: jl2012 on May 21, 2015, 06:40:44 PM As for progress, I'll improve the log message with the upcoming fixes. Thanks. I hope it could be logged by fixed time interval, not by block interval. It would be very helpful for slow machine like mine. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on May 21, 2015, 10:03:48 PM As for progress, I'll improve the log message with the upcoming fixes. Thanks. I hope it could be logged by fixed time interval, not by block interval. It would be very helpful for slow machine like mine. That's a good idea, I'll do that. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: jl2012 on May 26, 2015, 03:30:45 PM As for progress, I'll improve the log message with the upcoming fixes. Thanks. I hope it could be logged by fixed time interval, not by block interval. It would be very helpful for slow machine like mine. That's a good idea, I'll do that. I have to give up at about block 290000. It takes ~20s to process a block. The worst thing is it freezes the VM My VM's config: Host: Hyper-V 2012 R2 OS: Debian jessie RAM: 29GB CPU: Intel i5 Quad-core HD: RAID 1 (non-SSD) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 07, 2015, 09:31:57 PM New version out in ffreeze, please test away. There's a wealth of robustness changes and it should also be a bit faster. However no one at Armory has experienced the issues Carlton Banks and btcchris ran into, so most of these changes were speculative. It's possible these bugs will still be there partially or in full.
As usual, looking forward to your bug reports and logs, and again thank you for the time and effort you put in helping us to test things out =) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 07, 2015, 10:52:08 PM Having trouble getting past this stage in the DB build:
Code: -INFO - 1433717171: (BlockUtils.cpp:1337) Looking for first unrecognized block CPU cores hit the floor, that block never gets found. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 07, 2015, 11:45:30 PM Having trouble getting past this stage in the DB build: Code: -INFO - 1433717171: (BlockUtils.cpp:1337) Looking for first unrecognized block CPU cores hit the floor, that block never gets found. Fresh DB folder? Is it maxing one core or all cores? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 07, 2015, 11:59:23 PM Having trouble getting past this stage in the DB build: Code: -INFO - 1433717171: (BlockUtils.cpp:1337) Looking for first unrecognized block CPU cores hit the floor, that block never gets found. Fresh DB folder? Is it maxing one core or all cores? Yes, fresh DB. CPU activity cuts out altogether (occasional 1% blip for python). Logs and gdb output all look uninteresting. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 08, 2015, 12:14:43 AM Having trouble getting past this stage in the DB build: Code: -INFO - 1433717171: (BlockUtils.cpp:1337) Looking for first unrecognized block CPU cores hit the floor, that block never gets found. Fresh DB folder? Is it maxing one core or all cores? Yes, fresh DB. CPU activity cuts out altogether (occasional 1% blip for python). Logs and gdb output all look uninteresting. This is weird I didn't touch that part of the code. I'll get back to you when it's fixed (hopefully I can reproduce it) What's the last line in the terminal? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 08, 2015, 12:30:15 AM This is weird I didn't touch that part of the code. I'll get back to you when it's fixed (hopefully I can reproduce it) It happens to me every time it runs. So far. (4-5 attempts) What's the last line in the terminal? Code: -DEBUG - 1433717168: (Blockchain.cpp:214) Organizing chain w/ rebuild The SIGTERM was generated by me, issuing a kill is the only way to get it shut down. This is consistent also. There is a difference to my setup, in that I'm using jessie instead of wheezy, but I don't think that's the source of the problem. I know, better to keep the setup the same, but it's a long story, and 93.1 works identically on jessie as it does on wheezy. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 08, 2015, 12:42:43 AM There is a difference to my setup, in that I'm using jessie instead of wheezy, but I don't think that's the source of the problem. I know, better to keep the setup the same, but it's a long story, and 93.1 works identically on jessie as it does on wheezy. It should work on any setup as long as the compiler is doing its job really. Can you get through with testnet? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 08, 2015, 09:26:33 PM Ok this really is totally consistent behaviour. Tried not rebuilding the Db, just let it carry on from where it hits the (infinite?) loop. The headers get read again (implying the data structure for the headers wasn't committed to disk in the failed run), then it hits the same loop. I could try this with testnet I guess, but that's a fair amount of setup to confirm something so fundamental and consistent.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 08, 2015, 11:30:29 PM Ok this really is totally consistent behaviour. Tried not rebuilding the Db, just let it carry on from where it hits the (infinite?) loop. The headers get read again (implying the data structure for the headers wasn't committed to disk in the failed run), then it hits the same loop. I could try this with testnet I guess, but that's a fair amount of setup to confirm something so fundamental and consistent. I've reproduced this issue and I'm on it. I've also got a clue on the other one in the process so I may get both by the next commit. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: jl2012 on June 09, 2015, 03:20:37 PM New version out in ffreeze, please test away. There's a wealth of robustness changes and it should also be a bit faster. However no one at Armory has experienced the issues Carlton Banks and btcchris ran into, so most of these changes were speculative. It's possible these bugs will still be there partially or in full. As usual, looking forward to your bug reports and logs, and again thank you for the time and effort you put in helping us to test things out =) Have you made any big change re supernode? In the previous ffreeze it did not complete even with a week. In this version it took only a few hours to complete. I can now get the details of a random tx with "armoryd.py gettransaction txid". I guess this means the supernode in running properly? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 09, 2015, 06:47:08 PM New version out in ffreeze, please test away. There's a wealth of robustness changes and it should also be a bit faster. However no one at Armory has experienced the issues Carlton Banks and btcchris ran into, so most of these changes were speculative. It's possible these bugs will still be there partially or in full. As usual, looking forward to your bug reports and logs, and again thank you for the time and effort you put in helping us to test things out =) Have you made any big change re supernode? In the previous ffreeze it did not complete even with a week. In this version it took only a few hours to complete. Quote I made changes to some stuff that makes supernode faster (some weird opts I should have rolled but figured it could wait for fear it would be unstable). The other issue, with pulling block data and the object lifespan remains. It's a rare occurrence but common enough to trip certain setup quite often (2 out of my 3 testers T_T). I ended rolled the optimization with the stability changes (it didn't cost that much more to add the opts at that stage) Now I'm also convinced that the code speeding up supernode is not responsible for the instability. That gives me room to add some more optimizations (again supernode only) without fear of making bugs more convoluted (and thus harder to isolated). Not having a clue what the issue was, I ended up fixing what you were suffering from (which was a lot more obvious than the remaining issue) in an attempt to fix the bug they are experiencing. In grand scheme of things it doesn't matter what order I fix these, since I have to go after both anyways. In the current scope, it doesn't speak so good of me since I thought I was fixing one thing and got the other instead =P Quote I can now get the details of a random tx with "armoryd.py gettransaction txid". I guess this means the supernode in running properly? Not really, that just means you are using supernode indeed (which is the only the DB format now that support random txhash resolution), but that doesn't mean supernode is working per se. What you need to do to test it is to add a random bitcoin address to a wallet and getledger on the wallet, which will display that random address history along with the rest. You can also verify this in the UI by importing some publicly known private keys to a wallet (kind of an oxymoron I know) and verify they get imported quasi instantly and that their history shows up in the main ledger. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Roy Badami on June 09, 2015, 07:01:50 PM (On OS X)
No errors building the database - but doesn't find any transactions and all wallet balances are 0.0 EDIT: Ah, OK, database appears to build OK judging by progress, but then scanning transactions seems to complete almost instantly (so evidently it actually failed - but didn't give an error). roy Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 09, 2015, 08:02:48 PM (On OS X) No errors building the database - but doesn't find any transactions and all wallet balances are 0.0 EDIT: Ah, OK, database appears to build OK judging by progress, but then scanning transactions seems to complete almost instantly (so evidently it actually failed - but didn't give an error). roy Lemme see the last few lines of your log file. Also, what's the top block displayed in Armory? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Roy Badami on June 09, 2015, 08:15:17 PM Ah, after several restarts of Armory it finally started scanning - made it most of the way there - now has transactions up to late 2014. There's virtually nothing interesting in armorylog.txt.
I get the following error at startup (which I presume is largely harmless?) Code: 2015-06-09 20:54 (INFO) -- announcefetch.py:271 - Fetching: https://bitcoinarmory.com/announce.txt And apart from that littereally nothing of interest; just a couple of new blocks that came in during the scan: Code: 2015-06-09 20:57 (INFO) -- ArmoryQt.py:3132 - Current block number: 360187 EDIT: Fixed log paste Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Roy Badami on June 09, 2015, 08:50:34 PM Just reran it having removed the database and log files. Takes about 12 minutes to go through receiving headers, organizing blockchain and building databases - and then goes through the scan transactions phase in well under a minute (and ends up showing no transactions and 0.0 balance). I'll send you full logs of that run if you let me know the best way to get them to you.
EDIT: Block number displayed in the GUI is correct. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 09, 2015, 09:01:45 PM Gimme the part of the log where it posts the timings after the scan is over. I want to see the last block it scanned.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Roy Badami on June 09, 2015, 09:12:46 PM Ah, interesting. In armorycpp.log, it looks like it got to around block 130,000 and then readjusted the number of threads.
Code: -WARN - 1433882811: (BlockUtils.cpp:1071) Scanning from 0 to 360192 Is this thread count adjustment something I can try turning off? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Roy Badami on June 09, 2015, 09:37:15 PM I should also point out that this is my first time building in a very long time (I think the last time was before you offered prebuilt Mac binaries) so in particular I have absolutely no evidence that there isn't something subtly borked in my build environment.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 09, 2015, 09:40:07 PM This isn't the thread adjustment, it's failing to grab the block data or the data has expired before it got to process it. It's consistent with the symptoms of the issue I'm going after ATM so I'll have you wait for the fix to try this again. Right now I don't think there is anything else you can do about it.
If you want to manually set your thread counts regardless, go inside BlockWriteBatcher.cpp and do the following: 1) comment out line 2004 to prevent thread adjustment 2) hardcode values in the BlockWriteBatcher ctor (line 112 for fullnode, 99 for supernode) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Roy Badami on June 09, 2015, 10:46:05 PM Thanks. I'm sure you're right, but I might try nailing down the thread count anyway. (Probably not tonight though given how long it takes to rebuild on this slow machine. Is there no way of doing an incremental build on a Mac?)
roy Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Roy Badami on June 10, 2015, 12:15:09 AM If you want to manually set your thread counts regardless, go inside BlockWriteBatcher.cpp and do the following: 1) comment out line 2004 to prevent thread adjustment 2) hardcode values in the BlockWriteBatcher ctor (line 112 for fullnode, 99 for supernode) Could well be just chance since the behaviour before was clearly somewhat nondeterministric. But I did the above and it synced my wallets perfectly first time (and I hadn't managed to get a successfully sync at all so far up till now). If I get the chance tomorrow I'll nuke the databases and try again to confirm whether this is repeatable. Just to confirm, only change I made was as you suggested: Code: diff --git a/cppForSwig/BlockWriteBatcher.cpp b/cppForSwig/BlockWriteBatcher.cpp Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: doug_armory on June 10, 2015, 12:17:42 AM I should also point out that this is my first time building in a very long time (I think the last time was before you offered prebuilt Mac binaries) so in particular I have absolutely no evidence that there isn't something subtly borked in my build environment. The OS X build process is mostly self-contained. Knock on wood and all but I've found that even the slightest hiccups cause Armory to not build properly. In other words, your environment's probably fine. :) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 10, 2015, 02:42:03 AM Could well be just chance since the behaviour before was clearly somewhat nondeterministric. But I did the above and it synced my wallets perfectly first time (and I hadn't managed to get a successfully sync at all so far up till now). If I get the chance tomorrow I'll nuke the databases and try again to confirm whether this is repeatable. Well yes and no. I expect the issue is object life span: one type of threads creates some data that gets cleaned up before the next type of threads is done with it. If you reduce the amount of threads per groups to the bare minimum, you also reduce the "surface area" for the bug and thus the chance it will occur. Quote Just to confirm, only change I made was as you suggested: That's the proper change to force thread count for all groups down to 1. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: jl2012 on June 10, 2015, 09:19:54 AM New version out in ffreeze, please test away. There's a wealth of robustness changes and it should also be a bit faster. However no one at Armory has experienced the issues Carlton Banks and btcchris ran into, so most of these changes were speculative. It's possible these bugs will still be there partially or in full. As usual, looking forward to your bug reports and logs, and again thank you for the time and effort you put in helping us to test things out =) Have you made any big change re supernode? In the previous ffreeze it did not complete even with a week. In this version it took only a few hours to complete. Quote I made changes to some stuff that makes supernode faster (some weird opts I should have rolled but figured it could wait for fear it would be unstable). The other issue, with pulling block data and the object lifespan remains. It's a rare occurrence but common enough to trip certain setup quite often (2 out of my 3 testers T_T). I ended rolled the optimization with the stability changes (it didn't cost that much more to add the opts at that stage) Now I'm also convinced that the code speeding up supernode is not responsible for the instability. That gives me room to add some more optimizations (again supernode only) without fear of making bugs more convoluted (and thus harder to isolated). Not having a clue what the issue was, I ended up fixing what you were suffering from (which was a lot more obvious than the remaining issue) in an attempt to fix the bug they are experiencing. In grand scheme of things it doesn't matter what order I fix these, since I have to go after both anyways. In the current scope, it doesn't speak so good of me since I thought I was fixing one thing and got the other instead =P Quote I can now get the details of a random tx with "armoryd.py gettransaction txid". I guess this means the supernode in running properly? Not really, that just means you are using supernode indeed (which is the only the DB format now that support random txhash resolution), but that doesn't mean supernode is working per se. What you need to do to test it is to add a random bitcoin address to a wallet and getledger on the wallet, which will display that random address history along with the rest. You can also verify this in the UI by importing some publicly known private keys to a wallet (kind of an oxymoron I know) and verify they get imported quasi instantly and that their history shows up in the main ledger. 1. I compiled the latest ffreeze. Removed previous databases folder. 2. I started armoryd in supernode mode. It finished "Reading raw blocks finished at file xxx offset xxxxx" in a few hours 3. At this time, I can get the details of a random tx with "armoryd.py gettransaction txid". However, I didn't tested by importing private key. 4. At block 360206, and armoryd says "BDM thread failed: The scanning process interrupted unexpectedly" and asks me to rebuild and rescan. I killed armoryd by Control-C. 5. I started armory-qt. I didn't request a rescan but it seems doing that automatically. 6. The console started to show "Finished applying blocks up to xxxxxx". This message did not show up before the crash. The progress became slower and slower. I knew it will never complete and closed armory-qt. 7. I restored the previous databased folder by last ffreeze. It was incomplete and I stopped it at about block 300,000 after more than a week of scanning 8. I started armory-qt. No re-scan. 9. I imported this private key: 5JdeC9P7Pbd1uGdFVEsJ41EkEnADbbHGq6p1BwFxm6txNBsQnsw (SHA256 of "1") for the address 12AKRNHpFhDSBDD9rSn74VAzZSL3774PxQ. It only shows the 2 txs in 2014 but not the latest ones of 2015. So I guess both databases are somehow corrupted? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 10, 2015, 03:48:37 PM 4. At block 360206, and armoryd says "BDM thread failed: The scanning process interrupted unexpectedly" and asks me to rebuild and rescan. I killed armoryd by Control-C. That's consistent with the bug I'm going after currently. That error is too aggressive, half the time you can ignore it and restart without rescanning. Quote 5. I started armory-qt. I didn't request a rescan but it seems doing that automatically. There will be an empty file named either rescan.flag or rebuild .flag in your datadir when the previous instance of Armory wants to signal the next one to rebuild/rescan. Delete it and it won't. Quote 7. I restored the previous databased folder by last ffreeze. It was incomplete and I stopped it at about block 300,000 after more than a week of scanning 8. I started armory-qt. No re-scan. That's odd, it should try to catch up from 300,000 onwards Quote 9. I imported this private key: 5JdeC9P7Pbd1uGdFVEsJ41EkEnADbbHGq6p1BwFxm6txNBsQnsw (SHA256 of "1") for the address 12AKRNHpFhDSBDD9rSn74VAzZSL3774PxQ. It only shows the 2 txs in 2014 but not the latest ones of 2015. Expectable if the scan ended around block 300,000 Quote So I guess both databases are somehow corrupted? Shouldn't be, but I guess they are. Actually what's off is that they don't pick up where they left. I'd say hang on to your 300k DB, I intent to add some more checks to allow the DB to fix itself (instead of rebuild/rescan). Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 19, 2015, 02:26:08 AM New version. This one should fix the segfaults mid scan. Still working on the others ones. Pull away =)
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 19, 2015, 09:50:25 AM Same problems (not segfault).
The Db build gets so far: Code: [Thread 0x7fffca34b700 (LWP 3812) exited] ...and gets stuck, python process is more or less idling ~1% CPU. No errors in the logs. Quitting Armory doesn't work in this state, it does respond to ending the process. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 19, 2015, 10:15:50 AM That's quite interesting considering I added verbose at this stage for this specific reason (was curious how deep it would get before failing). I guess it doesn't even get that far.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 19, 2015, 10:26:06 AM Double-checked the BTCARMORY_BUILD string, definitely compiling from the latest commit.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 19, 2015, 12:16:06 PM How long do you let this part run until you conclude it is hanging?
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 19, 2015, 12:36:51 PM How long do you let this part run until you conclude it is hanging? 5-10 mins. Could there be something intended happening for longer than that length of time that consumes ~1% CPU on all cores? It's not consistent with the behaviour of your first couple of 0.94 commits, they took seconds to handle that stage. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 19, 2015, 12:49:52 PM How long do you let this part run until you conclude it is hanging? 5-10 mins. Could there be something intended happening for longer than that length of time that consumes ~1% CPU on all cores? It's not consistent with the behaviour of your first couple of 0.94 commits, they took seconds to handle that stage. Glancing at the code, it's pretty old and inefficient, but that wouldn't warrant that long a wait. This is probably some sort of dead lock. Unfortunately I haven't managed to reproduce it in this version (which is why I thought it was fixed), so it's gonna take me some time to figure it out. I expect this is the last hurdle left for this code to be stable and functional. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 19, 2015, 01:06:55 PM How long do you let this part run until you conclude it is hanging? 5-10 mins. Could there be something intended happening for longer than that length of time that consumes ~1% CPU on all cores? It's not consistent with the behaviour of your first couple of 0.94 commits, they took seconds to handle that stage. Glancing at the code, it's pretty old and inefficient, but that wouldn't warrant that long a wait. This is probably some sort of dead lock. Unfortunately I haven't managed to reproduce it in this version (which is why I thought it was fixed), so it's gonna take me some time to figure it out. I expect this is the last hurdle left for this code to be stable and functional. Just to totally clarify, I cannot get this or the build from your last commit to behave in any way except for this bug. It does this every time I load it up. Carefully deleting old Db and logs every run. Tested that Armory 93.1 & 93.2 work normally on the same VM (I cloned it from a working bitcoin-configured VM to begin with, but you never know). I'm pretty sure what you're saying about deadlocks sounds about right, from what I recall about how threading works. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: doug_armory on June 19, 2015, 02:50:49 PM I'm pretty sure what you're saying about deadlocks sounds about right, from what I recall about how threading works. Deadlocks can be a nightmare to debug. Threads are great in many ways. Debugging is not one of them. :( Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 19, 2015, 02:58:29 PM I'm pretty sure what you're saying about deadlocks sounds about right, from what I recall about how threading works. Deadlocks can be a nightmare to debug. Threads are great in many ways. Debugging is not one of them. :( I heard there were new processor instructions to abstract a bit of the complexity away from the programming logic, but Intel screwed up the implementation somehow and it's not quite "on the streets" yet. But all the specs are finalised as far as I remember, I believe they're calling it "lock elision" in C++, although I may be remembering wrong. So there's some more C++ to render the old standard obsolete for us all to look forward to :D. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 19, 2015, 07:14:53 PM Just to totally clarify, I cannot get this or the build from your last commit to behave in any way except for this bug. It does this every time I load it up. Carefully deleting old Db and logs every run. Tested that Armory 93.1 & 93.2 work normally on the same VM (I cloned it from a working bitcoin-configured VM to begin with, but you never know). And that bit is what's disturbing me considering this part uses the same code across version. Again, this is pretty old code, so I'm puzzled as to why it starts failing now. Anyways, I think I got a clue as to what's going on. I heard there were new processor instructions to abstract a bit of the complexity away from the programming logic, but Intel screwed up the implementation somehow and it's not quite "on the streets" yet. But all the specs are finalised as far as I remember, I believe they're calling it "lock elision" in C++, although I may be remembering wrong. So there's some more C++ to render the old standard obsolete for us all to look forward to :D. That's pretty fancy. It doesn't really simplify much. It looks like it's meant to speed up interlocked operations that don't overlap in memory, and it's only in some Haswell CPUs so far. I suspect it's the kind of extra assembly optimization that I will personally never have to use, as either compilers or OSes will upgrade their locks with HLE. The way it looks like, there is little reason not to standardize it for every lock operation. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Roy Badami on June 19, 2015, 07:27:40 PM Could well be just chance since the behaviour before was clearly somewhat nondeterministric. But I did the above and it synced my wallets perfectly first time (and I hadn't managed to get a successfully sync at all so far up till now). If I get the chance tomorrow I'll nuke the databases and try again to confirm whether this is repeatable. Well yes and no. I expect the issue is object life span: one type of threads creates some data that gets cleaned up before the next type of threads is done with it. If you reduce the amount of threads per groups to the bare minimum, you also reduce the "surface area" for the bug and thus the chance it will occur. Quote Just to confirm, only change I made was as you suggested: That's the proper change to force thread count for all groups down to 1. Is there any point in my trying the new version or will this version just behave the same for me? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 19, 2015, 07:31:58 PM Could well be just chance since the behaviour before was clearly somewhat nondeterministric. But I did the above and it synced my wallets perfectly first time (and I hadn't managed to get a successfully sync at all so far up till now). If I get the chance tomorrow I'll nuke the databases and try again to confirm whether this is repeatable. Well yes and no. I expect the issue is object life span: one type of threads creates some data that gets cleaned up before the next type of threads is done with it. If you reduce the amount of threads per groups to the bare minimum, you also reduce the "surface area" for the bug and thus the chance it will occur. Quote Just to confirm, only change I made was as you suggested: That's the proper change to force thread count for all groups down to 1. Is there any point in my trying the new version or will this version just behave the same for me? I'd like you to run it, the main fix in this version is supposed to cover your issue. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Roy Badami on June 19, 2015, 07:36:11 PM Could well be just chance since the behaviour before was clearly somewhat nondeterministric. But I did the above and it synced my wallets perfectly first time (and I hadn't managed to get a successfully sync at all so far up till now). If I get the chance tomorrow I'll nuke the databases and try again to confirm whether this is repeatable. Well yes and no. I expect the issue is object life span: one type of threads creates some data that gets cleaned up before the next type of threads is done with it. If you reduce the amount of threads per groups to the bare minimum, you also reduce the "surface area" for the bug and thus the chance it will occur. Quote Just to confirm, only change I made was as you suggested: That's the proper change to force thread count for all groups down to 1. Is there any point in my trying the new version or will this version just behave the same for me? I'd like you to run it, the main fix in this version is supposed to cover your issue. Ah, glad I asked, then :-) I initially assumed that since I wasn't seeing a segfault there wasn't much point in trying it. I'll give it a go and report back. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 19, 2015, 07:41:40 PM I heard there were new processor instructions to abstract a bit of the complexity away from the programming logic, but Intel screwed up the implementation somehow and it's not quite "on the streets" yet. But all the specs are finalised as far as I remember, I believe they're calling it "lock elision" in C++, although I may be remembering wrong. So there's some more C++ to render the old standard obsolete for us all to look forward to :D. That's pretty fancy. It doesn't really simplify much. It looks like it's meant to speed up interlocked operations that don't overlap in memory, and it's only in some Haswell CPUs so far. I suspect it's the kind of extra assembly optimization that I will personally never have to use, as either compilers or OSes will upgrade their locks with HLE. The way it looks like, there is little reason not to standardize it for every lock operation. It's supposed to be some kind of speculative execution; the majority of the time that a lock is applied, there are no writes to data structures that break the intention of the code. I forget the full details though, this was years ago. Haswell was where it was borked, it was disabled in the firmware for all lines IIRC (TSX is what Intel call it). Suposedly the next 16nm Intel chip out in Q4 this year has it working correctly. Just to totally clarify, I cannot get this or the build from your last commit to behave in any way except for this bug. It does this every time I load it up. Carefully deleting old Db and logs every run. Tested that Armory 93.1 & 93.2 work normally on the same VM (I cloned it from a working bitcoin-configured VM to begin with, but you never know). And that bit is what's disturbing me considering this part uses the same code across version. Again, this is pretty old code, so I'm puzzled as to why it starts failing now. Anyways, I think I got a clue as to what's going on. Like I said, this didn't happen in the first couple of 0.94 pre-versions you came up with, not sure what you can infer from that. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 19, 2015, 07:51:00 PM Like I said, this didn't happen in the first couple of 0.94 pre-versions you came up with, not sure what you can infer from that. The usual conclusion in this case is: there always was an issue with this code, and changes in its vicinity revealed it. Otherwise it was probably occurring at low frequency, passing itself as a scan segfault of sorts. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Roy Badami on June 19, 2015, 08:35:00 PM Like I said, this didn't happen in the first couple of 0.94 pre-versions you came up with, not sure what you can infer from that. The usual conclusion in this case is: there always was an issue with this code, and changes in its vicinity revealed it. Otherwise it was probably occurring at low frequency, passing itself as a scan segfault of sorts. The usual case is "OMG how can that ever have worked?" :-) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Roy Badami on June 19, 2015, 09:01:12 PM Ok, first attempt got a considerable way through scanning transactions, then failed with a popup "BDM not ready"
This was in the logs Code: -INFO - 1434747557: (BlockUtils.cpp:1443) checking scan integrity EDIT: Before we got to this point we had two instances of: Code: -WARN - 1434747337: (BlockUtils.cpp:1448) Top scanned block does not match top block header Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 19, 2015, 09:58:23 PM Thanks, that's enough information for now
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Roy Badami on June 19, 2015, 10:25:33 PM Ok, second attempt and it ran all the way through successfully. Had eleven instances of the 'top scanned block' issue in the logs, but got all the way through and displays the correct balances.
EDIT: Both runs were with no existing databases directory. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 19, 2015, 10:28:25 PM Had eleven instances of the 'top scanned block' issue in the logs This is a marker that the multi threaded block pulling code is still somehow messy, i.e. the scan fails to pull data for a block and halts. The fixing part detects this, goes over all blocks past that point, realizes there is nothing wrong with the data (or there is and it tries to fix it) and resumes scanning from the highest committed block, on and on. Also the "attempting fix" code is something I have to go over, it's the stuff from 0.93 that is not really on point with the 0.94 changes. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Roy Badami on June 19, 2015, 10:43:11 PM Also the "attempting fix" code is something I have to go over, it's the stuff from 0.93 that is not really on point with the 0.94 changes. Ah. Hence the 'should not get here' error? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 19, 2015, 11:01:58 PM Also the "attempting fix" code is something I have to go over, it's the stuff from 0.93 that is not really on point with the 0.94 changes. Ah. Hence the 'should not get here' error? That one is somewhat nastier. It triggers if the the DB cannot find the hash of the top scanned block, which probably means the scan halted before it failed to commit any blocks at all (and returned an empty top scanned block hash). It could be the concurrency issue triggering early on or something unrelated, although I expect it's the former. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 20, 2015, 02:40:07 AM Here's a tentative fix for Roy's issue. I added some minor changes around the part Carlton is stuck on but I don't expect this is it yet. Pull away.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 20, 2015, 09:32:37 AM You have gain a little traction. Latest:
Code: -INFO - 1434792421: (BlockUtils.cpp:345) parsing headers in file 284 Still failing, but getting further through. "BDM not ready" appears as a warning dialog box, OK'ing that shuts Armory. Still a deterministic fault, so far, so part of the same issue maybe? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 20, 2015, 02:37:58 PM Still failing, but getting further through. "BDM not ready" appears as a warning dialog box, OK'ing that shuts Armory. Still a deterministic fault, so far, so part of the same issue maybe? This means you are through your original issue. I would like you to run it a few times to determine whether you are completely past that point or this was a fluke. As for the error, you are on the same thing as Roy, although the nastier form. From the looks of it, it didn't even try to scan anything! Do you have wallets registered? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 20, 2015, 02:52:52 PM Still failing, but getting further through. "BDM not ready" appears as a warning dialog box, OK'ing that shuts Armory. Still a deterministic fault, so far, so part of the same issue maybe? This means you are through your original issue. I would like you to run it a few times to determine whether you are completely past that point or this was a fluke. As for the error, you are on the same thing as Roy, although the nastier form. From the looks of it, it didn't even try to scan anything! Do you have wallets registered? No wallets in that run, been trying with and without. With wallets, logging output ends with the same error, and it also produces a little stats for the Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 20, 2015, 05:06:46 PM Previous bug seems to be squashed. Several trials and no sign of it.
BDM not ready error isn't consistent for me, I just added wallets to the armory home folder and it almost completed correctly... the scan finished, and the application was functioning fine. What didn't work was the shutdown, I got the same response as with the previous bug. Then when I tried re-opening Armory with the built database: Code: ******************************************************************************** Tried re-opening again, same Gtk-CRITICAL, then BDM not ready was thrown. New blocks didn't register in Armory, not in the logs or the GUI. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 20, 2015, 05:15:59 PM I'm told the Gtk-Critical thing is an issue with the Qt version that ships in wheezy (4.8.2, dates from 2012 or something). I use Debian wheezy for my Linux tests and I see this error all the time, without any issue, so I don't believe it is related to BDM snafus.
I've just pushed a small change to squash a dead lock and get rid of BDM not ready false positive errors. This should make it easier to identify the current state of the scanning bug. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 20, 2015, 05:53:48 PM I'm told the Gtk-Critical thing is an issue with the Qt version that ships in wheezy (4.8.2, dates from 2012 or something). I use Debian wheezy for my Linux tests and I see this error all the time, without any issue, so I don't believe it is related to BDM snafus. Up to date jessie here, Debian haven't updated much, it seems. Output from build 3fad5ecd54: Code: -INFO - 1434822571: (BlockUtils.cpp:395) reading blocks from file 285 Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 20, 2015, 06:06:18 PM Give it a couple more spins see if it always fails at the same block, please.
There is a backport version of Qt 4.8.6, that should get rid of the Gtk issue I think. EDIT: actually give it a minute before running this again, pushing some extra verbose. EDIT2: should be commit 0008c05 Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 20, 2015, 09:37:22 PM From 0008c05:
Code: -INFO - 1434835078: (BlockUtils.cpp:1403) Wrote blocks to DB in 78.2405s Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 20, 2015, 09:49:46 PM It seems to me like the thread count readjustment triggers the BDM not ready's. If it dodges the bug, it adjusts about 3 times during the first 160,000 blocks, then the threads totally smother the CPU cores. Sometimes it doesn't even handle the zeroth adjust, such as the run that generated the output in the previous post.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 20, 2015, 10:05:24 PM It seems to me like the thread count readjustment triggers the BDM not ready's. If it dodges the bug, it adjusts about 3 times during the first 160,000 blocks, then the threads totally smother the CPU cores. Sometimes it doesn't even handle the zeroth adjust, such as the run that generated the output in the previous post. You were experiencing that bug before I even added the thread adjustment part. There is something wrong with the code loop through the pull thread queues I'm thinking. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 20, 2015, 10:12:23 PM Disabled thread toggling, grab 874cf1e and let's see how far it gets.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 20, 2015, 10:28:03 PM There is a backport version of Qt 4.8.6, that should get rid of the Gtk issue I think. Not too sure about that theory, just checked and I'm running Qt core 4.8.6 right now, although I was using wheezy with standard Qt a few weeks ago when I first tested pre-0.94. Disabled thread toggling, grab 874cf1e and let's see how far it gets. Will give it a try. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 20, 2015, 11:06:17 PM Not far:
Code: -INFO - 1434841472: (BlockUtils.cpp:1403) Wrote blocks to DB in 78.0345s Same result for several trials. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 22, 2015, 01:12:21 AM Let's see what the latest push can do
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 22, 2015, 09:24:47 AM Previous commit did produce one successful scan, but it mostly generated the deadlock that happens when/just before the scan begins. Scan completed through.
19c1a72e7f seems to reliably avoid the scanning deadlock, or at least it's done so twice consecutively so far. Continuing testing, trying to get a few more data points. Scanning completes. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 22, 2015, 03:19:28 PM Previous commit did produce one successful scan, but it mostly generated the deadlock that happens when/just before the scan begins. Scan completed through. Forgot to initialize the atomic flag used to signal termination (I'm an idiot). Unlike to most std templates, atomic objects are constructed with an unknown state (which actually makes sense). Obviously I knew all that and I forgot to clear the flag in the ctor... If this proves to be stable, I'll reactivate the thread toggling (which relies heavily on this flag). Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 22, 2015, 05:44:31 PM Current push is stable enough to build and scan Db 4 times completely today, using extensive/varied wallets. Zero failures so far.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 22, 2015, 06:09:24 PM Current push is stable enough to build and scan Db 4 times completely today, using extensive/varied wallets. Zero failures so far. Aight, time to re-enable the thread toggling. Bracing myself =D Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 22, 2015, 06:18:18 PM Current push is stable enough to build and scan Db 4 times completely today, using extensive/varied wallets. Zero failures so far. Aight, time to re-enable the thread toggling. Bracing myself =D My CPU heatsink is doing the same :D Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 22, 2015, 06:20:10 PM Current push is stable enough to build and scan Db 4 times completely today, using extensive/varied wallets. Zero failures so far. Aight, time to re-enable the thread toggling. Bracing myself =D My CPU heatsink is doing the same :D I plan on adding at least a command line arg to limit the amount of available cores for the scan. The current setting is to squeeze as much CPU time as available. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 22, 2015, 09:56:43 PM Looks good. Made it through 3 full build/scans cleanly. Thread toggler only adjusts once, whereas it typically did so at least twice with the commit it was last enabled for (the thread toggler seemed to provoke the crashes before). CPU still gets tortured. It handles being interrupted during the scan OK too; it picked up where it left off fine, did 2 thread adjustments (the only time that happened) and finished the job. If this isn't good enough for the testing release, then it's probably pretty close.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 23, 2015, 09:49:30 PM If this isn't good enough for the testing release, then it's probably pretty close. Looks like we are going with this for the testing release + a couple minor bugs I'm still working on. Thanks for your invaluable help guys, in particular Carlton Banks =). I guess etotheipi will create a brand new topic for the testing builds, sometimes this week. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 23, 2015, 10:32:10 PM DB build performance seems now to be outpacing the increase to the blockchain size. With my 4 year old quad core machine, OS and blockchain running from an SSD, I'm getting ~45 minutes to set up Armory, and that's including transaction scanning for several wallets that pretty much covers every bitcoin transaction I've made since 2012. I dare not try, but I suspect that 93.x might take 2-3 times longer to do that job.
Great stuff, goatpig (and I think Nick was involved too?). Thanks. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 23, 2015, 10:58:09 PM DB build performance seems now to be outpacing the increase to the blockchain size. With my 4 year old quad core machine, OS and blockchain running from an SSD, I'm getting ~45 minutes to set up Armory, and that's including transaction scanning for several wallets that pretty much covers every bitcoin transaction I've made since 2012. I dare not try, but I suspect that 93.x might take 2-3 times longer to do that job. Keep in mind that most the effort was in the scanning code. The building code is still single threaded so a lot of speed can be squeezed there as well (in due time). Great stuff, goatpig (and I think Nick was involved too?). Thanks. Who's Nick o.o" Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 23, 2015, 11:33:49 PM DB build performance seems now to be outpacing the increase to the blockchain size. With my 4 year old quad core machine, OS and blockchain running from an SSD, I'm getting ~45 minutes to set up Armory, and that's including transaction scanning for several wallets that pretty much covers every bitcoin transaction I've made since 2012. I dare not try, but I suspect that 93.x might take 2-3 times longer to do that job. Keep in mind that most the effort was in the scanning code. The building code is still single threaded so a lot of speed can be squeezed there as well (in due time). That sounds impressive, I just assumed the building workload didn't translate well to a multi-threaded design. You know those people that say stuff like "3 GHz? 8 GB RAM? You can't even use that spec for anything! More processing than anyone could use!". Well, I'm imagining a machine with, say, an 8 core CPU, 32 GB RAM, a PCI express SSD with NVMe, plus a 100 Mbit internet connection. That's probably a mid level dev machine here and now, and I think it could setup Armory 0.94 and Core 0.10 in about 2 hours, going flat out. And the performance of both those pieces of software are set to improve further in the future? I'm slightly amazed that so much efficiency can be gained (and that we have hardware suited to the task :D) Great stuff, goatpig (and I think Nick was involved too?). Thanks. Who's Nick o.o"Lol, I have clearly forgotten the name. You have another dev who is a Linux type, Andy? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 24, 2015, 03:16:36 AM That sounds impressive, I just assumed the building workload didn't translate well to a multi-threaded design. As with most transactional DB, there is only 1 write transaction active at any given time, so you can't have several threads writing to the same DB concurrently. In this aspect, multi-threading the build code seems pointless. Because of this, I simply grand-fathered the original build code into the backend overhaul. It relied on mirroring the full content of the blockchain, and that made storage bandwidth the only significant factor in build speed, so I ignored most optimizations in this area. Now that it skips the blockchain copy in fullnode, a meaningful portion of the building phase is taken by block parsing and processing, so there is enough performance to gain to justify the effort of multi-threading this part. Quote You know those people that say stuff like "3 GHz? 8 GB RAM? You can't even use that spec for anything! More processing than anyone could use!". That's the same mentality that brings large IT companies to state that tablets and mobile devices will replace PCs in every home. For their use pattern, they don't need that kind of oomph, and in a way I believe no one but gamers and professionals needed that kind of oomph until large scale P2P networks appeared. Now with Bitcoin things are quite different. With the blockchain eventually moving into big data territory, not only are big PCs relevant, but the implementation performance matters too. The faster the software, the lower the barrier to entry. If anything, my intention with these changes is for the code to scale with whatever hardware it is thrown at, so that a 10yo computer still has a chance at syncing a wallet against a full node, and that I won't have to rework everything from the ground up 10 years from now either. Quote Well, I'm imagining a machine with, say, an 8 core CPU, 32 GB RAM, a PCI express SSD with NVMe, plus a 100 Mbit internet connection. That's probably a mid level dev machine here and now, and I think it could setup Armory 0.94 and Core 0.10 in about 2 hours, going flat out. And the performance of both those pieces of software are set to improve further in the future? I'm slightly amazed that so much efficiency can be gained (and that we have hardware suited to the task Cheesy) My current machine is 6 core cpu (12 execution threads with HT), 32GB ddr4, SATA SSD RAID0 and fiber internet (~250Mb/s). And I'm thinking I should go with 64GB RAM soon. I had thought of buying a dual CPU board and go with a couple Haswell v3 Xeons but I backed off. Can't blow all my bitcoins on a PC =P. For info, it takes 280~300 sec to scan the DB with ~5k addresses, about 80sec to build, and another ~100 sec for the header reading and what not (that could be reduced to 15sec with some effort). Quote Lol, I have clearly forgotten the name. You have another dev who is a Linux type, Andy? That would be njaard (I try to avoid referring to people by their name on public forums unless they make a point of doing so). He was part of the original effort in remodeling the backend, which was deployed in 0.93. Since then he was given other responsibilities and I'm the only left with the task of completing this todo list. It still has some large entries to deal with, like blocks over P2P, lite node, full granularity coin selection, Trezor. Andy is a Python dev. You can see some of his recent work in the hotfix release (0.93.2) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: AussieHash on June 24, 2015, 09:13:33 PM My current machine is 6 core cpu (12 execution threads with HT), 32GB ddr4, SATA SSD RAID0 and fiber internet (~250Mb/s). And I'm thinking I should go with 64GB RAM soon. I had thought of buying a dual CPU board and go with a couple Haswell v3 Xeons but I backed off. Can't blow all my bitcoins on a PC =P. For info, it takes 280~300 sec to scan the DB with ~5k addresses, about 80sec to build, and another ~100 sec for the header reading and what not (that could be reduced to 15sec with some effort). Perhaps your FAQ should list minimum and recommended hardware specs. I am running my full node on a $35 ARM single board computer with 1 GB ram. The letstalkbitcoin guys say they used to use armory, and created a new tipping address for every episode, but stopped using it because it became too slow. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 24, 2015, 09:21:00 PM My current machine is 6 core cpu (12 execution threads with HT), 32GB ddr4, SATA SSD RAID0 and fiber internet (~250Mb/s). And I'm thinking I should go with 64GB RAM soon. I had thought of buying a dual CPU board and go with a couple Haswell v3 Xeons but I backed off. Can't blow all my bitcoins on a PC =P. For info, it takes 280~300 sec to scan the DB with ~5k addresses, about 80sec to build, and another ~100 sec for the header reading and what not (that could be reduced to 15sec with some effort). Perhaps your FAQ should list minimum and recommended hardware specs. I am running my full node on a $35 ARM single board computer with 1 GB ram. The letstalkbitcoin guys say they used to use armory, and created a new tipping address for every episode, but stopped using it because it became too slow. You'll find this version will be comparatively faster on anything with more than one core, I think that includes most ARM SoC's these days. And because the database for Armory drops down to 120 MB, it's much less concerning to think about the future disk space. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 25, 2015, 12:08:07 AM My current machine is 6 core cpu (12 execution threads with HT), 32GB ddr4, SATA SSD RAID0 and fiber internet (~250Mb/s). And I'm thinking I should go with 64GB RAM soon. I had thought of buying a dual CPU board and go with a couple Haswell v3 Xeons but I backed off. Can't blow all my bitcoins on a PC =P. For info, it takes 280~300 sec to scan the DB with ~5k addresses, about 80sec to build, and another ~100 sec for the header reading and what not (that could be reduced to 15sec with some effort). Perhaps your FAQ should list minimum and recommended hardware specs. I am running my full node on a $35 ARM single board computer with 1 GB ram. The letstalkbitcoin guys say they used to use armory, and created a new tipping address for every episode, but stopped using it because it became too slow. You can't compare the resource cost of initial synchronization with that of maintenance. I expect it would take your board over 2 weeks to sync the blockchain from scratch, yet it can process new blocks at an acceptable pace. You probably had to sync the chain on an actual computer then move the DB and data over to your RPi (or whatever the board is). Same goes with Armory, maintenance cost is a fraction of what it takes to build the original database. Considering 0.94 hardware requirement, I expect that board of yours can run a fully sync'ed armory daemon just fine in fullnode. As for the letstalkbitcoin guys, they were probably using an old version that had no scalability. Pre 0.9x kept all data in RAM, pre 0.93 was single threaded, relied on LevelDB (what legos are to structural steel) and had serious scalability issues which made the backend come to a dead stop once it reached critical mass. As for 0.93, it scales with RAM (lots of it). 0.94 is the first version to reduce hardware requirements (120MB disk space from 30GB+, <2GB RAM from 8+) while increasing performance AND increasing scalability. This is finally approaching good functionality and stability. It is also trivial to toggle RAM and CPU usage in this version, so I will most likely add command args to tune that for expert users. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 25, 2015, 08:28:18 AM My current machine is 6 core cpu (12 execution threads with HT), 32GB ddr4, SATA SSD RAID0 and fiber internet (~250Mb/s). And I'm thinking I should go with 64GB RAM soon. I had thought of buying a dual CPU board and go with a couple Haswell v3 Xeons but I backed off. Can't blow all my bitcoins on a PC =P. For info, it takes 280~300 sec to scan the DB with ~5k addresses, about 80sec to build, and another ~100 sec for the header reading and what not (that could be reduced to 15sec with some effort). Perhaps your FAQ should list minimum and recommended hardware specs. I am running my full node on a $35 ARM single board computer with 1 GB ram. The letstalkbitcoin guys say they used to use armory, and created a new tipping address for every episode, but stopped using it because it became too slow. You can't compare the resource cost of initial synchronization with that of maintenance. I expect it would take your board over 2 weeks to sync the blockchain from scratch, yet it can process new blocks at an acceptable pace. And the bitcoin core developers will likely improve that balance when they eventually get around to using libsecp256k1 for transaction verification as well as signing. That's probably the most significant bottleneck on these ARM systems, lack of clockspeed to handle compute-expensive crypto functions. Cutting the demand for CPU cycles by 6 or 8 as is claimed could cut the intial sync time by a similar factor. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 25, 2015, 04:43:08 PM Cutting the demand for CPU cycles by 6 or 8 as is claimed could cut the intial sync time by a similar factor. I expect it will be that much of an improvement, OpenSSL is painfully slow at ECDSA. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 25, 2015, 05:29:42 PM 6e839bc fixed something for me unexpectedly, now getting 288 headers and 288 block files processed whereas before was getting either 285 or 286. Another difference is that the OS handles the interrupt overload when tx scanning alot more gracefully, background tasks in other domains are now given near full opportunity to take as contiguous processing cycles as they need, and the scanning appears to be hammering the CPU just as heavily. i.e. OS is usable simultaneously with scanning :D
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 25, 2015, 05:50:45 PM I changed an aggressive lock around key addressing that was meant to cover certain stuff for supernode, and is unnecessary for fullnode. Now it's limited to supernode. I guess that untangled some other part of the process.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: doug_armory on June 25, 2015, 08:24:11 PM 0.94 is the first version to reduce hardware requirements (120MB disk space from 30GB+, <2GB RAM from 8+) while increasing performance AND increasing scalability. Seriously everyone, 0.94 is shaping up nicely. I've been flying by the seat of my pants and using 0.94 regularly since the first major chances were committed. After some initial crashes (totally understandable given the nature of software development), it worked really well for me. I don't doubt that there are bugs here and there - Carlton Banks inadvertently showed us quite a few serious bugs - and I encourage everybody to bang on the test builds when they come out. I'm just saying that, for my purposes, 0.94 is quite nice and does go a bit easier on the system. :) Cutting the demand for CPU cycles by 6 or 8 as is claimed could cut the intial sync time by a similar factor. I expect it will be that much of an improvement, OpenSSL is painfully slow at ECDSA. I don't think it's that OpenSSL is painfully slow at ECDSA - I've found it to be okay, and probably a bit better than Crypto++ - as much as libsecp256k1 is specifically designed to optimize the hell out of working with the secp256k1 ECDSA curve (i.e., the curve Bitcoin uses). I believe Wladimir & Pieter have both reported 6-8x speed increases in verifications. They're just concerned about the possibility of a specially crafted Bitcoin transaction being validated in libsecp256k1 and not in OpenSSL or vice versa (i.e., a fork would occur). So, for now, libsecp256k1 is used only for signing, where it's apparently easier to test all the various use cases. Anyway, yes, once the switch is thrown and libsecp256k1 is used for verification, BC Core should speed up quite a bit. The main contributors are doing some pretty cool things if you pay attention to the commits and the related conversations. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 25, 2015, 10:50:34 PM I don't think it's that OpenSSL is painfully slow at ECDSA - I've found it to be okay, and probably a bit better than Crypto++ - as much as libsecp256k1 is specifically designed to optimize the hell out of working with the secp256k1 ECDSA curve (i.e., the curve Bitcoin uses). I believe Wladimir & Pieter have both reported 6-8x speed increases in verifications. They're just concerned about the possibility of a specially crafted Bitcoin transaction being validated in libsecp256k1 and not in OpenSSL or vice versa (i.e., a fork would occur). So, for now, libsecp256k1 is used only for signing, where it's apparently easier to test all the various use cases. Anyway, yes, once the switch is thrown and libsecp256k1 is used for verification, BC Core should speed up quite a bit. The main contributors are doing some pretty cool things if you pay attention to the commits and the related conversations. I should rephrase. What I meant is OpenSSL is painfully slow at secp256k1 because it has none of the specific optimizations for that curve. It is coded to sign any curve, so that's understandable. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 26, 2015, 10:46:42 AM What was added for 875ac898db seems to have killed off the GTK+ CRITICAL assertion fail warning, for me at least. It appeared before only in runs after the initial build and scan was shutdown and restarted, but now it doesn't appear even then. Not that it was causing any problems at that point, just thought it might help improve your understand of what's causing it.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 26, 2015, 11:03:32 AM That's really weird because the change affects code that takes place after scans, and the Gtk error message shows up early on startup (most likely when a build or scan operation is underway).
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 26, 2015, 12:54:39 PM That's really weird because the change affects code that takes place after scans, and the Gtk error message shows up early on startup (most likely when a build or scan operation is underway). That pattern didn't apply to my later experience of the error (see above), but then again, it didn't cause any crashes or otherwise aberrant behaviour. And now it's not appearing at all. Make of it what you will (if the long term goal is still to ditch Qt 4, then it's possibly not worth pursuing). Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 26, 2015, 01:40:07 PM Forget it completely lol, caught it again. I'm tempted to say that random instances of this warning can occur say 4-5 times consecutively for no apparent reason, likewise the inverse (no warning several times in a row).
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 29, 2015, 09:26:26 AM Supernode
Just giving it a test. It didn't tolerate being interrupted and restarted; I gave it 2 hours to complete yesterday, and it only got ~40% finished. Starting again today produced a std:error exception relating to reading a block file. Otherwise seemingly ok, will mess around with the fully built Db later today. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on June 29, 2015, 03:11:39 PM Same issue again with supernode, some console output from a restart after a successful quit:
Code: Do wallet checking : True Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on June 29, 2015, 09:29:43 PM Looks like some false positive being blown out of proportion. Sadly, I have some short term assignment till Tuesday so I have to put 0.94 on hold until then.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on July 03, 2015, 04:05:16 PM Looks like some false positive being blown out of proportion. Sadly, I have some short term assignment till Tuesday so I have to put 0.94 on hold until then. Following more testing, I think I may have been misinterpreting. Armory appears to deadlock upon std:exception file not found, but this actually seems to be attributable to some interaction between SDM mode and the supernode DB build code; running bitcoind manually will generate the exception, but not the apparent deadlock. I suspect if I'd walked away for 10 minutes then it may have continued correctly, and it's already been said that SDM needs an overhaul so I didn't try testing it with supernode again (I figured testing multi-threaded scans with SDM would be worthwhile seeing as it's the default Armory setting that new users will encounter, I can confirm that it works as well with multi-threaded scans as it did with the previous scanning logic) Supernode DB builds fine (given the many hours that takes), that code seems robust to hours of testing, including several deliberate quit/restarts. Segfaults ensue during tx scanning. Also tested 5c0497f03b today (rebuild+scan), no problems so far. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on July 03, 2015, 09:42:19 PM I have not had time to deal with your bug reports on supernode yet, working a few issues with parallel address scanning and zero conf balance. I expect I'll have time to go over your issues before the testing builds are released (that's the last item on my bug list btw).
Also, it is expected supernode users will manage their own Core instance, and I've personally never tested it with auto managed bitcoind. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: m.fridge on July 06, 2015, 07:52:35 PM I get a segmentation fault during the "Organizing Blockchain" step.
The last messages: Code: [New Thread 0xac9f1b40 (LWP 12492)] And the backtrace: Code: #0 0xb7d87062 in ?? () from /lib/libc.so.6 Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on July 06, 2015, 09:19:09 PM is it happening every time?
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: m.fridge on July 06, 2015, 09:27:01 PM Yes, I tried it 3 times. Although only the last time in gdb, where I got the backtrace.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Roy Badami on July 06, 2015, 09:30:24 PM Just to say that 5c0497f fixes the db build/scan issue for me. Correct balances and no errors in the logs.
I did get one crash though (not during db build/scan): Code: Crashed Thread: 5 Code: Thread 5 Crashed: Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on July 07, 2015, 09:26:49 AM Thanks for debug logs guys, will get to it soon.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: joel_ on July 07, 2015, 07:33:54 PM Just a quick question for devs - will Armory 0.94 finally support import of compressed private keys ?
Thank you very much for reply. And good luck with testing, hope I and other - less tech savvy people - might put our hands on 0.94.x binary soon! Or is it already out there ? Thanks. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: doug_armory on July 07, 2015, 09:15:47 PM Just a quick question for devs - will Armory 0.94 finally support import of compressed private keys ? Thank you very much for reply. I don't think so. I believe that will come when the v2.0 wallets are ready. (We're still working on them! I'm guessing they'll be in the post-0.94 release. Sorry for the wait.) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: joel_ on July 08, 2015, 07:03:22 PM I don't think so. I believe that will come when the v2.0 wallets are ready. (We're still working on them! I'm guessing they'll be in the post-0.94 release. Sorry for the wait.) Thank you very much for reply. Never mind, it is certainly not one of most important features. Just that yesterday I was trying to import Multibit PK's to Armory wallet, and failed miserably. And have found that I am not alone, when searching this forum for solution :)Anyway, I have started to use Armory recently and ditched all those SPV/ dumbed down web wallets and so far I am loving Armory advanced features! The only thing I hate, but seems fixed with 0.94 is that it eats double the blockchain space on hdd :) So let me wish to all devs of Armory good luck with making Armory the best bitcoin wallet. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: fallinglantern on July 13, 2015, 11:34:03 PM current revision of ffreeze (hash b627160) won't build on debian 7.8 32-bit:
Quote g++ -Icryptopp -Imdb -DUSE_CRYPTOPP -D__STDC_LIMIT_MACROS -I/usr/include/python2.7 -I/usr/include/python2.7 -std=c++11 -O2 -pipe -fPIC -c ScrAddrObj.cpp ScrAddrObj.cpp: In member function 'void ScrAddrObj::purgeZC(const std::set<BinaryData>&)': ScrAddrObj.cpp:160:45: error: invalid initialization of non-const reference of type 'TxRef&' from an rvalue of type 'TxRef' ScrAddrObj.cpp:175:46: error: invalid initialization of non-const reference of type 'TxRef&' from an rvalue of type 'TxRef' make[1]: *** [ScrAddrObj.o] Error 1 I've reverted to 90586da for the time being which does still compile. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: doug_armory on July 14, 2015, 04:25:06 AM current revision of ffreeze (hash b627160) won't build on debian 7.8 32-bit: Quote g++ -Icryptopp -Imdb -DUSE_CRYPTOPP -D__STDC_LIMIT_MACROS -I/usr/include/python2.7 -I/usr/include/python2.7 -std=c++11 -O2 -pipe -fPIC -c ScrAddrObj.cpp ScrAddrObj.cpp: In member function 'void ScrAddrObj::purgeZC(const std::set<BinaryData>&)': ScrAddrObj.cpp:160:45: error: invalid initialization of non-const reference of type 'TxRef&' from an rvalue of type 'TxRef' ScrAddrObj.cpp:175:46: error: invalid initialization of non-const reference of type 'TxRef&' from an rvalue of type 'TxRef' make[1]: *** [ScrAddrObj.o] Error 1 I've reverted to 90586da for the time being which does still compile. Confirmed. I have a fix but will check with goatpig before it gets checked in. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: doug_armory on July 14, 2015, 04:32:20 PM current revision of ffreeze (hash b627160) won't build on debian 7.8 32-bit: Quote g++ -Icryptopp -Imdb -DUSE_CRYPTOPP -D__STDC_LIMIT_MACROS -I/usr/include/python2.7 -I/usr/include/python2.7 -std=c++11 -O2 -pipe -fPIC -c ScrAddrObj.cpp ScrAddrObj.cpp: In member function 'void ScrAddrObj::purgeZC(const std::set<BinaryData>&)': ScrAddrObj.cpp:160:45: error: invalid initialization of non-const reference of type 'TxRef&' from an rvalue of type 'TxRef' ScrAddrObj.cpp:175:46: error: invalid initialization of non-const reference of type 'TxRef&' from an rvalue of type 'TxRef' make[1]: *** [ScrAddrObj.o] Error 1 I've reverted to 90586da for the time being which does still compile. Confirmed. I have a fix but will check with goatpig before it gets checked in. Fix has been checked in. Give it a go. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: fallinglantern on July 18, 2015, 05:21:37 PM Fix has been checked in. Give it a go. Works like a champ. Thank you very much. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Searinox on July 19, 2015, 08:15:20 PM Much good news around here. When do we start preliminarily testing the build cross-platform too? I would very much love to dive right in with a 0.9.4 Win x64 build despite whatever remaining uncertainties, and I doubt I'd be the only one. :)
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on July 19, 2015, 08:29:33 PM https://bitcointalk.org/index.php?topic=1112974.msg11911942#msg11911942
The 0.94 release makes use of multi-threading to improve syncing performance, but multi-threading is a notoriously difficult discipline from which to iron out the kinks. Your enthusiasm (as well as that displayed by others in the thread) will be very useful when ATI do put testing builds out; getting this sort of code as solid as can be is important to qualify for release quality. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 03, 2015, 06:21:00 PM New commit, calling out the testers =P
Most of the changes are stability and supernode improvements. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on August 03, 2015, 08:43:05 PM In standard (headless?) mode, I'm getting block Db rebuilds every quit/restart. No apparent errors in logs, either in the build or the restarts. Db rebuilt/rescanned using latest commit. Using 0.11 for Core.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 03, 2015, 09:54:39 PM In standard (headless?) mode, I'm getting block Db rebuilds every quit/restart Will look into it. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 04, 2015, 02:43:33 PM In standard (headless?) mode, I'm getting block Db rebuilds every quit/restart. No apparent errors in logs, either in the build or the restarts. Db rebuilt/rescanned using latest commit. Using 0.11 for Core. Fixed. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on August 04, 2015, 04:39:30 PM In standard (headless?) mode, I'm getting block Db rebuilds every quit/restart. No apparent errors in logs, either in the build or the restarts. Db rebuilt/rescanned using latest commit. Using 0.11 for Core. Fixed. Will try it out shortly. Here's something from a crash I got in supernode mode: Code: -WARN - 1438705827: (BlockWriteBatcher.cpp:505) Finished applying blocks up to 40000 Could try it again using gdb if that helps. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 05, 2015, 12:54:04 PM Fixed. Just a dumb omission on my part.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on August 05, 2015, 02:47:49 PM In standard (headless?) mode, I'm getting block Db rebuilds every quit/restart. No apparent errors in logs, either in the build or the restarts. Db rebuilt/rescanned using latest commit. Using 0.11 for Core. Fixed. Confirmed. With supernode, I'm now getting no thread toggling (1.5 days to scan history). Logging for the thread toggling got removed from headless too, and yet that mode clearly multithreads the scanning workload. I like the "resume initialising from blockfile xxx" behaviour, serious productivity boost when testing supernode. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 05, 2015, 04:20:29 PM With supernode, I'm now getting no thread toggling (1.5 days to scan history). Logging for the thread toggling got removed from headless too, and yet that mode clearly multithreads the scanning workload. That was way too much verbose anyways. After profiling a bit of profiling it seems thread toggling is pointless. Better off setting all processes to max thread count (as returned by std::thread::hardware_concurrency()). There already is a RAM ceiling coded in, so the different parts of the scan (reading data, parsing, serializing, writing) cannot get ahead of one another. In this case it's simpler to max out thread count for each and everyone of them and let the OS sort things out. Each part waits on the next one through mutexes and condition variables, so all these threads are sleeping until they're allowed to work again. No harm done and it squeezes as much CPU time as possible. On the other hand toggling is a pain to tune properly. With the current toggler, a mainnet fullnode scan takes me ~8m30. With all thread counts maxed out it takes short of 5m. Quote I like the "resume initialising from blockfile xxx" behaviour, serious productivity boost when testing supernode. A lot changed there. The DB is now write ahead only. The previous version would modify earlier entries to mark spent TxOuts. Now it always writes ahead and keeps spentness in a dedicated DB. It speeds the process a lot (reduces rewrites) and guarantees that the DB can overwrite data by starting at the top of the last properly committed batch with no risk of corrupting the dataset. The one thing that did get corrupted a lot was the balance and transaction count for each address. There's a whole new section of code to handle that now, independently of scanning history. You need context to compute balance, since you are tallying the effect of each TxOut and Txin for each address. The previous version of supernode tallied balance while scanning history. If the DB failed to resume in the exact same state as before it crashed, there was a decent chance a least one balance got corrupted, and that meant rescanning from scratch. This version separates the 2 processes entirely. It first scans history, then computes balances. This simplifies and speeds up a lot of code. First of all, keeping track of balance at all times creates a lot a rewrites: every time an address appears in a batch, you need to pull the existing balance from the DB, update it and write it back. Before splitting the 2 processes, 0.94 scanned supernode in 4h30. After splitting it, I scanned the history in 1h30 and built balance in 5min. The good part, besides the speed boost, is robustness. Since the 2 are now separate, I added an option in supernode to run only the balance tallying part for quick fixing a damaged DB. It's called "Rescan SSH". Should fix the DB in 5~20min depending on the machine. PS: There still is room for some very significant optimization, but I've concluded they are out of the scope of this release. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on August 05, 2015, 05:38:03 PM After splitting it, I scanned the history in 1h30 and built balance in 5min. The good part, besides the speed boost, is robustness. Since the 2 are now separate, I added an option in supernode to run only the balance tallying part for quick fixing a damaged DB. It's called "Rescan SSH". Should fix the DB in 5~20min depending on the machine. PS: There still is room for some very significant optimization, but I've concluded they are out of the scope of this release. What could be accounting for low CPU usage (20%) and a 2 days estimate scan time? The number of addresses Armory scans? One thing I perhaps should have mentioned is that I did not delete the database folder from previous experiences with supernode (which ended midway through a troubled tx scan once you hinted supernode changes a few weeks back) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 05, 2015, 06:25:17 PM Forgot to mention the DB format has changed quite a lot, you are better off getting rid of that older DB and starting fresh.
What could be accounting for low CPU usage (20%) and a 2 days estimate scan time? Eventually it all comes down to your drive's bandwidth. A lot of in depth optimization (that I skipped this time) would be to modify the DB engine to line up all new written data sequentially. Another big optimization I skipped would be to fragment to spentness DB. The history DB is easy to fragment into smaller subsets so you can throw a thread at each of them while keeping the subset per thread fairly small. It speeds up searches and reduces the effort to realign data within each subset (as opposed to one massive DB). The spentness DB however is one single block and is thus written to by a single thread. LMDB only allows a single writer per DB and a single transaction per thread (which is a common sense approach to a transactional DB design: 1 writer, unlimited readers). There's quite an effort to provide to split down that data in a way multiple threads can write several subsets concurrently. History is keyed by addresses so it's pretty simple to break it down into group of addresses. Spentness is keyed by block height & transaction index & txout index, so breaking down the subset is more complicated. You need to fragment it in a way where you can still resolve arbitrary searches without more context that a transaction hash and txout id. I got a good idea to implement that, and this change (which I would apply to the history DB as well) would allow for crazy fast scanning on SSD and moderately fast scanning on HDD. However it is massive, requires some dynamic parameters that will add a lot of interlocking in certain corner cases, and I had to wrap this version up at some point. Keep in mind that the target use case for supernode is still a medium to large server meant to run as the backend to a web service like bc.info or as a bootstrap server for litenodes. We don't have the time nor the resources to get supernode working on HDD. I'm not sure this will ever be a target hardware for supernode either, although I may err on that path for pure hobbyist satisfaction. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: jje on August 05, 2015, 06:35:06 PM got 0.94 working on fedora 22. very enthused about the smaller database. hats off to the devs!
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on August 07, 2015, 01:34:27 PM Just got to the point with re-building supernode Db where the scanning begins. I think the threading sync is a little wrong for my platform somehow, range of CPU usage is 5-15%, and I state it that way because it's very bursty. Still getting long estimates for a complete scan (1.5 days currently).
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 07, 2015, 05:42:19 PM What's your disk bandwidth at?
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on August 07, 2015, 05:55:12 PM Nothing installed that measures disk bandwidth, unless there's something in standard Gnome Debian that does. Using an SSD for the databases, not more than 2/3rds full. The drive responds as it ever has to other workloads.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: qertoip2 on August 08, 2015, 08:03:00 PM PLEASE checkout the ffreeze branch and help test! Once we get a little bit more testing we'll do a semi-official testing release with a proper bug bounty! After successful sync and several successful restarts, I run into the following error. This is happening with the Bitcoin Core 0.11 fully synced and running fine in the background. Code: ➜ git log Code: ➜ uname -a Code: ➜ ~ python /ArmoryQt.py If you need anything else please let me know. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 08, 2015, 08:30:57 PM Does it fix itself after a restart?
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: qertoip2 on August 09, 2015, 11:22:18 AM Does it fix itself after a restart? This happens every time I run Armory. I could probably reset this to working by removing ~/.armory/databases and starting over.In the GUI a popup shows "bad block meta value". Once OK-ed the Armory shuts down. This is 100% reproducible. Perhaps my "databases" would be useful for you? I can put it online if you need. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 09, 2015, 11:40:26 AM This happens every time I run Armory. I could probably reset this to working by removing ~/.armory/databases and starting over. Yes, you will have to rebuild & rescan or delete the DB folder Quote Perhaps my "databases" would be useful for you? I can put it online if you need. I would like that very much. If I can experience the issue first hand I can improve the DB repair code to catch that one too. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: qertoip2 on August 09, 2015, 01:10:08 PM I would like that very much. If I can experience the issue first hand I can improve the DB repair code to catch that one too. Armory 0.94 databases.zip with the issue:https://mega.co.nz/#!ws0w3ZiK!T0wpFyeHV8yJ9f-BASgsHqsJ3g48IZp4pwUJfbN-AQo Please find the ZIP password PM-ed. I've encrypted the ZIP just in case it has some privacy implications. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 09, 2015, 01:26:07 PM Thanks, that will help a lot.
Please find the ZIP password PM-ed. I've encrypted the ZIP just in case it has some privacy implications. It does, encrypting it is proper practice. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 09, 2015, 08:53:08 PM Try the latest commit with your damaged DB.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: qertoip2 on August 10, 2015, 08:10:43 AM Try the latest commit with your damaged DB. It fixed the issue, thank you so much! Although there was a minor glitch during the recovery. On the first run I got another error about missing headers, a suggestion to do a factory reset, and a forced close. I choose to not do a factory reset. On the second run it fully recovered (very quickly). The logs: Code: git pull origin ffreeze Code: ➜ BitcoinArmory git:(ffreeze) python ArmoryQt.py Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 10, 2015, 10:11:12 AM On the first run I got another error about missing headers, a suggestion to do a factory reset, and a forced close. I choose to not do a factory reset This error marker is very aggressive, you shouldn't do a factory reset unless you run into it a few times in a row. I should do something about it really, at least not let it trigger on missing headers only. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: m.fridge on August 14, 2015, 06:55:17 PM Hi,
my previously reported bug (post #156 in this thread) has been fixed. Thanks for that! However, I've got new issues here: When I first import a wallet before the database is built, then armory seems to stuck in an endless loop when beginning to scan transactions. The program consumes CPU and has high I/O, but it keeps staying at 4% scanning progress (even when running for one night). When I first build the database and import the wallet afterwards, armory crashes when it comes to transaction scanning. Here is backtrace of such a crash: Code: [Thread 0xabcf0b40 (LWP 15657) exited] Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 14, 2015, 11:17:33 PM I fixed this issue in the last commit a couple days ago. Pull again and give it a spin.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Searinox on August 16, 2015, 08:39:39 AM On the first run I got another error about missing headers, a suggestion to do a factory reset, and a forced close. I choose to not do a factory reset This error marker is very aggressive, you shouldn't do a factory reset unless you run into it a few times in a row. I should do something about it really, at least not let it trigger on missing headers only. Although I am running on 0.93.2 I can report as much. My desktop has been running Armory for more than a year now and I've had no issues or crashes at all. Zero. My laptop however, is a different issue. Whenever I wake it up from days of inactivity, Armory quickly throws a "missing headers" error and quits, only to return with full functionality on next launch. At first I didn't know if it was an issue with sleep or just it not being able to keep up with bitcoind but today for the first time it happened on the desktop. We had a power failure yesterday and after restoration there was no internet until just about now. During all this time Armory had been running uninterrupted. When the internet came back, bitcoind began catching up with a day's worth of blocks, and Armory threw the "missing headers" error. Could you please investigate if there are issues with Armory keeping up with bitcoind when it's behind by say 2-3 days and suddenly it gets a chance to catch up? I am re-reporting this in the ongoing bigger Armory thread to keep a 0.93 error off the 0.94 post but since you are investigating this now, I figured I'd mention it here as well. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 16, 2015, 09:15:54 AM Although I am running on 0.93.2 I can report as much. My desktop has been running Armory for more than a year now and I've had no issues or crashes at all. Zero. My laptop however, is a different issue. Whenever I wake it up from days of inactivity, Armory quickly throws a "missing headers" error and quits, only to return with full functionality on next launch. At first I didn't know if it was an issue with sleep or just it not being able to keep up with bitcoind but today for the first time it happened on the desktop. We had a power failure yesterday and after restoration there was no internet until just about now. During all this time Armory had been running uninterrupted. When the internet came back, bitcoind began catching up with a day's worth of blocks, and Armory threw the "missing headers" error. Could you please investigate if there are issues with Armory keeping up with bitcoind when it's behind by say 2-3 days and suddenly it gets a chance to catch up? I am re-reporting this in the ongoing bigger Armory thread to keep a 0.93 error off the 0.94 post but since you are investigating this now, I figured I'd mention it here as well. That's a 0.93 specific issue that has been fixed in 0.94 0.93 can't tell the difference between a mangled block and a block in the process of being written to disk (which appears mangled when Armory reads it). 0.94 side steps the issue by rereading partial blocks at each iteration until the partials either fill up or a valid block further in the file extends the chain. The true solution would be to listen on the new block P2P notification and grab data over the socket, but that's a whole new effort. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Searinox on August 16, 2015, 09:25:20 AM Yet another reason to anticipate 0.94. If I had the knowledge, I'd compile the source for Windows myself.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: m.fridge on August 16, 2015, 01:17:01 PM I fixed this issue in the last commit a couple days ago. Pull again and give it a spin. The problem still exists with the latest version. Here is the log, where the crash seems to have another cause than before. I also got a popup saying "Cannot find block with hash". Code: -ERROR - 1439730764: (BlockWriteBatcher.cpp:711) error grabbing block 0|0 in file #0, offset: 8, with a size of 285 bytes Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 16, 2015, 09:35:46 PM This looks severe. Do a Help -> Rescan and see if it you get the same output.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: m.fridge on August 17, 2015, 08:46:08 PM I did a rebuild and rescan and now end up with "std::bad_alloc". I thought that my harddrive might have problems and used a usb drive for the armory datadir, where I got the same error. Any ideas what's wrong here?
Code: [New Thread 0xabe84b40 (LWP 24368)] "std::bad_alloc" is shown as a popup. After okay-ing it, armory exits normally according to gdb. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 17, 2015, 09:26:04 PM bad_alloc means the process failed to allocate new memory. Most likely the code is trying to allocate a ridiculous amount of RAM (possibly some badly derserialized varInt).
Download the testnet chain and try to build that. Also, could you give me your first block file? (blk00000.dat) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: trev on August 18, 2015, 01:22:49 AM I built the ffreeze version of Armory under Linux and am running it with bitcoind 0.11.0.
So far everything is working fine. A minor issue happened when I restored a wallet from 0.92 and then tried "Addresses->View Address Book". Armory threw the following exception: File "ArmoryQt.py", line 3902, in execAddressBook DlgAddressBook(self, self, None, None, None).exec_() File "/home/userid/BitcoinArmory-ffreeze/qtdialogs.py", line 8133, in __init__ rowHeight = tightSizeStr(self.font, 'XygjpHI')[1] File "/home/userid/BitcoinArmory-ffreeze/qtdefines.py", line 215, in tightSizeStr fm = QFontMetricsF(QFont(obj)) TypeError: QFont(): argument 1 has unexpected type 'builtin_function_or_method' However, I keep on getting popups about Armory losing it's connection to bitcoind. (ERROR) Networking.py:359 - ***Connection to Satoshi client LOST! Attempting to reconnect... In ~/.bitcoin/debug.log there are bezillions of lines like this... .... 2015-08-18 00:54:43 keypool reserve 1 2015-08-18 00:54:43 keypool return 1 .... It looks like Armory is polling bitcoind on 127.0.0.1:8332 once a second to get the chain state. Each poll is a tcp connection; open and close. The problem is that I get a bucket load of tcp TIME_WAIT entries. Eventually the system runs out of socket resources, and then I think Armory thows the cannot connect popup. Is there an option to increase the polling interval to, let's say, 10 second? Or better still, could Armory open a single tcp connection to bitcoind and keep it open for the duration of the session? I like having Armory start bitcoind as a child process, and would prefer to not have to start it myself, which is the work-around I have seen given in previous posts. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on August 18, 2015, 01:46:05 AM I like having Armory start bitcoind as a child process, and would prefer to not have to start it myself, which is the work-around I have seen given in previous posts. The code Armory uses to launch bitcoind as a child process is already flagged by the devs as starting to look a little stale and old. I'm not sure that they're planning to overhaul it for this particular release ("ffreeze" means "feature freeze" I think), but it will happen eventually. I also get unstable and/or buggy behaviour when Armory manages bitcoind, although not quite the same issues you're getting. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 18, 2015, 09:22:37 AM Quote Is there an option to increase the polling interval to, let's say, 10 second? There is no command line argument for this purpose afaik. The current stance on this piece of code is that it works enough that we completely ignored it for the past few versions update. As such, I am completely unfamiliar with it. There is also no plan to fix it or overhaul it, as blocks over P2P would require a C++ implementation of the Bitcoin P2P protocol, in which case all communications with Core would be handled straight from the backend and the Python code dropped entirely, which is another reason I've made a point of ignoring it for so long. At any rate, if you feel adventurous, you can modify the Python hardcoded value. I think this line should do it: https://github.com/etotheipi/BitcoinArmory/blob/ffreeze/ArmoryQt.py#L6391 Change nextBeatSec=1 to what you see fit. You can use decimal values to signify smaller time increments (i.e. 1.2 translates to 1200ms) Quote Or better still, could Armory open a single tcp connection to bitcoind and keep it open for the duration of the session? I don't think that's the default reactor behavior. Dropping the socket after every iteration is indeed expensive compared to a keep-alive model, but still marginal in the scope of a localhost connection. The issue isn't so much the amount of sockets opened than the less than stellar Python GC and the implication that no one in Python bothers with explicit clean ups. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: trev on August 20, 2015, 02:44:18 PM However, I keep on getting popups about Armory losing it's connection to bitcoind. (ERROR) Networking.py:359 - ***Connection to Satoshi client LOST! Attempting to reconnect... I found that "maxconnections=6" in bitcoin.conf was causing bitcoind to reject connections from Armory on the localhost interface. When I changed it to "maxconnections=30", Armory started working as expected, and bitcond says it has 9 open connections. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 20, 2015, 05:13:57 PM However, I keep on getting popups about Armory losing it's connection to bitcoind. (ERROR) Networking.py:359 - ***Connection to Satoshi client LOST! Attempting to reconnect... I found that "maxconnections=6" in bitcoin.conf was causing bitcoind to reject connections from Armory on the localhost interface. When I changed it to "maxconnections=30", Armory started working as expected, and bitcond says it has 9 open connections. Oh right, there's that too. I personally do it the other way and add "addnode=127.0.0.1" to guarantee Core tries to keep the localhost connection with Armory alive. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: trev on August 24, 2015, 04:31:08 AM When running ffreeze and bitcoind 0.11.0 in Linux I got a "BDM thread failed: std::bad_alloc" crash.
Python traps the exception so gdb has nothing to report. I was doing a send from bitcoin-qt to Armory-watch-wallet. The exception happened when Armory received the broadcast transaction. I restarted Armory and it then saw the transaction when the first confirmation block arrived. Are there any debug printf's for identifying where the bad_alloc is occuring? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 24, 2015, 02:14:09 PM When running ffreeze and bitcoind 0.11.0 in Linux I got a "BDM thread failed: std::bad_alloc" crash. Python traps the exception so gdb has nothing to report. I was doing a send from bitcoin-qt to Armory-watch-wallet. The exception happened when Armory received the broadcast transaction. I restarted Armory and it then saw the transaction when the first confirmation block arrived. Are there any debug printf's for identifying where the bad_alloc is occuring? Are you using a x86 OS? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: trev on August 24, 2015, 03:58:12 PM Are you using a x86 OS? I'm using a home-built Linux running on a HP laptop. Linux sys 2.6.37.6 #11 SMP Sun Apr 14 21:52:49 EDT 2013 x86_64 Intel(R) Core(TM)2 CPU T5300 @ 1.73GHz GenuineIntel GNU/Linux I used gcc-4.9.2 to build the c++11 bits of Armory. Packages: swig-3.0.6 sip-4.16.9 PyQt-x11-gpl-4.11.4 psutil-2.1.3 Twisted-15.3.0 qt-4.8.6. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 24, 2015, 04:13:28 PM I'm pretty sure Core 2 can handle both x86 and x64 OSes, so this isn't really helping me. Browse to Armory's binary folder, and run this command and paste the result back:
Code: file _CppBlockUtils.so Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on August 24, 2015, 04:20:11 PM it does actually say x86_64 on the readout, apologies if that's what you meant.
Could it be something to do with the instruction set on the Core 2 era? I seem to remember that Core 2 was better specified than Core 2 Duo somehow, possibly not Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 24, 2015, 04:32:58 PM bad_alloc usually means the process failed to allocate more RAM. I'd like to make sure this is not happening on a x86 machine first before I investigate.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: trev on August 24, 2015, 04:40:20 PM I'm pretty sure Core 2 can handle both x86 and x64 OSes, so this isn't really helping me. Browse to Armory's binary folder, and run this command and paste the result back: Code: file _CppBlockUtils.so _CppBlockUtils.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, not stripped The machine only has 2GB dram, but it also has 12GB swap. I think you can also get this exception from a realloc with a bad ptr. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 24, 2015, 04:46:51 PM We don't use realloc directly, and very little malloc anyways.
It is most likely a std container failing to get enough memory. The issue isn't so much that the addressing failed but whether it is because it ran out of RAM or it was passed a bad ptr. If this is happening in a std container, which we rely heavily upon, then it's the former. Did the issue happen after scanning the whole chain or was it in a freshly started process? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: trev on August 24, 2015, 09:45:44 PM Did the issue happen after scanning the whole chain or was it in a freshly started process? I've been running Armory on the laptop for about a week. The initial scan happened without incident. It took me a few days to sort out the maxconnections issue. After that I've been running Armory continuously. The laptop is only used to run bitcoind and Armory. I got a first std::bad_alloc crash at 20150821_1449. I restarted Armory under gdb. I got the second std::bad_alloc crash at 20150823_2125. Python had exited cleanly so there was nothing for gdb to display. I restarted Armory under gdb, set a breakpoint at BDM_mainthread.cpp:453, and did run with --netlog. I am expecting to have to wait a couple of days for another crash. I'll keep on making small value transfers between Armory and bitcoin-qt. I want to see if the crash is spontaneous, or if it happens during some user initiated activity. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 24, 2015, 09:55:55 PM I got a first std::bad_alloc crash at 20150821_1449. I restarted Armory under gdb. I got the second std::bad_alloc crash at 20150823_2125. So you are saying it takes 24h+ to crash? Can you keep an eye on the process memory? I'll run some tests on my own. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: trev on August 26, 2015, 03:42:05 AM So you are saying it takes 24h+ to crash? Can you keep an eye on the process memory? I'll run some tests on my own. I got a third crash at 20150825_2016. It looks like Armory has to have been running for 24++ hrs in order to get to the condition that triggers the std::bad_alloc. The approximate 48 hr interval that I see is probably due to the fixed times of day that I use the PC. I sent 1.5BTC from bitcoin-qt to Armory-watch. Armory popped up the std::bad_alloc window a few seconds after bitcoin-qt said it had sent the TX. I think the problem occurs when Armory receives the raw TX. After restarting, it gets the TX from the block without a problem (but that's not after a 24++ hr delay). The memory usage remained fairly constant for the 48 hrs, including after the exception. Armory: VIRT = 1052M RES = 288M SHR = 19M Bitcoin-qt VIRT = 1121M RES = 404M SHR = 12M I'm not having any success debuging ArmoryQt.py with gdb. I'll just try to get a crash in less than 48 hrs. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 26, 2015, 06:40:30 AM So you are saying it takes 24h+ to crash? Can you keep an eye on the process memory? I'll run some tests on my own. I got a third crash at 20150825_2016. It looks like Armory has to have been running for 24++ hrs in order to get to the condition that triggers the std::bad_alloc. The approximate 48 hr interval that I see is probably due to the fixed times of day that I use the PC. I sent 1.5BTC from bitcoin-qt to Armory-watch. Armory popped up the std::bad_alloc window a few seconds after bitcoin-qt said it had sent the TX. I think the problem occurs when Armory receives the raw TX. After restarting, it gets the TX from the block without a problem (but that's not after a 24++ hr delay). The memory usage remained fairly constant for the 48 hrs, including after the exception. Armory: VIRT = 1052M RES = 288M SHR = 19M Bitcoin-qt VIRT = 1121M RES = 404M SHR = 12M I'm not having any success debuging ArmoryQt.py with gdb. I'll just try to get a crash in less than 48 hrs. What's the total size of your DB? The headers DB alone? https://github.com/etotheipi/BitcoinArmory/blob/ffreeze/cppForSwig/mdb/mdb.c#L588 Reduce this value to 16MB and rebuild a new DB with that, see if it suffers from the bad_alloc. Title: Blockchain not scanned up to the top? Post by: fenwick on August 26, 2015, 12:03:26 PM Blockchain not scanned up to the top!
So, I have compiled Armory from github, using the ffreeze branch. (My exact revision is 65dafec4ad888219abaa8897929364423f71ea05 ) I could successfully sync up my DB, and I also did two successful transactions. I know they were successful, because they show up (with plenty of confirmations) on blockchain.info. But then, after restarting Armory, the transactions don't show up. Here is the full story: - I did a successful transaction, which was immediately confirmed a few times. - However, I noticed that armory still showed 0 confirmations. - I did another transaction, and Armory said that this transaction might have been refused by the network. I want to blockchain.info to check, and I saw that the transaction waiting there with 0 confirmations. A few blocks have gone by, and it wasn't confirmed, so I suspected that maybe I should have set higher fees, and so I wanted to "undo" this transactions ... so I told Armory (in the help menu) to "Clear All Unconfirmed". - Meanwhile, I see that eventually, my transaction _were_ confirmed OK - However, after restarting armory, these transactions don't show up at all. Neither the first one, nor the second one. Likewise, the displayed balance is the one before the transactions. According to armory, it's as if those transactions have never happened. - Also, the label in the bottom right corner says "Connected (353000 blocks)", whereas according to the .bitcoin/debug.log, the current block number is 371598. Ha Armory forgot to scan the last 20000 blocks? That is going on here? Title: Ordering of transaction doesn't work Post by: fenwick on August 26, 2015, 01:07:21 PM On the list of transactions, clicking on the header fields of the table (saying "Date", Wallet", "Comments", etc.), I can see an arrow being moved, switched up/down, moved to the column I clicked, etc, but the displayed transactions are not re-ordered accordingly. In fact, they don't change at all. Title: Re: Ordering of transaction doesn't work Post by: Carlton Banks on August 26, 2015, 01:14:56 PM On the list of transactions, clicking on the header fields of the table (saying "Date", Wallet", "Comments", etc.), I can see an arrow being moved, switched up/down, moved to the column I clicked, etc, but the displayed transactions are not re-ordered accordingly. In fact, they don't change at all. currently feature-not-bug. Title: Re: Blockchain not scanned up to the top? Post by: fenwick on August 26, 2015, 01:44:18 PM Blockchain not scanned up to the top! So, I have compiled Armory from github, using the ffreeze branch. (My exact revision is 65dafec4ad888219abaa8897929364423f71ea05 ) I could successfully sync up my DB, and I also did two successful transactions. I know they were successful, because they show up (with plenty of confirmations) on blockchain.info. But then, after restarting Armory, the transactions don't show up. Here is the full story: - I did a successful transaction, which was immediately confirmed a few times. - However, I noticed that armory still showed 0 confirmations. - I did another transaction, and Armory said that this transaction might have been refused by the network. I want to blockchain.info to check, and I saw that the transaction waiting there with 0 confirmations. A few blocks have gone by, and it wasn't confirmed, so I suspected that maybe I should have set higher fees, and so I wanted to "undo" this transactions ... so I told Armory (in the help menu) to "Clear All Unconfirmed". - Meanwhile, I see that eventually, my transaction _were_ confirmed OK - However, after restarting armory, these transactions don't show up at all. Neither the first one, nor the second one. Likewise, the displayed balance is the one before the transactions. According to armory, it's as if those transactions have never happened. - Also, the label in the bottom right corner says "Connected (353000 blocks)", whereas according to the .bitcoin/debug.log, the current block number is 371598. Ha Armory forgot to scan the last 20000 blocks? That is going on here? After restarting Armory, and waiting for a long, long time again, the same situation returned. (Scanning finished at block 353055) This is the end of of the output: -INFO - 1440596237: (BlockUtils.cpp:395) reading blocks from file 322 -INFO - 1440596239: (BlockUtils.cpp:395) reading blocks from file 323 -INFO - 1440596241: (BlockUtils.cpp:395) reading blocks from file 324 -INFO - 1440596243: (BlockUtils.cpp:395) reading blocks from file 325 -INFO - 1440596244: (BlockUtils.cpp:1418) Wrote blocks to DB in 24.3227s -WARN - 1440596245: (BlockUtils.cpp:1114) Scanning from 353055 to 353055 -INFO - 1440596247: (BlockUtils.cpp:1458) checking scan integrity -INFO - 1440596247: (BlockUtils.cpp:1646) --- bwbDtor: 0s -INFO - 1440596247: (BlockUtils.cpp:1647) Scanned Block range in 0.300441s -INFO - 1440596247: (BlockUtils.cpp:1653) Finished loading at file 325, offset 73062124 -INFO - 1440596247: (BlockDataViewer.cpp:157) Enabling zero-conf tracking -DEBUG - 1440596422: (Blockchain.cpp:214) Organizing chain And indeed, it says the status is "Connected (353055 blocks)", ignoring the last ~20k blocks in the network. (While my Satoshi's log says: height=371612 ) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 26, 2015, 02:04:54 PM You are missing some block data starting 353055, which is why Armory can't scan any futher.
In your blocks folder, you should delete the last few blk files, probably up to blk00320.dat (315 to be on safe side) and the associated rev files (rev00320.dat and so on). Then start BitcoinQt, it will tell you its DB is corrupt and rebuild it from scratch, download the last few missing files in the process. Once that is done, start Armory, it should recover from that. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: fenwick on August 26, 2015, 02:11:53 PM You are missing some block data starting 353055, which is why Armory can't scan any futher. In your blocks folder, you should delete the last few blk files, probably up to blk00320.dat (315 to be on safe side) and the associated rev files (rev00320.dat and so on). Then start BitcoinQt, it will tell you its DB is corrupt and rebuild it from scratch, download the last few missing files in the process. Once that is done, start Armory, it should recover from that. OK, I will do that ... but I find it really weird that neither Satoshi, nor Armory detects the missing/corrupted block data. (Or at least none of them complains.) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on August 29, 2015, 05:47:07 PM Suggestion (possibly devs already thought this):
Create a package that isn't "armory" for the testing releases (armory-testing?). Rationale: the majority of users still use a monolithic OS; only one instance of a package can be installed on such a machine. There are alot of people out there who might benefit from using this upcoming test release instead of the main release, and the process would benefit from as wide a potential testing base as can be. A separate package would allow those who run release Armory to see whether 0.94, for instance, alleviates their issues or not, without having to contemplate uninstalling "armory-testing" and re-installing Armory main release. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: doug_armory on August 29, 2015, 07:20:42 PM Suggestion (possibly devs already thought this): Create a package that isn't "armory" for the testing releases (armory-testing?). Rationale: the majority of users still use a monolithic OS; only one instance of a package can be installed on such a machine. There are alot of people out there who might benefit from using this upcoming test release instead of the main release, and the process would benefit from as wide a potential testing base as can be. A separate package would allow those who run release Armory to see whether 0.94, for instance, alleviates their issues or not, without having to contemplate uninstalling "armory-testing" and re-installing Armory main release. Interesting idea. Admittedly, I'm not entirely sold on the idea of a separate package. There's too great a chance that people will be lazy and leave two copies on their system, potentially leading to weirdness down the road. For example, if somebody had 0.93-testing and tried to go back to 0.92, the Armory DB would keep getting nuked and lead to constant rebuilds. Unfortunately, I don't think "But only people who know what they're doing will install it!" is correct. It wouldn't be hard for us to miss posts on, say, Reddit, and end up with somebody suggesting people try a test version that fixes one problem but introduces another. (Just FYI, Reddit was just an example. Some of us do check Reddit on occasion. We typically prefer this forum, though.) That said, it's something to keep in mind moving forward. Pros and cons, as always. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on August 29, 2015, 07:52:52 PM Interesting idea. Admittedly, I'm not entirely sold on the idea of a separate package. There's too great a chance that people will be lazy and leave two copies on their system, potentially leading to weirdness down the road. For example, if somebody had 0.93-testing and tried to go back to 0.92, the Armory DB would keep getting nuked and lead to constant rebuilds. Unfortunately, I don't think "But only people who know what they're doing will install it!" is correct. It wouldn't be hard for us to miss posts on, say, Reddit, and end up with somebody suggesting people try a test version that fixes one problem but introduces another. Agreed. I'm looking at all these user complaints from 0.92-0.93 and trying to think of a way to wish them away without actual magic, but there is no good compromise (as the terminally pessimistic will tell you about everything :D). It'd be nice if more people were accustomed to using virtualisation, but I guess it's not that time yet. I'll probably be waiting impatiently for everyone to adopt some other new sophistication by then :-\ (Just FYI, Reddit was just an example. Some of us do check Reddit on occasion. We typically prefer this forum, though.) Really? Slightly amazed. I guess the Armory sub is much quieter than /r/bitcoin Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: trev on August 31, 2015, 05:36:02 AM What's the total size of your DB? The headers DB alone? https://github.com/etotheipi/BitcoinArmory/blob/ffreeze/cppForSwig/mdb/mdb.c#L588 Reduce this value to 16MB and rebuild a new DB with that, see if it suffers from the bad_alloc. I reduced MAX_MAPSIZE_INCEREMENT from 256MB to 16MB, re-compiled, and --rebuild. The databases are now: 20819968 2015-08-31 00:57 blocks 107724800 2015-08-31 00:57 headers After a couple of days of uptime I got the std::bad_alloc crash. I'm running an identical copy of bitcoin-qt/Armory on a newer desktop that has 16GB dram. I got the std::bad_alloc crash on that machine too, just like the laptop. I also got the std::alloc crash when sending to a "normal" wallet. In one instance so far, I started Armory at 23:50 and sent BTC at 00:10 and got std::alloc. Is there any code in Armory that uses "current time - previous time" and doesn't handle a negative value? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on August 31, 2015, 05:43:28 PM What's the total size of your DB? The headers DB alone? https://github.com/etotheipi/BitcoinArmory/blob/ffreeze/cppForSwig/mdb/mdb.c#L588 Reduce this value to 16MB and rebuild a new DB with that, see if it suffers from the bad_alloc. I reduced MAX_MAPSIZE_INCEREMENT from 256MB to 16MB, re-compiled, and --rebuild. The databases are now: 20819968 2015-08-31 00:57 blocks 107724800 2015-08-31 00:57 headers After a couple of days of uptime I got the std::bad_alloc crash. I'm running an identical copy of bitcoin-qt/Armory on a newer desktop that has 16GB dram. I got the std::bad_alloc crash on that machine too, just like the laptop. I also got the std::alloc crash when sending to a "normal" wallet. In one instance so far, I started Armory at 23:50 and sent BTC at 00:10 and got std::alloc. Is there any code in Armory that uses "current time - previous time" and doesn't handle a negative value? Comment out this line: https://github.com/etotheipi/BitcoinArmory/blob/ffreeze/ArmoryQt.py#L2619 It will prevent the ZC container from parsing any new tx. This is the most likely culprit atm. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: trev on September 11, 2015, 04:08:31 AM Comment out this line: https://github.com/etotheipi/BitcoinArmory/blob/ffreeze/ArmoryQt.py#L2619 It will prevent the ZC container from parsing any new tx. This is the most likely culprit atm. Since the crash occurs intermittently, I commented out addNewZeroConfTx() in one system and ran the second system unpatched. The patched system has not crashed so far, but the unpatched system has crashed several times. So, it looks like the problem is in the "zero confirmations" code. I will use the fix on both systems. Thanks for providing a good work-around patch. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: pop9119 on September 18, 2015, 01:53:24 PM Hey Armory folks,
In my occasional view of both bitcoin core and armory's code, I noticed that a recent commit to armory's ffreeze branch that put it in a gitian branch. I'm curious if this means that we are getting close to getting pre-built test builds of 0.94? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: qertoip2 on September 20, 2015, 11:28:29 AM The advertised branch ffreeze is missing: https://github.com/etotheipi/BitcoinArmory/tree/ffreeze
In the similar gitian-ffreeze branch make is broken (no 'Makefile', Makefile.old doesn't work) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: zombieslayer9099 on October 10, 2015, 01:24:53 AM I am on Debian based Linux.
Code: ./autogen.sh Code: python ArmoryQt.py Is there a fix for this? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: m.fridge on January 18, 2016, 03:20:49 PM They made Armory closed-source, which is why you don't find the latest tree on github anymore. There won't be any open releases. As far as I noticed, this step was done without any public communication.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on January 18, 2016, 10:17:16 PM Is there any chance that the 0.94 version will be released soon ? Late Jan/early Feb. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Carlton Banks on January 20, 2016, 12:26:07 AM Is there any chance that the 0.94 version will be released soon ? Late Jan/early Feb. Counting down... 8) Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on January 20, 2016, 12:53:47 AM Counting down... 8) oh boy, I'm in trouble now. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: T-rage_11 on January 23, 2016, 06:58:19 PM Is there any chance that the 0.94 version will be released soon ? Late Jan/early Feb. But we can't trust it anymore because it is closed source ... or am I wrong? Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on January 23, 2016, 08:44:21 PM Is there any chance that the 0.94 version will be released soon ? Late Jan/early Feb. But we can't trust it anymore because it is closed source ... or am I wrong? That's an ETA for a fully open source release. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Deth on February 01, 2016, 06:14:22 PM Is there any chance that the 0.94 version will be released soon ? Late Jan/early Feb. But we can't trust it anymore because it is closed source ... or am I wrong? That's an ETA for a fully open source release. Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on February 01, 2016, 06:17:41 PM There are some complications but I still intent to get something out in February. There will be an announcement shortly to clarify things up.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Deth on February 01, 2016, 06:20:59 PM Great. In early February I hope, as you stated before? ;)
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: goatpig on February 01, 2016, 06:25:11 PM No sorry. I can't reuse the existing 0.94 code, I'll have to put something together from the ground up.
Title: Re: Starting preliminary 0.94 testing - "Headless fullnode" Post by: Deth on February 01, 2016, 06:28:59 PM Ok :(
|