Just check the commit hash, you can't change git history without changing the commit hash.
True. Though goatpig didn't quote the hash of the changeset they reviewed - and although pf quoted the hash, they didn't say whether they'd reviewed the changes. Anyway, FWIW, I've briefly reviewed cad8d2d39b11cbbe1c728bcd7895620eedb90141 and all the changes seem fairly self-evidently benign. (But of course, you shouldn't take my word for it unless you know/trust me.) Just built it, and it seems to work. (Not used it for any transactions yet, but it successfully built and scanned the databases and displays the correct balances...) Thanks, bitsolutions, for your work on this - it's particularly valuable to me since I've just bought a new Mac laptop - mainly because two blockchains are now too large for my old one - and so would have found it incredibly frustrating if I could no longer use it for Armory anyway! I'd tip you but you I don't know where to send the coins :-) roy |
|
|
|
At the top it will say something like "merge <branch 1> from <branch 2>"
Thanks, knightdk, that's obvous now you explain it. I was just looking for an obvious clickable link, I guess. Although I also realise that if I just check out bitsoultions's branch, and also review the pull request on github at the time I do the checkout that tells me what changes I'm running - modulo the race condition that the branch (and pull request) could change as I'm checking it out. But it is good enough for me. |
|
|
|
|
Ok, so just reviewing bitsolution's changes, my one question for bitsolution (or anyone else who is Qt-savvy) is:
This change introduces a new dependency on qt-project.org. I presume this is a trustworthy source, since there are existing references to qt-project.org in upstream - but they all seem to be commented out AFAICS, so it seems this change does involve trusting a new domain.
Could someone explain to me the relationship between qt-project.org and the Qt project/qt.io, as my Google fu is failing me?
Thanks,
roy
EDIT TO ADD: I'm absolutely not suggesting there is anything untoward going on here - I'm sure there isn't. I'm just doing my due diligence and as Qt is not my area of expertese, I'm just trying to understand the provenance of the Qt code that bitsolutions is using. As there are (currently usused) references to this source in the official Armory code, I expect this source is trustworthy, but I just want to understand why it's there before I run this.
|
|
|
|
|
Dumb github question, but is there as easy way to apply a pull request to a local clone of the git repo? I can't even figure out how to download a pull request as a unified diff, let alone how to pull it properly with git.
The github help tells me to browse to the pull request and click "command line" but I don't see such a link.
EDIT: And many thanks to bitsolutions for doing (and sharing) the necessary work - and of course to Alan and goatpig and all at ATI for their continuing work on Armory!
EDIT^2: I'm also being dumb as there's no way (I think) to download the actual pull request that goatpig reviewed, since pull requests are mutable (for obvious reasons). The change is small enough it's easy enough to review, though, so if you posted a diff that would be just as good. I'm still curious as to the answer to my question, though.
EDIT^3: nm, I found a (good enough) answer that at least allows me to download the diff: browse to the pull request and then edit the URL to add ".patch" or ".diff" to the end of the URL Ugh! Did I tell you I hate github? EDIT^4: But .patch and .diff give different results though (I don't think they're substantively different but am failing to see why both exist). BTW, did I tell you I hate github?
|
|
|
|
So I'm just curious, I know for a fact that they share your personal info like SS# and other things you have to provide them to the government which they can choose to do whatever with.. Doesn't this scare anyone? I know it scares me, and I never do anything illegal with my Bitcoin, just the fact that they have ready access to this makes it feel something shady is going to happen...
The US government already knows your SSN - they're the ones who issued it! |
|
|
|
no it is not possible. unless the world has another major financial crisis. but it is fun to dream now and again.
Another financial crisis? We're no where near fixing the one that started in 2007, and likely it can't be fixed. The fact that the only effect of the ongoing financial crisis most people see right now is abnormally low interest rates (which most people probably think is a good thing) doesn't mean the crisis isn't still there. |
|
|
|
Just to say that 5c0497f fixes the db build/scan issue for me. Correct balances and no errors in the logs. I did get one crash though (not during db build/scan): Crashed Thread: 5
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x000000011981d001
Thread 5 Crashed:
0 _CppBlockUtils.so 0x000000010417b006 BtcUtils::readVarInt(unsigned char const*, unsigned long, unsigned int*) + 22
1 _CppBlockUtils.so 0x000000010417ac58 BinaryRefReader::get_var_int(unsigned char*) + 40
2 _CppBlockUtils.so 0x000000010419951e BtcUtils::TxCalcLength(unsigned char const*, unsigned long, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> >*, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> >*) + 78
3 _CppBlockUtils.so 0x00000001041bd443 Tx::unserialize(unsigned char const*, unsigned long) + 51
4 _CppBlockUtils.so 0x00000001041bd7b0 Tx::unserialize(BinaryRefReader&) + 32
5 _CppBlockUtils.so 0x000000010419db30 Tx::Tx(BinaryRefReader&) + 144
6 _CppBlockUtils.so 0x00000001041a71e5 StoredHeader::unserializeFullBlock(BinaryRefReader, bool, bool) + 1237
7 _CppBlockUtils.so 0x00000001041a8a66 StoredHeader::unserializeFullBlock(BinaryDataRef, bool, bool) + 54
8 _CppBlockUtils.so 0x00000001041ce52e std::__1::__function::__func<BlockDataManager_LevelDB::loadBlockHeadersStartingAt(ProgressReporter&, std::__1::pair<unsigned long, unsigned long long> const&, bool)::$_1, std::__1::allocator<BlockDataManager_LevelDB::loadBlockHeadersStartingAt(ProgressReporter&, std::__1::pair<unsigned long, unsigned long long> const&, bool)::$_1>, void (BinaryDataRef const&, std::__1::pair<unsigned long, unsigned long long> const&, unsigned int)>::operator()(BinaryDataRef const&, std::__1::pair<unsigned long, unsigned long long> const&, unsigned int&&) + 510
9 _CppBlockUtils.so 0x00000001041d279b BlockDataManager_LevelDB::BitcoinQtBlockFiles::readRawBlocksFromFile(BlockFileAccessor&, unsigned int const&, unsigned long long, unsigned long long, std::__1::function<void (BinaryDataRef const&, std::__1::pair<unsigned long, unsigned long long> const&, unsigned int)> const&) const + 1611
10 _CppBlockUtils.so 0x00000001041cf197 BlockDataManager_LevelDB::BitcoinQtBlockFiles::readHeaders(std::__1::pair<unsigned long, unsigned long long>, std::__1::function<void (BinaryDataRef const&, std::__1::pair<unsigned long, unsigned long long> const&, unsigned int)> const&, bool) const + 423
11 _CppBlockUtils.so 0x00000001041c6bdb BlockDataManager_LevelDB::loadBlockHeadersStartingAt(ProgressReporter&, std::__1::pair<unsigned long, unsigned long long> const&, bool) + 283
12 _CppBlockUtils.so 0x00000001041cb25e BlockDataManager_LevelDB::readBlkFileUpdate(BlockDataManager_LevelDB::BlkFileUpdateCallbacks const&) + 110
13 _CppBlockUtils.so 0x0000000104235956 BlockDataManagerThread::run() + 3798
14 _CppBlockUtils.so 0x00000001042349c9 BlockDataManagerThread::thrun(void*) + 9
15 libsystem_pthread.dylib 0x00007fff8af3f899 _pthread_body + 138
16 libsystem_pthread.dylib 0x00007fff8af3f72a _pthread_start + 137
17 libsystem_pthread.dylib 0x00007fff8af43fc9 thread_start + 13
|
|
|
|
Also the "attempting fix" code is something I have to go over, it's the stuff from 0.93 that is not really on point with the 0.94 changes.
Ah. Hence the 'should not get here' error? |
|
|
|
|
Ok, second attempt and it ran all the way through successfully. Had eleven instances of the 'top scanned block' issue in the logs, but got all the way through and displays the correct balances.
EDIT: Both runs were with no existing databases directory.
|
|
|
|
Ok, first attempt got a considerable way through scanning transactions, then failed with a popup "BDM not ready" This was in the logs -INFO - 1434747557: (BlockUtils.cpp:1443) checking scan integrity
-WARN - 1434747557: (BlockUtils.cpp:1448) Top scanned block does not match top block header
-ERROR - 1434747557: (BlockUtils.cpp:1464) shouldnt get to this line
-INFO - 1434747557: (BlockDataViewer.cpp:155) Enabling zero-conf tracking
-ERROR - 1434747557: (BDM_mainthread.cpp:429) BDM thread failed: BDM is not ready!
EDIT: Before we got to this point we had two instances of: -WARN - 1434747337: (BlockUtils.cpp:1448) Top scanned block does not match top block header
-WARN - 1434747337: (BlockUtils.cpp:1499) Issue is significant, attempting repairs
|
|
|
|
Like I said, this didn't happen in the first couple of 0.94 pre-versions you came up with, not sure what you can infer from that.
The usual conclusion in this case is: there always was an issue with this code, and changes in its vicinity revealed it. Otherwise it was probably occurring at low frequency, passing itself as a scan segfault of sorts. The usual case is "OMG how can that ever have worked?" :-) |
|
|
|
Could well be just chance since the behaviour before was clearly somewhat nondeterministric. But I did the above and it synced my wallets perfectly first time (and I hadn't managed to get a successfully sync at all so far up till now). If I get the chance tomorrow I'll nuke the databases and try again to confirm whether this is repeatable.
Well yes and no. I expect the issue is object life span: one type of threads creates some data that gets cleaned up before the next type of threads is done with it. If you reduce the amount of threads per groups to the bare minimum, you also reduce the "surface area" for the bug and thus the chance it will occur. Just to confirm, only change I made was as you suggested:
That's the proper change to force thread count for all groups down to 1. Is there any point in my trying the new version or will this version just behave the same for me? I'd like you to run it, the main fix in this version is supposed to cover your issue. Ah, glad I asked, then :-) I initially assumed that since I wasn't seeing a segfault there wasn't much point in trying it. I'll give it a go and report back. |
|
|
|
Could well be just chance since the behaviour before was clearly somewhat nondeterministric. But I did the above and it synced my wallets perfectly first time (and I hadn't managed to get a successfully sync at all so far up till now). If I get the chance tomorrow I'll nuke the databases and try again to confirm whether this is repeatable.
Well yes and no. I expect the issue is object life span: one type of threads creates some data that gets cleaned up before the next type of threads is done with it. If you reduce the amount of threads per groups to the bare minimum, you also reduce the "surface area" for the bug and thus the chance it will occur. Just to confirm, only change I made was as you suggested:
That's the proper change to force thread count for all groups down to 1. Is there any point in my trying the new version or will this version just behave the same for me? |
|
|
|
If you want to manually set your thread counts regardless, go inside BlockWriteBatcher.cpp and do the following:
1) comment out line 2004 to prevent thread adjustment
2) hardcode values in the BlockWriteBatcher ctor (line 112 for fullnode, 99 for supernode)
Could well be just chance since the behaviour before was clearly somewhat nondeterministric. But I did the above and it synced my wallets perfectly first time (and I hadn't managed to get a successfully sync at all so far up till now). If I get the chance tomorrow I'll nuke the databases and try again to confirm whether this is repeatable. Just to confirm, only change I made was as you suggested: diff --git a/cppForSwig/BlockWriteBatcher.cpp b/cppForSwig/BlockWriteBatcher.cpp
index 153558b..c11348f 100644
--- a/cppForSwig/BlockWriteBatcher.cpp
+++ b/cppForSwig/BlockWriteBatcher.cpp
@@ -109,6 +109,8 @@ BlockWriteBatcher::BlockWriteBatcher(
if (readers < 1)
readers = 1;
+ workers = readers = 1;
+
setThreadCounts(readers, workers, 1);
}
}
@@ -2001,7 +2003,7 @@ void BlockBatchProcessor::writeThread()
commitedId_.fetch_add(1, memory_order_release);
commitObject->writeTime_ = clock() - commitObject->writeTime_;
- commitObject->processor_->adjustThreadCount(commitObject);
+// commitObject->processor_->adjustThreadCount(commitObject);
thread cleanup(cleanUpThread, commitObject);
if (cleanup.joinable())
|
|
|
|
|
Thanks. I'm sure you're right, but I might try nailing down the thread count anyway. (Probably not tonight though given how long it takes to rebuild on this slow machine. Is there no way of doing an incremental build on a Mac?)
roy
|
|
|
|
|
I should also point out that this is my first time building in a very long time (I think the last time was before you offered prebuilt Mac binaries) so in particular I have absolutely no evidence that there isn't something subtly borked in my build environment.
|
|
|
|
Ah, interesting. In armorycpp.log, it looks like it got to around block 130,000 and then readjusted the number of threads. No more Only one more "finished applying" message after the thread readjust message - which takes it up to block 130,250 (which is before I was using bitcoin - so consistent with the wallets all showing zero balance). -WARN - 1433882811: (BlockUtils.cpp:1071) Scanning from 0 to 360192
-WARN - 1433882817: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 2500
-WARN - 1433882819: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 85000
-WARN - 1433882822: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 105000
-WARN - 1433882824: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 115000
-WARN - 1433882828: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 120000
-WARN - 1433882830: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 125000
-WARN - 1433882832: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 127500
-WARN - 1433882838: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 130000
-INFO - 1433882842: (BlockWriteBatcher.cpp:2067) &&&&&&&&& cumulatedReadTime: 41.2889s
-INFO - 1433882842: (BlockWriteBatcher.cpp:2068) &&&&&&&&& cumulatedWorkTime: 9.8054s
-INFO - 1433882842: (BlockWriteBatcher.cpp:2069) &&&&&&&&& cumulatedWriteTime: 0.186015s
-WARN - 1433882842: (BlockWriteBatcher.cpp:2083) Readjusting thread count:
-WARN - 1433882842: (BlockWriteBatcher.cpp:2084) 2 readers
-WARN - 1433882842: (BlockWriteBatcher.cpp:2085) 1 workers
-WARN - 1433882842: (BlockWriteBatcher.cpp:2086) 1 writers
-WARN - 1433882842: (BlockWriteBatcher.cpp:2087) 1 old reader count
-WARN - 1433882842: (BlockWriteBatcher.cpp:2088) 1 old worker count
-WARN - 1433882842: (BlockWriteBatcher.cpp:2089) 1 old writer count
-WARN - 1433882842: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 132500
-WARN - 1433882844: (BlockWriteBatcher.cpp:503) --- cumulatedBatchSleep: 0
-WARN - 1433882844: (BlockWriteBatcher.cpp:545) --- feedSleep: 0 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:548) --- workers: 0 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:551) --- writeStxo: 0.004949 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:553) --- writeStxo_grabMutex: 6.9e-05 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:556) --- waitingOnSerThread: 0.123388 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:558) --- waitForDataToWrite: 46.0291 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:562) --- checkForCollisions: 0.00029 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:564) --- getExistingKeys: 0.000519 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:566) --- getNewKeys: 0.002581 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:568) --- getSSHHeadersLock: 5.1e-05 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:570) --- computeDBKeys: 0.00322 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:572) --- getSshHeaders: 0.00015 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:574) --- finalizeSerialization: 0.048598 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:579) --- serializeBatch: 0.099822 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:581) --- updateSSH: 0.008117 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:583) --- serializeSubSsh: 8.6e-05 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:585) --- waitOnSSHser: 0.041361 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:589) --- waitOnWriteThread: 0.001888 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:592) --- putSSH: 0.151328 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:594) --- putSTX: 0.000495 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:597) --- getnextfeed: 22.2412 s
-WARN - 1433882844: (BlockWriteBatcher.cpp:599) --- inControlThread: 46.3596 s
-INFO - 1433882844: (BlockUtils.cpp:1451) --- bwbDtor: 0s
-INFO - 1433882844: (BlockUtils.cpp:1452) Scanned Block range in 46.472s
-INFO - 1433882844: (BlockUtils.cpp:1455) Finished loading at file 280, offset 31371188
-INFO - 1433882844: (BlockDataViewer.cpp:155) Enabling zero-conf tracking
-DEBUG - 1433882844: (Blockchain.cpp:214) Organizing chain
-INFO - 1433882844: (BlockUtils.cpp:1489) Loading block data... file 280 offset 31371188
-INFO - 1433882844: (BlockUtils.cpp:544) Reading raw blocks finished at file 280 offset 32869610
-WARN - 1433882844: (BlockUtils.cpp:1071) Scanning from 360193 to 360194
-WARN - 1433882845: (BlockWriteBatcher.cpp:503) --- cumulatedBatchSleep: 0
-WARN - 1433882845: (BlockWriteBatcher.cpp:545) --- feedSleep: 0 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:548) --- workers: 0 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:551) --- writeStxo: 0.000513 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:553) --- writeStxo_grabMutex: 6e-06 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:556) --- waitingOnSerThread: 0.00268 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:558) --- waitForDataToWrite: 0.339536 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:562) --- checkForCollisions: 4e-06 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:564) --- getExistingKeys: 0.000724 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:566) --- getNewKeys: 4e-06 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:568) --- getSSHHeadersLock: 4e-06 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:570) --- computeDBKeys: 2.7e-05 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:572) --- getSshHeaders: 1.4e-05 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:574) --- finalizeSerialization: 0.006073 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:579) --- serializeBatch: 0.001759 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:581) --- updateSSH: 0.000546 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:583) --- serializeSubSsh: 6e-06 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:585) --- waitOnSSHser: 0.005375 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:589) --- waitOnWriteThread: 5e-06 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:592) --- putSSH: 0.001174 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:594) --- putSTX: 3.7e-05 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:597) --- getnextfeed: 0.292328 s
-WARN - 1433882845: (BlockWriteBatcher.cpp:599) --- inControlThread: 0.356495 s
Is this thread count adjustment something I can try turning off? |
|
|
|
|
Just reran it having removed the database and log files. Takes about 12 minutes to go through receiving headers, organizing blockchain and building databases - and then goes through the scan transactions phase in well under a minute (and ends up showing no transactions and 0.0 balance). I'll send you full logs of that run if you let me know the best way to get them to you.
EDIT: Block number displayed in the GUI is correct.
|
|
|
|
Ah, after several restarts of Armory it finally started scanning - made it most of the way there - now has transactions up to late 2014. There's virtually nothing interesting in armorylog.txt. I get the following error at startup (which I presume is largely harmless?) 2015-06-09 20:54 (INFO) -- announcefetch.py:271 - Fetching: https://bitcoinarmory.com/announce.txt
2015-06-09 20:54 (ERROR) -- announcefetch.py:312 - Could not verify data in signed message block
Traceback (most recent call last):
File "/Users/roy/BitcoinArmory/osxbuild/workspace/Armory.app/Contents/MacOS/py/usr/lib/armory/announcefetch.py", line 304, in __runFetchSequence
sig, msg = readSigBlock(digestData)
File "/Users/roy/BitcoinArmory/osxbuild/workspace/Armory.app/Contents/MacOS/py/usr/lib/armory/jasvet.py", line 589, in readSigBlock
name = r.split(BEGIN_MARKER)[1].split(DASHX5)[0]
IndexError: list index out of range And apart from that littereally nothing of interest; just a couple of new blocks that came in during the scan: 2015-06-09 20:57 (INFO) -- ArmoryQt.py:3132 - Current block number: 360187
2015-06-09 20:57 (INFO) -- ArmoryQt.py:6235 - New Block! : 360189
2015-06-09 20:57 (INFO) -- ArmoryQt.py:6243 - Current block number: 360189
2015-06-09 20:59 (INFO) -- Networking.py:215 - Received new block. 0000000000000000141082c80
45e2baded1c074967f0fdf70fb65b3ebb55bd37
2015-06-09 20:59 (INFO) -- ArmoryQt.py:6235 - New Block! : 360190
2015-06-09 20:59 (INFO) -- ArmoryQt.py:6243 - Current block number: 360190
2015-06-09 21:04 (INFO) -- Networking.py:215 - Received new block. 00000000000000000b3c8380aab3c52af2302c8763acff016ad750c3765d7767
2015-06-09 21:04 (INFO) -- ArmoryQt.py:6235 - New Block! : 360191
2015-06-09 21:04 (INFO) -- ArmoryQt.py:6243 - Current block number: 360191
2015-06-09 21:14 (INFO) -- Networking.py:215 - Received new block. 0000000000000000144086516ccc3824128835fbef52eb26e0c0fc81723f0dac
2015-06-09 21:14 (INFO) -- ArmoryQt.py:6235 - New Block! : 360192
2015-06-09 21:14 (INFO) -- ArmoryQt.py:6243 - Current block number: 360192 EDIT: Fixed log paste |
|
|
|
|
Show Posts
