Bitcoin Unlimited has fixed the blk file bug in their latest release. In theory, this means that they should be compatible again and not cause 45 minute startup due to having to rescan everything. They added an "xversion" message at some point and it breaks things again (sigh). They technically broke protocol backwards compatibility with this change. f11092713 add missing DB_LAST_BLOCK write in WriteBatchSync ( #1504) (Greg Griffith) I have created a branch that maintains compatibility. I mostly coded this for myself, but it shows what you need to change to get things compatible. (Use at own risk etc.) (You need 1.5.1.0 to get the fix for the blk files.) There aren't that many changes needed - adds a -bch command line for Armory that switches to Bitcoin Cash magic net code - satoshi-port command line fix (I think this is already fixed with latest release) - sends ping message immediately after an verack to indicate that Armory node doesn't support xversion |
|
|
|
Dev is 0.97, if you gonna PR you should do it against the testing branch, with 0.96.5. If you want to build dev, you need to clone the libwebsockets repo, checkout v3.0.0 or higher, build it, then give that path to the configure script in this fashion:
Ironically, I originally had it relative to testing but then saw that dev was more advanced. Anyway, I submitted the change required.
I spent way more time on this tiny change that I intended  , at least I have a 18.04 environment now, so the time isn't lost.
./configure --with-own-lws=/my/path/to/lws
P.S: The client in dev cannot speak to db, I have not updated that part yet. Considering dropping swig and moving to CFFI to ease to migration to py3. I was 99% of the way there. I had it compiling but it couldn't find the shared library. |
|
|
|
I was going to submit a PR about the port thing and this involved building against the dev branch. This doesn't seem to be possible using Ubuntu 18.04 apt-get to get dependencies. In addition to the packages in the linux build instructions, I used the following. sudo apt-get install libprotobuf-dev protobuf-compiler python-websocket libwebsockets-dev libevent-dev
However, the libwebsockets-dev package in 18.04 (and 18.10) is not sufficiently advanced. I get an error that "lws_fop_fd_t" due to not in the include file. This typedef doesn't appear in the libwebsockets.h include file until version 2.2.0. In Ubuntu-18.04, apt-get uses version 2.0.3-3build1. Even in Ubuntu-18.10, apt-get still uses version 2.0.3-3build1. I realize that this is the dev branch and things aren't stable yet, but I thought I would flag this. Being able to build from source is an important feature for Armory. If websockets 2.2 functionality is required, would it be directly included in the repo, like some of the other dependencies? [Edit] I manually installed the package and looks like it needs v3.0 to work. |
|
|
|
The log files that I included for the satoshi-port fix were unintentionally misleading. Specifically: 2018-11-14 01:34:40 (WARNING) -- SDM.pyc:396 - Spawning DB with command: ./ArmoryDB.exe --db-type="DB_FULL" --cookie --satoshi-datadir="F:\bitcoin_data_dir\blocks" --datadir="G:\armory_data" --dbdir="G:\armory_data/database_dir --debug --satoshi-port=8500"
It doesn't actually pass the satoshi-port parameter at all to the db. The dbdir parameter is actually merged together for some reason. I suspect windows decided to "help". It ended up creating a directory with the name "database_dir --debug --satoshi-port=8500". The parameter passed to the database is: --dbdir="G:\armory_data/database_dir --debug --satoshi-port=8500"
I don't think I included the quotes when invoking things, but I may have messed up a copy/paste. The end result is that the python code doesn't actually forward the parameter to the database process. |
|
|
|
And possibly reorg speed? The revXXXXX.dat files hold the snapshot for the utxo changeset per block. Is it in sync with block alignment in blk files?
Yeah, the block's undo data is put in the corresponding rev file. Re-orging each block could require looking in a separate file for each block. |
|
|
|
As seen in the code you linked to, the style is actually confusing: the semantics do not outright suggest ReadLastBlockFile takes a reference, as it is not being checked for a status return. This kind of semantics suggest you are setting some internal value for object pblocktree instead. The following snippet would be a little easier to read: int result = pblocktree->ReadLastBlockFile(nLastBlockFile)
if(result != 0)
{
//error handling
}
//do something with nLastBlockFile
Right. Even better would be to create a local variable to indicate things. int storedLastBlockFile;
int result = pblocktree->ReadLastBlockFile(storedLastBlockFile)
if(result != 0)
{
//error handling
}
nLastBlockFile = storedLastBlockFile;
He kind of did that, since the new variable is called loadedblockfile, it just isn't used to update the global variable. With that in mind, the change in BU could very well be a mistake that led to a benign change in behavior, assuming the author of the change did not bother to acquire a broad understanding of this code before modifying it. He possibly saw a global variable changed by a local object and figured that's bad style, correcting for style without caring for the substance.
I agree that this was possibly overlooked. Having a pointer to the last file doesn't really do much if it isn't persisted over restarts. He may have decided that it was more efficient to fill in the free spaces. This change would make pruning less efficient too. Ideally, each block should have a set of files from approx the same time, so they can be deleted roughly in order. I wonder if the BU node writes to blk00000.dat on each restart if pruning is enabled. |
|
|
|
Hah, you learn something new everyday. In c++, variables can be changed by passing them to functions, thanks to the reference notation. I checked the validation.cpp file and could only see one place where there is a change to the variable. It is also changed using this code. pblocktree->ReadLastBlockFile(nLastBlockFile);
ReadLastBlockFile takes int& nFile as its input. I thought references like that were inherently constants. I think this effect is due to the changes to blockstorage.cpp (including some refactoring away from validation.cpp). The call to ReadLastBlockFile doesn't change the nLastBlockFile global variable. It just targets a local variable in the function. This means the global variable stays at its default of zero. This is what allows the node to write to previously completed blk*.dat files rather than only looking at new blk files. In the Bitcoin Core client, the call to ReadLastBlockFile has a side effect that initializes the nLastBlockFile variable. This is a Bitcoin Unlimited only behavior, I think, since Bitcoin Core and Bitcoin ABC haven't made the changes. |
|
|
|
I initially wanted this change for security reasons: Core needs to be exposed to the WAN but Armory does not need the WAN
Seems reasonable. Even if Armory accidentally connected to a remote node and received a false block hash, the local node's filesystem wouldn't have that block. In that case, Armory stalls rather than trusting the block data from the remote host. |
|
|
|
Once it writes to a blk file, it never checks files lower than that anymore. That is a RAM variable though, so it gets set back to zero each time a node is restarted.
Why would it need to check for previous files appends on new blocks? If the issue only happens on fresh node start (when free space in older block files is eligible), as long as Armory sees that one new block, it will work. I am not sure if they fully thought out the logic. I may have missed something when looking at the code on core that protects against the issue. I think it is a pretty bad policy no matter what. It's not like keeping all the blk files at the same size actually matters, and keeping nearby blocks in the same file is likely a help. They want a way to fill in spaces in the old block files but once a file overflows since last restart, they just don't bother with that one anymore. The sequence is - on startup set nLastBlockFile to 0 - when a new block arrives scan for a blk file with space starting at index = nLastBlockFile - Once it jumps to a new file, it sets nLastBlockFile to that value This means that nLastBlockFile never decreases, so once the node is running for a while it only writes to the latest blk file. On the next reboot, different spaces could end up being filled. Armory does not expect blocks to appear in order in files. It will check block data on disk from the last known block to have extended the chain. Therefor if you have up to blk01000.dat and the last top is in blk00997.dat, it will check #977 to #1000 on every new block notifications until it finds a block that extends the chain again.
Cool, so re-scanning problematic blk files wouldn't be an issue. The only question is detecting them. It looks like c++ doesn't have a portable way to get the last modified time. The assumption that later numbered blk files contain later data is violated by the code. In that situation, blk #800 could be updated and no amount of scanning after 977 will fix things. The ultimate solution would be blocks over P2P, but idk what kind of mess BCH will be by the time I'm done with that. Chances are Armory won't be compatible anymore at that point.
I don't think it is only restricted to Bitcoin Cash. The problem is much more likely to be exposed by the variable block sizes, but I think it is possible even if they are mostly stable blocks. You could get a 950kB space that doesn't get filled for a while and then someone is unlucky that they start their node just before a block that can fill that space arrives. It might actually "settle down" once it has synced and then been restarted a few times. This would "fill in" many of the spaces. As long as the first block found after a restart is bigger than any of the spaces, then it will write to the file with the highest number and that jumps nLastBlockFile to the max value. I think the workaround of wiping the armory database is probably acceptable, since the problem is rare and doesn't seem to affect the main chain. |
|
|
|
This might be a Bitcoin Cash only issue (or even Bitcoin Unlimited only issue). On the other hand, it might be a source of intermittent/hard to track down blockchain stalls with all the other clients. It looks like the Bitcoin (Cash) Unlimited client manages the block files in an unexpected way. When I look at the block files, some earlier block files have modified times that are much more recent than they should be. The equivalent Bitcoin Core code seems to work the same way, but I haven't actually checked. I did a full resync (deleted all the blk files and chainstate) and left the node running in the background. Once I noticed that it resynced, I restarted the node (before the 15:57 timestamps) and then started Armory after wiping the Armory databases. Armory resynced and then stalled. It said connected to the node. This is the end of "ls -lrt blk0*" and I assume the node completed resyncing around 6:45am. -rw------- 1 tiern tiern 120188494 Nov 19 06:41 blk01037.dat
-rw------- 1 tiern tiern 134182052 Nov 19 06:41 blk01038.dat
-rw------- 1 tiern tiern 134024033 Nov 19 06:42 blk01039.dat
-rw------- 1 tiern tiern 127342200 Nov 19 06:44 blk01040.dat
-rw------- 1 tiern tiern 131259910 Nov 19 06:45 blk01041.dat
-rw------- 1 tiern tiern 83886080 Nov 19 13:14 blk01042.dat
-rw------- 1 tiern tiern 134215061 Nov 19 15:57 blk00001.dat
-rw------- 1 tiern tiern 134215184 Nov 19 15:57 blk00004.dat
-rw------- 1 tiern tiern 134216866 Nov 19 15:57 blk00006.dat
-rw------- 1 tiern tiern 134213458 Nov 19 15:57 blk00007.dat
-rw------- 1 tiern tiern 134198809 Nov 19 15:57 blk00008.dat
-rw------- 1 tiern tiern 134207767 Nov 19 15:57 blk00009.dat
Bitcoin node code was changed to this method pretty early on (in this commit from Aug 2012). When a new block is received from the network, it scans all the blk files and finds the first one that has enough space to store the new block. This means that the file systems doesn't act as "append-only" overall. It is append-only for each file though. It has a variable called nLastBlockFile. This variable acts as a lower limit on the scan. Once it writes to a blk file, it never checks files lower than that anymore. That is a RAM variable though, so it gets set back to zero each time a node is restarted. I think the problem with Bitcoin Cash is that the blocks are highly variable in size. If the current blk file doesn't have enough to store a large block, the client will move on to the next blk index, leaving a large amount of free space. The next time the node is started, a few 50kB blocks will probably fit in spaces like that. It highlights the issue, but I think it could happen on the Bitcoin Core client too. I think this is not compatible with Armory? It assumes that the blk files are filled in order and there are no movements backwards. A solution would be to keep a record of "last-checked time" and then re-scan any old blk files that have a modified time greater or equal to the last-checked time. This check could be run any time a new block notification is received. It should only be a few blk files at any one time. Even better would be to record how much of each file was already processed, but that is probably to much hassle. Would re-scanning already scanned blk files cause problems for Armory? The workaround is to wipe the Armory database and let it re-scan everything if the block stalls but that is a sledge hammer. |
|
|
|
A quick look at the db code shows this argument is ignored as a whole. I'm about to push the RC for 0.96.5, I'll add it in there.
Great. |
|
|
|
Sorry about not including that info, I was mainly asking if it was a known issue. The log files (included below) don't contain that much info though. They look pretty much identical in both cases. The only difference is BitcoinP2P.cpp reporting connected to bitcoind in the db log file when it actually works. I had a look at the code, and I think the problem could be in cppForSwig/BlockDataManagerConfig.cpp. The btcPort_ variable is set to the #define value NODE_PORT_MAINNET rather than some kind of command line controlled variable. It is hard coded depending on main/test/regtest networks rather than allowing the port to be selected. Standard PortBitcoind started (manually) with this line in bitcoin.conf. (default port used) Armory started from command line with this. C:\Program Files (x86)\Armory\ArmoryQt.exe --datadir=G:\armory_data --dbdir=G:\armory_data/database_dir --debug
It successfully connects. dbLog.txt armorylog.txtAlternative Port (8500)Bitcoind started (manually) with this line in bitcoin.conf Armory started from command line with this. C:\Program Files (x86)\Armory\ArmoryQt.exe --datadir=G:\armory_data --dbdir=G:\armory_data/database_dir --debug --satoshi-port=8500
The GUI stays in offline mode. dbLog.txt armorylog.txt |
|
|
|
|
If you are sure that the keys are not in use for the main chain, then the easiest is to use Electron cash (or similar). That means that you won't have to download the entire bitcoin cash chain (as you would with the abc client).
I think Bitcoin.com wallet also protects your keys somehow.
Presumably, you just want to send your coins to an exchange?
|
|
|
|
Honestly I'm not sure who came up with the idea of HD wallets first.
It looks like the original idea came from Greg Maxwell, but Armory was the first implementation to run with it. It is also possible that it was simultaneously invented / repeatedly invented by multiple people. |
|
|
|
|
I have a node running on a non-standard port and am trying to get Armory to connect to it. It connects if I use the standard port, but not if I use a non-standard port.
I think there may be some issue with passing the command line argument over to the database process, but that is just a guess.
|
|
|
|
Armory "Root Keys" are proprietary I don't think the method is actually 'proprietary'. That would mean that nobody else was allowed to use their method. and, as far as I am aware, only work with Armory... Right, but that is only because nobody uses the Armory method. They could if they wanted. Instead they use the standard (BIP-32) hierarchical key generation method. Armory would probably have used BIP-32 as well, but was developed before BIP-32 was created. |
|
|
|
Edit, that change alone isn't enough, armory won't connect to bitcoinABC.
Check your node log, should tell you why Armory is getting rejected. This is the error message. 2018-06-26 23:51:31 PROCESSMESSAGE: INVALID MESSAGESTART version peer=127.0.0.1:56606 (53)
The endianess needs to be swapped. When expressed as an uint32_t, the magic value for Bitcoin Cash is 0xe8f3e1e3 rather than 0xe3e1f3e8. The updated version of your suggestion is as follows. https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/BlockUtils.cpp#L886 networkNode_ = make_shared<BitcoinP2P>("127.0.0.1", config_.btcPort_,
*(uint32_t*)config_.magicBytes_.getPtr());
Should be changed to: networkNode_ = make_shared<BitcoinP2P>("127.0.0.1", config_.btcPort_,
0xe8f3e1e3);
This connects properly which I tried it. I also wiped the contents of the Armory db. I don't know if that was required. It looks like Bitcoin Cash clients have 2 magic patterns, one for the network and one for disk storage. This means that even without any changes, Armory will sync with the blocks stored on the disk. It just doesn't do online updates (or broadcasts), since it can still read the files on the disk. |
|
|
|
Right, but TierNolan is saying that if one private key is exposed all private keys and addresses in that wallet can be found, not just that one - it only requires one private key and the data from a watch only wallet... both of which I imagine have been on an online machine. It needs to be a new/fresh Armory wallet, not just a new address in the same wallet.
This is important if you don't want to risk your coins.
Exactly. The risk is low but possible. The key point is that Armory generates private keys in a sequence. It uses the watch-only info and the previous private key to compute the next private key. Once you have 1 private key and the watch-only info, you can compute all the remaining private keys. (I am pretty sure earlier keys are safe) An example would be if an altcoin was created that had a malware in the official wallet itself. You download the official wallet for the altcoin and install it. It scans your computer for watch-only wallets and sends them to some server. It also sends any private keys that you import into the wallet. That allows the attacker to generate any later private keys. [Edit] Fair enough. The attacker needs the watching only wallet + 1 private key. If they can get the private key, then the watching only info could very easily be obtained too, since that is on the online computer. |
|
|
|
so i took the private key of this address and i import it to bitcoin abc wallet.
It is worth noting that if you import a private key into another wallet from Armory, then it weakens the security of your Armory wallet. If someone gets that private key, then they can sometimes figure out other private keys in your wallet. Ideally, the sequence is - move Bitcoins to new wallet - export private keys for the altcoin - move altcoins to a wallet that supports those coins |
|
|
|
Are you trying to turn your Pi into some sort of automated hardware signer?
It is intended as an alternative to using USB to transfer the files to/from the offline computer. The PI would be setup with Armory and the ethernet transfer program. After that, the only thing the offline computer needs to do is offline signing. I am also thinking about setting up the PI with a 50MB RAM drive as Armory's data directory. This means that all saved files from Armory are reset after each powerup. The process would be - create unsigned tx on PC - disconnect PC from internet - connect PC to PI via ethernet - upload unsigned transaction via ethernet transfer (PC - PI) -- save in RAM drive - start Armory on PI - restore wallet from paper backup fragments to RAM drive - sign transactions - download signed transactions via ethernet transfer (PI - PC) - check signed transaction on PC - disconnect PI from PC - connect PC to internet - broadcast signed transactions |
|
|
|
|
Show Posts
