Title: General questions arising when adding Namecoin support to Armory Post by: josephbisch on November 11, 2014, 02:13:35 AM So I am working on adding Namecoin support to Armory.
I think that I got a lot of the work done already. My issue is that Namecoin is based off of an old version of Bitcoin. Therefore it uses block files like ~/.namecoin/blk0001.dat instead of ~/.bitcoin/blocks/blk00000.dat, which is used by modern Bitcoin clients. Armory is currently designed to look for blocks of the latter naming convention. What I want to do is modify the C++ code to find the Namecoin blocks when the --namecoin flag is set and to find Bitcoin blocks when there is no --namecoin flag. I've figured out that I want to modify the getBlkFilename function in cppForSwig/BtcUtils.h (https://github.com/etotheipi/BitcoinArmory/blob/e48c04b205f7c2e2768c9d35f98fb69b05049455/cppForSwig/BtcUtils.h#L1319). The problem is I need to do so based on a variable called COIN in armoryengine/ArmoryUtils.py (it's not in the GitHub repo, just in my local repo). The variable COIN will either be the string 'Namecoin' or the string 'Bitcoin' based on flags set when running Armory. So my question is how would I go about accessing COIN from BtcUtils.h? Or is there some other way to determine the network in use that I am missing? Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: njaard on November 11, 2014, 03:13:13 AM Hi Joseph,
The short answer is "you don't". The useful answer is that Armory loads the C++ side from BDM.py. There it creates a BlockDataManagerConfig object. When you create a BlockDataManagerConfig, you could specify parameters there that affect the C++ side's behavior, like the file format convention for blk files. You're developing against the "dev" branch, right? You should. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: etotheipi on November 11, 2014, 04:30:40 AM FYI, here's the old transitional code for the same method:
https://github.com/etotheipi/BitcoinArmory/blob/v0.88-beta/cppForSwig/BtcUtils.h#L736 And as njaard said: you don't access python from C++. If you need something, put it in the C++ side, and call it from python. SWIG is amazing (being able to call C++ objects and methods as if they're native python objects), but it doesn't go both ways, and in fact it can get quite complicated to do so. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 11, 2014, 02:03:16 PM First of all, yes I am developing against the dev branch. When I am done, I plan on stashing my changes, bringing dev up-to-date, reapplying my changes, and making a git diff to email.
Looking at BlockDataManagerConfig and BDM.py, I see that the Python code passes the magic bytes to the C++ code. So I think the best way to do what I want to do is by checking the magic bytes. The comments in cppForSwig/BlockUtils.h reference a GetInstance() (https://github.com/etotheipi/BitcoinArmory/blob/ce382beb866fa75e6b2dc3b2cf41855ef3b04f99/cppForSwig/BlockUtils.h#L92) method. I don't see any GetInstance() method, so I assume it was removed? How would I safely get an instance of BlockDataManager_LevelDB? I think I want an instance of BlockDataManager_LevelDB, because that class has getMagicBytes(). Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: njaard on November 11, 2014, 04:41:46 PM You cannot get an instance of BlockDataManager_LevelDB from anywhere, anymore. But anywhere that calls getBlkFilename has an instance of BlockDataManagerConfig available.
I recommend you put a boolean in in BlockDataManagerConfig that sets things to namecoin mode, and then check that boolean wherever getBlkFilename is called, and maybe write a new function, or use etotheipi's old version and modify its parameters based on that. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 11, 2014, 08:51:22 PM I think I may have gotten in over my head with trying to add Namecoin support. I don't think I'm going to be able to finish the C++ parts (myself at least).
I hardcoded Namecoin as the network, because I couldn't figure out how to access the configuration data from within the detectAllBlkFiles method. As an aside, I made the variable a string, so that it can be used for future altcoin additions. Now I get a bunch of errors like the following when building the db. Code: -ERROR - 1415733082: (StoredBlockObj.cpp:332) Merkle root mismatch! Raw block data is corrupt! The merkle root calculation should be independent of the chain in use, so I don't think that is it. The dat file should be parsable. So I don't understand what caused the error. FWIW, I get these errors with Namecoin and Namecoin Testnet. I'll look more into it when I have time later, but I just wanted to post where I'm at now, in case anyone has any tips. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: njaard on November 11, 2014, 08:57:55 PM You got so far into the the file (229714131 bytes) that it makes me think maybe the file itself is corrupt. Try blowing away namecoind data and download its blockchain again?
But conversely, you got it in the testnet as well, so that seems unlikely. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 11, 2014, 09:04:37 PM If my math is right, it's only 229 MB into the file, when the whole namecoin blockchain as reported by bitinfocharts.com is 1.97 GB. My entire ~/.namecoin directory is 2.2 GB, and that includes the testnet blockchain.
Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: njaard on November 11, 2014, 09:19:19 PM Open up a hex editor on that file, go to that byte offset, and see if there's anything unusual.
Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: etotheipi on November 11, 2014, 09:43:33 PM If my math is right, it's only 229 MB into the file, when the whole namecoin blockchain as reported by bitinfocharts.com is 1.97 GB. My entire ~/.namecoin directory is 2.2 GB, and that includes the testnet blockchain. This m might be a simple question for a namecoin guru: what is the difference in block formats between Bitcoin and namecoin? It might be something stupid and simple. Could it have to do with the name reservation system? Do transactions have a different structure when they register namecoin names? Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 11, 2014, 09:59:46 PM I started redownloading the namecoin testnet blockchain before your post, so I used the hex editor on a different block (there were many that had the error).
The output: Code: joseph@crunchbang:~$ hexdump -s 229723055 -n 602 .namecoin/testnet/blk0001.dat It starts with the version/magic bytes (bffa feb5), which appear as fabfb5fe in the Armory code, but there are no errors regarding the magic bytes, so I guess it is supposed to be like that. Next is supposed to be the prev_block (025a 0000 0101 0001 dc8f c35c 6542 81ba 1967 8a6e 9340 f339 fd89 da53 044b a3f1), but that doesn't look like it has enough leading zeros. The big strings of zeros later on in a couple of places look suspicious too. I will try to find a namecoin expert to check to see if there is some difference in the format. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: njaard on November 11, 2014, 10:51:44 PM It's hexdump that is exchanging the order of each pair of bytes. Give it the option -C
Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 12, 2014, 12:08:06 AM Thanks, it is indeed the correct magic bytes when using the -C option.
I got a response (https://github.com/etotheipi/BitcoinArmory/issues/230) from one of the Namecoin guys, who referred me to another Namecoin guy that will know more. It looks like we have two main differences. One is that there are merged mining headers in addition to the normal data. Two is that Namecoin still uses BDB instead of LevelDB. But that isn't really an issue, since the underlying data format is the same (with the exception of the addition of the merged mining headers), right? A version of Namecoin based off of a more recent version of Bitcoin is in the works, but there is no ETA on it. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: njaard on November 12, 2014, 01:52:54 AM Whether they use BDB or LevelDB is not relevant because those blk files are not BDB or LevelDB files, so indeed that does not make a difference.
That their headers have a different format does make a difference (and I would not be surprised if they started using merged mining at the time of when your patched Armory failed). In order to support their slightly different block format, you'll need to modify the class BlockHeader. I would recommend, for the time being, not bother making a program that supports both bitcoin and namecoin without recompiling. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 12, 2014, 08:07:20 PM I took a look at BlockHeader (and saw how it has version and prevBlock and all that), but I think I really want the TxCalcLength function in BtcUtils.h. That is because the merged mining data comes before the transactions, so I need to offset the start of the transactions by the length of the merged mining data.
BlockObj.cpp calls TxCalcLength. I added a parameter to TxCalcLength to pass the block height and I used getBlockHeight for the parameter, but it is just uint32_max, so I guess it isn't set at that point in the execution. My plan is to increment the pointer in TxCalcLength to skip the merged mining data. I will use the block height to decide whether to increment or not, because the merged mining only starts at block 19200. If it was reading the transaction data incorrectly then that would explain the merkle root mismatch errors I was getting. In addition to the help from the Namecoin devs, I found some helpful information on the wiki (https://en.bitcoin.it/wiki/Merged_mining_specification). One thing I learned from the devs is that the hash of merge mined blocks won't necessarily have leading zeros as you can see here with the first merge mined block (http://explorer.namecoin.info/b/19200), which makes sense now that I think about it. So that explains why the block I was looking at with hexdump doesn't have leading zeros for it's prevBlockHash. The wiki page confirms that the auxpow block (auxpow refers to the Namecoin block when a block is merge mined) has it's extra data inserted between the nonce and txn_count. The extra data includes variable length fields, so it's not going to be trivial to calculate the offset to find where the txn_count starts. Boy, when I started I thought this would just be a matter of tweaking a few parameters in Python and changing the text to say Namecoin. I was sure wrong! Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 12, 2014, 10:59:03 PM As I said before, I think I got in over my head with the C++. I originally thought I could add Namecoin support by just modifying the Python code. Even when it became apparent that I would need to work with the C++ code, it looked a lot simpler than it does now.
It may seem premature to give up already considering that I probably have only spent a few days on Namecoin support, but it is apparent now that I won't be able to do it myself. I feel like I'm taking more than I am contributing at this point. Thank you to everyone who has helped me. I'll post the diff of what I have so far tomorrow and hopefully someone who knows more than I do comes along and finishes the C++ side of things. I should stick to what I know really well, which is Debian packaging and Python coding. :) Sorry if I got anyone excited at the prospect of Namecoin support in Armory. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 13, 2014, 02:59:25 PM So here is the diff so far:
Code: diff --git a/ArmoryQt.py b/ArmoryQt.py Let me know if you want it as a file uploaded somewhere. Some notes: The fee calculation logic may need to be changed for Namecoin. I did change the MIN_TX_FEE and MIN_RELAY_TX_FEE, but not the fee calculation logic. P2SH and multisig will need to be disabled until the new version of Namecoin that is rebased against Bitcoin Core is released. BlockDataManagerConfig.h has a string chain, which is set in the Python code to either 'Bitcoin' or 'Namecoin' depending on which is being used. The variable chain isn't actually being used right now, because I couldn't figure out how to use it. I have 'Namecoin' hardcoded in BlockUtils.cpp (and that part is working correctly to read the correct block files). If you want to switch between Namecoin and Bitcoin, you need to recompile. I think TxCalcLength needs to be modified in BtcUtils.h, so that Armory knows that the transaction part of the block starts after the merged mining data. A couple of posts ago I posted helpful information I found on the wiki about merged mining. The difficult part is that the merge mined data is not a constant size. And a block may not necessarily be merge mined even though it is at height 19200 or greater. Namecoin currently has the donate to ATI button hidden, because I don't know of a Namecoin donation address. I tested this, but at this stage all I could test was that the correct addresses were being generated and that the correct block files were detected. And finally, I did not change all the instances of the word Bitcoin to Namecoin, so that still needs to be done. I figured it would just make the diff harder to read at this point. Title: Programming Namecoin support for Armory Post by: josephbisch on November 14, 2014, 09:37:28 PM I said I gave up, but I continued playing around with block parsing (except I did so in Python, because I am more comfortable with that). I noticed that merge mined blocks have a different version than non merge mined blocks. That was the key I was missing. You need to check the 0x100 bit to see if a block has merge mined data or not.
Going back to the Armory code, I created a class that unserializes the merge mined data (no need to be able to serialize it, since we don't care about storing the actual data). I mirrored the structure of the BlockHeader class. I'm a little lost on where to call the unserialize method. I tried putting it in the blockHeaderCallback (https://github.com/etotheipi/BitcoinArmory/blob/dev/cppForSwig/BlockUtils.cpp#L666) after unserializing the block header, but before nTx. The problem with that is that only 90 bytes are provided to blockHeaderCallback. The 90 bytes consist of 80 for the block header and 10 for the nTx, which is a var_int. Obviously trying to unserialize the merge mined data results in an attempt to go past the end of the 90 bytes, which results in an error. I tried looking for some place earlier in the execution where we have the entire block's data, so that I could skip the merge mined data there, but couldn't find it. I don't think I want to modify the BlockHeader class, because the merge mined data is not actually part of the block header. It comes right after the block header, but before the number of transactions. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: goatpig on November 15, 2014, 12:00:37 PM blockHeaderCallback only cares for headers, it doesnt unserialize the blocks' content. It needs to know the end offset of each block, but I'm guessing block size in header is correct so any amount of extra data is irrelevant as long as the first 80 bytes are aligned like in Bitcoin.
In StoredHeader::unserializeFullBlock, this is where the magic happens: Quote vector<BinaryData> allTxHashes; BlockHeader bh(brr); uint32_t nTx = (uint32_t)brr.get_var_int(); createFromBlockHeader(bh); numTx_ = nTx; numBytes_ = HEADER_SIZE + BtcUtils::calcVarIntSize(numTx_); if(dataCopy_.getSize() != HEADER_SIZE) { LOGERR << "Unserializing header did not produce 80-byte object!"; return; } if (numBytes_ > brr.getSize()) { LOGERR << "Anticipated size of block header is more than what we have"; throw BlockDeserializingException(); } BtcUtils::getHash256(dataCopy_, thisHash_); for(uint32_t tx=0; tx<nTx; tx++) { //unserialize tx here } 1) Based on block version, either ignore the following block or define HEADER_SIZE_NMC and check against that: Quote if(dataCopy_.getSize() != HEADER_SIZE) { LOGERR << "Unserializing header did not produce 80-byte object!"; return; } 2) Push brr forward by the amount of unnecessary bytes in between the header and the block data. It should look something like this: Quote //Somewhere in the header #define HEADER_VERSION_NMC someval //header version number for NMC #define NMCHEADER_NUMBYTESTOIGNORE someotherval //amount of bytes to skip after the header to get to txdata /////// //back to unserializeFullBlock BtcUtils::getHash256(dataCopy_, thisHash_); if(bh.getVersion() == HEADER_VERSION_NMC) brr.advance(NMC_HEADER_NUMBYTESTOIGNORE); for(uint32_t tx=0; tx<nTx; tx++) { //unresialize tx here } This will get you through unserializing and adding blocks to DB, I suppose you'll run into other snafus, but I can't tell you were on top of my head. The best thing you could do is create a separate C++ unit test that inits the BDM with a NMC block, with all the necessary comments to identify the parts of the code that need changed. You could also consider overloading these calls in a dedicated version of the class. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 15, 2014, 05:35:31 PM Thanks for the help! I am able to receive and send testnet Namecoins with Armory! I haven't tried testing it with mainnet yet.
I did run into a small bug where, after sending a transaction, the pop-up in the taskbar and the transactions tab on the main screen say I only sent 0.005, when I really sent 1 plus a 0.005 fee. I am on the dev branch, but haven't brought it up to date for a few days. So maybe it is a known bug that is already fixed. I'll do more testing to determine whether this is Namecoin specific or not and see if it is fixed with an up-to-date dev branch. Once there is a confirmation the transactions tab correctly lists the amount as 1.005. Also, when double clicking on the transaction, the window that pops up displays the correct amount, even if there is not yet a confirmation. I need to disable P2SH and multisig for Namecoin for now. I might need to tweak the fee calculation code too. Then I'll try to write that unit test. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: picobit on November 15, 2014, 06:10:38 PM Thanks for the help! I am able to receive and send testnet Namecoins with Armory! I haven't tried testing it with mainnet yet. I did run into a small bug where, after sending a transaction, the pop-up in the taskbar and the transactions tab on the main screen say I only sent 0.005, when I really sent 1 plus a 0.005 fee. I am on the dev branch, but haven't brought it up to date for a few days. So maybe it is a known bug that is already fixed. I'll do more testing to determine whether this is Namecoin specific or not and see if it is fixed with an up-to-date dev branch. Once there is a confirmation the transactions tab correctly lists the amount as 1.005. Also, when double clicking on the transaction, the window that pops up displays the correct amount, even if there is not yet a confirmation. I need to disable P2SH and multisig for Namecoin for now. I might need to tweak the fee calculation code too. Then I'll try to write that unit test. If you sent it to yourself (i.e. another address in the same wallet) then the list on the transaction tab will only show the fee, since that is the net amount leaving the account. The symbol next to the transaction will be different - there is one for outgoing, one for incoming, and one for send to self. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 15, 2014, 06:11:55 PM Thanks for the help! I am able to receive and send testnet Namecoins with Armory! I haven't tried testing it with mainnet yet. I did run into a small bug where, after sending a transaction, the pop-up in the taskbar and the transactions tab on the main screen say I only sent 0.005, when I really sent 1 plus a 0.005 fee. I am on the dev branch, but haven't brought it up to date for a few days. So maybe it is a known bug that is already fixed. I'll do more testing to determine whether this is Namecoin specific or not and see if it is fixed with an up-to-date dev branch. Once there is a confirmation the transactions tab correctly lists the amount as 1.005. Also, when double clicking on the transaction, the window that pops up displays the correct amount, even if there is not yet a confirmation. I need to disable P2SH and multisig for Namecoin for now. I might need to tweak the fee calculation code too. Then I'll try to write that unit test. If you sent it to yourself (i.e. another address in the same wallet) then the list on the transaction tab will only show the fee, since that is the net amount leaving the account. The symbol next to the transaction will be different - there is one for outgoing, one for incoming, and one for send to self. I sent it from an address in Armory to an address in Namecoin-qt. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 15, 2014, 06:51:30 PM Was this line of this commit (https://github.com/etotheipi/BitcoinArmory/commit/e7f5664ee84fb55c8e6beb4cb806b5b2e14323f8#diff-d7f973127e62fd7cc685f9a7d3c5ae48L127) supposed to be removed? Because I am getting "AttributeError: 'ArmoryMainWindow' object has no attribute 'lastSDMState'" when trying to start ArmoryQt.py with an up-to-date dev branch. I'm sure this is probably already known, or some error on my part.
Edit: Nope, ArmoryQt.py does launch, but it gets stuck at the block syncing stage. It won't proceed to create the Armory db. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: goatpig on November 16, 2014, 01:04:19 AM the ZC value issue was fixed, but the dev branch is in shambles currently.
Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 16, 2014, 03:45:33 PM Thanks, it's good to know that the dev branch is in a broken state and it's not just me.
I pushed what I have so far to my GitHub fork (https://github.com/josephbisch/BitcoinArmory/tree/namecoin). I think I disabled multisig for Namecoin, but I haven't tested it due to the state of dev. Looking through the code, it appears that leaving P2SHBYTE as an empty string is enough to disable P2SH for Namecoin (because when doing comparisons, an empty string is never going to == a byte), so I think I have disabled that too. I looked at the fee calculation code and there are three functions for calculating fees. It looks like one of the more complex parts of the Namecoin integration. I guess I can leave the multisig calculation one alone for now, since Namecoin doesn't currently support multisig. That leaves calcMinSuggestedFees and calcMinSuggestedFeesNew and it looks like I need to modify both for Namecoin. I'll watch the dev branch to see when ArmoryQt.py is runnable again. In the meantime I'll finish up the Namecoin support including that unit test you asked for. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: CircusPeanut on November 16, 2014, 04:24:41 PM Was this line of this commit (https://github.com/etotheipi/BitcoinArmory/commit/e7f5664ee84fb55c8e6beb4cb806b5b2e14323f8#diff-d7f973127e62fd7cc685f9a7d3c5ae48L127) supposed to be removed? Because I am getting "AttributeError: 'ArmoryMainWindow' object has no attribute 'lastSDMState'" when trying to start ArmoryQt.py with an up-to-date dev branch. I'm sure this is probably already known, or some error on my part. Edit: Nope, ArmoryQt.py does launch, but it gets stuck at the block syncing stage. It won't proceed to create the Armory db. Thanks for the heads up. I put that line back so it won't crash there. There still seem to be issues with running BitcoinD from Armory on Testnet. I usually uncheck "Let Armory start Bitcoin..." in the Settings menu, and start BitcoinD myself when I'm working with Testnet. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 16, 2014, 04:41:04 PM Was this line of this commit (https://github.com/etotheipi/BitcoinArmory/commit/e7f5664ee84fb55c8e6beb4cb806b5b2e14323f8#diff-d7f973127e62fd7cc685f9a7d3c5ae48L127) supposed to be removed? Because I am getting "AttributeError: 'ArmoryMainWindow' object has no attribute 'lastSDMState'" when trying to start ArmoryQt.py with an up-to-date dev branch. I'm sure this is probably already known, or some error on my part. Edit: Nope, ArmoryQt.py does launch, but it gets stuck at the block syncing stage. It won't proceed to create the Armory db. Thanks for the heads up. I put that line back so it won't crash there. There still seem to be issues with running BitcoinD from Armory on Testnet. I usually uncheck "Let Armory start Bitcoin..." in the Settings menu, and start BitcoinD myself when I'm working with Testnet. Thanks, I tried putting the line back myself and had issues with both Namecoin and Bitcoin as my edit said, so I will try it with starting namecoind and bitcoind myself. Edit: Starting namecoind myself helped with the issues. I'm amazed at the speed of rebuilding and rescanning the db with the recent updates. It happened almost instantaneously (granted I was on Namecoin testnet, so there was a lot less data than Bitcoin mainnet, but still). Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 16, 2014, 08:08:41 PM If anyone reading this wants to help speed up the process of getting Namecoin support into Armory, they can help by testing Namecoin in Armory. If you do, make sure you only use testnet Namecoins, because this hasn't been thoroughly tested. To do so:
1) Git clone this repository (https://github.com/josephbisch/BitcoinArmory/tree/namecoin) 2) git checkout namecoin 3) make 4) python ArmoryQt.py --namecoin-testnet Those instructions work for me on Linux. They may vary for other OS. You might need to go to options, uncheck "let Armory run bitcoind in the background", and then manually run namecoind before starting Armory (run namecoind with the -testnet flag). I needed to do so. If you do test Namecoin integration, please let me know, even if you don't experience any errors. One thing to note is that I haven't changed the text, so Armory will still say Bitcoin and BTC even though you provided the --namecoin-testnet flag to it. But if you look at the addresses being generated, you will see that they are testnet addresses. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 16, 2014, 11:48:27 PM About that unit test for NMC, should I put it in CppBlockUtilsTests.cpp or make a separate file for it? I think I'll try to work on it tomorrow.
I think the unit test, text changes (replacing Bitcoin with Namecoin), and making it so Namecoin is not hardcoded in the C++ are the only things stopping this from being ready to be reviewed by Armory devs. But then again I haven't received any indication that anyone else besides me has been testing this code, so it is possible there are bugs I haven't discovered. I'm unsure about the fee calculation code, though in my tests it appeared to work correctly. I might have to hand the code off with Namecoin hardcoded in the C++, because I am really lost on how to access the config data. I should be ready sometime this week for the code to be reviewed! Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 17, 2014, 03:21:51 PM I pushed an update that replaces Bitcoin/bitcoins/BTC with a function (getCoinText()) that returns the correct name of coin in the correct format. I should have replaced most instances with the function except for a few cases. Those are:
References to Casascius coins. There are no NMC Casascius coins, so I guess I will just get rid of that text when NMC is activated. Or I could leave it as an example of the type of keys it is referring to. The torrent download (like in the settings panel). I don't think there is a torrent for NMC, so I will completely disable that. The "bitcoin:" links (I realized I still haven't changed that for NMC (namecoin:), so I still need to do that). Let me know if you discover another location where you don't see NMC or Namecoin or namecoins when you are testing with Namecoin enabled. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: goatpig on November 17, 2014, 04:49:57 PM About that unit test for NMC, should I put it in CppBlockUtilsTests.cpp or make a separate file for it? I think I'll try to work on it tomorrow. I think the unit test, text changes (replacing Bitcoin with Namecoin), and making it so Namecoin is not hardcoded in the C++ are the only things stopping this from being ready to be reviewed by Armory devs. But then again I haven't received any indication that anyone else besides me has been testing this code, so it is possible there are bugs I haven't discovered. I'm unsure about the fee calculation code, though in my tests it appeared to work correctly. I might have to hand the code off with Namecoin hardcoded in the C++, because I am really lost on how to access the config data. I should be ready sometime this week for the code to be reviewed! You should obviously fork the repo and keep your NMC unit test in a dedicated file to cover your changes so it doesn't get modified after merging in changes on our end. That way you have a static unit test to throw at our changes to make sure your modifications remain consistent pull to pull. dev should be back to a functional state since last Sunday afternoon. It may go under for a while again though. Depending on how large NMC mainnet is you may run in the issue I'm currently working on, so let me know and I'll get you to bypass it for now. Edit: Starting namecoind myself helped with the issues. I'm amazed at the speed of rebuilding and rescanning the db with the recent updates. It happened almost instantaneously (granted I was on Namecoin testnet, so there was a lot less data than Bitcoin mainnet, but still). As CircusPeanut said, testnet has never worked with auto managed bitcoind. There's some snafu with the path resolving on the Python side I think, but this is such low priority we have left this issue unattended for over a year now. As for the speed, I got rid of some scan concurrency recently, so that got a bit slower but in a few commits I'll crank the speed up again. For reference I get Bitcoin testnet supernode scanned in less than 3 minutes. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 17, 2014, 05:17:17 PM I already forked the repo (available here (https://github.com/josephbisch/BitcoinArmory/tree/namecoin) fyi), so I'll add a new file to it for the unit test under cppForSwig/gtest/.
I'm not sure what issue you are running into, but I just tried Namecoin mainnet and I am stuck synchronizing at 22%. It has been at most a few days since I last synced Namecoin mainnet, so it shouldn't be as low as 22%. I've been mostly testing Namecoin testnet anyway, so if it is your issue, then it isn't a big deal to me. Namecoin mainnet is just under a couple of gigs, fyi. I think I am going to make a separate thread for testing Namecoin integration. I'm not getting anyone to test it and this thread's poor choice of title probably has to do something with that. I'll continue posting my questions/comments about development here. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 17, 2014, 10:02:31 PM Is there an example test that inits the BDM with a Bitcoin block (so that I can adapt it to Namecoin)? Because I wrote a test that calls unserializeFullBlock and then checks a lot of the data in the block, but I don't think that is what you had in mind, because it has nothing to do with the BDM.
You can see what tests I have so far (https://github.com/josephbisch/BitcoinArmory/commit/5dc0350ad9772863e2820ded28835ec8712b6bbb). Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: goatpig on November 17, 2014, 11:52:04 PM You want to pass this test for BDM init operations: https://github.com/etotheipi/BitcoinArmory/blob/dev/cppForSwig/gtest/CppBlockUtilsTests.cpp#L6794
Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 18, 2014, 02:53:11 AM I pushed a bunch more changes, including more text changes and support for "namecoin:" URIs. Also I disabled the torrent for Namecoin.
Now I think the only things left are the unit test, support for Namecoin in the built-in downloader, and support for Namecoin in armoryd. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 18, 2014, 05:32:07 PM I copied that test over to my NamecoinTests.cpp file (removing the lockbox portion, since NMC doesn't support that currently). I also copied the necessary functions to run that test. When running the test, I get output indicating that it is running, but it appears to be stuck running that test.
I used createTestChain.py, modifying it to load the Namecoin genesis block instead of the Bitcoin one, setting fakeblock to True, and using the Namecoin magic bytes. The line "TheBDM.doInitialSyncOnLoad(nullProgress);" seems to be the one that is hanging. Does fakeblock need to be False when running createTestChain.py for this test? Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: goatpig on November 18, 2014, 06:35:35 PM I copied that test over to my NamecoinTests.cpp file (removing the lockbox portion, since NMC doesn't support that currently). I also copied the necessary functions to run that test. When running the test, I get output indicating that it is running, but it appears to be stuck running that test. I used createTestChain.py, modifying it to load the Namecoin genesis block instead of the Bitcoin one, setting fakeblock to True, and using the Namecoin magic bytes. The line "TheBDM.doInitialSyncOnLoad(nullProgress);" seems to be the one that is hanging. Does fakeblock need to be False when running createTestChain.py for this test? It can be either. We set it to false to skip difficulty (use 0 for each block's nonce) in order to create test chains a lot faster. The line hanging is the the one initializing the backend. You should set a few breakpoints there and see where it is choking. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 18, 2014, 08:47:00 PM I traced the issue to BlockWriteBatcher::applyBlocksToDB with gdb. I should have looked in the test log file initially, because there were errors there relating to BlockWriteBatcher. Specifically there seems to be a problem with the size of the headers. Which I don't understand, because the output of createTestChain.py seems to indicate that the headers are 80 bytes as expected.
Code: Log file opened at 1416341503: namecoinTestsLog.txt Edit: I put the block file through my Python parser and the data being output looks okay to me. These blocks don't even have the extra Namecoin data yet. And the weirdest thing is that this issue only presents itself when running the tests. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: goatpig on November 18, 2014, 11:44:08 PM I traced the issue to BlockWriteBatcher::applyBlocksToDB with gdb. I should have looked in the test log file initially, because there were errors there relating to BlockWriteBatcher. Specifically there seems to be a problem with the size of the headers. This is what I expect is happening (as a peudo traceback): Code: BlockWriteBatcher::applyBlocksToDB This method eventually gets to this: Code: // Unserialize the raw header into the SBH object If you modified HEADER_SIZE, it's going to choke on non NMC blocks. Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 18, 2014, 11:47:30 PM I saw that code. The problem is I don't modify HEADER_SIZE (because the actual 80 byte header is the same for Namecoin). Which is why I find all of this quite odd.
Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: josephbisch on November 19, 2014, 01:25:42 AM I just pushed a bunch of code. I still don't know what is wrong with the test, but I think I have all the actual code written (for ArmoryQt, I haven't really looked at armoryd yet). So I think it is okay to start reviewing the code. Though I still haven't heard from anyone else that tested the code besides myself. So there could be some obvious bugs that I am overlooking.
Title: Re: Accessing Python variable from C++ (for adding Namecoin support to Armory) Post by: goatpig on November 19, 2014, 02:38:00 AM As a certain someone said, I'm up to my eyeballs with work currently, so I can't spend time debugging your changes. The issue is most likely occurring when the blocks are committed to the DB. BlockWriteBatcher reads commited blocks from DB and builds up the transaction history. It doesn't modify block data so there is no reason properly committed blocks would choke at that stage.
You should try to apply a block to the DB using BlockDataManager_LevelDB::addRawBlockToDB, then pulling it out with LMDBBlockDatabase::getStoredHeader. That should help you isolate the deserialization issue. Title: Re: General questions arising when adding Namecoin support to Armory Post by: jonasbits on November 22, 2014, 12:44:04 AM The build was easy if you followed this guide https://bitcoinarmory.com/building-from-source/
At first I did not, and a lot of error told me to install this and that :-) I will try to look for things to improve, good work! Title: Re: General questions arising when adding Namecoin support to Armory Post by: josephbisch on December 07, 2014, 02:38:20 AM I've taken a break from working on this, but I have a question to ask on behalf of the Namecoin devs.
Would Armory devs merge Namecoin specific transaction types into Armory if someone were to code it? This could be used in conjunction with an Armory plugin to automatically renew domain names to prevent expiration. Title: Re: General questions arising when adding Namecoin support to Armory Post by: goatpig on December 10, 2014, 01:28:09 PM I've taken a break from working on this, but I have a question to ask on behalf of the Namecoin devs. Would Armory devs merge Namecoin specific transaction types into Armory if someone were to code it? This could be used in conjunction with an Armory plugin to automatically renew domain names to prevent expiration. I don't think we have any plans to commit non Bitcoin code to our official repo. However, I can make key methods virtual (or create a fully virtual parent)s so someone implementing NMC or other coins can derive our transaction and block classes for their own purposes easily. Title: Re: General questions arising when adding Namecoin support to Armory Post by: renee25 on March 29, 2015, 09:11:31 PM Looks like armory want to offer signature verification with a new field on the DNS records, i wonder if there's any way to make this compatible with NMC.
http://www.reddit.com/r/Bitcoin/comments/30m1p6/verisign_partners_with_armory_to_propose_a_new/ Title: Re: General questions arising when adding Namecoin support to Armory Post by: josephbisch on March 29, 2015, 09:21:28 PM Wow, this thread is a blast from the past!
It looks like the IETF spec currently has ACH, Bitcoin, and testnet Bitcoin. But it is general, so that Namecoin could be added in the future. |