[help request] Setting up ABE and Electrum for Terracoin

[crazy_rabbit]

So I'm trying to setup ABE for Terracoin with the final goal of setting up an Electrum Service for Terracoin, but I'm a bit stuck on the ABE setup.

I think I've gotten everything setup for ABE but I keep getting the following message when I try to run abe so it can scan the blockfile:

:~/bitcoin-abe$ python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve
Chain not found for magic number 'B\xba\xbeV' in block file /home/ubuntu/.terracoin/blocks/blk00000.dat at offset 0.  If file contents have changed, consider forcing a rescan: UPDATE datadir SET blkfile_number=1, blkfile_offset=0 WHERE dirname='/home/ubuntu/.terracoin/'

I really don't understand what the "Magic Number" part is talking about, but I tried forcing a rescan and that doesn't work. MySQL says there is an error in my syntax near the "dirname='......' part.

So I'm stumped at the moment. It seems to otherwise work, but this part is tricky.

[celkaris]

in network applications, a "magic byte" (or in our case, "magic bytes") is the first byte contained in a packet, part of a custom binary protocol, so application may immediately identify its own protocol by just inspecting this magic byte.

Terracoin "live net" uses the following :
( https://github.com/terracoin/terracoin/blob/master/src/main.cpp#L2709 )
unsigned char pchMessageStart[4] = { 0x42, 0xba, 0xbe, 0x56 };

[crazy_rabbit]

in network applications, a "magic byte" (or in our case, "magic bytes") is the first byte contained in a packet, part of a custom binary protocol, so application may immediately identify its own protocol by just inspecting this magic byte.

Terracoin "live net" uses the following :
( https://github.com/terracoin/terracoin/blob/master/src/main.cpp#L2709 )
unsigned char pchMessageStart[4] = { 0x42, 0xba, 0xbe, 0x56 };

Thanks! I didn't know that. A starting point in trying to get this working. :-)

[celkaris]

good luck, i earlier wanted to setup a home-made blockchain parser, with http/web rendering, but finally gave up when http://cryptocoinexplorer.com:3750/ went live

[dreamwatcher]

I can help....

You will need to edit the Datastore.py file both for simplicity (No need to have it checking for all the coins it is preset for) and I have found the IP address transaction REGEX structure needs to be changed (For both PPC and TRC)

In the Datastore.py file change:

Code:

CHAIN_CONFIG = [
    {"chain":"Terracoin",
     "code3":"TRC", "address_version":"\x00", "magic":"\x42\xba\xbe\x56"},
    ]

Just delete the other coin definitions.

The Regex for IP address transactions needs to be changed:

Code:

# Regex to match a pubkey ("IP address transaction") in txout_scriptPubKey.
SCRIPT_PUBKEY_RE = re.compile("\x21(.{33})\xac\\Z", re.DOTALL)

The Cryptocoin Explorer site uses this as its config file:

Code:

dbtype MySQLdb
connect-args {"user":"<your database user name for abe>","db":"<database name>","passwd":"<user password>"}
upgrade
port 3750
host 96.126.122.8
datadir = [{
        "dirname": "<your .terracoin directory (expressed as a full path) >",
        "chain":   "Terracoin",
        "code3":   "TRC",
        "address_version": "\u0000" }]

The host will default to localhost. If you need to have ABE available outside a local network, you will need to host set to the outside facing IP address (Like Cryptocoin explorer is at 96.126.122.8 )

Port defaults to 2750, but I have it set at 3750 on the server so it does not conflict with the PPC explorer.


If you have any more questions/issues I will try to help the best I can.

[crazy_rabbit]

I can help....
If you have any more questions/issues I will try to help the best I can.

Looks like it's working so far....Thanks! I'll let everyone know if I get Electrum Working. :-)

Now I have to wait some time for ABE to index everything.

[crazy_rabbit]

Got to the end of the Terracoin chain and got this error:

Exception at 12175175
Failed to catch up {'blkfile_number': 100000, 'dirname': u'/home/ubuntu/.terracoin/', 'chain_id': Decimal('1'), 'id': 1L, 'blkfile_offset': 12125505}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2389, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2442, in catch_up_dir
    store.import_blkdat(dircfg, ds, filename[0])
  File "Abe/DataStore.py", line 2566, in import_blkdat
    store.import_block(b, chain_ids = chain_ids)
  File "Abe/DataStore.py", line 1642, in import_block
    raise MerkleRootMismatch(b['hash'], tx_hash_array)
MerkleRootMismatch: Block header Merkle root does not match its transactions. block hash=4be7570e8f70eb093640c8468274ba759745a7aa2b7d25ab1e0421b259845014

 
I was running Terracoind at the same time as Abe, would that be a problem? Although at the time it got this exception, it was at: block_tx 33750 42496

And Terracoind was saying there were this many blocks: 33747

hmmm....

[dreamwatcher]

Got to the end of the Terracoin chain and got this error:

Exception at 12175175
Failed to catch up {'blkfile_number': 100000, 'dirname': u'/home/ubuntu/.terracoin/', 'chain_id': Decimal('1'), 'id': 1L, 'blkfile_offset': 12125505}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2389, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2442, in catch_up_dir
    store.import_blkdat(dircfg, ds, filename[0])
  File "Abe/DataStore.py", line 2566, in import_blkdat
    store.import_block(b, chain_ids = chain_ids)
  File "Abe/DataStore.py", line 1642, in import_block
    raise MerkleRootMismatch(b['hash'], tx_hash_array)
MerkleRootMismatch: Block header Merkle root does not match its transactions. block hash=4be7570e8f70eb093640c8468274ba759745a7aa2b7d25ab1e0421b259845014

 
I was running Terracoind at the same time as Abe, would that be a problem? Although at the time it got this exception, it was at: block_tx 33750 42496

And Terracoind was saying there were this many blocks: 33747

hmmm....

Nope, it is a problem with the way TRC and PPC handle transactions. Not really a problem with the coins, but ABE does not understand some of the tweaks that were put into them. The ABE creator even gave this advice at one time, and this is what I do now:

In Datastore.py: Comment out ( With the # like below)

Code:

 # Verify Merkle root.
        #if b['hashMerkleRoot'] != util.merkle(tx_hash_array):
        #raise MerkleRootMismatch(b['hash'], tx_hash_array)

IF somebody wants to spend the time reprogramming the Merkle root structure  that ABE expects, more power to them. But for now, this is a 'dirty' way around it.
If anybody has found a better way around this issue, please tell because I would like a real solution myself.

Also With Terracoin (not PPC), after this fix, there is a second issue with ABE and the way difficulty is calculated. This bug does not effect anything because the issue is with predicting difficulty and corrects itself once the block is ready for hashing. The warning will also not terminate ABE, you just end up with alot of screen spam if you are monitoring it. I have tried to fix this one, but it has fallen down on my list of things to do because the difficulty ABE produces matches what the  daemon says (It differs slightly a few decimal places down, but it is an insignificant amount).

Code:

File "Abe/util.py", line 57, in calculate_target
    return (nBits & 0xffffff) << (8 * ((nBits >> 24) - 3))
ValueError: negative shift count

Anyway, good luck with your project, and if you come up with better ideas about these bugs, please let me know. I am always looking to improve my projects.

[Sunny King]

Nope, it is a problem with the way TRC and PPC handle transactions. Not really a problem with the coins, but ABE does not understand some of the tweaks that were put into them. The ABE creator even gave this advice at one time, and this is what I do now:

In Datastore.py: Comment out ( With the # like below)

Code:

 # Verify Merkle root.
        #if b['hashMerkleRoot'] != util.merkle(tx_hash_array):
        #raise MerkleRootMismatch(b['hash'], tx_hash_array)

Sounds strange. PPC did not change the way bitcoin calculates and verifies the merkle root hash, I don't think terracoin did either. So it's a mystery to me why abe would complain about merkle root mismatch here.

Any further info?

[dreamwatcher]

Nope, it is a problem with the way TRC and PPC handle transactions. Not really a problem with the coins, but ABE does not understand some of the tweaks that were put into them. The ABE creator even gave this advice at one time, and this is what I do now:

In Datastore.py: Comment out ( With the # like below)

Code:

 # Verify Merkle root.
        #if b['hashMerkleRoot'] != util.merkle(tx_hash_array):
        #raise MerkleRootMismatch(b['hash'], tx_hash_array)

Sounds strange. PPC did not change the way bitcoin calculates and verifies the merkle root hash, I don't think terracoin did either. So it's a mystery to me why abe would complain about merkle root mismatch here.

Any further info?

I think it is mostly because Abe does not use Berkleydb (now leveldb in Bitcoin 0.7+) to acquire the block information. Instead it reads the BLK001 ...etc file and deserializes it looking for certain markers (Like the "magic key" to see the start of blocks). This makes it sensitive to any changes in structure, although John does continue to improve it.

For example, in PPC the extra transaction time stamp really threw it for a loop because it was not expecting it and thought it was part of the transaction hash. Software that uses the database libraries would have just skipped over that field because it wasn't looking for that field. However Abe reads everything directly from the file byte for byte looking for markers and assuming some information is always a certain length.

Anyway, John has released a new version and I will try to have a look at it when i have time. However the explorer site is running smoothly..<crossed fingers> and I find myself drawn to other projects.

[roy7]

With an extra year of improvements to ABE, do you know if an Electrum wallet for TRC/PPC is feasible yet?