John Tobey (OP)
|
|
July 24, 2013, 04:24:50 AM |
|
Is there any more information you need? Is there someway I can tell abe to connect to bitcoind with RPC over a specific port?
Sorry for the delayed reply. Please post the last few lines of output before "catch_up_rpc: abort" or "Exception: RPC load failed." Have you run Bitcoin with -reindex since adding txindex to bitcoin.conf? RPC is slower for the initial load, I think over twice as slow as blockfiles. Once caught up, it usually inserts blocks faster than the blockfile method, since RPC has already loaded most transactions by the time the block comes (assuming that you arrange for it to catch up frequently). Abe normally does not "read the blockfiles over and over again": it stores the current position in the database.
|
|
|
|
katriel
Newbie
Offline
Activity: 3
Merit: 0
|
|
July 24, 2013, 03:59:33 PM |
|
Abe is really terrific! I used it to create a transaction network and play around with that.
However, on my machine I left it running (writing to a SQLite database) for I think about two weeks and only got about 18 months into the blockchain (2.4 million transactions). The readme says it typically takes several days to get through the chain--is there something I'm probably doing wrong?
|
|
|
|
John Tobey (OP)
|
|
July 24, 2013, 04:21:50 PM |
|
Abe is really terrific! I used it to create a transaction network and play around with that.
Good to hear! However, on my machine I left it running (writing to a SQLite database) for I think about two weeks and only got about 18 months into the blockchain (2.4 million transactions). The readme says it typically takes several days to get through the chain--is there something I'm probably doing wrong?
I don't use SQLite for the full block chain, so I don't really know. One user, K1773R, reported a fast load with the database file in tmpfs (RAM). ( https://bitcointalk.org/index.php?topic=22785.msg2639709#msg2639709) I think K1773R uses SQLite. If your system has enough memory, you could try that. Abe takes a lot of space, though. I estimate 4-5 times the size of all block files combined. (~/.bitcoin/blocks/blk00???.dat in Unix) You can reduce this somewhat by setting keep-scriptsig=false or by using the no-statistics branch and default-trim-depth. See the comments in abe.conf for what those options entail. K1773R mentioned possibly distributing the SQLite file via torrent: https://github.com/jtobey/bitcoin-abe/issues/15
|
|
|
|
johntawaki
Member
Offline
Activity: 65
Merit: 10
|
|
July 24, 2013, 05:07:04 PM |
|
nice one, but I heard it's not working for some new coins, such as Diamond (DMD).
|
|
|
|
John Tobey (OP)
|
|
July 24, 2013, 05:43:46 PM |
|
nice one, but I heard it's not working for some new coins, such as Diamond (DMD).
Yes, several new coins trace their ancestry to PPCoin and need the "ppcoin" branch in Git. See: https://github.com/jtobey/bitcoin-abe/issues/19Another one, CopperLark, has its own format, which the "copperlark" branch handles. Ideally, Abe would automatically detect among known variants and handle them without code changes.
|
|
|
|
internetbitcoins
Newbie
Offline
Activity: 12
Merit: 0
|
|
August 13, 2013, 11:13:58 PM |
|
Hi, I run python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve but 2 or 3 times I had to restart the server, so now in the block table, I check on block_id and there are some missing blocks. Is there a way to recover them without restart all the process of reading all the blocks on the blockchain?? For example, any way to launch the python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve only for missing ids? Thanks in advances and for Abe!
|
|
|
|
John Tobey (OP)
|
|
August 14, 2013, 01:47:19 PM |
|
I run python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve but 2 or 3 times I had to restart the server, so now in the block table, I check on block_id and there are some missing blocks. block_id is not the same as block number (a.k.a. block_height). The gaps in block_id are not a problem.
|
|
|
|
unick
|
|
August 22, 2013, 11:40:07 PM |
|
Hey John Tobey, I am trying to set up the explorer for the DMD coin, I've read the issue about PPcoin branches, could you give any help on setting this up for DMD?
thanks
|
|
|
|
John Tobey (OP)
|
|
August 23, 2013, 12:54:23 AM |
|
Hey John Tobey, I am trying to set up the explorer for the DMD coin, I've read the issue about PPcoin branches, could you give any help on setting this up for DMD?
thanks
Sure, just post how far you got and any detailed error message. If the transactions have a new format, we may need to find how the coin's source code serializes CTransaction. Bitcoin example: src/core.h line 194.
|
|
|
|
unick
|
|
August 23, 2013, 03:23:01 AM |
|
Hey John Tobey, I am trying to set up the explorer for the DMD coin, I've read the issue about PPcoin branches, could you give any help on setting this up for DMD?
thanks
Sure, just post how far you got and any detailed error message. If the transactions have a new format, we may need to find how the coin's source code serializes CTransaction. Bitcoin example: src/core.h line 194. This is the error message I get when trying to read the block chain root@blockchains:/var/www/blockchain/bitcoin-abe/Abe# python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve failed to load /root/.Diamond/bitcoin.conf: [Errno 2] No such file or directory: u'/root/.Diamond/bitcoin.conf' catch_up_rpc: abort Opened /root/.Diamond/blk0001.dat Exception at 72056513853653130 Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('9'), 'loader': None, 'dirname': u'/root/.Diamond', 'id': 24L} Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2637, in catch_up store.catch_up_dir(dircfg) File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2895, in catch_up_dir store.import_blkdat(dircfg, ds, blkfile['name']) File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 3022, in import_blkdat b = store.parse_block(ds, chain_id, magic, length) File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 3053, in parse_block d['transactions'].append(deserialize.parse_Transaction(ds)) File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 88, in parse_Transaction d['txIn'].append(parse_TxIn(vds)) File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 46, in parse_TxIn d['sequence'] = vds.read_uint32() File "/usr/local/lib/python2.7/dist-packages/Abe/BCDataStream.py", line 71, in read_uint32 def read_uint32 (self): return self._read_num('<I') File "/usr/local/lib/python2.7/dist-packages/Abe/BCDataStream.py", line 110, in _read_num (i,) = struct.unpack_from(format, self.input, self.read_cursor) OverflowError: Python int too large to convert to C long
I am not sure if this is what you are looking for https://github.com/diamondcoinproject/diamondcoin/blob/dd388c28bf89abeaaad761982bb4d04f18995c27/src/serialize.h#L59
|
|
|
|
John Tobey (OP)
|
|
August 23, 2013, 09:22:56 PM |
|
Close enough, it's here, and the new field is strTxComment: https://github.com/diamondcoinproject/diamondcoin/blob/dd388c28bf89abeaaad761982bb4d04f18995c27/src/main.h#L452-462I've downloaded some of the DMD chain and got Abe to load it, but only over RPC, since this coin uses strange block hashing rules. Loading from the blockfile resulted in only Block 0 being displayed, since Abe could not link any block to its parent. Also, I had to disable verification and make Abe trust the RPC server's block hashes. (If you have info about the differences between DMD and Bitcoin block hashing, I may be able to fix this.) Please pull branch "diamond", copy Diamond.conf as bitcoin.conf, add "rpcport=17772" to that file if not already present, and use a config such as this: datadir [{"dirname":"/root/.Diamond", "chain":"Diamond", "code3":"DMD", "address_version":"\u005a"}] default-loader rpc
|
|
|
|
tacoman71
|
|
August 23, 2013, 10:48:59 PM |
|
You should make a newbie friendly .exe version of this that they can just double click to install.
|
Feeling generous? Like my post? Leave a tip at BTC: 1NZJ8cceqEiKDZGAJged2vNGCyfFMUEYPt
|
|
|
John Tobey (OP)
|
|
August 23, 2013, 11:07:22 PM |
|
You should make a newbie friendly .exe version of this that they can just double click to install.
I have too little time. I would happily show another developer around, should one wish to package it nicely.
|
|
|
|
b!z
Legendary
Offline
Activity: 1582
Merit: 1010
|
|
August 24, 2013, 08:01:42 AM |
|
Very cool. This looks like a good way to view the blockchain without relying on an external site.
|
|
|
|
unick
|
|
August 26, 2013, 07:46:22 PM |
|
Close enough, it's here, and the new field is strTxComment: https://github.com/diamondcoinproject/diamondcoin/blob/dd388c28bf89abeaaad761982bb4d04f18995c27/src/main.h#L452-462I've downloaded some of the DMD chain and got Abe to load it, but only over RPC, since this coin uses strange block hashing rules. Loading from the blockfile resulted in only Block 0 being displayed, since Abe could not link any block to its parent. Also, I had to disable verification and make Abe trust the RPC server's block hashes. (If you have info about the differences between DMD and Bitcoin block hashing, I may be able to fix this.) Please pull branch "diamond", copy Diamond.conf as bitcoin.conf, add "rpcport=17772" to that file if not already present, and use a config such as this: datadir [{"dirname":"/root/.Diamond", "chain":"Diamond", "code3":"DMD", "address_version":"\u005a"}] default-loader rpc
I am having the following error RPC data not understood: block hash mismatch Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('9'), 'loader': None, 'dirname': u'/root/.Diamond', 'id': 36L} Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2635, in catch_up raise Exception("RPC load failed") Exception: RPC load failed Abe initialized.
I have pulled the diamond branch from the repository I have cp Diamond.conf. to bitcoin.conf (making sure the rpc port was on 17772) change abe-my.conf accordingly while the server listens... I can't seem to have the RPC loaded... did I miss something ?
|
|
|
|
|
unick
|
|
August 27, 2013, 06:58:25 AM |
|
yep, Git tells me I'm up to date and I have manually checked for those changes and everything is there.
any thing else?
|
|
|
|
John Tobey (OP)
|
|
August 27, 2013, 03:19:31 PM |
|
yep, Git tells me I'm up to date and I have manually checked for those changes and everything is there.
any thing else?
Perhaps you have an older version installed on your system? Otherwise, I don't see how Abe could produce that message. What does this show: python -c 'import Abe.abe; print Abe.abe;'
|
|
|
|
unick
|
|
August 27, 2013, 03:48:42 PM |
|
yep, Git tells me I'm up to date and I have manually checked for those changes and everything is there.
any thing else?
Perhaps you have an older version installed on your system? Otherwise, I don't see how Abe could produce that message. What does this show: python -c 'import Abe.abe; print Abe.abe;' This is the output <module 'Abe.abe' from '/usr/local/lib/python2.7/dist-packages/Abe/abe.pyc'>
|
|
|
|
John Tobey (OP)
|
|
August 27, 2013, 04:12:26 PM |
|
This is the output <module 'Abe.abe' from '/usr/local/lib/python2.7/dist-packages/Abe/abe.pyc'> This means you are getting the previously installed version. You want the one from from 'Abe/abe.py'. Either install the new version (as root, python setup.py install) or set PYTHONPATH to the git clone directory (bitcoin-abe, not the Abe subdirectory). Normally, sys.path starts with an empty string, so if you "cd" to the correct directory, Python should find the local modules.
|
|
|
|
|