This is the error
Exception happened during processing of request from ('178.63.69.203', 49828)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 704, in finish
self.wfile.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
(I think nginx (load-balancing here) or the client times out because it's taking too long) No Abe stack frames here, so I don't have much to go by. on AuroraCoind debug.log I see this:
22478 ThreadRPCServer method=getrawtransaction
22479 ThreadRPCServer method=getrawtransaction
22480 ThreadRPCServer method=getrawtransaction
22481 ThreadRPCServer method=getrawtransaction
22482 ThreadRPCServer method=getrawtransaction
22483 ThreadRPCServer method=getrawtransaction
22484 ThreadRPCServer method=getrawtransaction
22485 ThreadRPCServer method=getrawtransaction
22486 ThreadRPCServer method=getrawtransaction
22487 ThreadRPCServer method=getrawtransaction
22488 ThreadRPCServer method=getrawtransaction
22489 ThreadRPCServer method=getrawtransaction
22490 ThreadRPCServer method=getrawtransaction
22491 ThreadRPCServer method=getrawtransaction
22492 ThreadRPCServer method=getrawtransaction
22493 ThreadRPCServer method=getrawtransaction
22494 ThreadRPCServer method=getrawtransaction
22495 ThreadRPCServer method=getrawtransaction
22496 ThreadRPCServer method=getrawtransaction
22497 ThreadRPCServer method=getrawtransaction
22498 ThreadRPCServer method=getrawtransaction
22499 ThreadRPCServer method=getrawtransaction
22500 ThreadRPCServer method=getrawtransaction
22501 ThreadRPCServer method=getrawtransaction
22502 ThreadRPCServer method=getrawtransaction
22503 ThreadRPCServer method=getrawtransaction
22504 ThreadRPCServer method=getrawtransaction
about 500 per secondspsql> select * from pg_stat_activity; reports only one IDLE connection Perhaps the HTTP request is triggering a catch-up which takes too long. Have you tried separating the loader from the server? One process runs Abe in an infinite loop passing --no-serve, and the web process uses --no-load (or datadir=[]). Then the web requests will not wait for data to load. |
|
|
|
What is, in your experience, the fastest db to use with Abe?
SQLite with connect-args=":memory:" or (according to K1773R) something entirely in RAM (tmpfs). Honestly, Abe could be much better optimized for both speed and space. Data could be further denormalized. Unindexed data such as scripts, and even the non-initial bytes of hashes, could be read from the blockfile or RPC. If you want speed, I suggest you compare (the now open source) BlockExplorer or other such projects. |
|
|
|
I have a question about mempool transactions regarding performance: So I'm running http://blockexplorer.auroracoin.eu and because I have allocated quite a machine to the task everything has been zappy and fine this morning. However when I checked back from work an hour later I saw loads of exceptions saying "error: [Errno 32] Broken pipe", the nginx I have in front reporting gateway timeouts. I'm hypothesizing the db queries are the bottleneck. I tried rebuilding the database (drop it completely and rebuild)... that didn't help, it started again right away. There are loads of mempool transactions in AURoracoin because we're being pool-hopping-attacked to the point where there hasn't been a block for 6 hours or so. Another instance I run with same setup, but on quite weak machine (vm) was able to cope with the load quite well and didn't suffer broken pipes. What fixed it was to re-initialize BOTH the blockchain of the AuroraCoind AND the db (just db didn't help). Now the question: how are mempool transactions handled and could the existance of many mempool transactions have considerable impact on db (or abe.py) performance? I'm a bit confused... does anyone have an idea what could've been causing this? It could be transactions being left open, suboptimal SQL, or  If you have a collection of Python stack traces or database process lists from during the timeouts, they might point to the offender(s). |
|
|
|
This is how I setup the class... class HiroCoin(LtcScryptChain):
def __init__(chain, **kwargs):
chain.name = 'HiroCoin'
chain.code3 = 'HIC'
chain.address_version = "\x28"
chain.magic = "\xfe\xc3\xb9\xde"
Chain.__init__(chain, **kwargs)
datadir_conf_file_name = "hirocoin.conf"
datadir_rpcport = 9347
EDIT: This may or may not help, but all the block past block 0 are all in orphan_block table. Checking the code concludes this is because it cannot see the previous block hash correctly. Please see: https://bitcointalk.org/index.php?topic=22785.msg5785774#msg5785774 (and note that "policy":"NovaCoin" has an effect similar to the proposed "hashPrev":"scrypt"). If the algorithm is neither Bitcoin's nor NovaCoin's, we have to override block_header_hash in the new class: class Sha256Chain(Chain):
def block_header_hash(chain, header):
return util.double_sha256(header) class LtcScryptChain(Chain):
def block_header_hash(chain, header):
import ltc_scrypt
return ltc_scrypt.getPoWHash(header) |
|
|
|
I haven't kept up with the development of Abe, but is P2SH/Multisig supported now?
Apart from an issue affecting Namecoin and the question whether to include "escrow" outputs in address balances, yes, on the multisig branch, thanks to Jouke's sponsorship. |
|
|
|
With current master version of ABE, If I try to load with rpc, it will work but the frontend crashes as int(ss) is of the NoneType.
My knee-jerk fix for this type of bug is to replace " int(ss)" with " None if ss is None else int(ss)". |
|
|
|
Thanks John, will give this a shot. Actually trying this on HiroCoin which is a fork of litecoin. EDIT : Am I being dumb or have I missed something ? ImportError: No module named ltc_scrypt Install ltc_scrypt as per README.txt: Here is where I think the issue is (block_id 2) : *************************** 1. row ***************************
block_id: 1
block_hash: ae94c51e61c87f32a2a6c755df00b1ffa130c73b1bd165aaccaa3f90f8d243a9
block_version: 1
block_hashMerkleRoot: b0019d92bc054f7418960c91e252e7d24c77719c7a30128c5f6a827c73095d2a
block_nTime: 1394723131
block_nBits: 504365040
block_nNonce: 1234746574
block_height: 0
prev_block_id: NULL
search_block_id: NULL
block_chain_work: 0000000000000000000000000000000000000000000000000000000000000000000000100010
block_value_in: 0
block_value_out: 40000000000
block_total_satoshis: 40000000000
block_total_seconds: 0
block_satoshi_seconds: 0
block_total_ss: 0
block_num_tx: 1
block_ss_destroyed: 0
*************************** 2. row ***************************
block_id: 2
block_hash: 54a642a1aabb71a5b9d57a6455ba4ab7f69641e2ebe24ca18dfdc4bd9d98e198
block_version: 2
block_hashMerkleRoot: cf42adf0dbda83114f23f3c771700b1daf107aaf57f935f1d6bfcc568f38aa4f
block_nTime: 1394867933
block_nBits: 504365040
block_nNonce: 264378
block_height: NULL
prev_block_id: NULL
search_block_id: NULL
block_chain_work: NULL
block_value_in: 0
block_value_out: 40000000000
block_total_satoshis: -1
block_total_seconds: NULL
block_satoshi_seconds: NULL
block_total_ss: NULL
block_num_tx: 1
block_ss_destroyed: NULL
If you mean all those NULLs, they result from the incorrect block hash algorithm. Abe does not see block_id 1 as the parent of block_id 2, so it can not compute chain statistics or even height (block number). See if ltc_scrypt helps. |
|
|
|
Shows only the first block. The thing I do not understand.
When Abe loads many blocks but shows only the genesis block, it means that Abe is using the wrong algorithm to calculate the block header hash as it appears in the next block's hashPrev field. By default, Abe uses double-SHA256, the same as Bitcoin. Many coins use scrypt. I would like to make this selectable by datadir=[{...,"hashPrev":"scrypt"}], but for now it requires a few lines of code. Look for "NovaCoin" in the sources for an example. Hi John, Could you elaborate abit further on this? much appreciated. What few lines of code are needed? Come to think of it, if the format is exactly NovaCoin's, you might not need a code change. In datadir, add "policy":"NovaCoin" along with "chain" etc. The changes are mostly in Abe/Chain.py: add a class defined like NovaCoin (or CryptoCash) and a corresponding conditional in the create function. In Abe/DataStore.py, where it says {"chain":"NovaCoin"},
{"chain":"CryptoCash"},add a similar line. Edit: and if it works, post your changes, or better yet, create a GitHub pull request so I can add them. |
|
|
|
Ok tried what you said John, but I reposted that quote since I get the same message as previously when I tryed to comment out the MerkleRoot check.
It goes pass the 370th byte but fails at byte 1677721600398062
"Exception at <impossibly huge file offset>" means an unexpected block format. Where Abe expects a block size or script length, there is some other data. Abe interprets it as a huge number, uses it as the blockfile pointer, and reports the error as coming from there. See: https://github.com/bitcoin-abe/bitcoin-abe/issues/4. Try either the ppcoin patch or RPC mode. |
|
|
|
Is there a reason why my questions don't get answered ?
Two possible reasons: bad timing (I am busy) and no obvious answer. You have several questions, and it is not clear what you are currently using in terms of configuration and modifications to the code. You mentioned commenting out the Merkle root check, but your latest error shows it still occurring. Are you able to load blocks without the check? I have converted your hex dump back to binary, commented out the Merkle root check, installed the ltc_scrypt module, and loaded the block with this command: python -m Abe.abe --dbtype sqlite3 --connect-args :memory: --port 2750 --datadir '[{"dirname":".", "chain":"GrowthCoin", "policy":"NovaCoin"}]' |
|
|
|
is it possible to get address balance from command line query? I get the following error python -m Abe.abe --query /chain/FairCoin/q/addressbalance/fc44xJawRecE9xrZwGHMUtAfxWeS3CJqtP
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 2374, in <module>
sys.exit(main(sys.argv[1:]))
File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 2370, in main
serve(store)
File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 2236, in serve
}, start_response)
File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 276, in __call__
handler(page)
File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 415, in handle_chain
chain = abe.chain_lookup_by_name(symbol)
File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 397, in chain_lookup_by_name
ret = abe.store.get_chain_by_name(symbol)
File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 819, in get_chain_by_name
return store.chains_by.name.get(name, None)
AttributeError: 'DataStore' object has no attribute 'chains_by'
maybe I have the syntax wrong I'm sorry for the cryptic message. The problem is that you have to give --config (or --dbtype and --connect-args) along with --query if the query involves the database, as this one does. |
|
|
|
is it safe to assume that when using the RPC loader rather than the blkfile loader that it doesn't matter the algo of the coin since bitcoin-abe is using rpc call to get data ?
I guess so. I'm trying to setup fuguecoin using RPC here's my config datadir += [{"dirname": "/home/azureuser/.fuguecoin",
"loader": "rpc",
"chain": "FugueCoin",
"code3": "FC",
"conf":"fuguecoin.conf",
"address_version" : "\u0023",
"magic" : "\u00f9\u00be\u00b4\u00d4"
}]
i has getting "RPC data not understood: block hash mismatch" so i figured out the genesis transaction and block 0 hash I tried this import-tx += 010000006eb7dc52010000000000000000000000000000000000000000000000000000000000000 000ffffffff7604ffff001d020f274c6c4a616e2032302c20323031342031323a3430616d204544 542e204e65776567672074656173657220737567676573747320746865205553206f6e6c696e652 072657461696c206769616e74206d617920626567696e20616363657074696e6720626974636f69 6e20736f6f6effffffff0100000000000000000000000000 then got store = DataStore.new(args)
File "Abe/DataStore.py", line 216, in __init__
store.maybe_import_binary_tx(chain_name, str(hex_tx).decode('hex'))
UnboundLocalError: local variable 'chain_name' referenced before assignmentso i changed my config file to this: import-tx += {"chain":"FugueCoin" ,"tx":"010000006eb7dc52010000000000000000000000000000000000000000000000000000000000000 000ffffffff7604ffff001d020f274c6c4a616e2032302c20323031342031323a3430616d204544 542e204e65776567672074656173657220737567676573747320746865205553206f6e6c696e652 072657461696c206769616e74206d617920626567696e20616363657074696e6720626974636f69 6e20736f6f6effffffff0100000000000000000000000000"} no i'm getting this File "Abe/DataStore.py", line 2323, in maybe_import_binary_tx
chain = store.chains_by.name(chain_name)
TypeError: 'dict' object is not callablechanged that line from chain = store.chains_by.name(chain_name) to chain = store.chains_by.name.get(chain_name) and getting File "Abe/BCDataStream.py", line 110, in _read_num
(i,) = struct.unpack_from(format, self.input, self.read_cursor)
struct.error: unpack_from requires a buffer of at least 4 bytes  Thanks, I've fixed the last two errors. You will still get block hash mismatch unless you disable the check or add support for your coin. |
|
|
|
Shows only the first block. The thing I do not understand.
When Abe loads many blocks but shows only the genesis block, it means that Abe is using the wrong algorithm to calculate the block header hash as it appears in the next block's hashPrev field. By default, Abe uses double-SHA256, the same as Bitcoin. Many coins use scrypt. I would like to make this selectable by datadir=[{...,"hashPrev":"scrypt"}], but for now it requires a few lines of code. Look for "NovaCoin" in the sources for an example. is there a way to get a raw block before abe splits it and puts it in the database, maybe there are new fields added in this coin, or the data got moved around, I can then change the abe code to make it work See how ds_parse_block and ds_parse_block_header are overridden in Abe/Chain.py. However, note that this module is new and under heavy development. |
|
|
|
Shows only the first block. The thing I do not understand.
When Abe loads many blocks but shows only the genesis block, it means that Abe is using the wrong algorithm to calculate the block header hash as it appears in the next block's hashPrev field. By default, Abe uses double-SHA256, the same as Bitcoin. Many coins use scrypt. I would like to make this selectable by datadir=[{...,"hashPrev":"scrypt"}], but for now it requires a few lines of code. Look for "NovaCoin" in the sources for an example. |
|
|
|
Enabled log-sql: EXEC:
INSERT INTO tx (tx_id, tx_hash, tx_version, tx_lockTime, tx_size)
VALUES (%s, %s, %s, %s, %s) (88379L, 'c333a53f0174166236e341af9cad795d21578fb87ad7a1b6d2cf8aa9c722083c', 1, 0, 998171)
EXEC: INSERT INTO txout_seq () VALUES () ()
EXEC: SELECT LAST_INSERT_ID() ()
FETCH: (115923L,)
EXEC:
INSERT INTO txout (
txout_id, tx_id, txout_pos, txout_value,
txout_scriptPubKey, pubkey_id
) VALUES (%s, %s, %s, %s, %s, %s) (115923L, 88379L, 0, 0, 'fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff ffff
fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff fffff [...cut... many, many, MANY, 'f']
This is the query that screws everything up Edit: WELL, found the problem: http://blockexplorer.com/testnet/tx/c333a53f0174166236e341af9cad795d21578fb87ad7a1b6d2cf8aa9c722083ca crazy block in testnet blockchain. Now I have to figure out how to skip this block. I don't know when I'll have time to fix this, but I'm glad we see it on testnet before main. Thanks. |
|
|
|
psycopg2.DataError: value too long for type character(3) Either shorten your chain's "code3" to 3 characters or widen the database column: ALTER TABLE chain ALTER COLUMN chain_code3 TYPE VARCHAR(4); |
|
|
|
ok I found where NovaCoin took it's config from in Chain.py... and like I said they both have the same magic number so I hardcoded the FairCoin names into that file, witch I find is the unelegant way to go but it works for now... If you have a better solution, I would love to know  edit: well after putting more thoughts to it... I just added a conditional statement for the FairCoin name, it's a bit more clean this way... still if there is a better solution (as an update to the code will wipe this setting) let me know (or maybe update the github with this conditional statement) I'd put the FairCoin parameters in abe.conf datadir=[{...}] and not in Python. That should survive upgrades. See the datadir examples in abe.conf. Given a simple directory name (datadir="...") Abe relies on the magic number. We could support two chains with the same magic number, but we would need some way to distinguish them. That's why I introduced the expanded syntax for datadir, a JSON array of objects. |
|
|
|
Hi, I'm trying to setup GrowthCoin block explorer. This is what I get root@grw:~/bitcoin-abe# python -m Abe.abe --config /home/abe/abe-mysql.conf --commit-bytes 100000 --no-serve
Opened /root/.GrowthCoin/blk0001.dat
Chain not found for magic number a5efdbfd in block file /root/.GrowthCoin/blk0001.dat at offset 0. This coin is new to Abe. Please follow the example "datadir" syntax in abe.conf, and add datadir = [{....}] to your abe-mysql.conf file. If the currency is scrypt-based, it may need more changes discussed recently in this thread. |
|
|
|
|
Show Posts
