This seems to work. Note that the database has no function to translate an address into a public key hash. I assume you can do this in PHP. The example uses address 12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S, whose hash is 11b366edfc0a8b66feebae5c2e25a7b6a5d1cf31. The hash appears in two places in the query. SELECT SUM(value) / 100000000 AS balance FROM (
SELECT -txout.txout_value AS value
FROM pubkey
JOIN txout ON txout.pubkey_id=pubkey.pubkey_id
JOIN txin ON txin.txout_id=txout.txout_id
JOIN block_tx ON block_tx.tx_id=txout.tx_id
JOIN block b ON b.block_id=block_tx.block_id
JOIN chain_candidate cc ON cc.block_id=b.block_id
WHERE
pubkey.pubkey_hash = LOWER('11B366EDFC0A8B66FEEBAE5C2E25A7B6A5D1CF31') AND
cc.chain_id = 1 AND
cc.in_longest = 1
UNION ALL
SELECT txout.txout_value AS value
FROM pubkey
JOIN txout ON txout.pubkey_id=pubkey.pubkey_id
JOIN block_tx ON block_tx.tx_id=txout.tx_id
JOIN block b ON b.block_id=block_tx.block_id
JOIN chain_candidate cc ON cc.block_id=b.block_id
WHERE
pubkey.pubkey_hash = LOWER('11B366EDFC0A8B66FEEBAE5C2E25A7B6A5D1CF31') AND
cc.chain_id = 1 AND
cc.in_longest = 1
) a;
|
|
|
|
Is there documentation on the table setup?
How do I query information on a particular address? (total received balance, transaction history,etc)
No, sorry, no docs on the tables. Have you searched this thread for the answer? |
|
|
|
The import completed. SQL results: SELECT block_id FROM block WHERE block_hash LIKE '000003b4ef02c872071895b0308232%';
gave me empty results... Thanks. It looks as if MMMCoin uses a scrypt-like hash function. Abe relies on the block header containing the previous block's Bitcoin hash (double SHA256). Other scrypt-using chains apparently still use SHA256 for the previous-block pointer, or otherwise Abe couldn't link their blocks together. If this is right, I actually prefer MMM's design, although it breaks Abe. To fix it, we will need a Python implementation of MMMCoin's hashing algorithm. Probably one exists, we just have to find and test it, and I'll think about how to support it compatibly. |
|
|
|
I don't understand the part about "and then bring abe up to date". I'm assuming it means import everything now so I'm going ahead with that now.
Yes. Just to be sure the hash does not refer to a block received since Abe last read the data. Not sure if this helps or matters but from reading the forums through Google's translator it seems like changes were made recently. The original client / network (if I read correctly) got hit with a 51% attack. A new client was released a few days ago with fixes for that. I don't know if it affects the block chain or not (beyond my knowledge of block chains).
Possible I guess. I don't know if I'll have time to build it, but thanks. I am curious about Abe's console output. Have you tried loading the currency to a fresh database with no other datadirs configured? I'm curious what Abe prints. |
|
|
|
I had to setup a new box to run alternate currencies because the existing box is getting too heavily loaded.
Thanks for providing this! Importing this new chain to the same MySQL db running on a 3rd box looks like it runs fine from the prompt but when looking at it from the web interface, it shows with 0 blocks. Clicking on block 0 shows a block but no links to the rest...
By "runs fine" do you mean it logs a lot of block_tx records in the new chain? Are you comfortable issuing SQL queries? I'd like to know what is in the database. If you have a block hash that should have been loaded, try finding its block_id. You can do this with the partial hash shown in ~/.bitcoin/debug.log (or MMMCoin equivalent) as "best=...". For example, if you see best=000000000661131faf72 and then bring Abe up to date, try: SELECT block_id FROM block WHERE block_hash LIKE '000000000661131faf72%'; If this gives a block_id, for example 12165, see whether the block_id is attached to the right chain: SELECT cc.*, c.chain_name FROM chain_candidate cc JOIN chain c USING (chain_id) WHERE block_id = 12165;
I may need the first few blocks (or say 10kB) of MMM's blk0001.dat to figure this out. I don't know anything about the new chain, and it may do something unusual that would prevent Abe from loading it. |
|
|
|
Before I went to upgrade, I found that Abe had stopped working. My block file just exceeded 1GB in size a couple hours ago which is causing this error: Failed to catch up {'blkfile_number': 1, 'dirname': 'C:\\Documents and Settings\
\Jon\\Application Data\\Bitcoin', 'chain_id': None, 'id': 1L, 'blkfile_offset':
0}
Traceback (most recent call last):
File "Abe\DataStore.py", line 2169, in catch_up
store.catch_up_dir(dircfg)
File "Abe\DataStore.py", line 2184, in catch_up_dir
ds = open_blkfile()
File "Abe\DataStore.py", line 2180, in open_blkfile
ds.map_file(open(filename, "rb"), 0)
File "Abe\BCDataStream.py", line 27, in map_file
self.input = mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ)
WindowsError: [Error 8] Not enough storage is available to process this command It is because I'm running a 32-bit system. Apparently this issue will eventually affect Linux 32-bit systems as well because of the way mmap works; except that it will occur when the block file hits 2 or 3 GB in size. Is Abe particularly tied to mmap? If it is, it seems like it will soon be (or is now) for 64 bit systems only. Interesting. I don't think it will affect Linux, because bitcoin starts a new file (blk0002.dat) before 2GB: // FAT32 filesize max 4GB, fseek and ftell max 2GB, so we must stay under 2GB
if (ftell(file) < 0x7F000000 - MAX_SIZE)
{
nFileRet = nCurrentBlockFile;
return file;
}
No mention of mmap, but I think a change from 0x7F000000 to 0x3F000000 is worth proposing, or at least making it a compile-time option: diff --git a/src/main.cpp b/src/main.cpp
index 9951952..2d838b7 100644
--- a/src/main.cpp
+++ b/src/main.cpp
@@ -1773,6 +1773,10 @@ FILE* OpenBlockFile(unsigned int nFile, unsigned int nBlockPos, const char* pszM
static unsigned int nCurrentBlockFile = 1;
+#ifndef BITCOIN_MAX_BLKFILE_SIZE
+#define BITCOIN_MAX_BLKFILE_SIZE 0x7F000000
+#endif
+
FILE* AppendBlockFile(unsigned int& nFileRet)
{
nFileRet = 0;
@@ -1784,7 +1788,7 @@ FILE* AppendBlockFile(unsigned int& nFileRet)
if (fseek(file, 0, SEEK_END) != 0)
return NULL;
// FAT32 filesize max 4GB, fseek and ftell max 2GB, so we must stay under 2GB
- if (ftell(file) < 0x7F000000 - MAX_SIZE)
+ if (ftell(file) < BITCOIN_MAX_BLKFILE_SIZE - MAX_SIZE)
{
nFileRet = nCurrentBlockFile;
return file;
Abe inherits the use of mmap() from Bitcointools, so if you can verify the bug exists there, we might get Gavin interested in fixing it. |
|
|
|
Strange. I'll try to debug it if you give me steps to reproduce it. (Steps could include "git clone" and building Litecoin.) |
|
|
|
|
Atheros:
Thanks for the report. I fixed the sort() bug, made the limit configurable and committed the change to github as 945ed46829.
I have only 63938 blocks loaded in my MySQL database, and the longest address history (1XPTgDRhN8...) has 3207 rows, apparently too few for a good test. That address loaded much faster with a low limit than with none. However, MySQL "explain" showed it still having to scan and sort the whole history, so as you observed, it did not solve the problem. Adding an index on block.block_nTime did not help, so I went BlockExplorer's route and now display no history if the limit is exceeded. To show partial history efficiently might require significant table and index overhead.
Let me know if the latest commit does not solve the load problem.
|
|
|
|
@Atheros Good question. This patch is not the most polished piece of code, but you may try it, editing ADDRESS_HISTORY_ROWS_MAX in Abe/abe.py to your liking: diff --git a/Abe/abe.py b/Abe/abe.py
index 1982434..4811e01 100755
--- a/Abe/abe.py
+++ b/Abe/abe.py
@@ -92,6 +92,8 @@ HEIGHT_RE = re.compile('(?:0|[1-9][0-9]*)\\Z')
HASH_PREFIX_RE = re.compile('[0-9a-fA-F]{0,64}\\Z')
HASH_PREFIX_MIN = 6
+ADDRESS_HISTORY_ROWS_MAX = 500 # XXX hardcoded limit
+
NETHASH_HEADER = """\
blockNumber: height of last block in interval + 1
time: block time in seconds since 0h00 1 Jan 1970 UTC
@@ -930,12 +932,13 @@ class Abe:
count[txpoint['is_in']] += 1
txpoints = []
- rows = []
- rows += abe.store.selectall("""
+ max_rows = ADDRESS_HISTORY_ROWS_MAX
+ in_rows = abe.store.selectall("""
SELECT
b.block_nTime,
cc.chain_id,
b.block_height,
+ block_tx.tx_pos,
1,
b.block_hash,
tx.tx_hash,
@@ -949,13 +952,22 @@ class Abe:
JOIN txout prevout ON (txin.txout_id = prevout.txout_id)
JOIN pubkey ON (pubkey.pubkey_id = prevout.pubkey_id)
WHERE pubkey.pubkey_hash = ?
- AND cc.in_longest = 1""",
- (dbhash,))
- rows += abe.store.selectall("""
+ AND cc.in_longest = 1
+ ORDER BY
+ b.block_nTime,
+ cc.chain_id,
+ b.block_height,
+ block_tx.tx_pos,
+ txin.txin_pos
+ LIMIT ?""",
+ (dbhash, max_rows + 1))
+ truncated = in_rows[max_rows] if len(in_rows) > max_rows else None
+ out_rows = abe.store.selectall("""
SELECT
b.block_nTime,
cc.chain_id,
b.block_height,
+ block_tx.tx_pos,
0,
b.block_hash,
tx.tx_hash,
@@ -968,11 +980,41 @@ class Abe:
JOIN txout ON (txout.tx_id = tx.tx_id)
JOIN pubkey ON (pubkey.pubkey_id = txout.pubkey_id)
WHERE pubkey.pubkey_hash = ?
- AND cc.in_longest = 1""",
- (dbhash,))
+ AND cc.in_longest = 1""" + ("""
+ AND (b.block_nTime < ? OR
+ (b.block_nTime = ? AND
+ (cc.chain_id < ? OR
+ (cc.chain_id = ? AND
+ (b.block_height < ? OR
+ (b.block_height = ? AND
+ block_tx.tx_pos <= ?))))))"""
+ if truncated else "") + """
+ ORDER BY
+ b.block_nTime,
+ cc.chain_id,
+ b.block_height,
+ block_tx.tx_pos,
+ txout.txout_pos
+ LIMIT ?""",
+ (dbhash,
+ truncated[0], truncated[0], truncated[1], truncated[1],
+ truncated[2], truncated[2], truncated[3], max_rows + 1)
+ if truncated else
+ (dbhash, max_rows + 1))
+
+ if len(out_rows) > max_rows:
+ if truncated is None or truncated > out_rows[max_rows]:
+ truncated = out_rows[max_rows]
+ if truncated:
+ truncated = truncated[0:3]
+
+ rows = in_rows + out_rows
rows.sort()
for row in rows:
- nTime, chain_id, height, is_in, blk_hash, tx_hash, pos, value = row
+ if truncated and row >= truncated:
+ break
+ (nTime, chain_id, height, tx_pos,
+ is_in, blk_hash, tx_hash, pos, value) = row
txpoint = {
"nTime": int(nTime),
"chain_id": int(chain_id),
@@ -1008,7 +1050,11 @@ class Abe:
body += abe.short_link(page, 'a/' + address[:10])
- body += ['<p>Balance: '] + format_amounts(balance, True)
+ body += ['<p>']
+ if truncated:
+ body += ['<strong>Results truncated</strong></p>']
+ else:
+ body += ['<p>Balance: '] + format_amounts(balance, True)
for chain_id in chain_ids:
balance[chain_id] = 0 # Reset for history traversal.
The maximum applies to sends and receives independently, so for example, an address with 600 receives and 100 spends will show the first 500 receives and whichever of the spends occurred before the 501st receive. I replace the balance with "Results truncated", since it is probably inaccurate, and it may never have been the right balance (may even be negative) due to out-of-order blocks and transactions. Switching to FastCGI should give you a bit better parallelism, too, though FCGI mode may need fixes for Windows. You could try Python's multithreaded socket server with the built-in HTTP but I won't be surprised if you hit bugs. Let us know how it goes. |
|
|
|
So I'm stuck at block 42156 on the testnet for now then? It doesn't sound like a quick upgrade / change to Abe.  I've committed a fix. By default, only Bitcoin and Testnet are allowed to have any block version number. If there are more such chains, uncomment ignore-bit8-chains and add them to the list in abe.conf. |
|
|
|
The output from the SQL query is as follows:
(snipped)
Wow! I guess everybody creates their own block version to go along with the magic number and address version. When you say namecoin like merged mining do you mean any chain that's compatible to be merged with namecoin? In this case, it would be namecoin, ixcoin, i0coin, geistgeld, coiledcoin, devcoin and soon groupcoin (not imported yet).
That list is probably right. In practice, it's whoever started with a copy of Namecoin's merged-mining code and didn't change it too much. Small chains try to be compatible with tools like Abe. Abe tries to be compatible with big chains. That would be values 1, 1048577, 1048833, 196609, 262145, 262401, 257, 131073, 131329, 196609, 196865, 65537, 65793; and then whatever groupcoin reports and any other future chains. Is this correct?
That looks about right. Would commenting out those lines to get the import running again, then un-commenting afterwords be an option or will every block from this point onwards have the problem?
I don't know. It depends whether the Testnet block was a one-time test or that miner solved more blocks. Strange things happen on Testnet. |
|
|
|
|
A less fragile workaround would be to run a separate Abe process for loading Testnet using --no-serve and a copy of the software with the lines commented out. Your server process would not be able to add new testnet blocks at every request the way it does for other chains, and you would have to schedule the Testnet process to run every minute or so. But you might prefer this, since it should survive any new block versions introduced by the chains.
|
|
|
|
So I'm stuck at block 42156 on the testnet for now then? It doesn't sound like a quick upgrade / change to Abe. To be compatible with Testnet, but incompatible with Namecoin and chains that use its proof-of-work format, remove or comment out these lines in Abe/deserialize.py, lines 175-176 in the latest commit: if d['version'] & (1 << 8):
d['auxpow'] = parse_AuxPow(vds)
To be compatible with both, until I have a better fix, I suggest you run this query: SELECT DISTINCT c.chain_name, b.block_version
FROM block b
JOIN chain_candidate cc ON (b.block_id = cc.block_id)
JOIN chain c ON (cc.chain_id = c.chain_id) You will see something like: chain_name | block_version
------------+---------------
NcTestnet | 65537
NcTestnet | 1
Namecoin | 1
Namecoin | 65537
Namecoin | 65793
Bitcoin | 1
NcTestnet | 65793
Make a list of the values under block_version that belongs to a network with Namecoin-style merged mining. For the above output, the list would be (1, 65537, 65793) corresponding to Namecoin and NcTestnet. Then add " and d['version'] in (LIST)" after "d['version'] & (1 <<  " on line 175, e.g.: if (d['version'] & (1 << 8)) and d['version'] in (1, 65537, 65793):
d['auxpow'] = parse_AuxPow(vds)
Let us know how it goes. |
|
|
|
|
The testnet contains a block with an unusual value in the version field. Abe thinks it is a merged-mining block. I mean to provide a workaround, especially since the main BTC chain is vulnerable to it, but haven't got to it yet. Abe will need a new kind of chain metadata with flags for features such as merged mining, rather than rely on a universal interpretation of block version.
|
|
|
|
Edit: Now that I look at it a bit more.. I think it is an orphaned block but also a bug where the block version code isn't being applied to properly show the correct addresses.
Correct you are. |
|
|
|
Everyone already has a copy of the blockchain. All we need is a GUI or CGI to read it.
I would donate to that cause. For me (as a programming novice) it sounds like not-too-complicated to program a blockexplorer/blockchaininfo clone which works with the local blockchain copy? Someone make a bounty thread? Tried Abe? https://bitcointalk.org/index.php?topic=22785.0 |
|
|
|
had to quit app and rerun it and after using ctrl-c i lack one record :
More likely it seems to me, you have one unassigned block_id. This is not the same as a missing block. block_id is an internal identifier, not visible in any HTML or API output. So it doesn't really matter what the block_id is. It will vary from one Abe instance to another for the same block. Abe allocated a block_id but hadn't used it when you stopped it, so it remains unused. This is not a problem. By the way, the visible "block number" is called "height" internally. Often height and block_id are numerically close, depending on how you load the data, but they will diverge if, for example, your block file contains side chains or you load Testnet or alt chain data along with the main BTC block chain. Is here any possibility to recheck for missing data?
It would be nice to have a validation tool that checks for missing data. We don't have that. Part of it is covered by the foreign key constraints, and there is a tool (Abe/verify.py) that verifies the transaction Merkle hashes, but it could do much more. If you haven't done anything like delete table rows, and barring any bugs, if Abe shows a block number, it has all data from the genesis through that block. Another thing - when i have bitcoin running - can i rerun app to check for new blocks, or i have to stop bitcoin?
I have never noticed a problem running both at once. And btw - how is this tool different from Bitcointools ? i am new to this tools, was using crappy json-rpc methods given in bitcoind : p
Abe borrows some low-level code from Bitcointools and adds many features, notably SQL database support, speed improvements through indexing, and the HTML interface similar to Bitcoin Block Explorer. |
|
|
|
|
Simple answer, Abe runs on many databases, all of them transactional. This knowledge permits one to code with a certain set of assumptions that may not hold in a MyISAM environment. Fearing inconsistencies in users' MyISAM tables, I put in the check that you discovered.
I think the changes needed to support MyISAM's semantics safely would be too invasive. To use MyISAM unsafely is easy, just disable the check in Abe.DataStore._test_transaction, but don't expect support when you start to see wrong data.
|
|
|
|
Just out of curiosity why don't you support MyISAM engine? I tried replacing innodb with myisam in the source but then abe complains about inability to create a transactional database. Isn't MyISAM faster than InnoDB? I don't know just read some articles comparing them so I'm asking you.
As I understand, MyISAM is non-transactional. It does not support rollback. See: http://en.wikipedia.org/wiki/ACIDAbe relies on the ability to roll back the current transaction and recover when interrupted or in the event of an error. You may find MyISAM works for a while, but when something goes wrong, typically you will have to rebuild the database. With transactional tables, it normally suffices to restart Abe. |
|
|
|
I've seen somewhere an Abe install with the SolidCoin2 chain but I can't seem to get it going on my install. File "Abe/DataStore.py", line 1493, in import_block
raise MerkleRootMismatch(b['hash'], tx_hash_array)
MerkleRootMismatch: Block header Merkle root does not match its transactions. block hash=725ca2009ba1b949a820d3a48511dc21c7f8d76d06d8afb6abb5988f5a911a8b
I won't be surprised if SC2 has changed the Merkle tree hashing algorithm. If you trust your copy of the block chain, you may get around this by simply deleting these lines from Abe/DataStore.py: # Verify Merkle root.
if b['hashMerkleRoot'] != util.merkle(tx_hash_array):
raise MerkleRootMismatch(b['hash'], tx_hash_array)
A long-term solution would be to add support for different algorithms per chain, or at least an option to skip the verification in a given chain. |
|
|
|
|
Show Posts
