Please run the script in this post: https://bitcointalk.org/index.php?topic=22785.msg707302#msg707302

I've committed a small change that I hope will stop these MySQL duplicate key errors.

I know of one, possibly two bugs affecting this situation.  First, Abe sometimes crashes by exceeding Python's recursion limit when rescanning.  This does not produce wrong data, and I work around it by deleting rows from chain_candidate, updating chain.chain_last_block_id, and retrying.

I have noticed slight deviations in statistical values like Coin Days Destroyed from one Abe instance to another.  (The values on the homepage depend on server system time, but on a given block's page they should be identical.)  I suspect rescanning may create statistical discrepancies.  The ones I've seen were slight (under 1%).

I do not know of any bugs affecting address balances, transaction links, etc.  But I can offer no guarantee.

eurekafag:

Yes, you are right about not needing sqlite2.

Please run "sqlite3 abe.sqlite" from the command line and issue
The output should be "str" indicating that the workaround for this SQLite limitation is in effect.  (A result of integer arithmetic is too big for 64 bits.  The workaround forces the use of floating point.)

If int_type is not "str", please update via git where I've added support for --int-type=str.  Delete abe.sqlite and abe.sqlite-journal and retry with int-type=str in your config.

This error takes more thought.  I got this error with SQLite and thought I'd worked around it in the latest code.  Please list your versions of Abe, pysqlite, and SQLite.  I could not produce the error using Abe latest, pysqlite-2.6.0, and libsqlite3 3.7.4-2ubuntu5.  I tried this command, and it succeeds past block_tx 2261 2291:
Note that SQLite will slow down horribly if you make it to block 70000+.  Another database engine such as PostgreSQL or MySQL will not have the same error, which is specific to SQLite.

I think this will do it.  Note that the question is a little ambiguous, since a transaction can have more than one sending address.  I'll count every transaction where the given address is among the senders.  This will unfortunately double-count transactions where it appears in two inputs.  But at least it should illustrate the table relationships.

Note the hashes of the two addresses:
sender = 12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S = 11B366EDFC0A8B66FEEBAE5C2E25A7B6A5D1CF31
receiver = 1Q2TWHE3GMdB6BZKafqwxXtWAWgFt5Jvm3 = FC916F213A3D7F1369313D5FA30F6168F9446A2D

Note that Abe stores hashes as lowercase hex strings when it does not figure out how to use a binary type.  Only SQLite has a binary type that Abe can use, so for SQLite instead of LOWER('hex string') you would put X'hex string', IIRC.  (To force use of hex, pass --binary-type=hex when first creating the tables.)

someguy123,

I'm afraid I have little time to devote to new features--or much of anything beyond explaining how things work.  Patches are welcome.  I suggest starting a thread about the design of such a system--or integration into existing Python template systems, about which I know nothing.

-John

txout.pubkey_id receives coins in transaction txout.tx_id.

txin.txout_id points to txout.pubkey_id that sends coins in txin.tx_id.

Is this clear?

Have you tried scanning with one process and serving with another?


Not as elegant as an option to do it all in one, I'll grant.

Sorry, I think I confused you by linking to the algorithm going from hash to address.

To go from address to hash, you would reverse it: decode base58, discard last 4 bytes (checksum), discard first byte (version), and hex-encode the result.

If you want the date/time a transaction appeared in the chain, it is block.block_nTime (seconds since 0:00 1 Jan 1970 UTC).  You find the block row by joining tx to block_tx to block to chain_candidate, restricting in_longest=1 and chain_id=1 (for BTC).  Note that date is not a property of transactions.  (tx has a lock_time but it is unused.)  A transaction doesn't say when it was signed, and it doesn't know when it will become part of the chain.  The joins could produce different results at different times because of reorganizations, which change the value of chain_candidate.in_longest.

abe.py: decode_check_address or decode_address.  They return two values: address_version and pubkey_hash.  The "version" should be "\0" for BTC.

Also there is base58.py: bc_address_to_hash_160 returning only the hash.

Here's the algorithm in pseudocode: https://en.bitcoin.it/wiki/Protocol_specification#Addresses

andrehorta,

I appreciate your goal!

To extend Abe with new page types and HTTP-API functions, I suggest you follow the model of ecdsa.org.  See contrib/ecdsa.py for sample code.  It subclasses Abe and defines methods named with a "handle_" prefix.  Abe calls handle_* when it gets a request for /*/... .  Unfortunately, the structure of the "page" argument is not currently documented and may change, so you will have to figure it out from the existing handlers in abe.py and ecdsa.py.  Likewise, Abe maps /q/* and /chain/.../q/* to the q_* method with "page" and "chain" arguments; chain is None when /chain/ does not appear in the URL.

Pasted from doc/FAQ.html:

To find an address history in the database, you must first convert the address to a public key hash and look up its pubkey_id in the pubkey table.  As an example, the SQL for /address/ pages is in Abe/abe.py: handle_address.  pubkey_id links to the txout table, where txout_value is the amount received.  Abe internally uses units of 1 satoshi (.00000001 BTC), so it would store 1 BTC as txout_value=100000000 (1e8).

When money leaves an address, there will be a row in txin referencing an older txout row (via txin.txout_id) where txout.pubkey_id refers to the address.

To find when transactions entered the block chain, you must follow the joins from txin/txout to tx, block_tx, block, and chain_candidate.  Abe stores blocks that are on side-chains, and you should normally ignore those by restricting results to where chain_candidate.in_longest=1.  Abe is designed to support multiple currencies in one database, so you should also restrict by chain_id, where the values are listed in the chain table, and BTC is chain_id=1 by default.  block.block_nTime is the time recorded in the block header in units of seconds since 1 Jan 1970 UTC.

I hope this helps you get started.

That site is now six days behind, and your address appeared on the network only after the site stopped updating.  I have messaged the site maintainer but had no response yet.  If you (or anyone) would like to host the site with an up-to-date block chain and get free monitoring from me via PM I will happily switch the DNS record to your server.


On your server's home page, what block number does it show for BTC?  Are you running the latest bitcoind?  Can you post the last few lines of Abe's output?

@tiker

Regarding address_version in datadir configuration, I've added the following documentation:


To prevent the web process from reading blk*, I give it an empty directory in datadir.

To remove a chain would be a nice feature.  If I remember right, the steps are:

1. Find chain_id in the chain table.  (SELECT chain_id FROM chain WHERE chain_code3='CLC')
2. Delete from chain_candidate where chain_id = ?(value found in #1)
3. Delete from datadir where chain_id = ?(value found in #1)
4. Delete from chain where chain_id = ?(value found in #1)

This does not delete the chain's block data, which is harmless apart from taking space.  You could delete all rows from block, block_tx, tx, txin, block_txin, txout, and pubkey that relate only to the deleted chain.  That would be a nice function to have.

There must be a problem with the identifier sequences.  You can save the database by setting them to correct values.

For portability, Abe supports several methods of ID generation (called "sequences" in some DBMSs).  You can find out which implementation it chose with:
I assume this value is 'mysql' in your case.  The 'mysql' sequence implementation associates an empty table with just one column (an auto_increment) with each sequenced table.  For example, the next `block_seq`.`id` would become the next `block`.`block_id`.  Apparently, the dump/load process did not preserve the tables' internal counters.

This script might fix things for you.
If you have a chance to try it, please let us know the result.

See the comments about "datadir" in the sample abe.conf.

Abe has logic to attach orphaned blocks and reorganize a forked chain.  As far as I know it works, but it is the area I would most like to test when I have time.  Relevant code: adopt_orphans and _offer_block_to_chain in Abe/DataStore.py.

Abe does not validate blocks beyond what's needed to "checksum" a chain up to a trusted current-block hash.  Complete block validation is very hard and not on my priority list, though I might add hooks to use external logic.  (Wrapping Abe.DataStore.import_block with a subclass might suffice.)

I don't expect a problem feeding Abe deserialized data.  You would need a structure like that created by Abe.deserialize.parse_Block with one extra element: 'hash' whose value is the block header hash as a binary string.  The structure is based on Gavin's BitcoinTools.  You would pass that structure "b" to store.import_block(b, frozenset([1])).  (chain_id 1 = main BTC chain)  Abe.DataStore.import_blkdat does this for every block in blk0*.dat that was not previously loaded.

One way to shrink the footprint would be to add support for binary SQL types.  Abe supports the SQL 1992 BIT type and tests for it at installation, but the only database that passes the test is SQLite, which is unsuitable for large servers.  On MySQL and all the others, Abe falls back to binary_type=hex and stores scripts and hashes in hexadecimal, wasting half the bytes.  Relevant code is in DataStore: configure_binary_type, _set_sql_flavour (beneath the line "val = store.config.get('binary_type')"), and _sql_binary_as_hex, where Abe translates DDL from standard BIT types to CHARs of twice the length.

Another improvement would be to remove unneeded features (or, ideally, make them optional) such as the Coin-Days Destroyed calculation (block_tx.satoshi_seconds_destroyed etc.) and the unused pubkey.pubkey column.

Abe verifies proof of work and, as of 0.6, transaction Merkle trees on import.  Yes, an export/import tool like this should come with caveats about trust.  There's a verify.py script (possibly out of date) that verifies the Merkle roots already loaded, and it would be simple to add proof-of-work checks there or as part of an import tool.  Of course, if it is part of a system for fast loading of a local, known good block chain, it's not so vulnerable.

Edit: By "verifies proof of work" I do not mean checking hashes against the target or difficulty, just verifying that the "previous block hash" is indeed the hash of the previous block's header.  Adding a target check would be nice, though challenging for alternative chains that represent target and proof differently.

Yup, or someone with time to spare might write export and import functions, dumping and loading the data in a bitcoin-specific, db-neutral format.  If that runs pretty fast, write a translator from block files to that format, and it might approach the speed of torrent+mysql for the initial load.  The main thing, I suspect, is to create indexes after the tables have data.

It's not that the dumps would be different, but the file offset stored in the datadir table would probably not be at a block boundary in the local block file.  The solution is to reset the pointer.  Abe should have a command-line option for this or even do it automatically, but currently we do it with:
The next run will spend a few(?) minutes scanning the block file, skipping blocks already loaded via the dump.

Thanks for the torrent!

If you stick to this schedule, there is no problem.

In theory, yes, but I would not be surprised if a court still considered the combination a work based on Abe.  RMS discussed this in 1992 in regard to linking executables and libraries.  I think the AGPL is designed to reproduce the situation in the context of online services.

But it seems to me you intend to stay within the spirit of the license, so I apologize for going off topic.