Yes, Abe could store blockfile offsets and read from the files as bitcoind does, or store hashes and fetch raw data by RPC to bitcoind.  A table with just address and transaction identifier (hash or blockfile coordinates), indexed by address, would be useful all alone.  So would a table mapping transaction to block number.  Ideally, the whole database would be built of minimal pieces such as these, and the loader would consult metadata to figure out what it has to do.

Well, if you mean the BTC chain, or any kind of server use, I agree.  It would be nice to add some advice about back-end selection in the docs.

On the other hand, given Abe's SQL abstraction layer, any optimizations tend to benefit all databases, unless we start parallel loading, which SQLite won't support.

I'd like to support a lot more configurable features.  Currently, there is keep-scriptsig and use-firstbits, both of which control the existence of certain columns in the database.  The no-statistics branch also omits some columns and a table.  We could have a bare-bones option that omits transaction inputs and outputs from the database, fetching them as needed from bitcoind.  At the other extreme, we could support denormalization (copying columns into dependent tables) for faster queries at the cost of insertion time and space.  This motivated me to clean up the SQL abstraction layer (not yet merged from no-statistics to master).  I want to abstract over more SQL dialect differences to facilitate configuration changes that modify table structure.

Keeping the front end usable under all configurations would be a challenge, but not to worry: we can declare some configurations unsupported by the HTML front end.  One might even imagine a zero-init mode that starts loading and using new blockchain data before it fills in the old.

Of course, optimization could be easier if we commit to MySQL or PostgreSQL, but I think there is enough low-hanging fruit to postpone that idea for a while.

I'll appreciate it.  You may want to peruse the GitHub issue.

I think so, but I don't have profile data.

A bug has caused http://abe.john-edwin-tobey.org to lose track of the bitcoins outstanding as of Block 230560.  Details are in TODO.txt.

This affects the experimental no-statistics branch and possibly the master branch.  It appears to depend on multiple concurrent loading processes.  For best results, use the master branch and avoid concurrent loading until this is fixed.  To avoid concurrent loading, either use a single process (not FastCGI) or have all but one process use an empty datadir list ("datadir []" in abe.conf).  When I set up FastCGI, I use datadir=[] and create a separate loader job that runs Abe continuously in a loop with the live datadir(s) and --no-serve.

Sorry for any inconvenience!

@salfter

Thanks for the education.  It would be pretty easy to add JSON and JSONP support for API functions.  I'll add it to the to-do list and create an issue on GitHub.  Patches are welcome.

In theory, both branches work with and without txindex.  You only need txindex if you want to load data over RPC from bitcoind.  Benefits of RPC loading are (1) you see unconfirmed transactions, and (2) large new blocks show up faster because most of the transactions are already in (provided that you frequently force a catch-up by loading a page or running in --no-serve mode).  See the comments about "default-loader" in abe.conf or Issue #17 about setting up RPC.

You might even manage to run in RPC mode without txindex, given an already loaded database, but this situation is vulnerable to a condition where bitcoind processes an output and its spending input before Abe loads the output.  So I specify txindex.

Note that I mistakenly documented a need to run "bitcoind -rescan" the first time with txindex in effect; that should be "bitcoind -reindex" instead.

In practice, I sometimes see the following error [EDIT: fixed] with bitcoind 0.8.1 when a blockfile other than the current one contains trailing NULs (zero bytes).
I work around it by trimming the NULs from the file (using /usr/bin/truncate in Linux and the blkfile_offset value from the error trace) and restarting.  I would like to fix this in the code.

In theory, reading the blockfiles while bitcoind runs is unsafe due to race conditions and the use of an undocumented feature.  In practice, I do not recall this ever causing a problem until Bitcoin 0.8, and the problems related to running simultaneously with 0.8.x are fixed.  Abe checks each block's Merkle root so as not to load a corrupt or incomplete block, so at worst, an error should involve resetting the datadir row to the start of the current file (or pass --rescan to Abe for a slower but easier fix).


I am sure there is some file that will be a certain size, but I don't know it.  My blocks, index, and chainstate directories are similar to yours in size, and I have a confirmed working txindex.  The way I know is by finding a transaction all of whose outputs are spent, and fetching it with "bitcoind getrawtransaction".  If it says "transaction not in index" bad, if it gives you a hex string, good.


Thank you for your support.

Answered via PM.  If anything useful results from this exercise, one of us will post it.

This would be a nice feature but is not supported currently.  If you have good SQL skills and a few hours to spare, I can outline the steps.  But if you can afford to wait for the reload, that will be much easier.

Thanks to deflation, I am asking 4.5 BTC.

EDIT: still "at current rates".  IOW, $800 USD.

The experimental branch (no-statistics) now imports UNCONFIRMED (MEMPOOL) transactions (as well as blocks) over RPC from a running bitcoind, provided that:

• Bitcoind is new enough to support getrawmempool, getrawtransaction, getblock, and getblockhash (tested 0.8.x; 0.7.x may work)
• If using bitcoind 0.8.x, -txindex is in effect (see here)
• The datadir.chain_id column is populated: for BTC, this means execute the following SQL: UPDATE datadir SET chain_id = 1 WHERE chain_id IS NULL

My demonstration server satisfies these requirements and includes unconfirmed transactions in search results.

This feature ought to be easily portable to the master branch, in case anyone wants it.  However, I'd feel more comfortable with a configuration option to tell Abe whether or not to use RPC.  By default, it tries RPC and falls back to blockfile scans if that fails.  Also, the new RPC code could use testing and improvements.

Well, that's pretty perverse, but all right, we should be multithreaded.


If you have any trouble with README-FASTCGI.txt, I'd like to know.

Easier, though not as well tested: define a server class in abe.py:


Then add the text in bold: httpd = make_server(args.host, port, abe, server_class=HttpServer)

There is also SocketServer.ForkingMixIn in case ThreadingMixIn gives weird errors.

Thanks to the market, the asking price is down to 8 "at current rates" (c. $750).  But pledges have also been "at current rates"... danged volatility.

Like a zero-fee transaction, eventually it'll get in.  Probably.    In the absence of a business model, my priorities are driven by my personal needs, which lately are about space efficiency, bitcoind 0.8.x compatibility, and integrating contributions from other developers.  Donations do encourage me to work, at roughly $40/hour for features that I want to add anyway, up to $100 for site-specific things that don't improve Abe for most users.  (I'd quote a price in BTC, but BTC/USD is too volatile, as I have ranted in other threads, for reasons that led me to create Abe in the first place.)

I'd like to get clear what "multisig support" means.  Now that I have a full database, I can search for new transaction types.  There have been a few hundred transaction outputs that use OP_CHECKMULTISIG as specified in BIP 11 M-of-N Standard Transactions.  Blockchain.info calls these "escrow" outputs.  Examples: 1 2.  Given an address that redeemed an M-of-N output, Blockchain.info lists the transaction, as in this example.  But if the address did not sign it, its page does not list the transaction: example.  Mapping from addresses back to these transactions will take some extra work.

There have also been a few hundred outputs as in BIP 16 Pay to Script Hash (P2SH).  These have receiving addresses that start with "3" as per BIP 13.  Example: 1.  These output scripts are opaque, but the redeeming input reveals one or more addresses.  As far as I know, Blockchain.info does not correlate an address so revealed with the original P2SH transaction.  Perhaps it would be useful to do so, and this would require some extra work.

There have also been a few thousand regular pubkey outputs as in this example that Abe can not parse because the public key is shorter than usual.  This has nothing to do with multisig and is really a bug in Abe, but it would be nice to fix.  There are also a thousand or so outputs of miscellaneous types unrecognized by Abe.  They may all be nonstandard, but it would be nice to make sense of any that are frequent or demonstrably redeemable.

ThomasV sent me this:


It is not directly usable: Abe would have to pass the script to the function where currently it passes pubkey_hash to another function.  But this would be very easy and would add at least some value regarding P2SH.

A full implementation of M-of-N, P2SH, and mapping from address back to M-of-N transaction would take me about 20 hours, so figure 11 BTC at current rates.

FYI, http://abe.john-edwin-tobey.org is current with the BTC chain using the no-statistics branch and the HOMEPAGE template variable to make chain/Bitcoin the default view.  It uses the new abe_loader script (in tools/) to stay up-to-date, and the FCGI process has read-only SQL permissions.

Problems?  Please report them here or by email to jtobey@john-edwin-tobey.org.

Thanks for the report.  It's too bad that junk got into the Namecoin blockchain.  Do you have the actual script length?  http://explorer.dot-bit.org/tx/1474553 shows the transaction size as 32767, and the long script would be most of that, but that explorer may have truncated it.  You can verify this with something like:


I'll consider simply increasing the column width, but that has the side effect of making MySQL create the column as type TEXT rather than VARCHAR (unless using binary-type=binary).  This may not be so bad.  The column does not take part in any joins, and the vanilla app never searches it.  Anyone?

Yes, that information is lost.  All you see is total received minus total sent.  It would be possible to store the total received by each address, but it would only be a lower bound on the real value, since addresses themselves are trimmed when empty.

Any particular use case in mind?

The wiki page links to http://abe.bitcoinstats.org:2750/, which appears current.

Sorry, I haven't written this feature.  You can remove the currency from the main page by finding its chain_id (in the chain table) and deleting all matching rows from chain_candidate and chain, I think.  It's been a while since I did this.

Then reclaiming the block and tx space is a matter of identifying rows in a half dozen tables and deleting them in dependency order, being careful not to delete other chains' data.  Or you could disable keys, delete in any order, and enable keys again, but this is not advisable while other chains may be loading.  I seem to remember "clean" chain removal taking me 20-30 minutes.  In 1-2 hours I could develop a utility that does it all, but it's still a to-do.

I sympathize, but if complete satisfaction were a criterion, there would be no Abe.  See: Release Early, Release Often.