If you intend to release the source and are concerned about bugs affecting its output (misleading users about their bitcoin holdings) then I suggest you use the Bitcoin testnet during beta.

If you want to test the site live on the Internet and are concerned about security bugs (exploits) then I suggest you protect it with a password and give it to only a few trustworthy testers.  Also, use security best practices where possible, such as a dedicated OS account or virtual host with no access to non-empty wallets.

If you want to get something out in public quickly but are afraid of exploits or misguided forks of your code, use common sense.  If the site is non-commercial (free to use and free of advertising) I won't care much about the license until its proprietary features make it the most popular site running Abe.

@MORA,

Congratulations, and thanks for asking about the license.  Short answer: I am not a lawyer, the software license does not cover the data, and nothing in the AGPL prevents you from putting your own code in the public domain if that code's dependencies are compatible with the GPLv3 or AGPL.

Now, it gets a little hairy if you offer a proprietary service based on Abe's tables, and it needs a running Abe to keep those tables up to date.  Maybe the law would consider that a "work based on" Abe even though the service only directly reads the tables.  If in doubt, describe your plan to me.  If I find it in keeping with the spirit of collaboration and the goals of Abe and Bitcoin, I will write a license exception giving it explicit permission to use Abe.

Yes, that will work.  For the common case where the transaction is on the main branch, you can just subtract its block height from the longest chain's height.  chain_candidate.in_longest will equal 1 when a block is on main.  For the top block, you can use:
or as in DataStore.get_block_number:
This is probably the right thing even if the transaction is not on the main branch, since users won't care about confirmations on dead ends.

Yes.  You can also force a catch-up by running the program with --no-serve in another process from the command line while the server listens.  But this won't help with the database idle connection timeouts.

MORA,

Thanks for the comments and workaround.  Indeed, db idle timeouts are a problem.  I use two workarounds but haven't settled on a default approach.  I have a cron job request the homepage every minute to trigger the catch_up code.  And there is a "catch_up_thread" branch in git that automatically does this on a separate thread.  It may need merging with the master branch to get the latest features, and I have not tested it as well.  ThomasV implemented it for ecdsa.org and I think uses it in Electrum.

Reconnecting automatically is a good idea.  I think your patch will work in practice, although I see a slight chance of database corruption if it tries to reconnect in the middle of a transaction.  The chance is remote, since transaction durations won't normally approach the idle timeout, but the 12-hour init makes me extremely cautious about corruption.   My long-term plan is to test "begin transaction" for portability and explicitly start each transaction.  The start of a transaction would be the time to reconnect if needed.  For now, let me know if the workarounds prove inadequate.

Yes, I have been thinking about supporting firstbits.  I would consider a patch that adds address and first block_height (or block_id) to pubkey.  If I were doing it myself, I would try a new table "firstbits" with address_version, pubkey_id, block_id, firstbits.  "address" could be an optional field for applications that want it.  Storing firstbits directly would give us simple two-way lookups.

Putting address or firstbits in pubkey would make me nervous about chain splits (where each side remains active) and firstbits adoption by alt chains.  However, ideally I'd like to support this design for apps that want denormalization for performance.  So any design involving a new table would add a view to make it look as if the fields were in pubkey, and a design that adds columns to pubkey should wrap it with a view that provides constant "00" address_version.

Abe doesn't yet have a way to turn on or off features such as firstbits.  I would like to let users turn features on or off at install time: firstbits, coin-days destroyed, namecoin stuff, etc.  I would store a flag in configvar for each feature (such as 'firstbits'='yes'/'no') and skip the processing associated with deselected features.

This is just my vision at the moment, I don't have any code beyond what you see.  Any patch that looks useful to somebody, I'd probably accept.  If it compromises too much in some area, I'd put it on its own branch until the compromises become options.

By the way, I don't know how firstbits.com would handle two addresses with the same unique prefix first appearing in the same block.  I would give the shorter prefix to the address in the first transaction within the block, with ties going to the first txout within a transaction.

Please try with --commit-bytes=100000 and let me know.  This will prevent a commit after every tx insertion but may lead to errors when concurrent processes insert.  I recently changed this setting from being the default.

How is system load?  Is the database server busy with CPU?  IO?  Is there free memory?

When I have time, I will try some load testing.

Come to think of it, a plain old stack trace would be better than nothing if you have one around...

Any error message in the log or browser?

If it persists, I find an easy way to "profile" abe is to interrupt it (ctrl-C) and note the stack trace, then restart, let it get going, and repeat a few times.  If the interrupt usually happens in the same query or two, I know what to optimize.

Fixed, thanks.

Hmm, that's a new one to me.  Could you post a way to reliably produce the error in my own environment?  Or the next best thing would be to run with --log-sql and post a section of log including, say, 5 SQL commands leading up to the error.

I've read the simplified summary (thanks).  The issue, if one exists, is certainly not worth changing the "protocol" (block acceptance rules).

Users want their transactions to get into blocks.  So they pay miners (perhaps indirectly) to include them.  Problem solved!  The payment need not be in the form of transaction fees.  Matching currency users with miners is far beyond the proper scope of block validation.  Ask Mr. Market for a solution.  Hint: entrepreneur talks to pool ops and users, develops service, takes a cut.

Or did I misread it?

Just uploaded an example (untested by me) of subclassing Abe.abe to add handle_* functions.  I suspect q_* would work too.

https://github.com/jtobey/bitcoin-abe/blob/master/contrib/ecdsa.py

If you come to rely on an undocumented interface, the best way to keep it around is to document it and ask me to commit the doc.

This is a Newbies-friendly copy of: https://bitcointalk.org/index.php?topic=22785.0

!!! Announcing Abe, the open-source block chain browser !!!

While theymos' Bitcoin Block Explorer has served us well, there are benefits that can only be had by an open-source version of this critical Bitcoin infrastructure.

• Privacy: you can view information without revealing your interest in it.
• Extensibility: you can add features that you want.
• Trust: you can audit the source code.
• Reliability: you can run it on your own server free of charge.

Abe provides much of Block Explorer's interface, including the list of recent blocks, a search tool, and detailed block, transaction, and address history pages.  Abe can also present multiple currencies such as Bitcoin, Namecoin, Tenebrix, and whatever FooCoin tomorrow may bring.  Abe lets you page back and forth in the block list and presents some new statistics like Average Coin Age.

Abe's interactive performance approaches Block Explorer's, though it is untested under BBE loads.  Abe supports a subset of Block Explorer's API.

Source code on Github: https://github.com/bitcoin-abe/bitcoin-abe
Demonstration site: http://abe.john-edwin-tobey.org/ (a 512MB VPS, can't handle heavy loads)
Requirements: Python and an RDBMS; tested with PostgreSQL, SQLite, MySQL, Oracle, ODBC, and DB2.
Donations: 1PWC7PNHL1SgvZaN7xEtygenKjWobWsCuf (BTC) and NJ3MSELK1cWnqUa6xhF2wUYAnz3RSrWXcK (NMC).

Enjoy!

December 6, 2012: Version 0.7.2 released.
August 31, 2011: Version 0.6 released.
August 16, 2011: Version 0.5 released.
July 15, 2011: See changes since July 4.

Thanks for sharing!

I wonder if your GetTransactions function would work as a VIEW.  That might translate easily to MySQL etc.

I have no experience with nginx, but FastCGI should work the same on any server.  I might add that the "sudo" overhead is not essential, it's just how I set things up to keep any Abe exploits out of Apache and vice versa.  If your security policy doesn't need that separation, I suggest naming the script /usr/lib/cgi-bin/abe and removing --watch-pid="$1":


And set "Alias / /usr/lib/cgi-bin/abe/" or the nginx equivalent in the server configuration.

When you have finished following the instructions, the file /home/user/cgi-bin/abe should exist and be executable.  Issue "ls -l /home/user/cgi-bin/abe" and post its output if unsure.

Then test the site and post any error message from your server log or browser.

Oh, the runtime CPU and memory cost would be tiny, I'm sure.  But now, as a code reader, I must scan past three new globals to see the meatier parts such as pindexBest right above them.  It's not that I don't like the feature, or even (maybe) enabling it by default, but adding things in this way gradually makes the program harder to understand.

Even for uint64 on 32-bit systems?  And what if parts of the same getinfo result reflect changes that other parts don't reflect?

Luke-Jr, thanks for the patch.

Assuming it is bug free (which I'm not sure... does it mutex the reads of shared data like nPooledTx?) my concern is code bloat for a feature with a dozen(?) potential users.  Not to say the code base isn't currently bloated, but now seems a good time to mention refactoring options.

Imagine compile-time options similar to -DUSE_UPNP and -DGUI controlling inclusion of esoteric features.  (I note that optional features should in no way affect the validation rules, since we want everyone to agree on those.)  Luke's features, and probably over half the existing RPC cruft, could go in a separate file, ideally behind a module interface such as VinceD's hooks.  Imagine "extensions.cpp" defining CExtension extensions[], each element's existence controlled by an #ifdef, and each CExtension implementing a bunch of hooks, one of which the RPC table constructor would call and allow to insert methods.

Anyone?

You have to replace "USER" with your login, for example /home/terrytibbs/cgi-bin/abe.  And the doc tells you later how to create that file.
Edit: the file could be anywhere, ~/cgi-bin is just a suggested location.

Congrats!  Too bad I didn't last the whole time.  The miner's lifestyle took its toll on me.

John.Tobey