block_id is not the same as block number (a.k.a. block_height).  The gaps in block_id are not a problem.

Please pull the latest and see if it gets you past the error.  I would have pushed it earlier but did not have my github keys handy.

Thanks.

I suspect this comes from new script types.  I have a possible fix here in diff format.  I'll push it to master soon.

I am curious what transaction triggered the error.  I'll ask more about that if the fix doesn't work.

Thanks for the report.  Please mention the git revision (git rev-parse HEAD) and any local code changes.

I don't know of any offhand.

Well, I do not spend much time on new features these days, but if you have a particular function in mind, I'll at least comment on how best to do it.  Several Block Explorer API functions already work more or less compatibly.

Sorry for the long delay, but I am just busy with other things.  Try pulling the latest, then run:


replacing abe.conf as appropriate, and show me the output.  If it says that it deleted "from block", run Abe with --rescan to reload the chain.  That ought to wipe just the block headers from just that chain, but it may be slow due to a MySQL issue.  I suggest first trying it on a copy of the live database.

Also, try to configure your system so that only process loads chain data at a time.

Do you mean errors like this starting at MegaCoin block 41798?  It puzzles me.  From what I can piece together (without direct access to your database) this block could not figure out its block_value_in and/or block_value_out, yet it knows its parent block.  This is not supposed to happen.  Maybe one of the transactions failed to load because of bugs affecting parallel loading.

You might try deleting the block and its descendants and rescanning the chain.  Perhaps I can write an tool to help with this, but I have another priority at the moment.

This is typical of ppcoin-derived currencies.  Have you tried the "ppcoin" branch?  See: https://github.com/jtobey/bitcoin-abe/issues/4


This can result from a block being skipped or incorrectly loaded.  Have you tried rescanning the blockfiles?  Use --rescan or, if you have many coins in one database, you can rescan just the one by setting blkfile_number=1, blkfile_offset=0 in the appropriate row in the datadir table.

Yes, several new coins trace their ancestry to PPCoin and need the "ppcoin" branch in Git.  See: https://github.com/jtobey/bitcoin-abe/issues/19

Another one, CopperLark, has its own format, which the "copperlark" branch handles.

Ideally, Abe would automatically detect among known variants and handle them without code changes.

Good to hear!


I don't use SQLite for the full block chain, so I don't really know.  One user, K1773R, reported a fast load with the database file in tmpfs (RAM).  (https://bitcointalk.org/index.php?topic=22785.msg2639709#msg2639709)  I think K1773R uses SQLite.  If your system has enough memory, you could try that.  Abe takes a lot of space, though.  I estimate 4-5 times the size of all block files combined.  (~/.bitcoin/blocks/blk00???.dat in Unix)  You can reduce this somewhat by setting keep-scriptsig=false or by using the no-statistics branch and default-trim-depth.  See the comments in abe.conf for what those options entail.

K1773R mentioned possibly distributing the SQLite file via torrent: https://github.com/jtobey/bitcoin-abe/issues/15

Sorry for the delayed reply.  Please post the last few lines of output before "catch_up_rpc: abort" or "Exception: RPC load failed."  Have you run Bitcoin with -reindex since adding txindex to bitcoin.conf?

RPC is slower for the initial load, I think over twice as slow as blockfiles.  Once caught up, it usually inserts blocks faster than the blockfile method, since RPC has already loaded most transactions by the time the block comes (assuming that you arrange for it to catch up frequently).

Abe normally does not "read the blockfiles over and over again": it stores the current position in the database.

If you have not enabled firstbits, use SQL to update chain.chain_address_version with the correct value.  That should magically fix all the addresses.  To find the correct value, you could start loading the chain in a new database, identically configured except for that address version.  Then:

SQL> SELECT chain_name, chain_address_version FROM chain;

SQL> UPDATE chain SET chain_address_version='value' WHERE chain_name='chain';

This may require conversion to and from a printable format such as hex, if you use binary_type=binary, which is now the default.  In MySQL, replace chain_address_version with HEX(chain_address_version) in the select statement, and replace 'value' with UNHEX('hex value') in the update.

Ideally, we would have a command-line and web interface to manage chains.

Nevertheless, if I recall correctly, examples do exist, presumably where one address is not generated in the usual way (and coins sent to it are therefore dead).

I edited my last reply to ensure that "collision" includes collision with longer substrings.  For example, 1SgtspiQ... has a substring 1Sgtspi which collides not with 1SgTspiK's base representation, but with a firstbits address derived from it.  So the base representation of 1SgtspiQ... must include the Q.

Thank you, and sorry for the delay!


I don't think this is true or needed.  Some addresses have no firstbits base representation, because an address differing only by case appeared earlier.


This is good but could be clearer.  How about:
(Appearance implies "in a transaction output" as specified below.)



I don't have a strong preference.  Abe does use enough characters to distinguish from addresses later in the block.  I think the code was simpler before I did this.  I don't think conceptually either way is more complex; you choose between defining "order of appearance" and "same or earlier block".  I strongly suggest finding an example and checking blockchain.info's treatment of same-block collisions, since that site obviously has a lot of work put into it and may be unwilling to change.  And let's try to include piuk.

http://blockexplorer.coinworld.us/chain/AlphaCoin/q
etc. replacing AlphaCoin with coin name.

Jan, given SgtSpike's and my openness to change, would you care to work on a precise definition?

That's okay, perhaps we'll get it done here now...


Thanks for digging that up.  It looks as if the transactions are indeed generated by end users (of a modified client) who do see the affected addresses.  I don't have a full database handy, but I am curious how many OP_NOPs are in the chain and whether they still occur.  If it's a tiny percentage and none recently, I'm less concerned.  But here, a firstbits assigned in an OP_NOP output would indeed be reassigned to another address, so I suggest we consider it carefully.


Yes, and I fear that "what the protocol accepts" and what future users will consider an "appearance of an address" may be computationally infeasible for an unvarying standard such as firstbits should be.  For example, each normal address (starting with 1) has a unique corresponding, standard 1-of-1 P2SH output script and address starting with 3.  If you know the 1-address, you can find the 3-address, but not vice versa, at least until the output is spent.  Does the spending of such an output constitute appearance of the 1-address?  Some might say yes, others no.  The firstbits standard must give an answer for every possible case like that that may appear in the future.

Firstbits must be derived only from the blockchain, obviously.  "Bitcoin addresses" never occur directly in the blockchain; rather, they are extracted by matching scripts against templates and associating script types with well-known "version bytes".  If your address were spelled out in the block whenever it sent or received coins, we could trivially define first appearance.  Since that is not the case, it is up to us to cope with the complexity.

It is still an open question to me, whether to ignore non-functional trailing script fragments such as Eligius's OP_NOP.  The whole class of them would have to be easily identified, or if a special case for the single OP_NOP would yield better backward compatibility, I would consider just that.  I do not have time right now to find out the answers, but if someone more interested than I comes up with the data, I will comment on it. 

Perhaps I read it here:



Cleaner, I suppose, although I, too, originally based it on order within the block.  If it is possible (is it?) for an earlier tx to spend the output of a later tx in the same block, we would have to be careful about whether the earlier one's input script triggers "appearance" of the address in case a competing address appears in a third transaction between those two.  The rule about "all addresses appearing in the same block" avoids that issue, while unfortunately making some address prefixes ineligible ever to become firstbits.


Thanks for the willingness to clean up.  I think in this case, the change would never result in a firstbits becoming the firstbits of another address.  Rather, some (rare) firstbits would become non-firstbits, ineligible due to appearing first in two or more addresses within a block.


The wiki describes the original two standard transactions: https://en.bitcoin.it/wiki/Script#Scripts.  P2SH is in BIP 16: https://en.bitcoin.it/wiki/BIP_0016.  The authoritative reference is the IsStandard and Solver functions in src/script.cpp.


Yes.  In fact, here is a case where Abe does the other thing, and the standard that I propose will require a change.

I assume that the reference client does not generate these OP_NOP transactions, nor does any client designed with interoperability in mind.  The last time I checked, there were only a handful of them.  I can only speculate why someone went to the trouble of making them.  I remember first seeing them in Abe, checking BlockExplorer, and adding a tiny change to make Abe handle them compatibly.  The problem is, if we allow OP_NOP at the end, where do we draw the line?  Do we allow multiple OP_NOPs or other operations?  I guess we could just as easily define the script types by how scripts start, and if firstbits.com has always done this, that might convince me.  I consider a full, exact match cleaner.  In practice, I think anything after the standard portion gets ignored, unless it would make the script or block too big.  I haven't checked whether IsStandard allows trailing junk, but I have the impression that developers mostly frown on it.

Thanks for your comments.

Let me refresh your memory.   https://bitcointalk.org/index.php?topic=16217.msg957933#msg957933


Thank you, but this is not a precise specification.


IIRC, this contradicts information given earlier, that neither address would get 1sgtspike: both new firstbits would contain enough characters to distinguish them from each other, regardless of their ordering within the block.


True, but if so, it is not for lack of trying.  Rather than rename the function in Abe, why don't we finish the standard and bring ourselves into agreement.  Here is where we left off: https://bitcointalk.org/index.php?topic=16217.msg960077#msg960077