Versions 0.4.1 and 0.5 released today. 0.4.1 contains mostly minor fixes since 0.4. 0.5 contains dramatic speed improvements, new back-ends, and more. New in 0.5 - 2011-08-16* Big speed improvement for address history and transaction pages. * Big load time improvement for SQLite: below 10 hours for the BTC chain. * MySQL supported. * Oracle supported, but slow due to lack of transparent bind variable use in cx_Oracle. * BBE-compatible HTTP API functions: nethash totalbc addresstohash hashtoaddress hashpubkey checkaddress * New HTTP API functions: translate_address decode_address * Online list of API functions (/q). * Native BeerTokens currency support. * Many minor improvements; see the Git log. New in 0.4.1 - 2011-08-16* Security enhancement: refer to orphan blocks by hash, not height. * Fixed bugs affecting new chains defined via the configuration. * Warn, do not exit, if a block file is missing or unparsable. * Abe parses the new merged-mining block field, CAuxPow. * Decrement the value returned by getblockcount for compatibility. * Bug fix: remove '-' from parenthesized amounts. * Fixed previous/next block links on /chain/CHAIN/b/NUMBER pages. * Accept "var += val" in configuration as equivalent to "var = val" where "var" has not been defined. * Added --commit-bytes option to adjust the database commit interval. * Minor robustness and cosmetic improvements. Enjoy! |
|
|
|
Still thinking about various design issues though.
One more request relevant to Abe. Would you please design the library with alternative chains in mind? This is a must-have for Abe, which has good support for non-BTC currencies and a correspondingly inclined user base. CREATE DOMAIN amount_type AS NUMERIC(16,  CHECK (VALUE < 21000000 AND VALUE >= 0); 21 million is BTC-specific. CREATE DOMAIN address_type AS VARCHAR(110);
I'm not sure what you have in mind for addresses, but bear in mind that different currencies use different "address type" bytes and thus different address spaces. Abe doesn't store addresses at all, just the public key hash160. It efficiently looks up an address by extracting the pubkey hash (base 58 decode) and using that as a key. It even does initial substring searches for addresses by converting the substring into one or a few pubkey hash ranges. Address owners can (and do) use the same key pair in different networks under different names (addresses). I expect this to become more common, and a frequent query will be to find balances of a given pubkey hash in all known currencies. I also anticipate chain splits where both sides remain active indefinitely, especially with the advent of merged mining. I like your span_left/span_right system for tracking small side chains, since it gives me a quick way to tell if any given block is an ancestor to another. But I'm not sure how to apply it to this case. Would you consider moving the span columns to a new table indexed by block_id and chain_id, corresponding to Abe's chain_candidate? Cool stuff. Thanks. |
|
|
|
I'm not so sure that it's good to prune out the previous_output_hash since when getting an entire block, you want the entire info (which includes the hashes)- especially for displaying on a block explorer type website, hashing the transaction or performing the OP_CHECKSIG .etc
Good points. |
|
|
|
I'll probably add cumulative difficulty into the blocks very soon.
Cool. Still thinking about various design issues though.
I might suggest one thing from experience with Abe... CREATE TABLE inputs (
...
previous_output_id INT,
previous_output_hash hash_type NOT NULL,
previous_output_index BIGINT NOT NULL,
A hash per input will be a large fraction of storage space. Abe avoids this by putting unlinked inputs in a separate table and then deleting them when it finds the output: CREATE TABLE txin (
txin_id NUMERIC(26) PRIMARY KEY,
tx_id NUMERIC(26) NOT NULL,
txin_pos NUMERIC(10) NOT NULL,
txout_id NUMERIC(26),
...)...
CREATE TABLE unlinked_txin (
txin_id NUMERIC(26) PRIMARY KEY,
txout_tx_hash BIT(256) NOT NULL,
txout_pos NUMERIC(10) NOT NULL,
FOREIGN KEY (txin_id) REFERENCES txin (txin_id)
)
After the unlinked_txin is deleted, you can get the previous output's transaction hash via txin.txout_id to txout.tx_id to tx.tx_hash. |
|
|
|
Import status: now at block 62,000 (after 1.5 days). Note that I have a pretty slow machine that only uses 30W :-), not exactly made for db loads.
But still, that's too long. Block 62,000 is only about 5% of the way through, since the blocks are bigger now. I've committed rewrites of two queries that MySQL did not fully optimize. My (very informal) testing suggests a speedup of 10x. Let me know if it helps. |
|
|
|
Are you on IRC? You should hit me up on #bitcoinconsultancy
Thanks, but I have too many time obligations (ages 5 and 2) to justify regular IRC use. You can quickly query the total difficulty for a chain using a single SQL command:
SELECT SUM(to_difficulty(bits_head, bits_body)) FROM blocks WHERE span_left=X AND span_right=X;
Will give you the total difficulty in chain X (you have to have to_difficulty() defined as: bits_body * 2^[8*(bits_head - 3)] )
Great, but this presumably reads every block row. Are you familiar with the /q/nethash function? Abe reads every Nth (default: 144th) block's block_chain_work and subtracts the previous from the current value as part of the netHashPerSecond computation. So I do only one query that returns (at the moment) 980 rows for 979 resulting values. (It reads two table rows for every row it returns due to the trick by which I implement "every Nth" in portable SQL.) I suppose not all applications would benefit from this optimization, and I could bolt it onto your schema via trigger or patch if necessary. (Joys of open source!) |
|
|
|
Import status: now at block 62,000 (after 1.5 days). Note that I have a pretty slow machine that only uses 30W :-), not exactly made for db loads.
But still, that's too long. Block 62,000 is only about 5% of the way through, since the blocks are bigger now. While waiting for the import, I reverse-engineered the schema using mysql-workbench.
Thanks! Please update to the latest commit on the master branch and restart your import (it'll pick up where it left off) since I've indexed txin.txout_id. Not sure if it will help here, though. I won't be surprised if there's unneeded table scanning on import. PostgreSQL took a few hours on my 4-year-old laptop last I tried, but Pg may implement foreign keys differently, and Abe currently relies on foreign keys for implicit indexing. You could add txout->pubkey since MySQL supports null foreign keys. Apparently some do not. Anyway, txout.pubkey_id is indexed, I hope? |
|
|
|
Ooh! Lots of yumminess coming down the pike. I look forward to using your library to help display transactions before they get into a block. It'd be nice to harmonize our database schemata somewhat. While I wrap my head around libbitcoin's span_left/span_right reorg strategy, I suggest storing total_difficulty in blocks (where it is immutable) rather than chains. Abe relies on this (block_chain_work) to estimate network hash rate over a range of blocks by fetching just the start and end blocks. Thanks! |
|
|
|
First of all, John, thanks a lot for Abe, it's awesome as far as I can tell.
Thanks! I'm currenlty importing blocks into a mysql db (at block 16000 after about 1.5 hours), it'll takes ages (as in: days), but that's ok for me.
Thanks for the report. I'm considering a totally new initial import design with table indexes replaced by Python variables. One problem I found was with the datadir table:
[...]
Maybe you should consider using an INT PK here and an index on dirname if you need it?
Good idea. Done. The upgrade is not so bad, since Abe has machinery for it. This is actually the seventeenth schema change handled by --upgrade.  Other than that, keep going! I sent my small donation Thanks, I hope it works for you! |
|
|
|
|
Peter, these graphs are great. Would you mind making them for alternate chains such as Namecoin or releasing your code so that others may do so? I can provide a /q/nethash or similar API if that's what you used.
|
|
|
|
Hi Yanz, This error UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in position 0: ordinal not in range(128)
is fixed in the latest commit. Thanks! But I don't think it explains the "lockup". Are you sure Abe isn't just loading data? For SQLite the full block chain can take a week or longer. SQLite is okay for small, experimental block chains, though even there, you could wait hours. (Optimization of the initial data load should be a high priority enhancement. Currently, it's slow but simple, since it uses the same mechanism as the run-time new block import, which can not be optimized the way I have in mind.) Let me know how it goes. |
|
|
|
|
I for one would work more on FOSS and Bitcoin if it paid better--more reliably, more directly.
I'm skeptical of the bounty model where the first to finish gets the prize, while the loser may have done better work. Could you envision a variant where the pot is split according to value contributed after (say) 30 days? And if value contributed is less than 100% of what was requested, part or all goes back to the donors?
|
|
|
|
The problem with demurrage [...]
Good point. |
|
|
|
I realize I'm all over the place here, but here is another way to decrease they coin supply temporarily: have the protocol sell bonds. In this way, the protocol can borrow coins from users (reducing the coin supply), then make interest payments, and return the money at maturity. This would be a good way to handle a temporary oversupply of coins, but not a good way to handle a coin supply that needs to keep shrinking (since more coins are ultimately introduced by these bonds through the interest payments).
So people would have an option and mechanism to buy the bonds? I kind of prefer a dynamic demurrage-like factor that just multiplies every coin's value by a number based on the linked commodity price/index. People are already used to seeing their account balance go up and down by small amounts ("interest" and "fees" or perhaps "taxes"). |
|
|
|
There is, I believe, a much simpler way to exchange between bitcoins and stablecoins.
Sure. My proposal is about linking to prices outside the cryptocurrency world. The bootstrap currency, if you will. E.g., Goldcoin. |
|
|
|
Both coin programs I'm proposing, would need to include the bitcoin like program, and an exchange. There is not currently an exchange attached to any version of bitcoin to my knowledge. I'm working on a distributed exchange outside bitcoin ( https://github.com/macourtney/Dark-Exchange), and you could use a centralized exchange like MtGox or TradeHill, but that would defeat the decentralized aspect of a bitcoin like program. More work and though is needed in this area. removed option expiration. How about a very limited type of exchange within the main chain: Every block contains a price inserted by the miner. A block solution must contain a signature on a standard contract using the coinbase output keypair. The contract is an option to: buy at the quoted price plus a standard spread (e.g., 1%) or sell at the quoted price minus a standard spread an amount of the underlying commodity equal to the block reward. The option can only be exercised by solving a block with this block as its parent. To exercise such an option, a miner must set a special flag in the next block. The two miners have to find each other and settle up (no need to specify how) before either can claim his block reward. They must both sign something using their coinbase keypair and submit the signatures to the network before the network will recognize their block rewards. Perhaps the spreads don't have to be standard, the contract can specify any prices straddling the quote. Miners would be motivated to offer good terms in order to encourage the network to accept their blocks and thus validate their rewards. |
|
|
|
The value returned with the SQL is now actually less than the "ideal" number, as some miners have thrown away block reward BTC. I might make a /q page for it eventually, as the difference is interesting.
Abe now supports totalbc but reports the measured number (e.g., 7017349.98999999) rather than the "ideal" value BBE gives. I do not plan to implement totalbc for future blocks. http://abe.john-edwin-tobey.org/chain/Bitcoin/q/totalbc |
|
|
|
So many questions awaiting empirical answers! I strongly suspect that someone can design a better cryptocurrency, but I set high standards and look forward to seeing these and many other proposals in competition. Some good software could open the field of currency design to anybody with an idea to test, not just C++ programmers with large time budgets. Namecoin and MultiCoin have made good progress, but we have a long way to go. I am thinking specifically of: - Bitcoin client plugin interface: plugins can define block acceptance rules.
- Bindings to facilitate plugin development in high-level languages.
- Set of highly configurable plugins extending MultiCoin's advanced configurability.
- Multiple currencies handled by one client process.
- Multiple currencies on one p2p network with a facility for negotiating which chains to relay to which peers.
- Further development of merged mining:
- Multiple auxiliary chains mined together.
- Multiple proof-of-work formats per chain.
- Effective difficulty able to vary according to proof-of-work size/complexity.
- New type of "inv" message for broadcasting a new currency definition (plugin code and parameters).
- Block chain browser (Abe) and exchange (Bitcoin Central?) automatically supporting new currencies.
- Graphical wizard/tools to put it all together: plugin architecture, configuration, testing, and launch to the network. Marketing is still a Phase 2 feature.
I have the C++, network, database, and thread programming background, and I will eventually do this if I live long enough, but I can not donate enough time to it in the foreseeable future. Will you help me elaborate this vision until someone picks it up and runs with it? |
|
|
|
I've added the network hash rate estimates. It's pretty compatible with Block Explorer and adds optional start and stop arguments for fetching less than all history. http://abe.john-edwin-tobey.org/q/nethashHoping someone converts this into graphs like these for alt chains... |
|
|
|
|
Show Posts
