Here's the output filtering all output size <= 0.01BTC


To be honest I would have thought the saving would be greater. Regardless unless the official position is that bitcoin won't support transactions under 0.01BTC you couldn't filter them. The majority of space is taken up by transaction hashes and scripts so if there was a good way to compress these then you could reduce the size equally as well.

One possible scheme would be:
Order the ledger by transaction hash in ascending order. The hash is recorded as a var_int value between the difference of the previous hash.
Scripts are then stored at the end of the ledger, with duplicate scripts being written only once. The CTxOut then points to an script index instead of having the script written inline.

I will play around with some different formats.

Adjusting fees based on the ledger size effect is an excellent idea.

Announcing beta testing of blockchain.info's new e-wallet service.

http://blockchain.info/wallet

I believe this is the first e-wallet service to offer pure javascript transaction signing and wallet encryption/decryption. At no time do you share your private keys or password with our server. Your private keys are stored AES encrypted on servers so even if our database is compromised your wallet will still be secure.

• Live updating interface if your browser support websockets
• Send transactions with no fees
• Client side wallet decryption, no sharing of private keys
• Instant backups upon new key generation
• Export private keys
• Import private easy using a pywallet dump

This is early beta testing so do not expect everything to work perfectly and be sure you always keep a backup of your private keys on your own PC. I'm looking for feedback on browser compatibility and any errors that occur.

I wrote some test code to produce a vector of all unspent outputs (Basically the ledger).



Sample output:


+ You would need to hold a certain number of recent blocks, at least until all chain forks can be resolved or 2016 to calculate the difficulty target. Your probably looking at 85% reduction in disk size and the same in block validation time. I wouldn't even know where to begin calculating an for estimate for merkel pruning would you have to download the full merkel trees?

Edit: Come to think about it, assuming the requirement of a correct balance ledger was enforced you don't even need the latest full blocks. You could take the latest balance ledger from any chain head and download the block headers only to verify the proof of work. That way your looking at more than 85% reduction in chain size.

It would be much simpler if enforcing a correct ledger hash could be done, but it would make adoption much more difficult as it would require 50% of all miners to upgrade at once.

A solution this problem would be to have miners include the hash of previous ledger and block height they believe to be correct. During the initial blockchain download the client would continue to download blocks until there at least one ledger can be agreed upon. Some kind of decaying time window would need to be implemented so if the majority of hashing power is agreed on one "ledger chain" a minority of clients cannot force a full blockchain download.

You don't have trust any node's version of unspent reality, you have to trust 50% of the hashing power's version of unspent reality - something which you kind of have to do anyway. Although the the consequences for a malicious entity gaining 50% hashing power would be more severe (though they would need to have 50% for a long time).

Skybuck: I'm not sure were on the same page. I wasn't really talking about a separate balance chain. The problem with a chain is its a chain so grows indefinatly thus block validation time, disk space, bandwidth requirements all grow indefinitely. At some point you should be able to look into the past and  say these transactions are no longer under contention and be able to archive/compress them.

Because I allow store orphaned blocks the block height cannot be used as a blocks primary key, instead I use a key called block index. If you can lookup by block index then please do as I cache by block index so lookups are faster, otherwise use .

Added two new api calls:


Also regarding usage of this api, if you are displaying data on a public website then please a link back to blockchain.info. + I have a 200GB bandwidth limit so please go easy (or donate).

124621 is an invalid block index (this is different from a block height)


I'll take a look at this, possibly my server time is out.


I started generating data at odd times, not at midnight. Wow transaction fees have really plummeted I will adjust the rounding.

There was a ton of BTC moved around recently e.g. http://blockchain.info/tx-index/12730431/8a4852cf51365059042d024eb0838ce14333b8de5d3c64394459bfac38a73a24 i think that is what has could the spike.

Redirect from pi.uk.com should be up.

I've got a ton of good suggestions from people but not much time to implement them so apologies if I don't get round to them right away. Got a great new feature coming out sometime this week which will make using bitcoin easier for a lot of people.

It's worth noting that this method would reduce block validation time considerably, something which merkel pruning would not help with. I agree that there are better optimisations that can be implemented first but I still think it might be worth discussing for the future. Can you see any obvious flaws in the proposal Gavin?

I've modified to the original proposal to simplify to process of generating a ledger.

I don't believe the merkel tree pruning proposal by Satoshi is workable. If a client cannot validate blocks then it cannot be trusted to have any network participation.

Should be fixed.


Great minds (or should i say bitcoin obsessed minds) think alike. Already working on it.


I'm not a miner so I don't really understand hopping. But If pool owners want me to the delay the "found by" stats then i will do.

Sort of miners would be generating a hash that says "this is the result of all transactions at this block", as long as enough hashing power agrees with this state then it is accepted.


There would be no 100% safe point. I guess the idea would be most clients would hold blocks from the past few weeks (possibly less, few days) but miners would would hold blocks for a much longer period. If your holding the blocks for the past year then an attacker would need to put in a years worth of hashing power to beat it.


Yes assuming every node only had the past two weeks blocks.


Satoshi's paper is pretty vague on this, where is it explained how you prove a transaction was in a block with the merkel tree? With pruning you still have to hold all unspent outputs, including the transaction hash and scriptPubKey. This method clusters unspent txOutputs and no longer requires the transaction hash.

Edit: Even if you can prove a transaction is in a block with the merkel tree you cannot validate a block without all transactions. If you can't validate a block then how do you know you haven't been given false data?

If clients cannot validate a block with only the headers then what is to stop a malicious attacker generating a set of block headers with fake hashes and difficulty targets. There would be no way for nodes to determine which chain is valid without downloading the transactions.

The problem:

The blockchain will not scale how it is used currently. There is some mention of pruning unspent outputs mention on https://en.bitcoin.it/wiki/Scalability however this method still requires storage of all blockheaders, meaning there is still a unlimited cap on the blockchain size. Merkel tree pruning will not help to any significant extent as you maybe be able to tell if a transaction is in a block, however you cannot validate that block without all transactions. If lightweight clients cannot validate blocks then they cannot mine, relay blocks or relay transactions there is almost not point to them validating anything at all and might as well use a centralised blockchain.

a) A smaller blockchain helps lower the barrier of entry for the new users.
b) With less risk of blockchain bloat transaction fees could be lowered
c) The larger the blockchain the less users who will run the client and the more centralised the network becomes.

Proposed solution.

At certain points in time the client generates a snapshot of the of every unspent tx output in the chain. This snapshot encapsulates the state of the blockchain upto, but not including, that block.  When a miner produces a block he generates a SHA256 hash of this ledger and includes the hash it in the blocks coinbase.

When a client begins the initial block chain download it starts from the chain head and works backwards. The client downloads a minimum of 2016 blocks before it will accept a ledger hash. If there is a fork in the chain the client will continue to download blocks until it finds a pair of blocks that at least one ledger hash can be agreed upon. When an identical ledger is found the chain with the best proof of work wins. When the client accepts a hash it will ask the node to provide it with the full ledger corresponding to it which can be self hashed and verified. If the node doesn't have the ledger for that hash it may ask other nodes, if no nodes have a copy then it should continue to download past blocks until it can find a hash and a full copy of the ledger.

To validate a transaction the client locates the each txIn outpoint in the unspent ledger and checks the corresponding script for validity. The client checks the validity of a transaction by looking at the txOutputs in it's latest ledger and at in the transactions included in blocks after. Therefore nodes do need to not generate a balance sheet every transaction instead they would keep a balance sheet for approximately two weeks (2016 blocks) before regenerating. Two weeks has been chosen as a base value because it provides enough blocks to use for difficulty targeting, however nodes are free to keep more or less blocks depending on their storage capacity.

When the client decides it is time to generate a new ledger it looks through the chain for a more recent block which has a ledger hash in it's coinbase and is at least 2016 blocks behind the chain head. It then generates a new ledge sheet for that block and checks that the hash matches. It if matches then it is free to purge all transactions/blocks before that time. If the hash does not match then it is important to note the client does not reject the chain, as long as the proof of work is valid. The order of transactions is already decided by the order in the blockchain so the client would simply wait until a miner produces a hash it can agree with, it should not purge transactions until a hash is found. Miners may want to keep blocks for a longer period of time to ensure they have the necessary proof of work should it be needed.

Would this fork the chain?

No. Miners are free to insert whatever data they like into their block's coinbase. Clients that wish to hold a entire blockchain history can simply ignore it.

How much data would clients need to hold?


This is the initial estimate with compression it maybe possible to halve this value.

Could you mine without the entire blockchain?
Yes. The network could operate fully without any node having the entire blockchain. It is possible that a chain fork could go so far into the past that no nodes have a copy of the chain long enough to resolve the split, however this is extremely unlikely without a malicious attacker having 51% hashing power for a significant period of time.

How would this be adopted, would all miners need to switch immediately?
There needs to be at least one miner producing a ledger hash around every two weeks. So initially this would be possible to implement with only a small pool adopting the scheme. The more frequent miners produce a ledger the more efficiently clients will be able to prune old transactions.

File format
The initial proposed file format would simply be a dump of all unspent txOutputs, in the order they appeared in the blockchain, in the same format as they are serialised over the nework. This has the advantage that any bitcoin client that participates on a network level can decode the file with minimum effort.

The file will probably need to be indexed after it is downloaded as it will not be suitable for locating a txOutput efficiently. The file format for the ledger will be included in the coinbase along with the hash. In the file there will be many duplicate scripts and transaction hashes giving the possibility of much greater compression in future.

Coinbase
Magic Value - File format - Ledger size - Hash
uint32_t, uint16_t, uint64_t, uint256_t

** magic value is a flag indicating this coinbase holds a ledger hash


/Discuss. Feel free to point out any glaringly obvious flaws

The standard bitcoin client will likely not connect to over 1000 nodes as it's limited by select()'s fd_set size of 1024. I'm not sure if this is true of windows. My client, which uses kqueue, has connected to over 5000 nodes at once.

Basic api available @ http://blockchain.info/api. I'll be adding more methods in time.

I've noticed the problem with some of the log graphs before and i can't seem to fix it. I think it's a bug in highcharts, hopefully the next version will fix it.

I've updated the favicon and the final/usd balance on the address page, both good suggestions. It should show the timestamps for new transactions, I guess for transactions with no accurate timestamp i could show the block time. Thanks for the feedback,

Spent some time on this today, I redesigned it using bootstrap since that seems to be what a lll the cool kids are using 

In order to see some data such as transaction sizes and scripts you need to click "Show advanced" on the bottom of the page.

Strange transactions can now be found at http://blockchain.info/strange-transactions

Thank you to the person that donated, much appreciated.

All Feedback welcome.

http://www.judgejudy.com/submit_your_case I'd love to see her figure this one out. 

When i opened the new client for the first time I made this mistake, I'm sure other people will as well. "Address book" in the 0.40 client clearly separates sending and receiving address.

Do you know if they shutdown the dc with the pool? No i didn't rename it, "Vladamir (Pool)" is a different ip i got from http://eu1.bitcoincharts.com/bitcoin-wiki/wiki/Fallback_Nodes.html.

Great interview, well done.

Wasn't mainframe mining run by Vladimir who had his own datacenter? He could still be mining on his own, the node is definitely still active. If no then it's probably just an anomaly.


Oops. Now changed.