[BOUNTY - CANCELLED BTC] : read only blockchain patch for bitcoind

[neofutur]

 hi all,

 Currently, if I have bitcoind running on 20 machines, or 20 bitcoind running on the same machine ( vps for example ), each of them need to download and store 2 GB blockchain, total 40 GB diskspace used and much bandwidth wasted to download it 20 times.

Code:

 ls -alh .bitcoin/blk*
-rw------- 1 neofutur neofutur 1.1G 2012-03-21 00:36 .bitcoin/blk0001.dat
-rw------- 1 neofutur neofutur 401M 2012-03-21 00:36 .bitcoin/blkindex.dat

 The problem is the same for a vps provider like bitvps, hosting bitcoin users, 2 GB blockchain X 20 vps makes much diskspace and bandwidth wasted for no reason.

 Also the initial blockchain download is taking hours , often more than one full day . . .

 Also its nearly impossible to use bitcoind on a small device like a eeepc ( 4 GB diskspace ), a phone, a raspberrypi . . .

 I think the easiest way to make it better would be to have a "read-only" blockchain, one of the bitcoind writes it, and the others only use it read only to check transactions.

 Actually bitcoind is not able to use a read only blockchain, that could be written and updated by 1 bitcoind , and used by all the other bitcoinds on the same server ( in the case of VPS ) or mounted readonly as a sshfs or other secure network filesystem.

 I thought it could be simple and easy to hack bitcoind to allow this ( I tried ugly debugging on https://github.com/neofutur/bitcoin only to find its really not easy )

 So, I start a bounty for whoever can produce a working patch or fork of bitcoind , allowing this "sharing blockchain" feature, allowing a service provider to be his own blockchain provider and save much diskspace and bandwidth.

 Also, the "read only blockchain" is just an idea, one way to implement it that seems to be the easy way imho, but you can also provide a better way to do it . . .

another ( better ) way to address the problem, from gmaxwell :

As I've pointed out before on IRC,  this isn't the way to accomplish what you want. A "one write many read" blockchain can not be accomplished with a simple patch— it would take either significant and hazardous rewrite of BDB, or a hazardous rewrite of bitcoin to use something custom instead of BDB.   Even if someone managed to pull it off I'm doubtful we'd take it upstream.

Instead, it appears to be generally agreed that we want to split the blockchain and the wallet— so you could have one trusted blockchain and N wallets (GUI or CLI/RPC) talking to it across the network.

 The patch or fork will have to provide an option like "--readonly" at bitcoind start .

 the patch have to be a git pull request on github and be clean enough to have a chance of being accepted upstream.

 the implemented solution have to build and work on linux .

 I start the bounty with 5 BTC, and I know others will add btcs soon

 Feel free to ask for details here or on #bitcoin-hosting channel : http://gw.gd/irc
 Read also :

http://bitcoin.stackexchange.com/questions/3199/read-only-blockchain-in-bitcoind-patch-ideas
http://bitcoin.stackexchange.com/questions/2584/is-reclaiming-disk-space-already-implemented-how-effective-will-it-be

Some clues ?

Bluematt have a good beginning :

Code:

< BlueMatt> neofutur: done: take https://github.com/TheBlueMatt/bitcoin/tree/cblockstore and change https://github.com/TheBlueMatt/bitcoin/blob/cblockstore/src/blockstore.h#L127 to return false

useful links from weex :

It appears the devs have discussed this though a bounty like this is probably the only way to get code written. See Mike Hearn's post here: https://bitcointalk.org/index.php?topic=57435.msg892772#msg892772

Another reason for this project to happen(linking roughly to where relevant discussion is): https://bitcointalk.org/index.php?topic=80531.msg892743#msg892743

Rebroad have a good idea to make it more secure for the client :

Code:

(20:32) <   rebroad> neofutur, the thing is... each separate bitcoind is going to want to recover an old database if the centralized one is changed in a way they don't like...
(20:33) <   rebroad> neofutur, so ideally it would be stored in two places owned by 2 separate parties where users of it can be sure that they can recover to old versions
(20:33) <   rebroad> (at least 2)

possible problem :

Code:

(21:21) <   luke-jr> neofutur: rebroad: the problem you'll run into is that the blk files are NOT identical across bitcoin client instances
(21:21) <   rebroad> luke-jr, could they be though, in some theoretical node that's not been written yet?
(21:22) <   rebroad> luke-jr, with one file per block
(21:22) <   luke-jr> triptyl: have you given any thought to doing it without breaking old clients?
(21:22) <   luke-jr> rebroad: probably
(21:22) <  gmaxwell> triptyl: it would be better to call it "blinded" rather than anonymous.
(21:23) <   rebroad> luke-jr, I think it would be worth adapting bitcoin{d,-qt} to allow the blocks to be stored this way, so that they are more VPS friendly

 weex is suggesting another way to implement it :

Code:

As someone who has enjoyed using BitcoinSpinner a bit, why not patch bitcoind so it supports direct calls via BCCAPI ( http://code.google.com/p/bccapi/ ) or the protocol it uses?

PS : I have no problem with redefining the bounty goal,  my idea of readonly blockchain is possibly a bad idea, and we could need some kind of network protocol and a blockchain server, lets say all comments are welcome, and the goal of the bounty can change, blockchain server is probably a better idea, but I thought it would be a good simple / easy first step to begin with a read only blockchain

Current bounty : 15 btc
 +5 btc from raccon under some conditions ( see his comment below : https://bitcointalk.org/index.php?topic=71542.msg813297#msg813297 ) : https://bitcointalk.org/index.php?topic=71542.msg813297#msg813297
+10 from rapeghost
+5 for weex

[Raoul Duke]

I may be saying something that isn't true, but: Can't Armory do that? Or is it that other client, Electrum or what's its name?

Or I may be confusing the features in both of them.
If I recall correctly, Armory can have watch only wallets, which isn't what you want. You want a watch only blockchain.
But Electrum(?) can read the blockchain from a central server, who may or may not be managed by yourself, if I'm not mistaken.

[neofutur]

yes it seems some other clients ( java or python code ) do it, but the bounty is for the old , secure and trusted C code : bitcoind

[a_raccoon]

I'm willing to back active development in two areas.

1) A thin client / server blockchain standard.

That is, a single remote machine worries about the blockchain, and clients can request balances off of it (sacrificing anonymity), and interact directly with this server for transactions.  The wallet can be stored on either their mobile device (for servers that are run by external entities, like MtGox), or can be stored on the server (for servers that are run at home).

I don't care about actual server/client implementation so much as a defined protocol set and working libraries, open license.

2) Super Blocks.  An advancement in the storage of the block chain, whereby the block chain (ledger) is processed for addresses that still hold a positive balance, and the ledger becomes closed and cryptographically archived such that ALL clients can securely depend on a simple cryptographic hash of the whole Super Block.  The culled addresses with balances begin a new Super Block.

This way, new clients may download the entire block chain, but would not have to hash every transaction in the old Super Blocks, as they are known by all to be accurate and trustworthy by a simple file hash.  Only the transactions in the current Super Block block-chain would have to be articulately hashed.  This would also allow for optical storage and transfer of Super Blocks to be distributed on DVD or Blu-Ray instead of internet download, such as for Merchant POS terminals and servers.

I will fund 5 BTC, to each, given the active development of these projects, once they have established a name for themselves and their work in the IRC Bitcoin community.

[rapeghost]

BitVPS is adding another 5 BTC to this bounty.

Conditions: The only thing i care about is that it works on Linux.

[Luke-Jr]

Conditions: The only thing i care about is that it works on Linux.

Not that it's secure?

[weex]

neofutur has requested something based on "the old , secure and trusted C code : bitcoind". As someone who has enjoyed using BitcoinSpinner a bit, why not patch bitcoind so it supports direct calls via BCCAPI ( http://code.google.com/p/bccapi/ ) or the protocol it uses?

[rapeghost]

Bump + Weex:
I attempted this already. The API's are incompatible.

We need it based on Bitcoind.

I'm adding another 5 BTC bounty to this making the total bounty: 20 BTC

[weex]

Bump + Weex:
I attempted this already. The API's are incompatible.

We need it based on Bitcoind.

I'm adding another 5 BTC bounty to this making the total bounty: 20 BTC

That's why I said patch bitcoind!

It appears the devs have discussed this though a bounty like this is probably the only way to get code written. See Mike Hearn's post here: https://bitcointalk.org/index.php?topic=57435.msg892772#msg892772

Another reason for this project to happen(linking roughly to where relevant discussion is): https://bitcointalk.org/index.php?topic=80531.msg892743#msg892743

By the way... I'll add 5 BTC as well. Would love to see this happen!

[gmaxwell]

Currently, if I have bitcoind running on 20 machines, or 20 bitcoind running on the same machine ( vps for example ), each of them need to download and store 2 GB blockchain, total 40 GB diskspace used and much bandwidth wasted to download it 20 times.
[...]
 I think the easiest way to make it better would be to have a "read-only" blockchain, one of the bitcoind writes it, and the others only use it read only to check transactions.

As I've pointed out before on IRC,  this isn't the way to accomplish what you want. A "one write many read" blockchain can not be accomplished with a simple patch— it would take either significant and hazardous rewrite of BDB, or a hazardous rewrite of bitcoin to use something custom instead of BDB.   Even if someone managed to pull it off I'm doubtful we'd take it upstream.

Instead, it appears to be generally agreed that we want to split the blockchain and the wallet— so you could have one trusted blockchain and N wallets (GUI or CLI/RPC) talking to it across the network.

[someguy123]

Currently, if I have bitcoind running on 20 machines, or 20 bitcoind running on the same machine ( vps for example ), each of them need to download and store 2 GB blockchain, total 40 GB diskspace used and much bandwidth wasted to download it 20 times.
[...]
 I think the easiest way to make it better would be to have a "read-only" blockchain, one of the bitcoind writes it, and the others only use it read only to check transactions.

As I've pointed out before on IRC,  this isn't the way to accomplish what you want. A "one write many read" blockchain can not be accomplished with a simple patch— it would take either significant and hazardous rewrite of BDB, or a hazardous rewrite of bitcoin to use something custom instead of BDB.   Even if someone managed to pull it off I'm doubtful we'd take it upstream.

Instead, it appears to be generally agreed that we want to split the blockchain and the wallet— so you could have one trusted blockchain and N wallets (GUI or CLI/RPC) talking to it across the network.

I completely agree here.
On my VPS, if I give someone access and they need to run their own bitcoind, it ends up with an extra 2gb+ space being chewed up by THEIR blockchain, if instead, my personal bitcoind could handle the blockchain, and the other users bitcoind's would simply control their own wallets and use the single blockchain instance provided by my primary bitcoind, it'd save a lot of time and resources for setting up several bitcoind's.

So essentially we're looking to have separate daemons for the wallet management and the blockchain validation?

[lulzplzkthx]

Why not have a flag in each client "Download blockchain" and a path to where the Blockchain (should) reside. That way those without the checkmark simply try to read the file, and those with the checkmark read and write to the file.

[neofutur]

Why not have a flag in each client "Download blockchain" and a path to where the Blockchain (should) reside. That way those without the checkmark simply try to read the file, and those with the checkmark read and write to the file.

 that was the initial idea, but its not as simple as saying it, code it if you think its so simple

[Nyaaan]

Chmod the blockchain after it is downloaded so nobody can write to it.

[rapeghost]

* stabs Nyaaan in the face *

[rini17]

I am trying to add multiple independent wallets/users support to bitcoind. It is planned to work this way: There will be one bitcoind normally running, but it will use multiple wallet.dat files. Every RPC user will have its own file, and they will be authenticated by existing RPC username/password (so, no change necessary at the client).

The code is at: https://github.com/jurov/bitcoin . I have some issues with testnet, so the code is almost untested as of now. It will create two files wallet.dat and wallet2.dat . The first one is accessed normally by username and password from bitcoin.conf, the second one has "2" added in the end of both username and password, for example:

#access wallet.dat
% bitcoind -rpcuser=user -rpcpassword=pass getaccountaddress ""  
mkdkmraXRMmVw115NHuF1qippjqGpUravx
# access wallet2.dat
% bitcoind -rpcuser=user2 -rpcpassword=pass2 getaccountaddress ""
msVoNGgtq31rbqkngmpc5uqV2w6tSBu9MH

However I want to know first whether there is enough interest in this solution. I guess the work necessary to bring it up to par quality and especially security-wise will be around 150 BTC.

[rapeghost]

fuck. yeah.

[weex]

rapeghost, way to stab for effect!

rini17, I think there is interest...the more code people see the more likely they'll be to pony up some funds to pay for development. You might want to break your 150BTC estimate into smaller chunks that can be paid per milestone.

This thread is about "a patch" but to think longterm about this, it may help to get some feedback from the core dev team. From what I've been told on IRC, the direction they want to go is to to seperate the blockchain and network code from the wallet handling code. I haven't been able to find a github issue that specified this succinctly so creating such an issue might be a good start.

Or just hack it out...either way!

[rini17]

After some testing and fixes, the prototype is now able to move funds between the two wallets and get correct results from getinfo/listtransactions.

rini17, I think there is interest...the more code people see the more likely they'll be to pony up some funds to pay for development. You might want to break your 150BTC estimate into smaller chunks that can be paid per milestone.

I can see these milestones:
1. Adding/removing users (preferably without need to restart bitcoind)
2. Permissions (for commands like "stop")
3. Performance. Maybe this turns out to be non-issue, but from my experience with bitcoind it may be possible for users to DoS each other.

This thread is about "a patch" but to think longterm about this, it may help to get some feedback from the core dev team. From what I've been told on IRC, the direction they want to go is to to seperate the blockchain and network code from the wallet handling code. I haven't been able to find a github issue that specified this succinctly so creating such an issue might be a good start.

Yes, I plan to contact the dev team. The code is very well written and multiple wallet support is already halfway there. Thus this patch isn't very invasive and 90% of it is in RPC code. It should not be an obstacle for future updates.

[tevirk]

I can't help feeling that the better way is to separate out the wallet-handling code, with an RPC interface between that and the blockchain + network code, and an option of running it the current way with the two modules in one prcess.  Still, there may be situations where multiple wallets in one client is better.