Separate wallet and daemon even further

[nuggetbram]

At the moment the daemon and qt-wallet are separate, yet the qt-wallet contains and needs all the data within the daemon. Would it not be possible to have one wallet that draws on the daemon, treating it like a dll?

I've got tons of btc and other altcoin wallets after updates, and would love it if I could just replace the daemon in the same folder as the wallet (which could then be cross compatible with other (similar) altcoin daemons).

Is that at all possible?

[piotr_n]

Is that at all possible?

Great question. Though, I would rather ask: how is it possible that the wallet has not been split from the node by now?

My personal research comes to a hugely biased conclusion that the bocoin core dev team just does not care much about security of your wallet, and that is why they are reluctant to separate it from the online part, instead integrating it more and more with stuff like openssl, which (even they silently admit) nobody really knows what it does inside.

But they definitely have this goal high on the agenda. In fact they have had this goal on their agenda for many years already - like since forever.
The only problem is that there have always been "lower-hanging fruits" to reach. Some of the low-hanging fruits took a couple of years to develop and nobody wants to use them, but well... you cannot say that they didn't develop anything. And besides, it's just a myth that what they have developed instead ain't good for your privacy at all.

From what I see at the news flash from their conferences, separating wallet from the online part is still very high on their agenda.
Except that, from what I see at the github, it is unfortunately still far behind many other "low-hanging fruits", that keep coming and coming...

Now, if you care about my personal estimate about some realistic schedule, as for this specific team of engineers: I don't believe they will achieve the goal within the next 10 years.
In other words: if you need a wallet that would be separate from a node, sooner you will learn bitcoin programming and make one yourself.
And you hear it from a guy who actually did it.

[DeathAndTaxes]

It is possible it just hasn't been seen as a priority.  There are a lot of advantages as it would speed the development of alternate clients.  The "node" is the hardest part.  It has to be 100% exactly compatible with all existing nodes.  There are countless gotcha and traps for new developers.  If the node was separated from the wallet it would make development of alternatives safer.  One simple example would be deterministic wallets.  If today you wanted to develop a deterministic wallet you have three options a) write a complete node & wallet stack from ground up (and pray you don't introduce some critical bug), b) fork bitcoin-core and then need to continually patch and update it forever as "node" portion of bitcoin-core is updated, c) develop a wallet which uses bitcoin-core as a node backend (i.e. Armory).   None of those are particularly attractive.  However in bitcoin-core (wallet) and bitcoin-node (0% wallet code, 100% node connectivity) were separate processes you could fork bitcoin-core to make bitcoin-HD and have it connect to bitcoin-node.  From that point changes to bitcoin-core would not result in dependencies you would only be concerned with outward facing changes to bitcoin-node.  Your client could support updating node support by just having the user download and install the latest version of bitcoin-node.

I would prefer a full break where all wallet functionality is removed from bitcoind as well and bitcoind became just a generic node and on Windows OS this would run as a service 24/7 in the background.  Even better would be refactoring it into three components bitcoin-lib (a set of libraries containing the functionality of a node aka dll), bitcoin-node (an implementation of that library), bitcoin-core (a wallet which provides a reference implementation of how to connect to bitcoin-node).  I think I wrote a post about this over two years ago so as piotr_n points out it probably isn't going to happen.  I am not bashing the dev core team and I my C skills aren't at a level this type of refactoring would require.  I would gladly contribute towards a bounty to see this done but the critical part would be for it to become the mainline repo.  Simply having a compatible but different project isn't much good as it would need to be continually updated to match changes to the mainline client.  So this would require more than just funding it would also require support and commitment from the core team.

As a side note hopefully this encourages some alt-coin developer to do it right from day 0.  Having it tightly integrated makes future development much harder and once the coin is launched changes are occurring on an active network which must always remain backwards compatible.   If nothing else "ease of development" is one intangible attribute that could set an altcoin part in a sea of worthless clones (obviously it couldn't be the only thing).

[jgarzik]

Tangible progress is being made.

Disable-wallet support is rolled out in 0.9.  This is the first step towards a public daemon being a separate process from the wallet.  This mode permits disabling of wallet at compile time (no BDB or wallet code) or runtime.

Headers-first sync is the next step.  This will enable a faster, more secure chain sync... and also enable a further wallet split.

At that point, the wallet may run as a separate client, talking to the public blockchain engine.

[DeathAndTaxes]

Tangible progress is being made.

Disable-wallet support is rolled out in 0.9.  This is the first step towards a public daemon being a separate process from the wallet.  This mode permits disabling of wallet at compile time (no BDB or wallet code) or runtime.

Headers-first sync is the next step.  This will enable a faster, more secure chain sync... and also enable a further wallet split.

At that point, the wallet may run as a separate client, talking to the public blockchain engine.

jgarzik,

Glad to hear it, that sounds great.  Also hopefully my post wasn't taken as criticism.  As noted I lack the level of C development required for a refactoring of that scale so I don't want anyone to think I am throwing stones.   Maybe some developer who eventually writes a new altcoin see the real difficulty and sheer amount of work needed to (carefully) break the tight coupling of UI, wallet, and node logic introduced in the satoshi client and starts a greenfield project on a better platform.

This mode permits disabling of wallet at compile time (no BDB or wallet code)

I look forward to this not just for the reduced code footprint but also for losing the bdb dependency.

[piotr_n]

Come on man, who are you trying to fool?
You know very well that the current node's architecture is completely unfeasible to support multiple external wallets.
You need to change the architecture - and you know it.
The disable-wallet support does not mean shit, because you can disable wallet anywhere, while enabling an external wallet - that is a challenge.

And I am not saying that it is a challenge that would exceed skills of any of you guys, but you as a team, let's be real; you have been implementing a deterministic wallet for like three years already. Far smaller project, but for all I know it still does not have a chance to get released withing the next ten months...

So I am not surprised that you did not come to deny my ten year estimate for splitting the wallet from the node - at least we are on the same page here

But sincerely, if you don't want to loose the race, you need to get your shit together, my dear bitcoin core development team.
Because your default argument of "Satoshi left his code for us, so get on you knees, suckers" isn't going to work forever.

[kjj]

You can already hook up an external wallet using the RPC functions.  Optionally, add the notifiers to reduce polling.

[piotr_n]

You can already hook up an external wallet using the RPC functions.  Optionally, add the notifiers to reduce polling.

We probably have a different understanding of what a wallet is.

For me wallet would be saying to the node like "give me all the current UTXO records for these addresses: <here go the addresses>"
And then, after a second or two, the node would reply with the list of the UTXO records, that the wallet can potentially spend.

For all I know the current RPC does not support such things.
And the architecture does not make it easy to add such a support.

[kjj]

You can already hook up an external wallet using the RPC functions.  Optionally, add the notifiers to reduce polling.

We probably have a different understanding of what a wallet is.

For me wallet would be saying to the node like "give me all the current UTXO records for these addresses: <here go the addresses>"
And then, after a second or two, the node would reply with the list of the UTXO records, that the wallet can potentially spend.

For all I know the current RPC does not support such things.
And the architecture does not make it easy to add such a support.

Your wallet is just a lightweight version of a general wallet.  You are correct that the current API does not support lightweight wallets.  See the second link in my sig for a look at exactly what changes would be needed to support them.

But a heavier wallet works just fine with what we already have.  Actually, the only thing that is hard to do today is sweep privkeys.

[piotr_n]

Your wallet is just a lightweight version of a general wallet.  You are correct that the current API does not support lightweight wallets.  See the second link in my sig for a look at exactly what changes would be needed to support them.

But a heavier wallet works just fine with what we already have.  Actually, the only thing that is hard to do today is sweep privkeys.

I don't know what you are talking about.
What is a difference between lightweight wallet, general wallet and a heavier wallet?

I guess we have a general misunderstanding on the cultural level, because where I come from a heavy wallet is one that has lots of money in it, and bitcoin-qt is definitely not where I'd be willing to keep a lot of my money

Besides, why are you telling me what changes would be needed?
It's like saying: I can fly to the moon, because that is what I would need, in order to get there: ...
it is exactly what I am talking about mentioning the ten years: your post is three years old and you haven't done a thing to add the support for whatever problem you believe it would solve.
Are you waiting for other people to implement your ideas in your code for you, or what?



EDIT:
I'm not an expert in your code, but IMHO the biggest issue you will meet adding support for multiple external wallets (which I understand is the ultimate goal) isn't going to be in the wallet, but in the blockchain engine part.
The current approach, where you have a wallet file that has its own state which is tied to a specific state of the block chain - this is unfeasible for external wallets.
A wallet owner does not want to keep any part of his wallet in an online node, for a real-time syncing with the chain. He wants to come to my house, connect his hardware wallet to my PCs and spend his coins from his wallet using my node, without waiting half an hour for a -rescan.
Now he cannot - not with your software. And it isn't a matter of API.

[kjj]

A lightweight wallet has limited capabilities, relying instead on external services.  The term is widely understood in programming, engineering, etc.  Armory is a good example of a general external wallet.  It exists today, and indeed has for several years now.  But it is far from lightweight.

I have a job, a life, and indeed a variety of other interests.  In addition, I rarely use C++, so working on the official client is difficult for me, and because of the aforementioned other interests, I have not yet got around to investing enough time to improve my skills in that area (and probably never will).  However, all is not lost.  What I have built works very well for me, though it will never be suitable for general release to the public.  And other people are implementing similar things in their own way, so it isn't like the world is deprived of any great benefit by my continued inattention.

My point is that it is possible to use the existing tools to implement a bitcoin wallet outside of the standard node, today.  Bringing that wallet up to your standards will require a bit more, such as an extra service that implements the extra RPC calls that I identified 3 years ago.  But that isn't hard to do at all.

Bitcoin is a volunteer effort.  The developers either follow their own whims, or the whims of their employers.  Unless you are footing the bill, there isn't much point in complaining that your pet project isn't getting attention.

[piotr_n]

A lightweight wallet has limited capabilities, relying instead on external services.  The term is widely understood in programming, engineering, etc.  Armory is a good example of a general external wallet.  It exists today, and indeed has for several years now.  But it is far from lightweight.

Then we definitely have a different understanding of what a definition of wallet, in the bitcoin software, is.

For me "wallet" is a component that creates & stores private keys and signs messages (mostly transactions) with them - that's basically all it does.
Wallet according to my definition might have some network API (to control it), but it definitely does not deal with peers, nor any relying.

I believe the author of OP uses a similar definition of the wallet as mine, rather than yours.

But if for you "wallet" is actually a peer-to-peer client along with the wallet, then now you made me wonder; what are you guys actually planning to separate, from what, in your software?
And what do you actually disabled with the disable-wallet switch? Apparently I got it all wrong

[nuggetbram]

My interpretation was that the wallet is anywhere coins are kept for storage - so technically even bitcoind is a wallet...

At this stage we don't have anyway of connecting to the network without a wallet, which is what this topic is about.

[piotr_n]

My interpretation was that the wallet is anywhere coins are kept for storage - so technically even bitcoind is a wallet...

But now you make me wonder what "are kept" actually means.
For all we know "coins are kept for storage" in the blockchain.
Knowing the private key(s) you can destroy your entire bitcoind setup and you will still be able to spend the coins.

At this stage we don't have anyway of connecting to the network without a wallet, which is what this topic is about.

What do you mean? Isn't it why they made the disable-wallet mode in 0.9?
And what is the actual application of such a feature? Because I cannot figure out anything useful to do with it, having the API such a creature provides.

IMO what the topic is about is to make the component that stores private keys and signs transactions separate from the p2p node part.
So you could e.g. run the wallet component on a more secured hardware; like a hardware wallet. Isn't that what the whole fuzz is about?

[nuggetbram]

I phrased that pretty terribly didn't I 

That's more what I meant - being able to connect and setup a node without anything else attached, only the node/networking software

[piotr_n]

I phrased that pretty terribly didn't I  

That's more what I meant - being able to connect and setup a node without anything else attached, only the node/networking software

Yes, but as I added later, with the API the current node provides, in the diable-wallet mode the software is pretty useless.

What you really need from such kind of software is to be able to query the UTXO database.
Without it such a node is there just to generate an unnecessary traffic and consume resources.
It's pretty useless if you cannot even check a balance of an address, let alone fetching its unspent outputs.

[davec]

I would prefer a full break where all wallet functionality is removed from bitcoind as well and bitcoind became just a generic node and on Windows OS this would run as a service 24/7 in the background.

....

As a side note hopefully this encourages some alt-coin developer to do it right from day 0.  ....

This very thing was done in btcd from the ground up.  It was specifically designed this way for the many of the very reasons mentioned in this thread.

btcd has no notion of wallets built into it at all and only provides a chain server that does the core block chain validation and networking.  For those unfamiliar with the project, it is not an alt-coin, rather it is an alternative implementation of bitcoind (Bitcoin Core nowadays) which separates things out into nicely compartmentalized pieces.  As mentioned in the recent Beta announcement, it is mature enough and provides all the pieces necessary at this point that it is theoretically possible to run the entire Bitcoin network solely on btcd.  NOTE:  We firmly believe there needs to be multiple implementations, so we wouldn’t want this situation to occur at all, but it serves to illustrate the current completeness of btcd.

Since it was designed with wallet separation in mind, it does not suffer from the issues brought up by piotr_n in this thread.  It provides a JSON RPC API that is mostly compatible with Bitcoin Core.  However, it extends the API a bit to also provide Websockets for real-time updates to things happened to the chain such as block connects and disconnects which is then used by a separate process btcwallet to perform wallet services functionality such as querying balances, creating addresses, etc.

This design intentionally makes it possible, and quite likely, there will ultimately be several external wallet implementations, including hardware wallets, which only use btcd for chain services.

[piotr_n]

I remember when I freshly got myself interestied with Bitcoin, the topic of the wallet protocol was pretty popular back then. There is even an old wiki page about it: https://en.bitcoin.it/wiki/Wallet_protocol

But the idea somehow got abandoned, though honestly I am not sure what for, because I cannot say that any significant progress has been made in other parts of the software.

Some common API between the online part and the wallets seems to be even more desired now than ever before.
The Bitcoin Foundation used to promise to standardize this kind of stuff, but they apparently have been too occupied with exchanging business cards for the last three years.
Not that any sane person had actually hoped that they would help to standardize anything, except their own profits.

[Cryddit]

If I understand the goals correctly, Bitcoind is going to have all the "wallet" stuff removed from it, and then your "wallet" will be a separate program that communicates with your bitcoind using RPC or similar.

That means, bitcoind will be in the business of getting and sending messages, keeping track of transactions and blocks, and will have no idea which transactions might belong to anyone on the local machine and will have no access whatsoever to any privkeys under any circumstances. 

Your wallet on the other hand will be communicating with a single local instance of bitcoind, and unless you and the owner of some remote node agree to the configuration, will not talking to anything over the network.  It's going to be keeping track of your balance and txouts, and when it's unlocked it'll be able to get keys for your  txouts - but it's going to ask bitcoind for anything to do with confirmations, transactions, or blocks. 

When you pay someone, your wallet will notify bitcoind of a new transaction (which bitcoind will then propagate to the rest of the network).  When someone pays you, bitcoind won't notice, or care, because there are no keys that are 'special' to it.  It'll just check and log the transaction the same as it checks and logs all the other transactions.  But when your wallet asks it about transactions involving a particular pubkey, it'll cough up all the transactions that changed that key's balance.