bitcoin core RPC compatible lite wallet proxy?

[monsterer]

Is there a bitcoin core RPC compatible lite wallet available which proxies onto a full node for the data?

Use case: multiple merchant wallets, one blockchain, one API on one machine without having to duplicate the blockchain.

I have a full node running, which is great, but I need multiple wallets... Having lite wallet(s) which would get their data from my full node would be awesome and not have any of the trust issues a lite wallet would normally have.

I don't want to have to implement another RPC wrapper, so this needs to be compatible with the core RPC API.

Cheers, Paul.

[CIYAM]

Unfortunately the problem is UTXO indexing (by address) which is not done by Bitcoin Core (except for its own wallet).

Personally I think it's a pity this feature isn't added into Bitcoin Core as it makes sense for it to be able to provide this information (rather than having to add another server like Electrum on top of it).

I don't know what the core devs reason is for not implementing this (beyond it not being a high priority) and would actually help out if lack of developer time was the only issue (am guessing that it isn't though).

[2112]

It is a complex project, I'm not aware of anything open source.

Also the RPC paradigm isn't really well suited for this concept: the communication between blockchain server component and wallet component must be two-way not one-way request-response.

A producer-subscriber paradigm (like e.g. FIX protocol) is much better way to implement this, and once correctly implemented it nearly completely obviates the need for one-way RPC.

There were very buggy attempts to implement what you're asking by effectively replacing subscription pattern by a busy-wait loop with very frequent polling. But this was always ending with coding disaster that wouldn't handle chain reorganizations correctly.

[achow101]

I believe that bitcoin armory kind of does that. It used the bitcoin rpc to get all of its data, but I don't know whether you would consider that to be a lite wallet.

[CIYAM]

I believe that bitcoin armory kind of does that. It used the bitcoin rpc to get all of its data, but I don't know whether you would consider that to be a lite wallet.

As it requires an enormous amount of RAM to even run (unless it has dramatically changed design recently) I certainly don't think you could ever classify it as a "lite wallet".

[CIYAM]

It is a complex project, I'm not aware of anything open source.

Indexing UTXOs by address is not complex at all - and is exactly what projects like Electrum already do (clearly you are not very aware).

[2112]

Indexing UTXOs by address is not complex at all - and is exactly what projects like Electrum already do (clearly you are not very aware).

Are you talking about that Electrum client that uses Stratum protocol (designed by slush) to talk to Electrum servers?

Those Electrum servers that couldn't keep up with the Bitcoin transaction stream during recent stress tests and were getting hours behind w.r.t. the live network?

Or are you talking about something else?

[CIYAM]

Those Electrum servers that couldn't keep up with the Bitcoin transaction stream during recent stress tests and were getting hours behind w.r.t. the live network?

Clearly you just don't know when to give up - you do realise that Electrum is not the only software that does UTXO indexing or do you not?

Maybe you should go and study indexing for a while before making more ridiculous posts.

Hint - check what the OP is asking about and stop going off on stupid tangents trying your best to show off some sort of technical knowledge that you clearly lack.

[Envrin]

http://synala.com/

Would do the trick.  You can either setup multiple wallets with different BIP32 key pairs within one installation of Synala, or install it multiple times, and just enter the same RPC connection settings in each install.  Would require you to have one full node running, but you could run the multiple systems / wallets off that one node without problem.

Only thing is, during install it'll ask you to add lines to the bitcoin.conf file like:

Code:

walletnotify=/usr/bin/php /home/site1/public_html/synala/bitcoind/process_tx.php %s
blocknotify=/usr/bin/php home/site1/public_html/synala/bitcoind/process_block.php %s

You'll need to change those to run a bash script instead for the multiple sites.  So instead you would have for example:

Code:

walletnotify=/home/coind/process_tx.sh %s
blocknotify=/home/coind/process_block.sh %s

Then create the bash files with the appropriate PHP commands being run, one for each install.  So for example, /home/coind/process_tx.sh would contain:

Code:

#!/bin/sh
/usr/bin/php /home/site1/public_html/synala/bitcoind/process_tx.php ${1}
/usr/bin/php /home/site2/public_html/synala/bitcoind/process_tx.php ${1}
/usr/bin/php /home/site3/public_html/synala/bitcoind/process_tx.php ${1}

Hope that helps!

[2112]

On my computer there is nothing about UTXO indexing in the original post.

I'm going to quote both posts until I see the clarification.

Those Electrum servers that couldn't keep up with the Bitcoin transaction stream during recent stress tests and were getting hours behind w.r.t. the live network?

Clearly you just don't know when to give up - you do realise that Electrum is not the only software that does UTXO indexing or do you not?

Maybe you should go and study indexing for a while before making more ridiculous posts.

Hint - check what the OP is asking about and stop going off on stupid tangents trying your best to show off some sort of technical knowledge that you clearly lack.

Is there a bitcoin core RPC compatible lite wallet available which proxies onto a full node for the data?

Use case: multiple merchant wallets, one blockchain, one API on one machine without having to duplicate the blockchain.

I have a full node running, which is great, but I need multiple wallets... Having lite wallet(s) which would get their data from my full node would be awesome and not have any of the trust issues a lite wallet would normally have.

I don't want to have to implement another RPC wrapper, so this needs to be compatible with the core RPC API.

Cheers, Paul.

[Envrin]

Also the RPC paradigm isn't really well suited for this concept: the communication between blockchain server component and wallet component must be two-way not one-way request-response.

Huh?  It's somewhat two-way.  Not completely, but there are "walletnotify" and "blocknotify" config options available, and combine that with the "importaddress" function if you're not using the wallet.dat file, and it will instantly notify your software of a new tx that's assigned to one of your addresses and every new block.

As for block reorgs, that's quite simple too.  For example, say you set it for 4 confirmations required.  Every time a new block is generated, your software immediately fires, adds 1 confirmation to each tx, then looks up each unconfirmed tx that has 4 confirmations, and ensures it's still a valid tx.  If not, bitcoind will show the confirmations at -1, meaning block reorg so you abandon that tx.  If it's still valid and showing 4 confs within the blockchain though, then you confirm it in your system.

Or am I missing something here?

[CIYAM]

Or am I missing something here?

You are quite correct - I wouldn't pay much attention to the posts of the other person.

[Envrin]

Maybe he's talking about writing your own node or something, I don't know.  If that's the case, then yes, that's quite difficult when dealing with block reorgs, ensuring you get all txs and blocks from the P2P network without skipping a beat, ensure everything is valid, etc.

However, if you're willing to leverage Bitcoin Core, then everything is quite simple.

[monsterer]

Unfortunately the problem is UTXO indexing (by address) which is not done by Bitcoin Core (except for its own wallet).

The watch only wallet mode of bitcoin core doesn't support listunspent?

[CIYAM]

Unfortunately the problem is UTXO indexing (by address) which is not done by Bitcoin Core (except for its own wallet).

The watch only wallet mode of bitcoin core doesn't support listunspent?

I assume it indexes the UTXOs for watch only addresses in exactly the same way (the only difference being that you don't have the private keys) so yes it should work with the "listunspent" RPC command.

[Envrin]

The watch only wallet mode of bitcoin core doesn't support listunspent?

Just checked and confirmed, yes, the "listunspent" function does list all UTXOs from watch-only addresses added via the "importaddress" function.  Should note, the watch-only inputs have the "spendable" attribute set to false, but that's the only difference i can see.

Wouldn't really recommend that method for multiple reasons, and better to keep your own database of UTXOs for your addresses, but your software, so your choice.

EDIT:  If you're by chance using a HD wallet, you somewhat have to keep a list of UTXOs yourself anyway.  Otherwise, if you're going to rely solely on "listunspent", you're not going to know what key index was used to generate the child key / address from the master BIP32 key, and that piece of information is quite important if you ever want to send the funds.

[2112]

Also the RPC paradigm isn't really well suited for this concept: the communication between blockchain server component and wallet component must be two-way not one-way request-response.

Huh?  It's somewhat two-way.  Not completely, but there are "walletnotify" and "blocknotify" config options available, and combine that with the "importaddress" function if you're not using the wallet.dat file, and it will instantly notify your software of a new tx that's assigned to one of your addresses and every new block.

As for block reorgs, that's quite simple too.  For example, say you set it for 4 confirmations required.  Every time a new block is generated, your software immediately fires, adds 1 confirmation to each tx, then looks up each unconfirmed tx that has 4 confirmations, and ensures it's still a valid tx.  If not, bitcoind will show the confirmations at -1, meaning block reorg so you abandon that tx.  If it's still valid and showing 4 confs within the blockchain though, then you confirm it in your system.

Or am I missing something here?

What you describe is acceptable only to one-person shops or similar setups.

Most organizations will have clear interdepartmental security delineations and will have a preexisting accounting system already based on the (accounting, securities, etc.) prevailing standards.

"walletnotify" and "blocknotify" are really hacks that cross the security domain in wrong directions: from the Internet front end to the accounting servers. Only desperados will approve such architecture. The other problem with event callbacks are:
1) is that they require both caller & callee systems both online at the same time
2) they have no indication of the backlog of pending callbacks.

While conceptually handling chain reorganizations isn't terribly difficult it is also the feature that most often goes nearly completely untested. It is one of the most common failure modes in various solutions mentioned on this forum.

The other problem is purported scalability of having one blockchain and multiple wallet. Again the practice had show that people handle they wallet-synchronization code with tight poll loops and other similar lameness, that in practice multiple copies of bitcoind take less CPU and bandwidth resources only more storage. I think Stratum & Electrum is a good example: Stratum protocol in theory handles asynchronous notifications well, but its practical implementation in Electrum is plagued by horrible inefficiencies.

The last thing is that even if you solve all the above problems you still don't have a clean interface to the industry-standard accounting tools: Fix protocol in securities and various "middleware/message queueing" (like e.g. ZeroMQ) in general accounting. On the other hand the raw bitcoind can be run with -privdb=0 (and minimal patches) and allow to fully integrate with accounting back-ends thanks to the support for X/Open distributed transaction monitoring in BerkeleyDB than handles the wallet.

In summary: individual problems aren't difficult by themselves, the complexity is in integrating all of them. The sad state is that various folks peddle incomplete solutions plagued with tight poll loops and with untested/incorrect handling of chain reorganizations. The only exception was bitsofproof but it is back to being closed source. Almost nothing has changed in this space since I joined this forum sometime mid-2011.

You seem to be an advocate of Synala. I have one question: did your solution survived at least one accounting audit conducted to the prevailing accounting standards in your country?

Well, on the positive side I can at least observe that folks ceased to propose "use built-in accounts in bitcoind". Before those got officially marked as obsoleted this was the closest thing to the willful accounting fraud.

[Envrin]

Ok, we're talking about two completely different things here.  You're talking about large organizations with multiple corporations and bank accounts spread across the globe, who deal with 10s or 100s of millions in funds, and report to the likes of FinCEN, correct?  In that case, I doubt an out of the box solution would even be possible, because every system would have to be customized for that specific business (model).  I'm assuming folks like Bitstamp go with custom solutions, because they have millions in their budget to do so.

And yes, my solutions are meant for the small business owners, the 1 - 5 man shops or similar.  People that make say $100k - $500k/year.  For my target audience, my solutions work beautifully.

If you're looking towards folks like BitStamp, et al, then sounds as though the market is wide open for you, if you can put together even a base system that can be easily customized.  You're basically talking about writing your own node, correct?  That's a fairly adventurous task, and many have failed at trying, including myself.  Could be lucrative if you managed to pull it off though.  Probably wouldn't get too many clients, but when you're signing $1.5mm contracts, you don't really need many.

EDIT:  You mentioned the bitcoind & "accounting" servers must both be online, but that's not true.  If bitcoind goes down, when it comes back online, it'll begin firing all necessary "blocknotify" and "walletnotify" commands as it downloads the blockchain from where it left off.  Then just write a quick JSON API to communicate between the servers, queue the txs / blocks, send JSON request, if a proper response is received remove tx / block from queue.  If proper response not received, leave in queue and retry at regular intervals until proper response is received.

[2112]

Ok, we're talking about two completely different things here.  You're talking about large organizations with multiple corporations and bank accounts spread across the globe, who deal with 10s or 100s of millions in funds, and report to the likes of FinCEN, correct?  In that case, I doubt an out of the box solution would even be possible, because every system would have to be customized for that specific business (model).  I'm assuming folks like Bitstamp go with custom solutions, because they have millions in their budget to do so.

And yes, my solutions are meant for the small business owners, the 1 - 5 man shops or similar.  People that make say $100k - $500k/year.  For my target audience, my solutions work beautifully.

If you're looking towards folks like BitStamp, et al, then sounds as though the market is wide open for you, if you can put together even a base system that can be easily customized.  You're basically talking about writing your own node, correct?  That's a fairly adventurous task, and many have failed at trying, including myself.  Could be lucrative if you managed to pull it off though.  Probably wouldn't get too many clients, but when you're signing $1.5mm contracts, you don't really need many.

EDIT:  You mentioned the bitcoind & "accounting" servers must both be online, but that's not true.  If bitcoind goes down, when it comes back online, it'll begin firing all necessary "blocknotify" and "walletnotify" commands as it downloads the blockchain from where it left off.  Then just write a quick JSON API to communicate between the servers, queue the txs / blocks, send JSON request, if a proper response is received remove tx / block from queue.  If proper response not received, leave in queue and retry at regular intervals until proper response is received.

I don't buy your claim that your solutions "work beautifully". You are basically serving the market of "everyone knows the root password of the server". You can't properly provide elementary security like, e.g.

1) the web design subcontractor can't have the passwords to the accounting server
2) the are two independent web store fronts: the production one that is open 24*7 and the staging one for beta testing that is open intermittently and possibly managed by a different webmaster

It is also very telling that in your last paragraph you foresee closing down web front-end while having the accounting open. But the most real businesses operate in an opposite way: the web front end is open 24*7 but accounting is regularly being closed (weekly, monthly, quarterly) for periodic reporting and auditing.

I understand that there is a big market for "fly-by-nights" that essentially operate until the first audit by taxation authorities or first investigation by police/prosecution/regulators. I can't blame you for serving it, but I will stay away and always advise everyone else to stay away too.

There's a big disconnect between the two markets you mentioned: one-password shop with up to five people and multibillion corporation. The best example is what Microsoft targets with Server Essentials (a.k.a.) Small Business Server. The organizations of up to about 50 people who tend to subcontract various non-essential activities to the specialist firms. The people who know how to produce reports using Microsoft Access and why every commercial SQL database engine has some sort of per-column security.

Anyway, returning back to the ideas described in the opening post of this thread: it would be nice to have a Bitcoin engine that works somewhat like any other database engine. In the sense that you can start the engine itself to track the Bitcoin P2P network and then attach and detach wallets like one would attach and detach databases to the SQL engine.

But this isn't going to happen under current Bitcoin Core Development Team. In 2012 I mentioned something similar in the "What's the problem with getting Bitcoin compliant with GAAP???" and Gavin Andresen replied:

When I hear that, I hear "you should stop doing what you're doing for a year or two or three and re-implement the whole thing."

Yeah... no.  As much as that would be a fun project, I don't think that is the job of the team working on the existing reference implementation. Keeping the existing software and network running as smoothly as possible is the highest priority.

I think the current governance situation in the Core team could lead to the change in the general direction of the development. Time will show.

[monsterer]

Wouldn't really recommend that method for multiple reasons, and better to keep your own database of UTXOs for your addresses, but your software, so your choice.

Doesn't that mean you need to manually parse the blockchain for UTXOs to your scripts/addresses? That means calling getblock and then getrawtransaction for each transaction in every block?