Raw TX API & other private keys: 'watch' or 'listunspent' other addresses?

[HanSolo]

Looks like the raw transactions API could be useful to run a connected client, but keep signing keys elsewhere. (Elsewhere = only  occasionally connected in some limited hardened way.)

This would be easier if the connected client was able to 'listunspent' outputs available to arbitrary addresses.

Is there a way to register watched addresses in the standard client?

Or, could 'listunspent' be extended to take any non-wallet address as an optional parameter?

[Gavin Andresen]

Is there a way to register watched addresses in the standard client?

Not yet. There is a pull request implementing bloom filters that should make that easy to implement.

Or, could 'listunspent' be extended to take any non-wallet address as an optional parameter?

No. The reference implementation doesn't keep a master map of all addresses to unspent transaction outputs, and adding such an index for the small number of services that need to look up arbitrary addresses doesn't make sense.

I'd suggest you -blocknotify and the getblock() RPC call to maintain your own index of address --> unspent txout.

[CIYAM]

No. The reference implementation doesn't keep a master map of all addresses to unspent transaction outputs, and adding such an index for the small number of services that need to look up arbitrary addresses doesn't make sense.

I understand the point but I think that as fewer and fewer people other than miners and services end up choosing to use the "reference implementation" (due to the size of the block chain and time it takes to process) wouldn't having such an index (as an option) make at least a little sense (which would only come into effect after a rescan assuming the option wasn't originally turned on)?

My view is that the "reference implementation" is actually what we want to make things easy and possible for services (rather than each having to roll their own). If this is wrong then why bother with the "raw tx" API at all (as certainly no normal end-user will ever use it)?

[TangibleCryptography]

No. The reference implementation doesn't keep a master map of all addresses to unspent transaction outputs, and adding such an index for the small number of services that need to look up arbitrary addresses doesn't make sense.

I understand the point but I think that as fewer and fewer people other than miners and services end up choosing to use the "reference implementation" (due to the size of the block chain and time it takes to process) wouldn't having such an index (as an option) make at least a little sense (which would only come into effect after a rescan assuming the option wasn't originally turned on)?

My view is that the "reference implementation" is actually what we want to make things easy and possible for services (rather than each having to roll their own). If this is wrong then why bother with the "raw tx" API at all (as certainly no normal end-user will ever use it)?

I have to agree with this.  Having bitcoind be a platform that enables other developers and service providers to leverage the "low level plumbing" done by developers with superior knowledge of the "quirks" of the protocol (i.e. Gavin et al) allows it to pay dividends.

I think that is something which is really missing (and note this isn't a complaint I haven't built it so I can't complain that others haven't) but rather just an observation.  At the same time the bitcoin-qt becomes less useful to end users due to the increasing requirements of full nodes, it becomes even more valuable as a platform for developers to build higher level services and tools.


Just throwing out an idea here but I think the GUI needs to be decoupled from the "bitcoin engine".   Imagine a scenario where there are three related projects.  

The bitcoin class library (think openSSL, or bouncy castle but a bitcoin equivelent).  Contains the data structures and algorithms necessary to communicate with the protocol and store, organize, and access blockchain data.

The bitcoin reference engine is a non-GUI implementation of the bitcoin class library and handles communication between a client (any client) and the network.

The bitcoin-qt (GUI client) is branched off and handles the high level implementation of user interaction and uses the bitcoin engine.  This would allow other GUI developers to be able to build drop in replacements leveraging the work already done in the engine.

Once again just an observation but it seems to me the "bitcoin reference client" is really three projects tightly coupled. That makes building derivative works difficult and limits the feature-set that the lower level functionality can provide.  Everything is seen through the lens of what is best for this GUI client, a client that is increasingly not the optimal solution for manner users.

WARNING:  The above was written before I had my morning dose of caffeine so may be semi-incoherent.  

[jgarzik]

The GUI is already decoupled from the "engine" as demonstrated by how bitcoind and bitcoin-qt are built from the same source code.  bitcoind is the base, built with zero GUI libraries.  bitcoin-qt is simply "bitcoind + GUI"

In general, there is wide agreement that separation should be improved, along similar lines like ones you suggest:

• core class libraries
• blockchain engine
• wallet handling
• graphical user interface

Separating out the class libraries would enable more applications to share the core code.
Separating out wallet handling into a separate process (and executable) will greatly increase security, ensuring that any bug in the P2P network triggered by a remote attacker would never overflow or corrupt the wallet process.
etc.

But as >90% of the full nodes are the reference implementation -- which remains the most complete and secure implementation to date -- keeping the network running securely has to take priority over these other things.

If you know C++... help!  Unlike what the tinfoil hats would have you believe, the dev team is still volunteers working on our own time (hopefully Gavin will be funded by BF real soon now, easing the pressure a bit).  It is easy to join the dev team, too:  just submit a useful pull request.  No forms to sign or anything

So file this under...   we agree with you, but need more developer resources and bandwidth to make that happen.

[BCB]

"tinfoil hats" == Atlas et el

[HanSolo]

Is there a way to register watched addresses in the standard client?

Not yet. There is a pull request implementing bloom filters that should make that easy to implement.

Or, could 'listunspent' be extended to take any non-wallet address as an optional parameter?

No. The reference implementation doesn't keep a master map of all addresses to unspent transaction outputs, and adding such an index for the small number of services that need to look up arbitrary addresses doesn't make sense.

I'd suggest you -blocknotify and the getblock() RPC call to maintain your own index of address --> unspent txout.

Yes, external indexing by watching blocks should be a suitable workaround.

Thank you for a quick and definitive answer!

[2112]

The GUI is already decoupled from the "engine" as demonstrated by how bitcoind and bitcoin-qt are built from the same source code.

Actualy that is not true. bitcoind is a functional subset of bitcoin-qt in the core functionality of "sendtoaddress". And bitcoin-qt still suffers from the inversion of control problem.

I just checked the master github:

Code:

static bool noui_ThreadSafeAskFee(int64 nFeeRequired, const std::string& strCaption)
{
return true;
}

So the inversion of control in the fee calculation/addition is still there.

This isn't a patch or pull request. It is an architectural issue that requires extensive changes.

[Gavin Andresen]

This isn't a patch or pull request. It is an architectural issue that requires extensive changes.

Right, and we don't accept "change the world" pull requests because the risks of introducing a catastrophic bug outweigh the benefits, and rewriting everything invalidates all of the careful code review that's been done over the past few years.

However, we ARE moving towards better architecture as different parts of the code get worked on to support new features or improve scalability or fix bugs.  For an example of inversion-of-control done right, look for use of boost signals and slots; for example:

Code:

boost::signals2::signal<void (CWallet *wallet, const uint256 &hashTx, ChangeType status)> NotifyTransactionChanged;

[2112]

Right, and we don't accept "change the world" pull requests because the risks of introducing a catastrophic bug outweigh the benefits, and rewriting everything invalidates all of the careful code review that's been done over the past few years.

However, we ARE moving towards better architecture as different parts of the code get worked on to support new features or improve scalability or fix bugs.  For an example of inversion-of-control done right, look for use of boost signals and slots; for example:

Code:

boost::signals2::signal<void (CWallet *wallet, const uint256 &hashTx, ChangeType status)> NotifyTransactionChanged;

I'm 100% with you on both issues: I understand that you need maintain the continuity of safety and I understand that you need to build new features in the incremental way.

But during the year since I made the inversion-of-control observation I had a very productive and educational conversation with slush in the Stratum thread. The implicit knowledge from that thread is that there is a wide gap between the stated knowledge of the Bitcoin application programmers using PR-languages (Perl,PHP,Python,Ruby,etc) and their actual skill. slush never stated it explicitly, but I had to infer it from the discrepancy of the discussion in that thread and various blog posts of the people trying to interface with both Stratum-engine and bitcoind RPC-interface.

So from the discussion with slush I learned that in order to have a decent chance at achieving full-application-stack correctness you'll need to somewhat dumb down the ideal solution. Otherwise the impedance mismatch will be still there it will just move upward into the higher tiers of the application. Your best architectural choices will be of little value if the application programmers will not be able to correctly apply them in their end-user interfaces.

To this extent I would suggest somewhat less ambitious additions to sendtoaddress: (1) upper limit of the fee (2) explicit random seed for the stochastic knapsack solver in the SelectCoins. I think (1) is fairly obvious. (2) will allow correct (and testable!) implementation of the iterative choice of a fee.

Finally, I have to say that I do not share your optimism about the applicability and suitability of boost. I have a deep feeling that the C++/boost community will struggle for many years without 100% correct implementation of exception handling in the multithreaded environment. I make this observation by comparison with IBM mainframe implementation of PL/1 exceptions (ON statement) and multiprocessing/multithreading (TASK type). IBM sometime late 60's/early 70's realized that this requires a tree of stacks to be 100% correct and made appropriate changes to the low-level function call sequences. I venture to guess that the broad C++ community will continue to struggle with the currently prevailing paradigm of collection of stacks (one per thread) and various ad-hoc partially correct solutions where lot of functionality is described in the standards as undefined.

OK, thanks for your patience while I was typing this post through my partially operational connection. Let me know if I need to clarify anything.

[jl2012]

Is there a way to register watched addresses in the standard client?

Not yet. There is a pull request implementing bloom filters that should make that easy to implement.

Or, could 'listunspent' be extended to take any non-wallet address as an optional parameter?

No. The reference implementation doesn't keep a master map of all addresses to unspent transaction outputs, and adding such an index for the small number of services that need to look up arbitrary addresses doesn't make sense.

I'd suggest you -blocknotify and the getblock() RPC call to maintain your own index of address --> unspent txout.

Even getblock is used, it only returns the txid, and gettransaction does not show the tx if is a non-wallet id. What should I do?

Also, is there an ETC for the support of "watch-only addresses"?

[kjj]

Is there a way to register watched addresses in the standard client?

Not yet. There is a pull request implementing bloom filters that should make that easy to implement.

Or, could 'listunspent' be extended to take any non-wallet address as an optional parameter?

No. The reference implementation doesn't keep a master map of all addresses to unspent transaction outputs, and adding such an index for the small number of services that need to look up arbitrary addresses doesn't make sense.

I'd suggest you -blocknotify and the getblock() RPC call to maintain your own index of address --> unspent txout.

Even getblock is used, it only returns the txid, and gettransaction does not show the tx if is a non-wallet id. What should I do?

Also, is there an ETC for the support of "watch-only addresses"?

getrawtransaction will return non-wallet transaction, and you can decode the output with decoderawtransaction

[gmaxwell]

getrawtransaction will return non-wallet transaction, and you can decode the output with decoderawtransaction

It does not for spent transactions in GIT, and won't by default (at least, and probably not at all) in 0.8.

I have to agree with this.  Having bitcoind be a platform that enables other developers and service providers to leverage the "low level plumbing" done by developers with superior knowledge of the "quirks" of the protocol (i.e. Gavin et al) allows it to pay dividends.

Echoing what Jeff said— agreement isn't what will make a difference here, working on it will. Even if you don't have the time or background to contribute specifically the features you want contributing to testing will help get more done. (We appear to be primarily testing constrained, as we'd like to advance the level of software quality in the reference client but have a large preexisting deficit)

For the most part thoughts like this aren't "no don't!" they're "sure, that sounds like a fine idea … someday when someone has time" and anyone technically sophisticated enough to find bitcoin and post in the form can probably do something to help that someday come about a bit sooner.