Bitcoin Forum
December 14, 2017, 07:26:40 AM *
News: Latest stable version of Bitcoin Core: 0.15.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: Raw TX API & other private keys: 'watch' or 'listunspent' other addresses?  (Read 1492 times)
HanSolo
Jr. Member
*
Offline Offline

Activity: 59


Don't everyone thank me at once.


View Profile
October 13, 2012, 04:51:33 AM
 #1

Looks like the raw transactions API could be useful to run a connected client, but keep signing keys elsewhere. (Elsewhere = only  occasionally connected in some limited hardened way.)

This would be easier if the connected client was able to 'listunspent' outputs available to arbitrary addresses.

Is there a way to register watched addresses in the standard client?

Or, could 'listunspent' be extended to take any non-wallet address as an optional parameter?

Never tell me the odds. ⓑ 1J5wQvCz2mn3f4TxPFtExKs253GyeKKkBb
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1513236400
Hero Member
*
Offline Offline

Posts: 1513236400

View Profile Personal Message (Offline)

Ignore
1513236400
Reply with quote  #2

1513236400
Report to moderator
Gavin Andresen
Legendary
*
qt
Offline Offline

Activity: 1652


Chief Scientist


View Profile WWW
October 13, 2012, 01:08:41 PM
 #2

Is there a way to register watched addresses in the standard client?
Not yet. There is a pull request implementing bloom filters that should make that easy to implement.
Quote
Or, could 'listunspent' be extended to take any non-wallet address as an optional parameter?
No. The reference implementation doesn't keep a master map of all addresses to unspent transaction outputs, and adding such an index for the small number of services that need to look up arbitrary addresses doesn't make sense.

I'd suggest you -blocknotify and the getblock() RPC call to maintain your own index of address --> unspent txout.


How often do you get the chance to work on a potentially world-changing project?
CIYAM
Legendary
*
Offline Offline

Activity: 1862


Ian Knowles - CIYAM Lead Developer


View Profile WWW
October 13, 2012, 01:14:35 PM
 #3

No. The reference implementation doesn't keep a master map of all addresses to unspent transaction outputs, and adding such an index for the small number of services that need to look up arbitrary addresses doesn't make sense.

I understand the point but I think that as fewer and fewer people other than miners and services end up choosing to use the "reference implementation" (due to the size of the block chain and time it takes to process) wouldn't having such an index (as an option) make at least a little sense (which would only come into effect after a rescan assuming the option wasn't originally turned on)?

My view is that the "reference implementation" is actually what we want to make things easy and possible for services (rather than each having to roll their own). If this is wrong then why bother with the "raw tx" API at all (as certainly no normal end-user will ever use it)?

With CIYAM anyone can create 100% generated C++ web applications in literally minutes.

GPG Public Key | 1ciyam3htJit1feGa26p2wQ4aw6KFTejU
TangibleCryptography
Sr. Member
****
Offline Offline

Activity: 476


Tangible Cryptography LLC


View Profile WWW
October 13, 2012, 02:36:56 PM
 #4

No. The reference implementation doesn't keep a master map of all addresses to unspent transaction outputs, and adding such an index for the small number of services that need to look up arbitrary addresses doesn't make sense.

I understand the point but I think that as fewer and fewer people other than miners and services end up choosing to use the "reference implementation" (due to the size of the block chain and time it takes to process) wouldn't having such an index (as an option) make at least a little sense (which would only come into effect after a rescan assuming the option wasn't originally turned on)?

My view is that the "reference implementation" is actually what we want to make things easy and possible for services (rather than each having to roll their own). If this is wrong then why bother with the "raw tx" API at all (as certainly no normal end-user will ever use it)?


I have to agree with this.  Having bitcoind be a platform that enables other developers and service providers to leverage the "low level plumbing" done by developers with superior knowledge of the "quirks" of the protocol (i.e. Gavin et al) allows it to pay dividends.

I think that is something which is really missing (and note this isn't a complaint I haven't built it so I can't complain that others haven't) but rather just an observation.  At the same time the bitcoin-qt becomes less useful to end users due to the increasing requirements of full nodes, it becomes even more valuable as a platform for developers to build higher level services and tools.


Just throwing out an idea here but I think the GUI needs to be decoupled from the "bitcoin engine".   Imagine a scenario where there are three related projects.  

The bitcoin class library (think openSSL, or bouncy castle but a bitcoin equivelent).  Contains the data structures and algorithms necessary to communicate with the protocol and store, organize, and access blockchain data.

The bitcoin reference engine is a non-GUI implementation of the bitcoin class library and handles communication between a client (any client) and the network.

The bitcoin-qt (GUI client) is branched off and handles the high level implementation of user interaction and uses the bitcoin engine.  This would allow other GUI developers to be able to build drop in replacements leveraging the work already done in the engine.

Once again just an observation but it seems to me the "bitcoin reference client" is really three projects tightly coupled. That makes building derivative works difficult and limits the feature-set that the lower level functionality can provide.  Everything is seen through the lens of what is best for this GUI client, a client that is increasingly not the optimal solution for manner users.

WARNING:  The above was written before I had my morning dose of caffeine so may be semi-incoherent.  
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
October 13, 2012, 05:31:19 PM
 #5

The GUI is already decoupled from the "engine" as demonstrated by how bitcoind and bitcoin-qt are built from the same source code.  bitcoind is the base, built with zero GUI libraries.  bitcoin-qt is simply "bitcoind + GUI"

In general, there is wide agreement that separation should be improved, along similar lines like ones you suggest:
  • core class libraries
  • blockchain engine
  • wallet handling
  • graphical user interface

Separating out the class libraries would enable more applications to share the core code.
Separating out wallet handling into a separate process (and executable) will greatly increase security, ensuring that any bug in the P2P network triggered by a remote attacker would never overflow or corrupt the wallet process.
etc.

But as >90% of the full nodes are the reference implementation -- which remains the most complete and secure implementation to date -- keeping the network running securely has to take priority over these other things.

If you know C++... help!  Unlike what the tinfoil hats would have you believe, the dev team is still volunteers working on our own time (hopefully Gavin will be funded by BF real soon now, easing the pressure a bit).  It is easy to join the dev team, too:  just submit a useful pull request.  No forms to sign or anything Smiley

So file this under...   we agree with you, but need more developer resources and bandwidth to make that happen.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
BCB
CTG
VIP
Legendary
*
Offline Offline

Activity: 1050


BCJ


View Profile
October 13, 2012, 05:43:35 PM
 #6

"tinfoil hats" == Atlas et el
HanSolo
Jr. Member
*
Offline Offline

Activity: 59


Don't everyone thank me at once.


View Profile
October 13, 2012, 07:12:28 PM
 #7

Is there a way to register watched addresses in the standard client?
Not yet. There is a pull request implementing bloom filters that should make that easy to implement.
Quote
Or, could 'listunspent' be extended to take any non-wallet address as an optional parameter?
No. The reference implementation doesn't keep a master map of all addresses to unspent transaction outputs, and adding such an index for the small number of services that need to look up arbitrary addresses doesn't make sense.

I'd suggest you -blocknotify and the getblock() RPC call to maintain your own index of address --> unspent txout.


Yes, external indexing by watching blocks should be a suitable workaround.

Thank you for a quick and definitive answer!

Never tell me the odds. ⓑ 1J5wQvCz2mn3f4TxPFtExKs253GyeKKkBb
2112
Legendary
*
Offline Offline

Activity: 1988



View Profile
October 14, 2012, 02:25:05 AM
 #8

The GUI is already decoupled from the "engine" as demonstrated by how bitcoind and bitcoin-qt are built from the same source code.
Actualy that is not true. bitcoind is a functional subset of bitcoin-qt in the core functionality of "sendtoaddress". And bitcoin-qt still suffers from the inversion of control problem.

I just checked the master github:
Code:
static bool noui_ThreadSafeAskFee(int64 nFeeRequired, const std::string& strCaption)
{
return true;
}
So the inversion of control in the fee calculation/addition is still there.

This isn't a patch or pull request. It is an architectural issue that requires extensive changes.

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
Gavin Andresen
Legendary
*
qt
Offline Offline

Activity: 1652


Chief Scientist


View Profile WWW
October 14, 2012, 02:37:56 AM
 #9

This isn't a patch or pull request. It is an architectural issue that requires extensive changes.
Right, and we don't accept "change the world" pull requests because the risks of introducing a catastrophic bug outweigh the benefits, and rewriting everything invalidates all of the careful code review that's been done over the past few years.

However, we ARE moving towards better architecture as different parts of the code get worked on to support new features or improve scalability or fix bugs.  For an example of inversion-of-control done right, look for use of boost signals and slots; for example:
Code:
boost::signals2::signal<void (CWallet *wallet, const uint256 &hashTx, ChangeType status)> NotifyTransactionChanged;

How often do you get the chance to work on a potentially world-changing project?
2112
Legendary
*
Offline Offline

Activity: 1988



View Profile
October 14, 2012, 03:42:28 AM
 #10

Right, and we don't accept "change the world" pull requests because the risks of introducing a catastrophic bug outweigh the benefits, and rewriting everything invalidates all of the careful code review that's been done over the past few years.

However, we ARE moving towards better architecture as different parts of the code get worked on to support new features or improve scalability or fix bugs.  For an example of inversion-of-control done right, look for use of boost signals and slots; for example:
Code:
boost::signals2::signal<void (CWallet *wallet, const uint256 &hashTx, ChangeType status)> NotifyTransactionChanged;
I'm 100% with you on both issues: I understand that you need maintain the continuity of safety and I understand that you need to build new features in the incremental way.

But during the year since I made the inversion-of-control observation I had a very productive and educational conversation with slush in the Stratum thread. The implicit knowledge from that thread is that there is a wide gap between the stated knowledge of the Bitcoin application programmers using PR-languages (Perl,PHP,Python,Ruby,etc) and their actual skill. slush never stated it explicitly, but I had to infer it from the discrepancy of the discussion in that thread and various blog posts of the people trying to interface with both Stratum-engine and bitcoind RPC-interface.

So from the discussion with slush I learned that in order to have a decent chance at achieving full-application-stack correctness you'll need to somewhat dumb down the ideal solution. Otherwise the impedance mismatch will be still there it will just move upward into the higher tiers of the application. Your best architectural choices will be of little value if the application programmers will not be able to correctly apply them in their end-user interfaces.

To this extent I would suggest somewhat less ambitious additions to sendtoaddress: (1) upper limit of the fee (2) explicit random seed for the stochastic knapsack solver in the SelectCoins. I think (1) is fairly obvious. (2) will allow correct (and testable!) implementation of the iterative choice of a fee.

Finally, I have to say that I do not share your optimism about the applicability and suitability of boost. I have a deep feeling that the C++/boost community will struggle for many years without 100% correct implementation of exception handling in the multithreaded environment. I make this observation by comparison with IBM mainframe implementation of PL/1 exceptions (ON statement) and multiprocessing/multithreading (TASK type). IBM sometime late 60's/early 70's realized that this requires a tree of stacks to be 100% correct and made appropriate changes to the low-level function call sequences. I venture to guess that the broad C++ community will continue to struggle with the currently prevailing paradigm of collection of stacks (one per thread) and various ad-hoc partially correct solutions where lot of functionality is described in the standards as undefined.

OK, thanks for your patience while I was typing this post through my partially operational connection. Let me know if I need to clarify anything.

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
jl2012
Legendary
*
Offline Offline

Activity: 1750


View Profile
January 10, 2013, 02:19:27 AM
 #11

Is there a way to register watched addresses in the standard client?
Not yet. There is a pull request implementing bloom filters that should make that easy to implement.
Quote
Or, could 'listunspent' be extended to take any non-wallet address as an optional parameter?
No. The reference implementation doesn't keep a master map of all addresses to unspent transaction outputs, and adding such an index for the small number of services that need to look up arbitrary addresses doesn't make sense.

I'd suggest you -blocknotify and the getblock() RPC call to maintain your own index of address --> unspent txout.



Even getblock is used, it only returns the txid, and gettransaction does not show the tx if is a non-wallet id. What should I do?

Also, is there an ETC for the support of "watch-only addresses"?

Donation address: 374iXxS4BuqFHsEwwxUuH3nvJ69Y7Hqur3 (Bitcoin ONLY)
LRDGENPLYrcTRssGoZrsCT1hngaH3BVkM4 (LTC)
PGP: D3CC 1772 8600 5BB8 FF67 3294 C524 2A1A B393 6517
kjj
Legendary
*
Offline Offline

Activity: 1302



View Profile
January 10, 2013, 02:42:30 AM
 #12

Is there a way to register watched addresses in the standard client?
Not yet. There is a pull request implementing bloom filters that should make that easy to implement.
Quote
Or, could 'listunspent' be extended to take any non-wallet address as an optional parameter?
No. The reference implementation doesn't keep a master map of all addresses to unspent transaction outputs, and adding such an index for the small number of services that need to look up arbitrary addresses doesn't make sense.

I'd suggest you -blocknotify and the getblock() RPC call to maintain your own index of address --> unspent txout.



Even getblock is used, it only returns the txid, and gettransaction does not show the tx if is a non-wallet id. What should I do?

Also, is there an ETC for the support of "watch-only addresses"?

getrawtransaction will return non-wallet transaction, and you can decode the output with decoderawtransaction

p2pcoin: a USB/CD/PXE p2pool miner - 1N8ZXx2cuMzqBYSK72X4DAy1UdDbZQNPLf - todo
I routinely ignore posters with paid advertising in their sigs.  You should too.
gmaxwell
Moderator
Legendary
*
qt
Offline Offline

Activity: 2366



View Profile
January 10, 2013, 03:14:49 AM
 #13

getrawtransaction will return non-wallet transaction, and you can decode the output with decoderawtransaction
It does not for spent transactions in GIT, and won't by default (at least, and probably not at all) in 0.8.

I have to agree with this.  Having bitcoind be a platform that enables other developers and service providers to leverage the "low level plumbing" done by developers with superior knowledge of the "quirks" of the protocol (i.e. Gavin et al) allows it to pay dividends.
Echoing what Jeff said— agreement isn't what will make a difference here, working on it will. Even if you don't have the time or background to contribute specifically the features you want contributing to testing will help get more done. (We appear to be primarily testing constrained, as we'd like to advance the level of software quality in the reference client but have a large preexisting deficit)

For the most part thoughts like this aren't "no don't!" they're "sure, that sounds like a fine idea … someday when someone has time" and anyone technically sophisticated enough to find bitcoin and post in the form can probably do something to help that someday come about a bit sooner.

Bitcoin will not be compromised
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!