[PATCH] monitoraddress/blocks

[Gavin Andresen]

This patch adds the following JSON-RPC commands to bitcoin/bitcoind:

• monitoraddress <bitcoinaddress> <url> [monitor=true]
  When coins are sent to <bitcoinaddress> POST JSON transaction info to <url>.
  If <bitcoinaddress> is 'allwallet' then monitor coins sent to all of your addresses.
  Pass false as third param to stop monitoring.
• monitorblocks <url> [monitor=true] [startblockcount=0]
  POST block information to <url> as blocks are added to the block chain.
  [monitor] true will start monitoring, false will stop.
  Pass [startblockcount] to start monitoring at/after block with given blockcount.
• listmonitored
  Returns list describing where blocks and transactions are being POSTed.
• getblock <hash|number>
  Returns details of the block with <hash> (hexadecimal) or <number>.
• gettransaction <hash>
  Returns details of transaction with <hash> (hexadecimal).

This patch also modifies the "sendtoaddress" function so it returns the transaction ID on a successful send (instead of the string "sent").

If you use the monitor* functionality to POST information be sure to think through the security of your application.  For example, if you use monitoraddress to get notified of customer payments you should think about whether or not a customer could fake a payment by POSTing bogus information to your web server.

Full source code is at: http://github.com/gavinandresen/bitcoin-git/tree/monitorreceived
As always, bug reports, suggestions for improvement and feedback is welcome.
Updated monitoraddress/getblock commands as discussed below

[nelisky]

Kudos to you!

I will have to find the time to merge this into my private version, but this opens a whole new world of possibilities to me, in terms of code organization mainly.

[jgarzik]

My preferred arrangement would be a 'monitorall' command, which POSTs each incoming transaction to a URL, regardless of bitcoinaddress, as long as it's a wallet transaction.

[Gavin Andresen]

My preferred arrangement would be a 'monitorall' command...

Good Idea, and that aught to be easy.

I'm thinking it should be:
  monitoraddress allwallet <url>
... instead of a separate monitor command.

Come to think of it, maybe I should combine the getblockby* routines into one; there's no chance of mistaking a 64-character hex block hash for a 5-digit (or, in many years, a six or seven or ten digit) block number.

[jgarzik]

Come to think of it, maybe I should combine the getblockby* routines into one; there's no chance of mistaking a 64-character hex block hash for a 5-digit (or, in many years, a six or seven or ten digit) block number.

I thought about that before, too, but the rpc interface is internally typed.  Sure, we can specify a string for the parameter, and interpret the string as either a block hash or a block number, but satoshi might grumble at the overloading of parameters?  None of the existing RPC commands have multi-use parameters.

[jgarzik]

• getblockbyhash <hash>
  Returns details of the block with <hash> (hexadecimal).
• getblockbynumber <number>
  Returns details of block <number> in the longest block chain.
• gettransaction <hash>
  Returns details of transaction with <hash> (hexadecimal).

Your changes appear to come darned close to superceding my getblock and listtransaction patches.  With a few minor changes, you can completely supercede my patches:

• add listtransactions or gettransactions, calling your txToJSON, for each wallet tx matching the given query parameters
• add the following fields to txToJSON from my patch: bitcoin address, label, category (credit, debit, generated, etc.), amount
• any reason why you didn't use 'getblockbycount' as elsewhere discussed?  I named mine that based on your own logic

[Gavin Andresen]

I just updated the code/patch:
   monitoraddress allwallet <url>
... gets triggered for all transactions that go into your wallet.  That includes 'change' transactions that can occur when you send somebody coins, which I expect will strike some people as a bug and others as an important feature.

And I combined the getblockby methods into one.  I know I would've never remembered  "is it getblockybycount or getblockbynumber or getblockbyheight or getblockbydepth" -- I will remember "getblock".

[jgarzik]

I just updated the code/patch:
   monitoraddress allwallet <url>
... gets triggered for all transactions that go into your wallet.  That includes 'change' transactions that can occur when you send somebody coins, which I expect will strike some people as a bug and others as an important feature.

And I combined the getblockby methods into one.  I know I would've never remembered  "is it getblockybycount or getblockbynumber or getblockbyheight or getblockbydepth" -- I will remember "getblock".

+3 all great stuff.

I strongly prefer this new 'getblock' arrangement of yours.  My only question was whether satoshi might not like it.

[nelisky]

Any chance of getting a patch against the vanilla svn? I have a heavily tweaked code base (cuda and a collection of patches).

[Gavin Andresen]

I should have been clear:  this is a patch against the latest 'vanilla' svn.

[nelisky]

I should have been clear:  this is a patch against the latest 'vanilla' svn.

Oh, well, thanks again Boy, you're fast...

[jgarzik]

... gets triggered for all transactions that go into your wallet.  That includes 'change' transactions that can occur when you send somebody coins, which I expect will strike some people as a bug and others as an important feature.

BTW...  how do change transactions work?  what bitcoin address is chosen to receive the change?

[Gavin Andresen]

Today:  a brand-new keypair (address) is created and added to your wallet when you have change.

Soon (I hope; I think satoshi is working on it...) bitcoin will pre-generate a bunch of addresses to use for change, and use one of them (and re-generate a bunch when it runs out).

I'd kind of like an option to put change back into one of the addresses it came from; that'd be simpler, quicker, and would make your wallet smaller...

[mizerydearia]

gavin: Would you be willing to provide a patch file for monitoraddress/blocks so that I may include it in my gentoo linux ebuild?

Here it is

RE: keeping it current:  If you have git, this would work:
  # First time:
  git clone git://github.com/gavinandresen/bitcoin-git.git
  cd bitcoin-git
  git diff --no-prefix origin/svn origin/monitorreceived

  # Subsequent times:
  cd bitcoin-git
  git fetch
  git diff --no-prefix origin/svn origin/monitorreceived

Thanks gavin

[mizerydearia]

gavin: A question: If I continuously generate a new diff of your source code compared to latest svn repository and claim that as a patch, wouldn't that additionally produce a patch to an outdated or previous version?  e.g. if your patch initially worked with svn 158 (example) and currently latest is svn 500 (example), producing a patch comparing your source to latest svn and making it as a patch would then convert the bitcoin client to be very outdated and similar to svn 158.  This is my understanding.  Is this accurate/true?

As of right now the patch you supplied is not compatible with svn 161 and while I could technically create a diff comparison to generate a new patch, I fear that it will additionally revert changes that are unrelated to monitor functionality and ultimately produce a monitor-specific patch that reverts data not related to monitor capabilities.

Any suggestion on how to proceed in maintaining/preserving updated patch for this feature is appreciated.

gavin: If you are too busy to maintain a patch, perhaps someone could take over?  Is a btc bounty/donation helpful?  I, for one, will offer 5btc towards someone who will actively maintain this patch.

-Update-

Here is my attempt to produce a patch for Bitcoin svn 161:

I have established Bitcoin SVN in /downloads/svn/bitcoin/.
I have established gavin's Bitcoin-git in /downloads/git/bitcoin-git/.

Reviewing gavin's initial patch I know that only the following files are patched:
db.cpp
db.h
init.cpp
main.cpp
main.h
net.cpp
rpc.cpp
rpc.h

So I will generate a diff for only these files and additionally I will manually review each diff to make sure it is similar to gavin's submitted patch and does not include reverting unrelated changes due to further changes in Bitcoin svn.

db.cpp:
LC_ALL=C TZ=UTC0 diff -Napur /downloads/svn/bitcoin/trunk/db.cpp /downloads/git/bitcoin-git/db.cpp
Comparing this diff to gavin's I see

Code:

-        if (pindex->nHeight < 74000 && !mapArgs.count("-checkblocks"))
+        if (pindex->nHeight < nBestHeight-2500 && !mapArgs.count("-checkblocks"))

which I believe should not be included in the patch.  I will figure out how to manually exclude this and to preserve the patch as being correct in syntax.  Yep, it's unnecessary, and removing that block of patch, the remaining block of patch for db.cpp is identical.

Humm, manually modifying `diff`s is difficult, especially with rpc.cpp

Okay, yeah, I definitely give up.  I was doing pretty well until rpc.cpp when I realized that gavin's code contains WAY TOO MANY other patches besides only monitoraddress and monitorblocks and I have no idea how to remove these pieces from the patch data.

Additionally, using gavin's git-diff suggestion wasn't usable considering the svn code in his git repo is outdated.

If it is of any help, here is a manually compiled collection of diffs (using same switches as link above): http://pastebin.com/tN7hRZWH up to and not including rpc.cpp  (check file list above)

[doublec]

Here's another approach to automatically generate a patch (or an update repository). Clone Gavin's repository and create a branch for your own changes based on origin/monitorreceived. Then rebase your branch on top of origin/svn. Git will then automagically apply the changes in the monitorreceived branch on top of the new svn changes. You can then either work on your new branch or generate a diff to get monitorreceived's changes only. Here's the steps:

git clone git://github.com/gavinandresen/bitcoin-git.git
cd bitcoin-git
git checkout -b mybranch origin/monitorreceived
git rebase origin/svn
...now you can build the new code updated to svn...

To get a diff:

git diff -U8 origin/svn >myfile.patch

I just tested and the rebase occurs without requiring use intervention.

You also merge if you prefer instead of rebasing:

git clone git://github.com/gavinandresen/bitcoin-git.git
cd bitcoin-git
git checkout -b mybranch origin/monitorreceived
git merge origin/svn

[mizerydearia]

Code:

$ git checkout -b mybranch origin/monitorreceived
Branch mybranch set up to track remote branch monitorreceived from origin.
Switched to a new branch 'mybranch'
$ git rebase origin/svn
First, rewinding head to replay your work on top of it...
Applying: Clean monitorreceived branch
Applying: Update README.md
Using index info to reconstruct a base tree...
<stdin>:9: new blank line at EOF.
+
warning: 1 line adds whitespace errors.
Falling back to patching base and 3-way merge...
No changes -- Patch already applied.
Applying: Renaming monitorreceivedbyaddress, adding monitorblocks, listmonitored
Applying: Send proper JSON-RPC syntax, and refactored code a bit
Applying: monitorblocks is working
Applying: Use boost::xpressive for regex support (eliminates linking with regex lib)
Applying: Oops, need to cast command-line params to correct type
Applying: Tweak some code, and tweaked listmonitored a bit
Applying: sendtoaddress return txid instead of 'sent'
Applying: Implemented getblockbyhash/number and gettransaction
Applying: monitoraddress allwallet <url>  to monitor all to-wallet transactions.
Applying: Replace getblockbyhash/number with single getblock that takes hash or number

...now you can build the new code updated to svn...

I do not understand this step.  Did my previous steps update Bitcoin svn code to revision 161?  I do not understand "rebasing."

I am a git newbie and therefore it may be worthwhile or better time spent for someone who is experienced to maintain or produce a patch.  I could learn, but I have other priorities I must tend to.

[doublec]

I do not understand this step.  Did my previous steps update Bitcoin svn code to revision 161?  I do not understand "rebasing."

Poor wording on my part. At that step (after you've done the 'git rebase or git merge') you have a repository equivalent to svn revision 161 with the monitor changes applied. The 'git diff' command that follows produces a diff of only the monitor changes.

[doublec]

With the existing patch (and the git repository) if you call monitoraddress with a monitor value of 'false' (to remove a monitor) but pass a URL that is not being monitored then the bitcoin daemon process will exit. Possibly it's indexing into the map of monitor URL's, not finding it and throwing an exception.

[Gavin Andresen]

I updated most of my git branches to svn rev 161 last night, including monitorreceived.  That's easy:
   git svn fetch   (but you have to have an "svn-remote" setup in your .git/config)
   git merge refs/remotes/svn/trunk    (I could git rebase instead, but merge seems less magical to me)
    ... fix any merge conflicts and test then git commit, if needed  (usually there are none)
   git push   (to push up to github)

Repeated on each branch (I'll try to keep svn, svnTEST and monitorreceived up-to-date).

Keeping the patch file up-to-date is another couple of steps, and I could/should automate it.

doublec: thanks for the bug report, I'll look into it as soon as I get this machine setup.