CheapSweep: a script for low-cost address sweeping

[salfter]

I had a bunch of small transactions accumulated from mining, change, the various free-bitcoin sources, etc. and wanted to consolidate all of them onto one address.  I tried sweeping them with Armory, but it choked on the transaction when I went to offline-sign it.  Even before that, though, it said it was going to cost BTC0.024 to send, which seems a bit out of line.  I tried importing the privkeys to bitcoind and sending them directly from there, but it also insisted on a hefty fee.  I tried the bitcoin-nftf fork; it didn't work any better.

I then tried crafting a raw transaction of 66 inputs totaling about BTC0.025, one output, and a BTC0.0001 fee. bitcoind accepted it; after about three hours, it was confirmed. I'm not in a hurry, so this is acceptable. If you need faster turnaround, this script is probably not for you.

What follows is a shell script to automate this process.  You pick one or more source addresses from your wallet and a destination address that may or may not be in your wallet.  The fee is up to you.  By default, it will sweep inputs with at least 6 confirmations to the destination address; if you're sweeping from an address that receives generated coin from P2Pool, you'll probably want to include "-c 120" to avoid trying to send immature coin. The only dependency is a running bitcoind (0.7 or later).  The rest is fairly standard shell-script programming, with grep, sed, tr, and bc doing most of the munging.

Code:

#!/bin/bash

# CheapSweep v0.1
# Scott Alfter
# scott@alfter.us
# Donations: 1TipSAXbE6owdU24bcBDJKmL8JRxQe5Yu

help()
{
cat <<EOF >&2
Usage: $0 [options] -d destaddr addr1 addr2 ...

options: -d|--destaddr  destination address (REQUIRED)
         -f|--fee       fee to subtract from inputs (default: 0)
         -c|--confirm   minimum confirmations to include input (default: 6)
         -n|--no-send   don't send; dump the raw transaction to stdout
EOF
}

fee=0.0
minconfirm=6
OPTS=$(getopt -o d:f:c:hn --long destaddr:,fee:,confirm:,help,no-send -- "$@")
eval set -- "$OPTS"
while true; do
  case "$1" in
    -d|--destaddr)  destaddr="$2"; shift 2;;
    -f|--fee)       fee="$2"; shift 2;;
    -c|--confirm)   minconfirm="$2"; shift 2;;
    -n|--no-send)   nosend=1; shift 1;;
    -h|--help)      help; exit 1;;
    --)             shift; break;;
    *)              echo "Internal error"; exit 1;;
  esac
done
if [ "$destaddr" == "" ]
then
  help
  exit 1
fi

addrs=$(echo $* | sed "s/^/[\"/;s/ /\",\"/g;s/\$/\"]/")
total=$(bitcoind listunspent $minconfirm 21000000 $addrs | grep amount | sed "s/.*: //;s/,//" | tr "\n" "+" | sed "s/+\$/\n/" | bc)
total=$(echo $total - $fee | bc)

tx=$(bitcoind signrawtransaction $(bitcoind createrawtransaction [$(bitcoind listunspent $minconfirm 21000000 $addrs | egrep "txid|vout" | sed "s/\"txid/{\"txid/;s/\"vout\" : \([0-9]*\),/\"vout\" : \1},/" | tr -d "\n" | tr -d " " | sed "s/,\$//")] {\"$destaddr\":$total}) | grep \"hex\" | sed "s/.*: \"//;s/\",//")

if [ "$nosend" != "" ]
then
  echo $tx
else
  bitcoind sendrawtransaction $tx
fi

[caffeinewriter]

Very nice And quite a simple script as well. Bravo.

[Stephen Gornick]

What follows is a shell script to automate this process.

Very cool.  Every so often someone wants to clean out their wallet or redeem the losing bets on SatoshiDICE and this will be the tool I suggest to them.  Thanks for sharing this!

[salfter]

One more tip: If you use my script, you might want to make this change in ~/.bitcoin/bitcoin.conf, as recommended in https://en.bitcoin.it/wiki/Free_transaction_relay_policy:

Code:

addnode=173.242.112.53

[salfter]

...and some statistics regarding the transfers I made while figuring this out:

https://docs.google.com/spreadsheet/ccc?key=0AhDiish0IKqvdHpUOVQzZDZQbS1LMWRFNmFvRXYtVVE

24.253.13.34 is the router for my home network; my mining rig is on this network, and it's where CheapSweep was developed and tested. 173.242.112.67 is most likely on the same subnet as 173.242.112.53, which the Bitcoin wiki recommends for relaying free transactions.  Two other IPs also appear.

Also interesting: Eclipse mined five of the six blocks containing my transactions.

[Prospero]

I'm using 0.8 beta of bitcoind and the script fails to work because I think the JSON format might have changed. The script looks like it is expecting each "amount: $num" to be on a separate line, but in 0.8 the JSON format is compact so there are no newlines. I think right before `grep amount`, `perl -pe 's/,/,\n/g'` (or equivalent) should be used to add a newline after every comma.

[DeathAndTaxes]

Eventually it would be nice for scripts like this to work their way into the clients.   "It appears you wallet has become fragmented.  Would you like to consolidate some coins.  The coins will be unavailable until confirmed".

Anything that reduces the size of the pruned database is a positive.  It would be optimal if miners could detect these types of transactions (where the size of the output is smaller than the size of the input) and allow them as space allows even with no fee. 

EVERYONE (including miners) benefits from reducing the size of the pruned database.

[Bitsky]

If coincontrol would make it into the main client, users could even select which addresses should be defragmented. It shouldn't be too complex to display the fragmentation for each address.
And if the single recipient address is also in the source address list then it should be obvious that it's a coin-combination (which could be fee-free so users agree to it).

[salfter]

I'm using 0.8 beta of bitcoind and the script fails to work because I think the JSON format might have changed. The script looks like it is expecting each "amount: $num" to be on a separate line, but in 0.8 the JSON format is compact so there are no newlines. I think right before `grep amount`, `perl -pe 's/,/,\n/g'` (or equivalent) should be used to add a newline after every comma.

Hmm...haven't tried that version yet.  I'll have to look into it.

[etotheipi]

Does bitcoind RPC interface broadcast arbitrary transactions for you?  My understanding is that "stock" bitcoind will not accept such a 50 kB transaction without a substantial fee -- it would be DOA.  Obviously other nodes might use different fee rules, but I want to understand why your own bitcoind took it? 

Also, the latest version of Armory has coin-control -- but only at the address level.  If your coins are split up between lots of addresses, you can do this house-cleaning, though you'll have to pay a full fee, because stock bitcoind won't accept it and forward it if you don't.

[Prospero]

It sounds like this ticket might be related to this: https://github.com/bitcoin/bitcoin/issues/1675

[K1773R]

just tested this and sadly it dosnt work.
$addrs and $totals arent calucalted correctly, i tested it with testnet (yes i changed the script apropriately):

Code:

$ ./cheapsweep.sh -n -f 0 -d n1vCS8kqNQJSnFu3xA1ma64RafUEZ7oAtJ `bitcoind -testnet -rpcport=18334 listaddressgroupings | grep '",' | sed "s/            \"//" | sed "s/\",//" | xargs echo`
error: Error parsing JSON:c
error: Error parsing JSON:c
error: {"code":-3,"message":"Invalid amount"}
error: {"code":-1,"message":"signrawtransaction <hex string> [{\"txid\":txid,\"vout\":n,\"scriptPubKey\":hex},...] [<privatekey1>,...] [sighashtype=\"ALL\"]\nSign inputs for raw transaction (serialized, hex-encoded).\nSecond optional argument (may be null) is an array of previous transaction outputs that\nthis transaction depends on but may not yet be in the blockchain.\nThird optional argument (may be null) is an array of base58-encoded private\nkeys that, if given, will be the only keys used to sign the transaction.\nFourth optional argument is a string that is one of six values; ALL, NONE, SINGLE or\nALL|ANYONECANPAY, NONE|ANYONECANPAY, SINGLE|ANYONECANPAY.\nReturns json object with keys:\n  hex : raw transaction with signature(s) (hex-encoded string)\n  complete : 1 if transaction has a complete set of signature (0 if not)"}

$total is 0 and $address is "c"

[salfter]

Does bitcoind RPC interface broadcast arbitrary transactions for you?

If you create raw transactions, the usual checks AFAICT are bypassed. I've sent transactions up to about 10K or so with tiny fees (BTC0.0001-BTC0.0005).  I'm not sure how big my latest zero-fee transactions have been, but they've all gone through within (usually) 1-3 hours.

[salfter]

I'm using 0.8 beta of bitcoind and the script fails to work because I think the JSON format might have changed. The script looks like it is expecting each "amount: $num" to be on a separate line, but in 0.8 the JSON format is compact so there are no newlines. I think right before `grep amount`, `perl -pe 's/,/,\n/g'` (or equivalent) should be used to add a newline after every comma.

Something must've changed in the past few weeks, as now that I've gotten a chance to test CheapSweep against a beta build, it's working properly.  I made no changes to the script...just had it sweep some free coins a little bit ago.  The bitcoind I'm running was built from whatever was on GitHub last Friday.

[Prospero]

I think the issue is that there are many versions of sed and it's hard to maintain portability between the versions. Even replacing sed with gsed (I'm on a Mac) didn't help here. But after translating to perl I got reasonable results. I haven't translated the last call to sed yet.

Code:

# addrs=$(echo $* | sed "s/^/[\"/;s/ /\",\"/g;s/\$/\"]/")
addrs=$(echo $* | perl -ne 'printf "[%s]", join ",", map qq("$_"), split " "')
# total=$(bitcoind listunspent $minconfirm 21000000 "$addrs" | grep amount | sed "s/.*: //;s/,//" | tr "\n" "+" | sed "s/+\$/\n/")
total=$(
    bitcoind listunspent $minconfirm 21000000 "$addrs" \
        | perl -ne '($a)=/^\s*"amount" : (\d+\.\d+),$/; $a||next;$sum+=$a } END {print $sum'
)

[K1773R]

I think the issue is that there are many versions of sed and it's hard to maintain portability between the versions. Even replacing sed with gsed (I'm on a Mac) didn't help here. But after translating to perl I got reasonable results. I haven't translated the last call to sed yet.

Code:

# addrs=$(echo $* | sed "s/^/[\"/;s/ /\",\"/g;s/\$/\"]/")
addrs=$(echo $* | perl -ne 'printf "[%s]", join ",", map qq("$_"), split " "')
# total=$(bitcoind listunspent $minconfirm 21000000 "$addrs" | grep amount | sed "s/.*: //;s/,//" | tr "\n" "+" | sed "s/+\$/\n/")
total=$(
    bitcoind listunspent $minconfirm 21000000 "$addrs" \
        | perl -ne '($a)=/^\s*"amount" : (\d+\.\d+),$/; $a||next;$sum+=$a } END {print $sum'
)

unfortunately i cant code in perl, are you planing to complete it?

[Prospero]

unfortunately i cant code in perl, are you planing to complete it?

It turns out the last statement works as is. I didn't test it before because I assumed I had to translate it also. So the only statements I needed to change are in my above post.

[K1773R]

unfortunately i cant code in perl, are you planing to complete it?

It turns out the last statement works as is. I didn't test it before because I assumed I had to translate it also. So the only statements I needed to change are in my above post.

EDIT: err, nvm

[K1773R]

i just found the problem: [ and ] have to be escaped

Code:

tx=$(bitcoind signrawtransaction $(bitcoind createrawtransaction \[$(bitcoind listunspent $minconfirm 21000000 $addrs | egrep "txid|vout" | sed "s/\"txid/{\"txid/;s/\"vout\" : \([0-9]*\),/\"vout\" : \1},/" | tr -d "\n" | tr -d " " | sed "s/,\$//")\] {\"$destaddr\":$total}) | grep \"hex\" | sed "s/.*: \"//;s/\",//")

[K1773R]

i updated the script so it works with linux:

Code:

#!/bin/bash

# CheapSweep v0.1
# Scott Alfter
# scott@alfter.us
# Donations: 1TipSAXbE6owdU24bcBDJKmL8JRxQe5Yu

BITCOIND="bitcoind"
#BITCOIND="bitcoind -testnet -rpcport=18334"

help()
{
cat <<EOF >&2
Usage: $0 [options] -d destaddr addr1 addr2 ...

options: -d|--destaddr  destination address (REQUIRED)
         -f|--fee       fee to subtract from inputs (default: 0)
         -c|--confirm   minimum confirmations to include input (default: 6)
         -n|--no-send   dont send; dump the raw transaction to stdout
EOF
}

fee=0.0
minconfirm=6
OPTS=$(getopt -o d:f:c:hn --long destaddr:,fee:,confirm:,help,no-send -- "$@")
eval set -- "$OPTS"
while true; do
  case "$1" in
    -d|--destaddr)  destaddr="$2"; shift 2;;
    -f|--fee)       fee="$2"; shift 2;;
    -c|--confirm)   minconfirm="$2"; shift 2;;
    -n|--no-send)   nosend=1; shift 1;;
    -h|--help)      help; exit 1;;
    --)             shift; break;;
    *)              echo "Internal error"; exit 1;;
  esac
done
if [ "$destaddr" == "" ]
then
  help
  exit 1
fi

addrs=$(echo "$*" | sed "s/^/\[\"/;s/ /\",\"/g;s/\$/\"\]/")
echo "$addrs";

total=$($BITCOIND listunspent $minconfirm 21000000 "$addrs" | grep amount | sed "s/.*: //;s/,//" | tr "\n" "+" | sed "s/+\$/\n/" | bc)
total=$(echo $total - $fee | bc)
echo $total;

tx=$($BITCOIND signrawtransaction $($BITCOIND createrawtransaction \[$($BITCOIND listunspent $minconfirm 21000000 "$addrs" | egrep "txid|vout" | sed "s/\"txid/{\"txid/;s/\"vout\" : \([0-9]*\),/\"vout\" : \1},/" | tr -d "\n" | tr -d " " | sed "s/,\$//")\] {\"$destaddr\":$total}) | grep \"hex\" | sed "s/.*: \"//;s/\",//")

if [ "$nosend" != "" ]
then
  echo $tx
else
  $BITCOIND sendrawtransaction $tx
fi