Bitcoin Forum
June 16, 2024, 08:04:37 PM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1] 2 »  All
  Print  
Author Topic: CheapSweep: a script for low-cost address sweeping  (Read 9004 times)
salfter (OP)
Hero Member
*****
Offline Offline

Activity: 651
Merit: 501


My PGP Key: 92C7689C


View Profile WWW
November 29, 2012, 04:55:17 PM
Last edit: December 04, 2012, 06:44:09 PM by salfter
 #1

I had a bunch of small transactions accumulated from mining, change, the various free-bitcoin sources, etc. and wanted to consolidate all of them onto one address.  I tried sweeping them with Armory, but it choked on the transaction when I went to offline-sign it.  Even before that, though, it said it was going to cost BTC0.024 to send, which seems a bit out of line.  I tried importing the privkeys to bitcoind and sending them directly from there, but it also insisted on a hefty fee.  I tried the bitcoin-nftf fork; it didn't work any better.

I then tried crafting a raw transaction of 66 inputs totaling about BTC0.025, one output, and a BTC0.0001 fee. bitcoind accepted it; after about three hours, it was confirmed. I'm not in a hurry, so this is acceptable. If you need faster turnaround, this script is probably not for you.

What follows is a shell script to automate this process.  You pick one or more source addresses from your wallet and a destination address that may or may not be in your wallet.  The fee is up to you.  By default, it will sweep inputs with at least 6 confirmations to the destination address; if you're sweeping from an address that receives generated coin from P2Pool, you'll probably want to include "-c 120" to avoid trying to send immature coin. The only dependency is a running bitcoind (0.7 or later).  The rest is fairly standard shell-script programming, with grep, sed, tr, and bc doing most of the munging.

Code:
#!/bin/bash

# CheapSweep v0.1
# Scott Alfter
# scott@alfter.us
# Donations: 1TipSAXbE6owdU24bcBDJKmL8JRxQe5Yu

help()
{
cat <<EOF >&2
Usage: $0 [options] -d destaddr addr1 addr2 ...

options: -d|--destaddr  destination address (REQUIRED)
         -f|--fee       fee to subtract from inputs (default: 0)
         -c|--confirm   minimum confirmations to include input (default: 6)
         -n|--no-send   don't send; dump the raw transaction to stdout
EOF
}

fee=0.0
minconfirm=6
OPTS=$(getopt -o d:f:c:hn --long destaddr:,fee:,confirm:,help,no-send -- "$@")
eval set -- "$OPTS"
while true; do
  case "$1" in
    -d|--destaddr)  destaddr="$2"; shift 2;;
    -f|--fee)       fee="$2"; shift 2;;
    -c|--confirm)   minconfirm="$2"; shift 2;;
    -n|--no-send)   nosend=1; shift 1;;
    -h|--help)      help; exit 1;;
    --)             shift; break;;
    *)              echo "Internal error"; exit 1;;
  esac
done
if [ "$destaddr" == "" ]
then
  help
  exit 1
fi

addrs=$(echo $* | sed "s/^/[\"/;s/ /\",\"/g;s/\$/\"]/")
total=$(bitcoind listunspent $minconfirm 21000000 $addrs | grep amount | sed "s/.*: //;s/,//" | tr "\n" "+" | sed "s/+\$/\n/" | bc)
total=$(echo $total - $fee | bc)

tx=$(bitcoind signrawtransaction $(bitcoind createrawtransaction [$(bitcoind listunspent $minconfirm 21000000 $addrs | egrep "txid|vout" | sed "s/\"txid/{\"txid/;s/\"vout\" : \([0-9]*\),/\"vout\" : \1},/" | tr -d "\n" | tr -d " " | sed "s/,\$//")] {\"$destaddr\":$total}) | grep \"hex\" | sed "s/.*: \"//;s/\",//")

if [ "$nosend" != "" ]
then
  echo $tx
else
  bitcoind sendrawtransaction $tx
fi

Tipjars: BTC 1TipsGocnz2N5qgAm9f7JLrsMqkb3oXe2 LTC LTipsVC7XaFy9M6Zaf1aGGe8w8xVUeWFvR | My Bitcoin Note Generator | Pool Auto-Switchers: zpool MiningPoolHub NiceHash
Bitgem Resources: Pool Explorer Paper Wallet
caffeinewriter
Hero Member
*****
Offline Offline

Activity: 532
Merit: 500



View Profile
November 29, 2012, 05:22:54 PM
 #2

Very nice Smiley And quite a simple script as well. Bravo.

Stephen Gornick
Legendary
*
Offline Offline

Activity: 2506
Merit: 1010


View Profile
November 29, 2012, 10:35:23 PM
 #3

What follows is a shell script to automate this process.

Very cool.  Every so often someone wants to clean out their wallet or redeem the losing bets on SatoshiDICE and this will be the tool I suggest to them.  Thanks for sharing this!

Unichange.me

            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █


salfter (OP)
Hero Member
*****
Offline Offline

Activity: 651
Merit: 501


My PGP Key: 92C7689C


View Profile WWW
November 29, 2012, 11:09:18 PM
 #4

One more tip: If you use my script, you might want to make this change in ~/.bitcoin/bitcoin.conf, as recommended in https://en.bitcoin.it/wiki/Free_transaction_relay_policy:

Code:
addnode=173.242.112.53

Tipjars: BTC 1TipsGocnz2N5qgAm9f7JLrsMqkb3oXe2 LTC LTipsVC7XaFy9M6Zaf1aGGe8w8xVUeWFvR | My Bitcoin Note Generator | Pool Auto-Switchers: zpool MiningPoolHub NiceHash
Bitgem Resources: Pool Explorer Paper Wallet
salfter (OP)
Hero Member
*****
Offline Offline

Activity: 651
Merit: 501


My PGP Key: 92C7689C


View Profile WWW
November 29, 2012, 11:34:31 PM
 #5

...and some statistics regarding the transfers I made while figuring this out:

https://docs.google.com/spreadsheet/ccc?key=0AhDiish0IKqvdHpUOVQzZDZQbS1LMWRFNmFvRXYtVVE

24.253.13.34 is the router for my home network; my mining rig is on this network, and it's where CheapSweep was developed and tested. 173.242.112.67 is most likely on the same subnet as 173.242.112.53, which the Bitcoin wiki recommends for relaying free transactions.  Two other IPs also appear.

Also interesting: Eclipse mined five of the six blocks containing my transactions.

Tipjars: BTC 1TipsGocnz2N5qgAm9f7JLrsMqkb3oXe2 LTC LTipsVC7XaFy9M6Zaf1aGGe8w8xVUeWFvR | My Bitcoin Note Generator | Pool Auto-Switchers: zpool MiningPoolHub NiceHash
Bitgem Resources: Pool Explorer Paper Wallet
Prospero
Full Member
***
Offline Offline

Activity: 194
Merit: 100


View Profile
January 09, 2013, 06:56:51 PM
 #6

I'm using 0.8 beta of bitcoind and the script fails to work because I think the JSON format might have changed. The script looks like it is expecting each "amount: $num" to be on a separate line, but in 0.8 the JSON format is compact so there are no newlines. I think right before `grep amount`, `perl -pe 's/,/,\n/g'` (or equivalent) should be used to add a newline after every comma.

1Ja2AxA8hfFMrPwSTV9nP6kq7bp3f7x734
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1079


Gerald Davis


View Profile
January 09, 2013, 07:13:49 PM
 #7

Eventually it would be nice for scripts like this to work their way into the clients.   "It appears you wallet has become fragmented.  Would you like to consolidate some coins.  The coins will be unavailable until confirmed".

Anything that reduces the size of the pruned database is a positive.  It would be optimal if miners could detect these types of transactions (where the size of the output is smaller than the size of the input) and allow them as space allows even with no fee. 

EVERYONE (including miners) benefits from reducing the size of the pruned database.
Bitsky
Hero Member
*****
Offline Offline

Activity: 576
Merit: 514


View Profile
January 09, 2013, 07:42:19 PM
 #8

If coincontrol would make it into the main client, users could even select which addresses should be defragmented. It shouldn't be too complex to display the fragmentation for each address.
And if the single recipient address is also in the source address list then it should be obvious that it's a coin-combination (which could be fee-free so users agree to it).

Bounty: Earn up to 68.7 BTC
Like my post? Feel free to drop a tip to 1BitskyZbfR4irjyXDaGAM2wYKQknwX36Y
salfter (OP)
Hero Member
*****
Offline Offline

Activity: 651
Merit: 501


My PGP Key: 92C7689C


View Profile WWW
January 09, 2013, 11:24:55 PM
 #9

I'm using 0.8 beta of bitcoind and the script fails to work because I think the JSON format might have changed. The script looks like it is expecting each "amount: $num" to be on a separate line, but in 0.8 the JSON format is compact so there are no newlines. I think right before `grep amount`, `perl -pe 's/,/,\n/g'` (or equivalent) should be used to add a newline after every comma.

Hmm...haven't tried that version yet.  I'll have to look into it.

Tipjars: BTC 1TipsGocnz2N5qgAm9f7JLrsMqkb3oXe2 LTC LTipsVC7XaFy9M6Zaf1aGGe8w8xVUeWFvR | My Bitcoin Note Generator | Pool Auto-Switchers: zpool MiningPoolHub NiceHash
Bitgem Resources: Pool Explorer Paper Wallet
etotheipi
Legendary
*
Offline Offline

Activity: 1428
Merit: 1093


Core Armory Developer


View Profile WWW
January 10, 2013, 03:13:31 AM
 #10

Does bitcoind RPC interface broadcast arbitrary transactions for you?  My understanding is that "stock" bitcoind will not accept such a 50 kB transaction without a substantial fee -- it would be DOA.  Obviously other nodes might use different fee rules, but I want to understand why your own bitcoind took it? 

Also, the latest version of Armory has coin-control -- but only at the address level.  If your coins are split up between lots of addresses, you can do this house-cleaning, though you'll have to pay a full fee, because stock bitcoind won't accept it and forward it if you don't.

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
Prospero
Full Member
***
Offline Offline

Activity: 194
Merit: 100


View Profile
January 10, 2013, 04:01:36 AM
 #11

It sounds like this ticket might be related to this: https://github.com/bitcoin/bitcoin/issues/1675

1Ja2AxA8hfFMrPwSTV9nP6kq7bp3f7x734
K1773R
Legendary
*
Offline Offline

Activity: 1792
Merit: 1008


/dev/null


View Profile
January 13, 2013, 08:53:23 AM
 #12

just tested this and sadly it dosnt work.
$addrs and $totals arent calucalted correctly, i tested it with testnet (yes i changed the script apropriately):

Code:
$ ./cheapsweep.sh -n -f 0 -d n1vCS8kqNQJSnFu3xA1ma64RafUEZ7oAtJ `bitcoind -testnet -rpcport=18334 listaddressgroupings | grep '",' | sed "s/            \"//" | sed "s/\",//" | xargs echo`
error: Error parsing JSON:c
error: Error parsing JSON:c
error: {"code":-3,"message":"Invalid amount"}
error: {"code":-1,"message":"signrawtransaction <hex string> [{\"txid\":txid,\"vout\":n,\"scriptPubKey\":hex},...] [<privatekey1>,...] [sighashtype=\"ALL\"]\nSign inputs for raw transaction (serialized, hex-encoded).\nSecond optional argument (may be null) is an array of previous transaction outputs that\nthis transaction depends on but may not yet be in the blockchain.\nThird optional argument (may be null) is an array of base58-encoded private\nkeys that, if given, will be the only keys used to sign the transaction.\nFourth optional argument is a string that is one of six values; ALL, NONE, SINGLE or\nALL|ANYONECANPAY, NONE|ANYONECANPAY, SINGLE|ANYONECANPAY.\nReturns json object with keys:\n  hex : raw transaction with signature(s) (hex-encoded string)\n  complete : 1 if transaction has a complete set of signature (0 if not)"}

$total is 0 and $address is "c"

[GPG Public Key]
BTC/DVC/TRC/FRC: 1K1773RbXRZVRQSSXe9N6N2MUFERvrdu6y ANC/XPM AK1773RTmRKtvbKBCrUu95UQg5iegrqyeA NMC: NK1773Rzv8b4ugmCgX789PbjewA9fL9Dy1 LTC: LKi773RBuPepQH8E6Zb1ponoCvgbU7hHmd EMC: EK1773RxUes1HX1YAGMZ1xVYBBRUCqfDoF BQC: bK1773R1APJz4yTgRkmdKQhjhiMyQpJgfN
salfter (OP)
Hero Member
*****
Offline Offline

Activity: 651
Merit: 501


My PGP Key: 92C7689C


View Profile WWW
January 14, 2013, 09:20:16 PM
 #13

Does bitcoind RPC interface broadcast arbitrary transactions for you?

If you create raw transactions, the usual checks AFAICT are bypassed. I've sent transactions up to about 10K or so with tiny fees (BTC0.0001-BTC0.0005).  I'm not sure how big my latest zero-fee transactions have been, but they've all gone through within (usually) 1-3 hours.

Tipjars: BTC 1TipsGocnz2N5qgAm9f7JLrsMqkb3oXe2 LTC LTipsVC7XaFy9M6Zaf1aGGe8w8xVUeWFvR | My Bitcoin Note Generator | Pool Auto-Switchers: zpool MiningPoolHub NiceHash
Bitgem Resources: Pool Explorer Paper Wallet
salfter (OP)
Hero Member
*****
Offline Offline

Activity: 651
Merit: 501


My PGP Key: 92C7689C


View Profile WWW
January 28, 2013, 04:37:03 PM
 #14

I'm using 0.8 beta of bitcoind and the script fails to work because I think the JSON format might have changed. The script looks like it is expecting each "amount: $num" to be on a separate line, but in 0.8 the JSON format is compact so there are no newlines. I think right before `grep amount`, `perl -pe 's/,/,\n/g'` (or equivalent) should be used to add a newline after every comma.

Something must've changed in the past few weeks, as now that I've gotten a chance to test CheapSweep against a beta build, it's working properly.  I made no changes to the script...just had it sweep some free coins a little bit ago.  The bitcoind I'm running was built from whatever was on GitHub last Friday.

Tipjars: BTC 1TipsGocnz2N5qgAm9f7JLrsMqkb3oXe2 LTC LTipsVC7XaFy9M6Zaf1aGGe8w8xVUeWFvR | My Bitcoin Note Generator | Pool Auto-Switchers: zpool MiningPoolHub NiceHash
Bitgem Resources: Pool Explorer Paper Wallet
Prospero
Full Member
***
Offline Offline

Activity: 194
Merit: 100


View Profile
January 28, 2013, 08:32:17 PM
 #15

I think the issue is that there are many versions of sed and it's hard to maintain portability between the versions. Even replacing sed with gsed (I'm on a Mac) didn't help here. But after translating to perl I got reasonable results. I haven't translated the last call to sed yet.
Code:
# addrs=$(echo $* | sed "s/^/[\"/;s/ /\",\"/g;s/\$/\"]/")
addrs=$(echo $* | perl -ne 'printf "[%s]", join ",", map qq("$_"), split " "')
# total=$(bitcoind listunspent $minconfirm 21000000 "$addrs" | grep amount | sed "s/.*: //;s/,//" | tr "\n" "+" | sed "s/+\$/\n/")
total=$(
    bitcoind listunspent $minconfirm 21000000 "$addrs" \
        | perl -ne '($a)=/^\s*"amount" : (\d+\.\d+),$/; $a||next;$sum+=$a } END {print $sum'
)

1Ja2AxA8hfFMrPwSTV9nP6kq7bp3f7x734
K1773R
Legendary
*
Offline Offline

Activity: 1792
Merit: 1008


/dev/null


View Profile
February 01, 2013, 11:16:51 PM
 #16

I think the issue is that there are many versions of sed and it's hard to maintain portability between the versions. Even replacing sed with gsed (I'm on a Mac) didn't help here. But after translating to perl I got reasonable results. I haven't translated the last call to sed yet.
Code:
# addrs=$(echo $* | sed "s/^/[\"/;s/ /\",\"/g;s/\$/\"]/")
addrs=$(echo $* | perl -ne 'printf "[%s]", join ",", map qq("$_"), split " "')
# total=$(bitcoind listunspent $minconfirm 21000000 "$addrs" | grep amount | sed "s/.*: //;s/,//" | tr "\n" "+" | sed "s/+\$/\n/")
total=$(
    bitcoind listunspent $minconfirm 21000000 "$addrs" \
        | perl -ne '($a)=/^\s*"amount" : (\d+\.\d+),$/; $a||next;$sum+=$a } END {print $sum'
)
unfortunately i cant code in perl, are you planing to complete it?

[GPG Public Key]
BTC/DVC/TRC/FRC: 1K1773RbXRZVRQSSXe9N6N2MUFERvrdu6y ANC/XPM AK1773RTmRKtvbKBCrUu95UQg5iegrqyeA NMC: NK1773Rzv8b4ugmCgX789PbjewA9fL9Dy1 LTC: LKi773RBuPepQH8E6Zb1ponoCvgbU7hHmd EMC: EK1773RxUes1HX1YAGMZ1xVYBBRUCqfDoF BQC: bK1773R1APJz4yTgRkmdKQhjhiMyQpJgfN
Prospero
Full Member
***
Offline Offline

Activity: 194
Merit: 100


View Profile
February 02, 2013, 01:10:13 AM
 #17

unfortunately i cant code in perl, are you planing to complete it?
It turns out the last statement works as is. I didn't test it before because I assumed I had to translate it also. So the only statements I needed to change are in my above post.

1Ja2AxA8hfFMrPwSTV9nP6kq7bp3f7x734
K1773R
Legendary
*
Offline Offline

Activity: 1792
Merit: 1008


/dev/null


View Profile
February 02, 2013, 01:40:03 AM
Last edit: February 02, 2013, 12:09:39 PM by K1773R
 #18

unfortunately i cant code in perl, are you planing to complete it?
It turns out the last statement works as is. I didn't test it before because I assumed I had to translate it also. So the only statements I needed to change are in my above post.
EDIT: err, nvm

[GPG Public Key]
BTC/DVC/TRC/FRC: 1K1773RbXRZVRQSSXe9N6N2MUFERvrdu6y ANC/XPM AK1773RTmRKtvbKBCrUu95UQg5iegrqyeA NMC: NK1773Rzv8b4ugmCgX789PbjewA9fL9Dy1 LTC: LKi773RBuPepQH8E6Zb1ponoCvgbU7hHmd EMC: EK1773RxUes1HX1YAGMZ1xVYBBRUCqfDoF BQC: bK1773R1APJz4yTgRkmdKQhjhiMyQpJgfN
K1773R
Legendary
*
Offline Offline

Activity: 1792
Merit: 1008


/dev/null


View Profile
February 02, 2013, 03:25:00 AM
 #19

i just found the problem: [ and ] have to be escaped
Code:
tx=$(bitcoind signrawtransaction $(bitcoind createrawtransaction \[$(bitcoind listunspent $minconfirm 21000000 $addrs | egrep "txid|vout" | sed "s/\"txid/{\"txid/;s/\"vout\" : \([0-9]*\),/\"vout\" : \1},/" | tr -d "\n" | tr -d " " | sed "s/,\$//")\] {\"$destaddr\":$total}) | grep \"hex\" | sed "s/.*: \"//;s/\",//")

[GPG Public Key]
BTC/DVC/TRC/FRC: 1K1773RbXRZVRQSSXe9N6N2MUFERvrdu6y ANC/XPM AK1773RTmRKtvbKBCrUu95UQg5iegrqyeA NMC: NK1773Rzv8b4ugmCgX789PbjewA9fL9Dy1 LTC: LKi773RBuPepQH8E6Zb1ponoCvgbU7hHmd EMC: EK1773RxUes1HX1YAGMZ1xVYBBRUCqfDoF BQC: bK1773R1APJz4yTgRkmdKQhjhiMyQpJgfN
K1773R
Legendary
*
Offline Offline

Activity: 1792
Merit: 1008


/dev/null


View Profile
February 02, 2013, 04:34:36 AM
Last edit: March 12, 2013, 08:24:43 PM by K1773R
 #20

i updated the script so it works with linux:
Code:
#!/bin/bash

# CheapSweep v0.1
# Scott Alfter
# scott@alfter.us
# Donations: 1TipSAXbE6owdU24bcBDJKmL8JRxQe5Yu

BITCOIND="bitcoind"
#BITCOIND="bitcoind -testnet -rpcport=18334"

help()
{
cat <<EOF >&2
Usage: $0 [options] -d destaddr addr1 addr2 ...

options: -d|--destaddr  destination address (REQUIRED)
         -f|--fee       fee to subtract from inputs (default: 0)
         -c|--confirm   minimum confirmations to include input (default: 6)
         -n|--no-send   dont send; dump the raw transaction to stdout
EOF
}

fee=0.0
minconfirm=6
OPTS=$(getopt -o d:f:c:hn --long destaddr:,fee:,confirm:,help,no-send -- "$@")
eval set -- "$OPTS"
while true; do
  case "$1" in
    -d|--destaddr)  destaddr="$2"; shift 2;;
    -f|--fee)       fee="$2"; shift 2;;
    -c|--confirm)   minconfirm="$2"; shift 2;;
    -n|--no-send)   nosend=1; shift 1;;
    -h|--help)      help; exit 1;;
    --)             shift; break;;
    *)              echo "Internal error"; exit 1;;
  esac
done
if [ "$destaddr" == "" ]
then
  help
  exit 1
fi

addrs=$(echo "$*" | sed "s/^/\[\"/;s/ /\",\"/g;s/\$/\"\]/")
echo "$addrs";

total=$($BITCOIND listunspent $minconfirm 21000000 "$addrs" | grep amount | sed "s/.*: //;s/,//" | tr "\n" "+" | sed "s/+\$/\n/" | bc)
total=$(echo $total - $fee | bc)
echo $total;

tx=$($BITCOIND signrawtransaction $($BITCOIND createrawtransaction \[$($BITCOIND listunspent $minconfirm 21000000 "$addrs" | egrep "txid|vout" | sed "s/\"txid/{\"txid/;s/\"vout\" : \([0-9]*\),/\"vout\" : \1},/" | tr -d "\n" | tr -d " " | sed "s/,\$//")\] {\"$destaddr\":$total}) | grep \"hex\" | sed "s/.*: \"//;s/\",//")

if [ "$nosend" != "" ]
then
  echo $tx
else
  $BITCOIND sendrawtransaction $tx
fi

[GPG Public Key]
BTC/DVC/TRC/FRC: 1K1773RbXRZVRQSSXe9N6N2MUFERvrdu6y ANC/XPM AK1773RTmRKtvbKBCrUu95UQg5iegrqyeA NMC: NK1773Rzv8b4ugmCgX789PbjewA9fL9Dy1 LTC: LKi773RBuPepQH8E6Zb1ponoCvgbU7hHmd EMC: EK1773RxUes1HX1YAGMZ1xVYBBRUCqfDoF BQC: bK1773R1APJz4yTgRkmdKQhjhiMyQpJgfN
Pages: [1] 2 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!