Can you provide some more information here? Were you running the release binaries? What version? What operating system? How did you kill the process? What state was it in when you brought it back up? What errors did you receive? Would it be possible for you to provide the courrupted wallet and database/ directory to me?
I want to start by saying I think bitcoind overall is solid. This whole experiment started informally. A friend of mine is working on a project that requires accounts and I'm mostly exploring the topic out of curiosity. I saw a lot of posts recommending not to use the account features, and I wanted to see for myself how far I can take things before they break.
I was running an older version (which happened to be installed with my Armory instance), 8.2.2-beta (80202). You're absolutely right, I should probably try this again on the latest version.
I'll be happy to provide the database files to you (it's all on testnet), but they are currently very large. Wallet.dat is 85MB. Contact me directly please and I'll send you a download link.
I am running on Windows 7 and making calls from Python 2.7.
I didn't intentionally kill the process, but when I initially set up my code I used this construct, which seems to have caused the problem:
process = subprocess.Popen([r'C:\Program Files (x86)\Bitcoin\daemon\bitcoind.exe', '-testnet', '-rpcuser=test', '-rpcpassword=test1'])
time.sleep(20) #give bitcoind time to start up; a smarter way would be to check the network connection in a loop, but for our purposes this is fine
try:
#run various tests in a loop
finally:
process.terminate()
The terminate() call is what appears to cause the corruption. Later on I got a bit smarter, as I gained experience both with bitcoind and Python:
process = subprocess.Popen([r'C:\Program Files (x86)\Bitcoin\daemon\bitcoind.exe', '-testnet', '-rpcuser=test', '-rpcpassword=test1'])
time.sleep(20) #give bitcoind time to start up; a smarter way would be to check the network connection in a loop, but for our purposes this is fine
try:
#run various tests in a loop
finally:
if client is not None:
client.stop()
time.sleep(5)
process.terminate()
But things are still not 100% OK. I currently get this when I start up:
Warning: Warning: error reading wallet.dat! All keys read correctly, but transaction data or address book entries might be missing or incorrect.
In its current state, whether the issue is with the wallet database or with the chain database, I am unable to perform some of the wallet operations:
>>> client.getbalance()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\jsonrpc\proxy.py", line 45, in __call__
raise JSONRPCException(resp['error'])
JSONRPCException
I assume you were spending unconfirmed coins in these transactions? Taking several seconds per-spend is a known artifact of the current software behavior— the code that traverses unspent coins has factorial-ish complexity. While it could be improved— there are patches available, and simply disabling spending unconfirmed outputs avoids it—, since the overall network capacity is not very great I've mostly considered this bug helpful at discouraging inept denial of service attacks so I haven't personally considered it a priority. (And most of the people who've noticed it who have mentioned it to me appear to have just been conducting tests or attempting denial of service attacks…)
I'm not sure but I believe the inputs were all confirmed. I started out with 5 confirmed BTC and sent 0.0001 to a random address in the wallet on each iteration. Code is below.
It seems to me all or most of the delay was not in code but rather with disk operations, and more specifically flushing wallet.dat (which is now 85MB). In any case I don't consider this to be a major issue.
This is the code I used in my test:
Populate wallet with 50K accounts and test duration of moving funds internally between accounts:
import subprocess
import time
import datetime
import os
import shutil
import timeit
from jsonrpc import ServiceProxy
from random import randrange
#config
#location of an "empty" wallet file (only the default ('') account exists)
blankWalletFile = r'C:\Users\User\AppData\Roaming\Bitcoin\testnet3\wallet.empty.dat'
#location of the live wallet file; this file will be backed up and then restored after the test is complete
liveWalletFile = r'C:\Users\User\AppData\Roaming\Bitcoin\testnet3\wallet.dat'
#the number of accounts to be created
account_count = 50000
#the number of random transfers to perform between accounts
transfer_count = 10000
def resetWallet(account_count = 1):
#back up live wallet
shutil.copy(liveWalletFile, liveWalletFile + '.bak')
if account_count > 1: #see if we already have a wallet file with this number of accounts
source_wallet_file = liveWalletFile + '.' + str(account_count)
if not os.path.isfile(source_wallet_file):
print('wallet file does not exist; starting with a blank file')
source_wallet_file = blankWalletFile
else:
print('wallet file exists; re-using existing wallet file')
#overwrite live wallet with empty wallet
shutil.copy(source_wallet_file, liveWalletFile)
def restoreLiveWallet():
#make a copy of this test wallet file for future use
shutil.copy(liveWalletFile, liveWalletFile + '.' + str(account_count))
#overwrite test wallet with live wallet backup
shutil.move(liveWalletFile + '.bak', liveWalletFile)
def createAccounts(client, count):
for i in range(0, count):
client.move('', 'account%d' % i, 1e-8)
#print(client.getbalance('account%d' % i))
def performRandomTransfers(client, count):
account1 = ''
account2 = getRandomAccount(account_count)
for i in range(0, count):
client.move(account1, account2, 1e-8)
account1 = account2
account2 = getRandomAccount(account_count, account1)
def getRandomAccount(account_count, except_account = ''):
account = except_account
while account == except_account:
account = 'account%d' % randrange(0, account_count)
return account
def main():
print(datetime.datetime.now().time())
resetWallet(account_count)
print('starting bitcoind...')
process = subprocess.Popen([r'C:\Program Files (x86)\Bitcoin\daemon\bitcoind.exe', '-testnet', '-rpcuser=test', '-rpcpassword=test1'])
time.sleep(10) #give bitcoind time to start up; a smarter way would be to check the network connection in a loop, but for our purposes this is fine
print('bitcoind started')
try:
client = ServiceProxy("http://test:test1@localhost:18332")
actual_account_count = len(client.listaccounts())
print ('there are currently {} accounts'.format(actual_account_count))
to_create_count = account_count - actual_account_count + 1
if actual_account_count < account_count:
print('creating %d accounts' % to_create_count)
total_time = timeit.timeit(lambda: createAccounts(client, to_create_count), number=1)
print('time elapsed creating {0:d} accounts: {1:.2f} seconds'.format(to_create_count, total_time))
print(datetime.datetime.now().time())
print 'performing %d random transfers' % transfer_count
total_time = timeit.timeit(lambda: performRandomTransfers(client, transfer_count), number=1)
print 'time elapsed performing {0:d} transfers: {1:.2f} seconds'.format(transfer_count, total_time)
print('there are currently {} accounts'.format(len(client.listaccounts())))
#print(client.listaccounts())
print(datetime.datetime.now().time())
print('shutting down bitcoind...')
client.stop()
time.sleep(5)
finally:
process.terminate()
restoreLiveWallet()
if __name__ == "__main__":
main()
Perform external transfers between accounts:
import subprocess
import time
import datetime
import os
import shutil
import timeit
from jsonrpc import ServiceProxy
from random import randrange
#config
transfer_count = 50 #the number of random transfers to perform between accounts
account_count = 50000 #number of existing accounts in the wallet
def performRandomTransfer(client, actual_count):
account1 = getRandomAccount()
account2 = getRandomAccount(account1)
address = client.getaccountaddress(account2)
if (client.getbalance(account1) > 1e-4):
tx = client.sendfrom(account1, address, 1e-4)
elif client.getbalance() > 1e-4:
tx = client.sendtoaddress(address, 1e-4)
else:
raise Exception('No balance available in any account')
actual_count[0] += 1
print(tx)
def getRandomAccount(except_account = ''):
account = except_account
while account == except_account:
account = 'account%d' % randrange(0, account_count)
return account
def main():
print(datetime.datetime.now().time())
print('starting bitcoind...')
process = subprocess.Popen([r'C:\Program Files (x86)\Bitcoin\daemon\bitcoind.exe', '-testnet', '-rpcuser=test', '-rpcpassword=test1'])
time.sleep(20) #give bitcoind time to start up; a smarter way would be to check the network connection in a loop, but for our purposes this is fine
print('bitcoind started')
try:
client = ServiceProxy("http://test:test1@localhost:18332")
print(datetime.datetime.now().time())
print 'performing %d random transfers' % transfer_count
actual_count = [0]
total_time = timeit.timeit(lambda: performRandomTransfer(client, actual_count), number=transfer_count)
print 'time elapsed performing {0:d} transfers: {1:.2f} seconds'.format(actual_count[0], total_time)
print(datetime.datetime.now().time())
finally:
print('shutting down bitcoind...')
if client is not None:
client.stop()
time.sleep(5)
process.terminate()
if __name__ == "__main__":
main()
