Scalability of BitCoinD for a wallet service

[mtbitcoin]

Hi

I am builing a free wallet service (like instawallet) and would like to know what are your thoughts on solely relying on the bitcoind daemon for storing all transactions as opposed to storing them in an external database for lookups. I have concerns that if we were to run into hundreds of users and transactions that bitcoind would not be able to handle such volumes concurrently.

Are there any hard limits of to how far the daemon can scale or is this limited by the underlying hardware?

Thanks

[Gavin Andresen]

Are there any hard limits of to how far the daemon can scale or is this limited by the underlying hardware?

No hard limits, but the bitcoind "accounts" code hasn't been optimized for that.  For example, computing an account's balance loops through all transactions in the wallet, so will take an increasing amount of time the more transactions you have in the wallet.

[mtbitcoin]

Hi Gavin

Thank you for taking the time to reply

Any suggestions as to how one should go about this then for high volume transactions? I've looked at the previous posts and wiki entries, most of these points towards using the "Accounts" feature. If the transactions were recorded to an external database, the biggest issue i see with this is polling the  addresses when a payment is expected and also dealing with redundant payments.

Thanks
-MT

[kjj]

Can I ask why you don't want to use your own code to manage balances and transactions?  It seems to me that you need to do those things anyway, at least in most scenarios that I'm familiar with.

[blueadept]

I'd take a look at this thread:

https://bitcointalk.org/index.php?topic=25094.0

This fork makes it easier to poll/import transactions into your own DB and then do whatever you need with them rather than relying on bitcoind to keep accounts straight.  I'm using it for my own purposes (though I'm not accessing it from PHP but Python; I still need to keep track of transactions in a RDBMS).

[2112]

Can I ask why you don't want to use your own code to manage balances and transactions?  

I'm not mtbitcoin, but I can give you an answer to this: the current code is horribly complex and doesn't obey any rules demanded by any legitimate accounting methods. For example it routinely re-dates the transactions and can disappear them upon block-chain reorganization.

The whole idea of stochastically-solve-the-inverted-one-dimensional-knapsack-problem before you can compute the transaction fee is hair raising to any legitimate accountant. On the other hand it is music to the ears of people planning to pull some sort of a fraud.

[mtbitcoin]

Can I ask why you don't want to use your own code to manage balances and transactions?  It seems to me that you need to do those things anyway, at least in most scenarios that I'm familiar with.

By having the bitcoind manage the balance and transactions I could avoid any synchronization and integration issues. This would also simplify the development process

[mtbitcoin]

I'd take a look at this thread:

https://bitcointalk.org/index.php?topic=25094.0

This fork makes it easier to poll/import transactions into your own DB and then do whatever you need with them rather than relying on bitcoind to keep accounts straight.  I'm using it for my own purposes (though I'm not accessing it from PHP but Python; I still need to keep track of transactions in a RDBMS).

Thank you for the tip. However, all things being equal I would prefer to stick with the mainstream releases

[Xenland]

Honestly, I think its more work to go based off the bit coin daemon, if you want to move funds from a receiving bit coin address to a specific address in the same wallet it will have to go through the block chain which will add more wait time in waiting for even just 1 confirmation. Its just easier to use a database and periodically scan your transactions and then update the database with Bitcoin addresses with corresponding information like confirmations, balance, etc,etc. This method can help save block time and its more efficient to look up account information as well, since if you relied on the daemon for information you'd have to rescan every request which is inefficient. With a database you can cache all data in the wallet in one swoop then have the program query the database.

[2112]

Honestly, I think its more work to go based off the bit coin daemon, if you want to move funds from a receiving bit coin address to a specific address in the same wallet it will have to go through the block chain which will add more wait time in waiting for even just 1 confirmation.

You seem to be missing two things:
1) RPC move between the accounts instantly
2) wallet.dat can be modified live with correctly compiled BerkeleyDB program that uses the same DBENV. wallet.db is a database and BerkDB is a quite powerful DBMS, its just that most programmers (including the 'core development group') are unfamiliar with it and don't understand its features.

[Xenland]

Honestly, I think its more work to go based off the bit coin daemon, if you want to move funds from a receiving bit coin address to a specific address in the same wallet it will have to go through the block chain which will add more wait time in waiting for even just 1 confirmation.

You seem to be missing two things:
1) RPC move between the accounts instantly
2) wallet.dat can be modified live with correctly compiled BerkeleyDB program that uses the same DBENV. wallet.db is a database and BerkDB is a quite powerful DBMS, its just that most programmers (including the 'core development group') are unfamiliar with it and don't understand its features.

So your saying I can move bitcoins from one bitcoin address to another bitcoin address with out waiting for confirmations from the block chain?

can anyone else verify this?

[2112]

So your saying I can move bitcoins from one bitcoin address to another bitcoin address with out waiting for confirmations from the block chain?

Not "bitcoin address" but "bitcoin account". Please read carefully the output from "bitcoind help" and the original post.

[Xenland]

Okay yeah so in that case yeah you can manage it, I guess I just prefer MySql to manage it instead

[2112]

I guess I just prefer MySql to manage it instead

And herein lies the problem. Switching from BerkeleyDB to any relational DBMS is a quite nontrivial tasks. Many people undertook it and nobody delivered a working code yet. Thankfully Gavin isn't accepting patches that modify the basic behavior of the Bitcoin with respect to block chain reorganizations.

As for those who modify the underlying database without understanding the code: this was probably the story of MyBitcoin if one takes Ted Williams' posts on the face value. Some programmers completely cannot grasp the concept of chain reorganization and write code that is subject to an abuse that the unmodified Satoshi client is completely invulnerable.

[Xenland]

I guess I just prefer MySql to manage it instead

And herein lies the problem. Switching from BerkeleyDB to any relational DBMS is a quite nontrivial tasks. Many people undertook it and nobody delivered a working code yet. Thankfully Gavin isn't accepting patches that modify the basic behavior of the Bitcoin with respect to block chain reorganizations.

As for those who modify the underlying database without understanding the code: this was probably the story of MyBitcoin if one takes Ted Williams' posts on the face value. Some programmers completely cannot grasp the concept of chain reorganization and write code that is subject to an abuse that the unmodified Satoshi client is completely invulnerable.

So in your opinion you believe that it is better to handle with berkelyDB given that you know what your doing?

[2112]

So in your opinion you believe that it is better to handle with berkelyDB given that you know what your doing?

My professional opinion is as follows: (1) for the short-term do a recompilation and any small modifications to the Satoshi client to let it inter-operate correctly with the rest of your infrastructure; (2) for the long-term keep your fingers crossed that the libbitcoin team (bitcoinconsultancy?) delivers on their promises.

BerkeleyDB isn't that difficult to master and the documentation is top-notch superb. I'm actually quite positively surprised with what had happened to it under Oracle's wing. The new release even has the SQL built in. But you'll have to at least recompile and possibly make number of small modifications.

[Gavin Andresen]

Optimizing the accounts code to add a berkeley db index table that indexed wallet transactions by account, and that cached account balances (and invalidated or updated the cache on receive/send) shouldn't be terribly hard for somebody who already knows c++ and berkeley db.

It is not on my short-term TODO list because there are too many other higher priority things on my TODO list, but a nice clean well-tested upward-compatible patch would be most welcome.

PS:  for ClearCoin, I used the "bitcoind keeps track of the bitcoins" architecture, and I never regretted it-- no problems with synchronization, less possibility for MtGox-like hacks that create mythical bitcoin balances out of thin air by adding an entry to a database.

[mtbitcoin]

Looking at the various arguements brought forward, I am going ahead with the "let bitcoind manage" the transactions approach using the "Accounts" feature as a means of storing balances. At the moment I'd rather deal with scalability issues vs having unscynchronized balances between a separate DB vs wallet.dat.

There does not seem to any performance metrics as to what would constitute an upper limit for storing transactions in the wallet.dat. I have tried running some simulations by dumping more than 200k "move" transactions into the wallet and so far it appears to be running fine. I am assuming that the hardware in use would also play a critical role as to how far this would scale. However, the real tests would be concurrent use.

Thank you for all your suggestions and if may add this in as a feature requests (somewhere down the long list), it would be nice to be able to store the transactions into a separate DB i.e odbc, mysql, etc

Cheers
-MT

[Steve]

You should talk to Jan at instawallet about scaling.  I know he has struggled off and on with scaling and would probably be able to offer some good advice on what to do and what to avoid.

[Jan]

Looking at the various arguements brought forward, I am going ahead with the "let bitcoind manage" the transactions approach using the "Accounts" feature as a means of storing balances. At the moment I'd rather deal with scalability issues vs having unscynchronized balances between a separate DB vs wallet.dat.

If you want to scale really big with this approach you need several servers each running bitcoind, and each managing their share of accounts. A load balancer for this setup is fairly trivial to set up.