mtbitcoin (OP)
Legendary
Offline
Activity: 876
Merit: 1000
Etherscan.io
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
August 31, 2011, 04:58:11 AM |
|
Hi
I am builing a free wallet service (like instawallet) and would like to know what are your thoughts on solely relying on the bitcoind daemon for storing all transactions as opposed to storing them in an external database for lookups. I have concerns that if we were to run into hundreds of users and transactions that bitcoind would not be able to handle such volumes concurrently.
Are there any hard limits of to how far the daemon can scale or is this limited by the underlying hardware?
Thanks
|
|
|
|
Gavin Andresen
Legendary
Offline
Activity: 1652
Merit: 2300
Chief Scientist
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 01, 2011, 01:51:27 PM |
|
Are there any hard limits of to how far the daemon can scale or is this limited by the underlying hardware?
No hard limits, but the bitcoind "accounts" code hasn't been optimized for that. For example, computing an account's balance loops through all transactions in the wallet, so will take an increasing amount of time the more transactions you have in the wallet.
|
How often do you get the chance to work on a potentially world-changing project?
|
|
|
mtbitcoin (OP)
Legendary
Offline
Activity: 876
Merit: 1000
Etherscan.io
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 02, 2011, 11:08:16 AM |
|
Hi Gavin
Thank you for taking the time to reply
Any suggestions as to how one should go about this then for high volume transactions? I've looked at the previous posts and wiki entries, most of these points towards using the "Accounts" feature. If the transactions were recorded to an external database, the biggest issue i see with this is polling the addresses when a payment is expected and also dealing with redundant payments.
Thanks -MT
|
|
|
|
kjj
Legendary
Offline
Activity: 1302
Merit: 1026
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 02, 2011, 03:38:30 PM |
|
Can I ask why you don't want to use your own code to manage balances and transactions? It seems to me that you need to do those things anyway, at least in most scenarios that I'm familiar with.
|
17Np17BSrpnHCZ2pgtiMNnhjnsWJ2TMqq8 I routinely ignore posters with paid advertising in their sigs. You should too.
|
|
|
blueadept
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 02, 2011, 05:25:26 PM |
|
I'd take a look at this thread: https://bitcointalk.org/index.php?topic=25094.0This fork makes it easier to poll/import transactions into your own DB and then do whatever you need with them rather than relying on bitcoind to keep accounts straight. I'm using it for my own purposes (though I'm not accessing it from PHP but Python; I still need to keep track of transactions in a RDBMS).
|
|
|
|
2112
Legendary
Offline
Activity: 2128
Merit: 1073
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 02, 2011, 07:00:56 PM Last edit: September 02, 2011, 08:37:35 PM by 2112 |
|
Can I ask why you don't want to use your own code to manage balances and transactions? I'm not mtbitcoin, but I can give you an answer to this: the current code is horribly complex and doesn't obey any rules demanded by any legitimate accounting methods. For example it routinely re-dates the transactions and can disappear them upon block-chain reorganization. The whole idea of stochastically-solve-the-inverted-one-dimensional-knapsack-problem before you can compute the transaction fee is hair raising to any legitimate accountant. On the other hand it is music to the ears of people planning to pull some sort of a fraud.
|
|
|
|
mtbitcoin (OP)
Legendary
Offline
Activity: 876
Merit: 1000
Etherscan.io
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 02, 2011, 07:35:26 PM |
|
Can I ask why you don't want to use your own code to manage balances and transactions? It seems to me that you need to do those things anyway, at least in most scenarios that I'm familiar with.
By having the bitcoind manage the balance and transactions I could avoid any synchronization and integration issues. This would also simplify the development process
|
|
|
|
mtbitcoin (OP)
Legendary
Offline
Activity: 876
Merit: 1000
Etherscan.io
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 02, 2011, 07:40:07 PM |
|
I'd take a look at this thread: https://bitcointalk.org/index.php?topic=25094.0This fork makes it easier to poll/import transactions into your own DB and then do whatever you need with them rather than relying on bitcoind to keep accounts straight. I'm using it for my own purposes (though I'm not accessing it from PHP but Python; I still need to keep track of transactions in a RDBMS). Thank you for the tip. However, all things being equal I would prefer to stick with the mainstream releases
|
|
|
|
Xenland
Legendary
Offline
Activity: 980
Merit: 1003
I'm not just any shaman, I'm a Sha256man
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 02, 2011, 08:39:48 PM |
|
Honestly, I think its more work to go based off the bit coin daemon, if you want to move funds from a receiving bit coin address to a specific address in the same wallet it will have to go through the block chain which will add more wait time in waiting for even just 1 confirmation. Its just easier to use a database and periodically scan your transactions and then update the database with Bitcoin addresses with corresponding information like confirmations, balance, etc,etc. This method can help save block time and its more efficient to look up account information as well, since if you relied on the daemon for information you'd have to rescan every request which is inefficient. With a database you can cache all data in the wallet in one swoop then have the program query the database.
|
|
|
|
2112
Legendary
Offline
Activity: 2128
Merit: 1073
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 02, 2011, 09:14:39 PM |
|
Honestly, I think its more work to go based off the bit coin daemon, if you want to move funds from a receiving bit coin address to a specific address in the same wallet it will have to go through the block chain which will add more wait time in waiting for even just 1 confirmation.
You seem to be missing two things: 1) RPC move between the accounts instantly 2) wallet.dat can be modified live with correctly compiled BerkeleyDB program that uses the same DBENV. wallet.db is a database and BerkDB is a quite powerful DBMS, its just that most programmers (including the 'core development group') are unfamiliar with it and don't understand its features.
|
|
|
|
Xenland
Legendary
Offline
Activity: 980
Merit: 1003
I'm not just any shaman, I'm a Sha256man
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 02, 2011, 10:43:00 PM |
|
Honestly, I think its more work to go based off the bit coin daemon, if you want to move funds from a receiving bit coin address to a specific address in the same wallet it will have to go through the block chain which will add more wait time in waiting for even just 1 confirmation.
You seem to be missing two things: 1) RPC move between the accounts instantly 2) wallet.dat can be modified live with correctly compiled BerkeleyDB program that uses the same DBENV. wallet.db is a database and BerkDB is a quite powerful DBMS, its just that most programmers (including the 'core development group') are unfamiliar with it and don't understand its features. So your saying I can move bitcoins from one bitcoin address to another bitcoin address with out waiting for confirmations from the block chain? can anyone else verify this?
|
|
|
|
2112
Legendary
Offline
Activity: 2128
Merit: 1073
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 02, 2011, 10:46:58 PM |
|
So your saying I can move bitcoins from one bitcoin address to another bitcoin address with out waiting for confirmations from the block chain?
Not "bitcoin address" but "bitcoin account". Please read carefully the output from "bitcoind help" and the original post.
|
|
|
|
Xenland
Legendary
Offline
Activity: 980
Merit: 1003
I'm not just any shaman, I'm a Sha256man
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 02, 2011, 10:52:56 PM |
|
Okay yeah so in that case yeah you can manage it, I guess I just prefer MySql to manage it instead
|
|
|
|
2112
Legendary
Offline
Activity: 2128
Merit: 1073
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 02, 2011, 11:06:33 PM |
|
I guess I just prefer MySql to manage it instead
And herein lies the problem. Switching from BerkeleyDB to any relational DBMS is a quite nontrivial tasks. Many people undertook it and nobody delivered a working code yet. Thankfully Gavin isn't accepting patches that modify the basic behavior of the Bitcoin with respect to block chain reorganizations. As for those who modify the underlying database without understanding the code: this was probably the story of MyBitcoin if one takes Ted Williams' posts on the face value. Some programmers completely cannot grasp the concept of chain reorganization and write code that is subject to an abuse that the unmodified Satoshi client is completely invulnerable.
|
|
|
|
Xenland
Legendary
Offline
Activity: 980
Merit: 1003
I'm not just any shaman, I'm a Sha256man
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 02, 2011, 11:14:28 PM |
|
I guess I just prefer MySql to manage it instead
And herein lies the problem. Switching from BerkeleyDB to any relational DBMS is a quite nontrivial tasks. Many people undertook it and nobody delivered a working code yet. Thankfully Gavin isn't accepting patches that modify the basic behavior of the Bitcoin with respect to block chain reorganizations. As for those who modify the underlying database without understanding the code: this was probably the story of MyBitcoin if one takes Ted Williams' posts on the face value. Some programmers completely cannot grasp the concept of chain reorganization and write code that is subject to an abuse that the unmodified Satoshi client is completely invulnerable. So in your opinion you believe that it is better to handle with berkelyDB given that you know what your doing?
|
|
|
|
2112
Legendary
Offline
Activity: 2128
Merit: 1073
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 02, 2011, 11:58:17 PM Last edit: September 03, 2011, 12:09:55 AM by 2112 |
|
So in your opinion you believe that it is better to handle with berkelyDB given that you know what your doing?
My professional opinion is as follows: (1) for the short-term do a recompilation and any small modifications to the Satoshi client to let it inter-operate correctly with the rest of your infrastructure; (2) for the long-term keep your fingers crossed that the libbitcoin team (bitcoinconsultancy?) delivers on their promises. BerkeleyDB isn't that difficult to master and the documentation is top-notch superb. I'm actually quite positively surprised with what had happened to it under Oracle's wing. The new release even has the SQL built in. But you'll have to at least recompile and possibly make number of small modifications.
|
|
|
|
Gavin Andresen
Legendary
Offline
Activity: 1652
Merit: 2300
Chief Scientist
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 03, 2011, 02:07:59 AM |
|
Optimizing the accounts code to add a berkeley db index table that indexed wallet transactions by account, and that cached account balances (and invalidated or updated the cache on receive/send) shouldn't be terribly hard for somebody who already knows c++ and berkeley db.
It is not on my short-term TODO list because there are too many other higher priority things on my TODO list, but a nice clean well-tested upward-compatible patch would be most welcome.
PS: for ClearCoin, I used the "bitcoind keeps track of the bitcoins" architecture, and I never regretted it-- no problems with synchronization, less possibility for MtGox-like hacks that create mythical bitcoin balances out of thin air by adding an entry to a database.
|
How often do you get the chance to work on a potentially world-changing project?
|
|
|
mtbitcoin (OP)
Legendary
Offline
Activity: 876
Merit: 1000
Etherscan.io
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 03, 2011, 10:49:10 AM |
|
Looking at the various arguements brought forward, I am going ahead with the "let bitcoind manage" the transactions approach using the "Accounts" feature as a means of storing balances. At the moment I'd rather deal with scalability issues vs having unscynchronized balances between a separate DB vs wallet.dat.
There does not seem to any performance metrics as to what would constitute an upper limit for storing transactions in the wallet.dat. I have tried running some simulations by dumping more than 200k "move" transactions into the wallet and so far it appears to be running fine. I am assuming that the hardware in use would also play a critical role as to how far this would scale. However, the real tests would be concurrent use.
Thank you for all your suggestions and if may add this in as a feature requests (somewhere down the long list), it would be nice to be able to store the transactions into a separate DB i.e odbc, mysql, etc
Cheers -MT
|
|
|
|
Steve
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 03, 2011, 01:09:38 PM |
|
You should talk to Jan at instawallet about scaling. I know he has struggled off and on with scaling and would probably be able to offer some good advice on what to do and what to avoid.
|
|
|
|
Jan
Legendary
Offline
Activity: 1043
Merit: 1002
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
September 04, 2011, 11:42:57 AM |
|
Looking at the various arguements brought forward, I am going ahead with the "let bitcoind manage" the transactions approach using the "Accounts" feature as a means of storing balances. At the moment I'd rather deal with scalability issues vs having unscynchronized balances between a separate DB vs wallet.dat.
If you want to scale really big with this approach you need several servers each running bitcoind, and each managing their share of accounts. A load balancer for this setup is fairly trivial to set up.
|
Mycelium let's you hold your private keys private.
|
|
|
|