Bitcoin Forum
December 11, 2016, 06:23:08 AM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: [1] 2 »  All
  Print  
Author Topic: Scalability of BitCoinD for a wallet service  (Read 2416 times)
mtbitcoin
Legendary
*
Offline Offline

Activity: 876


Etherscan.io


View Profile
August 31, 2011, 04:58:11 AM
 #1

Hi

I am builing a free wallet service (like instawallet) and would like to know what are your thoughts on solely relying on the bitcoind daemon for storing all transactions as opposed to storing them in an external database for lookups. I have concerns that if we were to run into hundreds of users and transactions that bitcoind would not be able to handle such volumes concurrently.

Are there any hard limits of to how far the daemon can scale or is this limited by the underlying hardware?

Thanks

EtherScan::Ethereum Block Explorer | BlockScan::Counterparty Block Explorer
1481437388
Hero Member
*
Offline Offline

Posts: 1481437388

View Profile Personal Message (Offline)

Ignore
1481437388
Reply with quote  #2

1481437388
Report to moderator
1481437388
Hero Member
*
Offline Offline

Posts: 1481437388

View Profile Personal Message (Offline)

Ignore
1481437388
Reply with quote  #2

1481437388
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481437388
Hero Member
*
Offline Offline

Posts: 1481437388

View Profile Personal Message (Offline)

Ignore
1481437388
Reply with quote  #2

1481437388
Report to moderator
1481437388
Hero Member
*
Offline Offline

Posts: 1481437388

View Profile Personal Message (Offline)

Ignore
1481437388
Reply with quote  #2

1481437388
Report to moderator
1481437388
Hero Member
*
Offline Offline

Posts: 1481437388

View Profile Personal Message (Offline)

Ignore
1481437388
Reply with quote  #2

1481437388
Report to moderator
Gavin Andresen
Legendary
*
qt
Offline Offline

Activity: 1652


Chief Scientist


View Profile WWW
September 01, 2011, 01:51:27 PM
 #2

Are there any hard limits of to how far the daemon can scale or is this limited by the underlying hardware?
No hard limits, but the bitcoind "accounts" code hasn't been optimized for that.  For example, computing an account's balance loops through all transactions in the wallet, so will take an increasing amount of time the more transactions you have in the wallet.


How often do you get the chance to work on a potentially world-changing project?
mtbitcoin
Legendary
*
Offline Offline

Activity: 876


Etherscan.io


View Profile
September 02, 2011, 11:08:16 AM
 #3

Hi Gavin

Thank you for taking the time to reply

Any suggestions as to how one should go about this then for high volume transactions? I've looked at the previous posts and wiki entries, most of these points towards using the "Accounts" feature. If the transactions were recorded to an external database, the biggest issue i see with this is polling the  addresses when a payment is expected and also dealing with redundant payments.

Thanks
-MT

EtherScan::Ethereum Block Explorer | BlockScan::Counterparty Block Explorer
kjj
Legendary
*
Offline Offline

Activity: 1302



View Profile
September 02, 2011, 03:38:30 PM
 #4

Can I ask why you don't want to use your own code to manage balances and transactions?  It seems to me that you need to do those things anyway, at least in most scenarios that I'm familiar with.

p2pcoin: a USB/CD/PXE p2pool miner - 1N8ZXx2cuMzqBYSK72X4DAy1UdDbZQNPLf - todo
I routinely ignore posters with paid advertising in their sigs.  You should too.
blueadept
Full Member
***
Offline Offline

Activity: 225


View Profile
September 02, 2011, 05:25:26 PM
 #5

I'd take a look at this thread:

https://bitcointalk.org/index.php?topic=25094.0

This fork makes it easier to poll/import transactions into your own DB and then do whatever you need with them rather than relying on bitcoind to keep accounts straight.  I'm using it for my own purposes (though I'm not accessing it from PHP but Python; I still need to keep track of transactions in a RDBMS).

Like my posts?  Connect with me on LinkedIn and endorse my "Bitcoin" skill.
Decentralized, instant off-chain payments.
2112
Legendary
*
Offline Offline

Activity: 1708



View Profile
September 02, 2011, 07:00:56 PM
 #6

Can I ask why you don't want to use your own code to manage balances and transactions?  
I'm not mtbitcoin, but I can give you an answer to this: the current code is horribly complex and doesn't obey any rules demanded by any legitimate accounting methods. For example it routinely re-dates the transactions and can disappear them upon block-chain reorganization.

The whole idea of stochastically-solve-the-inverted-one-dimensional-knapsack-problem before you can compute the transaction fee is hair raising to any legitimate accountant. On the other hand it is music to the ears of people planning to pull some sort of a fraud.

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
mtbitcoin
Legendary
*
Offline Offline

Activity: 876


Etherscan.io


View Profile
September 02, 2011, 07:35:26 PM
 #7

Can I ask why you don't want to use your own code to manage balances and transactions?  It seems to me that you need to do those things anyway, at least in most scenarios that I'm familiar with.

By having the bitcoind manage the balance and transactions I could avoid any synchronization and integration issues. This would also simplify the development process

EtherScan::Ethereum Block Explorer | BlockScan::Counterparty Block Explorer
mtbitcoin
Legendary
*
Offline Offline

Activity: 876


Etherscan.io


View Profile
September 02, 2011, 07:40:07 PM
 #8

I'd take a look at this thread:

https://bitcointalk.org/index.php?topic=25094.0

This fork makes it easier to poll/import transactions into your own DB and then do whatever you need with them rather than relying on bitcoind to keep accounts straight.  I'm using it for my own purposes (though I'm not accessing it from PHP but Python; I still need to keep track of transactions in a RDBMS).

Thank you for the tip. However, all things being equal I would prefer to stick with the mainstream releases

EtherScan::Ethereum Block Explorer | BlockScan::Counterparty Block Explorer
Xenland
Legendary
*
Offline Offline

Activity: 980


I'm not just any shaman, I'm a Sha256man


View Profile
September 02, 2011, 08:39:48 PM
 #9

Honestly, I think its more work to go based off the bit coin daemon, if you want to move funds from a receiving bit coin address to a specific address in the same wallet it will have to go through the block chain which will add more wait time in waiting for even just 1 confirmation. Its just easier to use a database and periodically scan your transactions and then update the database with Bitcoin addresses with corresponding information like confirmations, balance, etc,etc. This method can help save block time and its more efficient to look up account information as well, since if you relied on the daemon for information you'd have to rescan every request which is inefficient. With a database you can cache all data in the wallet in one swoop then have the program query the database.
2112
Legendary
*
Offline Offline

Activity: 1708



View Profile
September 02, 2011, 09:14:39 PM
 #10

Honestly, I think its more work to go based off the bit coin daemon, if you want to move funds from a receiving bit coin address to a specific address in the same wallet it will have to go through the block chain which will add more wait time in waiting for even just 1 confirmation.
You seem to be missing two things:
1) RPC move between the accounts instantly
2) wallet.dat can be modified live with correctly compiled BerkeleyDB program that uses the same DBENV. wallet.db is a database and BerkDB is a quite powerful DBMS, its just that most programmers (including the 'core development group') are unfamiliar with it and don't understand its features.

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
Xenland
Legendary
*
Offline Offline

Activity: 980


I'm not just any shaman, I'm a Sha256man


View Profile
September 02, 2011, 10:43:00 PM
 #11

Honestly, I think its more work to go based off the bit coin daemon, if you want to move funds from a receiving bit coin address to a specific address in the same wallet it will have to go through the block chain which will add more wait time in waiting for even just 1 confirmation.
You seem to be missing two things:
1) RPC move between the accounts instantly
2) wallet.dat can be modified live with correctly compiled BerkeleyDB program that uses the same DBENV. wallet.db is a database and BerkDB is a quite powerful DBMS, its just that most programmers (including the 'core development group') are unfamiliar with it and don't understand its features.


So your saying I can move bitcoins from one bitcoin address to another bitcoin address with out waiting for confirmations from the block chain?

can anyone else verify this?
2112
Legendary
*
Offline Offline

Activity: 1708



View Profile
September 02, 2011, 10:46:58 PM
 #12

So your saying I can move bitcoins from one bitcoin address to another bitcoin address with out waiting for confirmations from the block chain?
Not "bitcoin address" but "bitcoin account". Please read carefully the output from "bitcoind help" and the original post.

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
Xenland
Legendary
*
Offline Offline

Activity: 980


I'm not just any shaman, I'm a Sha256man


View Profile
September 02, 2011, 10:52:56 PM
 #13

Okay yeah so in that case yeah you can manage it, I guess I just prefer MySql to manage it instead
2112
Legendary
*
Offline Offline

Activity: 1708



View Profile
September 02, 2011, 11:06:33 PM
 #14

I guess I just prefer MySql to manage it instead
And herein lies the problem. Switching from BerkeleyDB to any relational DBMS is a quite nontrivial tasks. Many people undertook it and nobody delivered a working code yet. Thankfully Gavin isn't accepting patches that modify the basic behavior of the Bitcoin with respect to block chain reorganizations.

As for those who modify the underlying database without understanding the code: this was probably the story of MyBitcoin if one takes Ted Williams' posts on the face value. Some programmers completely cannot grasp the concept of chain reorganization and write code that is subject to an abuse that the unmodified Satoshi client is completely invulnerable.

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
Xenland
Legendary
*
Offline Offline

Activity: 980


I'm not just any shaman, I'm a Sha256man


View Profile
September 02, 2011, 11:14:28 PM
 #15

I guess I just prefer MySql to manage it instead
And herein lies the problem. Switching from BerkeleyDB to any relational DBMS is a quite nontrivial tasks. Many people undertook it and nobody delivered a working code yet. Thankfully Gavin isn't accepting patches that modify the basic behavior of the Bitcoin with respect to block chain reorganizations.

As for those who modify the underlying database without understanding the code: this was probably the story of MyBitcoin if one takes Ted Williams' posts on the face value. Some programmers completely cannot grasp the concept of chain reorganization and write code that is subject to an abuse that the unmodified Satoshi client is completely invulnerable.

So in your opinion you believe that it is better to handle with berkelyDB given that you know what your doing?
2112
Legendary
*
Offline Offline

Activity: 1708



View Profile
September 02, 2011, 11:58:17 PM
 #16

So in your opinion you believe that it is better to handle with berkelyDB given that you know what your doing?
My professional opinion is as follows: (1) for the short-term do a recompilation and any small modifications to the Satoshi client to let it inter-operate correctly with the rest of your infrastructure; (2) for the long-term keep your fingers crossed that the libbitcoin team (bitcoinconsultancy?) delivers on their promises.

BerkeleyDB isn't that difficult to master and the documentation is top-notch superb. I'm actually quite positively surprised with what had happened to it under Oracle's wing. The new release even has the SQL built in. But you'll have to at least recompile and possibly make number of small modifications.

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
Gavin Andresen
Legendary
*
qt
Offline Offline

Activity: 1652


Chief Scientist


View Profile WWW
September 03, 2011, 02:07:59 AM
 #17

Optimizing the accounts code to add a berkeley db index table that indexed wallet transactions by account, and that cached account balances (and invalidated or updated the cache on receive/send) shouldn't be terribly hard for somebody who already knows c++ and berkeley db.

It is not on my short-term TODO list because there are too many other higher priority things on my TODO list, but a nice clean well-tested upward-compatible patch would be most welcome.

PS:  for ClearCoin, I used the "bitcoind keeps track of the bitcoins" architecture, and I never regretted it-- no problems with synchronization, less possibility for MtGox-like hacks that create mythical bitcoin balances out of thin air by adding an entry to a database.

How often do you get the chance to work on a potentially world-changing project?
mtbitcoin
Legendary
*
Offline Offline

Activity: 876


Etherscan.io


View Profile
September 03, 2011, 10:49:10 AM
 #18

Looking at the various arguements brought forward, I am going ahead with the "let bitcoind manage" the transactions approach using the "Accounts" feature as a means of storing balances. At the moment I'd rather deal with scalability issues vs having unscynchronized balances between a separate DB vs wallet.dat.

There does not seem to any performance metrics as to what would constitute an upper limit for storing transactions in the wallet.dat. I have tried running some simulations by dumping more than 200k "move" transactions into the wallet and so far it appears to be running fine. I am assuming that the hardware in use would also play a critical role as to how far this would scale. However, the real tests would be concurrent use.

Thank you for all your suggestions and if may add this in as a feature requests (somewhere down the long list), it would be nice to be able to store the transactions into a separate DB i.e odbc, mysql, etc

Cheers
-MT

EtherScan::Ethereum Block Explorer | BlockScan::Counterparty Block Explorer
Steve
Hero Member
*****
Offline Offline

Activity: 868



View Profile WWW
September 03, 2011, 01:09:38 PM
 #19

You should talk to Jan at instawallet about scaling.  I know he has struggled off and on with scaling and would probably be able to offer some good advice on what to do and what to avoid.

(gasteve on IRC) Does your website accept cash? https://bitpay.com
Jan
Legendary
*
Offline Offline

Activity: 1042



View Profile
September 04, 2011, 11:42:57 AM
 #20

Looking at the various arguements brought forward, I am going ahead with the "let bitcoind manage" the transactions approach using the "Accounts" feature as a means of storing balances. At the moment I'd rather deal with scalability issues vs having unscynchronized balances between a separate DB vs wallet.dat.

If you want to scale really big with this approach you need several servers each running bitcoind, and each managing their share of accounts. A load balancer for this setup is fairly trivial to set up.

Mycelium let's you hold your private keys private.
Pages: [1] 2 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!