Bitcoin Forum
August 19, 2018, 02:13:39 AM *
News: Latest stable version of Bitcoin Core: 0.16.2  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: What is the best SPV for handling large number of users?  (Read 3049 times)
BTCDDev
Sr. Member
****
Offline Offline

Activity: 256
Merit: 250


View Profile
October 28, 2015, 03:02:28 AM
 #1

Just like the title asked. I need an SPV for handling a large number of users. Needs to have support for accounting.

Running Ubuntu.

Currently looking at picocoin and libbitcoin.

BitcoinDark: RPHWc5CwP9YMMbvXQ4oXz5rQHb3pKkhaxc
Top Donations: juicybirds 420BTCD ensorcell 84BTCD Stuntruffle: 40BTCD
Top April Donations: juicybirds 420BTCD; ensorcell: 42BTCD
1534644819
Hero Member
*
Offline Offline

Posts: 1534644819

View Profile Personal Message (Offline)

Ignore
1534644819
Reply with quote  #2

1534644819
Report to moderator
1534644819
Hero Member
*
Offline Offline

Posts: 1534644819

View Profile Personal Message (Offline)

Ignore
1534644819
Reply with quote  #2

1534644819
Report to moderator
1534644819
Hero Member
*
Offline Offline

Posts: 1534644819

View Profile Personal Message (Offline)

Ignore
1534644819
Reply with quote  #2

1534644819
Report to moderator
BOUNTY PORTALS
BLOG
WHERE BOUNTY MANAGEMENT
MEETS AUTOMATION
SIGNATURE CAMPAIGNS
TWITTER
FACEBOOK
MEDIA CAMPAIGNS
AND MORE!
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1534644819
Hero Member
*
Offline Offline

Posts: 1534644819

View Profile Personal Message (Offline)

Ignore
1534644819
Reply with quote  #2

1534644819
Report to moderator
gmaxwell
Moderator
Legendary
*
Offline Offline

Activity: 2478
Merit: 1383



View Profile
October 28, 2015, 05:39:58 AM
 #2

Services providing services to other users should almost certainly not be using SPV.

Bitcoin will not be compromised
jim618
Legendary
*
Offline Offline

Activity: 1708
Merit: 1000



View Profile WWW
October 28, 2015, 01:51:13 PM
 #3

Yes got to agree with Greg on this.

We did some consultancy for an exchange and recommended they ran Core nodes for their network layer.
SPV is perfectly ok for single users but you want full nodes to support large numbers of users.

MultiBit HD   Lightweight desktop client.                    Bitcoin Solutions Ltd   Bespoke software. Consultancy.
BTCDDev
Sr. Member
****
Offline Offline

Activity: 256
Merit: 250


View Profile
October 28, 2015, 02:21:23 PM
 #4

Fair enough, running a full node was my original plan anyway.

However, I've only heard negative things about using bitcoind's accounting system, that it should not be used in production environments.

Alternative method?

BitcoinDark: RPHWc5CwP9YMMbvXQ4oXz5rQHb3pKkhaxc
Top Donations: juicybirds 420BTCD ensorcell 84BTCD Stuntruffle: 40BTCD
Top April Donations: juicybirds 420BTCD; ensorcell: 42BTCD
CIYAM
Legendary
*
Offline Offline

Activity: 1876
Merit: 1000


Ian Knowles - CIYAM Lead Developer


View Profile WWW
October 28, 2015, 02:41:14 PM
 #5

You most likely do not want to use the "bitcoind" idea of "accounts" (it is not the same idea that anyone who has ever done bookkeeping could make any sense of) and the wallet has scaling issues.

I think you are probably best to use the RPC to extract UTXO information from "bitcoind" into accounts that you keep in an RDBMS (from memory that will require a special flag for indexing but admittedly I haven't actually used that so you'll need to read up on exactly what that does).

The key command to retrieve UTXOs is "listunspent" although normally that will only output UTXOs that belong to your "bitcoind wallet".

Another very important thing to understand is you need to keep your RDBMS accounts "in sync" with the blockchain (small re-orgs of one or two blocks do occur and you should be very careful about unconfirmed txs).

EDIT: Hmm... it seems you still can't do "listunspent" for an address that is not in your wallet (am rather surprised that hasn't been added as it makes sense for "bitcoind" to have that feature so people don't resort to using things like blockchain.info). The indexing flag is only for txid lookup (which can be problematic due to malleability as mentioned).

With CIYAM anyone can create 100% generated C++ web applications in literally minutes.

GPG Public Key | 1ciyam3htJit1feGa26p2wQ4aw6KFTejU
_mr_e
Legendary
*
Offline Offline

Activity: 817
Merit: 1000



View Profile
October 28, 2015, 03:05:28 PM
 #6

You most likely do not want to use the "bitcoind" idea of "accounts" (it is not the same idea that anyone who has ever done bookkeeping could make any sense of) and the wallet has scaling issues.

I think you are probably best to use the RPC to extract UTXO information from "bitcoind" into accounts that you keep in an RDBMS (from memory that will require a special flag for indexing but admittedly I haven't actually used that so you'll need to read up on exactly what that does).

The key command to retrieve UTXOs is "listunspent" although normally that will only output UTXOs that belong to your "bitcoind wallet".

Another very important thing to understand is you need to keep your RDBMS accounts "in sync" with the blockchain (small re-orgs of one or two blocks do occur and you should be very careful about unconfirmed txs).


This sounds like something that should have already been done 1000x by now, is there really not a better way or some kind of library or wrapper that handles this stuff?
CIYAM
Legendary
*
Offline Offline

Activity: 1876
Merit: 1000


Ian Knowles - CIYAM Lead Developer


View Profile WWW
October 28, 2015, 03:11:09 PM
 #7

This sounds like something that should have already been done 1000x by now, is there really not a better way or some kind of library or wrapper that handles this stuff?

Although I think it is a pity that "listunspent" (assuming some sort of indexing flag has been put into the config) can't be used to find UTXO's for any address the main problem that you have in trying to link "bitcoind" to an external RDMBS is the whole "re-org" thing.

In the RDBMS world once you "commit" a transaction you can no longer "roll it back" but in the blockchain world this is simply not the case - thus marrying the two technologies is always going to result in some issues (as are often reported about for blockchain.info).

For the Bitcoin wallet I developed I have a "Check Balance" button to issue a "listunspent" in order to make sure it keeps in sync (of course that check could be automated but it would still need to be done prior to trying to create a new tx just in case the UTXO set has changed).

With CIYAM anyone can create 100% generated C++ web applications in literally minutes.

GPG Public Key | 1ciyam3htJit1feGa26p2wQ4aw6KFTejU
BTCDDev
Sr. Member
****
Offline Offline

Activity: 256
Merit: 250


View Profile
October 28, 2015, 03:14:08 PM
 #8

You most likely do not want to use the "bitcoind" idea of "accounts" (it is not the same idea that anyone who has ever done bookkeeping could make any sense of) and the wallet has scaling issues.

I think you are probably best to use the RPC to extract UTXO information from "bitcoind" into accounts that you keep in an RDBMS (from memory that will require a special flag for indexing but admittedly I haven't actually used that so you'll need to read up on exactly what that does).

The key command to retrieve UTXOs is "listunspent" although normally that will only output UTXOs that belong to your "bitcoind wallet".

Another very important thing to understand is you need to keep your RDBMS accounts "in sync" with the blockchain (small re-orgs of one or two blocks do occur and you should be very careful about unconfirmed txs).

EDIT: Hmm... it seems you still can't do "listunspent" for an address that is not in your wallet (am rather surprised that hasn't been added as it makes sense for "bitcoind" to have that feature so people don't resort to using things like blockchain.info). The indexing flag is only for txid lookup (which can be problematic due to malleability as mentioned).



This is ok for hot wallets, but I will want to keep large funds stored offline in a cold wallet.

I could keep cold wallet addresses in the hotwallet as watch-only, but how would I relate an unspent to a cold address to calculate the balance of a specific user?

BitcoinDark: RPHWc5CwP9YMMbvXQ4oXz5rQHb3pKkhaxc
Top Donations: juicybirds 420BTCD ensorcell 84BTCD Stuntruffle: 40BTCD
Top April Donations: juicybirds 420BTCD; ensorcell: 42BTCD
CIYAM
Legendary
*
Offline Offline

Activity: 1876
Merit: 1000


Ian Knowles - CIYAM Lead Developer


View Profile WWW
October 28, 2015, 03:19:09 PM
 #9

This is ok for hot wallets, but I will want to keep large funds stored offline in a cold wallet.

I could keep cold wallet addresses in the hotwallet as watch-only, but how would I relate an unspent to a cold address to calculate the balance of a specific user?

The "listunspent" output includes a "spendable" attribute (which I assume will be "false" for watch-only addresses). So you can determine the total balance and the "immediately spendable" balance also (if that is the concern).

Bear in mind if you do use a "bitcoind wallet" then you are going to have big problems once it has hundreds of thousands of addresses (from what I've read from others it doesn't scale well).

Although I've not looked into the code my guess is that even if you provide "listunspent" with a set of addresses it will iterate through every UTXO that the wallet contains (so will get slower and slower as the UTXOs increase).

With CIYAM anyone can create 100% generated C++ web applications in literally minutes.

GPG Public Key | 1ciyam3htJit1feGa26p2wQ4aw6KFTejU
CIYAM
Legendary
*
Offline Offline

Activity: 1876
Merit: 1000


Ian Knowles - CIYAM Lead Developer


View Profile WWW
October 29, 2015, 03:49:46 AM
 #10

I think perhaps look into running an Electrum server https://roll.urown.net/server/bitcoin/electrum-server.html (and https://github.com/spesmilo/electrum-server/) as it will index the UTXOs by address (the key thing needed really).

Note that it also requires "bitcoind" to be running (so you'll need a lot of disk space and presumably some decent hardware as well although if you are planning to service thousands of users am guessing you would already be planning for that).

With CIYAM anyone can create 100% generated C++ web applications in literally minutes.

GPG Public Key | 1ciyam3htJit1feGa26p2wQ4aw6KFTejU
BTCDDev
Sr. Member
****
Offline Offline

Activity: 256
Merit: 250


View Profile
October 29, 2015, 04:07:33 AM
 #11

I think perhaps look into running an Electrum server https://roll.urown.net/server/bitcoin/electrum-server.html (and https://github.com/spesmilo/electrum-server/) as it will index the UTXOs by address (the key thing needed really).

Note that it also requires "bitcoind" to be running (so you'll need a lot of disk space and presumably some decent hardware as well although if you are planning to service thousands of users am guessing you would already be planning for that).


I've got the hardware. I think I'll just use bitcoind and keep user addresses and balance in a rdbms, requiring a high level of confirms to reduce the chance of reorgs to near 0. If total wallet unspents become large, I'll move excess funds to multisig offline wallet.

BitcoinDark: RPHWc5CwP9YMMbvXQ4oXz5rQHb3pKkhaxc
Top Donations: juicybirds 420BTCD ensorcell 84BTCD Stuntruffle: 40BTCD
Top April Donations: juicybirds 420BTCD; ensorcell: 42BTCD
CIYAM
Legendary
*
Offline Offline

Activity: 1876
Merit: 1000


Ian Knowles - CIYAM Lead Developer


View Profile WWW
October 29, 2015, 04:25:15 AM
 #12

I've got the hardware. I think I'll just use bitcoind and keep user addresses and balance in a rdbms, requiring a high level of confirms to reduce the chance of reorgs to near 0. If total wallet unspents become large, I'll move excess funds to multisig offline wallet.

So basically for incoming funds you can use "listunspent" for the wallet determining account balances according to specific addresses (and adjusting your RDBMS account balances accordingly) and then move the funds to cold storage periodically.

So basically you need an index for Address and User (which presumably is only unique with the combination as each user should probably be encouraged to use different addresses for each deposit).

To handle re-orgs you might want to keep two balance figures one for amounts already shifted into cold storage and one for the total current set of UTXOs for each user and you'd probably only want to move into cold storage those UTXOs that are older than any re-org you think might occur (say 100 blocks to be safe even if a huge fork occurs).

You might also want to keep a third balance of "unconfirmed funds" just so a User can known that funds they have just sent are "on the way" (without them having to use blockchain.info or something else to check).

With CIYAM anyone can create 100% generated C++ web applications in literally minutes.

GPG Public Key | 1ciyam3htJit1feGa26p2wQ4aw6KFTejU
BTCDDev
Sr. Member
****
Offline Offline

Activity: 256
Merit: 250


View Profile
October 29, 2015, 04:46:52 AM
 #13

I've got the hardware. I think I'll just use bitcoind and keep user addresses and balance in a rdbms, requiring a high level of confirms to reduce the chance of reorgs to near 0. If total wallet unspents become large, I'll move excess funds to multisig offline wallet.

So basically for incoming funds you can use "listunspent" for the wallet determining account balances according to specific addresses (and adjusting your RDBMS account balances accordingly) and then move the funds to cold storage periodically.

So basically you need an index for Address and User (which presumably is only unique with the combination as each user should probably be encouraged to use different addresses for each deposit).

To handle re-orgs you might want to keep two balance figures one for amounts already shifted into cold storage and one for the total current set of UTXOs for each user and you'd probably only want to move into cold storage those UTXOs that are older than any re-org you think might occur (say 100 blocks to be safe even if a huge fork occurs).

You might also want to keep a third balance of "unconfirmed funds" just so a User can known that funds they have just sent are "on the way" (without them having to use blockchain.info or something else to check).


All good advice, thanks a lot.

Yes, I will mark an address as used or not, whenever a user wants to know his deposit address, if it's received funds already, it will generate him a new one.

I've been researching the bitcoind api, I am already familiar with zeromq (we use nanomsg in SuperNET) so it appeals to me as a much superior method of notifying my application of incoming funds than walletnotify (which spawns a new process(!) on each notification)

However, it looks like the zeromq flags don't allow for pushing the tx hashes of only those transactions which contain outputs to my wallet. I forked btc and added one in myself, if it works well then I may submit a pull request at some point. Seems like a big oversight.

Also, yeah the unconfirmed balance as a separate field is a good idea.

BitcoinDark: RPHWc5CwP9YMMbvXQ4oXz5rQHb3pKkhaxc
Top Donations: juicybirds 420BTCD ensorcell 84BTCD Stuntruffle: 40BTCD
Top April Donations: juicybirds 420BTCD; ensorcell: 42BTCD
Envrin
Sr. Member
****
Offline Offline

Activity: 318
Merit: 250



View Profile
October 29, 2015, 07:14:35 AM
 #14

http://synala.com/

Might do the trick for you.  Will easily support a large number of users, can handle both product purchases and invoices, keeps all tx data you need, generates a new address for each deposit, has a hook system so you can easily run additional PHP code when various actions occur, etc.  Free, and open sourced on Github.  It does require a full node on your server though, although doesn't use the wallet.dat file.

If you need a software system that's a little more commercial / enterprise level, we do have that available, and just drop me a PM with requirements.
CIYAM
Legendary
*
Offline Offline

Activity: 1876
Merit: 1000


Ian Knowles - CIYAM Lead Developer


View Profile WWW
October 29, 2015, 08:20:43 AM
 #15


Looks interesting although from the "How secure are incoming funds?" FAQ I found this:

Quote
...While Synala does allow the convenience of online singing...

Am guessing you meant "signing". Cheesy

With CIYAM anyone can create 100% generated C++ web applications in literally minutes.

GPG Public Key | 1ciyam3htJit1feGa26p2wQ4aw6KFTejU
Envrin
Sr. Member
****
Offline Offline

Activity: 318
Merit: 250



View Profile
October 30, 2015, 01:06:56 AM
 #16


Whoops, thanks for pointing the small typo out.  All fixed. Smiley
BTCDDev
Sr. Member
****
Offline Offline

Activity: 256
Merit: 250


View Profile
November 02, 2015, 06:53:47 PM
 #17

A little update, I've got it working really well, associating user addresses (unique for each deposit) with each user in my rdbms, implementing the cold wallet storage now.

It was tricky to efficiently monitor user deposits in real time, as currently you need to use walletnotify, which as I stated before launches a new process each time an incoming tx is seen, and the zeromq implementation only allows for sending every seen tx on the network, which would force me to use rpc call for EVERY tx!

I opened a pull request here https://github.com/bitcoin/bitcoin/pull/6925

but it was closed, so for now others who want to monitor deposits will have to do so in a loop tx-->bitcoind-->my application-->bitcoind rpc-->my application




BitcoinDark: RPHWc5CwP9YMMbvXQ4oXz5rQHb3pKkhaxc
Top Donations: juicybirds 420BTCD ensorcell 84BTCD Stuntruffle: 40BTCD
Top April Donations: juicybirds 420BTCD; ensorcell: 42BTCD
tl121
Sr. Member
****
Offline Offline

Activity: 279
Merit: 250


View Profile
November 07, 2015, 12:56:33 AM
 #18

Services providing services to other users should almost certainly not be using SPV.

It would be very helpful if you would explain your reasoning behind this.  From a functional point of view, SPV either works or it doesn't.  If it doesn't work properly on behalf of multiple users, it's not clear why it would work properly for a single user.  If there is a performance problem with SPV supporting a service with multiple users, it's not clear why there would be a performance problem with those same nodes supporting multiple SPV clients.

This seems to be related to the basic design of bitcoin core, which has (since the beginning) combined three separate functions that would be clearly separated with well defined interfaces in any proper software engineered implementation:  1) maintaining the distributed ledger and associated local database, 2) providing database and notifications on behalf of client applications, and 3) providing an easy to use graphic user interface suitable for bitcoin newbies.



Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!