Bitcoin Forum

Hi, I am looking for bitcoind alternative for server app. It should have RPC API and similar functions as bitcoind - accounts and transaction functions, address functions, everything accessible in specific account. (account has many addresses and transactions, etc).
Which wallet do mining pools use?

bitcoinJ is not suitable for server:
source: https://code.google.com/p/bitcoinj/wiki/Limitations (https://code.google.com/p/bitcoinj/wiki/Limitations)

bitcoind is nightmare:
https://github.com/bitcoin/bitcoin/pull/2703 (https://github.com/bitcoin/bitcoin/pull/2703) - yeah, documented command which does not work as documented. Why not? Worse is, this makes use of bitcoind accounts potentially dangerous because one can not easily implement check balance and move atomically.
Semaphore can be used (php, does not work on Windows) http://www.re-cycledair.com/php-dark-arts-semaphores (http://www.re-cycledair.com/php-dark-arts-semaphores)
On Windows, blocking can be implemented: https://locallost.net/?p=1091 (https://locallost.net/?p=1091)
But, both methods have impact on performance.
No watching only waller support (without impractical workarounds)

Thank you.

no ideas?

I think (subjectively) bitcoind does well at the network protocol but does not so well at managing all the data. The data can be more structured thus easier for use rather than being raw, but then the data would take up more disk space, which would push bitcoind farther away from personal desktop use. I have heard (but not actually tried to test it myself) some people have customized the bitcoind so that it can be used only as a network filter that handles all the inbound/outbound network messages, wallets and accounts managements are handled by homemade utilities.

accounts might be removed altogether. I'm not sure what you're trying to achieve/what the problem is. you can try btcd, but it's not widely used in production AFAIK. it works pretty smoothly and has websockets for example. Supporting alternative implementations is probably not even a good idea, but clearly this is how things should be. bitcoind is one monolithic blob.

I think current state of affairs with bitcoind is very sub-optimal in many ways. just shows how young this project really is. it would help if there where more developers. I don't understand what people are working on. I mean there isn't even proper documentation and install scripts. The core devs have other things to do, and there are not many supporting devs.

Yes, it is not optimal in managing accounts. But still provides better/faster solution to custom solution with limited human resources.
I am trying to implement online service which needs user accounts and access to bitcoin network.
I have to know unconfirmed balance of each customer and I need to know confirmed balance of each customer. I need to be able to link addresses to transactions. I need micropayments (the bitcoind provides move command).
Coinbase or blockchain.info are not suitable, because they do not have as good APIs as I need and it would take 3 coinbase-accounts-per-customer design to achieve the same functionality as minconf parameter in bitcoind API provides - coinbase do not provide such thing directly. blockchain.info does, but does not provide microtransactions. Those services provide high level presentation APIs, not tools for app development.
And I do not want to be dependent on third party APIs.
Sure, I could build everything by myself, but there are many problems such as data integrity between my app and wallet software, HUGE number of problems with non-atomic functionality, etc.

So, my question is, what wallet and account solution mining pools use? What software cryptsy, btc-e and bitstamp use?

custom stuff. it's not easy/cheap to have 100% security. there is no account solution. you have bitcoind which generates N addresses. then you have your own account database and do a 1:1 mapping. move is a simple as doing an update on your own database.


I don't understand - what are the details? no bitcoin transaction is atomic for small latencies. what kind of micropayments?? bitcoind has not implemented micropayments, only bitcoinj. I've not seen anybody using it.

https://blog.conformal.com/btcd-not-your-moms-bitcoin-daemon/

yes, I've tested btcd and has websockets, it's very well documented, etc. It uses go-channels which make the threading model much better.

But if you implement things on top of it, you have to put in a lot of work to ensure this works as you expect. So if you want to handle a lot of money it's unlikely to be a good option.

For micropayments see : https://code.google.com/p/bitcoinj/wiki/WorkingWithMicropayments

Note:
if you handle a lot of money you have to know what you're doing. you might be better off using merchant API's.

Don't use account.  Don't even plan on using accounts.  It is an uttterly worthless "feature" which has no place in bitcoind to begin with.

Bitcoind works flawlessly for its core functionality and that is running as a full node & wallet. "Account" level features should be in your application business logic and put in your application data store.

bitcoind for network connectivity and transaction processing.  With all high level functions in custom application and database.

If you had an application which needed to connect to the US banking network would you build it all inside your banks online banking interface?  Of course not.  You would build a high level application that issued commands and pulled data from your banks interface.

Look into libbitcoin and obelisk. Full rewrites of bitcoin with performance in mind.

Data integrity: Lets say I do not use bitcoind accounts. I have N addresses in Wallet and my own SQL database with addr-user (n:1) mapping. I really need user to have many addresses. So, when I display addr A to some user, I have to check for transactions to the addr A in some point in time and update my SQL - cron could be my friend in this case.
But, I have to scan all the addresses in my SQL with current balance to find out user balance! I have to use "deep" scan - I can not assume each address has static balance - the wallet sends btc from addresses it decides are optimal. So, I have to use for example blockchain.info API where I can pass many addresses with minconf and it returns balance from corresponding TXs which have minconf confirmations.
Edit: yes, it is abstraction "address has balance" - addresses have transactions, transactions have amounts, so "addresses have balances"
I am sure bitcoind's rpc functions do  not support many addresses as input and not sure if it could retrieve balance at minconf from one addr either.
It is pointless to cache user balances in SQL, because it may be changed at every user withdrawal. No way to say "send ONLY from THIS address".
And many more problems...
Atomic or synchronization problems: User wants withdrawal:
0. I need to set set semaphore/lock or make this withdrawal function serialized.
1. I deep-rescan his balance B1.
2. From this moment, the actual balance A1 >= B1 (he could send funds to his addr in the meantime).
3. The second withdrawal (attack attempt) comes at this time. Because of point zero 0, it waits blocked.
4a. If point 0. does not exists, each of two or more server instances/threads retrieved balance B1 (in fact, B1 and B2, B1==B2 or B1!=B2) and I command my wallet "send B1 amount to some user address" - two times!
4b. If point 0. exists, I commant my wallet "send B1 amount to some user address" - only once.
5. Withdrawal function ends and unlocks second attempt to withdrawal, which continues from point 1.

bitcoind provides sendfrom <account> <amount> [minconfirm=1] which is atomic function and fails if there is not enough balance "in the minconfirm state". test-and-do atomic operation.

The bitcoind account functionality in terms of problems like this is almost perfect. Yes, there are problems like it test-and-do balance with no fees includes and it may results in negative account balance, but this is intended and there are reasons for this behavior.
It is difficult (at least for me) to code two separate systems which must be synchronized.

Before you try to build whatever you're building, you really should learn how Bitcoin works so that you don't base your application's design on incorrect assumptions.

Thank, I will check that. But I am worry about Go's performance.

See bitcoinJ limitations https://code.google.com/p/bitcoinj/wiki/Limitations (https://code.google.com/p/bitcoinj/wiki/Limitations)
It is suitable for desktop wallets, but not for server apps.

That would be the worst possible solution.  bitcoind supports block and tx callback notifications.  At an application level you don't need to be concerned with the "balance at each address" (an abstraction which doesn't even exist at the protocol level) instead you only care about the balance per user.  When a tx comes in, determine which user it belongs to.



Once again who cares what the current balance of an individual address is, in a pool fund situation.  Joe has deposited 5 BTC, you owe Joe 5 BTC, Joe's "balance" is 5 BTC.  

Of course it changes on ever withdraw (and every transfer and ever deposit).

If a user has the following tx stored in the database
Deposit 10 BTC to deposit address x, tx id y
Withdraw 2 BTC to address x, tx id y
Transfer 1 BTC to user z

What is the user's current balance?

If those records were in a table couldn't you get the balance at any moment in time with a SELECT?


Then the issue isn't coding but architecture.    Forget Bitcoin a second.  Imagine you ran a USD based service.  Customers can deposit funds, customers can withdraw funds, customers can transfer funds to other customers accounts.  Are you telling me there is no possible way you could keep track of user balances without doing it all inside the banking system?   If the answer is no then replace bank/USD with bitcoind/BTC and that is your solution.  If the answer is yes, then please don't develop or launch a service without a lot more research.

Even if bitcoind did everything you wanted it to do it doesn't scale trying to use it as a real time balance data feed.  Use it as it was intended, as a interface to the bitcoin network.  You can then scale out if needed (multiple instances).

Have you read the full context?
The design was about Bitcoin integration with web service in the wrong way.
Why do you think I do not know how Bitcoin works and at what level of understanding I should be to code the app?

For one thing, you're talking about the balance of an address. That means it's virtually certain that you're tracking the wrong things.

D&T appears to have more time/patience than I so you should ask him about it.

The design I provided was example how not to do it. Read the context and quote above it.

I was refering to blockchain.info API which provides "balance of multiple address".

I understand that balance is retrieved from TXs. But TXs can not be assigned to users without addresses in my case - I need abstractions "addresses have balance" or, "account has balance"

I agree it has to be done this way in big projects. The question is if bitcoind API is good enough for my project. The answer is "no", but not because the design, but because it sucks in implementation.

Well to clarify bitcoind "core" works as expected.  IMHO the accounts functionality have always been next to worthless (there are other edge cases which can cause unexpected problems).   Most people wouldn't miss it if accounts were moved completely and it would avoid these kind of x-y questions.

there was a discussion about the account system recently: https://github.com/bitcoin/bitcoin/issues/3664 it was argued to get rid of accounts for the reasons mentioned above. instead it was suggested to (re-) introduce tagging.

what i would like to say are some server side implementations. I just went through all github repos with the word bitcoin in them and on the server side the only thing I have found is this.
https://github.com/kangasbros/django-bitcoin

there is definitely a lot of room for many more sample apps for a more plug & play environment. most merchant integration happens via commercial API's which is not good IMO.


you have a set of balances in your database which reflect past payments to a set of addresses. then you rescan and they are up to date. there is no realtime aspect to this. you have to keep track of all internal tx, to prevent double spend. any tx going outside need confirmations, see also https://en.bitcoin.it/wiki/Transaction_Malleability

I think bitcoind accounts are not worthless because custom reimplementation of the functionality cost long time for each custom implementation and testing.
It is already implemented and tested and provides methods you would maybe implemented, so it saves time in general.
Yes, if you implement totally custom account ledger because it has to be specific, bitcoind account functions are useless - you just need assign id to each address or whatever.
But in my situation I do not have resources (time/people) to reimplement it safely with testing.
Its bad design decision that the same wallet user use as client is on the server. But the client is still not final, so...

I would say that it is easier to be aware of and fix edge cases than to properly reimplement accounts.
What problems bitcoind accounts have? Everyone across the web says about problems, but I have encountered only a few:
1. Fee is added after sendfrom method internal check, so account may have (slightly) negative balance after withdrawal.
2. In the move method, the minconfirmations parameter does not work and is useless - but the wiki documentation describes it. The workaround is to use semaphore/lock/synchronized method/synchronized block and test balance, then move.
3. "" account is weird and I do not like that some methods work with "" implicitly. I just made my RPC interface ignore the methods and "" account.
4. No watching-only wallet support. This is the main reason I am looking for bitcoind alternative, because the cold storage design rocks. I am fine with warning and forcing users to not having large amounts in my app until there is nice cold storage solution or my app became more popular (do not worry, I am not launching any magic). I know the pywallet workaround with keypool and AFAIK it is tbe best solution for bitcoind.

Thanks for the django-bitcoin, seems interesting.
Yes, I am aware of issues of unconfirmed transactions. bitcoind's balance related functions have optional minconf parameter, so working with balances is practical and secure, so I hope the recommended software in this thread have rich APIs

Why do you need minconf? also forget about accounts. they will likely be removed. an account system is nothing else than a custom tags for addresses you control anyway.

with rpc you can simple iterate over the addresses and check for number of confirms. here is a snippet for bitcoinrpc.


Yes, you need atomicity. The way you do that is check for internal orders versus withdrawals. Say you have a service like eBay running on bitcoind. If userA has 1 BTC and purchases a good from userB, you need to make sure userA does not place an order and withdraws at the same time. you need to calculate the correct balance at each time, and withdrawals have to be subtracted immediately. then you can add extra security measures depending on your needs. you simply can't do bitcoin transactions in realtime (without a dedicated channel). you have to let users deposit and withdraw through the blockchain and make sure on your side that you have not an internal double spend/malleability atttack (https://en.bitcoin.it/wiki/Transaction_Malleability).

it would be very good if bitcoind would introduce a proper threading model /hooks (like btcd). It is however extremely unlikely that you reach a performance bottleneck with this pattern, and core devs have other things to worry about. this is largely the problem that not enough code is being shared, but build by merchants. merchants/exchanges are too busy making money, instead of helping the ecosystem, if they are not busy losing money for customers.

again, remember bitcoin does not operate in realtime. every unconfirmed transaction is to be treated as a double spend/fraud.

minconf is useful because I can get users unconfirmed balance, confirmed balance (3x/6x).. So, I can decide if it is OK to resend bitcoins away or move value to other account - for example I allow withdrawal only with 6 confirms but allow to move offblockchain with 3 confirms.

If I wanted to reimplement getbalance I have to use the algo you suggest: retrieve all the addresses, skip irrelevant, skipp addr with confirm. of last TX < 6...
If I have thousands of addresses and many users I would skip most of them. It is probable that the bitcoind's implementation is efficient (I do not know), but at least already tested and implemented so I can use getbalance account minconf
The atomicity: If I went to implement atomic send, I need to [lockA]; getBalance; checkBalance; ?withdraw; [unlockA] and [lockA]; getBalance; checkBalance; ?payForSomething; [unlockA]
This can be replaced with atomic sendfrom. Yes, I have to implement my own lock [lockMove] getBalance; checkBalance; move [unlockMove] and some locks on higher layer of my app, but thats it.
As prototype of my app, bitcoind accounts are suitable. Its not the design which I do not like, but the implementation.
I will change the design with new bitcoind without accounts (next 0.9 should have accounts) or when some other client has watching only wallet for server.

Is there any opensource alternative to bitcoind + pywallet + keypool workaround for watching only wallet?

AFAIK the current 0.8.6 does not have problem with transaction malleability. http://bitcoin.stackexchange.com/questions/22304/is-transaction-malleability-an-issue-when-receiving-payments (http://bitcoin.stackexchange.com/questions/22304/is-transaction-malleability-an-issue-when-receiving-payments)
Should I know something?

I've been wondering that as well...