Bitcoin Forum
November 17, 2018, 01:46:36 AM *
News: Latest Bitcoin Core release: 0.17.0 [Torrent].
 
   Home   Help Search Login Register More  
Pages: [1] 2 »  All
  Print  
Author Topic: Bitcoind alternative - for server  (Read 2522 times)
miso
Newbie
*
Offline Offline

Activity: 37
Merit: 0


View Profile
March 08, 2014, 03:30:42 AM
 #1

Hi, I am looking for bitcoind alternative for server app. It should have RPC API and similar functions as bitcoind - accounts and transaction functions, address functions, everything accessible in specific account. (account has many addresses and transactions, etc).
Which wallet do mining pools use?

bitcoinJ is not suitable for server:
Quote
All transactions that were ever relevant to the wallet are loaded into memory, all the time, and re-written every time the wallet is saved. ... the algorithms used for selecting coins and working with unspent transactions have quadratic time complexity or worse. In time we'll probably switch to a log structured wallet file format to solve this.
source: https://code.google.com/p/bitcoinj/wiki/Limitations

bitcoind is nightmare:
https://github.com/bitcoin/bitcoin/pull/2703 - yeah, documented command which does not work as documented. Why not? Worse is, this makes use of bitcoind accounts potentially dangerous because one can not easily implement check balance and move atomically.
Semaphore can be used (php, does not work on Windows) http://www.re-cycledair.com/php-dark-arts-semaphores
On Windows, blocking can be implemented: https://locallost.net/?p=1091
But, both methods have impact on performance.
No watching only waller support (without impractical workarounds)

Thank you.
1542419196
Hero Member
*
Offline Offline

Posts: 1542419196

View Profile Personal Message (Offline)

Ignore
1542419196
Reply with quote  #2

1542419196
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1542419196
Hero Member
*
Offline Offline

Posts: 1542419196

View Profile Personal Message (Offline)

Ignore
1542419196
Reply with quote  #2

1542419196
Report to moderator
1542419196
Hero Member
*
Offline Offline

Posts: 1542419196

View Profile Personal Message (Offline)

Ignore
1542419196
Reply with quote  #2

1542419196
Report to moderator
miso
Newbie
*
Offline Offline

Activity: 37
Merit: 0


View Profile
March 08, 2014, 11:37:19 AM
 #2

no ideas?
ning
Full Member
***
Offline Offline

Activity: 173
Merit: 100



View Profile
March 08, 2014, 12:30:33 PM
 #3

I think (subjectively) bitcoind does well at the network protocol but does not so well at managing all the data. The data can be more structured thus easier for use rather than being raw, but then the data would take up more disk space, which would push bitcoind farther away from personal desktop use. I have heard (but not actually tried to test it myself) some people have customized the bitcoind so that it can be used only as a network filter that handles all the inbound/outbound network messages, wallets and accounts managements are handled by homemade utilities.
benjyz
Full Member
***
Offline Offline

Activity: 140
Merit: 100


View Profile
March 08, 2014, 12:54:51 PM
 #4

accounts might be removed altogether. I'm not sure what you're trying to achieve/what the problem is. you can try btcd, but it's not widely used in production AFAIK. it works pretty smoothly and has websockets for example. Supporting alternative implementations is probably not even a good idea, but clearly this is how things should be. bitcoind is one monolithic blob.

I think current state of affairs with bitcoind is very sub-optimal in many ways. just shows how young this project really is. it would help if there where more developers. I don't understand what people are working on. I mean there isn't even proper documentation and install scripts. The core devs have other things to do, and there are not many supporting devs.
miso
Newbie
*
Offline Offline

Activity: 37
Merit: 0


View Profile
March 08, 2014, 01:27:33 PM
 #5

Yes, it is not optimal in managing accounts. But still provides better/faster solution to custom solution with limited human resources.
I am trying to implement online service which needs user accounts and access to bitcoin network.
I have to know unconfirmed balance of each customer and I need to know confirmed balance of each customer. I need to be able to link addresses to transactions. I need micropayments (the bitcoind provides move command).
Coinbase or blockchain.info are not suitable, because they do not have as good APIs as I need and it would take 3 coinbase-accounts-per-customer design to achieve the same functionality as minconf parameter in bitcoind API provides - coinbase do not provide such thing directly. blockchain.info does, but does not provide microtransactions. Those services provide high level presentation APIs, not tools for app development.
And I do not want to be dependent on third party APIs.
Sure, I could build everything by myself, but there are many problems such as data integrity between my app and wallet software, HUGE number of problems with non-atomic functionality, etc.

So, my question is, what wallet and account solution mining pools use? What software cryptsy, btc-e and bitstamp use?
benjyz
Full Member
***
Offline Offline

Activity: 140
Merit: 100


View Profile
March 08, 2014, 02:18:21 PM
 #6

Quote
So, my question is, what wallet and account solution mining pools use? What software cryptsy, btc-e and bitstamp use?

custom stuff. it's not easy/cheap to have 100% security. there is no account solution. you have bitcoind which generates N addresses. then you have your own account database and do a 1:1 mapping. move is a simple as doing an update on your own database.

Quote
Sure, I could build everything by myself, but there are many problems such as data integrity between my app and wallet software, HUGE number of problems with non-atomic functionality, etc.

I don't understand - what are the details? no bitcoin transaction is atomic for small latencies. what kind of micropayments?? bitcoind has not implemented micropayments, only bitcoinj. I've not seen anybody using it.
justusranvier
Legendary
*
Offline Offline

Activity: 1400
Merit: 1006



View Profile WWW
March 08, 2014, 02:20:36 PM
 #7

https://blog.conformal.com/btcd-not-your-moms-bitcoin-daemon/
benjyz
Full Member
***
Offline Offline

Activity: 140
Merit: 100


View Profile
March 08, 2014, 02:25:41 PM
 #8

yes, I've tested btcd and has websockets, it's very well documented, etc. It uses go-channels which make the threading model much better.

But if you implement things on top of it, you have to put in a lot of work to ensure this works as you expect. So if you want to handle a lot of money it's unlikely to be a good option.

For micropayments see : https://code.google.com/p/bitcoinj/wiki/WorkingWithMicropayments

Note:
if you handle a lot of money you have to know what you're doing. you might be better off using merchant API's.
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1005


Gerald Davis


View Profile
March 08, 2014, 02:26:10 PM
 #9

Don't use account.  Don't even plan on using accounts.  It is an uttterly worthless "feature" which has no place in bitcoind to begin with.

Bitcoind works flawlessly for its core functionality and that is running as a full node & wallet. "Account" level features should be in your application business logic and put in your application data store.

DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1005


Gerald Davis


View Profile
March 08, 2014, 02:29:15 PM
 #10

So, my question is, what wallet and account solution mining pools use? What software cryptsy, btc-e and bitstamp use?

bitcoind for network connectivity and transaction processing.  With all high level functions in custom application and database.

If you had an application which needed to connect to the US banking network would you build it all inside your banks online banking interface?  Of course not.  You would build a high level application that issued commands and pulled data from your banks interface.
paullinator
Newbie
*
Offline Offline

Activity: 37
Merit: 0



View Profile
March 08, 2014, 03:48:08 PM
 #11

Look into libbitcoin and obelisk. Full rewrites of bitcoin with performance in mind.
miso
Newbie
*
Offline Offline

Activity: 37
Merit: 0


View Profile
March 08, 2014, 04:36:18 PM
 #12

Quote
So, my question is, what wallet and account solution mining pools use? What software cryptsy, btc-e and bitstamp use?

custom stuff. it's not easy/cheap to have 100% security. there is no account solution. you have bitcoind which generates N addresses. then you have your own account database and do a 1:1 mapping. move is a simple as doing an update on your own database.

Quote
Sure, I could build everything by myself, but there are many problems such as data integrity between my app and wallet software, HUGE number of problems with non-atomic functionality, etc.

I don't understand - what are the details? no bitcoin transaction is atomic for small latencies. what kind of micropayments?? bitcoind has not implemented micropayments, only bitcoinj. I've not seen anybody using it.
Data integrity: Lets say I do not use bitcoind accounts. I have N addresses in Wallet and my own SQL database with addr-user (n:1) mapping. I really need user to have many addresses. So, when I display addr A to some user, I have to check for transactions to the addr A in some point in time and update my SQL - cron could be my friend in this case.
But, I have to scan all the addresses in my SQL with current balance to find out user balance! I have to use "deep" scan - I can not assume each address has static balance - the wallet sends btc from addresses it decides are optimal. So, I have to use for example blockchain.info API where I can pass many addresses with minconf and it returns balance from corresponding TXs which have minconf confirmations.
Edit: yes, it is abstraction "address has balance" - addresses have transactions, transactions have amounts, so "addresses have balances"
I am sure bitcoind's rpc functions do  not support many addresses as input and not sure if it could retrieve balance at minconf from one addr either.
It is pointless to cache user balances in SQL, because it may be changed at every user withdrawal. No way to say "send ONLY from THIS address".
And many more problems...
Atomic or synchronization problems: User wants withdrawal:
0. I need to set set semaphore/lock or make this withdrawal function serialized.
1. I deep-rescan his balance B1.
2. From this moment, the actual balance A1 >= B1 (he could send funds to his addr in the meantime).
3. The second withdrawal (attack attempt) comes at this time. Because of point zero 0, it waits blocked.
4a. If point 0. does not exists, each of two or more server instances/threads retrieved balance B1 (in fact, B1 and B2, B1==B2 or B1!=B2) and I command my wallet "send B1 amount to some user address" - two times!
4b. If point 0. exists, I commant my wallet "send B1 amount to some user address" - only once.
5. Withdrawal function ends and unlocks second attempt to withdrawal, which continues from point 1.

bitcoind provides sendfrom <account> <amount> [minconfirm=1] which is atomic function and fails if there is not enough balance "in the minconfirm state". test-and-do atomic operation.

The bitcoind account functionality in terms of problems like this is almost perfect. Yes, there are problems like it test-and-do balance with no fees includes and it may results in negative account balance, but this is intended and there are reasons for this behavior.
It is difficult (at least for me) to code two separate systems which must be synchronized.
justusranvier
Legendary
*
Offline Offline

Activity: 1400
Merit: 1006



View Profile WWW
March 08, 2014, 04:40:24 PM
 #13

Data integrity: Lets say I do not use bitcoind accounts. I have N addresses in Wallet and my own SQL database with addr-user (n:1) mapping. I really need user to have many addresses. So, when I display addr A to some user, I have to check for transactions to the addr A in some point in time and update my SQL - cron could be my friend in this case.
But, I have to scan all the addresses in my SQL with current balance to find out user balance! I have to use "deep" scan - I can not assume each address has static balance - the wallet sends btc from addresses it decides are optimal. So, I have to use for example blockchain.info API where I can pass many addresses with minconf and it returns balance from corresponding TXs which have minconf confirmations. I am sure bitcoind's rpc functions do  not support many addresses as input and not sure if it could retrieve balance at minconf from one addr either.
It is pointless to cache user balances in SQL, because it may be changed at every user withdrawal. No way to say "send ONLY from THIS address".
And many more problems...
Atomic or synchronization problems: User wants withdrawal:
0. I need to set set semaphore/lock or make this withdrawal function serialized.
1. I deep-rescan his balance B1.
2. From this moment, the actual balance A1 >= B1 (he could send funds to his addr in the meantime).
3. The second withdrawal (attack attempt) comes at this time. Because of point zero 0, it waits blocked.
4a. If point 0. does not exists, each of two or more server instances/threads retrieved balance B1 (in fact, B1 and B2, B1==B2 or B1!=B2) and I command my wallet "send B1 amount to some user address" - two times!
4b. If point 0. exists, I commant my wallet "send B1 amount to some user address" - only once.
5. Withdrawal function ends and unlocks second attempt to withdrawal, which continues from point 1.
Before you try to build whatever you're building, you really should learn how Bitcoin works so that you don't base your application's design on incorrect assumptions.
miso
Newbie
*
Offline Offline

Activity: 37
Merit: 0


View Profile
March 08, 2014, 04:48:23 PM
 #14


Thank, I will check that. But I am worry about Go's performance.

yes, I've tested btcd and has websockets, it's very well documented, etc. It uses go-channels which make the threading model much better.

But if you implement things on top of it, you have to put in a lot of work to ensure this works as you expect. So if you want to handle a lot of money it's unlikely to be a good option.

For micropayments see : https://code.google.com/p/bitcoinj/wiki/WorkingWithMicropayments


Note:
if you handle a lot of money you have to know what you're doing. you might be better off using merchant API's.
See bitcoinJ limitations https://code.google.com/p/bitcoinj/wiki/Limitations
Quote
The Wallet code doesn't scale, at all. All transactions that were ever relevant to the wallet are loaded into memory, all the time, and re-written every time the wallet is saved. This results in a simple on-disk format accessible to many kinds of apps, but has poor performance for heavy users. Also, even if spent transactions are removed, the algorithms used for selecting coins and working with unspent transactions have quadratic time complexity or worse. In time we'll probably switch to a log structured wallet file format to solve this.
It is suitable for desktop wallets, but not for server apps.
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1005


Gerald Davis


View Profile
March 08, 2014, 04:52:01 PM
 #15

So, my question is, what wallet and account solution mining pools use? What software cryptsy, btc-e and bitstamp use?
Data integrity: Lets say I do not use bitcoind accounts. I have N addresses in Wallet and my own SQL database with addr-user (n:1) mapping. I really need user to have many addresses. So, when I display addr A to some user, I have to check for transactions to the addr A in some point in time and update my SQL - cron could be my friend in this case.

That would be the worst possible solution.  bitcoind supports block and tx callback notifications.  At an application level you don't need to be concerned with the "balance at each address" (an abstraction which doesn't even exist at the protocol level) instead you only care about the balance per user.  When a tx comes in, determine which user it belongs to.

Quote
But, I have to scan all the addresses in my SQL with current balance to find out user balance! I have to use "deep" scan - I can not assume each address has static balance - the wallet sends btc from addresses it decides are optimal. So, I have to use for example blockchain.info API where I can pass many addresses with minconf and it returns balance from corresponding TXs which have minconf confirmations. I am sure bitcoind's rpc functions do  not support many addresses as input and not sure if it could retrieve balance at minconf from one addr either.

Quote
It is pointless to cache user balances in SQL, because it may be changed at every user withdrawal. No way to say "send ONLY from THIS address".

Once again who cares what the current balance of an individual address is, in a pool fund situation.  Joe has deposited 5 BTC, you owe Joe 5 BTC, Joe's "balance" is 5 BTC.  

Quote
It is pointless to cache user balances in SQL, because it may be changed at every user withdrawal. No way to say "send ONLY from THIS address".
Of course it changes on ever withdraw (and every transfer and ever deposit).

If a user has the following tx stored in the database
Deposit 10 BTC to deposit address x, tx id y
Withdraw 2 BTC to address x, tx id y
Transfer 1 BTC to user z

What is the user's current balance?

If those records were in a table couldn't you get the balance at any moment in time with a SELECT?

Quote
It is difficult (at least for me) to code two separate systems which must be synchronized.

Then the issue isn't coding but architecture.    Forget Bitcoin a second.  Imagine you ran a USD based service.  Customers can deposit funds, customers can withdraw funds, customers can transfer funds to other customers accounts.  Are you telling me there is no possible way you could keep track of user balances without doing it all inside the banking system?   If the answer is no then replace bank/USD with bitcoind/BTC and that is your solution.  If the answer is yes, then please don't develop or launch a service without a lot more research.

Even if bitcoind did everything you wanted it to do it doesn't scale trying to use it as a real time balance data feed.  Use it as it was intended, as a interface to the bitcoin network.  You can then scale out if needed (multiple instances).
miso
Newbie
*
Offline Offline

Activity: 37
Merit: 0


View Profile
March 08, 2014, 04:57:05 PM
 #16

Before you try to build whatever you're building, you really should learn how Bitcoin works so that you don't base your application's design on incorrect assumptions.
Have you read the full context?
The design was about Bitcoin integration with web service in the wrong way.
Why do you think I do not know how Bitcoin works and at what level of understanding I should be to code the app?
justusranvier
Legendary
*
Offline Offline

Activity: 1400
Merit: 1006



View Profile WWW
March 08, 2014, 05:02:38 PM
 #17

Why do you think I do not know how Bitcoin works and at what level of understanding I should be to code the app?
For one thing, you're talking about the balance of an address. That means it's virtually certain that you're tracking the wrong things.

D&T appears to have more time/patience than I so you should ask him about it.
miso
Newbie
*
Offline Offline

Activity: 37
Merit: 0


View Profile
March 08, 2014, 05:19:07 PM
 #18

Why do you think I do not know how Bitcoin works and at what level of understanding I should be to code the app?
For one thing, you're talking about the balance of an address. That means it's virtually certain that you're tracking the wrong things.

D&T appears to have more time/patience than I so you should ask him about it.
The design I provided was example how not to do it. Read the context and quote above it.

I was refering to blockchain.info API which provides "balance of multiple address".
Quote
addressbalance/Address - Get the balance of an address (in satoshi). Multiple addresses separated by | Do not use to process payments without the confirmations parameter

I understand that balance is retrieved from TXs. But TXs can not be assigned to users without addresses in my case - I need abstractions "addresses have balance" or, "account has balance"
miso
Newbie
*
Offline Offline

Activity: 37
Merit: 0


View Profile
March 08, 2014, 05:34:02 PM
 #19

So, my question is, what wallet and account solution mining pools use? What software cryptsy, btc-e and bitstamp use?

bitcoind for network connectivity and transaction processing.  With all high level functions in custom application and database.

If you had an application which needed to connect to the US banking network would you build it all inside your banks online banking interface?  Of course not.  You would build a high level application that issued commands and pulled data from your banks interface.


I agree it has to be done this way in big projects. The question is if bitcoind API is good enough for my project. The answer is "no", but not because the design, but because it sucks in implementation.
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1005


Gerald Davis


View Profile
March 08, 2014, 06:43:24 PM
 #20

Well to clarify bitcoind "core" works as expected.  IMHO the accounts functionality have always been next to worthless (there are other edge cases which can cause unexpected problems).   Most people wouldn't miss it if accounts were moved completely and it would avoid these kind of x-y questions.
Pages: [1] 2 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!