Bitcoin Forum

Hey all, I created a question here: http://bitcoin.stackexchange.com/questions/10878/is-bitcoind-reliable-when-depending-on-it-for-web-services (http://bitcoin.stackexchange.com/questions/10878/is-bitcoind-reliable-when-depending-on-it-for-web-services)

was wandering as there are a lot of technical minded people with bitcoin knowledge and knowledge of bitcoind and it's application in web services could answer it for me as I want to start work on a small project but my worry is scalability later on in running a service.

Here's the question:

"What I mean by the title is, say If I create a service with bitcoind being used to create and store large numbers of accounts and address, is it likely to take the strain? Can I setup multiple instances of bitcoind to access information about the network or would I need to one large instance of a bitcoin daemon to handle everything?

If not is there an alternative to be used that can keep up with an expanding web service?"

Please feel free to discuss here but if you have an experienced answer to give I'd hope you'd answer the question for me on stack exchange and so it's filed away easily for others to find.

You don't mention what "service" you intend on providing, so it's hard to say.  Generally it's not advisable to use bitcoind directly as your sole accounting system because there is no synchronous replication of metadata and internal transactions (e.g. "move"s).

Ideally it's just to read data about transactions and get address balances but it needs each account to potentially be read once per hour.

An alternative that can really take the heat of a web service is the BOP Enterprise Bitcoin Server.

You can run several instances of it and send requests to them through a load balancing message bus.
See the example repository https://github.com/bitsofproof/bop-explorer that shows a simple REST API forwarding to the bus.
The server code is in https://github.com/bitsofproof/supernode

Bits of Proof will launch its offer in two days in San Jose, dramatically reducing the pain of integrating with the Bitcoin protocol.

Interesting so, I can run the server myself or is this a service being sold so I'd have to pay per request for example? Secondly, is there much in the way of documentation for the REST API? I'm assuming you don't have any other examples or libraries that aren't Java?

You can run the server yourself, and we have also hosted offers, as of:

Bits of Proof launches world's first enterprise-ready Bitcoin server (http://www.prlog.org/12137996-bits-of-proof-launches-worldss-first-enterprise-ready-bitcoin-server.html)

we even have a complementary offer during Bitcoin 2013.

Pricing is not by request, but we offer maintenance and support contracts of several levels and hosted servers, the later will be priced by vCPU allocated.

The hosted solution saves you the effort to set up a server for development and I hope you will find it care free for your production.
We will have shared and dedicated servers and also load balanced server farms depending on your need.

The primary API is the Bus, that speaks STOMP and is transporting Protobuf format messages, so you can connect e.g. with python, ruby or PHP, I am sure you find examples using both STOMP and Protobuf quickly with google for your choice of language.

The rest API  in the bop-explorer is just a demo how easy is to create one. The BCCAPI that powers the Android app is also technically a REST adapter over the Bus.

The clients side Java code using the Bus API adds a layer of abstraction, that can surely be achieved with other languages. I hope that a community around the product will share those that are popular. You are not exposed to java directly if you do not want it. In the hosted case you do not even have to care of compiling and running the server using java.

The weakest point in our offer is the documentation, but this is just temporary as all resources went into development to hit the target for Bitcoin 2013, and we did. Next focus is documentation. I believe the example code you find in our GitHub (https://github.com/bitsofproof) repos give a head start.

Sounds great, genuinely excited, have you got particular locations for running hosted solutions, like will you be running some one cloud providers like Amazon or will you be purely in house?

I'd love to put some time in to make a PHP client library if I can get the time for it, obviously might be quite difficult until you guys have full documentation down.

Bits of Proof already operates several servers at world class data centers in Germany.
Incremental server locations will be aligned with our customer's need and preference.

In my experience bitcoind has been stable.

It's also fairly slow, and has been unable to handle minimal (~3k users) "getBalance/listtransactions/move" load on btct.co even with a caching layer on top of it for frequent queries.

I tackled the missing replication with frequent backups + external transaction logging.  (such that I should be able to restore the wallet, then replay the transactions.)  Rolling out a drbd implementation is next on my list.

Unfortunately I've also stumbled into a fairly major bug for a webservice.  It's really strange but listtransactions does not return transactions in chronological order for me anymore.

https://bitcointalk.org/index.php?topic=205647.0

Not sure what's going on there.

Hope that helps somewhat.  ;)

That is an excellent idea, VPN powered Virtual Server, multiple connections from strategic locations, excellent. Multi Coin management, This will definitly make it possible to go beyond the 500kb/block limit.

so how far have you been able to implement it?

Is there by any chance someone where that documents all the possible messages that can be set and expected results etc? Even as a really rough document?

It is implemented.
Product release is tomorrow.

Just a few days of patience please. We are in conference mode. Probably I get it written up while in the airport lounge :)

I try quick and dirty here:

All messages are STOMP ByteArray where the payload is Protobuf as defined in https://github.com/bitsofproof/supernode/blob/master/api/src/main/protobuf/BCSAPIMessage.proto

Messages go to queues and topics. Queues are read by a single server instance (this does the load balancing) while topics are broadcast to all server and clients listening.

The following broadcast is allowed from clients to servers
/topic/newTransaction - send a new Transaction
/topic/newBlock - send a newly mined block

Synchronous requests from client to server:
Client sends a message to a request queue and sets the reply-to queue of the message to a temporary queue /queue/temp.XXXXX
The server that picks up the request will send reply to that temp queue.

The following synchronous requests are allowed from clients to a server:
/queue/blockRequest - send a Hash message to get a Block message on the reply queue
/queue/headerRequest - send a Hash message to get a Block message on the reply queue, but transactions field is empty
/queue/transactionRequest - send a Hash message to get a Transaction message on the reply queue
/queue/filterRequest - send a FilterRequest message to get a ongoing feed of Transaction messages on the reply queue that match the Bloom filter, this will keep going until you disconnect
/queue/matchRequest - send an ExactMatchRequest to scan the blockchain for transactions matching the filter, where filter is a list of byte vectors. The byte vectors can be addresses, keys or any data in a transaction script. This scans once starting from an optional time point "after" in the message. Resulting transactions will appear on the reply queue.
/queue/scanRequest - works like match request with bloom filter
/queue/ping send a ping msg to server you get back the same. This is to measure latency or to check availability.

Thanks so ideally I'm guessing to monitor any of the addresses I've got to either:

1) do a MatchRequest for each address with my address as the receiving address?

Or

2) do I just create a filterRequest with a filter for my receiving address and I'll essentially get my transactions that way?

if so do you have a rough guess on how long these requests are likely to take?

The matchRequest or scanRequest is to scan the block chain. This would be useful if you import a private key or rescan history for a wallet.
You can submit a long list of patterns (addresses) with a single matchRequest, that is more effective, than one by one. The scanRequest is the same but with abloom filter, this would be useful for to scan for an ever larger set of patterns.

The filterRequest is an ongoing subscription to transactions coming in and matching the filter.

Beware that the bloom filter (for scanRequest or filterRequest) is probabilistic so you can get some transactions you do not care, but it reduces the regular transaction feed by magnitudes. The filter also gets updated automatically, so it will e.g. match for transactions spending a transaction that previously matched the filter. You might want to periodically reset the filter on a long standing connection.

The filter is a bitmap in style of a bloom filter at the moment I can only refer to https://github.com/bitsofproof/supernode/blob/master/api/src/main/java/com/bitsofproof/supernode/api/BloomFilter.java
how it is compiled from byte patterns (keys, addresses, outpoints) you want to filter for.

The Java class https://github.com/bitsofproof/supernode/blob/master/api/src/main/java/com/bitsofproof/supernode/api/DefaultAccountManager.java
shows how filter subscription and scanning is used to implement a client side wallet.

Ok, so I think I've got this down, basically the idea should be rather than directly accessing the 'supernode' from a stateless PHP execution I should probably be creating something like a worker service that will subscribe to batches of addresses and then start storing the results of such like when an address has reached a target amount received?

Cheers for all the help, good luck with the full release  :)

I suggest to have some cache at your side that is initialized with a scan and than subscribes for updating ttansactions.
The stateless calls can consult that cache. I would persist the cache and rescan from the server at startup from the time point it was last persisted. This avoids having inconsistent state with the servrer and gives best performance.

Don't use the "accounts" feat. it doesn't scale.
Manage accounts yourself in whatever data store you're using.

Accounts does not scale if implemented naively.

A combination of local cache and subscription can however scale as i described before. BTW this is the logic of the wallet too, just tightly coupled in the satoshi implementation

Please proceed to getting a clue.
The accounts feature does not scale in bitcoind, once you pass a few 10k accounts it starts slowing everything down.
These problems instantly go away as soon as you start doing the accounting externally.

This is something I've been wondering recently. So if you expect to manage dozens thousands addresses, then it's better not to use bitcoind wallet at all, manage everything on your own (on the level of unspent outputs which your software choses to build rawtransaction) and operate on rawtransactions? Use bitcoind only as block reader and then for pushing tx?

I cant work out a decent way to handle a high number of users with bitcoind - the problem im having is seeing what new transactions have come in.

listtransactions appears to be severely flawed - you have to walk backwards though the list (how do I avoid missing something if it can be re-organised at any time?).

List unspent is wallet wide, and it would only work if the entire transaction is cleared from the wallet after it comes in (i'm assuming change addresses would show up here after a transaction)

having one account for the entire site, with 24 hour throwaway deposit addresses could work (they'd only be polled waiting for incoming transactions for 24 hours) - but its not ideal for the user.

walletnotify might work. but its not supported by litecoind (yet?)

how does everyone else do it?

Busy doing development on a Bitcoin service at the moment. Although we haven't launched yet (soon!), we've tested the system and everything seems to working quite well so far. (Not sure on scalability of course).

You have to use the listsinceblock command. This makes getting new transactions much clearer. The ones that come in, you're sure will be recent transactions you have not seen before. Then you sort the list via the 'timereceived' parameter. Depending on how you handle transactions, but once a new transaction is confirmed, you save the blockhash. Now when you poll for new transactions, you have a place to start from.

You'll probably have to create a replica of the wallet transactions in another DB for persistency (if the server goes down etc).

I hope this helps a bit more, it's hard to give exact advice if I'm not exactly sure what you need. However, I would definitely recommend looking into the listsinceblock command to get your transactions, rather than listtransactions.

Polling = fail