BitCount: enterprise-grade account management for Bitcoin

[arosca]

I am proposing an open source project for robust, scalable Bitcoin account management. Before I get started I would like to know from the community:

• is there something similar out there there that I have overlooked? I don't want to re-invent the wheel
• is there a need/interest for what I am describing below? I have a need for it myself but have other workarounds and will probably not go through the trouble of structuring the project the same way if there isn't enough interest out there
• are there any developers interested in contributing?

The account features built into bitcoind are a sorely needed feature for any system that manages customer Bitcoin accounts and needs to keep track of customer balances. (Read: any website that provides wallet services for customers.) However, as mentioned in countless forum posts and even in the official wiki, accounts are not a robust feature of bitcoind. I believe the fundamental paradigms are sound and the existing API is good and has all or most of the required functionality, but the implementation lacks.

In my opinion there are three major issues that need to be addresses:

• account meta data storage is commingled with wallet (private key) storage
• account management features are integrated in the same process that manages the blockchain
• the wallet (and indirectly, account) backup functionality is insufficient, and is currently limited essentially to file backups

I am proposing a system with the high level architecture outlined in the diagram below:
https://creately.com/diagram/huhfk2yj1/szMUs5NfNPyIl3KipDC2tPzfxg%3D
http://i.snag.gy/2nTcM.jpg

In essence, a separate server component (which is the core goal of this project) is responsible for handling communication with the client (e.g. a website) and maintaining account information. This server can utilize any arbitrary database technology for storing account data and balances. (My plan is to start with a MySQL adapter.) The server utilizes an arbitrary Bitcoin client (initially bitcoind) for performing network operations (sending and receiving Bitcoin) but does not store any account data with bitcoind. Optionally, the server has knowledge of and can also store private keys for any addresses generated as part of the Bitcoin operations. (This constitutes an additional level or redundancy for wallet data.)

The server exposes a JSON-RPC API identical to the one exposed by bitcoind, and is thus a drop-in replacement for bitcoind. However, any account management calls are handled by the server, and only network-specific operations (e.g. receiving or sending funds) are relayed to bitcoind.

The fundamental goal is to provide an option for high availability in a server environment, something that bitcoind is just not designed to do. In addition however certain services, such as sending funds to an external address, can be better controlled, restricted, or authorized. Better wallet security can be enforced as well, as there is no direct access from the public-facing server (e.g. a website) to the machine running bitcoind.

To reiterate: please kindly reply to this post and let me know if this is something that would be useful to the community, and if you have any interest in contributing to such a project. I will also be happy to answer any questions about architecture, implementation technology, etc.

[2112]

It doesn't seem like you've ever done any distributed transaction processing.

http://en.wikipedia.org/wiki/Transaction_processing_system

Please post something that gives us readers an idea that you've ever in your life committed or rolled back a single distributed transaction. Something like Microsoft Excel 97 transacting with Microsoft Access 97 running on Windows 95 OSR2 with Microsoft Distributed Transaction Coordinator (MSDTC) would be fine by me. Typing COMMIT or ROLLBACK into a single SQL server doesn't count.

[arosca]

It doesn't seem like you've ever done any distributed transaction processing.

At my day job I design data warehouse systems for large financial firms.

Do you have any specific concern, question, or constructive comment?

[jonald_fyookball]

Sounds like a stack that could be used to build an online wallet, exchange, etc.
Am I correct, or is there another use case you had in mind?

[Kenneron]

I think normal QT wallet works fine

[arosca]

Sounds like a stack that could be used to build an online wallet, exchange, etc.
Am I correct, or is there another use case you had in mind?

Yes, that's what I'm thinking of. Instead of rolling your own account management, you would have an off-the-shelf component. That component exists inside bitcoind right now but it's not very robust.

[arosca]

I am currently working and speaking to enterprise operations that want to/already use bitcoins, they have account management in place so they basically just extend it to add bitcoins, leaving the bitcoind to really just be a gateway to the network. They don't storage private keys with it or even addresses.

Well, that's what I'm proposing: if you are building a website, you wouldn't have to build the account management part (including keeping track of account balances and a log of transactions that led to those balances) from scratch.

[jonald_fyookball]

build it and they will come

[arosca]

build it and they will come

If that were true I'd be filthy rich by now 

[gweedo]

I am currently working and speaking to enterprise operations that want to/already use bitcoins, they have account management in place so they basically just extend it to add bitcoins, leaving the bitcoind to really just be a gateway to the network. They don't storage private keys with it or even addresses.

Well, that's what I'm proposing: if you are building a website, you wouldn't have to build the account management part (including keeping track of account balances and a log of transactions that led to those balances) from scratch.

If you are building a website, you already have some kind of account manager, and it would be easier implement bitcoin address. Trust me I am working with some large corporations and large corporation software to get bitcoins to them. They all have legacy software and aren't giving it up for bitcoins.

[arosca]

If you are building a website, you already have some kind of account manager

Well, no, not if you're building a new system from scratch That's the use case I have in mind.

[2112]

It doesn't seem like you've ever done any distributed transaction processing.

At my day job I design data warehouse systems for large financial firms.

Do you have any specific concern, question, or constructive comment?

Thanks and I'm sorry for not responding timely.

What you've just told me is that you have no OLTP experience and moreover you don't even have an interest in OLTP technology. "Large financial firms" are the mainstay of the vendors of the http://en.wikipedia.org/wiki/Teleprocessing_monitor and pretty much every of their product internally uses the http://en.wikipedia.org/wiki/Two-phase_commit_protocol .

My following comment will be constructive for the prospective users of any such Bitcoin accounting package that portends to use the bitcoind RPC calls in the "enterprise" environment:

Simple test of robustness and idempotency: suspend the bitcoin daemon right after it received the "send*" RPC call. Do it with "kill -STOP" or store the ".bitcoin" directory on a Fibre Channel and fail the disk. Wait for the upper layer to timeout and then either resume the daemon with "kill -CONT" or plug back the disk. Oftentimes the recipients will receive exactly double of the intended amount.

This isn't some theoretical scenario, it had already happened to GLBSE and ASICminer when they used the naive integration of bitcoind with their SQL server without observing the 2PC. Obviously bitcoind wasn't intentionally suspended by them, it was just made arbitrarily slow by either simultaneously running backups of the same physical hard drives or by excessive swapping of the host machine.

[arosca]

What you've just told me is that you have no OLTP experience and moreover you don't even have an interest in OLTP technology. "Large financial firms" are the mainstay of the vendors of the http://en.wikipedia.org/wiki/Teleprocessing_monitor and pretty much every of their product internally uses the http://en.wikipedia.org/wiki/Two-phase_commit_protocol .

Look, if the point you're trying to make is "you don't have the skills to build this project"*, why not just get involved? If you are too busy to code, perhaps you can get involved in an advisory role?

Every post on the topic of account management seems to eventually lead to "the stuff built into bitcoind is not good enough and may be removed in the future; build your own." The moment you're separating account functionality out of bitcoind and into a separate process you're already dealing with distributed transactions. Would it not be better to have a publicly scrutinized and tested subsystem for this?


*even though you have absolutely no idea who I am and what my experience level is, and you've apparently concluded this based on a few paragraphs I wrote and what I do for a living

[2112]

What you've just told me is that you have no OLTP experience and moreover you don't even have an interest in OLTP technology. "Large financial firms" are the mainstay of the vendors of the http://en.wikipedia.org/wiki/Teleprocessing_monitor and pretty much every of their product internally uses the http://en.wikipedia.org/wiki/Two-phase_commit_protocol .

Look, if the point you're trying to make is "you don't have the skills to build this project"*, why not just get involved? If you are too busy to code, perhaps you can get involved in an advisory role?

Every post on the topic of account management seems to eventually lead to "the stuff built into bitcoind is not good enough and may be removed in the future; build your own." The moment you're separating account functionality out of bitcoind and into a separate process you're already dealing with distributed transactions. Would it not be better to have a publicly scrutinized and tested subsystem for this?


*even though you have absolutely no idea who I am and what my experience level is, and you've apparently concluded this based on a few paragraphs I wrote and what I do for a living

Many people tried to influence the Bitcoin core development team towards the use proper ACID techniques. You can review my past posts as well as several other members of bitcointalk.org. It isn't going to happen for many reasons:

1) it isn't a priority for them
2) nobody there has any appreciable experience with DBMS and OLTP, and they don't have time to acquire it
3) majority of Bitcoin users demand lightweight solutions even if they are seat-of-the-pants and aren't ACID-compliant
4) in the history of patches submitted to the project leader there were several that purported to provide some enterprise-level functionality, but also changed the core Bitcoin protocol behavior with respect to the chain reorganizations. This has understandably soured him to accepting any more "enterprise" contributions
5) Bitcoin milieu has already experienced several fraudsters (or "criminally optimistic people") that made promises very similar to yours

You may think of yourself as a trailblazer and Bitcoin do-gooder, but thus far you've pretty closely followed the footsteps of some previous scams. The onus is actually on you. I don't care about your credentials or experience or who you work for. Just post something that shows your technical competence in the OLTP domain and not in the domain of writing sales blurbage and the use of Microsoft Visio. What you've posted thus far would be graded as a fail by any professor at any OLTP-related course at any reputable school.

[jonald_fyookball]

What you've just told me is that you have no OLTP experience and moreover you don't even have an interest in OLTP technology. "Large financial firms" are the mainstay of the vendors of the http://en.wikipedia.org/wiki/Teleprocessing_monitor and pretty much every of their product internally uses the http://en.wikipedia.org/wiki/Two-phase_commit_protocol .

Look, if the point you're trying to make is "you don't have the skills to build this project"*, why not just get involved? If you are too busy to code, perhaps you can get involved in an advisory role?

Every post on the topic of account management seems to eventually lead to "the stuff built into bitcoind is not good enough and may be removed in the future; build your own." The moment you're separating account functionality out of bitcoind and into a separate process you're already dealing with distributed transactions. Would it not be better to have a publicly scrutinized and tested subsystem for this?


*even though you have absolutely no idea who I am and what my experience level is, and you've apparently concluded this based on a few paragraphs I wrote and what I do for a living

Many people tried to influence the Bitcoin core development team towards the use proper ACID techniques. You can review my past posts as well as several other members of bitcointalk.org. It isn't going to happen for many reasons:

1) it isn't a priority for them
2) nobody there has any appreciable experience with DBMS and OLTP, and they don't have time to acquire it
3) majority of Bitcoin users demand lightweight solutions even if they are seat-of-the-pants and aren't ACID-compliant
4) in the history of patches submitted to the project leader there were several that purported to provide some enterprise-level functionality, but also changed the core Bitcoin protocol behavior with respect to the chain reorganizations. This has understandably soured him to accepting any more "enterprise" contributions
5) Bitcoin milieu has already experienced several fraudsters (or "criminally optimistic people") that made promises very similar to yours

You may think of yourself as a trailblazer and Bitcoin do-gooder, but thus far you've pretty closely followed the footsteps of some previous scams. The onus is actually on you. I don't care about your credentials or experience or who you work for. Just post something that shows your technical competence in the OLTP domain and not in the domain of writing sales blurbage and the use of Microsoft Visio. What you've posted thus far would be graded as a fail by any professor at any OLTP-related course at any reputable school.

I'm sure you know a lot about OLTP and that's great,
it could be an important topic in the future and its
nice to know the Bitcoin community has someone
with expertise in this specific niche.
 
I don't think the OP is about using OLTP but rather
simply building a scalable computing infrastructure.

(Bitcoin can't even handle more than 7 tps
and right now we're doing one)

[2112]

seat-of-the-pants

I just realized that this was a bad choice of metaphor on my part. Here is a better one:

And when talking about "enterprise-grade" accounting most here have something like this on their mind:

or, apparently, something like that:

Credit goes to Mike Caldwell for clueing me on the "receipts-on-a-spike" metaphor some years back.

[arosca]

What you've posted thus far would be graded as a fail by any professor at any OLTP-related course at any reputable school.

I didn't realize I had posted enough detail for you to draw such a conclusion, but if that is indeed the case, by all means, please educate me. I don't mean to be sarcastic. I would truly appreciate your input. How would you make it better?

You may think of yourself as a trailblazer and Bitcoin do-gooder, but thus far you've pretty closely followed the footsteps of some previous scams. The onus is actually on you.

Agreed. The goal of my post is to gauge if there is any community interest in such a project. Time will tell if I actually deliver something useful or beneficial.

[2112]

I didn't realize I had posted enough detail for you to draw such a conclusion, but if that is indeed the case, by all means, please educate me. I don't mean to be sarcastic. I would truly appreciate your input. How would you make it better?

Oh, sure. I already posted my suggestions. But let me restate them:

1) Figure out a way to plug into industry-standard transaction monitor systems. Any or all, your choice

2) Figure out a way for bitcoind to operate in some sort of two-phase commit protocol. Start by understanding why the old fashioned credit card charge actually consists of "authorization" and "capture".

3) bitcoind uses BerkeleyDB for wallet storage. BDB in turn supports X/OPEN transaction monitor interface. Research if it is possible to actually enable that functionality without doing major re-factoring of the existing code.

And actually at a top priority:

0) Implement just about any school-grade, educational example of a working distributed transaction system, e.g. something like I've suggested in my first message in this thread: two programs from Microsoft Office on two Windows machines using MSDTC. If you don't like Windows then work with Linux/Tuxedo/free VM images provided by Oracle (I wrote about this a while back already).

[arosca]

Thank you very much! That's good feedback.

It may actually be the case that coordinating transactions internally it simpler than it might seem at first. I do not intend to interfere substantially with bitcoind. The goal of this project is strictly to provide account management services. All network operations (send/receive funds) would be relayed to bitcoind.

 In my opinion the majority of operations are limited to two components at a time (server+database; server+bitcoind; bitcoind+database). The only operations I can think of that involve all 3 components are sendfrom and sendmany. But, the devil is in the details, and your points are well taken.

Thank you again!