This is very wise opinion. From the accounting/auditing point of view the present wallet model is badly broken. Transactions should be stored in a sequential csv file that is instantly usable by any of the accounting packages for businesses or consumers.

Since the wallet.dat database is essentially append-only this would be relatively simple modification to the existing code. Berkeley DB already supports flat-text database files, but the database model need to be changed from Btree to RecNo.

For the managed hosting environment (multiple bitcoind each with its own wallet, but reasonably good security) I would work as follows:

1) create a separate DBENV for the blkindex.dat
2) convert stdio.h access to blkNNNN.dat into DB access to additional named databases inside blkindex.dat, which already supports multiple databases, take care that block insertion is idempotent
3) convert the new environment from (1) from direct calls into RPC calls
4) run berkeley_db_svc RPC server serving a single copy of blockchain inside blkindex.dat for all the client bitcoin-daemons, each of which is only managing its own wallet in a separate (original) DBENV

This modification will give you best bang for your development buck in the reasonably secure managed hosting environment.

If you know how to recompile you can try disabling memory mapping in the calls to db->open. The flag is DB_NOMMAP. It could slightly improve the disk cache hit ratio and it will greatly improve TLB hit ratio, especially on the virtualized servers.

I'll say don't give up on that goal. There may be an incremental way to redesign it.

1) Create a separate DB_ENV just for the wallet.
2) Change single B-tree DB into three RecNo DB-s backed by flat-text files. This is quite decently supported by the Berkeley DB, especially because the wallet database is essentially an append-only. The 3 files would be public-keys, private-keys and transaction-log.
3) I see lot of people here proposing public-key operation only for shared/managed hosts. But I have yet to see anyone discussing proper transaction auditing as done by the accountants. This is why I was thinking about 3 separate databases, the 3rd being the one given to the auditors.
4) During the transition (and possibly afterwards) the default client will continue to use Oracle BDB as its only database interface.

This incremental approach could assuage Gavin's worries that the patches/pull would try to surreptitiously change the basic protocol and principles of operation for Bitcoin.

I see that we are on the second page of comments on this thread and nobody had noticed one major feature of this acquisition:

bitomat.pl banking relationship is in the SEPA zone.

I guess you'll have to learn your own lessons when you get DDoS-ed for the first time, quite possibly by malfunctioning simple-minded software, not by intentional hackers.
You'll have to do it yourselves. But be aware of the multithreaded patches by Joel Katz:

https://bitcointalk.org/index.php?topic=22585.0

JSON-RPC is a really bad choice for a real-time network protocol. Basically its use cannot be distinguished from a Distributed Denial of Service Attack.

Mt.Gox suffers from it: uses JSON-RPC for market information thus when there's more than about several hundreds of users their DDoS protection service kicks in and users cannot trade in real-time any more.

Here's a link to me trying to explain it to the MtGox users:

https://bitcointalk.org/index.php?topic=35357.msg439514#msg439514

In addition to the above this patch doesn't seem to have any mechanism for garbage collecting stale notification callbacks. From the cursory look I would guess that it doesn't even try to be idempotent, i.e. one could add multiple identical callback URLs with no limit that I could see.

Accounts in bitcoind are fiction. They are essentially labels on transactions. The default label on the transaction is an empty string thus the default account has name "". Don't worry about deletion now. bitcoind currently doesn't handle properly transactions stalled in the "unconfirmed" state and requires manual fixups. Once this bug is fixed you can then worry about cleaning up stale accounts.

Satoshi client a.k.a. bitcoin a.k.a. bitcoind is not designed to do what you are planning. The design will not change as a matter of policy of the "core development group". This is/was meant to be "peer to peer" project and it frowns on "middlemen" as a matter of ideology. Your long term hope is only in libbitcoin project.

Short term you can implement notification callbacks in the block and transaction verification code. You'll need to employ (or be) a competent C++ programmer.

Your worries about saving space are premature. In order to fully participate in the bitcoin economy you'll have to store the whole block chain. Maintaining additional index of interesting transactions will be a pittance in terms of disk space.

To better understand the mindset of this forum may I suggest that you go through the history of my posts (it is 3 pages now) and look into other posts on the threads I replied. I really tried to stick to the technical issues.

Good luck!

This is probably true only for the professionals with unusually low lucrum cessans. For the fishes I swim with the opportunity cost seemed too big.

I understand your point and I disagree with you. Bitcoind is particularly bad for small-time users:

1) it cannot be used on shared or managed hosting plans
2) you have to trust you website manager with check-writing authority on your BTC account
3) the support infrastructure uses unsuitable protocols (JSON instead of FIX)

See my reply in other thread about the plight of small-time traders:

https://bitcointalk.org/index.php?topic=35357.msg439514#msg439514

Big-time users can afford dedicated servers, dedicated connections to the exchanges and dedicated accounting and infrastructure management teams.

I believe you. But adding more servers isn't a solution. The solution is to provide a rapid trading platform that behaves differently than the typical DDoS tool. Most of the online brokers have a Java applet just for that very purpose: it opens and maintains a single connection. I've heard that it is possible to do the same using Ajax and Websockets.

Again: adding more servers helps nobody. They will be as vulnerable to a DDoS as they are right now. The solution is to change the protocols from dumb to smart (JSON to FIX). Good luck with that!

http://en.wikipedia.org/wiki/Financial_Information_eXchange

This downornot.com service is bullshit. It does trip over pretty much any commercial DDoS defense product. I don't know whose DDoS prevention service MtGox uses, but for Prolexic Technologies' it has more than 50% false negative rate. This means a legit HTTP/HTTPS GET/POST from a browser connects through the defense while downornot.com shows the site as down.

I haven't seen the code you had to port & integrate. But from your description it was little-endian single-threaded C code not using exceptions. Here you have many more additional dimensions of complexity.

Satoshi bitcoin client is really a trailblazer in the annals of bad software design. Can you name any other software that uses public-key cryptography and stores both public and private keys in a single object? I presume most people got acquainted with asymmetric cryptography with PGP. Even then, about twenty years ago, there were two separate files: PUBRING.PGP and SECRING.PGP.

Now you have people saying single WALLET.DAT "isn't all that bad" design. I just checked, you think so too. Who erased your memory?

But presumably when you were working on the "bad code" the "bad programmers" were already dismissed and had no commit access to the codebase, right?

Well, here the situation is different. The "core development team" has full access to the code base and is actively moving the target because they disagree with the "new brooms". Do you understand the difference?

Dude, are you nuts? What does "official block-chain", "official checkpoint" mean in this project? "Fully compatible" with what? With the bugs implemented in the "Satoshi client"?

I work with "professional software development compan(ies)" not with Polyannas and other people full of hope that everything will be all right.
 

This isn't secret at all, they are in broad agreement with the goals of libbitcoin:
1) sensible modulatization and abstraction layers
2) make the engine embeddable in other projects, eg. PHP engines
3) byte-endian cleanness
4) sensible test harness
AFAIK the "core development team" only agrees that (4) is a valid goal.

I just wanted to add that I'm not representing any "big fish(es)", I'm just a "pilot fish".

Except that the Java code is unofficial, incomplete implementation and is, well, in Java and not C/C++. Anyone seriously developing an "alternative client" must really develop a "replacement peer" which has a hope of getting accepted by majority of the current peers and miners. Anything else is just a hobby project.

My personal guess is that BitcoinJ is really BitcoinD(alvik) as far as motivation and genesis of it.

I can tell you this much: real professional software development compan(ies) took a look at the current source code and its history and gave estimates & quotes that were unacceptable to some "big fish(es)" in the finance industry.

Obviously, there are "real professional software developers" that will take on anything knowing upfront that they will go over the budget and behind the schedule, but I have no contact with those.

There's a number of serious problems under the hood. You have to dig into the C++ code to see them.