No, don't "bring it on".

The project needs to grow gradually so the software can be strengthened along the way.

I make this appeal to WikiLeaks not to try to use Bitcoin.  Bitcoin is a small beta community in its infancy.  You would not stand to get more than pocket change, and the heat you would bring would likely destroy us at this stage.

That's a good optimisation.  I'll add that next time I update SVN.

More generally, we could also consider this:

        dbenv.set_lk_max_objects(10000);
        dbenv.set_errfile(fopen(strErrorFile.c_str(), "a")); /// debug
        dbenv.set_flags(DB_AUTO_COMMIT, 1);
+       dbenv.set_flags(DB_TXN_NOSYNC, 1);
        ret = dbenv.open(strDataDir.c_str(),
                         DB_CREATE     |
                         DB_INIT_LOCK  |
                         DB_INIT_LOG   |

We would then rely on dbenv.txn_checkpoint(0, 0, 0) in CDB::Close() to flush after wallet writes.

What was this wallet used with?  An early accounts patch or git build?

It's while loading the wallet.  I assume it must be in this:

    else if (strType == "acentry")
    {
        string strAccount;
        ssKey >> strAccount;
        uint64 nNumber;
        ssKey >> nNumber;
        if (nNumber > nAccountingEntryNumber)
            nAccountingEntryNumber = nNumber;
    }

You could check that with this:

    else if (strType == "acentry")
    {
        string strAccount;
        assert(!ssKey.empty());
        ssKey >> strAccount;
        uint64 nNumber;
        if (ssKey.size() != 8 )
            printf("***** %s %d\n", strAccount.c_str(), ssKey.size());
        assert(ssKey.empty() == false);
        ssKey >> nNumber;
        if (nNumber > nAccountingEntryNumber)
            nAccountingEntryNumber = nNumber;
    }


Was there an interim version of accounts on git at some point that had just ("acentry", "account") for the key?

If you have gdb, you could run it in gdb and do a backtrace.

gdb --args bitcoin ...
run
(wait for exception)
bt

It seems like you're inclined to assume everything is wrong more than is actually so.

Writing the block index is light work.  Building the tx index is much more random access per block.  I suspect reading all the prev txins is what's slow.  Read caching would help that.  It's best if the DB does that.  Maybe it has a setting for how much cache memory to use.

Already does that.  See CDB.  The lifetime of the (for instance) CTxDB object is only to support database transactions and to know if anything is still using the database at shutdown.

If it was doing that it would be much slower.  It's supposed to be only once a minute or 500 blocks:

    if (strFile == "blkindex.dat" && IsInitialBlockDownload() && nBestHeight % 500 != 0)
        nMinutes = 1;
    dbenv.txn_checkpoint(0, nMinutes, 0);

Probably should add this:
    if (!fReadOnly)
        dbenv.txn_checkpoint(0, nMinutes, 0);

Does transaction commit imply flush?  That seems surprising to me.  I assume a database op wrapped in a transaction would be logged like any other database op.  Many database applications need to wrap almost every pair of ops in a transaction, such as moving money from one account to another. (debit a, credit b)  I can't imagine they're required to batch all their stuff up themselves.

In the following cases, would case 1 flush once and case 2 flush twice?

case 1:
write
write
write
write
checkpoint

case 2:
begin transaction
write
write
commit transaction
begin transaction
write
write
commit transaction
checkpoint

Contorting our database usage will not be the right approach.  It's going to be BDB settings and caching.

Very much not.  Using multiple copies of wallet.dat is not recommended or supported, in fact all of Bitcoin is designed to defeat that.  Both copies will get screwed up.

If you're trying to consolidate your generated coins into one wallet, a better solution now is to run getwork miners on the additional systems.  jgarzik has a CPU miner, and it supports tcatm's 4-way SSE2, so on Windows it's up to twice as fast as the built-in SHA if you have an AMD or recent Intel (core 3, 5 or 7).

New demonstration CPU miner available:
http://bitcointalk.org/index.php?topic=1925.0

Despite everything else said, the current next step is:
In particular, I suspect that more read caching might help a lot.

Then something more specific was wrong.  That's not due to normal initial download time.  Without more details, it can't be diagnosed.  If it was due to slow download, did it speed up after 10-20 minutes when the next block broadcast should have made it switch to a faster source?  debug.log might have clues.  How fast is their Internet connection?  Was it steadily slow, or just slow down at one point?

The 74000 checkpoint is not enough to protect you, and does nothing if the download is already past 74000.  -checkblocks does more, but is still easily defeated.  You still must trust the supplier of the zipfile.

If there was a "verify it" step, that would take as long as the current normal initial download, in which it is the indexing, not the data download, that is the bottleneck.

80 bytes per header and no indexing work.  Might take 1 minute.

The data is mostly hashes and keys and signatures that are uncompressible.

The speed of initial download is not a reflection of the bulk data transfer rate of the protocol.  The gating factor is the indexing while it downloads.

ribuck's description is spot on.

Pool operators can modify their getwork to take one additional parameter, the address to send your share to.

The easy way for the pool operator would be to wait until the next block is found and divy it up proportionally as:
user's near-hits/total near-hits from everyone

That would be easier and safer to start up.  It also has the advantage that multiple hits from the same user can be combined into one transaction.  A lot of your hits will usually be from the same people.

The instant gratification way would be to pay a fixed amount for each near-hit immediately, and the operator takes the risk from randomness of having more or less near-hits before a block is found.

Either way, the user who submits the hit that solves the block should get an extra amount off the top, like 10 BTC.

New users wouldn't really even need the Bitcoin software.  They could download a miner, create an account on mtgox or mybitcoin, enter their deposit address into the miner and point it at anyone's pool server.  When the miner says it found something, a while later a few coins show up in their account.

Miner writers better make sure they never false-positive near-hits.  Users will depend on that to check if the pool operator is cheating them.  If the miner wrongly says it found something, users will look in their account, not find anything, and get mad at the pool operator.

You should try it with tcatm's 4-way SSE2 SHA in sha256.cpp.  It compiles fine as a C file, just rename sha256.cpp to sha256.c.  I was able to get it to work in simple tests on Windows, but not when linked in with Bitcoin.  It may have a better chance of working as part of a C program instead of C++.

Currently it's only enabled in the Linux build, so if you get it to work you could make it available to Windows users.  It's about 100% speedup on AMD CPUs.

That's what it does, it returns true/false.

Laszlo does them, but I haven't asked him to do one for a while because there wasn't anything major.  I'll ask him to do this version.

I tested it on a slow 7 year old drive, where bandwidth and CPU were clearly not the bottleneck.  Initial download took 1 hour 20 minutes.

If it's taking a lot longer than that, certainly 24 hours, then it must be downloading from a very slow node, or your connection is much slower than around 15KB per sec (120kbps), or something else is wrong.  It would be nice to know what appears to be the bottleneck when that happens.

Every 10 minutes or so when the latest block is sent, it should have the chance to change to a faster node.  When the latest block is broadcast, it requests the next 500 blocks from other nodes, and continues the download from the one that sends it fastest.  At least, that's how it should work.

It may only need more read caching.  It has to read randomly all over blk0001.dat and blkindex.dat to index.  It can't assume the file is smaller than memory, although it currently still is.  Caching would be effective, since most dependencies are recent.

Someone should experiment with different Berkeley DB settings and see if there's something that makes the download substantially faster.  If something substantial is discovered, then we can work out the particulars.

We checkpoint every 500 blocks.

Version 0.3.17 is now available.

Changes:
- new getwork, thanks m0mchil
- added transaction fee setting in UI options menu
- free transaction limits
- sendtoaddress returns transaction id instead of "sent"
- getaccountaddress <account>

The UI transaction fee setting was easy since it was still there from 0.1.5 and all I had to do was re-enable it.

The accounts-based commands: move, sendfrom and getbalance <account> will be in the next release.  We still have some more changes to make first.

Downloads:
http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.3.17/

It's not the downloading that takes the time, it's verifying and indexing it.

Bandwidthwise, it's more efficient than if you downloaded an archive.  Bitcoin only downloads the data in blk0001.dat, which is currently 55MB, and builds blkindex.dat itself, which is 47MB.  Building blkindex.dat is what causes all the disk activity.

During the block download, it only flushes the database to disk every 500 blocks.  You may see the block count pause at ??499 and ??999.  That's when it's flushing.

Doing your own verifying and indexing is the only way to be sure your index data is secure.  If you copy blk0001.dat and blkindex.dat from an untrusted source, there's no way to know if you can trust all the contents in them.

Maybe Berkeley DB has some tweaks we can make to enable or increase cache memory.

A revised version of getwork is now in the official client, but the miners need to be updated a little to use it.

getwork does the byte-reversing.  midstate, data and hash1 are already big-endian, and you pass data back still big-endian, so you work in big-endian and don't have to do any byte-reversing.  They're the same data that is passed to the ScanHash_ functions.  You can take midstate, data and hash1, put them in 16-byte aligned buffers and pass them to a ScanHash_ function, like ScanHash(pmidstate, pdata + 64, phash1, nHashesDone).  If a nonce is found, patch it into data and call getwork.

I should probably change the ScanHash_ functions to use pdata instead of pdata + 64 so they're consistent.

target is little endian, it's supposed to be the same as how m0mchil's did it.  (if it's not, then it should be fixed)  That's the only case where you would use byte reverse.  I think you do it like: if ByteReverse((unsigned int*)hash[6]) < (unsigned int*)target[6].

This is the new spec.  It shouldn't be hard to update your miner to use it.

The changes are:
- It does not return work when you submit a possible hit, only when called without parameter.
- The block field has been split into data and hash1.
- state renamed to midstate for consistency.
- extranonce not needed.

It's not an exact drop-in replacement.  I wanted to clean up the interface a little.  It only requires a few changes.

ScanHash_ functions aren't going away.  BTW, the interface of this is designed to mirror the parameters of that (midstate, data, hash1).

I uploaded a redesign of m0mchil's getwork to SVN rev 189 (version 31601)

m0mchil's external bitcoin miner idea has solved a lot of problems.  GPU programming is immature and hard to compile, and I didn't want to add additional dependencies to the build.  getwork allows these problems to be solved separately, with different programs for different hardware and OSes.  It's also convenient that server farms can run a single Bitcoin node and the rest only run getwork clients.

The interface has a few changes:

getwork [data]
If [data] is not specified, returns formatted hash data to work on:
  "midstate" : precomputed hash state after hashing the first half of the data
  "data" : block data
  "hash1" : formatted hash buffer for second hash
  "target" : little endian hash target
If [data] is specified, tries to solve the block and returns true if it was successful.  [data] is the same 128 byte block data that was returned in the "data" field, but with the nonce changed.

Notes:
- It does not return work when you submit a possible hit, only when called without parameter.
- The block field has been separated into data and hash1.
- data is 128 bytes, which includes the first half that's already hashed by midstate.
- hash1 is always the same, but included for convenience.
- Logging of "ThreadRPCServer method=getwork" is disabled, it would be too much junk in the log.

Thanks m0mchil for keeping up on the updates!

GPU miners, please upgrade as soon as possible to shut down the free transaction abuse!  This version has the new priority-based limit on free transaction spam.

Before SVN 184, compiling transactions into a block used an n^2 algorithm.  The new efficient single-pass algorithm is orders of magnitude quicker.  (O(n) vs O(n^2)/2 algorithm, n=200 maybe 10 to 100 times quicker)

I'm doing something like that.  Priority is a more formalised version of the concept you're describing.

Yes, like this.  And the no-priority-requirement area is 3K, about a dozen transactions per block.

I just uploaded SVN rev 185 which has a minimal priority requirement for free transactions.  Transaction floods are made up of coins that are re-spent over and over, so they depend on their own 0 conf transactions repeatedly.  0 conf transactions have 0 priority, so free transactions like that will have to wait for one transaction to get into a block at a time.

Version 0.3.15 doesn't write transactions using 0 conf dependencies unless that's all it has left, so normal users shouldn't usually have a problem with this.

I think this is a good compromise short of making the default fee 0.01.  It's not so much to ask that free transactions can only be used to turn coins over so often.  If you're using free transactions, you're taking charity and there has to be some limit on how often you can use it with the same coins.

We've always said free transactions may be processed more slowly.  You can help ensure your transactions go through quickly by adding -paytxfee=0.01.

We can't safely do OP_BLOCKNUMBER.  In the event of a block chain reorg after a segmentation, transactions need to be able to get into the chain in a later block.  The OP_BLOCKNUMBER transaction and all its dependants would become invalid.  This wouldn't be fair to later owners of the coins who weren't involved in the time limited transaction.

nTimeLock does the reverse.  It's an open transaction that can be replaced with new versions until the deadline.  It can't be recorded until it locks.  The highest version when the deadline hits gets recorded.  It could be used, for example, to write an escrow transaction that will automatically permanently lock and go through unless it is revoked before the deadline.  The feature isn't enabled or used yet, but the support is there so it could be implemented later.