Thanks!  I hope others who like python, will appreciate having a solid python interface to the blockchain.  

Just as an update, I have been working steadily on more functionality.  I just recently got a SelectCoins algorithm implemented, and my first real product is going to be an optimal, offline wallet tool   I have all the pieces, I just gotta put them together and battle PyQt4 a bit more.  I'll also be adding an "examples" directory to demonstrate how to best use the library.  There's already some good example code in testswig.py and the PBE blockexplorer demo, but those are designed for testing, not for clarity of code usage.

There's not really any overlap between what we're encoding and what PGP is encoding, so I'm not sure how necessary it is.  However, I agree that Base64 encoding would result in less characters on-screen, and there may be other things to learn from PGP.  Though, if anything, I would expect Base58 would be better since any BTC software would already have a method for the conversion.  Either way, any of this is open for discussion.

The other difference is that PGP basically encodes everything into a single opaque block of Base64, whereas I'm interested in keeping a little bit of human-readable information outside of the encoding.  To someone who doesn't know/care what's between the begin/end tags, it doesn't make a difference.  But it makes a huge difference for those of us who are inclined to visually inspect these code blocks.  I envision getting an email where someone copied the TxDP into the email and said "please add second signature and broadcast."  I don't have to fiddle with any programs to know that they forgot to sign it.

So, this is temporarily in gist, but I'd really like to get it into something more like what you have here, or at least figure out how to get gist to apply some kind of reasonable formatting to it.  Perhaps someone can help with this.  Once I get it pretty'd up, I'll forward it to the btcdev mailing list.    EDIT: got converted to a "markdown" document in Gist, now it looks pretty good!  I might leave it on there for now, and duplicate it on the wiki when it's back up

By the way, I just posted BIP 0010 to gist:  https://gist.github.com/1321518

Unfortunately, I couldn't figure out why word-wrapping is failing, so you might have to click the "raw" link to see it rendered in plain-text with wrapping.  

Keep in mind that the entire second half of that BIP is actually pseudo-code.  So it may appear long at first glance, but it's only half of what it seems.  On that note, I saved the gist as .py so if you examine the code from the original link, it will be a little easier to read with code highlighting.

This is exactly why I made the following charts:
    https://bitcointalk.org/index.php?topic=29416.0

The address construction process is illustrated at the bottom.  Tx-serialization at the top.  And when you're finally an expert at those, you can take a shot at OP_CHECKSIG (don't forget to reverse endian before signing!).  Yeah, this stuff is complicated, but it's really a one-time investment.  I printed out a couple of these diagrams and keep them nearby when coding...

I'm having a tough time believing that ATI cards are as different as you suggest.  NVIDIA GPUs tend to share 16-48 kB of shared memory and it takes 1-4 clock cycles.  There is nothing faster on the GPU than shared mem, except for the couple registers each thread has.   

However, you do make a good point:  extremely random accesses to memory changes the story quite a bit.  Global memory running 100-200x slower than shared mem usually isn't a problem because you don't do it very often, and sequential accesses can get you a 32x boost.  However, if the computation required each thread to continually/randomly access global memory, it probably would take quite a hit in efficiency.

I'm not much of an ATI/OpenCL programmer, but I've done quite a bit of CUDA programming (see my CUDA fractal viewer), and there's a lot of similarities between the way these chips store and access memory.  

I'm not too familiar with the L1-L3 cache on GPUs -- they tend to operate with some kind of global memory (the number you see on the graphics card box), and then various banks of shared memory and graphics caches.  On most CUDA hardware, there is 16-48kB of shared memory split between 256 threads.  It's fast as hell, but that's not a lot of memory.  Since a lot of programs would need more shared memory than that, they have to use the global memory anyway:  and they still do just fine getting 5-100x speed up over CPUs.    On the other hand, requiring the GPU to store memory in your RAM, would be disastrous.  It takes like 1000x more time to access RAM than it does its own global memory (don't quote me on that number).

Therefore, in order to disarm GPUs of their dominance, I believe you really need to force them to exceed their global memory capacity.  Here's what you'll find in a high-end machine with a good CPU and a solid GPU (take 6970 for exactly)

CPU:  4 cores, 8GB RAM  (2 GB/core)
GPU:  1600 cores, 2 GB RAM (1-2 MB/core)

The issue with hashing is that it requires only a couple kB of RAM, so there's nothing stopping the GPU from hitting full occupancy (maximum parallelization).  However, if the hashing were replaced by an operation that required 100 MB RAM per thread, the GPU would only be able to use 20/1600 cores.  It'd probably be no better than the CPU.  There's ways to do this:  you could replace sha256(sha256(a)) with having to sequentially hash the string 3,000,000 times, and concatenate all the hashes into a single string (requiring 100 MB to store), then execute some operation on that string that requires the thread to store all of it in memory at once (hashing can be done in pieces, but there are other options that would require it).  

Each GPU thread would have to use global memory, and only 20 would fit.  You might as well use your CPU (which whose four cores are probably faster than the GPU's 20).  100 MB is good, because a lot of computers can spare 400 MB of RAM without crippling the computer.

I think it's a mistake to assume that users won't be using Bitcoin from their own computers.  One core attraction of Bitcoin is the decentralization, and by assuming that end-users will flock to centralized "access points," we're creating a self-fulfilling prophecy.  We won't try to make it usable by the end-user directly, and then there will have to be third-parties to dumb it down for users.

My attitude is to not concede to third-parties until we know we have to.  Polish the hell out of the interfaces, make it as intuitive as possible, and give the users a chance to manage their own money.  If a user can succeed without paying a fee to a third-party, why would he?  Obviously, the 2-of-3 escrow transactions have a third-party, by definition, but that's a different topic than whether users will be managing coins on their computer or through a service.  I think light-clients that don't download the blockchain (or take 4-48 hours to load), with encrypted wallets, will make a huge difference in usability for "regular" users.  (most regular people use online banking... why not BTC?)

Additionally, I think one of the most important things, is the ability for management at small companies to handle BTC without having to hire/divert employees to deal with it.  Any benefits of BTC seen by small companies would be overrun by the cost to maintain it.  And without companies using it, BTC won't get very much attention.

Regardless, I totally agree about hiding all this stuff behind an "Advanced" tab/menu.  I think "regular" users should be shielded from confusing themselves with complex Bitcoin operations, until they explicitly decide they want them.  

So along those lines, I'm working on a BIP right now for multi-sig exchanges, which can be handled much like PGP/GPG msgs and sigs (armored ascii, easy to copy/print/email).  I think it gives us an opportunity to standardize this process so all BTC programs can interoperate without trouble.  Maybe I'm being too optimistic...    I'll post a link to the BIP when I get it up, but right now the wiki is inaccessible

EDIT:  Btw, I'm not ignoring the rest of your post, I just wanted to throw it out there that my initial assumptions are different, and I think there's reasons to be optimistic.  I'm going to work out some more details in my head before responding.

Here's an annoying interface detail I realized while trying to implement this:

(1) I create a "distribution proposal" for the funds -- it's an unsigned transaction
(2) I hash this object to give it a unique ID, to reference when talking to the other parties ("Hey, did you ever sign transaction 9bf3218a?")
(3) Everyone signs the tx, and it is broadcast, but the official tx ID (its hash) changes because of the signature
(4) No one actually knows what the ID will be until the packet is signed and broadcast
(5) Interested parties will have to search their own txs looking for one that hashes to 9bf3218a when you remove the sigs

Sure, step 5 can be automated to some extent, but this is annoying.  People will ultimately have to track two different IDs for the same tx.  This is crazy confusing for folks who have a questionable understanding of how Bitcoin works.  It also requires a second step:  perhaps you are trying to keep track of your transactions in a DB.  You have to update the DB once when you get the DP, and then wait until it shows up in the blockchain to update the DB again with the new hash.

How can we intuitively present these ideas to users that wish to use them?
Is it possible for the community to use consistent terminology so that it doesn't change between programs?

It may sound trivial to some, but these are details that really confuse the hell out of end-users that don't understand it like the developers do.  Users need a way to reference these transactions both before and after.  Perhaps "unsigned tx ID" and "broadcast ID."  Also, it would probably be good to use a different encoding for them.  Regular TxIDs (broadcast) are usually displayed in hex.  Perhaps "unsigned Tx ID"s can be presented in Base58 with a couple static leading characters to add some consistency across all of them.

I guess we also would benefit from coming up with a "standard" serialization process for distributing and collecting signatures.  Though, I will probably make a BIP for that...

I know you just threw this idea out there off the top of your head so this isn't a real criticism, I'm just expanding on your initial idea.  I think this would severely neglect your outputs with tiny values--after some point in time, you're going to be left with thousands of tiny inputs... of which you're going to have to use many dozen to execute subsequent transactions.   For example:  if you have an output in your wallet for 0.0001 BTC, you're going to have to wait 1,440,000 blocks for it to accrue the same priority as a 1 BTC output that is 1 day old.

I started investigating this with the idea that maybe I can even it out a bit by simply modifying the BTC amount before multiplying by numConfirm.   After playing with this in a spreadsheet, I see this alone won't solve the entire problem, but perhaps a combination could.  Here's three options:


("days til maturity" is the time until the BTC amount achieves the same prioritization as 1.0 BTC after 1 day)

The techniques are sorted from steepest to shallowest (left to right).  I was just making up equations that I know will guarantee maturity of all outputs before the sun runs out of energy.   Personally, I think the cube-root (technique 1) has a lot of promise, making sure that your tiny TxOuts will eventually get spent.  I'm going to consider trying Gavin's idea, but applying technique 1 before sorting the values, and occasionally throw in the top value from technique 2 or 3, to make sure I'm not letting too many tiny inputs accumulate.

Couldn't this similarly be used for PoS transactions?  For instance, you walk up to the cashier in a 7-11 to buy a candy bar.  You pull out your phone, but you normally will have to wait 20 seconds for it to find peers and connect to them.  Instead, the store's BTC-processor has a small wifi antenna with restricted network "BTC" -- you can instantly connect to it and send the tx to the 7-11 computer which will broadcast for you (assuming you have enough balance in your wallet already).  Also, the 7-11 can be guaranteed to see the tx first, instead of having to wait for it to propagate around the world.

I guess all of this has to be considered in the context of: "is there a problem with accepting 'side-channel' tx-data?"  I could see an attacker figuring out how to flood/DoS your client by throwing millions of tx's through the side-channel (if they can access it), but perhaps that side-channel can be treated just like another peer and cut-off if something suspicious is detected.  I don't see any obvious reasons this would be a risky feature to add.

Alternatively (on the topic of non-instantaneous tx broadcasting)... what percentage of the "networking engine" would you need implemented in order to initiate a "light" connection to bitcoind through localhost?  If tx-broadcasting is all I need, then I bet I could implement only a small subset of the networking protocol and still get bitcoind to accept my tx.  It looks like I'd only have to implement a few messages for network version exchange, inv msg.  I could skip bootstrapping/IRC, flood detection, peer DB building, etc. 

My understanding is that transactions fees are not enforced at the protocol level, they are enforced at the network/node/miner level.  Therefore, if you try to make 1,000,000 tx of 0.000001 BTC each, no node on the network would accept those transactions without fees.  But what is stopping the attacker from using his own miner to include these million transactions in a block?  

Can't he use an altered version of the software to accept Tx's without fees, and work on creating a block with these 1,000,000 tx?  Sure, it might take him a few days/months depending on his hardware, but he would eventually get a block into the chain like that, and all nodes/miners will perceive the 1,000,000 transactions as valid.  Is this correct?
Similarly, can't the attacker make a tx with 1000 outputs, as long as they kept it within the max (500 kB?) and include that in the chain in the same manner?  

If this is true, that would seem to be a "vulnerability" in the network.  You'd only need a couple people devoting their resources to this to really bloat the hell out of the blockchain.

EDIT:  I forgot to mention this would be a networking inconvenience too, as the network tries to circulate all the new blockdata

Haha, thanks for the suggestion, but it's a little too late for me.  I already have like 7,000 lines of C++ code, imported into python through SWIG.  However, I do python development in other projects, too, so I may keep your recommendations in mind.

I had never used SWIG before starting this effort, and now that I have battled it and come out victorious, I'm quite impressed with it.  It understands most of the "nominal" C++ programming structures, and usually does a good job of mapping every C++ function and class (with methods) to identical python calls.  There's a few quirks with it, but i got most of it ironed with one afternoon of frustration. 

But no need to pollute Sam's thread with this any longer.  Just throwing it out my positive recommendation for SWIG.  It's been a lifesaver for BTC client development in python.

Agreed.  I shouldn't have been so hasty to say "accept any blockheaders."  You should download it from a couple different peers and as long as any one of your peers is honest, you'll be able to receive and quickly determine the longest chain.  This is, by definition, the "correct" chain.    Any longer chain that is invalid will soon be outpaced by the correct chain and any one honest peer will set you straight.

Thus, I would argue if someone shows you a tx, you can "quickly" download the entire blockheaders from the network, get the merkle tree with that tx-hash in it, and confirm it matches a blockheader more than 6 blocks deep in the header list.  I would trust that transaction.

This is mostly unnecessary.  It's extraordinarily difficult for someone to give you fake blockheaders, because they would need an extraordinary amount of computing power to give you fake headers that match the difficulty of the block (hash has enough leading zero-bits). i.e. proof-of-work.  You get the headers from a few different peers, and you can verify the leading zeros and accumulate all the difficulty values to get the longest chain.  Unless the attacker has more than 50% of the global computation in his control, he won't be able to feed you a chain of headers longer than the "actual, legit" blockchain headers.

I say mostly unnecessary because technically, if the attacker has a lot of computing power and luck, he might be able to feed you a blockheader list with 1 or 2 fake blockheaders at the top even with less than 50% global computation speed.  But unless he has 51%+, the "actual, legit" blockchain will be extended to be longer within few blocks, and your client will correct itself within an hour. 

tl;dr : It's so difficult to produce fake blockheaders, that, unless you are handling huge amounts of BTC, it is perfectly fine to trust any headers you receive on a lightweight node (as long as its hash has the leading zeros).  I'm sure someone will flame me for this statement...

As a BTC client developer who is not interested in modifying the networking behavior/protocol, I see three reasons I need to implement it anyway:

• (1) Get new block data as it becomes available on the network
• (2) Get transactions that have not yet made it into the blockchain
• (3) Broadcast transactions after they have been signed

If I leave bitcoind running, I got (1) covered by watching blk0001.dat.  I also have (2) covered by using the new JSON-RPC call "getMemoryPool."  However, I do not see anything that covers (3).

Is there a known risk to adding a JSON-RPC call that allows one to hand bitcoind a pre-signed transaction, and have it verify&forward it as if it was received from a peer?  If this functionality was available, I believe I (and many other developers) would have everything we need to shortcut the whole networking reimplementation, and instead implement a few JSON-RPC calls.  This would be good for both new developers and the BTC network. 

2-of-2 escrow works in 99% of cases, except for when one of the parties "disappears."  What if one party dies?  Goes out of business?  Loses their wallet?  Then that money is locked forever.  If you have a third party, they can assist when this happen.   It can kind of act like insurance, too.  

For small amounts of money, I think the 2-of-2 escrow is fine... but you really need a third key in the mix to make sure nothing goes wrong.

People aren't paying you for work that was done in the past, they're paying you for work you will do... they compensate the same number of BTC you would've mined had they not bribed your miner to do the work.  This is a perfect system, because in addition to being "fair", it's also the most convenient medium for paying for the work:  all the owners of the computational units clearly want BTC (that's why they're mining in the first place), and transferring BTC requires no third-parties or additional fees in order to request the work.  The party can request compute time instantly from anywhere, 24 hours a day with no downtime, and not have to deal with anyone else except the miner itself.  It turns BTC into a computational resource that happens to mine these "tokens" with its spare compute power.

Then, the value of BTC would have to gravitate towards the "accepted" value of computation time.

You can download the headers in less than 5 minutes.  This gives you a definitive map of the data you should receive to fill in the rest.  Therefore, if the client only uses the network to pick out 15 MB of blockheaders with the longest chain/work, then he can get the other 600 MB from anywhere and be confident he's getting the right data.  This doesn't really require any more trust than downloading the data normally.

And compression wouldn't achieve too much:  most of the blockchain is hashes which are, by design, supposed to "random" sequences of bits.  Random data is not very compressible (in fact, you can use compression algorithms to test your encryption/hashing algorithms:  if the output will compress more than 5%-10%, then it's not sufficiently random).

On a separate related note ... We have discussed useful proof-of-work ideas on the forums many times.  There just is not any feasible way to change the BTC proof-of-work without a catastrophic disruption to the entire system.   And of course, there's just not a whole lot of useful problems out there that are suitable to replace hashing...

However  it would be possible to create a dual-purpose mining program that would achieve this goal outside the BTC network.  The program would run the BTC mining, but could also have GPU kernels uploaded to it by anyone who wants work done.  The miner would transmit a BTC address and the requester pays the same BTC that miner would've produced.  Then the miner can switch to doing their work because it's a financially neutral decision. 

YES I am well aware that the BTC network needs the mining for security.  But, I don't envision that there would be enough demand to draw enough miners away from BTC to make an impact on the security of the network.  If there is enough demand, then that will drive the price of BTC up further and people will start buying more hardware (or perhaps, the special mining program would be designed to not switch if more than 50% of the miners are currently diverted.   Yes, this raises questions... this is mostly just off the top of my head)

This would create an extremely stable economy around BTC:  it would create a tangible backing to the work the miners are doing:  1 hour of GPU time can be bought for 0.01 BTC, so how much are people/companies will to pay for 1 hour of GPU time?  If it's $1, then suddenly BTC price will start gravitating towards $100/BTC because people will know they can always buy $100 worth of computation with 1 BTC.  There's no magic, faith or acts-of-god required:  BTC is worth exactly how much institutions/companies are willing to pay for computation time.  It would turn BTC into a global, pay-per-compute cloud that anyone in the world can use -- if they can afford to buy the BTC to feed the miners.

We just need to convince someone who is extraordinarily rich, to put up $700 million dollars to "back" Bitcoin for a 5-10 years.  They basically publicly announce that they will buy BTC for $100 each, and then the BTC must have that value, because you can always sell to that rich guy for $100.  In a way, he'd be like an exchange, but using his own money instead of others'.

Now it sounds ludicrous... who would risk that kind of money?  Well, how much of a risk is it, really?  People will start using BTC, because they know each is worth $100 and will realize the extra benefits they get out of it that aren't available with other currencies.  Merchants, buyers, savings accounts, etc, can all be converted to BTC because they know it can be converted to USD at any time.  "People" will eventually start to accept BTC as having value of $100, not because that one person is offering $100 for each one, but now because everyone else is.  At some point (probably many years), BTC would completely self-sustain as an independent currency without the billionaire backing it.  

At this point, he can basically just sell it off through the natural demand for it.  In fact, price is likely to go up as it actually becomes a useful commodity, since $700 million is not enough to support the economies that would be utilizing it.  

This is basically, exactly what happend with the USD (without the price spike, since they created enough to support the economy).  The government used to say "Each of these pieces of green paper can be traded back to the government for an equiv amount of gold".  Eventually they stopped doing that, yet USD still maintained its value depite losing its "official" backing.  This theory would only fail if there's no legitimacy to BTC -- if there's no extra value to holding BTC over USD, then why use it?  But when people start understanding the liquidity, the potential anonymity, the convenience, the low fees, the built-in escrow, etc, people will have a reason to continue using trading it. By then, there would be a critical mass of people/companies/businesses that deal with it and confirm its value that it would actually have that value.