Thanks.  Yeah all the endian changes make slogging through the code a real chore.  Any small performance gains are not worth the loss in transparency.

It is almost like a secret handshake that each developer needs to learn.

A quick question when you have the time (no rush), looking for clarification on tx_k & tx_v.

tx_k?
Looks like tx_k is the raw hex dump of the key as stored in the wallet db.  I would just be tx_id (tx_hash) however it looks like it is 02 + "tx" + reversed tx hash.  With the 02 being the length of the following string "tx".  Is that correct?

I assume this means that given an arbitrary tx hash one needs to reverse it and prepend 0x027478 to find the key/value pair in the db.  I know you didn't write the wallet but any insight as to why?  Why not just make the key the tx_id?  I mean if we start having hash collisions it means a lot more problems then just incorrect db reads.   


tx_v?
tx_v is the raw hex dump of the value stored in the wallet.db which should be the complete raw tx without any additional encoding.
Right?

(if it helps line 1687 in pywallet.py)

Agreed.  Maybe "file" was the wrong word I don't really care how bdb is structuring it on disk.  I am more interested in the structure and format of the various key value pairs which make up the wallet database.  So when bdb returns the next key as "XXXX" one knows what the hell it is and how to parse the stored value.  

Updated OP for clarity.

No problem.  Coins sent https://blockchain.info/tx/02f632bea4b192f899108b3706e001e828eed546eac0b19321154278c034c7c3  

Wait nobody has ever donated?  WTF? Soapbox mode engaged.

Public service announcement.  If pywallet has made your life easier (and you know it has), then show some support by sending some coins.  Come on this is a great tools to crack the byzantine mess that is wallet.dat.  Eventually the wallet will move to leveldb which is going to require a significant rewrite of pywallet.  Hint: developers tend not to continually support projects they see no support from the community.

BTW if you are reading this PSA, in the "Pywallet: manage your wallets/addresses/keys/tx's" thread, well the odds are it probably applies to you (otherwise you wouldn't be here).


Weird that is not what I am getting.

Here is a unmodifed copy & paste of one key (a heads up to scammers, I removed this unused key from my keypool so nothing to steal)


This is on pywallet running on Windows, using a wallet exported from v0.8.3 client in case that somehow makes a difference.



DOH, I never tried.  I assumed passphrase was required for encrypted wallets.  Yeah it does exactly that and that is exactly what I was looking for.  I feel a little stupid that I needed assistance with that, but still glad I asked.  Pure awesome.

Thanks JackJack, pywallet is an awesome tool.  Having a read-only wallet conversion option in pywallet is a good fit. 

I will still work on a "watching wallet" conversion tool for my own use.  If nothing else it will be some interesting development. 

It gave us 200 or so years of breathing room but now we are right back in the same place.  Bad news is it is hard to secede from yourself.

Correct.  In this context only the entity actually building and broadcasting a block are "miners". So yes the pool server, solo miners (ASICMiner), and setups like p2pool.  Basically if you are broadcasting the block yourself via bitcoind then you are a "miner".   IMHO "pool workers" aren't really miners and calling them that is inaccurate.  They are just computing power providers (CPPs). I actually coined that term in a request to FinCEN for an administrative ruling to highlight the distinction between the entity creating new blocks/coins and entities merely providing the resources.  We wouldn't call the power company or ISPs "miners" although electrical power and connectivity are required inputs to creating new coins/blocks.


Yes however the risk is centralization.  It isn't that the network "couldn't" handle unlimited blocks it is that we might not like the consequence of unlimited blocks. As the block sizes gets larger and larger it becomes more difficult to keep orphans to a manageable level.  Orphans directly affect profitability and for pools it is a double hit.  High orphans mean less gross revenue per unit of hashing power but it also means the pool is less competitive so miners move to another pool.  The pools revenue per unit of hashing power is reduced and the overall hashing power is reduced too.  So pools have a very large incentive to manage orphans.    Now it is important to remember it doesn't matter how long it takes for ALL nodes to get your block just how long it takes for a majority of miners to get your block.  The average connect between major miners is what matters.  

If the average connection can handle the average block then it is a non-issue.  However imagine if it can't, and orphan rates go up across the board.  Pools are incentive to reduce orphans so imagine if x pools/major solo miners (enough to make up say 60% of total hashing power) moved all their servers to the same datacenter (or for redundancy the same mirrored sets of datacenters around the world).  Those pools would have essentially unlimited free bandwidth at line speed (i.e. 1Gbps for cheap and 10Gbps for reaosnable cost).  The communication between pools wouldn't be over the open (slow) internet but near instaneous on a private network with boatloads of excessive bandwidth.  This is very simple to accomplish if the miners are in the same datacenter.  The miners just share one LAN connection on a private switch to communicate directly.   Now for these pools a 40MB, 400MB, or even 4000MB block is a non issue.  They can relay it to each other, verify, and start on the next block in a fraction of a second.  Near 0% orphan rates and reduced bandwidth costs.  For other miners however the burden of these large blocks means very high oprhan rates.  How long do you think it will take before CPPs abandon their pool with 5% orphan rates for ones with near zero.  That isn't good for decentralization of the network.

I don't want to sound doomsday but this why BANDWIDTH (not stupid disk space) is the critical resource and one which requires some careful thought when raising the block limit. It is important that average block size not exceed what a miner on an average connection can broadcast to peers in a reasonable amount of time (3 seconds = ~0.5% orphan rate) on an average public internet connection.  Granted those are pretty vague terms.  Obviously 1MB block is below that critical level and 1GB block is obviously above it.  Is 100MB fine today? How about in 10 years? Is 5MB fine today? How about 2MB and doubling every two years? I am not saying I have the answers but that is the kind of thing we (community at large not just Gavin et all) need to think about critically before saying "yeah lets go to unlimited blocks and let the market figure it out".  I have no doubt the market will figure it out however one might not like what end state it reaches.

The good news is bandwidth is still increasing rapidly and the cost per unit of data is falling just as rapidly.  This is true both at the last mile (residential connection) and in datacenters.  So it is an problem which is manageable as long as average block size doesn't eclipse that growth.


Don't take any of this as "set in stone" it is more like when they ask you "how many windows are there in New York City?" in an interview.  Nobody cares what the exact number, what the interviewer is looking for is what logic will you use to come up with an answer.  If someone thinks my logic is flawed (and it certainly might be) well that is fine and I would love to hear it.  If someone can convince me otherwise that is even better.  However show me some contrary logic. If the counterargument is merely "unlimited or it is censorship" well that doesn't really get us anywhere.

There are four different bandwidth bottlenecks.

Miners
Miners are somewhat unique in that they have to broadcast a block very quickly (say 3 second or less target) to avoid excessive orphans and the loss of revenue that comes with it.  This means their bandwidth requirements are "peaky".  That can be smoothed out somewhat with protocol optimizations however I would expect to see miners run into a problem first.  The good news is it is not that difficult for pools or even solo-miners to setup their bitcoind node in a datacenter where bandwidth is more available and at lower cost.

Non mining full nodes
Full nodes don't need to receive blocks within seconds.  The positive is that as long as they receive them in a reasonable amount of time they can function.  The negative is that unless we start moving to split wallets these nodes are likely on residential connections which have limited upstream bandwidth.  A split wallet is where you have a private bitcoind running in a datacenter and your local wallet has no knowledge of the bitcoind network, it just communicates securely with your private bitcoind.  An example of this today would be electrum client connecting to your own private secure electrum server.

Bootstrapping nodes
Another issue to consider is that if full nodes are close to peak utilization you can't bootstrap new nodes.  Imagine a user has a 10 Mbps connection but the transaction volume is 9 Mbps.  The blockchain is growing at 9Mbps per second so the user is only "gaining" on the end of chain at 1 Mbps.  If the blockchain is say 30 GB it will take not ~7 hours (10Mbps) but 70 hours to catch up.
The good news here is there is some slack because most residential connections have more downstream bandwidth then upstream bandwidth and for synced nodes the upstream bandwidth is the critical resource.  

SPV nodes
The bandwidth requirements for SPV nodes are negligible and unlikely to be an issue which is a good thing however SPV don't provide for network security.  While SPV are important so casual users are not hit with the rising costs of running a full node at the same time we want to ensure that the ability to run a full node for enthusiasts remains an realistic option.  Maybe not everyone can run a full node but it shouldn't be out the reach of a the majority of potential users (i.e. requires 200Mbps symetric low latency connectivity, 20TB of storage, enterprise grade RAID controller, 64GB or RAM, and quad xeon processors).  How many full nodes are needed.  Well more is always better, there is no scenario where more hurts us in any way so it is more likely how few can we "get away with" and stay above that number.  If 100 enough?  Probably not.  1,000? 10,000? 100,000?  10% of users, 1% of users?  I don't have the answer it is just something to think about.  Higher requirements for full nodes means less full nodes but more on chain trust-less transaction volume.  It is a tradeoff, a compromise and there is no perfect answer.  1GB blocks are non-optimal in that they favor volume over decentralization too much.  Staying at 1MB blocks forever is non-optimal in that it favors decentralization over volume too much.  The optimal point is probably somewhere in the middle and the middle will move with technology.  We don't need to get the exact optimal point, there likely is a large range which "works" the goal should be to keep it down the middle of the lane.

Well currently our company (Tangible Cryptography) services are halted as we dance with the state but we ran the site fastcash4bitcoins.com The bitcoind interface was used to track customer deposits and update a SQL database.  All high level functions are between the application and the RDBMS.  A service polls bitcoind periodically and updated the database with low level transaction data.

That is the important part.  Large verifiable trusted companies using Bitcoin don't remove the choice of (psuedo) anonymity.  The payment protocol Gavin is working on actually combines both.  It allows one to verify the recipient (using CA and signed payment orders) while still making the tx itself psuedo anonymous.  If you don't want the gubbermint tracking your battery or tinfoil purchases made on Amazon with Bitcoins you could use a mixer.  This is something that can't be done with CC for example. 


I think a lot of the "issues" with Bitcoin are really issues with user naivety.  Bitcoin is like digital cash.  It is an imperfect metaphor but it works for the most part.  Most people would not hand a bag stuffed with cash (hereafter refered to as "bag-o-cash" to a "company" rep standing in an random alley which changes every day that has no ID, no name, no method of verification.  You don't even need to post warnings about how/why you shouldn't do that.  "Thanks for the bag of cash, wait here and I will get your product ... soon (sucker)".  However for some people they essentially do the EXACT same thing with Bitcoins and are surprised that they got scammed.  Somehow it is less risk because .... Bitcoin.

I think it is good for all Bitcoin users to stop before pushing [Send] and just ask themselves "If Bitcoin didn't exist and my only option of payment was fiat cash would I feel comfortable handing this "person" a bag-o-cash given the information I have and can verify?"  If the answer is "no" then you absolutely shouldn't be sending them Bitcoins either, so take your hand off the mouse and slow down.  It really is that simple.

If NewEgg, Amazon, or namecheap tomorrow started accepting only cash I would have issues with the risk of paying that way (it would be annoying but that is a different topic).  Likewise if they accept Bitcoins I wouldn't have any more or less issue then paying them by cash.  Escrow in those cases simply adds cost and expense.

On the other hand if some guy on Craigslist wanted to sell me a pre-order "super ultra hyper Bitcoin ASIC miner" but I had to buy today (they only have one left) and I need to pay in advance by sending a bag-o-cash to a PO BOX and wait 3 months I "probably"* would decline.  The same guy asking for Bitcoins instead doesn't make is less risky so the answer is still no.

I can't explain it beyond absolute naivety on the part of some users.  The problem is that enough "foolish" users makes it a target rich environment.  That brain dead scammer knows his pay cash and wait 3 months for magical beans won't work but change cash to Bitcoin and magical beans to ASICs and there is a line of suckers fighting to get rid of their wealth.  The amount of scamming will not go down until the "street smarts" of the users go up and the bad news is it will take a lot more losses for that to happen.  It likely will take time for "common sense" to catch up with the technology.  It doesn't help that credit cards have made the average consumer brain-dead when it comes to managing risk.  CCs transfer all the risk (and consequences for asinine decision making) from the consumer to the merchant so why bother managing risk.  Hell you can post your CC on facebook homepage (so you don't have to look in your wallet) and it costs you absolutely nothing. Consumers don't need to be "smart" with CC like they do with cash (or Bitcoins).  It will take time to deprogram the average user.

These aren't technological issues and they can't be solved by technology.  


* "Probably" means WTF? Are you on crack?

Yeah very nice set of capabilities.  The fact that it is "built into windows" and requires no installation makes it useful for rapid deployment at least in windows environment.   With a data store abstraction layer a variety of storage systems could be supported.  ESENT could just be the default (providing no install instant availability for windows users).

Performance even on random lookups seems to be exceptional.  The bad news is that it looks like only certain datatypes can be used for keys.  Would be nice to store blocks with blockhash in binary (256 bit BigInteger) as the key.  Not sure if this is a limit of ESENT or just the current implementation of the managed code wrapper).  Still if performance using a hexadecimal string representation of that block hash has good enough performance it many be a non-issue.

One thing that may be worth considering is use ESENT for storing the blockchain and then use something like SQL Server (w/ Express for users without installed RDBMS) to store: wallet data (keypairs, tx history, user metadata), the UXTO, and a smaller table of just block headers.  The UXTO is less than 250MB, block headers since genesis block are about 15MB and most wallets are under a couple MB.  We have one massive (8,000+ keys, 20,000 txs) wallet and it is only 40MB.  All that should fit fine within the constraints of Express.

I am making a project to dump the blockchain into ESENT and benchmark performance (sequential inserts, random reads).

You can't mine XRP period.  Even the creators didn't mine it.  They simply (like fiat) said "Let there be 100 billion XRP" and there was.  Then they decided unilaterally (he who holds the money makes the rules) that they would keep half of it as profit and transferred the other half to a OpenCoin.  OpenCoin has given away some unknown % of their half.  There is no mining of XRP, there will never be any mining of XRP.  It is like saying "at least you can mine Amazon Coins or Facebook credits with less resources".

If you find nothing else our company has additional visual studio enterprise licenses which are not being used.  I imagine we could work something out (i.e. if necessary hire you as a contractor for 1 bitcent per year for the sole purpose of developing c# open source code to the public domain or under copyleft license, renewable annually).  I need to speak to counsel about it but if interested send me a PM. 

No ulterior motive here.  The public codebase for Bitcoin + .NET has been pretty much been non-existent.  C# is my preferred language so everything I have written has needed to be custom built (use a modified headless version of bitcoind running as a windows service with all interaction through a c# to JSON/RPC wrapper.    It would be useful to get a solid managed code library going (stop reinventing the wheel) if I can support that with a license it works for me.

First of a massive thank you to JackJack.  Not only is this a useful tool but I find python a more expressive language than the C/C++ used in the reference client.  This makes pywallet a useful tool for learning the wallet format.

Is the address in your sig best address for donations for further pywallet development?

Also when pywallet exports keys it provides multiple values like this


addr, compressed, encrypted_prvikey, pubkey, reserve are all self explanatory.

However a little confused on hexsec, sec, and secret. Hexsec and sec appear to be identical in my dump for all keys.  Will they always be?  At first I though secret was private key in hexadecimal and sec was private key in import format but I am confused by fact that there are three entries and the naming "hexsec" is obviously not in hexadecimal.

Lastly couldn't find it but is there an option to do a dump of an encrypted wallet (without providing passphrase).

As an example something like


For disaster recovery I print out plain text versions of company wallets (as well as normal binary backups) an encrypted dump would be an awesome feature.

So doing some code spelunking in both pywallet and QT client I think I have figured most of it out.

version = version which last updated the wallet.
minversion = minimum version required to read wallet (throws error in older clients).
tx = transaction record involving a key in the wallet.
name = user entered label of a key (possibly a tx/account as well? or just keys?)
acc & acentry = used in QT accounting system (not really interested)
bestblock = highest block height seen/scanned by client
mkey = has nothing to do w/ Bitcoin keys, it contains info on encrypting/decrypting the wallet.
pool = list of public keys in keypool with their order (seems redundant by ok)

so that leaves "key", "wkey", and "ckey" which appear to all be related to bitcoin keypairs (addresses).

Why three different key-types for handling Bitcoin keypairs?
What is the difference?

Then there is "orderpostnext" and "cscript" which I can't make heads or tails of.

You aren't by any chance a student are you?

https://multibit.org/

It is linked to and recommended from bitcoin.org

There is nothing they can do.  It doesn't matter where your wallet is (even your own wallet) Bitcoin is designed to be irreversible.

Imagine you handed a stranger with no name, address, or ID, a bag of cash with $300 USD inside and he said "wait right here I will be back in a couple days with your phone", two days pass and the guy hasn't shown up.  How would you reverse this transaction?

Sorry for you to learn the hard way, but in the future before clicking [Send] ask yourself "do I trust this person and know enough about them that I would hand them a bag of cash and know they would delivery?"  If the answer is no then don't send your coins.  It is the EXACT same thing.

Your continual claim that golden blocks are novel and nothing like superblocks when the code is bit for identical to LuckyCoin.  This could be a mere mistake because you are woefully ignorant on crypto-currencies however even after being shown the exact code with link to "your" repository showing identical code you continue to make a false statement.

Sorry that is a lie.

Once again in case you "missed it" for the third or fourth time:
https://bitcointalk.org/index.php?topic=256460.msg2759962#msg2759962



No you can't.

Because you are an ass and most people don't want the stress of working with you.  They saw your 30+ pages of nonsense and life stories, and lies, and false promises, and bashing prior programmers, and worthless rants.

You are a liability.  Not the code, not the developer, not the coin ... you.  The only people who would take you up on your offer are those who have no reputation to risk and enjoy pain.

By the letter of the guidance there is no exemption for using a registered service.  Yes it sounds stupid and it is improbable that FinCEN would go after each miner but if you exchange virtual currency for virtual or real currency you are required to register unless an exemption applies.  One exemption is for not for profit, infrequent activity. 

So to answer your question.

Miner exchanges virtual currency on exchange for USD?  Yes an MSB (I doubt this will be enforced but the guidance provides no exemption)
Miner exchanges coins for bullion?  No.  Sells of goods or services (other than money transmission) are exempt
Miner exchanges one virtual currency for another?  Yes an MSB

Once again I don't necessarily agree with this, nor do I think it makes much sense, and it is highly unlikely FinCEN will (or even will be able to) enforce the regs to the letter of the guidance but that is what they advised.