Exact binary map of database blockchain?!

[etotheipi]

@kjj
If its really all so "simple" - why isn't it likewise simply explained?

Ah!
What do you mean with "bitcoin is nested" ?? What do you mean with "a lot of the complicated bits are in the interactions"
To begin with: what exactly is that "interaction"?
So far I only heard from transfers/transactions.
If you mean the same, what exactly do you mean with "complicated bits" (bits?)?

LvM - the point was that you are going to have to put in some elbow grease like the rest of us and figure out the minutiae.  Once you look at some binary maps and match them up against the serializations described on the protocol page, it will seem a lot simpler.  Only so much of it can be spelled out for you.  Trying to spell it out even further isn't all that productive, because you're not going to "get it" until you dive in anyway, and then those documents will seem entirely sufficient.  

100% of the details may not be there.  But most of them are.  It's accurate.  Ask questions.  Look at other implementations.  

Open blocks/blk00000.dat and read the first 300 bytes.  Then match it up against the protocol page and what I wrote above.  You'll see the magic bytes, the numBytes, the 80-byte header, the numTx, then a list of Tx.  You can then jump to an arbitrary block.  Jump to block 170 which has the first real transaction (it has two tx: a coinbase tx and then are regular pay-to-address transaction).  

[kjj]

@kjj
If its really all so "simple" - why isn't it likewise simply explained?

Ah!
What do you mean with "bitcoin is nested" ?? What do you mean with "a lot of the complicated bits are in the interactions"
To begin with: what exactly is that "interaction"?
So far I only heard from transfers/transactions.
If you mean the same, what exactly do you mean with "complicated bits" (bits?)?

Maybe you would be better off asking specific questions.  All of the information has been provided, but we are apparently unable to translate our answers into a form that is simple enough for you. 

[LvM]

The question is specific, absolutely precise, simple, quite normal and most important if third party datas must be used.
I would and could answer the same question regarding my datas/databases within minutes.

The blockchain must be used by each client, by all users of BTC.
So I can't help but assume: the details are unknown or hidden deliberately.
Top secret

[etotheipi]

The question is specific, absolutely precise, simple, quite normal and most important if third party datas must be used.
I would and could answer the same question regarding my datas/databases within minutes.

The blockchain must be used by each client, by all users of BTC.
So I can't help but assume: the details are unknown or hidden deliberately.
Top secret

LvM, I don't know what the problem is.  That protocol page very precisely lays out all of the serializations of block data and messages.  And it's accurate.  Everyone who has developed any Bitcoin software has used the information on that page, and errors subsequently flushed out if they were found.  It's not 100% complete, but it's accurate in what it says, even if it's not the most visually pleasing/organized descriptions possible. 

What kjj was alluding to was that there's deeper meanings to a lot of these things.  And while documentation could be improved, a lot of you just get from experience.  If the documentation is lacking, help fix it.  But for where you're at, there is more than enough there. It describes the binary maps exactly as you wanted.  Like I said, just open blocks/blk00000.dat and look at the first 300 bytes in a hex editor and compare against the data on that page.  You'll get it.  But it can't be spoon-fed to you.

[kjj]

Ahh, fuck it.  I was trying to cut you some slack because I figured english wasn't your first language, but you found it out.

All of it is top secret.  The published source code is fake, the extensive documentation on the wiki is all fake, even the posts on the forum and to the mailing list are fake.  Blocks are really just incomprehensible blocks of random data

[tyrion70]

Lvm, what bank do u work for? And what is your job there? I expect you do functional maintenance or something similar? I work at a bank as well, and I develop software as well. I concur that the bitcoin specs aren't anything near to the specs for like a SEPA transfer, but you forget one thing: because banks all have their own software (most still written in cobol :-p), banks have a need for perfect specs. Bitcoin is bitcoin. One client which uses the wallet and blockchain. There can never be discussion between bitcoin and itself, it simply works.

Thats what I, kjj and ethotheipi tried to explain, ethotheipi even went much farther than that giving you the closest thing to specs I've ever seen on the subject (thanks for that btw!)

Bottom line, if you want exact specs, read the code, bitcoin is not a bank and is not by any means "required" to hand out specs or api's. If you have a specific question (other than gimme the specs) because of a project you are working on, ask the questions instead of the specs.

Lastly, don't try harder, try different ;-)

[LvM]

@etotheipi

Be sure, any halfway experienced developer having to communicate with other Computers/datas
first of all studies the BASICS, often called "protocols".

This was one of the first things I did, of course - weeks ago.
Spent hours and days searching usable, complete and reliable infos, starting with Satoshis paper (in this respect totally unusable).
I really couldn't and can't believe it.

I am working with databases since decades, first programming them myself (yes!), then using half a dozen of all the very similar DBMS (dBase, SQL, MySQL, BdB ...).

The basics are logically always the same: Headers, records, fields, indexfiles.
If variable field lens are used, we need delimiters or len specifiers for them.
Any encryption, hashes, Endianness, compression...  are next to be explained.

I normally have no problem with all that. Normal electronic banking belongs to the easiest things I do.
Since databases never are loaded into memory they normally are not splitted as in BTC with its numerous blkxxx.dat + revxxx.dat
This and much more I would like to understand.

Seearching for infos about BTC I found a mess of superficial, inconstistent mixtures of this and that, advertising and promotion mixed with technical infos, even "Messages" and DB-structures -although complete different things- mixed together, nothing fully explained.

Why James, Bitrick, etotheipi tried to provide what I am searching for?
Thats why I asked my silly questions.

Where eg. is the date/time of transfers ? The "lock_time" is a different thing, mostly zero.
The block creation time might be interesting for miners or other internals, not for applications.

Just because you mentioned it above, lets take another example:
Above you write about "BlockHeader(80)"
whereas in the https://en.bitcoin.it/wiki/Protocol_specification
the field sizes of Block Headers are 81.
You know it yourself:
One byte difference on this level is enough to crash a complete app.

[LvM]

Satoshi Nakamoto, naming himself the "founder" of BTC might be an expert in cryptology.

This might explain that all what he writes in his basic article
"Bitcoin: A Peer-to-Peer Electronic Cash System"
http://bitcoin.org/bitcoin.pdf
has NOTHING todo with basic knowledge of money and bookkeeping.

His main factual problem is his so called "prevent double-spending".
Normally "a trusted third party [bank] is required to prevent double-spending" he tells us.

Sorry.
Never heard that "double-spending" i.e. spending the same money (or other things) twice is possible at all.

In case of real coins (and all other things) each child knows that, I hope.
In case of cashless payments its exactly the same since each payment reduces my disposable fund.

Where no money, there no payment.
Controlling that by software designed to manage whichever currency is normally the easiest thing at all.

Security and privacy are very important, but totally different questions.
They have as in all other cases (cars, gold or potatoes) logically nothing todo with the things protected.

But Satoshi Nakamoto sees that quite different, stylizing prevent of "double-spending" of BTC to the main task BTC has to care for.

Talking about Bitcoins in its title the central topic BTC is mentioned in his paper not even ONCE.
Instead he only speaks about important but secondary problems, about hashes, keys, "proof-of-work" etc.
But first comes the basic logic, then security and encryption - everywhere.

This might explain that even the basic logic of BTC is at least for me not understandable at all.
Perhaps he didn't understand it himself.

[LvM]

Nakamoto/BTC vs. bookkeeping
As I wrote

Bitcoins are money.

The basic logic of money transfers from A to B normally belongs to the easiest things in the world.
Technical things like encryption or the structure of involved databases change nothing in the very, very simple logic of money transfers.

In short:
A minus for the payer is a plus for the receiver.
At least time and reason of transfers are always added normally.
All is collected and balanced in the accounts of payers/receivers.
Easiest bookkeeping.

Some of these things either do not exist in BTC or simply are somehow hidden.
Where e.g. is the field for the transaction date/time?
I assume that there IS one, since Armory restored by the blockchain date and time of my transactions.

Lost after a restore were my manual "Comments" of transactions only.
BTC does indeed not have a field for the reason of transfers.
Other things like the strange "changings" not to mention....
Sorry, but even after reading quite a lot I am far away to understand that.

Satoshi Nakamoto writes:

Abstract. A purely peer-to-peer version of electronic cash would allow online payments to be sent directly from one party to another without going through a financial institution. Digital signatures provide part of the solution, but the main benefits are lost if a trusted third party is still required to prevent double-spending.

We propose a solution to the double-spending problem using a peer-to-peer network.
The network timestamps transactions by hashing them into an ongoing chain of hash-based proof-of-work, forming a record that cannot be changed without redoing the proof-of-work. The longest chain not only serves as proof of the sequence of events witnessed, but proof that it came from the largest pool of CPU power. As long as a majority of CPU power is controlled by nodes that are not cooperating to attack the network, they'll generate the longest chain and outpace attackers. The network itself requires minimal structure. Messages are broadcast on a best effort basis, and nodes can leave and rejoin the network at will, accepting the longest proof-of-work chain as proof of what happened while they were gone.
...
In this paper, we propose a solution to the double-spending problem using a peer-to-peer distributed timestamp server to generate computational proof of the chronological order of transactions. The system is secure as long as honest(??) nodes collectively control more CPU power than any cooperating group of attacker nodes.

9. Combining and Splitting Value (unbelievable!!))
Although it would be possible to handle coins individually (), it would be unwieldy to make a SEPARATE TRANSACTION FOR EVERY CENT() IN A TRANSFER.
To allow value to be split and combined,() transactions contain multiple inputs and outputs. Normally there will be either a single input
from a larger previous transaction or multiple inputs combining smaller amounts, and at most two outputs:
one for the payment, and one returning the change, if any, back to the sender. ?
(unbelievable!!)

12. Conclusion
We have proposed a system for electronic transactions without relying on trust.
[cmnt: misleading! we rely on trust. Trusted "party" is just like a bank the public blockchain!!]

We started with the usual framework of coins made from digital signatures, which provides strong control of ownership, but is incomplete without a way to prevent double-spending.
To solve this, we proposed a peer-to-peer network using proof-of-work to record a public history of transactions that quickly becomes computationally impractical for an attacker to change if honest(?) nodes control a majority of CPU power.
The network is robust in its unstructured simplicity. Nodes work all at once with little coordination. They do not need to be identified, since messages are not routed to any particular place and only need to be delivered on a best effort basis. Nodes can
leave and rejoin the network at will, accepting the proof-of-work chain as proof of what
happened while they were gone. They vote with their CPU power, expressing their acceptance of
valid blocks by working on extending them and rejecting invalid blocks by refusing to work on
them. Any needed rules and incentives can be enforced with this consensus mechanism.
========================

It might be a sacred cow, but I can't help it:
The basic logic of BTC as explained by Nakamoto himself, is wrong.

This does not mean, BTC is not usable at all.
But most complications have nothing todo with encryption or the public peer-to-peer database/network.
These are secondary technical questions, used except the public database in all "normal" electronic banking.

The problems have todo with the fact, that Satoshi Nakamoto as cryptologist forgot to implement the basics of normal bookkeeping.
He ignored the fact, that his blockchain is logically nothing else than a normal bank and should work like that, of course.
His main factual interest ("prevent double-spending") would be -since logically impossible- no special problem at all.

[kjj]

Where eg. is the date/time of transfers ? The "lock_time" is a different thing, mostly zero.
The block creation time might be interesting for miners or other internals, not for applications.

Just because you mentioned it above, lets take another example:
Above you write about "BlockHeader(80)"
whereas in the https://en.bitcoin.it/wiki/Protocol_specification
the field sizes of Block Headers are 81.
You know it yourself:
One byte difference on this level is enough to crash a complete app.

Ahh, good.  Specific questions.

Transactions don't have timestamps, but blocks do.  The timestamp on a block is considered to be the time that all of the transactions inside it happened.

Lock_time is indeed something else.  A transaction is not allowed to be included in a block until the lock_time has expired.  If lock_time is less than 500,000,000, it is considered to be a block index.  If greater, a UNIX timestamp.

The header network message is different from the header that gets hashed.  The first 80 bytes get hashed.  The network message includes that 80 bytes, plus a variable length integer transaction count.  I think I mentioned in an earlier post that the exact spot where the block header ends and the block body begins was a bit fuzzy.

P.S.  Your commentary on the whitepaper is wrong, deeply wrong.  You may know a bit about banking, but you don't seem to know anything about the history of cryptocurrencies.  Bitcoin is the solution to problems that you don't seem to even understand as problems.

[jackjack]

Ahh, fuck it.  I was trying to cut you some slack because I figured english wasn't your first language, but you found it out.

All of it is top secret.  The published source code is fake, the extensive documentation on the wiki is all fake, even the posts on the forum and to the mailing list are fake.  Blocks are really just incomprehensible blocks of random data

OH LORD! Why did you write it publicly?? 
I thought all Hero Members had to sign the Biggest Lie Of All-Time Non-Disclosure!!

[2112]

Oh Lordy! Those are the best unintentionally funny posts in a long time. I just quote them for the future reference, when I have to retell the story of "professional code monkey contra Bitcoin". Not just plain, garden-variety, cowboy code monkey a PROFESSIONAL code monkey.

When "Who moved my cheese?" was published and became an instant hit I keept saying that the book is so exagerrated that it becomes an unintentional comedy. Now I know that this book was not an exagerration.

I'm going to quote the signature too, because it makes the whole story even funnier:

Anforderungen an eine PROFESSIONELLE BTC-Anwendung https://bitcointalk.org/index.php?topic=189669
BANKGEHEIMNIS mit BTC gleich NULL!? https://bitcointalk.org/index.php?topic=188383 Antwort: Ja, wenn man nicht höllisch aufpaßt.

@etotheipi

Be sure, any halfway experienced developer having to communicate with other Computers/datas
first of all studies the BASICS, often called "protocols".

This was one of the first things I did, of course - weeks ago.
Spent hours and days searching usable, complete and reliable infos, starting with Satoshis paper (in this respect totally unusable).
I really couldn't and can't believe it.

I am working with databases since decades, first programming them myself (yes!), then using half a dozen of all the very similar DBMS (dBase, SQL, MySQL, BdB ...).

The basics are logically always the same: Headers, records, fields, indexfiles.
If variable field lens are used, we need delimiters or len specifiers for them.
Any encryption, hashes, Endianness, compression...  are next to be explained.

I normally have no problem with all that. Normal electronic banking belongs to the easiest things I do.
Since databases never are loaded into memory they normally are not splitted as in BTC with its numerous blkxxx.dat + revxxx.dat
This and much more I would like to understand.

Seearching for infos about BTC I found a mess of superficial, inconstistent mixtures of this and that, advertising and promotion mixed with technical infos, even "Messages" and DB-structures -although complete different things- mixed together, nothing fully explained.

Why James, Bitrick, etotheipi tried to provide what I am searching for?
Thats why I asked my silly questions.

Where eg. is the date/time of transfers ? The "lock_time" is a different thing, mostly zero.
The block creation time might be interesting for miners or other internals, not for applications.

Just because you mentioned it above, lets take another example:
Above you write about "BlockHeader(80)"
whereas in the https://en.bitcoin.it/wiki/Protocol_specification
the field sizes of Block Headers are 81.
You know it yourself:
One byte difference on this level is enough to crash a complete app.

Satoshi Nakamoto, naming himself the "founder" of BTC might be an expert in cryptology.

This might explain that all what he writes in his basic article
"Bitcoin: A Peer-to-Peer Electronic Cash System"
http://bitcoin.org/bitcoin.pdf
has NOTHING todo with basic knowledge of money and bookkeeping.

His main factual problem is his so called "prevent double-spending".
Normally "a trusted third party [bank] is required to prevent double-spending" he tells us.

Sorry.
Never heard that "double-spending" i.e. spending the same money (or other things) twice is possible at all.

In case of real coins (and all other things) each child knows that, I hope.
In case of cashless payments its exactly the same since each payment reduces my disposable fund.

Where no money, there no payment.
Controlling that by software designed to manage whichever currency is normally the easiest thing at all.

Security and privacy are very important, but totally different questions.
They have as in all other cases (cars, gold or potatoes) logically nothing todo with the things protected.

But Satoshi Nakamoto sees that quite different, stylizing prevent of "double-spending" of BTC to the main task BTC has to care for.

Talking about Bitcoins in its title the central topic BTC is mentioned in his paper not even ONCE.
Instead he only speaks about important but secondary problems, about hashes, keys, "proof-of-work" etc.
But first comes the basic logic, then security and encryption - everywhere.

This might explain that even the basic logic of BTC is at least for me not understandable at all.
Perhaps he didn't understand it himself.

Nakamoto/BTC vs. bookkeeping
As I wrote

Bitcoins are money.

The basic logic of money transfers from A to B normally belongs to the easiest things in the world.
Technical things like encryption or the structure of involved databases change nothing in the very, very simple logic of money transfers.

In short:
A minus for the payer is a plus for the receiver.
At least time and reason of transfers are always added normally.
All is collected and balanced in the accounts of payers/receivers.
Easiest bookkeeping.

Some of these things either do not exist in BTC or simply are somehow hidden.
Where e.g. is the field for the transaction date/time?
I assume that there IS one, since Armory restored by the blockchain date and time of my transactions.

Lost after a restore were my manual "Comments" of transactions only.
BTC does indeed not have a field for the reason of transfers.
Other things like the strange "changings" not to mention....
Sorry, but even after reading quite a lot I am far away to understand that.

Satoshi Nakamoto writes:

Abstract. A purely peer-to-peer version of electronic cash would allow online payments to be sent directly from one party to another without going through a financial institution. Digital signatures provide part of the solution, but the main benefits are lost if a trusted third party is still required to prevent double-spending.

We propose a solution to the double-spending problem using a peer-to-peer network.
The network timestamps transactions by hashing them into an ongoing chain of hash-based proof-of-work, forming a record that cannot be changed without redoing the proof-of-work. The longest chain not only serves as proof of the sequence of events witnessed, but proof that it came from the largest pool of CPU power. As long as a majority of CPU power is controlled by nodes that are not cooperating to attack the network, they'll generate the longest chain and outpace attackers. The network itself requires minimal structure. Messages are broadcast on a best effort basis, and nodes can leave and rejoin the network at will, accepting the longest proof-of-work chain as proof of what happened while they were gone.
...
In this paper, we propose a solution to the double-spending problem using a peer-to-peer distributed timestamp server to generate computational proof of the chronological order of transactions. The system is secure as long as honest(??) nodes collectively control more CPU power than any cooperating group of attacker nodes.

9. Combining and Splitting Value (unbelievable!!))
Although it would be possible to handle coins individually (), it would be unwieldy to make a SEPARATE TRANSACTION FOR EVERY CENT() IN A TRANSFER.
To allow value to be split and combined,() transactions contain multiple inputs and outputs. Normally there will be either a single input
from a larger previous transaction or multiple inputs combining smaller amounts, and at most two outputs:
one for the payment, and one returning the change, if any, back to the sender. ?
(unbelievable!!)

12. Conclusion
We have proposed a system for electronic transactions without relying on trust.
[cmnt: misleading! we rely on trust. Trusted "party" is just like a bank the public blockchain!!]

We started with the usual framework of coins made from digital signatures, which provides strong control of ownership, but is incomplete without a way to prevent double-spending.
To solve this, we proposed a peer-to-peer network using proof-of-work to record a public history of transactions that quickly becomes computationally impractical for an attacker to change if honest(?) nodes control a majority of CPU power.
The network is robust in its unstructured simplicity. Nodes work all at once with little coordination. They do not need to be identified, since messages are not routed to any particular place and only need to be delivered on a best effort basis. Nodes can
leave and rejoin the network at will, accepting the proof-of-work chain as proof of what
happened while they were gone. They vote with their CPU power, expressing their acceptance of
valid blocks by working on extending them and rejecting invalid blocks by refusing to work on
them. Any needed rules and incentives can be enforced with this consensus mechanism.
========================

It might be a sacred cow, but I can't help it:
The basic logic of BTC as explained by Nakamoto himself, is wrong.

This does not mean, BTC is not usable at all.
But most complications have nothing todo with encryption or the public peer-to-peer database/network.
These are secondary technical questions, used except the public database in all "normal" electronic banking.

The problems have todo with the fact, that Satoshi Nakamoto as cryptologist forgot to implement the basics of normal bookkeeping.
He ignored the fact, that his blockchain is logically nothing else than a normal bank and should work like that, of course.
His main factual interest ("prevent double-spending") would be -since logically impossible- no special problem at all.

[LvM]

@kjj

Transactions don't have timestamps, but blocks do.  The timestamp on a block is considered to be the time that all of the transactions inside it happened.

In reality all transactions even in the same block differ in time.
Nonetheless they all use the same timestamp? Can't really believe that.
Nonetheless that exactly the chronological order of transfers avoids double-spending?

Armory restored date+time of all my different transactions (as it seems correctly)
Was this something like a trick? Visionary power

P.S.  Your commentary on the whitepaper is wrong, deeply wrong. 

Why

You may know a bit about banking, but you don't seem to know anything about the history of cryptocurrencies.  Bitcoin is the solution to problems that you don't seem to even understand as problems.

Sorry, modesty is my middle name. But I can assure that I know more than "a bit" about banking and money, legally, economically, logically.
"History" and/or renaming money to "cryptocurrencies" changes what??

Be sure:
the basic logic I talk/ed about, was, is and will always be the same - since 100thousand years, forever.

Proofed very well in the other only aggressive comments, totally free of any arguments.

[LvM]

But might be that anybody understands "OH LORD" or "Funny" as an argument.
ROFL!

[Apocalyptic]

@LvM

I suggest you stop posting this non-sense, you are only embarassing yourself. By now nobody in the community will take you seriously and your "(unbelievable !!)" irritating comments on the whitepapers just demonstrate you fail to grasp the very basis of bitcoin.

[jackjack]

@LvM

I suggest you stop posting this non-sense, you are only embarassing yourself. By now nobody in the community will take you seriously and your "(unbelievable !!)" irritating comments on the whitepapers just demonstrate you fail to grasp the very basis of bitcoin.

Please no
That's too funny to stop now

[2112]

@LvM

I presume that you are German and because you title yourself PROFESSIONAL you hold a Dipl.Ing. certificate or some such.

As a holder of such certificate you have some ethical obligations, e.g.

a) do not consult outside of your area of expertise
b) continuously educate yourself on the progress in your area.

You failed on both counts.

Germany has a long history of an effective regulation, most famously the German Beer Purity Law from 1516. Violations of that law were punished by burning all the tools used to brew and/or sell beer in a negligent way. The repeated offenders were publicly whipped at a pillory.

Here you are getting off easy: nobody's going to burn your computers and you will be subject just to the some ridicule, not a corporal punishment.

If you need inspiration on how to fuse the Bitcoin technology with the traditional banking and finance technology take an example from grau and his bitsofproof.

https://bitcointalk.org/index.php?topic=122013.msg1350149#msg1350149

Edit: corrected the spelling of Dipl.Ing. (Diplom-Ingenieur, German equivalent of Master of Science)

[leijurv]

Please post more, LvM! You're making us laugh.

[LvM]

If falsificaton of clearly analyzed facts is impossible the word "nonsens" certainly is a very strong argument also.

So far we have:
"OH LORD"
"Funny"
"Nonsens"

Forgotten something? Any other strong or even stronger arguments?

[2112]

The problems have todo with the fact, that Satoshi Nakamoto as cryptologist forgot to implement the basics of normal bookkeeping.
He ignored the fact, that his blockchain is logically nothing else than a normal bank and should work like that, of course.
His main factual interest ("prevent double-spending") would be -since logically impossible- no special problem at all.

Today I had an opportunity to open the first edition of Bruce Schneier's "Applied Cryptography" from 1994. The second paragraph of the "Digital Cash" chapter explains double-spends in his rare style that is both highly entertaining and highly educational. Since we now have year 2013 this marks nearly two decades of you being out of touch with the progress in your claimed field of expertise.

Happily, there is a complicated protocol that allows for authentic but untraceable messages. Lobbyist Alice can transfer digital "cash" to Senator Bob so that newspaper reporter Eve does not know the identity of Alice. Bob can then deposit that electronic money into his bank account, even though the bank has no idea who Alice is. But if Alice tries to bribe two different Congresspersons with the same piece of digital cash, which is easy to try with a bit copy program, she will be detected by the bank. And if Bob tries to deposit the same piece of digital cash into two different accounts, he will be detected--but Alice will remain anonymous.