Bitcoin Forum

Satoshi's Original Bitcoin Client - An Operational View

Preface
---------

I thought my client was taking too long to download the block chain and it
did not appear to operate smoothly. I thought I could do something to decrease
the block download time. So I downloaded the code and dug in.  Ultimately,
I failed to find the silver bullet to eliminate the long download delays
(big suprise!). But I did manage to penetrate the C++ code and figure
out how things worked for the most part.

So, I decided to write down my understanding of the code from an operational
perspective, to spare those who are not fluent in C++ from having to wade
through the code, which is quite dense and bit of a chore to pick apart,
when they really just want to know "how it works".

My focus was initially on the block download process, but I decided to
go ahead and cover all the major operational aspects I could (before losing
interest ;). I do think I found some areas for improvement, but that is not
the point of these articles. I will try to make it clear when I am stating
the facts versus when I am writing commentary.

I intend these articles to go into the Wiki at some point but I also
thought it would be useful to open topics in the forum in order to
allow for review in case I made a mistake or missed something big,
and for reference.


Overview
------------

This series of articles will focus on how the Satoshi bitcoin client
program operates, and less so on the protocol details and the rules
for processing blocks and transactions.

Satoshi's bitcoin client is a C++ program, so be sure to look for code in
both the .cpp and the .h header files. Also, the program is multithreaded.
This leads to some complexity and the use of certain code patterns to deal
with concurrency that may be unfamiliar to many programmers. Also, the
code is aggresive in the use of C++ constructs, so it will help to be
fluent with map, multmap, set, string, vector, iostream, and templates.


For information on how the bitcoin protocol works, see:
    The original Satoshi whitepaper:
        http://bitcoin.org/bitcoin.pdf
    The articles on the bitcoin.it Wiki:
        https://en.bitcoin.it/wiki/Category:Technical
    With special mention of the protocol specification:
       https://en.bitcoin.it/wiki/Protocol_specification
    And the protocol rules:
       https://en.bitcoin.it/wiki/Protocol_rules


-- Operations --

The client is oriented around several major operations, including:

    Initialization and Startup
        Upon startup, the client performs various initilization routines
        including starting multiple threads to handle concurrent operations.

    Node Discovery
        The client uses various techniques find out about other bitcoin
        nodes that may exist.

    Node Connectivity
        The client initiates and maintains connections to other nodes.

    Sockets and Messages
        The client processes messages from other nodes and sends
        messages to other nodes using socket connections.
    
    Block Exchange
        Nodes advertise their inventory of blocks to each other and
        exchange blocks to build block chains.

    Transaction Exchange
        Nodes exchange and relay transactions with each other.
        The client associates transactions with bitcoin addresses in the
        local wallet.

    Wallet Services
        The client can create transactions using the local wallet.
        The client associates transactions with bitcoin addresses in the
        local wallet. The client provides a service for managing
        the local wallet.

    RPC Interface
        The client offers an JSON-RPC interface over HTTP over sockets
        to perform various operational functions and to manage the local
        wallet.

    User Interface
        The user interface code is scheduled to be superseded by bitcoin-qt.
        Therefore, it is not covered in further detail.

See their individual articles for more detail on each of these operations.


-- fClient Mode --

It is worth noting that there is code in the client to allow it to
operate in a mode where it only downloads block headers.
The implementation is intended to be used as a lightweight client mode which
can operate without verifying and storing all blocks and transactions.

This is controlled by the fClient variable in the code which is currently
hard coded to false.  This is currently not considered to be finished code.

This mode is known as fClient mode and the phrase Simplified Payment
Verification (or SPV) mode has also been used to describe a lightweight
client approach.



-- Main Thread Level Functions --

init.cpp:
    main()
    ExitTimeout
    Shutdown
net.cpp:
    StartNode
    ThreadGetMyExternalIP
    ThreadMapPort
    ThreadSocketHandler
    ThreadOpenConnections
    ThreadMessageHandler
rpc.cpp:
    ThreadRPCServer
irc.cpp:
    ThreadIRCSeed
db.cpp:
    ThreadFlushWalletDB
ui.cpp:
    ThreadDelayedRepaint
    SendingDialogStartTransfer


-- Significant Classes By File --

net.cpp/.h:
    CNode:  handes one socket connection
    CInv
    CAddress
    CMessageHeader
    CRequestTracker

main.cpp/.h:
    CDiskTxPos
    CInPoint
    COutPoint
    CTxIn
    CTxOut

    CTransaction
    CMerkleTx
    CTxIndex

    CBlock
    CBlockIndex
    CDiskBlockIndex
    CBlockLocator

    CAlert : CUnsignedAlert

wallet.cpp/.h
    CWallet : CKeyStore
    CReserveKey
    CWalletTx : CMerkleTx
    CWalletKey
    CAccount
    CAccountingEntry

db.cpp/.h:
    CTxDB
    CKeyPool
    CWalletDB

bignum.h
    CBigNum

util.h
    CCriticalSection: used for thread contention


--
Search on "Satoshi Client Operation" for more articles in this series.

Transaction Exchange : https://bitcointalk.org/index.php?topic=41730.0
Block Exchange : https://bitcointalk.org/index.php?topic=41729.0
Sockets and Messages : https://bitcointalk.org/index.php?topic=41727.0
Node Connectivity : https://bitcointalk.org/index.php?topic=41726.0
Node Discovery : https://bitcointalk.org/index.php?topic=41722.0
Initialization and Thread Startup : https://bitcointalk.org/index.php?topic=41719.0

I'm only part way through 2 of yr articles so far but I just wanna say thanks for an awesome job.  This sort of documentation is exactly what new devs need to get across the client quickly.

Are you planning to publish this series on a site somewhere?  It would be great to have it all in one place where people can bookmark it and find it easily for reference.  It's going to get buried eventually in the forum.

Thank you very much for this. I haven't had the time to go over everything in detail, but from what I've seen, it is correct and well researched. Please make sure this ends up on the wiki - people wanting to contribute need this kind of information.

I didn't yet read all of this, but it seems like a terrific job! Thanks for doing this!

I suggest putting it somewhere, like a wiki, and adding some images and formatting for better presentation.

I haven't read all of this yet either, but it looks like an amazing piece of badly needed documentation. Thank you so much.

Bitrick, this is very good information.  It is exactly the kind of information I wanted, as someone who wants to get involved in BTC development but doesn't have the advanced C++ software skills needed to comprehend the Satoshi client code.  I look forward to using this as a secondary reference in my work, and helping iron out the details.

I would recommend you put links to each of the other posts at the bottom of this "Overview" post, and then one of the admins make this thread sticky.  I think people would expect to see this kind of information when they come to the "Development & Technical Discussion" forum,  and right now this is the most concise yet seemingly-complete version of this information I've seen so far.

Nice work!
-Eto

I don't have the time to read or use these info but they look really awesome -> wiki!

Excellent stuff!

Are you satoshi?  ::)

Very nice bitrick!

By the way, this patch speeds up initial download quite a lot:
  https://github.com/gavinandresen/bitcoin-git/commit/042a619709fab1329e8286c6aedbb2cdc8eb3497

...as do these, which have already been pulled into git head:
  https://github.com/bitcoin/bitcoin/commit/fb45259967032d409bca4d542b55414a7c522fba
  https://github.com/bitcoin/bitcoin/commit/ec74e8a44338202bfb82faa2cef4611cc37e7fa5

Thanks Gavin!
The first patch is the most promising. I actually tested turning off VerifySignature last week and calculated around double the block processing rate, when CPU limited. I was hoping for more, but every bit helps.

The last two patches I have less hope for. I know 0.3.23 can disconnect inappropriately, but in my testing any such disconnects should only result in a "brief" stall (brief == a few minutes of a multi-hour process). Note that the client should not disconnect a connection until all incoming and outgoing buffers are drained, so even if there is a disconnect, all the blocks queued prior to that can still get processed, so there should not be totally unproductive ongoing disconnections. (Or did I misunderstand the nature of the 0.3.23 problem?) I did not see a lot of 0.3.23 disconnect delays in my testing but maybe I just got lucky.
 
Thanks everyone for the nice comments. To reiterate: I do plan on putting this information into the wiki for future reference (and thanks for the suggestion Eto).

p.s. I am not Satoshi.

Thanks Bitrick.

these are quiet educational, but it will take me days to digest as a newbie programmer.

Great job! ;D

Thanks bitrick! I have not seen this information elsewhere.

Subscribe.

Instresting, abandon downloading blockchains from .323 client speed the downloading up.

nice job!

Watch.

I find it impressive you were able to synthesize a summary from such a complex set of code.

This is some interesting info for me, post your address so I can send a few bitcents ;-)

This reminds me of my Pseudo Bitcoin Client, were the idea is to develop a bitcoin client from scratch in each language to better help others what is happening in the Bitcoin client in their native (programming) lanauge and as well as documentation in Pseudo Code to allow others that only know English(but have a solid understanding of hashing functions) to understand fundamentally whats going on in the official Bitcoin client.

So far I just have Generating Bitcoin addresses in java code.
I would love to expand on it with different language however I don't know how the Bitcoin client works well enough to do it my self, anyways here is the link https://github.com/Xenland/Bitcoin-Pseudocode-Client/tree/gh-pages

I wonder how much time will I need to study the Satoshi's client..  ???

A lot!  But a lot less, if you ask questions!  :)

Yeah, but I think I still need to study bitcoin protocols a little more. I read the Satoshi's Bitcoin paper, but guess that's not enough to understand all that code... I really don't feel 'smart' right now... :)

Wow, very very in depth.

A++

This is a great asset to the community, well done and thank you for your hard work.

An Operational View https://bitcointalk.org/index.php?topic=41718.0
Initialization and Thread Startup: https://bitcointalk.org/index.php?topic=41719.0
Node Discovery: https://bitcointalk.org/index.php?topic=41722.0
Node Connectivity: https://bitcointalk.org/index.php?topic=41726.0
Sockets and Messages: https://bitcointalk.org/index.php?topic=41727.0
Block Exchange: https://bitcointalk.org/index.php?topic=41729.0
Transaction Exchange: https://bitcointalk.org/index.php?topic=41730.0

Would be nice to have all these valuable articles in ONE Posting or ONE new thread.

yes. this is a good public resource, thanks.

there are also some fairly good docs about the Bitcoin client mechanics on the BitcoinJ wiki: http://code.google.com/p/bitcoinj/

Thx man, this saved me a lot of searching.

Great job! Thanks.

i am think same

nice job!

this is exactly what I was looking for, thanks! One question though, being this post is over two years old now, is the original content still correct?

I second this question as I am trying to get a deeper understanding of bitcoin protocol, thanx

Amazing. Thank you so much.

An interesting reading.

nicely done. thanks

Very helpful. Thanks!
You've inspired me to do a complete line by line technical analysis of the code and writing it up and posting here over the next few weeks.

Ooh Konstantin.  I cannot wait.  I think thousands of programmers would be interested in that.  But do it on the latest client.  And somebody should do libcoin as well.

Thanks for providing the info.  I'm definitely interested!  ;D

what version of the code is this?

Old one, you can see the thread was made in 2011. From then to now the code has changed A LOT.

but which version? I have source for:

0.1.0
0.1.2
0.3.24

https://bitcointalk.org/index.php?topic=41730.msg2753401#msg2753401

seems that this s not version 0.1.0

The thread does say Original Bitcoin client, so probably from 0.3.23 and earlier?

I don't have enough time to read this info but at first sight it looks really cool!

It is a good guess~ :D

ha ha  the same question...   so good..

Thanks for doing this!
Hope this jod can help bitcoin become more strong

Great job!

I have worked with allot of open source applications before like VLC and I can tell you this sort
of clear concise documentation without fluff is invaluable.

We newbies stand in debt to you.

this documentation is so old, it borders on useless.  Some of these files don't exist and many of them have been refactored and/or moved.  Maybe when the dev team actually wants people to be able to help, they will take a moment to point people in the right direction.

See this reply :

I think they are trying to make it simpler.
Kindly,
       MZ

I do see that this thread was created in 2011, and probably longer relevant towards the current Satoshi Client; I would still like to thank you for your efforts in closing the gap between developers and users. I have been picking apart some of the older clients as well and with the way the code is laid out, it can quickly become overwhelming. Although my findings were similar to yours, it was a pleasurable read that helped me evaluate my findings with a better since of clarity.

Thanks!

that's really a nice instructions.

You deserve much appreciation dude.
Really it is very very helpful for endeavors like me. I have been looking for this kind of primary hand guide to start with. The best thing is you also mentioned what things need to be equipped with for understanding the source code. I'm going to make use of it now.

Thank you very much.

Everything I need to work on Satochi's and mine them, good information, thanks for the help!