Bitcoin Forum
November 10, 2024, 03:55:07 PM *
News: Latest Bitcoin Core release: 28.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1] 2 3 »  All
  Print  
Author Topic: Satoshi Client Operation: Overview  (Read 64630 times)
bitrick (OP)
Member
**
Offline Offline

Activity: 64
Merit: 140


View Profile
September 06, 2011, 03:38:09 AM
Last edit: September 23, 2011, 12:49:48 PM by Gavin Andresen
Merited by ABCbits (22)
 #1

Satoshi's Original Bitcoin Client - An Operational View

Preface
---------

I thought my client was taking too long to download the block chain and it
did not appear to operate smoothly. I thought I could do something to decrease
the block download time. So I downloaded the code and dug in.  Ultimately,
I failed to find the silver bullet to eliminate the long download delays
(big suprise!). But I did manage to penetrate the C++ code and figure
out how things worked for the most part.

So, I decided to write down my understanding of the code from an operational
perspective, to spare those who are not fluent in C++ from having to wade
through the code, which is quite dense and bit of a chore to pick apart,
when they really just want to know "how it works".

My focus was initially on the block download process, but I decided to
go ahead and cover all the major operational aspects I could (before losing
interest Wink. I do think I found some areas for improvement, but that is not
the point of these articles. I will try to make it clear when I am stating
the facts versus when I am writing commentary.

I intend these articles to go into the Wiki at some point but I also
thought it would be useful to open topics in the forum in order to
allow for review in case I made a mistake or missed something big,
and for reference.


Overview
------------

This series of articles will focus on how the Satoshi bitcoin client
program operates, and less so on the protocol details and the rules
for processing blocks and transactions.

Satoshi's bitcoin client is a C++ program, so be sure to look for code in
both the .cpp and the .h header files. Also, the program is multithreaded.
This leads to some complexity and the use of certain code patterns to deal
with concurrency that may be unfamiliar to many programmers. Also, the
code is aggresive in the use of C++ constructs, so it will help to be
fluent with map, multmap, set, string, vector, iostream, and templates.


For information on how the bitcoin protocol works, see:
    The original Satoshi whitepaper:
        http://bitcoin.org/bitcoin.pdf
    The articles on the bitcoin.it Wiki:
        https://en.bitcoin.it/wiki/Category:Technical
    With special mention of the protocol specification:
       https://en.bitcoin.it/wiki/Protocol_specification
    And the protocol rules:
       https://en.bitcoin.it/wiki/Protocol_rules


-- Operations --

The client is oriented around several major operations, including:

    Initialization and Startup
        Upon startup, the client performs various initilization routines
        including starting multiple threads to handle concurrent operations.

    Node Discovery
        The client uses various techniques find out about other bitcoin
        nodes that may exist.

    Node Connectivity
        The client initiates and maintains connections to other nodes.

    Sockets and Messages
        The client processes messages from other nodes and sends
        messages to other nodes using socket connections.
    
    Block Exchange
        Nodes advertise their inventory of blocks to each other and
        exchange blocks to build block chains.

    Transaction Exchange
        Nodes exchange and relay transactions with each other.
        The client associates transactions with bitcoin addresses in the
        local wallet.

    Wallet Services
        The client can create transactions using the local wallet.
        The client associates transactions with bitcoin addresses in the
        local wallet. The client provides a service for managing
        the local wallet.

    RPC Interface
        The client offers an JSON-RPC interface over HTTP over sockets
        to perform various operational functions and to manage the local
        wallet.

    User Interface
        The user interface code is scheduled to be superseded by bitcoin-qt.
        Therefore, it is not covered in further detail.

See their individual articles for more detail on each of these operations.


-- fClient Mode --

It is worth noting that there is code in the client to allow it to
operate in a mode where it only downloads block headers.
The implementation is intended to be used as a lightweight client mode which
can operate without verifying and storing all blocks and transactions.

This is controlled by the fClient variable in the code which is currently
hard coded to false.  This is currently not considered to be finished code.

This mode is known as fClient mode and the phrase Simplified Payment
Verification (or SPV) mode has also been used to describe a lightweight
client approach.



-- Main Thread Level Functions --

init.cpp:
    main()
    ExitTimeout
    Shutdown
net.cpp:
    StartNode
    ThreadGetMyExternalIP
    ThreadMapPort
    ThreadSocketHandler
    ThreadOpenConnections
    ThreadMessageHandler
rpc.cpp:
    ThreadRPCServer
irc.cpp:
    ThreadIRCSeed
db.cpp:
    ThreadFlushWalletDB
ui.cpp:
    ThreadDelayedRepaint
    SendingDialogStartTransfer


-- Significant Classes By File --

net.cpp/.h:
    CNode:  handes one socket connection
    CInv
    CAddress
    CMessageHeader
    CRequestTracker

main.cpp/.h:
    CDiskTxPos
    CInPoint
    COutPoint
    CTxIn
    CTxOut

    CTransaction
    CMerkleTx
    CTxIndex

    CBlock
    CBlockIndex
    CDiskBlockIndex
    CBlockLocator

    CAlert : CUnsignedAlert

wallet.cpp/.h
    CWallet : CKeyStore
    CReserveKey
    CWalletTx : CMerkleTx
    CWalletKey
    CAccount
    CAccountingEntry

db.cpp/.h:
    CTxDB
    CKeyPool
    CWalletDB

bignum.h
    CBigNum

util.h
    CCriticalSection: used for thread contention


--
Search on "Satoshi Client Operation" for more articles in this series.

Transaction Exchange : https://bitcointalk.org/index.php?topic=41730.0
Block Exchange : https://bitcointalk.org/index.php?topic=41729.0
Sockets and Messages : https://bitcointalk.org/index.php?topic=41727.0
Node Connectivity : https://bitcointalk.org/index.php?topic=41726.0
Node Discovery : https://bitcointalk.org/index.php?topic=41722.0
Initialization and Thread Startup : https://bitcointalk.org/index.php?topic=41719.0
shads
Sr. Member
****
Offline Offline

Activity: 266
Merit: 254


View Profile
September 06, 2011, 08:04:35 AM
 #2

I'm only part way through 2 of yr articles so far but I just wanna say thanks for an awesome job.  This sort of documentation is exactly what new devs need to get across the client quickly.

Are you planning to publish this series on a site somewhere?  It would be great to have it all in one place where people can bookmark it and find it easily for reference.  It's going to get buried eventually in the forum.

PoolServerJ Home Page - High performance java mining pool engine

Quote from: Matthew N. Wright
Stop wasting the internet.
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1072
Merit: 1181


View Profile WWW
September 06, 2011, 08:18:40 AM
Merited by ABCbits (1)
 #3

Thank you very much for this. I haven't had the time to go over everything in detail, but from what I've seen, it is correct and well researched. Please make sure this ends up on the wiki - people wanting to contribute need this kind of information.

I do Bitcoin stuff.
Alex Zee
Member
**
Offline Offline

Activity: 112
Merit: 10



View Profile WWW
September 06, 2011, 09:12:14 AM
 #4

I didn't yet read all of this, but it seems like a terrific job! Thanks for doing this!

I suggest putting it somewhere, like a wiki, and adding some images and formatting for better presentation.

BTC Monitor - systray price ticker
RipTalk.org - new Ripple forum
Nubarius
Sr. Member
****
Offline Offline

Activity: 310
Merit: 253


View Profile
September 06, 2011, 10:50:09 AM
 #5

I haven't read all of this yet either, but it looks like an amazing piece of badly needed documentation. Thank you so much.
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428
Merit: 1093


Core Armory Developer


View Profile WWW
September 06, 2011, 09:02:52 PM
Merited by ABCbits (1)
 #6

Bitrick, this is very good information.  It is exactly the kind of information I wanted, as someone who wants to get involved in BTC development but doesn't have the advanced C++ software skills needed to comprehend the Satoshi client code.  I look forward to using this as a secondary reference in my work, and helping iron out the details.

I would recommend you put links to each of the other posts at the bottom of this "Overview" post, and then one of the admins make this thread sticky.  I think people would expect to see this kind of information when they come to the "Development & Technical Discussion" forum,  and right now this is the most concise yet seemingly-complete version of this information I've seen so far.

Nice work!
-Eto

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
jackjack
Legendary
*
Offline Offline

Activity: 1176
Merit: 1280


May Bitcoin be touched by his Noodly Appendage


View Profile
September 06, 2011, 10:19:14 PM
 #7

I don't have the time to read or use these info but they look really awesome -> wiki!

Own address: 19QkqAza7BHFTuoz9N8UQkryP4E9jHo4N3 - Pywallet support: 1AQDfx22pKGgXnUZFL1e4UKos3QqvRzNh5 - Bitcointalk++ script support: 1Pxeccscj1ygseTdSV1qUqQCanp2B2NMM2
Pywallet: instructions. Encrypted wallet support, export/import keys/addresses, backup wallets, export/import CSV data from/into wallet, merge wallets, delete/import addresses and transactions, recover altcoins sent to bitcoin addresses, sign/verify messages and files with Bitcoin addresses, recover deleted wallets, etc.
dinker
Member
**
Offline Offline

Activity: 103
Merit: 10



View Profile
September 06, 2011, 10:52:00 PM
 #8

Excellent stuff!

Are you satoshi?  Roll Eyes

Help Me Help You Donations:
14kP6tNtrz3woESs9nEE5aDB81QTybGyyZ
Gavin Andresen
Legendary
*
qt
Offline Offline

Activity: 1652
Merit: 2301


Chief Scientist


View Profile WWW
September 06, 2011, 11:25:47 PM
Merited by ABCbits (1)
 #9

Very nice bitrick!

By the way, this patch speeds up initial download quite a lot:
  https://github.com/gavinandresen/bitcoin-git/commit/042a619709fab1329e8286c6aedbb2cdc8eb3497

...as do these, which have already been pulled into git head:
  https://github.com/bitcoin/bitcoin/commit/fb45259967032d409bca4d542b55414a7c522fba
  https://github.com/bitcoin/bitcoin/commit/ec74e8a44338202bfb82faa2cef4611cc37e7fa5

How often do you get the chance to work on a potentially world-changing project?
bitrick (OP)
Member
**
Offline Offline

Activity: 64
Merit: 140


View Profile
September 07, 2011, 12:04:39 AM
 #10


Thanks Gavin!
The first patch is the most promising. I actually tested turning off VerifySignature last week and calculated around double the block processing rate, when CPU limited. I was hoping for more, but every bit helps.

The last two patches I have less hope for. I know 0.3.23 can disconnect inappropriately, but in my testing any such disconnects should only result in a "brief" stall (brief == a few minutes of a multi-hour process). Note that the client should not disconnect a connection until all incoming and outgoing buffers are drained, so even if there is a disconnect, all the blocks queued prior to that can still get processed, so there should not be totally unproductive ongoing disconnections. (Or did I misunderstand the nature of the 0.3.23 problem?) I did not see a lot of 0.3.23 disconnect delays in my testing but maybe I just got lucky.
 
Thanks everyone for the nice comments. To reiterate: I do plan on putting this information into the wiki for future reference (and thanks for the suggestion Eto).

p.s. I am not Satoshi.
netrin
Sr. Member
****
Offline Offline

Activity: 322
Merit: 251


FirstBits: 168Bc


View Profile
October 15, 2011, 03:26:49 PM
 #11

Thanks Bitrick.

Greenlandic tupilak. Hand carved, traditional cursed bone figures. Sorry, polar bear, walrus and human remains not available for export.
d.james
Sr. Member
****
Offline Offline

Activity: 280
Merit: 250

Firstbits: 12pqwk


View Profile
October 17, 2011, 05:19:43 PM
 #12

these are quiet educational, but it will take me days to digest as a newbie programmer.

You can not roll a BitCoin, but you can rollback some. Cheesy
Roll me back: 1NxMkvbYn8o7kKCWPsnWR4FDvH7L9TJqGG
GideonGono
Hero Member
*****
Offline Offline

Activity: 2156
Merit: 501


View Profile WWW
November 03, 2011, 02:15:54 PM
 #13

Great job! Grin
RodeoX
Legendary
*
Offline Offline

Activity: 3066
Merit: 1147


The revolution will be monetized!


View Profile
November 03, 2011, 03:29:11 PM
 #14

Thanks bitrick! I have not seen this information elsewhere.

The gospel according to Satoshi - https://bitcoin.org/bitcoin.pdf
Free bitcoin in ? - Stay tuned for this years Bitcoin hunt!
finway
Hero Member
*****
Offline Offline

Activity: 714
Merit: 500


View Profile
November 04, 2011, 08:03:24 AM
Last edit: November 04, 2011, 08:21:43 AM by finway
 #15

Subscribe.

Instresting, abandon downloading blockchains from .323 client speed the downloading up.

quartz92
Member
**
Offline Offline

Activity: 72
Merit: 10


View Profile
November 08, 2011, 02:10:51 AM
 #16

nice job!
btc_artist
Full Member
***
Offline Offline

Activity: 154
Merit: 102

Bitcoin!


View Profile WWW
November 29, 2011, 04:25:42 PM
 #17

Watch.

BTC: 1CDCLDBHbAzHyYUkk1wYHPYmrtDZNhk8zf
LTC: LMS7SqZJnqzxo76iDSEua33WCyYZdjaQoE
R-
Full Member
***
Offline Offline

Activity: 238
Merit: 100

Pasta


View Profile WWW
April 05, 2012, 10:32:52 AM
 #18

I find it impressive you were able to synthesize a summary from such a complex set of code.
malevolent
can into space
Legendary
*
Offline Offline

Activity: 3472
Merit: 1724



View Profile
May 16, 2012, 09:14:58 PM
 #19


This is some interesting info for me, post your address so I can send a few bitcents ;-)

Signature space available for rent.
Xenland
Legendary
*
Offline Offline

Activity: 980
Merit: 1003


I'm not just any shaman, I'm a Sha256man


View Profile
August 14, 2012, 02:38:23 AM
 #20

This reminds me of my Pseudo Bitcoin Client, were the idea is to develop a bitcoin client from scratch in each language to better help others what is happening in the Bitcoin client in their native (programming) lanauge and as well as documentation in Pseudo Code to allow others that only know English(but have a solid understanding of hashing functions) to understand fundamentally whats going on in the official Bitcoin client.

So far I just have Generating Bitcoin addresses in java code.
I would love to expand on it with different language however I don't know how the Bitcoin client works well enough to do it my self, anyways here is the link https://github.com/Xenland/Bitcoin-Pseudocode-Client/tree/gh-pages
Pages: [1] 2 3 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!