cbitcoin - Bitcoin implementation in C. Currently in development.

[jim618]

I just thought I would mention that bitcoinj has moved it's wallet file format over to protobuf.
There is a pure C protobuf project here:

http://code.google.com/p/protobuf-c/

I am just adding support for the new format into MultiBit and the wallet files are much smaller and so load in double quick time.

I am not sure where you have got to yet in your work on wallet storing / reading but wanted to mention it. Potentially cbitcoin and bitcoinj could have byte-compatible wallets.

There is a proto file representing all the messages already all done with the key, Bitcoin address etc structure - you could directly produce your cbitcoin wallet store/ load code from that.

Edit: The proto file is here:
http://code.google.com/p/bitcoinj/source/browse/core/src/bitcoin.proto?name=release-0.5

[MatthewLM]

For cbitcoin I do not plan to implement wallet functionality. The reason is because there are many ways to store keys and so it makes sense to add that functionality onto cbitcoin if making a client with it. bitcoinj is a bit different in that it's a library more specifically for clients using SPV. cbitcoin will be a bit more low level, providing powerful access to bitcoin features. I want cbitcoin to be more diverse.

I plan to make a client that will work on-top of cbitcoin which may include a higher level library or some sort of API perhaps similar to bitcoind. I'll get to that eventually and may use protocol buffers, I don't know yet.

[MatthewLM]

I moved the topic since the library isn't technically an alternative bitcoin client.

I'm currently doing some tests on transactions signing and OP_CHECKSIG/OP_CHECKMULTISIG. I've done over 250 tests for the script interpreter and that works nicely.

[MatthewLM]

I'm currently implementing the message structures with serialisation code for each structure. After that I'll implement code used to manage network connections, to pass on de-serialised messages from nodes and to send messages to nodes. Then I'll work on the blockchain validation code.

I want this library to be low level so that the library wont implement the network responses itself and relay addresses, relay blocks, download and verify the block chain (There will be validation code but it will need configuring to the requirements of the programmer) etc. Instead the library will give the interface for programmers to build a node to their specifications.

But don't worry if you think a client will not come from this. I plan to write a client library on top of cbitcoin, and GUI applications will come too. Do not get over excited since a stable/secure client is far off as there will need to be extensive testing on top of the development. Also remember this library should become suitable for various bitcoin projects and not only basic clients.

Once again, if anyone wishes to contribute to this project it would be very helpful (and easy) for people to contribute with the testing. At the moment there are two obvious things which need more testing. The "testCBByteArray" file needs tests for all the functions. The test files for the structures that inherit CBMessage need tests for fail cases where the de-serialisation and serialisation is supposed to fail (And give an appropriate error message). More script tests or any tests elsewhere are great.

I've given up developing iPhone apps to put a lot of my time towards this, so hopefully it will come along reasonably quickly.

There will be inevitable costs to this project. If anyone would like to help this project financially, donations are very welcome: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9

Thank you.

[Haplo]

I'll send a BTC your way in a day or so. I'd have probably donated more by now but I really had no idea how many retarded hoops I was going to have to jump through to buy BTC 

[MatthewLM]

Thank you very much. Sorry buying bitcoin isn't as easy as it would ideally be. 

[MatthewLM]

I've implemented all the message objects except the IP transaction ones which I wont implement. The message objects need testing for the cases in which they are supposed to fail and need to be scrutinised.

However, for now, I'm going to work on the code for the network. This will be the code that can send messages back and forth to and from the network and also some code that automates some common processes such as relaying network addresses and finding new nodes etc. I'm thinking I'll make a structure called CBNetworkCommunicator that works on a events based system:

eg. With "void (* onBlockRecieved)(CBNetworkCommunicator *,CBNetworkAddress *,CBBlock *)" a block is de-serialised from the network and sent to the function pointed to by onBlockRecieved with the CBNetworkCommunicator object and CBNetworkAddress where the block was sent from.

I will need to add networking and threading functions to CBDependencies.h for connecting to nodes and managing connections in separate threads.

After this I'll implement the block validation. I'll add functions to CBBlock.h/.c for this and use an CBEvents object so the validation code can ask other functions for block/transaction information (eg. "CBTransactionOutput * (* getPrevOutput)(CBByteArray * transactionHash,u_int32_t outputIndex)"). The programmer using the library will be responsible for the storage of blocks and retrieving the information for the validator. The programmer will also be responsible for the organisation of the block chain, cbitcoin will only validate single blocks at a time. cbitcoin should have code for the checking of block difficulty, timestamps etc. but the programmer using cbitcoin may have to pass some of this information around. This is because of the configurable/modular nature of cbitcoin (Remember that cbitcoin is supposed to be relatively low-level).

Shortly after that I should be able to put cbitcoin into the alpha development stage and I will move onto the higher level client library and applications for that (More info later). Developing a client library on top of cbitcoin may highlight issues, so I will begin proper testing after developing the client library. Both libraries and the client applications can be tested together and will eventually enter the beta testing stage. This stage has a focus on public testing but people will be free to help test the code during the alpha stage too. Eventually the code will be considered secure and stable for final release.

I've been looking a little into independent software reviews. It seems most software is tested internally and there is little interest in external auditing of software. I have the feeling that's because it would be very expensive. Does anyone know much about external auditing of software?

[MatthewLM]

I've figured out how cbitcoin will handle sockets. cbitcoin will have dependencies for events-based, non-blocking sockets. It should work with libevent though I need to learn more about how to use lubevent to make sure my library is compatible. My threading dependencies should work with POSIX threads nicely.

Here are the threading and socket dependencies as I have them so far (These are weakly linked function prototypes):

Code:

// THREADING DEPENDENCIES

/*
 @brief Starts a new thread. The thread should be joinable. 
 @param threadID The ID for the thread to be set with newly allocated memory, used to refer to the threads in other functions.
 @param startFunction The function to be called as the starting point of the new thread.
 @param args A pointer to the arguments for the starting function.
 @returns true if the thread was successfully created and false if the thread could not be created.
 */
bool CBStartThread(void * threadID,void * (*startFunction)(void *),void * args);
/*
 @brief Ends the calling thread.
 */
void CBEndThread(void);
/*
 @brief Waits for a thread to terminate.
 @param threadID The ID for the thread.
 */
void CBThreadJoin(void * threadID);
/*
 @brief Frees any resources related to a thread, including the ID.
 @param threadID The ID for the thread.
 */
void CBFreeThread(void * threadID);
/*
 @brief Creates a new mutex.
 @param mutexID The mutex id to be set with newly allocated new memory.
 @returns true if the mutex was successfully created and false if the mutex could not be created.
 */
bool CBNewMutex(void * mutexID);
/*
 @brief Frees a new mutex including the ID. It is assured that no thread locks are active before this is called as threads should be canceled.
 @param mutexID The mutex id to free.
 */
void CBFreeMutex(void * mutexID);
/*
 @brief Obtains a mutex lock when next available.
 @param mutexID The mutex id.
 */
void CBMutexLock(void * mutexID);
/*
 @brief Unlocks a mutex for other threads.
 @param mutexID The mutex id.
 */
void CBMutexUnlock(void * mutexID);

// NETWORKING DEPENDENCIES

/*
 @brief Creates a new TCP/IPv6 socket and binds it to the given IPv6 address and port. The socket should use a non-blocking mode.
 @param socketID The socket id to be set with newly allocated new memory.
 @returns true if the socket was successfully created and false if the socket could not be created.
 */
bool CBNewSocket(void * socketID);
/*
 @brief Binds the local address and a port number to a socket.
 @param socketID The socket id
 @param port The port to bind to.
 @returns true if the bind was sucessful and false otherwise.
 */
bool CBSocketBind(void * socketID,u_int16_t port);
/*
 @brief Begin connecting to an external host with a socket. This should be non-blocking.
 @param socketID The socket id
 @param IP 16 bytes for an IPv6 address to connect to.
 @param port Port to connect to.
 @param onConnect Pointer to the function that should be called when the connection has been sucessful.
 @param onConnectArg Pointer to send to "onConnect".
 @returns true if the function was sucessful and false otherwise.
 */
bool CBSocketConnect(void * socketID,u_int8_t * IP,u_int16_t port,void (*onConnect)(void *),void * onConnectArg);
/*
 @brief Begin listening for incomming connections on a bound socket. This should be non-blocking. 
 @param socketID The socket id
 @param maxConnections The maximum incomming connections to allow.
 @param onAccept Pointer to the function that should be called when the socket is ready for accepting an incomming connection.
 @param onAcceptArg Pointer to send to "onAccept".
 @returns true if function was sucessful and false otherwise.
 */
bool CBSocketListen(void * socketID,u_int16_t maxConnections,void (*onAccept)(void *),void * onAcceptArg);
/*
 @brief Accepts an incomming connections on a bound socket. This should be non-blocking.
 @param socketID The socket id
 @param connectionSocketID A socket id for a new socket for the connection.
 @param IP 16 bytes to be set by the function for the IP of the incomming connection.
 @returns true if function was sucessful and false otherwise.
 */
bool CBSocketAccept(void * socketID,void * connectionSocketID,u_int8_t * IP);
/*
 @brief Sets a function pointer for the event where a socket is available for sending data.
 @param socketID The socket id
 @param onCanSend The function to call for the event.
 @param onCanSendArg A pointer to pass to the "onCanWrite" function.
 @returns true if function was sucessful and false otherwise.
 */
bool CBSocketCanSendEvent(void * socketID,void (*onCanSend)(void *),void * onCanSendArg);
/*
 @brief Sets a function pointer for the event where a socket is available for receiving data.
 @param socketID The socket id
 @param onCanReceive The function to call for the event.
 @param onCanWriteArg A pointer to pass to the "onCanWrite" function.
 @returns true if function was sucessful and false otherwise.
 */
bool CBSocketCanReceiveEvent(void * socketID,void (*onCanReceive)(void *),void * onCanReceiveArg);
/*
 @brief Sends data to a socket. This should be non-blocking.
 @param socketID The socket id to send to.
 @param data The data bytes to send.
 @param len The length of the data to send.
 @returns The number of bytes actually sent and any number less than 0 on failure that suggests further data cannot be sent.
 */
int32_t CBSocketSend(void * socketID,u_int8_t * data,u_int32_t len);
/*
 @brief Receives data from a socket. This should be non-blocking.
 @param socketID The socket id to receive data from.
 @param data The data bytes to write the data to.
 @param len The length of the data.
 @returns The number of bytes actually written into "data", 0 on connection closure and any number less than 0 on failure.
 */
int32_t CBSocketReceive(void * socketID,u_int8_t * data,u_int32_t len);
/*
 @brief Closes a socket. The id should be freed, as well as any other data relating to this socket.
 @param socketID The socket id to be closed.
 */
void CBCloseSocket(void * socketID);

[MatthewLM]

Sockets are driving me mad. Different platforms use completely different ways to represent IPv6 addresses. This includes unix platforms which do not use fully compatible BSD sockets.

So if anyone knows a portable way to connect to IPv6 addresses please look here: http://stackoverflow.com/questions/11511339/portable-ipv6-connections-with-bsd-posix-sockets

Edit: I just sent 35.95515 bitcoins to an address made with cbitcoin. The first practical use of cbitcoin!

[jgarzik]

Well, the portable representation of an IPv6 address is the network representation.

You may need some platform-specific code to store it in the right structure and struct member.

bitcoin/bitcoin.git (satoshi client) works on linux/mac/win, and supports non-blocking IPv6 sockets.  The system calls and addressing are the same in C and C++.

[MatthewLM]

I looked briefly at the bitcoin socket code. It didn't look very nice from what I saw and it was quite disorganised. It looked like the sockets were blocking as I saw a section which looped through reading received data.

Anyway I think I found the solution to setting IPv6 addresses:

Code:

bool CBSocketConnect(u_int64_t socketID,u_int8_t * IP,u_int16_t port){
	// Create sockaddr_in6 information for a IPv6 address
	struct sockaddr_in6 address;
	memset(&address, 0, sizeof(address)); // Clear sockaddr_in6 structure.
	address.sin6_family = AF_INET6; // IPv6
	memmove(&address.sin6_addr, IP, 16); // Move IP address into place. POSIX standards should allow this.
	address.sin6_port = htons(port); // Port number to network order
	if (connect((evutil_socket_t)socketID, (struct sockaddr *)&address, sizeof(address)) != 0)
		// Connection failure
		return false;
	return true;
}

Apparently that should work on POSIX compliant platforms. It is not tested yet and may fail on windows.

The reason the socketID is a u_int64_t is to support any integer that it may be for portability. In this implementation it is casting to evutil_socket_t which is a file descriptor on unix/linux and a pointer on windows.

[Haplo]

sent donatey :X

[MatthewLM]

sent donatey :X

Thank you very much. Donations currently are sure to go to the outsourcing of graphic design for the GUI clients and possibly outsourcing some programming jobs, though the more voluntary help I can find, the cheaper it will all be. At the moment though I'm the only person working on this.

I'm currently reasonably close to implementing the networking code. Once that is done I'll implement the block validation code and then cbitcoin can reach the alpha testing stage.

I was thinking about making the address addition code synchronous with the network event loop, so that addresses can be added outside the network thread. I think I'll bin that idea though, so you will need to stop the network loop before adding additional addresses. I see little reason why anyone would want to do this. You simply give the CBNetworkCommunicator the addresses before you start it and when closing the application, the CBNetworkCommunicator is stopped and the addresses can be taken out and saved to disk.

[MatthewLM]

I'm currently testing the networking code. I'm reasonably near alpha-stage as I've just got to do the block-chain validation code after this.

But I've got an issue. Is anyone good with sockets? connect() is giving me ENOENT:

http://stackoverflow.com/questions/11916719/berkeley-sockets-connect-returns-1-with-errno-set-to-enoent

I'm not sure what that suggests is wrong. I don't think most implementations even give ENOENT errors.

Thank you for any help.

[kokjo]

i don't know about OSX. but linux's man page for connect(2). does not say that connect can return ENOENT.

what kind of socket do you make, ipv6 or ipv4? not for the connect() call, but for the socket() call.

[MatthewLM]

I've confirmed what the error is: LLDB is rubbish, never use it. LLDB does not work with errno properly. Use GDB.

Yes, errno was set as EINPROGRESS all along. It was the stupid debugger that caused all the confusion.

[MatthewLM]

Hmm. Now I'm getting an unexpected connection closure via 0 returned from read(). I printed out some of the data to try and understand what was happening:

Connection initiated by A
Connection accepted by B
Connection OK
A sent version header
A sent version payload
B received version header
B received version payload
B sent verack
B sent version header
B send version payload
A received verack
Connection closed

This is pretty odd, I don't know where to start looking... The thing is, neither A or B close the connection explicitly, they both receive 0 from read().

[kokjo]

Hmm. Now I'm getting an unexpected connection closure via 0 returned from read(). I printed out some of the data to try and understand what was happening:

Connection initiated by A
Connection accepted by B
Connection OK
A sent version header
A sent version payload
B received version header
B received version payload
B sent verack
B sent version header
B send version payload
A received verack
Connection closed

This is pretty odd, I don't know where to start looking... The thing is, neither A or B close the connection explicitly, they both receive 0 from read().

A does not receive the version from B?
my guess: its A that closes the connection.

can you post the version packets+headers?

EDIT: what clients are you testing with? both cbitcoin, or is one of them the satoshi client? which one then?

[MatthewLM]

Thanks for the suggestions. I don't really know what caused it but I did see an issue with attempts to allocate zero memory. I fixed all that by using NULL to represent nothing as I was intending to. After fixing all that and some other issues the problem went away.

Now the handshake works and I've committed an update to github.

EDIT: what clients are you testing with? both cbitcoin, or is one of them the satoshi client? which one then?

At the moment just cbitcoin but I will try connecting to the satoshi client as well.

Edit: Pings work fine. Moving onto address discovery.

[kokjo]

Thanks for the suggestions. I don't really know what caused it but I did see an issue with attempts to allocate zero memory. I fixed all that by using NULL to represent nothing as I was intending to. After fixing all that and some other issues the problem went away.

Now the handshake works and I've committed an update to github.

EDIT: what clients are you testing with? both cbitcoin, or is one of them the satoshi client? which one then?

At the moment just cbitcoin but I will try connecting to the satoshi client as well.

Edit: Pings work fine. Moving onto address discovery.

Nice! im only testing against the satoshi client for now. im also making an alternative client in python.
so far i can download and verify blocks and transactions(mostly... have implemented a hacky replacement for the scripting, just ripping out the sig and publickey and checks if it verifyes...). i have no working wallet for now, but i can send hand crafted transactions.