cbitcoin - Bitcoin implementation in C. Currently in development.

[MatthewLM]

Good luck with that python project. I take it that the client will use the SPV model?

cbitcoin wont be a bitcoin client but will provide the low-level interface for creating a client. After cbitcoin reaches alpha stage I will start working on the client on top of cbitcoin which is currently code-named BitEagle.

And I'm close to alpha stage. I've just got to sort out the address discovery and do the block validation code. cbitcoin wont relay/download transactions or blocks. cbitcoin only has a low level interface to the bitcoin network and also provides handshakes, pings and address discovery. You can choose to turn these features on or off using flags, so for instane you may not want address discovery if you plan to connect to specific nodes only and you may wish to implement your own behaviour.

[jgarzik]

Nice! im only testing against the satoshi client for now. im also making an alternative client in python.
so far i can download and verify blocks and transactions(mostly... have implemented a hacky replacement for the scripting, just ripping out the sig and publickey and checks if it verifyes...). i have no working wallet for now, but i can send hand crafted transactions.

If you are using python, please consider using the 'bitcoin' module found in https://github.com/jgarzik/pynode

It already does full script and block verification.

The only major to-do is chain reorg, and that does not impact the 'bitcoin' module at all.

[kokjo]

Good luck with that python project. I take it that the client will use the SPV model?

no, it will be a full blown client. but it will be very modular so a SPV would be very simple to make with the codebase

cbitcoin wont be a bitcoin client but will provide the low-level interface for creating a client. After cbitcoin reaches alpha stage I will start working on the client on top of cbitcoin which is currently code-named BitEagle.

awesome!

And I'm close to alpha stage. I've just got to sort out the address discovery and do the block validation code. cbitcoin wont relay/download transactions or blocks. cbitcoin only has a low level interface to the bitcoin network and also provides handshakes, pings and address discovery. You can choose to turn these features on or off using flags, so for instane you may not want address discovery if you plan to connect to specific nodes only and you may wish to implement your own behaviour.

i python i have created a seperat class, that controls block down/up-loading behavior, and another class that keeps track of the addresses.
...and i have not implemented ping yet, because im lazy and its not really an important feature(but connections closes sometimes...)

happy coding!

[kokjo]

Nice! im only testing against the satoshi client for now. im also making an alternative client in python.
so far i can download and verify blocks and transactions(mostly... have implemented a hacky replacement for the scripting, just ripping out the sig and publickey and checks if it verifyes...). i have no working wallet for now, but i can send hand crafted transactions.

If you are using python, please consider using the 'bitcoin' module found in https://github.com/jgarzik/pynode

It already does full script and block verification.

The only major to-do is chain reorg, and that does not impact the 'bitcoin' module at all.

+1 will look into it.

[MatthewLM]

When you get to validating scripts fully it would be a good idea to test the scripts with the scripts the original client uses. I've made a file with many tests written in my script format, with a line for the script followed by 1 if the script should be valid and 0 is it should not. This includes tests from the C++ client: https://github.com/MatthewLM/cbitcoin/blob/master/test/scriptCases.txt

For the scripts I use this BNF:

Code:

<digit> = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" | "a" | "b" | "c" | "d" | "e" | "f" | "A" | "B" | "C" | "D" | "E" | "F"
 <hex_digits> ::= <digit> | <digit> <hex_digits>
 <hex> ::= "0x" <hex_digits>
 <operation_name> = "0" | "FALSE" | "1NEGATE" | "RESERVED" | "1" | "TRUE" | "2" | "3" | "4" |  "5" | "6" | "7" | "8" | "9" | "10" | "11" | "12" | "13" | "14" | "15" | "16" | "NOP" | "VER" | "IF" | "NOTIF" | "VERIF" | "VERNOTIF" | "ELSE" | "ENDIF" | "VERIFY" | "RETURN" | "TOALTSTACK" | "FROMALTSTACK" | "2DROP" | "2DUP" | "3DUP" | "2OVER" | "2ROT" | "2SWAP" | "IFDUP" | "DEPTH" | "DROP" | "DUP" | "NIP" | "OVER" | "PICK" | "ROLL" | "ROT" | "SWAP" | "TUCK" | "CAT" | "SUBSTR" | "LEFT" | "RIGHT = 0x81" | "SIZE" | "INVERT" | "AND" | "OR" | "XOR" | "EQUAL" | "EQUALVERIFY" | "RESERVED1" | "RESERVED2" | "1ADD" | "1SUB" | "2MUL" | "2DIV" | "NEGATE" | "ABS" | "NOT" | "0NOTEQUAL" | "ADD" | "SUB" | "MUL" | "DIV" | "MOD" | "LSHIFT " | "RSHIFT" | "BOOLAND" | "BOOLOR" | "NUMEQUAL" | "NUMEQUALVERIFY" | "NUMNOTEQUAL" | "LESSTHAN" | "GREATERTHAN" | "LESSTHANOREQUAL" | "GREATERTHANOREQUAL" | "MIN" | "MAX" | "WITHIN" | "RIPEMD160" | "SHA1" | "SHA256" | "HASH160" | "HASH256" | "CODESEPARATOR" | "CHECKSIG" | "CHECKSIGVERIFY" | "CHECKMULTISIG" | "CHECKMULTISIGVERIFY" | "NOP1" | "NOP2" | "NOP3" | "NOP4" | "NOP5" | "NOP6" | "NOP7" | "NOP8" | "NOP9" | "NOP10"
 <operation> ::= "OP_" <operation_name>
 <lines> ::= "\n" | "\n" <lines>
 <seperator> ::= <lines> | " "
 <operation_or_hex> ::= <operation> | <hex>
 <seperated_operation_or_hex> ::= <operation_or_hex> <seperator>
 <script_body> ::= <seperated_operation_or_hex> | <seperated_operation_or_hex> <script_body> | ""
 <lines_or_none> ::= <lines> | ""
 <operation_or_hex_or_none> ::= <operation_or_hex> | ""
 <script> ::= <lines_or_none> <script_body> <operation_or_hex_or_none> 

I do now have an issue with EPIPE: http://stackoverflow.com/questions/11990700/broken-pipe-epipe-on-connection-to-loopback-address

[MatthewLM]

The EPIPE problem might be with libevent. I'll try libev (which is apparently better...). If anyone knows libev and would like to help, please get in touch.

[kokjo]

 signal(SIGPIPE, SIG_IGN) ?

[MatthewLM]

That is for ignoring SIGPIPE. I disable SIGPIPE via SO_NOSIGPIPE and MSG_NOSIGNAL.

The problem is the EPIPE that shouldn't be there. The connection over the loopback IP is not always working for some reason.

UPDATE: It seems the other side of the connection does not accept before the connection event is received..

Code:

CN CONNECTION WAIT
CN CONNECTED OK
CN CONNECTION WAIT
CN SENT version

When printing everything out it says the connection is OK but there was not accept. There should be at least a "L1 TRIED ACCEPT" in there.

UPDATE 2: Edge triggered events also fail.

UPDATE 3: Adding the events after trying the connection does nothing to help.

[MatthewLM]

So I made an attempted implementation of the networking code using libev. The problem is it doesn't work. If anyone can get it to work, please help. I've comited the changes. This is the new libev code: https://github.com/MatthewLM/cbitcoin/blob/master/dependencies/sockets/CBLibevSockets.c

When I run it, it seems to occasionally partially work any other times it will not work. For instance here:

Code:

CN CONNECTION WAIT
CN CONNECTION WAIT
L2 TRIED ACCEPT
L2 MADE ACCEPT
L2 HAS ACCEPTED OK
CN CONNECTED OK
CN CONNECTED OK

The connecting CBNetworkCommunicator waits for the connections, the second listening CBNetworkCommunicator accepts the connection but then the connecting CBNetworkCommunicator thinks both connections are fine when only one was accepted. Then the next time I run it:

Code:

CN CONNECTION WAIT
CN CONNECTION WAIT

So the connecting CBNetworkCommunicator waits for the connections but they do not get accepted.

Why is this so hard? It surely isn't meant to be this hard.

Edit: To allow time for any help to arrive (if ever) I'll skip this for now and move onto block validation and P2SH.

[MatthewLM]

I modified the libevent code to detect socket errors during the connect event.

Code:

void CBDidConnect(evutil_socket_t socketID,short eventNum,void * arg){
	CBEvent * event = arg;
	if (eventNum & EV_TIMEOUT) {
		// Timeout for the connection
		event->loop->onTimeOut(event->loop->communicator,event->node,CB_TIMEOUT_CONNECT);
	}else{
		int optval = -1;
		socklen_t optlen = sizeof(optval);
		getsockopt(socketID, SOL_SOCKET, SO_ERROR, &optval, &optlen);
		if (optval)
			// Act as timeout
			event->loop->onTimeOut(event->loop->communicator,event->node,CB_TIMEOUT_CONNECT);
		else
			// Connection successful
			event->onEvent.ptr(event->loop->communicator,event->node);
	}
}

Occasionally optval is given as 61 (ECONNREFUSED). Still begs the question, why is there a ECONNREFUSED for a loopback connection?

Edit: Fixed it, needed to use SO_REUSEADDR.

Problem now seems to be that the connecting CBNetworkCommunicator is not replying to getaddr requests.

Edit 2: Well the reason why is because the addresses are not broadcast. So you might expect the connecting CBNetworkCommunicator to relay the addresses when they are broadcast but this is only done for unsolicited addr messages and at the time the CBNetworkCommunicator has made the original getaddr request so it does not relay the addresses. As far as I can tell this is successfully following the behaviour of the original client. I'll simply set the addresses as being public so that they are always given in getaddr responses.

[MatthewLM]

Good news. Just now my networking code past the unit test for the first time. I'm fast approaching alpha stage now. I just need to implement the block validation code. I suspect that will be easier than the horrid networking code. Also I should finnish the libev implementation of the networking.

[MatthewLM]

So far I've discovered the way bitcoin uses targets is silly. For instance, what is the point in blocks containing the target? That's calculated by clients. Also the way you can't have a mantissa over 7FFFFF is stupid. The fact the original client uses a bignum to do calculations is stupid. So yes, it's silly.

Validating blocks is not as nice as I thought it was. I keep finding weird parts of the bitcoin protocol.

[jgarzik]

So far I've discovered the way bitcoin uses targets is silly. For instance, what is the point in blocks containing the target?

At a minimum, you do not need the entire blockchain history just to validate the block matches the target.

Also the way you can't have a mantissa over 7FFFFF is stupid.

By "stupid" ITYM avoiding pointless, redundant support that would exist only to enhance the artificial sense of completeness that some computer scientists need.

It's quite efficient, considering the limited storage (32 bits) into which a much larger number is stored.

[MatthewLM]

At a minimum, you do not need the entire blockchain history just to validate the block matches the target.

But you need the history to validate the proof of work regardless. Why not just check the proof of work when you are able to calculate the target all the way to the block. Is there any point in doing it earlier?

Here's my implementation of retargeting and proof of work validation: https://github.com/MatthewLM/cbitcoin/blob/master/src/CBValidationFunctions.c

[Pieter Wuille]

At a minimum, you do not need the entire blockchain history just to validate the block matches the target.

But you need the history to validate the proof of work regardless. Why not just check the proof of work when you are able to calculate the target all the way to the block. Is there any point in doing it earlier?

It's certainly faster to be able to do some checks before requiring the parent to exist. This allows you to validate the PoW of received orphan blocks, before bothering to request parent. Also, it allows doing some checks on a received block before needing a lock on the block tree data structure.

[MatthewLM]

Useless complaining over 4 bytes per block anyway. 

[MatthewLM]

I'm very close to alpha stage now but I'm getting horrible heap corruption. Goodness knows how long it will take to debug...

[kokjo]

Good luck with that.

(thats why i don't code in C, )

[MatthewLM]

Thanks... The power afforded by C does sometimes cause problems like this....

It's got even worse since the stack has screwed up. Like here, where the hell did main() go???

Code:

(gdb) bt
#0  0x0000000100266d4d in CBNewAddressManager (events=0x5fbff838) at /Users/matt/Programming/BitEagle_Projects/cbitcoin/src/structures/CBObject/CBMessage/CBAddressManager/CBAddressManager.c:40
#1  0x0000000100000e24 in start ()

Luckily for me, this type of awkward error is rare but when it happens it's horrible.

...Almost certain the root cause is some stack corruption, not heap corruption. Eventually I'll figure it out...

Edit: Figured out what I thought was the problem was actually completely fine... I must be too tired. I will come back to this tomorrow.

[jgarzik]

valgrind is your friend