[ANNOUNCE] picocoin and libccoin -- C-based bitcoin library and client

[jgarzik]

Source code URL:  https://github.com/jgarzik/picocoin/

I'd like to announce another bitcoin implementation, which is really two useful pieces in one:

     libccoin - a bitcoin library, written in C
     picocoin - A lightweight, C-based SPV bitcoin wallet client

libccoin supports all key network datastructures (block, transaction, etc.), script parsing and validation, transaction and block validation, a "headers-only" or full block database, and many other features essential to any bitcoin client.  libccoin passes all key encoding, script and transaction tests available in the Satoshi reference bitcoin client.

picocoin is much more under construction.  When complete, it will be a very low resource, command line / JSON-driven bitcoin wallet.  Advanced security features already implemented include required wallet encryption, fork-based process separation of P2P networking and wallet (and chroot/SELinux jailing coming soon), something that the reference Satoshi client does not even support.

Status:  Alpha quality, developer release.  Passes reference client base58/script/transaction tests, but is still a developer-only preview.

Feature list:

• Liberal Mit/X11 free software license
• A full-feature bitcoin support library.  The library will not be limited to "what picocoin needs."
• Supports all core data structures and network messages
• Full script implementation
• Passes hundreds of available reference client tests
• Supports multiple block chains: main or testnet3
• Very low resource usage (cpu, disk, and memory)
• Small codebase (both source code and compiled object)
• Fast.  Loads all block headers in under 2 seconds.
• Supports advanced thin-client features such as bloom filtering, an upcoming proposal that will reduce client bandwidth usage.
• Works on big endian machines, as well as little endian machines
• Multi-platform: Linux, Mac OSX, BSD are working.  Windows support in progress.
• Library works on Windows.  picocoin will work on Windows with some modifications, but be a bit less secure than other platforms due to lack of fork.
• Improved security:  fork-based process separation firewall between networking and wallet code -- your wallet is never directly exposed to the network.
• Follows the philosophy of "do, not hype."  This library is already far more secure and capable than other libraries hyped as the "future of bitcoin" by their authors.

Code contributions are welcome (see github URL above).
Comments are welcome.
Donations are welcome too (see signature).

[jgarzik]

Reserved for FAQ etc.

[marcus_of_augustus]

Nice work, thanks. This should be useful.

Edit: did you use anything from cbitcoin https://github.com/MatthewLM/cbitcoin ?

[jgarzik]

Edit: did you use anything from cbitcoin https://github.com/MatthewLM/cbitcoin ?

Definitely not.

[2112]

• Very low resource usage (cpu, disk, and memory)
• Small codebase (both source code and compiled object)
• Multi-platform: Linux supported currently.  Should work on OSX/BSD/Windows with minimal modifications.

Comments are welcome.

Jeff, I have one comment that has value only very early in the design of your architecture. Since you've made a choice of C as a implementation language I suggest that you add one more target to the list of supported platforms: standalone a.k.a. bare metal.

It will be nearly impossible to convert your code later unless you very early make a commitment to supporting the event-driven programming like using lwIP-style network interface in addition to socket-style networking.

The same architectural choice needs to be made early to accomodate various lightweight flash-storage access methods instead of Unix-style open/lseek/read/write/close. Same for fork/wait/pthreads versus any RTOS-style explicit synchronization primitives.

One possible advantage of such choices would be that instead of relying purely on donations you may be able to obtain funds via cooperation with future manufacturers of the future Bitcoin devices that will greatly benefit from the "very low resource usage".

[jgarzik]

Jeff, I have one comment that has value only very early in the design of your architecture. Since you've made a choice of C as a implementation language I suggest that you add one more target to the list of supported platforms: standalone a.k.a. bare metal.

Yes, embedded usage is another target.

Each module in libccoin is carefully designed to minimize internal dependencies.  The core data structures, address generation, script execution and transaction validation are wholly independent of any filesystem or network design.

These modules may be used on a non-POSIX flash filesystem, with zero network support, today.

picocoin (the client) requires certain network, filesystem and process features, but libccoin (the library) does not.  This is by design.

[jimbobway]

Do you think you could give a distinct (and cool) name for this library?  There are several now and it gets confusing.  For example:

libcoin - Gronager
libccoin - jgarzik
libbitcoin - genjix
cbitcoin - MathhewLM

How about GarzikBitcoinLib?

[dserrano5]

AES encryption is applied to the wallet.  Passphrase is specified via environment variable PICOCOIN_PASSPHRASE.

Is this secure? My understanding is that the environment isn't a safe place to store/transmit information.

[jgarzik]

AES encryption is applied to the wallet.  Passphrase is specified via environment variable PICOCOIN_PASSPHRASE.

Is this secure? My understanding is that the environment isn't a safe place to store/transmit information.

That's temporary.  It will input via fd like GPG soon.

[2112]

Yes, embedded usage is another target.

Each module in libccoin is carefully designed to minimize internal dependencies.  The core data structures, address generation, script execution and transaction validation are wholly independent of any filesystem or network design.

These modules may be used on a non-POSIX flash filesystem, with zero network support, today.

picocoin (the client) requires certain network, filesystem and process features, but libccoin (the library) does not.  This is by design.

But even the definitions in your "ccoin" include files are deeply incompatible with any standalone environment, even if it uses GCC toolchain. Major rework would be required even now, when there's just couple of hundred lines of code.

The big-endian compatibility is already super-confusing. Istead of using uint32_t in bu256_t use an "union le_int32" to at least flag the mixing of little endian uint32_t with guint32_t.

From my extensive commercial experience of porting from Linux/Posix/Intel-only code to anything standalone or lightweight OS will unfortunatly go through the "first deny then rewrite from almost scratch" stages.

If I may suggest two things:

1) from the start build also on Mac Leopard or Snow Leopard to verify the endian-neutrality with "-arch ppc" with Rosetta on Intel CPU. Edit: Never mind, I forgot that you have access to the Linux on POWER and z/Arch through your association with RedHat.
2) obtain any embedded development/simulation environment. If you have any Xilinx Spartan6 boards (with memory) laying around you could use them to test using MicroBlaze CPU with Xilinx EDK/SDK.

Thank you for yor time.

[jgarzik]

But even the definitions in your "ccoin" include files are deeply incompatible with any standalone environment, even if it uses GCC toolchain. Major rework would be required even now, when there's just couple of hundred lines of code.

This seems quite an exaggeration, with no supporting evidence of what needs a major rework.  I do embedded programming for the day job, so I don't see this.  The Linux kernel is as bare metal as it gets, and must work in all environments, all platforms.  

The main chore of embedded use is simply the dependencies, most notably OpenSSL, and to a lesser extent GLib.  Once you have that, or its replacement, embedded use is straightforward.

The big-endian compatibility is already super-confusing. Istead of using uint32_t in bu256_t use an "union le_int32" to at least flag the mixing of little endian uint32_t with guint32_t.

Using "union" decreases the performance and quality of code output for zero gain.  It is trivial enough to use sparse with a type specifier to guarantee endian purity, like the Linux kernel does, if that is desired ("__le32" etc.)

1) from the start build also on Mac Leopard or Snow Leopard to verify the endian-neutrality with "-arch ppc" with Rosetta on Intel CPU. Edit: Never mind, I forgot that you have access to the Linux on POWER and z/Arch through your association with RedHat.

The code has long since been verified to work on big endian.

[Bitcoin Oz]

Do you think you could give a distinct (and cool) name for this library?  There are several now and it gets confusing.  For example:

libcoin - Gronager
libccoin - jgarzik
libbitcoin - genjix
cbitcoin - MathhewLM

How about GarzikBitcoinLib?

Just call it gbitcoin 

[BkkCoins]

So am I understanding right? I could run picocoin and it would be lightweight like Electrum but also support the Bitcoin network by interacting on the network like a full satoshi client. I've been using Electrum mostly to get away from having to use around 5GB disk space and for quick install/startup but I'd really prefer to be supporting Bitcoin more by being another validating node in it's network.

[2112]

This seems quite an exaggeration, with no supporting evidence of what needs a major rework.  I do embedded programming for the day job, so I don't see this.  The Linux kernel is as bare metal as it gets, and must work in all environments, all platforms.  

For starters, you'll have heck of a time to remove your dependency on glib.h from GTK. But there's more. You could actually try to compile your code from within the Linux kernel tree to see for yourself.

For a die-hard Linux developer like you I think the minimum definition of bare-metal would be uClinux. Full Linux doesn't count anymore for a true embedded development.

The main chore of embedded use is simply the dependencies, most notably OpenSSL, and to a lesser extent GLib.  Once you have that, or its replacement, embedded use is straightforward.

Both GLib and OpenSSL are heavy duty harvester combines. Removing the dependencies on them will at least halve the resource usage.

For a RedHat-ter at least make the code work with newlib instead of glib. The required knowledge should be no more than the internal phone call away for you. This includes a whole sermon on why you don't want to build a dependency on them.

Using "union" decreases the performance and quality of code output for zero gain.  It is trivial enough to use sparse with a type specifier to guarantee endian purity, like the Linux kernel does, if that is desired ("__le32" etc.)

The code has long since been verified to work on big endian.

As I've noted: I've forgot that you are one of the very few Bitcoin developers with regular access to big-endian hardware. I'm thinking that you will continue to be the only one developer who can test picocoin on the big-endian architecture. The mojibake-endian problem will continue to badly affect the future contributors/users of your library if you aren't going to use your own medicine (that is "sparse" and "__le32")

The performance loss for passing unions around may have been true for GCC 2.96. I'm not normally working with Linux kernel and/or GCC.

But I have a great, positive experience with code of the following style(C++ used for brevity, ... expands to long #ifdef trees using platform- and compiler-specific intrinsics)

Code:

union be_int {
  unsigned char b[4];
  int i;
  inline operator =(int a) { ... }
  inline operator int() { ... }
};

The explicit strong-typing of endianness blocks the vast majority of possible endian errors and allows to produce meaningfull, endian-bug-free contributions from people who are exclusively using little-endian hardware. In my (rather extensive) experience it actually increases the performance because programmers can visualize the data structures better and can spend more time designing the algorithms instead of debugging trivial omissions or transpositions.

All in all, I apologise for jumping in into the battle between you and MatthewLM. Regretfully I can't easily build neither of yours code, because I'm almost exclusively working on the closed-source platforms like Windows, MacOS, etc. and standalone environments not relying on GCC/GLIB/autogen. But it is a shame that another C implementation of Bitcoin is useless for the developers of Bitcoin hardware wallets.

[jgarzik]

So am I understanding right? I could run picocoin and it would be lightweight like Electrum but also support the Bitcoin network by interacting on the network like a full satoshi client. I've been using Electrum mostly to get away from having to use around 5GB disk space and for quick install/startup but I'd really prefer to be supporting Bitcoin more by being another validating node in it's network.

picocoin is only a lightweight client.  It is not a "full node" and probably never will be.

The included library, libccoin, does provide all the tools and gadgetry needed to build your own "full node" bitcoin client that takes 5GB disk space, if you have the programming chops to do that.

[jgarzik]

For starters, you'll have heck of a time to remove your dependency on glib.h from GTK.

You mean "remove your dependency on glib.h from libccoin", I presume?  GTK has no relevance to this, besides being the administrative umbrella project for GLib.

Removing the GLib dependency is straightforward for any developer.  GLib is used for a few ADTs like variable length strings, hash tables and arrays.

Both GLib and OpenSSL are heavy duty harvester combines. Removing the dependencies on them will at least halve the resource usage.

For a RedHat-ter at least make the code work with newlib instead of glib. The required knowledge should be no more than the internal phone call away for you. This includes a whole sermon on why you don't want to build a dependency on them.

Not sure you understand the problem space...  First, newlib is wholly different from GLib, and does not provide the ADTs that GLib provides.  Second, replacing GLib and OpenSSL would not change resource usage much at all.  ADTs, Big Integer support, SHA hashing and ECDSA are required regardless of lib chosen.  They would simply be replaced with...  code that did the same thing under name other than "OpenSSL" or "GLib."

The performance loss for passing unions around may have been true for GCC 2.96. I'm not normally working with Linux kernel and/or GCC.

It continues to be true for all known compilers I've seen or heard of, when compared with a simple, native machine integer as used by the current implementation.

All in all, I apologise for jumping in into the battle between you and MatthewLM. Regretfully I can't easily build neither of yours code, because I'm almost exclusively working on the closed-source platforms like Windows, MacOS, etc. and standalone environments not relying on GCC/GLIB/autogen.

It will soon be building on MacOSX and Windows.  Will post when this happens (volunteers willing to post building outputs are helpful).  As stated, it is a developer-only release right now, so there are plenty of rough edges.

But it is a shame that another C implementation of Bitcoin is useless for the developers of Bitcoin hardware wallets.

I respectfully disagree with that assessment.  Anyone looking at the code may see precisely what dependencies need replacing, for an embedded environment.  If anyone actually developing hardware or embedded bitcoin solutions wishes to contact me, I would be happy to illustrate how it would work.

[ldrgn]

Everything builds and tests pass on MacOS X 10.7.  Good job on this code, you've written the client Satoshi should have.

[jgarzik]

Everything builds and tests pass on MacOS X 10.7.  Good job on this code, you've written the client Satoshi should have.

Appreciated.  Thanks in particular for testing OSX.  That was a big item on the checklist.

[Steve]

Very nice!  looking forward to playing around with it.

[2112]

It continues to be true for all known compilers I've seen or heard of, when compared with a simple, native machine integer as used by the current implementation.

OK, show of hands. You must have heard of the existience of Microsoft Visual Studio C++/C compiler. I wrote a short test program that tests strong-typed little-endian integer on the Intel architecture, which will be the most important case.

Code:

#include <cstdio>

union le_int {
		unsigned char b[4];
		int i;
		void operator =(int a) { i = a; }
		operator int() { return i; }
};

int lehmer1(int n)
{
	int r = 1;
	while (n--)
		r = (16807 * r) % 2147483647;
	return r;
}

int lehmer2(int n)
{
	union le_int r;

	r = 1;
	while (n--)
		r = (16807 * r) % 2147483647;
	return r;
}

void main()
{
	printf("%d\n",lehmer1(2112));
	printf("%d\n",lehmer2(2112));
}

and the generated "Release Win32" code, with everything at default:

Code:

00311000  push        edi  
	printf("%d\n",lehmer1(2112));
00311001  mov         ecx,840h  
00311006  mov         edx,1  
0031100B  mov         edi,7FFFFFFFh  
00311010  imul        edx,edx,41A7h  
00311016  mov         eax,edx  
00311018  cdq  
00311019  idiv        eax,edi  
0031101B  dec         ecx  
0031101C  jne         main+10h (0311010h)  
0031101E  push        edx  
0031101F  push        312100h  
00311024  call        dword ptr ds:[312090h]  
0031102A  add         esp,8  
	printf("%d\n",lehmer2(2112));
0031102D  mov         ecx,840h  
00311032  mov         edx,1  
00311037  imul        edx,edx,41A7h  
0031103D  mov         eax,edx  
0031103F  cdq  
00311040  idiv        eax,edi  
00311042  dec         ecx  
00311043  jne         main+37h (0311037h)  
00311045  push        edx  
00311046  push        312100h  
0031104B  call        dword ptr ds:[312090h]  
00311051  add         esp,8 

Could anyone here can give an example of a popular compiler that produces different code for those two little functions? Maybe the test program needs to be somehow changed to trip up the optimizer? I'm seriously asking. I have code like this (and pure C equivalent with #define macros instead of operators()) in wide deployment for many years now on a variety of platforms.

I really only have my Windows laptop with me. I would have to ask people to unpack and boot my usual development environment for remote access.

Meanwhile I'm going to see if I can trip-up the Visual Studio somehow by rewriting this code in various ways.