[ANNOUNCE] picocoin and libccoin -- C-based bitcoin library and client

[jgarzik]

Thanks to denisx on IRC, who helped get picocoin/libccoin going on FreeBSD and GLib < 2.30 as well.

[ShadowOfHarbringer]

First thing, i like the idea very much.

One small suggestion though:

Do you think you could give a distinct (and cool) name for this library?  There are several now and it gets confusing.  For example:

libcoin - Gronager
libccoin - jgarzik
libbitcoin - genjix
cbitcoin - MathhewLM

How about GarzikBitcoinLib?

+1

Seriously jgarzik, these names are getting more and more confusing.

How about calling it glibbitcoin, jgbitcoin or libbitcoin-jg ?

[2112]

Removing the GLib dependency is straightforward for any developer.  GLib is used for a few ADTs like variable length strings, hash tables and arrays.

Not sure you understand the problem space...  First, newlib is wholly different from GLib, and does not provide the ADTs that GLib provides.  Second, replacing GLib and OpenSSL would not change resource usage much at all.  ADTs, Big Integer support, SHA hashing and ECDSA are required regardless of lib chosen.  They would simply be replaced with...  code that did the same thing under name other than "OpenSSL" or "GLib."

If I don't understand the problem space then you don't understand the solution space. GLib's ADT heavily depend on glibc-style memory allocation and both depend on GCC extensions and cross-compiling with MinGW instead of Visual Studio or any other native Windows compiler.

Both GLib and OpenSSL are extremely resource intensive and the only way to get respectable performance out of them is by running on a host with paging MMU and TLB. This excludes majority of standalone / lightweight OSes.

The problem/solution space doesn't demand OpenSSL. It requires BigNums, ECDSA, SHA-256. Any multiuser OS is just a serious hindrance for an lightweight implementation on a lean hardware. All those tasks are best performed not on the main CPU but on the attached GPU or DSP array. When the GPU can be directly called without the overhead of going through OpenGL/OpenCL the whole new world of possibilities opens up. As far as I know currently only PowerVR GPUs have an open source toolchain. This means Texas Instruments OMAP chips. In a year or two probably someone will do the same for the Broadcom VideoCore chips in Raspberry Pi. The performance and power efficiency gained from MIMD implemenations will be immense.

It will soon be building on MacOSX and Windows.

The salient point for Windows is: is it going to be possible to build it using a native Windows compiler like Visual Studio, OpenWATCOM or Borland/Embarcadero? Because another MinGW-only deliverable will be impossible to integrate into the Windows ecosystem.

[grau]

Source code URL:  https://github.com/jgarzik/picocoin/

Great to see a well written alternative for the thin client!

Follows the philosophy of "do, not hype."  This library is already far more secure and capable than other libraries hyped as the "future of bitcoin" by their authors.

Add do not FUD.

[jgarzik]

Thanks to jrmithdobbs for helping in bringing up OpenBSD (and more FreeBSD help from denisx).

Here's the autogen.sh and configure magic he used for OpenBSD.  Some are specific to his system, but it is an illlustrative example:

Code:

AUTOMAKE_VERSION=1.11 AUTOCONF_VERSION=2.68 ./autogen.sh && \
CC=clang LD=llvm-ld CFLAGS='-O2 -Wall -g -I/usr/local/include/event2 \
-I/opt/OpenBSD/5.1/amd64/jansson-2.4/include -I/usr/local/include/event2' \
 LDFLAGS='-L/opt/OpenBSD/5.1/amd64/jansson-2.4/lib -L/usr/local/lib -L/usr/lib' \
      ./configure --prefix=/opt/OpenBSD/5.1/amd64/picocoin-0.0 && \
      make all check

[Peter Todd]

As I've noted: I've forgot that you are one of the very few Bitcoin developers with regular access to big-endian hardware. I'm thinking that you will continue to be the only one developer who can test picocoin on the big-endian architecture. The mojibake-endian problem will continue to badly affect the future contributors/users of your library if you aren't going to use your own medicine (that is "sparse" and "__le32")

FWIW while I've never tried doing it QEMU can apparently emulate the big-endian PowerPC architecture. Here is one guys notes about getting it working from two years ago, specifically to test endian issues.

[gmaxwell]

As I've noted: I've forgot that you are one of the very few Bitcoin developers with regular access to big-endian hardware. I'm thinking that you will continue to be the only one developer who can test picocoin on the big-endian architecture. The mojibake-endian problem will continue to badly affect the future contributors/users of your library if you aren't going to use your own medicine (that is "sparse" and "__le32")

The frequent invocations of RedHat upthread are weird and misplaced.  _I_ did the initial BE testing for Jeff. Lots of people have access to BE systems and pretty much anyone who wants to can have access to one (for a couple bucks on ebay if nothing else).  ... but nothing Jeff is doing makes handling the endlessness hard as directly evidenced by the fact that he got it running fine on BE without using one.  Your 'solution' of pushing unions everywhere for that is an odd one and I'm happy to not work on your codebase.

Now that you've made your numerous complaints would you please cut it out? The tangent is going nowhere. If you want bitcoin software written to your particular, and somewhat odd, specifications feel free to do so yourself or to pay someone to do so.

[jgarzik]

How about calling it glibbitcoin, jgbitcoin or libbitcoin-jg ?

Bleh  If someone actually does come up with a cool name I'd consider it

But anything directly related to (a) my name or (b) a dependency lib's name is right out.

[marcus_of_augustus]

I think "libccoin" is fine ... clean and to the point like the code.

[jimbobway]

cbitcoinlib!

Sorta like Yahoo! but cbitcoinlib!
 
Sounds exciting when I read it.  

Search engines will also find it when people type "c bitcoin library"

[wladston]

Any ideas if this client would be suitable for RPC-like usage ? I'm currently running bitcoind on a server to power up my web service via RPC calls. But it eats a HUGE share of memory, and over the weeks it keeps getting slow, so I have to restart it often.

[Mike Hearn]

You could use an SPV client to reduce resource usage, but if you're running a web service that handles money, it's a really good idea to just pony up the resources for a full node. Security matters!

[maaku]

AES encryption is applied to the wallet.  Passphrase is specified via environment variable PICOCOIN_PASSPHRASE.

Is this secure? My understanding is that the environment isn't a safe place to store/transmit information.

That's temporary.  It will input via fd like GPG soon.

Please leave the environment-specified password as an option, however. In some environments (12-factor stacks like Heroku, for example) that remains the best available option.


I'm a little confused, does this project do full validation or not? Am I understanding it correctly that picocoin is and will remain a SPV thin-client, but libccoin implements (or will implement if it doesn't already) everything required to build a full validating node?

[jgarzik]

I'm a little confused, does this project do full validation or not? Am I understanding it correctly that picocoin is and will remain a SPV thin-client, but libccoin implements (or will implement if it doesn't already) everything required to build a full validating node?

Yes, libccoin implements everything needed to do full validation.

If you wanted to build a "full node" with libccoin, you could do that.

[jgarzik]

picocoin's library successfully validates both main and testnet3 block chains, including scripts.  ("chain-verf" test in its testsuite)

It is working on Linux, MacOSX, FreeBSD and OpenBSD.  The library compiles on Windows, but the client requires additional Windows work to be functional.

[2112]

The frequent invocations of RedHat upthread are weird and misplaced.  _I_ did the initial BE testing for Jeff. Lots of people have access to BE systems and pretty much anyone who wants to can have access to one (for a couple bucks on ebay if nothing else).  ... but nothing Jeff is doing makes handling the endlessness hard as directly evidenced by the fact that he got it running fine on BE without using one.  Your 'solution' of pushing unions everywhere for that is an odd one and I'm happy to not work on your codebase.

Now that you've made your numerous complaints would you please cut it out? The tangent is going nowhere. If you want bitcoin software written to your particular, and somewhat odd, specifications feel free to do so yourself or to pay someone to do so.

I apologize for misattributing anyone's work. I didn't actually read the commit log. From the way you wrote I have to assume that Jeff did 100% of work and you did 100% of testing. It doesn't really matter who did what, so long as it is getting done at a reasonable frequency.

The "somewhat odd" comment is really strange. It is the standard way to strenghten the C type system from structural equivalence to name equivalence. The type systems taxonomy is the standard curriculum of pretty much every CS or SE course at the university level. In fact the C++ code I posted above is patterned after the sample C/C++ code and a paper that was included with the "cfront" C++ compiler distribution from Bjarne Stroustrup that I've seen on some dusty VAX at my school. I didn't claim the autorship and in fact the requirement of not overusing type-cast-ing is one of the standard admonishments in the C/C++ project specifications.

On the other hand the 1995-1997 copyright dates on glib.h show that the base GTK/GNOME code was designed during the darkest days of GCC, predating the EGCS fork: http://en.wikipedia.org/wiki/GNU_Compiler_Collection#EGCS_fork . At that time GCC was widely used as a baseline: it would reliably produce largest and slowest code after taking the longest time to build it; so almost anything looked better in comparison. From what I've heard from one of the Linux distribution maintainers was that many sub-optimal choices in open source packages were motivated by the desire to avoid tripping up GCC to produce very slow or incorrect code. Those days are long gone.

It really doesn't take an advanced computer science education to understand why poking holes in C/C++ type system by using unsafe casts is not a good architectural choice. It may be a good short term choice. Open source projects thrive on finding a group of people who while "scratching their own itch" actually scratch in the same place. It may well be the truth that many programmers here enjoy lookin at every uint32 and ponder wheteher to use it directly or through GUINT32_{TO,FROM}_LE macros.

I also know from the past experience that overall such people are not a majority, and would consider the above task a software engineering equivalent of digging diteches and filling them back in. For every one person who posted here there is at least 10 who read this thread with understanding. Of those 10 at least 2-3 will copy/paste the example code I wrote for a future reference.

So, yes, go ahead and play with the following somewhat longer example C code. Modify it to make mistakes and see which way is better for yourself.

Code:

#include <stdio.h>
#ifdef _MSC_VER
#include <stdlib.h>
#define inline __inline
#define __builtin_bswap32 _byteswap_ulong
#endif
#ifdef __GNUC__
#define _byteswap_ulong __builtin_bswap32
#endif
/*
 * type safe little endian
 */
union le_int {
		unsigned char b[4];
		int i;
};
/* little endian setter on little endian host */
static inline union le_int h2l_int(int a)
{
	union le_int t;

	t.i = a;
	return t;
}
/* little endian getter on little endian host */
static inline int l2h_int(union le_int a)
{
	return a.i;
}
/*
 * type safe big endian
 */
union be_int {
		unsigned char b[4];
		int i;
};
/* big endian setter on little endian host */
static inline union be_int h2b_int(int a)
{
	union be_int t;

	t.i = _byteswap_ulong(a);
	return t;
}
/* big endian getter on little endian host */
static inline int b2h_int(union be_int a)
{
	return _byteswap_ulong(a.i);
}

/* excerpted from the Glib.h include tree */
typedef unsigned int guint32;
typedef signed int gint32;
#define GUINT32_SWAP_LE_BE(val) ((guint32) __builtin_bswap32 ((gint32) val))
#define GINT32_TO_LE(val)	((gint32) (val))
#define GINT32_TO_BE(val)	((gint32) GUINT32_SWAP_LE_BE (val))
#define GINT32_FROM_LE(val)	(GINT32_TO_LE (val))
#define GINT32_FROM_BE(val)	(GINT32_TO_BE (val))
/* end of Glib.h excerpt */

/* little endian type unsafe */
int lehmer1(int n)
{
	int r = GINT32_TO_LE(1);

	while (n--)
		r = GINT32_TO_LE((16807 * GINT32_FROM_LE(r)) % 2147483647);
	return GINT32_FROM_LE(r);
}
/* little endian type safe */
int lehmer2(int n)
{
	union le_int r = h2l_int(1);

	while (n--)
		r = h2l_int((16807 * l2h_int(r)) % 2147483647);
	return l2h_int(r);
}
/* big endian type unsafe */
int lehmer3(int n)
{
	int r = GINT32_TO_BE(1);

	while (n--)
		r = GINT32_TO_BE((16807 * GINT32_FROM_BE(r)) % 2147483647);
	return GINT32_FROM_BE(r);
}
/* big endian type safe */
int lehmer4(int n)
{
	union be_int r = h2b_int(1);

	while (n--)
		r = h2b_int((16807 * b2h_int(r)) % 2147483647);
	return b2h_int(r);
}

int main()
{
	printf("%d\n",lehmer1(2112));
	printf("%d\n",lehmer2(2112));
	printf("%d\n",lehmer3(2112));
	printf("%d\n",lehmer4(2112));
	return 0;
}

[2112]

FWIW while I've never tried doing it QEMU can apparently emulate the big-endian PowerPC architecture. Here is one guys notes about getting it working from two years ago, specifically to test endian issues.

Thanks. The other option for big-endian testing on Intel little-endian laptop is running an IBM mainframe Linux eg. Debian or RedHat S390 on the Hercules emulator.

http://www.hercules-390.org/

[jgarzik]

Recent progress:

• libccoin now signs transactions.  Handling chain re-org is now the last hurdle for proper backend client support
• Building on older Debian/Ubuntu Linux now supported

[gmaxwell]

The "somewhat odd" comment is really strange. It is the standard way to strenghten the C type system from structural equivalence to name equivalence. The type systems taxonomy is the standard curriculum of pretty much every CS or SE course at the university level. In fact the C++ code I posted above is patterned after the sample C/C++ code and a paper that was included with the "cfront" C++ compiler distribution from Bjarne Stroustrup that I've seen on some dusty VAX at my school. I didn't claim the autorship and in fact the requirement of not overusing type-cast-ing is one of the standard admonishments in the C/C++ project specifications.

Nice lecture, I suspect that you didn't get my argument or I misunderstood what you're insisting on.  I'm not saying that using the typesystem and avoiding typecasting all over the code is weird, I'm saying that endenanness is usually compartmentalized— you can handle it cleanly when serializing and deserializing and only at those boundaries, and in the manner specified for the kind of serialization being performed. Everywhere is just a single fundamental type, and there is no need to carry a union around and to pollute the whole codebase with type specifiers which are not only not useful but acutely dangerous— as adhoc serialization using the alternative references littered in the codebase may miss the required byteorder functions.

Programming is foremost an art of communicating between programmers. Forcing every machine integer into a specialized union type when the only conversions needed are at specialized serialization points weakens that communication and probably increases the frequency of errors. It certainly isn't the convention in the OSS C ecosystem.

[paybitcoin]

Hey, thanks for libccoin!!! I used it for a brute forcer for Casascius's BIP38 contest and it was very clearly laid out (at least from what I used on the library side.)

I did find a possible bug, though:

When generating a public key and Bitcoin address from a private key, the EC key is using the compressed form, when I believe it should be using the uncompressed form. I added this workaround here, but I don't know if it is the right way to go.

https://github.com/notespace/picocoin/commit/736235acabf08f2eae8856673bfd68574c8600fe

I am generating the private key bytes from the outputs of the EC multiply operation required by BIP 38, then using it like this:

bp_key_init(&wallet)
bp_key_secret_set(&wallet,finalKey,32)
bp_pubkey_get(&wallet, ((void **) &pubKey), &pubKeylen);
btcAddress = bp_pubkey_get_address(&wallet, 0);

Also, you can see the whole code for the bip38 cracker here, all code is in main.c:

https://github.com/notespace/bip38-cracker