Bitcoin Forum
December 11, 2017, 04:16:17 AM *
News: Latest stable version of Bitcoin Core: 0.15.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: 1 2 3 4 5 6 7 8 9 10 11 [All]
  Print  
Author Topic: cbitcoin - Bitcoin implementation in C. Currently in development.  (Read 19896 times)
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 10, 2012, 12:25:44 AM
 #1

Hello.

I am currently making a bitcoin library named cbitcion. This is a very incomplete library in the early stages of development. The point of the library is to bring powerful bitcoin features directly to the C programming language, with a focus on documentation, portability and powerful features. Hopefully, considering this is C, it should have good performance as well. Perhaps the library could be used or modified for use in embedded device applications.

The library will be low level. It is not intended to be a client library for easy bitcoin functionality. Instead it should give programmers the tools to make bitcoin applications from basic abstraction on the bitcoin protocol. The design of cbitcoin should hopefully allow programmers to make a diverse range of bitcoin applications.

Feedback and contributions are welcome. This is the code: https://github.com/MatthewLM/cbitcoin/

I've given up developing iPhone apps to put a lot of my time towards this, so hopefully it will come along reasonably quickly.

There will be inevitable costs to this project. If anyone would like to help this project financially, donations are very welcome: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9

Do not hesitate to contact me regarding this project via the email address cbitcoin@thelibertyportal.com

Thanks!

Matthew M

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1512965777
Hero Member
*
Offline Offline

Posts: 1512965777

View Profile Personal Message (Offline)

Ignore
1512965777
Reply with quote  #2

1512965777
Report to moderator
1512965777
Hero Member
*
Offline Offline

Posts: 1512965777

View Profile Personal Message (Offline)

Ignore
1512965777
Reply with quote  #2

1512965777
Report to moderator
1512965777
Hero Member
*
Offline Offline

Posts: 1512965777

View Profile Personal Message (Offline)

Ignore
1512965777
Reply with quote  #2

1512965777
Report to moderator
hamdi
Hero Member
*****
Offline Offline

Activity: 714



View Profile
May 10, 2012, 12:31:11 AM
 #2

great

Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1050


View Profile WWW
May 10, 2012, 01:01:16 AM
 #3

If you're going to rewrite C++ in C (by building virtual function tables yourself), why not code it in C++ in the first place? If compatibility is a problem, you can always expose a C interface to the code.

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 10, 2012, 01:09:15 AM
 #4

The number one reason is that I've never liked C++. I get on with C much better.

C++ programmers can pretty much read and use C code with little problem. C++ code is not so easy for solely C programmers. Not only can the library be used by C programmers, it can be written by C programmers that don't get on with C++.

And implementing OOP features in C is not as hard as it sounds once you know how.

But yes, that's a common question.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
May 11, 2012, 04:09:11 AM
 #5

I fully approve of this effort.  Many told me that it was a waste of time to reimplement Bitcoin, but there's no better way to learn everything than to do it yourself.  It's not always pleasant (especially that goddamned endianness!), but it is extraordinarily educational and rewarding by the time you do get it.  I feel like anyone who's going to participate in serious discussions about protocol, client design/features, or contribute to any Bitcoin project at all, should have this kind of experience.

I started about 10 months ago, and did a little bit of documentation as I went.  I expect you'll find this useful Smiley

https://bitcointalk.org/index.php?topic=29416.0

Feel free to PM me if you have any implementation questions.  I've been through just about all of it by now.  Half of it in C++, half in Python...
(though I never bothered with bignum/base58 in C++... I track everything by hash160 values and let Python deal with Base58 using its native bignum capability).

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
finway
Hero Member
*****
Offline Offline

Activity: 714


View Profile
May 11, 2012, 05:31:49 AM
 #6

I fully approve of this effort.  Many told me that it was a waste of time to reimplement Bitcoin, but there's no better way to learn everything than to do it yourself.  It's not always pleasant (especially that goddamned endianness!), but it is extraordinarily educational and rewarding by the time you do get it.  I feel like anyone who's going to participate in serious discussions about protocol, client design/features, or contribute to any Bitcoin project at all, should have this kind of experience.

I started about 10 months ago, and did a little bit of documentation as I went.  I expect you'll find this useful Smiley

https://bitcointalk.org/index.php?topic=29416.0

Feel free to PM me if you have any implementation questions.  I've been through just about all of it by now.  Half of it in C++, half in Python...
(though I never bothered with bignum/base58 in C++... I track everything by hash160 values and let Python deal with Base58 using its native bignum capability).
+1

jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
May 11, 2012, 06:34:33 AM
 #7

I'm currently looking at the bignum division: http://stackoverflow.com/questions/10522379/bignum-division-with-an-unsigned-8-bit-integer-c The current algoritm was rather much a curiosity, only now have I looked for a better algorithm. This will go into the base 58 converter functions which is what I'll be working on (Maybe those can be done differently also?).

Why?  Just use OpenSSL for bignum and move on.  People have already written that part.

Bitcoin-specific data structures for storing blocks, transactions, etc. are the fundamental building blocks.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
casascius
Mike Caldwell
VIP
Legendary
*
Offline Offline

Activity: 1372


The Casascius 1oz 10BTC Silver Round (w/ Gold B)


View Profile WWW
May 11, 2012, 06:45:50 AM
 #8

Why?  Just use OpenSSL for bignum and move on.  People have already written that part.

Bitcoin-specific data structures for storing blocks, transactions, etc. are the fundamental building blocks.

If he manages to pull this off without a dependency on OpenSSL, his code is going to be able to run on my credit card machines.

Companies claiming they got hacked and lost your coins sounds like fraud so perfect it could be called fashionable.  I never believe them.  If I ever experience the misfortune of a real intrusion, I declare I have been honest about the way I have managed the keys in Casascius Coins.  I maintain no ability to recover or reproduce the keys, not even under limitless duress or total intrusion.  Remember that trusting strangers with your coins without any recourse is, as a matter of principle, not a best practice.  Don't keep coins online. Use paper or hardware wallets instead.
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
May 11, 2012, 04:33:06 PM
 #9

Why?  Just use OpenSSL for bignum and move on.  People have already written that part.

Bitcoin-specific data structures for storing blocks, transactions, etc. are the fundamental building blocks.

If he manages to pull this off without a dependency on OpenSSL, his code is going to be able to run on my credit card machines.

It is rarely a good idea to reinvent the wheel, when it comes to security-sensitive, time-tested, well-reviewed-by-experts cryptography code.

Once bignum has been reinvented, what about ECDSA and similar gadgetry?


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1050


View Profile WWW
May 11, 2012, 04:39:53 PM
 #10

From the perspective of being educational, I have absolutely no problem with people reimplementing code. They may stumble upon bugs, and even a partial implementation can have niche purposes (for example only the networking code for a crawler, or fast blockchain manipulation code for rescanning addresses).

That said, trying to understand existing code also has great educational value; in particular when you're trying to improve things, because that confronts you with things you thought you understood but didn't. In my opinion, this is a better way for helping the community than by starting from scratch - that just takes longer before it is useful.

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
May 11, 2012, 04:40:08 PM
 #11

It is rarely a good idea to reinvent the wheel, when it comes to security-sensitive, time-tested, well-reviewed-by-experts cryptography code.

This.  Also why is avoiding dependency on OpenSSL important?  Something I am missing?
mb300sd
Legendary
*
Offline Offline

Activity: 1260

Drunk Posts


View Profile WWW
May 11, 2012, 06:40:43 PM
 #12

Looks like your writing this to work on 8-bit CPUs? This may be very useful if you can keep it C-only and 8-bit compatible... Single chip embedded bitcoin client anyone?

1D7FJWRzeKa4SLmTznd3JpeNU13L1ErEco
casascius
Mike Caldwell
VIP
Legendary
*
Offline Offline

Activity: 1372


The Casascius 1oz 10BTC Silver Round (w/ Gold B)


View Profile WWW
May 11, 2012, 07:41:18 PM
 #13

It is rarely a good idea to reinvent the wheel, when it comes to security-sensitive, time-tested, well-reviewed-by-experts cryptography code.

This.  Also why is avoiding dependency on OpenSSL important?  Something I am missing?

If you ripped the code from OpenSSL it would work fine.  The devices just don't have gobs of memory for libraries and they have their own compiler that has idiosyncrasies that make it not compile everything that can be compiled in Linux.  Cause it is not Linux.

This would be a 32 bit CPU, no need for 8 bits.

Companies claiming they got hacked and lost your coins sounds like fraud so perfect it could be called fashionable.  I never believe them.  If I ever experience the misfortune of a real intrusion, I declare I have been honest about the way I have managed the keys in Casascius Coins.  I maintain no ability to recover or reproduce the keys, not even under limitless duress or total intrusion.  Remember that trusting strangers with your coins without any recourse is, as a matter of principle, not a best practice.  Don't keep coins online. Use paper or hardware wallets instead.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 11, 2012, 07:52:45 PM
 #14

This library aims to have zero dependencies except the standard C library making it small and easily portable. This doesn't mean I will reinvent the wheel in every single case. There are many open source implementations of the things my library will need. I can adapt these tried and tested algorithms to suit the specific needs of this library. In some cases it may be more or less copy and paste and in other cases it may require tweaks or substantial modifications.

At the moment I'm implementing specialist functions that include operations between a bignum and an 8 bit integer. This is for the base-58 encoder/decoder. If anyone knows a better solution to the base-58 stuff, please tell me or feel free to work on it yourself. When I'll be going operations between two bignums it should be simple to derive my code from bignum libraries and then I can derive the cryptography code from open-source libraries such as OpenSSL too.

If he manages to pull this off without a dependency on OpenSSL, his code is going to be able to run on my credit card machines.

Well wouldn't that be interesting. You would need an implementation of standard library features including dynamic memory allocation but I plan absolutely no additional dependencies other than the standard C library. Of-course, feel free to contribute anything, including ideas.

I fully approve of this effort.  Many told me that it was a waste of time to reimplement Bitcoin, but there's no better way to learn everything than to do it yourself.  It's not always pleasant (especially that goddamned endianness!), but it is extraordinarily educational and rewarding by the time you do get it.  I feel like anyone who's going to participate in serious discussions about protocol, client design/features, or contribute to any Bitcoin project at all, should have this kind of experience.

I started about 10 months ago, and did a little bit of documentation as I went.  I expect you'll find this useful Smiley

https://bitcointalk.org/index.php?topic=29416.0

Feel free to PM me if you have any implementation questions.  I've been through just about all of it by now.  Half of it in C++, half in Python...
(though I never bothered with bignum/base58 in C++... I track everything by hash160 values and let Python deal with Base58 using its native bignum capability).

Thanks. I am doing this largely to learn as well as to create something functional. The images you have made a pretty nice. Would I be able to use them in my documentation? I do hope to build a nice documentation for my library.

For the next few days I'll be busy with other things but I'll when I'm back to this library I hope to sort of the base-58 code and also modify the OOP so that the virtual function tables are initialised at compile time which makes more sense.

Looks like your writing this to work on 8-bit CPUs? This may be very useful if you can keep it C-only and 8-bit compatible... Single chip embedded bitcoin client anyone?

The library should be compilable with C compilers providing C99 with the standard library is supported (At least everything the library uses). Perhaps the library will need minor tweaks for compilers that aren't compatible with some features in C99 but that's not going to cause many headaches I sure. The biggest headache would be to implement the standard library features for microcontrollers with no out of the box standard C library support.

But are you saying some 8-bit architectures come with compilers that only allow 8 bit types? That would be pretty rubbish. The compilers should support up-to 64 bit integer types (int64_t). I believe long long types should be at least 64 bits so a compiler that keeps to the C specification properly should support it.

If anyone understands microcontrollers well and can advise on how the library can be designed with microcontroller portability in mind, please share your knowledge.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
May 11, 2012, 08:05:32 PM
 #15

Thanks. I am doing this largely to learn as well as to create something functional. The images you have made a pretty nice. Would I be able to use them in my documentation? I do hope to build a nice documentation for my library.

Go ahead.  Attribution is preferable, but I'm really just happy if other people get use out of them.  Maybe I can save you the 2 days I spent battling endianness and the OP_CHECKSIG wiki page trying to figure out how the hell to implement it!

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 11, 2012, 08:22:54 PM
 #16

Under the images I can put "Pictures created by Alan Reiner, developer of Bitcoin Armory (http://bitcoinarmory.com/)"?

I'm hoping following bitcoinj will also help (It's very easy to read), the issue is that bitcoinj doesn't implement everything so I'll have to look into other code that does implement things like the script properly. The C++ client has the code but it harder to read. Documentation no doubt helps me understand what I'm doing rather than do a piece by piece port, so I can make better code.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
May 11, 2012, 08:42:14 PM
 #17

It is rarely a good idea to reinvent the wheel, when it comes to security-sensitive, time-tested, well-reviewed-by-experts cryptography code.

This.  Also why is avoiding dependency on OpenSSL important?  Something I am missing?

If you ripped the code from OpenSSL it would work fine.  The devices just don't have gobs of memory for libraries and they have their own compiler that has idiosyncrasies that make it not compile everything that can be compiled in Linux.  Cause it is not Linux.

This would be a 32 bit CPU, no need for 8 bits.

Ummm...

OpenSSL is heavily used in the resource-constrained, embedded space, and runs on Windows, Linux, uclinux (embedded linux), VxWorks (an embedded OS), OSX, Solaris, HPUX, OSF1/Tru64, IBM mainframes, ancient MS-DOS systems, netware, and others old and new.  It works very well on almost every known 32- and 64-bit  platform in existence.

It does not require "gobs of memory"; the state structures for AES or SHA algorithms are almost always smaller than 1,024 bytes.  Compiled code size is already small and compact.

Copying OpenSSL code into your own codebase will not save any memory, regardless of linking style.  Libraries are internally split into modules and demand-paged in.  At best it will save disk space for unused code, at a cost of always lagging behind whenever security problems in the code are found.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
doldgigger
Full Member
***
Offline Offline

Activity: 170


View Profile
May 11, 2012, 08:49:05 PM
 #18

If you're going to rewrite C++ in C (by building virtual function tables yourself), why not code it in C++ in the first place? If compatibility is a problem, you can always expose a C interface to the code.
In order to have a super-lean runtime and dev environment, for example...

19orEcoqXQ5bzKbzbAnbQrCkQC5ahSh4P9
Feel free to PM me for consulting and development services.
casascius
Mike Caldwell
VIP
Legendary
*
Offline Offline

Activity: 1372


The Casascius 1oz 10BTC Silver Round (w/ Gold B)


View Profile WWW
May 11, 2012, 09:37:53 PM
 #19

It is rarely a good idea to reinvent the wheel, when it comes to security-sensitive, time-tested, well-reviewed-by-experts cryptography code.

This.  Also why is avoiding dependency on OpenSSL important?  Something I am missing?

If you ripped the code from OpenSSL it would work fine.  The devices just don't have gobs of memory for libraries and they have their own compiler that has idiosyncrasies that make it not compile everything that can be compiled in Linux.  Cause it is not Linux.

This would be a 32 bit CPU, no need for 8 bits.

Ummm...

OpenSSL is heavily used in the resource-constrained, embedded space, and runs on Windows, Linux, uclinux (embedded linux), VxWorks (an embedded OS), OSX, Solaris, HPUX, OSF1/Tru64, IBM mainframes, ancient MS-DOS systems, netware, and others old and new.  It works very well on almost every known 32- and 64-bit  platform in existence.

It does not require "gobs of memory"; the state structures for AES or SHA algorithms are almost always smaller than 1,024 bytes.  Compiled code size is already small and compact.

Copying OpenSSL code into your own codebase will not save any memory, regardless of linking style.  Libraries are internally split into modules and demand-paged in.  At best it will save disk space for unused code, at a cost of always lagging behind whenever security problems in the code are found.



Oh, yes, I should clarify.  VeriFones use a persistent battery-backed RAM-based file system, so disk space essentially equals RAM.  I am referring to the size of the library, not how much memory it allocates.  And some of the other devices I've developed for have very limited flash storage where fitting the entire compiled openssl lib takes up significant space.  I suppose it's laughable to consider a couple megabytes to be "gobs" but in the context of these cheapo devices, it is.  Also for the VeriFones they don't use GCC so sometimes things take refactoring for their dinosaur compiler to be able to handle it (and makefiles in particular).  Of course I understand having outdated tools is sucky, but that's also why you can find these things on eBay for so cheap.  The Omni 3200 model has an even older DOS-based compiler, but someone who bites the bullet and makes a working app for it suddenly breathes new life into a $30-on-eBay all-in-one-computer with a printer to boot.

Even with security bugs (not saying security bugs are OK), but possession is nine tenths of the law with these old school devices.  Suppose somebody finds a way to be able to perform a side channel attack on it and steal keys with RF emissions or something, they also need to gain physical access to the device to do it, making a widespread attack not all that feasible.  Their network support is rudimentary (often the units with Ethernet ports are merely Ethernet-to-Serial converters on an internal tty that is completely silent unless you write an app to access it) so the odds of someone making a key-disclosing network attack against them is pretty much moot.

Companies claiming they got hacked and lost your coins sounds like fraud so perfect it could be called fashionable.  I never believe them.  If I ever experience the misfortune of a real intrusion, I declare I have been honest about the way I have managed the keys in Casascius Coins.  I maintain no ability to recover or reproduce the keys, not even under limitless duress or total intrusion.  Remember that trusting strangers with your coins without any recourse is, as a matter of principle, not a best practice.  Don't keep coins online. Use paper or hardware wallets instead.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 11, 2012, 09:40:16 PM
 #20

Would people be more universally happy if my library required developers using the library to pass cryptographic functionality to the library using function pointers? That way people can use whatever cryptography library they like that matches their needs, and my library will focus on everything else unique to bitcoin.

Good idea?

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
May 12, 2012, 12:30:23 AM
 #21

Oh, yes, I should clarify.  VeriFones use a persistent battery-backed RAM-based file system, so disk space essentially equals RAM.  I am referring to the size of the library, not how much memory it allocates.   And some of the other devices I've developed for have very limited flash storage where fitting the entire compiled openssl lib takes up significant space.

Then link statically.  Or use one of the many embedded tools out there that ensures your lib has only the code your filesystem actually uses.

This is a solved problem, and does not exclude OpenSSL use in any way.

Quote
Also for the VeriFones they don't use GCC so sometimes things take refactoring for their dinosaur compiler to be able to handle it (and makefiles in particular).  Of course I understand having outdated tools is sucky, but that's also why you can find these things on eBay for so cheap.  The Omni 3200 model has an even older DOS-based compiler, but someone who bites the bullet and makes a working app for it suddenly breathes new life into a $30-on-eBay all-in-one-computer with a printer to boot.

And OpenSSL likely supports outdated OS's and tools...  they maintain support for MS-DOS to this day because that remains a common embedded target.



Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
2112
Legendary
*
Online Online

Activity: 1988



View Profile
May 12, 2012, 03:01:51 AM
 #22

Would people be more universally happy if my library required developers using the library to pass cryptographic functionality to the library using function pointers?
Good idea?
No, nobody is planning on swapping the low-level crypto layer at runtime. If you want to abstract it do so using preprocessor macros. I have two other important suggestions for a from-the-scratch implementation:

1) at all times maintain the code as endian-neutral and with proper alignment. If you don't have a Macintosh OSX 10.[456], then build yourself a Hackintosh with Rosetta and test your code using "gcc -arch ppc"
2) resolve the critical inversion of control issue that is present in the Satoshi client. I wrote an post about this a while back. I'll put the link directly to it once I find it. Here they are:

https://bitcointalk.org/index.php?topic=44330.msg538044#msg538044
https://bitcointalk.org/index.php?topic=55842.msg664634#msg664634

Good luck.

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
jothan
Full Member
***
Offline Offline

Activity: 184


Feel the coffee, be the coffee.


View Profile
May 12, 2012, 03:57:53 AM
 #23

It is rarely a good idea to reinvent the wheel, when it comes to security-sensitive, time-tested, well-reviewed-by-experts cryptography code.

This.  Also why is avoiding dependency on OpenSSL important?  Something I am missing?

If you ripped the code from OpenSSL it would work fine.  The devices just don't have gobs of memory for libraries and they have their own compiler that has idiosyncrasies that make it not compile everything that can be compiled in Linux.  Cause it is not Linux.

This would be a 32 bit CPU, no need for 8 bits.

Would Nettle fill the bill ? Crypto routines only, no SSL/TLS. I think GnuTLS has a Nettle extension for ECC/ECDSA.

http://www.lysator.liu.se/~nisse/nettle/

Bitcoin: the only currency you can store directly into your brain.

What this planet needs is a good 0.0005 BTC US nickel.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 12, 2012, 06:43:41 PM
 #24

Would people be more universally happy if my library required developers using the library to pass cryptographic functionality to the library using function pointers?
Good idea?
No, nobody is planning on swapping the low-level crypto layer at runtime. If you want to abstract it do so using preprocessor macros. I have two other important suggestions for a from-the-scratch implementation:

1) at all times maintain the code as endian-neutral and with proper alignment. If you don't have a Macintosh OSX 10.[456], then build yourself a Hackintosh with Rosetta and test your code using "gcc -arch ppc"
2) resolve the critical inversion of control issue that is present in the Satoshi client. I wrote an post about this a while back. I'll put the link directly to it once I find it. Here they are:

https://bitcointalk.org/index.php?topic=44330.msg538044#msg538044
https://bitcointalk.org/index.php?topic=55842.msg664634#msg664634

Good luck.

1. So you are saying test my code works on both x86-64 and ppc architectures? Well I'm writing it in C with no dependencies or assembly so no reason why not as long as I follow the usual guidelines for programming portable C code.

2. You mean to keep all the components separated? Well this is the plan and I was asking if I should completely separate the cryptography code. With function pointers dependency injection can be done. I'll likely do something like that for threading functionality, without having a dependency on a threading library. I could include the cryptography with the library but it seems it might cause controversy and leaving the cryptography code outside the library satisfies portability and flexibility concerns.

Would Nettle fill the bill ? Crypto routines only, no SSL/TLS. I think GnuTLS has a Nettle extension for ECC/ECDSA.

http://www.lysator.liu.se/~nisse/nettle/

I don't see why not allow people to choose what they want. It cannot be hard to create wrapper functions around these libraries to pass these functions with functionality like void (*sha256)(u_int8_t *, u_int16_t ,u_int8 [32]) or something similar.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
doldgigger
Full Member
***
Offline Offline

Activity: 170


View Profile
May 12, 2012, 07:43:28 PM
 #25

Would people be more universally happy if my library required developers using the library to pass cryptographic functionality to the library using function pointers?
Good idea?
No, nobody is planning on swapping the low-level crypto layer at runtime. If you want to abstract it do so using preprocessor macros.
Even though swapping the crypto layer at runtime is unlikely, function pointers are the more flexible alternative as they allow the developer to choose his crypto layer without having to recompile the bitcoin library.

Just have a look at the OpenSSL library: When you want to swap the memory allocator there, you also do it through function pointers, not preprocessor macros. Still, for obvious reasons nobody will want to switch the allocator at runtime...

19orEcoqXQ5bzKbzbAnbQrCkQC5ahSh4P9
Feel free to PM me for consulting and development services.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 12, 2012, 08:49:26 PM
 #26

It is perhaps conceivable that there are some one set of cryptography functions that provide better performance but less security and another set that are slower but provide greater security. A programmer may switch the functions to use the increased security when it is needed or whatever.

But this is not the point in using function pointers, by using function pointers the cryptography functions can be chosen by the programmer using the library and can be configured when the program starts. This is the primary reason. With great flexibility the library can be used as a single dynamic library across various applications; indeed the library will not need to be recompiled for changing requirements.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
2112
Legendary
*
Online Online

Activity: 1988



View Profile
May 13, 2012, 03:21:22 AM
 #27

1. So you are saying test my code works on both x86-64 and ppc architectures? Well I'm writing it in C with no dependencies or assembly so no reason why not as long as I follow the usual guidelines for programming portable C code.
What I meant to convey is the following: the Bitcoin protocol itself contains a multitude of design errors related to byte-order and word-alignment. Those errors are compounded by additional errors in the Satoshi client implementation. If you really plan on writing a portable code you'll need to fix those errors in implementation and work around the errors in protocol to maintain the wire compatibility. This is a nontrivial undertaking. If you don't have access to a native big-endian platform like SPARC or z/Arch, then the cheapest way to debug those problem is by working under OS/X.

2. You mean to keep all the components separated? Well this is the plan and I was asking if I should completely separate the cryptography code. With function pointers dependency injection can be done. I'll likely do something like that for threading functionality, without having a dependency on a threading library. I could include the cryptography with the library but it seems it might cause controversy and leaving the cryptography code outside the library satisfies portability and flexibility concerns.

What I meant to convery is the following: the Satoshi client implementation is  non-deterministic and non-testable (even while single-threaded) due to to horrible architectural choice of wrapping a stochastic knapsack solver for coin selection and fee calculation inside a control-inverted subroutine that is used in sendtoaddress() API. You'll need to seriously re-architect the API to come up with a deterministic library that will produce deterministic fees with non-trivial wallets.

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
2112
Legendary
*
Online Online

Activity: 1988



View Profile
May 13, 2012, 03:32:35 AM
 #28

Even though swapping the crypto layer at runtime is unlikely, function pointers are the more flexible alternative as they allow the developer to choose his crypto layer without having to recompile the bitcoin library.
For any platform where the above would be useful, the following is sufficient:
1) declare the appropriate entry points as "extern";
2) use the platform's native dynamic linking apparatus.

Using additional layer of indirection is completely pointless and has the following disadvantages:
1) it defeats many optimizations that the tool-chain could apply.
2) it defeats many automatic static-analysis and debugging tools.
3) slows down the resulting code (especially if the CPU doesn't have a branch predictor) and increases the energy consumption during execution (more CPU instructions and more cache misses).

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
May 13, 2012, 03:36:45 AM
 #29

Even though swapping the crypto layer at runtime is unlikely, function pointers are the more flexible alternative as they allow the developer to choose his crypto layer without having to recompile the bitcoin library.
For any platform where the above would be useful, the following is sufficient:
1) declare the appropriate entry points as "extern";
2) use the platform's native dynamic linking apparatus.

Using additional layer of indirection is completely pointless and has the following disadvantages:
1) it defeats many optimizations that the tool-chain could apply.
2) it defeats many automatic static-analysis and debugging tools.
3) slows down the resulting code (especially if the CPU doesn't have a branch predictor) and increases the energy consumption during execution (more CPU instructions and more cache misses).

All true.

Nevertheless, a wide swath of very fundamental libs provide hooks for similarly fundamental services, most notably memory allocation and m-t locking primitives.

...regardless, all this is really discussing useless details.  The main focus, I hope, should be that cbitcoin is actually doing something unique to bitcoin like parsing blocks and talking network, and not simply wasting a lot of time reimplementing basic software services like crypto or hash or data codec.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 13, 2012, 06:16:46 PM
 #30

Remember, anyone would be free to put in cryptography code directly into their own modified versions of the library. Makes sense, at least for now, to use function pointers I think.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 15, 2012, 08:14:37 PM
 #31

The base 58 conversion is working, time to move on. Could do with some more varied testing but for now the tests will do (Contributions to testing are more than welcome).

I'm all ears to suggestions. Now is the best time to pass suggestions to me as it's early on in development.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
realnowhereman
Hero Member
*****
Offline Offline

Activity: 504



View Profile
May 16, 2012, 11:18:41 AM
 #32

Development has kind of stalled lately, but in case it's of any use to you..

https://github.com/andyparkins/additup

It's C++ so won't be copy-and-pastable.  But it talks enough of the protocol to receive and parse blocks and transactions and has base58 (or anything else if you wanted it) support; plus a small test app to manipulate keys and addresses.

1AAZ4xBHbiCr96nsZJ8jtPkSzsg1CqhwDa
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 16, 2012, 07:08:16 PM
 #33

Thanks. It may be useful to refer to parts that I can't get from bitcoinj such as the script interpreter.  Wink

It seems I've made a little problem. I'm storing bitcoin addresses backwards at the moment since my base-58 conversion code works with little-endian data. Probably makes sense to reverse the result of the base-58 conversion so they are stored the right way around instead of reversing hashes and whatever.

etotheipi was right about the endianness issue! :-)

I have to make some interesting decisions. Should I have a reference counted string structure? It might help with the bitcoin address strings.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
realnowhereman
Hero Member
*****
Offline Offline

Activity: 504



View Profile
May 16, 2012, 08:26:53 PM
 #34

Thanks. It may be useful to refer to parts that I can't get from bitcoinj such as the script interpreter.  Wink

Yep; the interpreter is there and most of the ops; except (of course) for the difficult ones that actually check signatures :-)

It seems I've made a little problem. I'm storing bitcoin addresses backwards at the moment since my base-58 conversion code works with little-endian data. Probably makes sense to reverse the result of the base-58 conversion so they are stored the right way around instead of reversing hashes and whatever.

etotheipi was right about the endianness issue! :-)

Ain't that the truth.

I have to make some interesting decisions. Should I have a reference counted string structure? It might help with the bitcoin address strings.

Just abstract your string access to a set of routines, then you can make that decision later.  First pass: just use library code.

1AAZ4xBHbiCr96nsZJ8jtPkSzsg1CqhwDa
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 16, 2012, 10:07:34 PM
 #35

Well I decided the easiest and most sensible thing to do was indeed make a new structure inheriting CBObject named CBString as a wrapper for strings which does reference counting. Then it is also consistent with everything else.

Addresses work! I think I'll start working on the script interpreter next.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Haplo
Full Member
***
Offline Offline

Activity: 168



View Profile
May 17, 2012, 12:07:59 AM
 #36

If you're going to rewrite C++ in C (by building virtual function tables yourself), why not code it in C++ in the first place? If compatibility is a problem, you can always expose a C interface to the code.

Because C++ has craptastic performance and tramples all over C conventions, both for no reason. -The Linux Kernel Dev Team

Also why gnome decided to start from scratch with gobject, although I can't vouch for gobject's characteristics either.

That said, I approve of this project, and will probably donate to that end soonish.

Thanks. It may be useful to refer to parts that I can't get from bitcoinj such as the script interpreter.  Wink

It seems I've made a little problem. I'm storing bitcoin addresses backwards at the moment since my base-58 conversion code works with little-endian data. Probably makes sense to reverse the result of the base-58 conversion so they are stored the right way around instead of reversing hashes and whatever.

etotheipi was right about the endianness issue! :-)

I have to make some interesting decisions. Should I have a reference counted string structure? It might help with the bitcoin address strings.


I don't see what reference counting has to do with endianness, but generally it's only useful for memory management of long-lived objects. Are you really writing your own memory management code? o.0

I'm So Meta, Even This Acronym
GroundRod
Full Member
***
Offline Offline

Activity: 206


View Profile
May 17, 2012, 12:45:48 AM
 #37

A project after my own heart, will be checking in from time to time, see if I can make sense of some of the things your talking about. 

Wish I had more time this summer, this would be a fun one for me too...if by chance I get to the point of being helpful, will let you know MatthewLM!
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 17, 2012, 01:22:11 AM
 #38

If you're going to rewrite C++ in C (by building virtual function tables yourself), why not code it in C++ in the first place? If compatibility is a problem, you can always expose a C interface to the code.

Because C++ has craptastic performance and tramples all over C conventions, both for no reason. -The Linux Kernel Dev Team

"you can write object-oriented code (useful for filesystems etc) in C, _without_ the crap that is C++." That made me smile.

People tell me C++ can be just as efficient as C. Whether or not there is a performance issue, I don't care. I don't like C++, I like C.

That said, I approve of this project, and will probably donate to that end soonish.

Well that would be very kind, thank you.  Smiley

I don't see what reference counting has to do with endianness, but generally it's only useful for memory management of long-lived objects. Are you really writing your own memory management code? o.0

In my post I was talking about two separate issues. The endianness is a bit annoying but it's fine and I did decide to use reference counting for strings in the end.

The memory management is a simple reference counter. It is nothing complicated and makes the library much more straight-forward when passing objects everywhere, as no-doubt will be the case with a library like this. You can see what I've done by looking at the CBObject files. Look at the testCBAddress.c file for an example of using a CBAddress object with memory management included.

A project after my own heart, will be checking in from time to time, see if I can make sense of some of the things your talking about. 

Wish I had more time this summer, this would be a fun one for me too...if by chance I get to the point of being helpful, will let you know MatthewLM!

Thanks! PM me anytime or send an email to cbitcoin@thelibertyportal.com.

Next I'm going to implement the script interpreter. A lot of op-codes to do and some complexities here and there but I think it wont be too difficult. I will probably start on saturday since I'm busy until then with other things.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Haplo
Full Member
***
Offline Offline

Activity: 168



View Profile
May 17, 2012, 01:56:23 AM
 #39

I don't see what reference counting has to do with endianness, but generally it's only useful for memory management of long-lived objects. Are you really writing your own memory management code? o.0

In my post I was talking about two separate issues. The endianness is a bit annoying but it's fine and I did decide to use reference counting for strings in the end.

The memory management is a simple reference counter. It is nothing complicated and makes the library much more straight-forward when passing objects everywhere, as no-doubt will be the case with a library like this. You can see what I've done by looking at the CBObject files. Look at the testCBAddress.c file for an example of using a CBAddress object with memory management included.

Ah gotcha. I feel that though, retain/release is at least 10 times easier than trying to place deallocs correctly (which is sometimes impossible). You mightcould use an already-been-designed C garbage collector instead and save yourself the trouble in that case. Refcounting is very inefficient when many objects die, and overall doesn't perform any better than a GC, which is zero effort and no chance of screwing anything up Tongue.

Honestly, weak-typed imperative languages suck for memory management x.x

I'm So Meta, Even This Acronym
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 17, 2012, 05:49:53 PM
 #40

Using a tracing garbage collector might increase performance a tiny amount but at the cost of portability. It would add a dependency to the library that I don't want. The reference counting is simple enough in my opinion. Reference counting can be optimised by removing redundant retain/release calls and in fact might make the library easier to use. Here is an example of what I might do:

At the moment, as shown in the testCBAddress code, you make a CBAddress object like this:

Code:
CBString * addstr = CBNewStringByCopyingCString("1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9");
CBAddress * add = CBNewAddressFromString(addstr, false, &events, &dep);
CBGetObjectVT(addstr)->release(&addstr);

As you can see the CBString is no longer needed by the calling function, so it is released. This is a redundant release. The CBNewAddressFromString constructor can be modified so that it assumes control over the reference of the calling function. That means the constructor no longer retains the object. This removes the need for a retain and a release call. THe code could be written like this:

Code:
CBAddress * add = CBNewAddressFromString(CBNewStringByCopyingCString("1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9"), false, &events, &dep);

The CBAddress object makes a reference to the CBString, assuming the calling function no longer needs it. However in the cases where the function will need the CBString, something like this is needed:

Code:
CBString * addstr =  CBNewStringByCopyingCString("1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9");
CBGetObjectVT(addstr)->retain(addstr);
CBAddress * add = CBNewAddressFromString(addstr, false, &events, &dep);
// Continue using CBString

In fact if caching is enabled by passing true to the second argument in CBNewAddressFromString, then the calling function doesn't need to call retain on the CBString until the CBAddress is released, since the CBAddress would keep the CBString. This level of optimisation is of-course more confusing and probably practically worthless. It could also introduce problems with future compatibility. I don't want my library to guarantee that an object will hold another object for it's lifetime.

But the method of making the calling function responsible for retaining an object if it needs it after passing it to another object could make some things easier. Perhaps I could make special constructors like CBNewAddressByTakingString which assumes the calling function no longer needs the reference. And these constructors can be made for particular objects where it may be common that another object will no longer be needed after passing it to the object. So you'd have something like this:

Code:
CBAddress * add = CBNewAddressByTakingString(CBNewStringByCopyingCString("1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9"), false, &events, &dep);

Maybe calling it CBNewAddressByStealingString would avoid any confusion. I will of-course make such things as clear as possible in the documentation.

So question: Should all constructors and methods that take objects not retain them so that the calling function needs to have a retain if necessary, should it only be some methods/constructors (Clearly named to avoid confusion), or none like it is now?

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Haplo
Full Member
***
Offline Offline

Activity: 168



View Profile
May 17, 2012, 08:33:41 PM
 #41

Eh, I can understand not wanting to include any extra libraries, but I'm not so sure about GC having any effect on portability (except maybe on card-readers, which is hard to predict).

I see what you're trying to do with removing some unnecessary refcounts, which has been done (probably more effectively) in automatic reference counting systems, but I wouldn't recommend trying to do it manually. Basically, the case you listed (passing control of an object completely to another object with zero effect on count) is rarer than the more common case where an object is passed temporarily to an object which quickly uses and releases it with no effect on refcount (ie the original owner maintains control). If you want to cut out code like that, I'd suggest doing it on an expendable copy of your code later, and just counting everything for now. You're more likely to break stuff otherwise, which defeats the point of manual refcounting.


Btw, refcounting isn't only inefficient for many dead objects because of extra unnecessary counts; tracing is just that much more effective/efficient on dead objects. Age-oriented GC exploits that by having the young generation collected by a tracer and the old generation automatically refcounted. Personally I wouldn't go for all that complexity for a bitcoin library. It's not like you're decoding video for playback or anything, so a stop-the-world, copying tracing collector would be pretty efficient and work just fine, if you decide to go that direction.

I'm So Meta, Even This Acronym
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 17, 2012, 11:04:10 PM
 #42

Tracing GC would not be good for portability with embedded devices right?
I really can't see how reference counting is a major performance issue at all since the retain/release calls are in-between a lot of other code which is much more intensive.


Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Haplo
Full Member
***
Offline Offline

Activity: 168



View Profile
May 18, 2012, 12:08:13 AM
 #43

Tracing GC would not be good for portability with embedded devices right?

It depends on if your embedded application has actual real-time requirements from bitcoin. I don't see bitcoin having any such requirements since most of what it does is networking and/or churning away at validating the blockchain, updating current balances and so on. Very little of it is actual user-interaction, which might be taken care of by a separate gui layer with different memory management.

If you actually need real-time performance for any reason, you'd have to use either something which has no pauses, ie refcounting with a freelist/standard C alloc/dealloc, or a concurrent GC. Concurrent GCs perform worse overall than stop-the-world, but they have minimal pauses so that things like real-time video chat or a phone's normal calling routines aren't choppy.

I know that they don't use garbage collection in the iOS obj-c framework, but I don't know what other phone developers do. I know that it's possible to compile GC for an embedded system, but there may be other reasons against it's use. Theoretically it shouldn't break things though, since the OS kernel being preemptible and using a realtime scheduler are the main mechanisms for enabling real-time performance. I would ask someone who actually has some experience with programming for phones. They would know better than I do.

I really can't see how reference counting is a major performance issue at all since the retain/release calls are in-between a lot of other code which is much more intensive.

Memory management is one of the biggest overheads incurred by any and every program. It also performs no useful work in regards to a program's 'business' code. As a result, every single instruction of MM overhead is significant WRT overall performance. Just to give you the gist, experiments with region-based memory management and compile time GC in mercury showed speedups of 100% or more for some applications, and no less than 33% speedup for any, while using 66% less memory (or less) than with standard GC. Of course, that was a declarative language with strong-static typing, modes, determinism, and scope declarations, but I think it should give you a good idea of what to expect with changes to memory management in general.

I'm So Meta, Even This Acronym
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 18, 2012, 01:24:21 AM
 #44

Well what garbage collecting library would you recommend for C (This? http://www.hpl.hp.com/personal/Hans_Boehm/gc/)? Also does the mercury programming language use memory like C does? If it used the stack where memory management doesn't matter, then it could be an OK comparison. Remember only dynamically allocated memory needs to be managed.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Haplo
Full Member
***
Offline Offline

Activity: 168



View Profile
May 18, 2012, 01:43:29 AM
 #45

Well what garbage collecting library would you recommend for C (This? http://www.hpl.hp.com/personal/Hans_Boehm/gc/)? Also does the mercury programming language use memory like C does? If it used the stack where memory management doesn't matter, then it could be an OK comparison. Remember only dynamically allocated memory needs to be managed.

Declarative languages are strange compared to imperative languages like C. By default, prolog (and its derivatives like mercury) use what they call a WAM, which basically means everything is allocated to the stack. However, mercury also has the option to use a Boehm collector, and does allocate things to the heap. I don't know everything about how that works in mercury, and unfortunately as awesome a language as it is, it isn't well documented (and I haven't gotten around to digging through the installed examples yet).

Also, declarative languages using a WAM can generally end up with poor memory usage due to the FILO nature of stacks, and thus started using garbage collectors to collect memory on the stack o.0.

I'm not sure if there are collectors available for C besides the boehm. I did, however, find a nifty article on using the boehm library and getting good mileage from it: from linuxjournal.

EDIT: And they get better performance out of the Boehm than they do from malloc/dealloc.. which really surprises me. The only consideration is that you'll want to make sure to disable any compiler optimizations that mess with pointers in your build. The Boehm library is also pretty configurable, so you can make it do whatever you need it to do (ie for embedded systems).

I'm So Meta, Even This Acronym
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 18, 2012, 04:14:27 AM
 #46

Thanks for the feedback, it's made me think a bit. I'm thinking garbage collection may not even be required such that I can make the library without reference counting or tracing GC. I could make one version with GC and one not where objects just have to be freed at the right time.

I'll think more about this later... Should be asleep but I seem to have insomnia. Sad

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
May 18, 2012, 05:09:41 AM
 #47

Some of this is putting the cart before the horse Smiley

You cannot know what is the best allocator until you have an idea of the lifetime and usage pattern of the allocated objects...


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 18, 2012, 05:44:31 AM
 #48

Well this is the issue, what can be figured out now and what has to be fixed later. ;-) If referencing counting can be rejected early on then it makes coding easier; no need to completely re-implement memory management at a later stage.

I'd prefer to try and get it right to begin with but of-course if I'm wrong to begin with then it will just have to be changed.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Haplo
Full Member
***
Offline Offline

Activity: 168



View Profile
May 19, 2012, 06:41:09 AM
 #49

Well this is the issue, what can be figured out now and what has to be fixed later. ;-) If referencing counting can be rejected early on then it makes coding easier; no need to completely re-implement memory management at a later stage.

I'd prefer to try and get it right to begin with but of-course if I'm wrong to begin with then it will just have to be changed.

Well, reference counting does have one advantage. After reading through that link I sent you I realized that for C you still have to set your pointers to null in order to tell the GC that your pointer is dead. In terms of assuring correctness you'd still need reference counting to avoid dangling pointers x.x.

Weak types = no automatic GC. I'd suggest another language but most languages suck Tongue.

I'm So Meta, Even This Acronym
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 19, 2012, 07:51:36 PM
 #50

No, you wont need reference counting to get rif of dangling pointers, you simply replace calls to "release" with a NULL assignment.

Also I guess the CG will also collect garbage when pointers go out of scope. Not sure if that is true but it makes sense so.

Maybe tomorrow or the next day I might do a test with ref counting vs tracing CG vs manual malloc/free placement. I might do a test where a loop creates CBAddress objects with random data until it finds an address beginning with particular characters. That would test a mixture of object creation/destruction and algorithm execution which may be a fair test even though the test would never be a proper representation of the final library.

But I'm not doing any more today.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Haplo
Full Member
***
Offline Offline

Activity: 168



View Profile
May 20, 2012, 12:06:31 AM
 #51

No, you wont need reference counting to get rif of dangling pointers, you simply replace calls to "release" with a NULL assignment.

Also I guess the CG will also collect garbage when pointers go out of scope. Not sure if that is true but it makes sense so.

Maybe tomorrow or the next day I might do a test with ref counting vs tracing CG vs manual malloc/free placement. I might do a test where a loop creates CBAddress objects with random data until it finds an address beginning with particular characters. That would test a mixture of object creation/destruction and algorithm execution which may be a fair test even though the test would never be a proper representation of the final library.

But I'm not doing any more today.

Ah I think I get it now. As long as you always copy pointers when passing values into functions, then you get the same effect as refcounting. I would definitely like to see a test showing that that has lower overhead than dealloc/refcounting o.0. It looks to me like you'd be doing the same work as with refcounting, except without any explicit calls to dealloc.

I don't know about scope, but letting pointers fall out of scope is bad form anyway Tongue.

I'm So Meta, Even This Acronym
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 20, 2012, 11:36:32 AM
 #52

A tracing GC looks through the stack finding pointers to memory and if that memory is an object which the GC is responsible for it recursively does the same think until it finds no more reachable objects. The rest of the objects are collected. So all you need to ensure is that when you are done with a reference you point the reference somewhere else. This can indeed be done by pointing to NULL but is not always necessary as you may reuse pointers for deprecate objects where you reassign pointers to new objects. And as I've said a reference can go out of scope. No need to assign NULL to a pointer that becomes unreachable afterwards.

 

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 20, 2012, 10:37:34 PM
 #53

Perhaps using the garbage collector would be nice since I'm now struggling with a memory leak made by CBCreateAddressVT. The virtual function table should get freed from everything I see. :-( Once again I've forgotten to pass a pointer by reference (ie. pointer to a pointer). :-( Easy enough to fix.

No idea how to install libgc on OSX. Sad

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Haplo
Full Member
***
Offline Offline

Activity: 168



View Profile
May 21, 2012, 12:53:35 AM
 #54

As I said, C is not the best language for memory management x.x.

I'm So Meta, Even This Acronym
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 21, 2012, 02:29:41 AM
 #55

Well it doesn't matter that I cannot do any test. While it would be interesting Boehms GC may be insecure and reference counting is not too much of a headache using the right tools; it didn't take me long to fix the memory leaks. So now I'll move onto just testing the virtual functions vs ordinary functions before moving on to the script.

I was able to test removing the redundant virtual functions and it gives a speedup of about 3% which is not worth the change. Now that I did that I'll move onto the script.

Edit: Just have the arithmetic and cryptography codes to implement in the script interpreter. If anyone wants to help, the script interpreter certainly does need scrutinising.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 23, 2012, 10:49:07 PM
 #56

The script interpreter is implemented except for the signature checking codes. If anyone wants to help, the interpreters should be scrutinised and in particular checked against the bitcoin protocol to ensure it operates correctly. This is it:

https://github.com/MatthewLM/cbitcoin/tree/master/cbitcoin/src/structures/CBScript

The interpreter is here:

https://github.com/MatthewLM/cbitcoin/blob/master/cbitcoin/src/structures/CBScript/CBScript.c#L105

All documentation is in the header file.

The tests are in this file:

https://github.com/MatthewLM/cbitcoin/blob/master/cbitcoin/test/testCBScript.c

Next I'll implement the transaction structures before moving on to signature checking. Then I can implement the transaction validation and move onto blocks.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Gavin Andresen
Legendary
*
qt
Offline Offline

Activity: 1652


Chief Scientist


View Profile WWW
May 24, 2012, 03:42:16 PM
 #57

Your unit tests look like they'll miss lots of edge cases.

I've been working on cross-implementation unit tests for Script, and am actually working on more tests today (and am working on a testnet reset that will embed the "should validate" tests into the testnet block chain).

JSON format tests are here:
  https://github.com/bitcoin/bitcoin/tree/master/src/test/data

(read by https://github.com/bitcoin/bitcoin/blob/master/src/test/script_tests.cpp )

How often do you get the chance to work on a potentially world-changing project?
realnowhereman
Hero Member
*****
Offline Offline

Activity: 504



View Profile
May 24, 2012, 04:01:34 PM
 #58

Your unit tests look like they'll miss lots of edge cases.

I've been working on cross-implementation unit tests for Script, and am actually working on more tests today (and am working on a testnet reset that will embed the "should validate" tests into the testnet block chain).

JSON format tests are here:
  https://github.com/bitcoin/bitcoin/tree/master/src/test/data

(read by https://github.com/bitcoin/bitcoin/blob/master/src/test/script_tests.cpp )

I'm not familiar enough with Satoshi client internals to easily understand what script_tests is doing.  Could you confirm my understanding of the format of the two JSON files?

Obviously it's an array of arrays, but some of the inner arrays have two elements and some have three.  I guess the first two elements are the two component scripts?  Which one is which?  Is the third element simply an error message to be issued if that test fails?  What is considered a fail -- i.e. is the script meant to always reduce to one answer and that answer should be TRUE (or FALSE depending on which file is used)?

1AAZ4xBHbiCr96nsZJ8jtPkSzsg1CqhwDa
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 24, 2012, 05:17:42 PM
 #59

Thanks Gavin. I do need to implement more tests, I will do this at a later time, first I need to implement the transaction structures properly and OP_CHECKSIG. I'll have to use some of the tests you posted!

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
kokjo
Legendary
*
Offline Offline

Activity: 1050

You are WRONG!


View Profile
May 24, 2012, 07:45:06 PM
 #60

sup

"The whole problem with the world is that fools and fanatics are always so certain of themselves and wiser people so full of doubts." -Bertrand Russell
Gavin Andresen
Legendary
*
qt
Offline Offline

Activity: 1652


Chief Scientist


View Profile WWW
May 25, 2012, 01:15:06 AM
 #61

I'm not familiar enough with Satoshi client internals to easily understand what script_tests is doing.  Could you confirm my understanding of the format of the two JSON files?

The format is a list of pairs, where the first item is the scriptSig required to spend and the second is the scriptPubKey. The third item is ignored (useful for comments).

script_valid.json contains only valid scriptSig/scriptPubKey pairs.
script_invalid.json contains only invalid scriptSig/scriptPubKey pairs.

valid/invalid are defined by the rules of transaction validation, and the unit test actually constructs transactions and runs the verification routine on them to make sure that they succeed or fail, as expected.


How often do you get the chance to work on a potentially world-changing project?
realnowhereman
Hero Member
*****
Offline Offline

Activity: 504



View Profile
May 25, 2012, 08:43:32 AM
 #62

Thanks.

I note that the difficult ones, CHECKSIG, etc, aren't included.  I guess that's because there is no transaction for the script to operate on?

Is there a plan to include those difficult ones in the unit tests later?  Or will they be part of the transaction unit tests?

I ask only so I know whether to consider creating a fake transaction for the test scripts to run inside?

1AAZ4xBHbiCr96nsZJ8jtPkSzsg1CqhwDa
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 25, 2012, 02:30:15 PM
 #63

The transaction tests are here: https://github.com/bitcoin/bitcoin/blob/master/src/test/transaction_tests.cpp

Hopefully I'll get around to starting transaction validation next week, though I'll be busy over the next few days.

If anyone would like to contribute, improving the test code would certainly be helpful.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
May 30, 2012, 06:04:10 PM
 #64

I just had a brain wave. There is no need to do copying of data for serialisation and de-serialisation of bitcoin messages like bitcoinj does. I can store the serialised byte data and just access that through pointers. Should be more efficient.

It's problematic trying to use a Java library as a basis for a C library. :-)

I need to store integers separately in the systems endianness.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
June 02, 2012, 10:49:04 PM
 #65

Now I've implemented transaction serialisation and deserialisation. Here is the test code for that: https://github.com/MatthewLM/cbitcoin/blob/master/cbitcoin/test/testCBTransaction.c

Next I'll move onto transaction signing and validation. After that I can work on the blocks, block chain, block validation and then the networking.

Once again, if anyone wishes to aid in anyway the best way is to scrutinise the code for bugs and problems and improve the tests.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
June 03, 2012, 12:35:46 PM
 #66

(registering to this thread)

MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
June 03, 2012, 04:59:57 PM
 #67

I'm assuming with SIGHASH_SINGLE the output scripts are blank but the VarInt is still there, so each output up to before the input index will have 9 bytes, 8 for the value and 1 for the VarInt. Can anyone verify that? Or is it just the value with 8 bytes? Trying to track it down in the C++ client was taking too long.

Also how are output value signs represented as serialised data? Similar to the script, except as little endian? Twos compliment? The serialisation code in the C++ client is just too over my head for me to understand.

Edit: Apparently the C++ client is platform dependent on the x64 architecture meaning the signed integers should be twos compliment and little endian.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
June 04, 2012, 06:49:15 AM
 #68

I'm assuming with SIGHASH_SINGLE the output scripts are blank but the VarInt is still there, so each output up to before the input index will have 9 bytes, 8 for the value and 1 for the VarInt. Can anyone verify that? Or is it just the value with 8 bytes? Trying to track it down in the C++ client was taking too long.

Also how are output value signs represented as serialised data? Similar to the script, except as little endian? Twos compliment? The serialisation code in the C++ client is just too over my head for me to understand.

I'm not sure I understand exactly your problem, but maybe it could help if I tell you that I had lots of difficulty trying to understand the bitcoin serialisation protocol, until I read Gavin's bitcointools python code.  It is much easier to understand than C++.  It helped me a lot for my perl library.

MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
June 04, 2012, 01:05:52 PM
 #69

Unfortunately from what I see it doesn't help with my SIGHASH issue. I made a question here: http://bitcoin.stackexchange.com/questions/3890/for-sighash-single-do-the-outputs-other-than-at-the-input-index-have-8-bytes-or

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
fellowtraveler
Sr. Member
****
Offline Offline

Activity: 440


View Profile
June 04, 2012, 02:22:05 PM
 #70

If I had to write something in straight-C, I would use the D language.

co-founder, Monetas
creator, Open-Transactions
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
June 04, 2012, 03:03:42 PM
 #71

Except D isn't C.  Smiley

OpenSSL is terribly undocumented. I'm currently looking at checking transaction ECDSA signatures using:

Quote
bool (*ecdsaVerify)(u_int8_t *,u_int8_t *,const u_int8_t *); /**< Function for verifying ECDSA signatures for a 70 byte signature against a 32 byte hash using a 66 byte ECDSA public key. */

The first parameter is the 70 byte signature, the second is the 32 byte SHA256 digest and the last is the public key data. For testing I need to implement this function. I'm using OpenSSL for the tests but it doesn't document converting public key bytes to an EC_KEY object. Not good at all! But I looked at the bitcoin source code and I saw o2i_ECPublicKey so I'll use that...

Quote
bool ecdsaVerify(u_int8_t * signature,u_int8_t * hash,const u_int8_t * pubKey){
   EC_KEY * key = EC_KEY_new_by_curve_name(NID_secp256k1);
   o2i_ECPublicKey(&key, &pubKey, 66);
   ECDSA_verify(0, hash, 32, signature, 70, key);
}

Hopefully this is the right way. Am I right thinking that the signatures are 70 bytes and the public keys are 66 bytes?

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Gavin Andresen
Legendary
*
qt
Offline Offline

Activity: 1652


Chief Scientist


View Profile WWW
June 04, 2012, 03:17:45 PM
 #72

Hopefully this is the right way. Am I right thinking that the signatures are 70 bytes and the public keys are 66 bytes?

Signatures are BER-encoded data structures, and can be an arbitrary number of bytes (if they're DER-encoded, which is the strict subset of BER encoding, then they're 70-something bytes).

Public keys are either 33 or 65 bytes (not counting the "push the next N bytes onto the stack" CSCript opcode).

I've got to say you make me nervous; you seem to be following a "make it work for a couple of test cases then move on" style of development, which is a bad way to create a secure, robust codebase.

PS: I sympathize with you RE: OpenSSL's lack of documentation....


How often do you get the chance to work on a potentially world-changing project?
kokjo
Legendary
*
Offline Offline

Activity: 1050

You are WRONG!


View Profile
June 04, 2012, 03:23:33 PM
 #73

If I had to write something in straight-C, I would use the D language.
if i had to use X, i would use something different and unrelated Y.

WTF? the reason for using C is that its simple, small and portable. which i suspect that D is not.

"The whole problem with the world is that fools and fanatics are always so certain of themselves and wiser people so full of doubts." -Bertrand Russell
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
June 04, 2012, 03:42:18 PM
 #74

@Gavin Andresen

Thanks for the information. Don't get worried about the tests. I'm fully aware they are incomplete. The tests can always be fully implemented later on. I welcome anyone that is interested to contribute to the testing code.

So I'm following "make it work for a couple of test cases then move on and come back later to fully implement the tests".

I'll also need to come back later and complete the documentation. I'm documenting to some degree as I go along but there are parts that I'll be better able to document later.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
June 06, 2012, 02:37:37 PM
 #75

New problem: http://stackoverflow.com/questions/10906524/openssl-i2o-ecpublickey-not-providing-any-bytes

Any ideas?

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
fellowtraveler
Sr. Member
****
Offline Offline

Activity: 440


View Profile
June 06, 2012, 03:05:25 PM
 #76

If I had to write something in straight-C, I would use the D language.
if i had to use X, i would use something different and unrelated Y.

WTF? the reason for using C is that its simple, small and portable. which i suspect that D is not.

Anyone who writes C programs would do well to research the D language.

Ditto for C++, but for entirely different reasons.

D is the perfect replacement for both (each for different reasons.)

I have programmed C and C++ for over 20 years, everything from controller cards to Windows, to Linux, to Mac, to Android, and FreeBSD. (Do your own research, just giving my opinion. Take it for whatever it's worth.)

-FT





co-founder, Monetas
creator, Open-Transactions
kokjo
Legendary
*
Offline Offline

Activity: 1050

You are WRONG!


View Profile
June 06, 2012, 03:35:04 PM
 #77

If I had to write something in straight-C, I would use the D language.
if i had to use X, i would use something different and unrelated Y.

WTF? the reason for using C is that its simple, small and portable. which i suspect that D is not.

Anyone who writes C programs would do well to research the D language.

Ditto for C++, but for entirely different reasons.

D is the perfect replacement for both (each for different reasons.)

I have programmed C and C++ for over 20 years, everything from controller cards to Windows, to Linux, to Mac, to Android, and FreeBSD. (Do your own research, just giving my opinion. Take it for whatever it's worth.)

-FT
Language fanboy detected.

hey, if i had to write something in C, i would use python.

"The whole problem with the world is that fools and fanatics are always so certain of themselves and wiser people so full of doubts." -Bertrand Russell
fellowtraveler
Sr. Member
****
Offline Offline

Activity: 440


View Profile
June 06, 2012, 03:45:22 PM
 #78

Language fanboy detected.

hey, if i had to write something in C, i would use python.

Fair enough. However, Python was not specifically written as a replacement for C and C++.

D *was* specifically written as a replacement for C and C++.

Therefore, I think you are comparing apples and oranges.

For example, D, like C, would be an appropriate tool for writing a system-level driver.  Python, on the other hand, most likely would NOT be an appropriate tool for such a task.

(Apples and oranges...)



co-founder, Monetas
creator, Open-Transactions
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
June 06, 2012, 09:15:35 PM
 #79

Does anyone know how to install OpenSSL 1.0.1c on OSX? After changing to the unarchived directory I ran:

Code:
./config
make
make test
sudo make install

But I still have the old shipped version.

I had a feeling updating might fix my i2o_ECPublicKey problem.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
June 16, 2012, 07:29:44 PM
 #80

The OpenSSL problems are solved. I've worked on a script compiler (text to byte code) and this is the syntax:

Code:
<digit> = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" | "a" | "b" | "c" | "d" | "e" | "f" | "A" | "B" | "C" | "D" | "E" | "F"
 <hex_digits> ::= <digit> | <digit> <hex_digits>
 <hex> ::= "0x" <hex_digits>
 <operation_name> = "0" | "FALSE" | "1NEGATE" | "RESERVED" | "1" | "TRUE" | "2" | "3" | "4" |  "5" | "6" | "7" | "8" | "9" | "10" | "11" | "12" | "13" | "14" | "15" | "16" | "NOP" | "VER" | "IF" | "NOTIF" | "VERIF" | "VERNOTIF" | "ELSE" | "ENDIF" | "VERIFY" | "RETURN" | "TOALTSTACK" | "FROMALTSTACK" | "2DROP" | "2DUP" | "3DUP" | "2OVER" | "2ROT" | "2SWAP" | "IFDUP" | "DEPTH" | "DROP" | "DUP" | "NIP" | "OVER" | "PICK" | "ROLL" | "ROT" | "SWAP" | "TUCK" | "CAT" | "SUBSTR" | "LEFT" | "RIGHT = 0x81" | "SIZE" | "INVERT" | "AND" | "OR" | "XOR" | "EQUAL" | "EQUALVERIFY" | "RESERVED1" | "RESERVED2" | "1ADD" | "1SUB" | "2MUL" | "2DIV" | "NEGATE" | "ABS" | "NOT" | "0NOTEQUAL" | "ADD" | "SUB" | "MUL" | "DIV" | "MOD" | "LSHIFT " | "RSHIFT" | "BOOLAND" | "BOOLOR" | "NUMEQUAL" | "NUMEQUALVERIFY" | "NUMNOTEQUAL" | "LESSTHAN" | "GREATERTHAN" | "LESSTHANOREQUAL" | "GREATERTHANOREQUAL" | "MIN" | "MAX" | "WITHIN" | "RIPEMD160" | "SHA1" | "SHA256" | "HASH160" | "HASH256" | "CODESEPARATOR" | "CHECKSIG" | "CHECKSIGVERIFY" | "CHECKMULTISIG" | "CHECKMULTISIGVERIFY" | "NOP1" | "NOP2" | "NOP3" | "NOP4" | "NOP5" | "NOP6" | "NOP7" | "NOP8" | "NOP9" | "NOP10"
 <operation> ::= "OP_" <operation_name>
 <lines> ::= "\n" | "\n" <lines>
 <seperator> ::= <lines> | " "
 <operation_or_hex> ::= <operation> | <hex>
 <seperated_operation_or_hex> ::= <operation_or_hex> <seperator>
 <script_body> ::= <seperated_operation_or_hex> | <seperated_operation_or_hex> <script_body> | ""
 <lines_or_none> ::= <lines> | ""
 <operation_or_hex_or_none> ::= <operation_or_hex> | ""
 <script> ::= <lines_or_none> <script_body> <operation_or_hex_or_none>

What do people think? The compiler automatically converts hex into push operations.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
jim618
Legendary
*
Offline Offline

Activity: 1708



View Profile WWW
June 16, 2012, 08:31:05 PM
 #81

I just thought I would mention that bitcoinj has moved it's wallet file format over to protobuf.
There is a pure C protobuf project here:

http://code.google.com/p/protobuf-c/

I am just adding support for the new format into MultiBit and the wallet files are much smaller and so load in double quick time.

I am not sure where you have got to yet in your work on wallet storing / reading but wanted to mention it. Potentially cbitcoin and bitcoinj could have byte-compatible wallets.

There is a proto file representing all the messages already all done with the key, Bitcoin address etc structure - you could directly produce your cbitcoin wallet store/ load code from that.

Edit: The proto file is here:
http://code.google.com/p/bitcoinj/source/browse/core/src/bitcoin.proto?name=release-0.5

MultiBit HD   Lightweight desktop client.                    Bitcoin Solutions Ltd   Bespoke software. Consultancy.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
June 16, 2012, 09:02:45 PM
 #82

For cbitcoin I do not plan to implement wallet functionality. The reason is because there are many ways to store keys and so it makes sense to add that functionality onto cbitcoin if making a client with it. bitcoinj is a bit different in that it's a library more specifically for clients using SPV. cbitcoin will be a bit more low level, providing powerful access to bitcoin features. I want cbitcoin to be more diverse.

I plan to make a client that will work on-top of cbitcoin which may include a higher level library or some sort of API perhaps similar to bitcoind. I'll get to that eventually and may use protocol buffers, I don't know yet.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
June 18, 2012, 08:54:49 PM
 #83

I moved the topic since the library isn't technically an alternative bitcoin client.

I'm currently doing some tests on transactions signing and OP_CHECKSIG/OP_CHECKMULTISIG. I've done over 250 tests for the script interpreter and that works nicely.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
July 04, 2012, 10:51:05 PM
 #84

I'm currently implementing the message structures with serialisation code for each structure. After that I'll implement code used to manage network connections, to pass on de-serialised messages from nodes and to send messages to nodes. Then I'll work on the blockchain validation code.

I want this library to be low level so that the library wont implement the network responses itself and relay addresses, relay blocks, download and verify the block chain (There will be validation code but it will need configuring to the requirements of the programmer) etc. Instead the library will give the interface for programmers to build a node to their specifications.

But don't worry if you think a client will not come from this. I plan to write a client library on top of cbitcoin, and GUI applications will come too. Do not get over excited since a stable/secure client is far off as there will need to be extensive testing on top of the development. Also remember this library should become suitable for various bitcoin projects and not only basic clients.

Once again, if anyone wishes to contribute to this project it would be very helpful (and easy) for people to contribute with the testing. At the moment there are two obvious things which need more testing. The "testCBByteArray" file needs tests for all the functions. The test files for the structures that inherit CBMessage need tests for fail cases where the de-serialisation and serialisation is supposed to fail (And give an appropriate error message). More script tests or any tests elsewhere are great.

I've given up developing iPhone apps to put a lot of my time towards this, so hopefully it will come along reasonably quickly.

There will be inevitable costs to this project. If anyone would like to help this project financially, donations are very welcome: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9

Thank you.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Haplo
Full Member
***
Offline Offline

Activity: 168



View Profile
July 05, 2012, 02:44:02 AM
 #85

I'll send a BTC your way in a day or so. I'd have probably donated more by now but I really had no idea how many retarded hoops I was going to have to jump through to buy BTC  Undecided

I'm So Meta, Even This Acronym
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
July 05, 2012, 03:51:27 PM
 #86

Thank you very much. Sorry buying bitcoin isn't as easy as it would ideally be.  Wink

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
July 13, 2012, 06:55:34 PM
 #87

I've implemented all the message objects except the IP transaction ones which I wont implement. The message objects need testing for the cases in which they are supposed to fail and need to be scrutinised.

However, for now, I'm going to work on the code for the network. This will be the code that can send messages back and forth to and from the network and also some code that automates some common processes such as relaying network addresses and finding new nodes etc. I'm thinking I'll make a structure called CBNetworkCommunicator that works on a events based system:

eg. With "void (* onBlockRecieved)(CBNetworkCommunicator *,CBNetworkAddress *,CBBlock *)" a block is de-serialised from the network and sent to the function pointed to by onBlockRecieved with the CBNetworkCommunicator object and CBNetworkAddress where the block was sent from.

I will need to add networking and threading functions to CBDependencies.h for connecting to nodes and managing connections in separate threads.

After this I'll implement the block validation. I'll add functions to CBBlock.h/.c for this and use an CBEvents object so the validation code can ask other functions for block/transaction information (eg. "CBTransactionOutput * (* getPrevOutput)(CBByteArray * transactionHash,u_int32_t outputIndex)"). The programmer using the library will be responsible for the storage of blocks and retrieving the information for the validator. The programmer will also be responsible for the organisation of the block chain, cbitcoin will only validate single blocks at a time. cbitcoin should have code for the checking of block difficulty, timestamps etc. but the programmer using cbitcoin may have to pass some of this information around. This is because of the configurable/modular nature of cbitcoin (Remember that cbitcoin is supposed to be relatively low-level).

Shortly after that I should be able to put cbitcoin into the alpha development stage and I will move onto the higher level client library and applications for that (More info later). Developing a client library on top of cbitcoin may highlight issues, so I will begin proper testing after developing the client library. Both libraries and the client applications can be tested together and will eventually enter the beta testing stage. This stage has a focus on public testing but people will be free to help test the code during the alpha stage too. Eventually the code will be considered secure and stable for final release.

I've been looking a little into independent software reviews. It seems most software is tested internally and there is little interest in external auditing of software. I have the feeling that's because it would be very expensive. Does anyone know much about external auditing of software?

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
July 15, 2012, 11:18:13 PM
 #88

I've figured out how cbitcoin will handle sockets. cbitcoin will have dependencies for events-based, non-blocking sockets. It should work with libevent though I need to learn more about how to use lubevent to make sure my library is compatible. My threading dependencies should work with POSIX threads nicely.

Here are the threading and socket dependencies as I have them so far (These are weakly linked function prototypes):

Code:
// THREADING DEPENDENCIES

/*
 @brief Starts a new thread. The thread should be joinable.
 @param threadID The ID for the thread to be set with newly allocated memory, used to refer to the threads in other functions.
 @param startFunction The function to be called as the starting point of the new thread.
 @param args A pointer to the arguments for the starting function.
 @returns true if the thread was successfully created and false if the thread could not be created.
 */
bool CBStartThread(void * threadID,void * (*startFunction)(void *),void * args);
/*
 @brief Ends the calling thread.
 */
void CBEndThread(void);
/*
 @brief Waits for a thread to terminate.
 @param threadID The ID for the thread.
 */
void CBThreadJoin(void * threadID);
/*
 @brief Frees any resources related to a thread, including the ID.
 @param threadID The ID for the thread.
 */
void CBFreeThread(void * threadID);
/*
 @brief Creates a new mutex.
 @param mutexID The mutex id to be set with newly allocated new memory.
 @returns true if the mutex was successfully created and false if the mutex could not be created.
 */
bool CBNewMutex(void * mutexID);
/*
 @brief Frees a new mutex including the ID. It is assured that no thread locks are active before this is called as threads should be canceled.
 @param mutexID The mutex id to free.
 */
void CBFreeMutex(void * mutexID);
/*
 @brief Obtains a mutex lock when next available.
 @param mutexID The mutex id.
 */
void CBMutexLock(void * mutexID);
/*
 @brief Unlocks a mutex for other threads.
 @param mutexID The mutex id.
 */
void CBMutexUnlock(void * mutexID);

// NETWORKING DEPENDENCIES

/*
 @brief Creates a new TCP/IPv6 socket and binds it to the given IPv6 address and port. The socket should use a non-blocking mode.
 @param socketID The socket id to be set with newly allocated new memory.
 @returns true if the socket was successfully created and false if the socket could not be created.
 */
bool CBNewSocket(void * socketID);
/*
 @brief Binds the local address and a port number to a socket.
 @param socketID The socket id
 @param port The port to bind to.
 @returns true if the bind was sucessful and false otherwise.
 */
bool CBSocketBind(void * socketID,u_int16_t port);
/*
 @brief Begin connecting to an external host with a socket. This should be non-blocking.
 @param socketID The socket id
 @param IP 16 bytes for an IPv6 address to connect to.
 @param port Port to connect to.
 @param onConnect Pointer to the function that should be called when the connection has been sucessful.
 @param onConnectArg Pointer to send to "onConnect".
 @returns true if the function was sucessful and false otherwise.
 */
bool CBSocketConnect(void * socketID,u_int8_t * IP,u_int16_t port,void (*onConnect)(void *),void * onConnectArg);
/*
 @brief Begin listening for incomming connections on a bound socket. This should be non-blocking.
 @param socketID The socket id
 @param maxConnections The maximum incomming connections to allow.
 @param onAccept Pointer to the function that should be called when the socket is ready for accepting an incomming connection.
 @param onAcceptArg Pointer to send to "onAccept".
 @returns true if function was sucessful and false otherwise.
 */
bool CBSocketListen(void * socketID,u_int16_t maxConnections,void (*onAccept)(void *),void * onAcceptArg);
/*
 @brief Accepts an incomming connections on a bound socket. This should be non-blocking.
 @param socketID The socket id
 @param connectionSocketID A socket id for a new socket for the connection.
 @param IP 16 bytes to be set by the function for the IP of the incomming connection.
 @returns true if function was sucessful and false otherwise.
 */
bool CBSocketAccept(void * socketID,void * connectionSocketID,u_int8_t * IP);
/*
 @brief Sets a function pointer for the event where a socket is available for sending data.
 @param socketID The socket id
 @param onCanSend The function to call for the event.
 @param onCanSendArg A pointer to pass to the "onCanWrite" function.
 @returns true if function was sucessful and false otherwise.
 */
bool CBSocketCanSendEvent(void * socketID,void (*onCanSend)(void *),void * onCanSendArg);
/*
 @brief Sets a function pointer for the event where a socket is available for receiving data.
 @param socketID The socket id
 @param onCanReceive The function to call for the event.
 @param onCanWriteArg A pointer to pass to the "onCanWrite" function.
 @returns true if function was sucessful and false otherwise.
 */
bool CBSocketCanReceiveEvent(void * socketID,void (*onCanReceive)(void *),void * onCanReceiveArg);
/*
 @brief Sends data to a socket. This should be non-blocking.
 @param socketID The socket id to send to.
 @param data The data bytes to send.
 @param len The length of the data to send.
 @returns The number of bytes actually sent and any number less than 0 on failure that suggests further data cannot be sent.
 */
int32_t CBSocketSend(void * socketID,u_int8_t * data,u_int32_t len);
/*
 @brief Receives data from a socket. This should be non-blocking.
 @param socketID The socket id to receive data from.
 @param data The data bytes to write the data to.
 @param len The length of the data.
 @returns The number of bytes actually written into "data", 0 on connection closure and any number less than 0 on failure.
 */
int32_t CBSocketReceive(void * socketID,u_int8_t * data,u_int32_t len);
/*
 @brief Closes a socket. The id should be freed, as well as any other data relating to this socket.
 @param socketID The socket id to be closed.
 */
void CBCloseSocket(void * socketID);

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
July 16, 2012, 08:08:35 PM
 #89

Sockets are driving me mad. Different platforms use completely different ways to represent IPv6 addresses. This includes unix platforms which do not use fully compatible BSD sockets.

So if anyone knows a portable way to connect to IPv6 addresses please look here: http://stackoverflow.com/questions/11511339/portable-ipv6-connections-with-bsd-posix-sockets

Edit: I just sent 35.95515 bitcoins to an address made with cbitcoin. The first practical use of cbitcoin!

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
July 16, 2012, 11:50:34 PM
 #90

Well, the portable representation of an IPv6 address is the network representation.

You may need some platform-specific code to store it in the right structure and struct member.

bitcoin/bitcoin.git (satoshi client) works on linux/mac/win, and supports non-blocking IPv6 sockets.  The system calls and addressing are the same in C and C++.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
July 17, 2012, 12:38:21 AM
 #91

I looked briefly at the bitcoin socket code. It didn't look very nice from what I saw and it was quite disorganised. It looked like the sockets were blocking as I saw a section which looped through reading received data.

Anyway I think I found the solution to setting IPv6 addresses:

Code:
bool CBSocketConnect(u_int64_t socketID,u_int8_t * IP,u_int16_t port){
// Create sockaddr_in6 information for a IPv6 address
struct sockaddr_in6 address;
memset(&address, 0, sizeof(address)); // Clear sockaddr_in6 structure.
address.sin6_family = AF_INET6; // IPv6
memmove(&address.sin6_addr, IP, 16); // Move IP address into place. POSIX standards should allow this.
address.sin6_port = htons(port); // Port number to network order
if (connect((evutil_socket_t)socketID, (struct sockaddr *)&address, sizeof(address)) != 0)
// Connection failure
return false;
return true;
}

Apparently that should work on POSIX compliant platforms. It is not tested yet and may fail on windows.

The reason the socketID is a u_int64_t is to support any integer that it may be for portability. In this implementation it is casting to evutil_socket_t which is a file descriptor on unix/linux and a pointer on windows.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Haplo
Full Member
***
Offline Offline

Activity: 168



View Profile
August 07, 2012, 04:35:49 AM
 #92

sent donatey :X

I'm So Meta, Even This Acronym
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
August 07, 2012, 12:12:13 PM
 #93

sent donatey :X

Thank you very much. Donations currently are sure to go to the outsourcing of graphic design for the GUI clients and possibly outsourcing some programming jobs, though the more voluntary help I can find, the cheaper it will all be. At the moment though I'm the only person working on this.

I'm currently reasonably close to implementing the networking code. Once that is done I'll implement the block validation code and then cbitcoin can reach the alpha testing stage.

I was thinking about making the address addition code synchronous with the network event loop, so that addresses can be added outside the network thread. I think I'll bin that idea though, so you will need to stop the network loop before adding additional addresses. I see little reason why anyone would want to do this. You simply give the CBNetworkCommunicator the addresses before you start it and when closing the application, the CBNetworkCommunicator is stopped and the addresses can be taken out and saved to disk.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
August 11, 2012, 09:10:34 PM
 #94

I'm currently testing the networking code. I'm reasonably near alpha-stage as I've just got to do the block-chain validation code after this.

But I've got an issue. Is anyone good with sockets? connect() is giving me ENOENT:

http://stackoverflow.com/questions/11916719/berkeley-sockets-connect-returns-1-with-errno-set-to-enoent

I'm not sure what that suggests is wrong. I don't think most implementations even give ENOENT errors.

Thank you for any help.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
kokjo
Legendary
*
Offline Offline

Activity: 1050

You are WRONG!


View Profile
August 12, 2012, 07:43:41 AM
 #95

i don't know about OSX. but linux's man page for connect(2). does not say that connect can return ENOENT.

what kind of socket do you make, ipv6 or ipv4? not for the connect() call, but for the socket() call.

"The whole problem with the world is that fools and fanatics are always so certain of themselves and wiser people so full of doubts." -Bertrand Russell
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
August 12, 2012, 01:12:58 PM
 #96

I've confirmed what the error is: LLDB is rubbish, never use it. LLDB does not work with errno properly. Use GDB.

Yes, errno was set as EINPROGRESS all along. It was the stupid debugger that caused all the confusion.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
August 13, 2012, 07:11:56 PM
 #97

Hmm. Now I'm getting an unexpected connection closure via 0 returned from read(). I printed out some of the data to try and understand what was happening:

Connection initiated by A
Connection accepted by B
Connection OK
A sent version header
A sent version payload
B received version header
B received version payload
B sent verack
B sent version header
B send version payload
A received verack
Connection closed

This is pretty odd, I don't know where to start looking... The thing is, neither A or B close the connection explicitly, they both receive 0 from read().

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
kokjo
Legendary
*
Offline Offline

Activity: 1050

You are WRONG!


View Profile
August 14, 2012, 07:16:57 AM
 #98

Hmm. Now I'm getting an unexpected connection closure via 0 returned from read(). I printed out some of the data to try and understand what was happening:

Connection initiated by A
Connection accepted by B
Connection OK
A sent version header
A sent version payload
B received version header
B received version payload
B sent verack
B sent version header
B send version payload
A received verack
Connection closed

This is pretty odd, I don't know where to start looking... The thing is, neither A or B close the connection explicitly, they both receive 0 from read().
A does not receive the version from B?
my guess: its A that closes the connection.

can you post the version packets+headers?

EDIT: what clients are you testing with? both cbitcoin, or is one of them the satoshi client? which one then?

"The whole problem with the world is that fools and fanatics are always so certain of themselves and wiser people so full of doubts." -Bertrand Russell
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
August 14, 2012, 03:15:37 PM
 #99

Thanks for the suggestions. I don't really know what caused it but I did see an issue with attempts to allocate zero memory. I fixed all that by using NULL to represent nothing as I was intending to. After fixing all that and some other issues the problem went away.

Now the handshake works and I've committed an update to github.

EDIT: what clients are you testing with? both cbitcoin, or is one of them the satoshi client? which one then?

At the moment just cbitcoin but I will try connecting to the satoshi client as well.

Edit: Pings work fine. Moving onto address discovery.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
kokjo
Legendary
*
Offline Offline

Activity: 1050

You are WRONG!


View Profile
August 15, 2012, 07:32:50 AM
 #100

Thanks for the suggestions. I don't really know what caused it but I did see an issue with attempts to allocate zero memory. I fixed all that by using NULL to represent nothing as I was intending to. After fixing all that and some other issues the problem went away.

Now the handshake works and I've committed an update to github.

EDIT: what clients are you testing with? both cbitcoin, or is one of them the satoshi client? which one then?

At the moment just cbitcoin but I will try connecting to the satoshi client as well.

Edit: Pings work fine. Moving onto address discovery.
Nice! im only testing against the satoshi client for now. im also making an alternative client in python. Tongue
so far i can download and verify blocks and transactions(mostly... have implemented a hacky replacement for the scripting, just ripping out the sig and publickey and checks if it verifyes...). i have no working wallet for now, but i can send hand crafted transactions.

"The whole problem with the world is that fools and fanatics are always so certain of themselves and wiser people so full of doubts." -Bertrand Russell
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
August 15, 2012, 07:29:48 PM
 #101

Good luck with that python project. I take it that the client will use the SPV model?

cbitcoin wont be a bitcoin client but will provide the low-level interface for creating a client. After cbitcoin reaches alpha stage I will start working on the client on top of cbitcoin which is currently code-named BitEagle.

And I'm close to alpha stage. I've just got to sort out the address discovery and do the block validation code. cbitcoin wont relay/download transactions or blocks. cbitcoin only has a low level interface to the bitcoin network and also provides handshakes, pings and address discovery. You can choose to turn these features on or off using flags, so for instane you may not want address discovery if you plan to connect to specific nodes only and you may wish to implement your own behaviour.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
August 15, 2012, 07:38:26 PM
 #102

Nice! im only testing against the satoshi client for now. im also making an alternative client in python. Tongue
so far i can download and verify blocks and transactions(mostly... have implemented a hacky replacement for the scripting, just ripping out the sig and publickey and checks if it verifyes...). i have no working wallet for now, but i can send hand crafted transactions.

If you are using python, please consider using the 'bitcoin' module found in https://github.com/jgarzik/pynode

It already does full script and block verification.

The only major to-do is chain reorg, and that does not impact the 'bitcoin' module at all.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
kokjo
Legendary
*
Offline Offline

Activity: 1050

You are WRONG!


View Profile
August 16, 2012, 07:30:22 AM
 #103

Good luck with that python project. I take it that the client will use the SPV model?
no, it will be a full blown client. but it will be very modular so a SPV would be very simple to make with the codebase

Quote
cbitcoin wont be a bitcoin client but will provide the low-level interface for creating a client. After cbitcoin reaches alpha stage I will start working on the client on top of cbitcoin which is currently code-named BitEagle.
awesome!

Quote
And I'm close to alpha stage. I've just got to sort out the address discovery and do the block validation code. cbitcoin wont relay/download transactions or blocks. cbitcoin only has a low level interface to the bitcoin network and also provides handshakes, pings and address discovery. You can choose to turn these features on or off using flags, so for instane you may not want address discovery if you plan to connect to specific nodes only and you may wish to implement your own behaviour.
i python i have created a seperat class, that controls block down/up-loading behavior, and another class that keeps track of the addresses.
...and i have not implemented ping yet, because im lazy and its not really an important feature(but connections closes sometimes...)

happy coding!

"The whole problem with the world is that fools and fanatics are always so certain of themselves and wiser people so full of doubts." -Bertrand Russell
kokjo
Legendary
*
Offline Offline

Activity: 1050

You are WRONG!


View Profile
August 16, 2012, 07:38:34 AM
 #104

Nice! im only testing against the satoshi client for now. im also making an alternative client in python. Tongue
so far i can download and verify blocks and transactions(mostly... have implemented a hacky replacement for the scripting, just ripping out the sig and publickey and checks if it verifyes...). i have no working wallet for now, but i can send hand crafted transactions.

If you are using python, please consider using the 'bitcoin' module found in https://github.com/jgarzik/pynode

It already does full script and block verification.

The only major to-do is chain reorg, and that does not impact the 'bitcoin' module at all.


+1 will look into it. Smiley

"The whole problem with the world is that fools and fanatics are always so certain of themselves and wiser people so full of doubts." -Bertrand Russell
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
August 16, 2012, 02:33:05 PM
 #105

When you get to validating scripts fully it would be a good idea to test the scripts with the scripts the original client uses. I've made a file with many tests written in my script format, with a line for the script followed by 1 if the script should be valid and 0 is it should not. This includes tests from the C++ client: https://github.com/MatthewLM/cbitcoin/blob/master/test/scriptCases.txt

For the scripts I use this BNF:

Code:
<digit> = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" | "a" | "b" | "c" | "d" | "e" | "f" | "A" | "B" | "C" | "D" | "E" | "F"
 <hex_digits> ::= <digit> | <digit> <hex_digits>
 <hex> ::= "0x" <hex_digits>
 <operation_name> = "0" | "FALSE" | "1NEGATE" | "RESERVED" | "1" | "TRUE" | "2" | "3" | "4" |  "5" | "6" | "7" | "8" | "9" | "10" | "11" | "12" | "13" | "14" | "15" | "16" | "NOP" | "VER" | "IF" | "NOTIF" | "VERIF" | "VERNOTIF" | "ELSE" | "ENDIF" | "VERIFY" | "RETURN" | "TOALTSTACK" | "FROMALTSTACK" | "2DROP" | "2DUP" | "3DUP" | "2OVER" | "2ROT" | "2SWAP" | "IFDUP" | "DEPTH" | "DROP" | "DUP" | "NIP" | "OVER" | "PICK" | "ROLL" | "ROT" | "SWAP" | "TUCK" | "CAT" | "SUBSTR" | "LEFT" | "RIGHT = 0x81" | "SIZE" | "INVERT" | "AND" | "OR" | "XOR" | "EQUAL" | "EQUALVERIFY" | "RESERVED1" | "RESERVED2" | "1ADD" | "1SUB" | "2MUL" | "2DIV" | "NEGATE" | "ABS" | "NOT" | "0NOTEQUAL" | "ADD" | "SUB" | "MUL" | "DIV" | "MOD" | "LSHIFT " | "RSHIFT" | "BOOLAND" | "BOOLOR" | "NUMEQUAL" | "NUMEQUALVERIFY" | "NUMNOTEQUAL" | "LESSTHAN" | "GREATERTHAN" | "LESSTHANOREQUAL" | "GREATERTHANOREQUAL" | "MIN" | "MAX" | "WITHIN" | "RIPEMD160" | "SHA1" | "SHA256" | "HASH160" | "HASH256" | "CODESEPARATOR" | "CHECKSIG" | "CHECKSIGVERIFY" | "CHECKMULTISIG" | "CHECKMULTISIGVERIFY" | "NOP1" | "NOP2" | "NOP3" | "NOP4" | "NOP5" | "NOP6" | "NOP7" | "NOP8" | "NOP9" | "NOP10"
 <operation> ::= "OP_" <operation_name>
 <lines> ::= "\n" | "\n" <lines>
 <seperator> ::= <lines> | " "
 <operation_or_hex> ::= <operation> | <hex>
 <seperated_operation_or_hex> ::= <operation_or_hex> <seperator>
 <script_body> ::= <seperated_operation_or_hex> | <seperated_operation_or_hex> <script_body> | ""
 <lines_or_none> ::= <lines> | ""
 <operation_or_hex_or_none> ::= <operation_or_hex> | ""
 <script> ::= <lines_or_none> <script_body> <operation_or_hex_or_none>

I do now have an issue with EPIPE: http://stackoverflow.com/questions/11990700/broken-pipe-epipe-on-connection-to-loopback-address

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
August 19, 2012, 11:19:40 PM
 #106

The EPIPE problem might be with libevent. I'll try libev (which is apparently better...). If anyone knows libev and would like to help, please get in touch.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
kokjo
Legendary
*
Offline Offline

Activity: 1050

You are WRONG!


View Profile
August 20, 2012, 11:41:43 AM
 #107

 signal(SIGPIPE, SIG_IGN) ?

"The whole problem with the world is that fools and fanatics are always so certain of themselves and wiser people so full of doubts." -Bertrand Russell
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
August 20, 2012, 01:41:40 PM
 #108

That is for ignoring SIGPIPE. I disable SIGPIPE via SO_NOSIGPIPE and MSG_NOSIGNAL.

The problem is the EPIPE that shouldn't be there. The connection over the loopback IP is not always working for some reason.

UPDATE: It seems the other side of the connection does not accept before the connection event is received..

Code:
CN CONNECTION WAIT
CN CONNECTED OK
CN CONNECTION WAIT
CN SENT version

When printing everything out it says the connection is OK but there was not accept. There should be at least a "L1 TRIED ACCEPT" in there.

UPDATE 2: Edge triggered events also fail.

UPDATE 3: Adding the events after trying the connection does nothing to help.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
August 21, 2012, 05:32:04 PM
 #109

So I made an attempted implementation of the networking code using libev. The problem is it doesn't work. If anyone can get it to work, please help. I've comited the changes. This is the new libev code: https://github.com/MatthewLM/cbitcoin/blob/master/dependencies/sockets/CBLibevSockets.c

When I run it, it seems to occasionally partially work any other times it will not work. For instance here:

Code:
CN CONNECTION WAIT
CN CONNECTION WAIT
L2 TRIED ACCEPT
L2 MADE ACCEPT
L2 HAS ACCEPTED OK
CN CONNECTED OK
CN CONNECTED OK

The connecting CBNetworkCommunicator waits for the connections, the second listening CBNetworkCommunicator accepts the connection but then the connecting CBNetworkCommunicator thinks both connections are fine when only one was accepted. Then the next time I run it:

Code:
CN CONNECTION WAIT
CN CONNECTION WAIT

So the connecting CBNetworkCommunicator waits for the connections but they do not get accepted.

Why is this so hard? It surely isn't meant to be this hard.

Edit: To allow time for any help to arrive (if ever) I'll skip this for now and move onto block validation and P2SH.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
August 22, 2012, 01:35:38 PM
 #110

I modified the libevent code to detect socket errors during the connect event.

Code:
void CBDidConnect(evutil_socket_t socketID,short eventNum,void * arg){
CBEvent * event = arg;
if (eventNum & EV_TIMEOUT) {
// Timeout for the connection
event->loop->onTimeOut(event->loop->communicator,event->node,CB_TIMEOUT_CONNECT);
}else{
int optval = -1;
socklen_t optlen = sizeof(optval);
getsockopt(socketID, SOL_SOCKET, SO_ERROR, &optval, &optlen);
if (optval)
// Act as timeout
event->loop->onTimeOut(event->loop->communicator,event->node,CB_TIMEOUT_CONNECT);
else
// Connection successful
event->onEvent.ptr(event->loop->communicator,event->node);
}
}

Occasionally optval is given as 61 (ECONNREFUSED). Still begs the question, why is there a ECONNREFUSED for a loopback connection?

Edit: Fixed it, needed to use SO_REUSEADDR.

Problem now seems to be that the connecting CBNetworkCommunicator is not replying to getaddr requests.

Edit 2: Well the reason why is because the addresses are not broadcast. So you might expect the connecting CBNetworkCommunicator to relay the addresses when they are broadcast but this is only done for unsolicited addr messages and at the time the CBNetworkCommunicator has made the original getaddr request so it does not relay the addresses. As far as I can tell this is successfully following the behaviour of the original client. I'll simply set the addresses as being public so that they are always given in getaddr responses.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
August 25, 2012, 09:30:03 PM
 #111

Good news. Just now my networking code past the unit test for the first time. I'm fast approaching alpha stage now. I just need to implement the block validation code. I suspect that will be easier than the horrid networking code. Also I should finnish the libev implementation of the networking.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
August 26, 2012, 11:44:27 PM
 #112

So far I've discovered the way bitcoin uses targets is silly. For instance, what is the point in blocks containing the target? That's calculated by clients. Also the way you can't have a mantissa over 7FFFFF is stupid. The fact the original client uses a bignum to do calculations is stupid. So yes, it's silly.

Validating blocks is not as nice as I thought it was. I keep finding weird parts of the bitcoin protocol.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
August 27, 2012, 02:33:08 PM
 #113

So far I've discovered the way bitcoin uses targets is silly. For instance, what is the point in blocks containing the target?

At a minimum, you do not need the entire blockchain history just to validate the block matches the target.

Quote
Also the way you can't have a mantissa over 7FFFFF is stupid.

By "stupid" ITYM avoiding pointless, redundant support that would exist only to enhance the artificial sense of completeness that some computer scientists need.

It's quite efficient, considering the limited storage (32 bits) into which a much larger number is stored.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
August 27, 2012, 03:33:42 PM
 #114

Quote
At a minimum, you do not need the entire blockchain history just to validate the block matches the target.

But you need the history to validate the proof of work regardless. Why not just check the proof of work when you are able to calculate the target all the way to the block. Is there any point in doing it earlier?

Here's my implementation of retargeting and proof of work validation: https://github.com/MatthewLM/cbitcoin/blob/master/src/CBValidationFunctions.c

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1050


View Profile WWW
August 27, 2012, 05:06:08 PM
 #115

Quote
At a minimum, you do not need the entire blockchain history just to validate the block matches the target.

But you need the history to validate the proof of work regardless. Why not just check the proof of work when you are able to calculate the target all the way to the block. Is there any point in doing it earlier?

It's certainly faster to be able to do some checks before requiring the parent to exist. This allows you to validate the PoW of received orphan blocks, before bothering to request parent. Also, it allows doing some checks on a received block before needing a lock on the block tree data structure.

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
August 27, 2012, 05:11:11 PM
 #116

Useless complaining over 4 bytes per block anyway.  Cheesy

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
September 01, 2012, 09:09:46 PM
 #117

I'm very close to alpha stage now but I'm getting horrible heap corruption. Goodness knows how long it will take to debug...


Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
kokjo
Legendary
*
Offline Offline

Activity: 1050

You are WRONG!


View Profile
September 01, 2012, 09:26:33 PM
 #118

Good luck with that.

(thats why i don't code in C, Tongue )

"The whole problem with the world is that fools and fanatics are always so certain of themselves and wiser people so full of doubts." -Bertrand Russell
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
September 01, 2012, 09:34:49 PM
 #119

Thanks... The power afforded by C does sometimes cause problems like this....

It's got even worse since the stack has screwed up. Like here, where the hell did main() go???

Code:
(gdb) bt
#0  0x0000000100266d4d in CBNewAddressManager (events=0x5fbff838) at /Users/matt/Programming/BitEagle_Projects/cbitcoin/src/structures/CBObject/CBMessage/CBAddressManager/CBAddressManager.c:40
#1  0x0000000100000e24 in start ()

Luckily for me, this type of awkward error is rare but when it happens it's horrible.

...Almost certain the root cause is some stack corruption, not heap corruption. Eventually I'll figure it out...

Edit: Figured out what I thought was the problem was actually completely fine... I must be too tired. I will come back to this tomorrow.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
September 01, 2012, 11:57:16 PM
 #120

valgrind is your friend Smiley

Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
September 02, 2012, 11:59:21 AM
 #121

It would be... if it worked on Mountain Lion. Maybe I should try to run the code on Linux, which I need to do anyway.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
notme
Legendary
*
Offline Offline

Activity: 1848


View Profile
September 02, 2012, 04:07:21 PM
 #122

+21 million for valgrind

https://www.bitcoin.org/bitcoin.pdf
While no idea is perfect, some ideas are useful.
12jh3odyAAaR2XedPKZNCR4X4sebuotQzN
Gavin Andresen
Legendary
*
qt
Offline Offline

Activity: 1652


Chief Scientist


View Profile WWW
September 02, 2012, 04:13:01 PM
 #123

Does valgrind really not work on Mountain Lion?  Good reason for me not to upgrade... (runs great on Snow Leopard).

How often do you get the chance to work on a potentially world-changing project?
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
September 02, 2012, 05:47:08 PM
 #124

No, it's not working properly on Mountain Lion yet.

Never mind for me since I need to ensure my library is working in Linux anyway. The problem I have is trying to get it working smoothly on a virtual machine. This worked nice in the past but the computer is too hungry for RAM now so I really need to upgrade the RAM.

The reason I upgraded to Mountain Lion from Snow Leopard is because Apple wont let you upgrade Xcode otherwise... Well, at least the new mission control and application exposé thing is quite good.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
September 05, 2012, 09:39:02 PM
 #125

Well valgrind helped me find a few problems... unfortunately not the most annoying one that is keeping me back. I'm also having problems with libev. Does anyone have any experience with it? My current implementation is here: https://github.com/MatthewLM/cbitcoin/blob/master/dependencies/sockets/CBLibevSockets.c

It runs but gets stuck doing nothing.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
September 06, 2012, 10:13:42 PM
 #126

I figured out that my code was timing out and after I fixed that I'm getting a very strange valgrind error: http://stackoverflow.com/questions/12309178/valgrind-strange-syscall-param-socketcall-sendtomsg-points-to-uninitialised-b

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Remember remember the 5th of November
Legendary
*
Offline Offline

Activity: 1610

Reverse engineer from time to time


View Profile
September 12, 2012, 01:09:30 PM
 #127

If this library will be libevent dependant, it's already not portable, since libevent is hardly found for Windows.

And I also found this bit of code

Quote
void CBSecureRandomSeed(uint64_t gen){
   FILE * f = fopen("/dev/urandom", "r"); // Using urandom for speed.
   fread((void *)gen, 32, 1, f);
}

There is no /dev/urandom on Windows.

Though I suppose, since it's still early in development, Windows support could wait, but then again when you add more dependencies which are for Unix machines(I have not seen a port of libevent for Windows, for instance), it will be tough to add Windows support.

BTC:1AiCRMxgf1ptVQwx6hDuKMu4f7F27QmJC2
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
September 12, 2012, 03:10:13 PM
 #128

Sorry if it is not clear but cbitcoin is not dependent on libevent. The CBNetworkCommunicator code is dependent on a networking library but the dependency is satisfied via the implementation of functions. The implementation may use any networking library providing it can satisfy the requirements of cbitcoin (It also requires timer functions).

cbitcoin by itself is completely standard C99 using nothing but the standard C99 libraries. The dependencies are defined with weak linked function prototypes. You could say that isn't C99 and is a compiler addition, though the library would work by building it with the dependencies as well. The weak linking is done through pragma directives and works with gcc.

Also I thought libevent was available for windows. I understand cbitcoin is not Windows friendly because Windows is not C99 friendly. I don't even have Windows so I will be relying on other people to produce windows compatibility.

At the moment I've tested cbitcoin on OSX Mountain Lion and Linux Mint 13. It works minus the strange problems with the CBNetworkCommunicator code (please feel free to help with this). I've implemented everything even though it's not all properly working so I'll be making an alpha release shortly...

... I think I bizarrely just found the problem I have with the CBNetworkCommunicator... My eyes just saw something and I'm angry that GDB failed to detect it. Watchpoints do not seem to work properly...

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Remember remember the 5th of November
Legendary
*
Offline Offline

Activity: 1610

Reverse engineer from time to time


View Profile
September 12, 2012, 07:00:03 PM
 #129

Windows will be tough, some C applications usually require little to no modification to compile under GCC(Windows) whereas some may be much harder to do.
It really depends on how much OS specific APIs you use.

BTC:1AiCRMxgf1ptVQwx6hDuKMu4f7F27QmJC2
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
September 12, 2012, 07:13:56 PM
 #130

Well cbitcoin uses no OS-specific libraries, it just requires implementations of some functions. Provided with cbitcoin are solutions that work on Unix/Linux OSes (Tested Linux and OSX) but there is no reason why Windows implementations cannot be written. You can use cbitcoin without implementing those functions but some parts will not work. For example you cannot build and use merkle trees without implementing the SHA-256 function.

I've implementing networking with libevent but I also have an experimental libev implementation which does not work and I do not understand how to get working (If anyone understands libev, please help!).

As for the problems I'm having with the CBNetworkCommunicator code, I've realised the problem is with clashing connections which I forgot I didn't solve properly (the original client doesn't solve this problem but I'm trying to). I've implemented a method for favouring a connection when two clash but I'm getting a few issues with the expected behaviour of getaddr and addr. I'll find the issue and solve it. Then I will release cbitcoin as an alpha release and try to spur some more interest in the project as a result.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
randy-waterhouse
Jr. Member
*
Offline Offline

Activity: 41


View Profile
September 12, 2012, 11:10:31 PM
 #131

Have you considered converting to an autotools build format for the library?

I could do this for you .... nb: the src tree structure would probably have to be flattened out (i.e. changed) quite a bit.
notme
Legendary
*
Offline Offline

Activity: 1848


View Profile
September 12, 2012, 11:14:03 PM
 #132

If this library will be libevent dependant, it's already not portable, since libevent is hardly found for Windows.

And I also found this bit of code

Quote
void CBSecureRandomSeed(uint64_t gen){
   FILE * f = fopen("/dev/urandom", "r"); // Using urandom for speed.
   fread((void *)gen, 32, 1, f);
}

There is no /dev/urandom on Windows.

Though I suppose, since it's still early in development, Windows support could wait, but then again when you add more dependencies which are for Unix machines(I have not seen a port of libevent for Windows, for instance), it will be tough to add Windows support.

From http://en.wikipedia.org/wiki//dev/random
Quote
A counterpart to /dev/random is /dev/urandom ("unlocked"/non-blocking random source[4]) which reuses the internal pool to produce more pseudo-random bits. This means that the call will not block, but the output may contain less entropy than the corresponding read from /dev/random. While it is still intended as a pseudorandom number generator suitable for most cryptographic purposes, it is not recommended for the generation of long-term cryptographic keys.

If this seed has anything to do with the private keys, you really should be using /dev/random.  If it blocks, just have the user wiggle their mouse or ask their cat walk on the keyboard until you get enough entropy.

https://www.bitcoin.org/bitcoin.pdf
While no idea is perfect, some ideas are useful.
12jh3odyAAaR2XedPKZNCR4X4sebuotQzN
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
September 13, 2012, 01:00:46 AM
 #133

Have you considered converting to an autotools build format for the library?

I could do this for you .... nb: the src tree structure would probably have to be flattened out (i.e. changed) quite a bit.

I do like the tree structure. I do undertand that doesn't work very well for make files, though you can use bash to copy the files into a temporary directory, no?

Python was the easiest way for me at the moment. Feel free to play with it.

If this seed has anything to do with the private keys, you really should be using /dev/random.  If it blocks, just have the user wiggle their mouse or ask their cat walk on the keyboard until you get enough entropy.

It hasn't got anything to do with private keys. It's to do with the address manager where the security is not that critical.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
notme
Legendary
*
Offline Offline

Activity: 1848


View Profile
September 13, 2012, 02:46:26 AM
 #134

Have you considered converting to an autotools build format for the library?

I could do this for you .... nb: the src tree structure would probably have to be flattened out (i.e. changed) quite a bit.

I do like the tree structure. I do undertand that doesn't work very well for make files, though you can use bash to copy the files into a temporary directory, no?

Python was the easiest way for me at the moment. Feel free to play with it.

If this seed has anything to do with the private keys, you really should be using /dev/random.  If it blocks, just have the user wiggle their mouse or ask their cat walk on the keyboard until you get enough entropy.

It hasn't got anything to do with private keys. It's to do with the address manager where the security is not that critical.

Good to know.

https://www.bitcoin.org/bitcoin.pdf
While no idea is perfect, some ideas are useful.
12jh3odyAAaR2XedPKZNCR4X4sebuotQzN
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
September 13, 2012, 10:38:23 PM
 #135

I've released the first alpha version of cbitcoin: https://bitcointalk.org/index.php?topic=109312.0

Everything is implemented. You may notice a lot of bitcoin features are not included. This is because cbitcoin provides the fundamental functions of bitcoin and doesn't try to be specific. It cannot be called a client library (I'll be making a client library later that uses cbitcoin for BitEagle).

I wont make this beta until I'm personally happy with the testing I've done and I've implemented BitEagle on top of it. By implementing the client software on-top it will allow me to detect problems. BitEagle is the codename for the client which is in the conceptual stage at the moment.

I figured out how to use the git branches (I think) so I'll use the alpha branch to make fixes to the 1.0 release and the master branch will be for new features etc. for new releases.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
September 29, 2012, 12:39:19 PM
 #136

I'm not sure I will ever succeed to do this, but I will try to make a Perl6 interface to this library, using the NativeCall module.   If anyone wants to help, please PM me.
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
September 29, 2012, 01:01:58 PM
 #137

To compile your libraries on my 32-bits, debian GNU/linux host, I had to change line 94 of BUILD.py (why not use a makefile, btw?):

Code:
cflags += " -m64"

into:

Code:
cflags += " -m32"

Also, in src/structures/CBBigInt/CBBigInt.c, line 43:

Code:
for (u_int8_t x = a.length - 1;; x--) {

into:

Code:
for (uint8_t x = a.length - 1;; x--) {

Then all compilation seems fine, but linking fails:

LINKING ./build/bin/libcbitcoin.so
gcc: error: unrecognized command line option ‘-flat_namespace’


I just removed this option and linking succeeded.  Hope it's ok.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
September 29, 2012, 02:45:40 PM
 #138

I'm not sure I will ever succeed to do this, but I will try to make a Perl6 interface to this library, using the NativeCall module.   If anyone wants to help, please PM me.

Good luck with this. If you have any questions about the library just send an email to cbitcoin@thelibertyportal.com And keep in note it's still in alpha and probably has all sorts of problems some of which are known already (CBNetworkCommunicator has issues).

Quote
To compile your libraries on my 32-bits, debian GNU/linux host, I had to change line 94 of BUILD.py

I should have a 32bit option in there.

Quote
why not use a makefile, btw?

The reason I don't have a makefile is because it's not a trivial thing to compile and I'm not good with makefiles. A makefile would be nice be it's not something I want to produce right now. Anyone is highly welcome to create a makefile however. I do ask that the tree structure of the source and header files remains intact, so you'd need some sort of bash code to work through that (One thing that made me use python instead).

Thanks for spotting the u_int8_t problem in the master branch. I've fixed that.

The flat_namespace problem should also be fixed for both the alpha and master branch.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
September 29, 2012, 03:18:45 PM
 #139

Does anyone understand the RAND_add OpenSSL function. How do I figure out what to pass into the entropy argument? I literally have no clue, the docs are terrible. I'm adding entropy from a text string given through the terminal eg. dskjbapohspigpivsbvo98sg2pib3r2oivbspvow48fiqpbfivbsv=s like a monkey on a type-writer.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
misterbigg
Hero Member
*****
Offline Offline

Activity: 924


GET IN - Smart Ticket Protocol - Live in market!


View Profile
September 29, 2012, 03:30:23 PM
 #140

Just use OpenSSL for bignum and move on.

OpenSSL is a giant pain in the ass to build.

My preference for a repository is that everything compile by just pushing a button. No external dependencies (especially "wget", ick). It should not be necessary to install anything else. No python, no separate repositories, etc...

If I use an external dependency then I bring it in to my repository using "git subtree", which makes it very easy to work with. Sometimes I produce amalgamations of external libraries to make them easy to integrate (for example, I have produced an amalgamated version of FreeType).

If someone is better at C than C++ I have to question their proficiency in general. Learning new languages is part of being a good programmer.

This having been said, the GPL is the kiss of death for this project. It is an encumberance that virtually assures that cbitcoin will never become serious to corporations.

For some examples of git subtree and amalgamations, have a look at the repositories in my signature.


               ████
             ███  ███
           ████     ███
         ███  ███    ███
       ████     ███    ███
     ███  ███     ███    ███
   ████     ███     ███   ██
 ███  ███     █████████████████
███     ███     ███           ██
 ███      ███     ██          ██
   ███      ██████████      ███
     ███      ██████      ███
       ███      ██      ███
         ███          ███
           ███      ███
             ███  ███
               ████

GUTS
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
smart-ticket protocol for events
live product with market traction!
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
  BTC ANN
  WEBSITE
  BLOG
   
  SANDBOX
  WHITEPAPER
  BOUNTY
   
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
September 29, 2012, 03:58:37 PM
 #141

Does anyone understand the RAND_add OpenSSL function. How do I figure out what to pass into the entropy argument? I literally have no clue, the docs are terrible. I'm adding entropy from a text string given through the terminal eg. dskjbapohspigpivsbvo98sg2pib3r2oivbspvow48fiqpbfivbsv=s like a monkey on a type-writer.

Well keyboard input is not the best source of entropy so I'm using dev/random. This is just for the addressGenerator example program.

Quote
My preference for a repository is that everything compile by just pushing a button. No external dependencies (especially "wget", ick). It should not be necessary to install anything else. No python, no separate repositories, etc...

cbitcoin compiles with no external dependencies, only the standard C library. However it requires that you implement functions when you use the library through weak linking.

Quote
Learning new languages is part of being a good programmer.

And what there is no point in learning a new language? It's just a waste of time then. I don't like C++ at all. It annoys me. Unfortunately I have no choice but to read the C++ bitcoin code to help with my code.

Quote
This having been said, the GPL is the kiss of death for this project. It is an encumberance that virtually assures that cbitcoin will never become serious to corporations.

I'm not interested in corporations that want to mix my project with their own copyright restricted software. I'm using a "copyleft" license for a good reason.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
September 29, 2012, 04:25:30 PM
 #142

Just use OpenSSL for bignum and move on.

OpenSSL is a giant pain in the ass to build.

My preference for a repository is that everything compile by just pushing a button. No external dependencies (especially "wget", ick). It should not be necessary to install anything else. No python, no separate repositories, etc...

I'm surprised by this.  I mean, openssl might be hard to use and to link, but what's the alternative exactly?  Reinventing the wheel each time you do some code with cryptography?
misterbigg
Hero Member
*****
Offline Offline

Activity: 924


GET IN - Smart Ticket Protocol - Live in market!


View Profile
September 29, 2012, 04:28:51 PM
 #143

I'm surprised by this.  I mean, openssl might be hard to use and to link, but what's the alternative exactly?  Reinventing the wheel each time you do some code with cryptography?

Unfortunately, there is no alternative. I haven't had a need for OpenSSL in any of my projects. I tried to amalgamate it for a friend of mine but there's no chance of that happening, the code base is too difficult to work with.


               ████
             ███  ███
           ████     ███
         ███  ███    ███
       ████     ███    ███
     ███  ███     ███    ███
   ████     ███     ███   ██
 ███  ███     █████████████████
███     ███     ███           ██
 ███      ███     ██          ██
   ███      ██████████      ███
     ███      ██████      ███
       ███      ██      ███
         ███          ███
           ███      ███
             ███  ███
               ████

GUTS
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
smart-ticket protocol for events
live product with market traction!
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
  BTC ANN
  WEBSITE
  BLOG
   
  SANDBOX
  WHITEPAPER
  BOUNTY
   
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
September 29, 2012, 04:28:57 PM
 #144

Just use OpenSSL for bignum and move on.

OpenSSL is a giant pain in the ass to build.

My preference for a repository is that everything compile by just pushing a button. No external dependencies (especially "wget", ick). It should not be necessary to install anything else. No python, no separate repositories, etc...

I'm surprised by this.  I mean, openssl might be hard to use and to link, but what's the alternative exactly?  Reinventing the wheel each time you do some code with cryptography?

Yeah, it is silly and not realistic for any large software, because your maintenance effort scales up to involve tracking $N third party repos for security fixes and other updates.

When Fedora or SuSE Linux distros push out zero-day fixes for OpenSSL or zlib, for example, your project remains vulnerable.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
September 29, 2012, 04:51:48 PM
 #145

Unfortunately, there is no alternative. I haven't had a need for OpenSSL in any of my projects. I tried to amalgamate it for a friend of mine but there's no chance of that happening, the code base is too difficult to work with.

That's quite ironic for someone who just wrote:

If someone is better at C than C++ I have to question their proficiency in general. Learning new languages is part of being a good programmer.

I mean, for a good programmer, learning how to reuse existing code seems to me at least as important as learning new languages.

Lots of people link their code to the openssl libraries.  They might be complaining about the process being painful, but in the end they eventually do it.  They don't say:  "nah screw this, it's just too difficult".
misterbigg
Hero Member
*****
Offline Offline

Activity: 924


GET IN - Smart Ticket Protocol - Live in market!


View Profile
September 29, 2012, 04:57:56 PM
 #146

Lots of people link their code to the openssl libraries.  They might be complaining about the process being painful, but in the end they eventually do it.  They don't say:  "nah screw this, it's just too difficult".

I said it was too difficult to amalgamate (https://github.com/vinniefalco/Amalgamate).


               ████
             ███  ███
           ████     ███
         ███  ███    ███
       ████     ███    ███
     ███  ███     ███    ███
   ████     ███     ███   ██
 ███  ███     █████████████████
███     ███     ███           ██
 ███      ███     ██          ██
   ███      ██████████      ███
     ███      ██████      ███
       ███      ██      ███
         ███          ███
           ███      ███
             ███  ███
               ████

GUTS
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
smart-ticket protocol for events
live product with market traction!
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
  BTC ANN
  WEBSITE
  BLOG
   
  SANDBOX
  WHITEPAPER
  BOUNTY
   
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
September 29, 2012, 05:01:27 PM
 #147

I said it was too difficult to amalgamate (https://github.com/vinniefalco/Amalgamate).

Ok.  My bad.   I did not know about this concept.  I guess it's kind of a recent technique and openssl being quite an old library, it might not fit well into it.
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
September 30, 2012, 09:57:15 AM
 #148

The reason I don't have a makefile is because it's not a trivial thing to compile and I'm not good with makefiles. A makefile would be nice be it's not something I want to produce right now. Anyone is highly welcome to create a makefile however. I do ask that the tree structure of the source and header files remains intact, so you'd need some sort of bash code to work through that (One thing that made me use python instead).

Still, it would be great if we could compile your library with a classic "./configure && make"

I'm not good enough a GNU programmer, but if someone here is, please help.

Meanwhile, I'll read the automake manual and I'll see what I can do.

Anyway your project really seem like an awesome step towards a GNU version of bitcoin (C language + GNU license).  Please keep up.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
September 30, 2012, 01:56:55 PM
 #149

Thanks grondilu. If you or anyone else can help create a Makefile that would be excellent. It might be best to create a fork of cbitcoin on github for this task and when it is completed a pull request can be made to the master branch. This is a good idea even if you cannot get it to work since others may be able to modify it and get it to work.

And yes I'll keep it up the best I can. I've been thinking about how the plans for this project can be improved and I hopefully I'll be able to announce these improvements in the next few weeks when I have everything sorted. If anyone has any suggestions it's a good time to give them to me now. :-)

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
wabber
Member
**
Offline Offline

Activity: 85


View Profile
September 30, 2012, 02:49:57 PM
 #150

If someone is better at C than C++ I have to question their proficiency in general. Learning new languages is part of being a good programmer.

So someone who doesn't like C++ and prefers to use C is a bad programmer? Well I prefer C.
misterbigg
Hero Member
*****
Offline Offline

Activity: 924


GET IN - Smart Ticket Protocol - Live in market!


View Profile
September 30, 2012, 05:15:27 PM
 #151

So someone who doesn't like C++ and prefers to use C is a bad programmer? Well I prefer C.

No but disregard my comment, I see why the decision was made, thanks to all for providing their insights.


               ████
             ███  ███
           ████     ███
         ███  ███    ███
       ████     ███    ███
     ███  ███     ███    ███
   ████     ███     ███   ██
 ███  ███     █████████████████
███     ███     ███           ██
 ███      ███     ██          ██
   ███      ██████████      ███
     ███      ██████      ███
       ███      ██      ███
         ███          ███
           ███      ███
             ███  ███
               ████

GUTS
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
smart-ticket protocol for events
live product with market traction!
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
  BTC ANN
  WEBSITE
  BLOG
   
  SANDBOX
  WHITEPAPER
  BOUNTY
   
deadserious
Full Member
***
Offline Offline

Activity: 121



View Profile
October 01, 2012, 03:46:33 AM
 #152

This having been said, the GPL is the kiss of death for this project. It is an encumberance that virtually assures that cbitcoin will never become serious to corporations.

Agree 100%.  And not only limiting for corporations.  Even freeware, public domain, small independent but commercial software will be hindered by this.  LGPL I might understand, but GPL?  That's a serious problem.
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
October 01, 2012, 09:55:15 AM
 #153

Agree 100%.  And not only limiting for corporations.  Even freeware, public domain, small independent but commercial software will be hindered by this.  LGPL I might understand, but GPL?  That's a serious problem.

The linux kernel is under GPL.  That did not prevent Google from using it with Androïd.

Also, if they don't want to use cbitcoin because it's GPL, at least the very existence of cbitcoin will show them it's possible to port bitcoin into C.  So they'll rewrite their own C code.   That's good enough to me.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 01, 2012, 06:57:27 PM
 #154

The GPL is a serious problem? For some people maybe. For me? No.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
notme
Legendary
*
Offline Offline

Activity: 1848


View Profile
October 01, 2012, 07:00:23 PM
 #155

The GPL is a serious problem? For some people maybe. For me? No.

Don't mind the freedom haters.  It's your code, license it how you'd like.

https://www.bitcoin.org/bitcoin.pdf
While no idea is perfect, some ideas are useful.
12jh3odyAAaR2XedPKZNCR4X4sebuotQzN
deadserious
Full Member
***
Offline Offline

Activity: 121



View Profile
October 02, 2012, 07:15:19 PM
 #156

Agree 100%.  And not only limiting for corporations.  Even freeware, public domain, small independent but commercial software will be hindered by this.  LGPL I might understand, but GPL?  That's a serious problem.

The linux kernel is under GPL.  That did not prevent Google from using it with Androïd.

Linking to anything GPL requires your code to also be GPL (or license compatible).  That means that you cannot link to the library without your software also being open source. For me to use cbitcoin I would have to compile it as an EXE and then use it by calling out to the process. 

That is why many library developers chose to go with LGPL. It allows you to link to the open source code and protect your source code.  If we changed or improved upon cbitcoin in any way, those changes would still have to be made public.
deadserious
Full Member
***
Offline Offline

Activity: 121



View Profile
October 02, 2012, 07:16:51 PM
 #157

The GPL is a serious problem? For some people maybe. For me? No.

It is your problem if you hope this to have any serious adoption. 

If you are intending to keep it small, niche and useless to a vast portion of the community that would use it, then no, I guess it's not your problem.
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
October 02, 2012, 07:18:15 PM
 #158

That is why many library developers chose to go with LGPL. It allows you to link to the open source code and protect your source code.  If we changed or improved upon cbitcoin in any way, those changes would still have to be made public.

+1

Most libraries are LGPL'd or similar.

A GPL'd library requires that all users be GPL'd.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 02, 2012, 07:43:19 PM
 #159

That means that you cannot link to the library without your software also being open source.


Yes, that's the point.

If you are intending to keep it small, niche and useless to a vast portion of the community that would use it, then no, I guess it's not your problem.

Well grondilu pointed out the linux kernel. That's not exactly "small, niche and useless to a vast portion of the community that would use it".

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
kokjo
Legendary
*
Offline Offline

Activity: 1050

You are WRONG!


View Profile
October 02, 2012, 07:48:25 PM
 #160

woohoo!!! licence flame war incomming!

"The whole problem with the world is that fools and fanatics are always so certain of themselves and wiser people so full of doubts." -Bertrand Russell
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
October 02, 2012, 07:54:03 PM
 #161

Well grondilu pointed out the linux kernel. That's not exactly "small, niche and useless to a vast portion of the community that would use it".

Irrelevant example, comparing apples and oranges.  The Linux kernel is not a library.

Applications do not link directly with the Linux kernel, therefore applications (obviously!) may be any license.

Not so, with cbitcoin.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
October 02, 2012, 08:00:31 PM
 #162

Well grondilu pointed out the linux kernel. That's not exactly "small, niche and useless to a vast portion of the community that would use it".

Irrelevant example, comparing apples and oranges.  The Linux kernel is not a library.

Applications do not link directly with the Linux kernel, therefore applications (obviously!) may be any license.

I'm must say I'm quite surprised by this GPL hate.

Fair enough.  If MatthewLM wants to stick with GPL (I really hope he will), then cbitcoin will only be linked with GPL applications, which is totally fine imho.

In long term, I think bitcoin will need at least one GPL implementation, for reasons too long to explain.  It does not matter if not everyone uses it.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 02, 2012, 08:10:40 PM
 #163

I'll come back to this issue at a later date. Once again I said I'm thinking through cbitcoin over the next few weeks. This is one thing I will look at and consider what license is best. So you just have to stay tuned. I do think the GPL has got problems, so yes I will figure this out. The LGPL might not be perfect either, so we will see.

Was there this discussion with the AGPL with Armory? I'm not using the AGPL before anyone is scared.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
October 02, 2012, 08:16:49 PM
 #164

I'll come back to this issue at a later date. Once again I said I'm thinking through cbitcoin over the next few weeks. This is one thing I will look at and consider what license is best. So you just have to stay tuned. I do think the GPL has got problems, so yes I will figure this out. The LGPL might not be perfect either, so we will see.

I gave my opinion but as you might have understood, I'm not a professional programmer, not even an experienced or talented one.  So maybe indeed GPL is not a good fit for a library.  The only example of a GPL library I know is the GNU libc library itself.

I suggest that during those few weeks you seek wisdom in the FSF community, on IRC or something.   Hell, maybe you can send an email to RMS himself  (I'm serious, I think this is an important enough topic, and we now know that RMS is aware of bitcoin).
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
October 02, 2012, 08:17:09 PM
 #165

Well grondilu pointed out the linux kernel. That's not exactly "small, niche and useless to a vast portion of the community that would use it".

Irrelevant example, comparing apples and oranges.  The Linux kernel is not a library.

Applications do not link directly with the Linux kernel, therefore applications (obviously!) may be any license.

I'm must say I'm quite surprised by this GPL hate.

Where is the hate, in my message?  I am simply stating facts.  Linux kernel developers have been deeply involved in the arcane legal issues of licenses and linking for well over a decade.

And even excluding my kernel work, you will see that many of my projects are GPL license: https://github.com/jgarzik    (basically all the non-python projects are GPL'd)

The Linux kernel example is simply not applicable to the cbitcoin licensing situation, because it is not a library against which people directly link their end user applications.

The facts are:

1) If cbitcoin is LGPL, an application using cbitcoin may be any license.
2) If cbitcoin is GPL, an application using cbitcoin must be GPL.
3) Linux kernel is not a library against which user applications directly link.  The kernel is GPL, but applications using the kernel's system call interface may be any license.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 02, 2012, 08:28:12 PM
 #166

Quote
If cbitcoin is LGPL, an application using cbitcoin may be any license.

You see I don't want that. People will come and make their proprietary software using my free software. I'd be benefiting them with free software when they are not willing to make their software free. If I could have a license which allows for linking with anything providing that the derivative work (using the library) allows for certain freedoms for users including right to copy and distribute without limitation, but would allow for some limitations of the non-GPL compatible licenses, then that would be good I think.

It's all very nasty to think about which is why I'm saying give me a few weeks alongside everything else and I'll come back with a plan for cbitcoin.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
October 02, 2012, 08:37:45 PM
 #167


Just so you know, I've just sent an email to RMS about this.  As I said, I think it's an important enough issue to bother him.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 02, 2012, 08:40:38 PM
 #168

Alright, thanks. Would be interesting to get an answer.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
deadserious
Full Member
***
Offline Offline

Activity: 121



View Profile
October 02, 2012, 08:59:56 PM
 #169

Well grondilu pointed out the linux kernel. That's not exactly "small, niche and useless to a vast portion of the community that would use it".

The linux kernel is not a library.  I can create closed source software that runs on the linux kernel.

However, a vast number of the libraries that make up linux are LGPL, so that just proves my point.
deadserious
Full Member
***
Offline Offline

Activity: 121



View Profile
October 02, 2012, 09:02:39 PM
 #170

The only example of a GPL library I know is the GNU libc library itself.

GNU libc is not GPL.  It's LGPL.
deadserious
Full Member
***
Offline Offline

Activity: 121



View Profile
October 02, 2012, 09:03:47 PM
 #171

You see I don't want that. People will come and make their proprietary software using my free software. I'd be benefiting them with free software when they are not willing to make their software free. If I could have a license which

Just because something is closed source doesn't mean it isn't free.
kokjo
Legendary
*
Offline Offline

Activity: 1050

You are WRONG!


View Profile
October 02, 2012, 09:07:11 PM
 #172

You see I don't want that. People will come and make their proprietary software using my free software. I'd be benefiting them with free software when they are not willing to make their software free. If I could have a license which

Just because something is closed source doesn't mean it isn't free.
free as in free speech, not free beer.

"The whole problem with the world is that fools and fanatics are always so certain of themselves and wiser people so full of doubts." -Bertrand Russell
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
October 02, 2012, 09:09:22 PM
 #173


On a practical level, it is doubtful that the user community would trust a closed source implementation with their money, when so many open source implementations exist.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
notme
Legendary
*
Offline Offline

Activity: 1848


View Profile
October 02, 2012, 09:10:48 PM
 #174

You see I don't want that. People will come and make their proprietary software using my free software. I'd be benefiting them with free software when they are not willing to make their software free. If I could have a license which

Just because something is closed source doesn't mean it isn't free.

When you use RMS's definition of freedom it does.

Closed source means your handing over your computer power to the programmer without being able to audit what he does with it.

https://www.bitcoin.org/bitcoin.pdf
While no idea is perfect, some ideas are useful.
12jh3odyAAaR2XedPKZNCR4X4sebuotQzN
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
October 02, 2012, 09:12:38 PM
 #175

The only example of a GPL library I know is the GNU libc library itself.

GNU libc is not GPL.  It's LGPL.

Is it?  Oh shit it is.  I really thought it was GPL.   My bad.   I guess you guys win.
misterbigg
Hero Member
*****
Offline Offline

Activity: 924


GET IN - Smart Ticket Protocol - Live in market!


View Profile
October 02, 2012, 10:10:01 PM
 #176

People will come and make their proprietary software using my free software. I'd be benefiting them with free software when they are not willing to make their software free.

My opinion is that the benefit of having moneyed commercial interests invest energy into the Bitcoin ecosystem outweighs the disadvantage of having more proprietary software.

You only need to compare the success of OS X and iOS to the success of Linux to see how proprietary software can be superior for end users.


               ████
             ███  ███
           ████     ███
         ███  ███    ███
       ████     ███    ███
     ███  ███     ███    ███
   ████     ███     ███   ██
 ███  ███     █████████████████
███     ███     ███           ██
 ███      ███     ██          ██
   ███      ██████████      ███
     ███      ██████      ███
       ███      ██      ███
         ███          ███
           ███      ███
             ███  ███
               ████

GUTS
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
smart-ticket protocol for events
live product with market traction!
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
  BTC ANN
  WEBSITE
  BLOG
   
  SANDBOX
  WHITEPAPER
  BOUNTY
   
deadserious
Full Member
***
Offline Offline

Activity: 121



View Profile
October 02, 2012, 10:15:40 PM
 #177

On a practical level, it is doubtful that the user community would trust a closed source implementation with their money, when so many open source implementations exist.

If you are talking about wallet ripoffs, then you are probably right.  But what about POS systems, ATM firmware, ERP applications, all which could use some bitcoin integration but are unable to open source for one reason or another.  Maybe the codebase uses other third-party closed source code... maybe there are corporate license restrictions to conform to a specific industry law... maybe there are security implications for open sourcing the code.

The point is that there are so many untold reasons why someone may want to use the cbitcoin library in their code to further bitcoin adoption.  Pigeonholing it with GPL unnecessarily limits it, especially considering that the LGPL was designed specifically with libraries in mind to fix this issue.

I think the main question is what is the end goal.  To expand the uptake of bitcoin?  Or make sure that developers of open source software have a competitive advantage over the corporations?  If we were further along with bitcoin, I think you could make an argument for a GPL library. Unfortunately, unless those with closed source interests have an easy and cheap way into bitcoin, they aren't going to spend their money.  Bitcoin isn't big enough.

If LGPL is going to be a no go... let me make a suggestion.  How about a dual license.  GPL and a closed source license where the closed source licensee pays some fee to use the library how they see fit.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 02, 2012, 11:51:18 PM
 #178

I do notice now there are some things I misunderstood about the GPL license, despite reading it several times. It maybe that a custom license will be required for cbitcoin to solve issues with the GPL but also protect against proprietary software also.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
misterbigg
Hero Member
*****
Offline Offline

Activity: 924


GET IN - Smart Ticket Protocol - Live in market!


View Profile
October 03, 2012, 12:08:08 AM
 #179

protect against proprietary software also.

What exactly is it that you want to "protect" against?

Check out this article:
The GPL Family of Licenses Sees a Decline in Adoption


               ████
             ███  ███
           ████     ███
         ███  ███    ███
       ████     ███    ███
     ███  ███     ███    ███
   ████     ███     ███   ██
 ███  ███     █████████████████
███     ███     ███           ██
 ███      ███     ██          ██
   ███      ██████████      ███
     ███      ██████      ███
       ███      ██      ███
         ███          ███
           ███      ███
             ███  ███
               ████

GUTS
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
smart-ticket protocol for events
live product with market traction!
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
  BTC ANN
  WEBSITE
  BLOG
   
  SANDBOX
  WHITEPAPER
  BOUNTY
   
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
October 03, 2012, 08:18:43 AM
 #180

My opinion is that the benefit of having moneyed commercial interests invest energy into the Bitcoin ecosystem outweighs the disadvantage of having more proprietary software.

You only need to compare the success of OS X and iOS to the success of Linux to see how proprietary software can be superior for end users.

If the benefit you're talking about is only about mass adoption, I do not care.  Even if it means a "better quality", I do not care either.  I prefer to use a Free Software rather than a proprietary software, even if it has a lower quality.
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
October 03, 2012, 08:32:47 AM
 #181


So?  With this kind of argumentum ad populum, I guess there would be no one to advocate for open-source licenses at all in the first place, considering how little popular they were at some point in the history of computing.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 03, 2012, 06:33:43 PM
 #182

What I'm thinking about is a new license that aims to satisfy these two things:

1. Does not require source distribution alongside binary distribution of derivative works.
2. Places restriction on the license of derivative works but is more compatible with other licenses for linking purposes.

The second is the most awkward. It would need to be some sort of compromise between the GPL and LGPL to allow for compatibility but at the same time prevent proprietary derivative works.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
October 03, 2012, 06:45:27 PM
 #183

What I'm thinking about is a new license that aims to satisfy these two things:

1. Does not require source distribution alongside binary distribution of derivative works.
2. Places restriction on the license of derivative works but is more compatible with other licenses for linking purposes.

The second is the most awkward. It would need to be some sort of compromise between the GPL and LGPL to allow for compatibility but at the same time prevent proprietary derivative works.

LGPL prevents distribution of proprietary copies of the library.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 03, 2012, 07:05:14 PM
 #184

Hmm. I'm not used to all the legal-speak sorry. It does go over my head a lot of it.  Cheesy

What about linking to the library which is what I really mean to say. That's a different story isn't it?

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
notme
Legendary
*
Offline Offline

Activity: 1848


View Profile
October 03, 2012, 07:08:18 PM
 #185

Hmm. I'm not used to all the legal-speak sorry. It does go over my head a lot of it.  Cheesy

What about linking to the library which is what I really mean to say. That's a different story isn't it?

Linkability is precisely the distinction between GPL and LGPL.  GPL can only link with GPL.  LGPL can link with anything, but anyone who distributes the library must still provide source and any modifications done to the library.

https://www.bitcoin.org/bitcoin.pdf
While no idea is perfect, some ideas are useful.
12jh3odyAAaR2XedPKZNCR4X4sebuotQzN
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 03, 2012, 07:59:31 PM
 #186

Do they have to provide the source with binary distributions?

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
misterbigg
Hero Member
*****
Offline Offline

Activity: 924


GET IN - Smart Ticket Protocol - Live in market!


View Profile
October 03, 2012, 07:59:45 PM
 #187

LGPL is certainly preferred over GPL.

I use the MIT License (which is a permissive license) for all of my open source work (in my signature). Many developers have thanked me for providing things like DSP Filters under a permissive license to encourage commercial usage.


               ████
             ███  ███
           ████     ███
         ███  ███    ███
       ████     ███    ███
     ███  ███     ███    ███
   ████     ███     ███   ██
 ███  ███     █████████████████
███     ███     ███           ██
 ███      ███     ██          ██
   ███      ██████████      ███
     ███      ██████      ███
       ███      ██      ███
         ███          ███
           ███      ███
             ███  ███
               ████

GUTS
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
smart-ticket protocol for events
live product with market traction!
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
  BTC ANN
  WEBSITE
  BLOG
   
  SANDBOX
  WHITEPAPER
  BOUNTY
   
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
October 03, 2012, 08:02:46 PM
 #188

What about linking to the library which is what I really mean to say. That's a different story isn't it?

LGPL:

Proprietary, closed source versions of the library are NOT permitted.
Proprietary, closed source applications using cbitcoin library are permitted.

GPL:

Proprietary, closed source versions of the library are NOT permitted.
Proprietary, closed source applications using cbitcoin library are NOT permitted.

In either case, LGPL or GPL, your cbitcoin code remains free software.  Nobody is permitted to modify and distribute cbitcoin without also providing source code.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 03, 2012, 08:20:40 PM
 #189

Proprietary, closed source applications using cbitcoin library are permitted.

That is what I was saying and I thought (I shouldn't have said "derivative"). I don't want proprietary applications using my library but I do not mind if they are closed-source. I do not consider closed source applications as necessarily proprietary. I consider them to be free software as long as they allow for free distribution, reverse-engineering and modification. Open-source is just an option.

So as you see I don't really want LGPL or GPL. In fact there is no license that exists which I can find which matches what I want. I want the license to be as anti-copyright as possible (Which is ironic since it's using copyright against itself). The only exception is to allow for compatibility with other open-source licenses. cbitcoin will need compatibility with the OpenSSL license which GPL isn't compatible with I realised.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
October 03, 2012, 08:59:19 PM
 #190

Proprietary, closed source applications using cbitcoin library are permitted.

That is what I was saying and I thought (I shouldn't have said "derivative"). I don't want proprietary applications using my library but I do not mind if they are closed-source. I do not consider closed source applications as necessarily proprietary. I consider them to be free software as long as they allow for free distribution, reverse-engineering and modification. Open-source is just an option.

The rest of the world does not consider that free software.

Closed source does not permit easy reverse engineering or modification, by its very nature.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 03, 2012, 09:37:25 PM
 #191

Well I don't care what the rust of the world thinks is free (as in liberty). I don't think trying to force people to include source code in their distributions is free. That's my opinion. If it means I have to make a new license to support this opinion then that's what I'll do.

Obviously I wont write the license, I'll get help with that.

If "the rest of the world" thought that way then why don't they all use the AGPL? Why does server-side code get treated differently?

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
misterbigg
Hero Member
*****
Offline Offline

Activity: 924


GET IN - Smart Ticket Protocol - Live in market!


View Profile
October 03, 2012, 10:06:38 PM
 #192

Well I don't care what the rust of the world thinks is free (as in liberty). I don't think trying to force people to include source code in their distributions is free.

I agree! Placing the restriction that source must be provided in order to distribute a binary is certainly non-free (as in liberty). But it's not exactly clear what you are trying to accomplish with your license. Is it just that you want to make sure no one can make money off your work? What about the license used by Bitcoin-Qt and Bitcoind? Are those licenses okay with you? Because they are both permissive licenses (equivalent to MIT License, I think).

How does this resonate with you:

Quote
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
 


               ████
             ███  ███
           ████     ███
         ███  ███    ███
       ████     ███    ███
     ███  ███     ███    ███
   ████     ███     ███   ██
 ███  ███     █████████████████
███     ███     ███           ██
 ███      ███     ██          ██
   ███      ██████████      ███
     ███      ██████      ███
       ███      ██      ███
         ███          ███
           ███      ███
             ███  ███
               ████

GUTS
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
smart-ticket protocol for events
live product with market traction!
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
  BTC ANN
  WEBSITE
  BLOG
   
  SANDBOX
  WHITEPAPER
  BOUNTY
   
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 03, 2012, 11:22:01 PM
 #193

I'm going to come back to this another time. It's really hard knowing what to do. As I've said I've got a lot to think about, so it is all up in the air at the moment.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
deadserious
Full Member
***
Offline Offline

Activity: 121



View Profile
October 04, 2012, 12:02:10 AM
 #194

I'm going to come back to this another time. It's really hard knowing what to do. As I've said I've got a lot to think about, so it is all up in the air at the moment.

I have a good amount of experience in open source licensing so if you need to bounce ideas off of someone I can help, though to be honest, I'm philosophically quite a bit apart from you.  All of my recent open source work has been published under MIT allowing anyone to do pretty much whatever they please with it.  I don't care if people make money off of my work.  I can always charge them for support on the backend or make money off my reputation in the community.  It's been very successful for me so far. 
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
October 04, 2012, 12:05:22 PM
 #195


After some thought, and a few mail exchanges with RMS, my opinion is that it would be nice if Mathew could stick to GPL (yes, GPL, not LGPL).    People would indeed not be able to link to cbitcoin without releasing their source code.  So be it.
misterbigg
Hero Member
*****
Offline Offline

Activity: 924


GET IN - Smart Ticket Protocol - Live in market!


View Profile
October 04, 2012, 01:45:49 PM
 #196


After some thought, and a few mail exchanges with RMS, my opinion is that it would be nice if Mathew could stick to GPL (yes, GPL, not LGPL).    People would indeed not be able to link to cbitcoin without releasing their source code.  So be it.

Well what's the point of emailing RMS? His answer is always GPL...


               ████
             ███  ███
           ████     ███
         ███  ███    ███
       ████     ███    ███
     ███  ███     ███    ███
   ████     ███     ███   ██
 ███  ███     █████████████████
███     ███     ███           ██
 ███      ███     ██          ██
   ███      ██████████      ███
     ███      ██████      ███
       ███      ██      ███
         ███          ███
           ███      ███
             ███  ███
               ████

GUTS
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
smart-ticket protocol for events
live product with market traction!
    ███
███
███
███
███
███
███
███
███
███
███
███
███
███
   
  BTC ANN
  WEBSITE
  BLOG
   
  SANDBOX
  WHITEPAPER
  BOUNTY
   
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 04, 2012, 02:06:50 PM
 #197

Yes, you would expect GPL to be the only answer. I did email him my idea. I expect to have criticism on it.  Smiley

I have a good amount of experience in open source licensing so if you need to bounce ideas off of someone I can help, though to be honest, I'm philosophically quite a bit apart from you.  All of my recent open source work has been published under MIT allowing anyone to do pretty much whatever they please with it.  I don't care if people make money off of my work.  I can always charge them for support on the backend or make money off my reputation in the community.  It's been very successful for me so far. 

I don't care if people make money using cbitcoin. That would be great. You clearly do not understand. I do not want proprietary software being made using cbitcoin. By that I mean software with use, copy and/or modification restrictions.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
kjj
Legendary
*
Offline Offline

Activity: 1302



View Profile
October 04, 2012, 05:04:27 PM
 #198


After some thought, and a few mail exchanges with RMS, my opinion is that it would be nice if Mathew could stick to GPL (yes, GPL, not LGPL).    People would indeed not be able to link to cbitcoin without releasing their source code.  So be it.

Well what's the point of emailing RMS? His answer is always GPL...

No, it isn't.  He has a fully nuanced answer.  I will attempt to condense and summarize, but he explains it better, so it is worth the time to read it.

Basically, if you have a library that does something that everyone already does, using LGPL is the right way to go, as it allows people to use free libraries with their non-free software.  If they don't use the LGPL library, they will just use some other library that is worse (for the free software community).  It is better for the world to have free alternatives for common things, than not to.

But, if you have a library that does something that no one else does, it would be better (for the free software community) if the library was full GPL.  This way, developers that don't particularly support the free software ideals will have to decide if keeping their software non-free is worth the effort of duplicating the GPL library.  Some developers will choose to make their software free to avoid duplicating that effort, which increases the freedom in the world.

p2pcoin: a USB/CD/PXE p2pool miner - 1N8ZXx2cuMzqBYSK72X4DAy1UdDbZQNPLf - todo
I routinely ignore posters with paid advertising in their sigs.  You should too.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 04, 2012, 08:00:21 PM
 #199

Here's a draft copy of a largely modified GPLv3 license which is closer to what I want.

Code:
DRAFT LICENSE

1. Definitions

"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.

"The Program" refers to any copyrightable work licensed under this
License.  Each licensee is addressed as "you".  "Licensees" and
"recipients" may be individuals or organizations.

To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy.  The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.

A "covered work" means either the unmodified Program or a work based
on the Program.

To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy.  Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.

To "convey" a work means any kind of propagation that enables other
parties to make or receive copies.  Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.

2. Permissions

All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met.  This License explicitly affirms your unlimited
permission to run the unmodified Program.  The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work.  This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.

You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force.

You may convey verbatim copies of any part of the Program's source code
 as it is. These copies must remain licensed under this license.

You may convey a work based on the Program, or the modifications to
produce it from the Program, providing the work is licensed under the
entire terms of this license and any applicable additional terms allowed
in section 3.

3. Additional Terms

Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:

a) Disclaiming warranty or limiting liability differently from the
    terms of sections 15 and 16 of this License; or

b) Requiring preservation of specified reasonable legal notices or
    author attributions in that material or in the Appropriate Legal
    Notices displayed by works containing it; or

c) Prohibiting misrepresentation of the origin of that material, or
    requiring that modified versions of such material be marked in
    reasonable ways as different from the original version; or

d) Limiting the use for publicity purposes of names of licensors or
    authors of the material; or

e) Declining to grant rights under trademark law for use of some
    trade names, trademarks, or service marks; or

f) Requiring indemnification of licensors and authors of that
    material by anyone who conveys the material (or modified versions of
    it) with contractual assumptions of liability to the recipient, for
    any liability that these contractual assumptions directly impose on
    those licensors and authors.

  g) Requiring copyright attributions or legal notices in advertising
    material.

You may not add any further license terms to the material. Additional
terms can only apply to the material you add to the covered work and
cannot be applied to this Program.

Additional obligations may be placed upon you under an aggreement with any
party. You may supplement additional terms to this license with an
agreement between you and another party providing these terms comply with
section 7.

4. Termination.

You may not propagate or modify a covered work except as expressly
provided under this License.  Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 8).

However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.

Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.

Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License.  If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 6.

5. Acceptance Not Required for Having Copies.

You are not required to accept this License in order to receive or
run a copy of the Program.  Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance.  However,
nothing other than this License grants you permission to propagate or
modify any covered work.  These actions infringe copyright if you do
not accept this License.  Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.

6. Automatic Licensing of Downstream Recipients.

Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License and any applicable
additional terms allowed in section 3.  You are not
responsible for enforcing compliance by third parties with this License.

An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations.  If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph.

You may not initiate litigation (including a cross-claim or counterclaim
in a lawsuit) alleging that any patent claim is infringed by making,
using, selling, offering for sale, or importing the Program or any portion
of it.

7. No Surrender of Others' Freedom.

If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from your obigations under this License.  If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all.  For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.

8. Patents

A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based.  The
work thus licensed is called the contributor's "contributor version".

A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version.  For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.

Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.

In the following two paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement).  To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.

If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.

A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License.  You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.

Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.

9. Disclaimer of Warranty

THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

10. Limitation of Liability

IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.

11. Interpretation of Sections 9 and 10.

If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
deadserious
Full Member
***
Offline Offline

Activity: 121



View Profile
October 04, 2012, 09:01:33 PM
 #200

But, if you have a library that does something that no one else does, it would be better (for the free software community) if the library was full GPL.  This way, developers that don't particularly support the free software ideals will have to decide if keeping their software non-free is worth the effort of duplicating the GPL library.  Some developers will choose to make their software free to avoid duplicating that effort, which increases the freedom in the world.

He has a point if the market is such that a commercial entity would invest in that closed source alternative.  Bitcoin however is looking to encourage uptake, not discourage it.  By putting that financial and technical hurdle in front of business, you are killing innovation before it starts.
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
October 04, 2012, 09:19:48 PM
 #201

It is generally inadvisable to come up with licenses on your own.

Smart lawyers spent years on the current licenses, including rounds of open review and comments across multiple jurisdictions in the world.

It is doubtful lone efforts will match that sufficiently to make a bulletproof license, compared to existing ones.

Typically the choice is GPL (and variants like LGPL) or BSD (and variants like MIT/X11).  Those licenses are much more likely to have established court and legal precedent.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1050


View Profile WWW
October 04, 2012, 09:35:05 PM
 #202

Well what's the point of emailing RMS? His answer is always GPL...

No, it isn't.  He has a fully nuanced answer.  I will attempt to condense and summarize, but he explains it better, so it is worth the time to read it.

By the way, at the London conference, we had a discussion with RMS about Bitcoin and the license of its implementation. Afterwards in his talk, he acknowledged that a weaker license than GPL may be appropriate for things like Bitcoin, to increase the ability of adopting the system.

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 05, 2012, 07:42:41 PM
 #203

Smart lawyers spent years on the current licenses, including rounds of open review and comments across multiple jurisdictions in the world.

It is doubtful lone efforts will match that sufficiently to make a bulletproof license, compared to existing ones.

Well I plan to have a lawyer look over the license before I would use it. Many people use customer software licenses which might not have as much review as the popular licenses you talk about but at least have lawyers look at them.

And the license will be a modified GPL license, only changing necessary parts to satisfy my requirements. If you look at what I've done so far I've just removed a lot of the GPL restrictions and added an additional part to the "Additional Terms" section in attempt to create compatibility with the Apache License 1.0 which is needed for OpenSSL.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Peter Todd
Legendary
*
expert
Offline Offline

Activity: 1106


View Profile
October 05, 2012, 11:02:32 PM
 #204

Smart lawyers spent years on the current licenses, including rounds of open review and comments across multiple jurisdictions in the world.

It is doubtful lone efforts will match that sufficiently to make a bulletproof license, compared to existing ones.

Well I plan to have a lawyer look over the license before I would use it. Many people use customer software licenses which might not have as much review as the popular licenses you talk about but at least have lawyers look at them.

That's nice and all, but now if I'm going to use your software, I need to get my lawyers to look over the license. On top of that, if I write some software using your library, and distribute it to others, they need to have their lawyers look over it. With the standard licenses all our lawyers have already done this saving a tonne of effort. These days younger lawyers with a copyright law specialty might have even studied the standard open-source licenses in school. If I were a developer considering whether to use your license or not, I'd take one look at it and reject it purely out of the uncertainty.

Even the standard licenses are often misunderstood:

What about linking to the library which is what I really mean to say. That's a different story isn't it?

LGPL:

Proprietary, closed source versions of the library are NOT permitted.
Proprietary, closed source applications using cbitcoin library are permitted.

GPL:

Proprietary, closed source versions of the library are NOT permitted.
Proprietary, closed source applications using cbitcoin library are NOT permitted.

In either case, LGPL or GPL, your cbitcoin code remains free software.  Nobody is permitted to modify and distribute cbitcoin without also providing source code.


jgarzik got the gist of the LGPL right, and probably already knows about the point I'm about to make, but something a lot of people don't realize is that when you distribute LGPL using proprietary software in addition to distributing any changes to the LGPL library you must also make it possible for your end users to link your binaries to their own changed version of that library. While this is rarely an issue with dynamic libraries, let alone libraries in interpreted languages, it does mean you have to take special steps to distribute a statically linked binary. From the LGPL itself:

Quote
Code:
d) Do one of the following:

    0) Convey the Minimal Corresponding Source under the terms of this License, and the Corresponding
Application Code in a form suitable for, and under terms that permit, the user to recombine or relink the
Application with a modified version of the Linked Version to produce a modified Combined Work, in the
manner specified by section 6 of the GNU GPL for conveying Corresponding Source.
    1) Use a suitable shared library mechanism for linking with the Library. A suitable mechanism is one that
(a) uses at run time a copy of the Library already present on the user's computer system, and (b) will
operate properly with a modified version of the Library that is interface-compatible with the Linked Version.

FWIW I just started a new timestamping project, written in Python, and have licensed the client command line utility and core library under the LGPL (>=3). However the server daemon and library functions related to implementing that server are license under the FSF AGPL, (>=3) which requires source-code distribution not only to those you distribute the software too, but also those who you allow to access servers running that software over a network. Basically that means that there is an RPC command called "getsourceurl" and to comply with the license if you modify the source for the server you need to put that source code up somewhere publicly (github would be ideal) and in the config file set the sourcecode url to the new location. I'm also going to include some explanation in the LICENSE file making it clear that trivial changes are just that and don't trigger that requirement. Similarly that config files are not considered a part of the sourcecode.

My thinking is quite like RMS's stance on Bitcoin: I want the timestamping system itself to become widely used, even integrated into proprietary applications. However I'd like to see the timestamping servers remain absolutely open source, and indeed, the system itself is most secure and tamper-proof in an "open source" environment where everyone validates each others timestamps.

Of course, even though the AGPL is an official Free Software Foundation license it's still obscure enough that I may be making the same mistake you are.


If you still really really want to use something different from the standard FSF licenses at least consider dual licensing, or licensing + additional license exemptions. For instance if you licensed your library under the standard GPL-3, but then included an additional statement like:

Quote
As an additional permission, above and beyond the standard rights granted by the GPL3.0, you may also distribute a binary containing code from this library, provided that you allow such binary to be freely distributed.

This clause only adds additional permissions to the GPL3 license; if you do not require such permissions, you are free to ignore this clause and follow the terms in the standard GPL3 license.

This kind of thing has precedent already with the GPL linking exemption http://en.wikipedia.org/wiki/GPL_linking_exception used by some projects. Another example is in the modified GPL used by some ADA programs: http://en.wikipedia.org/wiki/GNAT_Modified_General_Public_License Finally the GPL font exemption: http://en.wikipedia.org/wiki/GPL_font_exception

At least with GPL+exemptions and GPL+other dual licensing a developer can look at the license file and immediately see that they can treat the project as GPL licensed and be "in the clear" by complying with the GPL. Lawyers don't need to be involved, and everyone already knows exactly what's required to comply.

MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 06, 2012, 12:38:49 AM
 #205

Thank you for that comment. I will likely use a dual license with the GPL.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Peter Todd
Legendary
*
expert
Offline Offline

Activity: 1106


View Profile
October 06, 2012, 01:11:10 AM
 #206

Thank you for that comment. I will likely use a dual license with the GPL.

Sounds great!

randy-waterhouse
Jr. Member
*
Offline Offline

Activity: