Bitcoin Forum

Simple question, why is OpenSSL needed in the official client?

It is used to do the ECDSA public/private key generation as well as the SHA256 and RIPEMD hashes.

Oh - yes - also the wallet encryption would be using OpenSSL functions.

We use it for ECDSA signing and verification, to generate private keys, for the SHA256 and RIPEMD160 hashing functions. and for AES (used in wallet encryption).

So no actual SSL? Because OpenSSL is a bulky nightmare to build and use. I'm contemplating writing my own bitcoind / GUI front end, but I would much like to avoid OpenSSL at all costs.

Open source libraries for all the operations that you described are available, most of them in cryptopp (http://www.cryptopp.com/).

Indeed I think that it should be possible to go a bit more "minimalist" (I personally don't use boost either for the same reason).

(wow - I just became a "hero")

And that was my next question, why is boost needed?

My preference is to have a repository that requires no external dependencies, just pull the repo and press the "build" button. I find the use of Qt quite disturbing as well. Last I checked "slots:" is not standard C++. And I don't think that the official client can be compiled in either Xcode or Visual Studio (doesn't it require some Qt tools to build?), which is quite shocking.

Oh yes, RPC-SSL uses actual SSL, but it's certainly possible to avoid using that (and disabling RPC-SSL support).


Certainly - but OpenSSL is also an open-source library, and it does everything we need.

Seems we are on the same wavelength in regards to 3rd party libraries.

My own project only requires 3rd party libraries for DB, PDF and FCGI (I even wrote my own regex class to avoid the need for an extra library).

Although I do appreciate the quality of code such as boost (I used to contribute to comp.lang.c++.moderated years ago whilst it was being developed) it really can be a pain when a project has so many 3rd party libraries or tools (that's why I currently am not able to build bitcoin).

from the top of my head: interprocess communication, threads, locking, filesystem interaction, some datatypes, unions, program options, asio.


The GUI is a Qt application - obviously it needs Qt to build.

If you just need the daemon, there is no dependency on Qt or the Qt toolchain.

I looked at the build steps for the official client and ran home screaming to mommy.

Would there be any interest in a clean C++ implementation that had no external dependencies and could be easily compiled in Xcode / Visual Studio?

Cool, but instead of degrading OpenSSL's image, try to to find a way to remove that monster known as Boost.

Hmm... I wasn't trying to degrade OpenSSL's image at all (it is a 3rd party library option that I use in my project and I think it is very high quality software).

I don't really have a problem with Boost either apart from the fact I do think it became a little bloated by perhaps trying to become a bit like .net or the like.

Boost is the lesser of these evils. Last time I tried to build OpenSSL there was a rats nest of external dependencies and build tools that I had to install. Boost at least, once you unpack it you just type "bootstrap" followed by "bjam." And if you are using just the header-only module of Boost, there is no need to build at all.

OpenSSL is so easy to build that I've built it for Android, too. Not to mention is builds fine on Windows.

Are you talking about Bitcoin-Qt or bitcoind?

We used to maintain a project for Visual Studio, but as none of the current developers use Windows, it got outdated. If someone wants to revive and maintain it, that is very welcome.

Unsure how the OSX builds happen, Gavin does those.

No external dependencies... we currently depend on OpenSSL, BDB, boost, libminiupnpc (optional), Qt (optional). OpenSSL shouldn't be too hard to swap for something else. There are already alternative clients that don't use OpenSSL, afaik. BDB is going to be dropped and replaced by LevelDB (included in the source tree, not a dependency). Boost would be a lot of work to change. The others are optional.

I could possibly be interested in doing this (I think I can cope with installing Boost but am really not interested in the whole MingW stuff) - does it support just using "nmake" rather than the IDE (I only use the console)?

As far as I know it is not possible to compile a Qt application using the Visual Studio IDE, you need their preprocessing junk to make sense of "slots" and "signals" keywords.

I have no clue about Windows programming or build environments. But if you make it work with nmake, then it will...

The Qt GUI is built using qmake, which supports many environments. Presumably you want to adapt the qmake project file to work on your platform, rather than maintaining a separate one.

Okay - can you give me the repository URL so I can have a look at it (am hoping it is github)?

There's a link on bitcoin.org. The source code is at http://github.com/bitcoin/bitcoin.

I would be interested in hearing about your experiences with this, if you would care to "blog" your progress :-)

Sorry - but I don't know how that helps me find a Visual Studio version (it is not one of the branches AFAICT).

I do have the software on my computer but it requires MingW to build (am I missing something obvious?).

I don't think its a separate branch, I think you just need to either open the existing .vcxproj (if it exists) or make a new one and add the sources to it.

Well, it was not maintained, so it was removed from the repository.

Just checked, it was removed between 0.5.0 and 0.6.0. The latest makefile.vc file is here (https://github.com/bitcoin/bitcoin/blob/3f64fa1369bde45034e451c4093e0852e59a1cdf/src/makefile.vc). You can use it as a starting point, but a lot changed since it was functional.

Thanks Pieter - but ouch - this is/was a manually maintained makefile (maybe you guys might be interested in my system that can automatically generate make files and handle dependencies even with the free VC++ compiler).

I don't know if I am really going to have enough free time to do this as I'm flat out on my own project (just have a look at how many commits I have made in https://github.com/ciyam/ciyam to get an idea) but if I do find a day or so to do this I will.

You should give unity builds a try. "SimpleDJ" in my signature uses that style.

I'm sure there are probably several good build systems out there - am comfortable with my own as it works perfectly for Windows and Linux - but manually putting together a large makefile is definitely a huge PITA (especially if it needs to be specifically told about headers).

I fully agree, and I think the other developers do as well.

It's just that all experiments with more automated build systems failed - either they didn't work for all currently supported platforms, or weren't maintained after being written.

Again, help is welcome.

Unity build is a system for organizing source files, it is not a separate tool and doesn't affect your Makefile or project file, except to make it orders of magnitude smaller. You can read about it here:

http://buffered.io/posts/the-magic-of-unity-builds

If you have Visual Studio or Xcode you can open up my "SimpleDJ" application (in my sig) and see how this all works.

Okay - the system I have developed works with an XML like "makefile" that is minimal (only source files and required libraries for each dependency).

It then uses OS specific "templates" that take it's structure and turn it into an actual platform specific makefile (my system can do VC++, BCB and g++ makefiles currently).

Maintenance of the system is actually very minimal (don't think I've had to change it much in the last few years actually). Header dependencies (only for VC++ in my system as both BCB and g++ handle these automatically) are handled via a specific tool (also not that complicated).

Link not working in China (perhaps blocked?) - will try and have a look at that tomorrow though.

To be more specific lets look at the official client sources. The qt/ directory contains all of the source files for the associated user interface classes:

https://github.com/bitcoin/bitcoin/tree/master/src/qt

We have

aboutdialog.cpp
aboutdialog.h
addressbookpage.cpp
...
walletmodel.h

Right now each of these is individually mentioned in the Makefile. What we could do instead is create a new "unity" .cpp / .h pair of files and manually #include these sources:

qt_ui.h

qt_ui.cpp

Given, it looks a little strange with .cpp files as includes, but the results are quite impressive. You can remove all of the #include lines from every associated .cpp and .h and just merge them into one non-repeated group of includes at the top of qt_ui.h and qt_ui.cpp. Preferably the bulk of includes would go into qt_ui.cpp (so they have a smaller scope).

It is instantly possible to see what the group of classes external include dependencies are (they are all in one place).

Adding, renaming, or removing individual UI component source files is done by editing the .h and .cpp, no changes to the Makefile are required. All of the individual .cpp / .h in the Makefile are replaced with the one qt_ui.h and qt_ui.cpp pair.

It is true that the granularity of compilation now becomes this larger set of classes, but the trade-off is well worth it.

The tool for VC++ that I am using just scans all the quoted #include's from the source files to work out the dependencies.

Provided you stick to a consistent coding style it solves this problem easily and without anything weird (just requires a couple of tools and the makefile template).

What are the advantages of LevelDB vs BDB?

Faster, less prone to corruption (in our setting), better compatibility between versions. At least, hopefully.

Also see this (http://bitcoin.stackexchange.com/questions/4809/what-technology-will-supersede-the-berkleydb-format-what-are-the-current-scalab) SE question.

Why not sqlite for the wallet (and maybe the blockchain) ?

Although not familiar with LevelDB I can say I was quite impressed with SQLite (used to use it for my own project although unfortunately it's not designed for concurrency due to its rather global locking approach).

SQLite works great with multithreaded apps you just need to set the appropriate locking model option. If you mean concurrency in the sense of different processes accessing the same database then yeah I agree (but that's not SQLite's target audience).

Interesting - my problem was actually with multi-threading not different processes - will have to look into that again if they have improved the locking (I last looked at that a few years back).

In my opinion, even sqlite is overkill for the wallet. It's yet another dependency (something you don't like, right?), and all we need is a simple key-value store that is read at startup and loaded into memory. Probably we'll move to a very simple custom append-only format with checksums.

For the blockchain: performance. LevelDB is exactly what we need: not more than a key-value store with atomic writes, with very good performance and consistency.

I use SQLite for all document formats, no matter how simple. Because it is transactional, robust, and performs well. I have taken the soci C++ wrapper and remodeled it to provide a very nice system of binding to C++ that takes advantage of all the object oriented metaphors. So it is simple to map primitive data types and object types to and from database entries.

As for integration and dependencies, SQLite is quite easy to integrate it comes as a single .c / .h file pair that you just add to your existing project. No Makefile, no build settings, nothing.

Let me point out that I don't like external dependencies. My opinion is that a repository should stand on its own. This is done by bringing in the sources for external dependencies directly into the source tree for the repo (I use "git-subtree" for that). I'm not a fan of using dynamic libraries at all. For desktop applications I much prefer to have everything "baked in" to a single .exe. Then there is no question about what people are running. Yes it is true that if there is a security vulnerability the software needs to be updated (you can't just update some shared .DLL) but I prefer this over the alternative.

To build an application it should be possible to clone the repository, open the associated IDE project file (Xcode or Visual Studio), press "Build", and get a correct result. Nothing more. No extra tools (no Python, no CMake, no Perl, nothing).

Agreed - the simpler the better IMO (have developed my own object DB but have yet to make it ACID).


Also agreed - you might be interested to take a look at the DB I developed as it works with streaming operators and fits very nicely with C++.

Good for you, but it certainly doesn't convince me for the wallet use case. The data being stored in there consists of complex data structures (keys, transactions), for which a serialization framework already exists. The only thing being written would be byte sequences. The advantage of SQL-based systems is easier aggregation of data from the database, but if the database layer can't inspect the data, that doesn't make much sense.

You could write an SQL-based wallet implementation that splits out wallet information over several SQL tables (I believe genjix' libbitcoin does that), and take advantage of what SQL adds, but at that point I think you need to rewrite the wallet implementation pretty much from scratch.


I still prefer not needing to integrate anything at all.


Yes, this is what we're going to do for LevelDB, as it doesn't even come as a dynamic library. Dependencies certainly complicate development - especially for portable applications - but in some cases there is just no better solution.

I've done that myself. I have a few projects where all I have are a handful of blobs, without any structure to the data. I still use SQLite because it is transactional and preserves the integrity of the writes, without me having to roll my own solution. LevelDB is a good alternative for that as well but if you ever want to start using relational database features then you'd either need two DBs or migrate code (both of which are acceptible, if you absolutely need relational features).

I do not know if this was mentioned already but it is important. Any bitcoin implementation that does not use OpenSSL must be compatible with OpenSSL if it is doing full validation of the block chain. The reason is because OpenSSL has a broken version of ECDSA, ie. it doesn't follow the ECDSA standards and does it own thing. Other ECDSA implementations may not take the OpenSSL differences into account and if a bitcoin implementation does not successfully implement ECDSA as OpenSSL does, then it could allow someone to create a fork between the two implementations.

HOLY!!!!!

Thanks for this heads up!!!!

Really?


Perhaps with optional build-openssl and build-bdb steps first, if your distro does not provide.  :)

If your OS makes it more difficult, get a better OS ;p

That's somewhat of an exaggeration. Let me explain.

Bitcoin uses DER encoding for signatures and EC private keys, and a serialization defined by the SEC2 standard for EC public keys. OpenSSL follows both these standards perfectly. It creates fully compliant ones, and parses them perfectly.

However, as DER-encoded signatures and SEC encoded public keys end up in the block chain, they are effectively part of Bitcoin's (implied) protocol specification as well.

Unfortunately, OpenSSL also supports signatures that do not follow the strict DER encoding (it supports anything encoded in BER, and even that is not required). For public keys, it accepts both encodings defined by SEC, plus a weird third own one. Because these non-standard signatures and public keys have always been accepted by every full node on the network (as those simply used OpenSSL), they became part of Bitcoin's implied protocol as well. For a long time nobody noticed this, but today there are alternative implementations that systematically create non-standard signatures for example.

There's an effort being done to outlaw these, as it will make the live of developers for alternative implementations easier. See pull request 1742 (https://github.com/bitcoin/bitcoin/pull/1742) for more information.

Yes, really. I'm on Windows. And I consider downloading and building external dependencies to be part of the build. Looking at build-msw.txt (https://github.com/bitcoin/bitcoin/blob/master/doc/build-msw.txt), I see:

Need to download boost, miniupnpc, BDB, and OpenSSL
Follow the build procedures for each of these. OpenSSL in particular requires Perl, a bunch of other stuff.

Then there's mingw32-make. What's that? I don't have that, I use Visual Studio. MinGW is a hassle.


Trolling aside, Visual Studio is currently the most productive environment for building C++ desktop applications (for me at least). At least once a year I evaluate the alternatives and nothing has dethroned it yet, although I desperately wish it would happen.

Rejecting keys and encodings which are not compliant with SEC standards in blocks could cause forking problems. There would need to be miner support to prevent the bad signatures or keys being included in blocks, before a client change was to take place.

Obviously you must know this already but I was just mentioning it.

It would be an improvement to get this through.

Sad that for such a new piece of technology, we're already struggling to maintain backward compatibility with older hacks.

I think users with older clients, holders of older bitcoins quite appreciate the struggle to maintain backwards compat.

Nobody wants to wake up in the morning, to discover that their money is unspendable outside of a required upgrade.

So have these non-standard keys or signatures already been included in the block-chain? That would cause a problem if ever a protocol change was wanted. You'd be stuck requiring to validate the signatures and keys as OpenSSL does it.

I think a new client could include a check point, validate the block hashes and then just assume that any tx / sig was valid (without checking it) if it precedes the check point?


Yeah I'm not saying backward compatibility is bad, but that it is already necessary for such a young project.

You could do that and hard code unspent outputs (That exist before the checkpoint) for validation purposes.

Yes, a lot (close to 100000, IIRC).

We're not doing signature verification right now for things buried deep enough in the chain and protected by a checkpoint. Once a significant part of the chain has only standard signatures left (I haven't seen any non-standard pubkey, except on testnet where I tried it myself), code to support the non-standard signatures is only required for implementation that want to do full validation of the entire history (which, imho, there should be). It's more of a benefit for lighter types of clients, which don't verify history.

Well regardless I looked into CryptoPP (http://www.cryptopp.com/) and it supports all the standard ECDSA signature formats as well as the odd one used in OpenSSL. So it would be a suitable alternative to OpenSSL (for ECDSA).

And it supports the hybrid keys too?

There's a function DSAConvertSignatureFormat() which handles the various formats. Not sure about the hybrid keys, can you go into more detail?

Thanks

Pieter knows moe about this. Basically OpenSSL has the uncompressed and compressed ECDSA public key formats which are defined in the SEC standards but it also has a hybrid format. http://openssl.sourcearchive.com/documentation/1.0.0e-3/crypto_2ec_2ec_8h_aa2c7ec2902c397d59393d357921e14b5.html Literally that is all the documentation gives you. I have no idea where Pieter Wuille learned all this stuff.

Right. SEC defines the compressed (0x02/0x03 + 32-byte X coordinate) and uncompressed (0x04 + 32-byte X coordinate + 32-byte Y coordinate) formats, but doesn't specify the hybrid format (0x06/0x07 + 32-byte X coordinate + 32-byte Y coordinate). From that link, it seems that the hybrid format may have been specified by that ANSI X9.62, but I won't spend $100 to buy the standard...

Anyway, it's that hybrid format that's bothersome. Making an implementation support it is trivial though - just change the 0x06 or 0x07 byte to a 0x04 and pass it that way to the crypto library.

I just learnt this from the OpenSSL source code, and the SEC document (see the secg (http://www.secg.org/index.php?action=secg,docs_secg) site).

It seems like almost every technical thread about bitcoin{d,-qt} needs to take a detour into the DB-land.

I just wanted to stress that the "append-only" is the key concept to understand what is required architecturally to implement Bitcoin efficiently. Incidentally it is also a key ingredient to make any Bitcoin implementation GAAP-compliant.

It is alway scary experience to post here a link to some Microsoft's web property. But this thread already has at least 2 posters who aren't scared of MSFT, so here it goes:

http://blogs.msdn.com/b/pathelland/archive/2007/06/14/accountants-don-t-use-erasers.aspx

LevelDB unfortunately will not be "exactly what we need" unless a significant reachitecturing is undertaken.

Mike Hearn had explained this succintly. I'll find the link and post it here.

https://bitcointalk.org/index.php?topic=94453.msg1048149#msg1048149

Gavin Andressen had estimated that the rearchitecturing of storage layer could take up to 3 years anywhere from 1 to 3 years while seriously impacting other intended deliverables. Again I'll find the link and post it here.

https://bitcointalk.org/index.php?topic=101011.msg1170970#msg1170970

That's simply false.

Keywords such as "slots:" are defined away with a macro when compiling the actual code. There is no specific preprocessor for Qt (there is the "moc" compiler, but it adds class introspection information and such that is linked as a separate file, it does not transform source code).

I have compiled Bitcoin-Qt with visual studio 2010 express. I have no time to maintain a parallel build system though, mingw works perfectly for our purposes.  I'm using Qt Creator as IDE, which is similar enough to VS but without need to buy an expensive license after 30 days.

But if you'd like to maintain a VS build system I can send you what I have.

BTW@ 2112 you can use bitcoin with leveldb right now if you want. See pull request https://github.com/bitcoin/bitcoin/pull/1677

Not too surprising, it's one hell of a weakness right now. BDB is just not the right fit for how we use it.


I was talking about the storage system, not wallet semantics (which is what you want changed to conform to those rules). Even if we move to an append-only wallet file, that doesn't mean anything will observably change.


Mike Hearn also implemented a first Bitcoin-on-LevelDB port himself (see pull request 1619 (https://github.com/bitcoin/bitcoin/pull/1619)), which was abandoned after I modified it to work on top of my rewrite of the validation engine (see pull request 1677 (https://github.com/bitcoin/bitcoin/pull/1677)).

The problem you probably were referring to is the fact that Bitcoin relied on reading uncommitted data during block validation, something that isn't supported by LevelDB (it just has atomic writes, no real database transactionality). Mike solved that in his port by writing a tiny caching layer around LevelDB. I solved it by avoiding the need for such operations altogether, with a nice performance improvement along the way.


He wasn't talking about the storage layer. He was referring to your request for changing the wallet semantics. I disagree that it would take that long, by the way, but I disagree it's within our scope right now. There are enough alternative wallets already, those are in a perfect place to experiment with different types of wallets.

By the way, changing the wallet storage (but just that) to use an append-only format is also already implemented (by me), but it had some issues left, and I felt other things were more important to work on.

Actually ever since Herb Sutter joined MS their compiler has been extremely standards compliant (although you do have to flick some compiler switches to get it to adhere correctly).

I don't actually use the IDE myself (being a console guy). The only big problem with not doing that is the lack of auto-dependencies (at least in the free version) although it isn't really that hard to construct your own make system (which I did for my own project) and there are other free ones out there.

Thanks to everyone for their informative replies, this has provided useful insights into some of the thinking that goes on in the development process.