Why is OpenSSL needed in the official client?

[misterbigg]

In my opinion, even sqlite is overkill for the wallet. It's yet another dependency (something you don't like, right?)

I use SQLite for all document formats, no matter how simple. Because it is transactional, robust, and performs well. I have taken the soci C++ wrapper and remodeled it to provide a very nice system of binding to C++ that takes advantage of all the object oriented metaphors. So it is simple to map primitive data types and object types to and from database entries.

As for integration and dependencies, SQLite is quite easy to integrate it comes as a single .c / .h file pair that you just add to your existing project. No Makefile, no build settings, nothing.

Let me point out that I don't like external dependencies. My opinion is that a repository should stand on its own. This is done by bringing in the sources for external dependencies directly into the source tree for the repo (I use "git-subtree" for that). I'm not a fan of using dynamic libraries at all. For desktop applications I much prefer to have everything "baked in" to a single .exe. Then there is no question about what people are running. Yes it is true that if there is a security vulnerability the software needs to be updated (you can't just update some shared .DLL) but I prefer this over the alternative.

To build an application it should be possible to clone the repository, open the associated IDE project file (Xcode or Visual Studio), press "Build", and get a correct result. Nothing more. No extra tools (no Python, no CMake, no Perl, nothing).

[CIYAM]

In my opinion, even sqlite is overkill for the wallet. It's yet another dependency (something you don't like, right?), and all we need is a simple key-value store that is read at startup and loaded into memory. Probably we'll move to a very simple custom append-only format with checksums.

For the blockchain: performance. LevelDB is exactly what we need: not more than a key-value store with atomic writes, with very good performance and consistency.

Agreed - the simpler the better IMO (have developed my own object DB but have yet to make it ACID).

I use SQLite for all document formats, no matter how simple. Because it is transactional, robust, and performs well. I have taken the soci C++ wrapper and remodeled it to provide a very nice system of binding to C++ that takes advantage of all the object oriented metaphors. So it is simple to map primitive data types and object types to and from database entries.

As for integration and dependencies, SQLite is quite easy to integrate it comes as a single .c / .h file pair that you just add to your existing project. No Makefile, no build settings, nothing.

Also agreed - you might be interested to take a look at the DB I developed as it works with streaming operators and fits very nicely with C++.

[Pieter Wuille]

In my opinion, even sqlite is overkill for the wallet. It's yet another dependency (something you don't like, right?)

I use SQLite for all document formats, no matter how simple. Because it is transactional, robust, and performs well. I have taken the soci C++ wrapper and remodeled it to provide a very nice system of binding to C++ that takes advantage of all the object oriented metaphors. So it is simple to map primitive data types and object types to and from database entries.

Good for you, but it certainly doesn't convince me for the wallet use case. The data being stored in there consists of complex data structures (keys, transactions), for which a serialization framework already exists. The only thing being written would be byte sequences. The advantage of SQL-based systems is easier aggregation of data from the database, but if the database layer can't inspect the data, that doesn't make much sense.

You could write an SQL-based wallet implementation that splits out wallet information over several SQL tables (I believe genjix' libbitcoin does that), and take advantage of what SQL adds, but at that point I think you need to rewrite the wallet implementation pretty much from scratch.

As for integration and dependencies, SQLite is quite easy to integrate it comes as a single .c / .h file pair that you just add to your existing project. No Makefile, no build settings, nothing.

I still prefer not needing to integrate anything at all.

Let me point out that I don't like external dependencies. My opinion is that a repository should stand on its own. This is done by bringing in the sources for external dependencies directly into the source tree for the repo (I use "git-subtree" for that). I'm not a fan of using dynamic libraries at all.

Yes, this is what we're going to do for LevelDB, as it doesn't even come as a dynamic library. Dependencies certainly complicate development - especially for portable applications - but in some cases there is just no better solution.

[misterbigg]

The only thing being written would be byte sequences.

I've done that myself. I have a few projects where all I have are a handful of blobs, without any structure to the data. I still use SQLite because it is transactional and preserves the integrity of the writes, without me having to roll my own solution. LevelDB is a good alternative for that as well but if you ever want to start using relational database features then you'd either need two DBs or migrate code (both of which are acceptible, if you absolutely need relational features).

[MatthewLM]

I do not know if this was mentioned already but it is important. Any bitcoin implementation that does not use OpenSSL must be compatible with OpenSSL if it is doing full validation of the block chain. The reason is because OpenSSL has a broken version of ECDSA, ie. it doesn't follow the ECDSA standards and does it own thing. Other ECDSA implementations may not take the OpenSSL differences into account and if a bitcoin implementation does not successfully implement ECDSA as OpenSSL does, then it could allow someone to create a fork between the two implementations.

[misterbigg]

I do not know if this was mentioned already but it is important. Any bitcoin implementation that does not use OpenSSL must be compatible with OpenSSL if it is doing full validation of the block chain. The reason is because OpenSSL has a broken version of ECDSA, ie. it doesn't follow the ECDSA standards and does it own thing.

HOLY!!!!!

Thanks for this heads up!!!!

[jgarzik]

I looked at the build steps for the official client and ran home screaming to mommy.

Really?

Code:

$ cd bitcoin/src
$ make -f makefile.unix

Perhaps with optional build-openssl and build-bdb steps first, if your distro does not provide.  

If your OS makes it more difficult, get a better OS ;p

[Pieter Wuille]

I do not know if this was mentioned already but it is important. Any bitcoin implementation that does not use OpenSSL must be compatible with OpenSSL if it is doing full validation of the block chain. The reason is because OpenSSL has a broken version of ECDSA, ie. it doesn't follow the ECDSA standards and does it own thing. Other ECDSA implementations may not take the OpenSSL differences into account and if a bitcoin implementation does not successfully implement ECDSA as OpenSSL does, then it could allow someone to create a fork between the two implementations.

That's somewhat of an exaggeration. Let me explain.

Bitcoin uses DER encoding for signatures and EC private keys, and a serialization defined by the SEC2 standard for EC public keys. OpenSSL follows both these standards perfectly. It creates fully compliant ones, and parses them perfectly.

However, as DER-encoded signatures and SEC encoded public keys end up in the block chain, they are effectively part of Bitcoin's (implied) protocol specification as well.

Unfortunately, OpenSSL also supports signatures that do not follow the strict DER encoding (it supports anything encoded in BER, and even that is not required). For public keys, it accepts both encodings defined by SEC, plus a weird third own one. Because these non-standard signatures and public keys have always been accepted by every full node on the network (as those simply used OpenSSL), they became part of Bitcoin's implied protocol as well. For a long time nobody noticed this, but today there are alternative implementations that systematically create non-standard signatures for example.

There's an effort being done to outlaw these, as it will make the live of developers for alternative implementations easier. See pull request 1742 for more information.

[misterbigg]

I looked at the build steps for the official client and ran home screaming to mommy.

Really?

Yes, really. I'm on Windows. And I consider downloading and building external dependencies to be part of the build. Looking at build-msw.txt, I see:

Need to download boost, miniupnpc, BDB, and OpenSSL
Follow the build procedures for each of these. OpenSSL in particular requires Perl, a bunch of other stuff.

Then there's mingw32-make. What's that? I don't have that, I use Visual Studio. MinGW is a hassle.

If your OS makes it more difficult, get a better OS ;p

Trolling aside, Visual Studio is currently the most productive environment for building C++ desktop applications (for me at least). At least once a year I evaluate the alternatives and nothing has dethroned it yet, although I desperately wish it would happen.

[MatthewLM]

Unfortunately, OpenSSL also supports signatures that do not follow the strict DER encoding (it supports anything encoded in BER, and even that is not required). For public keys, it accepts both encodings defined by SEC, plus a weird third own one. Because these non-standard signatures and public keys have always been accepted by every full node on the network (as those simply used OpenSSL), they became part of Bitcoin's implied protocol as well. For a long time nobody noticed this, but today there are alternative implementations that systematically create non-standard signatures for example.

There's an effort being done to outlaw these, as it will make the live of developers for alternative implementations easier. See pull request 1742 for more information.

Rejecting keys and encodings which are not compliant with SEC standards in blocks could cause forking problems. There would need to be miner support to prevent the bad signatures or keys being included in blocks, before a client change was to take place.

Obviously you must know this already but I was just mentioning it.

It would be an improvement to get this through.

[misterbigg]

Sad that for such a new piece of technology, we're already struggling to maintain backward compatibility with older hacks.

[jgarzik]

Sad that for such a new piece of technology, we're already struggling to maintain backward compatibility with older hacks.

I think users with older clients, holders of older bitcoins quite appreciate the struggle to maintain backwards compat.

Nobody wants to wake up in the morning, to discover that their money is unspendable outside of a required upgrade.

[MatthewLM]

So have these non-standard keys or signatures already been included in the block-chain? That would cause a problem if ever a protocol change was wanted. You'd be stuck requiring to validate the signatures and keys as OpenSSL does it.

[misterbigg]

So have these non-standard keys or signatures already been included in the block-chain? That would cause a problem if ever a protocol change was wanted. You'd be stuck requiring to validate the signatures and keys as OpenSSL does it.

I think a new client could include a check point, validate the block hashes and then just assume that any tx / sig was valid (without checking it) if it precedes the check point?

I think users with older clients, holders of older bitcoins quite appreciate the struggle to maintain

Yeah I'm not saying backward compatibility is bad, but that it is already necessary for such a young project.

[MatthewLM]

So have these non-standard keys or signatures already been included in the block-chain? That would cause a problem if ever a protocol change was wanted. You'd be stuck requiring to validate the signatures and keys as OpenSSL does it.

I think a new client could include a check point, validate the block hashes and then just assume that any tx / sig was valid (without checking it) if it precedes the check point?

You could do that and hard code unspent outputs (That exist before the checkpoint) for validation purposes.

[Pieter Wuille]

So have these non-standard keys or signatures already been included in the block-chain? That would cause a problem if ever a protocol change was wanted. You'd be stuck requiring to validate the signatures and keys as OpenSSL does it.

Yes, a lot (close to 100000, IIRC).

We're not doing signature verification right now for things buried deep enough in the chain and protected by a checkpoint. Once a significant part of the chain has only standard signatures left (I haven't seen any non-standard pubkey, except on testnet where I tried it myself), code to support the non-standard signatures is only required for implementation that want to do full validation of the entire history (which, imho, there should be). It's more of a benefit for lighter types of clients, which don't verify history.

[misterbigg]

Well regardless I looked into CryptoPP and it supports all the standard ECDSA signature formats as well as the odd one used in OpenSSL. So it would be a suitable alternative to OpenSSL (for ECDSA).

[MatthewLM]

And it supports the hybrid keys too?

[misterbigg]

And it supports the hybrid keys too?

There's a function DSAConvertSignatureFormat() which handles the various formats. Not sure about the hybrid keys, can you go into more detail?

Thanks

[MatthewLM]

Pieter knows moe about this. Basically OpenSSL has the uncompressed and compressed ECDSA public key formats which are defined in the SEC standards but it also has a hybrid format. http://openssl.sourcearchive.com/documentation/1.0.0e-3/crypto_2ec_2ec_8h_aa2c7ec2902c397d59393d357921e14b5.html Literally that is all the documentation gives you. I have no idea where Pieter Wuille learned all this stuff.