Bitcoin Forum
May 19, 2019, 03:53:25 PM *
News: Latest Bitcoin Core release: 0.18.0 [Torrent] (New!)
 
   Home   Help Search Login Register More  
Pages: « 1 2 [3] 4 »  All
  Print  
Author Topic: Why is OpenSSL needed in the official client?  (Read 3820 times)
misterbigg
Legendary
*
Offline Offline

Activity: 1064
Merit: 1001



View Profile
September 29, 2012, 06:23:56 PM
 #41

In my opinion, even sqlite is overkill for the wallet. It's yet another dependency (something you don't like, right?)

I use SQLite for all document formats, no matter how simple. Because it is transactional, robust, and performs well. I have taken the soci C++ wrapper and remodeled it to provide a very nice system of binding to C++ that takes advantage of all the object oriented metaphors. So it is simple to map primitive data types and object types to and from database entries.

As for integration and dependencies, SQLite is quite easy to integrate it comes as a single .c / .h file pair that you just add to your existing project. No Makefile, no build settings, nothing.

Let me point out that I don't like external dependencies. My opinion is that a repository should stand on its own. This is done by bringing in the sources for external dependencies directly into the source tree for the repo (I use "git-subtree" for that). I'm not a fan of using dynamic libraries at all. For desktop applications I much prefer to have everything "baked in" to a single .exe. Then there is no question about what people are running. Yes it is true that if there is a security vulnerability the software needs to be updated (you can't just update some shared .DLL) but I prefer this over the alternative.

To build an application it should be possible to clone the repository, open the associated IDE project file (Xcode or Visual Studio), press "Build", and get a correct result. Nothing more. No extra tools (no Python, no CMake, no Perl, nothing).
1558281205
Hero Member
*
Offline Offline

Posts: 1558281205

View Profile Personal Message (Offline)

Ignore
1558281205
Reply with quote  #2

1558281205
Report to moderator
1558281205
Hero Member
*
Offline Offline

Posts: 1558281205

View Profile Personal Message (Offline)

Ignore
1558281205
Reply with quote  #2

1558281205
Report to moderator
NEW GAME FORMAT
JACKPOT UP TO $8000+
Guess The Symbols Of a Real Ethereum Hash
PLAY NOW
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
CIYAM
Legendary
*
Offline Offline

Activity: 1890
Merit: 1003


Ian Knowles - CIYAM Lead Developer


View Profile WWW
September 29, 2012, 06:27:41 PM
 #42

In my opinion, even sqlite is overkill for the wallet. It's yet another dependency (something you don't like, right?), and all we need is a simple key-value store that is read at startup and loaded into memory. Probably we'll move to a very simple custom append-only format with checksums.

For the blockchain: performance. LevelDB is exactly what we need: not more than a key-value store with atomic writes, with very good performance and consistency.

Agreed - the simpler the better IMO (have developed my own object DB but have yet to make it ACID).

I use SQLite for all document formats, no matter how simple. Because it is transactional, robust, and performs well. I have taken the soci C++ wrapper and remodeled it to provide a very nice system of binding to C++ that takes advantage of all the object oriented metaphors. So it is simple to map primitive data types and object types to and from database entries.

As for integration and dependencies, SQLite is quite easy to integrate it comes as a single .c / .h file pair that you just add to your existing project. No Makefile, no build settings, nothing.


Also agreed - you might be interested to take a look at the DB I developed as it works with streaming operators and fits very nicely with C++.

With CIYAM anyone can create 100% generated C++ web applications in literally minutes.

GPG Public Key | 1ciyam3htJit1feGa26p2wQ4aw6KFTejU
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1050
Merit: 1006


View Profile WWW
September 29, 2012, 06:36:55 PM
 #43

In my opinion, even sqlite is overkill for the wallet. It's yet another dependency (something you don't like, right?)

I use SQLite for all document formats, no matter how simple. Because it is transactional, robust, and performs well. I have taken the soci C++ wrapper and remodeled it to provide a very nice system of binding to C++ that takes advantage of all the object oriented metaphors. So it is simple to map primitive data types and object types to and from database entries.

Good for you, but it certainly doesn't convince me for the wallet use case. The data being stored in there consists of complex data structures (keys, transactions), for which a serialization framework already exists. The only thing being written would be byte sequences. The advantage of SQL-based systems is easier aggregation of data from the database, but if the database layer can't inspect the data, that doesn't make much sense.

You could write an SQL-based wallet implementation that splits out wallet information over several SQL tables (I believe genjix' libbitcoin does that), and take advantage of what SQL adds, but at that point I think you need to rewrite the wallet implementation pretty much from scratch.

Quote
As for integration and dependencies, SQLite is quite easy to integrate it comes as a single .c / .h file pair that you just add to your existing project. No Makefile, no build settings, nothing.

I still prefer not needing to integrate anything at all.

Quote
Let me point out that I don't like external dependencies. My opinion is that a repository should stand on its own. This is done by bringing in the sources for external dependencies directly into the source tree for the repo (I use "git-subtree" for that). I'm not a fan of using dynamic libraries at all.

Yes, this is what we're going to do for LevelDB, as it doesn't even come as a dynamic library. Dependencies certainly complicate development - especially for portable applications - but in some cases there is just no better solution.

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
misterbigg
Legendary
*
Offline Offline

Activity: 1064
Merit: 1001



View Profile
September 29, 2012, 06:40:55 PM
 #44

The only thing being written would be byte sequences.

I've done that myself. I have a few projects where all I have are a handful of blobs, without any structure to the data. I still use SQLite because it is transactional and preserves the integrity of the writes, without me having to roll my own solution. LevelDB is a good alternative for that as well but if you ever want to start using relational database features then you'd either need two DBs or migrate code (both of which are acceptible, if you absolutely need relational features).
MatthewLM
Legendary
*
Offline Offline

Activity: 1092
Merit: 1000



View Profile WWW
September 29, 2012, 07:09:27 PM
 #45

I do not know if this was mentioned already but it is important. Any bitcoin implementation that does not use OpenSSL must be compatible with OpenSSL if it is doing full validation of the block chain. The reason is because OpenSSL has a broken version of ECDSA, ie. it doesn't follow the ECDSA standards and does it own thing. Other ECDSA implementations may not take the OpenSSL differences into account and if a bitcoin implementation does not successfully implement ECDSA as OpenSSL does, then it could allow someone to create a fork between the two implementations.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
misterbigg
Legendary
*
Offline Offline

Activity: 1064
Merit: 1001



View Profile
September 29, 2012, 07:41:48 PM
 #46

I do not know if this was mentioned already but it is important. Any bitcoin implementation that does not use OpenSSL must be compatible with OpenSSL if it is doing full validation of the block chain. The reason is because OpenSSL has a broken version of ECDSA, ie. it doesn't follow the ECDSA standards and does it own thing.

HOLY!!!!!

Thanks for this heads up!!!!
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1554
Merit: 1005


View Profile
September 29, 2012, 07:50:27 PM
 #47

I looked at the build steps for the official client and ran home screaming to mommy.

Really?

Code:
$ cd bitcoin/src
$ make -f makefile.unix

Perhaps with optional build-openssl and build-bdb steps first, if your distro does not provide.  Smiley

If your OS makes it more difficult, get a better OS ;p


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1050
Merit: 1006


View Profile WWW
September 29, 2012, 07:56:10 PM
 #48

I do not know if this was mentioned already but it is important. Any bitcoin implementation that does not use OpenSSL must be compatible with OpenSSL if it is doing full validation of the block chain. The reason is because OpenSSL has a broken version of ECDSA, ie. it doesn't follow the ECDSA standards and does it own thing. Other ECDSA implementations may not take the OpenSSL differences into account and if a bitcoin implementation does not successfully implement ECDSA as OpenSSL does, then it could allow someone to create a fork between the two implementations.

That's somewhat of an exaggeration. Let me explain.

Bitcoin uses DER encoding for signatures and EC private keys, and a serialization defined by the SEC2 standard for EC public keys. OpenSSL follows both these standards perfectly. It creates fully compliant ones, and parses them perfectly.

However, as DER-encoded signatures and SEC encoded public keys end up in the block chain, they are effectively part of Bitcoin's (implied) protocol specification as well.

Unfortunately, OpenSSL also supports signatures that do not follow the strict DER encoding (it supports anything encoded in BER, and even that is not required). For public keys, it accepts both encodings defined by SEC, plus a weird third own one. Because these non-standard signatures and public keys have always been accepted by every full node on the network (as those simply used OpenSSL), they became part of Bitcoin's implied protocol as well. For a long time nobody noticed this, but today there are alternative implementations that systematically create non-standard signatures for example.

There's an effort being done to outlaw these, as it will make the live of developers for alternative implementations easier. See pull request 1742 for more information.

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
misterbigg
Legendary
*
Offline Offline

Activity: 1064
Merit: 1001



View Profile
September 29, 2012, 08:05:12 PM
 #49

I looked at the build steps for the official client and ran home screaming to mommy.

Really?

Yes, really. I'm on Windows. And I consider downloading and building external dependencies to be part of the build. Looking at build-msw.txt, I see:

Need to download boost, miniupnpc, BDB, and OpenSSL
Follow the build procedures for each of these. OpenSSL in particular requires Perl, a bunch of other stuff.

Then there's mingw32-make. What's that? I don't have that, I use Visual Studio. MinGW is a hassle.

Quote
If your OS makes it more difficult, get a better OS ;p

Trolling aside, Visual Studio is currently the most productive environment for building C++ desktop applications (for me at least). At least once a year I evaluate the alternatives and nothing has dethroned it yet, although I desperately wish it would happen.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092
Merit: 1000



View Profile WWW
September 29, 2012, 08:15:32 PM
 #50

Unfortunately, OpenSSL also supports signatures that do not follow the strict DER encoding (it supports anything encoded in BER, and even that is not required). For public keys, it accepts both encodings defined by SEC, plus a weird third own one. Because these non-standard signatures and public keys have always been accepted by every full node on the network (as those simply used OpenSSL), they became part of Bitcoin's implied protocol as well. For a long time nobody noticed this, but today there are alternative implementations that systematically create non-standard signatures for example.

There's an effort being done to outlaw these, as it will make the live of developers for alternative implementations easier. See pull request 1742 for more information.


Rejecting keys and encodings which are not compliant with SEC standards in blocks could cause forking problems. There would need to be miner support to prevent the bad signatures or keys being included in blocks, before a client change was to take place.

Obviously you must know this already but I was just mentioning it.

It would be an improvement to get this through.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
misterbigg
Legendary
*
Offline Offline

Activity: 1064
Merit: 1001



View Profile
September 29, 2012, 08:16:53 PM
 #51

Sad that for such a new piece of technology, we're already struggling to maintain backward compatibility with older hacks.
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1554
Merit: 1005


View Profile
September 29, 2012, 08:20:27 PM
 #52

Sad that for such a new piece of technology, we're already struggling to maintain backward compatibility with older hacks.

I think users with older clients, holders of older bitcoins quite appreciate the struggle to maintain backwards compat.

Nobody wants to wake up in the morning, to discover that their money is unspendable outside of a required upgrade.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
MatthewLM
Legendary
*
Offline Offline

Activity: 1092
Merit: 1000



View Profile WWW
September 29, 2012, 08:25:20 PM
 #53

So have these non-standard keys or signatures already been included in the block-chain? That would cause a problem if ever a protocol change was wanted. You'd be stuck requiring to validate the signatures and keys as OpenSSL does it.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
misterbigg
Legendary
*
Offline Offline

Activity: 1064
Merit: 1001



View Profile
September 29, 2012, 08:27:19 PM
 #54

So have these non-standard keys or signatures already been included in the block-chain? That would cause a problem if ever a protocol change was wanted. You'd be stuck requiring to validate the signatures and keys as OpenSSL does it.

I think a new client could include a check point, validate the block hashes and then just assume that any tx / sig was valid (without checking it) if it precedes the check point?

I think users with older clients, holders of older bitcoins quite appreciate the struggle to maintain

Yeah I'm not saying backward compatibility is bad, but that it is already necessary for such a young project.
MatthewLM
Legendary
*
Offline Offline

Activity: 1092
Merit: 1000



View Profile WWW
September 29, 2012, 08:31:17 PM
 #55

So have these non-standard keys or signatures already been included in the block-chain? That would cause a problem if ever a protocol change was wanted. You'd be stuck requiring to validate the signatures and keys as OpenSSL does it.

I think a new client could include a check point, validate the block hashes and then just assume that any tx / sig was valid (without checking it) if it precedes the check point?


You could do that and hard code unspent outputs (That exist before the checkpoint) for validation purposes.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1050
Merit: 1006


View Profile WWW
September 29, 2012, 08:33:28 PM
 #56

So have these non-standard keys or signatures already been included in the block-chain? That would cause a problem if ever a protocol change was wanted. You'd be stuck requiring to validate the signatures and keys as OpenSSL does it.

Yes, a lot (close to 100000, IIRC).

We're not doing signature verification right now for things buried deep enough in the chain and protected by a checkpoint. Once a significant part of the chain has only standard signatures left (I haven't seen any non-standard pubkey, except on testnet where I tried it myself), code to support the non-standard signatures is only required for implementation that want to do full validation of the entire history (which, imho, there should be). It's more of a benefit for lighter types of clients, which don't verify history.

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
misterbigg
Legendary
*
Offline Offline

Activity: 1064
Merit: 1001



View Profile
September 29, 2012, 08:35:39 PM
 #57

Well regardless I looked into CryptoPP and it supports all the standard ECDSA signature formats as well as the odd one used in OpenSSL. So it would be a suitable alternative to OpenSSL (for ECDSA).
MatthewLM
Legendary
*
Offline Offline

Activity: 1092
Merit: 1000



View Profile WWW
September 29, 2012, 08:43:04 PM
 #58

And it supports the hybrid keys too?

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
misterbigg
Legendary
*
Offline Offline

Activity: 1064
Merit: 1001



View Profile
September 29, 2012, 08:44:17 PM
 #59

And it supports the hybrid keys too?

There's a function DSAConvertSignatureFormat() which handles the various formats. Not sure about the hybrid keys, can you go into more detail?

Thanks
MatthewLM
Legendary
*
Offline Offline

Activity: 1092
Merit: 1000



View Profile WWW
September 29, 2012, 08:51:56 PM
 #60

Pieter knows moe about this. Basically OpenSSL has the uncompressed and compressed ECDSA public key formats which are defined in the SEC standards but it also has a hybrid format. http://openssl.sourcearchive.com/documentation/1.0.0e-3/crypto_2ec_2ec_8h_aa2c7ec2902c397d59393d357921e14b5.html Literally that is all the documentation gives you. I have no idea where Pieter Wuille learned all this stuff.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
Pages: « 1 2 [3] 4 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!