Bitcoin Forum
April 24, 2024, 11:18:02 PM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1] 2 »  All
  Print  
Author Topic: Bitcoind CentOS Clean Compile On AWS EC2  (Read 7360 times)
neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
June 20, 2013, 10:56:33 PM
Last edit: June 21, 2013, 06:55:43 AM by neonzeon
 #1

PRIMARY GOAL: A clean compile of bitcoind on a freshly instantiated Amazon EC2 server


Join this journey by adding the topic to your watch list (especially you, weex and kjj Smiley

http://t1.gstatic.com/images?q=tbn:ANd9GcSr3Ymfo70la581hp_n1dgmEXJysnXAqVvXotXzCLqYMhqXeuqa

1. What I have so far found does NOT work:


1.1 Michael Steurer's guide http://www.staff.tugraz.at/michael.steurer/Bitcoin-CentOS5.5-V1.pdf

Why? 1.1.1 Only a few of the (huge) boost library components are needed
        1.1.2 The libraries are statically linked, but then make calls like dlopen() to load dynamic libraries
        1.1.3 At least in my compile the boost had python!!! dependencies

1.2 David Sterry (weex) and kjj's GIT solution at https://github.com/weex/bitcoind-centos

Why? 1.2.1 The file makefile.new contains several undefined variables such as $(BOOST_INCLUDE_PATH) $(BDB_INCLUDE_PATH) $(OPENSSL_INCLUDE_PATH)
        1.2.2 Since it's based on the Steurer guide above, the static/dynamic issue remain

2.3 Laszlo Hanyecz's guide at  http://btcsec.com/files/other/bitcoin-linuxbuild.pdf
     This is the famous dude who paid 10000 bitcoins to have pizza delivered to his house in Florida.
     This amazing story happened right here on this forum https://bitcointalk.org/?topic=137.0

Why? The guide is old and wxwidgets is not required anymore, but is the basis for all the other guides, so it's good to have handy.


2. PLAN


2.1 Get a free amazon EC2 cloud server as in http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html
2.2 Instantiate "the latest" CentOS image
2.3 Install the necessary build tools
2.4 Build the libraries that bitcoind needs
2.5 Build bitcoind itself
2.6 Discuss, document & refine every step until they are clear enough for an unsophisticad user to read the document & repeat the feat.







1714000682
Hero Member
*
Offline Offline

Posts: 1714000682

View Profile Personal Message (Offline)

Ignore
1714000682
Reply with quote  #2

1714000682
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
1714000682
Hero Member
*
Offline Offline

Posts: 1714000682

View Profile Personal Message (Offline)

Ignore
1714000682
Reply with quote  #2

1714000682
Report to moderator
1714000682
Hero Member
*
Offline Offline

Posts: 1714000682

View Profile Personal Message (Offline)

Ignore
1714000682
Reply with quote  #2

1714000682
Report to moderator
1714000682
Hero Member
*
Offline Offline

Posts: 1714000682

View Profile Personal Message (Offline)

Ignore
1714000682
Reply with quote  #2

1714000682
Report to moderator
neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
June 20, 2013, 11:09:42 PM
 #2

Day 1...

1. Get an Amazon AWS EC2 account and instantiate my free-for-a-year CentOS machine in the EC2.

2. I will also have to download putty.exe on my Windows machine in order to SSH into the cloud machine.

I think I will call my centos machine bitCloud...

Gonna read http://www.techrepublic.com/blog/datacenter/getting-to-the-command-line-starting-up-your-aws-ec2-machine-with-ssh/5106 and report back when bitCloud is up and running.
neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
June 21, 2013, 07:11:15 AM
Last edit: June 23, 2013, 06:34:14 PM by neonzeon
 #3

Day 2...

a. Got the Amazon Web Services account set up (easy), and installed the free ECS instance of this Centos http://goo.gl/QX7N6

b. Amazon generated a public-private key pair for me, which I could download.

c. Installed putty.exe on my windows machine.

d. Used puttygen.exe (that came with putty) to import the Amazon-generated key into putty.

e. Was then able to use putty to connect to the ssh port of my new machine as root.

f. Installed all the development tools onto the centos server by typing  yum groupinstall 'Development Tools'
AliceWonder
Full Member
***
Offline Offline

Activity: 168
Merit: 100



View Profile
June 21, 2013, 07:18:37 AM
 #4

Hi,

I've been using CentOS for years.
Easiest thing really is to create a local install which mirrors the packages you have installed on the remote server. It's free, why not?

Then you can do things like code locally and rsync over ssh to your remote server and it works.

QuarkCoin - what I believe bitcoin was intended to be. On reddit: http://www.reddit.com/r/QuarkCoin/
walletrecoveryservices
Member
**
Offline Offline

Activity: 89
Merit: 21



View Profile WWW
June 21, 2013, 07:57:33 AM
 #5

another too

Providing Cryptocurrency Wallet, Password and Seed Recovery Services since 2013
neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
June 21, 2013, 03:13:04 PM
 #6


Easiest thing really is to create a local install which mirrors the packages you have installed on the remote server.


Its a great idea. I'm doing it on a spare box I happen to have.

However, one of the key reason for the Amazon Web Services EC2 machine is that anyone interested in helping/joining can exactly duplicate this setup for free or at very low cost.

Is there an easy way to use the EC2 install image to burn a boot CD?

serp
Sr. Member
****
Offline Offline

Activity: 245
Merit: 250


@serp


View Profile WWW
June 21, 2013, 03:42:12 PM
 #7

A few things to watch if you are using the free tier is the number of I/O's you use on your EBS (you get 2 million/month free) and your bandwidth (you get 15 GB/month free).  I've not tried it but if you plan to run it continuously then you might want to keep an eye on these things.  I'd be interested in hearing how it goes.

Also, I understand if you want to stick with CentOS, but I wanted to mention it would probably be super easy if you wanted to just use one of their Ubuntu server images.  You could just install the O/S then type 'sudo apt-get install bitcoind' and probably be done.

neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
June 22, 2013, 01:36:19 AM
 #8

I wanted to mention it would probably be super easy if you wanted to just use one of their Ubuntu server images.  
You could just install the O/S then type 'sudo apt-get install bitcoind' and probably be done.

Serp, if you have access to a bitcoind built on ubuntu, could you post the output of a

Code:
ldd bitcoind

command?

In fact, any ldd output, especially on a 6.4 CentOS would be great.
neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
June 23, 2013, 05:35:21 PM
Last edit: June 23, 2013, 06:35:32 PM by neonzeon
 #9

DAY 3...

I found this link http://wiki.centos.org/HowTos/RebuildSRPM on the CentOS wiki.

Quote
Even if the software is not available in CentOS, you should always try to get or build a RPM for the software, since the advantages of using a package management system will compensate for the work you will have building the package.

I agree, so Google found me Tom van der Woerdt http://bitcoin.stackexchange.com/questions/10467/how-do-i-go-about-installing-a-bitcoin-daemon-in-centos-linux

He says
Quote
This is the bitcoin.spec file I use to build these packages

Code:
Name:           bitcoin
Version:        0.8.1
Release:        1%{?dist}
Summary:        bitcoin
License:        GPL
URL:            http://bitcoin.org
Source0:        http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-%{version}/bitcoin-%{version}-linux.tar.gz
%description
bitcoin
%package qt
Summary:        bitcoin-qt
%description qt
bitcoin-qt
%prep
%setup -q -n bitcoin-%{version}-linux
%build
%install
mkdir -p %{buildroot}%{_bindir}
%ifarch x86_64
cp -R bin/64/* %{buildroot}%{_bindir}
%endif
%ifarch i686
cp -R bin/32/* %{buildroot}%{_bindir}
%endif
%clean
rm -rf %{buildroot}
%files
%doc README COPYING
/usr/bin/bitcoind
%files qt
/usr/bin/bitcoin-qt
%changelog

I think this is the way to go, though I would really like to build a non-qt version of bitcoind.

This might be a good start and a first attempt to know more about how an rpm package works.



neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
June 23, 2013, 09:24:58 PM
Last edit: June 24, 2013, 10:58:59 AM by neonzeon
 #10

First Bitcoind Compile Attempt on CentOS 6.4

Woohoo! Today I found some time to prepare for an initial compile of bitcoind from https://github.com/bitcoin/bitcoin.github.com.

1. First I wiped the existing AWS EC2 machine, re-instantiating a fresh CentOS 6.4 machine on the Amazon Cloud and logging in via ssh/putty.

Code:
[root]$ echo "Your fresh AWS EC2 machine is $(cat /proc/version*)"
Your fresh AWS EC2 machine is Linux version 2.6.32-279.el6.x86_64 (mockbuild@c6b9.bsys.dev.centos.org)

2. Install dependencies as far as I could figure them out

Code:
yum update
yum install 'Development tools'
yum install openssl-devel
yum install boost
yum install db4

3. Add a new user abc and pull the bitcoin source code from github

Code:
adduser abc; su abc; mkdir src; cd src
git clone git://github.com/bitcoin/bitcoin.git
cd /home/abc/src/bitcoin/src

4. Change the variable USE_UPNP in file makefile.unix so the library miniUPNP is not included.

I apologise for using the obscure sed editor here but this way anyone can accurately duplicate the result.

Code:
 sed -i -e "s/^USE_UPNP.*/USE_UPNP:=-/" makefile.unix

5. Finally, try a compile of bitcoind as per the instructions in the bitcoin/doc directory

Code:
[abc]$ cd /home/abc/src/bitcoin/src
[abc]$ make -f makefile.unix

6. The result? Well, a bunch of files compiled happily until this:

Code:
g++ -c -O2 -pthread -Wall -Wextra -Wformat -Wformat-security -Wno-unused-parameter -g -DBOOST_SPIRIT_THREADSAFE -D_FILE_OFFSET_BITS=64 -I/home/abc/src/bitcoin/src -I/home/abc/src/bitcoin/src/obj -DUSE_IPV6=1 -I/home/abc/src/bitcoin/src/leveldb/include -I/home/abc/src/bitcoin/src/leveldb/helpers -DHAVE_BUILD_INFO -fno-stack-protector -fstack-protector-all -Wstack-protector -D_FORTIFY_SOURCE=2  -MMD -MF obj/alert.d -o obj/alert.o alert.cpp

alert.cpp:6:53: warning: boost/algorithm/string/classification.hpp: No such file or directory
alert.cpp:7:46: warning: boost/algorithm/string/replace.hpp: No such file or directory
alert.cpp:8:29: warning: boost/foreach.hpp: No such file or directory

In file included from alert.h:13, from alert.cpp:11:

util.h:23:29: warning: boost/version.hpp: No such file or directory
util.h:24:28: warning: boost/thread.hpp: No such file or directory
util.h:25:32: warning: boost/filesystem.hpp: No such file or directory

<snip>

Doh. It seems as if a bunch of boost files are not found.

7. Not going to give up that easily. Go back to root, yum-install boost-development package, try again.

Code:
[abc]$ exit # To root user
[root]# yum install boost-devel
Package boost-devel-1.41.0-17.el6_4.x86_64 installed
[root]# su abc # Become user abc again
[abc]$ cd /home/abc/src/bitcoin/src # where the makefile is
[abc]$ make -f makefile.unix

The above errors go away!

Code:
/bin/sh ../share/genbuild.sh obj/build.h
g++ -c -O2 -pthread -Wall -Wextra -Wformat -Wformat-security -Wno-unused-parameter -g -DBOOST_SPIRIT_THREADSAFE -D_FILE_OFFSET_BITS=64 -I/home/abc/src/bitcoin/src -I/home/abc/src/bitcoin/src/obj -DUSE_IPV6=1 -I/home/abc/src/bitcoin/src/leveldb/include -I/home/abc/src/bitcoin/src/leveldb/helpers -DHAVE_BUILD_INFO -fno-stack-protector -fstack-protector-all -Wstack-protector -D_FORTIFY_SOURCE=2  -MMD -MF obj/key.d -o obj/key.o key.cpp
key.cpp:5:27: warning: openssl/ecdsa.h: No such file or directory
key.cpp:16: error: ‘EC_KEY’ was not declared in this scope
key.cpp:55: error: ‘ECDSA_SIG’ was not declared in this scope

The new errors are kind-of expected, because I read somewhere that openssl gets compiled without EC (elliptic curves) due to some patent issue.  

8. New quest would be to find an openssl with EC turned on.

In https://bitcointalk.org/index.php?topic=9476.msg991835#msg991835 Jeff Garzik, bitcoin core dev team guru, says:
Quote
From: Jeff Garzik, bitcoin core dev team   
Re: ECDSA dropped out of openssl 1.0.0b  June 26, 2012, 05:39:39 PM

The preferred "fixes" for Red Hat, CentOS, Fedora systems are, if you want to do it yourself,

1) Download SRPM
2) Download associated source code from openssl.org
3) Edit SPECS/openssl.spec,
   a) replacing source tarball filename with the downloaded one
   b) comment out all references to source1
   c) remove the "no-ec" stuff from the configure line
4) rebuild with "rpmbuild -ba SPECS/openssl.spec" or similar
5) install build rpms found in RPMS/

Not yet sure what all those steps would translate to on my AWS EC2 CentOS box, but will be finding out over the next few days.  


8. Ok, think I have Jeff's step 1 figured out - it's downloading a .src.rpm fetch from the CentOS mother code vault Smiley

Code:
[root]# su abc # Currently root, become user abc
[abc]$ cd /home/abc # Go to abc's home directory
[abcl]$ # Use curl to download the openssl SRPM (Source RPM) package
[abc]$  curl -O http://vault.centos.org/6.4/os/Source/SPackages/openssl-1.0.0-27.el6.src.rpm
(4.11 MB/s) - “openssl-1.0.0-27.el6.src.rpm” saved [3419292/3419292]

Comments & tips welcome.

I also read this http://wiki.centos.org/TipsAndTricks/YumAndRPM - everything is not 100% clear yet but 14.1 seems important.
Quote
14. Get set up for rebuilding packages as yourself, not root
Sometimes you just have to rebuild that package - maybe only to use some configuration option which just isn't there in the official package.
So here's how to rebuild your packages in your home directory - with your own user account.

Material to read:

http://wiki.centos.org/HowTos/SetupRpmBuildEnvironment
http://wiki.centos.org/HowTos/RebuildSRPM

Thanks TrevorH1 Smiley

9. So, below a first attempt at taking apart a Source RPM package...

Code:
[root]# yum install rpm-build                                                   # from the instructions at http://wiki.centos.org/HowTos/SetupRpmBuildEnvironment
[root]# yum install redhat-rpm-config                                           # from the instructions at http://wiki.centos.org/HowTos/SetupRpmBuildEnvironment
[root]# su abc # become the non-root user
[abc]$ rpm -ivh /home/abc/openssl-1.0.0-27.el6.src.rpm                  # from the instructions at http://bradthemad.org/tech/notes/patching_rpms.php

Holy cow, that seems like a lot of work. Former east coast racing sailor, the guitar-playing Bradthemad says on his site
Quote
Part of the philosophy behind RPM is that the build process should be repeatable, as explained here. It may be small consolation when you just want make a simple change on your own system, where building once is good enough, but that's the way it is.

Ktks, got it Bradthemad. Time to suck it up and code on. And if you're still following this post you're as hardcore as yours truly to get bitcoind going on CentOS Smiley

10. Start fiddling with the source in openssl SRPM

Code:
These steps have been deleted

11. Now have to figure out which config (or source) files to edit in order to re-enable the elliptic curve code in openssl on CentOS 6.4...


Hero Member kano on bitcointalk: https://bitcointalk.org/index.php?topic=85228.0  says we have to change

enable-ec enable-ecdh enable-ecdsa  

Also, it seems that you have to replace the .bz2 file in the source with a .gz fresh file from www.openssl.org, and then update the openssl.spec file to reflect that.
Code:
[abc]$ cd /home/abc/rpmbuild/SOURCES
[abc]$ mv  openssl-1.0.0-usa.tar.bz2  openssl-1.0.0-usa.tar.bz2-original
[abc]$ curl -O http://www.openssl.org/source/openssl-1.0.0j.tar.gz
[abc]$ mv  openssl-1.0.0j.tar.gz  openssl-1.0.0-usa.tar.gz

In SOURCES/openssl.spec:

search for
 Source1: hobble-openssl
(line 29) and change it (comment it out) to:
 #Source1: hobble-openssl

search for
 %{SOURCE1} > /dev/null
(line 133) and change it (comment it out) to:
 #%{SOURCE1} > /dev/null

search fpr
  Source: openssl-%{version}-usa.tar.bz2
(line nnn) and change it to
Source: openssl-%{version}j.tar.gz

Thank you kano!

12. A few more changes to the openssl.spec file and the we rebuild the package with rpmbuild.

Code:
[abc]$ sed -i -e "s/no-ec/enable_ec/; s/no-ecdh/enable-ecdh/; s/no-ecdsa/enable-ecdsa/" /home/abc/rpmbuild/SPECS/openssl.spec  # Used sed editor to delete the no-ecdsa and no-ec directives from the specfile
[abc]$ cd /home/abc/rpmbuild
[abc]$ rpmbuild -ba SPECS/openssl.spec  # From instructions at http://wiki.centos.org/HowTos/RebuildSRPM

The above rpmbuild of the openssl package results in an error

Code:
+ umask 022
+ cd /home/abc/rpmbuild/BUILD
+ LANG=C
+ export LANG
+ unset DISPLAY
+ cd /home/abc/rpmbuild/BUILD
+ rm -rf openssl-1.0.0
+ /bin/tar -xf -
+ /usr/bin/gzip -dc /home/abc/rpmbuild/SOURCES/openssl-1.0.0j.tar.gz
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd openssl-1.0.0
/var/tmp/rpm-tmp.IOaGqm: line 38: cd: openssl-1.0.0: No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.IOaGqm (%prep)

It's close, but not exactly there.  From the above, when  /usr/bin/gzip -dc /home/abc/rpmbuild/SOURCES/openssl-1.0.0j.tar.gz  runs, the archive gets extracted to openssl-1.0.0j instead of openssl-1.0.0.  Therefore, when the cd openssl-1.0.0 happens, a "no such directory" is thrown, because everything is in openssl-1.0.0j.

So, seems close, but not 100% there.  Comments welcomed.
neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
June 24, 2013, 05:22:23 PM
Last edit: June 24, 2013, 05:47:46 PM by neonzeon
 #11

DAY 4...

I'm a little frustrated being so close AND so far away.

However, the Bitcoin Mayor of Las Vegas, Tuxavant, summed up my feelings nicely in this thread https://bitcointalk.org/index.php?topic=85228.msg939240
Quote
Even so, this will prove to be a very enlightening and educational experience for me as I've wanted to help maintain other packages too. This ECDSA thing has motivate me to finally learn how this stuff works.

I've already learned more about packages and source RPMs than what I thought I would have to Smiley

Amazingly, this is a classic problem according to Kano, who has been running Linux for 16 years.
Quote
Since there really is no one who has written this, that is easy to find on the net, for the last 7+ years this issue has existed ...

Note to self and others: "that is easy to find on the net"  I'm trying to remember the search terms I originally typed in to find this information. I think it was some text from the build error during the compile.  

If you remember what you typed to get here, please add a comment with the text "My Search Terms" somewhere prominent.

Below is Kano's instructions repeated. It's for Fedora Core 16 though. I'm going to analyze and annotate them with what I've read so far.

I suspect one of the key reasons he was successful was because he did everything as root.  
The CentOS instructions explicitly say that's living dangerously.
From the "Yum and RPM Tricks" at http://wiki.centos.org/TipsAndTricks/YumAndRPM
Quote
14. Get set up for rebuilding packages as yourself, not root

Sometimes you just have to rebuild that package - maybe only to use some configuration option which just isn't there in the official package. Or because you have found some great package which you really cannot find in the repositories, but the site only gives you RPMs for another distribution. So you have to grab the src.rpm and rebuild it for yourself.  But you really do not want to do it as root. So here's how to rebuild your packages in your home directory - with your own user account.

With that warning in mind, Kano's Instructions from https://bitcointalk.org/index.php?topic=85228.msg946289#msg946289

Code:
Build openssl-1.0.0j with EC on fc16
------------------------------------

All this is run from root.

1) Find the correct SRC RPM

Update fc16 to the latest everything and get
 openssl-1.0.0j-1.fc16.src.rpm
from any mirror

or later if there is a later version
(this was written 7-Jun-2012)

N.B. these details are specific for openssl-1.0.0j
but most developers should be able to adapt this to later versions

2) rpm -Uvh openssl-1.0.0j-1.fc16.src.rpm

3) cd ~/rpmbuild/SPECS

4) vim openssl.spec

look for "./Configure" at the start of a line
(in 1.0.0j-1.fc16 it is line 219)
about 3 lines down from that you will see:
 enable-cms enable-md2 no-idea no-mdc2 no-rc5 no-ec no-ecdh no-ecdsa \
change it to look like:
 enable-cms enable-md2 no-idea no-mdc2 no-rc5 enable-ec enable-ecdh enable-ecdsa \

search for
 Source1: hobble-openssl
(line 29) and change it (comment it out) to:
 #Source1: hobble-openssl

search for
 %{SOURCE1} > /dev/null
(line 133) and change it (comment it out) to:
 #%{SOURCE1} > /dev/null

go back to the top and increase "Release:"

line 24 increment the "Release:" number
e.g. change
 Release: 1%{?dist}
to
 Release: 2%{?dist}

5) install rpm-build
yum install rpm-build

6) cd ~/rpmbuild/SOURCES/

in ~/rpmbuild/SOURCES/ there is a file called "openssl-1.0.0j-usa.tar.xz"

rename it to "openssl-1.0.0j-usa.tar.xz.orig" (or whatever else you like)

get a full replacement for the tar file, at http://www.openssl.org/source/

 http://www.openssl.org/source/openssl-1.0.0j.tar.gz

and put it in the directory: ~/rpmbuild/SOURCES/

then rename it to "openssl-1.0.0j-usa.tar.xz"

7) cd ~/rpmbuild/SPECS

 rpmbuild -bb openssl.spec

8) You now have the RPM files you need in ~/rpmbuild/RPMS/*/


I'll start annotating in the next post.
neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
June 24, 2013, 07:19:05 PM
Last edit: June 24, 2013, 10:37:50 PM by neonzeon
 #12

Kano's Instructions Modified / Annotated for CentOS 6.4

Before Kano's stuff we do some preparation for a new user as well as prepare per http://wiki.centos.org/TipsAndTricks/YumAndRPM

We start as the root user, create a new user "abcd" and then continue work as this new user.  That way, everything is fresh.

[root] yum install rpm-build # Install the package builder
[root] useradd abcd # Add a new user named abcd
[root] su abcd # Become user abcd
[abcd] cd /home/abcd
[abcd] mkdir /home/abcd/rpmbuild/{BUILD,RPMS,SPECS,SOURCES,SRPMS} # Build the directory structure for rpmbuild
[abcd] # The next two lines create a special rpmbuild file that tells the packager where we want to build and who we are
[abcd] echo "%_topdir /home/abcd/rpmbuild" > /home/abcd/.rpmmacros
[abcd] echo "%packager Test User <testuser@example.com>" >> /home/abcd/.rpmmacros

1) Find the correct SRC RPM
Update fc16 to the latest everything and get  openssl-1.0.0j-1.fc16.src.rpm from any mirror

[abcd] cd /home/abcd ; curl -O http://vault.centos.org/6.4/os/Source/SPackages/openssl-1.0.0-27.el6.src.rpm

2) rpm -Uvh openssl-1.0.0j-1.fc16.src.rpm

[abcd] rpm --install openssl-1.0.0-27.el6.src.rpm

Note: I'm getting a warning which I'm going to ignore (for now) : group mockbuild does not exist - using root

3) cd ~/rpmbuild/SPECS

[abcd] cd ~/abcd/rpmbuild/SPECS

4) vim openssl.spec

look for "./Configure" at the start of a line
(in 1.0.0j-1.fc16 it is line 219)
about 3 lines down from that you will see:
 enable-cms enable-md2 no-idea no-mdc2 no-rc5 no-ec no-ecdh no-ecdsa \
change it to look like:
 enable-cms enable-md2 no-idea no-mdc2 no-rc5 enable-ec enable-ecdh enable-ecdsa \


I'm going to use the sed editor because you can find it on almost any linux box and you can do these changes as little one-line scripts.

[abcd] sed -i -e "s/no-ec/enable-ec/; s/no-ecdh/enable-ecdh/; s/no-ecdsa/enable-ecdsa/" /home/abcd/rpmbuild/SPECS/openssl.spec

search for
 Source1: hobble-openssl
(line 29) and change it (comment it out) to:
 #Source1: hobble-openssl

search for
 %{SOURCE1} > /dev/null
(line 133) and change it (comment it out) to:
 #%{SOURCE1} > /dev/null


[abcd] sed -i -e "s/^Source1: hobble-openssl/#&/; s/^%.SOURCE1. /#&/" ~/rpmbuild/SPECS/openssl.spec

go back to the top and increase "Release:"

line 24 increment the "Release:" number
e.g. change
 Release: 1%{?dist}
to
 Release: 2%{?dist}


For now, I'm going to keep the version unchanged unless it seems important to do so. (can always force the package update)

5) install rpm-build
yum install rpm-build


We completed this step right in the beginning when we were still the root user, so can ignore now

6) cd ~/rpmbuild/SOURCES/

in ~/rpmbuild/SOURCES/ there is a file called "openssl-1.0.0j-usa.tar.xz"

rename it to "openssl-1.0.0j-usa.tar.xz.orig" (or whatever else you like)


[abcd] cd ~/rpmbuild/SOURCES/ ; mv openssl-1.0.0-usa.tar.bz2 old-usa-bz2-file-not-needed

get a full replacement for the tar file, at http://www.openssl.org/source/

 http://www.openssl.org/source/openssl-1.0.0j.tar.gz

and put it in the directory: ~/rpmbuild/SOURCES/

then rename it to "openssl-1.0.0j-usa.tar.xz"


[abcd] cd ~/rpmbuild/SOURCES/ ; curl -O http://www.openssl.org/source/openssl-1.0.0j.tar.gz # Download sources from openssl.org
[abcd] gunzip -d openssl-1.0.0j.tar.gz ; bzip2 -z  openssl-1.0.0j.tar  # We have to convert to .bz2 because that is required in spec file openssl.spec
[abcd] mv openssl-1.0.0j.tar.bz2 openssl-1.0.0-usa.tar.bz2 # We have to rename it to this because that is what the spec file openssl.spec is looking for

7) cd ~/rpmbuild/SPECS

 rpmbuild -bb openssl.spec


I'm going with -ba instead of -bb because -ba seems to build everything. Hope that doesn't bite us later.

[abcd]  cd ~/rpmbuild/SPECS
[abcd] rpmbuild -ba openssl.spec

DOH - The instructions break here. The problem is as follows:

In the original openssl-1.0.0-usa.tar.bz2, all the source get extracted to rpmbuild/SOURCE/openssl-1.0.0

In the downloaded openssl-1.0.0j (which we rename), source go to rpmbuild/SOURCE/openssl-1.0.0j  <<< note the extra "j" at the end.

I somehow have to modify the "root" directory in the tar file to be openssl-1.0.0 instead of openssl-1.0.0j.


Here's the clue from the rpmbuild output. The last thing that works is the extraction of the archive.  
It then tries a cd openssl-1.0.0 which fails. However the directory rpmbuild/BUILD/openssl-1.0.0j (note the j at the end) exists and is populated with the source files.

Code:
+ /usr/bin/bzip2 -dc /home/abcd/rpmbuild/SOURCES/openssl-1.0.0-usa.tar.bz2 | /bin/tar -xf -
+ cd openssl-1.0.0
error:  line 38:  cd: openssl-1.0.0: No such file or directory

Below is a fairly ugly hack as a workaround attempt.
Basically, I take the wrongly installed openssl-1.0.0j, rename it to1.0.0 and then reconvert to bz2.

Code:
 cd /home/abcd/rpmbuild/BUILD
 mv openssl-1.0.0j openssl-1.0.0 # Get rid of the "j" at the end
 tar -cvf openssl-1.0.0j.tar openssl-1.0.0 # Archive to tar
 bzip2 -z openssl-1.0.0j.tar # Convert to bz2
 mv openssl-1.0.0j.tar.bz2 ~/rpmbild/SOURCES/ openssl-1.0.0-usa.tar.bz2 # Rename and move to SOURCES

With this hack, during rpmbuild, the archive extracts correctly to BUILD/openssl-1.0.0.

The build now fails because the patches in the SOURCES directory do not fully correlate to the source in he "fake" openssl-1.0.0-usa.bz2 we created.

This seems to be the flaw in kano's instructions, and we'll have to figure that out.

For those interested, here is some output from the build process so far http://pastebin.com/z5nx844J

Cool You now have the RPM files you need in ~/rpmbuild/RPMS/

I wish...

[abcd] ls ~/rpmbuild/RPMS/
[abcd] echo woo hoo !!!
neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
June 25, 2013, 12:18:52 AM
Last edit: June 25, 2013, 12:45:24 AM by neonzeon
 #13

Brief Build Instructions
Code:
userdel -rf abcd # Wipe user abcd
useradd abcd # Add a new user named abcd
su abcd # Become user abcd
mkdir ~/rpmbuild ; mkdir ~/rpmbuild/{BUILD,RPMS,SPECS,SOURCES,SRPMS} # Build the directory structure for rpmbuild
echo "%_topdir /home/abcd/rpmbuild" > ~/.rpmmacros
echo "%packager Test User <testuser@example.com>" >> ~/.rpmmacros
cd ~ ; curl -O http://vault.centos.org/6.4/os/Source/SPackages/openssl-1.0.0-27.el6.src.rpm
rpm --install openssl-1.0.0-27.el6.src.rpm
cd ~/rpmbuild/SPECS
sed -i -e "s/no-ec/enable-ec/; s/no-ecdh/enable-ecdh/; s/no-ecdsa/enable-ecdsa/" ~/rpmbuild/SPECS/openssl.spec
sed -i -e "s/^Source1: hobble-openssl/#&/; s/^%.SOURCE1. /#&/" ~/rpmbuild/SPECS/openssl.spec
rpmbuild -ba openssl.spec 1>e1 2>e2 &
neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
June 25, 2013, 01:41:06 AM
Last edit: June 25, 2013, 02:09:39 AM by neonzeon
 #14

Briefer Build Instructions

Code:
userdel -rf abcd ; useradd abcd ; su abcd # Wipe user abcd, create fresh user abcd, become user abcd
echo "%_topdir /home/$USER/rpmbuild" > ~/.rpmmacros ; echo "%packager Test User <testuser@example.com>" >> ~/.rpmmacros  # Tell rpm where to install files and fake id
cd ~ ; curl -O http://vault.centos.org/6.4/os/Source/SPackages/openssl-1.0.0-27.el6.src.rpm # Download the source package from CentOS
rpm --install openssl-1.0.0-27.el6.src.rpm 1>e1 2>e2 # Extract the source package to /rpmbuild/SOURCES
cd ~/rpmbuild/SPECS # This is where the package .spec file is extracted to
sed -i -e "s/no-ec/enable-ec/; s/no-ecdh/enable-ecdh/; s/no-ecdsa/enable-ecdsa/" ~/rpmbuild/SPECS/openssl.spec # Edit the .spec file
sed -i -e "s/^Source1: hobble-openssl/#&/; s/^%.SOURCE1. /#&/" ~/rpmbuild/SPECS/openssl.spec # Edit the .spec file
rpmbuild -bb openssl.spec 1>e1 2>e2 & # Rebuild the package using the edited .spec file
neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
June 25, 2013, 02:18:48 AM
 #15

In the "Briefer Build Instructions" above the last line is

Code:
rpmbuild -bb openssl.spec 1>e1 2>e2 &

The "&" at the end of the command means that the compile process is sent to the background.

That means you get your prompt back while the compile is still running.

You can pull the job to the foreground using the fg command.

The stdout and stderr messages are sent to files e1 and e2 respectively.

You can watch the stdout and stderr output during the compile with the following:

tail -f e1

tail -f e2

At the end of the compile, all status and error messages are saved in files e1 and e2.

You can browse them with

less -i e1

less -i e2

While browsing with less, you can use search, for instance type /error to search for "error" text.
neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
June 25, 2013, 02:28:12 AM
 #16

Here is the last bit of stderr at the end of the rpmbuild command. It doesn't look too bad. 

Any pointers and tips appreciated.

Code:

dsa_gen.c: In function 'FIPS_dsa_generate_pq':
dsa_gen.c:218: warning: unused variable 'mont'
dsa_gen.c:217: warning: unused variable 'g'

eng_openssl.c: In function 'bind_helper':
eng_openssl.c:144: warning: implicit declaration of function 'ECDH_OpenSSL'
eng_openssl.c:144: warning: passing argument 2 of 'ENGINE_set_ECDH' makes pointer from integer without a cast ../../include/openssl/engine.h:465: note: expected 'const struct CDH_METHOD *' but argument is of type 'int'
eng_openssl.c:147: warning: implicit declaration of function 'ECDSA_OpenSSL'
eng_openssl.c:147: warning: passing argument 2 of 'ENGINE_set_ECDSA' makes pointer from integer without a cast ../../include/openssl/engine.h:466: note: expected 'const struct CDSA_METHOD *' but argument is of type 'int'

evp_enc.c:102: warning: initialization from incompatible pointer type

p_lib.c:318: error: expected declaration specifiers or '...' before 'EC_KEY'
p_lib.c: In function 'EVP_PKEY_set1_EC_KEY':
p_lib.c:320: warning: implicit declaration of function 'EVP_PKEY_assign_EC_KEY'
p_lib.c:320: error: 'key' undeclared (first use in this function)
p_lib.c:320: error: (Each undeclared identifier is reported only once
p_lib.c:320: error: for each function it appears in.)
p_lib.c:322: warning: implicit declaration of function 'EC_KEY_up_ref'
p_lib.c: At top level:
p_lib.c:326: error: expected '=', ',', ';', 'asm' or '__attribute__' before '*' token

make[2]: *** [p_lib.o] Error 1
make[1]: *** [subdirs] Error 1
make: *** [build_crypto] Error 1

error: Bad exit status from /var/tmp/rpm-tmp.aIHqo8 (%build)

neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
June 26, 2013, 02:33:09 PM
Last edit: June 27, 2013, 04:22:34 AM by neonzeon
 #17

DAY 6

Focusing on building openssl.

Instantiate a fresh CentOS 6.4 server and then...

Code:
yum -y update  # Update everything
yum -y groupinstall 'Development tools'
yum -y install zlib-devel
yum -y install krb5-devel
 
cat /etc/centos-release
# CentOS release 6.4 (Final)
 
uname -rpo
# 2.6.32-279.el6.x86_64 x86_64 GNU/Linux
 
gcc --version
# gcc 4.4.7 20120313 (Red Hat 4.4.7-3)

We now have all the tools to rebuild openssl. We are going to build as user "abcd".

Code:
userdel -rf abcd ; useradd abcd ; su abcd # Wipe user abcd, create fresh user abcd, become user abcd
echo "%_topdir /home/$USER/rpmbuild" > ~/.rpmmacros ; # Tell rpm where to install files
echo "%packager Test User <testuser@example.com>" >> ~/.rpmmacros  # Tell rpm a test user id
echo "%_query_all_fmt %%{name}-%%{version}-%%{release}.%%{arch}" >> ~/.rpmmacros # Tell rpm the architecture
cd ~ ; curl -O http://vault.centos.org/6.4/os/Source/SPackages/openssl-1.0.0-27.el6.src.rpm # Get openssl source
rpm --install openssl-1.0.0-27.el6.src.rpm 1>e1 2>e2 # Extract the source package to /rpmbuild/SOURCES
cd ~/rpmbuild/SPECS # This is where the package .spec file is extracted to
sed -i -e "s/no-ec/enable-ec/; s/no-ecdh/ /; s/no-ecdsa/ /" ~/rpmbuild/SPECS/openssl.spec
sed -i -e "s/^Source1: hobble-openssl/#&/; s/^%.SOURCE1. /#&/" ~/rpmbuild/SPECS/openssl.spec # Edit the .spec file
rpmbuild -bb openssl.spec 1>e1 2>e2 &    # Rebuild the package using the edited .spec file

# The above rebuild of openssl-1.0.0j takes a long time - at least 20 minutes.
 
jobs  # Command to check if the compile is done
# [1]+  Running  rpmbuild -bb openssl.spec > e1 2> e2 &
 
tail -f e2 # or "tail -f e1" (ctl-c to exit)   command to check the build progress
 
# When the build finishes, the compiler output is in files e1 and e2 in ~/rpmbuild/SPECS
 
less -i e2  # Command to let you browse the build errors. Search for "error" with /error
 
# ====================================================================
# =========================== After the build ========================
# ====================================================================
 
Note: openssl-1.0.0-27.el6.src.rpm compiles 100% correct AS DOWNLOADED - evidence that the build environment is OK.
 
However, we make the following five changes (see the sed lines above) to enable elliptic curves:
 
s/no-ec/enable-ec/
s/no-ecdh/ /
s/no-ecdsa/ /
s/^Source1: hobble-openssl/#&/
s/^%.SOURCE1. /#&/
 
When the package is built with these five changes in, there are build errors.
 
THIS IS WHERE I NEED YOUR HELP.  
 
Build errors here, near the end of the file: http://pastebin.com/8aGxEd6n

I find them extremely hard to resolve due to my inexperience.

p_lib.c:318: error: expected declaration specifiers or '...' before 'EC_KEY'

Looking at the preprocessor output with gcc -E, I saw that EC_KEY is undefined.
 
I found this handy: http://fossies.org/dox/openssl-1.0.1e/index.html
 
EC_KEY is defined in /crypto/include/ec.h
 
If I manually edit p_lib.c to include ec.h, this error vanishes.

So, for some reason, ec.h is not included in p_lib.c. I wonder why and how to properly fix it.
 
Comments and suggestions welcome.

neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
July 01, 2013, 01:37:17 AM
Last edit: July 01, 2013, 01:55:27 AM by neonzeon
 #18

DAY 7

I've been asking for help in several places regarding compiling openssl with EC.

A frequently asked question was "Why are you not using mock to build the package?".

So, after some searching, mock is a package builder from Fedora. It is installed from the "EPEL" repository.

To make a long story short, when you build with mock, you get the same errors.

Below is a summary of the steps needed to try building openssl (with EC) using mock on a fresh CentOS 6.4 box.

Build instructions: http://pastebin.centos.org/3016/

neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
July 01, 2013, 01:41:08 AM
 #19


A kind soul in #centos on freenode irc with the handle pj was very helpful.

First, he confirmed the build errors in p_lib.c.

Then he created a patch to fix that error, and immediately ran into another error in p_lib.c.

So, it seems as if it's not just a simple fix in openssl.spec.

Some patches will have to be created. Off to learning how to patch...
neonzeon (OP)
Newbie
*
Offline Offline

Activity: 23
Merit: 0


View Profile
July 01, 2013, 08:37:00 AM
Last edit: July 04, 2013, 07:10:29 PM by neonzeon
 #20

Success! OpenSSL with EC Builds on CentOS 6.4

Finally, openssl with EC builds on Centos! This is basically due to the help from <pj> in #centos on freenode IRC.

The method was actually VERY close to Kano's instructions (see previous posts)

We start with the CentOS openssl source RPM, and "install" it (which means splitting it into Sources (consisting of patches and a .tar.bz2) and Specs (containing openssl.spec)

Then we download a different, but version-synchronized  .tar.gz version of the source from openssl.org. The new source will replace the existing .tar.bz2 file.

A patch file (to patch a test error) is also needed from openssl.org. The patch fixes this error http://openssl.6102.n7.nabble.com/OpenSSL-1-0-0b-testssl-fails-td11009.html

With the .tar.gz and the patch from openssl.org dowloaded, the openssl.spec file is edited to

1) Point to the newly downloaded .tar.gz as source
2) Change no-EC to enable-EC (enable ellptic curves)
3) Disable the "hobble" script (which erases elliptic curve source files)
4) Change the release number
5) Add a patch entry for the newly downloaded patch file

After that, we create a new source rpm using the modified spec file and replacement source (.tar.gz).

Finally, use mock, a Fedora tool used by Centos package builders, to build a new openssl package from the newly created source rpm.

Below are all the steps, condensed, that you need to take on a fresh CentOS 6.4 box, to compile openssl with enable-ec.

Code:
yum -y update  # Update all packages on new machine
yum -y groupinstall 'Development tools'
yum -y localinstall --nogpgcheck http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm # Install EPEL (EL6 extra packages) repository
yum -y install fedora-packager  # Install mock from EPEL repository
userdel -rf abcd ; useradd -G mock abcd ; su abcd
cd ~ ; curl -O http://vault.centos.org/6.4/os/Source/SPackages/openssl-1.0.0-27.el6.src.rpm
/usr/bin/mock ~/openssl-1.0.0-27.el6.src.rpm
rm -rf /home/abcd/build ; mv /var/lib/mock/epel-6-x86_64/root/builddir/build/ /home/abcd ; # Move to a safe place
cd /home/abcd/build/SOURCES
curl -O http://www.openssl.org/source/openssl-1.0.0.tar.gz # Download corresponding source tarball from openssl
curl -o patch300.patch http://cvs.openssl.org/patchset?cn=19998 # Download this patch to fix a test error
cd ../SPECS
sed -i -e "s/no-ec/enable-ec/; s/no-ecdh/enable-ecdh/; s/no-ecdsa/enable-ecdsa/" openssl.spec # Enable EC
sed -i -e "s/^Source1: hobble-openssl/#&/; s/^%.SOURCE1. /#&/" openssl.spec # Disable the "hobble" script
sed -i -e "s/^Release.*dist\}/&.EC.1/" openssl.spec # Also change release number by adding .EC.1
sed -i -e "s/-usa.tar.bz2/.tar.gz/" openssl.spec # Change the source tarball
sed -i -e "s/^Patch78.*/&\nPatch300: patch300.patch\n/" openssl.spec # Add the new patch
sed -i -e "s/^%patch78.*/&\n%patch300 -p1 \n/" openssl.spec # Add the new patch again
/usr/bin/mock --buildsrpm --spec  ~/build/SPECS/openssl.spec --sources  ~/build/SOURCES # Do a source rebuild
cp /var/lib/mock/epel-6-x86_64/root/builddir/build/SRPMS/openssl-1.0.0-27.el6.EC.1.src.rpm /home/abcd
cd ~ ; /usr/bin/mock --rebuild openssl-1.0.0-27.el6.EC.1.src.rpm


Tip Jar: 1KaJZTmvvk2CPYmRPWALU63o2AZkMEMvJk

Pages: [1] 2 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!