Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: Alex Zee on September 19, 2011, 07:50:47 AM



Title: Proposal: make it so anybody could easily compile the client
Post by: Alex Zee on September 19, 2011, 07:50:47 AM

I've tried once to compile it. Gave up.

I don't understand why can't you bundle all the needed libraries and distribute the complete source code with the client?

1) You will get more trust. What's the point of doing code review if you can't compile a binary version from it?

2) You will get more developers involved. I, personally, can't spend several days, figuring out just how to compile it. But if it were a matter of loading it into VS and hitting Build, I might work on small features or optimizations.

And the most annoying is that I don't see any rational reasons not to do it! Why hasn't it been done from the beginning? Are you trying to impose some sort of an artificial "entry barrier"?


P.S. I am specifically talking about Windows.


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: wumpus on September 19, 2011, 08:11:34 AM
bitcoin-qt has very simple windows build instructions, only a few steps, and the dependencies are available in a downloadable (signed) package:

https://github.com/laanwj/bitcoin-qt


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: mtbitcoin on September 19, 2011, 08:12:38 AM

I've tried once to compile it. Gave up.

I don't understand why can't you bundle all the needed libraries and distribute the complete source code with the client?

1) You will get more trust. What's the point of doing code review if you can't compile a binary version from it?

2) You will get more developers involved. I, personally, can't spend several days, figuring out just how to compile it. But if it were a matter of loading it into VS and hitting Build, I might work on small features or optimizations.

And the most annoying is that I don't see any rational reasons not to do it! Why hasn't it been done from the beginning? Are you trying to impose some sort of an artificial "entry barrier"?


P.S. I am specifically talking about Windows.

+1 on this


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: Alex Zee on September 19, 2011, 08:15:07 AM
bitcoin-qt has very simple windows build instructions, only a few steps, and the dependencies are available in a downloadable (signed) package:

https://github.com/laanwj/bitcoin-qt

Is there a reason why they are "available in a downloadable (signed) package" and not included into the main source distribution?

BTW, I would like to test the new qt client, can you give a direct link to download?


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: P4man on September 19, 2011, 08:16:57 AM
Its easy to compile on linux. Its easy to install the binary on windows (and possible to compile if you know what you are doing, though I havent tried). Really, thats all thats needed IMO. If you are wary of using closed source software, you shouldnt be running windows.


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: wumpus on September 19, 2011, 08:17:38 AM
Is there a reason why they are "available in a downloadable (signed) package" and not included into the main source distribution?
Because they are binaries, they do not belong in a source package.
Quote
BTW, I would like to test the new qt client, can you give a direct link to download?
tar.gz: https://github.com/laanwj/bitcoin-qt/tarball/master
zip: https://github.com/laanwj/bitcoin-qt/zipball/master


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: Alex Zee on September 19, 2011, 08:23:37 AM
Because they are binaries, they do not belong in a source package.

Is there a rational reason why "binaries do not belong in a source package"?

So the whole idea of LIB files is a flawed concept?

Anyway, make them source code too, don't make excuses.

Learn from other projects, where you have a single solution and all the projects to build all the libraries and compiling is a matter of one click. Preferably the solution should be for multiple VS or an old one, like 2005 - it's easy to convert projects up and hard to convert down.



Title: Re: Proposal: make it so anybody could easily compile the client
Post by: payb.tc on September 19, 2011, 08:25:36 AM
Its easy to compile on linux. Its easy to install the binary on windows (and possible to compile if you know what you are doing, though I havent tried). Really, thats all thats needed IMO. If you are wary of using closed source software, you shouldnt be running windows.

what if you want to make a small modification before compiling?

in that case trusting closed source windows software has nothing to do with it.


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: Alex Zee on September 19, 2011, 08:27:13 AM
This is what I need to do just to test your client???

Quote
Windows build instructions:

* Download the QT Windows SDK and install it. You don't need the Symbian stuff, just the desktop Qt.

* Download and extract the dependencies archive [1], or compile openssl, boost and dbcxx yourself.

* Copy the contents of the folder "deps" to "X:\QtSDK\mingw", replace X:\ with the location where you installed the Qt SDK. Make sure that the contents of "deps\include" end up in the current "include" directory.

* Open the .pro file in QT creator and build as normal (ctrl-B)

[1] PGP signature: https://download.visucore.com/bitcoin/qtgui_deps_1.zip.sig (signed with RSA key ID 610945D0


Oh, hell no. Thank you very much. I will test it when there is a pre-built binary.


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: P4man on September 19, 2011, 08:40:52 AM
what if you want to make a small modification before compiling?

in that case trusting closed source windows software has nothing to do with it.

Trust is the only reason I can see why Joe Six pack ('s smarter brother) would want to compile. But if you are a developer wanting to change the code, Im sure you can figure out how to compile it?


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: Alex Zee on September 19, 2011, 08:46:08 AM
Let me explain, why I am so pissed off about it.

It's because I take it as a personal offense when people try to invade my time.

And that's exactly what every crappy open-source project tries to do - it wants  me to jump through hoops just for the pleasure of its makers.

Why? Because crappy open-source projects are usually written by a bunch of schoolboys with too much time on their hands. They adore Linux because they can afford time to play this little game of "type letters into command prompt and see what happens". They adore spending countless hours trying to figure stuff out.

Now I am not saying that you are a bunch of schoolboys or that this is a crappy open-source project, but then stop acting like one!

It takes just one developer, (who already have all the necessary libraries on his machine with versions that match his code), to bundle them and make it a one-click compile to save many man-hours for all the people who will later use it! Failing to do so is an insult to all of them.

Just because thousands of monkeys on the Internet do it this way doesn't mean you have to too. Do it the smart way, not the monkey way.

[Just venting. Feeling better already :) ]


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: wumpus on September 19, 2011, 08:48:39 AM
Anyway, make them source code too, don't make excuses.
You can also easily do that yourself if you have any skills working with IDEs. It's funny how people's requirements can become unreasonably high so quickly when they're not paying for something.

Oh, hell no. Thank you very much. I will test it when there is a pre-built binary.
A pre-built binary? wtf, a few posts ago you were talking about building it yourself because you want to make some changes, and so suddenly you want a prebuilt binary.

You can download one from www.bitcoin.org. In a few months it will have the new interface.

Please remind me to never try to be helpful to you again.

Quote
It's because I take it as a personal offense when people try to invade my time.
I'm now seeing it as an offense you're trying to invade my time. Welcome to my ignore list.


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: Alex Zee on September 19, 2011, 08:54:38 AM
I'm now seeing it as an offense you're trying to invade my time. Welcome to my ignore list.

Developers are few and people who will be using their project are many. That's enough reason to try and make it as easy for them as possible, to save their time and not yours.

But whatever.


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: Alex Zee on September 19, 2011, 09:02:38 AM
It's funny how people's requirements can become unreasonably high so quickly when they're not paying for something.

It's funny how such a simple thing as bundling all the libraries, since you already have them, and creating projects for them becomes an "unreasonably high requirement".

Of course, it's better to not include them for no reason whatsoever and then force everyone who will try to help you develop the project to spend time jumping from site to site, searching for the exact versions, downloading and placing into proper folders and doing the whole setup again.

"Come on in, developers, help me develop it! But you need to dance for an hour and upload the video of your dance on YouTube, before I let you work on my project!"

That's a cool attitude. It goes great with "hey, I am not being paid, so I am free to do a crappy job".


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: wumpus on September 19, 2011, 09:18:19 AM
It's funny how such a simple thing as bundling all the libraries, since you already have them, and creating projects for them becomes an "unreasonably high requirement".
Then why not do it and contribute the result, to save "all those many other people" the work.

Oh wait, this is just about you.



Title: Re: Proposal: make it so anybody could easily compile the client
Post by: Alex Zee on September 19, 2011, 09:26:42 AM
Then why not do it and contribute the result, to save "all those many other people" the work.

Oh wait, this is just about you.

Because you already have all the libraries installed! You already have it set up!

Why don't you just publish the whole project, not just a part that requires lots of dependencies?

If you dare me, hell, sure, I can do it. I would be glad to help all the future possible developers who might want to get involved.

It just would be much easier for you, since you already have it on your machine.

Besides, the "full" distribution should be supported by those, who work on it "full time".

It would be cumbersome for me to follow your release schedule and update a stand-alone "full" package for others.



Title: Re: Proposal: make it so anybody could easily compile the client
Post by: Alex Zee on September 19, 2011, 09:31:11 AM
Let's make it practical.

If you don't have time to create VS projects - let me do it.

Developers. Do just this step one: include all the source into the main branch. QT includes, what have you. All. No downloads required.

I will do the projects.


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: P4man on September 19, 2011, 09:31:38 AM
You realize those libraries might be published under different licenses, some of which would prevent him from redistributing them? I dont know if its the case here, but you usually cant just go and put "everything thats needed" in a tarball for your convenience.

BTW, I just compiled bitcoin-qt. Copy pasted 3 commands (yes, I use linux) and its working and looking great. Thank you John Smith.


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: wumpus on September 19, 2011, 09:35:04 AM
Because you already have all the libraries installed! You already have it set up!
No I don't. I don't even have Visual Studio and rarely use Windows. And when I do I use the Qt SDK with Qt Creator, for which I also published the four simple steps for building in the README file...

Quote
Why don't you just publish the whole project, not just a part that requires lots of dependencies?
Because on non-windows platforms it is not normal to bundle all the libraries a project makes use of. They are installed as separate packages.

Even on Windows, many people already use the libraries such as boost and openssl so have them installed in their own locations.

Releasing a tangle of source files that include copied boost, upnp, openssl headers and source files would certainly be useful for some people, but could also create more problems than it's worth.

Quote
If you dare me, hell, sure, I can do it. I would be glad to help all the future possible developers who might want to get involved.
Yes, maybe you should do that. That's a much better attitude :)


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: Alex Zee on September 19, 2011, 09:51:08 AM
You raise valid points.

But if we want to attract more developers to Bitcoin, and not only "core full-time" developers, but people who will do minor optimizations and bug-fixes in their little spare time, we need to have a one-click setup. It's crucial.

Yes, a developer might use some other libraries, but they can be of different versions. And most importantly - our hypothetical developer probably wouldn't care if they will be duplicated in Bitcoin distribution.

He installed the Bitcoin client, he noticed a bug or had an idea for a "20-minute work" feature. All that he needs is to go to the "source", click on solution, make some tweaks in code, fix a bug, whatever, and compile himself a new binary.

If he is required to spend 40 minutes on top of those 20 minutes just to make it compile, he might just give up.

So, Ok, if there are nobody else to do a full Windows build, I will do it.

But now I am confused, which version to work with. Is your bitcoin-qt going to be the new official master branch as is? Or will there be some kind of merging process?



Title: Re: Proposal: make it so anybody could easily compile the client
Post by: wumpus on September 19, 2011, 10:02:28 AM
Developers. Do just this step one: include all the source into the main branch. QT includes, what have you. All. No downloads required.
This is not possible license-wise. You can externally release a tarball with all source code and dependencies, if you make it clear that it includes project+dependencies.

However, including the source and headers of entire frameworks such as boost and Qt into the git branch would be extremely unwise and also wasteful of space. LGPL and GPL allow linking against, but not copying the files into your project, except by making the entire project (L)GPL.

If you don't believe that, name one project that does this...

Quote
But now I am confused, which version to work with. Is your bitcoin-qt going to be the new official master branch as is? Or will there be some kind of merging process?
It's in the process of being merged.


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: HostFat on September 19, 2011, 10:09:39 AM
Is it possible to make a bat that uses wget to download all sources? ;D


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: Alex Zee on September 19, 2011, 10:11:04 AM
Ok, what I will do is create a separate package with a single addition - a "windows" folder where I will put everything required to build it on Windows.

It's in the process of being merged.

That's kinda vague :)

When do you guys plan to make the official 0.5 branch?


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: Remember remember the 5th of November on September 19, 2011, 10:41:21 AM
I am glad to see a person who thinks like me.

+1 AlexZ!


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: Furyan on September 20, 2011, 12:06:40 PM
Is it possible to make a bat that uses wget to download all sources? ;D

This is what my pushpool/bitcoin installer does. I can set up a new (Linux) pool server in minutes without having any of the dependencies on the machine.  No, I won't share it, it's a mess and is quite brittle :)

License questions aside, it can be difficult to include all the required dependencies for a project, particularly anything built in C++ on Windows.  Windows/MSVC runtime has a *&(@$ load of library dependencies it doesn't advertise that can bite you hard when deploying an application.  The redistributables do a good job of addressing 99.8% of the problems so usually just distributing the msvc redistributables will do the job, but then there are the edge cases...

Qt's full build is over 30 GB.  The core libraries (release) are only a couple hundred kb, but debug ones are about 10gb total.  Any developer will need the debug ones if they want to do their own compiling without worrying about dependencies.  I know of no place currently that will host binary lib drops that big, so cuts/choices would have to be made - a Qt GUI project could include only the QtGui.dll and QtGuid.dll files, which amounts to about 10 MB. But you wouldn't be able to link/include functionality from any other Qt libraries.

The situation is much simpler with the .NET Framework on Windows. It includes everything, and because you can run the framework installer as part of a setup package, you don't even need to include the framework, just include the redistributable installer.  And because with .NET, a dependency list is as simple as a list of DLL's you drop in your exe dir, including non-Framework libraries is trivial.

Remember, all this is licensing aside.  Redistribution is a hot topic in software licensing and you can get burned pretty badly by it if you're not careful.


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: Alex Zee on September 20, 2011, 07:17:06 PM
Well, I finally managed to create a stand-alone 1-click Windows build.

You can download it here (25 Mb):

http://qbizy.com/bitcoin-one-click-win-build-0.4.0rc2.7z

It's based on the latest official version (0.4.0 RC 2).

You will need Visual Studio 2005 to build it (if you'll manage to compile it in later versions - please post here).

--

The source code is not modified in any way, except one line in the "util.h" - I've added "typedef int pid_t;" because it didn't recognize this type.

The code that uses it should probably be #ifdef'ed with "__WXMSW__". Developers, please fix it!

Another thing - the "bitcoin.ico" converted to uncompressed format so VS's resource compiler would understand it.

Also, developers, please put the updated icon into the main distribution:

http://qbizy.com/bitcoin-vs2005-compatible-ico.zip

--

No UPNP yet. Is it important? No daemon too. Latest OpenSSL 1.0.0e is used instead of 1.0.0d.

EXE file size is 6 instead of 11 Mb.

Still tries to load "WMIApSrv" driver, same as the official client. Need to find how to turn this off...

To save space only the essential dependencies were included. For example, no debug libraries. All the added Windows stuff is in the single "windows" folder.

Comments and ideas appreciated.


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: speeder on September 20, 2011, 07:24:20 PM
+1 to Alex Zee point.


I never fiddled with the sources myself, because my time is too precious to waste trying to compile stuff.


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: casascius on September 21, 2011, 04:50:07 AM
I too am frustrated by difficulty getting this to compile (on Linux).

I too want to contribute a little here and there.  Even if the README was a bit of a HOWTO that said go get this, go get that, I'd do it.

I am hoping that someone will be able to walk me through getting past whatever is throwing me for a loop.  It looks related to OpenSSL.

http://pastebin.com/wYDFLzds


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: nhodges on September 21, 2011, 05:16:39 AM
Well, I finally managed to create a stand-alone 1-click Windows build.

You can download it here (25 Mb):

http://qbizy.com/bitcoin-one-click-win-build-0.4.0rc2.7z

It's based on the latest official version (0.4.0 RC 2).

You will need Visual Studio 2005 to build it (if you'll manage to compile it in later versions - please post here).

--

The source code is not modified in any way, except one line in the "util.h" - I've added "typedef int pid_t;" because it didn't recognize this type.

The code that uses it should probably be #ifdef'ed with "__WXMSW__". Developers, please fix it!

Another thing - the "bitcoin.ico" converted to uncompressed format so VS's resource compiler would understand it.

Also, developers, please put the updated icon into the main distribution:

http://qbizy.com/bitcoin-vs2005-compatible-ico.zip

--

No UPNP yet. Is it important? No daemon too. Latest OpenSSL 1.0.0e is used instead of 1.0.0d.

EXE file size is 6 instead of 11 Mb.

Still tries to load "WMIApSrv" driver, same as the official client. Need to find how to turn this off...

To save space only the essential dependencies were included. For example, no debug libraries. All the added Windows stuff is in the single "windows" folder.

Comments and ideas appreciated.


You can submit a pull request to the core project on github, I believe.


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: Furyan on September 21, 2011, 10:18:59 PM
I too am frustrated by difficulty getting this to compile (on Linux).

I too want to contribute a little here and there.  Even if the README was a bit of a HOWTO that said go get this, go get that, I'd do it.

I am hoping that someone will be able to walk me through getting past whatever is throwing me for a loop.  It looks related to OpenSSL.

http://pastebin.com/wYDFLzds

makefile.unix?  What OS are you compiling on?


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: casascius on September 22, 2011, 09:57:54 PM
I finally got it to work, my issue was that I had two different versions of openssl, one with includes at /usr/local/include/openssl and the other with /usr/include/openssl, and they were managing to include one another's include files due to the way the include search path apparently works.  Would yield screenfuls of compile errors all related to ECDSA.  So removing one of them solved the problem.

SO yeah, that's not the fault of the Bitcoin project.  But having a list of the dependencies and the typical instructions to install them (either HTTP URL, git clone path, apt-get package name, etc.) in the README wouldn't hurt.


Title: Re: Proposal: make it so anybody could easily compile the client
Post by: ShadowOfHarbringer on September 28, 2011, 05:18:24 AM

I've tried once to compile it. Gave up.

I don't understand why can't you bundle all the needed libraries and distribute the complete source code with the client?

1) You will get more trust. What's the point of doing code review if you can't compile a binary version from it?

2) You will get more developers involved. I, personally, can't spend several days, figuring out just how to compile it. But if it were a matter of loading it into VS and hitting Build, I might work on small features or optimizations.

And the most annoying is that I don't see any rational reasons not to do it! Why hasn't it been done from the beginning? Are you trying to impose some sort of an artificial "entry barrier"?


P.S. I am specifically talking about Windows.

It is more difficult to compile anything on windows by default.
If you want to EASILY compile things, move to an Open Source UNIX-like system (such as Linux), which is designed for ease of compilation by default.

However, having a proper ./configure script on UNIX/Linux wouldn't hurt. For now it is not possible to do the following on the Bitcoin client sources:

Code:
./configure
make
make install

Which is a little annoying. Other then that, compilation on UNIX is not problematic.