Nightly Builds

[Matt Corallo]

I just finished setting up a nightly build server.  It currently only builds i686 for linux, but I'm hoping to add Windows soon.  It also includes an up to date copy of the blockchain with each nightly.  
It can be found at bitcoin.bluematt.me.  Thanks goes to validus for the bandwidth.  

Edit: URL is now bitcoin.bluematt.me/bitcoin-nightly.  It is now DNS load balanced.  Thanks to validus and vrs for bandwidth.

[1715340544]

1715340544

[1715340544]

1715340544

[1715340544]

1715340544

[1715340544]

1715340544

[1715340544]

1715340544

[Matt Corallo]

At the request of tcatm and jgarzik, I think it would be nice to have a version that includes various patches.  It might be updated nightly or just on new releases.  I'm thinking patches like getblock and other similar patches which aren't of use to enough people to make it into mainline, but are sensible useful patches.  
Please list any patches you want to see included.

[tcatm]

* xlisttransactions http://bitcointalk.org/index.php?topic=611.0
* getblock https://www.bitcoin.org/smf/index.php?topic=724.0

[jgarzik]

my patches
---------------------------------------------
xlisttransactions RPC: http://yyz.us/bitcoin/patch.bitcoin-listtransactions
getblockbycount RPC: http://yyz.us/bitcoin/patch.bitcoin-getblock
dumpblock RPC: http://yyz.us/bitcoin/patch.bitcoin-dumpblock
settxfee RPC: http://yyz.us/bitcoin/patch.bitcoin-settxfee
print POW failures: http://yyz.us/bitcoin/patch.bitcoin-pow-fail


other peoples' hackery
-----------------------------------------------------
- gavin's monitor* stuff
- gavin's portoption branch
- UPNP



things to avoid
----------------------------------------------------
Breaking changes, like Luke JR's RPC versioning changes.


administrivia
---------------------------------------------------
IMO require people to submit (or write yourself) a summary of each patch, so that people know exactly what's in there.

[Matt Corallo]

things to avoid
----------------------------------------------------
Breaking changes, like Luke JR's RPC versioning changes.

I'll try, but for something like this I can't make any guarantees.

administrivia
---------------------------------------------------
IMO require people to submit (or write yourself) a summary of each patch, so that people know exactly what's in there.

Good idea, care to submit a summary of yours?

[jgarzik]

Associated patch summaries...

xlisttransactions RPC: http://yyz.us/bitcoin/patch.bitcoin-listtransactions

Adds "xlisttransactions" RPC call.  This lists all wallet transactions, using logic similar to the GUI's display code.

getblockbycount RPC: http://yyz.us/bitcoin/patch.bitcoin-getblock

Adds "getblockbycount" RPC call.  This dumps the block on the main chain at the specified height, in its entirety.  All data fields, including all transactions, are dumped in a single JSON response.

dumpblock RPC: http://yyz.us/bitcoin/patch.bitcoin-dumpblock

A clone of "getblockbycount" RPC call.  Same behavior as "getblockbycount" -- dumps block at specified height.  Adds a "_fulldump" boolean to the output, indicating whether or not the entire block is present on disk (preparing for lightweight block chain support), hoping to address some objections of satoshi.

settxfee RPC: http://yyz.us/bitcoin/patch.bitcoin-settxfee

Add "settxfee" RPC call.  This permits changing of the 'paytxfee' variable at runtime.

print POW failures: http://yyz.us/bitcoin/patch.bitcoin-pow-fail

Print a response for every work solution submitted to bitcoind.  Since we already logged successful proof-of-work solutions, the only thing remaining was to log when a proof-of-work check failed, which this patch does.

[genjix]

RPC methods return strings instead of floats:
https://github.com/genjix/bitcoin/tree/strrpc

Immediate initialisation of RPC + isinitialized in getinfo:
https://github.com/genjix/bitcoin/tree/isinitialized

Name resolution for addresses (genjix@foo.org -> 1kjdskjds3232dsjds4343):
https://github.com/genjix/bitcoin/tree/bitnom

Maybe you could setup a way to generate custom builds for distribution.

[jgarzik]

I was thinking this would be mostly about non-breaking changes, such as those patches people are already applying manually and running in production.

RPC methods return strings instead of floats:
https://github.com/genjix/bitcoin/tree/strrpc

disagree - breaking change

Immediate initialisation of RPC + isinitialized in getinfo:
https://github.com/genjix/bitcoin/tree/isinitialized

disagree - breaking change (people must rewrite their code to expect the RPC server to appear early)

Name resolution for addresses (genjix@foo.org -> 1kjdskjds3232dsjds4343):
https://github.com/genjix/bitcoin/tree/bitnom

That's more like it   I think this sort of change -- presuming it doesn't break any existing APIs, which I don't think it does -- would be perfect for these "useful patches" nightly builds.

Maybe you could setup a Bitcoin cloud build service? I login... add my branch and click generate builds.

Sounds like a great bitcoin business or website for somebody...

[Matt Corallo]

I agree with jgarzik, I want this version to be stable and compatible, but add new features which might be considered "beta" or might just not have any use in mainline.  

The current version can be see here.

Maybe you could setup a way to generate custom builds for distribution.

Great idea.  It wouldn't be hard to do with simple bash scripts.  The problem is currently I compile the new version on my server and upload it to a VPS provided by validus because I don't have the bandwidth to provide good service.  Also, the compilation might take a while so it would need powerful servers if it was to serve more than one simultaneous connection.  If anyone knows of a place where I can get a free, powerful server with plenty of bandwidth I'd set it up (or would be willing to pay for one, donation address is in my sig  ).  

[genjix]

I'm not going to re-ignite an argument over those 2 first patches, but if this is for stable nightly builds, then please don't include my third branch yet- it's not complete.

There's 2 issues I need to conceptually solve. (privacy concern + a security scheme)

[Matt Corallo]

I'm not going to re-ignite an argument over those 2 first patches, but if this is for stable nightly builds, then please don't include my third branch yet- it's not complete.

There's 2 issues I need to conceptually solve. (privacy concern + a security scheme)

My goal is to have two branches.  One nightly, stable build from the current git version of bitcoin.  One which is either nightly or in sync with bitcoin releases (haven't made up my mind yet).  The second one will be patched with all kinds of "beta" patches but which don't break existing functionality. 

[Matt Corallo]

First release of the patched branch has been uploaded.  I will try to get the bitnom included later today, but it currently conflicts with sendmany so it needs a bit of work to get it to merge properly.  Included patches are listed in the README.md file.  I also need to add Gavin's monitor stuff. 

[devrandom]

Would be cool if the builds used the gitian process:

http://bitcointalk.org/index.php?topic=2926.20

This would allow people to verify that the builds are not compromised.

[Matt Corallo]

Would be cool if the builds used the gitian process:

http://bitcointalk.org/index.php?topic=2926.20

This would allow people to verify that the builds are not compromised.

Sadly that is not possible as I cannot run VMs within the existing build VMs.  However, I am looking into building stable releases (of -patched, and potentially of mainline) with gitian. 

[devrandom]

Would be cool if the builds used the gitian process:

http://bitcointalk.org/index.php?topic=2926.20

This would allow people to verify that the builds are not compromised.

Sadly that is not possible as I cannot run VMs within the existing build VMs.  However, I am looking into building stable releases (of -patched, and potentially of mainline) with gitian. 

Ah, yes, running VM within VM is a problem.  I hear that's possible in some configurations, but haven't looked into it.

I'm available if you run into any issues with building and to compare hashes.

[Matt Corallo]

Would be cool if the builds used the gitian process:

http://bitcointalk.org/index.php?topic=2926.20

This would allow people to verify that the builds are not compromised.

Sadly that is not possible as I cannot run VMs within the existing build VMs.  However, I am looking into building stable releases (of -patched, and potentially of mainline) with gitian. 

Ah, yes, running VM within VM is a problem.  I hear that's possible in some configurations, but haven't looked into it.

I'm available if you run into any issues with building and to compare hashes.

After building the VMs, attempting to run the build script results in apt errors fetching from my host name.  ie "Failed to fetch from http://HostHostName:...".  My host name will not resolve via DNS which is apparently the problem here. 

[devrandom]

After building the VMs, attempting to run the build script results in apt errors fetching from my host name.  ie "Failed to fetch from http://HostHostName:...".  My host name will not resolve via DNS which is apparently the problem here. 

Looking into this.

[devrandom]

After building the VMs, attempting to run the build script results in apt errors fetching from my host name.  ie "Failed to fetch from http://HostHostName:...".  My host name will not resolve via DNS which is apparently the problem here. 

Try doing:

Code:

export MIRROR_HOST=10.0.2.2
bin/make-base-vm

I think I'll make that the default.

[Matt Corallo]

After building the VMs, attempting to run the build script results in apt errors fetching from my host name.  ie "Failed to fetch from http://HostHostName:...".  My host name will not resolve via DNS which is apparently the problem here. 

Try doing:

Code:

export MIRROR_HOST=10.0.2.2
bin/make-base-vm

I think I'll make that the default.

No change at all.

[devrandom]

After building the VMs, attempting to run the build script results in apt errors fetching from my host name.  ie "Failed to fetch from http://HostHostName:...".  My host name will not resolve via DNS which is apparently the problem here. 

Try doing:

Code:

export MIRROR_HOST=10.0.2.2
bin/make-base-vm

I think I'll make that the default.

No change at all.

Strange.  Could you try:

* While make-base-vm is running, run

Code:

ps auxww | grep vmbuilder

and make sure that the mirror arguments refer to 10.0.2.2
* Make sure that you didn't specify -i or --skip-image
* I just realized that the --quiet flag to gbuild actually does --skip-image due to a bug .  If you had this flag on, it didn't pick up the new base vm image.

I've just pushed a new version that fixed the latter and makes 10.0.2.2 the default.

I'm on irc if you want to chat instead of messaging here.

[devrandom]

New version pushed just now.

The default is now to use 127.0.0.1 while bootstrapping (on the host) and 10.0.2.2 in the VM.  This will always hit the host.

Let me know if it works for you.

[jgarzik]

It would be nice if directories such as http://bitcoin.bluematt.me/bitcoin-nightly/ included a README that described the contents of the directory / purpose of the project / etc.

[Luke-Jr]

things to avoid
----------------------------------------------------
Breaking changes, like Luke JR's RPC versioning changes.

More trolling, I see.

For anyone interested, my RPCv1 changes do NOT break anything, and are completely backward compatible.

[Garrett Burgwardt]

things to avoid
----------------------------------------------------
Breaking changes, like Luke JR's RPC versioning changes.

More trolling, I see.

For anyone interested, my RPCv1 changes do NOT break anything, and are completely backward compatible.

Weren't those the ones that didn't actually do anything?

[Luke-Jr]

things to avoid
----------------------------------------------------
Breaking changes, like Luke JR's RPC versioning changes.

More trolling, I see.

For anyone interested, my RPCv1 changes do NOT break anything, and are completely backward compatible.

Weren't those the ones that didn't actually do anything?

No.


More importantly, both the oldest and newest Win32 nightlies on the site crash when stopped.

[Matt Corallo]

More importantly, both the oldest and newest Win32 nightlies on the site crash when stopped.

I can't reproduce, could you provide more information (ie Windows version, etc)

[Luke-Jr]

More importantly, both the oldest and newest Win32 nightlies on the site crash when stopped.

I can't reproduce, could you provide more information (ie Windows version, etc)

Windows 7, vanilla fresh install.
1. Run start /B bitcoind -rpcpassword=1
(wait for it to finish startup)
2. Run bitcoind -rpcpassword=1 stop
3. Windows says it crashed (and db error visible)

[Matt Corallo]

Windows 7, vanilla fresh install.
1. Run start /B bitcoind -rpcpassword=1
(wait for it to finish startup)
2. Run bitcoind -rpcpassword=1 stop
3. Windows says it crashed (and db error visible)

Ok, so I wasn't able to fix that on Windows Server 2003.  I'd assume it would work on XP but I gave up on windows.  Now the win32 nightlies are compiled on Ubuntu via https://github.com/TheBlueMatt/bitcoin/tree/crosscompile.  They appear to work 100% now.  If you find any additional issues, please notify me.

[khal]

On linux, bitcoind, 64bits :
./bitcoind
=> bitcoind doesn't display "bitcoin server starting" and doesn't return to the next line of the terminal.

But, from another terminal, i can send rpc commands and all seams to work.
I can launch "./bitcoind stop" and bitcoind stops in the first terminal.

The problem doesn't appear with the -daemon option.
This problem didn't exist in the 0.3.20.2 release.

[Matt Corallo]

On linux, bitcoind, 64bits :
./bitcoind
=> bitcoind doesn't display "bitcoin server starting" and doesn't return to the next line of the terminal.

But, from another terminal, i can send rpc commands and all seams to work.
I can launch "./bitcoind stop" and bitcoind stops in the first terminal.

The problem doesn't appear with the -daemon option.
This problem didn't exist in the 0.3.20.2 release.

That isn't a problem with the nightlies.  That was a conscious change made and will be in the 0.3.21 release (I believe its in the 0.3.21 RC now). 

[LZ]

Thanks for your builds.

[Venkatesh Srinivas]

Would you be interested in nightly builds for DragonFly BSD and OpenBSD?

[ctoon6]

if it is possible, could you do a 64 bit windows build? just to see how much faster it is.

[Matt Corallo]

Would you be interested in nightly builds for DragonFly BSD and OpenBSD?

Currently I do all the building on one Ubuntu VM, so adding two more VMs is quite a bit more effort.  If you want to do them though, setting up a cronjob to build is quite easy.

if it is possible, could you do a 64 bit windows build? just to see how much faster it is.

The Linux 32/64 difference is unnoticeable, the only reason a Linux 64-bit binary is provided is the 32-bit one will fail to load due to 32-bit BDB on 64-bit Ubuntu attempting to load 64-bit plugins.  Because they are both compiled with gcc, the difference for Windows should be virtually identical to that on Linux.  That and setting up a 64-bit build for windows would be substantially more effort.

Sorry to those who have noticed the builds have not been happening over the past couple days, Im hoping to get them back running today or tomorrow (Im in the process of switching build servers).

[Gavin Andresen]

Would you be interested in nightly builds for DragonFly BSD and OpenBSD?

Yes, would you be interested in building?

[Venkatesh Srinivas]

Yes; was asking to see if they could be hosted at the same place or would have to be elsewhere.