Nightly Builds

[Matt Corallo]

I just finished setting up a nightly build server.  It currently only builds i686 for linux, but I'm hoping to add Windows soon.  It also includes an up to date copy of the blockchain with each nightly.  
It can be found at bitcoin.bluematt.me.  Thanks goes to validus for the bandwidth.  

Edit: URL is now bitcoin.bluematt.me/bitcoin-nightly.  It is now DNS load balanced.  Thanks to validus and vrs for bandwidth.

[Matt Corallo]

At the request of tcatm and jgarzik, I think it would be nice to have a version that includes various patches.  It might be updated nightly or just on new releases.  I'm thinking patches like getblock and other similar patches which aren't of use to enough people to make it into mainline, but are sensible useful patches.  
Please list any patches you want to see included.

[tcatm]

* xlisttransactions http://bitcointalk.org/index.php?topic=611.0
* getblock https://www.bitcoin.org/smf/index.php?topic=724.0

[jgarzik]

my patches
---------------------------------------------
xlisttransactions RPC: http://yyz.us/bitcoin/patch.bitcoin-listtransactions
getblockbycount RPC: http://yyz.us/bitcoin/patch.bitcoin-getblock
dumpblock RPC: http://yyz.us/bitcoin/patch.bitcoin-dumpblock
settxfee RPC: http://yyz.us/bitcoin/patch.bitcoin-settxfee
print POW failures: http://yyz.us/bitcoin/patch.bitcoin-pow-fail


other peoples' hackery
-----------------------------------------------------
- gavin's monitor* stuff
- gavin's portoption branch
- UPNP



things to avoid
----------------------------------------------------
Breaking changes, like Luke JR's RPC versioning changes.


administrivia
---------------------------------------------------
IMO require people to submit (or write yourself) a summary of each patch, so that people know exactly what's in there.

[Matt Corallo]

things to avoid
----------------------------------------------------
Breaking changes, like Luke JR's RPC versioning changes.

I'll try, but for something like this I can't make any guarantees.

administrivia
---------------------------------------------------
IMO require people to submit (or write yourself) a summary of each patch, so that people know exactly what's in there.

Good idea, care to submit a summary of yours?

[jgarzik]

Associated patch summaries...

xlisttransactions RPC: http://yyz.us/bitcoin/patch.bitcoin-listtransactions

Adds "xlisttransactions" RPC call.  This lists all wallet transactions, using logic similar to the GUI's display code.

getblockbycount RPC: http://yyz.us/bitcoin/patch.bitcoin-getblock

Adds "getblockbycount" RPC call.  This dumps the block on the main chain at the specified height, in its entirety.  All data fields, including all transactions, are dumped in a single JSON response.

dumpblock RPC: http://yyz.us/bitcoin/patch.bitcoin-dumpblock

A clone of "getblockbycount" RPC call.  Same behavior as "getblockbycount" -- dumps block at specified height.  Adds a "_fulldump" boolean to the output, indicating whether or not the entire block is present on disk (preparing for lightweight block chain support), hoping to address some objections of satoshi.

settxfee RPC: http://yyz.us/bitcoin/patch.bitcoin-settxfee

Add "settxfee" RPC call.  This permits changing of the 'paytxfee' variable at runtime.

print POW failures: http://yyz.us/bitcoin/patch.bitcoin-pow-fail

Print a response for every work solution submitted to bitcoind.  Since we already logged successful proof-of-work solutions, the only thing remaining was to log when a proof-of-work check failed, which this patch does.

[genjix]

RPC methods return strings instead of floats:
https://github.com/genjix/bitcoin/tree/strrpc

Immediate initialisation of RPC + isinitialized in getinfo:
https://github.com/genjix/bitcoin/tree/isinitialized

Name resolution for addresses (genjix@foo.org -> 1kjdskjds3232dsjds4343):
https://github.com/genjix/bitcoin/tree/bitnom

Maybe you could setup a way to generate custom builds for distribution.

[jgarzik]

I was thinking this would be mostly about non-breaking changes, such as those patches people are already applying manually and running in production.

RPC methods return strings instead of floats:
https://github.com/genjix/bitcoin/tree/strrpc

disagree - breaking change

Immediate initialisation of RPC + isinitialized in getinfo:
https://github.com/genjix/bitcoin/tree/isinitialized

disagree - breaking change (people must rewrite their code to expect the RPC server to appear early)

Name resolution for addresses (genjix@foo.org -> 1kjdskjds3232dsjds4343):
https://github.com/genjix/bitcoin/tree/bitnom

That's more like it   I think this sort of change -- presuming it doesn't break any existing APIs, which I don't think it does -- would be perfect for these "useful patches" nightly builds.

Maybe you could setup a Bitcoin cloud build service? I login... add my branch and click generate builds.

Sounds like a great bitcoin business or website for somebody...

[Matt Corallo]

I agree with jgarzik, I want this version to be stable and compatible, but add new features which might be considered "beta" or might just not have any use in mainline.  

The current version can be see here.

Maybe you could setup a way to generate custom builds for distribution.

Great idea.  It wouldn't be hard to do with simple bash scripts.  The problem is currently I compile the new version on my server and upload it to a VPS provided by validus because I don't have the bandwidth to provide good service.  Also, the compilation might take a while so it would need powerful servers if it was to serve more than one simultaneous connection.  If anyone knows of a place where I can get a free, powerful server with plenty of bandwidth I'd set it up (or would be willing to pay for one, donation address is in my sig  ).  

[genjix]

I'm not going to re-ignite an argument over those 2 first patches, but if this is for stable nightly builds, then please don't include my third branch yet- it's not complete.

There's 2 issues I need to conceptually solve. (privacy concern + a security scheme)

[Matt Corallo]

I'm not going to re-ignite an argument over those 2 first patches, but if this is for stable nightly builds, then please don't include my third branch yet- it's not complete.

There's 2 issues I need to conceptually solve. (privacy concern + a security scheme)

My goal is to have two branches.  One nightly, stable build from the current git version of bitcoin.  One which is either nightly or in sync with bitcoin releases (haven't made up my mind yet).  The second one will be patched with all kinds of "beta" patches but which don't break existing functionality. 

[Matt Corallo]

First release of the patched branch has been uploaded.  I will try to get the bitnom included later today, but it currently conflicts with sendmany so it needs a bit of work to get it to merge properly.  Included patches are listed in the README.md file.  I also need to add Gavin's monitor stuff. 

[devrandom]

Would be cool if the builds used the gitian process:

http://bitcointalk.org/index.php?topic=2926.20

This would allow people to verify that the builds are not compromised.

[Matt Corallo]

Would be cool if the builds used the gitian process:

http://bitcointalk.org/index.php?topic=2926.20

This would allow people to verify that the builds are not compromised.

Sadly that is not possible as I cannot run VMs within the existing build VMs.  However, I am looking into building stable releases (of -patched, and potentially of mainline) with gitian. 

[devrandom]

Would be cool if the builds used the gitian process:

http://bitcointalk.org/index.php?topic=2926.20

This would allow people to verify that the builds are not compromised.

Sadly that is not possible as I cannot run VMs within the existing build VMs.  However, I am looking into building stable releases (of -patched, and potentially of mainline) with gitian. 

Ah, yes, running VM within VM is a problem.  I hear that's possible in some configurations, but haven't looked into it.

I'm available if you run into any issues with building and to compare hashes.

[Matt Corallo]

Would be cool if the builds used the gitian process:

http://bitcointalk.org/index.php?topic=2926.20

This would allow people to verify that the builds are not compromised.

Sadly that is not possible as I cannot run VMs within the existing build VMs.  However, I am looking into building stable releases (of -patched, and potentially of mainline) with gitian. 

Ah, yes, running VM within VM is a problem.  I hear that's possible in some configurations, but haven't looked into it.

I'm available if you run into any issues with building and to compare hashes.

After building the VMs, attempting to run the build script results in apt errors fetching from my host name.  ie "Failed to fetch from http://HostHostName:...".  My host name will not resolve via DNS which is apparently the problem here. 

[devrandom]

After building the VMs, attempting to run the build script results in apt errors fetching from my host name.  ie "Failed to fetch from http://HostHostName:...".  My host name will not resolve via DNS which is apparently the problem here. 

Looking into this.

[devrandom]

After building the VMs, attempting to run the build script results in apt errors fetching from my host name.  ie "Failed to fetch from http://HostHostName:...".  My host name will not resolve via DNS which is apparently the problem here. 

Try doing:

Code:

export MIRROR_HOST=10.0.2.2
bin/make-base-vm

I think I'll make that the default.

[Matt Corallo]

After building the VMs, attempting to run the build script results in apt errors fetching from my host name.  ie "Failed to fetch from http://HostHostName:...".  My host name will not resolve via DNS which is apparently the problem here. 

Try doing:

Code:

export MIRROR_HOST=10.0.2.2
bin/make-base-vm

I think I'll make that the default.

No change at all.

[devrandom]

After building the VMs, attempting to run the build script results in apt errors fetching from my host name.  ie "Failed to fetch from http://HostHostName:...".  My host name will not resolve via DNS which is apparently the problem here. 

Try doing:

Code:

export MIRROR_HOST=10.0.2.2
bin/make-base-vm

I think I'll make that the default.

No change at all.

Strange.  Could you try:

* While make-base-vm is running, run

Code:

ps auxww | grep vmbuilder

and make sure that the mirror arguments refer to 10.0.2.2
* Make sure that you didn't specify -i or --skip-image
* I just realized that the --quiet flag to gbuild actually does --skip-image due to a bug .  If you had this flag on, it didn't pick up the new base vm image.

I've just pushed a new version that fixed the latter and makes 10.0.2.2 the default.

I'm on irc if you want to chat instead of messaging here.