Development process straw-man

[Gavin Andresen]

With Satoshi's blessing, and with great reluctance, I'm going to start doing more active project management for bitcoin.

Everybody please be patient with me; I've had a lot of project management experience at startups, but this is the first open source project of any size I've been involved with.  Anyway, I've created an integration/staging tree at:
  https://github.com/bitcoin/bitcoin

... and am proposing this for a development process:

Straw-man plan for Bitcoin development (open source vets, please slap me around and help make this better):

Developers work in their own trees, then submit pull requests when they think their feature is ready.

Requests get discussed (where? here in the bitcoin forums?) and if there's broad consensus they're a good thing, well written, match coding style, etc. then they're merged into the 'master' branch.

master branch is regularly built and tested (by who? need people willing to be quality assurance testers), and periodically pushed to the subversion repo to become the official, stable, released bitcoin.

We'll create feature branches if/when there are major new features being worked on by several people.

Discussion, feedback, etc, especially from people with experience leading or working on other open source projects, is very welcome.

[davout]

That sounds really good!
Simple question : why not drop svn altogether ? git does a much better job, it's easier to follow, easier to fork

Also I think a really good thing would be a test suite, it would make it much easier to check that some pulled patch doesn't break anything, also it could be really useful for bug-fixing, having submission of a failing test and submission of a patch being two required steps for a fix inclusion in the source.

[BioMike]

And a changelog for every release in the tarball would be nice.

[jgarzik]

And a changelog for every release in the tarball would be nice.

At which depth of information?  The full changelog is always available via SVN (and the various source tree browsers on the web), and it seems like a waste to simply "svn log > ChangeLog" for each release.

[jgarzik]

This might sound strange at first glance, but I would encourage the "Linux kernel" development process, something I call "throwing spaghetti at a wall, so see what sticks."

Following the open source philosophy of "release early, release often" one sends patches (or git pull requests) as soon as a change is available and passes a developer's simple "it works" tests.  If you know the change is likely to be controversial, or you cannot guess whether or not it will be accepted, change is marked with 'RFC' in the subject line.

Project maintainer (satoshi or gavin) and community comment on the change.  Often, maintainer can be lazy and wait to observe community reaction to a change, before commenting.  Public feedback is critically important.  Developers will post dumb or off-the-cuff patches.  Project maintainers and the community are expected to educate new developers via respectful feedback, posted in an open forum.

If feedback is given, developer is expected to revise their change, and repost the patch/pull request.  Often this step is repeated many, many times until the change is "right."  Sometimes the developer is a jackhole who persists in reposting the same annoying change over and over.  We ignore these people.  Sometimes maintainer gives "time is just not right for that change; come back in 6 months" -- this is disappointing, but inevitable.

If no feedback is given, and the change is not controversial, maintainer accepts patch / pulls request into git repo.

Public feedback, especially from project maintainers, is important.  Existing developers must teach new developers why a change should look the way it does.

[Cdecker]

As far as I see it the Bitcoin client has 3 interfaces:

• The User interface
• The API interface
• The Network interface

Apart from the first there are others relying on the fact that the interfaces do not change, so I think that changes to them should be slowed to a regular update intervall and should be well documented, as it would be incredibly hard to reverse engineer them at every change.

This is also supported by this ticket: https://github.com/bitcoin/bitcoin/issues#issue/5

[Mike Hearn]

Perhaps controversial but for what it's worth, Subversion is a lot simpler and easier to use than git. I don't know if it's still true but git also had problems working on Windows in the past.

You might want to look into reviewboard, if patches are becoming an issue. I'd definitely recommend NOT having a workflow in which people code random stuff and then it's pulled into the main repo, but rather people upload patches that are then reviewed by either you or Satoshi via some code review tool.

[kiba]

Perhaps controversial but for what it's worth, Subversion is a lot simpler and easier to use than git. I don't know if it's still true but git also had problems working on Windows in the past.

Simpler?

[davout]

Perhaps controversial but for what it's worth, Subversion is a lot simpler and easier to use than git. I don't know if it's still true but git also had problems working on Windows in the past.

Simpler?

+1, git being harder is a misconception, it's just different
it handles lots of things much more intuitively than svn does.

but that's kindof off topic

[jgarzik]

You might want to look into reviewboard, if patches are becoming an issue. I'd definitely recommend NOT having a workflow in which people code random stuff and then it's pulled into the main repo, but rather people upload patches that are then reviewed by either you or Satoshi via some code review tool.

Any workflow that skips code review is bonkers.  I don't think anybody has ever suggested that.

And git has plenty of tools for code review.  If you need help, just ask.

[jgarzik]

OK, here's a test of the new devel process

See http://bitcointalk.org/index.php?topic=2371.0

[wumpus]

I just did my first pull request ever It was much simpler than I thought.

Git isn't hard to use, but it seems to be still the case that documentation on git is very fragmented, and generally hard to understand if you just want to do something simple. It assumes every user wants to start juggling around branches, HEADs, and dreams about hash tags. Luckily, there is Google and a zillion of 'tips and tricks' pages.

[pj]

Developers work in their own trees, then submit pull requests when they think their feature is ready.

Requests get discussed (where?

Yes - where?

I found that an email list was nearly ideal for such work when in the Linux kernel (the famous lkml list.)

This allows threaded discussions, keeping a visible history, and a wide variety of client readers.

Web forums have the history and the threading, but are limited in what they can do compared to
some of the fancier email clients in the hands of an experienced user.

IRC is not threaded, and tends to be more off the cuff conversational.

On a separate topic, I would suggest that a mercurial (hg) interface to the tree, running alongside
the git interface, would be cool.   Aha -- I just noticed http://hg-git.github.com/.  My wish is granted.

[davout]

I just did my first pull request ever It was much simpler than I thought.

Git isn't hard to use, but it seems to be still the case that documentation on git is very fragmented, and generally hard to understand if you just want to do something simple. It assumes every user wants to start juggling around branches, HEADs, and dreams about hash tags. Luckily, there is Google and a zillion of 'tips and tricks' pages.

http://progit.org/book/ was my best friend for a week

[genjix]

http://www.youtube.com/watch?v=OFkgSjRnay4

learnt git in an hour from that video.

[Gavin Andresen]

Requests get discussed (where?

Yes - where?

I found that an email list was nearly ideal for such work when in the Linux kernel (the famous lkml list.)

Let's try using the github Pull Request system for discussing pull requests:
  https://github.com/bitcoin/bitcoin/pulls

That ties the discussion to the proposed code changes, is a natural way to create separate 'threads', and will be a convenient place to point to in release notes.

And lets talk about potential code changes (pre pull request) here in this forum.

I'd rather not introduce Yet Another Place (we've already got these forums, IRC chat, and github) to talk about bitcoin development.

[jgarzik]

Let's try using the github Pull Request system for discussing pull requests:
  https://github.com/bitcoin/bitcoin/pulls

I'd rather not introduce Yet Another Place (we've already got these forums, IRC chat, and github) to talk about bitcoin development.

These two statements contradict each other, because github is Yet Another Place, too.

I do not see any point in moving the discussion of some bitcoin changes to a special place outside the forum.  Discuss changes on the forum just like we've always done, regardless of whether that change is a proposal, a patch, or a pull request.

The forum is perfectly adequate for discussions.

[tfoote]

This seems like a good proposal.  One of the most important aspects will be documentation both of the product and the process. 

I'm interested in the project and have developed a patch and until I found this email I had no clue how to submit it.  Can I suggest that a development overview page be created wiki https://en.bitcoin.it/wiki/Main_Page to house this sort of instructions.  Maybe something like https://en.bitcoin.it/wiki/Development with links from the wiki Main_Page and the main website. 

Another thing to point out is where tickets should be opened.  I tried for quite a while to find ticket tracking on the sourceforge site.  There should probably also be a link from the sourceforge site to github. 

[Gavin Andresen]

I'm interested in the project and have developed a patch and until I found this email I had no clue how to submit it.  Can I suggest that a development overview page be created wiki https://en.bitcoin.it/wiki/Main_Page to house this sort of instructions.  Maybe something like https://en.bitcoin.it/wiki/Development with links from the wiki Main_Page and the main website.

Done.  Please feel free to make it better.

Another thing to point out is where tickets should be opened.  I tried for quite a while to find ticket tracking on the sourceforge site.  There should probably also be a link from the sourceforge site to github. 

Good idea.

[Gavin Andresen]

I do not see any point in moving the discussion of some bitcoin changes to a special place outside the forum.  Discuss changes on the forum just like we've always done, regardless of whether that change is a proposal, a patch, or a pull request.

The forum is perfectly adequate for discussions.

OK.  Lets talk here.