The realm of the build czar

## Build Czar Duties

The main duties of the Build Czar are summarized as follows

- Continuously monitor the builds using the Scoreboard as one of the primary source of information.
- Notify developers who broke compilation to fix the errors as soon as possible, ideally by the next day. A red color in the "Compile" column is not at all acceptable - the Build Czar needs to ensure that these problems are identified and fixed in a timely manner. If possible, the Build Czar should let developers know what the source of problems might be. It is quite possible that developers who checked in the code or users who provided the patch may not have resources to investigate the issues, so the Builds Czar's help is essential to keep things moving ahead.
- Keep an eye on the tests that are run in every build. Anything abnormal needs to be notified to the right developer. The Build Czar should try helping the developer by providing stack traces (in case of crashes) or other details like printouts with debugging level turned on.
- Some tests fail in the daily builds for many reasons like known bugs, transient timeouts etc. Make sure that no new test failures show up. This guy knows most of the information. Ask him to help you out with known problems.
- Keep an eye on the footprint and performance stats. Any abnormal changes should be brought to the attention of the developer responsible for it or to the devo group.
- Keep the builds ticking. Any red on the "Last Finished" column in the Scoreboard should be fixed. The link to the "Build Name" indicates the machine where the build is being run.
- The builds don't cover all the possible configurations. If you get a bug report about a compile error in a particular configuration, try setting up a build to make sure that it doesn't show up again if it has been fixed.
- Keep an eye on the bugzilla entries that are registered by users and developers. Decide on the bugs that need to be fixed for the beta and pain developers for an ETA.

  The document here talks about the powers of a build Czar.

  The Build Czar is empowered to set up more builds on his own for his convenience. This page has a step by step instructions on how to do that.

  The build czar can get the build configuration by looking at the config portion of the scoreboard.

  Pro-active involvement by the build czar is necessary. Being a pro-active build czar requires monitoring the git repository carefully and responding quickly to suspected changes to keep the repo stays stable.

  After an email from the Coverity scan project check the Coverity project and check that there are no new errors reported. If so, contact the person who introduced the error.

  Check the daily test statistics email to determine any drop in test failures. Do note that when there are a lot of compile errors the test results are really unusable.

  ## Recipe for Cutting a Beta/Minor Kit

  The build czar is also in charge for the release of the beta. Cutting a beta is as simple as cutting butter if things go well. Here is the procedure followed while cutting a beta:

  1. The whole process takes somewhere between 3-4 hours.
  2. We suggest you take advantage of GNU Screen so that even if your SSH session is interrupted, the cutting process can continue. This command must be installed on the machines we use to cut the release.
     - type `screen` to start screen.
     - execute commands as normal. Note that Ctrl-A is special in screen, so you need to type Ctrl-A-A to send a Ctrl-A to the shell
     - should your session be interrupted, reconnect and type `screen -x`
     - when finished, just type exit twice
  3. After login check that you can,
     `ssh bczar@download.dre.vanderbilt.edu`
     to ensure that this succeeds. If not fix the problem, if ok exit again back to
  4. Prior to starting this, gather aggregate release notes from all developers. This is usually in the form of an email plea asking to update all NEWS files in the archive. These NEWS files are used as part of the release notes for the release.
  5. Make sure your release system has all the needed tools. This can be achieved on Fedora using:
     - `yum install perl svn screen pysvn automake doxygen bzip2 tar gzip openssh graphviz zip libtool GitPython`
     - `yum update`
     or on OpenSuSE
     - `zypper install perl screen python-pysvn automake doxygen bzip2 tar gzip openssh graphviz zip libtool python-gitpython`
     - `zypper update`
     If you want to perform a full build with qt support, than run:
     - `yum install ntp rubygem-rmagick bison libxerces-c-devel psmisc yum-utils gdb unzip glibc-devel libasan bison redhat-lsb perl-Pod-Usage rubygems clang make patch libcgroup-devel ant setuptool system-config-network-tui system-config-firewall-tui lcov gnuplot java-1.7.0-openjdk git-svn perl svn screen pysvn automake doxygen bzip2 tar gzip openssh graphviz zip libtool gcc-c++ boost-devel valgrind openssl-devel gcc qt4 fltk-devel bzip2-devel rsync openssl lzo-devel zziplib-devel acpid acpi nfs-utils java xerces-c xerces-c-devel mc qt qt-devel icecream ruby ruby-devel lksctp-tools-devel git telnet GitPython NetworkManager wget mailx`
     For some optional i686 packages run
     - `yum install openssl-devel.i686 xerces-c-devel.i686 glibc-devel.i686 glibc.i686 libstdc++.i686 ncurses-libs.i686 zlib.i686`
  6. Setup your ssh access to github according to the github ssh documentation
  7. Checkout a new workspace on a Fedora system with the last public release and with all patches installed.
  - The best place to create the workspace is under `/export/anduriltmp/bczar` (if you are on anduril). Don't use the home directory itself, it is an NFS share and not really fast.
  - Checkout like this:
    - `git clone https://github.com/DOCGroup/ATCD.git ATCD`
    - `git clone https://github.com/DOCGroup/ATCD.git MPC`
  8. Set $DOC_ROOT to point to the new workspace you checked out.
  9. Set an environment variable SIGNATURE indicating your full name. This is used to fill the ChangeLog entry.
  - For example,`export SIGNATURE="Johnny Willemsen"`
  10. Set an environment variable MAILID indicating your mail id. This is used to fill the mail id portion of the ChangeLog entry.
  - For example,`export MAILID="jwillemsen@remedy.nl"`
  11. Change directories to `$DOC_ROOT`
  12. Tag the release by executing
      `ACE/bin/make_release.py --beta --update --tag --push`
      This will only take a couple minutes to complete and once done successfully, you can carry on with BOTH creating the kits and generating the doxygen documentation in parallel. NOTE that `--beta` should be replaced with `--minor` or `--major` as appropriate.
  After the repository has been tagged check each file listed below to make sure version numbers are updated as expected.
  In the EXTREMELY unlikely event that something goes wrong during the tagging of the repo, the following files must be returned to the state they were in before the release process started and then checked back into git:
  `
  - ACE/ChangeLog
  - ACE/PROBLEM-REPORT-FORM
  - ACE/VERSION
  - ACE/TAO/ChangeLog
  - ACE/TAO/PROBLEM-REPORT-FORM
  - TAO/VERSION
  - CIAO/ChangeLog
  - CIAO/PROBLEM-REPORT-FORM
  - CIAO/VERSION
  - CIAO/ciao/Version.h
  - TAO/tao/Version.h
  ace/Version.h`
In most cases, a
`svn revert -R *`
from DOC_ROOT will suffice.
The tag will also need to be removed (both in Middleware and MPC): ACE+TAO+CIAO-X_Y_Z (where X is the ACE Major version number, and Y & Z are the Minor and Beta release numbers of the release that is to be restarted).

E.g.:
`cd ATCD && git tag -d ACE+TAO+CIAO-X_Y_Z
cd MPC && git tag -d ACE+TAO+CIAO-X_Y_Z
`
Note that this only needs to be done if the tagging fails. If kit creation fails, simply restart that process.

- Create the kits by executing
  `ACE/bin/make_release.py --kit`
  This will take somewhere around 2-4 hours to complete.
  - These commands only tags and creates the kits for the software itself, not documentation, this can be started in parallel with this activity.
  - The kits end up in `$DOC_ROOT/packages`

  To summarize, the following is a transcript of the steps up to this point executing successfully:

  `$ ssh ..
  No default printer
  screen
  cd $HOME
  rm -rf DOC_ROOT
  mkdir DOC_ROOT
  cd DOC_ROOT
  export DOC_ROOT=$PWD
  export SIGNATURE="Johnny Willemsen"
  export MAILID=jwillemsen@remedy.nl
  git clone https://github.com/DOCGroup/ATCD.git ATCD
  git clone https://github.com/DOCGroup/MPC.git MPC
  cd ATCD
  ACE/bin/make_release.py --beta --update --verbose
  ACE/bin/make_release.py --beta --tag --verbose
  ACE/bin/make_release.py --beta --push --verbose
  ACE/bin/make_release.py --kit
  `

  Feel free to cut and paste with suitable edits.

- The packages end up by default under $DOC_ROOT/package-<PID>, you can copy them to the webserver using the following commands. (Note that <PID> needs to be the numerical pid of the process that created the kit, use
  `ls -ald` to determine the correct filename.) At the moment you execute these commands all users can download these packages.
`scp $DOC_ROOT/package-<PID>/ACE* bczar@download.dre.vanderbilt.edu:/export/www/download.dre/ACE+TAO-distribution
`
- After the repository is tagged you can also start generating the doxygen documentation in parallel with the kit generation above.
  - Login to a release system you prepared with the same packages as above:
    `screen`
  - After login check that you can,
    `ssh bczar@download.dre.vanderbilt.edu`
    to ensure that this succeeds. If not fix the problem, if ok exit again back to your release system.
  - `cd /tmp`
    and remove the contents of the doxygen directory and recreate it again with
  `rm -rf doxygen`
  `mkdir doxygen`
  `cd doxygen`
  If you create the doxygen documentation on `naboo.dre.vanderbilt.edu` than make sure you use `/web/users/isisbuilds/tmp/ACE_wrappers` as working directory
  - Update the workspace with the right version tag (replace the X_Y_Z with the ACE version number being released e.g. 5_6_7)
    `svn co git clone https://github.com/DOCGroup/ATCD.git --depth 1 --branch ACE+TAO+CIAO-6_3_2 ATCD
    `
  - Change to the ATCD directory using
    cd ATCD
  - Set the needed environment variables using
    `export ACE_ROOT=$PWD/ACE`
    `export TAO_ROOT=$PWD/TAO`
    `export CIAO_ROOT=$PWD/CIAO`
    `export DANCE_ROOT=$PWD/DAnCE`
    `export DDS_ROOT=`
  - Check the doxygen version at the shell by executing the command:
    `doxygen --version` This should be at least 1.6.2
  Now you are ready to create documentation
- `cd $ACE_ROOT`
  `$ACE_ROOT/bin/generate_rel_manpages`
  When this is ready copy the resulting files using
  `scp ACE-html.tar.gz ACE-html.tar.bz2 ACE-html.zip ACE-html.tar.gz.md5 ACE-html.tar.bz2.md5 ACE-html.zip.md5 bczar@download.dre.vanderbilt.edu:/export/www/download.dre/ACE+TAO-distribution`
`screen
cd $HOME
rm -rf doxygen
mkdir doxygen
cd doxygen
git clone https://github.com/DOCGroup/ATCD.git --depth 1 --branch ACE+TAO+CIAO-6_3_2 ATCD
cd ATCD
export ACE_ROOT=$PWD/ACE
export TAO_ROOT=$PWD/TAO
export CIAO_ROOT=$PWD/CIAO
export DANCE_ROOT=$PWD/DAnCE
export DDS_ROOT=
cd ACE
$ACE_ROOT/bin/generate_rel_manpages
scp ACE-html.tar.gz ACE-html.tar.bz2 ACE-html.zip ACE-html.tar.gz.md5 ACE-html.tar.bz2.md5 ACE-html.zip.md5 bczar@download.dre.vanderbilt.edu:/export/www/download.dre/ACE+TAO-distribution`
- While doxygen churns, format a release announcement, including the release notes gathered from developers.
  - Get from bugzilla the bugs fixed. Use the following query as start query and update the start date.
  - Let Doug Schmidt review these before you do anything with them.
- Make sure the new version is available in Bugzilla.

- We now have a bczar Bugzilla user (bczar@dre.vanderbilt.edu) with full privileges which points to the bczar user at ISIS. To gain access to this account as a new build czar, you could update the ~/.forward file on one of the ISIS hosts (for example bczar@naboo.dre.vanderbilt.edu) with your own email address (but be aware that if you leave this ~/.forward file in effect, you will get innundated with cron mail messages from all of the ISIS lab build machines, so it is probably best to remove it after obtaining the Bugzilla password). From the "Bugzilla - Main Page" (http://bugzilla.dre.vanderbilt.edu/index.cgi) click on one of the various "LogIn" links, and from this login page you should be able to "Submit Request" to change a forgotten password. If you enter bczar@dre.vanderbilt.edu and click "Submit Request" this will email you the password change link to bczar@dre.vanderbilt.edu, which will then in turn be forwarded to your own email account. Simply copy this link into your browser to set your own bczar password for the next steps.
- here is the description of how to add a new version through Bugzilla.
- go to any Bugzilla "Query" page, you should see a grey box at the bottom. click "Log in" link to log in as bczar@dre.vanderbilt.edu.
- look at the grey box at bottom again. You will see several links following "Edit". Click on the "Product" link.
- you are then at "Select product" page. You should see a list components, i.e., ACE CIAO TAO. Click on the product you want to edit.
- you are then at "Edit product" page. Scroll down a bit, you should see a list of all versions coming with this product. At the very beginning of the list, you should see "Edit versions" link. Click this link.
- you should see a "Add a new version" text box and a "Add" link just above the grey box at the bottom. Click on this link
- you are then at "Add version of [Name of the product]" page.
- you are able to add the new version now.

- Back on naboo.dre.vanderbilt.edu once the doxygen generation has finished, update the documentation at www.dre.vanderbilt.edu/Doxygen.

- `cd /web/www/Doxygen`
- Create a temporary directory. eg. tmp and change Directory to this-
  `mkdir tmp`
  `cd tmp`
- scp file from the download server -
  `scp bczar@download.dre.vanderbilt.edu:/export/www/download.dre/ACE+TAO-distribution/ACE-html.tar.bz2 .`
- Unzip and untar the files -
  `bunzip2 ACE-html.tar.bz2;tar -xvf ACE-html.tar`
- back out of the temporary directory
  `cd ..`
- Create directory 'Current Version No' for example 5.8.1 and change directory into this new one
  `mkdir 5.8.1
  cd 5.8.1`
- Move contents of the temporary directory's html to this directory -
  `mv ../tmp/html .`
- Now back our of this directory and remove the already existing softlink to the "Micro" directory -
  `cd ..
  rm Micro`
- Create softlink "Micro" linking it to new Documentation using -
  `ln -s 5.8.1/html Micro`. If this is a minor release also update the `Stable` link.
- Remove the directory tmp -
  `rm -rf tmp`

- On anduril.dre.vanderbilt.edu where the kit was being generated and once BOTH the kit and doxygen generation have finished their work, you should also move the packages to the previous versions directory with the appropriate decorators.
  - `cd /export/www/download.dre/ACE+TAO-distribution`
  - Check that the packages in this directory are all from today
  - Modify `/export/anduriltmp/bczar/copy_script.sh` to use the correct ACE version X.Y.Z and run it.
  - Update the copy_script.sh file for the new micro release
- Validate the packages on the webserver whether they are really containing the new release. Make at least one build where you run the TAO Hello world test and check if the libraries are having the correct version number.
- Update in the autobuild archive the file configs/scoreboard/releases.xml with the made release (version number and release date). This is used by the integrated scoreboard on http://scoreboard.theaceorb.nl Remember to do a changelog entry.
- Update the ACE_wrappers repo (remember to create a changelog entry, and possibly archive the old changelog to the changelog directory if this has become too long):
  - docs/Download.html to show the new release. Make sure you refer to the previous_versions directory, that way we can exactly track how many people download a specific version.
  - etc/index.hml to show the new doxygen package you installed
  - bin/diff-builds-and-group-fixed-tests-only.sh to give the correct default old_date for this release.
- Update the NEWS, TAO/NEWS, TAO/DAnCE/NEWS, and TAO/CIAO/NEWS files to have a new section for the next release.
- Update OpenSuSE Build service using `osc checkout devel:libraries:ACE osc add cp rpmbuild/ace-tao.spec . cp debian/control debian.control cp debian/dsc ace.dsc cp debian/changelog debian.changelog cp debian/rules debian.rules osc commit`
- Mail the approved release announcement out to, at minimum the following: `ciao-users@list.isis.vanderbilt.edu`, `ciao-announce@list.isis.vanderbilt.edu`, `tao-users@list.isis.vanderbilt.edu`, `tao-announce@list.isis.vanderbilt.edu`, `ace-users@list.isis.vanderbilt.edu`, `ace-announce@list.isis.vanderbilt.edu`. Do this as yourself (not as bugzilla). N.B. You will not be able to post to the users' lists unless you are subscribed to them. Odds are you will not be able to post to the announce lists at all. Ask someone else (like Doug or Johnny) to do this step.

  ## Tips to being a Build Czar

  1. Trust no one.
  2. Be careful with this guy, he is notorious in breaking builds (and fixing them as well...Rumour has it that it's actually a super-scalar, super-pipelined processor capable of out-of-order execution, in human incarnation).
  3. Don't forgive people who break ACE :-)
  4. If a build hasn't run in a long time (symptoms are a "red" in the Last Run column of the build scoreboard), delete the .disable file in /path/to/build/directory/BUILD_NAME/ by hand.
  5. Think of the group who wrote the scoreboard update script, every time you catch an otherwise not so obvious error with the help of the scoreboard. Tell DEVO group about it.
  6. Send a note to sysadmin@isis.vanderbilt.edu asking for the repo to be frozen. Provide them a list of names, including yourself and bczar to be granted write permission.
  7. Compile once on Win32, Linux and Solaris before cutting a beta.
  8. Trust the release script when making a release. Don't make tar balls by hand.
  9. When all hell breaks loose, don't wait for the nightly builds to monitor improvement. Instead manually start the builds.
  10. Maintain private up-to-date workspaces for problem platforms (read as Solaris).
  11. Don't hesitate to ask for help.
  12. When you get an account to access the svn repo, make sure you are added to the correct groups, for example, gid=100(users),5000(doc),5002(acetaodev),5003(cvs). Otherwise you will have problem to checkout various modules.
  13. Install your public key to the different machines you have frequent access to avoid typing password.
  14. Update this page if you have any more tips for future build czars :-). This page is in git under `ACE/docs/bczar/bczar.html`

  # The Realm of the Build Czar

  ## Build Czar Arthur

  Many years have passes since the days of the legendary Build Czar Arthur. His duties were given to him by the mystical Lady of the Lake, who outlined the first responsibilities of the Build Czar.

  `
  Then bespake the Lady of the Lake,
  And these were the words said shee:
  "I knoweth of thy deeds, thou noble Arthur,
  but thy task hath not finished for thee"
  "Thou shalt feitch thy trusty steed,
  And cleanse thy builds againe;
  Then shallt thy ryde hath finnished,
  When new kits released thee cann."
  Then bespake him noble Arthur
  And these were the words said he:
  "With what weapons shallt I use,
  To asure these from the devil free?"
  Then appeered before noble Arthur,
  Uppon the ground a sacred scroll
  Conjurred by the Lady of the Lake
  Borne of the earth in a roll.
  She saies, "Clasp this to thine selfe
  For thee shallt find need for it.
  It shall keep others in the cold,
  Only to be ressurected when thee sees fit."
  "Others shall join thy person,
  To ryde with thee in thy quest;
  Thee shallt be thankful of theire help,
  And to alsoe hold them steadfast."
  "But if theire talke too lodly rise,
  And causeth much damage to thine cuntry,
  He must come forth, and make proclamation,
  For the next one he shall be."
  So hath Arthur to the Lady spoke:
  "For I sweare, and save my othe,
  While enimes and evils I seeke,
  I shall fight against them bothe.
  `