This is a guide for Acumos software developers and testers about the continuous integration/continuous deployment (CI/CD) resources and processes supported by the Linux Foundation.
CI/CD Network Resources
These resources require a Linux Foundation identity.
|https://identity.linuxfoundation.org||The Linux Foundation identity management web site.|
|https://jira.linuxfoundation.org/servicedesk/customer/portals||The Linux Foundation release engineering team's service desk.|
|https://gerrit.acumos.org||The Gerrit code review server hosts the Git repositories and supports the review, merge and release processes. The review process basically requires all users to follow something like a conventional git pull-request process, but restricts the publication (push) of private branches.|
|https://jenkins.acumos.org||Jenkins is the continuous-integration server aka the build server. All users can see the contents of the Jenkins; no users can modify the configuration nor start a job in this Jenkins.|
|https://jenkins.acumos.org/sandbox||The Jenkins sandbox is a place for testing Jenkins job configurations. After requesting access, users can create jobs, reconfigure jobs, trigger builds etc. After JJB templates have been tested in the sandbox they can be submitted for the main Jenkins server.|
|https://jira.acumos.org||Jira is the web site that tracks issues, epics, stories, etc.|
This Nexus 2 repository holds Maven (i.e., jar) artifacts produced by builds. Snapshot and staging builds are all deployed to this repository. Release artifacts are created by promoting artifacts manually from the staging repository after suitable approval. Publicly accessible to users without credentials. All users should be able to access and browse artifacts through this URL.
This Nexus 3 server publishes external and internal Docker images. Supports the following registries:
Access to these registries requires authentication with this well-known username and password:
These credentials provide read-only access to all the registries above. Only the Jenkins server has write access to the snapshot and staging registries.
|https://sonar.acumos.org||Sonar analyzes Java code for quality issues and code coverage achieved by unit tests (JUnit).|
Development Procedures and Policies
This section guides the developer in submitting code, reviewing code, tracking the status of builds and requesting creation of release versions. These recommended practices come from the Linux Foundation.
Maven POM version numbers and dependencies
All Java project POM files should always use the suffix "-SNAPSHOT" in their project version strings, that's the "V" in the GAV triple defined at the top, at all times. This is slightly counter intuitive. The Jenkins jobs expect this and manage the version strings appropriately.
In contrast, Java project POM files should use release versions of dependencies (not SNAPSHOT versions). It may be necessary to use SNAPSHOT versions of dependencies for development, but that must be regarded as temporary. All releases given to the test team and published for use by the world MUST use only release versions.
A peculiarity of the LF build environment is that every Jenkins job that stages a release candidate strips all "-SNAPSHOT" suffixes within POM files. This affects the project's own version and all named dependencies.
Maven repositories and pluginRepositories URLs are managed externally
Per the LF, Java project POM files should have neither a "repositories" nor a "pluginRepositories" section. These sections can be used to store URLs of the servers that supply dependencies - jar and plugin files - other than the well-known Maven Central (central.maven.org) server. Those sections are provided on the LF Jenkins build server by a configured "settings.xml" file. Individual developers who build the software on their own machines must create a settings file at path ~/.m2/settings.xml (i.e., in their home directory where maven looks by default) with this information.
Always create a branch
When working with a git repository cloned from Gerrit you can create as many local branches as you like. Note that none of the branches will be pushed to the remote Gerrit server, the branches remain forever private. Creating branches for each task will allow you to work on multiple independent tasks in parallel, let you recover gracefully from various situations, and generally save aggravation and time. Also see these instructions on tagging and branching for releases:
Write good git commit messages
The Linux Foundation strongly recommends (and eventually will enforce) Gerrit commit message content. A git commit message must meet these requirements:
- The first line should have a short message, up to 50 characters
- The second line must be blank
- The body of the commit message should have a detailed description of the change
- The line before the footer must be blank
- The footer (last block of lines following a blank line) must consist of these lines:
- Issue-ID: line with a valid Acumos Jira ticket number, composed and inserted manually by the committer
- Change-Id: line, which is automatically generated and inserted by git review
- Signed-off-by line, which is automatically generated and inserted by git commit
An example is shown below:
Acumos uses Gerrit to automate the process of reviewing all code changes before they are committed to the Git repository. For a tutorial on git you might start here:
Here is a tutorial and reference on using Gerrit:
Also see this guide published by the LF Release Engineering team about using Gerrit:
Gerrit is designed to host text files – source code. It enforces a size threshold on every commit, the default limit is 5MB. Further the Linux Foundation prohibits committing binary files such as compiled executables, jar files, zip archives and so on. An exception is made for binary picture (image) files in GIF, JPG and PNG formats, but the size limit must still be honored.
The quickstart guides below describe the command-line procedures for performing common tasks on Gerrit. The command-line tool "git review" is the most reliable and can be used on any platform. Windows users should install "Git Bash" to gain support for command-line git.
Quickstart: Create and submit a change for review
git checkout -b my-local-branch
Quickstart: Revise your open gerrit review
git checkout my-local-branch
Quickstart: Revise any open gerrit review
# Look up the change number shown top-left in Gerrit web, here "999".
Quickstart: Resolve a conflicted review
git checkout master
Quickstart: Squash commits after you forgot the "--amend" flag
Reviewing and merging on Gerrit web site
Submitted reviews with changes should appear in the Gerrit web interface. The Jenkins job builder will publish test results, giving "Verified +1" if the test succeeds and -1 if the test fails. A contributor or committer can review it with a -1/0/+1. A committer then must approve it with a +2 rating and merge the code to the master branch. All actions are done on the gerrit.acumos.org web site.
The committer may take any of several actions, such as clicking on the "Reply" button, adding reviewers, adding a review comment, and moving the flags to +2 and +1
Once a committer/contributor approves it, the code can be merged to the master branch.
The CLM jobs are still scheduled to run every Saturday, this feature can be useful for debugging on demand. Commenting "run-clm" in a gerrit that is not merged, will not trigger the CLM job based on that revision but will trigger the job based on the tip of the branch. This job is designed to always run on the latest tip of the branch to avoid inconsistencies on the reports.
Staging a release candidate
Once testing of the snapshot version has been completed, prepare a release candidate in the staging area. First ensure that your POM file has only release versions in the dependency sections. Then trigger the appropriate Jenkins job to stage a release candidate as noted above. The job uses magic (ok a shell script) to strip the string "-SNAPSHOT" from every line of all of your project's POM files. This means any lingering dependency version string with -SNAPSHOT will also be modified, not just the project version string. If the modified build succeeds, the build outputs are deployed or pushed to a staging repository.
Jar files in the staging repository are not used by the LF Jenkins server - there is no Maven profile for this. My advice is to treat the staging repository as a brief resting stop on the way to full release status.
If you force a local build job to pull artifacts from a staging repository by fiddling your local settings, be warned that the staging server will give you the OLDEST artifact in the staging repository, not the newest. This is a but, and a very real practical limitation on the usefulness of the staging repository.
Releasing your project
Once testing against the staging repo version has been completed (see above ) and the project has determined that the artifact in the staged repository is ready for release, a release can then be performed as follows:
- Find the Jenkins stage job that created the release candidate. Look among its output logs for the file with the name: staging-repo.txt.gz, it will have a URL like this:
- Open a ticket with the LF service desk to sign the artifact(s) in the staging repo and promote them to the release repo. Example request is below.
- Login at https://jira.linuxfoundation.org/servicedesk/customer/portals
- Create a new request - search for "Release Artifacts"
- Paste in the Jenkins stage job URL and Jenkins log URL from above.
- Paste in the name of the autorelease repository.
Example service request content:
Once complete, the artifacts should appear in the Nexus2 release repository and/or Nexus3 release registry.
Finally, add a tag for the release on the Gerrit web site.
Capturing the release in Gerrit history
This addresses the need for a project to reproduce from git the EXACT, byte-for-byte state of the project's files as built, which is not a requirement for all projects. The Jenkins build server's release job modifies POM files to strip --SNAPSHOT suffixes. The Jenkins server stores in the local git repository the state of the project after that modification. That state is published via a git bundle file that is pushed to the log server, it is NOT pushed directly back to Gerrit, because Jenkins cannot push changes back to Gerrit. To add that item into project history, that commit can be downloaded, merged to the local repository and submitted via a Gerrit review into the global history. Steps:
- Download the taglist.log and project.bundle files from the log server
- Using the SHA code from taglist.log, checkout the commit hash
- Merge the project.bundle patch into the local repository
- Submit the change as a git review to Gerrit, and merge the patch set
- Git tag the release
- Push the release tag to Gerrit.