Jump to: navigation, search

Difference between revisions of "StarlingX/Developer Guide"

(Method (in brief))
 
(14 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
 +
 +
See the [https://docs.starlingx.io/contributor/index.html StarlingX Build Guide] for the latest information regarding StarlingX development practices.
 +
This wiki page has been deprecated.
 +
 +
<!--
 +
 
This section contains the steps for building a StarlingX ISO from Master branch.
 
This section contains the steps for building a StarlingX ISO from Master branch.
  
Line 67: Line 74:
 
<ol start="1"><li>Under your $HOME directory, clone the &lt;stx-tools&gt; project
 
<ol start="1"><li>Under your $HOME directory, clone the &lt;stx-tools&gt; project
  
<source lang="sh">$ cd $HOME
+
<source lang="sh">
$ git clone https://git.starlingx.io/stx-tools
+
$ git -C $HOME clone https://git.starlingx.io/stx-tools
 
</source></li></ol>
 
</source></li></ol>
  
Line 84: Line 91:
 
=== Setup Repository Docker Container ===
 
=== Setup Repository Docker Container ===
  
Run the following commands under a terminal identified as "One". <br>
+
Run the following commands under a ''terminal identified as'' "'''One'''". <br>
  
 
<ol start="1"><li>Navigate to the ''&lt;$HOME/stx-tools&gt;/centos-mirror-tool'' project directory:
 
<ol start="1"><li>Navigate to the ''&lt;$HOME/stx-tools&gt;/centos-mirror-tool'' project directory:
Line 395: Line 402:
 
* A server in the regional office performs a regular (daily?), automated builds using existing methods.  Call these the reference builds.
 
* A server in the regional office performs a regular (daily?), automated builds using existing methods.  Call these the reference builds.
 
* The builds are timestamped, and preserved for some time.  (a few weeks)
 
* The builds are timestamped, and preserved for some time.  (a few weeks)
* A build CONTEXT is captured, consisting of the SHA of each and every git that contributed to the build.
+
* A build CONTEXT is captured. This is a file produced by build-pkgs at location '$MY_WORKSPACE/CONTEXT'.  It is a bash script that can cd to each and every git and checkout the SHA that contributed to the build.
* For each package built, a file shall capture he md5sums of all the source code inputs to the build of that package.
+
* For each package built, a file shall capture he md5sums of all the source code inputs to the build of that package.  These files  are also produced by build-pkgs at location '$MY_WORKSPACE/<build-type>/rpmbuild/SOURCES/<pkg-name>/srpm_reference.md5'.
 
* All these build products are accessible locally (e.g. a regional office) via rsync (other protocols can be added later)
 
* All these build products are accessible locally (e.g. a regional office) via rsync (other protocols can be added later)
 
</li>
 
</li>
Line 410: Line 417:
 
** Using a config file to specify location of your reference build<source>
 
** Using a config file to specify location of your reference build<source>
 
mkdir -p $MY_REPO/local-build-data
 
mkdir -p $MY_REPO/local-build-data
 +
 
cat <<- EOF > $MY_REPO/local-build-data/build_avoidance_source
 
cat <<- EOF > $MY_REPO/local-build-data/build_avoidance_source
 
# Optional, these are already the default values.
 
# Optional, these are already the default values.
Line 427: Line 435:
 
** Using command line args to specify location of your reference build<source>
 
** Using command line args to specify location of your reference build<source>
 
build-pkgs --build-avoidance --build-avoidance-dir /localdisk/loadbuild/jenkins/StarlingX_Reference_Build --build-avoidance-host stx-builder.mycompany.com --build-avoidance-user jenkins
 
build-pkgs --build-avoidance --build-avoidance-dir /localdisk/loadbuild/jenkins/StarlingX_Reference_Build --build-avoidance-host stx-builder.mycompany.com --build-avoidance-user jenkins
 +
</source>
 +
* Prior to your build attempt, you need to accept the host key.  This will prevent rsync failures on a yes/no prompt. (you should only have to do this once)<source>
 +
grep -q $BUILD_AVOIDANCE_HOST $HOME/.ssh/known_hosts
 +
if [ $? != 0 ]; then
 +
    ssh-keyscan $BUILD_AVOIDANCE_HOST >> $HOME/.ssh/known_hosts
 +
fi
 
</source>
 
</source>
 
* build-pkgs will:
 
* build-pkgs will:
Line 434: Line 448:
 
** The designer may have additional commits not present in the reference build, or uncommitted changes.  Affected packages will identified by the differing md5sum's, and the package is re-built. (5+ min, depending on what packages have changed)
 
** The designer may have additional commits not present in the reference build, or uncommitted changes.  Affected packages will identified by the differing md5sum's, and the package is re-built. (5+ min, depending on what packages have changed)
 
</li>
 
</li>
 +
 +
* What if no valid reference build is found?  Then build-pkgs will fall back to a regular build.
 
</ol>
 
</ol>
  
Line 441: Line 457:
 
* Each build is saved to a unique directory, and preserved for a time that is reflective of how long a designer might be expected to work on a private branch without syncronizing with the master branch.  e.g. 2 weeks.
 
* Each build is saved to a unique directory, and preserved for a time that is reflective of how long a designer might be expected to work on a private branch without syncronizing with the master branch.  e.g. 2 weeks.
 
* The MY_WORKSPACE directory for the build shall have a common root directory, and a leaf directory that is a sortable time stamp. Suggested format YYYYMMDDThhmmss.  e.g. <source lang="sh">$ sudo apt-get update
 
* The MY_WORKSPACE directory for the build shall have a common root directory, and a leaf directory that is a sortable time stamp. Suggested format YYYYMMDDThhmmss.  e.g. <source lang="sh">$ sudo apt-get update
 +
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
 
BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ')
 
BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ')
MY_WORKSPACE=/localdisk/loadbuild/jenkins/StarlingX/${BUILD_TIMESTAMP}
+
MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${BUILD_TIMESTAMP}
 
</source>
 
</source>
 
* Designers can access all build products over the internal network of the regional office.  The current prototype employs rsync. Other protocols that can efficiently share/copy/transfer large directories of content can be added as needed.
 
* Designers can access all build products over the internal network of the regional office.  The current prototype employs rsync. Other protocols that can efficiently share/copy/transfer large directories of content can be added as needed.
 +
 +
==== Advanced usage ====
 +
 +
Can the reference build itself use build avoidance? Yes<br>
 +
Can it reference itself?  Yes.<br>
 +
In either case we advise caution.  To protect against any possible 'divergence from reality', you should limit how many steps removed a build avoidance build is from a full build.<br>
 +
Suppose we want to implement a self referencing daily build, except that a full build occurs every Saturday.  To protect ourselves from a build failure on Saturday we also want a limit of 7 days since last full build.  You build script might look like this ...<source>
 +
...
 +
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
 +
BUILD_AVOIDANCE_HOST="stx-builder.mycompany.com"
 +
FULL_BUILD_DAY="Saturday"
 +
MAX_AGE_DAYS=7
 +
 +
LAST_FULL_BUILD_LINK="$BUILD_AVOIDANCE_DIR/latest_full_build"
 +
LAST_FULL_BUILD_DAY=""
 +
NOW_DAY=$(date -u "+%A")
 +
BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ')
 +
MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${BUILD_TIMESTAMP}
 +
 +
# update software
 +
repo init -u ${BUILD_REPO_URL} -b ${BUILD_BRANCH}
 +
repo sync --force-sync
 +
$MY_REPO_ROOT_DIR/stx-tools/toCOPY/generate-cgcs-centos-repo.sh
 +
$MY_REPO_ROOT_DIR/stx-tools/toCOPY/populate_downloads.sh
 +
 +
# User can optionally define BUILD_METHOD equal to one of 'FULL', 'AVOIDANCE', or 'AUTO'
 +
# Sanitize BUILD_METHOD
 +
if [ "$BUILD_METHOD" != "FULL" ] && [ "$BUILD_METHOD" != "AVOIDANCE" ]; then
 +
    BUILD_METHOD="AUTO"
 +
fi
 +
 +
# First build test
 +
if [ "$BUILD_METHOD" != "FULL" ] && [ ! -L $LAST_FULL_BUILD_LINK ]; then
 +
    echo "latest_full_build symlink missing, forcing full build"
 +
    BUILD_METHOD="FULL"
 +
fi
 +
 +
# Build day test
 +
if [ "$BUILD_METHOD" == "AUTO" ] && [ "$NOW_DAY" == "$FULL_BUILD_DAY" ]; then
 +
    echo "Today is $FULL_BUILD_DAY, forcing full build"
 +
    BUILD_METHOD="FULL"
 +
fi
 +
 +
# Build age test
 +
if [ "$BUILD_METHOD" != "FULL" ]; then
 +
    LAST_FULL_BUILD_DATE=$(basename $(readlink $LAST_FULL_BUILD_LINK) | cut -d '_' -f 1)
 +
    LAST_FULL_BUILD_DAY=$(date -d $LAST_FULL_BUILD_DATE "+%A")
 +
    AGE_SECS=$(( $(date "+%s") - $(date -d $LAST_FULL_BUILD_DATE "+%s") ))
 +
    AGE_DAYS=$(( $AGE_SECS/60/60/24 ))
 +
    if [ $AGE_DAYS -ge $MAX_AGE_DAYS ]; then
 +
        echo "Haven't had a full build in $AGE_DAYS days, forcing full build"
 +
        BUILD_METHOD="FULL"
 +
    fi
 +
    BUILD_METHOD="AVOIDANCE"
 +
fi
 +
 +
#Build it
 +
if [ "$BUILD_METHOD" == "FULL" ]; then
 +
    build-pkgs --no-build-avoidance
 +
else
 +
    build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER
 +
fi
 +
if [ $? -ne 0 ]; then
 +
    echo "Build failed in build-pkgs"
 +
    exit 1
 +
fi
 +
 +
build-iso
 +
if [ $? -ne 0 ]; then
 +
    echo "Build failed in build-iso"
 +
    exit 1
 +
fi
 +
 +
if [ "$BUILD_METHOD" == "FULL" ]; then
 +
    # A successful full build.  Set last full build symlink.
 +
    if [ -L $LAST_FULL_BUILD_LINK ]; then
 +
        rm -rf $LAST_FULL_BUILD_LINK
 +
    fi
 +
    ln -sf $MY_WORKSPACE $LAST_FULL_BUILD_LINK
 +
fi
 +
...
 +
</source>
 +
 +
One final wrinkle.
 +
We can ask build avoidance to preferentially use the full build day rather than the most recent build, as the reference point of the next avoidance build via use of  '--build-avoidance-day <day-name>'. e.g. substitute this line into the above.  <source>
 +
build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER --build-avoidance-day $FULL_BUILD_DAY
 +
 +
# or perhaps, with a bit more shuffling of the above script.
 +
 +
build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER --build-avoidance-day $LAST_FULL_BUILD_DAY
 +
</source>
 +
The advantage is that our build is never more than one step removed from a full build (assuming the full build was successful).<br>
 +
The disadvantage is that by end of week the reference build is getting rather old.  During active weeks, builds times might be approaching that of a full build.
 +
 +
-->

Latest revision as of 18:46, 16 July 2019


See the StarlingX Build Guide for the latest information regarding StarlingX development practices. This wiki page has been deprecated.