Difference between revisions of "StarlingX/Developer Guide"
Scott.little (talk | contribs) (→Method (in brief)) |
|||
(14 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
+ | |||
+ | |||
+ | See the [https://docs.starlingx.io/contributor/index.html StarlingX Build Guide] for the latest information regarding StarlingX development practices. | ||
+ | This wiki page has been deprecated. | ||
+ | |||
+ | <!-- | ||
+ | |||
This section contains the steps for building a StarlingX ISO from Master branch. | This section contains the steps for building a StarlingX ISO from Master branch. | ||
Line 67: | Line 74: | ||
<ol start="1"><li>Under your $HOME directory, clone the <stx-tools> project | <ol start="1"><li>Under your $HOME directory, clone the <stx-tools> project | ||
− | <source lang="sh">$ | + | <source lang="sh"> |
− | + | $ git -C $HOME clone https://git.starlingx.io/stx-tools | |
</source></li></ol> | </source></li></ol> | ||
Line 84: | Line 91: | ||
=== Setup Repository Docker Container === | === Setup Repository Docker Container === | ||
− | Run the following commands under a terminal identified as "One". <br> | + | Run the following commands under a ''terminal identified as'' "'''One'''". <br> |
<ol start="1"><li>Navigate to the ''<$HOME/stx-tools>/centos-mirror-tool'' project directory: | <ol start="1"><li>Navigate to the ''<$HOME/stx-tools>/centos-mirror-tool'' project directory: | ||
Line 395: | Line 402: | ||
* A server in the regional office performs a regular (daily?), automated builds using existing methods. Call these the reference builds. | * A server in the regional office performs a regular (daily?), automated builds using existing methods. Call these the reference builds. | ||
* The builds are timestamped, and preserved for some time. (a few weeks) | * The builds are timestamped, and preserved for some time. (a few weeks) | ||
− | * A build CONTEXT is captured | + | * A build CONTEXT is captured. This is a file produced by build-pkgs at location '$MY_WORKSPACE/CONTEXT'. It is a bash script that can cd to each and every git and checkout the SHA that contributed to the build. |
− | * For each package built, a file shall capture he md5sums of all the source code inputs to the build of that package. | + | * For each package built, a file shall capture he md5sums of all the source code inputs to the build of that package. These files are also produced by build-pkgs at location '$MY_WORKSPACE/<build-type>/rpmbuild/SOURCES/<pkg-name>/srpm_reference.md5'. |
* All these build products are accessible locally (e.g. a regional office) via rsync (other protocols can be added later) | * All these build products are accessible locally (e.g. a regional office) via rsync (other protocols can be added later) | ||
</li> | </li> | ||
Line 410: | Line 417: | ||
** Using a config file to specify location of your reference build<source> | ** Using a config file to specify location of your reference build<source> | ||
mkdir -p $MY_REPO/local-build-data | mkdir -p $MY_REPO/local-build-data | ||
+ | |||
cat <<- EOF > $MY_REPO/local-build-data/build_avoidance_source | cat <<- EOF > $MY_REPO/local-build-data/build_avoidance_source | ||
# Optional, these are already the default values. | # Optional, these are already the default values. | ||
Line 427: | Line 435: | ||
** Using command line args to specify location of your reference build<source> | ** Using command line args to specify location of your reference build<source> | ||
build-pkgs --build-avoidance --build-avoidance-dir /localdisk/loadbuild/jenkins/StarlingX_Reference_Build --build-avoidance-host stx-builder.mycompany.com --build-avoidance-user jenkins | build-pkgs --build-avoidance --build-avoidance-dir /localdisk/loadbuild/jenkins/StarlingX_Reference_Build --build-avoidance-host stx-builder.mycompany.com --build-avoidance-user jenkins | ||
+ | </source> | ||
+ | * Prior to your build attempt, you need to accept the host key. This will prevent rsync failures on a yes/no prompt. (you should only have to do this once)<source> | ||
+ | grep -q $BUILD_AVOIDANCE_HOST $HOME/.ssh/known_hosts | ||
+ | if [ $? != 0 ]; then | ||
+ | ssh-keyscan $BUILD_AVOIDANCE_HOST >> $HOME/.ssh/known_hosts | ||
+ | fi | ||
</source> | </source> | ||
* build-pkgs will: | * build-pkgs will: | ||
Line 434: | Line 448: | ||
** The designer may have additional commits not present in the reference build, or uncommitted changes. Affected packages will identified by the differing md5sum's, and the package is re-built. (5+ min, depending on what packages have changed) | ** The designer may have additional commits not present in the reference build, or uncommitted changes. Affected packages will identified by the differing md5sum's, and the package is re-built. (5+ min, depending on what packages have changed) | ||
</li> | </li> | ||
+ | |||
+ | * What if no valid reference build is found? Then build-pkgs will fall back to a regular build. | ||
</ol> | </ol> | ||
Line 441: | Line 457: | ||
* Each build is saved to a unique directory, and preserved for a time that is reflective of how long a designer might be expected to work on a private branch without syncronizing with the master branch. e.g. 2 weeks. | * Each build is saved to a unique directory, and preserved for a time that is reflective of how long a designer might be expected to work on a private branch without syncronizing with the master branch. e.g. 2 weeks. | ||
* The MY_WORKSPACE directory for the build shall have a common root directory, and a leaf directory that is a sortable time stamp. Suggested format YYYYMMDDThhmmss. e.g. <source lang="sh">$ sudo apt-get update | * The MY_WORKSPACE directory for the build shall have a common root directory, and a leaf directory that is a sortable time stamp. Suggested format YYYYMMDDThhmmss. e.g. <source lang="sh">$ sudo apt-get update | ||
+ | BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build" | ||
BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ') | BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ') | ||
− | MY_WORKSPACE= | + | MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${BUILD_TIMESTAMP} |
</source> | </source> | ||
* Designers can access all build products over the internal network of the regional office. The current prototype employs rsync. Other protocols that can efficiently share/copy/transfer large directories of content can be added as needed. | * Designers can access all build products over the internal network of the regional office. The current prototype employs rsync. Other protocols that can efficiently share/copy/transfer large directories of content can be added as needed. | ||
+ | |||
+ | ==== Advanced usage ==== | ||
+ | |||
+ | Can the reference build itself use build avoidance? Yes<br> | ||
+ | Can it reference itself? Yes.<br> | ||
+ | In either case we advise caution. To protect against any possible 'divergence from reality', you should limit how many steps removed a build avoidance build is from a full build.<br> | ||
+ | Suppose we want to implement a self referencing daily build, except that a full build occurs every Saturday. To protect ourselves from a build failure on Saturday we also want a limit of 7 days since last full build. You build script might look like this ...<source> | ||
+ | ... | ||
+ | BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build" | ||
+ | BUILD_AVOIDANCE_HOST="stx-builder.mycompany.com" | ||
+ | FULL_BUILD_DAY="Saturday" | ||
+ | MAX_AGE_DAYS=7 | ||
+ | |||
+ | LAST_FULL_BUILD_LINK="$BUILD_AVOIDANCE_DIR/latest_full_build" | ||
+ | LAST_FULL_BUILD_DAY="" | ||
+ | NOW_DAY=$(date -u "+%A") | ||
+ | BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ') | ||
+ | MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${BUILD_TIMESTAMP} | ||
+ | |||
+ | # update software | ||
+ | repo init -u ${BUILD_REPO_URL} -b ${BUILD_BRANCH} | ||
+ | repo sync --force-sync | ||
+ | $MY_REPO_ROOT_DIR/stx-tools/toCOPY/generate-cgcs-centos-repo.sh | ||
+ | $MY_REPO_ROOT_DIR/stx-tools/toCOPY/populate_downloads.sh | ||
+ | |||
+ | # User can optionally define BUILD_METHOD equal to one of 'FULL', 'AVOIDANCE', or 'AUTO' | ||
+ | # Sanitize BUILD_METHOD | ||
+ | if [ "$BUILD_METHOD" != "FULL" ] && [ "$BUILD_METHOD" != "AVOIDANCE" ]; then | ||
+ | BUILD_METHOD="AUTO" | ||
+ | fi | ||
+ | |||
+ | # First build test | ||
+ | if [ "$BUILD_METHOD" != "FULL" ] && [ ! -L $LAST_FULL_BUILD_LINK ]; then | ||
+ | echo "latest_full_build symlink missing, forcing full build" | ||
+ | BUILD_METHOD="FULL" | ||
+ | fi | ||
+ | |||
+ | # Build day test | ||
+ | if [ "$BUILD_METHOD" == "AUTO" ] && [ "$NOW_DAY" == "$FULL_BUILD_DAY" ]; then | ||
+ | echo "Today is $FULL_BUILD_DAY, forcing full build" | ||
+ | BUILD_METHOD="FULL" | ||
+ | fi | ||
+ | |||
+ | # Build age test | ||
+ | if [ "$BUILD_METHOD" != "FULL" ]; then | ||
+ | LAST_FULL_BUILD_DATE=$(basename $(readlink $LAST_FULL_BUILD_LINK) | cut -d '_' -f 1) | ||
+ | LAST_FULL_BUILD_DAY=$(date -d $LAST_FULL_BUILD_DATE "+%A") | ||
+ | AGE_SECS=$(( $(date "+%s") - $(date -d $LAST_FULL_BUILD_DATE "+%s") )) | ||
+ | AGE_DAYS=$(( $AGE_SECS/60/60/24 )) | ||
+ | if [ $AGE_DAYS -ge $MAX_AGE_DAYS ]; then | ||
+ | echo "Haven't had a full build in $AGE_DAYS days, forcing full build" | ||
+ | BUILD_METHOD="FULL" | ||
+ | fi | ||
+ | BUILD_METHOD="AVOIDANCE" | ||
+ | fi | ||
+ | |||
+ | #Build it | ||
+ | if [ "$BUILD_METHOD" == "FULL" ]; then | ||
+ | build-pkgs --no-build-avoidance | ||
+ | else | ||
+ | build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER | ||
+ | fi | ||
+ | if [ $? -ne 0 ]; then | ||
+ | echo "Build failed in build-pkgs" | ||
+ | exit 1 | ||
+ | fi | ||
+ | |||
+ | build-iso | ||
+ | if [ $? -ne 0 ]; then | ||
+ | echo "Build failed in build-iso" | ||
+ | exit 1 | ||
+ | fi | ||
+ | |||
+ | if [ "$BUILD_METHOD" == "FULL" ]; then | ||
+ | # A successful full build. Set last full build symlink. | ||
+ | if [ -L $LAST_FULL_BUILD_LINK ]; then | ||
+ | rm -rf $LAST_FULL_BUILD_LINK | ||
+ | fi | ||
+ | ln -sf $MY_WORKSPACE $LAST_FULL_BUILD_LINK | ||
+ | fi | ||
+ | ... | ||
+ | </source> | ||
+ | |||
+ | One final wrinkle. | ||
+ | We can ask build avoidance to preferentially use the full build day rather than the most recent build, as the reference point of the next avoidance build via use of '--build-avoidance-day <day-name>'. e.g. substitute this line into the above. <source> | ||
+ | build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER --build-avoidance-day $FULL_BUILD_DAY | ||
+ | |||
+ | # or perhaps, with a bit more shuffling of the above script. | ||
+ | |||
+ | build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER --build-avoidance-day $LAST_FULL_BUILD_DAY | ||
+ | </source> | ||
+ | The advantage is that our build is never more than one step removed from a full build (assuming the full build was successful).<br> | ||
+ | The disadvantage is that by end of week the reference build is getting rather old. During active weeks, builds times might be approaching that of a full build. | ||
+ | |||
+ | --> |
Latest revision as of 18:46, 16 July 2019
See the StarlingX Build Guide for the latest information regarding StarlingX development practices.
This wiki page has been deprecated.