Jump to: navigation, search

Difference between revisions of "StarlingX/Developer Guide"

 
(220 intermediate revisions by 12 users not shown)
Line 1: Line 1:
This section contains the steps for building a StarlingX ISO.
+
 
 +
 
 +
See the [https://docs.starlingx.io/contributor/index.html StarlingX Build Guide] for the latest information regarding StarlingX development practices.
 +
This wiki page has been deprecated.
 +
 
 +
<!--
 +
 
 +
This section contains the steps for building a StarlingX ISO from Master branch.
  
 
== Requirements ==
 
== Requirements ==
Line 9: Line 16:
 
A workstation computer with:
 
A workstation computer with:
  
* Processor: x86_64 only supported architecture
+
* Processor: x86_64 is the only supported architecture
 
* Memory: At least 32GB RAM
 
* Memory: At least 32GB RAM
* Hard Disk: 100GB HDD
+
* Hard Disk: 500GB HDD
 
* Network: Network adapter with active Internet connection
 
* Network: Network adapter with active Internet connection
  
Line 22: Line 29:
 
* Android Repo Tool
 
* Android Repo Tool
 
* Proxy Settings Configured (If Required)
 
* Proxy Settings Configured (If Required)
 +
** See http://lists.starlingx.io/pipermail/starlingx-discuss/2018-July/000136.html for more details
 
* Public SSH Key
 
* Public SSH Key
  
Line 38: Line 46:
 
=== Installation Requirements and Dependencies ===
 
=== Installation Requirements and Dependencies ===
  
<ol start="1"><li>Install the required packages in an Ubuntu host system with:
+
<ol start="1">
 +
 
 +
==== Git ====
 +
<li>Install the required packages in an Ubuntu host system with:
 +
<source lang="sh">$ sudo apt-get install make git curl
 +
</source></li>
  
<source lang="sh">$ sudo apt-get install git
+
<li>Make sure to setup your identity
 +
<source lang="sh">$ git config --global user.name "Name LastName"
 +
$ git config --global user.email "Email Address"
 
</source></li>
 
</source></li>
<li><p>Install the required Docker CE packages in an Ubuntu host system. See [https://docs.docker.com/install/ Get Docker] for more information.</p></li>
+
 
<li><p>Install the required Android Repo Tool in an Ubuntu host system. See [https://source.android.com/setup/build/downloading#installing-repo Installing Repo] for more information.</p></li></ol>
+
==== Docker CE ====
 +
<li><p>Install the required Docker CE packages in an Ubuntu host system. See [https://docs.docker.com/install/linux/docker-ce/ubuntu/#os-requirements Get Docker CE for Ubuntu] for more information.</p></li>
 +
 
 +
==== Android Repo Tool ====
 +
<li><p>Install the required Android Repo Tool in an Ubuntu host system. Follow the 2 steps in "Installing Repo" section from [https://source.android.com/setup/build/downloading#installing-repo Installing Repo] to have Andriod Repo Tool installed.</p></li>
  
 
=== Install Public SSH Key===
 
=== Install Public SSH Key===
Line 53: Line 72:
 
=== Install stx-tools project ===
 
=== Install stx-tools project ===
  
<ol start="1"><li>Clone the &lt;stx-tools&gt; project
+
<ol start="1"><li>Under your $HOME directory, clone the &lt;stx-tools&gt; project
  
<source lang="sh">$ git clone git://git.openstack.org/openstack/stx-tools
+
<source lang="sh">
 +
$ git -C $HOME clone https://git.starlingx.io/stx-tools
 
</source></li></ol>
 
</source></li></ol>
  
Line 71: Line 91:
 
=== Setup Repository Docker Container ===
 
=== Setup Repository Docker Container ===
  
<ol start="1"><li>Navigate to the ''&lt;stx-tools&gt;/centos-mirror-tool'' project directory:
+
Run the following commands under a ''terminal identified as'' "'''One'''". <br>
  
<source lang="sh">$ cd stx-tools/centos-mirror-tools/
+
<ol start="1"><li>Navigate to the ''&lt;$HOME/stx-tools&gt;/centos-mirror-tool'' project directory:
 +
 
 +
<source lang="sh">$ cd $HOME/stx-tools/centos-mirror-tools/
 
</source></li>
 
</source></li>
 +
 
<li>If necessary you might have to set http/https proxy in your Dockerfile before building the docker image.
 
<li>If necessary you might have to set http/https proxy in your Dockerfile before building the docker image.
  
<source lang="sh">ENV http_proxy "http://your.actual_http_proxy.com:your_port" && \
+
<source lang="sh">
    https_proxy "https://your.actual_https_proxy.com:your_port" && \
+
ENV http_proxy " http://your.actual_http_proxy.com:your_port "
    ftp_proxy "http://your.actual_ftp_proxy.com:your_port"
+
ENV https_proxy " https://your.actual_https_proxy.com:your_port "
RUN echo "proxy=http://your-proxy.com:port" >> /etc/yum.conf
+
ENV ftp_proxy " http://your.actual_ftp_proxy.com:your_port "
 +
RUN echo " proxy=http://your-proxy.com:port " >> /etc/yum.conf
 
</source></li>
 
</source></li>
<li>Build your ''&lt;name&gt;:&lt;tag&gt;'' base container image with '''e.g.''' ''aarcemor:centos-mirror-repository''
+
<li>Build your ''&lt;user&gt;:&lt;tag&gt;'' base container image with '''e.g.''' ''user:centos-mirror-repository''
  
<source lang="sh">$ docker build -t aarcemor:centos-mirror-repository -f Dockerfile .
+
<source lang="sh">$ docker build --tag $USER:centos-mirror-repository --file Dockerfile .
 
</source></li>
 
</source></li>
<li>Launch a ''&lt;name&gt;'' docker container using previously created Docker base container image ''&lt;name&gt;:&lt;tag&gt;'' '''e.g.''' ''aarcemor-centos-mirror-repository''. As /localdisk is defined as the workdir of the container, the same folder name should be used to define the volume. The container will start to run and populate a logs and output folders in this directory. The container shall be run from the same directory where the other scripts are stored.
+
<li>Launch a ''&lt;user&gt;'' docker container using previously created Docker base container image ''&lt;user&gt;:&lt;tag&gt;'' '''e.g.''' ''<user>-centos-mirror-repository''. As /localdisk is defined as the workdir of the container, the same folder name should be used to define the volume. The container will start to run and populate a logs and output folders in this directory. The container shall be run from the same directory where the other scripts are stored.
  
<source lang="sh">$ docker run -itd --name aarcemor-centos-mirror-repository -v $(pwd):/localdisk aarcemor:centos-mirror-repository bash
+
<source lang="sh">$ docker run -itd --name $USER-centos-mirror-repository --volume $(pwd):/localdisk $USER:centos-mirror-repository
</source></li>
+
</source>
<li>Execute the ''&lt;name&gt;'' docker container '''e.g.''' ''aarcemor-centos-mirror-repository''
+
'''Note''':  the above command will create the container in background, this mean that you need to attach it manually. The advantage of this is that you can enter/exit from the container many times as you want.
 
+
</li></ol>
<source lang="sh">$ docker exec -it aarcemor-centos-mirror-repository bash
 
</source></li></ol>
 
 
 
=== Import GPG Keys ===
 
 
 
<ol start="1"><li>Inside the docker container, import the keys into the local GPG keyring and query public key information:
 
 
 
<source lang="none"># rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY*
 
# rpm -qi gpg-pubkey-\*
 
</source></li></ol>
 
  
 
=== Download Packages ===
 
=== Download Packages ===
  
<ol start="1"><li>Enter the following command to download the required packages to populate the CentOS Mirror Repository:
+
<ol start="1">
 +
<li>Attach to the docker repository previously created
 +
<source lang="none">$ docker exec -it <CONTAINER ID> /bin/bash</source>
 +
</li>
 +
<li>Inside Repository Docker container, enter the following command to download the required packages to populate the CentOS Mirror Repository:
 +
<source lang="none"># bash download_mirror.sh</source>
 +
</li>
  
<source lang="none"># bash download_mirror.sh
 
</source></li>
 
 
<li>Monitor the download of packages until it is complete. When download is complete, the following message is displayed:
 
<li>Monitor the download of packages until it is complete. When download is complete, the following message is displayed:
  
Line 124: Line 142:
 
<ol start="1"><li>Verify there are no missing or failed packages:
 
<ol start="1"><li>Verify there are no missing or failed packages:
  
<source lang="none"># cat output/3rd_rpms_missing_L1.txt output/3rd_srpms_missing_L1.txt output/centos_rpms_missing_L1.txt output/centos_srpms_missing_L1.txt
+
<source lang="none">
# cat output/3rd_rpms_fail_move_L1.txt output/3rd_srpms_fail_move_L1.txt output/centos_rpms_fail_move_L1.txt output/centos_srpms_fail_move_L1.txt
+
# cat logs/*_missing_*.log
 +
# cat logs/*_failmove_*.log
 
</source></li>
 
</source></li>
<li><p>In case there are missing or failed ones due to network instability (or timeout), you should download them manually, to assure you get all RPMs listed in &quot;rpms_from_3rd_parties.lst&quot; and &quot;rpms_from_centos_repo.lst&quot;.</p></li>
 
<li><p>After all packages were succesfully downloaded, remove all i686 RPMs packages and change ''output'' directory ownership:</p>
 
  
<source lang="none"># find ./output -name "*.i686.rpm" | xargs rm -f
+
<li><p>In case there are missing or failed ones due to network instability (or timeout), you should download them manually, to assure you get all RPMs listed in '''rpms_3rdparties.lst'''/'''rpms_centos.lst'''/'''rpms_centos3rdparties.lst'''.</p></li>
# chown 751:751 -R ./output
+
 
</source></li></ol>
+
</li></ol>
 +
 
 +
=== Packages Structure ===
 +
<p>The following is a general overview of the packages structure that you will have after having downloaded the packages</p>
 +
 
 +
<source lang="none">
 +
/home/<user>/stx-tools/centos-mirror-tools/output
 +
└── stx-r1
 +
    └── CentOS
 +
        └── pike
 +
            ├── Binary
 +
            │   ├── EFI
 +
            │   ├── images
 +
            │   ├── isolinux
 +
            │   ├── LiveOS
 +
            │   ├── noarch
 +
            │   └── x86_64
 +
            ├── downloads
 +
            │   ├── integrity
 +
            │   └── puppet
 +
            └── Source
 +
</source></li>
  
 
=== Create CentOS Mirror Repository ===
 
=== Create CentOS Mirror Repository ===
  
<ol start="1"><li>From a console of the workstation, create a ''mirror/CentOS'' directory under your ''starlingx'' workspace directory:
+
Outside your Repository Docker container, in another terminal identified as "'''Two'''", run the following commands:
 +
 
 +
<ol start="1"><li>From terminal identified as "'''Two'''", create a ''mirror/CentOS'' directory under your ''starlingx'' workspace directory:
  
 
<source lang="sh">$ mkdir -p $HOME/starlingx/mirror/CentOS/
 
<source lang="sh">$ mkdir -p $HOME/starlingx/mirror/CentOS/
 
</source></li>
 
</source></li>
<li>Copy the built CentOS Mirror Repository built under ''&lt;stx-tools&gt;/centos-mirror-tool'' to the ''$HOME/starlingx/mirror/CentOS'' workspace directory.
+
<li>Copy the built CentOS Mirror Repository built under ''$HOME/stx-tools/centos-mirror-tool'' to the ''$HOME/starlingx/mirror/'' workspace directory.
 +
 
 +
<source lang="sh">$ cp -r $HOME/stx-tools/centos-mirror-tools/output/stx-r1/ $HOME/starlingx/mirror/CentOS/
 +
</source></li></ol>
 +
 
 +
==Create StarlingX Packages==
 +
===Setup Building Docker Container===
 +
 
 +
<ol start="1"><li>From terminal identified as "'''Two'''", create the workspace folder
 +
 
 +
<source lang="sh">$ mkdir -p $HOME/starlingx/workspace
 +
</source></li>
 +
 
 +
<li>Navigate to the '' $HOME/stx-tools'' project directory:
 +
 
 +
<source lang="sh">$ cd $HOME/stx-tools
 +
</source></li>
 +
 
 +
<li>Copy your git options to "toCopy" folder
 +
 
 +
<source lang="sh">$ cp ~/.gitconfig toCOPY
 +
</source></li>
 +
 
 +
<li>Create a ''&lt;localrc&gt;'' file
 +
 
 +
<source lang="sh">
 +
$ cat <<- EOF > localrc
 +
# tbuilder localrc
 +
MYUNAME=$USER
 +
PROJECT=starlingx
 +
HOST_PREFIX=$HOME/starlingx/workspace
 +
HOST_MIRROR_DIR=$HOME/starlingx/mirror
 +
EOF
 +
</source></li>
 +
 
 +
<li>If necessary you might have to set http/https proxy in your Dockerfile.centos73 before building the docker image.
 +
 
 +
<source lang="sh">ENV http_proxy  "http://your.actual_http_proxy.com:your_port"
 +
ENV https_proxy "https://your.actual_https_proxy.com:your_port"
 +
ENV ftp_proxy "http://your.actual_ftp_proxy.com:your_port"
 +
RUN echo "proxy=$http_proxy" >> /etc/yum.conf && \
 +
    echo -e "export http_proxy=$http_proxy\nexport https_proxy=$https_proxy\n\
 +
export ftp_proxy=$ftp_proxy" >> /root/.bashrc
 +
</source></li>
 +
 
 +
<li>Base container setup
 +
If you are running in fedora system, you will see  " .makeenv:88: *** missing separator.  Stop. " error, to continue :
 +
 
 +
* delete the functions define in the .makeenv ( module () { ... } )
 +
* delete the line 19 in the Makefile and ( NULL := $(shell bash -c "source buildrc ... ).
 +
 
 +
 
 +
<source lang="sh">$ make base-build
 +
</source></li>
 +
 
 +
<li>Build container setup
 +
 
 +
<source lang="sh">
 +
$ make build
 +
</source></li>
 +
 
 +
<li>Verify environment variables
 +
 
 +
<source lang="sh">$ bash tb.sh env
 +
</source></li>
 +
 
 +
<li>Build container run
 +
 
 +
<source lang="sh">$ bash tb.sh run
 +
</source></li>
 +
 
 +
<li>Execute the built container:
 +
 
 +
<source lang="sh">$ bash tb.sh exec
 +
</source></li></ol>
 +
 
 +
===Download Source Code Repositories===
 +
 
 +
<ol start="1"><li>From terminal identified as "Two", now inside the Building Docker container, Internal environment
 +
 
 +
<source lang="sh">$ eval $(ssh-agent)
 +
$ ssh-add
 +
</source></li>
 +
 
 +
<li>Repo init
 +
 
 +
<source lang="sh">
 +
$ cd $MY_REPO_ROOT_DIR
 +
$ repo init -u https://git.starlingx.io/stx-manifest -m default.xml
 +
</source></li>
 +
 
 +
<li>Repo sync
 +
 
 +
<source lang="sh">
 +
$ repo sync -j`nproc`
 +
</source></li>
 +
 
 +
<li> Tarballs Repository
 +
 
 +
<source lang="sh">
 +
$ ln -s /import/mirrors/CentOS/stx-r1/CentOS/pike/downloads/ $MY_REPO/stx/
 +
</source>
 +
Alternatively you can run the populate_downloads.sh script to copy the tarballs instead of using a symlink.
 +
 
 +
<source lang="sh">
 +
$ populate_downloads.sh /import/mirrors/CentOS/stx-r1/CentOS/pike/
 +
</source>
 +
 
 +
</li>
 +
 
 +
Outside the container
 +
 
 +
<li>From another terminal identified as "Three", Mirror Binaries
 +
 
 +
<source lang="sh">$ mkdir -p $HOME/starlingx/mirror/CentOS/tis-installer
 +
$ cp $HOME/starlingx/mirror/CentOS/stx-r1/CentOS/pike/Binary/images/pxeboot/initrd.img $HOME/starlingx/mirror/CentOS/tis-installer/initrd.img-stx-0.2
 +
$ cp $HOME/starlingx/mirror/CentOS/stx-r1/CentOS/pike/Binary/images/pxeboot/vmlinuz $HOME/starlingx/mirror/CentOS/tis-installer/vmlinuz-stx-0.2
 +
$ cp $HOME/starlingx/mirror/CentOS/stx-r1/CentOS/pike/Binary/LiveOS/squashfs.img $HOME/starlingx/mirror/CentOS/tis-installer/squashfs.img-stx-0.2
 +
</source></li></ol>
 +
 
 +
===Build Packages===
 +
 
 +
<ol start="1"><li>Back to the Building Docker container, terminal identified as "'''Two'''"</li>
 +
 
 +
<li><strong>Temporal!</strong> Build-Pkgs Errors
 +
Be prepared to have some missing / corrupted rpm and tarball packages generated during [[StarlingX/Developer_Guide#Build_the_CentOS_Mirror_Repository]] which will make the next step to fail, if that happens please download manually those missing / corrupted packages. <br>
 +
</li>
 +
 
 +
<li><strong>Update the symbolic links</strong>
 +
<source lang="sh">
 +
$ generate-cgcs-centos-repo.sh /import/mirrors/CentOS/stx-r1/CentOS/pike/
 +
</source></li>
  
<source lang="sh">$ cp -r stx-tools/centos-mirror-tools/output/stx-r1/ $HOME/starlingx/mirror/CentOS/
+
<li>Build-Pkgs
 +
 
 +
<source lang="sh">
 +
$ build-pkgs
 +
</source></li>
 +
 
 +
<li><strong>Optional!</strong> Generate-Cgcs-Tis-Repo <br>
 +
This step is optional but will improve performance on subsequent builds. The cgcs-tis-repo has the dependency information that sequences the build order; To generate or update the information the following command needs to be executed after building modified or new packages. <br>
 +
</li>
 +
 
 +
<source lang="sh">
 +
$ generate-cgcs-tis-repo
 +
</source>
 +
</ol>
 +
 
 +
==Build StarlingX ISO==
 +
 
 +
<ol start="1"><li>Build-Iso
 +
 
 +
<source lang="sh">$ build-iso
 
</source></li></ol>
 
</source></li></ol>
  
= Work in Progress... =
+
==Build installer==
 +
To get your StarlingX ISO ready to use, you will need to create the init files that will be used to boot the ISO as well to boot additional controllers and compute nodes. Note that this procedure only is needed in your first build and every time the kernel is upgraded.
 +
 
 +
Once you had run build-iso, run:
 +
 
 +
<source lang="sh">
 +
$ build-pkgs --installer
 +
</source>
 +
 
 +
This will build ''rpm'' and ''anaconda'' packages. Then run:
 +
 
 +
<source lang="sh">
 +
$ update-pxe-network-installer
 +
</source>
 +
 
 +
The ''update-pxe-network-installer'' covers the steps detailed in ''$MY_REPO/stx/stx-metal/installer/initrd/README''. This script will create three files on ''/localdisk/loadbuild/<user>/<project>/pxe-network-installer/output''.
 +
 
 +
<source>
 +
new-initrd.img
 +
new-squashfs.img
 +
new-vmlinuz
 +
</source>
 +
 
 +
Then, rename them to:
 +
 
 +
<source>
 +
initrd.img-stx-0.2
 +
squashfs.img-stx-0.2
 +
vmlinuz-stx-0.2
 +
</source>
 +
 
 +
There are two ways to use these files:
 +
 
 +
# Store the files in the ''/import/mirror/CentOS/tis-installer/'' folder  for future use.
 +
# Store it in an arbitrary location and modify the ''$MY_REPO/stx/stx-metal/installer/pxe-network-installer/centos/build_srpm.data'' file to point to these files.
 +
 
 +
Now, the ''pxe-network-installer'' package needs to be recreated and the ISO regenerated.
 +
 
 +
<source lang="sh">
 +
$ build-pkgs --clean pxe-network-installer
 +
$ build-pkgs pxe-network-installer
 +
$ build-iso
 +
</source>
 +
 
 +
Now your ISO should be able to boot.
 +
 
 +
=== Additional notes ===
 +
* In order to get the first boot working this complete procedure needs to be done. However, once the init files are created, these can be stored in a shared location where different developers can make use of them. Updating these files is not a frequent task and should be done whenever the kernel is upgraded.
 +
* StarlingX is in active development, so it is possible that in the future the '''0.2''' version will change to a more generic solution.
 +
 
 +
 
 +
==Build Avoidance==
 +
 
 +
===Purpose===
 +
 
 +
Greatly reduce build times after a repo sync for designers working within a regional office.  Starting from a new workspace, build-pkgs typically requires 3+ hours. Build avoidance typically reduces this step to ~20min
 +
 
 +
===Limitations===
 +
 
 +
* Little or no benefit for designers who refresh a pre-existing workspace at least daily. (download_mirror.sh, repo sync, generate-cgcs-centos-repo.sh, build-pkgs, build-iso).  In these cases an incremental build (reuse of same workspace without a 'build-pkgs --clean') is often just as efficient.
 +
* Not likely to be useful to solo designers, or teleworkers that wish to compile on there home computers.  Build avoidance downloads build artifacts from a reference build, and WAN speeds are generally to slow.
 +
 
 +
===Method (in brief)===
 +
 
 +
<ol start="1"><li> Reference Builds
 +
* A server in the regional office performs a regular (daily?), automated builds using existing methods.  Call these the reference builds.
 +
* The builds are timestamped, and preserved for some time.  (a few weeks)
 +
* A build CONTEXT is captured. This is a file produced by build-pkgs at location '$MY_WORKSPACE/CONTEXT'.  It is a bash script that can cd to each and every git and checkout the SHA that contributed to the build.
 +
* For each package built, a file shall capture he md5sums of all the source code inputs to the build of that package.  These files  are also produced by build-pkgs at location '$MY_WORKSPACE/<build-type>/rpmbuild/SOURCES/<pkg-name>/srpm_reference.md5'.
 +
* All these build products are accessible locally (e.g. a regional office) via rsync (other protocols can be added later)
 +
</li>
 +
 
 +
<li> Designers
 +
* Request a build avoidance build.  Recommended after you have just done a repo sync.  e.g.<source>
 +
repo sync
 +
generate-cgcs-centos-repo.sh
 +
populate_downloads.sh
 +
build-pkgs --build-avoidance
 +
</source>
 +
* Additional arguments, and/or environment variables, and/or a config file unique to the regional office, are used to specify a URL to the reference builds.
 +
** Using a config file to specify location of your reference build<source>
 +
mkdir -p $MY_REPO/local-build-data
 +
 
 +
cat <<- EOF > $MY_REPO/local-build-data/build_avoidance_source
 +
# Optional, these are already the default values.
 +
BUILD_AVOIDANCE_DATE_FORMAT="%Y%m%d"
 +
BUILD_AVOIDANCE_TIME_FORMAT="%H%M%S"
 +
BUILD_AVOIDANCE_DATE_TIME_DELIM="T"
 +
BUILD_AVOIDANCE_DATE_TIME_POSTFIX="Z"
 +
BUILD_AVOIDANCE_DATE_UTC=1
 +
BUILD_AVOIDANCE_FILE_TRANSFER="rsync"
 +
 
 +
# Required, unique values for each regional office
 +
BUILD_AVOIDANCE_USR="jenkins"
 +
BUILD_AVOIDANCE_HOST="stx-builder.mycompany.com"
 +
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
 +
EOF
 +
</source>
 +
** Using command line args to specify location of your reference build<source>
 +
build-pkgs --build-avoidance --build-avoidance-dir /localdisk/loadbuild/jenkins/StarlingX_Reference_Build --build-avoidance-host stx-builder.mycompany.com --build-avoidance-user jenkins
 +
</source>
 +
* Prior to your build attempt, you need to accept the host key.  This will prevent rsync failures on a yes/no prompt. (you should only have to do this once)<source>
 +
grep -q $BUILD_AVOIDANCE_HOST $HOME/.ssh/known_hosts
 +
if [ $? != 0 ]; then
 +
    ssh-keyscan $BUILD_AVOIDANCE_HOST >> $HOME/.ssh/known_hosts
 +
fi
 +
</source>
 +
* build-pkgs will:
 +
** From newest to oldest, scan the CONTEXTs of the various reference builds.  Select the first (most recent) context which satisfies the following requirement.  For every git, the SHA specified in the CONTEXT is present.
 +
** The selected context might be slightly out of date, but not by more than a day (assuming daily reference builds).
 +
** If the context has not been previously downloaded, then download it now.  Meaning download select portions of the reference build workspace into the designer's workspace.  This includes all the SRPMS, RPMS, MD5SUMS, and misc supporting files.  (~10 min over office LAN)
 +
** The designer may have additional commits not present in the reference build, or uncommitted changes.  Affected packages will identified by the differing md5sum's, and the package is re-built. (5+ min, depending on what packages have changed)
 +
</li>
 +
 
 +
* What if no valid reference build is found?  Then build-pkgs will fall back to a regular build.
 +
</ol>
 +
 
 +
===Reference builds===
 +
 
 +
* The regional office implements an automated build that pulls the latest StarlingX software and builds it on a regular basis.  e.g. a daily.  Perhaps implemented by Jenkins, cron, or similar tools.
 +
* Each build is saved to a unique directory, and preserved for a time that is reflective of how long a designer might be expected to work on a private branch without syncronizing with the master branch.  e.g. 2 weeks.
 +
* The MY_WORKSPACE directory for the build shall have a common root directory, and a leaf directory that is a sortable time stamp. Suggested format YYYYMMDDThhmmss.  e.g. <source lang="sh">$ sudo apt-get update
 +
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
 +
BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ')
 +
MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${BUILD_TIMESTAMP}
 +
</source>
 +
* Designers can access all build products over the internal network of the regional office.  The current prototype employs rsync. Other protocols that can efficiently share/copy/transfer large directories of content can be added as needed.
 +
 
 +
==== Advanced usage ====
 +
 
 +
Can the reference build itself use build avoidance? Yes<br>
 +
Can it reference itself?  Yes.<br>
 +
In either case we advise caution.  To protect against any possible 'divergence from reality', you should limit how many steps removed a build avoidance build is from a full build.<br>
 +
Suppose we want to implement a self referencing daily build, except that a full build occurs every Saturday.  To protect ourselves from a build failure on Saturday we also want a limit of 7 days since last full build.  You build script might look like this ...<source>
 +
...
 +
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
 +
BUILD_AVOIDANCE_HOST="stx-builder.mycompany.com"
 +
FULL_BUILD_DAY="Saturday"
 +
MAX_AGE_DAYS=7
 +
 
 +
LAST_FULL_BUILD_LINK="$BUILD_AVOIDANCE_DIR/latest_full_build"
 +
LAST_FULL_BUILD_DAY=""
 +
NOW_DAY=$(date -u "+%A")
 +
BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ')
 +
MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${BUILD_TIMESTAMP}
 +
 
 +
# update software
 +
repo init -u ${BUILD_REPO_URL} -b ${BUILD_BRANCH}
 +
repo sync --force-sync
 +
$MY_REPO_ROOT_DIR/stx-tools/toCOPY/generate-cgcs-centos-repo.sh
 +
$MY_REPO_ROOT_DIR/stx-tools/toCOPY/populate_downloads.sh
 +
 
 +
# User can optionally define BUILD_METHOD equal to one of 'FULL', 'AVOIDANCE', or 'AUTO'
 +
# Sanitize BUILD_METHOD
 +
if [ "$BUILD_METHOD" != "FULL" ] && [ "$BUILD_METHOD" != "AVOIDANCE" ]; then
 +
    BUILD_METHOD="AUTO"
 +
fi
 +
 
 +
# First build test
 +
if [ "$BUILD_METHOD" != "FULL" ] && [ ! -L $LAST_FULL_BUILD_LINK ]; then
 +
    echo "latest_full_build symlink missing, forcing full build"
 +
    BUILD_METHOD="FULL"
 +
fi
 +
 
 +
# Build day test
 +
if [ "$BUILD_METHOD" == "AUTO" ] && [ "$NOW_DAY" == "$FULL_BUILD_DAY" ]; then
 +
    echo "Today is $FULL_BUILD_DAY, forcing full build"
 +
    BUILD_METHOD="FULL"
 +
fi
 +
 
 +
# Build age test
 +
if [ "$BUILD_METHOD" != "FULL" ]; then
 +
    LAST_FULL_BUILD_DATE=$(basename $(readlink $LAST_FULL_BUILD_LINK) | cut -d '_' -f 1)
 +
    LAST_FULL_BUILD_DAY=$(date -d $LAST_FULL_BUILD_DATE "+%A")
 +
    AGE_SECS=$(( $(date "+%s") - $(date -d $LAST_FULL_BUILD_DATE "+%s") ))
 +
    AGE_DAYS=$(( $AGE_SECS/60/60/24 ))
 +
    if [ $AGE_DAYS -ge $MAX_AGE_DAYS ]; then
 +
        echo "Haven't had a full build in $AGE_DAYS days, forcing full build"
 +
        BUILD_METHOD="FULL"
 +
    fi
 +
    BUILD_METHOD="AVOIDANCE"
 +
fi
 +
 
 +
#Build it
 +
if [ "$BUILD_METHOD" == "FULL" ]; then
 +
    build-pkgs --no-build-avoidance
 +
else
 +
    build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER
 +
fi
 +
if [ $? -ne 0 ]; then
 +
    echo "Build failed in build-pkgs"
 +
    exit 1
 +
fi
 +
 
 +
build-iso
 +
if [ $? -ne 0 ]; then
 +
    echo "Build failed in build-iso"
 +
    exit 1
 +
fi
 +
 
 +
if [ "$BUILD_METHOD" == "FULL" ]; then
 +
    # A successful full build.  Set last full build symlink.
 +
    if [ -L $LAST_FULL_BUILD_LINK ]; then
 +
        rm -rf $LAST_FULL_BUILD_LINK
 +
    fi
 +
    ln -sf $MY_WORKSPACE $LAST_FULL_BUILD_LINK
 +
fi
 +
...
 +
</source>
 +
 
 +
One final wrinkle.
 +
We can ask build avoidance to preferentially use the full build day rather than the most recent build, as the reference point of the next avoidance build via use of  '--build-avoidance-day <day-name>'. e.g. substitute this line into the above.  <source>
 +
build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER --build-avoidance-day $FULL_BUILD_DAY
 +
 
 +
# or perhaps, with a bit more shuffling of the above script.
 +
 
 +
build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER --build-avoidance-day $LAST_FULL_BUILD_DAY
 +
</source>
 +
The advantage is that our build is never more than one step removed from a full build (assuming the full build was successful).<br>
 +
The disadvantage is that by end of week the reference build is getting rather old.  During active weeks, builds times might be approaching that of a full build.
 +
 
 +
-->

Latest revision as of 18:46, 16 July 2019


See the StarlingX Build Guide for the latest information regarding StarlingX development practices. This wiki page has been deprecated.