Jump to: navigation, search

Difference between revisions of "StarlingX/Developer Guide/Build System"

(build-srpms)
(Build ISO)
 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Structure ==
+
== Intro ==
 +
 
 +
== StarlingX Mirror ==
 +
 
 +
One of the features of StarlingX is to provide an ISO image ready to be installed in a deployment. This offers to the user a simple way to get everything configured and ready to use after installation. To get this done, StarlingX uses packages from upstream CentOS to provide a curated GNU/Linux distribution.
 +
 
 +
Four type of items are used by the StarlingX build system to create the ISO image:
 +
* RPMs : These are runtime or build dependencies for the system. Some of these packages are included in the ISO and other are just used for building.
 +
* Source RPMs : This is the source code of projects that are modified or extended by StarlingX. The build system will use these packages to create a new set of source rpms with patches applied to generate the final rpms.
 +
* Tarballs : Some componentes requires specific software that is not possible to find in a CentOS repository. The tarballs are used to generate source rpms with patches applied by StarlingX.
 +
* Binaries : These are files like `squashfs.img` or `initrd.img` used to create the ISO. The build system will modify the CentOS boot files to form the final ISO image.
 +
 
 +
In order to build StarlingX it is required to have all these files in place. To simplify this process a tool to download all these dependencies was created, however as upstream CentOS constantly changes is common to have issues downloading all the packages. This document tries to explain the process of downloading a mirror, verify that is correct and troubleshoot when is necessary.
 +
 
 +
== How `download_mirror.sh` works ==
 +
=== Creating the docker image ===
 +
=== Downloading the mirror ===
 +
=== Download all rpms ===
 +
==== Download tarballs ====
 +
=== Validating the mirror. ===
 +
==== GPG verification ====
 +
=== Troubleshooting ===
 +
==== Download packages manually ====
 +
 
 +
== StarlingX Build System ==
  
 
=== Environment Variables ===
 
=== Environment Variables ===
Line 545: Line 569:
  
 
=== Build ISO ===
 
=== Build ISO ===
 +
 +
http://lists.starlingx.io/pipermail/starlingx-discuss/2018-August/000745.html
  
 
<source lang="sh">
 
<source lang="sh">
Line 551: Line 577:
 
$ cat $MY_WORKSPACE/export/deps_detail.txt
 
$ cat $MY_WORKSPACE/export/deps_detail.txt
 
</source>
 
</source>
 +
 +
=== To Be Assigned ===
 +
 +
http://lists.starlingx.io/pipermail/starlingx-discuss/2018-July/000486.html
 +
 +
It is awesome!! Running “create_dependancy_cache.py” against the mirror (of RPMs) is to generate a new “dependancy-cache”, isn’t it?
 +
I had a successful parallel build (aka build-pkgs --parallel) inside the docker container.  ~1h45m on 24 core, 64G ram
 +
The prerequisite was a populated $MY_REPO/cgcs-tis-repo/dependancy-cache
 +
Currently we only generate the cache after the build in the 'generate-cgcs-tis-repo' step.  I'd like to see the cache stored in git and updated regularly by 'official' builds. 
 +
Note: The cache doesn't have to be perfect, so a cache that is out of date by a day or a week is still very useful.  build-pkgs/mockchain just needs a rough guide on build dependencies and potential dependency loops.

Latest revision as of 17:07, 20 August 2018

Intro

StarlingX Mirror

One of the features of StarlingX is to provide an ISO image ready to be installed in a deployment. This offers to the user a simple way to get everything configured and ready to use after installation. To get this done, StarlingX uses packages from upstream CentOS to provide a curated GNU/Linux distribution.

Four type of items are used by the StarlingX build system to create the ISO image:

  • RPMs : These are runtime or build dependencies for the system. Some of these packages are included in the ISO and other are just used for building.
  • Source RPMs : This is the source code of projects that are modified or extended by StarlingX. The build system will use these packages to create a new set of source rpms with patches applied to generate the final rpms.
  • Tarballs : Some componentes requires specific software that is not possible to find in a CentOS repository. The tarballs are used to generate source rpms with patches applied by StarlingX.
  • Binaries : These are files like `squashfs.img` or `initrd.img` used to create the ISO. The build system will modify the CentOS boot files to form the final ISO image.

In order to build StarlingX it is required to have all these files in place. To simplify this process a tool to download all these dependencies was created, however as upstream CentOS constantly changes is common to have issues downloading all the packages. This document tries to explain the process of downloading a mirror, verify that is correct and troubleshoot when is necessary.

How `download_mirror.sh` works

Creating the docker image

Downloading the mirror

Download all rpms

Download tarballs

Validating the mirror.

GPG verification

Troubleshooting

Download packages manually

StarlingX Build System

Environment Variables


[user@container ~]$ echo $MY_WORKSPACE
/localdisk/loadbuild/user/starlingx
[user@container ~]$ echo $MY_REPO
/localdisk/designer/user/starlingx/cgcs-root
[user@container ~]$ echo $MY_REPO_ROOT_DIR
/localdisk/designer/user/starlingx

Mirror


[user@container ~]$ tree -d -L 8 /import/
/import/
└── mirrors
    └── CentOS
        ├── stx-r1
        │   └── CentOS
        │       └── pike
        │           ├── Binary
        │           │   ├── EFI
        │           │   │   └── BOOT
        │           │   ├── images
        │           │   │   └── pxeboot
        │           │   ├── isolinux
        │           │   ├── LiveOS
        │           │   ├── noarch
        │           │   └── x86_64
        │           ├── downloads
        │           │   └── puppet
        │           └── Source
        └── tis-installer

18 directories
[user@container ~]$

Local Disk


[user@container ~]$ tree -d -L 2 /localdisk/
/localdisk/
├── designer
│   └── user
└── loadbuild
    ├── mock
    ├── mock-cache
    └── user

6 directories
[user@container ~]$

Designer

The 'designer' disk is for sourcecode.


[user@c6b5029b0279 ~]$ tree -d -L 4 /localdisk/designer/user/
/localdisk/designer/user/
└── starlingx
    ├── cgcs-root
    │   ├── build-data
    │   ├── build-tools
    │   │   ├── branching
    │   │   ├── build_guest
    │   │   ├── build_iso
    │   │   ├── build_minimal_iso
    │   │   ├── certificates
    │   │   ├── mirror_rebase
    │   │   ├── mk
    │   │   ├── repo_files
    │   │   └── signing
    │   ├── cgcs-centos-repo
    │   │   ├── Binary
    │   │   ├── Source
    │   ├── cgcs-tis-repo
    │   │   ├── Binary
    │   │   ├── dependancy-cache
    │   │   ├── downloads
    │   │   └── Source
    │   └── stx
    │       ├── downloads -> /import/mirrors/CentOS/stx-r1/CentOS/pike/downloads/
    │       ├── extras
    │       ├── git
    │       ├── middleware
    │       ├── recipes-bsp
    │       ├── stx-clients
    │       ├── stx-config
    │       ├── stx-fault
    │       ├── stx-gplv2
    │       ├── stx-gplv3
    │       ├── stx-gui
    │       ├── stx-ha
    │       ├── stx-integ
    │       ├── stx-metal
    │       ├── stx-nfv
    │       ├── stx-update
    │       └── stx-upstream
    └── stx-tools
        ├── centos-mirror-tools
        │   └── rpm-gpg-keys
        ├── release
        └── toCOPY
            └── cgcs_overlay

55 directories
[user@c6b5029b0279 ~]$

Loadbuild

The 'loadbuild' disk is for the build environment and the generated rpms and iso.


[user@c6b5029b0279 ~]$ tree -d -L 4 /localdisk/loadbuild/user/starlingx/
/localdisk/loadbuild/user/starlingx/
└── std
    ├── inputs
    │   └── stx
    │       ├── middleware
    │       ├── stx-clients
    │       └── stx-config
    ├── repo -> /localdisk/designer/user/starlingx/cgcs-root
    ├── rpmbuild
    │   ├── BUILD
    │   ├── RPMS
    │   ├── SOURCES
    │   ├── SPECS
    │   └── SRPMS
    └── srpm_assemble
        └── grubby
            └── rpmbuild

16 directories
[user@c6b5029b0279 ~]$

[user@c6b5029b0279 ~]$ tree -d -L 3 /localdisk/loadbuild/user/starlingx/
/localdisk/loadbuild/user/starlingx/
└── std
    ├── cache
    │   ├── b0
    │   ├── b1
    │   ├── b10
    │   ├── b11
    │   ├── b12
    │   ├── b13
    │   ├── b14
    │   ├── b15
    │   ├── b2
    │   ├── b3
    │   ├── b4
    │   ├── b5
    │   ├── b6
    │   ├── b7
    │   ├── b8
    │   └── b9
    ├── configs
    │   └── user-starlingx-tis-r5-pike-std
    ├── mock
    │   ├── b0
    │   ├── b1
    │   ├── b10
    │   ├── b11
    │   ├── b12
    │   ├── b13
    │   ├── b14
    │   ├── b15
    │   ├── b2
    │   ├── b3
    │   ├── b4
    │   ├── b5
    │   ├── b6
    │   ├── b7
    │   ├── b8
    │   └── b9
    ├── repo -> /localdisk/designer/user/starlingx/cgcs-root
    ├── results
    │   └── user-starlingx-tis-r5-pike-std
    ├── rpmbuild
    │   ├── BUILD
    │   ├── RPMS
    │   ├── SOURCES
    │   ├── SPECS
    │   └── SRPMS
    ├── srpm_assemble
    │   └── bash
    ├── tmp
    └── yum
        └── cache

51 directories
[user@c6b5029b0279 ~]$

[user@c6b5029b0279 ~]$ ls /localdisk/loadbuild/user/starlingx/std/inputs/stx/middleware/branding/recipes-control/wrs-branding/rpmbuild/
BUILD  BUILDROOT  RPMS  SOURCES  SPECS  SRPMS
[user@c6b5029b0279 ~]$

[user@c6b5029b0279 ~]$ ls /localdisk/loadbuild/user/starlingx/std/srpm_assemble/bash/rpmbuild/
BUILD  BUILDROOT  RPMS  SOURCES  SPECS  SRPMS
[user@c6b5029b0279 ~]$

[user@c6b5029b0279 ~]$ tree -d -L 3 /localdisk/loadbuild/user/starlingx/std/
/localdisk/loadbuild/user/starlingx/std/
├── cache
│   ├── b0
│   │   └── mock
│   ├── b1
│   │   └── mock
│   ├── b10
│   │   └── mock
│   ├── b11
│   │   └── mock
│   ├── b12
│   │   └── mock
│   ├── b13
│   │   └── mock
│   ├── b14
│   │   └── mock
│   ├── b15
│   │   └── mock
│   ├── b2
│   │   └── mock
│   ├── b3
│   │   └── mock
│   ├── b4
│   │   └── mock
│   ├── b5
│   │   └── mock
│   ├── b6
│   │   └── mock
│   ├── b7
│   │   └── mock
│   ├── b8
│   │   └── mock
│   └── b9
│       └── mock
├── configs
│   └── user-starlingx-tis-r5-pike-std
├── mock
│   ├── b0
│   │   ├── mock
│   │   └── root
│   ├── b1
│   ├── b10
│   ├── b11
│   ├── b12
│   ├── b13
│   ├── b14
│   ├── b15
│   ├── b2
│   ├── b3
│   ├── b4
│   ├── b5
│   ├── b6
│   ├── b7
│   ├── b8
│   └── b9
├── repo -> /localdisk/designer/user/starlingx/cgcs-root
├── result
├── results
│   └── user-starlingx-tis-r5-pike-std
│       ├── bash-4.2.46-29.el7_4.tis.3
│       └── repodata
├── rpmbuild
│   ├── BUILD
│   ├── RPMS
│   │   └── repodata
│   ├── SOURCES
│   │   └── bash-4.2.46-29.el7_4.tis.3.src.rpm
│   ├── SPECS
│   │   └── bash-4.2.46-29.el7_4.tis.3.src.rpm
│   └── SRPMS
│       └── repodata
├── srpm_assemble
│   └── bash
│       └── rpmbuild
├── tmp
└── yum
    └── cache

76 directories
[user@c6b5029b0279 ~]$

[user@c6b5029b0279 ~]$ tree -d -L 4 /localdisk/loadbuild/user/starlingx/
/localdisk/loadbuild/user/starlingx/
└── std
    ├── inputs
    │   └── stx
    │       ├── middleware
    │       ├── stx-clients
    │       ├── stx-config
    │       ├── stx-fault
    │       ├── stx-gplv2
    │       ├── stx-ha
    │       └── stx-integ
    ├── repo -> /localdisk/designer/user/starlingx/cgcs-root
    ├── rpmbuild
    │   ├── BUILD
    │   ├── RPMS
    │   ├── SOURCES
    │   ├── SPECS
    │   └── SRPMS
    └── srpm_assemble
        ├── bash
        │   └── rpmbuild
        ├── centos-release
        │   └── rpmbuild
        ├── crontabs
        │   └── rpmbuild
        ├── dhcp
        │   └── rpmbuild
        ├── dnsmasq
        │   └── rpmbuild
        ├── e2fsprogs
        │   └── rpmbuild
        ├── facter
        │   └── rpmbuild
        ├── grub2
        │   └── rpmbuild                                                                                                                             [12/1917]
        ├── grubby
        │   └── rpmbuild
        ├── haproxy
        │   └── rpmbuild
        ├── initscripts
        │   └── rpmbuild
        ├── iptables
        │   └── rpmbuild
        ├── iscsi-initiator-utils
        │   └── rpmbuild
        ├── lshell
        │   └── rpmbuild
        ├── netpbm
        │   └── rpmbuild
        ├── net-tools
        │   └── rpmbuild
        ├── nfs-utils
        │   └── rpmbuild
        ├── nss-pam-ldapd
        │   └── rpmbuild
        ├── openssh
        │   └── rpmbuild
        ├── parted
        │   └── rpmbuild
        ├── python
        │   └── rpmbuild
        ├── python-django
        │   └── rpmbuild
        ├── python-keyring
        │   └── rpmbuild
        ├── python-psycopg2
        │   └── rpmbuild
        ├── resource-agents
        │   └── rpmbuild
        ├── rsync
        │   └── rpmbuild
        ├── seabios
        │   └── rpmbuild
        ├── setup
        │   └── rpmbuild
        ├── util-linux
        │   └── rpmbuild
        └── vim
            └── rpmbuild

78 directories
[user@c6b5029b0279 ~]$

[user@container ~]$ tree -d -L 4 /localdisk/loadbuild/
/localdisk/loadbuild/
├── mock
│   ├── user-starlingx-tis-r5-pike-installer -> /localdisk/loadbuild/user/starlingx/installer
│   ├── user-starlingx-tis-r5-pike-installer.b0 -> /localdisk/loadbuild/user/starlingx/installer
│   ├── user-starlingx-tis-r5-pike-rt -> /localdisk/loadbuild/user/starlingx/rt
│   ├── user-starlingx-tis-r5-pike-rt.b0 -> /localdisk/loadbuild/user/starlingx/rt
│   ├── user-starlingx-tis-r5-pike-rt.b1 -> /localdisk/loadbuild/user/starlingx/rt
│   ├── user-starlingx-tis-r5-pike-rt.b2 -> /localdisk/loadbuild/user/starlingx/rt
│   ├── user-starlingx-tis-r5-pike-rt.b3 -> /localdisk/loadbuild/user/starlingx/rt
│   ├── user-starlingx-tis-r5-pike-std -> /localdisk/loadbuild/user/starlingx/std
│   ├── user-starlingx-tis-r5-pike-std.b0 -> /localdisk/loadbuild/user/starlingx/std
│   ├── user-starlingx-tis-r5-pike-std.b1 -> /localdisk/loadbuild/user/starlingx/std
│   ├── user-starlingx-tis-r5-pike-std.b2 -> /localdisk/loadbuild/user/starlingx/std
│   └── user-starlingx-tis-r5-pike-std.b3 -> /localdisk/loadbuild/user/starlingx/std
├── mock-cache
│   ├── user-starlingx-tis-r5-pike-installer -> /localdisk/loadbuild/user/starlingx/installer/cache
│   ├── user-starlingx-tis-r5-pike-installer.b0 -> /localdisk/loadbuild/user/starlingx/installer/cache
│   ├── user-starlingx-tis-r5-pike-rt -> /localdisk/loadbuild/user/starlingx/rt/cache
│   ├── user-starlingx-tis-r5-pike-rt.b0 -> /localdisk/loadbuild/user/starlingx/rt/cache
│   ├── user-starlingx-tis-r5-pike-rt.b1 -> /localdisk/loadbuild/user/starlingx/rt/cache
│   ├── user-starlingx-tis-r5-pike-rt.b2 -> /localdisk/loadbuild/user/starlingx/rt/cache
│   ├── user-starlingx-tis-r5-pike-rt.b3 -> /localdisk/loadbuild/user/starlingx/rt/cache
│   ├── user-starlingx-tis-r5-pike-std -> /localdisk/loadbuild/user/starlingx/std/cache
│   ├── user-starlingx-tis-r5-pike-std.b0 -> /localdisk/loadbuild/user/starlingx/std/cache
│   ├── user-starlingx-tis-r5-pike-std.b1 -> /localdisk/loadbuild/user/starlingx/std/cache
│   ├── user-starlingx-tis-r5-pike-std.b2 -> /localdisk/loadbuild/user/starlingx/std/cache
│   └── user-starlingx-tis-r5-pike-std.b3 -> /localdisk/loadbuild/user/starlingx/std/cache
└── user
    └── starlingx
        ├── export
        │   ├── dist
        │   ├── extra_cfgs
        │   ├── kickstarts
        │   ├── kickstart.work
        │   └── mirrors
        ├── installer
        │   ├── cache
        │   ├── configs
        │   ├── inputs
        │   ├── mock
        │   ├── repo -> /localdisk/designer/user/starlingx/cgcs-root
        │   ├── result
        │   ├── results
        │   ├── rpmbuild
        │   ├── srpm_assemble
        │   ├── tmp
        │   └── yum
        ├── mock-cache
        ├── rt
        │   ├── cache
        │   ├── configs
        │   ├── inputs
        │   ├── mock
        │   ├── repo -> /localdisk/designer/user/starlingx/cgcs-root
        │   ├── result
        │   ├── results
        │   ├── rpmbuild
        │   ├── srpm_assemble
        │   ├── tmp
        │   └── yum
        ├── std
        │   ├── cache
        │   ├── configs
        │   ├── inputs
        │   ├── mock
        │   ├── mock-user-10530
        │   ├── mock-user-17746
        │   ├── mock-user-26749
        │   ├── mock-user-30169
        │   ├── mock-user-6012
        │   ├── mock-user-9458
        │   ├── repo -> /localdisk/designer/user/starlingx/cgcs-root
        │   ├── result
        │   ├── results
        │   ├── rpmbuild
        │   ├── srpm_assemble
        │   ├── tmp
        │   └── yum
        ├── tmp
        └── yum
            └── cache
            └── cache

80 directories
[user@container ~]$

Build

  • build-pkgs-serial
  • build-pkgs-parallel
  • std build
  • rt build
  • installer build
  • build-srpms-serial
  • build-rpms-serial
  • build-srpms-parallel
  • build-rpms-parallel

build-srpms


      # we've found a file (ex centos/srpm_path) which lists a path to a source
      # RPM file
      #
      # The specified file can be of the form
      #
      # repo:path/to/file.src.rpm
      # mirror:path/to/file.src.rpm
      # /path/to/file.rpm
      # path/to/file.rpm
      #
      # If "repo:" is specified, then we search for the file relative to
      # $REPO_DOWNLOADS_ROOT (i.e. a path to the file in a "downloads subgit)
      #
      # If "mirror:" is specified, then we search for the file relateive to
      # $MIRROR_ROOT
      #
      # If "3rd_party:" is specified, then we search for the file relateive to
      # $THIRD_PARTY_ROOT
      #
      # An absolute path is parsed as an absolute path (mainly intended for
      # developer/experimental use without checking in files or messing with
      # your git repos)
      #
      # A lack of prefix (relative path name) is interpretted as "mirror:"
      # (legacy support for existing packages)
      #
      # Other prefixes (file:, http:, whatever:)are unsupported at this time

cgcs-root/build-tools/build-rpms-parallel:# adding dependant packages
cgcs-root/build-tools/build-rpms-parallel:   echo "adding dependant packages"
cgcs-root/build-tools/build-rpms-serial:# adding dependant packages
cgcs-root/build-tools/build-rpms-serial:   echo "adding dependant packages"

Debug

Mock

For serial build use:

$ export MY_BUILD_ENVIRONMENT=$USER-$PROJECT-$SRC_BUILD_ENVIRONMENT
$ export MY_DEBUG_BUILD_CFG_STD=$MY_WORKSPACE/std/configs/${MY_BUILD_ENVIRONMENT}-std/${MY_BUILD_ENVIRONMENT}-std.cfg

For parallel build use:

$ export MY_BUILD_ENVIRONMENT=$USER-$PROJECT-$SRC_BUILD_ENVIRONMENT
$ export MY_DEBUG_BUILD_CFG_STD=$MY_WORKSPACE/std/configs/${MY_BUILD_ENVIRONMENT}-std/${MY_BUILD_ENVIRONMENT}-std.b0.cfg

Enter your mock

$ mock -r $MY_DEBUG_BUILD_CFG_STD --shell

Build ISO

http://lists.starlingx.io/pipermail/starlingx-discuss/2018-August/000745.html

$ ls $MY_WORKSPACE/export/
cgts_deps_tmp.txt  deps_detail.txt  deps.txt  dist  last_test.txt  local.txt  mirrors  mounts_used.log  warnings.txt  yum.conf
$ cat $MY_WORKSPACE/export/deps_detail.txt

To Be Assigned

http://lists.starlingx.io/pipermail/starlingx-discuss/2018-July/000486.html

It is awesome!! Running “create_dependancy_cache.py” against the mirror (of RPMs) is to generate a new “dependancy-cache”, isn’t it? I had a successful parallel build (aka build-pkgs --parallel) inside the docker container. ~1h45m on 24 core, 64G ram The prerequisite was a populated $MY_REPO/cgcs-tis-repo/dependancy-cache Currently we only generate the cache after the build in the 'generate-cgcs-tis-repo' step. I'd like to see the cache stored in git and updated regularly by 'official' builds. Note: The cache doesn't have to be perfect, so a cache that is out of date by a day or a week is still very useful. build-pkgs/mockchain just needs a rough guide on build dependencies and potential dependency loops.