Difference between revisions of "StarlingX/Developer Guide/Build System"
(→Structure) |
|||
Line 1: | Line 1: | ||
− | == | + | # StarlingX Mirror |
+ | |||
+ | One of the features of StarlingX is to provide an ISO image ready to be installed in a deployment. This offers to the user a simple way to get everything configured and ready to use after installation. To get this done, StarlingX uses packages from upstream CentOS to provide a curated GNU/Linux distribution. | ||
+ | |||
+ | Four type of items are used by the StarlingX build system to create the ISO image: | ||
+ | * RPMs : These are runtime or build dependencies for the system. Some of these packages are included in the ISO and other are just used for building. | ||
+ | * Source RPMs : This is the source code of projects that are modified or extended by StarlingX. The build system will use these packages to create a new set of source rpms with patches applied to generate the final rpms. | ||
+ | * Tarballs : Some componentes requires specific software that is not possible to find in a CentOS repository. The tarballs are used to generate source rpms with patches applied by StarlingX. | ||
+ | * Binaries : These are files like `squashfs.img` or `initrd.img` used to create the ISO. The build system will modify the CentOS boot files to form the final ISO image. | ||
+ | |||
+ | In order to build StarlingX it is required to have all these files in place. To simplify this process a tool to download all these dependencies was created, however as upstream CentOS constantly changes is common to have issues downloading all the packages. This document tries to explain the process of downloading a mirror, verify that is correct and troubleshoot when is necessary. | ||
+ | |||
+ | == How `download_mirror.sh` works == | ||
+ | === Creating the docker image === | ||
+ | === Downloading the mirror === | ||
+ | === Download all rpms === | ||
+ | ==== Download tarballs ==== | ||
+ | === Validating the mirror. === | ||
+ | ==== GPG verification ==== | ||
+ | === Troubleshooting === | ||
+ | ==== Download packages manually ==== | ||
+ | |||
+ | == Build System == | ||
=== Environment Variables === | === Environment Variables === |
Revision as of 20:42, 15 August 2018
- StarlingX Mirror
One of the features of StarlingX is to provide an ISO image ready to be installed in a deployment. This offers to the user a simple way to get everything configured and ready to use after installation. To get this done, StarlingX uses packages from upstream CentOS to provide a curated GNU/Linux distribution.
Four type of items are used by the StarlingX build system to create the ISO image:
- RPMs : These are runtime or build dependencies for the system. Some of these packages are included in the ISO and other are just used for building.
- Source RPMs : This is the source code of projects that are modified or extended by StarlingX. The build system will use these packages to create a new set of source rpms with patches applied to generate the final rpms.
- Tarballs : Some componentes requires specific software that is not possible to find in a CentOS repository. The tarballs are used to generate source rpms with patches applied by StarlingX.
- Binaries : These are files like `squashfs.img` or `initrd.img` used to create the ISO. The build system will modify the CentOS boot files to form the final ISO image.
In order to build StarlingX it is required to have all these files in place. To simplify this process a tool to download all these dependencies was created, however as upstream CentOS constantly changes is common to have issues downloading all the packages. This document tries to explain the process of downloading a mirror, verify that is correct and troubleshoot when is necessary.
How `download_mirror.sh` works
Creating the docker image
Downloading the mirror
Download all rpms
Download tarballs
Validating the mirror.
GPG verification
Troubleshooting
Download packages manually
Build System
Environment Variables
[user@container ~]$ echo $MY_WORKSPACE
/localdisk/loadbuild/user/starlingx
[user@container ~]$ echo $MY_REPO
/localdisk/designer/user/starlingx/cgcs-root
[user@container ~]$ echo $MY_REPO_ROOT_DIR
/localdisk/designer/user/starlingx
Mirror
[user@container ~]$ tree -d -L 8 /import/
/import/
└── mirrors
└── CentOS
├── stx-r1
│ └── CentOS
│ └── pike
│ ├── Binary
│ │ ├── EFI
│ │ │ └── BOOT
│ │ ├── images
│ │ │ └── pxeboot
│ │ ├── isolinux
│ │ ├── LiveOS
│ │ ├── noarch
│ │ └── x86_64
│ ├── downloads
│ │ └── puppet
│ └── Source
└── tis-installer
18 directories
[user@container ~]$
Local Disk
[user@container ~]$ tree -d -L 2 /localdisk/
/localdisk/
├── designer
│ └── user
└── loadbuild
├── mock
├── mock-cache
└── user
6 directories
[user@container ~]$
Designer
The 'designer' disk is for sourcecode.
[user@c6b5029b0279 ~]$ tree -d -L 4 /localdisk/designer/user/
/localdisk/designer/user/
└── starlingx
├── cgcs-root
│ ├── build-data
│ ├── build-tools
│ │ ├── branching
│ │ ├── build_guest
│ │ ├── build_iso
│ │ ├── build_minimal_iso
│ │ ├── certificates
│ │ ├── mirror_rebase
│ │ ├── mk
│ │ ├── repo_files
│ │ └── signing
│ ├── cgcs-centos-repo
│ │ ├── Binary
│ │ ├── Source
│ ├── cgcs-tis-repo
│ │ ├── Binary
│ │ ├── dependancy-cache
│ │ ├── downloads
│ │ └── Source
│ └── stx
│ ├── downloads -> /import/mirrors/CentOS/stx-r1/CentOS/pike/downloads/
│ ├── extras
│ ├── git
│ ├── middleware
│ ├── recipes-bsp
│ ├── stx-clients
│ ├── stx-config
│ ├── stx-fault
│ ├── stx-gplv2
│ ├── stx-gplv3
│ ├── stx-gui
│ ├── stx-ha
│ ├── stx-integ
│ ├── stx-metal
│ ├── stx-nfv
│ ├── stx-update
│ └── stx-upstream
└── stx-tools
├── centos-mirror-tools
│ └── rpm-gpg-keys
├── release
└── toCOPY
└── cgcs_overlay
55 directories
[user@c6b5029b0279 ~]$
Loadbuild
The 'loadbuild' disk is for the build environment and the generated rpms and iso.
[user@c6b5029b0279 ~]$ tree -d -L 4 /localdisk/loadbuild/user/starlingx/
/localdisk/loadbuild/user/starlingx/
└── std
├── inputs
│ └── stx
│ ├── middleware
│ ├── stx-clients
│ └── stx-config
├── repo -> /localdisk/designer/user/starlingx/cgcs-root
├── rpmbuild
│ ├── BUILD
│ ├── RPMS
│ ├── SOURCES
│ ├── SPECS
│ └── SRPMS
└── srpm_assemble
└── grubby
└── rpmbuild
16 directories
[user@c6b5029b0279 ~]$
[user@c6b5029b0279 ~]$ tree -d -L 3 /localdisk/loadbuild/user/starlingx/
/localdisk/loadbuild/user/starlingx/
└── std
├── cache
│ ├── b0
│ ├── b1
│ ├── b10
│ ├── b11
│ ├── b12
│ ├── b13
│ ├── b14
│ ├── b15
│ ├── b2
│ ├── b3
│ ├── b4
│ ├── b5
│ ├── b6
│ ├── b7
│ ├── b8
│ └── b9
├── configs
│ └── user-starlingx-tis-r5-pike-std
├── mock
│ ├── b0
│ ├── b1
│ ├── b10
│ ├── b11
│ ├── b12
│ ├── b13
│ ├── b14
│ ├── b15
│ ├── b2
│ ├── b3
│ ├── b4
│ ├── b5
│ ├── b6
│ ├── b7
│ ├── b8
│ └── b9
├── repo -> /localdisk/designer/user/starlingx/cgcs-root
├── results
│ └── user-starlingx-tis-r5-pike-std
├── rpmbuild
│ ├── BUILD
│ ├── RPMS
│ ├── SOURCES
│ ├── SPECS
│ └── SRPMS
├── srpm_assemble
│ └── bash
├── tmp
└── yum
└── cache
51 directories
[user@c6b5029b0279 ~]$
[user@c6b5029b0279 ~]$ ls /localdisk/loadbuild/user/starlingx/std/inputs/stx/middleware/branding/recipes-control/wrs-branding/rpmbuild/
BUILD BUILDROOT RPMS SOURCES SPECS SRPMS
[user@c6b5029b0279 ~]$
[user@c6b5029b0279 ~]$ ls /localdisk/loadbuild/user/starlingx/std/srpm_assemble/bash/rpmbuild/
BUILD BUILDROOT RPMS SOURCES SPECS SRPMS
[user@c6b5029b0279 ~]$
[user@c6b5029b0279 ~]$ tree -d -L 3 /localdisk/loadbuild/user/starlingx/std/
/localdisk/loadbuild/user/starlingx/std/
├── cache
│ ├── b0
│ │ └── mock
│ ├── b1
│ │ └── mock
│ ├── b10
│ │ └── mock
│ ├── b11
│ │ └── mock
│ ├── b12
│ │ └── mock
│ ├── b13
│ │ └── mock
│ ├── b14
│ │ └── mock
│ ├── b15
│ │ └── mock
│ ├── b2
│ │ └── mock
│ ├── b3
│ │ └── mock
│ ├── b4
│ │ └── mock
│ ├── b5
│ │ └── mock
│ ├── b6
│ │ └── mock
│ ├── b7
│ │ └── mock
│ ├── b8
│ │ └── mock
│ └── b9
│ └── mock
├── configs
│ └── user-starlingx-tis-r5-pike-std
├── mock
│ ├── b0
│ │ ├── mock
│ │ └── root
│ ├── b1
│ ├── b10
│ ├── b11
│ ├── b12
│ ├── b13
│ ├── b14
│ ├── b15
│ ├── b2
│ ├── b3
│ ├── b4
│ ├── b5
│ ├── b6
│ ├── b7
│ ├── b8
│ └── b9
├── repo -> /localdisk/designer/user/starlingx/cgcs-root
├── result
├── results
│ └── user-starlingx-tis-r5-pike-std
│ ├── bash-4.2.46-29.el7_4.tis.3
│ └── repodata
├── rpmbuild
│ ├── BUILD
│ ├── RPMS
│ │ └── repodata
│ ├── SOURCES
│ │ └── bash-4.2.46-29.el7_4.tis.3.src.rpm
│ ├── SPECS
│ │ └── bash-4.2.46-29.el7_4.tis.3.src.rpm
│ └── SRPMS
│ └── repodata
├── srpm_assemble
│ └── bash
│ └── rpmbuild
├── tmp
└── yum
└── cache
76 directories
[user@c6b5029b0279 ~]$
[user@c6b5029b0279 ~]$ tree -d -L 4 /localdisk/loadbuild/user/starlingx/
/localdisk/loadbuild/user/starlingx/
└── std
├── inputs
│ └── stx
│ ├── middleware
│ ├── stx-clients
│ ├── stx-config
│ ├── stx-fault
│ ├── stx-gplv2
│ ├── stx-ha
│ └── stx-integ
├── repo -> /localdisk/designer/user/starlingx/cgcs-root
├── rpmbuild
│ ├── BUILD
│ ├── RPMS
│ ├── SOURCES
│ ├── SPECS
│ └── SRPMS
└── srpm_assemble
├── bash
│ └── rpmbuild
├── centos-release
│ └── rpmbuild
├── crontabs
│ └── rpmbuild
├── dhcp
│ └── rpmbuild
├── dnsmasq
│ └── rpmbuild
├── e2fsprogs
│ └── rpmbuild
├── facter
│ └── rpmbuild
├── grub2
│ └── rpmbuild [12/1917]
├── grubby
│ └── rpmbuild
├── haproxy
│ └── rpmbuild
├── initscripts
│ └── rpmbuild
├── iptables
│ └── rpmbuild
├── iscsi-initiator-utils
│ └── rpmbuild
├── lshell
│ └── rpmbuild
├── netpbm
│ └── rpmbuild
├── net-tools
│ └── rpmbuild
├── nfs-utils
│ └── rpmbuild
├── nss-pam-ldapd
│ └── rpmbuild
├── openssh
│ └── rpmbuild
├── parted
│ └── rpmbuild
├── python
│ └── rpmbuild
├── python-django
│ └── rpmbuild
├── python-keyring
│ └── rpmbuild
├── python-psycopg2
│ └── rpmbuild
├── resource-agents
│ └── rpmbuild
├── rsync
│ └── rpmbuild
├── seabios
│ └── rpmbuild
├── setup
│ └── rpmbuild
├── util-linux
│ └── rpmbuild
└── vim
└── rpmbuild
78 directories
[user@c6b5029b0279 ~]$
[user@container ~]$ tree -d -L 4 /localdisk/loadbuild/
/localdisk/loadbuild/
├── mock
│ ├── user-starlingx-tis-r5-pike-installer -> /localdisk/loadbuild/user/starlingx/installer
│ ├── user-starlingx-tis-r5-pike-installer.b0 -> /localdisk/loadbuild/user/starlingx/installer
│ ├── user-starlingx-tis-r5-pike-rt -> /localdisk/loadbuild/user/starlingx/rt
│ ├── user-starlingx-tis-r5-pike-rt.b0 -> /localdisk/loadbuild/user/starlingx/rt
│ ├── user-starlingx-tis-r5-pike-rt.b1 -> /localdisk/loadbuild/user/starlingx/rt
│ ├── user-starlingx-tis-r5-pike-rt.b2 -> /localdisk/loadbuild/user/starlingx/rt
│ ├── user-starlingx-tis-r5-pike-rt.b3 -> /localdisk/loadbuild/user/starlingx/rt
│ ├── user-starlingx-tis-r5-pike-std -> /localdisk/loadbuild/user/starlingx/std
│ ├── user-starlingx-tis-r5-pike-std.b0 -> /localdisk/loadbuild/user/starlingx/std
│ ├── user-starlingx-tis-r5-pike-std.b1 -> /localdisk/loadbuild/user/starlingx/std
│ ├── user-starlingx-tis-r5-pike-std.b2 -> /localdisk/loadbuild/user/starlingx/std
│ └── user-starlingx-tis-r5-pike-std.b3 -> /localdisk/loadbuild/user/starlingx/std
├── mock-cache
│ ├── user-starlingx-tis-r5-pike-installer -> /localdisk/loadbuild/user/starlingx/installer/cache
│ ├── user-starlingx-tis-r5-pike-installer.b0 -> /localdisk/loadbuild/user/starlingx/installer/cache
│ ├── user-starlingx-tis-r5-pike-rt -> /localdisk/loadbuild/user/starlingx/rt/cache
│ ├── user-starlingx-tis-r5-pike-rt.b0 -> /localdisk/loadbuild/user/starlingx/rt/cache
│ ├── user-starlingx-tis-r5-pike-rt.b1 -> /localdisk/loadbuild/user/starlingx/rt/cache
│ ├── user-starlingx-tis-r5-pike-rt.b2 -> /localdisk/loadbuild/user/starlingx/rt/cache
│ ├── user-starlingx-tis-r5-pike-rt.b3 -> /localdisk/loadbuild/user/starlingx/rt/cache
│ ├── user-starlingx-tis-r5-pike-std -> /localdisk/loadbuild/user/starlingx/std/cache
│ ├── user-starlingx-tis-r5-pike-std.b0 -> /localdisk/loadbuild/user/starlingx/std/cache
│ ├── user-starlingx-tis-r5-pike-std.b1 -> /localdisk/loadbuild/user/starlingx/std/cache
│ ├── user-starlingx-tis-r5-pike-std.b2 -> /localdisk/loadbuild/user/starlingx/std/cache
│ └── user-starlingx-tis-r5-pike-std.b3 -> /localdisk/loadbuild/user/starlingx/std/cache
└── user
└── starlingx
├── export
│ ├── dist
│ ├── extra_cfgs
│ ├── kickstarts
│ ├── kickstart.work
│ └── mirrors
├── installer
│ ├── cache
│ ├── configs
│ ├── inputs
│ ├── mock
│ ├── repo -> /localdisk/designer/user/starlingx/cgcs-root
│ ├── result
│ ├── results
│ ├── rpmbuild
│ ├── srpm_assemble
│ ├── tmp
│ └── yum
├── mock-cache
├── rt
│ ├── cache
│ ├── configs
│ ├── inputs
│ ├── mock
│ ├── repo -> /localdisk/designer/user/starlingx/cgcs-root
│ ├── result
│ ├── results
│ ├── rpmbuild
│ ├── srpm_assemble
│ ├── tmp
│ └── yum
├── std
│ ├── cache
│ ├── configs
│ ├── inputs
│ ├── mock
│ ├── mock-user-10530
│ ├── mock-user-17746
│ ├── mock-user-26749
│ ├── mock-user-30169
│ ├── mock-user-6012
│ ├── mock-user-9458
│ ├── repo -> /localdisk/designer/user/starlingx/cgcs-root
│ ├── result
│ ├── results
│ ├── rpmbuild
│ ├── srpm_assemble
│ ├── tmp
│ └── yum
├── tmp
└── yum
└── cache
└── cache
80 directories
[user@container ~]$
Build
- build-pkgs-serial
- build-pkgs-parallel
- std build
- rt build
- installer build
- build-srpms-serial
- build-rpms-serial
- build-srpms-parallel
- build-rpms-parallel
build-srpms
# we've found a file (ex centos/srpm_path) which lists a path to a source
# RPM file
#
# The specified file can be of the form
#
# repo:path/to/file.src.rpm
# mirror:path/to/file.src.rpm
# /path/to/file.rpm
# path/to/file.rpm
#
# If "repo:" is specified, then we search for the file relative to
# $REPO_DOWNLOADS_ROOT (i.e. a path to the file in a "downloads subgit)
#
# If "mirror:" is specified, then we search for the file relateive to
# $MIRROR_ROOT
#
# If "3rd_party:" is specified, then we search for the file relateive to
# $THIRD_PARTY_ROOT
#
# An absolute path is parsed as an absolute path (mainly intended for
# developer/experimental use without checking in files or messing with
# your git repos)
#
# A lack of prefix (relative path name) is interpretted as "mirror:"
# (legacy support for existing packages)
#
# Other prefixes (file:, http:, whatever:)are unsupported at this time
cgcs-root/build-tools/build-rpms-parallel:# adding dependant packages
cgcs-root/build-tools/build-rpms-parallel: echo "adding dependant packages"
cgcs-root/build-tools/build-rpms-serial:# adding dependant packages
cgcs-root/build-tools/build-rpms-serial: echo "adding dependant packages"
Debug
Mock
For serial build use:
$ export MY_BUILD_ENVIRONMENT=$USER-$PROJECT-$SRC_BUILD_ENVIRONMENT
$ export MY_DEBUG_BUILD_CFG_STD=$MY_WORKSPACE/std/configs/${MY_BUILD_ENVIRONMENT}-std/${MY_BUILD_ENVIRONMENT}-std.cfg
For parallel build use:
$ export MY_BUILD_ENVIRONMENT=$USER-$PROJECT-$SRC_BUILD_ENVIRONMENT
$ export MY_DEBUG_BUILD_CFG_STD=$MY_WORKSPACE/std/configs/${MY_BUILD_ENVIRONMENT}-std/${MY_BUILD_ENVIRONMENT}-std.b0.cfg
Enter your mock
$ mock -r $MY_DEBUG_BUILD_CFG_STD --shell
Build ISO
$ ls $MY_WORKSPACE/export/
cgts_deps_tmp.txt deps_detail.txt deps.txt dist last_test.txt local.txt mirrors mounts_used.log warnings.txt yum.conf
$ cat $MY_WORKSPACE/export/deps_detail.txt
To Be Assigned
http://lists.starlingx.io/pipermail/starlingx-discuss/2018-July/000486.html
It is awesome!! Running “create_dependancy_cache.py” against the mirror (of RPMs) is to generate a new “dependancy-cache”, isn’t it? I had a successful parallel build (aka build-pkgs --parallel) inside the docker container. ~1h45m on 24 core, 64G ram The prerequisite was a populated $MY_REPO/cgcs-tis-repo/dependancy-cache Currently we only generate the cache after the build in the 'generate-cgcs-tis-repo' step. I'd like to see the cache stored in git and updated regularly by 'official' builds. Note: The cache doesn't have to be perfect, so a cache that is out of date by a day or a week is still very useful. build-pkgs/mockchain just needs a rough guide on build dependencies and potential dependency loops.