Difference between revisions of "Build an Official ISO"

From SME Server
Jump to navigationJump to search
Line 18: Line 18:
 
You have to install a new plugin for mock : Iso_prepare<br />
 
You have to install a new plugin for mock : Iso_prepare<br />
  
copy the content of http://bugs.contribs.org/attachment.cgi?id=4059 and past it in  
+
copy the content of http://bugs.contribs.org/attachment.cgi?id=4059 and paste it in  
 
  sudo vim  /usr/lib/python2.6/site-packages/mockbuild/plugins/iso_prepare.py
 
  sudo vim  /usr/lib/python2.6/site-packages/mockbuild/plugins/iso_prepare.py
 +
 
====Folder structure, both local and buildsys====
 
====Folder structure, both local and buildsys====
  

Revision as of 20:52, 30 December 2014

PythonIcon.png Skill level: Developer
Risk of inconsistencies with Koozali SME Server methodology, upgrades & functionality is high. One must be knowledgeable about how changes impact their Koozali SME Server. Significant risk of irreversible harm.



Warning.png Work in Progress:
This page is a Work in Progress. The contents off this page may be in flux, please have a look at this page history the to see list of changes.


Introduction

These Notes have been kindly shared by Ian Wells, thanks to him.

discussion on the Iso build

bugzilla:7675

build_installer and build_ISO are meant to be run in a mock chroot with the stage tree nfs mounted into it. Each installer and ISO needs to built on the corresponding version (cos5 for sme8 and cos6 for sme9). You also need to have access to the extras repo as many of the packages needed to build the installer and ISO are contained in there. This repo is to house the needed packages that aren't going to be included in the ISO.

The os, updates, and updates-testing repos are included so we can stage the packages we are going to release without actually having to release them. Normally the updates-testing repo is used for verification (testing is for developer testing). During the ISO release schedule (usually very short) I hold moving packages to updates-testing until I have the ISO built.

The build_installer and build_ISO expect to be run from a directory that points to the head of the stage (ex. /build/stage/8). This is usually a symlink to 8.1 or 8.1beta1. The script fully expands the symlink and pulls off the last element to figure out what version we are going to build.

mock plugin : iso_prepare

You have to install a new plugin for mock : Iso_prepare

copy the content of http://bugs.contribs.org/attachment.cgi?id=4059 and paste it in

sudo vim  /usr/lib/python2.6/site-packages/mockbuild/plugins/iso_prepare.py

Folder structure, both local and buildsys

This document presumes that

~/builds_bin is builds_bin from CVS
~/cdrom.image is cdrom.image from CVS
~/anacondo-po contains the anaconda translations
~/smeserver contains a local mirror 
~/rpms is rpms from CVS

update_repos

update_repos (/build/smeserver/stage/bin/update_repos) is used to keep the repos in sync and to create the ‘stage’ from which the ISO is built. On the local build system update_repos can be run manually before building an ISO, whereas it runs every two hours on the buildsys.

The buildsys has two main folders that are relevant for building an ISO.

repo      => '/build/smeserver/repo/ ', eg '/build/smeserver/repo/testing/9/'
stage     => '/build/smeserver/stage/', eg  '/build/smeserver/stage/9/'
  • repo is the folder that replicates to the mirrors. Be careful of all changes here.
  • stage is the working directory. Note that update_repos expects the folder structure to be in place.

update_repos relies on RPM2 which can be found as an RPM. It also needs all GPG keys for every package that it sees, it can take a while to find them all.

yum install perl-RPM2
Important.png Note:
Note that update_repos copies smeupdates-testing packages to stage but not from smetest.


Local Only: Create the repo folder and the binary program folder

Local Only

mkdir -p /build/smeserver/repo
mkdir -p /build/smeserver/stage/bin

The content of builds_bin from CVS needs to be copied to /build/smeserver/stage/bin

Eg

/bin/cp ~/builds_bin/* /build/smeserver/stage/bin

Building the SME Server ISOs

Important.png Note:
Note I needed to edit mock cfg file to include /build in the chroot due to my local setup.


There is also an update to /usr/lib/python2.6/site-packages/mockbuild/mounts.py see http://bugs.contribs.org/show_bug.cgi?id=7675#c61

Preparation for a new ISO release

Server-Manager translations

Check translations from http://translate.contribs.org/patches/ if the filesize is >0 then need to patch smeserver-locale

Anaconda translations

Check installer translations from http://translate.contribs.org/anaconda/ I have no idea how to tell if there are updates get the latest files,

wget --mirror --no-parent -A.mo -P ~/anacondo-po http://translate.contribs.org/anaconda/

Check if installer needs re-building

If either anaconda has changed OR there is new content in cdrom.image (apart from the release note) then the installer needs to be rebuilt (build_installer). Otherwise just the ISO needs to be built (build_ISO).

Update cdrom.image content

Update cdrom.image to the latest in CVS
cd ~/cdrom.image 
cvs update -dPA


Anaconda version

Check which anaconda was used upstream, eg http://vault.centos.org/5.10/os/SRPMS/ and is it newer than the one in CVS. If it is then anaconda needs to be updated and patched. | The latest Anaconda also needs to be in smeextras.


SME Server 9

Local Only: Update local mirror as seen by the chroot

Local Only

rsync -av --delete --progress ~/smeserver/releases/9/ /build/smeserver/repo/9

Create release specific folder and link

mkdir -p /build/smeserver/stage/9.1/SRPMS
cd /build/smeserver/stage/
ln -s 9.1/ 9

Local Only

mkdir -p /build/smeserver/stage/extra/isolinux/

Create cdrom content

When starting from fresh rsync can copy cdrom.image content, it also generates the folder tree.

Important.png Note:
Note that this would remove everything else out of the target directory, eg translations.


Also note that the isolinux content goes into a temporary directory, /build/smeserver/stage/extra/isolinux/

rsync -a --delete --delete-excluded --exclude 'CVS/' --exclude 'isolinux/' ~/cdrom.image/sme9/ /build/smeserver/stage/9/x86_64
rsync -a --delete --delete-excluded --exclude 'CVS/' --exclude 'isolinux/' ~/cdrom.image/sme9/ /build/smeserver/stage/9/i386
rsync -a --delete --delete-excluded --exclude 'CVS/' ~/cdrom.image/sme9/isolinux/ /build/smeserver/stage/extra/isolinux/9    


OR Refresh cdrom content

If only refreshing the files, then remove the delete.

rsync -a --exclude 'CVS/' --exclude 'isolinux/' ~/cdrom.image/sme9/ /build/smeserver/stage/9/x86_64
rsync -a --exclude 'CVS/' --exclude 'isolinux/' ~/cdrom.image/sme9/ /build/smeserver/stage/9/i386
rsync -a --exclude 'CVS/' ~/cdrom.image/sme9/isolinux/ /build/smeserver/stage/extra/isolinux/9    


Create the remaining folders

Important.png Note:
Note: images and isolinux do not need to be created as they are removed anyway in build_installer


  • i386
cd /build/smeserver/stage/9/i386
ln -s Packages/ SME
mkdir -p /build/smeserver/stage/9/i386/repodata/
mkdir -p /build/smeserver/stage/9/i386/updates/po
mkdir -p /build/smeserver/stage/9/i386/updates/textw
mkdir -p /build/smeserver/stage/9/i386/updates/iw
mkdir -p /build/smeserver/stage/9/i386/updates/storage/devicelibs
  • x86_64
cd /build/smeserver/stage/9/x86_64
ln -s Packages/ SME
mkdir -p /build/smeserver/stage/9/x86_64/repodata/
mkdir -p /build/smeserver/stage/9/x86_64/updates/po
mkdir -p /build/smeserver/stage/9/x86_64/updates/textw
mkdir -p /build/smeserver/stage/9/x86_64/updates/iw
mkdir -p /build/smeserver/stage/9/x86_64/updates/storage/devicelibs
  • other directories
mkdir -p /build/smeserver/stage/iso/9/i386
mkdir -p /build/smeserver/stage/iso/9/x86_64
mkdir -p /build/smeserver/stage/iso/9/source

Copy anaconda translations

Anaconda translations need copying to /updates/po from ~/anacondo-po

/bin/cp -R ~/anacondo-po/translate.contribs.org/anaconda/sme9/* /build/smeserver/stage/9/i386/updates/po
/bin/cp -R ~/anacondo-po/translate.contribs.org/anaconda/sme9/* /build/smeserver/stage/9/x86_64/updates/po

Copy anaconda content

See ~/cdrom.image/sme9/updates/README.txt for details of what to copy. The patched anaconda files from ~/rpms/anaconda/sme9 will go to updates directory. The exact directory depends on the anaconda version, eg ~/rpms/anaconda/sme9/anaconda-13.21.215

cd ~/rpms/anaconda/sme9/
make prep
cd ~/rpms/anaconda/sme9/anaconda-13.21.215
/bin/cp -R storage /build/smeserver/stage/9/i386/updates/
/bin/cp -R textw/*.py /build/smeserver/stage/9/i386/updates/textw
/bin/cp -R iw/*.py /build/smeserver/stage/9/i386/updates/iw
/bin/cp *.py /build/smeserver/stage/9/i386/updates
/bin/cp lang-table /build/smeserver/stage/9/i386/updates
/bin/cp -R storage /build/smeserver/stage/9/x86_64/updates/
/bin/cp -R textw/*.py /build/smeserver/stage/9/x86_64/updates/textw
/bin/cp -R iw/*.py /build/smeserver/stage/9/x86_64/updates/iw
/bin/cp *.py /build/smeserver/stage/9/x86_64/updates
/bin/cp lang-table /build/smeserver/stage/9/x86_64/updates

Local Only: Hack: Permission Fixup

This is only needed on local server if permissions are incorrect. It should not be needed.

Local Only

sudo chmod -R a+w /build/smeserver/stage

Local Only: Create stage content

Important.png Note:
Note: This step occurs every two hours on the buildsys.


Local Only

/build/smeserver/stage/bin/update_repos

Prepare SME Server 8

Local Only: Update local mirror as seen by the chroot

rsync -av --delete --progress ~/smeserver/releases/8/ /build/smeserver/repo/8

Create release specific folder and link (or rename folder)

mkdir -p /build/smeserver/stage/8.1/SRPMS
cd /build/smeserver/stage/
ln -s 8.1/ 8
mkdir -p /build/smeserver/stage/extra/isolinux/


Create cdrom content

When starting from fresh rsync can copy cdrom.image content, it also generates the folder tree.

Important.png Note:
Note that this would remove everything else out of the target directory, eg translations.


Also note that the isolinux content goes into a temporary directory, /build/smeserver/stage/extra/isolinux/

rsync -a --delete --delete-excluded --exclude 'CVS/' --exclude 'isolinux/' ~/cdrom.image/sme8/ /build/smeserver/stage/8/x86_64
rsync -a --delete --delete-excluded --exclude 'CVS/' --exclude 'isolinux/' ~/cdrom.image/sme8/ /build/smeserver/stage/8/i386
rsync -a --delete --delete-excluded --exclude 'CVS/' ~/cdrom.image/sme8/isolinux/ /build/smeserver/stage/extra/isolinux/8    


OR Refresh cdrom content

If only refreshing the files, then remove the delete.

rsync -a --exclude 'CVS/' --exclude 'isolinux/' ~/cdrom.image/sme8/ /build/smeserver/stage/8/x86_64
rsync -a --exclude 'CVS/' --exclude 'isolinux/' ~/cdrom.image/sme8/ /build/smeserver/stage/8/i386
rsync -a --exclude 'CVS/' ~/cdrom.image/sme8/isolinux/ /build/smeserver/stage/extra/isolinux/8    


Create the remaining folders

Important.png Note:
Note: images and isolinux do not need to be created as they are removed anyway in build_installer


  • i386
cd /build/smeserver/stage/8/i386
ln -s Packages/ SME
mkdir -p /build/smeserver/stage/8/i386/repodata/
mkdir -p /build/smeserver/stage/8/i386/updates/po
/bin/cp ~/cdrom.image/sme8/SME/base/comps.xml /build/smeserver/stage/8/i386/repodata/
  • x86_64
cd /build/smeserver/stage/8/x86_64
ln -s Packages/ SME
mkdir -p /build/smeserver/stage/8/x86_64/repodata/
mkdir -p /build/smeserver/stage/8/x86_64/updates/po
/bin/cp ~/cdrom.image/sme8/SME/base/comps.xml /build/smeserver/stage/8/x86_64/repodata/
  • other directories
mkdir -p /build/smeserver/stage/iso/8/i386
mkdir -p /build/smeserver/stage/iso/8/x86_64
mkdir -p /build/smeserver/stage/iso/8/source

Copy anaconda translations

Anaconda translations need copying to /updates/po from ~/anacondo-po

/bin/cp -R ~/anacondo-po/translate.contribs.org/anaconda/sme8/* /build/smeserver/stage/8/i386/updates/po
/bin/cp -R ~/anacondo-po/translate.contribs.org/anaconda/sme8/* /build/smeserver/stage/8/x86_64/updates/po


Copy anaconda content

See ~/cdrom.image/sme8/updates/README.txt for details of what to copy. The patched anaconda files from ~/rpms/anaconda/sme8 will go to updates directory. The exact directory depends on the anaconda version, eg ~/rpms/anaconda/sme8/anaconda-11.1.2.263. One file (bootloaderInfo.py) also comes from booty. Finally lang-names needs to come from the actual anaconda RPM to updates which is a manual step.

cd ~/rpms/anaconda/sme8/
make prep
cd ~/rpms/anaconda/sme8/anaconda-11.1.2.263
/bin/cp anaconda bootloader.py findpackageset.py fsset.py instdata.py kickstart.py lang-table language.py packages.py partedUtils.py raid.py upgrade.py yuminstall.py /build/smeserver/stage/8/i386/updates/
/bin/cp anaconda bootloader.py findpackageset.py fsset.py instdata.py kickstart.py lang-table language.py packages.py partedUtils.py raid.py upgrade.py yuminstall.py /build/smeserver/stage/8/x86_64/updates/
cd textw
/bin/cp confirm_text.py progress_text.py timezone_text.py upgrade_bootloader_text.py upgrade_text.py /build/smeserver/stage/8/i386/updates/
/bin/cp confirm_text.py progress_text.py timezone_text.py upgrade_bootloader_text.py upgrade_text.py /build/smeserver/stage/8/x86_64/updates/
cd  ~/rpms/booty/sme8/
make prep
/bin/cp ~/rpms/booty/sme8/booty-0.80.6/bootloaderInfo.py /build/smeserver/stage/8/i386/updates/
/bin/cp ~/rpms/booty/sme8/booty-0.80.6/bootloaderInfo.py /build/smeserver/stage/8/x86_64/updates/

Local Only: Hack: Permission Fixup

This is only needed on local server if permissions are incorrect. It should not be needed.

  • Massage permissions before starting ISO building

Local Only

sudo chmod -R a+w /build/smeserver/stage

Local Only: Create stage content

Important.png Note:
Note: This step occurs every two hours on the buildsys.


/build/smeserver/stage/bin/update_repos

Build & Release SME Server 8

Build SME Server 8 32 bit ISO

mock -r smeserver-8-i386-iso --init
mock -r smeserver-8-i386-iso --cwd /build/smeserver/stage/8 --unpriv --chroot /build/smeserver/stage/bin/build_installer
mock -r smeserver-8-i386-iso --cwd /build/smeserver/stage/8 --unpriv --chroot /build/smeserver/stage/bin/build_ISO

Build SME Server 8 64 bit ISO

mock -r smeserver-8-x86_64-iso --init
mock -r smeserver-8-x86_64-iso --cwd /build/smeserver/stage/8 --unpriv --chroot /build/smeserver/stage/bin/build_installer
mock -r smeserver-8-x86_64-iso --cwd /build/smeserver/stage/8 --unpriv --chroot /build/smeserver/stage/bin/build_ISO

Prepare directory structure for release

SME 8.x test releases will go to /build/smeserver/repo/testing/8. Check that the link points to the correct testing release. Note when uploading Jigdos the ‘8’ link will not be pointing to the new directory.

mkdir -p /build/smeserver/repo/testing/8.1/iso/i386/
mkdir -p /build/smeserver/repo/testing/8.1/iso/source/
mkdir -p /build/smeserver/repo/testing/8.1/iso/x86_64/
mkdir -p /build/smeserver/repo/testing/8.1/smeos/i386/
mkdir -p /build/smeserver/repo/testing/8.1/smeos/SRPMS/
mkdir -p /build/smeserver/repo/testing/8.1/smeos/x86_64/

Copy SME Server 8 64 bit Jigdo to mirror

/bin/cp -a /build/smeserver/stage/iso/8/x86_64/smeserver-8.1-x86_64.jigdo /build/smeserver/repo/testing/8.1/iso/x86_64/
/bin/cp -a /build/smeserver/stage/iso/8/x86_64/smeserver-8.1-x86_64.template /build/smeserver/repo/testing/8.1/iso/x86_64/
rsync -a --delete /build/smeserver/stage/8/x86_64/ /build/smeserver/repo/testing/8.1/smeos/x86_64

Copy SME Server 8 32 bit Jigdo to mirror

/bin/cp -a /build/smeserver/stage/iso/8/i386/smeserver-8.1-i386.jigdo /build/smeserver/repo/testing/8.1/iso/i386/
/bin/cp -a /build/smeserver/stage/iso/8/i386/smeserver-8.1-i386.template /build/smeserver/repo/testing/8.1/iso/i386/
rsync -a --delete /build/smeserver/stage/8/i386/ /build/smeserver/repo/testing/8.1/smeos/i386

Copy SME Server 8 SRPMS Jigdo to mirror

/bin/cp -a /build/smeserver/stage/iso/8/source/smeserver-8.1-SRPMS.jigdo /build/smeserver/repo/testing/8.1/iso/source/
/bin/cp -a /build/smeserver/stage/iso/8/source/smeserver-8.1-SRPMS.template /build/smeserver/repo/testing/8.1/iso/source/
rsync -a --delete /build/smeserver/stage/8/SRPMS/ /build/smeserver/repo/testing/8.1/smeos/SRPMS

Copy SME Server 8 ISOs to mirror

rsync -a --delete /build/smeserver/stage/iso/8/ /build/smeserver/repo/testing/8.1/iso

Build & Release SME Server 9

Build SME Server 9 32 bit ISO

mock -r smeserver-9-i386-iso --init
mock -r smeserver-9-i386-iso --cwd /build/smeserver/stage/9 --unpriv --chroot /build/smeserver/stage/bin/build_installer
mock -r smeserver-9-i386-iso --cwd /build/smeserver/stage/9 --unpriv --chroot /build/smeserver/stage/bin/build_ISO

Build SME Server 9 64 bit ISO

mock -r smeserver-9-x86_64-iso --init
mock -r smeserver-9-x86_64-iso --cwd /build/smeserver/stage/9 --unpriv --chroot /build/smeserver/stage/bin/build_installer
mock -r smeserver-9-x86_64-iso --cwd /build/smeserver/stage/9 --unpriv --chroot /build/smeserver/stage/bin/build_ISO

Prepare directory structure for release

SME 9.x test releases will go to /build/smeserver/repo/testing/9. Check that the link points to the correct testing release. Note when uploading Jigdos the ‘9’ link will not be pointing to the new directory.

mkdir -p /build/smeserver/repo/testing/9.0/iso/i386/
mkdir -p /build/smeserver/repo/testing/9.0/iso/source/
mkdir -p /build/smeserver/repo/testing/9.0/iso/x86_64/
mkdir -p /build/smeserver/repo/testing/9.0/smeos/x86_64/
mkdir -p /build/smeserver/repo/testing/9.0/smeos/i386/

Copy SME Server 9 64 bit Jigdo to mirror

/bin/cp -a /build/smeserver/stage/iso/9/x86_64/smeserver-9.0-x86_64.jigdo /build/smeserver/repo/testing/9.0/iso/x86_64/
/bin/cp -a /build/smeserver/stage/iso/9/x86_64/smeserver-9.0-x86_64.template /build/smeserver/repo/testing/9.0/iso/x86_64/
rsync -a --delete /build/smeserver/stage/9/x86_64/ /build/smeserver/repo/testing/9.0/smeos/x86_64

Copy SME Server 9 32 bit Jigdo to mirror

/bin/cp -a /build/smeserver/stage/iso/9/i386/smeserver-9.0-i386.jigdo /build/smeserver/repo/testing/9.0/iso/i386/
/bin/cp -a /build/smeserver/stage/iso/9/i386/smeserver-9.0-i386.template /build/smeserver/repo/testing/9.0/iso/i386/
rsync -a --delete /build/smeserver/stage/9/i386/ /build/smeserver/repo/testing/9.0/smeos/i386

Copy SME Server 9 SRPMS Jigdo to mirror

/bin/cp -a /build/smeserver/stage/iso/9/source/smeserver-9.0-SRPMS.jigdo /build/smeserver/repo/testing/9.0/iso/source/
/bin/cp -a /build/smeserver/stage/iso/9/source/smeserver-9.0-SRPMS.template /build/smeserver/repo/testing/9.0/iso/source/
rsync -a --delete /build/smeserver/stage/9/SRPMS/ /build/smeserver/repo/testing/9.0/smeos/SRPMS

Copy SME Server 9 ISOs to mirror

rsync -a --delete /build/smeserver/stage/iso/9/ /build/smeserver/repo/testing/9.0/iso


Hard links

As an example how to create the hard links:

cp -lr /build/smeserver/repo/testing/9.0rc1/smeos/* /build/smeserver/repo/testing/9.0/smeos/
mkdir -p /build/smeserver/repo/testing/9.0/smeaddons/
mkdir -p /build/smeserver/repo/testing/9.0/smecontribs/
mkdir -p /build/smeserver/repo/testing/9.0/smedev/
mkdir -p /build/smeserver/repo/testing/9.0/smeextras/
mkdir -p /build/smeserver/repo/testing/9.0/smetest/
mkdir -p /build/smeserver/repo/testing/9.0/smeupdates/
mkdir -p /build/smeserver/repo/testing/9.0/smeupdates-testing/
cp -lr /build/smeserver/repo/testing/9.0rc1/smecontribs/* /build/smeserver/repo/testing/9.0/smecontribs/
cp -lr /build/smeserver/repo/testing/9.0rc1/smedev/* /build/smeserver/repo/testing/9.0/smedev/
cp -lr /build/smeserver/repo/testing/9.0rc1/smeextras/* /build/smeserver/repo/testing/9.0/smeextras/
cp -lr /build/smeserver/repo/testing/9.0rc1/smetest/* /build/smeserver/repo/testing/9.0/smetest/
cp -lr /build/smeserver/repo/testing/9.0rc1/smeupdates/* /build/smeserver/repo/testing/9.0/smeupdates/
cp -lr /build/smeserver/repo/testing/9.0rc1/smeupdates-testing/* /build/smeserver/repo/testing/9.0/smeupdates-testing/
cp -lr /build/smeserver/repo/testing/9.0rc1/smeaddons/* /build/smeserver/repo/testing/9.0/smeaddons/