autospec¶
autospec is a tool used to assist with the automated creation and
maintenance of RPM packaging in Clear Linux* OS. Where a standard
RPM build process using rpmbuild
requires a tarball and .spec
file to start, autospec requires only a
tarball and package name to start.
Description¶
The autospec tool attempts to infer the requirements of the .spec
file by analyzing the source code and Makefile
information. It
continuously runs updated builds based on new information discovered from
build failures until it has a complete and valid .spec
file. If
needed, you can influence the behavior of autospec and customize the build by providing optional control files to the autospec tool.
autospec uses mock as a sandbox to run the builds. Visit the mock wiki for additional information on using mock.
For a general understanding of how an RPM works, visit the rpm website or the RPM Packaging Guide.
How it works¶
Learn the autospec tool set up and process.
Prerequisites¶
The setup for building source in Clear Linux OS must be completed before using the autospec tool.
Refer to Setup environment to build source for instructions on completing the setup.
Create an RPM¶
The basic autospec process is described in the following steps:
The make autospec command generates a
.spec
file based on the analysis of code and existing control files.Any control files should be located in the same directory as the resulting
.spec
file. View the autospec README for more information on control files.autospec creates a build root with mock config.
autospec attempts to build an RPM from the generated
.spec
.autospec detects any missed declarations in the
.spec
.If build errors occur, autospec scans the build log to try to detect the root cause.
If autospec detects the root cause and knows how to continue, it restarts the build automatically at step 1 with updated build instructions.
Otherwise, autospec stops the build for user inspection to resolve the errors. Respond to the build process output by fixing source code issues and/or editing control files to resolve issues, which may include dependencies or exclusions. See autospec README for more information on control files.
The user resumes the process at step 1 after errors are resolved.
If a binary dependency doesn’t exist in Clear Linux OS, you must build it before running autospec again.
Following these steps, autospec continues to rebuild the package, based on
new information discovered from build failures, until it has a valid
.spec
. If no build errors occur, RPM packages are successfully built.
Examples¶
Complete Setup environment to build source before using these examples.
Example 1: Build RPM with an existing spec file¶
This example shows how to build a RPM from a pre-packaged upstream package
with an existing spec file. The example uses the dmidecode
package.
Navigate to the autospec workspace and clone the
dmidecode
package:cd ~/clearlinux make clone_dmidecode
Note
You can clone all package repos at once using the following command:
make [-j NUM] clone-packages
The optional NUM is the number of threads to use.
For a list of available packages, view the
~/clearlinux/projects/common/packages
file.Navigate to the local copy of the
dmidecode
package and build it:cd ~/clearlinux/packages/dmidecode/ make build
The resulting RPMs are in
./rpms
. Build logs and additional RPMs are in./results
.
Example 2: Build a new RPM¶
This example shows how to build a new RPM with no spec file. The example will create a simple helloclear RPM.
Navigate to the autospec workspace and build the helloclear RPM. The
Makefile
provides a make autospecnew that can automatically generate an RPM package using the autospec tool. You must pass the URL to the source tarball and the NAME of the RPM you wish to create:cd ~/clearlinux make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" NAME="helloclear"
The resulting RPMs are in
./packages/helloclear/rpms
. Build logs and additional RPMs are in./packages/helloclear/results
.
Example 3: Generate a new spec file with a pre-defined package¶
This example shows how to modify an existing package to create a custom RPM.
In this example you will make a simple change to the dmidecode
package
and rebuild the package.
Navigate to the autospec workspace and clone the
dmidecode
package:cd ~/clearlinux make clone_dmidecode
Navigate into the dmidecode directory:
cd packages/dmidecode
Open the
excludes
file with an editor and add these lines:/usr/bin/biosdecode /usr/bin/ownership /usr/bin/vpddecode /usr/share/man/man8/biosdecode.8 /usr/share/man/man8/ownership.8 /usr/share/man/man8/vpddecode.8
Note
These files aren’t needed by dmidecode, so we can remove them without any issues.
In the
dmidecode
directory, build the modifieddmidecode
package:make autospec
The resulting RPMs are in
./rpms
. Logs are in./results
.
Example 4: Provide control files to autospec¶
This example shows how to modify control files to correct build failures that autospec is unable to resolve. In this example, you will add a missing license and dependencies so autospec can complete a successful build.
Navigate to the autospec workspace:
cd ~/clearlinux
If you have not already, clone all upstream package repos:
make [-j NUM] clone-packages
The optional NUM is the number of threads to use.
Note
In a later step of this example, we will search the cloned package repos for a missing dependency.
Build the opae-sdk RPM:
make autospecnew URL="https://github.com/OPAE/opae-sdk/archive/0.13.0.tar.gz" NAME="opae-sdk"
This results in an error for a missing license file:
[FATAL] Cannot find any license or opae-sdk.license file!
Navigate to the package with build failures:
cd packages/opae-sdk
Add one or more valid license identifiers from the SPDX License List. In the example below, two different licenses are appropriate based on the opae-sdk project licensing:
echo "BSD-3-Clause MIT" > opae-sdk.license
Run autospec again:
make autospec
This results in a generic error:
[FATAL] Build failed, aborting
Open the build log to view the error details:
cat ./results/build.log
The build log contains details for the specific failures. In this instance, there are missing dependencies:
CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: CJSON_LIBRARY linked by target "opae-c++-utils" in directory /builddir/build/BUILD/opae-sdk-0.13.0/tools/c++utilslib json-c_LIBRARIES linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae libuuid_LIBRARIES linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae
Search the spec files of upstream Clear Linux OS packages to see if the json-c library is available. In this case, it does exist and we’ll add the json-c ‘dev’ package into the buildreq_add:
grep 'json-c\.so$' ~/clearlinux/packages/*/*.spec echo "json-c-dev" >> buildreq_add
Note
This search step works only if the user cloned all of the upstream package repos. In this example, upstream package repos were cloned in a previous step.
Search the spec files of upstream Clear Linux OS packages to see if the libuuid library is available. In this case, it exists in the util-linux package, so we’ll add util-linux-dev package into the buildreq_add:
grep 'libuuid\.so$' ~/clearlinux/packages/*/*.spec echo "util-linux-dev" >> buildreq_add
Run autospec again and find the successfully-generated RPMs in the
rpms
directory:make autospec
Note
If you need a dependency that does not exist in the Clear Linux OS repo, you must first build it manually (see Example 2: Build a new RPM), then add the repo so that autospec knows the package exists. For example:
cd ~/clearlinux/packages/<package-name> make repoadd make repostatus
You only need to add the dependency to the
buildreq_add
control file if autospec is not able to automatically find the correct dependency on its own.
Example 5: Update an existing package¶
The Clear Linux OS team prefers to carry no patches and seeks to make the latest releases work. If we do need patches, we use autospec to add, remove, or manage patches. The autospec control files are integral to the patch management process. Developers can expect a more streamlined approach to managing a large collection of packages with autospec.
Adding and submitting patches¶
To add patches to Clear Linux OS upstream, follow patching source code.
To submit a patch to upstream, follow contributing to an existing software package.
If you maintain a downstream derivative of Clear Linux OS and you want to integrate new or patched packages into your mix, follow the process in mixer.
Assuming you have followed the above process, autospec has generated a new spec file.
Refresh a package and inspect¶
In this example, we use autospec to refresh the m4 package and recreate RPM files.
Navigate to the top-level directory of the workspace
cd clearlinux
where clearlinux is the top level of the tooling workspace
Run the make_clone command and then navigate to the package.
make clone_m4 cd packages/m4
Make desired changes to the package, its control files, or other files.
Finally, run:
make autospec
To view spec file changes, run:
git show m4.spec
The output shows:
m4: Autospec creation for version 1.4.18 diff --git a/m4.spec b/m4.spec index f76c78d..97b846a 100644 --- a/m4.spec +++ b/m4.spec @@ -6,15 +6,14 @@ # Name : m4 Version : 1.4.18 -Release : 88 +Release : 89 URL : http://mirrors.kernel.org/gnu/m4/m4-1.4.18.tar.xz Source0 : http://mirrors.kernel.org/gnu/m4/m4-1.4.18.tar.xz -Source99 : http://mirrors.kernel.org/gnu/m4/m4-1.4.18.tar.xz.sig +Source1 : http://mirrors.kernel.org/gnu/m4/m4-1.4.18.tar.xz.sig Summary : No detailed summary available Group : Development/Tools ...
The following commands provide a more complete view of the changes.
git log -p
gitk
Test packaged software¶
After software has been packaged with autospec, the resulting RPMs can be tested for functionality before being integrated and deployed into a Clear Linux OS image with the Mixer tool.
The Clear Linux OS development tooling offers two ways to quickly test autospec generated RPMs.
Note
The methods outlined below should only be used for temporary testing on development systems.
Test in a Clear Linux OS virtual machine¶
The Clear Linux OS development tooling includes a method to install RPMs into a Clear Linux OS virtual machine running on the KVM hypervisor. Using a VM allows testing in a completely isolated environment.
To test an autospec-created package inside a VM:
Download the Clear Linux OS KVM image into the
~/clearlinux
directory asclear.img
. The location and nameclear.img.xz
is important for the tooling to work:cd ~/clearlinux curl -o clear.img.xz https://download.clearlinux.org/image/$(curl https://download.clearlinux.org/image/latest-images | grep '[0-9]'-kvm)
Extract the downloaded Clear Linux OS KVM image:
unxz -v clear.img.xz
Copy the QEMU start script and virtual firmware needed for KVM into the
~/clearlinux
directory:cp ~/clearlinux/projects/common/start_qemu.sh . cp /usr/share/qemu/OVMF.fd .
Run make install from the package’s autospec directory. The make install command mounts the downloaded Clear Linux OS KVM image and installs the autospec-created RPM into it:
cd ~/clearlinux/packages/<package-name> make install
The code that makes this possible can be viewed by searching for the install: target in the Makefile.common file on GitHub.
Return to the
~/clearlinux
directory and start the Clear Linux OS VM:cd ~/clearlinux/ sudo ./start_qemu.sh clear.img
A new Clear Linux OS VM will launch in the console. Log into the VM as root and set a new password for the VM.
Check that the software is installed in the Clear Linux OS VM as expected and perform any relevant tests.
After testing has been completed, the Clear Linux OS VM can be powered off and deleted:
poweroff rm clear.img
Test directly on a development machine¶
The Clear Linux OS development tooling also includes a method to extract autospec-created RPMs locally onto a Clear Linux OS development system for testing. Extracting an RPM directly onto a system offers quicker testing; however conflicts may occur and responsibility to remove the software after testing is up to the developer.
To test an autospec created package directly on the Clear Linux OS development system:
Run make install-local from the package’s autospec directory. The make install-local command extracts the RPM directly onto the filesystem of the running Clear Linux OS system:
cd ~/clearlinux/packages/<package-name> make install-local
The code that makes this possible can be viewed by searching for the install-local: target in the Makefile.common file on GitHub.
Check that the software is installed as expected and perform any relevant tests.
After testing has been completed, the software and any related files must be identified and deleted. The swupd repair --picky command can help restore the state of the
/usr
directory (see swupd) however any other files must be cleaned up manually.
References¶
Reference the autospec README for details regarding autospec commands and options.
Setup environment to build source¶
Setup of the workspace and tooling used for building source in Clear Linux OS is mostly automated for you with a setup script. It uses tools from the os-clr-on-clr bundle.
The setup script creates a workspace in the clearlinux
folder, with the
subfolders Makefile
, packages
, and projects
. The
projects
folder contains the main tools used for making packages in
Clear Linux OS autospec
and common
.
Follow these steps to setup the workspace and tooling for building source:
Install the os-clr-on-clr bundle:
sudo swupd bundle-add os-clr-on-clr
Download the
user-setup.sh
script:curl -O https://raw.githubusercontent.com/clearlinux/common/master/user-setup.sh
Make
user-setup.sh
executable:chmod +x user-setup.sh
Run the script as an unprivileged user:
./user-setup.sh
After the script completes, log out and log in again to complete the setup process.
Set your Git user email and username for the repos on your system:
git config --global user.email "you@example.com" git config --global user.name "Your Name"
This global setting is used by Clear Linux OS tools that make use of Git.