libguestfs/builder/virt-builder.pod

=begin html

<img src="virt-builder.svg" width="250"
  style="float: right; clear: right;" />

=end html

=head1 NAME

virt-builder - Build virtual machine images quickly

=head1 SYNOPSIS

 virt-builder os-version
    [-o|--output DISKIMAGE] [--size SIZE] [--format raw|qcow2]
    [--arch ARCHITECTURE] [--attach ISOFILE]
__CUSTOMIZE_SYNOPSIS__

 virt-builder -l|--list [--long] [--list-format short|long|json]

 virt-builder --notes os-version

 virt-builder --print-cache

 virt-builder --cache-all-templates

 virt-builder --delete-cache

 virt-builder --get-kernel DISKIMAGE
    [--format raw|qcow2] [--output OUTPUTDIR]

=head1 DESCRIPTION

Virt-builder is a tool for quickly building new virtual machines.  You
can build a variety of VMs for local or cloud use, usually within a
few minutes or less.  Virt-builder also has many ways to customize
these VMs.  Everything is run from the command line and nothing
requires root privileges, so automation and scripting is simple.

Note that virt-builder does not install guests from scratch.  It takes
cleanly prepared, digitally signed OS templates and customizes them.
This approach is used because it is much faster, but if you need to do
fresh installs you may want to look at L<virt-install(1)> and
L<oz-install(1)>.

The easiest way to get started is by looking at the examples in the
next section.

=head1 EXAMPLES

=head2 List the virtual machines available

 virt-builder --list

will list out the operating systems available to install.  A selection
of freely redistributable OSes is available as standard.  You can add
your own too (see below).

After choosing a guest from the list, you may want to see if there
are any installation notes:

 virt-builder --notes fedora-20

=head2 Build a virtual machine

 virt-builder fedora-20

will build a Fedora 20 image for the same architecture as virt-builder
(so running it from an i386 installation will try to build an i386
image, if available).  This will have all default
configuration (minimal size, no user accounts, random root password,
only the bare minimum installed software, etc.).

You I<do not> need to run this command as root.

The first time this runs it has to download the template over the
network, but this gets cached (see L</CACHING>).

The name of the output file is derived from the template name, so
above it will be C<fedora-20.img>.  You can change the output filename
using the I<-o> option:

 virt-builder fedora-20 -o mydisk.img

You can also use the I<-o> option to write to existing devices or
logical volumes.

 virt-builder fedora-20 --format qcow2

As above, but write the output in qcow2 format to C<fedora-20.qcow2>.

 virt-builder fedora-20 --size 20G

As above, but the output size will be 20 GB.  The guest OS is resized
as it is copied to the output (automatically, using
L<virt-resize(1)>).

 virt-builder fedora-20 --arch i386

As above, but using an i386 template, if available.

=head2 Setting the root password

 virt-builder fedora-20 --root-password file:/tmp/rootpw

Create a Fedora 20 image.  The root password is taken from the file
C</tmp/rootpw>.

Note if you I<don't> set I<--root-password> then the guest is given
a I<random> root password.

You can also create user accounts.  See L</USERS AND PASSWORDS> below.

=head2 Set the hostname

 virt-builder fedora-20 --hostname virt.example.com

Set the hostname to C<virt.example.com>.

=head2 Installing software

To install packages from the ordinary (guest) software repository
(eg. yum or apt):

 virt-builder fedora-20 --install "inkscape,@Xfce Desktop"

(In Fedora, C<@> is used to install groups of packages.  On Debian
you would install a meta-package instead.)

To update the core packages to the latest version:

 virt-builder debian-7 --update

For guests which use SELinux, like Fedora and Red Hat Enterprise
Linux, you may need to do SELinux relabelling after installing or
updating packages (see L</SELINUX> below):

 virt-builder fedora-20 --update --selinux-relabel

=head2 Customizing the installation

There are many options that let you customize the installation.  These
include: I<--run>/I<--run-command>, which run a shell script or
command while the disk image is being generated and lets you add or
edit files that go into the disk image.
I<--firstboot>/I<--firstboot-command>, which let you add
scripts/commands that are run the first time the guest boots.
I<--edit> to edit files.  I<--upload> to upload files.

For example:

 cat <<'EOF' > /tmp/yum-update.sh
 yum -y update
 EOF
 
 virt-builder fedora-20 --firstboot /tmp/yum-update.sh

or simply:

 virt-builder fedora-20 --firstboot-command 'yum -y update'

which makes the L<yum(8)> C<update> command run once the first time
the guest boots.

Or:

 virt-builder fedora-20 --edit '/etc/yum.conf: s/gpgcheck=1/gpgcheck=0/'

which edits C</etc/yum.conf> inside the disk image (during disk image
creation, long before boot).

You can combine these options, and have multiple options of all types.

=head1 OPTIONS

=over 4

=item B<--help>

Display help.

=item B<--arch> ARCHITECTURE

Use the specified architecture for the output image.  This means
there must be sources providing the requested template for the
requested architecture.

See also L</ARCHITECTURE>.

=item B<--attach> ISOFILE

During the customization phase, the given disk is attached to the
libguestfs appliance.  This is used to provide extra software
repositories or other data for customization.

You probably want to ensure the volume(s) or filesystems in the
attached disks are labelled (or use an ISO volume name) so that you
can mount them by label in your run-scripts:

 mkdir /tmp/mount
 mount LABEL=EXTRA /tmp/mount

You can have multiple I<--attach> options, and the format can be any
disk format (not just an ISO).

See also: I<--run>,
L</Installing packages at build time from a side repository>,
L<genisoimage(1)>, L<virt-make-fs(1)>.

=item B<--attach-format> FORMAT

Specify the disk format for the next I<--attach> option.  The
C<FORMAT> is usually C<raw> or C<qcow2>.  Use C<raw> for ISOs.

=item B<--cache> DIR

=item B<--no-cache>

I<--cache> DIR sets the directory to use/check for cached template
files.  If not set, defaults to either
C<$XDG_CACHE_HOME/virt-builder/> or C<$HOME/.cache/virt-builder/>.

I<--no-cache> disables template caching.

=item B<--cache-all-templates>

Download all templates to the cache and then exit.  See L</CACHING>.

Note this doesn't cache everything.  More templates might be uploaded.
Also this doesn't cache packages (the I<--install>, I<--update> options).

=item B<--check-signature>

=item B<--no-check-signature>

Check/don't check the digital signature of the OS template.  The
default is to check the signature and exit if it is not correct.
Using I<--no-check-signature> bypasses this check.

See also I<--fingerprint>.

=item B<--curl> CURL

Specify an alternate L<curl(1)> binary.  You can also use this to add
curl parameters, for example to disable https certificate checks:

 virt-builder --curl "curl --insecure" [...]

=item B<--delete-cache>

Delete the template cache.  See L</CACHING>.

=item B<--no-delete-on-failure>

Don't delete the output file on failure to build.  You can use this to
debug failures to run scripts.  See L</DEBUGGING BUILDS> for ways to
debug images.

The default is to delete the output file if virt-builder fails (or,
for example, some script that it runs fails).

=item B<--fingerprint> 'AAAA BBBB ...'

Check that the index and templates are signed by the key with the
given fingerprint.  (The fingerprint is a long string, usually written
as 10 groups of 4 hexadecimal digits).

You can give this option multiple times.  If you have multiple source
URLs, then you can have either no fingerprint, one fingerprint or
multiple fingerprints.  If you have multiple, then each must
correspond 1-1 with a source URL.

=item B<--format> qcow2

=item B<--format> raw

For ordinary builds, this selects the output format.  The default is I<raw>.

With I<--get-kernel> this specifies the input format.

=item B<--get-kernel> IMAGE

This option extracts the kernel and initramfs from a previously built
disk image called C<IMAGE> (in fact it works for any VM disk image,
not just ones built using virt-builder).

The kernel and initramfs are written to the current directory, unless
you also specify the I<--output> C<outputdir> B<directory> name.

The format of the disk image is automatically detected unless you
specify it by using the I<--format> option.

In the case where the guest contains multiple kernels, the one with
the highest version number is chosen.  To extract arbitrary kernels
from the disk image, see L<guestfish(1)>.  To extract the entire
C</boot> directory of a guest, see L<virt-copy-out(1)>.

=item B<--gpg> GPG

Specify an alternate L<gpg(1)> (GNU Privacy Guard) binary.  You can
also use this to add gpg parameters, for example to specify an
alternate home directory:

 virt-builder --gpg "gpg --homedir /tmp" [...]

=item B<-l>

=item B<--list>

=item B<--list --format> format

=item B<--list --long>

List available templates.

It is possible to choose with I<--format> the output format for the list
templates:

=over 4

=item B<short>

The default format, prints only the template identifier and, next to it,
its short description.

=item B<long>

Prints a textual list with the details of the available sources, followed
by the details of the available templates.

=item B<json>

Prints a JSON object with the details of the available sources and
the details of the available templates.

The C<version> key in the main object represents the "compatibility version",
and it is bumped every time the resulting JSON output is incompatible with
the previous versions (for example the structure has changed, or non-optional
keys are no more present).

=back

I<--long> is a shorthand for the C<long> format.

See also: I<--source>, I<--notes>, L</SOURCES OF TEMPLATES>.

=item B<--machine-readable>

This option is used to make the output more machine friendly
when being parsed by other programs.  See
L</MACHINE READABLE OUTPUT> below.

=item B<-m> MB

=item B<--memsize> MB

Change the amount of memory allocated to I<--run> scripts.  Increase
this if you find that I<--run> scripts or the I<--install> option are
running out of memory.

The default can be found with this command:

 guestfish get-memsize

=item B<--network>

=item B<--no-network>

Enable or disable network access from the guest during the installation.

Enabled is the default.  Use I<--no-network> to disable access.

The network only allows outgoing connections and has other minor
limitations.  See L<virt-rescue(1)/NETWORK>.

If you use I<--no-network> then certain other options such as
I<--install> will not work.

This does not affect whether the guest can access the network once it
has been booted, because that is controlled by your hypervisor or
cloud environment and has nothing to do with virt-builder.

Generally speaking you should I<not> use I<--no-network>.  But here
are some reasons why you might want to:

=over 4

=item 1.

Because the libguestfs backend that you are using doesn't support the
network.  (See: L<guestfs(3)/BACKEND>).

=item 2.

Any software you need to install comes from an attached ISO, so you
don't need the network.

=item 3.

You don't want untrusted guest code trying to access your host network
when running virt-builder.  This is particularly an issue when you
don't trust the source of the operating system templates.  (See
L</SECURITY> below).

=item 4.

You don't have a host network (eg. in secure/restricted environments).

=back

=item B<--no-sync>

Do not sync the output file on exit.

Virt-builder fsync's the output file or disk image when it exits.

The reason is that qemu/KVM's default caching mode is C<none> or
C<directsync>, both of which bypass the host page cache.  Therefore
these would not work correctly if you immediately started the guest
after running virt-builder - they would not see the complete output
file.  (Note that you should not use these caching modes - they are
fundamentally broken for this and other reasons.)

If you are not using these broken caching modes, you can use
I<--no-sync> to avoid this unnecessary sync and gain considerable
extra performance.

=item B<--notes> os-version

List any notes associated with this guest, then exit (this does not do
the install).

=item B<-o> filename

=item B<--output> filename

Write the output to C<filename>.  If you don't specify this option,
then the output filename is generated by taking the C<os-version>
string and adding C<.img> (for raw format) or C<.qcow2> (for qcow2
format).

Note that the output filename could be a device, partition or logical
volume.

When used with I<--get-kernel>, this option specifies the output
directory.

=item B<--print-cache>

Print information about the template cache.  See L</CACHING>.

=item B<--quiet>

Don't print ordinary progress messages.

=item B<--size> SIZE

Select the size of the output disk, where the size can be specified
using common names such as C<32G> (32 gigabytes) etc.

Virt-builder will resize filesystems inside the disk image
automatically.

If the size is not specified, then one of two things happens.  If the
output is a file, then the size is the same as the template.  If the
output is a device, partition, etc then the size of that device is
used.

To specify size in bytes, the number must be followed by the lowercase
letter I<b>, eg: S<C<--size 10737418240b>>.

=item B<--smp> N

Enable N E<ge> 2 virtual CPUs for I<--run> scripts to use.

=item B<--source> URL

Set the source URL to look for indexes.

You can give this option multiple times to specify multiple sources.

See also L</SOURCES OF TEMPLATES> below.

Note that you should not point I<--source> to sources that you don't
trust (unless the source is signed by someone you do trust).  See also
the I<--no-network> option.

=item B<-v>

=item B<--verbose>

Enable debug messages and/or produce verbose output.

When reporting bugs, use this option and attach the complete output to
your bug report.

=item B<-V>

=item B<--version>

Display version number and exit.

=back

=head2 Customization options

__CUSTOMIZE_OPTIONS__

=head1 REFERENCE

=head2 INSTALLING PACKAGES

There are several approaches to installing packages or applications in
the guest which have different trade-offs.

=head3 Installing packages at build time

If the guest OS you are installing is similar to the host OS (eg.
both are Linux), and if libguestfs supports network connections, then
you can use I<--install> to install packages like this:

 virt-builder fedora-20 --install inkscape

This uses the guest's package manager and the host's network
connection.

=head3 Updating packages at build time

To update the core set of packages in the template at build time:

 virt-builder fedora-20 --update

Most of the templates that ship with virt-builder come with a very
minimal selection of packages (known as a "JEOS" or "Just Enough
Operating System"), which are up to date at the time the template is
created, but could be out of date by the time you come to install an
OS from the template.  This option updates those template packages.

=head3 Installing packages at first boot

Another option is to install the packages when the guest first boots:

 virt-builder fedora-20 --firstboot-install inkscape

This uses the guest's package manager and the guest's network
connection.

The downsides are that it will take the guest a lot longer to boot
first time, and there's nothing much you can do if package
installation fails (eg. if a network problem means the guest can't
reach the package repositories).

=head3 Installing packages at build time from a side repository

If the software you want to install is not available in the main
package repository of the guest, then you can add a side repository.
Usually this is presented as an ISO (CD disk image) file containing
extra packages.

You can create the disk image using either L<genisoimage(1)> or
L<virt-make-fs(1)>.  For genisoimage, use a command like this:

 genisoimage -o extra-packages.iso -R -J -V EXTRA cdcontents/

Create a script that mounts the ISO and sets up the repository.  For
yum, create /tmp/install.sh containing:

 mkdir /tmp/mount
 mount LABEL=EXTRA /tmp/mount
 
 cat <<'EOF' > /etc/yum.repos.d/extra.repo
 [extra]
 name=extra
 baseurl=file:///tmp/mount
 enabled=1
 EOF
 
 yum -y install famousdatabase

For apt, create /tmp/install.sh containing:

 mkdir /tmp/mount
 mount LABEL=EXTRA /tmp/mount
 
 apt-cdrom -d=/tmp/mount add
 apt-get -y install famousdatabase

Use the I<--attach> option to attach the CD / disk image and the
I<--run> option to run the script:

 virt-builder fedora-20 \
   --attach extra-packages.iso \
   --run /tmp/install.sh

=head2 USERS AND PASSWORDS

The I<--root-password> option is used to change the root password
(otherwise a random password is used).  This option takes a password
C<SELECTOR> in one of the following formats:

=over 4

=item B<--root-password> file:FILENAME

Read the root password from C<FILENAME>.  The whole first line
of this file is the replacement password.  Any other lines are
ignored.  You should create the file with mode 0600 to ensure
no one else can read it.

=item B<--root-password> password:PASSWORD

Set the root password to the literal string C<PASSWORD>.

B<Note: this is not secure> since any user on the same machine can
see the cleartext password using L<ps(1)>.

=item B<--root-password> random

Choose a random password, which is printed on stdout.  The password
has approximately 120 bits of randomness.

This is the default.

=item B<--root-password> disabled

The root account password is disabled.  This is like putting C<*>
in the password field.

=item B<--root-password> locked:file:FILENAME

=item B<--root-password> locked:password:PASSWORD

=item B<--root-password> locked:random

The root account is locked, but a password is placed on the
account.  If first unlocked (using C<passwd -u>) then logins will
use the given password.

=item B<--root-password> locked

=item B<--root-password> locked:disabled

The root account is locked I<and> password is disabled.

=back

=head3 Creating user accounts

To create user accounts, use the L<useradd(8)> command with
L<--firstboot-command> like this:

 virt-builder --firstboot-command \
    'useradd -m -p "" rjones ; chage -d 0 rjones'

The above command will create an C<rjones> account with no password,
and force the user to set a password when they first log in.  There
are other ways to manage passwords, see L<useradd(8)> for details.

=head2 KEYBOARD LAYOUT

Because there are so many different ways to set the keyboard layout in
Linux distributions, virt-builder does not yet attempt to have a
simple command line option.  This section describes how to set the
keyboard for some common Linux distributions.

=head3 Keyboard layout with systemd

For distros that use systemd C<localectl>, use a command like this:

 virt-builder fedora-20 \
   --firstboot-command 'localectl set-keymap uk'

See L<localectl(1)> and
L<https://www.happyassassin.net/2013/11/23/keyboard-layouts-in-fedora-20-and-previously/>
for more details.

=head3 Keyboard layout using C</etc/sysconfig/keyboard>

For RHEL E<le> 6, Fedora E<le> 18 and similar, upload or modify the
keyboard configuration file using the I<--upload>, I<--write> or
I<--edit> options.  For example:

 virt-builder centos-6 \
   --edit '/etc/sysconfig/keyboard: s/^KEYTABLE=.*/KEYTABLE="uk"/'

The format of this file can be found documented in many places online.

=head3 Keyboard layout with Debian-derived distros

For Debian-derived distros using C</etc/default/keyboard>, upload or
modify the keyboard file using the I<--upload>, I<--write> or
I<--edit> options.  For example:

 virt-builder debian-7 \
   --edit '/etc/default/keyboard: s/^XKBLAYOUT=.*/XKBLAYOUT="gb"/'

See L<https://wiki.debian.org/Keyboard>.

=head2 LANGUAGE

Most Linux distributions support multiple locale settings so that you
can have guest messages printed in another language such as Russian.

However there is no single setting which controls this, since extra
packages may need to be installed to support console and X fonts, and
keyboard input methods.  The packages required, and their
configuration is highly distro-specific, and it is outside the scope
of virt-builder to do this.

This section contains examples for some common Linux distributions.

=head3 Setting Japanese in Fedora 20

 virt-builder fedora-20 \
   --size 20G \
   --update \
   --install @japanese-support \
   --install @xfce \
   --install xorg-x11-server-Xorg,xorg-x11-drivers,rsyslog \
   --link /usr/lib/systemd/system/graphical.target:/etc/systemd/system/default.target \
   --firstboot-command 'localectl set-locale LANG=ja_JP.utf8' \
   --firstboot-command 'localectl set-keymap jp' \
   --firstboot-command 'systemctl isolate graphical.target'

=head3 Setting Japanese in Debian 7 (Wheezy)

Note that although this enables Japanese in the text console too, it
is unlikely that you will see properly rendered Japanese there.
However Japanese is properly rendered in X applications and terminals.

 pkgs=locales,xfce4,\
 ibus,ibus-anthy,\
 ttf-sazanami-gothic,ttf-sazanami-mincho,\
 fonts-takao-mincho,\
 xfonts-intl-japanese,xfonts-intl-japanese-big,\
 iceweasel-l10n-ja,manpages-ja
 
 virt-builder debian-7 \
   --size 20G \
   --install $pkgs \
   --edit '/etc/inittab: s,^#([1-9].*respawn.*/sbin/getty.*),$1,' \
   --edit '/etc/locale.gen: s,^#\s*ja,ja,' \
   --write '/etc/default/locale:LANG="ja_JP.UTF-8"' \
   --run-command "locale-gen"

=head2 LOG FILE

Scripts and package installation that runs at build time (I<--run>,
I<--run-command>, I<--install>, I<--update>, but I<not> firstboot) is
logged in one of the following locations:

=over 4

=item C</tmp/builder.log>

On Linux, BSD and other guests.

=item C<C:\Temp\builder.log>

On Windows, DOS guests.

=item C</builder.log>

If C</tmp> or C<C:\Temp> is missing.

=back

If you don't want the log file to appear in the final image, then
use the I<--no-logfile> command line option.

=head2 INSTALLATION PROCESS

When you invoke virt-builder, installation proceeds as follows:

=over 4

=item *

The template image is downloaded.

If the template image is present in the cache, the cached version
is used instead.  (See L</CACHING>).

=item *

The template signature is checked.

=item *

The template is uncompressed to a tmp file.

=item *

The template image is resized into the destination, using
L<virt-resize(1)>.

=item *

Extra disks are attached (I<--attach>).

=item *

A new random seed is generated for the guest.

=item *

Guest customization is performed, in the order specified on the
command line.

=item *

SELinux relabelling is done (I<--selinux-relabel>).

=back

=head2 IMPORTING THE DISK IMAGE

=head3 Importing into libvirt

Import the disk image into libvirt using L<virt-install(1)>
I<--import> option.

 virt-install --import \
   --name guest --ram 2048 --disk path=disk.img,format=raw

Notes:

=over 4

=item 1.

You I<must> specify the correct format.  The format is C<raw> unless
you used virt-builder's I<--format> option.

=item 2.

You can run virt-install as root or non-root.  Each works slightly
differently because libvirt manages a different set of virtual
machines for each user.  In particular virt-manager normally shows the
root-owned VMs, whereas Boxes shows the user-owned VMs, and other
tools probably work differently as well.

=back

=head3 Importing into OpenStack

Import the image into Glance (the OpenStack image store) by doing:

 glance image-create --name fedora-20-image --file fedora-20.img \
   --disk-format raw --container-format bare \
   --is-public True

The I<--file> parameter is the virt-builder-generated disk image.  It
should match virt-builder's I<--output> option.  The I<--disk-format>
parameter should match virt-builder's I<--format> option (or C<raw> if
you didn't use that option).  The I<--container-format> should always
be C<bare> since virt-builder doesn't put images into containers.

You can use the S<C<glance image-show I<fedora-20-image>>> command to
display the properties of the image.

To boot up an instance of your image on a Nova compute node, do:

 nova boot fedora-20-server --image fedora-20-image \
   --flavor m1.medium

Use S<C<nova flavor-list>> to list possible machine flavors.  Use
S<C<nova list>> to list running instances.

=head3 Booting directly using qemu or KVM

The qemu command line is not very stable or easy to use, hence libvirt
should be used if possible.  However a command line similar to the
following could be used to boot the virtual machine:

 qemu-system-x86_64 \
   -machine accel=kvm:tcg \
   -cpu host \
   -m 2048 \
   -drive file=disk.img,format=raw,if=virtio

As with libvirt, it is very important that the correct format is
chosen.  It will be C<raw> unless the I<--format> option was used.

=head2 CONFIGURATION MANAGEMENT

=head3 Puppet

To enable the Puppet agent in a guest, install the package, point
the configuration at your Puppetmaster, and ensure the agent runs
at boot.

A typical virt-builder command would be:

 virt-builder fedora-20 \
   --hostname client.example.com \
   --update \
   --install puppet \
   --edit '/etc/puppet/puppet.conf:
       s/^\[agent\]/[agent]\n  server = puppetmaster.example.com/' \
   --run-command 'systemctl enable puppet' \
   --selinux-relabel

The precise instructions vary according to the Linux distro.  For
further information see:
L<http://docs.puppetlabs.com/guides/installation.htm>

=head2 DEBUGGING BUILDS

If virt-builder itself fails, then enable debugging (I<-v>) and report
a bug (see L</BUGS> below).

If virt-builder fails because some script or package it is installing
fails, try using I<--no-delete-on-failure> to preserve the output
file, and continue reading this section.

If virt-builder is successful but the image doesn't work, here are
some things to try:

=over 4

=item Use virt-rescue

Run L<virt-rescue(1)> on the disk image:

 virt-rescue -a disk.img

This gives you a rescue shell.  You can mount the filesystems from the
disk image on C</sysroot> and examine them using ordinary Linux
commands.  You can also chroot into the guest to reinstall the
bootloader.  The virt-rescue man page has a lot more information and
examples.

=item Use guestfish

Run L<guestfish(1)> on the disk image:

 guestfish -a disk.img -i

Use guestfish commands like C<ll /directory> and C<cat /file> to
examine directories and files.

=item Use guestmount

Mount the disk image safely on the host using FUSE and L<guestmount(1)>:

 mkdir /tmp/mp
 guestmount -a disk.img -i /tmp/mp
 cd /tmp/mp

To unmount the disk image do:

 fusermount -u /tmp/mp

=item Add a serial console

If the guest hangs during boot, it can be helpful to add a serial
console to the guest, and direct kernel messages to the serial
console.  Adding the serial console will involve looking at the
documentation for your hypervisor.  To direct kernel messages to the
serial console, add the following on the kernel command line:

 console=tty0 console=ttyS0,115200

=back

=head2 SOURCES OF TEMPLATES

virt-builder reads the available sources from configuration files,
with the I<.conf> extension and located in the following paths:

=over 4

=item *

$XDG_CONFIG_HOME/virt-builder/repos.d/ (C<$XDG_CONFIG_HOME> is
C<$HOME/.config> if not set).

=item *

$XDG_CONFIG_DIRS/virt-builder/repos.d/ (where C<$XDG_CONFIG_DIRS>
means any of the directories in that environment variable, or just C</etc/xdg>
if not set)

=back

Each I<.conf> file in those paths has a simple text format like the
following:

 [libguestfs.org]
 uri=http://libguestfs.org/download/builder/index.asc
 gpgkey=file:///etc/xdg/virt-builder/repos.d/libguestfs.gpg

The part in square brackets is the repository identifier, which is
used as unique identifier.

The following fields can appear:

=over 4

=item C<uri=URI>

The URI of the index file which this repository refers to.

This field is required.

=item C<gpgkey=URI>

This optional field represents the URI (although only I<file://> URIs
are accepted) of the key used to sign the index file.
If not present, the index file referred by I<uri=..> is not signed.

=item C<proxy=MODE>

This optional field specifies the proxy mode, to be used when downloading
the index file of this repository.  The possible values are:

=over 4

=item B<no>, B<off>

No proxy is being used at all, even overriding the system configuration.

=item B<system>

The proxy used is the system one.

=item I<anything else>

Specifies the actual proxy configuration to be used, overriding the system
configuration.

=back

If not present, the assumed value is to respect the proxy settings of the
system (i.e. as if B<system> would be specified).

=back

For serious virt-builder use, you may want to create your own
repository of templates.

=head3 Libguestfs.org repository

Out of the box, virt-builder downloads the file
L<http://libguestfs.org/download/builder/index.asc> which is an index
of available templates plus some information about each one, wrapped
up in a digital signature.  The command C<virt-builder --list> lists
out the information in this index file.

The templates hosted on libguestfs.org were created using shell
scripts, kickstart files and preseed files which can be found in the
libguestfs source tree, in C<builder/website>.

=head3 Setting up the repository

You can set up your own site containing an index file and some
templates, and then point virt-builder at the site by creating a
I<.conf> file pointing to it.

Note that if your index is signed, you will need to properly fill
I<gpgkey=..> in your I<.conf> file, making sure to deploy also the
GPG key file.

 virt-builder --source https://example.com/builder/index.asc \
    --fingerprint 'AAAA BBBB ...' \
    --list

You can host this on any web or FTP server, or a local or network
filesystem.

=head3 Setting up a GPG key

If you don't have a GnuPG key, you will need to set one up.  (Strictly
speaking this is optional, but if your index and template files are
not signed then virt-builder users will have to use the
I<--no-check-signature> flag every time they use virt-builder.)

To create a key, see the GPG manual
L<http://www.gnupg.org/gph/en/manual.html>.

Export your GPG public key:

 gpg --export -a "you@example.com" > pubkey
 
=head3 Create the templates

There are many ways to create the templates.  For example you could
clone existing guests (see L<virt-sysprep(1)>), or you could install a
guest by hand (L<virt-install(1)>).  To see how the templates were
created for virt-builder, look at the scripts in C<builder/website>

For best results when compressing the templates, use the following xz
options (see L<nbdkit-xz-plugin(1)> for further explanation):

 xz --best --block-size=16777216 disk

=head3 Creating and signing the index file

The index file has a simple text format (shown here without the
digital signature):

 [fedora-18]
 name=Fedora® 18
 osinfo=fedora18
 arch=x86_64
 file=fedora-18.xz
 checksum[sha512]=...
 format=raw
 size=6442450944
 compressed_size=148947524
 expand=/dev/sda3
 
 [fedora-19]
 name=Fedora® 19
 osinfo=fedora19
 arch=x86_64
 file=fedora-19.xz
 checksum[sha512]=...
 revision=3
 format=raw
 size=4294967296
 compressed_size=172190964
 expand=/dev/sda3

The part in square brackets is the C<os-version>, which is the same
string that is used on the virt-builder command line to build that OS.

After preparing the C<index> file in the correct format, clearsign it
using the following command:

 gpg --clearsign --armor index

This will create the final file called C<index.asc> which can be
uploaded to the server (and is the I<uri=..> URL).  As noted above,
signing the index file is optional, but recommended.

The following fields can appear:

=over 4

=item C<name=NAME>

The user-friendly name of this template.  This is displayed in the
I<--list> output but is otherwise not significant.

=item C<osinfo=ID>

This optional field maps the operating system to the associated
libosinfo ID.  Virt-builder does not use it (yet).

=item C<arch=ARCH>

The architecture of the operating system installed within the
template. This field is required.

=item C<file=PATH>

The path (relative to the index) of the xz-compressed template.

Note that absolute paths or URIs are B<not> permitted here.  This is
because virt-builder has a "same origin" policy for templates so they
cannot come from other servers.

=item C<sig=PATH>

B<This option is deprecated>.  Use the checksum field instead.

The path (relative to the index) of the GPG detached signature of the
xz file.

Note that absolute paths or URIs are B<not> permitted here.  This is
because virt-builder has a "same origin" policy for templates so they
cannot come from other servers.

The file can be created as follows:

 gpg --detach-sign --armor -o disk.xz.sig disk.xz

=item C<checksum[sha512]=7b882fe9b82eb0fef...>

The SHA-512 checksum of the B<compressed> file is checked after it is
downloaded.  To work out the signature, do:

 sha512sum disk.xz

Note if you use this, you don't need to sign the file, ie. don't use
C<sig>.  This option overrides C<sig>.

=item C<checksum=7b882fe9b82eb0fef...>

C<checksum> is an alias for C<checksum[sha512]>.

If you need to interoperate with virt-builder = 1.24.0 then you have
to use C<checksum> because that version would give a parse error with
square brackets and numbers in the key of a field.  This is fixed in
virt-builder E<ge> 1.24.1.

=item C<revision=N>

The revision is an integer which is used to control the template
cache.  Increasing the revision number causes clients to download the
template again even if they have a copy in the cache.

The revision number is optional.  If omitted it defaults to C<1>.

=item C<format=raw>

=item C<format=qcow2>

Specify the format of the disk image (before it was compressed).  If
not given, the format is autodetected, but generally it is better to
be explicit about the intended format.

Note this is the source format, which is different from the
I<--format> option (requested output format).  Virt-builder does
on-the-fly conversion from the source format to the requested output
format.

=item C<size=NNN>

The virtual size of the image in bytes.  This is the size of the image
when uncompressed.  If using a non-raw format such as qcow2 then it
means the virtual disk size, not the size of the qcow2 file.

This field is required.

Virt-builder also uses this as the minimum size that users can request
via the I<--size> option, or as the default size if there is no
I<--size> option.

=item C<compressed_size=NNN>

The compressed size of the disk image in bytes.  This is just used for
information (when using I<--list --long>).

=item C<expand=/dev/sdaX>

When expanding the image to its final size, instruct L<virt-resize(1)>
to expand the named partition in the guest image to fill up all
available space.  This works like the virt-resize I<--expand> option.

You should usually put the device name of the guest's root filesystem here.

It's a good idea to use this, but not required.  If the field is
omitted then virt-resize will create an extra partition at the end of
the disk to cover the free space, which is much less user-friendly.

=item C<lvexpand=/dev/VolGroup/LogVol>

When expanding the image to its final size, instruct L<virt-resize(1)>
to expand the named logical volume in the guest image to fill up all
available space.  This works like the virt-resize I<--lv-expand> option.

If the guest uses LVM2 you should usually put the LV of the guest's
root filesystem here.  If the guest does not use LVM2 or its root
filesystem is not on an LV, don't use this option.

=item C<notes=NOTES>

Any notes that go with this image, especially notes describing what
packages are in the image, how the image was prepared, and licensing
information.

This information is shown in the I<--notes> and I<--list> I<--long> modes.

You can use multi-line notes here by indenting each new line with at
least one character of whitespace (even on blank lines):

 notes=This image was prepared using
  the following kickstart script:
                                <-- one space at beginning of line
  part /boot --fstype ext3
  ...

=item C<hidden=true>

Using the hidden flag prevents the template from being listed by the
I<--list> option (but it is still installable).  This is used for test
images.

=back

=head3 Running virt-builder against multiple sources

It is possible to use multiple sources with virt-builder.
The recommended way is to deploy I<.conf> files pointing to the
index files. Another way is to specify the sources using
multiple I<--source> and/or I<--fingerprint> options:

 virt-builder \
   --source http://example.com/s1/index.asc \
   --source http://example.com/s2/index.asc

You can provide N or 1 fingerprints.  In the case where you
provide N fingerprints, N = number of sources and there is a 1-1
correspondence between each source and each fingerprint:

 virt-builder \
   --source http://example.com/s1/index.asc --fingerprint '0123 ...' \
   --source http://example.com/s2/index.asc --fingerprint '9876 ...'

In the case where you provide 1 fingerprint, the same fingerprint
is used for all sources.

You C<must> provide at least 1 fingerprint.

=head3 Licensing of templates

You should be aware of the licensing of images that you distribute.
For open source guests, provide a link to the source code in the
C<notes> field and comply with other requirements (eg. around
trademarks).

=head3 Formal specification of the index file

The index file format has a formal specification defined by the flex
scanner and bison parser used to parse the file.  This can be found in
the following files in the libguestfs source tree:

 builder/index-scan.l
 builder/index-parse.y

A tool called L<virt-index-validate(1)> is available to validate the
index file to ensure it is correct.

Note that the parser and tool can work on either the signed or
unsigned index file (ie. C<index> or C<index.asc>).

The index is always encoded in UTF-8.

=head2 CACHING

=head3 Caching templates

Since the templates are usually very large, downloaded templates are
cached in the user's home directory.

The location of the cache is C<$XDG_CACHE_HOME/virt-builder/> or
C<$HOME/.cache/virt-builder>.

You can print out information about the cache directory, including
which guests are currently cached, by doing:

 virt-builder --print-cache

The cache can be deleted if you want to save space by doing:

 virt-builder --delete-cache

You can download all (current) templates to the local cache by doing:

 virt-builder --cache-all-templates

To disable the template cache, use I<--no-cache>.

Only templates are cached.  The index and detached digital signatures
are not cached.

=head3 Caching packages

Virt-builder uses L<curl(1)> to download files and it also uses the
current C<http_proxy> (etc) settings when installing packages
(I<--install>, I<--update>).

You may therefore want to set those environment variables in order to
maximize the amount of local caching that happens.  See
L</ENVIRONMENT VARIABLES> and L<curl(1)>.

=head3 Local mirrors

To increase both speed and reliability of installing packages, you can
set up a local mirror of the target distribution, and point the guest
package manager at that.

Because of the order in which each phase of installation happens, you
cannot use I<--write> (to point the package manager at a repo)
followed by I<--install> (to install from that repo).  The I<--write>
and I<--install> steps run in the opposite order, regardless of their
order on the command line.  You have to do this using I<--run-command>
instead of I<--install>.

=head4 Using a local mirror with Fedora

To install a Fedora guest using a local mirror:

 virt-builder fedora 20 \
   --edit '/etc/yum.repos.d/fedora.repo:
       s{.*baseurl=.*}{baseurl=http://example.com/mirror/};
       s{.*metalink=.*}{};
   ' \
   --edit '/etc/yum.repos.d/fedora-updates.repo:
       s{.*baseurl=.*}{baseurl=http://example.com/mirror-updates/};
       s{.*metalink=.*}{};
   ' \
   --run-command 'yum -y update' \
   --run-command 'yum -y install pkg1 pkg2 ...'

=head4 Using a local mirror with Debian

Assuming that you are using C<apt-proxy> to mirror the repository, you
should create a new C<sources.list> file to point to your proxy (see
L<https://help.ubuntu.com/community/AptProxy>) and then do:

 virt-builder fedora 20 \
   --upload sources.list:/etc/apt/sources.list \
   --run-command 'apt-get -y update' \
   --run-command 'apt-get -y install pkg1 pkg2 ...'

=head2 DIGITAL SIGNATURES

Virt-builder uses GNU Privacy Guard (GnuPG or gpg) to verify that the
index and templates have not been tampered with.

The source points to an index file, which is optionally signed.

Virt-builder downloads the index and checks that the signature is
valid and the signer's fingerprint matches the specified fingerprint
(ie. the one specified in I<gpgkey=..> in the I<.conf>, or with
I<--fingerprint>, in that order).

For checking against the built-in public key/fingerprint, this
requires importing the public key into the user's local gpg keyring
(that's just the way that gpg works).

When a template is downloaded, its signature is checked in the same
way.

Although the signatures are optional, if you don't have them then
virt-builder users will have to use I<--no-check-signature> on the
command line.  This prevents an attacker from replacing the signed
index file with an unsigned index file and having virt-builder
silently work without checking the signature.  In any case it is
highly recommended that you always create signed index and templates.

=head2 ARCHITECTURE

Virt-builder can build a guest for any architecture no matter what the
host architecture is.  For example an x86-64 guest on an ARM host.

However certain options may not work, specifically options
that require running commands in the guest during the build process:
I<--install>, I<--update>, I<--run>, I<--run-command>.  You may need
to replace these with their firstboot-equivalents.

An x86-64 host building 32 bit i686 guests should work without any
special steps.

=head2 SECURITY

Virt-builder does not need to run as root (in fact, should not be run
as root), and doesn't use setuid, C<sudo> or any similar mechanism.

I<--install>, I<--update>, I<--run> and I<--run-command> are
implemented using an appliance (a small virtual machine) so these
commands do not run on the host.  If you are using the libguestfs
libvirt backend and have SELinux enabled then the virtual machine is
additionally encapsulated in an SELinux container (sVirt).

However these options will have access to the host's network and since
the template may contain untrusted code, the code might try to access
host network resources which it should not.  You can use
I<--no-network> to prevent this.

Firstboot commands run in the context of the guest when it is booted,
and so the security of your hypervisor / cloud should be considered.

Virt-builder injects a random seed into every guest which it builds.
This helps to ensure that TCP sequence numbers, UUIDs, ssh host keys
etc are truly random when the guest boots.

You should check digital signatures and not ignore any signing errors.

=head2 CLONES

If you wish to create many new guests of the same type, it is tempting
to run virt-builder once and then copy the output file.  You should
B<not> do this.  You should run virt-builder once for each new guest
you need.

The reason is that each clone needs to have (at least) a separate
random seed, and possibly other unique features (such as filesystem
UUIDs) in future versions of virt-builder.

Another thing you should I<not> do is to boot the guest, then clone
the booted disk image.  The reason is that some guests create unique
machine IDs, SSH host keys and so on at first boot, and you would not
want clones to have duplicate identities.

See also: L<virt-sysprep(1)>.

=head2 PERFORMANCE

The most important aspect of getting good performance is caching.
Templates gets downloaded into the cache the first time they are used,
or if you use the I<--cache-all-templates> option.  See L</CACHING>
above for further information.

Packages required for the I<--install> and I<--update> options are
downloaded using the host network connection.  Setting the
C<http_proxy>, C<https_proxy> and C<ftp_proxy> environment variables
to point to a local web cache may ensure they only need to be
downloaded once.  You can also try using a local package repository,
although this can be complex to set up and varies according to which
Linux distro you are trying to install.

=head3 Using I<--no-sync>

Use I<--no-sync>.  However read the caveats in the L</OPTIONS> section
above, since this can cause disk corruption if not used correctly.

=head3 Skipping virt-resize

Virt-builder can skip the virt-resize step under certain conditions.
This makes virt-builder much faster.  The conditions are:

=over 4

=item *

the output must be a regular file (not a block device), B<and>

=item *

the user did B<not> use the I<--size> option, B<and>

=item *

the output format is the same as the template format (usually raw).

=back

=head3 pxzcat

Virt-builder uses an internal implementation of pxzcat (parallel
xzcat) if liblzma was found at build time.  If liblzma was not found
at build time, regular C<xzcat> is used which is single-threaded.

=head3 User-Mode Linux

You can use virt-builder with the User-Mode Linux (UML) backend.  This
may be faster when running virt-builder inside a virtual machine
(eg. in the cloud).

To enable the UML backend, read the instructions in
L<guestfs(3)/USER-MODE LINUX BACKEND>.

Currently you have to use the I<--no-network> option.  This should be
fixed in a future version.

The qcow2 output format is not supported by UML.  You can only create
raw-format guests.

=head2 SELINUX

Guests which use SELinux (such as Fedora and Red Hat Enterprise Linux)
require that each file has a correct SELinux label.

Virt-builder does not know how to give new files a label, so there are
two possible strategies it can use to ensure correct labelling:

=over 4

=item Using I<--selinux-relabel>

This runs L<fixfiles(8)> just before finalizing the guest, which sets
SELinux labels correctly in the disk image.

Sometimes fixfiles is not possible during installation, in which case
this option falls back on:

=item Touching C</.autorelabel>

Guest templates may already contain a file called C</.autorelabel>, or
it is touched if I<--selinux-relabel> cannot run fixfiles.

For guests that use SELinux, this causes fixfiles to run at first
boot.  Guests will reboot themselves once the first time you use them,
which is normal and harmless.

=back

=head1 MACHINE READABLE OUTPUT

The I<--machine-readable> option can be used to make the output more
machine friendly, which is useful when calling virt-builder from other
programs, GUIs etc.

Use the option on its own to query the capabilities of the
virt-builder binary.  Typical output looks like this:

 $ virt-builder --machine-readable
 virt-builder
 arch
 config-file
 customize
 json-list
 pxzcat

A list of features is printed, one per line, and the program exits
with status 0.

=head1 ENVIRONMENT VARIABLES

For other environment variables which affect all libguestfs programs,
see L<guestfs(3)/ENVIRONMENT VARIABLES>.

=over 4

=item C<http_proxy>

=item C<https_proxy>

=item C<no_proxy>

Set the proxy for downloads.  These environment variables (and more)
are actually interpreted by L<curl(1)>, not virt-builder.

=item C<HOME>

Used to determine the location of the template cache, and the location
of the user' sources.  See L</CACHING> and L</SOURCES OF TEMPLATES>.

=item C<XDG_CACHE_HOME>

Used to determine the location of the template cache.  See L</CACHING>.

=item C<XDG_CONFIG_HOME>

Used to determine the location of the user' sources.  See
L</SOURCES OF TEMPLATES>.

=item C<XDG_CONFIG_DIRS>

Used to determine the location of the system sources.  See
L</SOURCES OF TEMPLATES>.

=back

=head1 EXIT STATUS

This program returns 0 if successful, or non-zero if there was an
error.

=head1 SEE ALSO

L<guestfs(3)>,
L<guestfish(1)>,
L<guestmount(1)>,
L<virt-copy-out(1)>,
L<virt-customize(1)>,
L<virt-install(1)>,
L<virt-rescue(1)>,
L<virt-resize(1)>,
L<virt-sysprep(1)>,
L<oz-install(1)>,
L<gpg(1)>,
L<curl(1)>,
L<virt-make-fs(1)>,
L<genisoimage(1)>,
L<fixfiles(8)>,
L<http://libguestfs.org/>.

=head1 AUTHOR

Richard W.M. Jones L<http://people.redhat.com/~rjones/>

=head1 COPYRIGHT

Copyright (C) 2013 Red Hat Inc.