Commit Graph

597 Commits

Author SHA1 Message Date
Richard W.M. Jones
ffdcb7cfeb build: Require <caml/unixsupport.h>
Require <caml/unixsupport.h> (an OCaml header file) and remove
alternate defintions of ‘Nothing’ and ‘unix_error’ which are defined
in this header file.

We require OCaml >= 3.11 which has this header file, so there is no
need to test for it or provide alternative definitions.

Thanks: Pino Toscano.
2017-07-18 18:42:23 +01:00
Richard W.M. Jones
2afe62eb60 builder: templates: Prevent script from being run as root.
Or with LIBVIRT_DEFAULT_URI=qemu:///system which is the same
thing.

In either case the images are created as user qemu.qemu and then
aren't readable or modifiable by later parts of the script.
2017-07-13 12:07:35 +01:00
Pino Toscano
178a20f6c3 builder: fix paths to mlstdutils & mlutils
Followup/fix of commit 61d4891ef4.
2017-07-12 09:52:46 +02:00
Richard W.M. Jones
0a73032120 builder: Add Fedora 26 for aarch64, armv7l, i686, ppc64, ppc64le.
All the secondary architectures except s390x, which we cannot handle
because qemu doesn't emulate that architecture sufficiently yet.
2017-07-11 15:17:58 +01:00
Richard W.M. Jones
5aaa91b524 builder: templates: Use full path for nvram file.
Works around a virt-install or libvirt bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1469491
2017-07-11 13:02:55 +01:00
Richard W.M. Jones
904578e8b9 builder: templates: Use dl.fedoraproject.org for secondary architectures.
Avoid redirecting to incomplete mirrors.
2017-07-11 12:00:58 +01:00
Richard W.M. Jones
48874066b9 builder: Add fedora-26 (x86_64) image. 2017-07-11 10:29:57 +01:00
Richard W.M. Jones
e6c89f9631 utils: Rename ‘guestfs-internal-frontend.h’ to ‘guestfs-utils.h’.
The reason it's not just ‘utils.h’ is because Pino is worried that we
might pick up /usr/include/utils.h from a rogue library.
2017-07-10 17:01:59 +01:00
Richard W.M. Jones
8be72935dc mllib, v2v: Split out OCaml utils bindings ‘common/mlutils’.
Create a module ‘C_utils’ containing functions like ‘drive_name’ and
‘shell_unquote’ which come from the C utilities.

The new directory ‘common/mlutils’ also contains the ‘Unix_utils’
wrappers around POSIX functions missing from the OCaml stdlib.
2017-07-10 17:01:59 +01:00
Richard W.M. Jones
61d4891ef4 mllib: Split ‘Common_utils’ into ‘Std_utils’ + ‘Common_utils’.
The new module ‘Std_utils’ contains only functions which are pure
OCaml and depend only on the OCaml stdlib.  Therefore these functions
may be used by the generator.

The new module is moved to ‘common/mlstdutils’.

This also removes the "<stdlib>" hack, and the code which copied the
library around.

Also ‘Guestfs_config’, ‘Libdir’ and ‘StringMap’ modules are moved
since these are essentially the same.

The bulk of this change is just updating files which use
‘open Common_utils’ to add ‘open Std_utils’ where necessary.
2017-07-10 17:01:59 +01:00
Pino Toscano
28c375701d builder: templates: pass empty proxy to d-i when not set
When there is no proxy set, pass an empty string to the Debian
Installer.  This way, the installer will not ask for a proxy.
2017-07-10 17:56:13 +02:00
Pino Toscano
941545bde6 builder: templates: save NVRAM before removing guest
When installing guests that need NVRAM variables, the cleanup of the
guest with `virsh undefine` will remove that file too, which is not what
we want.  Instead, compress the NVRAM file right before the cleanup,
to ensure we have it.

Also, fix the filename for it, removing the double "-nvram" suffixes.
2017-07-10 17:56:13 +02:00
Pino Toscano
a67346b2fe builder: templates: improve debian.preseed
- set the mirror to deb.debian.org, which is the official redirector, so
  generated images will use the closest mirror depending on their
  location
- automake the task selection, using the choice we want
- do a full upgrade of the distro, so the template is already up-to-date
- install grub on the default device, which should also be the only one
  in the automated installation
2017-07-10 17:56:13 +02:00
Pino Toscano
eb830e89dc builder: templates: generalize debian location
Add an helper function to translate our architecture to the Debian
naming, and use it to get the location for all the architectures.
2017-07-10 17:56:13 +02:00
Pino Toscano
2ff6938935 builder: templates: update Debian preseed from Stretch
Update our debian.preseed file from a Stretch example file, leaving all
the customization done already.
2017-07-10 17:56:13 +02:00
Richard W.M. Jones
4cea914d24 builder: make-template: Combine both os_variant_of_os functions into one.
Cosmetic only change, no functional change.
2017-07-10 16:49:32 +01:00
Richard W.M. Jones
dd4e95c636 builder: Add debian-9 template. 2017-06-27 10:04:45 +01:00
Richard W.M. Jones
669739a16c builder: make-template: Remove trailing \n in Debian index-fragment.
Caused the following error when running make-template:

  syntax error at line 22: syntax error
  virt-index-validate: ‘debian-9.index-fragment’ could not be validated, see errors above
2017-06-27 10:04:45 +01:00
Richard W.M. Jones
0e7c0877be builder: Support Debian 9 ("Stretch") in make-template. 2017-06-27 10:04:45 +01:00
Richard W.M. Jones
ab1620b8d0 builder: Add cirros-0.3.5. 2017-06-04 09:39:39 +01:00
Richard W.M. Jones
d606edfba4 builder: templates: Ensure a stable sort when creating index file. 2017-06-04 09:39:39 +01:00
Richard W.M. Jones
3ca0110adc builder: Fix Unicode quotes in index file.
Fixes commit 126ef5d082.
2017-06-04 09:39:23 +01:00
Richard W.M. Jones
c0fdeb36c7 daemon: Move the useful 'is_zero' function into common code.
This is largely a simple refactoring, but it combines another
definition of this function from virt-builder which had a slightly
different prototype.
2017-04-19 17:30:58 +01:00
Pino Toscano
56c4387b38 builder: various improvements to the documentation
- convert the Fedora 20 examples to Fedora 25
- convert the yum examples to dnf
- convert the Debian 7 examples to Debian 8
- mention that --core updates all the installed packages, not "core"
  ones
- convert i386 to i686 for the Fedora 32bit examples
- mention that autogenerated root passwords are printed on stdout
- mention --get-kernel is deprecated in favour of virt-get-kernel
- update URL to the Puppet installation documentation
- edit the repository index documentation to make it clearer
  non-compressed templates are supported
- remove old wrong paragraph about customization specified in command
  line not processed in the same order
- use --install instead of --run-command '$pkgmgr install ...'
- other minor changes
2017-04-11 22:54:52 +01:00
Richard W.M. Jones
069eb3d3df builder: Allow GnuPG v2 or v1 to be used (RHBZ#1438939).
All of the GPG commands we run are compatible with either v1 or v2, so
this simply involves looking for either gpg2 or gpg binaries at
runtime.
2017-04-06 16:29:45 +01:00
Richard W.M. Jones
ee206d7ba8 Use Unicode single quotes ‘’ in place of short single quoted strings throughout.
Only in end-user messages and documentation.  This change was done
mostly mechanically using the Perl script attached below.

I also changed don't -> don’t etc and made some other simple fixes.

See also: https://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html

----------
 #!/usr/bin/perl -w

use strict;
use Locale::PO;

my $re = qr{'([-\w%.,=?*/]+)'};

my %files = ();

foreach my $filename ("po/libguestfs.pot", "po-docs/libguestfs-docs.pot") {
    my $poref = Locale::PO->load_file_asarray($filename);

    foreach my $po (@$poref) {
        if ($po->msgid =~ $re) {
            my @refs = split /\s+/, $po->reference;
            foreach my $ref (@refs) {
                my ($file, $lineno) = split /:/, $ref, 2;
                $file =~ s{^\.\./}{};
                if (exists $files{$file}) {
                    push @{$files{$file}}, $lineno;
                } else {
                    $files{$file} = [$lineno];
                }
            }
        }
    }
}

foreach my $file (sort keys %files) {
    unless (-w $file) {
        warn "warning: $file is probably generated\n"; # have to edit generator
        next;
    }
    my @lines = sort { $a <=> $b } @{$files{$file}};

    #print "editing $file at lines ", join (", ", @lines), " ...\n";
    open FILE, "<$file" or die "$file: $!";
    my @all = ();
    push @all, $_ while <FILE>;
    close FILE;

    my $ext = $file;
    $ext =~ s/^.*\.//;

    foreach (@lines) {
        # Don't mess with verbatim sections in POD files.
        next if $ext eq "pod" && $all[$_-1] =~ m/^ /;

        unless ($all[$_-1] =~ $re) {
            # this can happen for multi-line strings, have to edit it
            # by hand
            warn "warning: $file:$_ does not contain expected content\n";
            next;
        }
        $all[$_-1] =~ s/$re/‘$1’/g;
    }

    rename "$file", "$file.bak";
    open FILE, ">$file" or die "$file: $!";
    print FILE $_ for @all;
    close FILE;
    my $mode = (stat ("$file.bak"))[2];
    chmod ($mode & 0777, "$file");
}
2017-04-04 18:47:37 +01:00
Richard W.M. Jones
8c58b62f9d Replace possessive ASCII apostrophe ('s) with Unicode apostrophe (’s).
Only replaced in end-user messages and documentation, not in code,
comments, or anything else that's not end-user visible.

See: https://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html
2017-03-31 10:38:04 +01:00
Richard W.M. Jones
4b6a482bc8 mllib: Combine small Unix library call bindings into Unix_utils module.
Concatenate six small modules containing Unix/POSIX library call
bindings into a single module called Unix_utils.

The previous modules and the library functions bound were:

 - Dev_t:   makedev(3), major(3), minor(3)
 - Exit:    _exit(2)
 - Fnmatch: fnmatch(3)
 - Fsync:   sync(2)
 - Mkdtemp: mkdtemp(3)
 - StatVFS: statvfs(2)
2017-02-22 14:55:12 +00:00
Richard W.M. Jones
a1b3d8d04b tests: Extend $TEST_FUNCTIONS with predefined functions for skipping tests etc.
Apply this change across all the shell scripts containing tests.

Additionally this defines the environment variables $abs_srcdir,
$abs_builddir, $top_srcdir, $top_builddir, $abs_top_srcdir and
$abs_top_builddir which can now be used throughout test scripts.
2017-02-21 17:23:22 +00:00
Richard W.M. Jones
0b3a5a0b00 GCC 7: Add __attribute__((noreturn)) to some usage functions which call exit.
This happens with GCC 7.0.1.  The errors were all of the form:

qemu-speed-test.c: In function 'main':
qemu-speed-test.c:153:7: error: this statement may fall through [-Werror=implicit-fallthrough=]
       usage (EXIT_SUCCESS);
       ^~~~~~~~~~~~~~~~~~~~
qemu-speed-test.c:155:5: note: here
     default:
     ^~~~~~~
2017-02-14 15:25:25 +00:00
Richard W.M. Jones
4ab07a9949 builder: templates: Call sync after running virt-install.
I don't think this will really make any difference.  However there is
some puzzling disk corruption in the recently built RHEL 5 & 6 i686
images, and I want to make absolutely sure they are not caused by an
unsynchronized disk image.
2017-02-09 17:52:45 +00:00
Richard W.M. Jones
eeb3f42d1f builder: templates: Add 32 bit (i386) templates for RHEL 5 and 6. 2017-02-09 13:54:46 +00:00
Richard W.M. Jones
23878acbbd builder: tests: Disable valgrind for test with multi-line parameters.
Breaks libtool, see:
https://bugzilla.redhat.com/show_bug.cgi?id=1420301

This updates commit 736a8465a6.
2017-02-08 11:58:34 +00:00
Richard W.M. Jones
f161c9ea57 Rename src/ to lib/ 2017-01-26 15:05:46 +00:00
Richard W.M. Jones
b53cec584d lib: Move utilities to new directory common/utils.
Just code motion.

This commit makes it clearer what is a utility and what is part of the
library.  It also makes it clear that we should rename:

  guestfs-internal-frontend.h -> utils.h
  guestfs-internal-frontend-cleanups.h -> cleanups.h (?)

but this commit does not make that change.
2017-01-26 15:05:46 +00:00
Cédric Bosdonnat
b9f510328a builder: extract Yajl helper functions to yajl.ml
Extract the yajl object_get_* helpers in the Yajl module since this
could be useful for any Yajl user code.
2017-01-05 09:25:11 +01:00
Pino Toscano
55bf7de97c Update copyright dates for 2017
Run the following command over the source:

  perl -pi.bak -e 's/(20[01][0-9])-2016/$1-2017/g' `git ls-files`

(Thanks Rich for the perl snippet, as used in past years.)
2017-01-03 16:48:21 +01:00
Richard W.M. Jones
25ed84c64d builder: Fixed CentOS 7.3 image.
The previous one had apparent filesystem corruption.

Fixes commit b16627d679.
2016-12-12 22:14:47 +00:00
Richard W.M. Jones
b16627d679 builder: Add CentOS 7.3 template.
Release notice:
https://lists.centos.org/pipermail/centos-announce/2016-December/022172.html
2016-12-12 20:14:20 +00:00
Richard W.M. Jones
c540676f5e Remove most instances of OCaml warning 52.
See:
http://caml.inria.fr/pub/docs/manual-ocaml/comp.html#s:comp-warnings
2016-12-09 09:31:25 +00:00
Richard W.M. Jones
07fb30b161 mllib: Add quote function to Common_utils module.
Remove multiple places where we let quote = Filename.quote
2016-12-09 09:31:25 +00:00
Richard W.M. Jones
601e05508b builder: Refactor command line parsing in make-template script.
This refactors command line parsing into a parse_cmdline function.
The function uses the Arg module to parse the command line instead of
ad hoc parsing.

This also enforces use of the ./run script to run the program, since
otherwise you can end up with mixed versions of the OCaml bindings and
the C library.

This updates commit 65a0570385.
2016-12-02 17:04:05 +00:00
Richard W.M. Jones
e4b73d67f2 builder: Update Fedora 25 and RHEL 7.3 ppc64 and ppc64le images.
Rebuild these images with the new make-template.ml script.  This fixes
the lack of virtio-scsi driver and broken initramfs.

These images all use LVM because without that both Fedora and RHEL put
the root filesystem on an extended partition (/dev/sda5) which
virt-resize cannot handle.

(Note the RHEL images are not public.)
2016-11-30 14:21:13 +00:00
Richard W.M. Jones
7a9f9b7cdc builder: ppc64: Enable virtio-scsi in templates.
Workaround for dracut bug RHBZ#1396217.
2016-11-30 14:21:13 +00:00
Richard W.M. Jones
65a0570385 builder: Rearrange how template-building scripts work.
Create a new directory (builder/template).  Integrate all of the
scripts into a single program, so that templates are generated more
consistently.

This also changes how the index file is generated.  The script now
generates the index file fragment and saves it under version control,
and then generates the final index file by concatenating these.
(Previously the index was written by hand which was tedious and
error-prone.)

The new script also saves the generated kickstart under version
control so it can be referenced later.
2016-11-30 14:20:20 +00:00
Richard W.M. Jones
06dc0e52eb Revert "builder: ppc64, ppc64le: Force add virtio-scsi driver."
It turns out that adding the virtio-scsi driver does not help
to fix the dracut problem.

This partially reverts commit f766c84c39,
keeping the change to index.
2016-11-24 18:46:26 +00:00
Richard W.M. Jones
f766c84c39 builder: ppc64, ppc64le: Force add virtio-scsi driver.
Works around dracut bug RHBZ#1396217.

Also update new Fedora ppc64 & ppc64le images.
2016-11-24 17:25:33 +00:00
Richard W.M. Jones
0cd54ff45b builder: Add Fedora 25 templates. 2016-11-24 10:35:45 +00:00
Richard W.M. Jones
0ae56ad711 builder: fedora-armv7l: Increase RAM for installation.
In Fedora 25 the guest runs out of RAM when loading the
Anaconda initramfs.
2016-11-24 09:52:17 +00:00
Richard W.M. Jones
f45d9d3cd2 builder: fedora-ppc64le: Require POWER8 CPU.
This is the same as commit 4919f4f740
for Fedora.
2016-11-23 22:01:57 +00:00