Create a module ‘C_utils’ containing functions like ‘drive_name’ and
‘shell_unquote’ which come from the C utilities.
The new directory ‘common/mlutils’ also contains the ‘Unix_utils’
wrappers around POSIX functions missing from the OCaml stdlib.
The new module ‘Std_utils’ contains only functions which are pure
OCaml and depend only on the OCaml stdlib. Therefore these functions
may be used by the generator.
The new module is moved to ‘common/mlstdutils’.
This also removes the "<stdlib>" hack, and the code which copied the
library around.
Also ‘Guestfs_config’, ‘Libdir’ and ‘StringMap’ modules are moved
since these are essentially the same.
The bulk of this change is just updating files which use
‘open Common_utils’ to add ‘open Std_utils’ where necessary.
Only in end-user messages and documentation. This change was done
mostly mechanically using the Perl script attached below.
I also changed don't -> don’t etc and made some other simple fixes.
See also: https://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html
----------
#!/usr/bin/perl -w
use strict;
use Locale::PO;
my $re = qr{'([-\w%.,=?*/]+)'};
my %files = ();
foreach my $filename ("po/libguestfs.pot", "po-docs/libguestfs-docs.pot") {
my $poref = Locale::PO->load_file_asarray($filename);
foreach my $po (@$poref) {
if ($po->msgid =~ $re) {
my @refs = split /\s+/, $po->reference;
foreach my $ref (@refs) {
my ($file, $lineno) = split /:/, $ref, 2;
$file =~ s{^\.\./}{};
if (exists $files{$file}) {
push @{$files{$file}}, $lineno;
} else {
$files{$file} = [$lineno];
}
}
}
}
}
foreach my $file (sort keys %files) {
unless (-w $file) {
warn "warning: $file is probably generated\n"; # have to edit generator
next;
}
my @lines = sort { $a <=> $b } @{$files{$file}};
#print "editing $file at lines ", join (", ", @lines), " ...\n";
open FILE, "<$file" or die "$file: $!";
my @all = ();
push @all, $_ while <FILE>;
close FILE;
my $ext = $file;
$ext =~ s/^.*\.//;
foreach (@lines) {
# Don't mess with verbatim sections in POD files.
next if $ext eq "pod" && $all[$_-1] =~ m/^ /;
unless ($all[$_-1] =~ $re) {
# this can happen for multi-line strings, have to edit it
# by hand
warn "warning: $file:$_ does not contain expected content\n";
next;
}
$all[$_-1] =~ s/$re/‘$1’/g;
}
rename "$file", "$file.bak";
open FILE, ">$file" or die "$file: $!";
print FILE $_ for @all;
close FILE;
my $mode = (stat ("$file.bak"))[2];
chmod ($mode & 0777, "$file");
}
Export a new environment variable for extra-data.d scripts, that
contains a YAML representation of the elements in use.
This is a new addition in d-i-b 2.0.
d-i-b 2.0 is written in Python, and thus it passes the information about
the interpreter used for it (sys.executable) to the scripts that need
it; this mechanism replaces the old discovery of what is the default
Python interpreter in the system.
Since we are not Python-based, look for 'python' and use it as default
interpreter, with the --python command line option to set a different
one.
The only place where it is used so far are extra-data.d scripts; in case
it will be used also in other out-of-chroot phases, a different solution
will be needed.
Output the get_image_element_array function like d-i-b does, together
with the eval stuff needed to make it work.
get_image_element_array is basically a bash array, with the keys are the
used elements, and the values are the paths of each element; it is a new
addition in d-i-b 2.0.
This is used so far only for the extra-data.d phase.
It looks like Unix.O_CLOEXEC does not exist in OCaml < 4, so resort to
manually using Unix.set_close_on_exec on the filedesc after opening it.
Thanks to: John Sucaet and Richard W.M. Jones.
Concatenate six small modules containing Unix/POSIX library call
bindings into a single module called Unix_utils.
The previous modules and the library functions bound were:
- Dev_t: makedev(3), major(3), minor(3)
- Exit: _exit(2)
- Fnmatch: fnmatch(3)
- Fsync: sync(2)
- Mkdtemp: mkdtemp(3)
- StatVFS: statvfs(2)
Apply this change across all the shell scripts containing tests.
Additionally this defines the environment variables $abs_srcdir,
$abs_builddir, $top_srcdir, $top_builddir, $abs_top_srcdir and
$abs_top_builddir which can now be used throughout test scripts.
Sadly, there are elements (hello "gentoo"!) that, during the cleanup.d
phase, wipe out almost everything in the /tmp of the guest, including
the /tmp/aux where virt-dib mounts the auxiliary data. Since that
removal excludes things starting with "in_target" (mostly to avoid
wiping the "in_target.d" that disk-image-create itself sets up, then
rename our "aux" subdirectory to "in_target.aux" to work it around.
Hopefully, the "gentoo" element will be fixed upstream:
https://review.openstack.org/#/c/436101/
Commit 8ee51ee396 re-scans for the
available scripts when running certain phases; OTOH, some of them may be
missing, usually due to the lack of scripts for that phase in the
selected set of elements.
Indeed, if there is no directory for a phase, safely raise Not_found so
the case is handled as if the phase was missing in the final_hooks
Hashtbl.
The real sudo does it as well, and leaving them when preserving the
environment (-E) maybe breaks the applications, as e.g. chroot will have
a TMPDIR path pointing outside of it.
When extracting the content of the guest as tar, save also the extended
attributes and the SELinux attributes. This makes sure guests exported
as tar, tgz, and as docker image will work fine afterwards.
This is what disk-image-create does as well.
Implement the "squash" output format, i.e. a squashfs compressed
filesystem.
This was implemented in diskimage-builder upstream as
commit 9d13084c4183b63587e1f5e4b03395a8df6538f6.
Recently, diskimage-builder moved the log cleanup from the 'base'
element to disk-image-create proper; the cleanup done is:
- truncate any file in /var/log
- remove *.log files in /root
This was implemented in diskimage-builder upstream as
commit 022d93ee822e71245af52c4cf8f8a8e82f599af3.
Scripts (especially root.d and extra-data.d) may add new scripts to the
existing set during their execution: in this case, re-read the list of
available scripts, so we do not miss any new addition.
It looks like scripts (in particular in the extra-data.d phase) may
in-flight add new scripts to the existing ones. As first step to
handle this situation, change the path of hooks passed for extra-data.d
scripts: instead of the local temporary directory, use the in-guest
location that is visible as result of the local fuse-based mount of the
guest.
Source the "die" script, part of the diskimage-builder library, when
running hooks that run outside the guest chroot. This is what
disk-image-create does, and scripts expect to use the "die" function
without sourcing the "die" script containing it.
Query awk for the list of environment variables, instead of trying to
extract the list from the output of `env`: the old approach breaks when
any of the environment variable contains more than one line.
Implement the "tgz" output format, i.e. "tar" compressed as gzip.
This was implemented in diskimage-builder upstream as
commit da41ee6012b064aa6901c871a1104a3a3933117e.
Implement a system similar to the operations in virt-sysprep, so each
output format has its own file and attributes (such as command line
arguments).
The result is that there is no more need to spread everywhere the job
required by each output format, such as checking for tools, or hooking
at the right point.
Make sure they are handled as options for the actual mkfs driver,
instead of mkfs itself. Also, improve the documentation of
--mkfs-options accordingly.
This was fixed in diskimage-builder upstream as
commit 5e2330d89c6c0e55b270464abea7e99a4d3246ad.
Add a new option to create files with the MD5 and SHA512 checksums of
all the disk image types produced.
This was implemented in diskimage-builder upstream as
commit 2ea5feca5c4b64867ac327736edfb20408f8840e and
commit 22952b7ea0543bb4f446752976d1d8ba232b021a.
Improve the error message that is shown when trying to selecting an
element, and another element provides it.
This was implemented in diskimage-builder upstream as
commit 452f7b8d5aaa6e85e3b01b970cd9cbc842b34b86.
Just code motion.
This commit makes it clearer what is a utility and what is part of the
library. It also makes it clear that we should rename:
guestfs-internal-frontend.h -> utils.h
guestfs-internal-frontend-cleanups.h -> cleanups.h (?)
but this commit does not make that change.
For a very long time we have maintained two sets of utility functions,
in mllib/common_utils.ml and generator/utils.ml. This changes things
so that the same set of utility functions can be shared with both
directories.
It's not possible to use common_utils.ml directly in the generator
because it provides several functions that use modules outside the
OCaml stdlib. Therefore we add some lightweight post-processing which
extracts the functions using only the stdlib:
(*<stdlib>*)
...
(*</stdlib>*)
and creates generator/common_utils.ml and generator/common_utils.mli
from that. The effect is we only need to write utility functions
once.
As with other tools, we still have generator-specific utility
functions in generator/utils.ml.
Also in this change:
- Use String.uppercase_ascii and String.lowercase_ascii in place
of deprecated String.uppercase/String.lowercase.
- Implement String.capitalize_ascii to replace deprecated
String.capitalize.
- Move isspace, isdigit, isxdigit functions to Char module.
By adding common CLEANFILES and DISTCLEANFILES variables to
common-rules.mk, we can remove these from most other Makefiles, and
also clean files more consistently.
Note that bin_PROGRAMS are already cleaned by 'make clean', so I
removed cases where these were unnecessarily added to CLEANFILES.
Allow to specify the format of the helper drive, to avoid relying on the
format autodetection. This is the same approach used also in all the
other tools.
Export FS_TYPE for the elements, so they can know what is the filesystem
created.
This was implemented in diskimage-builder upstream as
commit be521bdec6233530cc952be246a66f1957b0dd58.
Implement the "docker" output format, to import an image into docker.
This was implemented in diskimage-builder upstream as
commit 1187f9d5df6d6eddba478ac75d3834391480755f and
commit edc06a20e57bdf0a9a03949b54395008d73520dc.
Instead of running them before lanching the appliance with the disks and
then uploading the result after root.d hooks run, mount the root in the
local temporary directory using FUSE and then run the hooks on the
guest. Other than being closer to what diskimage-builder does, it also
avoids issues with the extra-data.d scripts assuming to find things
already, and we don't error out while trying to unpack files on the
guest.
Since virt-dib requires FUSE now, build it only if FUSE is supported.
podcheck.pl is run as part of the tests to perform various checks on
the documentation and the tool.
Currently we check only that the documented options matches the
options that the tool implements and vice versa. This commit would
also allow us (in future) to check --help, --long-options,
--short-options, --version output.
This commit includes scripts to run the tests and various fixes to the
manual pages to ensure that the tests pass.
Instead of linking with individual objects, which is very tedious,
build a proper library and link the other tools with it.
This doesn't make the resulting binaries any larger.
Since the function uses the StringMap module, I also had to
move this to mllib.
The function is renamed and the 'if verbose ()' part is moved into the
function, but apart from that, it's all just code motion.
