Replace strange $TEST_FUNCTIONS variable/expansion thing with
something more like what we use in nbdkit, a simple tests/functions.sh
script that gets sourced into each test script.
Update the common submodule to get:
commit 8137d47d0e654065391151eb275e3b64f230f6f5
Author: Richard W.M. Jones <rjones@redhat.com>
Date: Thu Feb 13 11:13:55 2025 +0000
mlcustomize, mltools: Replace $TEST_FUNCTIONS
TEST_FUNCTIONS is being removed from libguestfs and guestfs-tools (it
was removed from virt-v2v a while back). Make the same adjustment in
the common submodule.
(and some other commits which are not relevant to libguestfs)
Various errors like this:
In function ‘test_fuse’,
inlined from ‘main’ at test-fuse.c:133:11:
test-fuse.c:274:5: error: argument 1 null where non-null expected [-Werror=nonnull]
274 | fclose (fp);
| ^~~~~~~~~~~
In file included from test-fuse.c:26:
/usr/include/stdio.h: In function ‘main’:
/usr/include/stdio.h:183:12: note: in a call to function ‘fclose’ declared ‘nonnull’
183 | extern int fclose (FILE *__stream) __nonnull ((1));
| ^~~~~~
Run this command across the source:
perl -pi.bak -e 's/(20[012][0-9])-20[12][012]/$1-2023/g' `git ls-files`
and remove changes to po{,-docs}/*.po{,t} (these will be regenerated
later when we run 'make dist').
Previously we noted in a comment that stat("hello.txt") is cached (and
not called again), so the test of the link count failed. Something
has changed, possibly in the kernel, but it results in even more
aggressive caching so that an earlier, similar test also fails in the
same way. I checked by enabling debugging that the stat call doesn't
result in guestfs_lstatns being called, and the old value for the
statbuf was being returned.
Call the C-language helper key_store_requires_network() in guestfish and
guestmount.
(Short log for the "common" submodule, commit range
35467027f657..af6cb55bc58a:
Laszlo Ersek (12):
options: fix UUID comparison logic bug in get_keys()
mltools/tools_utils: remove unused function "key_store_to_cli"
mltools/tools_utils: allow multiple "--key" options for OCaml tools too
options: replace NULL-termination with number-of-elements in get_keys()
options: wrap each passphrase from get_keys() into a struct
options: add back-end for LUKS decryption with Clevis+Tang
options: introduce selector type "key_clevis"
options: generalize "--key" selector parsing for C-language utilities
mltools/tools_utils-c: handle internal type error with abort()
mltools/tools_utils: generalize "--key" selector parsing for OCaml utils
options, mltools/tools_utils: parse "--key ID:clevis" options
options, mltools/tools_utils: add helper for network dependency
).
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1809453
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Message-Id: <20220630122048.19335-4-lersek@redhat.com>
User-Mode Linux was an alternative hypervisor that could run the
appliance, instead of using qemu. It had many limitations including
lack of network, and UML support in Linux has been semi-broken for a
long time. It was also slower than KVM on baremeal in general and had
various corner cases which were much slower including the emulated
serial port which made bulk uploads and downloads painful. Also of
course it lacked qemu-specific features like qcow2 or any
network-backed disk, so many disk images could not be opened this way.
This was never supported in RHEL.
See-also: https://bugzilla.redhat.com/1144197
This experimental feature allowed you (in theory) to connect to an
existing instance of the libguestfs daemon. (Again, in theory) it
allowed you to attach to running guests. This didn't work well in
practice. If you want to do this, install qemu-guest-agent inside
your guest instead.
This also disables the --live options in guestfish and guestmount.
(The option now prints an error).
This was never supported in RHEL.
The daemon tests relied on this connection method to perform tests on
a bare daemon, so this removes those tests. They were not especially
valuable.
See-also: https://bugzilla.redhat.com/798980
As part of our efforts to clean up and simplify libguestfs, removing
gnulib deletes a large dependency that we mostly no longer use and
causes problems for new users trying to build the library from source.
A few modules from gnulib are still used (under a compatible license)
and these are copied into gnulib/lib/
This brings libguestfs into line with other projects which have a
separate include/ directory for the public header.
It's also the case that <guestfs.h> has never particularly belonged in
the lib/ subdirectory. Some tools add -Ilib/ but they only need
<guestfs.h> and not any other headers from that directory, and
separating out the public header allows us to clean those up. This is
certainly the case for examples, and some language bindings and some
tests.
In future I'm hopeful we can use this as the basis to tease out other
dependencies, as a prelude to separating them out from the repo.
Add a simple way to do not even provide prototypes of deprecated
functions in the C library: this way, users (like our tools) can build
against the library making sure to not use any deprecated function, not
even when compiler deprecation warnings are disabled.
Add it to the majority of our tools/internal libraries, and make sure
that it is not defined when building the API bridges of our bindings.
The proper file that should be included is `sys/xattr.h` as that comes from
`glibc` and not `attr/xattr.h` which ships with the `attr` utility.
New enough `attr` utility (at least 2.4.48 in my case) even includes a #warning
in `attr/xattr.h` for projects that still have this mistake in the code.
The majority of the tools have already options (--echo-keys &
--keys-from-stdin) to deal with LUKS credentials, although there is no
way to automatically provide credentials. --keys-from-stdin is
suboptimal, because it is a usable solution only when there is just one
device to open, and no other input passed via stdin to the tool (like
the commands for guestfish).
To overcome this limitation, introduce a new --key option in tools:
* --key /dev/device:file:/filename/with/key
* --key /dev/device:string:the-actual-key
this way it is possible to pass all the credentials needed for the
specific devices to open, with no risk of conflict with stdin, and also
in a secure way (when using the "file" way).
On the technical side: this adds a new "key_store" API for the C tools,
making sure it is used only when needed. Partially mirror it also for
the OCaml tools, although there will be a conversion to the C API
because the decryption helpers used are in the common C parts.
The device name is only used by guestfish (when using the -N option to
prepare drives). We constructed the device name very naively,
basically ‘sprintf ("/dev/sd%c", next_drive)’.
This stores the device index instead, and only constructs the device
name in guestfish. Also the device name is constructed properly using
guestfs_int_drive_name so it can cope with #drives > 26.
As a side effect of this change we can remove the extra parameter of
the add_drives macro.
Thanks: Pino Toscano
Only in end-user messages and documentation. This change was done
mostly mechanically using the Perl script attached below.
I also changed don't -> don’t etc and made some other simple fixes.
See also: https://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html
----------
#!/usr/bin/perl -w
use strict;
use Locale::PO;
my $re = qr{'([-\w%.,=?*/]+)'};
my %files = ();
foreach my $filename ("po/libguestfs.pot", "po-docs/libguestfs-docs.pot") {
my $poref = Locale::PO->load_file_asarray($filename);
foreach my $po (@$poref) {
if ($po->msgid =~ $re) {
my @refs = split /\s+/, $po->reference;
foreach my $ref (@refs) {
my ($file, $lineno) = split /:/, $ref, 2;
$file =~ s{^\.\./}{};
if (exists $files{$file}) {
push @{$files{$file}}, $lineno;
} else {
$files{$file} = [$lineno];
}
}
}
}
}
foreach my $file (sort keys %files) {
unless (-w $file) {
warn "warning: $file is probably generated\n"; # have to edit generator
next;
}
my @lines = sort { $a <=> $b } @{$files{$file}};
#print "editing $file at lines ", join (", ", @lines), " ...\n";
open FILE, "<$file" or die "$file: $!";
my @all = ();
push @all, $_ while <FILE>;
close FILE;
my $ext = $file;
$ext =~ s/^.*\.//;
foreach (@lines) {
# Don't mess with verbatim sections in POD files.
next if $ext eq "pod" && $all[$_-1] =~ m/^ /;
unless ($all[$_-1] =~ $re) {
# this can happen for multi-line strings, have to edit it
# by hand
warn "warning: $file:$_ does not contain expected content\n";
next;
}
$all[$_-1] =~ s/$re/‘$1’/g;
}
rename "$file", "$file.bak";
open FILE, ">$file" or die "$file: $!";
print FILE $_ for @all;
close FILE;
my $mode = (stat ("$file.bak"))[2];
chmod ($mode & 0777, "$file");
}
Apply this change across all the shell scripts containing tests.
Additionally this defines the environment variables $abs_srcdir,
$abs_builddir, $top_srcdir, $top_builddir, $abs_top_srcdir and
$abs_top_builddir which can now be used throughout test scripts.
This is mostly code motion but:
(1) I had to remove the compile-time COMPILING_GUESTFISH and
COMPILING_VIRT_RESCUE macros and replace them with runtime constants
and checks.
(2) I moved the fish/config.c file into this library.
Just code motion.
This commit makes it clearer what is a utility and what is part of the
library. It also makes it clear that we should rename:
guestfs-internal-frontend.h -> utils.h
guestfs-internal-frontend-cleanups.h -> cleanups.h (?)
but this commit does not make that change.
Run the following command over the source:
perl -pi.bak -e 's/(20[01][0-9])-2016/$1-2017/g' `git ls-files`
(Thanks Rich for the perl snippet, as used in past years.)
Create a new libfishcommon convenience static library to build just once
(instead of 12 times!) the majority of the guestfish sources used in the
rest of the C tools (mostly for command line stuff, inspection, and
mount).
The notable exceptions not using libfishcommon are guestfish itself, and
virt-rescue: both need to build at least one of the common sources using
additional CPPFLAGS.
Make use of the recently added 'getprogname' module in gnulib: replace
our guestfs_int_program_name with the getprogname() provided by the
module, since it does the same thing, and in a portable way.
As consequence of the above, use gnulib in a couple of tests that use
getprogname().
Since guestfs_int_program_name is gone, drop the configure checks
associated with it.
By adding common CLEANFILES and DISTCLEANFILES variables to
common-rules.mk, we can remove these from most other Makefiles, and
also clean files more consistently.
Note that bin_PROGRAMS are already cleaned by 'make clean', so I
removed cases where these were unnecessarily added to CLEANFILES.
By moving these two functions out of the common options parsing code,
it means we don't need to depend on all the other machinery of options
parsing, such as the global variables ("verbose"), libconfig, etc.
podcheck.pl is run as part of the tests to perform various checks on
the documentation and the tool.
Currently we check only that the documented options matches the
options that the tool implements and vice versa. This commit would
also allow us (in future) to check --help, --long-options,
--short-options, --version output.
This commit includes scripts to run the tests and various fixes to the
manual pages to ensure that the tests pass.
For guestfish, guestmount, remove '?' from short options. Currently
those tools don't process -?, so I believe these are erroneous:
$ guestfish -\?
Try `guestfish --help' for more information.
For virt-format, the -c, -d and -q options are removed. These options
just give errors because they appear in the short options list but not
in the case statement.
When possible, make the disk image format explicit when invoking tools
or using add-drive. This avoids warnings from qemu about the unspecified
format for the image, and also makes qemu slightly faster (skipping the
disk image probing).
Tests checking the image probing are not touched.
This changes also:
- old-style invocations of tools (`$tool $filename`) into new style
(`$tool -a $filename`)
- add-drive-ro/add-drive-with-if guestfish commands into add/add-drive
with explicit readonly/iface arguments
There should be no change in the tests results.
Improve the error messages produced by C-based tools in case of issues
with the command line options:
- explicitly mention to use -a/-d (and -A/-D in virt-diff)
- when extra arguments are found, mention the correct way to pass
options to certain command line switches (like --format)
- in virt-inspector, give a cleaner error message when neither -i nor
any -m is specified
In all the cases, keep the extra notice to use 'TOOL --help' to get more
help with it.
Like with the previous commit, this replaces instances of:
if (something_bad) {
fprintf (stderr, "%s: error message\n", guestfs_int_program_name);
exit (EXIT_FAILURE);
}
with:
if (something_bad)
error (EXIT_FAILURE, 0, "error message");
(except in a few cases were errno was incorrectly being ignored, in
which case I have fixed that).
It's slightly more complex than the previous commit because we must be
careful to:
- Remove the program name (since error(3) prints it).
- Remove any trailing \n character from the message.
Candidates for replacement were found using:
pcregrep --buffer-size 10M -M '\bfprintf\b.*\n.*\bexit\b' `git ls-files`
Wherever we had code which did:
if (something_bad) {
perror (...);
exit (EXIT_FAILURE);
}
replace this with use of the error(3) function:
if (something_bad)
error (EXIT_FAILURE, errno, ...);
The error(3) function is supplied by glibc, or by gnulib on platforms
which don't have it, and is much more flexible than perror(3). Since
we already use error(3), there seems to be no downside to mandating it
everywhere.
Note there is one nasty catch with error(3): error (EXIT_SUCCESS, ...)
does *not* exit! This is also the reason why error(3) cannot be
marked as __attribute__((noreturn)).
Because the examples can't use gnulib, I did not change them.
To search for multiline patterns of the above form, pcregrep -M turns
out to be very useful:
pcregrep --buffer-size 10M -M '\bperror\b.*\n.*\bexit\b' `git ls-files`
