test-parallel-mount-local reliably fails when my machines are
under load (like running the test suite in parallel). The
issue is /usr/libexec/gvfs-udisks2-volume-monitor fiddling with
the fuse mounts, preventing them from being unmounted.
./fuse/guestunmount defaults to 5 unmount retries with increasing
time backoff, but apparently that isn't enough. Bumping it to
use 10 retries makes things more reliable for me.
Signed-off-by: Cole Robinson <crobinso@redhat.com>
In commit 6d32773e81 ("tests: Run the tests in parallel.", 2021-03-18),
the working directory relative to which "test-parallel-mount-local" would
be launched (by the test machinery) changed from "tests/mount-local" to
just "tests".
While the relative pathname of the "guestunmount" executable was updated
inside "test-parallel-mount-local" accordingly, the relative pathname of
the FUSE client ("test-parallel-mount-local" itself, just invoked with
"--test") was not. This issue guarantees that the exec call fails in the
child, and so the test case always hangs.
Because we had removed "mount-local" from the end of the working
directory, prepend it now to the relative pathname of the FUSE client
executable.
Fixes: 6d32773e81
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20210902135124.15191-3-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
Each worker thread of "test-parallel-mount-local" performs the following
steps (among others):
(1) it starts an appliance dedicated to that thread, using a private
scratch disk image,
(2) exports a dedicated FUSE mount point on the host, exposing the file
system on the appliance's disk,
(3) launches a child process for manipulating the particular FUSE mount
point on the host,
(4) enters a FUSE request processing loop, translating requests between
the host kernel (coming in via the FUSE mount point) and the
appliance.
Items to note:
- The child process from step (3) consists of a single thread of execution
(see fork() in POSIX): a duplicate of the parent process's respective
worker thread.
- The child process from step (3) blocks on any FUSE mount point access on
the host until the worker thread in the parent process starts processing
FUSE requests, in step (4).
- The FUSE request processing in step (4), in the worker thread living in
the parent process, terminates if and only if the child process unmounts
the FUSE mount point originating from (2).
Should the exec call in step (3) fail for any reason, the child currently
jumps to the "error" label. This is wrong: under the error label, we call
guestfs_close() on the appliance -- but the appliance is owned by the
parent process's worker thread, not the child. What happens is that the
child kills off the appliance while the parent's worker thread is in the
FUSE request processing loop (4).
The "error" label was never meant to be reached by the child process -- if
exec fails for any reason, exit the child immediately. The parent will
remain in the FUSE request processing loop (4) forever, but no state will
be corrupted. For example, using another (interactive) session on the
host, the FUSE mount points can be interacted with, and if all of them are
manually unmounted, the FUSE request processing (4) completes in every
worker thread.
This patch does not fix the primary issue with
"test-parallel-mount-local", but removes "chaos" from the symptoms. The
next patch will fix the actual regression in this test case.
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20210902135124.15191-2-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
Before this change the tests ran in about 12m34 and afterwards in
about 6m20, although the real change is more dramatic if you only run
tests from the tests/ subdirectory (as language tests still run serially).
This breaks valgrinding for now, which I intend to fix properly later.
This brings libguestfs into line with other projects which have a
separate include/ directory for the public header.
It's also the case that <guestfs.h> has never particularly belonged in
the lib/ subdirectory. Some tools add -Ilib/ but they only need
<guestfs.h> and not any other headers from that directory, and
separating out the public header allows us to clean those up. This is
certainly the case for examples, and some language bindings and some
tests.
In future I'm hopeful we can use this as the basis to tease out other
dependencies, as a prelude to separating them out from the repo.
Add a simple way to do not even provide prototypes of deprecated
functions in the C library: this way, users (like our tools) can build
against the library making sure to not use any deprecated function, not
even when compiler deprecation warnings are disabled.
Add it to the majority of our tools/internal libraries, and make sure
that it is not defined when building the API bridges of our bindings.
Just code motion.
This commit makes it clearer what is a utility and what is part of the
library. It also makes it clear that we should rename:
guestfs-internal-frontend.h -> utils.h
guestfs-internal-frontend-cleanups.h -> cleanups.h (?)
but this commit does not make that change.
Run the following command over the source:
perl -pi.bak -e 's/(20[01][0-9])-2016/$1-2017/g' `git ls-files`
(Thanks Rich for the perl snippet, as used in past years.)
Make use of the recently added 'getprogname' module in gnulib: replace
our guestfs_int_program_name with the getprogname() provided by the
module, since it does the same thing, and in a portable way.
As consequence of the above, use gnulib in a couple of tests that use
getprogname().
Since guestfs_int_program_name is gone, drop the configure checks
associated with it.
Because of previous automated commits, such as changing 'guestfs___'
-> 'guestfs_int_', several function calls no longer lined up with
their parameters, and some lines were too long.
The bulk of this commit was done using emacs batch mode and the
technique described here:
http://www.cslab.pepperdine.edu/warford/BatchIndentationEmacs.html
The changes suggested by emacs were then reviewed by hand.
Because this 'feature' is broken (since 2013):
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=13928
we have to replace all instances of $(top_srcdir) in *_SOURCES lines
with a relative path. According to what I read, this shouldn't break
split builds, but I didn't test it.
The only things automake moans about now are:
* Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/\${ <-- HERE ([^ \t=:+{}]+)}/ at /usr/bin/automake line 3936.
- This is another bug in automake
* autoreconf: configure.ac: AM_GNU_GETTEXT is used, but not AM_GNU_GETTEXT_VERSION
- This is a feature, not a bug.
libguestfs has used double and triple underscores in identifiers.
These aren't valid for global names in C++.
The first step is to replace all guestfs___* (3 underscores) with
guestfs_int_*. We've used guestfs_int_* elsewhere already as a prefix
for internal identifiers.
This is an entirely mechanical change done using:
git ls-files | xargs perl -pi.bak -e 's/guestfs___/guestfs_int_/g'
Reference: http://stackoverflow.com/a/228797
The gnulib 'error' module uses 'program_name'. On some platforms --
but not Linux / glibc -- it references it as:
extern char *program_name;
This means when you compile libguestfs on non-glibc (eg. Mac OS X)
gnulib requires 'program_name' as an external string reference, which
we don't provide.
This change doesn't define this string reference for gnulib, but it
does change the name of the macro we use to avoid conflicts if we
eventually need to export 'program_name' as a string.
Thanks: Margaret Lewicka
Replaces code such as:
fd = open "test1.img"
ftruncate fd, size
close fd
g.add_drive "test1.img"
with the shorter and simpler:
g.add_drive_scratch size
It's simpler to use the glibc 'program_invocation_short_name(3)'
feature, and fall back to a generic solution. Also remove risky
assignments to argv[0].
This adds a common utility function (guestfs___exit_status_to_string)
and a common error function (guestfs___external_command_failed), and
uses them all over the library and tools when converting exit status
in error messages etc.
This mini-library runs the 'free -m' command and greps the output to
estimate the max. number of libguestfs appliances we could run in
parallel in the remaining free memory.
The second parameter passed into getline(3) is the size of the
allocated buffer, *NOT* the length of the returned line. This can be
confusing, so rename this parameter as 'allocsize' consistently
throughout the code.
This is just code motion.
These configure flags enable code profiling (with gprof) and code
coverage (with gcov) respectively.
Although this is a nice idea, it's not currently very useful.
Libtool mangles filenames in such a way that gcov cannot locate its
datafiles.
Profiling is of dubious utility with libguestfs which is not CPU-bound
and relies extensively on running external programs (oprofile-like
system profiling that took into account libguestfs + qemu or
libguestfs + qemu + the appliance + filesystem tools *would* be
useful).
Also neither flag will help in capturing data from the appliance.
'make extra-tests' was a monolithic set of tests that did all sorts of
things: valgrind, tests over local guests, tests with upstream qemu,
tests with upstream libvirt, tests with the appliance attach method.
This made it hard to perform individual tests, eg. just valgrind
testing. It was also hard to maintain because the tests were not
located in the same directories as the programs and sometimes
duplicated tests that were run elsewhere.
This commit splits up 'make extra-tests' into 5 separate targets:
make check-valgrind # run a subset of tests under valgrind
make check-valgrind-local-guests # test under valgrind with local guests
make check-with-appliance # test with attach-method == appliance
make check-with-upstream-qemu # test with an alternate/upstream qemu
make check-with-upstream-libvirt # test with an alternate/upstream libvirt
(You can also still run 'make extra-tests' which is now simply
a rule that runs the above 5 targets in order).
This replaces everything that was in the tests/extra directory,
so that has now gone.
libguestfs fails to build with --enable-fuse on openSuSE 11.4 and
earlier because the included fuse version does not include
libulockmgr.so. configure already used pkgconfig to retrieve the correct
CFLAGS, so there is no need to hardcode -lulockmgr.
With this change the build succeeds again.
Signed-off-by: Olaf Hering <olaf@aepfle.de>
