This gnulib feature abstracts away threads, locks and TLS, and also
allowed libguestfs to be linked with or without pthread. However
since pthread these days is part of glibc and so every program is
using pthread, and we want to get rid of gnulib as a dependency, just
use pthread directly.
Note this requires libvirt >= 7.1.0 which was only released in March 2021.
With an older libvirt you will see this error:
Original error from libvirt: unsupported configuration: Invalid mode attribute 'maximum' [code=67 int1=-1]
In theory we could check if this is supported by looking at the
libvirt capabilities and fall back, but this commit does not do that,
in the expectation that most people will be using the default backend
(direct) and on Fedora/RHEL we will add an explicit minimum version
dependency to the package.
qemu support has been around quite a bit longer (at least since 2017).
Fixes: commit 30f74f38bd
QEMU has a newish feature (from about 2017 / qemu 2.9) called -cpu max
which is supposed to select the best CPU, ideal for libguestfs.
After this change, on x86-64:
KVM TCG
Direct -cpu max -cpu max
(non-libvirt)
Libvirt <cpu mode="host-passthrough"> <cpu mode="host-model">
<model fallback="allow"/> <model fallback="allow"/>
</cpu> </cpu>
Thanks: Daniel Berrangé
Previously this was in common/utils. However it is not used anywhere
else, and guestfs-tools wants to remove gnulib dependencies, so move
this to libguestfs.
Bug originally reported here by trysis:
https://stackoverflow.com/questions/64273334/test-x-in-mounted-filesystem
If the user is root then we override normally access controls in FUSE,
see https://bugzilla.redhat.com/show_bug.cgi?id=1106548.
However this causes test -x to mark all files as executable. We
shouldn't let root execute any file, only ones which have the 'x' bit
set. Therefore this narrows the fix in bug 1106548 so it only applies
to read and write bits.
To test this I created a disk with guestfish which had an executable
and a non-executable file:
$ guestfish -N fs -m /dev/sda1
><fs> touch /file1
><fs> touch /file2
><fs> chmod 0755 /file1
><fs> ll /
total 24
drwxr-xr-x 3 root root 4096 Oct 12 14:04 .
drwxr-xr-x 19 root root 4096 Oct 12 14:04 ..
-rwxr-xr-x 1 root root 0 Oct 12 14:04 file1
-rw-r--r-- 1 root root 0 Oct 12 14:04 file2
drwx------ 2 root root 16384 Oct 12 14:04 lost+found
I then mounted and tested it as non-root:
$ guestmount -a test1.img -m /dev/sda1 /tmp/mnt -v -x
$ ls -l /tmp/mnt
total 16
-rwxr-xr-x. 1 root root 0 Oct 12 15:04 file1
-rw-r--r--. 1 root root 0 Oct 12 15:04 file2
drwx------. 2 root root 16384 Oct 12 15:04 lost+found
$ test -x /tmp/mnt/file1; echo $?
0
$ test -x /tmp/mnt/file2; echo $?
1
and as root:
$ sudo guestmount -a test1.img -m /dev/sda1 /tmp/mnt -v -x
$ test -x /tmp/mnt/file1; echo $?
0
$ test -x /tmp/mnt/file2; echo $?
0
In the debug output for non-root we can see the difference:
libguestfs: /file1: testing access mask X_OK: caller UID:GID = 1000:1000, file UID:GID = 0:0, file mode = 100755, result = OK
libguestfs: /file2: testing access mask X_OK: caller UID:GID = 1000:1000, file UID:GID = 0:0, file mode = 100644, result = EACCESS
and for root:
libguestfs: /file1: testing access mask X_OK: caller UID:GID = 0:0, file UID:GID = 0:0, file mode = 100755, result = OK
libguestfs: /file2: testing access mask X_OK: caller UID:GID = 0:0, file UID:GID = 0:0, file mode = 100644, result = OK
After this commit the root output changes to this (ie. same decision
as non-root):
libguestfs: /file1: testing access mask X_OK: caller UID:GID = 0:0, file UID:GID = 0:0, file mode = 100755, result = OK
libguestfs: /file2: testing access mask X_OK: caller UID:GID = 0:0, file UID:GID = 0:0, file mode = 100644, result = EACCESS
When guestfs_lvm_canonical_lv_name was called with a /dev/dm* or
/dev/mapper* name which was not an LV then a noisy error would be
printed. This would typically have happened with encrypted disks, and
now happens very noticably when inspecting Windows BitLocker-
encrypted guests.
This commit hides this error in all cases, although it is still logged
to debug. See comment and the thread below for detailed rationale.
https://www.redhat.com/archives/libguestfs/2020-October/thread.html#00055
This commit deprecates luks-open/luks-open-ro/luks-close for the more
generic sounding names cryptsetup-open/cryptsetup-close, which also
correspond directly to the cryptsetup commands.
The optional cryptsetup-open readonly flag is used to replace the
functionality of luks-open-ro.
The optional cryptsetup-open crypttype parameter can be used to select
the type (corresponding to cryptsetup open --type), which allows us to
open BitLocker-encrypted disks with no extra effort. As a convenience
the crypttype parameter may be omitted, and libguestfs will use a
heuristic (based on vfs-type output) to try to determine the correct
type to use.
The deprecated functions and the new functions are all (re-)written in
OCaml.
There is no new test here, unfortunately. It would be nice to test
Windows BitLocker support in this new API, however the Linux tools do
not support creating BitLocker disks, and while it is possible to
create one under Windows, the smallest compressed disk I could create
is 37M because of a mixture of the minimum support size for BitLocker
disks and the fact that encrypted parts of NTFS cannot be compressed.
Also synchronise with common module.
This brings libguestfs into line with other projects which have a
separate include/ directory for the public header.
It's also the case that <guestfs.h> has never particularly belonged in
the lib/ subdirectory. Some tools add -Ilib/ but they only need
<guestfs.h> and not any other headers from that directory, and
separating out the public header allows us to clean those up. This is
certainly the case for examples, and some language bindings and some
tests.
In future I'm hopeful we can use this as the basis to tease out other
dependencies, as a prelude to separating them out from the repo.
For the appliance of the QCOW2 format, the function get_root_uuid()
fails to get the UUID of the disk image.
In this case, let us read the first 256k bytes of the disk image with
the 'qemu-img dd' command. Then pass the read block to the 'file'
command.
Suggested-by: Denis V. Lunev <den@openvz.org>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Argon2 is the default LUKS Password-Based Key Derivation Function
(PBKDF) for some new guests such as RHEL 8.2 and Fedora. It is
designed to be "memory hard", meaning that by design it requires large
amounts of memory, making it expensive to brute-force. Unfortunately
the default for guests which had more than a few GB of RAM at install
time is to require about 1 GB of RAM to decrypt the block device,
which is considerably larger than the default available in the
libguestfs appliance.
To make it possible to open these encrypted disks we need to make the
appliance larger. This could be done as a one-off, and the current
workaround is simply to set LIBGUESTFS_MEMSIZE=2048 or a similar
amount. However since we don't know in advance whether we could be
dealing with an encrypted disk, partition, etc. or what PBKDF it uses,
the only way to deal with this in all circumstances is to increase the
default memsize. This commit increases it quite a lot (768 -> 1280)
which is unfortunate.
Note as there is some confusion on this point: Since libguestfs does
not attempt to decrypt disks in parallel, you only need ~ 1GB in
total, not per encrypted disk.
For a reproducer, see:
https://bugzilla.redhat.com/show_bug.cgi?id=1837765#c14
At the moment it is empty, so probably it does not exist. Remove it to
avoid adding spurious content to the pkg-config file in case that
variable will get a value in the future.
We use a similar trick to libvirt to allow external C programs that
use libguestfs to be compiled against the built (but not installed)
libguestfs with:
../libguestfs/run ./configure
make
What actually happens is we have a second pkg-config file
(lib/local/libguestfs.pc) which points to the locally built
libguestfs. The ./run script sets up PKG_CONFIG_PATH to point to this
directory. Assuming that ./configure is using pkg-config/pkgconf and
not some other half-baked solution it will pick up the libguestfs.pc
file from here which will set CFLAGS and LIBS appropriately.
Appliance device names are not reliable since the kernel no longer
enumerates virtio-scsi devices serially. Instead get the UUID of the
appliance and pass this as the parameter.
Note this requires supermin >= 5.1.18 (from around July 2017).
Linux from around 5.6 now enumerates individual disks in any order
(whereas previously it enumerated only drivers in parallel). This
means that /dev/sdX ordering is no longer stable - in particular we
cannot be sure that /dev/sda inside the guest is the first disk that
was attached to the appliance, /dev/sdb the second disk and so on.
However we can still use SCSI PCI device numbering as found in
/dev/disk/by-path. Use this to translate device names in and out of
the appliance.
Thanks: Vitaly Kuznetsov, Paolo Bonzini, Dan Berrangé.
In the guestfs_disk_create API we have traditionally allowed you to
set backingfile without setting backingformat. The meaning of this is
to let qemu autodetect the backing format when opening the overlay
disk.
However libvirt >= 6.0 refuses to even pass such disks to qemu (see
https://bugzilla.redhat.com/show_bug.cgi?id=1798148).
For this reason, move the autodetection earlier and make it explicit.
We now autodetect the format of the backing disk at the time of
creation of the overlay, and set that as the backing format in the
overlay disk itself, allowing libvirt to open the disk later.
This function doesn't work reliably with the proposed change to device
name translation. The reason is that strings returned by
Devsparts.list_devices contained translated names, so their indexes
did not correspond to the untranslated names used outside the
appliance..
We can avoid this and make the function much simpler and faster by
implementing it on the library side instead.
Nowadays there are hard drives and operating systems which support
"4K native" sector size. In this mode physical and logical block size
exposed to the operating system is equal to 4096 bytes.
GPT partition table (as a known example) being created in this mode will
place GPT header at LBA1 which is 4096 bytes. libguetfs is unable to
recognize partition table on such physical block devices or disk images.
The reason is that libguestfs appliance will look for a GPT header at
LBA1 which is seen at 512 byte offset.
In order to fix the issue we need a way to provide correct logical block
size for attached disks. Fortunately QEMU and libvirt already provides
a way to specify physical/logical block size per disk basis.
After discussion in a mailing list we agreed that physical block size is
rarely used and is not so important. Thus both physical and logical
block size will be set to the same value.
In this patch one more optional parameter 'blocksize' is added
to add_drive_opts API method. Valid values are 512 and 4096.
add_drive_scratch has the same optional parameter for a consistency and
testing purpose.
add-domain and add_libvirt_dom will pass logical_block_size value from
libvirt XML to add_drive_opts method.
Enhance the UEFI firmware lookup function with the information on the
libvirt firmware autoselection, allowing it to return a value to use for
the appliance.
At the moment no firmware is selected this way, so there is no behaviour
change.
If it is not possible to detect the distribution of a Linux OS, do not
propose "unknownX.Y" (where X is the major version number, and Y the
minor) as short osinfo ID. Just return "unknown" instead.
Remove gnulib modules that provide stuff clearly not used within
libguestfs (library, daemon, and C tools). Among directly and
indirectly modules used previous (and now no more), they are:
cycle-check
d-ino
dev-ino
dup3
dup3-tests
fcntl-safer
fcntl-safer-tests
fdopendir
fdopendir-tests
filevercmp
filevercmp-tests
ftell
ftell-tests
ftello
ftello-tests
fts
getaddrinfo
getaddrinfo-tests
getcwd
getcwd-tests
gnu-make
hostent
i-ring
i-ring-tests
iconv
iconv-tests
inet_ntop
inet_ntop-tests
isatty
isatty-tests
openat-safer
openat-safer-tests
opendirat
ptsname_r
ptsname_r-tests
read-file
read-file-tests
rewinddir
servent
ttyname_r
ttyname_r-tests
xgetcwd
Some of the removed modules are still used pulled indirectly as
dependency of other modules. There should be no behaviour change on
recent Linux distros, although older distros were not tested (adding
a module back is easy, anyway).
Remove accordingly unused automake variables, and ignored files.
Previously, is_custom_hv() used to compare the QEMU executable found
during configure to the hypervisor set to check whether it is a custom
one; however, the QEMU found at configure time can be different than
what libvirt was configured with.
This fixes the libvirt backend when libguestfs is configured with a
different QEMU, that now will be specified as emulator overriding the
libvirt one.
Return the right osinfo short IDs for some rolling Linux distributions,
such as Arch Linux, Gentoo, and Void Linux. Their IDs were recently
added to osinfo-db.
