Currently the guestfs_readdir() API can not list long directories, due to
it sending back the whole directory listing in a single guestfs protocol
response, which is limited to GUESTFS_MESSAGE_MAX (approx. 4MB) in size.
Introduce the "internal_readdir" action, for transferring the directory
listing from the daemon to the library through a FileOut parameter.
Rewrite guestfs_readdir() on top of this new internal function:
- The new "internal_readdir" action is a daemon action. Do not repurpose
the "readdir" proc_nr (138) for "internal_readdir", as some distros ship
the binary appliance to their users, and reusing the proc_nr could
create a mismatch between library & appliance with obscure symptoms.
Replace the old proc_nr (138) with a new proc_nr (511) instead; a
mismatch would then produce a clear error message. Assume the new action
will first be released in libguestfs-1.48.2.
- Turn "readdir" from a daemon action into a non-daemon one. Call the
daemon action guestfs_internal_readdir() manually, receive the FileOut
parameter into a temp file, then deserialize the dirents array from the
temp file.
This patch sneakily fixes an independent bug, too. In the pre-patch
do_readdir() function [daemon/readdir.c], when readdir() returns NULL, we
don't distinguish "end of directory stream" from "readdir() failed". This
rewrite fixes this problem -- I didn't see much value separating out the
fix for the original do_readdir().
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1674392
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20220502085601.15012-2-lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
User-Mode Linux was an alternative hypervisor that could run the
appliance, instead of using qemu. It had many limitations including
lack of network, and UML support in Linux has been semi-broken for a
long time. It was also slower than KVM on baremeal in general and had
various corner cases which were much slower including the emulated
serial port which made bulk uploads and downloads painful. Also of
course it lacked qemu-specific features like qcow2 or any
network-backed disk, so many disk images could not be opened this way.
This was never supported in RHEL.
See-also: https://bugzilla.redhat.com/1144197
LUKS support used to work best if the LUKS device resided on a partition,
and contained a Physical Volume for an LVM Volume Group. This scheme, also
called LVM-on-LUKS, is commonly created by installers of various Linux
distributions. (See RHBZ#1451665.)
Libguestfs now also supports the scheme wherein the LUKS device resides on
an LVM Logical Volume, and contains a filesystem. This is called
LUKS-on-LVM, it is the inverse of the above scheme, and is created by
installers of other Linux distributions. (See RHBZ#1658126.)
Both schemes are now decrypted by libguestfs-based utilities when
inspection is enabled (such as in "guestfish -i", virt-inspector,
virt-v2v), through the inspect_mount() function in utilities written in C,
and through the "inspect_decrypt" function in ones written in OCaml.
We don't seem to need an API like "list-luks-devices", as
"list-dm-devices" returns decrypted (i.e., opened) LUKS devices too; for
example, in the LUKS-on-LVM case:
> ><fs> list-dm-devices
> /dev/mapper/luks-0d619854-ccd5-43b1-8883-991fec5ef713
> /dev/mapper/luks-4e9e7a6f-a68c-42fd-92b4-8f4f2579a389
Thus, the subject TODO section is now out of date, and it's unclear what
remains "to do" there; let's just remove the section.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1658126
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20220223162120.16729-4-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
Linux from around 5.6 now enumerates individual disks in any order
(whereas previously it enumerated only drivers in parallel). This
means that /dev/sdX ordering is no longer stable - in particular we
cannot be sure that /dev/sda inside the guest is the first disk that
was attached to the appliance, /dev/sdb the second disk and so on.
However we can still use SCSI PCI device numbering as found in
/dev/disk/by-path. Use this to translate device names in and out of
the appliance.
Thanks: Vitaly Kuznetsov, Paolo Bonzini, Dan Berrangé.
This adds a new output mode to virt-v2v. virt-v2v -o rhv-upload
streams images directly to an oVirt or RHV >= 4 Data Domain using the
oVirt SDK v4. It is more efficient than -o rhv because it does not
need to go via the Export Storage Domain, and is possible for humans
to use unlike -o vdsm.
The implementation uses the Python SDK (‘ovirtsdk4’ module). An
nbdkit Python 3 plugin translates NBD calls from qemu into HTTPS
requests to oVirt via the SDK.
Build the P2V disk image and boot it. We don't actually use PXE
specifically, but we do test the whole PXE / kernel command line path
much more thoroughly.
This is a 'check-slow' test because it takes ages to run.
We also need to be more careful about PyString_FromString and similar
functions returning NULL on failure. Currently we don't check this
every time. This commit adds more checks, but is still not complete.
The existing APIs guestfs_stat, guestfs_lstat and guestfs_lstatlist
return a stat structure that contains atime, mtime and ctime fields
that store only the timestamp in seconds.
Modern filesystems can store timestamps down to nanosecond
granularity, and the ordinary glibc stat(2) wrapper will return these
in "hidden" stat fields:
struct timespec st_atim; /* Time of last access. */
struct timespec st_mtim; /* Time of last modification. */
struct timespec st_ctim; /* Time of last status change. */
with the following macros defined for backwards compatibility:
#define st_atime st_atim.tv_sec
#define st_mtime st_mtim.tv_sec
#define st_ctime st_ctim.tv_sec
It is not possible to redefine guestfs_stat to return a longer struct
guestfs_stat with room for the extra nanosecond fields, because that
would break the ABI of guestfs_lstatlist as it returns an array
containing consecutive stat structs (not pointers). Changing the
return type of guestfs_stat would break API. Changing the generator
to support symbol versioning is judged to be too intrusive.
Therefore this adds a new struct (guestfs_statns) and new APIs:
guestfs_statns
guestfs_lstatns
guestfs_lstatnslist
which return the new struct (or array of structs in the last case).
The old APIs may of course still be used, forever, but are deprecated
and shouldn't be used in new programs.
Because virt tools are compiled with -DGUESTFS_WARN_DEPRECATED=1, I
have updated all the places calling the deprecated functions. This
has revealed some areas for improvement: in particular virt-diff and
virt-ls could be changed to print the nanosecond fields.
FUSE now returns nanoseconds in stat calls where available, fixing
https://bugzilla.redhat.com/show_bug.cgi?id=1144891
Notes about the implementation:
- guestfs_internal_lstatlist has been removed and replaced by
guestfs_internal_lstatnslist. As the former was an internal API no
one should have been calling it, or indeed can call it unless they
start defining their own header files.
- guestfs_stat and guestfs_lstat have been changed into library-side
functions. They, along with guestfs_lstatlist, are now implemented
as wrappers around the new functions which just throw away the
nanosecond fields.
'-cpu host' makes the CPU look like the host CPU. This is useful
because it means the appliance can use all the host CPU's features,
eg. for fast checksumming or MD XORing.
'kvmclock' is the paravirtualized clock for qemu/KVM. It's much more
stable than relying on tsc or a virtualized device such as HPET, and
clock instability is one of the problems we have encountered when
running libguestfs on a heavily loaded machine, especially in a nested VM.
Note that we require qemu >= 1.2 (and have for a while) which has long
supported both of these options.
This large, but mainly mechanical commit, renames "attach method"
everywhere to "backend".
Backwards compatibility of the API (guestfs_{set,get}_attach_method)
and environment (LIBGUESTFS_ATTACH_METHOD) is maintained, but in new
code use guestfs_{set,get}_backend and LIBGUESTFS_BACKEND instead.
The default backend (launching qemu directly) is now called 'direct'
instead of 'appliance', although you can still use 'appliance' as a
synonym.
The cleanup of package managers cache is already implemented
by sprep_operation_package_manager_cache.ml, so remove this
TODO item.
Signed-off-by: Wanlong Gao <gaowanlong@cn.fujitsu.com>
Let tests/bigdirs/test-big-dirs.pl uses mke2fs to create small
inode ratio files.
Signed-off-by: Wanlong Gao <gaowanlong@cn.fujitsu.com>
RWMJ:
- Fix invocation of mke2fs with optional arguments.
- Run the test to check it works.
I tested this by comparing the output of virt-inspector over Windows
guests before and after the change, which was identical:
$ md5sum `ls -1 /tmp/*.before /tmp/*.after`
c292d6629b5a761eccb4a279754399b4 /tmp/Win2003.after
c292d6629b5a761eccb4a279754399b4 /tmp/Win2003.before
eb1e1ff29208a9ee46e9c100dfec26b2 /tmp/Win2012.after
eb1e1ff29208a9ee46e9c100dfec26b2 /tmp/Win2012.before
d060a95d7ffe5dce6c4e66feb80c2837 /tmp/Win7x32.after
d060a95d7ffe5dce6c4e66feb80c2837 /tmp/Win7x32.before
8914eee70ac4f8a0317659e09e00dcdc /tmp/Win7x32Dynamic.after
8914eee70ac4f8a0317659e09e00dcdc /tmp/Win7x32Dynamic.before
a2dcdfc0f9d64054640875aa791889e0 /tmp/Win7x32TwoDisks.after
a2dcdfc0f9d64054640875aa791889e0 /tmp/Win7x32TwoDisks.before
5ed49568a5147dce7517c99de41ebf2e /tmp/Win8previewx64.after
5ed49568a5147dce7517c99de41ebf2e /tmp/Win8previewx64.before
fdfc7d272b79a665ae3313ae1ae30660 /tmp/WinXP.after
fdfc7d272b79a665ae3313ae1ae30660 /tmp/WinXP.before
3c705444be664f1316b21c5d8d3cb0be /tmp/WinXPRecConsole.after
3c705444be664f1316b21c5d8d3cb0be /tmp/WinXPRecConsole.before
Note that we keep the old daemon calls, but rename them as
"internal_write" and "internal_write_append". This lets us implement
the new library-side calls more efficiently in the common case when
the uploaded content is smaller than the message buffer. In most
cases the new calls won't end up using a temporary file.
