Commit f7a24b2ea8 ("lib/qemu.c: Use machine type none when inspecting
QMP properties") changed the number of command line parameters used
before the '-qmp stdio' option when libguestfs queries qemu features.
This broke some tests which rely on the exact order of parameters.
Fixes: commit f7a24b2ea8
Updates: commit 5da8102f5f
Reimplement xfs_info by returning a hash table of values (rather than
a limited struct), and by writing it in OCaml with PCRE which makes
string parsing a lot simpler. This will now flexibly return all the
fields from the underlying xfs_info command, even (hopefully) future
fields.
Note the field values are returned as strings, because the actual
fields in xfs_info output are fairly random and free-form. There is a
trade off here between returning as much information as we can, and
requiring the user to do a bit of (simple) field parsing.
Fixes: https://issues.redhat.com/browse/RHEL-143673
Dan mentioned that there is a special machine type ("none") we can use
when querying for KVM. It has no CPU, memory, etc and does not run,
but you can still enable KVM for it.
Note we have to remove the -cpu parameter, otherwise qemu prints this
error:
qemu-kvm: apic-id property was not initialized properly
Updates: commit 5da8102f5f
Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
/usr/bin/ld: /tmp/ccvSGq6E.ltrans7.ltrans.o:(.data.rel.ro+0x1f8): undefined reference to `optgroup_selinuxrelabel_available'
The reason is that we didn't include optgroup_selinuxrelabel_available
on the fallback / no libselinux code path.
Reported-by: David Runge
Thanks: Toolybird
Fixes: https://github.com/libguestfs/libguestfs/issues/290
Fixes: commit ed40333a23
Add a new optional boolean argument 'keepdirlink' to tar_in that passes
--keep-directory-symlink to tar. This preserves existing symlinks to
directories when extracting, which is important for usrmerge systems
where /lib is a symlink to /usr/lib.
Without this option, extracting a tarball containing lib/modules/...
to / would replace the /lib symlink with a real directory, breaking
the system.
Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
When checking for QMP properties, we run a qemu command and interact
with the QMP console. However we also start the guest running (there
is no actual guest in this situation). This occasionally causes this
line to be printed:
libguestfs: generic_qmp_test: 3: {"timestamp": {"seconds": 1768823946, "microseconds": 898287}, "event": "GUEST_PANICKED", "data": {"action": "poweroff", "info": {"core": 0, "psw-addr": 0, "reason": "disabled-wait", "psw-mask": 562956395872256, "type": "s390"}}}\r\n
which confuses our parser.
As there is no reason to start the non-existent guest, add the -S
option which causes qemu to start up in a paused state.
For some reason this only happens on s390x but I think it could happen
on all architectures, so it may be a timing issue or something
particular about s390x firmware.
Commit 669eda1e24 ("lib/launch-direct.c: Simplify test for KVM, remove
qemu caching") made it so that failure of generic_qmp_test() will
cause launch to fail. However we still used debug messages to print
the error, so unless you have debugging enabled you wouldn't see any
error message if this function fails.
Updates: commit 669eda1e24
Devices associated with btrfs volumes are not reverse-translated
(e.g., btrfsvol:/dev/sdX to sdY).
Forward translation occurs, creating a path mismatch. This causes
errors in subsequent btrfs commands.
Thanks: Arye Yurkovsky
lvm returns logical volumes even if they're broken, for instance when a
physical volume is missing in their volume group.
In such cases, stat would fail to resolve the provided path.
Handle such cases by skipping such failures when finding the matching
lvm device.
In C23, strchr or strrchr on a const parameter now returns a const
pointer. We saw this error:
destpaths.c: In function ‘complete_dest_paths_generator’:
destpaths.c:165:9: error: assignment discards ‘const’ qualifier from pointer target type [-Werror=discarded-qualifiers]
165 | p = strrchr (text, '/');
| ^
The existing code had a bug which you can demonstrate by doing:
$ guestfish -N fs:btrfs:10G -m /dev/sda1 \
btrfs-subvolume-create /sub :
btrfs-subvolume-snapshot /sub /snap1 : \
btrfs-subvolume-snapshot /sub /snap123 : \
btrfs-subvolume-snapshot /sub /snap123456 : \
btrfs-subvolume-show /sub
...
libguestfs: error: appliance closed the connection unexpectedly.
This usually means the libguestfs appliance crashed.
As the code for parsing the output and creating the comma-separated
list of snapshots was unncessarily complicated in the first place,
simplify it. This also fixes the bug.
This also adds a regression test.
Thanks: Arye Yurkovsky
Link: https://lists.libguestfs.org/archives/list/guestfs@lists.libguestfs.org/thread/QV5VDHIH7WRUNAE54K6OEOKJMWL6M7EM/
Python is broken and requires that we include <Python.h> before all
other headers so it can make inadvisable definitions of
_POSIX_C_SOURCE and other things. This wasn't a problem before, but a
recent change to glibc makes this necessary now.
See also a similar commit in nbdkit:
f924c3c34b
I also removed the -Wcast-align suppression as that is no longer
needed.
It tests a corner case that we don't experience in real life (although
it can happen). Moreover the test often hangs, so we're not proving
anything except that there's a very obscure bug that we haven't fixed
and probably won't ever fix.
Part of this test uses `qemu-nbd -k SOCKPATH`, and passes
that socket to libvirt. SOCKPATH will have label user_home_t,
which svirt_t can't access, so running with selinux enforcing this
test will always fail.
Manually change the socket label to svirt_image_t which makes
selinux happy.
Note: libvirt does not relabel most (all?) externally managed socket
paths. I think this is correct, since in theory this could be a
socket shared with other processes, so its up to the user to ensure
DAC/MAC permissions are acceptable for their usecase.
Signed-off-by: Cole Robinson <crobinso@redhat.com>
really what we need is sgdisk in the appliance, but for test suite
purposes this check is enough IMO
Signed-off-by: Cole Robinson <crobinso@redhat.com>
To match other non-libtool tests.
Before: PASS: /path/to/libguestfs.git/tests/c-api/.libs/test-config
After: PASS: c-api/.libs/test-config
Signed-off-by: Cole Robinson <crobinso@redhat.com>
When iterating on commit 3c1554e7f2 ("lib: Add new app2_class field
for classifying applications") in early versions I was accidentally
passing NULL to match(). Unexpectedly this returned 1 (ie. matching).
This happened because we did not correctly handle other errors apart
from PCRE2_ERROR_NOMATCH from pcre2_match.
Emit a debug message when this happens, and return no match.
To test this I ran these commands, and there was no output:
$ make && LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 make check
$ find -name '*.log' | xargs grep 'match.*unexpected error'
Windows group policy objects (GPOs) are restrictions that can be added
by an administrator to Windows to lock down various operations. From
our point of view the ones that matter involve restricting the ability
to inject device drivers.
Previously virt-v2v detected group policy here:
9bb2e7d470/convert/convert_windows.ml (L69)
We would like to report group policy through the libguestfs API and
tools such as virt-inspector, so move the code that is used to detect
group policy to libguestfs. A new API is introduced that returns
whether group policy was found (only for Windows guests) during
inspection of the software registry.
Fixes: https://issues.redhat.com/browse/RHEL-125846
For McAfee VirusScan, the name is bogus and the display name is the
actual name:
<application>
<name>{CE15D1B6-19B6-4D4D-8F43-CF5D2C3356FF}</name>
<display_name>McAfee VirusScan Enterprise</display_name>
<version>8.8.04001</version>
<install_path>C:Program Files (x86)McAfeeVirusScan Enterprise\</install_path>
<publisher>McAfee, Inc.</publisher>
<url>http://www.mcafeesecurity.com/</url>
</application>
Simplify the existing ad hoc code so we just match name, display name
and publisher (if present) against all the virus scanning strings.
Existing virt-v2v code uses some simple heuristics for detecting
Windows anti-virus software:
7520185504/convert/windows.ml
Replicate exactly this code as a new field in the struct returned by
guestfs_inspect_get_applications2. Because of limitations with the
API, we must use one of the existing spare fields in the struct, and
it must have the same type (a string), so we are limited in the design
of this new API. I chose to return a primary classification for the
application, with the only classification possible so far being
"antivirus" (or "" if not). This allows the possibility of future
expansion of use of this field if we need to in future.
Fixes: https://issues.redhat.com/browse/RHEL-125846
This is currently broken, and gobject bits are slated to be
removed entirely, so just disable this test.
Signed-off-by: Cole Robinson <crobinso@redhat.com>
Convert to TypedData_Get_Struct which has been in ruby since 2009
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Cole Robinson <crobinso@redhat.com>
It was added in d4a1c3a778 for gnulib, but since then gnulib
is no longer a submodule, and it does not seem to be required
for the build anylonger.
Signed-off-by: Cole Robinson <crobinso@redhat.com>
This reverts "m4: Add junk required to make 'AM_GNU_GETTEXT' work"
commit b9f75ca5b8
And replaces AM_GNU_GETTEXT call with a simpler LIBINTL check
(recommended by claude).
For linux builds on glibc it doesn't appear we need anything
from autotools here, since libintl.h is always available.
For some other cases like macos I think this new minimal check
covers us in the common case.
There are definitely more obscure corner cases that gettext
m4 magic covers but I suspect none of them really matter for
any platforms libguestfs already functions on.
Signed-off-by: Cole Robinson <crobinso@redhat.com>
This tests adding a running libvirt domain to libguestfs.
This was never really a safe real-world example AFAICT because it
creates qemu overlays on top of disks that are opened writeable
by the source guest.
Nowadays qemu uses write locks to reject this type of behavior,
and the test fails.
Adjust things so the source VM opens its storage readonly, but
we are still confirming that add_libvirt_dom doesn't mess up
selinux labels.
Signed-off-by: Cole Robinson <crobinso@redhat.com>
If a user passes a custom hv, and uses add_libvirt_dom (like
in test820RHBZ912499.py) , this hits a bogus libvirt validation
check which rejects disabling svirt at both domain and disk level.
Work around it by skipping the redundant disk override when
svirt is disabled at the domain level.
Signed-off-by: Cole Robinson <crobinso@redhat.com>
On fedora, ./configure will set QEMU=qemu-kvm, but libvirt
domcaps will default to qemu-system-x86_64. The former is a symlink
to the latter, but libguestfs doesn't know that, so determines
the user requested a non-default hv override and changes some
settings as a result (like disabling svirt).
Check for the symlink case and consider it a non-custom config
if realpath matches libvirt's default
Signed-off-by: Cole Robinson <crobinso@redhat.com>
Add debugging about is_custom_hv result.
cache the initial call so we are only hitting the debug() once.
Signed-off-by: Cole Robinson <crobinso@redhat.com>
libvirt has DAC relabeled sockets for us for a decade, so we don't
need to do chowning here anymore
The chmod calls are still required, otherwise our created sockets
may be too permissive.
Update the comment to try and preserve the still relevant info,
though now it's a bit awkwardly placed.
Signed-off-by: Cole Robinson <crobinso@redhat.com>
https://bugzilla.redhat.com/2082806 is a private kernel bug (sigh) for
RHEL 8, where some missing backports caused 5-level page tables to
fail when qemu emulated them. This was fixed back in 2023, so let's
remove this workaround now.
On RHEL 10.2 aarch64 (only) we see:
$ echo '{ "execute": "qmp_capabilities" }' '{ "execute": "query-kvm" }' '{ "execute": "quit" }' | QEMU_AUDIO_DRV=none "/usr/libexec/qemu-kvm" -display none -machine "virt,accel=kvm:hvf:tcg" -qmp stdio
qemu-kvm: invalid accelerator hvf
qemu-kvm: falling back to KVM
{"QMP": {"version": {"qemu": {"micro": 0, "minor": 1, "major": 9}, "package": "qemu-kvm-9.1.0-15.el10_0.4"}, "capabilities": ["oob"]}}
qemu-kvm: warning: CPU model cortex-a57-arm-cpu is deprecated -- use 'host' / 'max'
qemu-kvm: kvm_init_vcpu: kvm_arch_init_vcpu failed (0): Invalid argument
Unfortunately we cannot use guestfs_int_get_cpu_model (as that
requires us to already know if KVM is supported), so we just have to
guess that -cpu max will work, at least enough for QMP to work.
Fixes: https://issues.redhat.com/browse/RHEL-121076
Reported-by: Xiang Hua Chen
virt-customize and friends have the following options:
--sm-register
--sm-unregister
--sm-attach
--sm-remove
--sm-credentials
These run `subscription-manager` commands inside the VM, which
is often necessary to get working package install for RHEL VMs.
These were added in 2015ish. Conceptually I understand why we
might want native support in the tools: virt-customize and
virt-builder have package management knowledge, and
for some distros subscription-manager is necessary.
But this support doesn't add much convenience over calling the commands
directly with `--run-command`. And on RHEL10, subscription-manager was
reworked and complete drops the backing commands for `--sm-attach` and
`--sm-remove`.
We _could_ make this code smarter, try to detect that situation,
and not error when subscription-manager is new enough. But if
subscription-manager itself doesn't care about maintaining that kind
of compat, I'm not sure why we should care either.
Instead we decided to just rip it all out and document the
alternatives.
common module is updated to sync related changes:
Cole Robinson (1):
mlcustomize: deprecate and remove --sm-* options
Richard W.M. Jones (3):
mlstdutils: Export List.find_opt
daemon, generator: Use power of 2 for initial size of Hashtbl.create
mlcustomize/inject_virtio_win.ml: Use viostor.inf instead of guestor
https://issues.redhat.com/browse/RHEL-113397https://issues.redhat.com/browse/RHEL-113398
Signed-off-by: Cole Robinson <crobinso@redhat.com>
test-parallel-mount-local reliably fails when my machines are
under load (like running the test suite in parallel). The
issue is /usr/libexec/gvfs-udisks2-volume-monitor fiddling with
the fuse mounts, preventing them from being unmounted.
./fuse/guestunmount defaults to 5 unmount retries with increasing
time backoff, but apparently that isn't enough. Bumping it to
use 10 retries makes things more reliable for me.
Signed-off-by: Cole Robinson <crobinso@redhat.com>
On macOS this fails with:
guestunmount.c:170:5: error: call to undeclared function 'sigaction'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
170 | sigaction (SIGINT, &sa, NULL);
| ^
1 error generated.
Reported-by: Mohamed Akram
Related: https://github.com/libguestfs/libguestfs/pull/188
macOS macfuse has an extra options parameter for the setxattr and
getxattr FUSE callbacks. The possible options are documented below.
However the underlying libguestfs APIs don't allow us to act on these
flags, so we ignore them for now.
(from https://manp.gs/mac/2/setxattr)
XATTR_NOFOLLOW
do not follow symbolic links. setxattr() normally sets attributes
on the target of path if it is a symbolic link. With this option,
setxattr() will act on the link itself.
XATTR_NOFOLLOW_ANY
do not follow any symbolic links encountered during pathname
resolution. An error is returned if a symlink is encountered
before the last component of path.
XATTR_CREATE
fail if the named attribute already exists.
XATTR_REPLACE
fail if the named attribute does not exist. Failure to specify
XATTR_REPLACE or XATTR_CREATE allows creation and replacement.
Reported-by: Mohamed Akram
Fixes: https://github.com/libguestfs/libguestfs/issues/180
Related: https://github.com/macfuse/macfuse/issues/1065
Previously we tested if KVM was available, and cached that for the
qemu binary. I think this was actually wrong. For example, if the
machine restarts, then the cache is still around, but KVM might be
enabled or disabled because of a new host kernel.
In any case, let's radically simplify this.
Test for KVM on each run. Consequently we can remove all the qemu
test caching stuff as it is no longer used anywhere.
I also tightened up the code that runs the QMP query-kvm command, so
now any unexpected output will cause a runtime failure. This command
ought to work, and if it breaks we ought to know about it and fix it.
Previously we ran 'qemu -device ?' to list devices, using the output
to test if qemu supported particular devices. This was extensively
used in the past, but after recent commits there are no further users
of this code, so we can remove it all completely.
It's always a good idea to provide proper entropy to the libguestfs
appliance (eg. for any cryptographic operations). We already assume
virtio, so we can assume virtio-rng is present.
We no longer use the libvirt version anywhere, except when reporting
the version. Remove this from the handle.
Simplify the remaining code. In particular:
* don't bother parsing the libvirt version, just print what
virGetVersion gives us
* guestfs_int_version_from_libvirt is dead code, so it can be removed
Libvirt 9.0.0 was released in January 2023, and it seems safe to
assume that if you're enabling the non-default backend, you can at
least use a new version of libvirt.
If you're using new libvirt, might as well also assume passt is
available.
Previously we parsed qemu -help output to get the version of qemu and
to test for other features. After prior commits, this is no longer
done, so we can remove the qemu -help invocation and associated code
completely.
This function is now only used in one place, to print the version of
qemu in direct mode, when debugging is enabled.
Remove this function and replace with a direct command invocation
('qemu --version'). We only need to run this command when debugging
is enabled, and we copy all of the output to the debug channel.
I have made the assumption here that qemu -version does not try to
create a display device. (The previous invocation of qemu -help
actually ran 'qemu -display none -help' indicating that this was not
always the case.)
This is actually an improvement on before, since now we get to see the
full output of 'qemu --version'. The new output looks like:
libguestfs: begin testing qemu features
libguestfs: command: run: /usr/bin/qemu-kvm
libguestfs: command: run: \ -version
libguestfs: qemu: QEMU emulator version 10.1.0 (qemu-10.1.0-8.fc44)
libguestfs: qemu: Copyright (c) 2003-2025 Fabrice Bellard and the QEMU Project developers
We already effectively assume that qemu is much newer than the 1.3.0
baseline previously documented. As one example, commit 47857751a7
("lib: direct: Remove test for qemu mandatory locking") assumes that
qemu >= 2.10. Because passt support is desirable in general, let's
assume that qemu is at least version 7.2.0, and document it.
qemu 7.2.0 was released in December 2022, nearly 3 years ago, and RHEL
9 is currently on qemu 9.1.0.
This keeps the SLIRP fallback path in case passt isn't installed, but
we should remove that fallback in future too.
We can safely assume that qemu supports -nodefaults and
-no-user-config, since these have been supported since forever.
-no-hpet was deprecated in qemu 8.0 and the option removed in early
2024, replaced with -machine hpet=off. HPET defaults to 'on' in
upstream qemu, and to 'off' in downstream RHEL rebuilds.
Since (for libguestfs) we can assume an up to date Linux kernel is
running inside the guest, and that the kernel will do the right thing
with regards to timers, we don't need to mess with qemu defaults. In
practice, Linux chooses kvm-clock.
Thanks: Thomas Huth, Daniel Berrange
We used to read the QMP schema so we could see if the binary supported
qemu mandatory locking. However after commit 47857751a7 ("lib:
direct: Remove test for qemu mandatory locking") this is dead code, so
remove it.
Updates: commit 47857751a7
This was a test for a 13 year old problem to do with problems
launching libguestfs appliances in parallel. That problem does not
affect current code.
Testing build/launch of libguestfs appliances is a valuable thing to
test. However we start libguestfs in parallel in both the current
test suite and in higher level tools, and this test was dubious (and
breaks with recent changes). Therefore remove the test.
Before 2011 it was recommended to use a prime number for the initial
size. In 2011 the OCaml hash table was reimplemented using a hash
function based on Murmur 3. Hashtbl.create now adjusts the initial
size to the next power of 2 (minimum 16). So replace obsolete
'Hashtbl.create 13' with 'Hashtbl.create 16'.
In filesystems that have many millions of files, the default behaviour
of setfiles is to build a huge hash table containing every filename.
This uses up lots of memory which prevents relabelling from happening
in the reduced memory environment of the libguestfs appliance.
I added the setfiles -A option to change this default behaviour. If
setfiles has the option then use it.
Fixes: https://issues.redhat.com/browse/RHEL-114292
Related: https://issues.redhat.com/browse/RHEL-111165
Related: https://issues.redhat.com/browse/RHEL-111505
It turns out that ubuntu-latest is the same as 24.04, so we're just
testing on the same version twice. Remove the duplication.
Thanks: Stephen Gallagher
Fixes: commit e218dd73cc
Add an API to do the equivalent of `chmod [-r] MODE PATH` for
NTFS filesystems.
Files created on a linux ntfs-3g mount can not change permissions
directly. New files and directories are created with rough windows
equivalent of `chmod 777`. These wide open permissions can generate
security warnings on windows after virt-v2v installs bits into
`Program Files\Guestfs`.
Behind the scenes we use `ntfssecaudit(8)` from `ntfsprogs`
which is already part of the appliance. We only expose the chmod-style
feature; the rest of `ntfssecaudit` is concerned reporting and
managing fine grained windows security info which is way more than
we need.
Also note, `ntfssecaudit` needs to run on an unmounted partition
so using this is more complicated than a traditional `chmod` call.
Related: https://issues.redhat.com/browse/RHEL-104352
Signed-off-by: Cole Robinson <crobinso@redhat.com>
This builds libguestfs and runs the in-tree tests, ie. the basic
'./configure && make && make check'.
We're using the "free for public repositories" plan. We run the tests
directly on an Ubuntu VM, versions 24.04 & latest. Also on Fedora 42,
43 in a container (running on the Ubuntu VM).
The tests are run on pushes to the master branch, and on pull
requests to the master branch.
GObject is not built, since the tests fail and anyway there are some
deep problems with the GObject bindings and they probably will be
removed.
Thanks: Stephen Gallagher, Cole Robinson
This change was done almost entirely automatically using the script
below. This uses the OCaml lexer to read the source files and extract
the strings and locations. Strings which are "candidates" (in this
case, longer than 3 lines) are replaced in the output with quoted
string literals.
Since the OCaml lexer is used, it already substitutes all escape
sequences correctly. I diffed the output of the generator and it is
identical after this change, except for UUIDs, which change because of
how Utils.stable_uuid is implemented.
Thanks: Nicolas Ojeda Bar
$ ocamlfind opt -package unix,compiler-libs.common find_strings.ml \
-o find_strings.opt -linkpkg
$ for f in $( git ls-files -- \*.ml ) ; do ./find_strings.opt $f ; done
open Printf
let read_whole_file path =
let buf = Buffer.create 16384 in
let chan = open_in path in
let maxlen = 16384 in
let b = Bytes.create maxlen in
let rec loop () =
let r = input chan b 0 maxlen in
if r > 0 then (
Buffer.add_substring buf (Bytes.to_string b) 0 r;
loop ()
)
in
loop ();
close_in chan;
Buffer.contents buf
let count_chars c str =
let count = ref 0 in
for i = 0 to String.length str - 1 do
if c = String.unsafe_get str i then incr count
done;
!count
let subs = ref []
let consider_string str loc =
let nr_lines = count_chars '\n' str in
if nr_lines > 3 then
subs := (str, loc) :: !subs
let () =
Lexer.init ();
let filename = Sys.argv.(1) in
let content = read_whole_file filename in
let lexbuf = Lexing.from_string content in
let rec loop () =
let token = Lexer.token lexbuf in
(match token with
| Parser.EOF -> ();
| STRING (s, loc, sopt) ->
consider_string s loc; (* sopt? *)
loop ();
| token ->
loop ();
)
in
loop ();
(* The list of subs is already reversed, which is convenient
* because we must the file substitutions in reverse order.
*)
let subs = !subs in
let new_content = ref content in
List.iter (
fun (str, loc) ->
let { Location.loc_start = { pos_cnum = p1 };
loc_end = { pos_cnum = p2 } } = loc in
let len = String.length !new_content in
let before = String.sub !new_content 0 (p1-1) in
let after = String.sub !new_content (p2+1) (len - p2 - 1) in
new_content := before ^ "{|" ^ str ^ "|}" ^ after
) subs;
let new_content = !new_content in
if content <> new_content then (
(* Update the file in place. *)
let new_filename = filename ^ ".new"
and backup_filename = filename ^ ".bak" in
let chan = open_out new_filename in
fprintf chan "%s" new_content;
close_out chan;
Unix.rename filename backup_filename;
Unix.rename new_filename filename
)
Pulls in the commits listed below. This has no effect as all changes
are confined to the common/mlcustomize subdirectory which we do not
use or ship.
Richard W.M. Jones (4):
mlcustomize/SELinux_relabel.ml: Add comment
mlcustomize/SELinux_relabel.ml: Use new guestfs_setfiles API
mlcustomize/SELinux_relabel.ml: Relabel every mountpoint
mlcustomize/firstboot.ml: Use quoted string literals for firstboot
Vadim Rozenfeld (1):
Modify the firstboot script to check the scripts execution return status
If HKLM\System\MountedDevices references a blank disk, then when we
try to search for the actual backing device we will get an error from
parted:
parted: /dev/sdb: parted exited with status 1: Error: /dev/sdb: unrecognised disk label: Invalid argument
Just ignore these errors instead of failing inspection.
Fixes: https://issues.redhat.com/browse/RHEL-108803
Reported-by: Ameen Barakat
Thanks: Ming Xie
The function 'map_registry_disk_blob_gpt' immediately below this one
has a debugging statement. Add the equivalent to the function
'map_registry_disk_blob_mbr'.
The output looks like:
map_registry_disk_blob_mbr: searching for MBR disk ID 31 32 33 34
map_registry_disk_blob_mbr: searching for MBR partition offset 00 00 00 10 00 00 00 00
The guestfs_selinux_relabel function was very hard to use. In
particular it didn't just do an SELinux relabel as you might expect.
Instead you have to write a whole bunch of code around it (example[1])
to make it useful.
Another problem is that it doesn't let you pass multiple paths to the
setfiles command, but the command itself does permit that (and, as it
turns out, will require it). There is no backwards compatible way to
extend the existing definition to allow a list parameter without
breaking API.
So deprecate guestfs_selinux_relabel. Reimplement it as
guestfs_setfiles. The new function is basically the same as the old
one, but allows you to pass a list of paths. The old function calls
the new function with a single path parameter.
[1] https://github.com/libguestfs/libguestfs-common/blob/master/mlcustomize/SELinux_relabel.ml
No existing OCaml functions have a StringList parameter, but we would
like to add one.
The original plan seems to have been to map these to 'string array'
types, but 'string list' is more natural, albeit marginally less
efficient. The implementation here just has to convert the 'char **'
into the OCaml linked list of values.
Previously calling 'sysroot_path "/dev"' for example would return the
string "/sysroot//dev". While this is not wrong, it confuses some
external programs (hello, setfiles), and it's not very "clean". Be a
bit more careful to avoid doubling the '/' character in the common case.
Encrypted root fs on SUSE distros will present itself like so:
```
/dev/mapper/cr_root / btrfs defaults 0 0
UUID=588905f9-bfa4-47b5-9fe8-893cb8ad4a0b /var btrfs subvol=/@/var 0 0
... more subvols here ...
UUID=8a278363-3042-4dea-a878-592f5e1b7381 swap btrfs defaults 0 0
/dev/mapper/cr_root /.snapshots btrfs subvol=/@/.snapshots 0 0
cr_root UUID=5289379a-a707-41b5-994c-c383f7ed54cc none x-initrd.attach
```
This breaks `-i` inspection, since libguestfs doesn't know what
/dev/mapper/cr_root is supposed to be, and nothing in the appliance
will autopopulate that path. This isn't a problem on Fedora, where
it uses UUID= instead of a /dev/mapper path.
Currently when we see /dev/mapper as a mount prefix, we only attempt
to do some LVM name mapping. This extends libguestfs to check
/etc/crypttab first. If we find an entry for the mapper path, and it
points to the encrypted luks UUID, we use that UUID to build the
associated /dev/disk/by-id/dm-uuid-CRYPT-* path, which is a symlink
to the unencrypted /dev/dm-X path
Resolves: https://issues.redhat.com/browse/RHEL-93584
Signed-off-by: Cole Robinson <crobinso@redhat.com>
Also some mdadm configuration files. This is useful for debugging.
The output looks like this:
info: /etc/fstab in /dev/VG/Root
LABEL=BOOT /boot ext2 default 0 0$
LABEL=ROOT / ext2 default 0 0$
Fixes: https://issues.redhat.com/browse/RHEL-106490
This saves us going into a loop if virDomainDestroyFlags keeps
returning -EBUSY quickly, which apparenrly can happen in containers.
The equivalent 'direct' backend code sleeps for 2 seconds in this case.
This is just code movement.
Signed-off-by: Cole Robinson <crobinso@redhat.com>
RWMJ: Renamed and moved the function for consistency with surrounding
code.
Many years ago we used to pass acpi=off on the Linux kernel command
line. In commit db1f811b2 we stopped doing that (around 2016).
However unless you also use:
<features>
<acpi/>
</features>
then it turns out that libvirt disables ACPI generation at the qemu
level. None of this mattered until SeaBIOS 1.17 changed its
behaviour, causing ACPI to be required for virtio devices to work.
Updates: commit db1f811b29
Related: https://bugzilla.redhat.com/show_bug.cgi?id=2372329
Thanks: Gerd Hoffmann
This test has never been run. It was originally added in
commit 36d6df671e ("tests/http: Add a test of HTTP protocol.", 2013).
However even when it was added, it was commented out.
Updates: commit 36d6df671e
We removed the final uses of custom printf formatters in
commit 0b3c6cc0c0 ("daemon: Remove remaining uses of custom printf %Q
and %R", 2022).
Updates: commit 0b3c6cc0c0
libosinfo changes the naming scheme it uses for SUSE starting with
major version 15. Previously it used names like "sles12" (or
"sles12sp1"), "sled12" for Server and Desktop variants. In 15+ it
uses "sle15" as there are no variants any longer (instead the
installer asks you what variant you want to install). We're only
interested in the Server variant. Change the name that we return to
"sle15" or "sle15sp1".
See: b0fa386699
Fixes: https://issues.redhat.com/browse/RHEL-95791
Thanks: Ming Xie, Victor Toso
Related: https://issues.redhat.com/browse/RHEL-95540
They will be removed in libguestfs 1.58 (the next but one version).
Currently they don't actually compile. The larger problem is that
they don't handle 64 bit quantities properly (using floats instead),
meaning that any disk size or offset above a certain size will be
improperly passed through the API, usually rounded to the nearest
53 bits.
In SLES guests in particular, btrfs snapshots seem to be used to allow
rollback of changes made to the filesystem. Dozens of snapshots may
be present. Technically therefore these are multi-boot guests. The
libguestfs concept of "root" of an operating system does not map well
to this, causing problems in virt-inspector and virt-v2v.
In this commit we ignore these duplicates. The test is quite narrow
to avoid false positives: We only remove a duplicate if it is a member
of a parent device, both are btrfs, both the snapshot and parent have
a root role, and the roles are otherwise very similar.
There may be a case for reporting this information separately in
future, although it's also easy to find this out now. For example,
when you see a btrfs root device returned by inspect_os, you could
call btrfs_subvolume_list on the root device to list the snapshots.
Fixes: https://issues.redhat.com/browse/RHEL-93109
Back in commit 8289aa1ad6 ("New APIs for guest inspection.", 2010)
when inspection was first added, we did inspection in the library, so
it was accurate to say that inspection information was stored "in the
handle". Much later, in commit 394d11be49 and commit 3a00c4d179
(2017) we moved inspection to the daemon, but left the comment the
same.
Fixes: commit 3a00c4d179
This function is used from other parts of the daemon, especially for
example with inspection. However it was difficult to follow exactly
what filesystems it was returning because of insufficient debugging
information.
create.c: In function 'disk_create_qcow2':
create.c:372:5: error: format not a string literal and no format arguments [-Werror=format-security]
372 | debug (g, cmd_stdout);
| ^~~~~
Fixes: commit 606aa1d182
https://issues.redhat.com/browse/RHEL-92239
After this, output looks like
$ ./run guestfish --ro --format=qcow2 -a test.img
libguestfs: error: qemu-img: qemu-img: /home/crobinso/src/libguestfs/tmp/libguestfsFlxnb0/overlay1.qcow2: Image is not in qcow2 format Could not open backing image. : qemu-img exited with error status 1.
To see full error messages you may need to enable debugging.
...
Signed-off-by: Cole Robinson <crobinso@redhat.com>
The old btrfs-fsck API used "btrfs check" which appears to be broken
or deprecated. The real tool you should use is "btrfs scrub". We
have already implemented that API, but it is very awkward to use from
libguestfs. In particular there's no existing way to run the scrub
and wait for it to finish.
Fix this by deprecating btrfs-fsck. Implement a new API
btrfs-scrub-full which runs btrfs scrub in the foreground, waits for
it to finish, and handles errors. It's much more like fsck tools in
other filesystems.
Thanks: Eric Sandeen
Fixes: https://issues.redhat.com/browse/RHEL-91936
This pulls in the commits below, requiring us to replace all uses of
String.is_prefix and String.is_suffix.
Mostly done with Perl like this, and carefully checked by hand
afterwards since this doesn't get everything right:
$ perl -pi.bak -e 's/String.is_prefix ([^[:space:]\)]+) ([^[:space:]\)]+)/String.starts_with \2 \1/g' -- `git ls-files`
Richard W.M. Jones (3):
mlstdutils: Fix comment that still referred to the old function names
mldrivers: Link to gettext-stub if ocaml-gettext is enabled
mlstdutils: Rename String.is_prefix -> starts_with, is_suffix -> ends_with
Pull in these commits which require minor changes:
Richard W.M. Jones (3):
mlstdutils: Remove Std_utils.identity
mlstdutils: Remove Std_utils.protect
mlstdutils: Remove List.filter_map
Adds these commits:
Richard W.M. Jones (5):
mlstdutils: Modify List.take, List.drop to match OCaml 5.3
mlstdutils: Rename List.dropwhile -> drop_while, takewhile -> take_while
mlstdutils: Add List.last function
mlstdutils: Move List module first
mlstdutils: Add String.common_prefix, longest_common_prefix
zfs-fuse has been unmaintained for a very long time. In fact, the
website has gone. This means that whatever we need for zfs (probably
some kind of license-compatible kernel module) is not going to be
provided by zfs-fuse.
See: https://bugzilla.redhat.com/show_bug.cgi?id=2214965
A customer case was found where /etc/fstab contained multiple root
mountpoints, something like:
LABEL=System / xfs ...
LABEL=Boot /boot ext2 ...
LABEL=System / xfs ...
This causes libguestfs and virt-v2v to fail. Either (on RHEL 9) we
try to mount the second instance of / which gives an error. Or (on
upstream kernels) we are able to mount the second instance but then
libguestfs gets confused when trying to unmount them.
In this case as the mounted devices are the same we can just delete
the duplicate. It's also possible that there could be multiple
non-identical root mountpoints, in which case we have to pick one, and
this code arbitrarily picks the first[*] (but emits a warning).
We don't do anything for non-root mountpoints.
Update common submodule to add 'List.same' function from mlstdutils.
[*] Which one is "the first" depends on what version of ocaml-augeas
we are using. ocaml-augeas version 0.6 Augeas.matches function
returns entries in reverse order (compared to augeas itself). This is
fixed in version 0.7:
http://git.annexia.org/?p=ocaml-augeas.git;a=commitdiff;h=b703b92e3d26690aa6f7b822132049ce5435983e
Fixes: https://issues.redhat.com/browse/RHEL-90168
We could match on 1- and 2-digit RHEL issues, thus matching strings
like "RHEL-9" and "RHEL-10". No such Jira issues exist for us.
(cherry picked from commit 846ecb2b3840212c6f2cad6a15a6b33cd2e912bf)
We didn't correctly check for an error from the ioctl. Thus when
using a raw block device you would often see a bogus warning in
debugging output.
Related: https://issues.redhat.com/browse/MTV-2441
Commit e11583a03a ("ocaml: Don't embed -L../lib/.libs in final
mlguestfs.cma or .cmxa") was correct in itself, but as a side effect
caused mlguestfs.cma to be linked to the system libguestfs. If system
libguestfs was not installed, the build would fail entirely with:
ocamlmklib -o mlguestfs \
-ldopt '-Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 ' \
libguestfsocaml_a-guestfs-c.o libguestfsocaml_a-guestfs-c-actions.o libguestfsocaml_a-guestfs-c-errnos.o ../common/utils/libguestfsocaml_a-cleanups.o ../common/utils/libguestfsocaml_a-stringlists-utils.o guestfs.cmo \
\
-lguestfs
/usr/bin/ld: cannot find -lguestfs: No such file or directory
Fix (from Mohamed Akram) is to add -L../lib/.libs to the -ldopt flag.
This does not embed the option in the final mlguestfs.cma file, but
seems to only use it while building the library.
Updates: commit e11583a03a
Reported-by: Mohamed Akram
Fixes: https://github.com/libguestfs/libguestfs/issues/191
$ ocamlobjinfo /usr/lib64/ocaml/guestfs/mlguestfs.cma
File /usr/lib64/ocaml/guestfs/mlguestfs.cma
Force custom: no
Extra C object files: -lmlguestfs -L../lib/.libs -lguestfs
...
(and the same in mlguestfs.cmxa). This was caused because we run
ocamlmklib with this -L parameter so that OCaml tests and examples
will link with the local libguestfs.so (instead of the
system-installed version), but for other reasons that doesn't actually
work either.
Comment from Eric Blake:
> We don't have any hard-links in our source. GNU cp -d is the same as
> cp -P --preserve=links, but --preserve=links only matters for hard
> links, so losing it doesn't hurt our usage. I'm inclined to just go
> with cp -P unconditionally rather than needing the configure.ac
> logic for $(CP_D).
Fixes: https://github.com/libguestfs/libguestfs/issues/183
Reported-by: Mohamed Akram
Thanks: Eric Blake
These libraries are only needed if building the daemon (so not if
using ./configure --disable-daemon). Move the tests so we only check
for these libraries if the daemon is enabled in the build.
The daemon also requires ocaml-augeas (it uses both the augeas C
library directly and the OCaml bindings for augeas, for reasons).
However that test is already disabled if the daemon is not being
built.
Reported-by: Mohamed Akram
Fixes: commit dfa9dee775
Fixes: commit 8a723ca62e
Fixes: commit 228d49bb84
Fixes: https://github.com/libguestfs/libguestfs/issues/184
libmagic (part of the "file" command) was required in earlier versions
of libguestfs, but this requirement was effectively removed in
commit b48da89dd6 ("daemon: Reimplement ‘file_architecture’ API in
OCaml") back in 2017. Or to be more precise, we now use the "file"
command inside the daemon, which may or may not use libmagic but we
no longer link to the library directly.
Reported-by: Mohamed Akram
Fixes: commit b48da89dd6
Related: https://github.com/libguestfs/libguestfs/issues/184
Linux + LVM supports device names like /dev/disk/by-id/dm-uuid-LVM-
followed by two concatenated UUIDs, firstly for the volume group and
secondly for the logical volume. We can reverse those to get the
device name (/dev/VG/LV).
fstab entries look like:
# / was on /dev/vg0/lv-0 during curtin installation
/dev/disk/by-id/dm-uuid-LVM-OzFWT6NHkstr1hcmrWRRMDGPn9xdZj1YOOycQ533186x288FdU6UubU3OlnWJz6D / ext4 defaults 0 1
# /usr was on /dev/vg0/lv-1 during curtin installation
/dev/disk/by-id/dm-uuid-LVM-OzFWT6NHkstr1hcmrWRRMDGPn9xdZj1YZu53m4ZssZ8Jeb3I14RAJwIj5YlHIb9P /usr ext4 defaults 0 1
The upshot of this fix is that we are now able to correctly inspect
and run virt-v2v on Ubuntu 22+ guests with split /usr. In particular,
we correctly map /etc/fstab entries like the above to LV device names,
which means that /usr merging now works correctly.
Reported-by: Jaroslav Spanko
Thanks: Daniel Berrange
Fixes: https://issues.redhat.com/browse/RHEL-87493
These were previously written in very convoluted C which had to deal
with parsing the crazy output of the "lvm" command. In fact the
parsing was so complex that it was generated by the generator. It's
easier to do this in OCaml.
These are basically legacy APIs. They cannot be expanded and LVM
already supports many more fields. We should replace these with APIs
for getting single named fields from LVM.
This was implemented wrongly. In the XDR protocol, UUIDs are fixed
buffers of length 32. We can just use memcpy to copy from the OCaml
string to the UUID, but we have to ensure the string length returned
by OCaml is correct (if not we just assert, it's an internal error).
(It didn't even compile before, so we know it was never used).
Ubuntu 22= uses /dev/disk/by-uuid/ followed by a filesystem UUID in
fstab entries. Resolve these to mountables.
A typical fstab entry looks like this:
# /boot was on /dev/vda2 during curtin installation
/dev/disk/by-uuid/b4e56462-5a64-4272-b76d-f5e58bd8f128 /boot ext4 defaults 0 1
The comment is generated by the installer and appears in the fstab.
This entry would be translated to /dev/sda2.
This acts just like FString except that we do reverse device name
translation on it. The only use is in the 'pvs-full' API where we
will use it (in a subsequent commit) to reverse translate the pv_name
field (a device name) before returning it from the daemon.
Compare this to the 'pvs' API which also returns a list of device
names, but using the generator's 'RStructList (RDevice,...)' return
type, where RDevice is similarly reverse translated.
Note in the library-side bindings, because the name has already been
translated in the daemon, we just treat it exactly the same as
FString. The vast majority of this patch is this mechanical change.
Cole Robinson (2):
mltools: decouple and simplify osinfo device support checks
mlcustomize: disable `--inject-virtio-win osinfo`
Richard W.M. Jones (3):
mltools: Fix de-oUnit-ized tests
mltools: Unreference various objects
Revert "mltools: Unreference various objects"
And update customize docs to match
Signed-off-by: Cole Robinson <crobinso@redhat.com>
These are not required by libguestfs. They were used by virt-builder,
but that tool has now been moved to a separate project
(guestfs-tools).
Fixes: commit 733d2182b6
Useful for debugging problems caused by the host kernel. In
particular we were looking at a problem with passt creating a user
namespace but didn't know what exact kernel was being used.
Internal error type did not implement the necessary traits to be
treated as a proper Rust error. This would cause ergonomic issues
with manual error handling and with error handling crates like
anyhow. Implement the Display and Error trait for internal Error
type. This also fixes the build warnings.
Signed-off-by: Jacob Reger <regerjacob@gmail.com>
Fixed break in Rust bindings where callback registering expected a
*const i8, but were provided with a *const c_char. Changed to be both
*const c_char in alignment with the libguestfs documentation.
Signed-off-by: Jacob Reger <regerjacob@gmail.com>
These APIs allow you to capture output from guest commands that
generate more output than the protocol limit allows.
Thanks: Nijin Ashok
Fixes: https://issues.redhat.com/browse/RHEL-80159
There were several, large *.o files left over after 'make clean':
$ ls ruby/ext/guestfs/
actions-0.c actions-2.c actions-4.c actions-6.c handle.c
actions-0.o actions-2.o actions-4.o actions-6.o handle.o
actions-1.c actions-3.c actions-5.c actions.h module.c
actions-1.o actions-3.o actions-5.o extconf.rb.in module.o
tests/Makefile.am assumes the test image is located in
tests/syslinux/, but we actually created it in tests/. Fix the
location, and also ensure the test will clean it up after running
successfully.
Replace strange $TEST_FUNCTIONS variable/expansion thing with
something more like what we use in nbdkit, a simple tests/functions.sh
script that gets sourced into each test script.
Update the common submodule to get:
commit 8137d47d0e654065391151eb275e3b64f230f6f5
Author: Richard W.M. Jones <rjones@redhat.com>
Date: Thu Feb 13 11:13:55 2025 +0000
mlcustomize, mltools: Replace $TEST_FUNCTIONS
TEST_FUNCTIONS is being removed from libguestfs and guestfs-tools (it
was removed from virt-v2v a while back). Make the same adjustment in
the common submodule.
(and some other commits which are not relevant to libguestfs)
Fixes virt-builder error:
virt-builder: error: libguestfs error: inspect_os: execvp: sfdisk: No
such file or directory
Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
Debian has replaced the deprecated isc-dhcp-client (dhclient) with
dhcpcd-base (dhcpcd) as the default DHCP client [1]. This causes network
issues in virt-builder since isc-dhcp-client is no longer installed
by default.
Add support for dhcpcd (dhcpcd-base package) in Debian.
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1041066
Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
As far as I can tell using po4a-gettextize was wrong here. The right
subtool for combining multiple documents into a .pot is po4a-updatepo,
so use that one instead.
OCaml 4.08.0 was released on 2019-06-14, over 5 years ago. By
requiring a slightly later OCaml version, we can drop more
compatibility code which was only used by older versions.
Consistent with qemu & libvirt, this drops support for compiling
upstream libguestfs on RHEL 8 (ocaml-4.07.0-4.el8.x86_64).
Qemu policy:
https://www.qemu.org/docs/master/about/build-platforms.html
Libvirt policy:
https://libvirt.org/platforms.html
Update the common submodule, pulling in:
Richard W.M. Jones (4):
qemuopts: Add ability to add raw, unquoted output to qemu scripts
qemuopts: Fix missing break statement
mlstdutils: Remove Option module
Remove test for caml_alloc_initialized_string
Stop generating these files. They are currently only used by virt-v2v
-o qemu mode, and there are better ways to locate the UEFI files
there.
Update the common submodule to bring in:
Richard W.M. Jones (5):
mlcustomize: Add heuristic support for Windows Server 2025
mlcustomize/customize_run.ml: Move 'in' to new line
mlstdutils/guestfs_config: Define host_os
mlcustomize, mltools: Check guest OS is compatible before allowing --run
Remove mlv2v/ subdirectory
dhcpcd fails to update /etc/resolv.conf if is a dangling symlink, so remove it
if it is.
This happens on Arch Linux when systemd-resolved is enabled by symlinking
/etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf. The symlink is copied
into the supermin appliance, but the target file is not and this causes dhcpcd to
fail when it tries to update /etc/resolv.conf.
The handling of /etc/resolv.conf could be improved in dhcpcd but it's not their
job to decide when to remove the symlink. We do have the authority to decide
whether or not to use systemd-resolved or resolvconf in the appliance, so we can
remove it.
After previous changes, this library is no longer used. We have
switched to json-c, for better compatibility with libvirt.
(cherry picked from
guestfs-tools commit e6dcf7e3a7e9170978e57ce6df1b34f92fac5ae3)
We tested for QEMU >= 2.10 support for mandatory locking. I believe
this is for all practical purposes always enabled now (and qemu 2.10
is ancient history) so simply assume it's true always.
Although we don't use JSON_parser in libguestfs, update the submodule
just to get rid of another Jansson dependency:
Richard W.M. Jones (1):
mltools: Replace jansson with json-c
This will eventually replace Jansson for all JSON parsing. However
this commit simply introduces the new dependency in the configure
script and documents it.
I chose json-c 0.14 as the baseline since that is the version in RHEL 9.
Probably earlier versions would work.
All recent compilers support this (except MS compilers which we don't
care about). Assume it is supported. We test it in ./configure and
hard fail if it doesn't work.
We still define HAVE_ATTRIBUTE_CLEANUP but you can now assume it is
always defined and don't have to check it.
The guest was found to have these inspection fields:
type: windows
distro: windows
product_name: Windows Server 2025 Standard
product_variant: Server
version: 10.0
arch: x86_64
hostname: WIN-84V4KKQ30SM
build ID: 26080
windows_systemroot: /Windows
windows_software_hive: /Windows/System32/config/SOFTWARE
windows_system_hive: /Windows/System32/config/SYSTEM
windows_current_control_set: ControlSet001
Reported-by: Ming Xie
Fixes: https://issues.redhat.com/browse/RHEL-62935
Add these two files from the common/ submodule:
common/mlcustomize/v2v-customize-options.pod
common/mlcustomize/v2v-customize-synopsis.pod
Updates: commit fe1ce09242
openEuler is simliar to CentOS, but the ID is not lower-case string,
as below:
NAME="openEuler"
VERSION="24.03 (LTS)"
ID="openEuler"
VERSION_ID="24.03"
PRETTY_NAME="openEuler 24.03 (LTS)"
ANSI_COLOR="0;31"
Signed-off-by: Wang Guoquan <wangguoquan03@foxmail.com>
Virt-v2v already includes facilities for injecting QEMU guest agent
etc. We shouldn't add the virt-customize options for this.
Update common submodule to include:
Richard W.M. Jones (2):
mlcustomize: Move virt-customize modules to mlcustomize/
mlcustomize: Update generated options for virt-v2v
Pull in some fixes from the common submodule. None of these should be
applicable to libguestfs, they apply only to guestfs-tools and
virt-v2v.
Richard W.M. Jones (13):
mlcustomize: Add Inject_virtio_win.inject_blnsvr implementation
mlcustomize: firstboot: Use Linux path for Powershell script path
mlcustomize: firstboot: Use powershell.exe instead of path
mlcustomize: firstboot: Use Powershell -NoProfile flag
mlcustomize: Revert delay installation of qemu-ga MSI
mldrivers/linux_kernels.ml: Prefix general information with ^info:
mlcustomize: Use Start-Process -Wait to run qemu-ga installer
mlcustomize: Add Firstboot.firstboot_dir function
mlcustomize: Place powershell scripts into <firstboot_dir>\Temp
mlcustomize: Inject qemu-ga & blnsvr into <firstboot_dir>/Temp
mlcustomize: Write qemu-ga log file name to log.txt
mlcustomize: Add some comments to firstboot batch file
mlcustomize: Reboot Windows between each firstboot script
Since RHEL 7.4, the noop scheduler is no longer a thing. Trying to
set it results in the error:
+ echo noop
/init: line 108: echo: write error: Invalid argument
The current recommendation (https://access.redhat.com/solutions/5427)
is to use mq-deadline, but that's also the default so we don't have to
do anything.
A bigger reason to remove these lines is that kernel 6.11.0 has
introduced a hang where -- rarely -- the ext4 filesystem hangs if you
try to change the scheduler while handing a page fault, even if you're
setting a scheduler that doesn't exist. I couldn't get much detail
except for a couple of stack traces from different VMs:
crash> set 234
PID: 234
COMMAND: "modprobe"
TASK: ffff9f5ec3a22f40 [THREAD_INFO: ffff9f5ec3a22f40]
CPU: 0
STATE: TASK_UNINTERRUPTIBLE
crash> bt
PID: 234 TASK: ffff9f5ec3a22f40 CPU: 0 COMMAND: "modprobe"
#0 [ffffb21e002e7840] __schedule at ffffffffa718f6d0
#1 [ffffb21e002e78f8] schedule at ffffffffa7190a27
#2 [ffffb21e002e7908] __bio_queue_enter at ffffffffa67e121c
#3 [ffffb21e002e7968] blk_mq_submit_bio at ffffffffa67f358c
#4 [ffffb21e002e79f0] __submit_bio at ffffffffa67e1e3c
#5 [ffffb21e002e7a58] submit_bio_noacct_nocheck at ffffffffa67e2326
#6 [ffffb21e002e7ac0] ext4_mpage_readpages at ffffffffa65ceafc
#7 [ffffb21e002e7be0] read_pages at ffffffffa6381d17
#8 [ffffb21e002e7c40] page_cache_ra_unbounded at ffffffffa6381ff5
#9 [ffffb21e002e7ca8] filemap_fault at ffffffffa63761b5
#10 [ffffb21e002e7d48] __do_fault at ffffffffa63d1892
#11 [ffffb21e002e7d70] do_fault at ffffffffa63d2425
#12 [ffffb21e002e7da0] __handle_mm_fault at ffffffffa63d8c6b
#13 [ffffb21e002e7e88] handle_mm_fault at ffffffffa63d95c2
#14 [ffffb21e002e7ec8] do_user_addr_fault at ffffffffa60b34ea
#15 [ffffb21e002e7f28] exc_page_fault at ffffffffa7186e4e
#16 [ffffb21e002e7f50] asm_exc_page_fault at ffffffffa72012a6
RIP: 000055d16159f8d8 RSP: 00007ffdd4c1f340 RFLAGS: 00010206
RAX: 00000000000bec82 RBX: 00007ff2fd00dc82 RCX: 000055d1615b492a
RDX: 00007ffdd4c216b0 RSI: 00000000200bec82 RDI: 000055d185725960
RBP: 00007ffdd4c1f5a0 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000202 R12: 00000000200bec82
R13: 000055d185725960 R14: 00007ffdd4c216b0 R15: 000055d1615b9708
ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b
crash> set 230
PID: 230
COMMAND: "modprobe"
TASK: ffff98ce03ca3040 [THREAD_INFO: ffff98ce03ca3040]
CPU: 0
STATE: TASK_UNINTERRUPTIBLE
crash> bt
PID: 230 TASK: ffff98ce03ca3040 CPU: 0 COMMAND: "modprobe"
#0 [ffffaf9940307840] __schedule at ffffffff9618f6d0
#1 [ffffaf99403078f8] schedule at ffffffff96190a27
#2 [ffffaf9940307908] __bio_queue_enter at ffffffff957e121c
#3 [ffffaf9940307968] blk_mq_submit_bio at ffffffff957f358c
#4 [ffffaf99403079f0] __submit_bio at ffffffff957e1e3c
#5 [ffffaf9940307a58] submit_bio_noacct_nocheck at ffffffff957e2326
#6 [ffffaf9940307ac0] ext4_mpage_readpages at ffffffff955ceafc
#7 [ffffaf9940307be0] read_pages at ffffffff95381d1a
#8 [ffffaf9940307c40] page_cache_ra_unbounded at ffffffff95381ff5
#9 [ffffaf9940307ca8] filemap_fault at ffffffff953761b5
#10 [ffffaf9940307d48] __do_fault at ffffffff953d1895
#11 [ffffaf9940307d70] do_fault at ffffffff953d2425
#12 [ffffaf9940307da0] __handle_mm_fault at ffffffff953d8c6b
#13 [ffffaf9940307e88] handle_mm_fault at ffffffff953d95c2
#14 [ffffaf9940307ec8] do_user_addr_fault at ffffffff950b34ea
#15 [ffffaf9940307f28] exc_page_fault at ffffffff96186e4e
#16 [ffffaf9940307f50] asm_exc_page_fault at ffffffff962012a6
RIP: 0000556b7a7468d8 RSP: 00007ffde2ffb560 RFLAGS: 00000206
RAX: 00000000000bec82 RBX: 00007f5331a0dc82 RCX: 0000556b7a75b92a
RDX: 00007ffde2ffd8d0 RSI: 00000000200bec82 RDI: 0000556ba8edf960
RBP: 00007ffde2ffb7c0 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000202 R12: 00000000200bec82
R13: 0000556ba8edf960 R14: 00007ffde2ffd8d0 R15: 0000556b7a760708
ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2303267
Add INSTALL_OCAMLLIB parameter for allowing ocaml install to a user
defined path. If not defined, fallback to `ocamlc -where`.
Signed-off-by: Antonio Caggiano <quic_acaggian@quicinc.com>
The documentation for Module::Build is completely opaque on how you're
supposed to pass @CFLAGS@ correctly. However I noticed that we were
not passing -g through when generating Guestfs.so:
gcc -lpthread -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld' '-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1' '-Wl,--build-id=sha1' -L/usr/local/lib -fstack-protector-strong -lperl -o blib/arch/auto/Sys/Guestfs/Guestfs.so lib/Sys/Guestfs.o -L../lib/.libs -lguestfs
This debuginfo was not always being generated correctly. (For some
reason it still manages to be generated in Fedora and RHEL 9, but not
in RHEL 10, this part is still unclear to me.)
Anyway it seems we ought to pass at least -g to the linker, and we
might as well pass the full set of @CFLAGS@, hence this change.
The actual output of sfdisk --part-attrs is bizarre and doesn't match
the documentation. After looking at the source from util-linux, fix
the parsing to match what sfdisk produces.
Reported-by: Yongkui Guo
Fixes: commit c6c266a85d
Fixes: https://issues.redhat.com/browse/RHEL-35998
Commit d5b6f1df5f ("daemon: Allow parts of the daemon and APIs to be
written in OCaml.", 2017) contained a bug where in any OCaml function
that returns int64_t, the result was truncated to an int. This
particularly affected part_get_gpt_attributes as that returns large 64
bit numbers, but probably affects other functions too, undetected.
Fixes: commit d5b6f1df5f
This was only theoretically supported, via curl. It's unlikely that
it really worked as it was never tested.
If needed it's better to use nbdkit-curl-plugin instead (this applies
to http and ftp as well).
After many, many years, although libvirt does still often fail to
work, it's generally more secure to stick with libvirt than to try
running qemu directly. The main issue here is that people have
cargo-culted LIBGUESTFS_BACKEND=direct everywhere (even when it's not
necessary).
In case network was requested, but the host lacks both dhclient and
dhcpcd, skip the loop which waits for a resolv.conf update.
This reduces boot time by 10 seconds.
Signed-off-by: Olaf Hering <olaf@aepfle.de>
This was failing with recent Linux:
libguestfs: error: btrfs_subvolume_snapshot: /dir/test3: /dir/test6: ERROR: cannot snapshot '/sysroot/dir/test3': Invalid argument
I tried to change the test to use 1/1000 instead, but that fails with
a different error which I don't understand at all.
As we're not meant to be testing btrfs here, only that libguestfs can
translate between the guestfs API and btrfs commands and we know it
can do that, I simply deleted the sub-test entirely.
sfdisk can now do everything with GPT that sgdisk was needed for
before. In particular we are able to reimplement the following
functions using sfdisk:
- part_set_disk_guid (replace with sfdisk --disk-id)
- part_get_disk_guid
- part_set_disk_guid_random
- part_set_gpt_attributes (sfdisk --part-attrs)
- part_get_gpt_attributes
- part_set_gpt_guid (sfdisk --part-uuid)
- part_get_gpt_guid
- part_set_gpt_type (sfdisk --part-type)
- part_get_gpt_type
This allows us to drop the requirement for gdisk in many cases.
There is only one API remaining which requires gdisk, part_expand_gpt,
which we do not use in our tools. In a prior commit I already moved
this solitary function to a new source file (daemon/gdisk.c).
Fixes: https://issues.redhat.com/browse/RHEL-35998
This "new" parameter was added in 2014:
commit 8eab3194ce1737a167812d5e84d83b0dfc253fac
Author: Karel Zak <kzak@redhat.com>
Date: Mon Sep 15 12:37:52 2014 +0200
sfdisk: add --parttype
The patch also makes --{id,change-id,print-id} deprecated in favour
of --parttype. The original --id is too generic option name and the
--print-id and --change-id are unnecessary and inconsistent with
another sfdisk options (e.g. we don't have --change-bootable)
Also remove an extraneous / incorrect comment about parted. As
history has played out, sfdisk proves to be the better tool and parted
is a PITA.
These were regenerated by Dan Berrange from a known good Debian
container image. In addition they are stripped. The commands were:
$ podman run -it registry.gitlab.com/qemu-project/qemu/qemu/debian-loongarch-cross
# echo 'int main(){}' > bin.c
# loongarch64-unknown-linux-gnu-gcc bin.c -o bin-loongarch64
# loongarch64-unknown-linux-gnu-strip --strip-all bin-loongarch64
# echo '' > lib.c
# loongarch64-unknown-linux-gnu-gcc -shared lib.c -o lib-loongarch64.so
# loongarch64-unknown-linux-gnu-strip --strip-all lib-loongarch64.so
Signed-off-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
These binaries are not meant to be run, they are purely data files
used for testing. Remove the +x attribute to prevent accidentally
running them.
However to avoid breaking the phony guests, we need to chmod +x the
files when we upload them into those guests.
guestmount.1 depends on translated files blocksize-option.pod,
key-option.pod & keys-from-stdin-option.pod (via __INCLUDE__
directives). If these are not yet translated by the time we try to
generate guestmount.1 then it will fail with:
podwrapper.pl: key-option.pod: cannot find input file on path at /builddir/build/BUILD/libguestfs-1.50.1/podwrapper.pl line 672.
This happens especially in parallel builds. Fix this by writing the
guestmount.1 rule explicitly, with the correct dependencies.
I noticed that 1-byte translated POD files were being generated in the
output directory (po-docs/ja/). This seems to have happened because
po4a-translate was generating an error, but because we were
immediately pipeing the output into sed the error was suppressed.
By running them as two separate commands this cannot happen.
Fixes: commit bd896d68c0
The previous code split it on ',' which was completely wrong.
(It reveals the lack of testing however).
Fixes: commit c08032ebe2
Reported-by: Yongkui Guo
The list of patches is below. The one which matters for guestfish is
addition of --key all:... selector.
Andrey Drobyshev (1):
mldrivers: look for bootloader config in /boot/grub/grub.cfg in case of UEFI
Richard W.M. Jones (5):
mlxml: Include <libxml/parser.h> for xmlReadMemory
options/keys.c: Rewrite confusing match statement
options: Rewrite --key documentation fragment
options: Allow --key all:SELECTOR to be used to match any device
mltools/libosinfo-c.c: Fix off-by-one error
Fixes: https://issues.redhat.com/browse/RHEL-19367
Since OCaml 5.1.1, changes to custom blocks caused C finalizers that
call caml_enter_blocking_section to stop working (if they ever did
before). They are relatively inflexible compared to registering an
OCaml finalizer (Gc.finalise) to call Guestfs.close, so use that
instead.
Suggested-by: Guillaume Munch-Maccagnoni
See: https://github.com/ocaml/ocaml/issues/12820
See: db48794fa8
Using 'virt-customize --tar-in some.tar:/dir -a disk.img' will unpack
'some.tar' into '/dir' in the guest. Note that this will not work for
compressed tar files as written since the underlying guestfs_tar_in
function requires the compression type to be set explicitly and
defaults to no compression (it does not auto-detect or default to
compression).
OCaml 5.1 changes the names of these libraries for some reason.
Also in OCaml 5.1, if using those libraries you must link with -lzstd.
Since zstd was already described as "required" (although we only used
it in the appliance), there is no official change to the requirements,
but I have added a configure time check for the library.
Thanks: Jerry James <loganjerry@gmail.com>
Intermittent test failures in virt-filesystems showed that when using
the pvs_full API, the pv_name field in the returned list of structures
was not being reverse translated. As a result internal partition
names could appear in the output of virt-filesystems.
See: https://listman.redhat.com/archives/libguestfs/2023-July/032058.html
The just added --chown option previously used StringPair, splitting
the argument as ‘UID.GID:FILENAME’. However this will not work if we
ever extend this with the ability to use user or group names, since
they may contain dot (but not colon). Add a new StringTriplet type
and split the argument string three ways. The new option becomes:
virt-customize ... --chown UID:GID:FILENAME
Include the following commit from the common submodule:
commit e70d89a58dae068be2e19c7c21558707261af96a
Author: Richard W.M. Jones <rjones@redhat.com>
Date: Sat Jul 15 16:42:06 2023 +0100
customize: Update generated files for --chown with StringTriplet
Updates: commit d8e48bff21
On QEMU 7.2.0+, if "passt" is available, ask QEMU for passt ("stream")
rather than SLIRP ("user") networking.
For this, we need to run passt ourselves. Given that passt daemonizes by
default, start it with our traditional function guestfs_int_cmd_run(). Ask
passt to save its PID file, because in case something goes wrong before
we're completely sure the appliance (i.e. QEMU) is up and running, we'll
need to kill passt, the *grandchild*, ourselves.
Pass "--one-off" to passt (same as libvirt). This way, once we have proof
that QEMU has connected to passt (because the appliance shows signs of
life), we need not clean up passt ourselves -- once QEMU exits, passt will
see an EOF on the unix domain socket, and exit as well.
Passt is way more flexible than SLIRP, and passt normally intends to
imitate the host environment in the guest as much as possible. This means
that, when switching from SLIRP to passt, the guest would see changes to
the following:
- guest IP address,
- guest subnet mask,
- host (= gateway) IP address,
- host (= gateway) MAC address.
Extract the SLIRP defaults into the new macros NETWORK_GW_IP and
NETWORK_GW_MAC, and pass them explicitly to passt. In particular,
"tests/rsync/test-rsync.sh" fails without setting the host address
(NETWORK_GW_IP) properly.
(These artifacts can be verified in the appliance with "virt-rescue
--network", by running "ip addr", "ip route", and "ip neighbor" at the
virt-rescue prompt. There are four scenarios: two libguest backends, times
passt being installed or not installed.)
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20230714132213.96616-8-lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Introduce a small function for creating pathnames for PID files.
guestfs_int_make_pid_path() is something of an amalgamation of
guestfs_int_make_temp_path() [1] and guestfs_int_create_socketname() [2]:
- it creates a pathname under sockdir, like [2],
- it uses the handle's unique counter, like [1],
- it takes a name like both [1] and [2], but the name is not size-limited
like in [2], plus we hardcode the suffix from [1] as ".pid".
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Message-Id: <20230714132213.96616-7-lersek@redhat.com>
Consider the following inverted call tree (effectively a dependency tree
-- callees are at the top and near the left margin):
lazy_make_tmpdir() [lib/tmpdirs.c]
guestfs_int_lazy_make_tmpdir() [lib/tmpdirs.c]
guestfs_int_make_temp_path() [lib/tmpdirs.c]
guestfs_int_lazy_make_sockdir() [lib/tmpdirs.c]
guestfs_int_create_socketname() [lib/launch.c]
lazy_make_tmpdir() is our common workhorse / helper function that
centralizes the mkdtemp() function call.
guestfs_int_lazy_make_tmpdir() and guestfs_int_lazy_make_sockdir() are the
next level functions, both calling lazy_make_tmpdir(), just feeding it
different dirname generator functions, and different "is_runtime_dir"
qualifications. These functions create temp dirs for various, more
specific, purposes (see the manual and "lib/guestfs-internal.h" for more
details).
On a yet higher level are guestfs_int_make_temp_path() and
guestfs_int_create_socketname() -- they serve for creating *entries* in
those specific temp directories.
The discrepancy here is that, although all the other functions live in
"lib/tmpdirs.c", guestfs_int_create_socketname() is defined in
"lib/launch.c". That makes for a confusing code reading; move the function
to "lib/tmpdirs.c", just below its sibling function
guestfs_int_make_temp_path().
While at it, correct the leading comment on
guestfs_int_create_socketname() -- the socket pathname is created in the
socket directory, not in the temporary directory.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Message-Id: <20230714132213.96616-6-lersek@redhat.com>
We generate the <interface type="user"> element on libvirt 3.8.0+ already.
For selecting passt rather than SLIRP, we only need to insert the child
element <backend type='passt'>. Make that child element conditional on
libvirt 9.0.0+, plus "passt --help" being executable.
For the latter, place the new helper function guestfs_int_passt_runnable()
in "lib/launch.c" -- we're going to use the same function for the direct
backend as well.
This change exposes a number of (perceived) shortcomings in libvirt; I've
filed <https://bugzilla.redhat.com/show_bug.cgi?id=2222766> about those.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Message-Id: <20230714132213.96616-3-lersek@redhat.com>
Currently we #define NETWORK_ADDRESS as "169.254.0.0". That's a bug in
libguestfs, but it's been invisible -- thus far it's been canceled out by
*independent* bugs in *both* QEMU and libvirt.
(1) With the direct backend, the current definition of NETWORK_ADDRESS
results in the following QEMU command line option:
-netdev user,id=usernet,net=169.254.0.0/16
According to the QEMU documentation, the "net" property here is supposed
to specify the *exact* IP address that the guest will receive. The guest
however receives 169.254.2.15 (I've checked that with virt-rescue).
In other words, libguestfs doesn't do what the QEMU documentation says,
and QEMU doesn't do what the QEMU documentation says either. The end
result has been good enough -- but only until now.
(2) With the libvirt backend, the current definition of NETWORK_ADDRESS
results in the following domain XML snippet:
<interface type="user">
<model type="virtio"/>
<ip family="ipv4" address="169.254.0.0" prefix="16"/>
</interface>
which libvirt translates, in turn, to
-netdev {"type":"user","net":"169.254.0.0/16","id":"hostnet0"}
According to the domain XML documentation, the @address attribute here is
again supposed to specify the *exact* IP address that the guest will
receive. However, the guest receives 169.254.2.15 (I've checked that with
virt-rescue).
In other words, libguestfs doesn't do what the libvirt documentation says,
and libvirt doesn't do what the libvirt documentation says either. The end
result has been good enough -- but only until now.
Where things break down though is the subsequent passt enablement, in the
rest of this series. For example, when using the libvirt backend together
with passt, libvirt translates the @address attribute to passt's
"--address" option, but passt takes the address *verbatim*, and not as a
subnet base address. That difference is visible in the appliance; for
example, when running virt-rescue on a Fedora 38 image, and issuing "ip
addr". Namely, after enabling passt for the libvirt backend, the
guest-visible IP address changes from 169.254.2.15 to 169.254.0.0, which
is an IP address that makes no sense for an endpoint.
Fix the latent bug by specifying the actual guest IP address we want, in
NETWORK_ADDRESS.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Message-Id: <20230714132213.96616-2-lersek@redhat.com>
The last (only?) caller of guestfs_int_cmd_clear_close_files() disappeared
in commit e4c3968880 ("lib/info: Remove /dev/fd hacking and pass a true
filename to qemu-img info.", 2018-01-23), part of v1.37.36.
Simplify the code by removing guestfs_int_cmd_clear_close_files().
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20230711113906.107340-1-lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
When finalizing the handle we call guestfs_close. This function could
be long-running (eg. it may have to shut down the qemu subprocess), so
release the runtime lock.
Since OCaml 4 the old and confusing caml_enter_blocking_section and
caml_leave_blocking_section calls have been replaced with
caml_release_runtime_system and caml_acquire_runtime_system (in that
order). Use the new names.
Various errors like this:
In function ‘test_fuse’,
inlined from ‘main’ at test-fuse.c:133:11:
test-fuse.c:274:5: error: argument 1 null where non-null expected [-Werror=nonnull]
274 | fclose (fp);
| ^~~~~~~~~~~
In file included from test-fuse.c:26:
/usr/include/stdio.h: In function ‘main’:
/usr/include/stdio.h:183:12: note: in a call to function ‘fclose’ declared ‘nonnull’
183 | extern int fclose (FILE *__stream) __nonnull ((1));
| ^~~~~~
Since OCaml 4.07 (released 2018-07-10) the always-loaded standard
library module has been called Stdlib. The old Pervasives module was
finally removed in OCaml 5.
$ perl -pi.bak -e 's/Pervasives\./Stdlib./g' -- `git ls-files`
OCaml >= 4.07 is now required.
Also update the common submodule with:
commit d61cd820b49e403848d15c5deaccbf8dd7045370
Author: Jürgen Hötzel
Date: Sat May 20 18:16:40 2023 +0200
Add support for OCaml 5.0
Since SELinux userspace v3.4 [1], setfiles command supports "-T nthreads"
option, which allows parallel execution. "-T 0" allows using as many
threads as there're available CPU cores. This might speed up the process
of filesystem relabeling in case the appliance is being run with multiple
vCPUs. The latter is true for at least v2v starting from d2b64ecc67
("v2v: Set the number of vCPUs to same as host number of pCPUs.").
For instance, when running virt-v2v-in-place on my 12-core Xeon host
with SSD, with appliance being run with 8 vCPUs (the upper limit specified
in d2b64ecc67), and on the ~150GiB disk VM (physical size on the host),
I get the following results:
./in-place/virt-v2v-in-place -i libvirt fedora37-vm -v -x
Without this patch:
...
commandrvf: setfiles -F -e /sysroot/dev -e /sysroot/proc -e /sysroot/sys -m -C -r /sysroot -v /sysroot/etc/selinux/targeted/contexts/files/file_contexts /sysroot/^M
libguestfs: trace: v2v: selinux_relabel = 0
libguestfs: trace: v2v: rm_f "/.autorelabel"
guestfsd: => selinux_relabel (0x1d3) took 17.94 secs
...
With this patch:
...
commandrvf: setfiles -F -e /sysroot/dev -e /sysroot/proc -e /sysroot/sys -m -C -T 0 -r /sysroot -v /sysroot/etc/selinux/targeted/contexts/files/file_contexts /sysroot/^M
libguestfs: trace: v2v: selinux_relabel = 0
libguestfs: trace: v2v: rm_f "/.autorelabel"
guestfsd: => selinux_relabel (0x1d3) took 5.88 secs
...
So in my scenario it's getting 3 times faster.
[1] https://github.com/SELinuxProject/selinux/releases/tag/3.4
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
'X' in the setiles' stderr doesn't necessarily mean that option 'X'
doesn't exist. For instance, when passing '-T' we get: "setfiles:
option requires an argument -- 'T'".
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Since RHBZ#726528, filesystem.rpm doesn't include /selinux. setfiles
then gives us the warning: "Can't stat exclude path "/sysroot/selinux",
No such file or directory - ignoring."
Though the warning is harmless, let's get rid of it by checking the
existence of /selinux directory.
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Andrey Drobyshev (2):
inject_virtio_win: add Virtio_SCSI to block_type
inject_virtio_win: write the proper block controller PCI ID to Win registry
Richard W.M. Jones (2):
mlcustomize: Fix overlong comment
mlcustomize: Add accessors for block driver priority list
Roman Kagan (1):
inject_virtio_win: match only vendor/device/revision
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
This was not being set because of some impenetrable problem with
autoconf. The actual line which set the shell variable was simply
being deleted for some reason. Using an m4 definition works.
Updates: commit f68752462e
It seems as if this is just a variation on the previous libvirt
suppression.
==2042391== 61 (48 direct, 13 indirect) bytes in 1 blocks are definitely lost in
loss record 267 of 507
==2042391== at 0x4846464: calloc (vg_replace_malloc.c:1340)
==2042391== by 0x54EE318: g_malloc0 (gmem.c:163)
==2042391== by 0x5017D4C: virClassNew (virobject.c:189)
==2042391== by 0x523CCB9: virDataTypesOnce.lto_priv.0 (datatypes.c:111)
==2042391== by 0x4D8BE42: __pthread_once_slow (pthread_once.c:116)
==2042391== by 0x50345E1: virOnce (virthread.c:44)
==2042391== by 0x523CD6A: virGetConnect (datatypes.c:121)
==2042391== by 0x51FBC1F: virConnectOpenInternal (libvirt.c:893)
==2042391== by 0x51FCC11: virConnectOpenAuth (libvirt.c:1271)
==2042391== by 0x4A3ECAD: guestfs_int_open_libvirt_connection.constprop.0 (li
bvirt-auth.c:224)
==2042391== by 0x4A1D62A: launch_libvirt.lto_priv.0 (launch-libvirt.c:389)
==2042391== by 0x4993739: guestfs_launch (launch.c:114)
(cherry picked from guestfs-tools commit 8790af0266a808c8a04927bb5039be06cbc3da54)
OCaml PCRE.compile doesn't free up regexps if they are stored in
globals. However the valgrind suppression for this matched the old
function name, not the new PCRE2 name. Valgrind errors looked like:
==1799651== 145 bytes in 1 blocks are possibly lost in loss record 2,927 of 4,168
==1799651== at 0x484186F: malloc (vg_replace_malloc.c:381)
==1799651== by 0x4D5F2E5: pcre2_compile_8 (pcre2_compile.c:10250)
==1799651== by 0x29E077: guestfs_int_pcre_compile (pcre-c.c:196)
==1799651== by 0x2B725F: camlOVA__entry (OVA.ml:392)
==1799651== by 0x2859A8: caml_program (in /home/rjones/d/virt-v2v/v2v/virt-v2v)
==1799651== by 0x40F9D8: caml_start_program (in /home/rjones/d/virt-v2v/v2v/virt-v2v)
==1799651== by 0x40FD86: caml_startup_common (in /home/rjones/d/virt-v2v/v2v/virt-v2v)
==1799651== by 0x40FDDC: caml_startup (in /home/rjones/d/virt-v2v/v2v/virt-v2v)
==1799651== by 0x28523F: main (in /home/rjones/d/virt-v2v/v2v/virt-v2v)
(cherry picked from virt-v2v commit 2949109ff9f7a081b1a4b475b9f7bcbef5b45ee0)
OCaml 4.08 introduces a stdlib Option module which looks a bit like
ours but has a number of differences. In particular our functions
Option.may and Option.default have no corresponding functions in
stdlib, although there are close enough equivalents.
This change was automated using this command:
$ perl -pi.bak \
-e 's/Option.may/Option.iter/g; s/Option.default /Option.value ~default:/g' \
`git ls-files`
Update common module to include:
commit cffa077323fafcdfcf78e230c022afa891a6b3ff
Author: Richard W.M. Jones <rjones@redhat.com>
Date: Mon Feb 20 12:11:51 2023 +0000
mlstdutils: Rework the Option module to be compatible with stdlib
This reverts one part of commit 85235aec83 related to reference
counts. Py_BuildValue always increments the reference count (when it
succeeds) and I could not reproduce the Python 3 problem that was
described there.
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
The presence of this file complicates ./configure and you also have to
remember to update it on each release. Replace it with a simple
RELEASE_DATE set in ./configure when the version is updated.
I also had to make a minor change to the generator which was using
this file both to check it was run from the source directory and to
get an exclusive lock. We now use podwrapper.pl.in for this.
It fails when run twice with:
../run cargo clean
error: could not find `Cargo.toml` in `/home/rjones/d/libguestfs/rust` or any parent directory
As this is an optional step don't fail here.
We require libvirt >= 0.10.2, and we included code to check this at
configure-, compile- and run-time. Remove the checks at compile and
run time (keep the ./configure check). Libvirt 0.10.2 was released
over 10 years ago so it's safe to assume that everyone has it by now.
The event callback gets a buffer parameter which is usually something
like a log message. However as it comes from C it is not necessarily
well-formed (eg) UTF-8 but could contain any old sequence of bytes.
In the test case provided by the reporter, we failed to encode the
buffer as 'str' with this error:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 137: unexpected end of data
Use 'bytes' instead. Strictly speaking this changes the type
signature of the callbacks, but our existing Python tests which just
print the buffer using '%s' don't fail and in any case we don't
guarantee the stability of non-C APIs.
Reported-by: Yonatan Shtarkman
See: https://listman.redhat.com/archives/libguestfs/2023-February/030653.html
In the case that building the parameters to the Python event callback
fails, args was returned as NULL. We immediately tried to call
Py_INCREF on this which crashed. Returning NULL means the Python
function threw an exception, so print the exception and return (there
is no way to return an error here - the event is lost).
Reported-by: Yonatan Shtarkman
See: https://listman.redhat.com/archives/libguestfs/2023-February/030653.html
The tool only supports an older version of the diskimage-builder
metadata, and we do not have the time or inclination to update it to a
newer version.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1910039
Run this command across the source:
perl -pi.bak -e 's/(20[012][0-9])-20[12][012]/$1-2023/g' `git ls-files`
and remove changes to po{,-docs}/*.po{,t} (these will be regenerated
later when we run 'make dist').
Update common module to 3253cd99d135d5eb788a0477459b43a269e8cca6
No significant changes that affect virt-v2v now, but it adds some
extra functions to Std_utils that might be used one day.
Cherry picked from virt-v2v commit c7c9dac4f84580190b0e4f7c2e68970192f4bad3
It might be left over from a previous failed run. Best to unlink the
old file before starting qemu-nbd, so there's no possibility of
getting confused later when we wait for the file to appear.
This test fails for reasons I have not fully understood yet. However
one thing I noticed is that the Unix domain socket and PID file used
the tests are placed in the tests/ directory, not the tests/nbd/
subdirectory, so let's fix that:
Starting qemu-nbd fedora-nbd.img -t --pid-file /home/rjones/d/libguestfs-rhel-9.2/tests/nbd.pid --format raw -p 63668 ...
Starting qemu-nbd fedora-nbd.img -t --pid-file /home/rjones/d/libguestfs-rhel-9.2/tests/nbd.pid --format raw -p 60684 ...
Starting qemu-nbd fedora-nbd.img -t --pid-file /home/rjones/d/libguestfs-rhel-9.2/tests/nbd.pid --format raw -k /home/rjones/d/libguestfs-rhel-9.2/tests/unix.sock ...
Fixes: commit 6d32773e81
Add an API to return the build ID of the guest. This to allow a
future change to be able to distinguish between Windows 10 and Windows 11
which can only be done using the build ID.
For Windows we can read the CurrentBuildNumber key from the registry.
For Linux there happens to be a BUILD_ID field in /etc/os-release.
I've never seen a Linux distro that actually uses this.
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
The file(1) manual suggests using -S (disable seccomp) with -z since
the set of system calls provided by the seccomp policy does not allow
the subprocess to run. This is obvious when you use file -z on a
compressed file on a Linux distro that enables file's seccomp policy
(Arch does this, Fedora does not):
$ file -zbsL lib-i586.so.zst
Bad system call
I also fixed some incorrect text in the manual.
Thanks: Toolybird for pointing to this fix
Reported-by: David Runge
Fixes: https://github.com/libguestfs/libguestfs/issues/100
I cannot reproduce the originally reported error:
libguestfs: error: mkfs: xfs: /dev/VG/LV: Filesystem must be larger than 300MB.
Thanks: David Runge
Related: https://github.com/libguestfs/libguestfs/issues/100
$ ls -l `find -name c-ctype.h`
-rwxr-xr-x. 1 rjones rjones 9647 Dec 3 2021 ./gnulib/lib/c-ctype.h
$ chmod -x `find -name c-ctype.h`
$ ls -l `find -name c-ctype.h`
-rw-r--r--. 1 rjones rjones 9647 Dec 3 2021 ./gnulib/lib/c-ctype.h
RPM builds actually gave a warning about this which is how I noticed
the problem:
*** WARNING: ./usr/src/debug/guestfs-tools-1.48.2-2.fc36.x86_64/gnulib/lib/c-ctype.h is executable but has no shebang, removing executable bit
(cherry picked from
guestfs-tools commit 566267a3d447eb97b4a0637adbe3e45c09ba090f)
Commit 133a491677 ("Use guestfsd binary to auto-generate library
dependencies for appliance", October 2020) removed explicit
dependencies for various system packages that the daemon links
directly to, ie. all of these libraries:
$ objdump -p daemon/guestfsd | grep NEEDED | sort
NEEDED libacl.so.1
NEEDED libaugeas.so.0
NEEDED libcap.so.2
NEEDED libc.so.6
NEEDED libgcc_s.so.1
NEEDED libhivex.so.0
NEEDED libjansson.so.4
NEEDED libm.so.6
NEEDED libpcre2-8.so.0
NEEDED librpm.so.9
NEEDED libselinux.so.1
NEEDED libsystemd.so.0
NEEDED libtirpc.so.3
(plus libyara which I don't have installed.)
This avoids having to update these dependencies if they change, eg.
when we switched from PCRE to PCRE2 we did not need to update this
file.
However the same commit also incorrectly removed two apparent library
packages (libldm, libxml2) which the daemon does not link to, but
which we'd like to pull in because of tools they provide, in
particular ldmtool.
Re-add those two explicit dependencies.
Fixes: commit 133a491677
Many warnings such as:
src/session.c: In function 'guestfs_session_internal_test_rstruct':
src/session.c:14755:7: warning: the comparison will always evaluate as 'true' for the address of 'pv_uuid' will never be NULL [-Waddress]
14755 | if (ret->pv_uuid) memcpy (s->pv_uuid, ret->pv_uuid, sizeof (s->pv_uuid));
| ^~~
In file included from src/session.c:40:
../include/guestfs.h:551:8: note: 'pv_uuid' declared here
551 | char pv_uuid[32]; /* this is NOT nul-terminated, be careful when printing */
| ^~~~~~~
Deprecation warnings include:
src/optargs-xfs_growfs.c: In function 'guestfs_xfs_growfs_init':
src/optargs-xfs_growfs.c:311:13: warning: Deprecated pre-processor symbol: replace with "G_ADD_PRIVATE"
311 | o->priv = GUESTFS_XFS_GROWFS_GET_PRIVATE (o);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is based on the same change made here:
https://github.com/nzjrs/osm-gps-map/pull/78/files
setenv can call malloc and is not safe to call here. Glibc is usually
tolerant of this and we haven't had problems before, but if you use
GLIBC_TUNABLES glibc.malloc.check=1 (or any alternate malloc / libc
which serializes) then you would see hangs if starting multiple
libguestfs handles from different threads at the same time.
This commit also updates the common submodule to pick up:
commit 3c64bcdeaf684f05f46f3928b55aadafdfe72720
Author: Richard W.M. Jones <rjones@redhat.com>
Date: Fri Oct 14 11:07:21 2022 +0100
utils: Add function for copying the environment and adding new entries
libguestfs is currently calling setenv at an unsafe location between
fork and exec. To fix this we need a way to copy and modify the
environment before fork and then we can pass the modified environ to
execve-like functions. nbdkit already does the same so use that code.
This function is copied and adapted from here under a compatible license:
https://gitlab.com/nbdkit/nbdkit/-/blob/master/common/utils/environ.c
Thanks: Siddhesh Poyarekar
Downstream patched openssl in Fedora 37+ broke unless
/etc/crypto-policies/back-ends/opensslcnf.conf is present. Files in
this directory are generated by %post rules that use scripting
languages so cannot easily be created by supermin.
Add a symlink to the DEFAULT policy file if the configuration file
doesn't exist.
A symptom of this problem is the error:
Requested hash sha256 is not supported.
Failed to set pbkdf parameters.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2133884
Updates: commit d6ba398825
Downstream patched openssl in Fedora 37+ broke unless
/etc/crypto-policies/back-ends/opensslcnf.conf is present. Files in
this directory are generated by %post rules that use scripting
languages so cannot easily be created by supermin.
Force a copy of the host files into the appliance. This is not ideal
and is hopefully a temporary fix until Fedora's openssl is fixed.
A symptom of this problem is the error:
Requested hash sha256 is not supported.
Failed to set pbkdf parameters.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2133884
If the LVM ("lvm2") feature is not available, these calls would fail.
Really they ought to be part of the "lvm2" optgroup which would cause
the generator to call reply_with_unavailable_feature and generate the
correct ENOTSUP error. When vgscan was originally added in 2010 it
was not added to the optgroup, and when lvm_scan was later added in
2018 and deprecating vgscan, the same mistake was copied.
Before this commit they will try to run the lvm pvscan command which
will fail returning some other error (instead of ENOTSUP).
Fix this by turning the calls into no-ops if the LVM feature is not
available, since scanning for LVM objects when there is no LVM can be
safely turned into a no-op.
See also
https://listman.redhat.com/archives/libguestfs/2022-September/thread.html#29908
Also this updates the common module to pick up a related fix:
commit 4b4a5b84647b1496d034bcdff910930ca5f5c486
Author: Richard W.M. Jones <rjones@redhat.com>
Date: Fri Sep 23 15:18:43 2022 +0100
options: Don't attempt to scan LVs if "lvm2" feature is not available
Reported-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Fixes: 55dfcb2211 ("New API: lvm_scan, deprecate vgscan")
Fixes: 9752039e52 ("New API: vgscan")
gnulib itself has a replacement for <unistd.h> which redefines pipe2
as rpl_pipe2 (etc), which is why the apparently recursive call in the
implementation of pipe2 isn't actually recursive. Since I didn't copy
that file, none of that worked and instead on platforms which have
pipe2 it recursed.
Reported-by: Laszlo Ersek
Fixes: commit 908e41e556
commit 9d40590852e0755d4719adf97122758fa98e90f9
Author: Richard W.M. Jones <rjones@redhat.com>
Date: Tue Aug 16 16:19:29 2022 +0100
options/decrypt.c: Ignore #pragma GCC with clang
Added in 2021 as a workaround for GCC 11 and since fixed upstream.
On macOS (clang):
tilde.c:43:32: error: unknown warning group '-Wanalyzer-null-argument', ignored [-Werror,-Wunknown-warning-option]
^
tilde.c:86:32: error: unknown warning group '-Wanalyzer-null-argument', ignored [-Werror,-Wunknown-warning-option]
^
2 errors generated.
These were added for GCC 11. The problem has been fixed in GCC 12.
On macOS (clang) these produced errors like this:
tsk.c:75:32: error: unknown warning group '-Wanalyzer-file-leak', ignored [-Werror,-Wunknown-warning-option]
^
On macOS, several pages of errors like:
In file included from readdir.c:26:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/rpc/xdr.h:126:3: error: type name requires a specifier or qualifier
bool_t (*x_getlong)(struct __rpc_xdr *, int *);
^
launch.c:191:3: error: implicit declaration of function 'sigemptyset' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
sigemptyset (&sigset);
^
launch.c:192:3: error: implicit declaration of function 'sigaddset' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
sigaddset (&sigset, SIGTERM);
^
launch.c:193:3: error: implicit declaration of function 'sigprocmask' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
sigprocmask (SIG_UNBLOCK, &sigset, NULL);
^
3 errors generated.
commit 1bf092f3c22e93c152de9dea3f5c13df23dc571c
Author: Richard W.M. Jones <rjones@redhat.com>
Date: Tue Aug 16 15:08:40 2022 +0100
utils: Include <stdio.h> for FILE*
These were added in libguestfs 1.14, but never really used. Only a
handful of probes were available. When I was benchmarking libguestfs
in 2016 I didn't even use these probes because better/simpler
techniques were available.
The macOS rpcgen actually generates calls to xdr_uint64_t but doesn't
define them. Despite this we can just use xdr_int64_t instead since
it's just byte swapping.
We have traditionally used custom printf formatters %Q and %R, where
%Q replaces the argument with a shell-quoted string, and %R replaces
the argument with a sysroot-prefixed shell-quoted string. They are
actually pretty useful, but unfortunately only supported by glibc.
We only used them in about a dozen places in the daemon (much code
having been replaced by OCaml which does not need them).
In every remaining case we were constructing a command using code like
this:
asprintf_nowarn (&cmd,
"cd %Q && find -print0 | %s -0 -o -H %s --quiet", ...);
We can replace this with:
char *cmd;
size_t cmd_size;
fp = open_memstream (&cmd, &cmd_size);
fprintf (fp, "cd ");
shell_quote (dir, fp);
fprintf (fp, " && find -print0 | %s -0 -o -H %s --quiet", ...);
fclose (fp);
This code is attempting to construct a grub-install command like:
grub-install --root-directory=/sysroot/boot /dev/sda
In fact it was adding quoting to the --root-directory parameter where
it was not needed (because our "command" function uses exec).
Remove use of %R here (to avoid the extra quoting) and just use the
sysroot prefix directly.
This is required so we can determine the file architecture of
zstd-compressed Linux kernel modules as used by OpenSUSE and maybe
other distros in future.
Note that zstd becomes a required package, but it is widely available
in current Linux distros.
The package names come from https://pkgs.org/download/zstd and my own
research.
OCaml is required to compile libguestfs, however we should still be
able to disable the OCaml bindings. This didn't work because using
--disable-ocaml caused various configure tests to be skipped which are
required to compile the daemon. In particular the check for
caml_alloc_initialized_string, resulting in this error:
pcre-c.c:47:1: error: static declaration of ‘caml_alloc_initialized_string’ follows non-static declaration
caml_alloc_initialized_string (mlsize_t len, const char *p)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Also OCaml gettext is not required by libguestfs. There are no *.ml
files used by libguestfs which require translation.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2108425
Fixes: https://bugs.gentoo.org/820053
Fixes: commit 733d2182b6
qemu (7.0) does not support -cpu max for TCG.
Note this change is necessary but not sufficient for getting
libguestfs to run on RISC-V, because there is also currently no
working path to make -kernel work.
Previously we noted in a comment that stat("hello.txt") is cached (and
not called again), so the test of the link count failed. Something
has changed, possibly in the kernel, but it results in even more
aggressive caching so that an earlier, similar test also fails in the
same way. I checked by enabling debugging that the stat call doesn't
result in guestfs_lstatns being called, and the old value for the
statbuf was being returned.
Call the C-language helper key_store_requires_network() in guestfish and
guestmount.
(Short log for the "common" submodule, commit range
35467027f657..af6cb55bc58a:
Laszlo Ersek (12):
options: fix UUID comparison logic bug in get_keys()
mltools/tools_utils: remove unused function "key_store_to_cli"
mltools/tools_utils: allow multiple "--key" options for OCaml tools too
options: replace NULL-termination with number-of-elements in get_keys()
options: wrap each passphrase from get_keys() into a struct
options: add back-end for LUKS decryption with Clevis+Tang
options: introduce selector type "key_clevis"
options: generalize "--key" selector parsing for C-language utilities
mltools/tools_utils-c: handle internal type error with abort()
mltools/tools_utils: generalize "--key" selector parsing for OCaml utils
options, mltools/tools_utils: parse "--key ID:clevis" options
options, mltools/tools_utils: add helper for network dependency
).
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1809453
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Message-Id: <20220630122048.19335-4-lersek@redhat.com>
On riscv64:
readdir.c: In function ‘guestfs_impl_readdir’:
readdir.c:127:3: error: implicit declaration of function ‘unlink’ [-Werror=implicit-function-declaration]
127 | unlink (tmpfn);
| ^~~~~~
I also changed the #include lines to make them look a bit more
like use in other files.
On older GCC:
debug.c:116:32: error: unknown option after ‘#pragma GCC diagnostic’ kind [-Werror=pragmas]
116 | #pragma GCC diagnostic ignored "-Wanalyzer-mismatching-deallocation"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
make[3]: *** [Makefile:2039: guestfsd-debug.o] Error 1
The upstream bug (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99193)
has now been fixed so the workaround is not necessary with the latest
GCC, so just delete the workaround.
Starting with PHP8, arginfo is mandatory for PHP extensions. This patch
updates the generator for the PHP bindings to generate the arginfo
structures, using the Zend API macros. Only basic arginfo is added,
without full documentation of argument and return types, in order to
ensure compatibility with as many versions of PHP as possible.
In guestfs-tools commit 4fe8a03cd2d3 ('sysprep: remove lvm2's default
"system.devices" file', 2022-04-11), we disabled the use of LVM2's new
"devicesfile" feature, which could interfere with the cloning of virtual
machines.
We suspected in
https://bugzilla.redhat.com/show_bug.cgi?id=2072493#c6
that the same lvm2 feature could affect the libguestfs appliance itself,
but decided in
https://bugzilla.redhat.com/show_bug.cgi?id=2072493#c8https://bugzilla.redhat.com/show_bug.cgi?id=2072493#c10
that this would not be the case, because "appliance/init" already
constructed a pristine LVM_SYSTEM_DIR.
Unfortunately, that's not enough: due to the "use_devicesfile=1" default
(on RHEL9 anyway), some "lvm" invocation, possibly inside the
lvm-set-filter API, *creates* "$LVM_SYSTEM_DIR/devices/system.devices".
And then we get (minimally) warnings such as
> Please remove the lvm.conf global_filter, it is ignored with the devices
> file.
> Please remove the lvm.conf filter, it is ignored with the devices file.
when using the lvm-set-filter API.
Explicitly disable the "devices file" in "appliance/init", and also
whenever we rewrite "lvm.conf" -- that is, in set_filter()
[daemon/lvm-filter.c]. In the former, check for the feature by locating
the devicesfile-related utilities "lvmdevices" and "vgimportdevices". In
the C code, invoke the utilities with the "--help" option instead. (In
"appliance/init", I thought it was best not to call any lvm2 utilities
even with "--help", with our lvm2.conf still under construction there.) If
either utility is available, set "use_devicesfile = 0".
Cc: David Teigland <teigland@redhat.com>
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1965941
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20220530141027.16167-1-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
[lersek@redhat.com: style fix: break "devicesfile_feature" in the function
definition to a new line]
CentOS Stream has:
ID_LIKE="rhel fedora"
which confused the existing script. If there are multiple "likes"
arbitrarily pick the first one in the list.
Fixes: commit 63b722b6c0
We previously didn't bother to check the return values from any librpm
calls. In some cases where possibly the RPM database is faulty, this
caused us to return a zero-length list of installed applications (but
no error indication).
One way to reproduce this is given below. Note this reproducer will
only work when run on a RHEL 8 host (or more specifically, with
rpm <= 4.16):
$ virt-builder fedora-28
$ guestfish -a fedora-28.img -i rm /var/lib/rpm/Packages
$ guestfish --ro -a fedora-28.img -i inspect-list-applications /dev/sda4 -vx
...
chroot: /sysroot: running 'librpm'
error: cannot open Packages index using db5 - Read-only file system (30)
error: cannot open Packages database in
error: cannot open Packages index using db5 - Read-only file system (30)
error: cannot open Packages database in
librpm returned 0 installed packages
...
With this commit we get an error instead:
...
chroot: /sysroot: running 'librpm'
error: cannot open Packages index using db5 - Read-only file system (30)
error: cannot open Packages database in
ocaml_exn: 'internal_list_rpm_applications' raised 'Failure' exception
guestfsd: error: rpmtsInitIterator
guestfsd: => internal_list_rpm_applications (0x1fe) took 0.01 secs
libguestfs: trace: internal_list_rpm_applications = NULL (error)
libguestfs: error: internal_list_rpm_applications: rpmtsInitIterator
libguestfs: trace: inspect_list_applications2 = NULL (error)
libguestfs: trace: inspect_list_applications = NULL (error)
...
Not in this case, but in some cases of corrupt RPM databases it is
possible to recover them by running "rpmdb --rebuilddb" as a guest
command (ie. with guestfs_sh).
See-also: https://bugzilla.redhat.com/show_bug.cgi?id=2089623#c12
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2089623
Fixes: commit c9ee831aff
Reported-by: Xiaodai Wang
Reported-by: Ming Xie
Acked-by: Laszlo Ersek <lersek@redhat.com>
The current code for working out the distro uses the ID entry from
/etc/os-release, and then we map those strings into a smaller set of
values (basically, what package manager to use). However it was
suggested that we should try ID_LIKE first so that distros which act
like other distros would work. On an Arch Linux 32 system:
ID=arch32
ID_LIKE=arch
See-also: https://github.com/libguestfs/libguestfs/issues/81
Thanks: S D Rausty
Instead of continuing on regardless and failing with a weird error
later, error out early if we don't know about the distro and so cannot
set QUERY_FILES_CMD. This avoids situations like
https://github.com/libguestfs/libguestfs/issues/81
The `git-publish`[1] tool is a wrapper around `git-format-patch` and
`git-send-email`. It's a handy tool that automates some of the tedious
aspects of manual patch submission:
- Submitting a patch to the list (with a small config in place) is as
simple as `git publish`
- On next revisions, it automatically increments version numbers
- It auto-copies the list of To: and Cc: from your previous iteration
- It lets you preview/edit emails before submission
- You can also use standard `git-format-patch` and `git-send-email`
options with `git publish`
- You can send pull requests with `git publish --pull-request`
- It also provides custom hooks ... and more[2]
[1] https://github.com/stefanha/git-publish
[2] https://github.com/stefanha/git-publish/blob/master/git-publish.pod
Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com>
In https://bugzilla.redhat.com/show_bug.cgi?id=2082806 we've been
tracking an insidious qemu bug which intermittently prevents the
libguestfs appliance from starting. The symptoms are that SeaBIOS
starts and displays its messages, but the kernel isn't reached. We
found that the kernel does in fact start, but when it tries to set up
page tables and jump to protected mode it gets a triple fault which
causes the emulated CPU in qemu to reset (qemu exits).
This seems to only affect TCG (not KVM).
Yesterday I found that this is caused by using -cpu max which enables
the "la57" feature (5-level page tables[0]), and that we can make the
problem go away using -cpu max,la57=off. Note that I still don't
fully understand the qemu bug, so this is only a workaround.
I chose to disable 5-level page tables for both TCG and KVM, partly to
make the patch simpler, and partly because I guess it's not a feature
(ie. 57 bit linear addresses) that is useful for the libguestfs
appliance case, where we have limited physical memory and no need to
run any programs with huge address spaces.
I tested this by running both the direct & libvirt paths overnight. I
expect that this patch will fail with old qemu/libvirt which doesn't
understand the "la57" feature, but this is only intended as a
temporary workaround.
[0] Article about 5-level page tables as background:
https://lwn.net/Articles/717293/
Thanks: Laszlo Ersek
Fixes: https://answers.launchpad.net/ubuntu/+source/libguestfs/+question/701625
Acked-by: Laszlo Ersek <lersek@redhat.com>
Option "-C" of setfiles(8) causes setfiles(8) to exit with status 1 rather
than status 255 if it encounters relabeling errors, but no other (fatal)
error. Pass "-C" to setfiles(8) in "selinux-relabel", because we don't
want the "selinux-relabel" API to fail if setfiles(8) only encounters
relabeling errors.
(NB even without "-C", setfiles(8) continues traversing the directory
tree(s) and relabeling files across relabeling errors, so this change is
specifically about the exit status.)
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1794518
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20220511122345.14208-3-lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
The documentation currently says that the user should avoid passing
"--selinux-relabel" on the command line if the guest does not support
SELinux. However, the "is_selinux_guest" helper function in
"common/mlcustomize/SELinux_relabel.ml" already turns "--selinux-relabel"
into a no-op if some key SELinux files are absent from the guest, so there
is no need to caution the user.
This change is relevant because the subsequent patches will turn on
"--selinux-relabel" by default, and therefore "is_selinux_guest" will grow
in importance.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1554735
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2075718
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20220510102757.14466-2-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
Representing "iface" in the "drive_create_data" and "drive" structures is
now useless; the direct backend ignores "iface", while the libvirt one
rejects it unless it is empty. Unify both backends -- make them both
ignore "iface". (Which only relaxes the libvirt backend, so it cannot
cause compatibility problems.) This lets us remove the fields. Update the
documentation as well.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1844341
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20220504134155.11832-3-lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
In guestfs_readdir(), the daemon currently sends each XDR-encoded
"guestfs_int_dirent" to the library with a separate send_file_write()
call.
Determine the largest encoded size (from the longest filename that a
"guestfs_int_dirent" could carry, from readdir()'s "struct dirent"), and
batch up the XDR encodings until the next encoding might not fit in
GUESTFS_MAX_CHUNK_SIZE. Call send_file_write() only then.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1674392
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20220502085601.15012-3-lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Currently the guestfs_readdir() API can not list long directories, due to
it sending back the whole directory listing in a single guestfs protocol
response, which is limited to GUESTFS_MESSAGE_MAX (approx. 4MB) in size.
Introduce the "internal_readdir" action, for transferring the directory
listing from the daemon to the library through a FileOut parameter.
Rewrite guestfs_readdir() on top of this new internal function:
- The new "internal_readdir" action is a daemon action. Do not repurpose
the "readdir" proc_nr (138) for "internal_readdir", as some distros ship
the binary appliance to their users, and reusing the proc_nr could
create a mismatch between library & appliance with obscure symptoms.
Replace the old proc_nr (138) with a new proc_nr (511) instead; a
mismatch would then produce a clear error message. Assume the new action
will first be released in libguestfs-1.48.2.
- Turn "readdir" from a daemon action into a non-daemon one. Call the
daemon action guestfs_internal_readdir() manually, receive the FileOut
parameter into a temp file, then deserialize the dirents array from the
temp file.
This patch sneakily fixes an independent bug, too. In the pre-patch
do_readdir() function [daemon/readdir.c], when readdir() returns NULL, we
don't distinguish "end of directory stream" from "readdir() failed". This
rewrite fixes this problem -- I didn't see much value separating out the
fix for the original do_readdir().
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1674392
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20220502085601.15012-2-lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Before commit 3a00c4d179 ("Remove inspection from the C library and
switch to daemon/OCaml implementation") in 2017 the name parameter
passed to add_drive was used by inspection to override the device name
that is determined from fstab. None of our tools ever actually used
this parameter, and when the inspection code was moved inside the
daemon we stopped using this hint field at all.
So it's no longer used, and likely hasn't been used ever. Therefore
document that the field does nothing.
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
These were added to ocamldep in Jan 2012, over 10 years ago. They
were not present in RHEL 6, but we don't care about that now.
(cherry picked from virt-v2v commit f6108bbd661d3e922d07b47f00daa901ab846e59)
cdrtools writes "CDROM" into the Volume Identifier field in the PVD,
whereas genisoimage and xorriso write "ISOIMAGE". Recognise either
string as valid in the test.
Fixes: https://github.com/libguestfs/libguestfs/issues/79
Reported-by: David Runge
If the appliance is a QCOW2 image, function get_root_uuid_with_file()
fails to read ext filesystem signature (0x53EF at offset 0x438) from it.
This results in the following error:
libguestfs: error: /usr/lib64/guestfs/appliance/root: appliance is not
an extfs filesystem
The error itself is harmless, but misleading. So let's skip retrieving
the signature and UUID in case the image contains QCOW2 header. It's
safe because in this case we'll retrieve it later from RAW image dumped
from that QCOW2 by "qemu-img dd".
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
On RHEL 7 (rpm-devel-4.11.3-45.el7.x86_64):
rpm-c.c: In function ‘guestfs_int_daemon_rpm_start_iterator’:
rpm-c.c:97:44: error: ‘RPMVSF_MASK_NOSIGNATURES’ undeclared (first use in this function)
rpmtsSetVSFlags (ts, rpmtsVSFlags (ts) | RPMVSF_MASK_NOSIGNATURES);
^
rpm-c.c:97:44: note: each undeclared identifier is reported only once for each function it appears in
Fixes: commit aa6f8038f8
Bytes.get_uint8 was added in OCaml 4.08. To support OCaml >= 4.04 (in
particular, RHEL 8 has OCaml 4.07) we have to replace this function
with the equivalent native call. We can remove this commit once the
baseline OCaml moves up.
Updates: commit edfebee404
Test fails as it cannot find .xml file. Turns out $pwd=./tests and
and not ./tests/regressions as supposed in script.
Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy@openvz.org>
Test fails as it cannot find .in file. Turns out $srcdir=. (which is ./tests) and
and not ./tests/regressions as supposed in script. Same apply to
$abs_srcdir.
Also put .out file in ./tests/regressions too.
Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy@openvz.org>
Older distros (eg CentOS 6) used SHA-1 RPM package signatures which
some newer distros (eg RHEL 9.0) prevent us from verifying.
This resulted in packages with SHA-1 signatures being skipped by
librpm (there is a warning in debug output, but if you're not looking
at that then the package is silently ignored). In some cases
essential packages like the kernel were skipped, which would be
visible as a failure of virt-v2v. In other cases (eg virt-inspector)
you'd just see fewer installed packages in the <applications> list.
Since verifying package signatures is not essential for inspection,
disable this feature in librpm.
Reported-by: Xiaodai Wang
Thanks: Panu Matilainen
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2064182
Signed-off-by: Richard W.M. Jones <rjones@redhat.com>
This script is used to prepare the list of fixed bugs for release
notes. This commit contains miscellaneous fixes:
- Recognise other bug formats such as direct links.
- Ensure user is logged in so that BZ API doesn't silently truncate
results.
(cherry picked from virt-v2v commit fac1e4fce0d7114bd044b274788de01569089e18)
This was a feature that allowed you to add drives to the appliance
after launching it. It was complicated to implement, and only worked
for the libvirt backend (not "direct", which is the default backend).
It also turned out to be a bad idea. The original concept was that
appliance creation was slow, so to examine multiple guests you should
launch the handle once then hot-add the disks from each guest in turn
to manipulate them. However this is terrible from a security point of
view, especially for multi-tenant, because the drives from one guest
might compromise the appliance and thus the filesystems/drives from
subsequent guests.
It also turns out that hotplugging is very slow. Nowadays appliance
creation should be faster than hotplugging.
The main use case for this was virt-df, but virt-df no longer uses it
after we discovered the problems outlined above.
These APIs were an experimental feature for passing through 9p
filesystems from the host to the libguestfs appliance. It was never
possible to use this without hacking the qemu command line of the
appliance to add such drives by hand. It also didn't fit the
libguestfs model very well. And 9p is generally deprecated in
upstream qemu.
Note that for ABI reasons these APIs are not actually removed, they
have been changed so that they always return an error. These APIs
were actually hard-removed from all versions of RHEL.
See-also: https://bugzilla.redhat.com/921710
User-Mode Linux was an alternative hypervisor that could run the
appliance, instead of using qemu. It had many limitations including
lack of network, and UML support in Linux has been semi-broken for a
long time. It was also slower than KVM on baremeal in general and had
various corner cases which were much slower including the emulated
serial port which made bulk uploads and downloads painful. Also of
course it lacked qemu-specific features like qcow2 or any
network-backed disk, so many disk images could not be opened this way.
This was never supported in RHEL.
See-also: https://bugzilla.redhat.com/1144197
This experimental feature allowed you (in theory) to connect to an
existing instance of the libguestfs daemon. (Again, in theory) it
allowed you to attach to running guests. This didn't work well in
practice. If you want to do this, install qemu-guest-agent inside
your guest instead.
This also disables the --live options in guestfish and guestmount.
(The option now prints an error).
This was never supported in RHEL.
The daemon tests relied on this connection method to perform tests on
a bare daemon, so this removes those tests. They were not especially
valuable.
See-also: https://bugzilla.redhat.com/798980
Commit 6d32773e81 ("tests: Run the tests in parallel.") makes all of
the tests run in the same directory. Previously tests expected to be
run in their own subdirectories and so were freer about using generic
filenames. When run in parallel these filenames now clash. Fix
another case.
Fixes: commit 6d32773e81
Commit 6d32773e81 ("tests: Run the tests in parallel.") combined
multiple Makefiles into one. During this conversion I accidentally
made several mistakes - duplicating the EXTRA_DIST entry for
mountable/test-mountable-inspect.sh and omitting the backslash
continuation of the TESTS line.
Fixes: commit 6d32773e81
Commit e9eaf4d889 ("docs: Split release notes by release.") split the
old guestfs-release-notes(1) page by release, but left now-broken
links to guestfs-release-notes(1) in various places in the
documentation.
The easiest way to fix this is to recreate this page by renaming
guestfs-release-notes-historical(1) as guestfs-release-notes(1) and
adding links to the other release notes pages.
Fixes: commit e9eaf4d889
When running the OCaml bytecode tests, small but apparently
unimportant differences in the stack frames caused the existing
suppressions not to match. Adding an extra "..." will ignore these
differences allowing the tests to pass.
Valgrind output before this commit (note "UnknownInlinedFun" frame):
==1733400== 79 (48 direct, 31 indirect) bytes in 1 blocks are definitely lost in loss record 320 of 518
==1733400== at 0x484A464: calloc (vg_replace_malloc.c:1328)
==1733400== by 0x59685D0: g_malloc0 (gmem.c:136)
==1733400== by 0x531BB59: virClassNew (virobject.c:191)
==1733400== by 0x553A3BF: UnknownInlinedFun (datatypes.c:110)
==1733400== by 0x553A3BF: virDataTypesOnce (datatypes.c:121)
==1733400== by 0x49EA0C8: __pthread_once_slow (pthread_once.c:116)
==1733400== by 0x53312E9: virOnce (virthread.c:44)
==1733400== by 0x553A74A: UnknownInlinedFun (datatypes.c:121)
==1733400== by 0x553A74A: virGetConnect (datatypes.c:133)
==1733400== by 0x54FA94F: virConnectOpenInternal (libvirt.c:895)
==1733400== by 0x54FB883: virConnectOpenAuth (libvirt.c:1277)
==1733400== by 0x50E842A: guestfs_int_open_libvirt_connection.constprop.0 (libvirt-auth.c:224)
==1733400== by 0x50C6120: launch_libvirt.lto_priv.0 (launch-libvirt.c:390)
==1733400== by 0x5040E30: UnknownInlinedFun (launch.c:114)
==1733400== by 0x5040E30: guestfs_launch (actions-3.c:513)
==1733400==
==1733400== 256 bytes in 1 blocks are definitely lost in loss record 481 of 518
==1733400== at 0x484586F: malloc (vg_replace_malloc.c:381)
==1733400== by 0x137A0E: UnknownInlinedFun (memory.c:824)
==1733400== by 0x137A0E: caml_executable_name (unix.c:367)
==1733400== by 0x14C224: UnknownInlinedFun (startup_byt.c:502)
==1733400== by 0x14C224: caml_main (startup_byt.c:457)
==1733400== by 0x11CEE1: main (main.c:41)
LUKS support used to work best if the LUKS device resided on a partition,
and contained a Physical Volume for an LVM Volume Group. This scheme, also
called LVM-on-LUKS, is commonly created by installers of various Linux
distributions. (See RHBZ#1451665.)
Libguestfs now also supports the scheme wherein the LUKS device resides on
an LVM Logical Volume, and contains a filesystem. This is called
LUKS-on-LVM, it is the inverse of the above scheme, and is created by
installers of other Linux distributions. (See RHBZ#1658126.)
Both schemes are now decrypted by libguestfs-based utilities when
inspection is enabled (such as in "guestfish -i", virt-inspector,
virt-v2v), through the inspect_mount() function in utilities written in C,
and through the "inspect_decrypt" function in ones written in OCaml.
We don't seem to need an API like "list-luks-devices", as
"list-dm-devices" returns decrypted (i.e., opened) LUKS devices too; for
example, in the LUKS-on-LVM case:
> ><fs> list-dm-devices
> /dev/mapper/luks-0d619854-ccd5-43b1-8883-991fec5ef713
> /dev/mapper/luks-4e9e7a6f-a68c-42fd-92b4-8f4f2579a389
Thus, the subject TODO section is now out of date, and it's unclear what
remains "to do" there; let's just remove the section.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1658126
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20220223162120.16729-4-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
Create a new (fake) Fedora disk image with two partitions. /dev/sda1 is
the boot partition as usual, /dev/sda2 is used as an LVM PV. The VG has
four LVs, Root and LV1 through LV3.
Each LV holds a LUKS device (with a different key). Each decrypted LUKS
device holds an ext2 filesystem, with "/dev/mapper/Root-luks" holding the
root filesystem.
Each filesystem has a dedicated label (ROOT, LV1, LV2, LV3).
In the test case, run guestfish in inspector mode, twice.
In the first invocation, provide the LUKS passphrases by LV name. Also
specific to the first invocation, fetch the LUKS UUIDs by LV name.
In the second invocation, provide the LUKS passphrases by UUID.
In both invocations, after decryption, check the filesystem labels, the
/dev/mapper/* names generated for the decrypted LUKS block devices, and
the existence of "/etc/fedora-release" on the root filesystem.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1658126
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20220223162120.16729-3-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
$ git shortlog 5b5fac3e0b10..41126802097f
Laszlo Ersek (3):
Demote "Std_utils.wrap" to an internal function in Tools_utils
Tools_utils.wrap: only wrap text for TTYs
add common ("standard") option --wrap
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
GCC 12 gives a warning about our previous attempt to check the length
of the socket path. In the ensuing discussion it was pointed out that
it is easier to get snprintf to do the hard work. snprintf will
return an int >= UNIX_PATH_MAX if the path is too long, or < 0 if
there are other errors such as locale/encoding problems. So we should
just check for those two cases instead.
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/NPKWMTSJ2A2ABNJJEH6WTZIAEFTX6CQY/
Thanks: Martin Sebor and Laszlo Ersek
The 169.254.0.0/16 network specification (for the appliance) is currently
duplicated between the direct backend and the libvirt backend. In a
subsequent patch, we're going to need the network specification in yet
another spot; extract it now to the NETWORK_ADDRESS and NETWORK_PREFIX
macros (simply as strings).
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2034160
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20211223103701.12702-3-lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Tested-by: Richard W.M. Jones <rjones@redhat.com>
The <qemu:commandline> trick we use for adding our virtio-net-pci device
in the libvirt backend can conflict with libvirtd's and QEMU's PCI address
assignment. Try to mitigate that by placing our device in slot 0x1e on the
root bus. In practice this could only conflict with a "dmi-to-pci-bridge"
device model, which libvirtd itself places in slot 0x1e. However, given
the XMLs we generate, and modern QEMU versions, libvirtd has no reason to
auto-add "dmi-to-pci-bridge". Refer to
<https://libvirt.org/formatdomain.html#controllers>.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2034160
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20211223103701.12702-2-lersek@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Tested-by: Richard W.M. Jones <rjones@redhat.com>
Warning 6 "labels-omitted" is not useful. It's fine to omit labels on
positional arguments.
Example:
File "perl_edit.ml", line 30, characters 2-13:
30 | c_edit_file (verbose ()) g (Guestfs.c_pointer g) file expr
^^^^^^^^^^^
Warning 6 [labels-omitted]: label verbose was omitted in the application of this function.
The function is specified as:
external c_edit_file : verbose:bool -> Guestfs.t -> int64 -> string -> string -> unit
The complaint is that the verbose: label has been omitted from the
first argument when the function is called, but IMO this is a
stylistic thing, not a bug.
(cherry picked from
guestfs-tools commit 577f7aee4b1c720f4c4826115b49a0c3870b149e)
In OCaml 4.13:
File "perl_edit.ml", line 30, characters 2-13:
30 | c_edit_file (verbose ()) g (Guestfs.c_pointer g) file expr
^^^^^^^^^^^
Error (warning 6 [labels-omitted]): label verbose was omitted in the application of this function.
(cherry picked from
guestfs-tools commit a4930f5fad82e5358d565b8cf3610970e9646259)
In some places in the generator we were still generating "noalloc".
It was hidden from the previous regexp I used to replace these because
of string escaping.
Updates: commit a69cde79ca
Picks up this commit:
mlstdutils/std_utils.mli: Remove export of deprecated String.copy
Since OCaml strings are at long last immutable, the String.copy
function is deprecated. Stop exporting it from our Std_utils.String.
Any places we use it are wrong and should be fixed.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2030709
Thanks: label@rockylinux.org
---
RWMJ notes: I fixed the original patch so it compiled. This patch
sets osinfo to "rocky8", which doesn't exist in the osinfo db yet.
Arguably we might want to set this to "centos8", but we can see what
libosinfo decides to do. Here is partial virt-inspector output on a
Rocky Linux disk image:
$ ./run virt-inspector -a disk.img
<?xml version="1.0"?>
<operatingsystems>
<operatingsystem>
<root>/dev/rl/root</root>
<name>linux</name>
<arch>x86_64</arch>
<distro>rocky</distro>
<product_name>Rocky Linux 8.5 (Green Obsidian)</product_name>
<major_version>8</major_version>
<minor_version>5</minor_version>
<package_format>rpm</package_format>
<package_management>dnf</package_management>
<hostname>localhost.localdomain</hostname>
<osinfo>rocky8</osinfo>
<mountpoints>
<mountpoint dev="/dev/rl/root">/</mountpoint>
<mountpoint dev="/dev/sda1">/boot</mountpoint>
</mountpoints>
<filesystems>
<filesystem dev="/dev/rl/root">
<type>xfs</type>
<uuid>fed8331f-9f25-40cd-883e-090cd640559d</uuid>
</filesystem>
<filesystem dev="/dev/rl/swap">
<type>swap</type>
<uuid>6da2c121-ea7d-49ce-98a3-14a37fceaadd</uuid>
</filesystem>
<filesystem dev="/dev/sda1">
<type>xfs</type>
<uuid>4efafe61-2d20-4d93-8055-537e09bfd033</uuid>
</filesystem>
</filesystems>
The "is_partition_can_hold_filesystem" function calls
"Parted.part_get_gpt_type" on the partition if:
- the partition table type is GPT,
- or the partition table type is MBR, and the partition is primary or
logical.
The one entry in the fake MBR partition table described in the previous
patch passes the second branch of this check, therefore
"Parted.part_get_gpt_type" is reached, and it invokes "sgdisk -i 1" on the
disk.
Surprisingly (not), while "sgdisk -i" copes fine with valid MBR partition
tables, it chokes on the fake one. The output does not contain the
"Partition GUID code" line, and so "sgdisk_info_extract_field" throws an
exception.
Prevent calling "Parted.part_get_gpt_type" on a bogus MBR partition table,
similarly to the "extended entry in MBR partition table" case; the
difference is that the bogus primary entry, unlike a valid extended entry,
*can* hold a filesystem.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1931821
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
Message-Id: <20211125094954.9713-6-lersek@redhat.com>
"parted" incorrectly reports "loop" rather than "msdos" for the partition
table type, when the (fake) partition table comes from the "--mbr" option
of "mkfs.fat" (in dosfstools-4.2+), and the FAT variant in question is
FAT16 or FAT32. (See RHBZ#2026224.) Work this around by
- parsing the partition table ourselves, and
- overriding "loop" with "msdos" when appropriate.
Note that when the FAT variant is FAT12, "parted" fails to parse the fake
MBR partition table completely (see RHBZ#2026220), which we cannot work
around. However, FAT12 should be a rare corner case in libguestfs usage --
"mkfs.fat" auto-chooses FAT12 only below 9MB disk size, and even "-F 12"
can only be forced up to and including 255MB disk size.
Add the helper function "has_bogus_mbr" to the Utils module; we'll use it
elsewhere too.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1931821
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
Message-Id: <20211125094954.9713-5-lersek@redhat.com>
[lersek@redhat.com: drop "fun" keyword, and use partial application, in
the definition of "sec0at" [Rich]]
Since commit 994ca1f8eb ("daemon: Reimplement 'part_get_mbr_part_type'
API in OCaml.", 2018-05-02), we've not had any calls to
print_partition_table() that would pass a "false" argument for the
"add_m_option" parameter.
Remove the parameter, and inside part_get_mbr_part_type(), remove the dead
branch.
Relatedly, update the comment on the
"print_partition_table_machine_readable" OCaml function, originally from
commit 32e661f421 ("daemon: Reimplement ‘part_list’ API in OCaml.",
2017-07-27). Because print_partition_table() now passes "-m" to "parted"
unconditionally, and there are no use cases left that would *forbid* "-m",
"print_partition_table_machine_readable" is almost equivalent to
print_partition_table() -- modulo the enforcement of the "BYT;" header.
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
Message-Id: <20211125094954.9713-4-lersek@redhat.com>
The directory that readdir() and closedir() work on is BUS_PATH
("/sys/bus/virtio/drivers/9pnet_virtio"), not "/sys/block". Fix the error
messages that are sent when readdir() or closedir() fails.
(The invalid "sys/block" pathname could be a leftover from when the
directory reading logic was (perhaps) copied from "daemon/sync.c".)
Fixes: 5f10c33503
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
Message-Id: <20211125094954.9713-3-lersek@redhat.com>
Search the usage output of "mkfs.fat" for "--mbr[="; cache the result for
further invocations. If the option is supported, pass "--mbr=n" to
"mkfs.fat". This will prevent the creation of a bogus partition table
whose first (and only) entry describes a partition that contains the
partition table.
(Such a bogus partition table breaks "parted", which is a tool used by
libguestfs extensively, both internally and in public libguestfs APIs.)
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1931821
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
Message-Id: <20211125094954.9713-2-lersek@redhat.com>
Another test was removing all disk_*.img files, which removed the file
used by this test. This was ultimately caused by us using parallel
tests.
Put the disk image files back into the tests/gdisk/ subdirectory to
avoid this race.
Fixes: commit 6d32773e81
"my" variable $output masks earlier declaration in same scope at ./gdisk/test-expand-gpt.pl line 73.
"my" variable $end_sectors masks earlier declaration in same scope at ./gdisk/test-expand-gpt.pl line 78.
Actually cargo caches downloaded libraries. The previous change
caused cargo to download and rebuild these after make clean which is
overly aggressive. Use make distclean instead.
Updates: commit 1834f19d20
Commit e597fc5317 ("daemon/yara: fix undefined behavior due to Yara 4.0
API changes", 2021-10-12) prevents the daemon from using such a Yara
version that precedes 4.0.0.
If only yara < 4 is found, treat the library as absent, rather than
attempting and failing to compile the yara module of the daemon. Note the
version requirement in the documentation too.
Suggested-by: Eric Blake <eblake@redhat.com>
Suggested-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20211013133611.21599-4-lersek@redhat.com>
Acked-by: Eric Blake <eblake@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
Eliminate the AC_CHECK_LIB / AC_CHECK_HEADER tests for Yara, for the
following reasons:
- Upstream Yara has provided a pkg-config file since 2015, so the
(now-fixed) pkg-config check should always find it, without the
AC_CHECK_LIB / AC_CHECK_HEADER fallback branch.
- In a subsequent patch, we'll want to test for the incompatible Yara API
changes described at
<https://github.com/VirusTotal/yara/wiki/Backward-incompatible-changes-in-YARA-4.0-API>.
That's easy to do with pkg-config, but impossible with AC_CHECK_*,
without a custom test. Namely, both AC_CHECK_DECLS and AC_CHECK_TYPES
appear unable to check the parameter list of a function pointer typedef
(namely YR_CALLBACK_FUNC and YR_COMPILER_CALLBACK_FUNC). And writing a
dedicated test for this is overkill.
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20211013133611.21599-3-lersek@redhat.com>
Acked-by: Eric Blake <eblake@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
The upstream Yara project has always provided its "libyara/yara.pc.in"
file with "Name: yara" -- ever since its introduction in commit
334bd1a671ca ("Add support for pkg-config", 2015-01-08).
In spite of this, when the (optional) Yara dependency was added to
libguestfs in commit 2e24129da3 ("appliance: add yara dependency",
2017-05-02), PKG_CHECK_MODULES was invoked with [libyara] as
list-of-modules.
(That was clearly a bug. I'm unsure what Debian calls their Yara
pkg-config module, but calling it anything else than "yara" would be a
distribution bug: upstream projects provide pkg-config files specifically
so that dependent projects can find their dependencies *regardless of
distribution*.)
As a consequence, on Fedora today, the PKG_CHECK_MODULES macro always
fails, and only the AC_CHECK_LIB / AC_CHECK_HEADER branch finds Yara.
In a subsequent patch, we'll want to add a version requirement to the
PKG_CHECK_MODULES macro invocation, so at first, fix the pkg-config
identifier of Yara.
Fixes: 2e24129da3
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20211013133611.21599-2-lersek@redhat.com>
Acked-by: Eric Blake <eblake@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
Currently, the Yara test case ("yara/test-yara-scan.sh") fails, with the
following obscure error message:
> ><fs> yara-scan /text.txt
> libguestfs: error: deserialise_yara_detection_list:
Namely, the Yara rule match list serialization / de-serialization, between
the daemon and the library, is broken. It is caused by the following
incompatible pointer passing (i.e., undefined behavior), in function
do_internal_yara_scan(), file "daemon/yara.c":
> r = yr_rules_scan_fd (rules, fd, 0, yara_rules_callback, (void *) path, 0);
^^^^^^^^^^^^^^^^^^^
The prototype of yara_rules_callback() is:
> static int
> yara_rules_callback (int code, void *message, void *data)
however, in Yara commit 2b121b166d25 ("Track string matches using
YR_SCAN_CONTEXT.", 2020-02-27), which was included in the upstream v4.0.0
release, the rules callback prototype was changed as follows:
> diff --git a/libyara/include/yara/types.h b/libyara/include/yara/types.h
> index cad095cd70c2..f415033c4aa6 100644
> --- a/libyara/include/yara/types.h
> +++ b/libyara/include/yara/types.h
> @@ -661,6 +644,7 @@ struct YR_MEMORY_BLOCK_ITERATOR
>
>
> typedef int (*YR_CALLBACK_FUNC)(
> + YR_SCAN_CONTEXT* context,
> int message,
> void* message_data,
> void* user_data);
Therefore, the yara_rules_callback() function is entered with a mismatched
parameter list in the daemon, and so it passes garbage to
send_detection_info(), for serializing the match list.
This incompatible change was in fact documented by the Yara project:
https://github.com/VirusTotal/yara/wiki/Backward-incompatible-changes-in-YARA-4.0-API#scanning-callback
Gcc too warns about the incompatible pointer type, under
"-Wincompatible-pointer-types". However, libguestfs is built without
"-Werror" by default, so the warning is easy to miss, and the bug only
manifests at runtime.
(The same problem exists for yr_compiler_set_callback() /
compile_error_callback():
https://github.com/VirusTotal/yara/wiki/Backward-incompatible-changes-in-YARA-4.0-API#compiler-callback
except that this instance of the problem is not triggered by the test
case, as the rule list always compiles.)
Rather than simply fixing the parameter lists, consider the following
approach.
If Yara's YR_CALLBACK_FUNC and YR_COMPILER_CALLBACK_FUNC typedefs were not
for *pointer* types but actual *function* prototypes, then we could use
them directly in the declarations of our callback functions. Then any
future changes in the param lists would force a "conflicting types"
*compilation error* (not a warning). Illustration:
/* this is *not* a pointer type */
typedef int HELLO_FUNC (void);
/* function declarations */
static HELLO_FUNC my_hello_good;
static HELLO_FUNC my_hello_bad;
/* function definition, with explicit parameter list */
static int my_hello_good (void) { return 1; }
/* function definition with wrong param list -> compilation error */
static int my_hello_bad (int i) { return i; }
Unfortunately, given that the Yara-provided typedefs are already pointers,
we can't do this, in standard C anyway. Type derivation only allows for
array, structure, union, function, and pointer type derivation; it does
not allow "undoing" previous derivations.
However, using gcc's "typeof" keyword, the idea is possible. Given
YR_CALLBACK_FUNC, the expression
(YR_CALLBACK_FUNC)NULL
is a well-defined null pointer, and the function designator expression
*(YR_CALLBACK_FUNC)NULL
has the desired function type.
Of course, evaluating this expression would be undefined behavior, but in
the GCC extension expression
typeof (*(YR_CALLBACK_FUNC)NULL)
the operand of the "typeof" operator is never evaluated, as it does not
have a variably modified type. We can therefore use this "typeof" in the
same role as HELLO_FUNC had in the above example.
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20211011223627.20856-4-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
[lersek@redhat.com: clean up whitespace in "YR_RULE *rule"]
gcc emits the following warning:
> proto.c: In function ‘send_file_complete’:
> proto.c:437:10: error: ‘buf’ may be used uninitialized
> [-Werror=maybe-uninitialized]
> 437 | return send_file_chunk (g, 0, buf, 0);
> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In theory, passing the 1-byte array "buf", with indeterminate contents, to
xdr_bytes() ultimately, could be fine -- assuming xdr_bytes() never reads
the contents of the buffer, due to the buffer size being zero. However,
the xdr_bytes() manual does not seem to guarantee this (it also does not
explicitly permit passing a NULL buffer alongside size=0, which would be
even simpler for the caller).
In order to shut up the compiler, just zero-initialize the buffer --
that's simpler than adding diagnostics pragmas. The "maybe-uninitialized"
warning is otherwise very useful, so keep it globally enabled (per
WARN_CFLAGS / WERROR_CFLAGS).
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20211011223627.20856-3-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
In OCaml 4.13:
Alert ocaml_deprecated_cli: Setting a warning with a sequence of lowercase or uppercase letters,
like 'CDEFLMPSUVYZX', is deprecated.
Use the equivalent signed form: +C+D+E+F+L+M+P+S+U+V+Y+Z+X+52-3.
(cherry picked from
guestfs-tools commit fa4f59e1d99c08d7e0bae2a7cb54f254a6506d67)
The current "arg_string_list" and "free_string_list" implementations go
back to commit b6f01f3260 ("Add Go (language) bindings.", 2013-07-03).
There are two problems with them:
- "free_string_list" doesn't actually free anything,
- at the *first* such g.Internal_test() call site that passes an
Ostringlist member inside the Optargs argument, namely:
> g.Internal_test ("abc",
> string_addr ("def"),
> []string{},
> false,
> 0,
> 0,
> "123",
> "456",
> []byte{'a', 'b', 'c', '\000', 'a', 'b', 'c'},
> &guestfs.OptargsInternal_test{Ostringlist_is_set: true,
> Ostringlist: []string{}
> }
> )
the "golang/run-bindtests" case crashes:
> panic: runtime error: cgo argument has Go pointer to Go pointer
>
> goroutine 1 [running]:
> libguestfs.org/guestfs.(*Guestfs).Internal_test.func7(0xc000018180,
> 0xadfb60, 0xadfb80, 0xc000010048, 0x0, 0x0, 0x0, 0xae3e10, 0xae3e30,
> 0xade3a0, ...)
> golang/src/libguestfs.org/guestfs/guestfs.go:6729 +0xa9
> libguestfs.org/guestfs.(*Guestfs).Internal_test(0xc000018180, 0x4ee3a5,
> 0x3, 0xc000061be8, 0xc000061af8, 0x0, 0x0, 0xc000061a00, 0x0, 0x0, ...)
> golang/src/libguestfs.org/guestfs/guestfs.go:6729 +0x3c9
> main.main()
> golang/bindtests/bindtests.go:77 +0x151e
> exit status 2
> FAIL run-bindtests (exit status: 1)
In Daniel P. Berrangé's words [1],
> You're allowed to pass a Go pointer to C via CGo, but the memory that
> points to is not allowed to contained further Go pointers. So the struct
> fields must strictly use a C pointer.
One pattern to solve the problem has been shown on stackoverflow [2].
Thus, rewrite the "arg_string_list" and "free_string_list" functions
almost entirely in C, following that example.
While this approach is not the most idiomatic Go, as a solution exists
without C helper functions [3], it should still be acceptable, at least as
an incremental improvement, for letting "golang/run-bindtests" pass.
[1] https://listman.redhat.com/archives/libguestfs/2021-September/msg00118.html
[2] https://stackoverflow.com/questions/35924545/golang-cgo-panic-runtime-error-cgo-argument-has-go-pointer-to-go-pointer
[3] https://listman.redhat.com/archives/libguestfs/2021-September/msg00106.html
Cc: "Daniel P. Berrangé" <berrange@redhat.com>
Cc: "Richard W.M. Jones" <rjones@redhat.com>
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20210921192939.32468-1-lersek@redhat.com>
Tested-by: "Richard W.M. Jones" <rjones@redhat.com>
Acked-by: "Richard W.M. Jones" <rjones@redhat.com>
According to xfs_admin(8):
> -c 0|1 Enable (1) or disable (0) lazy-counters in the filesys‐
> tem.
>
> Lazy-counters may not be disabled on Version 5 su‐
> perblock filesystems (i.e. those with metadata CRCs en‐
> abled).
>
> [...]
According to mkfs.xfs(1):
> -m global_metadata_options
> Section Name: [metadata]
> These options specify metadata format options that ei‐
> ther apply to the entire filesystem or aren't easily
> characterised by a specific functionality group. The
> valid global_metadata_options are:
>
> [...]
>
> crc=value
> This is used to create a filesystem which
> maintains and checks CRC information in all
> metadata objects on disk. The value is ei‐
> ther 0 to disable the feature, or 1 to en‐
> able the use of CRCs.
>
> [...]
>
> By default, mkfs.xfs will enable metadata
> CRCs.
Consistently with the above, the first "xfs_admin" test case in
"generator/actions_core.ml", which attempts to disable lazy counters,
always fails:
> 534/550 test_xfs_admin_0
> libguestfs: error: xfs_admin: /dev/sda1: Cannot disable lazy-counters on V5 fs
We can resolve this test failure in three ways:
(1) Extend do_mkfs() [daemon/mkfs.c], possibly even introduce
do_mkfs_xfs(), and permit the caller to specify "-m crc=0" for
mkfs.xfs. Then use this option when the temporary filesystem is
created in the XFS test that disables lazy counters.
(2) Extend the "guestfs_int_xfsinfo" structure in the libguestfs-common
project, with an "xfs_crc" field. Extend parse_xfs_info()
[daemon/xfs.c] to populate the field from "meta-data=...crc=[01]".
Modify the test case to check the following post-condition:
xfs_crc || xfs_lazycount == 0
instead of the current
xfs_lazycount == 0
effectively ignoring "xfs_lazycount" when "xfs_crc" is set.
(3) Remove the test altogether that attempts to disable lazy counters
after filesystem creation.
Given that new XFS filesystems are created with metadata CRCs enabled by
default, and several XFS features depend on metadata CRCs being enabled,
this patch implements option (3).
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20210920052335.3358-4-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
The "test-md-and-lvm-devices" test case creates, among other things, a
RAID0 array (md127) that spans two *differently sized* block devices
(sda1: 20MB, lv0: 16MB).
In Linux v3.14, the layout of such arrays was changed incompatibly and
undetectably. If an array were created with a pre-v3.14 kernel and
assembled on a v3.14+ kernel, or vice versa, data could be corrupted.
In Linux v5.4, a mitigation was added, requiring the user to specify the
layout version of such RAID0 arrays explicitly, as a module parameter. If
the user fails to specify a layout version, the v5.4+ kernel refuses to
assemble such arrays. This is why "test-md-and-lvm-devices" currently
fails, with any v5.4+ appliance kernel.
Until we implement a more general solution (see the bugzilla link below),
work around the issue by sizing sda1 and lv0 identically. For this,
increase the size of sdb1 to 24MB: when one 4MB extent is spent on LVM
metadata, the resultant lv0 size (20MB) will precisely match the size of
sda1.
This workaround only affects sizes, and does not interfere with the
original purpose of this test case, which is to test various *stackings*
between disk partitions, software RAID (md), and LVM logical volumes.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=2005485
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20210920052335.3358-3-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
[lersek@redhat.com: remove stray empty line]
In commit 6d32773e81 ("tests: Run the tests in parallel.", 2021-03-18),
the "abs_srcdir" macro value that the 9p test would see changed from
".../tests/9p" to just ".../tests" -- the last component got dropped.
(Said commit updated some "abs_srcdir"-based references accordingly, for
example under "tests/disks", but "tests/9p/test-9p.sh" was missed.)
Therefore, the guest-visible location of the "/test-9p.sh" file changed to
"/9p/test-9p.sh", and a non-recursive listing of the guest-visible root
directory would not return the file. Thus, the test fails now.
Restore the host-side base directory to ".../tests/9p".
Fixes: 6d32773e81
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20210920052335.3358-2-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
bash can read input from a spawned process, and even provide input to
such process. This feature relies on /dev/fd/ being present. In the
past udev silently created this symlink, so this bash feature worked
more or less by accident. With recent systemd versions, such as 246
which is included in Leap 15.3, the symlink is not created anymore. As
a result scripts, such as /sbin/dhclient-script, fail to work
properly.
This symlink should have been created in version 1 of this variant of /init.
https://bugzilla.opensuse.org/show_bug.cgi?id=1190501
Signed-off-by: Olaf Hering <olaf@aepfle.de>
systemd-sysvinit contains the reboot command, which is used to
properly stop the VM. This was required by other packages, and as a
result always available. Since Leap 15.3 it will not be installed, and
as a result the VM will just panic because /init died.
If the appliance is started with --network, dhclient will run
/usr/sbin/dhclient-script, which in turn may call /sbin/netconfig to
update /etc/resolv.conf. Install sysconfig-netconfig to make sure DNS
resolving actually works.
Signed-off-by: Olaf Hering <olaf@aepfle.de>
Pass $(HIVEX_LIBS) with -cclib under the "daemon_utils_tests_LINK" target;
otherwise the OCaml compiler does not tell the linker where "-lhivex" can
be found, and the linking step fails if "-lhivex" is not on a system
library path.
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20210908133542.19002-3-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
"ocamlc -where" is supposed to "print the location of the standard library
and exit". While this directory contains core OCaml C header files, it
does not contain hivex-related C header files. Trim "guestfsd_CPPFLAGS"
accordingly.
Furthermore, the hivex module for OCaml may exist elsewhere than under the
OCaml standard library directory. Invoke "ocamlfind query hivex" to find
this module. This is what AC_CHECK_OCAML_PKG(hivex) does too, in
"m4/guestfs-ocaml.m4" and "m4/ocaml.m4".
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20210908133542.19002-2-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
This warning is bogus, caused by the analyzer cannot track that len ==
0 if roots == NULL. I just changed the code to make it easier to
analyze, this doesn't fix any real bug.
guestfs-c.c: In function 'guestfs_finalize':
guestfs-c.c:85:9: error: dereference of NULL '0B' [CWE-476] [-Werror=analyzer-null-dereference]
85 | caml_remove_generational_global_root (roots[i]);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Found by GCC -fanalyzer:
xattr.c:478:32: error: '%zu' directive output may be truncated writing between 1 and 19 bytes into a region of size 16 [-Werror=format-truncation=]
478 | snprintf (num, sizeof num, "%zu", nr_attrs);
| ^
xattr.c:478:32: note: directive argument in the range [0, 2305843009213693950]
/usr/include/bits/stdio2.h:71:10: note: '__builtin___snprintf_chk' output between 2 and 20 bytes into a destination of size 16
71 | return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
| ^
Short log:
Laszlo Ersek (2):
Makefile.am: supply missing $(LIBGUESTFS_CFLAGS)
Makefile.am: use $(LIBGUESTFS_LIBS) for linking OCaml test programs
Richard W.M. Jones (4):
tools: Refactor create_standard_options
tools: Add optional --program-name flag.
mltools/test-getopt.sh: Allow test to work for tools version 2
mltools: Add JSON_parser.object_get_bool
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
In commit 6d32773e81 ("tests: Run the tests in parallel.", 2021-03-18),
the working directory relative to which "test-parallel-mount-local" would
be launched (by the test machinery) changed from "tests/mount-local" to
just "tests".
While the relative pathname of the "guestunmount" executable was updated
inside "test-parallel-mount-local" accordingly, the relative pathname of
the FUSE client ("test-parallel-mount-local" itself, just invoked with
"--test") was not. This issue guarantees that the exec call fails in the
child, and so the test case always hangs.
Because we had removed "mount-local" from the end of the working
directory, prepend it now to the relative pathname of the FUSE client
executable.
Fixes: 6d32773e81
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20210902135124.15191-3-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
Each worker thread of "test-parallel-mount-local" performs the following
steps (among others):
(1) it starts an appliance dedicated to that thread, using a private
scratch disk image,
(2) exports a dedicated FUSE mount point on the host, exposing the file
system on the appliance's disk,
(3) launches a child process for manipulating the particular FUSE mount
point on the host,
(4) enters a FUSE request processing loop, translating requests between
the host kernel (coming in via the FUSE mount point) and the
appliance.
Items to note:
- The child process from step (3) consists of a single thread of execution
(see fork() in POSIX): a duplicate of the parent process's respective
worker thread.
- The child process from step (3) blocks on any FUSE mount point access on
the host until the worker thread in the parent process starts processing
FUSE requests, in step (4).
- The FUSE request processing in step (4), in the worker thread living in
the parent process, terminates if and only if the child process unmounts
the FUSE mount point originating from (2).
Should the exec call in step (3) fail for any reason, the child currently
jumps to the "error" label. This is wrong: under the error label, we call
guestfs_close() on the appliance -- but the appliance is owned by the
parent process's worker thread, not the child. What happens is that the
child kills off the appliance while the parent's worker thread is in the
FUSE request processing loop (4).
The "error" label was never meant to be reached by the child process -- if
exec fails for any reason, exit the child immediately. The parent will
remain in the FUSE request processing loop (4) forever, but no state will
be corrupted. For example, using another (interactive) session on the
host, the FUSE mount points can be interacted with, and if all of them are
manually unmounted, the FUSE request processing (4) completes in every
worker thread.
This patch does not fix the primary issue with
"test-parallel-mount-local", but removes "chaos" from the symptoms. The
next patch will fix the actual regression in this test case.
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20210902135124.15191-2-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
If /lib is a symlink to usr/lib, paths to shared libraries as
determined by ld.so may differ from dpkg's file lists.
We turn the filename search pattern into a glob expression by
prefixing it with a '*', so the required packages are found again:
$ dpkg -S /lib/x86_64-linux-gnu/libpcre2-8.so.0
dpkg-query: no path found matching pattern /lib/x86_64-linux-gnu/libpcre2-8.so.0
$ dpkg -S */lib/x86_64-linux-gnu/libpcre2-8.so.0
libpcre2-8-0:amd64: /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0
qemu 6.1 has decided to change qemu-img create so that a backing
format (-F) is required if a backing file (-b) is specified. Since we
don't want to change the libguestfs API to force callers to specify
this because that would be an API break, autodetect it.
This is similar to commit c8c181e8d9 ("launch: libvirt: Autodetect
backing format for readonly drive overlays").
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1998820
When using option `--with-distro`, `HAVE_RPM`, `HAVE_DPKG` and
`HAVE_PACMAN` are not defined and make the configure phase fail.
This makes sure that these conditionals are always defined.
Windows Server 2022 preview is identified as <osinfo>win2k16</osinfo>.
Although current osinfo-db does not have an entry "win2k22", return
this instead.
osinfo-db issue to add win2k22:
https://gitlab.com/libosinfo/osinfo-db/-/issues/82
Inspection information for the guest:
type: windows
distro: windows
product_name: Windows Server 2022 Datacenter
product_variant: Server
version: 10.0
arch: x86_64
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1997446
Reported-by: Yongkui Guo
On RISC-V there is no default machine type. Invoking QEMU requires to
specify a board model with the -M option. So let's define MACHINE_TYPE
as virt.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
$ guestfish -N fs:vfat:2G syslinux /dev/sda1
libguestfs: error: syslinux: Error converting to codepage 850 Invalid argument
...
This happens because of the default codepage requested by syslinux
(code page 850) combined with the appliance missing the iconv
converter for this codepage.
Reported-by: Yongkui Guo
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1990720
When creating and returning a Python struct we were adding fields from
the C struct, but did not reduce the ref count on the temporary value
after it had been moved to the struct, resulting in a memory leak.
Reported-by: 朱丹 <zhudan24@huawei.com>
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1985912
The `test` builtin/binary usually accepts `==` for string comparison, it is
mostly accepted for typos and people being used to double equals, but is not
documented and not always accepted either. Since autoconf uses the default
shell, it might just fail in some cases with:
./configure: 29986: test: xrustc: unexpected operator
./configure: 29990: test: xcargo: unexpected operator
Just change it to single equals as it is done everywhere else.
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
Commit 0f54df53d2 ("build: Remove gnulib") introduced a bug when I
rewrote existing code that used gnulib areadlink().
A missing "continue" statement on the path where fstatat(2) failed
caused fall-through to the case where it tries to use malloc(3) on the
value from the uninitialized stat buf. This caused a huge amount of
memory to be allocated, invoking the oom-killer inside the appliance.
Reported-by: Yongkui Guo
Fixes: commit 0f54df53d2
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1960217
Commit 2f587bbaec ("daemon: Read ISO9660 Primary Volume Descriptor
directly.") changed daemon/isoinfo.ml to read the PVD directly. This
was fine for guestfs_isoinfo_device which opens a device name, but did
not work for ISOs embedded within filesystems opened using
guestfs_isoinfo because we did not chroot into the filesystem first.
Example reproducer (run from the libguestfs source directory):
$ guestfish -N fs -m /dev/sda1 upload ./test-data/test.iso /test.iso
$ guestfish --ro -a test1.img -m /dev/sda1 isoinfo /test.iso
libguestfs: error: isoinfo: open: /test.iso: No such file or directory
After this fix:
$ guestfish --ro -a test1.img -m /dev/sda1 isoinfo /test.iso
iso_system_id:
iso_volume_id: ISOIMAGE
iso_volume_space_size: 2490
[etc.]
Reported-by: Yongkui Guo
Fixes: commit 2f587bbaec
Fixes: https://bugzilla.redhat.com/show_bug.cgi
warning: panic message is not a string literal
--> src/bin/event_leak.rs:9:30
|
9 | Err(e) => panic!(format!(" could not create handle {:?}", e)),
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(non_fmt_panic)]` on by default
= note: this is no longer accepted in Rust 2021
= note: this warning originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
QEMU has deprecated this option:
commit 166310299a1e7824bbff17e1f016659d18b4a559
Author: Daniel P. Berrangé
Date: Tue Oct 20 17:08:27 2020 +0100
os: deprecate the -enable-fips option and QEMU's FIPS enforcement
The -enable-fips option was added a long time ago to prevent the use of
single DES when VNC when FIPS mode is enabled. It should never have been
added, because apps are supposed to unconditionally honour FIPS mode
based on the '/proc/sys/crypto/fips_enabled' file contents.
In addition there is more to achieving FIPS compliance than merely
blocking use of certain algorithms. Those algorithms which are used
need to perform self-tests at runtime.
QEMU's built-in cryptography provider has no support for self-tests,
and neither does the nettle library.
If QEMU is required to be used in a FIPS enabled host, then it must be
built with the libgcrypt library enabled, which will unconditionally
enforce FIPS compliance in any algorithm usage.
Thus there is no need to keep either the -enable-fips option in QEMU, or
QEMU's internal FIPS checking methods.
In RHEL 8+, /usr/etc no longer exists. Since we were looking for this
directory in order to detect a separate /usr partition, those were no
longer detected, so the merging of /usr data into the root was not
being done. The result was incomplete inspection data and failure of
virt-v2v.
All Linux systems since forever have had /usr/src but not /src, so
detect this instead.
Furthermore the merging code didn't work, because we expected that the
root filesystem had a distro assigned, but in this configuration we
may need to look for that information in /usr/lib/os-release (not on
the root filesystem). This change makes the merging work even if we
have incomplete information about the root filesystem, so long as we
have an /etc/fstab entry pointing to the /usr mountpoint.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1949683
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1930133
Fixes: commit 394d11be49
./guestfs.go:4945:16: conversion from _Ctype_char (int8) to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
./guestfs.go:7320:16: conversion from _Ctype_char (int8) to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
./guestfs.go:7335:16: conversion from _Ctype_char (int8) to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
As part of our efforts to clean up and simplify libguestfs, removing
gnulib deletes a large dependency that we mostly no longer use and
causes problems for new users trying to build the library from source.
A few modules from gnulib are still used (under a compatible license)
and these are copied into gnulib/lib/
This gnulib feature abstracts away threads, locks and TLS, and also
allowed libguestfs to be linked with or without pthread. However
since pthread these days is part of glibc and so every program is
using pthread, and we want to get rid of gnulib as a dependency, just
use pthread directly.
It turns out we can read the information we need for the isoinfo API
directly from the ISO9660 PVD. We don't need to use either isoinfo or
xorriso. This also has the advantages of reducing by 1 the number of
dependencies in the appliance, and reducing potential vulnerability to
a crafted ISO file.
This also fixes timezone calculation for the datetime fields.
Thanks: Thomas Schmitt
Updates: commit efb8a766ca
Currently the guestfs_isoinfo and guestfs_isoinfo_device APIs run
isoinfo inside the appliance to extract the information.
isoinfo is part of genisoimage which is somewhat dead upstream.
xorriso is supposedly the new thing. (For a summary of the situation
see: https://wiki.debian.org/genisoimage).
This commit rewrites the parsing from C to OCaml to make it easier to
deal with, and allows you to use either isoinfo or xorriso.
Mostly the same fields are available from either tool, but xorriso is
a bit more awkward to parse.
This Debian page explains the upstream situation:
https://wiki.debian.org/genisoimage
On Fedora, xorriso provides a compatibility program called "mkisofs".
However this is not present in Debian. Hence the choice to look for
the program called "xorrisofs".
libguestfs 1.45.3 now reads the RPM database using librpm, which means
our old phony database created by db_dump can no longer work. Instead
provide a real (but very minimal) sqlite database.
This commit also fixes the virt-inspector test since the RPM database
contents are now different.
The child (chrooted) process wrote its answer on the pipe and then
exited. Meanwhile the parent waiting for the child to exit before
reading from the pipe. Thus if the output was larger than a Linux
pipebuffer then the whole thing would deadlock.
Latest btrfs seems to reject 512 byte sector size. It may be because
of the specific hardware that I'm running the test on. Anyway using a
4K sector size works.
libguestfs: error: mkfs_btrfs: /dev/sda1: ERROR: invalid sectorsize 512, expected range is [4K, 64K]
When shipping the libguestfs tarball we do not necessarily have
common/mlcustomize. If we run the generator in this situation then
don't try to generate files in the non-existent directory.
See-also: commit 7ced2b9354
This was returning "readdir: Invalid argument" which is actually
impossible (readdir(3) cannot fail with EINVAL). It turns out that
the problem is just errno from some other place leaking out.
This was deprecated in btrfs 4.14.1 and recently removed (see
btrfs-progs commit 4bd94dba8a "btrfs-progs: mkfs: remove alloc start
options and docs"). If the option is set simply ignore it.
Use qemu-nbd --pid-file option so we don't have to use an arbitrary
sleep.
Enable all parts of the test, since everything should work now with
various upstream bugs having been fixed in the 8 years since the test
was originally written.
libcap commit 177cd41803 ("A more compact form for the text
representation of capabilities.") changed the format used by
cap_to_text(3), breaking our test. Change the test to cope. This
will break with older libcap now, but there's not a lot we can do
about it.
Before this change the tests ran in about 12m34 and afterwards in
about 6m20, although the real change is more dramatic if you only run
tests from the tests/ subdirectory (as language tests still run serially).
This breaks valgrinding for now, which I intend to fix properly later.
This test was correctly copied into the new guestfs-tools repository
when virt-resize was moved there along with the other tools. However
it was never removed from libguestfs. We were effectively running the
test on the installed virt-resize.
Fixes: commit 733d2182b6
Note this requires libvirt >= 7.1.0 which was only released in March 2021.
With an older libvirt you will see this error:
Original error from libvirt: unsupported configuration: Invalid mode attribute 'maximum' [code=67 int1=-1]
In theory we could check if this is supported by looking at the
libvirt capabilities and fall back, but this commit does not do that,
in the expectation that most people will be using the default backend
(direct) and on Fedora/RHEL we will add an explicit minimum version
dependency to the package.
qemu support has been around quite a bit longer (at least since 2017).
Fixes: commit 30f74f38bd
This is used in virt-win-reg, but that tool have moved out to
guestfs-tools so the dependency is no longer needed by libguestfs
itself.
Fixes: commit 733d2182b6
This was only used for a single rule (check-valgrind-local-guests)
which ran "make check-valgrind" on local guests. This was never
really used by me and was fairly inadvisable anyway, so we can easily
remove it and thus remove the dependency on perl Sys::Virt.
CC virt_builder-setlocale-c.o
setlocale-c.c:38: error: "Val_none" redefined [-Werror]
38 | #define Val_none (Val_int (0))
|
In file included from /usr/lib64/ocaml/caml/alloc.h:24,
from setlocale-c.c:23:
/usr/lib64/ocaml/caml/mlvalues.h:395: note: this is the location of the previous definition
395 | #define Val_none Val_int(0)
|
QEMU has a newish feature (from about 2017 / qemu 2.9) called -cpu max
which is supposed to select the best CPU, ideal for libguestfs.
After this change, on x86-64:
KVM TCG
Direct -cpu max -cpu max
(non-libvirt)
Libvirt <cpu mode="host-passthrough"> <cpu mode="host-model">
<model fallback="allow"/> <model fallback="allow"/>
</cpu> </cpu>
Thanks: Daniel Berrangé
Previously this was in common/utils. However it is not used anywhere
else, and guestfs-tools wants to remove gnulib dependencies, so move
this to libguestfs.
Normally packagers should try to use ./configure --with-guestfs-path=...
However if they do not, then a regression in 1.44 caused the path to
constain the literal string "${exec_prefix}". This was because of
autoconf stupidity.
Try to resolve this by expanding ${exec_prefix} to something sensible.
The expansion will often be wrong (but at least it will now be a valid
path), hence the recommendation to always set --libdir and/or
--with-guestfs-path when configuring.
Fixes: commit 94ff24b880
Fixes: https://bugzilla.redhat.com/1915646
In commit 4bbbf03b8b we started to
ignore bogus GCC 11 warnings. Unfortunately earlier versions of GCC
don't know about those pragmas so give warnings [hence errors in
developer builds] like:
tsk.c:75:32: error: unknown option after '#pragma GCC diagnostic' kind [-Werror=pragmas]
Turn off these warnings.
Updates: commit 4bbbf03b8b
Current GNU tar does not restore all extended attributes. In
particular only user.* capabilities are restored (although all
are saved in the tarball).
To restore capabilities, SELinux security attributes, and other things
we need to use --xattrs-include=*
For further information on the tar bug, see:
https://bugzilla.redhat.com/show_bug.cgi?id=771927
This fixes:
"-supermin: ext2fs_namei: parent directory not found:
/var/lib/rpm: File not found by ext2_lookup"
This because chkconfig contains /var/lib/rpm/alternatives on Mageia
This package in Mageia enables optional support for Windows 10
"CompactOS" (file-level compression), read-only, which is sufficient
for inspecting Windows guests and doing certain types of modifications
to them. Virt-v2v appears to work, but it may be that anything that
involves modifying a compressed file might not work.
See commit e6764a5415
The ELF NEEDED are used to determine guestfsd's library dependencies
with help from the dynamic linker and the package manager.
This was prompted by Debian bug #972241 which was caused by a
libtirpc package renaming in Debian/unstable because the SONAME had
been changed.
Bug originally reported here by trysis:
https://stackoverflow.com/questions/64273334/test-x-in-mounted-filesystem
If the user is root then we override normally access controls in FUSE,
see https://bugzilla.redhat.com/show_bug.cgi?id=1106548.
However this causes test -x to mark all files as executable. We
shouldn't let root execute any file, only ones which have the 'x' bit
set. Therefore this narrows the fix in bug 1106548 so it only applies
to read and write bits.
To test this I created a disk with guestfish which had an executable
and a non-executable file:
$ guestfish -N fs -m /dev/sda1
><fs> touch /file1
><fs> touch /file2
><fs> chmod 0755 /file1
><fs> ll /
total 24
drwxr-xr-x 3 root root 4096 Oct 12 14:04 .
drwxr-xr-x 19 root root 4096 Oct 12 14:04 ..
-rwxr-xr-x 1 root root 0 Oct 12 14:04 file1
-rw-r--r-- 1 root root 0 Oct 12 14:04 file2
drwx------ 2 root root 16384 Oct 12 14:04 lost+found
I then mounted and tested it as non-root:
$ guestmount -a test1.img -m /dev/sda1 /tmp/mnt -v -x
$ ls -l /tmp/mnt
total 16
-rwxr-xr-x. 1 root root 0 Oct 12 15:04 file1
-rw-r--r--. 1 root root 0 Oct 12 15:04 file2
drwx------. 2 root root 16384 Oct 12 15:04 lost+found
$ test -x /tmp/mnt/file1; echo $?
0
$ test -x /tmp/mnt/file2; echo $?
1
and as root:
$ sudo guestmount -a test1.img -m /dev/sda1 /tmp/mnt -v -x
$ test -x /tmp/mnt/file1; echo $?
0
$ test -x /tmp/mnt/file2; echo $?
0
In the debug output for non-root we can see the difference:
libguestfs: /file1: testing access mask X_OK: caller UID:GID = 1000:1000, file UID:GID = 0:0, file mode = 100755, result = OK
libguestfs: /file2: testing access mask X_OK: caller UID:GID = 1000:1000, file UID:GID = 0:0, file mode = 100644, result = EACCESS
and for root:
libguestfs: /file1: testing access mask X_OK: caller UID:GID = 0:0, file UID:GID = 0:0, file mode = 100755, result = OK
libguestfs: /file2: testing access mask X_OK: caller UID:GID = 0:0, file UID:GID = 0:0, file mode = 100644, result = OK
After this commit the root output changes to this (ie. same decision
as non-root):
libguestfs: /file1: testing access mask X_OK: caller UID:GID = 0:0, file UID:GID = 0:0, file mode = 100755, result = OK
libguestfs: /file2: testing access mask X_OK: caller UID:GID = 0:0, file UID:GID = 0:0, file mode = 100644, result = EACCESS
When guestfs_lvm_canonical_lv_name was called with a /dev/dm* or
/dev/mapper* name which was not an LV then a noisy error would be
printed. This would typically have happened with encrypted disks, and
now happens very noticably when inspecting Windows BitLocker-
encrypted guests.
This commit hides this error in all cases, although it is still logged
to debug. See comment and the thread below for detailed rationale.
https://www.redhat.com/archives/libguestfs/2020-October/thread.html#00055
Previously callers were unable to distinguish a regular error (like an
I/O error) from the case where you call this API on something which is
valid but not a logical volume. Set errno to a known value in this
case.
In case any bare filesystems were decrypted using cryptsetup-open,
they would appear as /dev/mapper/name devices. Since list-filesystems
did not consider those when searching for filesystems, the unencrypted
filesystems would not be returned.
Note that previously this worked for LUKS because the common case
(eg. for Fedora) was that whole devices were encrypted and thoes
devices contained LVs, so luks-open + vgactivate would activate the
LVs which would then be found by list-filesystems. For Windows
BitLocker, the common case seems to be that each separate NTFS
filesystem is contained in a separate BitLocker wrapper.
This commit deprecates luks-open/luks-open-ro/luks-close for the more
generic sounding names cryptsetup-open/cryptsetup-close, which also
correspond directly to the cryptsetup commands.
The optional cryptsetup-open readonly flag is used to replace the
functionality of luks-open-ro.
The optional cryptsetup-open crypttype parameter can be used to select
the type (corresponding to cryptsetup open --type), which allows us to
open BitLocker-encrypted disks with no extra effort. As a convenience
the crypttype parameter may be omitted, and libguestfs will use a
heuristic (based on vfs-type output) to try to determine the correct
type to use.
The deprecated functions and the new functions are all (re-)written in
OCaml.
There is no new test here, unfortunately. It would be nice to test
Windows BitLocker support in this new API, however the Linux tools do
not support creating BitLocker disks, and while it is possible to
create one under Windows, the smallest compressed disk I could create
is 37M because of a mixture of the minimum support size for BitLocker
disks and the fact that encrypted parts of NTFS cannot be compressed.
Also synchronise with common module.
This brings libguestfs into line with other projects which have a
separate include/ directory for the public header.
It's also the case that <guestfs.h> has never particularly belonged in
the lib/ subdirectory. Some tools add -Ilib/ but they only need
<guestfs.h> and not any other headers from that directory, and
separating out the public header allows us to clean those up. This is
certainly the case for examples, and some language bindings and some
tests.
In future I'm hopeful we can use this as the basis to tease out other
dependencies, as a prelude to separating them out from the repo.
Use a LINGUAS file with the list of available translations instead of
defining them in a make variable. This way Weblate will be able to
update the list using an available addon, and we do not need to list
those not built.
Accordingly, rename the variable with built languages to
'linguas_translated'.
Start the message extraction from the toplevel source directory, so the
file references are relative to that, instead of relative to this
po-docs subdirectory.
Also update/regenerate podfiles accordingly.
Start the message extraction from the toplevel source directory, so the
file references are relative to that, instead of relative to this po
subdirectory.
With the Weblate adoption, we let it update the po files from the
catalog template. The po4a behaviour of extracting the template,
merging the existing translations, and creating the translated PODs at
once is problematic. Hence, split the extraction and the translated POD
generation in two.
Use po4a-gettextize to extract the catalog template only, not doing it
anymore automatically at each build. There is no more need for a
po4a.conf file.
Use po4a-translate to create translated PODs from the po files, keeping
the fixup of the generated files (to avoid spurious =encoding, etc).
Add a silent rule to hide the po4a-translate command lines by default.
These changes also allow us to get rid of the POD existance checks with
associated error message pointing to the update-po rule. Now each
translated POD file is generated because of make dependency, and it
depends only on its po file.
Signed-off-by: Pino Toscano <ptoscano@redhat.com>
The guestfish man page uses also additional POD snippets, so list them
as dependencies to make sure they are up-to-date.
This does not change the behaviour at the moment, however it will matter
when each traslated POD file will be generated on its own.
Weblate will handle the update of the po files from the translation
catalog, so avoid stomping on its feet by doing the same.
The translation catalog will be regenerated manually periodically.
Use a LINGUAS file with the list of available translations instead of
defining them in a make variable. This way Weblate will be able to
update the list using an available addon.
Signed-off-by: Pino Toscano <ptoscano@redhat.com>
For the appliance of the QCOW2 format, the function get_root_uuid()
fails to get the UUID of the disk image.
In this case, let us read the first 256k bytes of the disk image with
the 'qemu-img dd' command. Then pass the read block to the 'file'
command.
Suggested-by: Denis V. Lunev <den@openvz.org>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Even though it seems clear from the code that roots cannot be used
uninitialized, GCC 10.1 cannot seem to work it out (possibly an LTO
bug). Easiest way out here is to just initialize it.
rescue.c:396:37: error: 'roots' may be used uninitialized in this function [-Werror=maybe-uninitialized]
396 | CLEANUP_FREE_STRING_LIST char **roots;
| ^
I couldn't get GCC 10.1 to ignore this warning any longer, possibly
because I am using LTO. In any case dereferencing a pointer is
undefined behaviour, so let's use GCC's __builtin_trap() function
instead (also supported by clang).
debug.c: In function 'debug_segv':
debug.c:1002:8: error: null pointer dereference [-Werror=null-dereference]
1002 | *ptr = 1;
| ^
Generated code in virt-builder causes the error below. Since we
cannot control what is generated by bison, increase the limit.
CCLD virt-index-validate
index-parse.c: In function 'yyparse':
index-parse.c:1857:1: error: the frame size of 5152 bytes is larger than 5000 bytes [-Werror=frame-larger-than=]
1857 | }
| ^
lto1: all warnings being treated as errors
lto-wrapper: fatal error: gcc returned 1 exit status
Argon2 is the default LUKS Password-Based Key Derivation Function
(PBKDF) for some new guests such as RHEL 8.2 and Fedora. It is
designed to be "memory hard", meaning that by design it requires large
amounts of memory, making it expensive to brute-force. Unfortunately
the default for guests which had more than a few GB of RAM at install
time is to require about 1 GB of RAM to decrypt the block device,
which is considerably larger than the default available in the
libguestfs appliance.
To make it possible to open these encrypted disks we need to make the
appliance larger. This could be done as a one-off, and the current
workaround is simply to set LIBGUESTFS_MEMSIZE=2048 or a similar
amount. However since we don't know in advance whether we could be
dealing with an encrypted disk, partition, etc. or what PBKDF it uses,
the only way to deal with this in all circumstances is to increase the
default memsize. This commit increases it quite a lot (768 -> 1280)
which is unfortunate.
Note as there is some confusion on this point: Since libguestfs does
not attempt to decrypt disks in parallel, you only need ~ 1GB in
total, not per encrypted disk.
For a reproducer, see:
https://bugzilla.redhat.com/show_bug.cgi?id=1837765#c14
Python 2 reached end of life on 2020-01-01:
https://python3statement.org/https://pythonclock.org/
The minimum version required is now Python 3.4 (since that is the
version in Debian oldoldstable), but 3.6 is the minimum version that
I actually test.
Ubuntu 20.04 could not be built because their installer has changed in
a way which is not compatible with the current build script. This
needs some work to fix.
Replace the use of liberl_interface, which is removed in Erlang 23,
by libei. The implementation uses the ei_decode_iodata() function
which has been introduces only for Erlang 23, so it doesnt work with
earlier Erlang versions.
This new operation removes the Kerberos /etc/krb5.keytab file from the
guest.
Thanks to Christian Heimes and François Cami for the hints.
Related to RHBZ#1789592.
This new operation unenrolls the guest from a IPA server offline, by
removing the configuration files and certificates.
Thanks to Christian Heimes and François Cami for the hints.
Add a simple side effect to make operation flag that a regeneration of
the system CA store is needed. In case it is flagged, regenerate the
system CA store directly, or using a firstboot script in case of
incompatible architectures.
This change is almost a no-op, since no operation requires the
regeneration of the system CA store yet.
For unclear reasons this broke recently, although the breakage is only
reproducible in Fedora Koji. It appears to be caused by the
calculation of the internal autoconf variable $acl_libdirstem
changing. In the reproducer system:
checking for the common suffixes of directories in the library search path... lib,lib,lib64
On my local system:
checking for the common suffixes of directories in the library search path... lib64,lib64
My local system would be the correct one. The actual code that
calculates this does some crazy stuff with ‘gcc -print-search-dirs’
which would hint that this output has changed in some way that subtly
breaks the generated configure script. However even with match GCC
versions on my local system I could still not reproduce the issue.
None of this matters, as the easiest fix here is simply to stop using
internal autoconf variables at all. I also added some AC_MSG_*
statements so we can more easily see what's going on in the configure
output.
Fixes commit 0f79400c7f.
The kernel returns xattr names in a slightly peculiar format. We
parsed this format several times in the code. Refactor this parsing
so we only do it in one place.
At the moment it is empty, so probably it does not exist. Remove it to
avoid adding spurious content to the pkg-config file in case that
variable will get a value in the future.
You have to use:
../libguestfs/run ./configure
../libguestfs/run make
Use of the second ../libguestfs/run against make is unfortunate but I
believe it's unavoidable due to the way that ocamlfind works.
We use a similar trick to libvirt to allow external C programs that
use libguestfs to be compiled against the built (but not installed)
libguestfs with:
../libguestfs/run ./configure
make
What actually happens is we have a second pkg-config file
(lib/local/libguestfs.pc) which points to the locally built
libguestfs. The ./run script sets up PKG_CONFIG_PATH to point to this
directory. Assuming that ./configure is using pkg-config/pkgconf and
not some other half-baked solution it will pick up the libguestfs.pc
file from here which will set CFLAGS and LIBS appropriately.
Causes this error if you compile libguestfs 1.42.0 from the tarball
without invoking the generator:
make[4]: *** No rule to make target '../common/mlv2v/uefi.ml', needed by 'libguestfs.pot'. Stop.
Appliance device names are not reliable since the kernel no longer
enumerates virtio-scsi devices serially. Instead get the UUID of the
appliance and pass this as the parameter.
Note this requires supermin >= 5.1.18 (from around July 2017).
Linux from around 5.6 now enumerates individual disks in any order
(whereas previously it enumerated only drivers in parallel). This
means that /dev/sdX ordering is no longer stable - in particular we
cannot be sure that /dev/sda inside the guest is the first disk that
was attached to the appliance, /dev/sdb the second disk and so on.
However we can still use SCSI PCI device numbering as found in
/dev/disk/by-path. Use this to translate device names in and out of
the appliance.
Thanks: Vitaly Kuznetsov, Paolo Bonzini, Dan Berrangé.
In the guestfs_disk_create API we have traditionally allowed you to
set backingfile without setting backingformat. The meaning of this is
to let qemu autodetect the backing format when opening the overlay
disk.
However libvirt >= 6.0 refuses to even pass such disks to qemu (see
https://bugzilla.redhat.com/show_bug.cgi?id=1798148).
For this reason, move the autodetection earlier and make it explicit.
We now autodetect the format of the backing disk at the time of
creation of the overlay, and set that as the backing format in the
overlay disk itself, allowing libvirt to open the disk later.
This symbol is not present in Python 2.7 or 3.6. It's not really
necessary to call this, it just avoids a crash in certain corner cases
when the interpreter is shutting down. So make the call conditional
on the function existing.
Fixes commit e6f9e0b0f6.
This function doesn't work reliably with the proposed change to device
name translation. The reason is that strings returned by
Devsparts.list_devices contained translated names, so their indexes
did not correspond to the untranslated names used outside the
appliance..
We can avoid this and make the function much simpler and faster by
implementing it on the library side instead.
Although it's highly unlikely in normal use, while testing device name
translation patches it did happen and caused the test to segfault
instead of exiting with an error.
Already outdated, but rounded ;)
I literally just opened the 5yrs logo, changed the text and then done:
inkscape -z -o logo/fish-10yrs.{png,svg}
cp {logo,website}/fish-10yrs.svg
and then updated the rest of the files.
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
The current way to get the size of a filesystem is to query the size in
bytes of the device. However, this gives the whole size of the device
where a filesystem is stored, and it does not consider the actual size
for which the filesystem is configured (e.g. in case it was shrunk).
A simple reproducer for this is:
$ guestfish -N test.img=fs:ext4:2G resize2fs-size /dev/sda1 1073741824
As result, try to mount the filesystem, and get its actual statistics to
determine its full size. In case mounting fails, fall back to the
previous method, which is still a good value in the majority of the
cases.
Thanks to: Erik Skultety.
This patch adds '--blocksize' command line option for virt-make-fs
tool. This option allows specifying disk sector size as described in
'guestfs_add_drive_opts' libguestfs API.
On current Fedora releases the ocaml modules will fail to
link unless CFLAGS contains -fPIC.
The autogen.sh script only updates the 'gnulib' submodule,
and so the build will fail due to the missing 'common'
submodule. This needs to be manually initialized at checkout.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
The first split does not care about the whole string, it is just trying to get
the command name in front, so triml is just right.
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
Seeing `g.add_drive_opt :readonly => 1` allows one to imply
that ensuring writable access to drive should happen via
`g.add_drive_opt :readonly => 0`. However, the passed option
value gets passed down to C according to Ruby Boolean semantics,
that is, any value apart from `false` and `nil` will be true
(see RTEST in Ruby C API).
So its more idiomatic and provides a better hint if we use
`g.add_drive_opt :readonly => true` in Ruby samples.
Go API functions returned (<val>, *GuestfsError) that made
code like this fail to build:
n, err := os.Stdin.Read(buf)
if err != nil {
log.Fatal(err)
}
n, err = g.Pwrite_device(dev, buf[:n], off)
...
As err should be of error (interface) type as of the stdlib call,
and should be of *GuestfsError type as of the libguestfs call.
The concrete error value that libguestfs functions return can be
a *GuestfsError, but the function signature should have (<val>, error)
as return value.
This patch adds '--blocksize' command line option for virt-get-kernel
tool. This option allows specifying disk sector size as described in
'guestfs_add_drive_opts' libguestfs API.
This package in Fedora enables optional support for Windows 10
"CompactOS" (file-level compression), read-only, which is sufficient
for inspecting Windows guests and doing certain types of modifications
to them. Virt-v2v appears to work, but it may be that anything that
involves modifying a compressed file might not work.
I couldn't find the equivalent package in Debian or SUSE. It's
available in Arch AUR although I didn't verify that part of the change
actually works there (but should be safe because supermin ignores
packages that are not known about on the target system).
Nowadays there are hard drives and operating systems which support
"4K native" sector size. In this mode physical and logical block size
exposed to the operating system is equal to 4096 bytes.
GPT partition table (as a known example) being created in this mode will
place GPT header at LBA1 which is 4096 bytes. libguetfs is unable to
recognize partition table on such physical block devices or disk images.
The reason is that libguestfs appliance will look for a GPT header at
LBA1 which is seen at 512 byte offset.
In order to fix the issue we need a way to provide correct logical block
size for attached disks. Fortunately QEMU and libvirt already provides
a way to specify physical/logical block size per disk basis.
After discussion in a mailing list we agreed that physical block size is
rarely used and is not so important. Thus both physical and logical
block size will be set to the same value.
In this patch one more optional parameter 'blocksize' is added
to add_drive_opts API method. Valid values are 512 and 4096.
add_drive_scratch has the same optional parameter for a consistency and
testing purpose.
add-domain and add_libvirt_dom will pass logical_block_size value from
libvirt XML to add_drive_opts method.
Libvirt 6.0 now requires that every disk in the backing chain has an
explicit backing format. For example this will be rejected by
libvirt:
qemu-img create -f qcow2 -b backing-disk disk.qcow2
with the error:
Original error from libvirt: Requested operation is not valid:
format of backing image 'backing-disk' of image 'disk.qcow2' was not
specified in the image metadata (See
https://libvirt.org/kbase/backing_chains.html for troubleshooting)
[code=55 int1=-1]
Instead you have to use the -F option to specify the format, eg:
qemu-img create -f qcow2 -b backing-disk -F raw disk.qcow2
I believe this warning is bogus, but simply initializing the local
variable is enough to avoid it.
log.c: In function 'do_log':
log.c:390:7: error: 'comm_len' may be used uninitialized in this function [-Werror=maybe-uninitialized]
390 | printf (" %.*s", (int) comm_len, comm);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The current code was broken, as the field 1 of the exception value is
the error code (int), not an error string, and thus it would have
crashed. This did not happen in practice, as all the usage of
ocaml-augeas were only in the inspection code with ad-hoc exception
catching blocks.
Other than fixing the aforementioned issue, enhance the error reporting
to be as close as possible to what the current AUGEAS_ERROR() macro
does: error message, error minor message (if available), error details
(if available).
Enhance the UEFI firmware lookup function with the information on the
libvirt firmware autoselection, allowing it to return a value to use for
the appliance.
At the moment no firmware is selected this way, so there is no behaviour
change.
This fails with Fedora 32, and possibly earlier versions:
checking if OCaml ‘-runtime-variant _pic’ works... no
The reason is:
$ ocamlc -runtime-variant _pic test.ml -o test
File "test.ml", line 1:
Error: Cannot find file camlheader_pic
which may even be a packaging error in the Fedora package. However it
makes no sense to test the bytecode compiler since we don't use it on
any architecture we care about and bytecode doesn't even contain a
linked runtime. Changing the test to use ocamlopt instead of ocamlc
fixes the problem.
Properly mark the long line with the license classifier in setup.py:
the classifier is long on its own, and changing the code to stay within
79 characters would result in worse code.
Fix continuation indentation, and whitespaces around operators.
This is just code reformatting, with no behaviour changes; no content
changed beside whitespaces, so "git diff -w" gives an empty diff.
Add or remove empty lines to match the needed ones around
blocks/functions/etc.
Adapt the generation of guestfs.py to emit the separating empty line
before adding a new function/alias, rather than after it.
This is just formatting, no behaviour changes.
In Python 2 code branches use PyString_FromStringAndSize directly
instead of the guestfs_int_py_fromstringsize wrapper (which calls
PyString_FromStringAndSize anyway on Python < 3).
The PyString API is specific to Python < 3, replaced by PyBytes and
PyUnicode in Python 3+; hence, drop the compile time check, and use the
right API depending on the Python version.
If it is not possible to detect the distribution of a Linux OS, do not
propose "unknownX.Y" (where X is the major version number, and Y the
minor) as short osinfo ID. Just return "unknown" instead.
The OCaml and Python bindings directly use the utils.c source in
common/utils, mostly for guestfs_int_free_string_list. That source
contained also functions using gnulib functions, however without
linking to gnulib. When building with default build flags (e.g. without
as-needed mode), the gnulib symbols cannot be resolved, leading to
unusable OCaml and Python libraries.
As solution, update the common submodule to get the split of the split
of the stringlist functions in common/utils, and adapt the OCaml and
Python bindings:
- both now use stringlists-utils.c instead of utils.c
- fix the Python distutils setup to include only the sources really
needed
Move the interal static libraries as the last items in the list of
libraries of guestfsd, to make sure their symbols are used for all the
other libraries. This is because GCC resolves the symbols looking at
the arguments from the beginning to the end of the command line.
This currently does not cause failures, however it "just works" because
of the tricky situation set up.
The situation is the following:
1) common/utils contains few utility sources: one of them is utils.c,
which contains various functions -- for example
guestfs_int_free_string_list and guestfs_int_drive_name --, it is built
as utils.o, and bundled in the static library libutils.a
2) common/mlutils builds a OCaml library with bindings for some utility
functions in libutils.a, in particular guestfs_int_drive_name (but not
guestfs_int_free_string_list); there are two versions of this library,
one OCaml library (dllmlcutils.so) that links with libutils.a, and one
static library (libmlcutils.a), which cannot specify the libraries it
links to (as it is static)
3) when the daemon is linked, the command line was the following
(simplified):
$ gcc [...] -o guestfsd guestfsd-9p.o other_daemon_object.o [...] \
../common/utils/.libs/libutils.a [...] -lmlcutils [...]
Some of the objects of the daemon itself use
guestfs_int_free_string_list, and thus the compiler opens libutils.a
(it is after the objects in the command line) and picks utils.o, which
contains also guestfs_int_drive_name (not used directly in the daemon);
when linking later on with libmlcutils.a, the symbols for this static
library (like guestfs_int_drive_name) are already resolved, and thus
all the symbols are resolved, and the linking succeeds
This fragile situation can be easily broken by moving e.g.
guestfs_int_drive_name out of common/utils/utils.c to a new source (say
utils2.c) still built as part of libutils.a: since nothing before
-lmlcutils actually needs to pick utils2.o from libutils.a for symbols,
then GCC will not be able to resolve all the symbols in libmlcutils.a.
As solution, move libutils.a (and other internal static libraries) as
last libraries to link guestfsd to: this way, GCC knows where to find
all the symbols needed by all the objects and libraries specified in
the command line.
Prune from the list of sources where to extract messages various sources
with no messages:
- .pl and .pm files, as they do not contain messages: almost all the
.pl files are tests, and the only .pm file is the Perl Sys::Guestfs
module, which wraps the XS extension
- dummy.c sources; they are empty sources used to build OCaml-only
targets using automake
- gperf generated sources
- C/OCaml tests
As proposed in the Debian bug #946594 [1], the implicit .cmi/.cmo/.cmx
dependencies are tied to files in the source directory only, and thus
they break when either the .ml or the .mli files are in the build
directory. Since there is already VPATH set up by automake, rely on it
to locate the right source.
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=946594#36
Solution proposed by Vincent Danjean, thanks!
This reverts commit 802c5d2055 in this
file.
Add a copy of the libvirt-ocaml library, currently available at:
http://git.annexia.org/?p=ocaml-augeas.git;a=summary
This is a snapshot at commit 1cf5aef99b26a46529ca797547c0b49627fffe78,
which has all the features we need (and that builds fine).
It is expected to stay synchronized with upstream, until there is a new
upstream release, and it will be widespread enough.
This helper script will contain variables with results of configure
checks, so other scripts can source it.
Source it automatically in test-functions.sh, so every test can already
make use of it.
Invoke make-internal-documentation.pl by its path in the source
directory.
Also, depend on it to regenerate the documentation in case
make-internal-documentation.pl changes.
The current method of adding multiple --insert or --verbatim
parameters to the podwrapper command is not very easy to use because
it involves modifying the Makefile.am in every place where this is
used, plus under po-docs/$language/Makefile.am. It's better if the
POD file itself can do the inclusion.
This enhances support so that the special sequences
__INCLUDE:file.pod__
or
__VERBATIM:file.txt__
are treated as file inclusion.
The filename must be a plain file and is searched along a path
(supplied by the optional podwrapper --path parameter). The purpose
of the path is to allow translations to happen more easily. For
example we can include a particular POD fragment from common/options/
for the English version, but copy the translated file to
po-docs/$language/ for every translated version.
Remove gnulib modules that provide stuff clearly not used within
libguestfs (library, daemon, and C tools). Among directly and
indirectly modules used previous (and now no more), they are:
cycle-check
d-ino
dev-ino
dup3
dup3-tests
fcntl-safer
fcntl-safer-tests
fdopendir
fdopendir-tests
filevercmp
filevercmp-tests
ftell
ftell-tests
ftello
ftello-tests
fts
getaddrinfo
getaddrinfo-tests
getcwd
getcwd-tests
gnu-make
hostent
i-ring
i-ring-tests
iconv
iconv-tests
inet_ntop
inet_ntop-tests
isatty
isatty-tests
openat-safer
openat-safer-tests
opendirat
ptsname_r
ptsname_r-tests
read-file
read-file-tests
rewinddir
servent
ttyname_r
ttyname_r-tests
xgetcwd
Some of the removed modules are still used pulled indirectly as
dependency of other modules. There should be no behaviour change on
recent Linux distros, although older distros were not tested (adding
a module back is easy, anyway).
Remove accordingly unused automake variables, and ignored files.
Move the read_whole_file function to the common utilities of the daemon,
so other parts can use it. For this purpose, add an out parameter to
get the amount of bytes read.
Except from the parameter addition, this should be just refactoring.
xgetcwd is used only in a test, so there is no need to pull a gnulib
module just for it.
Switch to use getcwd directly with a fixed buffer: the tests would have
failed with paths longer than 992 characters, as the libvirt_uri would
have been truncated. Since there were no reports of issues, we can
assume that the current working directory will fit in 1024 characters;
adapt the size of libvirt_uri accordingly.
If you've registered a callback in Python and the handle is implicitly
closed when the Python interpreter exits, then it can be that the
following happens:
- Python interpreter is finalized.
- guestfs_close is called
- callback is invoked (eg. close event or to display a debug message)
- Python code runs from the callback wrapper
This cause a segfault on shutdown. Catch this case and stop the
callback from running (we lose the event but given the above sequence
of events there's not much we can do about it).
See:
https://bugzilla.redhat.com/show_bug.cgi?id=1773520#c7
All Py_* functions should be protected by the GIL.
Otherwise internal python data structures can get corrupted.
Move PyGILState_Ensure to the beginning of the block and
PyGILState_Release to the bottom.
Signed-off-by: Sam Eiderman <sameid@google.com>
The generator creates these files with 0444 mode, so they cannot be
overwritten by a simple ‘cp’ command. We could use ‘cp -f’ or ‘rm -f’.
Fixes commit 15394cb4dd.
With the proposed split we will only run the generator from the
libguestfs repo. When compiling virt-v2v or the guestfs-tools we will
need certain generated files to be present already in the
libguestfs-common repo, and therefore these files must be added to
git. Hopefully they won't change very often.
After the proposed split of the libguestfs repo, we will end up with
the following layout:
libguestfs.git
common -> git submodule libguestfs-common.git
generator
virt-v2v.git
common -> git submodule libguestfs-common.git
guestfs-tools.git
common -> git submodule libguestfs-common.git
The generator will only be able to write to libguestfs directories and
the common directory/submodule. This is mostly the case already with
only 6 exceptions:
customize/customize-options.pod
customize/customize-synopsis.pod
customize/customize_cmdline.ml
customize/customize_cmdline.mli
v2v/uefi.ml
v2v/uefi.mli
This commit moves these files around so they appear under common/ml*
It is somewhat unsatisfactory because it involves copying files
around, but there are some mitigating factors:
(1) Any changes now give us more freedom to develop faster and thus
clean things up in future.
(2) The v2v/uefi files ought to go away in future anyway.
This is simple code motion and should have no effect on the built
programs or tests.
These two modules are a dependency of virt-v2v. Since we intend to
split virt-v2v from the other OCaml virt-* programs, we cannot have a
dependency between virt-v2v and virt-customize. Instead we must move
the modules to a common directory (common/mlcustomize) and have both
tools depending on the modules from there.
This is simple refactoring and should not affect how the programs work
or are tested.
The existing output objects only used the s_name field so pass that
instead. (There was one minor exception: -o libvirt adjusted an error
message based on the hypervisor field but that was simply to fix.)
Existing conversions do not use anything except the source disks, so
only pass this.
This also renames src_disks -> source_disks in a few places for
consistency.
Pure refactoring with no change in function, designed to reduce
coupling between stages.
Use firstboot script to install MSI with QEMU-GA from virtio-win ISO or
oVirt/RHV guest tools ISO.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
For Linux the guest itself remembers the IP address associated with
each MAC address. Thus it doesn't matter if the interface type
changes (ie. to virtio-net), because as long as we preserve the MAC
address the guest will use the same IP address or the same DHCP
configuration.
However on Windows this association is not maintained by MAC address.
In fact the MAC address isn't saved anywhere in the guest registry.
(It seems instead this is likely done through PCI device type and
address which we don't record at the moment and is almost impossible
to preserve.) When a guest which doesn't use DHCP is migrated, the
guest sees the brand new virtio-net devices and doesn't know what to
do with them, and meanwhile the right static IPs are still associated
with the old and now-defunct interfaces in the registry.
We cannot collect the required information from within the guest.
However we can collect it outside the tool by some other means
(eg. using VMware Tools APIs) and present this information to virt-v2v
which then writes it into the Windows guest at firstboot time.
This commit adds the --mac ..:ip:.. sub-option which creates a
Powershell script to set network adapters at firstboot. An option
such as:
--mac 00:0c:29:e6:3d:9d:ip:192.168.0.89,192.168.0.1,24,192.168.0.254
approximately turns into this script:
# Wait for the netkvm (virtio-net) driver to become active.
$adapters = @()
While (-Not $adapters) {
Start-Sleep -Seconds 5
$adapters = Get-NetAdapter -Physical |
Where DriverFileName -eq "netkvm.sys"
}
$mac_address = '00-0c-29-e6-3d-9d'
$ifindex = (Get-NetAdapter -Physical |
Where MacAddress -eq $mac_address).ifIndex
if ($ifindex) {
New-NetIPAddress -InterfaceIndex $ifindex
-IPAddress '192.168.0.89'
-DefaultGateway '192.168.0.1'
-PrefixLength 24
Set-DnsClientServerAddress -InterfaceIndex $ifindex
-ServerAddresses ('192.168.0.254')
}
Thanks: Brett Thurber for diagnosing the problem and suggesting paths
towards a fix.
This experimental filter can be used to work around brief
interruptions in service, such as the network going down, firewalls
timing out connections etc., without requiring virt-v2v to be rerun.
For input methods which use nbdkit, we can cheaply add
nbdkit-rate-filter to control input-side network bandwidth. These
options control that filter. We can choose to set the bandwidth
statically and optionally change it dynamically:
--bandwidth 10M
# static bandwidth of 10 Mbps, no dynamic adjustment possible
--bandwidth 5M --bandwidth-file /tmp/bw
# initial static bandwidth of 5 Mbps, adjustable by writing to /tmp/bw
--bandwidth-file /tmp/bw
# no initial bandwidth cap, can be added later by writing to /tmp/bw
It only makes sense to control the input side since virt-v2v writes a
lot less data than it reads.
‘virt-v2v -i libvirtxml’ has a little-known feature where it can read
network disks over HTTP or HTTPS. This can be used to test VMware
conversions without needing VMware, although as far as I can tell the
current test suite does not use the feature. This commit changes this
functionality to use nbdkit-curl-plugin instead of the qemu curl
driver.
This change shouldn't affect functionality. The readahead size is
changed from a fixed 1M buffer to using the readahead filter which is
self-configuring.
See also commit 38bf2a0f97 which
originally introduced this functionality in 2017.
The readahead filter is a self-configuring filter that makes
sequential reads faster when the plugin is slow (and all of the
plugins we use here are always slow).
I observed the behaviour of the readahead filter with our qcow2
overlay when converting a guest from a vCenter source. Even when
doing random reads, qemu issues 64K reads which happen to also be the
minimum request size of the readahead filter, so there is no extra
overhead. When doing the sequential copy the readahead filter
performed better than qemu curl’s readahead because it scaled the
prefetched data appropriately on long contiguous stretches and then
shrunk it back to 64K around fragmented parts of the base image.
This uses the technique described in the nbdkit-probing(1) man page.
It should work with a wide range of versions of nbdkit, and is the one
which the nbdkit developers currently recommend.
Initially this is a like-for-like replacement, but in future commits
this will allow us to implement:
- password authentication (instead of SSH agent)
- bandwidth throttling
- readahead
Note this requires nbdkit >= 1.12.
In forthcoming commits we will be adding support for ssh, curl and
other features that require nbdkit >= 1.12.
As a prelude to that work, add generic code for querying ‘nbdkit
--dump-config’ and ‘nbdkit plugin --dump-plugin’ and checking the
minimum version number.
This changes the minimum version from 1.1.16 to 1.2, although that was
released about a year ago and is widely available, and in any case
we're going to require 1.12 in the next commit.
This refactoring takes the nbdkit-specific code from the ‘-it vddk’
mode and puts it into a separate module. The idea is to reuse this
module (in future commits) to support ssh, curl and rate limiting.
This refactoring is not quite neutral because it moves some of the
prechecking into the Nbdkit.create function. This means it will
happen a little bit later in the cycle (in input#source instead of
input#precheck - refer to the diagram in v2v/types.mli). However it's
still almost in the same place and importantly still happens before we
start copying.
cgo does not allow arbitrary CFLAGS to be used. Instead it contains a
list of flags (safelist) that are allowed to be passed to the compiler.
Sadly -U option (introduced in commit d8d8c856a1) is not among them.
See: https://github.com/golang/go/issues/23672
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
It makes little sense to require the oVirt certificate, especially when
the verification of the connection (-oo rhv-verifypeer) is disabled by
default. The only work done with the certificate in that case is
checking that it is a valid certificate file.
Hence, make -oo rhv-cafile optional, requiring it only when
-oo rhv-verifypeer is enabled.
By default the installer will set the hostname to what the DHCP returns,
exposing details of the machine where make-template.ml runs.
Instead, force "unassigned-hostname.unassigned-domain" as hostname, so
plays nicely with the hostname setting code in virt-customize.
According to the Erlang website:
The old legacy erl_interface library (functions with prefix erl_) is
deprecated as of OTP 22, and will be removed in OTP 23. This does
not apply to the ei library. Reasonably new gcc compilers will issue
deprecation warnings. In order to disable these warnings, define the
macro EI_NO_DEPR_WARN.
That's a shame and probably means we will have to drop the Erlang
bindings soon unless someone ports them to this new API (stable APIs
FTW people!). In the meantime add the flag to prevent warn-errors
about deprecation.
This way it is possible to override the UUIDs of the uploaded disks,
instead of letting RHV generate them.
This can be useful to force certain UUIDs, and to specify the disks in
--no-copy mode (which now can be used).
Check for the INITIALIZING state of the image transfer right away,
without waiting 5 seconds even before the first time: this way, if the
transfer is already in the right state then there is no need to wait.
Otherwise it complains about missing files that it has no rules for, for example
`builder/index-parser.c`.
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
Instead of waiting for the completion of the nbdkit transfers to get the
UUIDs of the disks, use the new #disk_copied hook to do that after each
disk is copied.
This has almost no behaviour on rhv-upload, except for the --no-copy
mode:
- previously it used to hit the 5 minute timeout while waiting for the
finalization of the first disk
- now it asserts on the different number of collected UUIDs vs the
actual targets; at the moment there is nothing else that can be done,
as this assumption is needed e.g. when creating the OVF file
Improve the way the precheck script checks for the specified resources:
- look directly for a data center with the specified storage domain
- get the storage domain object from the storage domains attached to the
data center found
- similarly, look for the specified cluster among the ones attached to
the data center found
When everything is found, return the UUID of the storage domain, and of
the cluster back to virt-v2v, which will store them.
Similarly, rework the createvm script to directly get the requested
cluster, instead of looking for it once again. Also, since the UUID of
the storage domain is available in virt-v2v already, use it directly
instead of using a placeholder.
This should fix a number of issues:
- unexisting/unattached storage domains are rejected outright
- the cluster is rejected if not part of the same data center of the
selected storage domain
- renaming the specified storage domain during the data copying will not
cause the conversion to fail (which will still use the specified
storage domain, no matter the new name)
Based on the hints by Daniel Erez in RHBZ#1612653.
This way it is possible to communicate data from the precheck script
back to virt-v2v.
For now there are no results, so the resulting JSON is discarded.
Split the VM existance check out of the precheck script to a new vmcheck
script, and invoke that in #prepare_targets. Invoke the precheck script
in #precheck, as now it can be run with only values of command line
options.
This does not change which checks are performed; however, an invalid
cluster name will make virt-v2v fail way earlier (even before connecting
to the source).
This removes only the tool itself, and all the bits strictly needed to
not break the build.
This is now available as separate tool in its own repository:
https://github.com/libguestfs/virt-p2v
Added in virt-p2v commit:
commit e83b6f50af34ce650063ecc520bfabab400e8e73
Author: Matthew Booth <mbooth@redhat.com>
Date: Fri Mar 26 09:40:20 2010 +0000
Add export to RHEV
Allow guests to be written to a RHEV NFS export storage domain.
Add 'rhev' output method and -osd command-line option.
Example command line:
virt-v2v -f virt-v2v.conf -ic 'esx://yellow.rhev.marston/' \
-o rhev -osd blue:/export/export RHEL3-32
This will connect to an ESX server and write the guest 'RHEL3-32' to
blue:/export/export, which is a pre-initialised RHEV export storage domain.
make[2]: *** No rule to make target '../p2v/about-authors.c', needed by 'internal-documentation.pod'. Stop.
Since we're going to remove the whole p2v subdirectory shortly anyway,
it was simplest to ignore the whole directory.
Avoids this error when building from git:
make[2]: *** No rule to make target '../common/mllibvirt/libvirt_c.c', needed by 'internal-documentation.pod'. Stop.
This file doesn't have any generated docs or gettext annotations in it
so adding it to these files is useless anyway.
Disable this warning/error in the Python 3.8 header files:
In file included from /usr/include/python3.8/abstract.h:837,
from /usr/include/python3.8/Python.h:147,
from actions.h:31,
from actions-6.c:34:
/usr/include/python3.8/cpython/abstract.h: In function '_PyVectorcall_Function':
/usr/include/python3.8/cpython/abstract.h:91:11: error: cast increases required alignment of target type [-Werror=cast-align]
91 | ptr = (vectorcallfunc*)(((char *)callable) + offset);
| ^
Previously to work around some problems in Python 2 header files we
had to include <Python.h> before any other config file.
For Python 3 which is all we really care about now this is no longer
needed. We can move the include from three files into the local
"actions.h" file, bringing all the Python definitions and workarounds
into a single place.
GtkAttachOptions is part of GtkTable, which is used only with GTK < 3.4;
however, these enum values are used also in the GtkGrid version of the
code, so they are needed also when disabling deprecated stuff.
As easy solution to make the current code working without deprecated
stuff of GTK, copy the GtkAttachOptions enum when using GtkGrid (i.e.
with GTK >= 3.4).
- feed the content directly to stdin, avoid the need of read (and write)
a temporary file
- read all the output at once, without a tail-recursive function
- apply trimming and first line discarding after closing the process
Isolate the logic for the memoized disk cache in a small module, so it
can be reused for other tools.
Other than refactoring, there should be no behaviour changes.
- add a no-op Connection.close(), as it called explicitly in the
close() callback of the nbdkit plugin (rhv-upload-plugin.py)
- fix the types of the 'id' variables, which are strings
- fix names of arguments & optional arguments in C<..> markers
- use https for URLs where possible
- fix links to other guestfs APIs
- use more C<..> markers for special tests, shell commands, values of
arguments, and names of fields
- link to command man pages where an explicit command is mentioned
- fix few incorrect documentation bits
This patch includes:
- Event callback handlers
- Tests related to events(410-430)
src/bin/event.rs and src/bin/event_leak.rs
are the PoCs that Boxes related to callbacks are
not leaked.
Without clarifying handle's lifetime, it is unable
to see how long the callbacks which the handle
owns will live. Then, Rust compiler will infer
that the callbacks have 'static lifetime. It is
not convenient for users.
BUGS
docs/C_SOURCE_FILES
po/POTFILES
po/POTFILES-ml
These files are normally updated by ‘make dist’ but as we've not had a
release in a while they had not been updated for some time.
Since there is no more need to build nbdkit from sources, then there is
no need to set $PATH with a custom build of nbdkit.
Followup of commit 0704d8eb0b.
Point to files in the source directory using the right variables, so
they are found also when the build directory is different than the
source directory.
While it is generated at configure time for dependency reasons, consider
it a generated source nevertheless: after the first run, there will be
dependency rules available, so automake will trigger the proper rule to
generate it again using generate-p2v-config.pl.
Soon only the virt-p2v authors will be available, so remove all the
other roles. This leaves only in the virt-p2v about dialog, which is a
mild regression compared to the current situation, although it is just
for user information.
Move (or copy the general ones) all the p2v entries in the top-level
.gitignore file to a new file specific for p2v. This will make it
easier to keep them when splitting p2v.
Copy from generator/p2v_config.ml also the bits for POD documentation,
adding them to generate-p2v-config.pl; this way,
virt-p2v-kernel-config.pod can be generated at build time directly,
instead of statically shipped as generator output.
Instead of generating the p2v kernel config using the OCaml generator,
create a Perl script to do this job, mostly at build time. This is done
to rely less on the generator for p2v, and because the generation of
these sources is quick enough that it can be done at build time (instead
of shipping the generated sources in dist tarballs).
The generate-p2v-config.pl mimics what generator/p2v_config.ml --
namings, and general structure are kept close to that for comparison.
The two C sources are created at build time by the script; however, the
p2v-config.h header is generated at configure time: this is done because
p2v-config.h is included by p2v.h (another header), which in turn is
included by all the p2v C sources -- automake is not able to properly
resolve the dependency, and thus it would not be generated properly.
Python 3.8 no longer links C extensions to -lpython, instead relying
on the fact that the python binary itself already contains those
symbols. This means $PYTHON_LIBS is empty and so the Python bindings
are not built.
Use a different test to see if the python module is available.
This way no non-namespaced OCaml C symbols are used, reducing the risk
of clashes with other code.
The only exception is ocaml-augeas, which does not build with
CAML_NAME_SPACE; it will be fixed upstream, and it affects only
ocaml-augeas itself.
While svirt_t can be used for sockets it does not always guarantee that it will
be accessible from a virtual machine. The VM might be running under svirt_tcg_t
context which will need a svirt_tcg_t label on the socket in order to access it.
There is, however, another label, svirt_socket_t, which is accessible from
virt_domain:
# sesearch -A -s svirt_t -c unix_stream_socket -p connectto
...
allow virt_domain svirt_socket_t:unix_stream_socket { ... connectto ... };
...
And virt_domain is a type attribute of both svirt_t and svirt_tcg_t:
# seinfo -x -a virt_domain
Type Attributes: 1
attribute virt_domain;
svirt_t
svirt_tcg_t
Resolves: https://bugzilla.redhat.com/1698437
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
It seems that on older Python 3 versions importlib.util is not imported
automatically when importlib is imported; as solution, import
importlib.util directly (as it is what is used, anyway).
Fixes commit f79129b8dc.
Previously, is_custom_hv() used to compare the QEMU executable found
during configure to the hypervisor set to check whether it is a custom
one; however, the QEMU found at configure time can be different than
what libvirt was configured with.
This fixes the libvirt backend when libguestfs is configured with a
different QEMU, that now will be specified as emulator overriding the
libvirt one.
We ship our own copy of it, so we do not need the external version.
(Also, the latest upstream version of ocaml-libvirt was already not
usable to build the test harness of v2v.)
Now that we have a proper libvirt connection object, use it directly to
refresh the storage pool, and define the final guest. This avoids
spawning a new virsh process twice, with no possibility to even share a
possible authentication required.
Store the Libvirt.Connect.t object as instance variable, so it can be
used also outside of prepare_targets. Use a private method to access
it, so there is no need to directly use the Lazy object.
Currently virt-v2v has few custom C-based functions for libvirt
operations, which are limited in what they do, and there is a lot of
duplicated code.
Instead, switch to ocaml-libvirt for all the libvirt interaction
currently done by the Libvirt_utils module. This has few advantages:
- each input & output module now opens a libvirt connection only once,
only when needed
- no need to pass URIs and passwords around, if not needed
- a wider range of libvirt APIs can now be used, with no need to create
bindings manually
The hierarchy of input_libvirt* classes is changed to take a Lazy object
with the libvirt connection, accessing it through a "proctected" method:
this way, the connection is opened only at the first access.
Also, the Libvirt_utils module now is just helpers around the Libvirt
module, to centralize error handling, and few common operations.
Add a copy of the libvirt-ocaml library, currently available at:
https://libvirt.org/git/?p=libvirt-ocaml.git;a=summary
This is a snapshot at commit d3ed8dcf1b0a6a8a855ceecbe0bb97f21e6665e3,
which has all the features we need (and that builds fine).
It is expected to stay synchronized with upstream, until there is a new
upstream release, and it will be widespread enough.
While there are input modes that do not use libvirt, making libvirt
mandatory for virt-v2v slightly simplifies the code now, and allow for
further improvements/integration with libvirt later on.
Adapt make-fedora-img.pl to not use deprecated APIs anymore:
- set_e2uuid -> set_uuid
- txz_in -> tar_in + compress:xz
This causes no changes in the generated test images.
Since we already assume Python >= 2.7, modernize this example to make it
work also on Python 3:
- use print() as function
- sort the mount points using a key for sorted(), instead of a
comparison function
- remove extra newline escape
- reident two lines according to the PEP 8 style
Add a simple way to do not even provide prototypes of deprecated
functions in the C library: this way, users (like our tools) can build
against the library making sure to not use any deprecated function, not
even when compiler deprecation warnings are disabled.
Add it to the majority of our tools/internal libraries, and make sure
that it is not defined when building the API bridges of our bindings.
Right now, deprecated functions of the library do not trigger any
compiler deprecation warning by default; they do that only if
GUESTFS_WARN_DEPRECATED=1 is defined. However, this is not something
that seems to be done often -- at least none of the projects using the
libguestfs C API does that.
Hence, do a small behaviour change to change this on the other way
round: now deprecated functions trigger compiler deprecation warnings by
default, using GUESTFS_NO_WARN_DEPRECATED to disable this (and revert
to the previous behaviour). Even though deprecated functions will not
be removed, we really want users to migrate away from them, as they were
deprecated for good reasons.
Define GUESTFS_NO_WARN_DEPRECATED where needed:
- in all the bindings, as they bind all the functions including the
deprecated ones
- in the guestfish actions, as it exposes almost all the APIs
- in the C API test, as it runs the automated tests of all the APIs that
have them
- for two tests that explicitly test deprecated functions
part-expand-gpt takes extreme cautions and doesn't proceed to writing
to the disk if the preliminary dry run of sgdisk has generated any
warnings on stdout.
This blocks the use of part-expand-gpt on disk shrink (with disk
resize being the main usecase for part-expand-gpt), because sgdisk dry
run produces a warning in that case.
So remove the excessive safety check, and leave it up to the caller.
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Roman Kagan <rkagan@virtuozzo.com>
This is not quite a neutral refactoring, because it means we now run
the getenforce command every time virt-v2v starts up. However it's a
trivial command that reads a single /sys file and it can't fail even
if the command is missing or on platforms that know nothing about
SELinux.
virt-v2v obviously cannot convert this kind of devices, since they are
specific to the host of the hypervisor. Thus, emit a warning about the
presence of direct network interfaces, so at least this can be noticed
when converting a guest.
Starting with 5.2.0, libvirt has a way to select automatically the
firmware for a guest using an attribute of the <os> tag. Hence, use
this information (when available, of course) to flag the firmware used
by the guest.
Return the right osinfo short IDs for some rolling Linux distributions,
such as Arch Linux, Gentoo, and Void Linux. Their IDs were recently
added to osinfo-db.
Consider Arch Linux as rolling distribution, so it is recognized using
/etc/os-release.
The end result does not change, although this makes Arch Linux inspected
using os-release only, instead of getting inspection details mixed from
both os-release and lsb-release.
Add "gentoo" as recognized ID in /etc/os-release, and consider it as
rolling distribution (so without VERSION_ID in os-release).
This avoids using a not-useful version read from /etc/gentoo-release,
e.g. "Gentoo Base System release 2.6".
Use a static list of rolling distros, so it is easier to handle them
differently; use this list for handling the lack of VERSION_ID in
os-release files.
This is just refactoring, no behaviour changes.
Add a new output mode to virt-v2v: similar to -o local, the written
metadata is a JSON file with the majority of the data that virt-v2v
knowns about (or collects) during the conversion.
This is meant to be used only when no existing output mode is usable,
and a guest needs to be converted to run on KVM anyway. The user of
this mode is supposed to use all the data in the JSON, as they contain
important details on how even run the guest (e.g. w.r.t. firmware,
drivers of disks/NICs, etc).
When the machine readable mode is enabled, print all the messages
(progress, info, warning, and errors) also as JSON in the machine
readable stream: this way, users can easily parse the status of the
OCaml tool, and report that back.
The formatting of the current date time into the RFC 3999 format is done
in C, because of the lack of OCaml APIs for this.
Allow to specify a file descriptor for the machine readable output.
Use the same assumption as done in v2v, i.e. that Unix.file_descr is
simply the int file descriptor.
Enhance the helper printf function for machine readable output to always
flush after each string: this way, readers of the machine readable
stream can get the output as soon as it is outputted.
Store log from MSI installer. Log file will be located in firstboot
scripts-done directory with name rhev-apt.log. The path has to be
double-quoted to handle spaces in path name properly.
Hopefully this can help resolve RHBZ#1584678 someday.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
The debug message was slightly changed too to better match the similar
message for ISO case. It refers to the root directory instead of the
specific subdirectory inside guest tools.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
The RHV Tools ISO is provided as Red Hat only product, and thus not
available for all the virt-v2v users. Hence, change the way we report
the status of the installation of the qemu guest agent from the RHV
Tools ISO:
- do not warn if virt-v2v does not know how to install the package for
the current guest
- do not warn if the ISO does not contain packages for the current guest
- on successful installations, show an info message
Related: RHBZ#1691659
Right now the code copies, and then tries to install, all the files
found for the directory of a distro. This does not take into account
differences in the architectures of the packages available, so a x86_64
package must be installed only on x86_64 guests.
As solution, filter the packages using the typical suffix used by
packages (so architecture + file extension), to pick only packages
compatible with the current guest.
Schema parsing was failing with errors such as:
libguestfs: QMP parse error: '[' or '{' expected near end of file (ignored)
This happened because the QMP command was actually completely failing
and never printing a result at all. This happens because the qemu
audio driver can't be set up without a console. We can suppress this
by setting the environment variable QEMU_AUDIO_DRV=none, which is the
same thing that libvirt does, and also the same thing that we are
already doing when launching the real appliance subprocess.
See also: https://bugzilla.redhat.com/show_bug.cgi?id=1692047
Make sure the create guest is always up-to-date: this way it is possible
to test the conversion even taking into account the latest upgrades
available.
ubuntu-server depends on open-vm-tools on Ubuntu, so if v2v tries to
uninstall open-vm-tools then dpkg will (rightfully) fail with a
dependency issue.
Since open-vm-tools is harmless after the conversion anyway (it will
not even run), then do not attempt to uninstall it if ubuntu-server is
installed too.
Followup of commit 2bebacf8bf.
Thanks to: Ming Xie.
When writing the libosinfo metadata in the libvirt XML, use the newly
added (in osinfo-db) ID for Windows Server 2019; sadly, this version of
Windows has the same version as Windows Server 2016, so distinguish it
by looking at its product name.
Use a better icon for RHEL guests, still provided by redhat-logos (or
equivalent in downstream distributions), and which fits a better
definition of logo for the distribution.
Thanks to Ray Strode for the hints.
Try to look for a well known kernel module (so far only virtio, or kvm)
to use for detecting the architecture of a kernel. This way, we can
avoid picking 3rd party modules that cause troubles.
Kernel modules can be also symlinks to files available outside the
"canonical" module directory; the "file" API, used by the
"file-architecture" API, return the actual type of a file (so symlinks,
block devices, etc), and thus "file-architecture" fails on symlinks.
To prevent this situation, canonicalize the path of the module picked
for architecture detection: this way, "file-architecture" will act on a
real file.
test-relabel.pl performs a full SELinux relabelling of the phony guest,
and the selinux_relabel API does not work if the "selinuxrelabel"
feature is not available.
The icon is installed by x11-themes/gentoo-artwork, which is not
installed by default (and most probably only on "desktop"
installations), although it's the best hit so far...
Newer versions do not have the 24px distributor.png icon; OTOH they have
the 48px version, so look for that one before the 24px one.
Also, bump the size limit to 10K, as newer versions of the icon are
bigger than 3K.
Isolate the code for looking for a PNG file among a specified list from
the icon_ubuntu implementation, and switch that function to it.
This is just code refactoring, with no behaviour changes.
Use NEVR when querying RPM for the list of files of a package, instead
of ENVR. Also, use the epoch only when non-zero, and version of RPM
supports it.
The approach is basically copied from what supermin does in its RPM
package handler.
So far RBufferOut return values, and FBuffer struct fields are 'str' on
all the versions of Python. Python 3 distinguishes between 'str'
(unicode strings), and 'bytes', with 'str' no more able to hold
arbitrary data.
For this reason, switch the return value of RBufferOut functions, and
FBuffer struct fields to bytes on Python 3: while this is a potentially
incompatibile change, this is the only way to handle safely sequences
of arbitrary bytes.
Make sure to reference the arguments, to make sure they are kept alive
during the function call; this is visible when setting an event handler
for the CLOSE event, and testing it with Python 3.
This does not seem to create a memory leak e.g. with Python 2.
Also, switch away from the quasi-internal PyEval_CallObject to the
public PyObject_CallObject, which takes care of doing safety checks.
Commit 4252a490bb dropped the PO files
for ca (Catalan) and pt_BR (Brazilian Portuguese). However it did not
drop these languages from the linguas line in the Makefile resulting
in a build failure.
./configure output now contains a distinct section for v2v and p2v:
--- Checking the virt-v2v and virt-p2v dependencies ---
checking for the nbdkit python plugin name... python3
checking for --with-gtk option... 2
checking for GTK... yes
checking for DBUS... yes
checking if we can build virt-p2v... yes, with Gtk 2
Thanks: Pino Toscano.
No functional change, but it does allow downstream distributions to
adjust the nbdkit Python plugin used by virt-v2v -o rhv-upload mode:
./configure --with-virt-v2v-nbdkit-python-plugin=...
When parsing "xdev"-kind devices, do not assume that the partition
number can be converted to integer: re_xdev accepts an empty part of the
partition number, so just handle as it is, as string.
This fixes a regression due to the conversion of the inspection code to
OCaml, as the old C version did not have this issue.
We've been carrying this exact patch in RHEL 7 for several years. It
reverts the change made in 2014 where we switched to using the virbr0
bridge for libguestfs networking instead of SLIRP. We thought SLIRP
was going to become unsupported in qemu, but recently there have been
more encouraging signs since it looks like SLIRP will be spun off as a
separate project, running as a modular process and properly secured
and supported.
This reverts commit 224de20b9a.
Do not assume that the Python plugin of nbdkit has the same name of the
Python interpreter.
Use the default upstream name of nbdkit to identify it; downstream
distributions must adjust this variable, in case they rename the Python
plugin of nbdkit.
Replace python 2 only "buffer" with "memoryview".
Falling back to emulated zero would fail with:
NameError: name 'buffer' is not defined
I did not test the changed code but it was not tested before so it is
unlikely to be worse.
Detected by pylint.
headers was initialized with a set literal {"key", "value"} instead of
dict literal {"key": "value"}.
Calling pread() will fail with:
AttributeError: 'set' object has no attribute 'items'
I did not test the changed code, but it was not tested before, so it is
unlikely to be worse.
Detected by pylint.
builder.libguestfs.org has almost run out of space. I would like to
move templates for older guests to the archive site (which is slower
but has unlimited space). The easiest way to do this is to add a
second repo.
This only affects obsolete/unsupported Fedora releases.
For a while libguestfs.org/builder has redirected to
builder.libguestfs.org. This change just makes virt-builder go direct
to the subsite instead of via the redirect.
It should not be error to use virtio-win ISO that does not have Linux
packages of QEMU Guest Agent. Only oVirt/RHV guest tools ISO has such
packages now. Regular virtio-win ISO does not have them and maybe never
will.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
Add few more characters for the devices of resume= entries in the
command line of grub: this way it is possible to match also /dev/mapper
devices.
This should require no further processing, since the names of the
/dev/mapper devices do not change after the conversion.
Commit c11a92751e chose some strangely
sized partitions for testing Linux MD devices. The disks were 100M
however the four partitions only covered the first 10M of disk space
(with the rest being unallocated and unused in the test).
This worked until Linux 4.20 where the ~2M-sized partitions become too
small for a Linux MD device.
This commit changes the test to use four 20M-sized partitions which is
sufficient space to create the MD device with recent kernels. I also
modified the partition allocation code to use explicit calculations
and variables, making it considerably clearer.
Tweak the limits of the files of package managers downloaded to list the
installed applications:
- bump RPM 'Packages' up to 500M: Fedora installations upgraded to each
new Fedora release have an ever-growing file, reported to be even
slightly bigger than 400M
- reduce the other files down to 50M: they are usually small, not bigger
than 1M-5M, so 50M is a good enough high threshold
Use different defines for the limits of the package manager files
downloaded during the inspection of applications. This way it will be
possible to tweak each of them without affecting the others.
This is a minor refactoring, with no behaviour change.
The previous support documentation said that for UEFI guests, when
using RHEL ≥ 7.3 host, you must use qemu-kvm-rhev. For RHEL 8 this is
inaccurate since qemu-kvm-rhev no longer exists.
To fix this I've dropped the whole sentence. It only applies to the
downstream product (RHEL) and so RHEL can add the right documentation
if it needs to.
Thanks: Ming Xie, Pino Toscano.
This option was removed from qemu for no apparent reason except to
break existing consumers. It does the same as -no-user-config, added
in May 2012, so use that instead.
There's a standardized libosinfo namespace for libvirt domain metadata. For now
it supports the id of the OS only. However that is still a very helpful feature
that is already supported in gnome-boxes and virt-manager (at least).
The discussion happened here:
https://www.redhat.com/archives/libosinfo/2018-September/msg00003.html
So let's add the support to local and libvirt outputs.
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
The proper file that should be included is `sys/xattr.h` as that comes from
`glibc` and not `attr/xattr.h` which ships with the `attr` utility.
New enough `attr` utility (at least 2.4.48 in my case) even includes a #warning
in `attr/xattr.h` for projects that still have this mistake in the code.
If the Content-Length header was missing from the headers returned by
the server then the test would fail with:
KeyError: 'content-length'
This happens on RHEL 7.6 in particular. We can fix this by using an
alternate method to fetch the header, which will return None instead
of throwing an exception if the header doesn't exist. See:
https://stackoverflow.com/a/19255675
If ‘virt-v2v -oo verify-server-certificate=false’ is used then
‘openstack --insecure’ flag is added whenever we invoke the openstack
command. This turns off SSL certificate validation. The default is
to verify the server certificate (which is the default of the
openstack command).
Create the fedora-btrfs.img as an empty file.
The only place this is used explicitly is tests/mountable/
test-mountable-inspect.sh, but that test already skips if !btrfs.
Also this is used via guests-all-good.xml, but the script that creates
this XML skips the file if it has zero size.
Previously:
$ virt-v2v -v -x -i disk fedora-28.img -o openstack -oo server-id=foo
virt-v2v: libguestfs 1.39.11fedora=29,release=1.fc29,libvirt (x86_64)
libvirt version: 4.5.0
virt-v2v: error: openstack: precheck failed, there may be a problem with
authentication, see earlier error messages
rm -rf '/var/tmp/null.dTxRFN'
Notice there is no "earlier error message".
After this commit the error changes to:
openstack: executable not found
virt-v2v: error: openstack: precheck failed, there may be a problem with
authentication, see earlier error messages
In case the appliance is disabled at configure, then all the phony
guests will not be built during the 'make check' time. Since the tests
already handle an empty windows.vmdk fine (in case ntfs-3g is not
available), then just create it empty in this case.
Reported by: Martin Kletzander.
In Gentoo the java alternatives are managed using jva-config (or eselect java),
but because the selection is treated as a configuration the symlink to the
current jvm selected is stored in /etc, particularly in the symlink
/etc/java-config-2/current-system-vm which works such as /usr/lib/jvm/default.
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
The primary virt-v2v(1) page now contains a more concise overview of
virt-v2v, along with things like detailed options documentation which
we would expect in a tool manual.
The documentation for VMware, Xen, local output, oVirt and OpenStack
has been moved to new pages where it is introduced in a more narrative
/ tutorial style. The idea is that people who are interested in
converting from or to these sources will be able to start at
virt-v2v(1) for an overview before moving to one of these new pages
for a friendlier introduction to the topic.
The support matrix has also been moved to a new page called
virt-v2v-support(1). The idea is that downstream packagers will be
more easily able to modify or completely replace this page according
to their support requirements.
When rewriting the VMware documentation, I dropped the section
"INPUT FROM VMWARE ESXi HYPERVISOR" which discussed using
virt-v2v-copy-to-local. This should no longer be necessary since you
can use SSH or VDDK. (See also previous commit).
virt-v2v is capable of accessing ESXi hypervisors now. There is only
one remaining use of this tool - when converting from Xen hypervisors
over ssh which are using host block devices for storage.
Commit ba82fb023d changed the default
libguestfs appliance memory from 500 -> 768MB.
Virt-v2v used a formula to calculate how much memory to use which is
based on this which was designed to result in around 2GB of memory:
500 * 20 / 5 = 2000
However after the commit the same formula now produces:
768 * 20 / 5 = 3072
This commit reduces the numerator in the formula to yield (about) 2GB
again:
768 * 14 / 5 = 2150
Some paths in the function are evaluated in libguestfs environment.
Previous commit copied the invalid construction. This is now fixed.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
Changed the function to be more generic and renamed.
The only change in behavior is in produced debug messages.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
Make use of APIs available in early GTK+ 3 versions, adding
compatibility functions/macros for older GTK+ versions.
There is no behaviour change, and it helps in porting to even newer GTK+
versions.
I think libvirt or virt-install has changed recently so that it
creates disk images which are not public readable. This has lead to
me uploading two sets of non-readable templates for virt-builder. Set
the mode explicitly to 0644.
Thanks: Laine Stump, Jacob Shivers.
Instead of assuming the latest Fedora version if /etc/redhat-release is
available, or the latest Debian with /etc/debian_version, use only
/etc/os-release. The possible name of the virt-builder template is
created by concatenating $ID and $VERSION_ID, which is generally a
better guess than what previously done, and better matches the host OS.
This affects only the case when os-version is not specified as command
line argument.
Add a "Shutdown" action to the Shutdown button in the conversion dialog,
before the Reboot action: this way it is possible to shutdown the
physical server directly from the GUI after the conversion.
To ease adding a proper Shutdown action in the conversion dialog, turn
the current Reboot button into a button that pops a menu up, with Reboot
as the only available action.
The menu is implemented using a popover when using "recent" versions of
GLib, and GTK+ 3, because of all the APIs needed. In older versions, a
simple toggle button with menu is used, which though needs a manual menu
firing when pressing the button itself (and not its side arrow).
This seems to be necessary for Windows Server Core 2012 + R2, although
it has not been necessary with virt-v2v previously. By adding this
simple comment-only change we will know this for future reference.
After the previous commit, wherever we had:
start_element ("foo") {
string ("bar");
} end_element ();
this can now be replaced by:
single_element ("foo", "bar");
Change virt-inspector so it uses the common set of macros. I also
added:
* single_element() and single_element_format():
creates <foo>bar</foo> which is used extensively by virt-inspector
* base64():
used by virt-inspector for the icon
In some places when generating XML output in C code we use some clever
macros:
start_element ("memory") {
attribute ("unit", "MiB");
string_format ("%d", g->memsize);
} end_element ();
This commit which is mostly refactoring moves the repeated definitions
of these macros into a common header file.
I also took this opportunity to change / clean up the macros:
- The macros are now documented properly.
- comment() and empty_element() macros are now available everywhere.
- Error handling has been made generic.
- Added do..while(0) around some of the macros to make them safe to
use in all contexts.
With recent Linux kernels, adding and partitioning 255 disks causes
the appliance to run out of memory. This causes a test failure in
tests/disks/test-255-disks.sh. This change gives the appliance enough
memory to complete the test.
Add oVirt specific elemnt to OVF. It represents the combination of
machine type (i440fx/q35) and firmware (BIOS/UEFI).
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
This option was added in error. It never had any effect and now the
nbdkit VDDK plugin ignores it. Virt-v2v users shouldn't have been
using it.
This removes the option completely (so if anyone was using it they
will now get an error).
When installing the VMware tools from tarball, the installation script
rebuilds the initramdisk of all the available kernels to inject the
missing kernel drivers; in the end, the information on which kernels
were changed is recorded in the internal "database" of the installation
answers. When uninstalling the VMware tools, the uninstallation script
reads the saved answers, and in the case of ramdisks will do the reverse
job done during the installation (which is done again when new kernels
are installed).
OTOH, virt-v2v already rebuilds the initramdisk of the default kernel,
not touching the other available kernels (no matter whether they have
the right modules or not). Hence, trick the answers "database" of the
VMware tools to discard all the information about ramdisks to restore on
uninstallation: this way they are not rebuilt, and the whole
uninstallation time will be reasonable enough (a couple of minutes or
so).
Linux distributions usually ship the open source VMware tools as
open-vm-tools (and open-vm-tools-desktop for the integration with X).
While they will not run already anymore after the conversion, uninstall
them during the conversion, to save some space in the converted guest.
It seems that all the newer versions of openSUSE have an 'opensuse-'
prefix in their distro ID; hence, check for that prefix at once, keeping
compatibility for the old "opensuse" ID.
Followup of commit a4e53bcbc5.
Support the ID found in openSUSE Leap 15.0 in addition to the other
SUSE-related IDs. While there is already --with-distro=ID to force the
ID without using os-release, this makes it possible to build OOTB.
Previously this output method was almost completely untested.
This commit adds a fake ovirtsdk4 module so we can test the
-o rhv-upload method fairly completely without needing an actual
oVirt instance around.
Thanks: Pino Toscano.
For real imageio servers the destination will always be https. This
change has no effect there.
However when testing we want to use an http server for simplicity. As
there is no certificate or cafile in this case the call to create the
context will fail.
This also simplifies creation of the context object and recognizes the
"insecure" flag for connecting to imageio.
Thanks: Nir Soffer.
Since Linux commit f663b5b38fff trimming vfat is now supported by
Linux. This broke the test which assumed it was not supported. Use
another filesystem (minix) which does not support trimming instead.
Thanks: Daniel P. Berrangé and Pino Toscano.
When using the direct backend, you should see the result of testing
the qemu binary for the availability of KVM:
libguestfs: qemu KVM: enabled
Thanks: Andrea Bolognani.
After this commit, all annocheck errors are fixed except for:
Hardened: virt-get-kernel: MAYB: Gaps were detected in the annobin coverage. Run with -v to list.
After discussion with the annocheck maintainers this gap in coverage
(which corresponds to the OCaml runtime) seems to be caused either by
the runtime not being linked with the right flags, or might be a bug
in annocheck itself. In any case it's not something that can be
resolved within the scope of libguestfs.
OCaml has a small runtime which is statically linked into the virt
tools (providing things like GC and primitives). Since OCaml 4.03 it
has been possible to select variants of this runtime, one of which is
compiled with -fPIC, using ‘ocamlopt -runtime-variant _pic’.
This has performance implications on i686, but is relatively free on
other architectures. Since it (in theory) adds to the security of the
final binary this commit enables it whenever it is available.
The majority of the tools have already options (--echo-keys &
--keys-from-stdin) to deal with LUKS credentials, although there is no
way to automatically provide credentials. --keys-from-stdin is
suboptimal, because it is a usable solution only when there is just one
device to open, and no other input passed via stdin to the tool (like
the commands for guestfish).
To overcome this limitation, introduce a new --key option in tools:
* --key /dev/device:file:/filename/with/key
* --key /dev/device:string:the-actual-key
this way it is possible to pass all the credentials needed for the
specific devices to open, with no risk of conflict with stdin, and also
in a secure way (when using the "file" way).
On the technical side: this adds a new "key_store" API for the C tools,
making sure it is used only when needed. Partially mirror it also for
the OCaml tools, although there will be a conversion to the C API
because the decryption helpers used are in the common C parts.
Instead of returning directly a Getopt.t handle, now
Tools_utils.create_standard_options returns a struct, which at the
moment contains only the Getopt.t handle. This way, it will be easy to
add more data needed for handling standard command line options.
This is mostly refactoring, with no functional changes.
As with the previous commit this requires that Windows guests have
been created first using the procedure from:
https://rwmj.wordpress.com/2018/09/13/creating-windows-templates-for-virt-builder/
For me:
PASS: test-v2v-conversion-of-windows-6.3-server.sh
PASS: test-v2v-conversion-of-windows-6.2-server.sh
PASS: test-v2v-conversion-of-windows-10.0-server.sh
If you don't have these templates in virt-builder then the tests will
skip.
This requires that Windows guests have been created using the
procedure outlined here:
https://rwmj.wordpress.com/2018/09/13/creating-windows-templates-for-virt-builder/
For me:
PASS: test-firstboot-windows-6.2-server.sh
PASS: test-firstboot-windows-6.3-server.sh
PASS: test-firstboot-windows-10.0-server.sh
An incidental change is that we dump the firstboot log from the guest
(even on success). If the firstboot fails this is very useful for
determining the real cause.
In nbdkit >= 1.7.2, nbdkit vddk --dump-plugin will print the shared
library name in normal output, which breaks this test.
The actual error when LD_LIBRARY_PATH is not set includes this line:
nbdkit: error: libvixDiskLib.so.6: cannot open shared object file: No such file or directory
so search instead for the error message "cannot open shared object file".
Provides support for building:
- Windows 7
- Windows Server 2008 R2
- Windows Server 2012
- Windows Server 2012 R2
- Windows Server 2016
Note that these images cannot be released to the public because of
obvious licensing issues. But this documents how we build them for
internal consumption so that others can also build them.
Thanks: Christophe Fergeau, Tomáš Golembiovský.
virt-sparsify in copying mode takes a huge amount of temporary space
and takes a very long time. In any case it's not really necessary for
modern guests since in-place sparsification is fully supported now.
Inspection code checks /etc/mdadm.conf to map MD device paths listed in
mdadm.conf to MD device paths in the guestfs appliance. However on some
operating systems (e.g. Ubuntu) mdadm.conf has alternative location:
/etc/mdadm/mdadm.conf.
This patch consider an alternative location of mdadm.conf as well.
This function will soon be used to generate Windows unattended.xml
files (as well as Debian preseed) so just rename it back to
‘make_kickstart’ rather than attempting to explain in the name every
way it could be used.
Straightforward refactoring. Apart from small modifications to clean
up the code and reorder the properties more logically there should be
no functional change.
See:
https://bugzilla.redhat.com/show_bug.cgi?id=1584678#c15
Fixes commit bcdbe6405c. However this
bug was copied directly from old virt-v2v which did the same thing
(from lib/Sys/VirtConvert/Converter/Windows.pm):
echo installing rhev-apt >>log.txt
"rhev-apt.exe" /S /v /qn >>log.txt
Thanks: Lev Veyde
With python 3, we have a nicer way to handle socket.error with errno set
to EPIPE (or ESHUTDOWN).
This is also more correct since in some cases (that I could not
reproduce yet with v2v), using e[0] with BrokenPipeError will fail with:
>>> OSError(errno.EPIPE, "Broken pipe")[0]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'BrokenPipeError' object is not subscriptable
For python 2 e[0] seems to work, but is leftover from historic python
version that used to raise a tuple instead of socket.error instance.
In python 2.7 library code e.args[0] is used. If we ever port this to
python 2 this is the best form.
This option prints the estimated size of the data that will be copied
from the source disk.
Currently this overestimates by the size of the qcow2 header, but for
real disk images that doesn't matter much.
For example:
$ virt-builder fedora-27
$ virt-v2v -i disk fedora-27.img -o null --print-estimate
[...]
virt-v2v: This guest has virtio drivers installed.
[ 44.0] Mapping filesystem data to avoid copying unused and blank areas
[ 44.5] Closing the overlay
disk 1: 1047920640
total: 1047920640
Really use the parameter of the "read_file" function, instead of
hardcoding "out". This does not change the behaviour, since all the
callers already use "out" as the file name to read.
Fixes commit 7e3689bfe0.
Add an optional argument for --machine-readable to select the output,
adding a new function to specifically write data to that output stream.
The possible choices are:
* --machine-readable: to stdout, like before
* --machine-readable=file:name-of-file: to the specified file
* --machine-readable=stream:stdout: explicitly to stdout
* --machine-readable=stream:stderr: explicitly to stderr
Adapt all the OCaml-based tools to use the new function, so the
--machine-readable choice is respected.
When creating the qemu-img command, use the guestfs_int_cmd_add_arg &
guestfs_int_cmd_add_arg_format APIs to add the proper filename directly,
without creating a string for it.
This should cause no functional change.
Remove:
common/utils/libxml2-utils.c
common/utils/libxml2-utils.h
These were accidentally added by
commit a63d02f8f1 because of a bad
interactive rebase.
It was convenient to have these as separate variants when we were only
using this type to generate JSON. However if we also use it to parse
JSON documents then integers in the document should only map to a
single variant.
Commit bd1c5c9f4d changed all the code
to use Jansson instead of yajl. However it didn't change the OCaml
module name (still Yajl).
This commit changes the module to a neutral name ("JSON_parser") and
moves it into common/mltools so it can be used by other tools.
This leaves us in a slightly awkward situation of having two JSON-ish
OCaml modules (JSON for creating trees and JSON_parser for parsing
them) with incompatible types. That is left for future work to
resolve. (It should be easier to do now that both modules live in the
same directory.)
This is just renaming and general refactoring. There should be no
change in functionality.
Avoids the warning:
libtoolize: Remember to add 'LT_INIT' to configure.ac.
This is the new name for AC_PROG_LIBTOOL, so I removed that.
However to use this macro we must enable AC_USE_SYSTEM_EXTENSIONS.
(AC_GNU_SOURCE was removed back in 2011).
This change helps to make libguestfs package build reproducible.
See https://reproducible-builds.org/ for why this is good.
Without this patch, building today's libguestfs in 2033, claims
Copyright (C) 2009-2033 Red Hat Inc.
which cannot be correct.
This affected files like
/usr/include/guestfs-gobject.h
/usr/lib/perl5/vendor_perl/5.26.2/x86_64-linux-thread-multi/Sys/Guestfs.pm
/usr/lib64/python2.7/site-packages/guestfs.py
/usr/lib64/ocaml/guestfs/guestfs.mli
Commits like 212762c593
will take care of updating the year.
Fix the error format so we actually format the arguments instead of
raising the format string and the arguments.
Here is an upload error formatted correctly with this change:
nbdkit: python[1]: error: /home/nsoffer/src/libguestfs/tmp/v2v.eC5yCl/rhv-upload-plugin.py:
pwrite: error: could not write sector offset 218911744 size 3584: 403 Forbidden:
b'{"explanation": "Access was denied to this resource.", "code": 403, "detail":
"Ticket u\'61ac0483-48e3-4984-84d6-438884ba8bb2\' expired", "title": "Forbidden"}'
The oVirt server may fail a PUT request before reading all request body.
However before closing the connection, it writes a detailed error
response, that will make debugging issues like expired tickets much
easier to handle. If we don't get the response and log the error, the
error may be lost when the daemon log is rotated.
Change pwrite() and emulate_zero() to get the response after EPIPE,
failing with the error response from the oVirt server.
Here is an example error log when a ticket expires during import:
nbdkit: python[1]: error: /home/nsoffer/src/libguestfs/tmp/v2v.pRoyXm/rhv-upload-plugin.py:
pwrite: error: ('%s: %d %s: %r', 'could not write sector offset 1841154048 size 1024', 403,
'Forbidden', b'{"explanation": "Access was denied to this resource.", "code": 403, "detail":
"Ticket u\'6071e16f-ec60-4ff9-a594-10b0faae3617\' expired", "title": "Forbidden"}')
Make use of the helper provided by Tools_utils.create_standard_options,
so there is no need to implement the logic in each tool.
This affects all the OCaml tools with --machine-readable, namely:
virt-builder, virt-builder-repository, virt-dib, virt-get-kernel,
virt-resize, virt-sparsify, and virt-v2v.
Store the machine-readable flag globally, just like done for
verbose/debug/etc, and enhance create_standard_options to provide
--machine-readable automatically.
These are file descriptors, not the high level OCaml in_channel/
out_channel type, so we would normally not refer to them as *_chan.
Just renaming, no functional change.
When parsing the libvirt XML, make sure to assign the IDs for disks
(s_disk_id) from 0 instead of 1. This does not change the actual
behaviour, just makes the IDs like in all the other input modes not
based on libvirt XML.
The output method used the s_disk_id field assuming it was a unique,
monotonically increasing number counting from 0. However this is not
the case, the input method simply has to set s_disk_id to be unique
for each disk.
Fixes commit cc04573927.
Thanks: Xiaodai Wang
When erroring out about duplicated parameters, say "more than once"
instead of "twice", since there can be more than two repeated
parameters.
Thanks to: Xiaodai Wang
Patch fe1a8866 added --mac option that allows mapping bridges/networks
based on the MAC address. It would be nice to have that listed in
machine-readable output
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
If the /etc/fstab of a guest contains devices specified with UUID or
LABEL, then the new OCaml inspection code will report the findfs failure
as general failure of the inspection. OTOH, the old C inspection code
simply ignored all the devices that cannot be resolved.
Hence, restore the old behaviour by ignoring unresolvable devices.
In commit 2894b8f392, #prepare_metadata
was added then later removed.
This commit provides the same data to #prepare_targets so that output
methods can create metadata early if necessary. (Note that creating
metadata here is optional, it should normally be created
by #create_metadata method after the copying step).
Remove this function (not available in any released stable version of
virt-v2v). It complicates the code and is not used anywhere.
This reverts commit 9ac4f99fdc.
The init_targets function is pretty odd, responsible for at least two
rather different functions: calculating the final format of each
target, and calling output#prepare_targets to get the output mode to
create each target.
Factor out the calculation of format into a separate function. This
allows us to enhance this feature, so that -o null can now properly
override the output format (whereas previously it relied on some
internals).
After this, init_targets becomes mostly a small wrapper around calling
output#prepare_targets and we can remove init_targets completely.
Previously we initialized the targets and called #prepare_targets very
early on. However this causes various problems:
- the targets struct is only half-initialized
(target_file is set to a dummy value)
- the targets struct is passed around unnecessarily
The ostensible reason for this was so that #prepare_targets is called
very early on so that errors can be displayed quickly. This is
important, but in practice it doesn't really matter if the user has to
wait for a few seconds for the conversion to happen first.
We also have the #precheck method which can do some pre-checking
functions, and that does run very early.
This is just refactoring, but it will allow some more significant
changes in how targets get initialized. There are some large
whitespace changes in this patch so it is better to look at it using
‘git show -w’.
As there is a 1-1 relationship between overlays and targets, it
doesn't matter where this structure actually lives. Move it from the
target struct to the overlay struct.
Commit 5a620137bf (in 2014) introduced
an output#check_target_free_space method but noted that it was not
used:
Provide an extra output method (#check_target_free_space) which output
modules may use to check that there is sufficient free space to
proceed, before conversion starts. None of the output modules
actually implement this at the moment.
As this method has never been used in nearly 4 years, remove it.
As a side effect, a lot more fields are now settable on the
kernel command line.
Existing kernel command lines & corresponding documentation should
still remain backwards compatible.
Since the previous commit introduced configuration sections, we can
now group more configuration settings together in these sections.
The largest part of this commit is a simple renaming of fields in the
struct.
Mostly refactoring to make it easier to add fields to this struct in
future.
I'd like to call the header p2v/config.h but that's not possible since
it conflicts with the autoconf-generated file.
The old vgscan API literally ran vgscan. When we switched to using
lvmetad (in commit dd162d2cd5) this
stopped working because lvmetad now ignores plain *scan commands
without the --cache option.
We documented that vgscan would rescan PVs, VGs and LVs, but without
activating them.
I have introduced a new API (lvm_scan) which scans or rescans PVs, VGs
and LVs. It has an optional activate parameter allowing activation of
any new LVs that are found.
With lvmetad this nicely maps to the single command:
pvscan --cache [--activate ay]
QEMU for x86 supports two machine types, "pc" (emulating the ancient
Intel i440FX chipset originally used by the Pentium Pro), and "q35"
(https://wiki.qemu.org/Features/Q35).
Currently virt-v2v does not set any machine type, so libvirt or the
target hypervisor will choose some default, probably i440fx. The
latest advice from the QEMU and libvirt communities is not to rely on
the default machine type but to specify what we need explicitly, but
it may also be that the libvirt configuration file has been changed to
set the default machine type to Q35 (either by the distro or the end
user).
None of this matters for reasonably new guests since they can boot
with either chipset. However there are some very old guests (notably
Windows XP) which cannot handle Q35.
This commit changes virt-v2v so it always tries to specify the machine
type explicitly (assuming the target supports that, and not all of
them do). For x86_64 guests this patch always selects i440fx (pc).
In future we hope to get the correct machine type for the guest from
libosinfo but this is not available yet.
For non-x86 architectures we select the "virt" model which will
probably only work for AArch64. More work is needed for POWER.
For secure boot we still have to force the machine type to Q35. In a
future version this forcing can be removed since any guest which is
using secure boot will be new enough that it'll be using Q35 anyway
(on x86).
Search for DC by 'storage.name=' to make it explicit.
I.e. "storage=" uses regex, so similar names can be
found in the search query. For example, searching for
a domain named FCSD, will find FCSD1 as well.
This allows specific NICs (identified by their source MAC address) to
be mapped to networks or bridges on the target. You can use the --mac
parameter to select this mapping, eg:
$ virt-v2v ... \
--mac 52:54:00:d0:cf:0e:network:mgmt \
--mac 52:54:00:d0:cf:0f:network:clientdata
The old --network and --bridge mappings can also be used but --mac
takes precedence.
Note this does not adjust MAC addresses inside the guest which is a
hard problem to solve. For this to work you must still carry over the
MAC addresses from the source to target hypervisor as that is how most
guests identify and associate functions with specific network
interfaces.
Due to a conflict with the IDs of the OVF standard, and the existing
implementation in ovirt-engine, the ID of QXL devices changed to a
different value.
As a consequence, change the ResourceType of QXL devices, but only in
OVirt flavour to avoid breaking vdsm mode.
See: https://bugzilla.redhat.com/show_bug.cgi?id=1598715#c5
virt-v2v moves guests, it doesn't clone them. Therefore we should try
to preserve the VM Generation ID (genid) as much as possible.
This has the ability to read the genid from VMware VMX files and
libvirt XML (but note RHBZ#1598348). It can also write the genid to
libvirt (‘-o libvirt’, ‘-o local’) and QEMU (‘-o qemu’).
We are missing support currently for all OVF-based formats (hence
‘-i ova’, and all oVirt/RHV output modes). It's unclear where we
would store the genid in this format. oVirt does not yet support
genid (see RHBZ#1118825).
Instead of storing the original network name (s_vnet_orig) in the
source NIC struct and then trying to guess what happened much later
when we're writing target metadata, get our newly created Networks
abstract data type to store the precise mapping explanation.
For direct upload, a suitable host must be in status 'Up'
and belong to the same datacenter as the created disk.
Added these criteria to the host search query.
Currently unused, this method will allow output methods to prepare the
target hypervisor to receive a guest before the actual copy/upload
begins.
For rhv-upload this is useful since RHV would prefer to have the
metadata for the guest before receiving disk data. The only metadata
unavailable here is target_stats.target_actual_size, but in the RHV API
case this is never available as we stream the data directly into RHV
rather than writing it to a local file.
This change avoids us threading the targets structure awkwardly
through several functions, and also makes it clear from the code that
the targets struct does not change after its creation in init_targets
(but the stats change).
Just code refactoring.
The optimization to start the transfer on the local host makes sense
only when using the rhv-direct=true option. When using a proxy, let the
engine choose a host.
When optimizing the connection using unix socket, we handle these cases:
- The local host is not an oVirt host (no /etc/vdsm/vdsm.id).
- The local host is an oVirt host, but is not registered with engine.
- Creating UnixHTTPConnection() fails. Unlikely and probably a bug in
the plugin, but we can recover by using the https connection.
The current code handle these cases silently, making it harder to
understand why the unix socket optimization did no happen. Add debug
message to make this clear.
Also comment in the error handler why we take this path instead of
failing the operation.
Old imageio proxy was using Authorization header for GET and PUT
requests. Remove unneeded authorization when sending OPTIONS request.
Remove unneeded duplicated comments about authorization for old
imageio, and replace them with a comment when we set needs_auth.
Add the podcheck.pl test, to make sure the documentation is in sync with
the available options.
Also fix the formats of options in the current POD, so it matches what
used already in other documentations, and what podcheck.pl expects.
In the case where virt-v2v runs on the same server as the imageio
daemon that we are talking to, it may be possible to optimize access
using a Unix domain socket.
This is only an optimization. If it fails or if we're not running on
the same server it will fall back to the usual HTTPS over TCP
connection.
Thanks: Nir Soffer, Daniel Erez.
Previously we lazily requested the server options in the can_*
callbacks. The can_* callbacks are always called by nbdkit straight
after open, so this just adds complexity for no benefit. This change
simply makes the code always fetch the server options during the open
callback.
This is — functionally at least — mostly just refactoring. However I
also added a useful debug message so we can see what features the
imageio server is offering.
"http" and "transfer" variables were missing in emulate_zero, so the
code would fail with NameError. This can happen only when communicating
with old imageio versions not supporting the "zero" feature.
Testing with qemu-img 2.12 show that we never send emulated zero request
because of the highestwrite mechanism, but it can break with older
qemu-img-rhev used on RHEL.
Python manual warns[1]:
Note that you must have read the whole response before you can send
a new request to the server.
The reason for this warning is exposed only when the server is using
keep alive connections. When the response is not read, sending a new
request will fail with:
httplib.ResponseNotReady
Even if Content-Length was 0 or the request has no content. The failure
looks like this when using --verbose:
nbdkit: python[1]: debug: zero count=33554432 offset=0 may_trim=1 fua=0
nbdkit: python[1]: debug: zero count=33554432 offset=33554432 may_trim=1 fua=0
nbdkit: python[1]: error: /home/nsoffer/src/libguestfs/tmp/rhvupload.Au2B5I/rhv-upload-plugin.py: zero: error: Request-sent
nbdkit: python[1]: debug: sending error reply: Input/output error
qemu-img: error writing zeroes at offset 0: Input/output error
Change all requests to read the whole response.
Tested with imageio patch supporting keep alive connections:
https://gerrit.ovirt.org/#/c/92296/
[1] https://docs.python.org/3.8/library/http.client.html#http.client.HTTPConnection.getresponse
The new ‘-o rhv-upload’ output mode contains a '-' character in the
name, but the regular expression which matched the output of the
virt-v2v command did not recognize '-' as a valid character. It ended
up mapping this to just "rhv" meaning two "rhv" entries would appear
in the list of output drivers.
Thanks: Ming Xie.
Virt-v2v transfers to newer versions of ovirt-engine can fail with:
ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "failed to parse a given ovf configuration ovf error: [empty name]: cannot read '//*/disksection' with value: null". HTTP response code is 400.
This was caused by a change made to oVirt:
https://gerrit.ovirt.org/#/c/91902/
so that it now requires the <Disk ovf:capacity> attribute.
Thanks: Arik Hadas
When sending request with small or no payload, it is simpler and
possibly more efficient to use the high level HTTPSConnection.request(),
instead of the lower level APIs.
The only reason to use the lower level APIs is to avoid copying the
payload, or on python 2, to use a bigger buffer size when streaming a
file-like object.
The 'localmountpoint' variable in the handle is available only when
building with FUSE support, so guard it in a proper #ifdef block.
Fixes commit 296370fb86.
Thanks to: Corentin Labbe (both reporting, and testing)
Simple refactoring of the order of parameters to input objects
so they are always in the order:
input_foo ; class name
input_conn ; command line -iX parameters, alphabetically
input_password ;
... ;
vddk_options ; extra class parameters
... ;
guestname ; remaining command line parameters
Consistent with the option -op added in
commit a4e181137a, use -ip to pass
passwords for the input side.
This replaces --password-file, and like that option you have to pass a
filename.
This also changes the code so we pass around the filename instead of
the password between functions. For ‘-it vddk’ this simplifies
things: we can pass the filename through to nbdkit and we never need
to read it in virt-v2v. For other input methods we eventually need to
read the password from the file at some lower level place in the code.
This increases the inactivity timeout for transfers from the default
(60 seconds in oVirt < 4.2.3, 600 seconds in >= 4.2.3), up to 1 hour,
so that we should never hit it for ordinary transfers.
Note this requires oVirt >= 4.2.3. The corresponding oVirt fix was
https://bugzilla.redhat.com/1563278
I also replaced the deprecated ‘image’ parameter with ‘disk’.
Thanks: Nir Soffer, Daniel Erez.
Factor out the common code used to hexdump the protocol when
./configure --enable-packet-dump is used.
It's not quite a straight refactor because I had to fix some
signedness bugs in the code which were not revealed before because
this code is usually disabled.
Extended MBR partitions cannot hold filesystems - filter them out.
RWMJ:
* Simplify is_mbr_extended function.
* Fix regression test rhbz1285847:
The test assumed that list-filesystems would return an extended MBR
partition (as "unknown"). However such partitions only contain
logical partitions inside them, not filesystems.
* Some minor whitespace changes.
1. Now we use GPT partition type to filter out LDM partitions.
2. We also check for filesystems on LDM volumes because LDM partitions
doesn't contain filesystems (list_ldm_partitions is not called anymore).
3. Obvious repetitive comments are moved to a function description.
Test guestfish list-filesystems command finds file system on partitioned md device and doesn't take into account md device itself (similar to as physical devices are filtered out if they are partitioned).
Instead of using part_to_dev to find such devices we open the device's
directory under /sys/block/<dev_name> and look for entries starting with
<dev_name>, eg. /sys/block/sda/sda1.
I saw several Windows disk images which contains strange registry entry
for mapped drives:
"\\DosDevices\\Y:"=hex(3):00,00,00,00,00,00,00,00,00,00,00,00
Which is decoded something like diskID = 0x0, partition starts at 0
bytes offset from the start of the disk. In addition to a Windows disk
image, I have attached dummy disk and made xfs file system on a whole
device without partitioning it. I mount xfs file system to a "/" and
then mkdir and mount other found file systems inside (/fs1, /fs2 etc.).
When we decode drive mappings we are looking for a disk with ID 0x0 (it
is 4 bytes somewhere LBA0). It is appeared that dummy non-partitioned
disk with xfs file system has zeros by offset where diskID is expected
to be). So the disk is considered as a candidate to search for
partition at offset 0. part-list command (and "parted" which is used
under the hood) reports there is 1 partition on "dummy" disk which
starts exactly at offset 0. And thus dummy device name and partition
number are simply concatenated together and corresponding drive mapping
is returned: Y => /dev/sdX1. But /dev/sdX1 is not existing block
device.
No matter either it is a bug in "parted" (or it works this way
by-design), let's protect ourself from this situation: in addition we
look for msdos partition table on a disk before making any further
assumptions.
When the daemon starts up it creates a fresh (empty) LVM configuration
and starts up lvmetad (which depends on the LVM configuration).
However this appears to cause problems: Some types of PV seem to
require lvmetad and don't work without it
(https://bugzilla.redhat.com/show_bug.cgi?id=1581810). If we don't
start lvmetad earlier, the device nodes are not created.
Therefore move the whole initialization step into appliance/init.
Two further changes had to be made:
Now we are using lvmetad all the time, using vgchange is incorrect.
With lvmetad activated early we must use ‘pvscan --cache --activate ay’
to scan all disks for PVs and activate any VGs on them (although the
documentation is complex, confusing and contradictory so I'm not
completely sure about this).
The ‘lvm_system_dir’ local variable in ‘daemon/lvm-filter.c’
previously contained the path of the directory above $LVM_SYSTEM_DIR
(eg. $LVM_SYSTEM_DIR = "/etc/lvm", lvm_system_dir = "/etc"). As this
was highly confusing, I have changed it so the local variable and the
environment variable have identical contents. This involved removing
the ‘lvm/’ component from a couple of paths since it is now included
in the local variable.
Use the $(srcdir) variable where needed, to make sure it builds also
with srcdir != builddir.
Furthermore, make sure that the OCaml dependencies calculation depend on
the generated output_rhv_upload_*_source.ml files, otherwise there will
be incomplete OCaml rules for them in the generated .depend.
Fixes commit cc04573927.
The current detection code for Linux kernels assumes that a kernel
package contains everything in it, i.e. the kernel itself, its modules,
and its configuration. However, since recent Ubuntu versions (e.g.
starting from 18.04) modules & config (with few more files) are split in
an own package, thus not detecting the modpath from installed vmlinuz
files.
To overcome this situation, rework this detection a bit:
1) find the vmlinuz file as before, but then immediately make sure it
exists by stat'ing it
2) find the modules path from the package as before:
2a) if found, extract the version in the same step
2b) if not found, get the kernel version from the vmlinuz filename,
and use it to detect the modules path
3) check that the modules path exists
The detection done in (2b) is based on the current packaging scheme
found in the most important Linux distributions (Fedora, RHEL, CentOS,
Debian, Ubuntu, openSUSE, AltLinux, and possibly more). The notable
exception is Arch Linux.
As additional change, do not assume the config file is in the same
package as vmlinuz, but directly look into the filesystem using the
version we already have.
Commit 4699c7b6e1 converted the null
output to use the null-co qemu driver with a JSON URL syntax --
especially the latter is only available in newer versions of qemu.
Even if this output mode is mostly for testing, check at runtime whether
the null-co + JSON way is possible, falling back to the creation of
thrown-away temporary files as before.
Newer versions of file slightly changed the output, removing the comma
between the type, and the architecture string.
Tweak the regular expression so:
- the comma is optional, but if missing then only the architecture
string will follow
- the architecture string has no commas
Starting OpenJDK 10, the 'javah' utility is no more provided [1], and
its functionality is provided by 'javac' itself. Hence, do not error
out on missing 'javah', and store whether it was found; in case it is
not, then:
1) assume 'javac' has the -h parameter to generate the C header, and
make use of it
2) tell the buildsystem that com_redhat_et_libguestfs_GuestFS.h depends
on the JAR, since the the header is generated at the step (1)
[1] https://bugs.java.com/view_bug.do?bug_id=JDK-8182758
Since it does not make much sense, then forbid this situation outright:
- change qemuopts_end_arg_list() to return an error if the current arg
list has no elements
- when creating the argv array, assert that each arg list is not empty
Some compilers do not manage to figure out that the members of it are
set only when search_appliance() in the end returns 1, which is already
checked. Help them a bit by resetting the appliance_files struct on our
own, so they will not report that 'appliance.kernel', and the others are
used as uninitialized.
Use the cleanup handlers to free the structs (and list of structs) in
case their OCaml->C transformation fails for any reason; use calloc()
to not try to use uninitialized memory.
In case of lists, avoid allocating the memory for the array if there
are no elements, since the returned pointer in that case is either NULL,
or a free()-only pointer; also, set the list size only after the array
is allocated, to not confuse the XDR routines.
"localmountpoint" parameter is allocated in JNI before calling
mount_local and freed afterward. But guestfs handle keeps reference
to passed "localmountpoint" parameter and will try to access it in
umount_local and free after mount_local_run caller thread ends
which leads to a crash (an attempt to access to already freed memory).
RWMJ: Remove ‘const’ from definition of localmountpoint, and
wrap a comment at 80 columns.
Instead of parsing 'parted' output OCaml implementation relies on the following facts:
1. MBR partition table can hold up to 4 "primary" partitions.
2. Partition with number greater then 4 is "logical" partition.
3. Partition with number less then or equal to 4 with MBR ID 0x05 or 0x0f is "extended" partition (http://thestarman.pcministry.com/asm/mbr/PartTypes.htm; https://en.wikipedia.org/wiki/Partition_type) or "primary" otherwise.
When creating an helper string for do_aug_match(), use a simpler
asprintf() with manual free(), since the code block is small enough.
This slightly helps static code analyzers.
Call realloc() directly with the pointer to the old data, instead of
assigning it to the destination variable, and using that one. The rest
of the code is the same, already properly checking for the results of
realloc(), so this mostly help static code analyzers.
free_last_match() frees the memory of the match passed as argument, so
accessing it is not possible after free_last_match(). Since all we
need is the return code, save it locally for later usage.
Check for hrefs like "../../../../dev/sda", or dubious symlinks in the
tarball, which could escape from the tarball into the host system.
We use realpath(3) to do this. Since this function does not work on
non-existent files, we must change the signature of get_file_ref so it
can return whether the file exists or not.
Factor out the complex code that handles dealing with multiple
different OVA file formats into a separate Parse_ova module.
This is largely straightforward code refactoring -- there should be no
significant functional change.
However:
- Parse_ova now checks up front if the OVA contains any compressed
disks and avoids the tar optimization in that case. This is a
regression for the case of an OVA containing a mix of both
compressed and uncompressed disks (we expect this to be rare).
The change is nevertheless good because it reduces the coupling
between two parts of the code.
- I had to simplify an error message.
We use ‘tar tRvf’ to parse the locations of files within the tarball.
However examination of tar.git:src/list.c shows various other messages
which can appear in the output:
block <offset>: ** Block of NULs **
block <offset>: ** End of File **
Indeed it was easy to produce the first message just by using modern
tar to create a tarball:
$ tar tRvf '/var/tmp/bz1570407-reproducer.ova'
block 0: -rw-r--r-- rjones/rjones 100 2018-04-22 17:06 RHEL7_3_042218_extra-disk1.vmdk.000000000
block 2: -rw-r--r-- rjones/rjones 243 2018-04-22 17:07 RHEL7_3_042218_extra.mf
block 4: -rw-r--r-- rjones/rjones 13066 2018-04-22 15:08 RHEL7_3_042218_extra.ovf
block 31: ** Block of NULs **
Ignore these messages.
Some OVA files generated by VMware have a *.mf file which contains
checksums for files which don't exist in the OVA. Ignore these
checksums.
Thanks: Nisim Simsolo.
As well as being consistent with other tests in this directory this
also fixes a problem when we ran the tests and the real virtio-win
package happened to be installed on the same machine.
The test used to pick up the virtio drivers from the virtio-win
package, resulting in different test output (because virtio drivers
had been installed).
Now we use the phony virtio drivers in all cases and also update the
test output (because virtio drivers are now always installed).
Because we checked the limitation of raw+sparse during command line
processing, it had the effect of forcing you to use ‘-of raw’ even if
the input was already in raw format. Move the checking to the output
mode to avoid this.
Fixes commit cc04573927.
Instead of storing the number of sockets, cores, and threads separately
with the possibility to have each of them optional, store them together,
so either we have a CPU topology or none. This slightly simplifies the
handling of CPU topology.
Most of the output modes/producers already considered a missing
information of a topology as "1" when any of the other was specified, so
the behaviour is unchanged. The only behaviour changes are when
parsing, and creating libvirt XMLs:
- libvirt requires all the three attributes to be specified, so when
reading use them only if <topology> has all of them
- for the same reason, write <topology> only when all of them are
specified
Adapt the vmx, and ova input modes to consider 1 threads when they have
the other information of a CPU topology, since none of them can provide
that information yet.
When updating the device names to the new names, handle GRUB_CMDLINE
entries (typically in grub2 configuration files) correctly: loop over
the whole configuration value, and replace each occurrency of
resume=/dev/device found.
The actual code is moved away from replace_if_device, since this
function now does only the actual replacement of the device string.
Some filesystems fall back silently to read-only if there are problems
such a dirty filesystem and an unrecoverable journal. Almost all
conversions involve writing to the root filesystem, so these will
inevitably fail later on with a strange error message.
Test the root filesystem is writable by creating and deleting a
temporary file, and if the creation fails then give better
diagnostics.
Reported-by: Piotr Kliczewski
This adds a new output mode to virt-v2v. virt-v2v -o rhv-upload
streams images directly to an oVirt or RHV >= 4 Data Domain using the
oVirt SDK v4. It is more efficient than -o rhv because it does not
need to go via the Export Storage Domain, and is possible for humans
to use unlike -o vdsm.
The implementation uses the Python SDK (‘ovirtsdk4’ module). An
nbdkit Python 3 plugin translates NBD calls from qemu into HTTPS
requests to oVirt via the SDK.
The guestfish prepared disks (-N option) all defaulted to 100M. This
has always been too small for btrfs (so for example ‘-N fs:btrfs’ has
failed for a long time), and can be too small for LVs with new LVM2 /
kernel >= 4.16.
This changes the default from 100M to 1G. The beginning of the 1.39
branch seems like a good time to make this change.
This is just a mechanincal change, so that the public documentation[*]
refers to the latest release Fedora versions, instead of the EOLed
versions.
While at it, also update the `virt-cutomize` Makefile.am
[*] http://libguestfs.org/virt-builder.1.html
Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com>
Add a way to generate OCaml interfaces for all the modules in the
daemon that implement APIs: this makes sure that for them the
interface of each function matches the actual API specified in the
generator.
This way the Mount module contains only the OCaml implementations of
mount-related daemon APIs.
This is simple refactoring, with no functional changes.
Don't clone online gnulib repo if the environment variable $GNULIB_SRCDIR
is available, and add an option '--gnulib-srcdir' for bootstrap to specify
a local source directory.
(borrowed some code which wrote by Jim Meyering and Eric Blake in libvirt)
Signed-off-by: Lin Ma <lma@suse.com>
Some of users probably don't know which packages they exactly need to
install while they experienced the error message:
libmagic (part of the "file" command) is required.
So add corresponding package information into error message.
Signed-off-by: Lin Ma <lma@suse.com>
It should error out instead of warning if users provide the option
'--with-default-backend=libvirt' and no libvirt devel package installed.
Signed-off-by: Lin Ma <lma@suse.com>
When writing the OVF in OVirt flavour, add a ovirt:id attribute to the
OperatingSystemSection tag: this attribute represents the numeric value
of the ostype ID, which is ignored by oVirt when parsing OVFs in API
mode.
- consider CentOS as RHEL, since oVirt has no CentOS OS mappings, and
CentOS is derived from RHEL
- use rhel_6_9_plus_ppc64 for RHEL 6 >= 6.9 on ppc64/ppc64le, while
rhel_6_ppc64 for RHEL 6 < 6.9
- use rhel_7_s390x for RHEL 7 on s390x
- use sles_11, and sles_11_ppc64 for any SLES greater than 11, not only
11
- use sles_12_s390x for SLES 11+ on s390x
- use ubuntu_16_04_s390x for Ubuntu 16.04+ on s390x
- use other_linux_ppc64 for any unmapped Linux distro on ppc64/ppc64le
- use other_linux_s390x for any unmapped Linux distro on s390x
Currently we have a bunch of ad hoc options like --vddk* and --vdsm*
(and proposed to add --rhv*) to handle extra parameters for input and
output modes/transports. This complicates the command line parsing
and also the clarity of the command line (becauseit's not very obvious
which options apply to which side of the conversion).
Replace these with a general mechanism for handling input and output
options.
Thus (for example):
--vddk-thumbprint=... becomes -io vddk-thumbprint=...
--vdsm-compat=0.10 -oo vdsm-compat=0.10
The responsibility for parsing input and output options moves into the
input and output drivers.
This improves error checking so it's harder now for wrong flags to be
included on the command line when they don't apply to the current mode.
The old option names are preserved for compatibility.
So far v2v did not read the model of network interfaces, resulting in
"virtio" as the model for such interfaces.
Start reading the model, if available, directly mapping E1000, while
other models are considered as Source_other_nic.
I've had several reports that it just puts people off. Possibly we
should delete or rewrite it but this moves it below the examples
section which is more important.
When parsing OVA files we have to work backwards to whether the
original VMware object was a network or a bridge. It seems as if the
presence of <rasd:Connection> is sufficient. I dropped the use of
<rasd:ElementName> since it is literally just an internal name for the
network element, and is not significant in how the network is used.
Also update documentation because you must use ‘--bridge’ when
converting VMware guests to RHV.
For a full explanation see:
https://bugzilla.redhat.com/show_bug.cgi?id=1559027#c2
Newer versions of qemu use file locking for the images by default, and
apparently that does not work with /dev/null. Since this test just
calls qemu-img to get the format of an empty image, create a temporary
one instead.
This is analogous to --print-source, except that it prints the overlay
and target disk information.
The output looks like below. Note there is one overlay and one target
section per disk.
Overlay and Target information (--print-target option):
overlay file: /home/rjones/d/libguestfs/tmp/v2vovlc687fe.qcow2
overlay device name: sda
overlay virtual disk size: 6442450944
overlay source qemu URI: /var/tmp/fedora-27.img
target file: [qemu] json:{ "file.driver": "nbd", "file.path": "/home/rjones/d/libguestfs/tmp/rhvupload.IWrzO6/nbdkit0.sock", "file.export": "/" }
target format: raw
target estimated size: 2274060540
While scanning the sources, some of the files (i.e. the cached
appliance) may be larger than 4G, and thus raise EOVERFLOW on 32bit
architectures.
Fixes commit b4e119d8b7.
One gnulib test creates a symlink to the same directory, causing our
iteration to read the same test directory over and over, every time
considering it as new level of subdirectory.
As solution (or workaround), when iterating through a directory consider
only regular files, and directories, ignoring any other file type
(symlinks included).
Add option for -o vdsm that enables output of the modified OVF. oVirt
engine should already be able to consume the OVF, but let's not take any
chances and enable it only by command line argument. It can be made
default later when it receives proper testing.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
For historical reasons the OVF used in RHV export domain contains some
deviations from the OVF standard. The format used in -o rhv has to
remain fixed but for -o vdsm and we could produce much nicer OVF. This
patch serves as a preparatory step to this.
The main reason for creating different OVF is that it can be used to
create VM by oVirt REST API. The RHV export domain flavor cannot be used
that way.
For now the virt-v2v behavior is unchanged. The modified output will be
enabled in some later patch.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
Check the generated OVF for -o rhv and -o vdsm outputs. Variable UUIDs
and date/times are filtered out. Make sure the the important UUIDs
(disk, volume, VM) are where we think they should be.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
GCC 8 thinks that the case drive_protocol_gluster may fall through, most
probably because the only code is a switch case that handles the
elements of an enum, and thus letting other fall through. In reality
this ought to not happen at all, so help GCC by break'ing the case,
which will then lead to the abort() at the end of
guestfs_int_drive_source_qemu_param.
Apparently the jansson library on 64bit installation does not provide
the non-64bit name, so jansson is not pulled in the appliance, and thus
the daemon does not start.
Updates commit eb4fbe96c9.
Even though the list of checks is very short, at least this migrates
from imperative checks to a "declarative" one.
There should be no behaviour change, other than using os-release if it
contains all the needed information.
Kali Linux is a Debian derivative, so add basic support for it by using
most of the Debian code paths. The only exception is the crypto
algorithm for passwords in passwd, which is always assumed as SHA512
(as Kali Linux is relatively new).
While YAJL mostly works fine, it did not see any active development in
the last 3 years. OTOH, Jansson is another JSON C implementation, with
a very liberal license, and a much nicer API.
Hence, switch all of libguestfs from YAJL to Jansson:
- configure checks, and buildsystem in general
- packages pulled in the appliance
- actual implementations
- contrib scripts
- documentation
This also makes use of the better APIs available (e.g. json_object_get,
json_array_foreach, and json_object_foreach). This does not change the
API of our OCaml Yajl module.
On distros with OCaml < 4.02 we need to create a compatibility Bytes
module. However we didn't create the interface file (bytes.mli) which
would mean that dependencies wouldn't be created correctly for
parallel builds. This commit uses ‘ocaml -i’ to create an interface
file which exports everything.
When checking the return value of guestfs_int_py_fromstring for string
fields of structs, add a newline to generated C code, so it is properly
indented.
Fixes commit 401c445636.
An installation of MS-DOS has various files in a /DOS directory,
which COMMAND.COM looking like a reasonable signal that its MS-DOS
or a very close relative there-of.
This is validated with an MS-DOS 6.22 install.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
There were some problems with the Ukrainian translations which I have
fixed. The 3 magic strings __CUSTOMIZE_OPERATIONS__,
__CUSTOMIZE_SYNOPSIS__ and __OPERATIONS__ were translated, breaking
substitutions in virt-builder, virt-customize and virt-sysprep manual
pages.
Do a configure check for the OPEN_UNSAFE flag in the OCaml binding of
Hivex, using it only when available. This makes it possible to use
hivex < 1.3.14 to build libguestfs (the daemon, actually).
Amend the building documentation accordingly, bringing the minimum
version of hivex back as it was before
commit 64f49df747.
libselinux defaults to "targeted" when no SELINUXTYPE is specified in
/etc/config/selinux. Hence do the same here, instead of failing because
of the missing key.
Add a slow test for checking SELinux relabeling on a Fedora 27 guest,
both with no changes, and with a modified configuration.
Since virt-builder Fedora 27 templates now use GPT, the root
filesystem has moved from /dev/sda3 to /dev/sda4. /dev/sda3 is now a
swap device which could not be mounted as the root device, although
systemd unhelpfully hides the error unless you run qemu interactively.
I think this fixes commit a06e50e263.
guestfs_list_filesystems uses mount/umount to discover btrfs
sub-volumes and since 1.37 it generates random mountpoint so it will
longer affect already mounted filesystems if either.
Ancient qemu 1.5 (in RHEL 7) does not understand the
file.file.filename= and file.driver= parameters. Go back to using the
old-style file= and format= parameters when we're not trying to set
the file.backing.file.locking=off parameter.
Fixes commit 9fe592808c.
Thanks: Yongkui Guo, Václav Kadlčík.
ocamlmklib will get confused if the -cclib parameter expands to
anything except a single -lxml2 argument. In any case it's not
necessary to specify the linker command here since every use of this
library links in a program to libxml2.
We routinely test the upstream code by running everything under
valgrind, and in any case _FORTIFY_SOURCE is usually defined by
downstream Linux distros and we can leave the optimization vs safety
decision to them.
See this bug: https://bugs.gentoo.org/640494
In some cases, the first stage bootloader needs the 'Legacy BIOS
bootable' flag to be set on the partition. This change copies all
flags (including this one) for each partition of the old disk to the
new one to avoid ending up with non-bootable disks.
Rename the sgdisk_info_extract_uuid_field to
sgdisk_info_extract_field in order to reuse it for other field types.
Just like its C ancestor, it now needs an extractor function to be
passed as parameter.
Test guestfish finds:
1. md device created from physical block device and LV,
2. md device created from LVs
3. LV created on md device
raid0 is used for md device because it is inoperable if one of its components is inaccessible so it is easy observable that md device is missing (raid1 in this case will be operable but in degraded state).
The issue:
- raid1 will be in degraded state if one of its components is logical volume (LV)
- raid0 will be inoperable at all (inacessible from within appliance) if one of its component is LV
- raidN: you can expect the same issue for any raid level depends on how many components are inaccessible at the time mdadm is running and raid redundency.
It happens because mdadm is launched prior to lvm AND it is instructed to run found arrays immediately (--run flag) regardless of completeness of their components.
Later (when lvm activates found LVs) md signature on LV might be recognized BUT newly found raid components could't be inserted into already running (in degraded state)
or marked as inoperable raid arrays.
The patch fixes the issue in the following way:
1. Found arrays won't be run immediately unless ALL expected drives (components) are present. Here '--no-degraded' flag comes into a play. See mdadm(8).
2. Second mdadm call (after LVM is scanned) will scan UNUSED yet devices and make an attempt to run all found arrays (even they will be in degraded state).
There is no performance penalty because second pass scans UNUSED yet devices. Here is 'boot-benchmark' before and after patch:
: libvirt backend : direct backend
------------------------------------------------
master : 835.2ms ±1.1ms : 670.4ms ±0.3ms
master+patch : 837.7ms ±2.4ms : 671.8ms ±0.2ms
According to 'guestfs_sync' API method documentation: "You should always call this if you have modified a disk image, before closing the handle."
So, 'guestfish --remote sync' is required because changes made on the disk (guestfish --remote rm /tail) should be visible to 'virt-tail' which works in different process and also accessing the same disk.
A you can see, other changes done via 'guestfish --remote' in this test are flushed via a 'sync' command.
The test is failing spontaneously beasue without 'sync' it very depends on outside factors like qemu caching policy, underlying host filesystem etc.
This allows us to easily see from trace output (‘virt-diff -x’) which
handle is being used for each call. Otherwise you get meaningless
output such as:
libguestfs: trace: download "/Windows/WindowsUpdate.log" "/tmp/virtdiffy9m5YW/a"
libguestfs: trace: download = 0
libguestfs: trace: download "/Windows/WindowsUpdate.log" "/tmp/virtdiffy9m5YW/b"
libguestfs: trace: download = 0
Thanks: Xiang Hua Chen
Someone following this example started by literally copying "olddisk"
to "newdisk", which doesn't work (the target must always be blank).
Clarify the wording.
This enhances the existing VMX input support allowing it to be
used over SSH to the ESXi server.
The original command (for local .vmx files) was:
$ virt-v2v -i vmx guest.vmx -o local -os /var/tmp
Adding ‘-it ssh’ and using an SSH remote path gives the new syntax:
$ virt-v2v \
-i vmx -it ssh \
"ssh://root@esxi.example.com/vmfs/volumes/datastore1/guest/guest.vmx" \
-o local -os /var/tmp
I anticipate that this input method will be widely used enough that it
deserves its own example at the top of the man page.
Previously the presence of the ‘--vddk <libdir>’ option magically
enabled VDDK mode. However we want to introduce other transports for
VMware conversions so this wasn't a very clean choice.
With this commit you must use ‘-it vddk’ to specify that you want VDDK
as a disk transport. The previous ‘--vddk <libdir>’ option has been
renamed to ‘--vddk-libdir <libdir>’ to be consistent with the other
passthrough options, and it is no longer required.
A new command line looks like:
$ export PATH=/path/to/nbdkit:$PATH
$ virt-v2v \
-ic 'vpx://root@vcenter.example.com/Datacenter/esxi?no_verify=1' \
| -it vddk \
| --vddk-libdir /path/to/vmware-vix-disklib-distrib \
--vddk-thumbprint xx:xx:xx:... \
"Windows 2003" \
-o local -os /var/tmp
where only the two lines marked with ‘|’ have changed.
This is an unexpected error, so fail hard instead of leaking
End_of_file exception.
Nothing that calls into the Urandom module expects or handles
End_of_file.
Certain transitions where the input and output filename are the same
are impossible, eg copying a file to itself. Don't add these.
Reported-by: David Kaylor.
This adds a ‘return’ statement as found in other programming
languages. You can use it like this:
with_return (fun {return} ->
some code ...
)
where ‘some code’ may either return implicitly (as usual), or may call
‘return x’ to immediately return ‘x’. All returned values must have
the same type.
The OCaml >= 4.04 implementation is by Petter A. Urkedal and octachron.
See this thread:
https://sympa.inria.fr/sympa/arc/caml-list/2017-11/msg00017.html
The version that works for any OCaml is by me. (Note that my version
cannot be nested).
However some existing functions had names which shadowed existing
functions in the List module, so I had to rename them:
assoc -> List.assoc_lbl
append -> List.push_back_list
prepend -> List.push_front_list
This is an extension of the previous commit.
We defined a number of functions on lists which are not provided by
the standard library. As with Char and String, let's extend List to
add these new functions to a List pseudo-module (really
Std_utils.List, but called List when you ‘open Std_utils’).
The initial exported functions are all List functions from OCaml 3.11
+ iteri + mapi. We can add other functions as needed.
When caching all the templates, use the architecture of each template,
instead of the architecture passed as --arch (or the host architecture,
as default). This way, the right destination filename will be used.
Fixes commit b1cf6246f3.
Thanks to: Erik Skultety.
It wasn't actually used for anything significant. Even worse
virt-v2v-copy-to-local made up a scheme out of thin air.
Just code refactoring, no functional change.
Commit aa9e0057b1 made the libvirt backend
use <shareable/> for the disk of the appliance, since at that time all
the instances were using the disk directly.
OTOH, commit 3ad44c8660 switched to
overlays for read-only disks, including the appliance, so effectively
protecting the appliance.
Because of this, the libvirt backend does not need <shareable/> anymore.
Thanks to: Daniel Berrange, Richard W.M. Jones, Peter Krempa.
When using grubby to get the default kernel of a guest, do not fail
with a bogus error like:
virt-v2v: error: libguestfs error: statns: statns_stub: path must start
with a / character
in case there is no default kernel that can be determined (e.g. because
of a bogus configuration).
QEMU does not accept options unrecognized by the block driver
in use. Disable locking only for read-only disks that are
file-backed, as that's the only block driver it is supported
with.
Signed-off-by: Lars Seipel <ls@slrz.net>
virt-builder-repository allows users to easily create or update
a virt-builder source repository out of disk images. The tool can
be run in either interactive or automated mode.
get_index now gets a new template parameter. Setting it to true will
make the index parsing less picky about missing important data. This
can be used to parse a partial index file.
Change Index.arch to the type (Arch of string | GuessedArch of string).
In a future commit, the index parser will allow arch not to be set
for some cases. Thus arch value will be guessed by inspecting the
image. However we need to distinguish between a set value and a guessed
one. Using this new type will help it:
match arch with
| Arch s -> (* This is a set value *)
| GuessedArch s -> (* This is a guessed value *)
Since glibc-2.26 removed the /usr/include/rpc/rpc.h, it caused problem even
though probing for libtirpc has taken place and worked properly. The problem is
that RPC_CFLAGS (usually `-I/usr/include/tirpc` on Linux) was not used in places
where it was needed. So the build failed with messages like the following:
In file included from guestfs_protocol.c:6:0:
guestfs_protocol.h:9:10: fatal error: rpc/rpc.h: No such file or directory
#include <rpc/rpc.h>
^~~~~~~~~~~
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
Because of vagaries in how the debug messages get printed they can get
split over lines:
gulibguestfs: trace: truncate_size = 0
estfsd: => truncate_size (0xc8) took 0.00 secs
and the code which filtered out those messages before comparison was
failing. This makes the code more liberal on how it matches and
filters out these messages.
Every .ml file now needs a corresponding .mli file so that
dependencies are generated correctly. The check-mli.sh script picks
up the problem:
FAIL: check-mli.sh
==================
./builder/osinfo_config.ml: missing ./builder/osinfo_config.mli
FAIL check-mli.sh (exit status: 1)
Fixes commit 46abca0ec3
and commit 02184f55f9.
If running the external command fails in "argv mode" (ie. when
not using the shell), then exit with either 126 or 127 as defined
by POSIX.
This is mostly the same as what bash does, see
execute_cmd.c:shell_execve in the bash sources.
Note: saving errno around perror(3) if necessary, otherwise you will
see different behaviour between verbose and non-verbose mode. In
non-verbose mode, perror(3) tried to print to a closed file
descriptor, failing and overwriting errno. That took some time to
debug.
This safe wrapper around Unix.openfile ensures that exceptions
escaping cannot leave unclosed files.
There are only a few places in the code where this wrapper can be used
currently. There are other occurences of Unix.openfile but they are
not suitable for replacement.
In OCaml 4.06 we need to link to daemon to libasmrun_pic.a (the OCaml
runtime linked with -fPIC). Otherwise you will see many errors like
this:
/usr/bin/ld: /usr/lib64/ocaml/libasmrun.a(startup_aux.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/lib64/ocaml/libasmrun.a(startup.o): relocation R_X86_64_32S against symbol `caml_data_segments' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/lib64/ocaml/libasmrun.a(fail.o): relocation R_X86_64_32 against symbol `caml_exn_Failure' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/lib64/ocaml/libasmrun.a(roots.o): relocation R_X86_64_32 against symbol `caml_frametable' can not be used when making a shared object; recompile with -fPIC
This commit use a configure-time test to find the best OCaml runtime.
The immediate issue is with Fedora/ppc64 and /ppc64le which currently
use extended partitions, breaking the virt-builder ‘--size’ parameter,
eg:
$ virt-builder --arch ppc64le fedora-26 --size 20G
...
[ 21.6] Resizing (using virt-resize) to expand the disk to 20.0G
virt-resize: error: /dev/sda5: partition not found in the source disk image
(this error came from '--expand' option on the command line). Try running
this command: virt-filesystems --partitions --long -a /var/tmp/vbf67b8c.img
However more generally MBR is broken and should die. GPT is supported
by all modern virtual bootloaders, so just default to it.
Notes:
* This is different from mandating a UEFI bootloader.
* I am not planning to rebuild any existing images except the
F26 ppc64 & ppc64le ones.
libfuse prints errors on stderr and there seems to be no way to change
that. It doesn't make any attempt to preserve errno either, so
printing an error based on errno is wrong.
Thanks: Assaf Gordon.
Add a function to properly write virt-builder source index entries.
Note that this function is very similar to Index.print_entry that is
meant for debugging purposes.
Add a simple OCaml-based implementation of reader of the osinfo-db:
the only interface is an iterator that invokes an user-supplied
function with each XML file found.
This implementation behaves like the current C implementation, and
still supports the old libosinfo db.
[RWMJ: Fixed trailing whitespace]
While 1.1.14 will technically still work, 1.1.16 adds the
‘selinux=yes|no’ flag to ‘--dump-config’ output. It's also the more
widely available and tested version at the time of writing.
Thanks: Ming Xie.
- move the space needed by mountpoint mapping to an own function,
outside of the checking loop
- remove the minimum limit of 100M on partitions, since existing
partitions that we check (e.g. /boot) may be smaller than that
- in case /boot is not on a separate mountpoint, enforce the space check
on the root mountpoint instead, since it is where /boot is in that
case
If visit_guest() fails, then it returns a null pointer; later on,
free_tree() is called unconditionally on the variables, thus
dereferencing null pointers.
As a solution, make free_tree() behave like free(), i.e. become no-op
with null pointers.
Remove two wrong error handling situations in btrfs_balance_status:
- if the calloc for 'ret' fails, the 'goto error' would try to
dereference the resulting null pointer (to free the items of the
return struct); hence, just return null directly, instead of jumping
to 'error'
- if the strdup() for 'btrfsbalance_status' fails, then the directly
return would leak 'ret'
When the LDM code was converted to the CLEANUP_* macros, a free()
invocation for a CLEANUP_FREE variable was left in the
ldmtool_diskgroup_volumes implementation, causing double-free on
success.
Updates commit 950951c67d.
Linux.file_owner is not used by any other function, so remove it.
Linux.is_file_owned is only used when removing kmod-xenpv on old RHEL
releases, and so is only required to work for RPM.
The old file_owner/is_file_owned functions were completely broken for
the RPM case. This replaces them with a simpler, working
implementation of is_file_owned only.
Thanks: Ming Xie for finding and reporting the original bug.
Some hypervisors including KVM and VMware expose SATA as a separate
device emulation from IDE. SATA devices are an evolution of IDE, but
in Linux they are handled like SCSI disks.
This change adds a new Source_SATA type to model this. In the Linux
conversion module we treat them like SCSI (ie. with "sd" prefix).
It's not very clear how to handle them for target bus assignment, so I
arbitrarily mapped them to the SCSI bus. It's possible we need a new
bus type to handle SATA properly. As the code only applies to
removable devices it's not very important to get this right.
Currently test820RHBZ912499.py fails with libvirt because libvirt
still doesn't work around qemu locking properly. Allow this test to
be skipped on a temporary basis using SKIP_TEST820RHBZ912499_PY=1
Latest libvirt (3.8, I think) made a very small adjustment to the
format of the generated -drive parameter when using iscsi, from:
file=iscsi://1.2.3.4:1234/iqn.2003-01.org.linux-iscsi.fedora,
to:
file=iscsi://1.2.3.4:1234/iqn.2003-01.org.linux-iscsi.fedora/0,
This makes a corresponding change to the test so that both patterns
can be recognized.
Commit 0b631d739b changed the debug
message format from the main loop in guestfsd. This test tried to
filter out these messages when comparing output, but that stopped
working. The problem was only evident when enabling debugging while
running the tests.
Previously we were abusing the ‘input#source ()’ and
‘output#prepare_targets’ methods to perform environmental checks.
Small refactoring, no functional change.
Make use of parse_key_value_strings to slightly simplify the parsing of
mdadm output, and the parsing of os-release, and lsb-release files.
In particular, the parsing of lsb-release is simplified a lot, bringing
it much like the parsing of os-release.
Add a simple helper to turn a list of strings into key/value pairs,
splitting by '=', with the possibility to apply a function to unquote
values.
Add also a simple unquote function.
This implements a missing feature from old virt-v2v, namely being able
to cope with a guest with snapshots. Note this only converts the top
(latest) snapshot. As in old virt-v2v it does NOT convert the whole
chain of snapshots.
This module had a selection of functions taking a different mix of
parameters and doing slightly different things. You could call one
function to return an https://... URL, or another function to return a
qemu URL, and there was a third function to get the session cookie but
you generally had to call that anyway (and it was implicitly called
when making the qemu URL!)
Integrate these into a single function which returns a struct
returning all possible values.
This is conceptually refactoring, except that the session cookie is no
longer memoized, but we didn't use this feature (of calling
get_session_cookie multiple times) anyway.
With modern libvirt, when fetching the XML of a VMware guest libvirt
passes us the datacenter path (dcpath). However with older libvirt we
had to guess this value, or else the user had to supply it on the
command line.
This commit removes all the guessing code and the --dcpath parameter
(which will now give an error).
This requires libvirt >= 1.2.20 for virt-v2v, released Oct 2015.
In case os-release does not contain VERSION_ID, and it is not an
exception (like Void Linux), then discard the results got from
os-release.
This matches what the old C code did -- e.g. see
commit 32d19e3289, and the changes in
commit 0251c0fcaa.
Instead of using ‘xpath_(string|int|int64)_default’ we can write the
equivalent code using ‘Option.default’. This is not quite so concise,
but may be easier to understand.
eg:
xpath_int_default xctx "xpath_expr" 10
->
Option.default 10 (xpath_int xctx "xpath_expr")
Inspired by ocaml-extlib, introduce a module for handling option
types.
We already had the ‘may’ function (which becomes ‘Option.may’). This
adds also ‘Option.map’ (unused), and ‘Option.default’ functions.
Note this does *not* introduce the unsafe ‘Option.get’ function from
extlib.
When testing if sudo -n requires a password, we tested for the prompt
earlier than testing for the magic sudo message ‘a password is
required’.
Since the shell will print the prompt just after the sudo message:
prompt$ sudo -n virt-v2v --version
sudo: a password is required
prompt$
the prompt nearly always matched and we missed the magic sudo message.
(The exception is in the case where we are running everything on
localhost where the sudo message could be read in a single call to
read(2) without seeing the prompt immediately afterwards. Even this
exception was non-deterministic.)
By swapping the priority of the sudo message and prompt we avoid this.
Looking at the debug output (enabled by editing common/miniexpect)
makes this clearer:
DEBUG: writing: sudo -n virt-v2v --version
DEBUG: buffer content: sudo: a password is required
ESC]0;rjones@hamr:~^G###bphcxtq5###
Fixes commit 5b6a8e0862.
Thanks: Ming Xie.
The upstream ocaml-augeas defines macros which may not be used in all
the cases, triggering a warning (which turns into error with -Werror)
because of -Wunused-macros.
Since the code is correct, disable -Wunused-macros with
-Wno-unused-macros before updating the embedded copy of ocaml-augeas.
The embedded copy of ocaml-augeas does Val_int(-1), which in turns
triggers warnings in newer GCC versions about "left shift of negative
value". The issue actually lies in the OCaml headers (mlvalues.h in
particular), and it was fixed in newer OCaml versions.
Since the code is actually correct, disable -Wshift-negative-value with
-Wno-shift-negative-value (checking whether the compiler has it).
Use also custom_compare_ext_default for the compare_ext field.
According to the git logs, this was introduced in OCaml 3.12.1, which is
earlier than out minimum required version.
mlaugeas is not touched by this change, since it is a copy of a 3rd
party library (and thus it will be fixed there first).
This function returns -1 if things like fork(2) fail, so it is right
to return an error here. If the PBMTEXT command fails then that's a
NOT_FOUND case.
We reimplemented some functions which can now be found in the OCaml
stdlib since 4.01 (or earlier). The functions I have dropped are:
- String.map
- |>
- iteri (replaced by List.iteri)
- mapi (replaced by List.mapi)
Note that our definition of iteri was slightly wrong: the type of the
function parameter was too wide, allowing (int -> 'a -> 'b) instead of
(int -> 'a -> unit).
I also added this new function to the Std_utils.String module as an
export from stdlib String:
- String.iteri
Thanks: Pino Toscano
Change the Planner.plan function so it returns an optional type. This
means it no longer raises Failure "plan" on error, so we can both
force the caller to deal with the error case and avoid Warning 52.
This avoids warning 52 in OCaml code such as:
try URI.parse_uri arg
with Invalid_argument "URI.parse_uri" -> ...
which prints:
Warning 52: Code should not depend on the actual values of
this constructor's arguments. They are only for information
and may change in future versions. (See manual section 8.5)
In the long term we need to change fish/uri.c so that we can throw
proper errors.
If you have a struct containing ‘field’, eg:
type t = { field : int }
then previously to pattern-match on this type, eg. in function
parameters, you had to write:
let f { field = field } =
(* ... use field ... *)
In OCaml >= 3.12 it is possible to abbreviate cases where the field
being matched and the variable being bound have the same name, so now
you can just write:
let f { field } =
(* ... use field ... *)
(Similarly for a field prefixed by a Module name you can use
‘{ Module.field }’ instead of ‘{ Module.field = field }’).
This style is widely used inside the OCaml compiler sources, and is
briefer than the long form, so it makes sense to use it. Furthermore
there was one place in virt-dib where we are already using this new
style, so the old code did not compile on OCaml < 3.12.
See also:
https://forge.ocamlcore.org/docman/view.php/77/112/leroy-cug2010.pdf
common/mlutils: Unix_utils.StatVFS.statvfs: This commit implements a
full-featured binding for the statvfs(3) function.
We then use this to reimplement the daemon statvfs API in OCaml.
Note that the Gnulib fallback is dropped in this commit. It
previously referenced non-existent field names in the fs_usage struct
so it didn't work. Also it's not necessary as POSIX has supported
statvfs(3) since 2001, it's supported in *BSD, macOS > 10.4, and there
is already a Windows fallback.
This directory which previously contained random modules and functions
now has an official purpose: to be the place for any OCaml utility
needed by the OCaml virt tools.
This is just code movement, I didn't (yet) rename or move any of the
modules.
Although it's not too likely that these libraries will contain
translatable strings, it's consistent to add them to po/POTFILES-ml
because mllib/*.ml are also in this file.
I also renamed the functions with the correct ‘guestfs_int_*’
namespace.
The corresponding change is made for OCaml C_utils and Visit.
No functional change, just code motion.
systemd defined an /etc/machine-id file which is supposed to contain a
unique, unchanging ID for the host. This file is initially zero-sized
and is meant to be set by systemd on the first boot of the system. In
virt-builder Fedora templates, the file is empty.
Unfortunately the Fedora kernel %post script requires the machine-id
to have been set, else the script exits with an error:
Running scriptlet: kernel-core-4.12.13-300.fc26.x86_64 209/209
Could not determine your machine ID from /etc/machine-id.
Please run 'systemd-machine-id-setup' as root. See man:machine-id(5)
warning: %posttrans(kernel-core-4.12.13-300.fc26.x86_64) scriptlet failed, exit status 1
This also leaves the kernel package half-installed. The files are
present in the filesystem, but important initialization is not done,
in particular the vmlinuz file is not copied into /boot.
A simple reproducer for this problem is:
$ virt-builder fedora-26 --update
which will leave the image with a half-installed kernel. (Add -v -x
to see the error above amongst the debug output).
This change makes virt-customize set /etc/machine-id to a random value
if the file exists and is zero sized. This is done unconditionally at
the same time as setting the random seed (a similar issue), and before
running any customize options such as installing or updating packages.
The old virt-v2v code ignored boot kernels with names like
"/boot/vmlinuz-*.rpmsave". The transscribed code did not because the
Str module requires ‘|’ to be escaped as ‘\|’.
This changes the code to use a simpler test.
Thanks: Pino Toscano
Commit dbe0b69f24 transscribed the Perl
regexp incorrectly so that it only matched the impossible case of
‘resume=/dev/X’ for a single non-whitespace character X.
This fixes the regular expression, referencing back to the original
Perl code in virt-v2v.
In the original conversion of virt-v2v from Perl
(commit 0131d6f666), the Perl regular
expression was incorrectly transscribed. ‘\b’ was not converted to
‘\\b’ so the new regexp was looking for an ASCII BEL character, not a
word boundary. Also ‘|’ was used, but Str requires ‘\|’ (ie. "\\|"
in the final source).
To fix these problems I converted the code to use PCRE, and went back
to the original virt-v2v code (virt-v2v.git:
lib/Sys/VirtConvert/Converter/Linux.pm) to find out what the Perl
regular expression should have been.
Note I have also removed ‘.*’ at the beginning and end of the regexp
because PCRE regexps are not anchored.
Only five simple flags are allowed so far, and not all of them are
actually used in any code. They are:
~anchored / PCRE_ANCHORED - implicit ^...$ around regexp
~caseless / PCRE_CASELESS - fold upper and lower case
~dotall / PCRE_DOTALL - ‘.’ matches anything
~extended / PCRE_EXTENDED - extended regular expressions
~multiline / PCRE_MULTILINE - ^ and $ match lines within the subject string
If configured with --without-ocaml, the build might fail because the
fix added in df5bd5741b was not active.
According to the automake documentation, it should be enough to set
BUILT_SOURCES.
This pure refactoring changes the convert () functions so that the
conversion operations are listed first, followed by the
implementations of those operations.
Add a --with-distro=ID argument for configure, so it is possible to
manually specify the distro to use for the packages (in case os-release
does not provide ID=.., or the ID is not recognized yet).
In the case when --with-distro is not set, keep doing the autodetection,
but using os-release only, i.e. dropping the checks for all the other
-release files -- since there is --with-distro, older distros with no
os-release can still be used.
RWMJ: Add documentation to guestfs-building(1).
Here is an example of a failure with the previous dependency
calculation:
$ make -j5
make: Entering directory '/home/rjones/d/libguestfs/common/mlstdutils'
OCAMLOPT guestfs_config.cmx
CC libmlstdutils_a-dummy.o
OCAMLOPT libdir.cmx
OCAMLCMI stringMap.cmi
OCAMLCMI stringSet.cmi
OCAMLCMI guestfs_config.cmi
OCAMLCMI std_utils.cmi
OCAMLC guestfs_config.cmo
OCAMLC libdir.cmo
OCAMLC stringMap.cmo
OCAMLC stringSet.cmo
OCAMLOPT stringMap.cmx
OCAMLOPT stringSet.cmx
OCAMLOPT std_utils.cmx
OCAMLC std_utils.cmo
ocamlfind ocamlc -package str,unix -I . -a guestfs_config.cmo libdir.cmo stringMap.cmo stringSet.cmo std_utils.cmo -o mlstdutils.cma
ocamlfind ocamlopt -package str,unix -I . -a guestfs_config.cmx libdir.cmx stringMap.cmx stringSet.cmx std_utils.cmx -o mlstdutils.cmxa
AR libmlstdutils.a
File "_none_", line 1:
Error: Files std_utils.cmx and guestfs_config.cmx
make inconsistent assumptions over interface Guestfs_config
make: *** [Makefile:2523: mlstdutils.cmxa] Error 2
make: Leaving directory '/home/rjones/d/libguestfs/common/mlstdutils'
What seems to be happening is that there is a rule:
std_utils.cmx : guestfs_config.cmi guestfs_config.cmx [...]
In this case, make chose to build guestfs_config.cmx and
guestfs_config.cmi in parallel (see the first 5 rules above). However
building guestfs_config.cmx also creates guestfs_config.cmi
(implicitly - this is not expressed in make dependencies, and make
doesn't "know" that guestfs_config.cmi has already been created).
Unfortunately the OCaml compiler doesn't create output files
atomically. Worse than that, it creates an intermediate version of
the output file, reads it back and then creates the final version. It
seems if the build of std_utils.cmi reads this intermediate version.
In any case, Std_utils sees the wrong hash of the Guestfs_config
module.
The above only happens where we have a *.ml file without a
corresponding *.mli file. That is because if there is a *.mli file,
ocamldep generates slightly different dependencies:
guestfs_config.cmx [...] : guestfs_config.cmi guestfs_config.ml
std_utils.cmx : guestfs_config.cmi guestfs_config.cmx [...]
std_utils.cmx still depends on both files, but there is an extra rule
which ensures that guestfs_config.cmx isn't built in parallel with
guestfs_config.cmi.
I tested this change by running this command:
$ while ( rm common/mlstdutils/.depend; make -C common/mlstdutils/ clean && make -C common/mlstdutils/ ) >& /tmp/log; do echo -n .; done
Before the change it would fail after about 100 iterations. After the
change it ran for 10000s iterations and did not fail ever.
Updates commit 6d0ad49d5e.
Since commit 65cfecb0f5,
‘guestfs_int_download_to_tmp’ was buggy because it did not deal with
the case where a guest had multiple roots. It cached the downloaded
file under a single name which was not distinguished by which root we
were looking at. As a result, if you inspected (eg.) the icon on such
a guest, then in some circumstances the same icon could be returned
for all roots (incorrectly).
This changes the function in a few ways:
- Don't cache downloaded files. It wasn't a useful feature of the
function in most scenarios. Instead a new name is generated for
every call.
- Use guestfs_int_make_temp_path.
- Allow an extension to be specified.
In the function ‘guestfs_int_make_temp_path’, allow an optional
extension to be specified. For example:
r = guestfs_int_make_temp_path (g, "ls", "txt");
will create paths like ‘<tmpdir>/ls123.txt’.
NULL can also be passed for the extension, which means no extension.
Move the last remaining function ‘guestfs_int_download_to_tmp’ to
lib/inspect-icon.c (the main, but not only user of this function).
Then remove lib/inspect.c entirely.
This is not quite code motion because I updated the comment for the
function to reflect what it does in reality.
Commit 4d3601eb4e made it clear that
ocaml-hivex is required, implying also that hivex is required.
However the configure test was still optional, and if you built
libguestfs without hivex you got a very long and confusing error in
the daemon/ subdirectory.
Thanks: Cédric Bosdonnat
We permit the following constructs in libguestfs code:
if (guestfs_some_call (g) == -1) {
fprintf (stderr, "failed: error is %s\n", guestfs_last_error (g));
}
and:
guestfs_push_error_handler (g, NULL, NULL);
guestfs_some_call (g);
guestfs_pop_error_handler (g);
Neither of these would be safe if we allowed the handle to be used
from threads concurrently, since the error string or error handler
could be changed by another thread.
Solve this in approximately the same way that libvirt does: by making
the error, current error handler, and stack of error handlers use
thread-local storage (TLS).
The implementation is not entirely straightforward, mainly because
POSIX doesn't give us useful destructor behaviour, so effectively we
end up creating our own destructor using a linked list.
Note that you have to set the error handler in each thread separately,
which is an API change (eg: if you set the error handler in one
thread, then pass the handle 'g' to another thread, the error handler
in the second thread appears to have reset itself back to the default
error handler). I haven't yet worked out a better way to solve this.
Acquire the per-handle lock on entering each public API function.
The lock is released by a cleanup handler, so we only need to use the
ACQUIRE_LOCK_FOR_CURRENT_SCOPE macro at the top of each function.
Note this means we require __attribute__((cleanup)). On platforms
where this is not supported, the code will probably hang whenever a
libguestfs function is called.
The only definitive list of public APIs is found indirectly in the
generator (in generator/c.ml : globals).
Define the types which will be used to communicate between the
different parts of the inspection code. The main types are:
fs corresponds to ‘struct inspect_fs’ in C code
root no direct correspondence with the C code, but in the C
code, ‘inspect_fs’ was overloaded to store roots
inspection_data
the inspection data which is incrementally collected about
each filesystem as we perform inspection steps
Other types have simple and obvious correspondences with the
equivalent C code.
Add some utility function which will be used by inspection.
Note that this commit has no effect on its own, it just links extra
dead code into the daemon.
After we move inspection code to the daemon, the library will no
longer have access to ‘struct inspect_fs’, and so we won't be able to
prefix downloads with the "root filesystem number".
Just remove this prefix (it's internal only). However it does mean
that this function can no longer cache downloaded files.
Unlike previous ‘daemon: Reimplement ...’ patches, this does not
reimplement the umount_all API completely (yet, but this
implementation could be completed in future and then replace the C
one). However it is necessary to have a version of umount_all which
we can call from the OCaml inspection code.
hivex has a function hivex_value_string. We were not calling it under
the mistaken belief that because hivex implements this using iconv,
the function wouldn't work inside the daemon. Instead we
reimplemented the functionality in the library.
This commit deprecates hivex_value_utf8 and removes the library side
code. It replaces it with a plain wrapper around hivex_value_string.
Thanks: Pino Toscano
This commit bundles the ocaml-augeas library (upstream here:
http://git.annexia.org/?p=ocaml-augeas.git;a=summary). It's identical
to the upstream version and should remain so.
We can work towards using system ocaml-augeas, when it's more widely
available.
In the case where we have a module.ml without a corresponding
module.mli file, ocamldep generates (correct) dependencies:
module.cmx module.cmi : module.ml
but we had no rule telling make how to generate the module.cmi
file from module.ml.
This didn't matter very much because when make came to build
module.cmx, the module.cmi file is generated as a side-effect.
However for highly parallel builds, the build ordering was still
incorrect. Any other module that depends on module.cmi could be built
in parallel. You would very occasionally see errors like this one:
File "_none_", line 1:
Error: Files index.cmx and utils.cmx
make inconsistent assumptions over interface Utils
Fixing this involves adding a ‘%.cmi: %.ml’ rule. However we have to
be careful that make doesn't run this rule instead of the ‘%.cmi: %.mli’
rule (if module.mli did exist). It turns out that GNU make says we
can depend on rule ordering in the Makefile for this.
I found that this only works correctly if we use "%"-style pattern
rules (not the ‘.ml.cmi:’ old-style rules).
This is *still* not a complete fix. Make still doesn't know that the
rules ‘%.cmo: %.ml’ and ‘%.cmx: %.ml’ also build the .cmi file as a
side-effect, so you can still occasionally see build failures.
However I could not work out how to add the extra information to the
dependencies without causing make itself to go into an infinite loop.
It may be that in the end we will have to get rid of pattern rules
completely and generate the complete OCaml rule set.
Because there are no *.mli in either of these directories, the .depend
file was not being built (silently). So dependencies were wrong.
Fixes commit 692195c6ba.
We ran some sed scripts over the output of ocamldep to get them into a
single line format. This actually broke the output of ocamldep,
combining multiple dependencies into single lines. I didn't debug
exactly why our sed rules were broken because modern ocamldep has an
option (-one-line) which make this hacking unnecessary.
Another useful option (-all) is added so that all dependencies are
made explicit.
Since old ocamldep in RHEL 6 doesn't support these options, we have to
detect them at configure time.
The historical reason we were trying to combine entries into a single
line was so we could sort them. That was only needed back when we
used to commit the dependency files (so the files were stable), but we
haven't committed dependency files in a long time.
QEMU >= 2.10 started to do mandatory locking. This checks the QMP
schema to see if we are using that version of qemu. (Note it is
sometimes disabled in downstream builds, and it was also enabled in
upstream prereleases with version 2.9.9x, so we cannot just check the
version number).
Rename the cache files like ‘qemu.stat’ etc so they include the qemu
binary "key" (ie. size and mtime) in the name. This allows a single
user to use multiple qemu binaries in parallel without conflicts.
This adds an extra test using QMP (the QEMU Monitor Protocol). This
allows us to get extra information about the qemu binary beyond what
is available from the version number or ‘qemu -help’.
The previous code duplicated a lot of common code for reading and
writing the cache file per data field. This change simply factors out
that common code. This makes it simpler to add new tests in future.
This is just refactoring, it should have no effect.
Rather unnecessarily this function returned the parsed qemu version.
This complicates further refactoring, so I have changed the function
not to return this, and instead there is a separate function you have
to call to get the version struct (‘guestfs_int_qemu_version’).
Apart from a tiny amount of extra copying this is simply refactoring
of the interface between the direct-mode backend and the qemu query
functions.
Unlike ordinary guestfs_int_cmd_run, the pipe version did not print
the command it was about to run when debugging was enabled.
Fixes commit e98bb86929.
These have been supported since forever, but it's good to have an
indication in the ‘virt-v2v --machine-readable’ output that they are
available:
$ virt-v2v --machine-readable
virt-v2v
libguestfs-rewrite
vcenter-https
xen-ssh
colours-option
vdsm-compat-option
[etc]
Thanks: Pino Toscano.
virt-v2v obviously cannot convert this kind of devices, since they are
specific to the host of the hypervisor. Thus, emit a warning about the
presence of passthrough host devices, so at least this can be noticed
when converting a guest.
Avoid passing an empty UUID string to mkswap, which mkswap does not
accept (correctly) as new UUID.
In addition, print a warning when the UUID of a swap partition changed,
since it may require manual fixups in the guest.
The previously selected 'atomic' recipe resulted in 2GB swap in a 6GB
volume. Also, we don't really need the boot partition, so just create a
partition using the whole disk space.
Commit 0a94cae15b added useful
headings to parts of the configure script. This refactors
the code by adding a common macro (‘HEADING’), and also changes
the output so it's clearer at a glance:
--- Checking for Haskell ---
checking for ghc... (cached) ghc
--- Checking for PHP ---
checking for php... (cached) php
checking for phpize... (cached) phpize
Modern ssh and RHEL 5 sshd are not always interoperable, requiring
enabling legacy crypto policies on the converion server (ssh client).
Thanks: Nikos Mavrogiannopoulos, Jakub Jelen.
Shuffle the order of the various CFLAGS used when building the Ruby
extension: first the flags from manywarnings (gnulib), then libguestfs
own CFLAGS, and then -DGUESTFS_PRIVATE=1 (so it is not overridden).
This matches also what happens in automake parts, and makes it possible
to override any flags from manywarnings also in the Ruby extension.
As the name suggests, caml_stat* functions are used for allocating
static blocks. The OCaml runtime doesn't bother freeing any of these
on exit. OCaml 4.05 allocates caml_executable_name this way, a newly
seen global variable. We might as well just ignore everything
allocated this way.
Further information here:
e7cf3b1846/byterun/caml/memory.h (L66)
These are generated in many different ways in the various
subdirectories, and sometimes not generated correctly. Introduce a
script to do this in one place, and hopefully correctly.
This is mostly simple refactoring, but I got rid of a couple of
things:
(1) The ‘make depend’ rule doesn't appear to be needed. automake (or
make?) seems to rebuild the ‘.depend’ file automatically just because
it is included.
(2) I got rid of the hairy path rewriting sed expression. Possibly
that is needed for srcdir != builddir.
Output each path where we attempt to find 'java', so it is easier to
spot whether the location for the current OS is missing. This also
removes the need to print the location next to the version, since the
location was already printed above.
All sorts of strings might be passed here hoping to make them
canonical LV names. We cannot be sure that the strings passed will be
devices which exist in the appliance.
For Device parameters we expect a block device name. However we were
only testing for "/dev/..." and so chardevs (from the appliance) could
be passed here, resulting in strange effects. This adds a function
is_device_parameter which tests for a valid block device name.
For Dev_or_Path parameters much the same, except we can also use the
is_device_parameter function elsewhere in the daemon to distinguish if
we were called with a device or path parameter. Previously we used a
simple test if the path begins with "/dev/...".
Reported by Mathieu Tarral.
The following functions were previously reimplemented in OCaml. This
commit replaces them with calls to the C functions:
- is_root_device
- prog_exists
- udev_settle
plus the internal get_verbose_flag function.
However note that we cannot do this for every utility function. In
particular the C function must not call any reply* functions.
When parts of the daemon were previously converted to OCaml, the
previous PCRE regexps were converted to Str regexps. Restore the
original PCRE regexps.
There was also one case where an original call to glob(3) was replaced
by a Str regexp, and this is replaced by a PCRE regexp (although it is
in fact identical in this instance).
This updates commit b48da89dd6
and commit eeda6edca1
and commit 2ca0fa778d.
oraclelinux is the same as redhat/centos/scientificlinux, so add it
where it is missing. This fixes amongst other things, running sysprep
on an Oracle Linux image where it would previously fail operations like
setting the hostname, saying that it was not supported on that distro.
Since the introduction of the TEST_FUNCTIONS framework in tests, with
commit 2cb0413049, using a non-bash shell
will not work anymore, since the framework currently relies on bash.
Hence, switch the interpreter of these tests to bash, so they can run
also on systems where /bin/sh is not bash (e.g. default Ubuntu
installations).
Since commit d5b6f1df5f, the daemon
requires ocaml-hivex; OTOH, nothing checks it is actually available, so
the build of the daemon will fail with a semi-cryptic error about the
lack of a directory in the OCaml install prefix.
As fix, check for the presence of the hivex module at build time,
failing earlier if not present. The check is performed only when the
daemon is enabled, as ocaml-hivex is not used for anything else than the
daemon.
This also reimplements the lv_canonical function in OCaml. We cannot
call the original C function because it calls reply_with_perror which
would break the OCaml bindings.
Move the list_filesystems API into the daemon, reimplementing it in
OCaml. Since this API makes many other API calls, it runs a lot
faster in the daemon.
The previously library-side ‘file_architecture’ API is reimplemented
in the daemon, in OCaml.
There are some significant differences compared to the C
implementation:
- The C code used libmagic. That is replaced by calling the ‘file’
command (because that is simpler than using the library).
- The C code had extra cases to deal with compressed files. This is
not necessary since the ‘file’ command supports the ‘-z’ option
which transparently looks inside compressed content (this is a
consequence of the change above).
This commit demonstrates a number of techniques which will be useful
for moving inspection code to the daemon:
- Moving an API from the C library to the OCaml daemon.
- Calling from one OCaml API inside the daemon to another (from
‘Filearch.file_architecture’ to ‘File.file’). This can be done and
is done with C daemon APIs but correct reply_with_error handling is
more difficult in C.
- Use of Str for regular expression matching within the appliance.
This change allows parts of the daemon to be written in the OCaml
programming language. I am using the ‘Main Program in C’ method along
with ‘-output-obj’ to create an object file from the OCaml code /
runtime, as described here:
https://caml.inria.fr/pub/docs/manual-ocaml/intfc.html
Furthermore, change the generator to allow individual APIs to be
implemented in OCaml. This is picked by setting:
impl = OCaml <ocaml_function>;
The generator creates ‘do_function’ (the same one you would have to
write by hand in C), with the function calling the named
‘ocaml_function’ and dealing with marshalling/unmarshalling the OCaml
parameters.
LVM is fine with a completely empty configuration file (meaning "all
defaults"), so start with one instead of copying the system
configuration file.
Also this means we can very easily implement lvm_set_filter
functionality without using Augeas, since we no longer have to worry
about existing filters being present.
Thanks: Alasdair Kergon, Zdenek Kabelac.
In SUSE guests, handle the case where
Bootloader::Tools::GetDefaultSection () returns undef.
Previously this would return an empty string and cause a bogus error
in subsequent code:
virt-v2v: error: libguestfs error: statns: statns_stub: path must start
with a / character
With OCaml < 4.02 when using the alternate Bytes module, this module
would be compiled twice during parallel builds, resulting in
occasional corruption. The reason for this is that the ocamldep file
mentions ‘bytes.cmo’ whereas the ‘$(OCAML_BYTES_COMPAT_ML)’ macro
expands to ‘../../common/mlstdutils/bytes.ml’. Make doesn't recognize
these as the same file.
Use an alternate way to specify this file to fix this.
GUESTFSD_EXT_CMD was used by OpenSUSE to track which external commands
are run by the daemon and package those commands into the appliance.
It is no longer used by recent SUSE builds, so remove it.
Thanks: Pino Toscano, Olaf Hering.
Even if ocamlopt is available, always build a bytecode version of
‘common/mlstdutils’.
Furthermore, because this library is pure OCaml, we should not be
using ‘ocamlmklib’. We should use ‘ocaml{c,opt} -a’ instead. This
doesn't make any difference for native code, but for bytecode it was
building a broken library.
The original reason for making this change is because the generator is
always built as bytecode, and it depended on
‘../common/mlstdutils/guestfs_config.cmo’ and
‘../common/mlstdutils/std_utils.cmo’. On native code platforms these
were not built before the generator and so the generator races to
build the .cmi and .cmo files. Since the generator doesn't have
correct dependencies covering the ‘common/mlstdutils’ directory you
can get a broken link on fast machines:
File "../common/mlstdutils/std_utils.ml", line 1:
Error: Corrupted compiled interface
../common/mlstdutils/guestfs_config.cmi
make[2]: *** [Makefile:1993: ../common/mlstdutils/std_utils.cmo] Error 2
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/builddir/build/BUILD/libguestfs-1.37.17/generator'
The C function mkdtemp(3) requires that the string ends with 6 'X'
characters, so appending a non-empty suffix causes the function to
raise EINVAL.
Luckily we only ever called this function with the last parameter "".
This is like the Perl chomp function, it removes a single \n from the
end of a string if present, else leaves the string alone.
I believe I found the only (two) places where such a function is used,
but there may be a few more lurking.
Require <caml/unixsupport.h> (an OCaml header file) and remove
alternate defintions of ‘Nothing’ and ‘unix_error’ which are defined
in this header file.
We require OCaml >= 3.11 which has this header file, so there is no
need to test for it or provide alternative definitions.
Thanks: Pino Toscano.
Previously the OCaml compiler was only required if building from git
but was at least theoretically optional if building from tarballs
(although this was never tested). Since we want to write parts of the
daemon in OCaml, this makes OCaml required for all builds.
Note that the ‘--disable-ocaml’ option remains, but it now only
disables OCaml bindings and OCaml virt tools. Using this option does
not disable the OCaml compiler requirement.
Also note that ‘HAVE_OCAML’ changes meaning slightly, so it now means
"build OCaml bindings and tools" (analogous to ‘HAVE_PERL’ and
others). The generator, daemon [in a future commit], and some utility
libraries needed by the generator or daemon do not test for this macro
because we can assume OCaml compiler availability.
The ‘vmware-uninstall-tools.pl’ script tries to rebuild the initrd.
On SUSE, if kdump initrd has been enabled, this would use the wrong
root device because ‘mkdumprd’ doesn't know what root device to use.
Fix that by setting the ‘rootdev’ environment variable.
See also https://bugzilla.redhat.com/1465849 (this is not a fix).
Currently we install the firstboot service under systemd target
‘default.target’. This change simply factors out this name.
Note that the name is not factored out in the code which deletes the
old ‘/etc/systemd/system/default.target.wants/firstboot.service’ file,
since that would have always been installed in the same location.
Or with LIBVIRT_DEFAULT_URI=qemu:///system which is the same
thing.
In either case the images are created as user qemu.qemu and then
aren't readable or modifiable by later parts of the script.
Place the Bytes fallback module in the right place (mlstdutils), with no
need to make it available directly also for generation, since it uses
mlstdutils now.
Fixes commit 61d4891ef4.
The gobject bindings are adequately covered in the usual manual pages:
guestfs(3). There is no need for separate generation of gtk-doc.
Also generating gtk documentation is the slowest part of the build,
and the tooling around gtk-doc is broken
(https://bugzilla.redhat.com/show_bug.cgi?id=1465665).
Note this removes the configure ‘--enable-gtk-doc’ option. Using this
option now gives a warning, but is otherwise ignored:
configure: WARNING: unrecognized options: --enable-gtk-doc
After the previous refactoring, we are able to link the daemon to
common/utils, and also remove some of the "duplicate" functions that
the daemon carried ("duplicate" in quotes because they were often not
exact duplicates).
Also this removes the duplicate reimplementation of (most) cleanup
functions in the daemon, since those are provided by libutils now.
It also allows us in future (but not in this commit) to move utility
functions from the daemon into libutils.
Create a module ‘C_utils’ containing functions like ‘drive_name’ and
‘shell_unquote’ which come from the C utilities.
The new directory ‘common/mlutils’ also contains the ‘Unix_utils’
wrappers around POSIX functions missing from the OCaml stdlib.
This refactoring change just moves the cleanup functions around in the
common/utils directory.
libxml2 cleanups are moved to a separate object file, so that we can
still link to libutils even if the main program is not using libxml2
anywhere. Similarly gnulib cleanups.
cleanup.c is renamed to cleanups.c.
A new header file cleanups.h is introduced which will replace
guestfs-internal-frontend-cleanups.h (fully replaced in a later commit).
The new module ‘Std_utils’ contains only functions which are pure
OCaml and depend only on the OCaml stdlib. Therefore these functions
may be used by the generator.
The new module is moved to ‘common/mlstdutils’.
This also removes the "<stdlib>" hack, and the code which copied the
library around.
Also ‘Guestfs_config’, ‘Libdir’ and ‘StringMap’ modules are moved
since these are essentially the same.
The bulk of this change is just updating files which use
‘open Common_utils’ to add ‘open Std_utils’ where necessary.
The ‘Xml’ module is a self-contained library of bindings for libxml2,
with no other dependencies.
Move it to a separate ‘common/mlxml’ directory.
This is not pure refactoring. For unclear reasons, the previous
version of ‘Xml.parse_file’ read the whole file into memory and then
called ‘xmlReadMemory’. This was quite inefficient, and unnecessary
because we could use ‘xmlReadFile’ to read and parse the file
efficiently. Changing the code to use ‘xmlReadFile’ also removes the
unnecessary dependency on ‘Common_utils.read_whole_file’.
The ‘Progress’ module is a self-contained library with the only
dependencies being:
- the C ‘progress’ implementation
Move it to a separate ‘common/mlprogress’ directory.
This change is pure code refactoring.
The ‘Visit’ module is a self-contained library with the only
dependencies being:
- the C ‘visit’ implementation
- the guestfs OCaml bindings
Move it to a separate ‘common/mlvisit’ directory.
This change is not entirely refactoring. Two other fixes are made:
- remove unsafe use of CLEANUP_FREE from a function which could
raise an OCaml exception (cleanup handlers would not be called
correctly if the exception is thrown)
- don't link directly to common/visit/visit.c, but instead use
the library (common/visit/libvisit.la)
When installing guests that need NVRAM variables, the cleanup of the
guest with `virsh undefine` will remove that file too, which is not what
we want. Instead, compress the NVRAM file right before the cleanup,
to ensure we have it.
Also, fix the filename for it, removing the double "-nvram" suffixes.
- set the mirror to deb.debian.org, which is the official redirector, so
generated images will use the closest mirror depending on their
location
- automake the task selection, using the choice we want
- do a full upgrade of the distro, so the template is already up-to-date
- install grub on the default device, which should also be the only one
in the automated installation
We were dropping the add_drive copyonread flag when using the libvirt
backend. This resulted in significant performance degradation (2x-3x
slower) when running virt-v2v against VMware servers.
Thanks: Kun Wei.
Currently -i libvirtxml mode only works for local files or NBD disks.
The purpose of NBD is to support virt-p2v.
This change adds support for network disks over http or https, ie:
<disk type='network' device='disk'>
<driver name='qemu' type='raw'/>
<source protocol="http" name="/scratch/disk.img">
<host name="server.example.com" port="80"/>
</source>
<target dev='hda' bus='ide'/>
</disk>
This is just for testing. It's especially useful for exercising curl
support in qemu without requiring VMware to be available.
This feature allows you to use different image formats for the fixed
appliance. The raw format is used by default.
Signed-off-by: Pavel Butsykin <pbutsykin@virtuozzo.com>
Caused the following error when running make-template:
syntax error at line 22: syntax error
virt-index-validate: ‘debian-9.index-fragment’ could not be validated, see errors above
Add a RELEASES file which contains the release date for each version
of libguestfs. When a new version is released, the configure script
checks that the date has not been omitted from the file.
This also fixes the release date for version 1.37.16 which I omitted
from commit 9455f21237.
New gnulib uses ‘-Wimplicit-fallthrough=5’ which warns (and hence
errors) on every switch statement with an implicit fallthrough.
We could use the new GCC attribute here:
#ifndef FALLTHROUGH
# if __GNUC__ < 7
# define FALLTHROUGH ((void) 0)
# else
# define FALLTHROUGH __attribute__ ((__fallthrough__))
# endif
#endif
but instead this changes the level to 4 which allows us to use
/*FALLTHROUGH*/ comments (which we are already using) to mark cases
where we expect fallthrough.
Gnulib installs two files: ‘GNUmakefile’ and ‘maint.mk’. The first
one runs the second one, and the second one sets LC_ALL=C universally.
However this breaks the ‘bugzilla’ script (or Python) because some
bugs contain non-ASCII characters in their Summary. The script prints
this error:
Traceback (most recent call last):
File "/usr/bin/bugzilla", line 1117, in <module>
main()
File "/usr/bin/bugzilla", line 1112, in main
_format_output(bz, opt, buglist)
File "/usr/bin/bugzilla", line 702, in _format_output
print(format_field_re.sub(bug_field, opt.outputformat))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 108-109: ordinal not in range(128)
The simplest fix for this is to unset LC_ALL before running the
external script, so that the normal locale settings are used.
The OCaml function ‘input_line’ throws an End_of_file exception if the
end of the file is read before any other input, ie. if the file here
is zero length. Return an empty string instead.
This can produce peculiar errors such as:
$ virt-customize ... --root-password file:/dev/null
libguestfs: uncaught OCaml exception in getopt callback: End_of_file
Remove (before opening round bracket) whitespaces in the documentation
of the Python binding, according to the PEP 8 specification.
This is just code reformatting, with no behaviour changes; no content
changed beside whitespaces, so "git diff -w" gives an empty diff.
Mostly modelled after a snippet implemented in dib, it is an helper
function to run multiple commands in parallel, waiting for all of them
at once, and returning all their exit codes. It is possible to pass
custom descriptors for collecting stdout and stderr of each command.
Common_utils.run_command is adapted to use few helper methods used by
run_commands, so all the existing code using it keeps working; in
addition, it gets labelled parameters for stdout and stderr FDs.
Add a simple unit tests for them.
LVM2 >= 2.02.171 requires the ‘--yes’ option to force pvresize to work
in various circumstances, eg. reducing the size of an existing PV.
Pass this flag unconditionally.
Note this does NOT break earlier versions which just ignore this flag.
If SIGTERM is blocked in the main program, then it ends up still being
blocked in the subprocess after we fork. This means that we cannot
kill qemu by sending SIGTERM to it. This commit fixes the problem by
unblocking SIGTERM unconditionally after fork.
Thanks: wtfuzz on IRC for reporting and analysis.
If you use the libguestfs tools which open disk images read-only
(eg. virt-df), with formats such as 'vdi', then you will see an error:
error: invalid value for backingformat parameter 'vdi'
This is because opening a disk image read-only will try to create a
qcow2 file with the original image as a backing file. However the
list of permitted backing formats was very restrictive and did not
include 'vdi' (nor many other uncommon formats).
Instead of using a whitelist for backing formats, just validate that
the string is alphanumeric and short.
Thanks: Mike Goodwin for reporting the bug.
Complementary fix of commit 2d25872df3.
On SLES 11 SP4 with kdump enabled mkinitrd calls mkdumprd which calls
mkinitrd, but mkdumprd doesn't have any clue of the root device.
Call mkinitrd with rootdev environment variable to tell them all
what device to use as root.
Tested-By: Cédric Bosdonnat <cbosdonnat@suse.com>
These were already included in the .ml implementation file, but just
weren't being exported in the interface, so there's no real change
here. It just allows these functions to be used.
Make use of the additional command line arguments, and API needed to
decrypt LUKS partitions. This extends to v2v the work done in other
OCaml tools with commit 6b26a0cce4,
since it seems to be working fine after a basic testing.
Related to: RHBZ#1362649
PCI devices don't exist/work. You would see errors such as:
qemu-system-s390x: -device virtio-rng-pci,rng=rng0: MSI-X support is mandatory in the S390 architecture
mkinitrd in SLE guests < 12 tries to get the root device by scanning the
fstab: this will fail, since v2v already remapped the devices from
hd*/sd* to vd* in the guest (including in its fstab).
Since we know what is the root device in the appliance, pass it to
mkinitrd directly, so it does not have to do guesswork.
Thanks to: Cédric Bosdonnat, for reporting the issue, and testing
the fix.
Verify the returned values of Python Object constructor functions
are not NULL before adding them to a collection.
This is particularly relevant when constructing Unicode strings in
Python 3 as they will return NULL if non UTF-8 characters are present.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
It looks like older versions of lvm2 (recent enough to have selectors)
do not recognize '1' (and '0') as boolean values. Switch to 'yes',
which seems to be supported.
Updates commit 7367945647.
Make use of functions and types that fit more, and that do the same job:
- use PyErr_NoMemory() on malloc failure
- use PyErr_SetFromErrno when setting an exception from an errno
- throw TypeError if not getting a list when required
The current need for #ifdef's based on the presence of
PyString_FromString makes both the OCaml code of the generator, and the
generated C code a mess to read.
Hence, add three simple wrappers to make both the OCaml, and C code more
readable, and easier to tweak in the future.
This should be just refactoring, with no actual behaviour changes.
Thanks to: Matteo Cafasso
If the parallel tools pigz or pxz are available, prefer them for
uncompressing gz- and xz-compressed OVA files respectively. If not
available then gzip or xz are used as normal.
In its current form this is very hard to implement because it requires
us to "unparse" the options, including removing any shell quoting.
It wasn't implemented at all for the libvirt backend.
Also contrary to the documentation, the configure script did not use
these options for testing, but constructed its own set of qemu test
options.
Previously the generator did not change any string returned from the
daemon. Thus guestfs_list_devices (for example) might return internal
device names like /dev/vda (if virtio-blk was in use).
This changes calls to the daemon so that returned strings are
annotated as plain strings, devices or mountables:
old ---> new
RString "uuid" RString (RPlainString "uuid")
RString "device" RString (RDevice "device")
RString "fs" RString (RMountable "fs")
For hash tables, keys and values must be annotated separately. For
example a hash table of mountables (keys) -> plain strings (values)
would be annotated like this:
old ---> new
RHashtable "fses" RHashtable (RMountable, RPlainString, "fses")
The daemon calls reverse_device_name_translation (currently a no-op)
for devices and mountables.
Note that this has no effect for calls which are handled on the
library side.
(cherry picked from commit 6b77cc196ecb8d7e1d73592ef65a189a7412c97c)
Remove much of the text detailing how device name translation
happened. Since we removed support for virtio-blk
(commit 9e0294f88f) and deprecated the
‘iface’ parameter, only /dev/sdX device names should be visible
through the public APIs, both in parameters and in return values from
calls like guestfs_list_devices and guestfs_list_partitions.
Note the above is in fact not true for the UML backend, but UML is
broken in the kernel and in any case this will be fixed later.
(cherry picked from commit 2727e589db216bf0731385966889a4f66dbfe225)
In particular the virt-rescue --scratch option makes it very easy to
add huge numbers of drives. Since the per-backend max_disks limit was
never checked anywhere you could get peculiar failures. Now you'll
get a clear error message:
$ virt-rescue --scratch=256
libguestfs: error: too many drives have been added, the current backend only supports 255 drives
This patch changes appliance search using paths with multiple directories. Now
all appliance checks will be done separately for each directory. For example
if the path LIBGUESTFS_PATH=/a:/b:/c, then all applainces are searched first in
/a, then in /b and then in /c. It allows to flexibly configure the libguestfs
to interact with different appliances.
Signed-off-by: Pavel Butsykin <pbutsykin@virtuozzo.com>
It is not something standard, and actually unused by libguestfs itself.
Possibly going to be used by the old virt-v2v (which was in Perl), but
never used for that either.
The device name is only used by guestfish (when using the -N option to
prepare drives). We constructed the device name very naively,
basically ‘sprintf ("/dev/sd%c", next_drive)’.
This stores the device index instead, and only constructs the device
name in guestfish. Also the device name is constructed properly using
guestfs_int_drive_name so it can cope with #drives > 26.
As a side effect of this change we can remove the extra parameter of
the add_drives macro.
Thanks: Pino Toscano
After the previous commit it become possible to construct various
"impossible" argument types, such as lists of FileIn strings. This
commit prevents these from happening.
Previously we had lots of types like String, Device, StringList,
DeviceList, etc. where Device was just a String with magical
properties (but only inside the daemon), and DeviceList was just a
list of Device strings.
Replace these with some simple top-level types:
String
StringList
and move the magic into a subtype.
The change is mechanical, for example:
old ---> new
FileIn "filename" String (FileIn, "filename")
DeviceList "devices" StringList (Device, "devices")
Handling BufferIn is sufficiently different from a plain String
throughout all the bindings that it still uses a top-level type.
(Compare with FileIn/FileOut where the only difference is in the
protocol, but the bindings can uniformly treat it as a plain String.)
There is no semantic change, and the generated files are identical
except for a minor change in the (deprecated) Perl
%guestfs_introspection table.
This patch improves the search of grub config on EFI partition. This
means that the config will be found not only for rhel but also for
many other distributions. Tests were performed on the following
distributions: centos, fedora, ubuntu, suse. In all cases, the config
path was /boot/efi/EFI/*distname*/grub.cfg
The main purpose of the patch is to improve support for converting of
vm with UEFI for most distributions. Unfortunately this patch does not
solve the problem for all distributions, for example Debian does not
store grub config on the EFI partition, therefore for such
distributions another solution is necessary.
Signed-off-by: Pavel Butsykin <pbutsykin@virtuozzo.com>
The yara_scan API parses the file generated by the daemon counterpart
function and returns the list of yara_detection structs to the user.
It writes the daemon's command output on a temporary file and parses it,
deserialising the XDR formatted yara_detection structs.
It returns to the caller the list of yara_detection structs generated by
the internal_yara_scan command.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
The internal_yara_scan runs the Yara engine with the previously loaded
rules against the given file.
For each rule matching against the scanned file, a struct containing
the file name and the rule identifier is returned.
The gathered list of yara_detection structs is serialised into XDR format
and written to a file.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
The yara_destroy API allows to claim resources back via the removal of
the previously loaded Yara rules.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
The yara_load API allows to load a set of Yara rules contained within a
file on the host.
Rules can be in binary format, as when compiled with yarac command, or
in source code format. In the latter case, the rules will be first
compiled and then loaded.
Subsequent calls of the yara_load API will result in the discard of the
previously loaded rules.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
At least two control files (packages and base.tar.gz) are necessary for the
supermin appliance.
Signed-off-by: Pavel Butsykin <pbutsykin@virtuozzo.com>
Many of the properties for disks are the same for all of them, so
collect them only once, instead of doing that for every disk.
Should be just code motion, with no behaviour change.
When converting a guest with UEFI firmware, set the also
hw_firmware_type=uefi property for all the disks of the guest, so Nova
can properly boot the guest.
In every instance where we used the ‘cancel_stmt’ parameter of
these macros:
ABS_PATH
NEED_ROOT
the value was only ever ‘cancel_receive ()’ or empty. We only use
‘cancel_receive’ for FileIn functions, so replace it with a simple
flag for whether the current function is a FileIn function.
This also removes some incorrect comments about macros that cannot be
used with FileIn functions when in fact they can.
As a simple consequence of the previous commit, every instance of the
‘fail_stmt’ parameter to one of the following macros:
RESOLVE_DEVICE
RESOLVE_MOUNTABLE
REQUIRE_ROOT_OR_RESOLVE_DEVICE
REQUIRE_ROOT_OR_RESOLVE_MOUNTABLE
is now ‘return’ and therefore we can replace it in the macro and drop
the parameter completely.
The generated code had:
...
done_no_free:
return;
}
There was no possible code between the label and the return statement,
so both the label and the return statement are redundant. The
instances of ‘goto done_no_free’ can simply be replaced by ‘return’.
The only small complication is that there is a label just before this
which contains some optional code. So we might have ended up with
generated code looking like:
done:
// if there was no optional code, this would be empty
}
which provokes a parse error in C. Therefore I had to add a semicolon
after the ‘done:’ label. This will be removed in a subsequent commit,
so it's just to make the current commit bisectable.
Replace the monolithic 'test-max-disks.pl' script with a test program
written in C. The program is completely equivalent to the old script,
except for the enhancement that it is able to detect if disks are
added to the appliance in the wrong order.
The tests themselves are split out into some shell scripts:
- test-27-disks: Fully tests 27 disks.
This is the minimum supported configuration for all backends.
- test-255-disks: Fully tests 255 disks.
This is the most disks that libguestfs up to 1.36 supported.
- test-add-lots-of-disks: Add ‘a lot’ of disks and exit (without
any further testing). This is meant to try to exercise > 255
disk case but without being as slow as a test of the max number
of disks (which takes ages, even for as few as 4000 disks).
- test-max-disks: Test the maximum possible number of disks.
The maximum depends on several factors, notably which backend
is in use, and the limit on the number of open files.
‘test-max-disks’ is a slow test.
swap.c: In function 'do_mkswap_U':
swap.c:62:9: error: argument 1 null where non-null expected [-Werror=nonnull]
if (strlen (label) > SWAP_LABEL_MAX) {
^~~~~~~~~~~~~~
virtio-scsi has been supported in qemu since 2012, and it is superior
in every respect to virtio-blk. There's no reason to still be using
virtio-blk.
virtio-scsi support was initially added in 2012
(commit 0c0a7d0d86).
You can still use virtio-blk using the (deprecated) iface parameter,
but don't do that in new code.
Add udev_settle_file() to run 'udevadm settle' with --exit-if-exists option. It
will slightly reduce the waiting-time for pending events if we need to wait
for events related to a particular device/file.
Signed-off-by: Pavel Butsykin <pbutsykin@virtuozzo.com>
RWMJ:
- Use local variable for MAX_ARGS.
- Use commandv instead of commandrv, fix checking of return code.
When running under valgrind, the process takes a fraction of a second
to start up, changing the "Opening the guest" timestamp, which broke
the test.
Fixes commit ca40078cdd.
- convert the Fedora 20 examples to Fedora 25
- convert the yum examples to dnf
- convert the Debian 7 examples to Debian 8
- mention that --core updates all the installed packages, not "core"
ones
- convert i386 to i686 for the Fedora 32bit examples
- mention that autogenerated root passwords are printed on stdout
- mention --get-kernel is deprecated in favour of virt-get-kernel
- update URL to the Puppet installation documentation
- edit the repository index documentation to make it clearer
non-compressed templates are supported
- remove old wrong paragraph about customization specified in command
line not processed in the same order
- use --install instead of --run-command '$pkgmgr install ...'
- other minor changes
This is a mostly complete implementation of a VMX parser and input
class for virt-v2v. It parses the name, memory size, CPU topology,
firmware, video, sound, hard disks, removable disks and network
interfaces from the VMX file. It only omits support for floppies and
SCSI CD-ROMs.
The input class is split into two major parts: a generic VMX file
parser (Parse_vmx), and the Input_vmx module which translates the VMX
tree into the source device model.
This also contains tests. There are simple unit tests of the
Parse_vmx module, and also some more complete parsing tests taken from
real guests.
When outputting libvirt XML, create virtio-rng, a memory balloon
device, and a pvpanic device, if the guest supports it.
Thanks: Daniel Berrangé for advice on memballoon
Extend the guestcaps structure so it records whether a guest supports
(or drivers were added) for virtio-rng, the virtio memory balloon, and
the ISA pvpanic device.
Previously the kernel_info field 'ki_supports_virtio' really meant
that the kernel supports virtio-net. That was used as a proxy to mean
the kernel supports virtio in general.
This change splits the field so we explicitly test for both virtio-blk
and virtio-net drivers, and store the results as separate fields.
The patch is straightforward, except for the change to the
'rebuild_initrd' function. Instead of making the module list
conditional on whether virtio-net is available and using the old
(probably wrong) fallback if it happens to be unavailable, this now
tries to enable the common virtio kernel modules (just the ones needed
for virtio-blk and virtio-net to work). The fallback is only used if
none of the common virtio modules can be found.
This also adds a print_kernel_info function (as the
string_of_kernel_info function was unwieldy) and exports it from the
module.
We want to detect if a Linux kernel is Xen PV only ("PV-only"). Such
a kernel will not boot on KVM, and if a guest has only PV-only
kernels, it will not be able to boot at all on the target.
Our previous test was wrong. It tested whether the xennet.ko module
exists. This module was renamed in more recent kernels (to
xen-netfront.ko), so it happened to not detect modern kernels as
PV-only, but this was by chance.
Modern kernel images can be compiled with Xen PV guest support. The
same image can boot on baremetal, KVM, Xen PV or Xen HVM. Testing if
the xennet (or xen-netfront) module exists is irrelevant to this.
This test, which is based on ideas from Laszlo Ersek and
https://wiki.xen.org/wiki/Xen_Project_Software_Overview#Guest_Types
uses the kernel config test CONFIG_X86_XEN || CONFIG_X86_64_XEN to
determine PV-only kernels.
Note that these CONFIG flags were never upstream, and existed only in
Linux kernels forked by Xen ("XenLinux"). By the time Xen guest
support was added upstream, it was implemented using pvops support, so
a single image could boot on Xen and non-Xen as described above, and
these flags were no longer used.
Updates commit 7eb219d193.
Thanks: Laszlo Ersek.
Only in end-user messages and documentation. This change was done
mostly mechanically using the Perl script attached below.
I also changed don't -> don’t etc and made some other simple fixes.
See also: https://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html
----------
#!/usr/bin/perl -w
use strict;
use Locale::PO;
my $re = qr{'([-\w%.,=?*/]+)'};
my %files = ();
foreach my $filename ("po/libguestfs.pot", "po-docs/libguestfs-docs.pot") {
my $poref = Locale::PO->load_file_asarray($filename);
foreach my $po (@$poref) {
if ($po->msgid =~ $re) {
my @refs = split /\s+/, $po->reference;
foreach my $ref (@refs) {
my ($file, $lineno) = split /:/, $ref, 2;
$file =~ s{^\.\./}{};
if (exists $files{$file}) {
push @{$files{$file}}, $lineno;
} else {
$files{$file} = [$lineno];
}
}
}
}
}
foreach my $file (sort keys %files) {
unless (-w $file) {
warn "warning: $file is probably generated\n"; # have to edit generator
next;
}
my @lines = sort { $a <=> $b } @{$files{$file}};
#print "editing $file at lines ", join (", ", @lines), " ...\n";
open FILE, "<$file" or die "$file: $!";
my @all = ();
push @all, $_ while <FILE>;
close FILE;
my $ext = $file;
$ext =~ s/^.*\.//;
foreach (@lines) {
# Don't mess with verbatim sections in POD files.
next if $ext eq "pod" && $all[$_-1] =~ m/^ /;
unless ($all[$_-1] =~ $re) {
# this can happen for multi-line strings, have to edit it
# by hand
warn "warning: $file:$_ does not contain expected content\n";
next;
}
$all[$_-1] =~ s/$re/‘$1’/g;
}
rename "$file", "$file.bak";
open FILE, ">$file" or die "$file: $!";
print FILE $_ for @all;
close FILE;
my $mode = (stat ("$file.bak"))[2];
chmod ($mode & 0777, "$file");
}
This patch fixes the problem of the first boot after installation virtio block
drivers on Windows 10 / Server 2016. The problem is related to the change in
the behavior of Windows-Kernel-Pnp. To fix we need to add a field "Version" to
HKLM/SYSTEM/DriverDatabase/DriverPackages/drv_inf_label then Windows-Kernel-Pnp
will be able to correctly install the virtio block driver.
Signed-off-by: Pavel Butsykin <pbutsykin@virtuozzo.com>
Otherwise the man page cannot contain any non-7-bit-ASCII characters.
The test error seen was:
Failed test 'POD test for blib/lib/Sys/Guestfs.pm'
at /usr/share/perl5/vendor_perl/Test/Pod.pm line 187.
Wide character in print at /usr/share/perl5/vendor_perl/Test2/Formatter/TAP.pm line 112.
blib/lib/Sys/Guestfs.pm (1811): Non-ASCII character seen before =encoding in 'caller’s'. Assuming UTF-8
Looks like you failed 1 test of 1.
Each scp command takes a considerable amount of time -- several
seconds -- because it must set up, authenticate and tear down a new
connection. Avoid this by combining several copies into a single
command.
We still have to use two scp commands because we want to check that
some files are copied and ignore failures in a second set of
informational files.
Ensure the full version of virt-v2v and virt-p2v is available in the
conversion log file, the only log file that we reliably get from users
and customers. "Full version" means the major, minor, release and
extra fields. The extra field is especially important as it contains
the downstream package release in Fedora, RHEL etc.
This change saves the virt-p2v version as a comment in the
physical.xml file, which is included in full in the conversion log (by
virt-v2v).
It also ensures that the initial virt-v2v debug message contains the
full version number including the 'extra' field.
$ cat virt-v2v-conversion-log.txt
virt-v2v: libguestfs 1.37.7local,libvirt (x86_64)
...
<!-- virt-p2v 1.37.7local,libvirt -->
It also adds 'p2v-version' and 'v2v-version' files in the virt-p2v
debug directory. These are strictly superfluous but could be useful
for end users.
$ cat p2v-version
virt-p2v 1.37.7local,libvirt
$ cat v2v-version
virt-v2v 1.37.7local,libvirt
GtkAboutDialog in GTK+ 3 can be configured with few standard licenses,
including GPL2+. Thus, just set that property, so the about dialog will
show its own license text, and there is no need for our custom one.
If you switch from ./configure --with-gtk=2 to =3 (or vice versa) the
dependencies files don't get rebuilt. As a result the wrong Gtk
libraries can be included in the ISO and virt-p2v won't run.
This make sure there is no mismatch between the size of the integer
value that Int64_val returns, and the size of the guestfs_h pointer.
This should fix the warning on 32bit environments (and thus build, when
--enable-werror is enabled).
Don't get the CPU information from libvirt, because including libvirt
and all dependencies in the virt-p2v ISO bloats everything.
Instead get most of the information we need from the util-linux
program 'lscpu'.
Unfortunately the CPU model cannot be retrieved.
Example output:
$ ./run virt-p2v --cmdline="p2v.dump_config_and_exit"
[...]
cpu vendor . . . Intel
cpu sockets . . 2
cpu cores . . . 8
cpu threads . . 1
flags . . . . . acpi apic pae
This updates commit 963d6c3be7.
Calculate the offset of the physical host's Real Time Clock (RTC) from
UTC and pass this to virt-v2v through the libvirt XML description of
the physical machine.
The libvirt XML is modified to add one of the following:
(no <clock/> element)
- if the RTC could not be read or there was some other time
calculation error.
<clock offset='utc' />
- if the RTC is the same as UTC.
<clock offset='localtime' />
- if the RTC is in local time.
It's not possible to distinguish between UTC and localtime in
timezones that lie along the Greenwich Meridian (obviously
including the UK), when daylight savings time is not in effect. In
that case, UTC is chosen.
Necessarily in timezones that use DST, this depends on when
virt-p2v is run, so in the unusual case where a physical machine is
switched off for a long time and then booted directly into virt-p2v
when DST has changed, the calculation can be wrong.
<clock offset='variable' basis='utc' adjustment='<seconds>' />
- if the RTC is some other offset to UTC, other than equal to UTC or
localtime.
Check for a "product.id" file in an architecture-specific subdirectory
of the main partition, and use its data to improve the data on the
media.
Only Mageia as distribution name is recognized there, since most
probably this file will not be available on other distros.
Export a new environment variable for extra-data.d scripts, that
contains a YAML representation of the elements in use.
This is a new addition in d-i-b 2.0.
d-i-b 2.0 is written in Python, and thus it passes the information about
the interpreter used for it (sys.executable) to the scripts that need
it; this mechanism replaces the old discovery of what is the default
Python interpreter in the system.
Since we are not Python-based, look for 'python' and use it as default
interpreter, with the --python command line option to set a different
one.
The only place where it is used so far are extra-data.d scripts; in case
it will be used also in other out-of-chroot phases, a different solution
will be needed.
Output the get_image_element_array function like d-i-b does, together
with the eval stuff needed to make it work.
get_image_element_array is basically a bash array, with the keys are the
used elements, and the values are the paths of each element; it is a new
addition in d-i-b 2.0.
This is used so far only for the extra-data.d phase.
Change mnemonics so 's' is not used twice:
Start Conversion: 's' -> 'c'
Output Connection: 'c' -> 'o'
Output Storage: continues to use 's'
Thanks: Ming Xie.
There were two related bugs in the virt-v2v error messages when source
inspection failed.
Firstly it would print the message about running 'virt-v2v -v -x' twice:
virt-v2v: error: inspection could not detect the source guest (or physical
machine).
Assuming that you are running virt-v2v/virt-p2v on a source which is
supported (and not, for example, a blank disk), then this should not
happen. You should run 'virt-v2v -v -x ... >& log' and attach the complete
log to a new bug report (see http://libguestfs.org).
No root device found in this operating system image.
If reporting bugs, run virt-v2v with debugging enabled and include the
complete output:
virt-v2v -v -x [...]
And secondly it would tell you to run virt-v2v -v -x even if you were
running with verbose messages.
Further fix for https://bugzilla.redhat.com/show_bug.cgi?id=1167623
which updates commit ab7197476d.
Thanks: Kun Wei.
Use a 64px icon which is in the menu-icons-default package, hopefully
part of the default installation of any flavour.
Updates commit 9928296c1c.
Thanks to: Xiang Hua Chen.
On Debian systems, usually daemons are run automatically after their
installations; since we are upgrading the template, and installing
packages such as openssh, network-manager, etc, this may cause few
daemon to stay running during the virt-builder execution, causing it to
fail when shutting down (since e.g. /sysroot/dev would be in use).
As workaround, use a temporary policy-rc.d policy layer [1]: give an
error code so no daemon is ever started at any point of package upgrades
and installation. The script is removed at the very end of the
building, so the final image will spawn daemons just fine when booted.
[1] https://people.debian.org/~hmh/invokerc.d-policyrc.d-specification.txt
Usually systemd is configured with prefix=/, so units are in
/lib/systemd. On a system with UsrMove, such as Fedora/RHEL7, /lib is
a symlink to /usr/lib, so both /lib/systemd and /usr/lib/systemd work;
OTOH, on non-UsrMove systems, such as Debian, the two paths are
different locations, and thus trying to edit system files as
/usr/lib/systemd/etc will fail.
As solution, just reference /lib/systemd for systemd files.
On Debian, network-manager-gnome contains nm-applet, which is what
launch-virt-p2v spawns. network-manager-gnome is already in the list of
packages for Debian, while network-manager-applet does not exist.
On Debian, the names of the packages containing the Gtk 2 and 3
libraries have different patterns, and thus just the name will not work
for Gtk 3.
Apply a bit of m4 to use a different package name on Gtk 2 and other
versions. It can be easily adapted to new models in future Gtk 4+,
should the naming pattern change from Gtk 3.
An old version of btrfs-progs (3.17 in this case) has a different
version string, so adapt our filtering to include this line as well.
Related to/updates commit 839ae5bcd5.
mdadm 4.0 no longer lets you create devices called literally "boot" or
"root", giving a very obscure error message:
https://bugzilla.redhat.com/show_bug.cgi?id=1433575
Work around this by calling them something else.
Move all the dependencies with the same name in all the distributions to
a single list at the end.
There should be no change to the package list used to build the p2v ISO.
Where supported, pass the source CPU vendor, model and topology to the
target hypervisor.
-i ova: We can get just cores per socket via a proprietary VMware
extension to OVF.
-i libvirt, virt-p2v: We can get all of these fields from the libvirt
XML.
-o libvirt, -o local: We can preserve all of the information in the
target XML.
-o rhv/vdsm: We can only pass through the topology, since neither
oVirt Engine nor the OVF format supports CPU vendor or model.
Topology uses an extension of OVF which is not part of the standard.
-o qemu: We preserve the topology only because versions of qemu vary
widely in their support for CPU models.
-o glance: Only the topology is preserved since that is all that
Glance currently supports.
Thanks: Pino Toscano.
In the fake <domain type='physical'> libvirt XML that we create to
describe the physical host, we did not accurately pass any information
about the host CPU except the number of cores (<vcpu/>).
This commit extracts detailed information about the vendor, model and
topology of the host CPU and adds that to the libvirt XML for
virt-v2v. Conveniently we can use libvirt capabilities to get this
information without needing to parse /proc/cpuinfo or similar
techniques.
The libvirt XML looks like this:
<domain type="physical">
...
<cpu match="minimum">
<vendor>Intel</vendor>
<model fallback="allow">Broadwell</model>
<topology sockets="1" cores="2" threads="2"/>
</cpu>
...
<features>
<acpi/>
<apic/>
<pae/>
</features>
It looks like Unix.O_CLOEXEC does not exist in OCaml < 4, so resort to
manually using Unix.set_close_on_exec on the filedesc after opening it.
Thanks to: John Sucaet and Richard W.M. Jones.
Unfortunately, there are very few osinfo types available for Debian and
Ubuntu, so make sure to use them only when the version is enough, and
use the higher version available also for any following version.
In cross-compilation, it is not possible to run tests, so the configure
aborts when checking for attribute cleanup.
The comment above the test hints at why we would need to do an actual
run test rather than a compile test: gcc only warns on unrecognised
attributes.
But warnings can be turned into error with -Werror.
Change the AC_RUN_IFELSE into an AC_COMPILE_IFELSE test that runs with
-Werror set. Save the CFLAGS before adding -Werror, and restore them
after the test.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
In the case where we are going to read the disk directly from the OVA
file (partial = true), we will create a qcow2 image backed by the OVA.
If running as root, libvirt will run qemu as a non-root user (because
of no qemu:///session support for root, which is a libvirt bug). qemu
will not be able to read the backing file and thus will fail.
To work around this problem, chmod the backing (OVA) file.
This isn't a complete fix since (eg) the OVA might be located in a
directory which isn't accessible by root, but there's only so much we
can do here until libvirt is fixed.
Thanks: Kun Wei.
Mixing the XML parsing with the other functions of this module made it
very hard to understand. Splitting the XML parsing into another
module simplifies the flow considerably.
This is just code refactoring and should not affect the semantics.
Support also a "socket" listen type with no explicitly specified socket
(which will be generated by libvirt).
Updates commit a4adf48915.
Related to: RHBZ#1378022
Commit a6330e9d3a enabled /run for
systemd-tmpfiles: while this works fine in most of the cases, there are
few tmpfiles configurations that still references /var/run instead of
/run. As result, include also /var/run in the systemd-tmpfiles
execution.
When converting and throwing away the output (-o null), the input
format should not really matter. However it does currently matter for
a couple of reasons:
(1) If the input is not raw or qcow2, then you'll get the error below
because virt-v2v wants to write the same output format as input format
but only allows raw or qcow2 as output formats:
virt-v2v: error: output format should be 'raw' or 'qcow2'.
(2) If you use -o null -oa preallocated then a non-sparse temporary
disk is created before being deleted, but that just wastes disk space
for no reason.
Therefore this change
(i) prevents '-o null -oa preallocated' from being specified on the
command line,
(ii) prevents '-o null -of <anything>',
(iii) forces the output (temporary file) to be raw sparse.
glibc in Fedora is currently configured with `--enable-obsolete-rpc',
so I guess we can see which way the wind is blowing.
(1) This changes our configure script to prefer libtirpc if it is
available.
If libtirpc is _not_ available then:
(a) Headers must be located in <rpc/xdr.h>, or the user must supply
the right CFLAGS.
(b) XDR functions must be located in one of -lportablexdr, -lrpc,
-lxdr, -lnsl or no library at all (ie. -lc), and the user must set
LDFLAGS if needed.
(2) We no longer add these paths automatically to $(CFLAGS)/$(LIBS).
Any part of libguestfs which needs <rpc/*.h> or the xdr_* functions
must use $(RPC_CFLAGS)/$(RPC_LIBS) explicitly.
(3) Previously Mac OS X had a workaround for the broken 64 bit support
in the supplied rpcgen. This workaround "activates" all the time if
you use tirpc, so breaking Linux after the above changes. tirpc is
supported on OS X, so I think it's just better to use that rather than
the broken rpcgen. For that reason I removed the workaround
completely.
Thanks: Roy Keene
`virt-rescue -a disk -i' does the right thing.
`-m' was previously an alternate form of `--memsize'. By sniffing the
parameter we can make `-m MB' continue to work, while also allowing
`-m' to be used as a short form for the `--mount' option.
This also removes most of the description of `--suggest' from the man
page, since it is no longer needed.
Instead of using "direct mode" (which was basically a quick hack),
virt-rescue now launches the appliance with a running daemon.
The daemon doesn't do much -- there is still a bash shell which the
user interacts with. The daemon is there simply to provide the
initial GUESTFS_LAUNCH_FLAG message and to handle shutdown a bit more
gracefully.
To interact with the shell, and replacing direct mode, virt-rescue now
prints out log messages (the output of the shell), and sends input
typed by the user directly to the console socket. This uses the
guestfs_internal_get_console_socket API added previously. Most of the
complexity behind this is hidden in virt-rescue.
This fully fixes the handling of ^C (RHBZ#1152819). Also there were
earlier reports that full screen commands like 'vim' didn't work well,
(RHBZ#1171654), but in this version vim appears to work fine, albeit
only using 80x24 of the screen because of the serial console.
Test:
- Bad SHA1 sum
- Bad SHA256 sum
- Invalid lines in manifest (x2)
- Good checksum and manifest
These tests were originally written by Tomáš Golembiovský. All I have
done is to integrate them into the virt-v2v test suite.
Avoids a lengthy conversion followed by failure if we discover at the
end that OVMF is not installed.
This also changes the order of the methods in -o libvirt and -o qemu
so that it matches the order in the class interface, and also
logically makes more sense.
Thanks: Christopher Brown
Create own blocks for all the parts dealing with FILE*: this way there
is no need to recycle the same FILE* variable for all the operations,
and have each block its own variable automatically cleaned up.
This also fixes a potential undefined behaviour on error: POSIX says
that after a call fclose(), a FILE* cannot be used anymore, not even
on fclose() failure. The previous behaviour for fclose == -1 was to jump
to the error label, which would then try to call fclose() again (since
the FILE* pointer was still non-null).
Filling some of their fields may cause the flow to skip to throw the
"out of memory" exception, and return immediately. To avoid leaking the
struct, or struct list, from the C implementation, use a cleanup handler
so there is no need to manually clean it up.
The previous code:
fcntl (fd, F_SETFL, O_NONBLOCK)
was technically incorrect, because it would have reset any
other flags on the file descriptor.
Thanks: Eric Blake
In case there are no event handlers registered with the handle,
get_all_event_callbacks will count 0 elements, trying to malloc a buffer
of that size. POSIX says that this can result in either a null pointer,
or an unusable pointer.
Short-circuit get_all_event_callbacks to allocate nothing when there are
no events, making sure to use its results only when there were events.
In case there are no event handlers registered with the handle,
get_all_event_callbacks will count 0 elements, trying to malloc a buffer
of that size. POSIX says that this can result in either a null pointer,
or an unusable pointer.
Short-circuit get_all_event_callbacks to allocate nothing when there are
no events, making sure to use its results only when there were events.
In case there are no event handlers registered with the handle,
get_all_event_callbacks will count 0 elements, trying to malloc a buffer
of that size. POSIX says that this can result in either a null pointer,
or an unusable pointer.
Short-circuit get_all_event_callbacks to allocate nothing when there are
no events, making sure to use its results only when there were events.
In case there are no event handlers registered with the handle,
get_all_event_callbacks will count 0 elements, trying to malloc a buffer
of that size. POSIX says that this can result in either a null pointer,
or an unusable pointer.
Short-circuit get_all_event_callbacks to allocate nothing when there are
no events, making sure to use its results only when there were events.
We aren't generating Java bindings for internal and other undocumented
methods, and therefore providing {@link #...} for deprecated methods
replaced by internal methods doesn't work. The easiest way is just to
check for this and turn such links into plain text.
Call windows_path with the actual path to resolve, instead of a null
pointer ('filename' is still null at that point), so Windows paths can
be properly resolved.
Use CLEANUP_FREE to properly dispose the temporary array used for
StringList, DeviceList, FilenameList, and OStringList parameters: this
way, an early return (jumping to the ret_error label) will not forget
cleaning them up.
When convering FBuffer fields of structs in each element of the return
list, make sure to allocate enough buffer to hold also the trailing null
character.
The JNI library uses CLEANUP_FREE macros, whose functions are built in
the internal libutils. Currently, trying to use functions that use
CLEANUP_FREE variables will cause the java execution to stop with a
symbol lookup error (for guestfs_int_cleanup_free).
It's not possible to define an action which takes a parameter called
'message' because the Erlang bindings use that as the name of an
internal variable. Solve this by renaming the Erlang internal
variable.
Fixes commit 84763d7fca.
Note this requires either the following fix in autoconf:
http://git.savannah.gnu.org/cgit/autoconf.git/commit/?id=e17a30e98
OR gnulib sys_types module plus gnulib
commit a512e041120e9012e69afa2f5c3adc196ec4999a (any gnulib more
recent than Sep 2016) which corrects the AC_HEADER_MAJOR macro in a
similar way.
add udev_settle calls to print_partition_table and
sgdisk_info_extract_field because the inspect-os calls
guestfs_part_get_parttype and guestfs_part_get_gpt_guid for all
parition devices found and this causes intermittent with opening block
devices that are certainly present yet RESOLVE_DEVICE macro would fail
wiht ENOENT.
This test failed on ppc64le with:
Failure:
</wrong argument type Fixnum \(expected Array\)/> was expected to be =~
<"wrong argument type Integer (expected Array)">.
In addition the test generated a warning:
tc_800_rhbz507346.rb:29: warning: ambiguous first argument; put parentheses or a space even after `/' operator
This commit fixes both of these and also makes it simpler and faster
by not bothering to launch the appliance.
Fixes commit 227b1eea90.
If libmagic isn't installed then the guestfs_file_architecture API
doesn't work. This means that inspection will always return
<arch>unknown</arch> for every guest. This subtly breaks a few
features. In particular it was reported that the
virt-builder/virt-customize --install option did not work because the
"unknown" architecture of the guest was not compatible with the host.
libmagic is a small, widely available C library, so the easiest fix is
just to make it mandatory.
Reported by: Solarix on IRC
This change refactors how the qemu shell script is generated, making
it a bit less likely that we'll get the quoting wrong. Parameters are
now added to a list, unquoted, and all quoting is done when printing
out the list.
Checked by running virt-v2v -o qemu before and after the change and
manually comparing the output (which is not identical, but still
correct).
Make the documentation in Convert_linux more generic than just
Enterprise distros using RPM. Also, rename the name of the module from
"enterprise-linux" to simply "linux".
In particular, read the URL, the source name, and both the summary and
the description. For the long description, add a small system to read
continuation lines.
Adapt the expected result of virt-inspector for Debian and Ubuntu phony
guests, so test-virt-inspector.sh still passes.
A message was translated without the trailing \n. As it was in a
verbatim section of POD, I actually reverted the whole translation.
The error was:
./uk.po:53533: 'msgid' and 'msgstr' entries do not both end with '\n'
msgfmt: found 1 fatal error
In fact po4a (or whatever calls msgfmt) seems to ignore the error and
continues which is another problem.
Fixes commit 794b6a1acf.
Commit 794b6a1acf added cs.po, but
because this wasn't also added to lingas_not_translated the file was
not included in EXTRA_DIST or processed properly.
Fixes commit 794b6a1acf.
Compiling po-docs/ja/guestfish.pod failed with:
Pod input around line 277: L<> starts or ends with whitespace
Pod input around line 362: L<> starts or ends with whitespace
I reverted the part of the L</...> link which had been translated.
The fix isn't quite correct -- headings *can* be translated, and the
corresponding internal links should be translated too. However
pod2man has problems dealing with such links and in any case the
partial translation was not correct, so revert to the English version
of the link.
Updates commit 794b6a1acf.
Somewhere between OCaml 4.01 and OCaml 4.04, <caml/memory.h> changed
the way that CAMLlocalX() values on the stack are initialized. In
OCaml 4.01:
#define CAMLlocal1(x) \
value x = 0; \
CAMLxparam1 (x)
but in OCaml 4.04:
#define CAMLlocal1(x) \
value x = Val_unit; \
CAMLxparam1 (x)
The code in mllib/visit-c.c assumed that the value would be
initialized to Val_unit, and used a check (exn != Val_unit) to see if
the value had been updated. This failed badly (with a segfault) on
OCaml 4.01 in Debian 8.
Resolve this by always initializing CAMLlocalX() values before use.
We are redy to restore the disabled test. The relevant libvirt fixes
should be available in 3.1.0.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
Libvirt < 3.1.0 lacks enough support for json: pseudo-URLs. Notably it
does not allow use of "raw" driver that we need.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
The test tests/regressions/rhbz914931.c works by causing the daemon to
segfault while writing to it.
For reasons unknown, when configured --without-libvirt, this causes
the test to fail receiving SIGPIPE (exit code 141). We can prevent
this by installing a signal handler to ignore SIGPIPE, so the signal
is converted to EPIPE which the code handles properly.
The old version of qemu-img available in Debian 8 doesn't support the
preallocation option for raw format. However this doesn't matter
because the default is preallocation=off (ie. sparse) for old and new
versions of qemu-img so we can just omit this parameter.
Fixes commit 53317e3f07
and commit fa83b47876
and commit 74ded0dbc1.
Slightly change the logic for installing RHEV-APT and VMDP:
- Only warn on failure to install RHEV-APT if the output hypervisor
requested it. Otherwise the warning is pointless because RHEV-APT
can't be used anyway.
- Make the warning clearer and actionable.
- Try to install VMDP unconditionally (rather than only if RHEV-APT
failed). It is always useful for SuSE hosts. However don't warn if
it's not available.
Previously the Convert_linux conversion module depended on one feature
of the output module (#keep_serial_console). This was extracted in an
ad-hoc way from the output module and passed as an extra parameter to
the conversion module.
Instead of doing it this way, just pass the output module into the
conversion module, so it can call output#keep_serial_console itself.
This is just a simplification of the existing code, but otherwise adds
no new features.
Non-Linux Unix guests may have static devices for CDs, so make sure to
skip them when reading /etc/fstab. This is the same as done for
/dev/fdN devices, i.e. floppy devices.
Currently imports from vCenter sometimes obey proxy environment
variables (eg. $http_proxy) and sometimes don't. The initial libvirt
connection to fetch metadata never uses the proxy, but because the
subsequent conversion and copying uses libcurl, it will pick up on
proxy environment variables.
This makes no sense, and in any case vCenter is really slow as it is
without putting another device into the data path. Therefore ensure
that libcurl does not see any proxy environment variables by unsetting
them.
Concatenate six small modules containing Unix/POSIX library call
bindings into a single module called Unix_utils.
The previous modules and the library functions bound were:
- Dev_t: makedev(3), major(3), minor(3)
- Exit: _exit(2)
- Fnmatch: fnmatch(3)
- Fsync: sync(2)
- Mkdtemp: mkdtemp(3)
- StatVFS: statvfs(2)
Apply this change across all the shell scripts containing tests.
Additionally this defines the environment variables $abs_srcdir,
$abs_builddir, $top_srcdir, $top_builddir, $abs_top_srcdir and
$abs_top_builddir which can now be used throughout test scripts.
This macro(?) expands to some shell script to source the
tests/test-functions.sh file from its correct location. The intention
is to use this in all tests, but in this commit only the existing
tests which already include test-functions.sh are modified.
Daemon 'proc_nr's have to be assigned monotonically and uniquely to
each daemon function. However in practice it can be difficult to work
out which is the next free proc_nr. Placing all of them into a single
table in a new file (proc_nr.ml) should make this easier.
Group the APIs logically and move them into new modules:
Actions_core:
Core APIs and anything that doesn't fit into another group, eg. launch.
(With some more effort this could be split further.)
Actions_augeas:
Augeas APIs, eg. aug-init.
Actions_debug:
Debug APIs.
Actions_hivex:
Hivex APIs, eg. hivex-open.
Actions_inspection:
Inspection APIs, eg. inspect-get-type.
Actions_properties:
Handle properties, eg. set-hv, get-hv.
Actions_tsk:
SleuthKit APIs, eg. filesystem-walk.
*_deprecated:
All of the above modules have deprecated variants, where we
place the deprecated actions.
Sort the functions so the output is stable.
This changes the order in which the C API tests run. Previously we
ran the newest tests first, which was useful when we were frequently
adding new APIs. Now we run them in sorted order.
Also rename the function 'uuidgen ()' (conflicts with a same-named
function in Common_utils) to 'stable_uuid'. Notice that the UUID is
now only computed once per run of the generator, whereas previously
the same value was computed over and over again.
Sadly, there are elements (hello "gentoo"!) that, during the cleanup.d
phase, wipe out almost everything in the /tmp of the guest, including
the /tmp/aux where virt-dib mounts the auxiliary data. Since that
removal excludes things starting with "in_target" (mostly to avoid
wiping the "in_target.d" that disk-image-create itself sets up, then
rename our "aux" subdirectory to "in_target.aux" to work it around.
Hopefully, the "gentoo" element will be fixed upstream:
https://review.openstack.org/#/c/436101/
Commit 8ee51ee396 re-scans for the
available scripts when running certain phases; OTOH, some of them may be
missing, usually due to the lack of scripts for that phase in the
selected set of elements.
Indeed, if there is no directory for a phase, safely raise Not_found so
the case is handled as if the phase was missing in the final_hooks
Hashtbl.
The real sudo does it as well, and leaving them when preserving the
environment (-E) maybe breaks the applications, as e.g. chroot will have
a TMPDIR path pointing outside of it.
The inspection code already computed the paths of the software hive
(twice!) and the system hive, plus we also recompute the same paths
elsewhere, in virt-v2v for example. Therefore it makes sense to store
the paths from the inspection code and make them available through two
new APIs.
Add a convenient tuple Registry.t for the currently open hive. It
contains the guestfs handle and the root node of a registry.
The functions with_hive_readonly and with_hive_write are modified to
pass this tuple to their callbacks.
For reasons unknown virt-v2v recomputed the CurrentControlSet from
first principles, and passed %systemroot% around to all functions.
However that data is available from the libguestfs inspection APIs.
Move the functions decode_utf16le, encode_utf16le, get_node,
with_hive_readonly and with_hive_write to common code in a new module
called Registry.
This also defines types for nodes and values, instead of using int64
directly.
Just code motion.
Adding a new optional parameter to hivex_open in
commit 1f99251223 reveals a subtle bug
in the virt-p2v slow test.
Because we didn't pass LIBGUESTFS_PATH through to the instance of
virt-v2v, it was running with the new binary and library code, but
with the installed appliance (or would have failed if libguestfs
wasn't installed on the host when running the test). In particular
this bug was revealed when the new virt-v2v binary passed the
GUESTFS_HIVEX_OPEN_UNSAFE flag to the guestfs_hivex_open call, which
the (old, installed) daemon did not recognize and rejected with an
error.
For the same reason we also have to pass in LIBGUESTFS_CACHEDIR to
make sure that supermin doesn't reuse the cached appliance from
/var/tmp.
When extracting the content of the guest as tar, save also the extended
attributes and the SELinux attributes. This makes sure guests exported
as tar, tgz, and as docker image will work fine afterwards.
This is what disk-image-create does as well.
Pass the HIVEX_OPEN_UNSAFE flag when opening hives for reading.
Do NOT pass it when opening hives for writing.
This should make inspection, virt-win-reg and virt-v2v more tolerant
about handling Windows Registry corruption, without increasing the
risk of causing new corruption in hives.
In hivex >= 1.3.14, there is a new HIVEX_OPEN_UNSAFE flag allowing
heuristics to be used to deal with corrupted hives. Map this flag
into the libguestfs API.
If the flag is not supported (because libguestfs was compiled with
hivex < 1.3.14) then the flag is ignored. This is safe behaviour:
opening corrupted hives will give an error, as happened previously.
Because v2v/test-harness is a subdirectory of v2v, and because both
paths are listed in $(DIST_SUBDIRS), using find $(DIST_SUBDIRS) will
list files in v2v/test-harness twice. (This probably happens in other
directories too, but I noticed it here.) The easiest fix for this is
simply to use 'sort -u' to remove the duplicates.
Implement the "squash" output format, i.e. a squashfs compressed
filesystem.
This was implemented in diskimage-builder upstream as
commit 9d13084c4183b63587e1f5e4b03395a8df6538f6.
Introduce a new API to create a new squashfs filesystem out of a path
in the guest. It can be configured to exclude paths based on patterns,
and to select which compression use for the filesystem.
The advantage of running mksquashfs directly in the appliance is that
ownerships are properly saved, as opposed to tar_out + local untar.
It will be useful also for APIs different than tar-out, so move it to
guestfsd.c, and add it a parameter to specify the function name that
invoked it.
This is mostly code motion.
Recently, diskimage-builder moved the log cleanup from the 'base'
element to disk-image-create proper; the cleanup done is:
- truncate any file in /var/log
- remove *.log files in /root
This was implemented in diskimage-builder upstream as
commit 022d93ee822e71245af52c4cf8f8a8e82f599af3.
Scripts (especially root.d and extra-data.d) may add new scripts to the
existing set during their execution: in this case, re-read the list of
available scripts, so we do not miss any new addition.
It looks like scripts (in particular in the extra-data.d phase) may
in-flight add new scripts to the existing ones. As first step to
handle this situation, change the path of hooks passed for extra-data.d
scripts: instead of the local temporary directory, use the in-guest
location that is visible as result of the local fuse-based mount of the
guest.
Source the "die" script, part of the diskimage-builder library, when
running hooks that run outside the guest chroot. This is what
disk-image-create does, and scripts expect to use the "die" function
without sourcing the "die" script containing it.
Query awk for the list of environment variables, instead of trying to
extract the list from the output of `env`: the old approach breaks when
any of the environment variable contains more than one line.
GCC 7.0.1 can determine if there is likely to be sufficient space in
the output buffer when using sprintf/snprintf, based on the format
string.
The errors were all either of this form:
bindtests.c:717:29: error: '%zu' directive output may be truncated writing between 1 and 19 bytes into a region of size 16 [-Werror=format-truncation=]
snprintf (strs[i], 16, "%zu", i);
^~~
bindtests.c:717:28: note: directive argument in the range [0, 2305843009213693951]
snprintf (strs[i], 16, "%zu", i);
^~~~~
or this form:
sync.c: In function 'fsync_devices':
sync.c:108:50: error: '%s' directive output may be truncated writing up to 255 bytes into a region of size 251 [-Werror=format-truncation=]
snprintf (dev_path, sizeof dev_path, "/dev/%s", d->d_name);
^~
Fixed by converting these into dynamic allocation, or making the
output buffer larger, whichever was easier.
There is a gnulib macro we can use to make this simpler for integers.
It requires a new gnulib module (intprops), but it turns out that we
were already pulling that in through dependencies, so the change to
bootstrap is a no-op. (thanks: Dan Berrange)
This happens with GCC 7.0.1. The errors were all of the form:
qemu-speed-test.c: In function 'main':
qemu-speed-test.c:153:7: error: this statement may fall through [-Werror=implicit-fallthrough=]
usage (EXIT_SUCCESS);
^~~~~~~~~~~~~~~~~~~~
qemu-speed-test.c:155:5: note: here
default:
^~~~~~~
Commit 74ded0dbc1 changes existing calls
to truncate to run 'guestfisk disk-create' instead. However this
fails if guestfish has not been installed. By using the ./run script,
we can use the just-built guestfish instead.
Fixes commit 74ded0dbc1.
This adds a contrib script which can be used to build the virt-p2v ISO
on top of RHEL 5 or RHEL 6, i686 (32 bit) or x86-64 (64 bit) base.
There is also a script for testing the ISOs produced this way.
I don't think this will really make any difference. However there is
some puzzling disk corruption in the recently built RHEL 5 & 6 i686
images, and I want to make absolutely sure they are not caused by an
unsynchronized disk image.
There were still a few places where we were testing bash rather than
the intended program. Add `libtool --mode=execute' to the beginning
of the @VG@ macro to try to get around this.
If supported, use socket activation to pass a pre-opened listening
socket to the NBD server. This means we can guarantee to choose an
unused port even if there are multiple instances of virt-p2v running
(ie. when testing) or if there are unexpected services running on the
same machine.
This change applies to both NBD servers, since both now (or will
shortly) support this feature.
In preparation for using socket activation, move the nbd_local_port
global inside nbd.c and make it private.
This is largely code motion. However I rearranged the order in which
the ssh data connection and the NBD server are started up. Previously
the ssh connection was made first, and the NBD server was then
started. Now the NBD server starts first. This happens so that nbd.c
can choose the local port, and when we implement socket activation it
will allow nbd.c to open the socket and choose a free port too.
This is almost pure code motion, but I changed the name and prototype
of the function 'wait_nbd' to make its purpose clearer, and to remove
the unnecessary timeout setting (which is hard-coded).
Parse the output disk as URI, and use all its attributes just like
it is done for the input disk. The only change is that the fsync of the
output disk is limited now for local URIs only, since it will not work
with remote protocols.
Implement the "tgz" output format, i.e. "tar" compressed as gzip.
This was implemented in diskimage-builder upstream as
commit da41ee6012b064aa6901c871a1104a3a3933117e.
Implement a system similar to the operations in virt-sysprep, so each
output format has its own file and attributes (such as command line
arguments).
The result is that there is no more need to spread everywhere the job
required by each output format, such as checking for tools, or hooking
at the right point.
Make sure they are handled as options for the actual mkfs driver,
instead of mkfs itself. Also, improve the documentation of
--mkfs-options accordingly.
This was fixed in diskimage-builder upstream as
commit 5e2330d89c6c0e55b270464abea7e99a4d3246ad.
Add a new option to create files with the MD5 and SHA512 checksums of
all the disk image types produced.
This was implemented in diskimage-builder upstream as
commit 2ea5feca5c4b64867ac327736edfb20408f8840e and
commit 22952b7ea0543bb4f446752976d1d8ba232b021a.
Improve the error message that is shown when trying to selecting an
element, and another element provides it.
This was implemented in diskimage-builder upstream as
commit 452f7b8d5aaa6e85e3b01b970cd9cbc842b34b86.
We don't have to always extract all files from the OVA archive. The OVA,
as defined in the standard, is plain tar. We can work directly over the
tar archive if we use correct 'offset' and 'size' options when defining
the backing file for QEMU. This puts much lower requirement on available
disk space.
Since the virt-v2v behaviour for OVA input now depends on QEMU version
available this affects some of the tests. Expected result of the
affected also has to depend on the QEMU used thus such tests will have
two *.expected files.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
The function looks up file in tar archive and returns a tuple containing
which at byte it starts and how long the file is.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
The function returns version of qemu-img as a tuple (major, minor), or
the value (0,9) in case there was an error detecting the version.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
In commit 08f82f2e3d we increased the
memory size to 800MB (on x86) so that the semodule command would work.
However it has been discovered that another SELinux command takes
large amounts of RAM (setfiles during the SELinux relabel step).
Therefore increase the memory size again, this time to 2000MB.
The information whether the disk is gzip compressed or not is stored
in the OVF. There is no reason to do the detection.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
- Make module name explicit, so it's more obvious which module a
function is defined in.
- Group lines of code by feature.
- Capitalize some module names properly.
Just code motion, no functional change.
Commit 004de6cf45 was not sufficient to
solve this problem. We are also using gperf in guestfish. Rewrite
the code in the same way to avoid having to prototype the hash
function.
Add code in virt-p2v so that it can use nbdkit (with the file plugin)
as an alternative to qemu-nbd.
This is controlled through the virt-p2v --nbd command line option,
allowing you to select which server (out of qemu-nbd or nbdkit) you
prefer (with a fallback). The default is: --nbd=qemu-nbd,nbdkit so
qemu-nbd will be used preferentially.
When testing virt-p2v (eg. on the host machine) this prevents us from
testing two instances of virt-p2v at the same time because both will
try to use the same port.
Generate the random filename using our utility function
guestfs_int_random_string. This also means that we will not need to
call srandom() in guestfish or virt-edit.
The prototype of the hash function changed in gperf 3.1:
* The 'len' parameter of the hash function and of the lookup function is now
of type 'size_t' instead of 'unsigned int'. This makes it safe to call these
functions with strings of length > 4 GB, on 64-bit machines.
This commit moves the guestfs_int_string_to_errno function into the
third (functions) section of the gperf file so it is able to use the
lookup function without needing a prototype.
Thanks: Marius Cirsta
The prototype of the hash function changed in gperf 3.1:
* The 'len' parameter of the hash function and of the lookup function is now
of type 'size_t' instead of 'unsigned int'. This makes it safe to call these
functions with strings of length > 4 GB, on 64-bit machines.
This change tries to detect the required type in ./configure
Thanks: Marius Cirsta
Three more pieces of common code are moved under the common/
subdirectory. This is just code motion.
Note that windows.[ch] wasn't even being used by guestfish. That code
was only used in other virt tools.
This is mostly code motion but:
(1) I had to remove the compile-time COMPILING_GUESTFISH and
COMPILING_VIRT_RESCUE macros and replace them with runtime constants
and checks.
(2) I moved the fish/config.c file into this library.
Just code motion.
This commit makes it clearer what is a utility and what is part of the
library. It also makes it clear that we should rename:
guestfs-internal-frontend.h -> utils.h
guestfs-internal-frontend-cleanups.h -> cleanups.h (?)
but this commit does not make that change.
This commit, which is just code motion, moves the common XDR protocol
code (libprotocol) and the common errno handling (liberrnostring) into
libraries which are each built once and shared between the library and
daemon.
The error is:
gobject/docs/gtk-doc.make:52: warning: CLEANFILES multiply defined in condition TRUE ...
gobject/docs/Makefile.am:95: 'gobject/docs/gtk-doc.make' included from here
common-rules.mk:28: ... 'CLEANFILES' previously defined here
gobject/docs/Makefile.am:18: 'subdir-rules.mk' included from here
subdir-rules.mk:20: 'common-rules.mk' included from here
In the ancient version of PCRE in RHEL 5, partial match support does
not support /.*/, returning PCRE_ERROR_BADPARTIAL. However the
equivalent regular expression /(?:.)*/ works fine, so use that
instead.
RHEL 5 has ancient Gtk 2.10 which lacks several functions and whole
features we need (eg the spinner). Add various macros and function
definitions to work around this.
The guestfs_canonical_device_name API was rewriting /dev/mdX as
/dev/sdX. This is wrong since (eg) /dev/sd0 is not a device name, so
if you pass the canonicalized name back to the API it will fail.
virt-v2v was one tool doing this.
Bug found by valgrind on aarch64:
==21350== Syscall param faccessat(pathname) points to unaddressable byte(s)
==21350== at 0x4F05274: access (access.c:29)
==21350== by 0x4954DC3: guestfs_int_get_uefi (appliance-uefi.c:90)
==21350== by 0x49752D7: launch_libvirt (launch-libvirt.c:413)
==21350== by 0x496FD83: guestfs_impl_launch (launch.c:98)
==21350== by 0x4902003: guestfs_launch (actions-3.c:145)
[...]
==21350== Address 0x0 is not stack'd, malloc'd or (recently) free'd
Because we used 'return' directly, the effect of the CAMLparam0()
macro was not undone, resulting in local roots corruption when the
visit callback throws an exception.
The visit_tests unit test picked this up, but the corruption was only
seen on aarch64.
Fixes commit 4a62e52111.
appliance-uefi.c: In function 'guestfs_int_get_uefi':
appliance-uefi.c:67:9: error: implicit declaration of function 'access' [-Werror=implicit-function-declaration]
if (access (codefile, R_OK) == 0 && access (varsfile, R_OK) == 0) {
^~~~~~
appliance-uefi.c:67:5: error: nested extern declaration of 'access' [-Werror=nested-externs]
if (access (codefile, R_OK) == 0 && access (varsfile, R_OK) == 0) {
^~
appliance-uefi.c:67:27: error: 'R_OK' undeclared (first use in this function)
if (access (codefile, R_OK) == 0 && access (varsfile, R_OK) == 0) {
^~~~
appliance-uefi.c:67:27: note: each undeclared identifier is reported only once for each function it appears in
cc1: all warnings being treated as errors
Fixes commit dd519e8a8e.
Run the following command over the source:
perl -pi.bak -e 's/(20[01][0-9])-2016/$1-2017/g' `git ls-files`
(Thanks Rich for the perl snippet, as used in past years.)
EDD (https://en.wikipedia.org/wiki/Enhanced_Disk_Drive) is a BIOS INT 13h
subcall for communicating the extended size of the boot disk over 8GB
to the OS. Since libguestfs doesn't use a boot disk, and the
appliance disk is limited to 4GB, and we use virtio-scsi, this is all
useless.
EDD is also broken currently on RHEL 7.3, see:
https://bugzilla.redhat.com/show_bug.cgi?id=1404287
Also the EDD probing takes significant extra time (about 80ms on my
laptop), and using edd=off reduces this time although doesn't entirely
eliminate it.
One of the tests for btrfs_subvolume_show tries to invoke it on the
toplevel subvolume, expecting its failure (since btrfs-progs used to not
be able to list that). However, since v4.8.3 btrfs-progs can do that
(even if the data returned for that subvolume is limited/incomplete),
and thus this test fails because the API does not fail anymore.
Since we can consider that functionality working (for some kind of it),
then drop this test.
Very recent versions of tar (most probably as a consequence of
CVE-2016-6321) may refuse archive members with '..', like the relative
paths to upper level directories.
Since these are just tests, simply copy the files in the temporary
directories where tar (or zip as well) is run, so all the files are in
the same directory.
The return value struct was freed using a simple free() instead of the
own cleanup function of each struct: this meant dynamically allocated
values (such as strings) were leaked.
Use the proper cleanup functions instead.
The return value struct was freed using a simple free() instead of the
own cleanup function of each struct: this meant dynamically allocated
values (such as strings) were leaked.
Use the proper cleanup functions instead.
The definition of BUILDFILES was accidentally removed, resulting
in the error:
cp /home/rjones/d/websites/libguestfs/download/builder/
cp: missing destination file operand after '/home/rjones/d/websites/libguestfs/download/builder/'
Fixes commit 65a0570385.
It fails with:
File "../builder/templates/make-template.ml", line 1, characters 0-1:
Parse error: [a_LIDENT] expected after "#" (in [implem])
End_of_file
An error occurs while processing.
In any case this doesn't matter because this file should not be
translated.
Fixes commit a1ea9a2599.
See:
http://caml.inria.fr/pub/docs/manual-ocaml/comp.html#ss%3Awarn57
I believe the code as written previously was incorrect. However we
are lucky because if neither clause matches then it will fall through
to displaying an error message, allowing the user to correct the
problem.
For a very long time we have maintained two sets of utility functions,
in mllib/common_utils.ml and generator/utils.ml. This changes things
so that the same set of utility functions can be shared with both
directories.
It's not possible to use common_utils.ml directly in the generator
because it provides several functions that use modules outside the
OCaml stdlib. Therefore we add some lightweight post-processing which
extracts the functions using only the stdlib:
(*<stdlib>*)
...
(*</stdlib>*)
and creates generator/common_utils.ml and generator/common_utils.mli
from that. The effect is we only need to write utility functions
once.
As with other tools, we still have generator-specific utility
functions in generator/utils.ml.
Also in this change:
- Use String.uppercase_ascii and String.lowercase_ascii in place
of deprecated String.uppercase/String.lowercase.
- Implement String.capitalize_ascii to replace deprecated
String.capitalize.
- Move isspace, isdigit, isxdigit functions to Char module.
Support for RHEV with RHEL 6 nodes required us to output the old style
qcow2 compat=0.10 images. Since RHEV 3.6 GA, RHEL 6 has not been
supported as a RHEV node type. Since RHV 4.1, compat=1.1 is
supported. (Support for compat=1.1 is uncertain in RHV 4.0 even on
RHEL 7 nodes.)
There are significant downsides to using qcow2 compat=0.10 instead of
the modern default (compat=1.1).
Therefore this patch does two things:
For -o rhev, it drops support for compat=0.10 completely. You must
use RHV 4.1.
For -o vdsm, it adds an interim flag (--vdsm-compat=0.10 or
--vdsm-compat=1.1) which controls the compat level of the qcow2 output
file. VDSM should use --vdsm-compat=1.1 when it is known that modern
qemu is available. We can make this the default later when all RHV
instances have moved to 4.1.
It also adds:
vdsm-compat-option
to the `virt-v2v --machine-readable' output to indicate that this flag
can be used.
Thanks: Yaniv Kaul, Michal Skrivanek.
Flag the filesystems for Linux /usr properly as USR role, and detect
some data out of it, like the distro information from an os-release
(if present), and the architecture (since the binaries used for our
architecture check will be available there only).
Later on, collect the results in a way similar to what is done for
CoreOS: for each non-CoreOS root, try to find its /usr filesystem, and
if found then merge what is missing from root; in the last case, also
override the distro inspection data (version, product name) if available
in /usr.
When checking for the existance of /dev/mapper devices found in the
fstab of a filesystem, using guestfs_exists means they are checked as
files in the guest, while they really appear as devices on the
appliance. Instead, try the lvm name resolution anyway, and ignore them
when they are reported as missing.
Thanks to: Richard W.M. Jones.
Fixes commit 96b6504b09.
Move the checks for empty xmlXPathObjectPtr, and for extracting the
result string out of it, to a new helper functions.
This is just code motion, there should be no behaviour changes.
Add a new inspect role for "/usr" partitions, and use that to mark the
/usr partition in CoreOS: this additional role allows to ease its lookup
later on, when merging its results into those of the root.
Introduce a new enum to classify the role of a filesystem, if available.
This will help later on when doing operations on non-root filesystems,
like detecting particular mountpoints such as /usr.
The new enum has only "root" as known role, which replaces the is_root
flag.
Tools could require the use of pseudo-terminals, so make sure we have
/dev/pts available in the appliance. The "command" API already
bind-mounts it when running commands, so this is the only bit needed.
This refactors command line parsing into a parse_cmdline function.
The function uses the Arg module to parse the command line instead of
ad hoc parsing.
This also enforces use of the ./run script to run the program, since
otherwise you can end up with mixed versions of the OCaml bindings and
the C library.
This updates commit 65a0570385.
initviocons package provides tools to resize the terminal. Having it
in the appliance will allow SUSE users to have proper line wrapping
in their terminal when using virt-rescue.
Rebuild these images with the new make-template.ml script. This fixes
the lack of virtio-scsi driver and broken initramfs.
These images all use LVM because without that both Fedora and RHEL put
the root filesystem on an extended partition (/dev/sda5) which
virt-resize cannot handle.
(Note the RHEL images are not public.)
Create a new directory (builder/template). Integrate all of the
scripts into a single program, so that templates are generated more
consistently.
This also changes how the index file is generated. The script now
generates the index file fragment and saves it under version control,
and then generates the final index file by concatenating these.
(Previously the index was written by hand which was tedious and
error-prone.)
The new script also saves the generated kickstart under version
control so it can be referenced later.
It turns out that adding the virtio-scsi driver does not help
to fix the dracut problem.
This partially reverts commit f766c84c39,
keeping the change to index.
Unfortunately I was unable to build s390 binaries since multilib was
dropped in Fedora 24 on s390x. Going from the source of the 'file'
command it seems as if it prints "32-bit" (the architecture is really
31 bit).
As noted by Pino in another patch, the logic passes the first member of
the struct which happens to be the right address to the callback
function.
This will break the callback if order of the members of the struct will
change.
As the callback is using the entire struct, better to pass the pointer
to the struct itself.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
If an environment variable such as XDG_RUNTIME_DIR or one of the
tmpdirs or cachedir is set to a non-existent directory, improve the
error message that the user will see so that (where possible) it
includes the environment variable or API call.
This is still not bullet-proof because it's hard to display the
environment variable if it is LIBGUESTFS_TMPDIR or
LIBGUESTFS_CACHEDIR, but the main problem is with XDG_RUNTIME_DIR
(because of systemd bugs).
Thanks: Hilko Bengen for identifying the bug.
This makes a number of small changes to how the internal documentation
is generated and what can be documented.
Header files are now permitted to contain internal documentation.
This works in the same wasy as .c files.
Also documentation can be added for structs as well as functions.
This commit also adds documentation to some header files and structs,
and fixes a few places which contained broken header documentation.
We do this by sending an Inhibit() message to logind and receiving a
file descriptor back, which we hold open until the conversion
completes (or fails). This is described here:
https://www.freedesktop.org/wiki/Software/systemd/inhibit/
This adds an additional optional dependency on DBus since we use DBus
to call the Inhibit() method.
Reported-by: Chris Cowley.
Create a single temporary directory for all the files created during the
virt-builder run (except big disk images, which already use the
libguestfs cache directory). A single directory means there is no need
to manually schedule all the temporary files and directories for removal
when the application quits.
Add a new optional parameter for the Curl ADT, so temporary files can be
created in a specified directory (which is supposed to be temporary, and
disposed only when the application quits).
$0 is set to the name of the test script, eg. test-settings.sh, and in
bash there is no way to change that.
Create a new environment variable, $script, which is set to the name
of the wrapper script (eg. test-settings-fedora-24.sh).
This should give more accurate error messages.
When TESTS_ENVIRONMENT already uses 'run', the VG variable
doesn't also need to use 'run'.
The specific problem is that if the command contains newlines
then double invocations of the 'run' script fails (in libtool).
ie the following command failed causing errors in check-valgrind:
$VG virt-builder phony-fedora \
-v --no-cache --no-check-signature $no_network \
...
--write '/etc/append4:line1
' \
...
Currently 'make install' installs the virt-p2v binary in
/usr/libexec/virt-p2v on the host. It is never supposed to be run
from there, even by another program, so use of /usr/libexec is
incorrect. It is only supposed to be copied into USB keys / ISOs /
etc created by virt-p2v-make-* scripts.
The other problem with shipping a "naked" binary on the host is that
packages built from that get all the dependencies of virt-p2v, for
example Gtk. This is unnecessary just for running the command line
scripts mentioned above.
This changes the Makefile and scripts so that the binary is stored
compressed in $libdir/virt-p2v/virt-p2v.xz. It is compressed to avoid
exposing the dependencies. It is stored under $libdir since the
binary is still architecture-dependent.
A further change is that when we copy the binary into the virt-p2v
ISO, it is now installed in /usr/bin instead of /usr/libexec. (And
note that we always use /usr/bin, not $bindir, since this path should
not need to be affected by the configuration of libguestfs).
Added:
- cdrtools: added as alternative to cdrkit
- multipath-tools: contains kpartx (in AUR)
Removed:
- ntfsprogs: the package is no longer available, it has been completely
replaced by ntfs-3g (already in packagelist.in)
- zfs-fuse: no longer in AUR
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
The OVF standard allows the use of SHA256 hashes in the manifest file.
Adding support for this.
One of the tests was updated to put SHA256 into manifest file.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
The regular expression for parsing the manifest line was wrong. There is
a mandatory space between '=' and the hash.
Another problem was that only the first line of the manifest file was
actually processed.
Also added some debugging info and warning to catch problems with
parsing.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
Try to improve the way packages of VMware tools are removed from
YUM-based guests:
- when filtering the package itself from its providers, do a stricter
check so either the provide is the unversioned package, or it is
exactly its own name
- if the package has no other providers, then going further will cause
the invocation of 'yum install' with no packages, and thus the package
itself will not be added to the list of packages to be removed; to
overcome this issue, just mark the package as "to be removed" in that
case
Related to: RHBZ#1155150
When checking whether a kernel supports virtio or it is Xen-based, it is
assumed that the feature has the kernel module for it; this will fail if
the feature is built-in in the kernel, misrepresenting it.
The solution is to check the kernel configuration whether the feature
is built-in, in case there is no module available.
This fixes the virtio detection on Ubuntu kernels, where virtio is
built in and not as module.
Detect only once which method must be used to get and set the default
Grub2 kernel in the guest: this avoids the same checks in list_kernels
and set_default_kernel.
Also, add a "no method" option as well: Debian/Ubuntu guests do not have
neither grubby nor Perl's Bootloader::Tools, so there is no way to know
what is the default kernel, nor to change it. In this case, add a
warning about this situation.
Move the Checksums module from virt-builder mostly as it is; the only
change is that on checksum mismatch an exception is raised rather than
invoking "error" directly: this way users of verify_checksum &
verify_checksums can do their own handling of the situation.
Debian is also doing a UsrMove/UsrMerge:
https://wiki.debian.org/UsrMerge
However it is not finalized that Debian will actually make this
change.
Since some Debian systems have /sbin as a symlink and other have /sbin
as a real directory, and we should avoid choosing a symlinked
directory for the daemon, the easiest fix is simply to probe /usr/sbin
before /sbin since under all scenarios (and Fedora too) /usr/sbin is a
real directory.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=838995
Create a new libfishcommon convenience static library to build just once
(instead of 12 times!) the majority of the guestfish sources used in the
rest of the C tools (mostly for command line stuff, inspection, and
mount).
The notable exceptions not using libfishcommon are guestfish itself, and
virt-rescue: both need to build at least one of the common sources using
additional CPPFLAGS.
All the define was doing in options.h was masking the declaration of
inspect_mount_root, which was always built-in in inspect.c (because of
the unconditional #define there) anyway.
Since this is common code used by all the C tools, try to avoid extra
knobs which add different code paths for no benefit.
Some of the C tools were building also config.c as part of the shared
sources from guestfish, and thus bringing a dependency on libconfig.
Since none of them actually read the libguestfs configuration at all,
then exclude fish/config.c from their build, and stop linking to
libconfig.
Using update-initramfs is the native way of updating initrd on Debian
based systems.
To add some modules to the image we can list them in file
/etc/initramfs-tools/modules.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
A disk of type 'volume' is stored as
<source pool='pool_name' volume='volume_name'/>
and its real location is inside the 'volume_name', as 'pool_name': in
this case, query libvirt for the actual path of the specified volume in
the specified pool.
Adjust the code so that:
- for_each_disk gets the virConnectPtr, needed to do operations with
libvirt
- when extracting the disk filename depending on the type, the code
snippet doing it can directly set 'filename', without setting an XPath
result variable
Only file-based volumes are supported for now; more types can be added
(with proper testing) later on.
Since libvirt 2.0.0, these two new <listen/> types have been
supported: https://libvirt.org/formatdomain.html#elementsGraphics
This change just copies that configuration over from the source to the
destination if the destination is also libvirt.
Since we previously used 'LNone' to mean "no parseable <listen/>
element" I also had to change previous uses of 'LNone' to 'LNoListen',
so we can use 'LNone' to mean "<listen type='none'>".
Thanks: Ming Xie.
NTFS file system always has the MFT file at inode 0. This reliable
information helps testing the API.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
Library's counterpart of the daemon's internal_find_inode command.
It writes the daemon's command output on a temporary file and parses it,
deserialising the XDR formatted tsk_dirent structs.
It returns to the caller the list of tsk_dirent structs generated by the
internal_find_inode command.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
The internal_find_inode command searches all entries referring to the
given inode and returns a tsk_dirent structure for each of them.
The command is able to retrieve information regarding deleted
or unaccessible files where other commands such as stat or find
would fail.
The gathered list of tsk_dirent structs is serialised into XDR format
and written to a file by the appliance.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
Make use of the additional command line arguments, and API needed to
decrypt LUKS partitions.
This affects only virt-customize, virt-get-kernel, virt-sparsify, and
virt-sysprep, as they are the main OCaml tools interacting with
user-provided images.
Expose via Common_utils the C functions & variables (part of guestfish)
that handle decryption of LUKS partitions, and the additional command
line arguments to tune the way they work. This way it will be easy to
provide (basic) crypto support also in OCaml-based tools.
Related to: RHBZ#1362649
With the current implementation, the root inode of the given partition
is ignored.
The root inode is now reported. Its name will be a single dot '.'
reproducing the TSK API.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
On Debian family of OSes Grub2 tools are prefixed with 'grub-', not with
'grub2-'. We have to detect the correct name of the tool to use it.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
When creating the XML for the new guest, always put the name of the pool
containing the disks, even when -os specified a pool UUID: libvirt does
not handle pool UUIDs for storage, but only names.
When running in verbose mode for debugging, run `glance --version` to
print the version of the glance client. This helps when dealing with
failures, or making its output actionable.
The glance client v1.0.0 defaults to the Image v2 API: this means that
an image can be referenced only by its UUID, and not anymore by the name
as well (which does not need to be unique). This caused our glance
invocation to set the properties for the newly created image to fail,
since we are using the name as identifier.
Instead of first creating the image and then setting all the properties
for it, set all the properties when creating the image: other than being
simpler, it also avoid parsing the output of the 'glance image-create'
command for the UUID ('id') of the image.
Create a new module [Linux_kernels] which does all kernel detection,
and also provides a place to define the kernel_info data structure.
This is essentially just code motion.
If the guest has no <Name> element in the OVF, previously we chose
"default" as the name. This changes that so it uses a name derived
from the basename of the OVA file instead.
For example:
virt-v2v -i ova /path/to/myguest.ova [...]
would use "myguest" as the name (assuming no <Name> was present).
Modifies the behaviour of
commit 1ae4252c93.
Upstream Perl is going to remove '.' from @INC (the include path for
modules) by default for the next major release (= 5.26) [1], as measure
to fix security issues. Debian already started backporting the fixes
for this [2], thus behaving this way in current Sid installations.
Since the affected Perl sources are only the local daemon testing
scripts, a simple fix is to force the 'requires' for the local
captive-daemon.pm module to start from the current directory: this way
there is no need to manually augment @INC, and only our local module is
loaded automatically.
[1] https://rt.perl.org/Public/Bug/Display.html?id=127810
[2] https://lists.debian.org/debian-devel-announce/2016/08/msg00013.html
Make use of the recently added 'getprogname' module in gnulib: replace
our guestfs_int_program_name with the getprogname() provided by the
module, since it does the same thing, and in a portable way.
As consequence of the above, use gnulib in a couple of tests that use
getprogname().
Since guestfs_int_program_name is gone, drop the configure checks
associated with it.
When running 'make maintainer-check-extra-dist', check also that all
generated files are included in the tarball. This is done so that end
users will not need OCaml to compile from tarball releases.
Previously we used an awkward hack to split up the large src/actions.c
into smaller files (which can benefit from being compiled in
parallel). This commit generalizes that code, so that we pass a
subsetted actions list to certain generator functions.
The output of the generator is identical after this commit and the
previous commit, except for the UUID encoded into tests/c-api/tests.c
since that is derived from the MD5 hash of generator/actions.ml.
This mostly mechanical change moves all of the libguestfs API
lists of functions into a struct in the Actions module.
It also adds filter functions to be used elsewhere to get
subsets of these functions.
Original code Replacement
all_functions actions
daemon_functions actions |> daemon_functions
non_daemon_functions actions |> non_daemon_functions
external_functions actions |> external_functions
internal_functions actions |> internal_functions
documented_functions actions |> documented_functions
fish_functions actions |> fish_functions
*_functions_sorted ... replacement as above ... |> sort
Commit ab2df2e659 attempted to rewrite
/etc/hosts if it contained the old hostname.
However this wasn't done reliably for a couple of reasons:
(1) Certain Debian/Ubuntu /etc/hostname contains
"localhost.localdomain" but still has "unassigned-hostname" or
"unassigned-hostname.unassigned-domain" in /etc/hosts.
(2) Even if (1) doesn't apply, you still need to split out the
hostname and domainname parts and deal with them separately.
Use /etc/os-release as first option, translating the distro name to the
current identifier used. The other options (the release files) are left
as following checks, avoiding them if any matches.
Don't remove the Processor and Intelppm nodes since that just breaks
the device driver.
The only remaining node being removed by the original code was
"rhelscsi" (the xenpv-win driver). I changed this so that instead of
deleting the whole node, it simply disables that driver. If you look
at RHBZ#737600, it's not even clear that deleting nodes was the right
fix for anything. I also renamed the function.
Reviewed-by: Roman Kagan <rkagan@virtuozzo.com>
Signed-off-by: Richard W.M. Jones <rjones@redhat.com>
When put on new virtual hardware Windows will start driver installation
for newly discovered devices.
The problem is that it happens asynchronously and concurrently with
other activities, in particular, the firstboot scripts. This may result
in conflicts (because a firstboot script may want to install or
uninstall a driver, etc.) and eventually in the system left in
inconsistent state.
In order to prevent it, add another firstboot script which calls a
special utility, pnp_wait, whose sole purpose is to wait until all
PnP-related activities are finished. (It does so via a WinAPI call
which isn't available from a script so it has to be a compiled .exe.
The source code for it can be found in the corresponding directory in
https://github.com/rwmjones/rhsrvany). This firstboot script is put
first, so that other firstboot scripts do not have to worry about
interactions with PnP manager any more.
One caveat is that on Windows XP and Windows 2003 the PnP manager always
fires the "Found New Hardware" wizard, which doesn't allow PnP to make
any progress until the user closes it. As a result, this firstboot
script would never finish.
To work it around, follow the Microsoft KB
(https://support.microsoft.com/en-us/kb/938596) and set up a registry
key to make the the PnP manager not fire the wizard; the key is restored
to its initial state upon PnP completion. Unfortunately this only works
on systems having the mentioned update installed; otherwise the user
will have to interact with the UI.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Reviewed-by: "Richard W.M. Jones" <rjones@redhat.com>
It will be used in new code in a followup patch.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Reviewed-by: "Richard W.M. Jones" <rjones@redhat.com>
If multiple tests (eg. from different branches) are running in
parallel, they will attempt to use and delete the same temporary pool.
Use a random pool name to avoid this.
Other functions, such as ``-i ova'', also call Xml.parse_memory.
Therefore the error is not always from XML generated by libvirt, the
XML might be generated from other sources too.
Thanks: Ming Xie.
Following the same style used for the firstboot tests in the
customize/ subdirectory, use a separate test script for each real
guest that we test. This has two advantages: (a) the tests run in
parallel (and are thus faster) and (b) automake "knows" about each
test and reports the results separately.
Methods in OCaml which don't take any parameters don't require the
dummy unit arg, ie writing:
method foo = ...
is fine. The reason you might need the unit arg is if you need to
create a closure from the method without calling it, for example if
you need to use the method in a callback.
In lablgtk2 the convention is to use unitless methods if either: the
method shouldn't be used as a callback; or: (conceptually) the method
doesn't change the object's internal state. Let's do that here.
The name of the initrd image on Debian-based systems is different from
what used on Fedora/RHEL/SUSE and derived; set a different regexp to
avoid making the current regexp even more complex.
Implement the 'remove', 'file_list_of_package', and 'file_owner' methods
of the Linux module for the "deb" package manager (dpkg basically, on
Debian and derived distributions).
Also allow it for the main conversion code.
Move the actual job in an helper function, so the common bits (like the
check of the size of 'packages' and the reload of Augeas) can be done
for all the package manager implementations.
This should be code motion with no behaviour change.
Old perl/POD from Debian Wheezy has a bug where <name>NAME</name>
gives the error:
Unknown E content in E</name>
even though this is in a verbatim section. Fudge the documentation to
avoid this.
In OCaml 3.12, type inference was not as smart and could not use local
information to disambiguate same-named fields in different structures.
The easiest fix for this is just to rename the field so it doesn't
have the same name as a field in the Customize_cmdline.ops struct.
In Debian Wheezy with libvirt 0.9.12.3, HAVE_LIBVIRT was defined, but
HAVE_LIBVIRT_BACKEND was undefined, so src/libvirt-auth.c failed to
compile.
It's easy, and better, to check the minimum version in the configure
script.
Create an object hierarchy to represent different bootloaders for Linux
guests, moving the separate handling of grub1 and grub2 in different
classes: this isolates the code for each type of bootloader together,
instead of scattering it all around.
This is mostly code refactoring, with no actual behaviour change.
String.map was added in OCaml 4.00.0. However we use this function
to implement String.lowercase_ascii etc.
Therefore include a definition of the function for older versions of
OCaml. (Debian Wheezy has OCaml 3.12.1.)
By adding common CLEANFILES and DISTCLEANFILES variables to
common-rules.mk, we can remove these from most other Makefiles, and
also clean files more consistently.
Note that bin_PROGRAMS are already cleaned by 'make clean', so I
removed cases where these were unnecessarily added to CLEANFILES.
There was some code in configure.ac dating back to 2009
(commit b9014d6a0d) which attempted to
add the javac option `-source 1.5'. I don't think this code ever
worked.
However, if -source 1.5 is added, then you get this warning:
warning: [options] bootstrap class path not set in conjunction with -source 1.5
warning: [options] source value 1.5 is obsolete and will be removed in a future release
warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
and that's very hard to reliably fix because you have to know somehow
a magic path. For more details, see:
https://blogs.oracle.com/darcy/entry/bootclasspath_older_sourcehttps://blogs.oracle.com/darcy/entry/how_to_cross_compile_for
Anyway, remove the voodoo from configure.ac.
As a side effect, this allows you to set the EXTRA_JAVAC_FLAGS
environment variable before ./configure with any extra javac flags
that you want.
Add also /boot/efi/EFI/redhat/grub.conf as configuration of Grub 1;
since the "grub" lens of Augeas does not handle this path, add a
transformation so Augeas can parse it.
Expose the aug_transform API through the library, so it's possible to
add/remove Augeas transformations to handle files in custom places using
existing lenses.
Add new scripts for:
- guestunmount
- virt-copy-in
- virt-copy-out
- virt-customize
- virt-dib
- virt-diff
- virt-get-kernel
- virt-p2v-make-disk
- virt-p2v-make-kickstart
- virt-p2v-make-kiwi
- virt-tar-in
- virt-tar-out
Also combine the separate virt-resize script into the general script
virt-alignment-scan. There wasn't really any reason to have separate
scripts.
By moving these two functions out of the common options parsing code,
it means we don't need to depend on all the other machinery of options
parsing, such as the global variables ("verbose"), libconfig, etc.
The code to parse btrfs subvol entries in /etc/fstab failed if the
entry had more than one comma-separated option, for example:
/dev/sda4 /home btrfs rw,user,subvol=foo 0 0
This commit fixes that code to use Augeas correctly.
Fixes commit 7ba0e10501.
Reported by: Zhongfu Li
https://bugs.launchpad.net/ubuntu/+source/libguestfs/+bug/1615337
Libvirt >= 2.1.0 now allows you to open files which have a "json:"
QEMU pseudo-URL as backingfile, whereas previously it would fail hard
in this case (RHBZ#1134878).
When virt-v2v performs conversions from Xen (over SSH) or vCenter
(over HTTPS) it uses these pseudo-URLs as backingfiles. We had to
tell people to use LIBGUESTFS_BACKEND=direct to avoid libvirt in this
situation.
This commit narrows the check so it will now only print the error if
libvirt < 2.1.0 and LIBGUESTFS_BACKEND=direct is not set. Also the
error is modified to tell users they can either upgrade libvirt or set
LIBGUESTFS_BACKEND=direct to work around the problem.
Note there is not an easy way apart from checking the version number
of libvirt to tell if the json pseudo-URL is supported.
As a side-effect, this commit also prints the libvirt version number
in debugging output when virt-v2v starts up, which is sometimes useful
information for narrowing down bugs (it is in fact already printed by
libguestfs, so this is duplicate information, but it's a bit easier to
find when it's at the top of the debug).
Thanks: Peter Krempa.
Make 'which' gracefully handle the case where $PATH is not set
(it will raise Executable_not_found, but that is the expected thing to
do).
Related to RHBZ#1367839.
From RHEL 7.3, Red Hat have decided to only compile the secure boot
version of OVMF on x86-64, with flags -D SECURE_BOOT_ENABLE -D SMM_REQUIRE.
The filename has also changed to reflect this - it is now
/usr/share/OVMF/OVMF_CODE.secboot.fd. The old file
/usr/share/OVMF/OVMF_CODE.fd is no longer shipped.
However switching to using this variant of OVMF for UEFI guests is not
just a matter of changing the filename. The new OVMF variant won't
run unless we also change:
- The qemu machine model, from the default ("pc" ==
"pc-i440fx-rhel7.3.0" or later) to Q35 ("q35" == "pc-q35-rhel7.3.0"
or later).
- Add <smm> under <features>.
- Set <loader ... secure="yes">.
NB: On RHEL the changes requires qemu-kvm-rhev. It is no longer
possible to convert UEFI guests using the basic qemu-kvm.
Thanks: Laszlo Ersek, Ming Xie.
Previously we had a list of UEFI paths in src/uefi.c, which was
accessed in virt-v2v using a private (guestfs_int_*) API and some C
binding code. This was clumsy and required the paths to be replicated
in the virt-v2v unit tests.
Instead just generate the list of paths from the generator, creating
src/uefi.c and v2v/uefi.ml with the same content.
Remove the C bindings and the virt-v2v unit tests associated with UEFI
paths.
This *may* be required by some filesystems in order for fstrim to
work. I'm not actually sure if this is true, but it's what
virt-sparsify --in-place does, and that utility has been tested a lot
more in regards to trimming.
This reverts the change made for RHBZ#1168144. The warning is now
always displayed.
It would be nice to make the warning actionable, but there is not a
lot that end users can do since fstrim is such a complex topic
interacting with all filesystem and storage layers.
Do not print warning for 'sr' devices when converting fstab. Not all
systems create the /dev/cdrom symlink for SCSI CD-ROM devices. Moreover,
on systems with multiple CD-ROMs, having entries for /dev/sr* devices
may be inevitable.
RWMJ: Use String.is_prefix instead of String.find, to more accurately
match on the device name.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
Signed-off-by: Richard W.M. Jones <rjones@redhat.com>
Conversion from Citrix Xen has now been tested and should work. Remove
the comment about it from the documentation.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
The replacement string was wrong. There are only two match groups in the
regular expression, not three.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
The Invalid_argument exception is there to catch unexpected situation
when rpm returns no output. Such situation should be reported rather
then hidden.
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
Because we run the external fstrim command we don't have access to the
kernel errno when it fails. However in the case where it prints this
specific error message, turn that into errno ENOTSUP.
Add makefile variables to enable silent rules for simple command
invocations, such as ocamlc, ocamlopt, javac, and erlc.
This reduces the log output when building with silent rules, still
showing the full command lines otherwise.
Allow to specify the format of the helper drive, to avoid relying on the
format autodetection. This is the same approach used also in all the
other tools.
Turn the rc.d symlinks for the guestfs-firstboot service as relative,
instead of absolute paths: the result is the same (the service works the
same), and this way is more coherent with symlinks created by
update-rc.d.
Currently we install a systemd service named firstboot.service and a
SysV service named virt-sysprep-firstboot. On systems where systemd is
the init system and runs with the SysV compatibility, the different
names make systemd handle them as different services, and thus trying to
run the firstboot script runner twice.
Rename both the systemd service and the SysV one to guestfs-firstboot:
the new name is less generic, and allows the systemd service to be
shadowed by the SysV service (and thus running just once).
Also cleanup the old services: the old SysV service can be removed
directly, since its former name had "virt-sysprep" in it, and so there
could not be much room for confusion and conflict. Regarding the old
systemd service: to avoid leaving it behind, a simple cleanup strategy
is in place, checking the content of the old firstboot.service to really
ensure we are removing one of our versions of this file.
Export FS_TYPE for the elements, so they can know what is the filesystem
created.
This was implemented in diskimage-builder upstream as
commit be521bdec6233530cc952be246a66f1957b0dd58.
Implement the "docker" output format, to import an image into docker.
This was implemented in diskimage-builder upstream as
commit 1187f9d5df6d6eddba478ac75d3834391480755f and
commit edc06a20e57bdf0a9a03949b54395008d73520dc.
Instead of running them before lanching the appliance with the disks and
then uploading the result after root.d hooks run, mount the root in the
local temporary directory using FUSE and then run the hooks on the
guest. Other than being closer to what diskimage-builder does, it also
avoids issues with the extra-data.d scripts assuming to find things
already, and we don't error out while trying to unpack files on the
guest.
Since virt-dib requires FUSE now, build it only if FUSE is supported.
run_command uses Unix.create_process which forks a child process, and
executes execve: the latter fails when the executable does not exist,
triggering the exit which, in older OCaml versions [1], also runs the
at_exit handlers.
Since there is not much that can be done to avoid this on the OCaml
side, to keep run_command working also in older OCaml version then
manually search for the existance of the given executable, exiting with
code 127 (as a shell does) in this case.
[1] http://caml.inria.fr/mantis/view.php?id=7209
If the user doesn't specify a format (ie. they want autodetection),
and the drive is set to read-only, previously we created a qcow2
overlay to protect the drive but didn't set the backing_fmt field.
However libvirt disabled this so now libvirt doesn't work at all on
qcow2 files that have no backing_fmt field set.
We must therefore do autodetection in libguestfs and use that to set
the backing_fmt field.
We were already doing autodetection in the non-readonly case, so this
commit refactors that code into a new function
("get_source_format_or_autodetect") and uses it on the readonly path
too.
Because we didn't give the type of 'g' there wasn't sufficient
information for the type checker to pick up this problem. By
specifying the real type of 'g' in a type annotation, that reveals the
problem.
Thanks: Pino Toscano, Nyoxi on IRC.
What was happening in file_owner function did not match the description
in the comment. When a path is owned by multiple packages the returned
string was in fact a concatenation of the names of all packages that own
it. E.g. for `Linux.is_file_owned g inspect "/etc"` the returned value
was "filesystemyum" (i.e. "filesystem" + "yum").
Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
$OSINFO_SYSTEM_DIR (or its /usr/share/osinfo fallback) represents the
top-level directory of the osinfo data, while we need to read the "os"
subdirectory of it.
Fixes commit c2ae46a9eb.
Directly query the attributes in the media node, instead of using an
XPath expression to read it.
Saves ~10ms when parsing the libosinfo DB on a Fedora 24 installation
(other than making the code easier).
Check the presence of Gtk properly depending on the value of --with-gtk:
if a specific version is chosen, then let PKG_CHECK_MODULES fail if that
version if not found, otherwise fallback from gtk3 to gtk2 to no gtk.
Also move few common AC_SUBST in a single place.
It looks like flex 2.6.1 changed [1] the return code for EOF in
yyinput. Therefore, use the right value depending on the version of
flex which generates the lexer.
[1] f863c9490e
Setup the volatile /run in the appliance also with the tmpfiles
configurations available. In particular, setting up correctly the lvm
bits allow lvmetad to run.
Commit 2e16e3e993 added lv_active=active
as additional condition when listing LVs, to ignore those with the
activationskip flag set. OTOH, this check is too broad, and matches
also other kind of LVs.
Change the condition to lv_skip_activation!=1, so matching precisely
what was meant, and only that.
Related to: RHBZ#1306666
Currently lvmetad is started in init, and thus using the system
(= appliance) configuration of lvm. Later on, in the daemon, a local
copy of the lvm configuration is setup, and set it for use using the
LVM_SYSTEM_DIR environment variable: this means only the programmes
executed by the daemon will use the local lvm configuration, and not
lvmetad.
Thus manually start lvmetad from the daemon, right after having setup
the local lvm configuration, and still without failing if it cannot be
executed.
Additionally, since lvmetad now respects the right configuration, make
sure to update its cache when rescanning the VGs by passing --cache to
vgscan.
When editing the lvm configuration to set the LVM filter, edit the
'global_filter' key in addition to 'filter': the latter is not used when
lvmetad is running, when only the former works.
Commit b91b39e06a changed the separator to
':', although this creates parsing issues when there are fields with
colons (for example lv_tags=imgbased:pool).
Change once more the separator, but this time using a non-printable
character such as '\r': while it will produce uglier debug logs, this
should greatly reduce the possibilities of conflicts with texts of
metadata.
Switch to guestfs_int_version_from_x_y_or_x to parse version numbers --
although, restrict the parsing to what could look like a valid version
number, to discard quickly version strings like "unknown".
This makes sure that Debian, Fedora, Mageia, and SLED ISOs have the
right version number.
More recent versions of libosinfo switched the internal directory with
the XML files of OSes to a different layout (still with the same XML
format), causing libguestfs to not read them anymore. Furthermore, the
internal directory is going to disappear soon, replaced by a public
osinfo database [1].
Revamp the way libguestfs reads the data: first try the upcoming osinfo
layout, falling back to the current libosinfo layout (which is the same
as osinfo), and then to the old flat layout.
[1] https://gitlab.com/libosinfo/libosinfo/blob/master/docs/database-layout.txt
-Wnull-dereference and -Wshift-overflow are new warnings in GCC 6, so do
not try to disable them with pragmas on older GCC versions.
Fixes commit a8e15ea924.
One -Wnull-dereference warning is real: we deliberately cause a
segfault in one of the tests.
There is a -Wshift-overflow bug in a Gtk 2 header.
The others are the result of shortcomings in GCC.
In all cases we have to add GCC diagnostic overrides to ignore
the warnings when compiling with ./configure --enable-werror.
The commit claimed that SELinux was disabled in the
virt-p2v-make-kickstart script, and so this was just making things
consistent. However this is not true. SELinux *is* enabled in the
kickstart version. There is some code to disable it, but it is
commented out (admittedly it's hard to tell because the code is buried
in a multi-line sed expression).
The commit message also claimed (this time correctly) that the
'--selinux-relabel' step caused an extra reboot each time virt-p2v is
run from the ISO. However we have recently fixed this so it can
usually do the relabelling during the build, not at boot time, so this
is no longer a problem.
This reverts commit 25ffcc7d43.
For OCaml tools this does essentially nothing useful because the
--help output is automatically generated from the options, and so
cannot be wrong. However for C tools this is a useful check.
It would be nice to generate C tools --help output, but there isn't
enough information in the getopt data to do that.
This commit also includes fixes to the --help output for a few tools.
This function allows to download file system data units (blocks) from
the given partition.
The API can be used to detect data hidden within filesystem bad blocks
or slack space.
Moreover for filesystems such as Ext3 and Ext4, this function is the
only way to retrieve deleted files. An example is given in the function
tests.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
Pass -q to virsh if virt-v2v was run with -q, to reduce its output in
quiet mode.
Unfortunately this does not currently work in virsh as it should, see
RHBZ#1358179, so only after that bug is fixed this change can actually
take effect.
ocamlmklib -g does not work in OCaml 4.01.0. It does work (and is
desirable) in OCaml 4.02.2. However since we would like to support
back to OCaml 3.11, we cannot use it.
In libguestfs 1.33.43 we managed to release a completely
non-functional virt-customize binary (it ran without error but did
nothing at all, whatever command line arguments you gave it).
Improve the test so this cannot happen again.
podcheck.pl is run as part of the tests to perform various checks on
the documentation and the tool.
Currently we check only that the documented options matches the
options that the tool implements and vice versa. This commit would
also allow us (in future) to check --help, --long-options,
--short-options, --version output.
This commit includes scripts to run the tests and various fixes to the
manual pages to ensure that the tests pass.
For guestfish, guestmount, remove '?' from short options. Currently
those tools don't process -?, so I believe these are erroneous:
$ guestfish -\?
Try `guestfish --help' for more information.
For virt-format, the -c, -d and -q options are removed. These options
just give errors because they appear in the short options list but not
in the case statement.
Store them directly in List_entries, an adding helper function to
convert from string.
Also use Getopt.Symbol for them, so there is no need to manually reject
invalid formats.
Instead of linking with individual objects, which is very tedious,
build a proper library and link virt-builder, virt-customize and
virt-sysprep to it.
This makes the binaries a tiny bit smaller because .cmxa/.a files
allow unused code to be removed by the linker, whereas explicitly
linking .cmx/.o files does not.
Instead of linking with individual objects, which is very tedious,
build a proper library and link the other tools with it.
This doesn't make the resulting binaries any larger.
This implements the --selinux-relabel option for virt-customize,
virt-builder and virt-sysprep. There is no need to autorelabel
functionality now.
Thanks: Stephen Smalley
Sadly, the dhclient-script shipped as part of isc-dhcp-client in Ubuntu
unconditionally reads from /etc/fstab without checking for its
existence. Since no package holds /etc/fstab, this file will not exist
in the appliance, cause dhclient to fail (actually keep looping calling
the failing dhclient-script) when the network is requested.
As a workaround, touch /etc/fstab just before enabling the network: if
that file exists nothing changes, while an empty file will be available
in the other case, making at least dhclient-script in Ubuntu working.
Since the function uses the StringMap module, I also had to
move this to mllib.
The function is renamed and the 'if verbose ()' part is moved into the
function, but apart from that, it's all just code motion.
Add a dummy description value to mark an option as "hidden", so it will
not be shown in the help text.
Mark few options as hidden:
- common: --short-options, --long-options, --debug-gc
- virt-sysprep: --dump-pod, --dump-pod-options
Since --debug-gc is now really considered internal, it is no more
documented.
Add a new Getopt module to mllib, to parse command line arguments with
handlers close to the ones used with Arg, but using getopt(3) (actually
getopt_long_only) to do the real parsing. This allow us to provide
options for OCaml tools with a syntax similar to the C tools, and use
the additional features getopt offers and Arg does not.
Getopt now handles every part of the command line handling, including
the output of short & long options.
Do a single-step conversion of Common_utils and all the OCaml tools to
the syntax of Getopt.
Move a couple of utility functions from Common_utils to Getopt, since
they fit better there (and Common_utils cannot be used in Getopt, as
the former already uses the latter).
As side-change due to the conversion, extra arguments for sysprep
operation can have more keys for the same argument.
In previous commit c5f12e47e4 we changed
the PATH environment variable to run the locally built virt-v2v, not
the system-installed virt-v2v.
However this meant that we were running the locally built virt-v2v
against the system-installed libguestfs.so.0. We also need to set
LD_LIBRARY_PATH.
Fixes commit c5f12e47e4.
In the case where we are building a Python dist (which can be
distributed separately), we generated calls to non-existent optargs
structures if an API was once no-optargs but had optargs added. A
typical error was this (library: 1.32.5, Python sdist: 1.33.42):
guestfs-py.c: In function ‘guestfs_int_py_glob_expand’:
guestfs-py.c:9648:40: error: storage size of ‘optargs_s’ isn’t known
struct guestfs_glob_expand_opts_argv optargs_s;
This is a simple solution to this, although it has the possible
disadvantage that such functions will not be bound at all.
Since commit 83e92b4a97, utils.c
includes "ignore-value.h". We copy utils.c into the python sdist
tarball, but it didn't not compile because of the missing header file.
Therefore we need to copy the header in too.
Fixes commit 83e92b4a97.
guestfs_set_uuid wants device as argument.
Moreover, btrfstune -U is unable to set uuid for subvolumes,
only unmounted partitions are supported.
Here we skip btrfs subvolumes.
Calling guestfs_is_lv on btrfs subvolume throws an error.
Here we workaround it by taking Mountable instead of Device
and returning 'false' for non-device mountables.
Access, modification, last status change and creation time in
Unix format as for statns.
Number of links pointing to a given entry.
If the entry is a symbolic link, report the its target path.
A new flag (DIRENT_COMPRESSED 0x04) indicating whether the file is
compressed using native filesystem compression support.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
This adds imperative list manipulation functions inspired by Perl.
The functions are passed list refs which get updated in place.
This allows us to replace some awkward pure functional code like:
let xs = ys in
let xs = if foo then xs @ zs else xs in
with:
let xs = ref ys in
if foo then append xs zs;
These are now passed using a curl configuration file, which is a
little bit safer than using command lines. virt-builder doesn't need
to pass usernames and passwords to curl, but if it ever does in future
this will be a lot safer.
Change the Curl module to use an ADT to store the name of the curl
binary and the arguments.
The callers in virt-v2v are changed accordingly.
This also adds a (currently unused) ?proxy argument to allow callers
to override the proxy. It also adds some safety arguments implicitly.
Declare most of the stringsbuf as CLEANUP_FREE_STRINGSBUF, so they are
freed completely on stack unwind: use take_stringsbuf() in return
places to take away from the stringsbuf its content, and remove all the
manual calls to free_stringslen (no more needed now).
This requires to not use free_stringslen anymore on failure in the
helper functions of stringsbuf, which now leave the content as-is (might
be still useful even on error).
This allows us to simplify the memory management of stringsbuf's, which
are not properly fully freed, fixing memory leaks in some error paths
(which were not calling free_stringslen).
If add_string_nodup fails free the passed string instead of leaking it,
as that string would have been owned by the stringbuf.
Adapt few places to this behaviour.
We have to download the old 32 bit explorer.exe in order to find the
icons as the 64 bit version doesn't contain any icons (where are they?).
Thus prefer the 32 bit (WoW64 subsystem) directory if found.
Fixes commit 7f16c346bb.
Thanks: Xiaoyun Hu
The current location doesn't exist unless you've installed GNOME,
which is not so common on Ubuntu. Unfortunately I couldn't find any
other location containing a clean, high quality logo.
This adds another low quality icon source, and also prevents any icon
being returned if the highquality flag was set (note this prevents
virt-manager from displaying an icon, but there's nothing we can do
about that, and it's no worse than the current situation).
Updates commit 1d0683964f.
Thanks: Xiaoyun Hu
This is now enabled by default and cannot be changed/disabled by the
user. The output is saved into /tmp/.../virt-v2v-conversion-log.txt
on the conversion server.
This makes sure that in the internal configure_kernel_modules function,
for virtio or SCSI block types:
a) the warnings about leftover Xen modules are printed
b) the changes in Augeas are saved
Fixes commit ee02191483.
First check for the existence of the directory /etc/modprobe.d, in case
using a file under it; this also skips all the other checks, since they
are not needed at all. Also /etc/modprobe.d exists on recent Linux
versions, so let's give priority to the more common methods.
When /etc/modprobe.d does not exist, check for the file to edit using a
single list of possible files, now in order of priority, where the first
find is used without checking further for the rest.
Also, make sure all the returned paths are absolute: they are used in
Augeas paths later on, so relative paths will not do anything useful.
We are now able to cancel the conversion instantly by sending ^C to
the remote virt-v2v process.
Also, this reverts:
"p2v: Poll to make Cancel Conversion button more responsive."
(commit 6da4941db7)
Previously we copied / creates files in the remote dir by running
complex shell commands like:
cat > file <<'EOF'
## the file was copied in here
EOF
This was a little hairy, but in particular it doesn't allow us to set
the ssh session to cooked mode (so we can send ^C to cancel a
conversion).
A cleaner way to do it is to use 'scp' to copy the files over.
The messages that come "through" from virt-v2v are already colourized.
However the other messages are not. This colourizes the ones
generated during kernel-mode conversions.
Note that because of the way journal/syslog works we have to write the
ansi_restore before the end of line, thus code like this:
ansi_magenta (stdout);
printf ("%s: %s", guestfs_int_program_name, data);
ansi_restore (stdout);
putchar ('\n');
This also adds a "Conversion finished successfully" message (in green).
Thanks: Ming Xie
Add powerpc64 (ppc64) and powerpc64le (ppc64le) cases to the
normalize_arch function. Also remove a few duplicate cases which are
already caught in the catch-all case at the end of the match.
In kernel conversion mode, we must ensure messages are flushed to the
journal as soon as they are received from virt-v2v, otherwise (now
that debugging is not the default) we won't see any messages at all
until the last moment.
Thanks: Ming Xie
This allows you to install extra packages in the disk/ISO. The
implementation of this option in virt-p2v-make-disk was particularly
simple and followed naturally from the previous commit.
- sanitize the logs, removing terminal color codes and carriage returns
- add an empty "properties" node
- put the log for skipped tests as "message" attribute, hoping it is
read from there
- do not blacklist the log of test-virt-rescue.pl, which should not
cause issues now
If an ssh server is completely unresponsive (ie. it doesn't even send
back ICMP errors), then eventually ssh times out after ConnectTimeout
seconds [ConnectTimeout is an ssh option].
Unfortunately the miniexpect timeout (default 60 seconds) was less
than ConnectTimeout so we never saw the ssh error. Instead we would
see the internal error:
remote server closed the connection unexpectedly,
waiting for: password prompt
This commit sets the ssh ConnectTimeout to an explicit value, and sets
the miniexpect timeout to be 20 seconds larger than ConnectTimeout, so
that in most cases we will see the ssh error message before miniexpect
times out.
Thanks: Ming Xie for finding and diagnosing the problem
In the GUI disks block, instead of using separate "Size" and "Model"
columns, combine that information into a markup column under "Device".
To make the GUI look consistent I also had to change the removables
"Device" column to be a markup column and embolden the device name
there too.
This is just refactoring, but it reveals and fixes a bug too. The
size_gb field was left as NULL when the --test-disk option was used.
Apparently passing NULL to gtk_list_store_set is fine, but just in
case I replaced it with "".
The error you would see is:
virt-v2v: error: insufficient free space in the conversion server temporary
directory /var/tmp (853.8M).
Either free up space in that directory, or set the LIBGUESTFS_CACHEDIR
environment variable to point to another directory with more than 1GB of
free space.
See also the virt-v2v(1) manual, section "Minimum free space check in the
host".
Also adds some documentation.
Thanks: Ming Xie and Xiaodai Wang
There is a tiny functional change in this patch, since overlay_dir is
now always evaluated once (eg. even in --inplace mode), whereas
previously it was evaluated twice but only in copying mode.
As described in the comment, this solves a number of problems with
non-standard remote configurations.
I tested this with:
- tcsh
- zsh
- ksh
and all behaved correctly.
Print a better error message when the non-root user on the conversion
server requires a password to use sudo, and p2v is told to use sudo.
See also RHZ#1340809.
Library's counterpart of the daemon's internal_filesystem_walk command.
It writes the daemon's command output on a temporary file and parses it,
deserialising the XDR formatted tsk_dirent structs.
It returns to the caller the list of tsk_dirent structs generated by the
internal_filesystem_walk command.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
The tests check whether the filesystem_walk command is able to retrieve
information regarding both existing and deleted files.
A NTFS image is used as Ext3+ filesystems deletion is more aggressive
in terms of metadata removal.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
- generator: Added tsk_dirent struct
The tsk_dirent struct contains the information gathered via TSK APIs.
The struct contains the following fields:
* tsk_inode: inode of a file
* tsk_type: type of file such as for dirwalk command
* tsk_size: file size in bytes
* tsk_name: path relative to its disk partition
* tsk_flags: bitfield containing extra information
* tsk_spare[1-5]: extra space for future usage
- configure: Added libtsk compile-time check
Ensure libtsk is available at compile time.
If not, daemon routines depending on it won't be available.
- API: internal_filesystem_walk
The internal_filesystem_walk command walks through the FS structures
of a disk partition and returns all the files or directories
which could be found.
The command is able to retrieve information regarding deleted
or unaccessible files where other commands such as stat or find
would fail.
The gathered list of tsk_dirent structs is serialised into XDR format
and written to a file by the appliance.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
The previous code treated floppy disks and CD-ROMs as the same kind of
thing, resulting in malformed libvirt XML. You would see the
following error when importing a guest into libvirt:
error: Failed to define domain from /tmp/v2vlibvirt063486.xml
error: internal error: Invalid floppy device name: hdb
because we incorrectly generated this bogus libvirt XML fragment:
<disk device='floppy' type='file'>
<driver name='qemu' type='raw'/>
<target dev='hdb' bus='ide'/>
</disk>
This commit models floppy devices as a distinct type, occupying their
own bus ("/dev/fdX"). When writing to libvirt, we generate correct
XML fragments, looking like this:
<disk device='floppy' type='file'>
<driver name='qemu' type='raw'/>
<target dev='fda'/>
</disk>
Miscellaneous other changes were required in the code. There is also
a regression test (see following commit).
Note this ignores floppy disks in '-o qemu' mode.
aarch64 image isn't buildable at the moment because of
Anaconda bug RHBZ#1348980.
ppc64 & ppc64le images are not buildable because of delays updating
the Fedora mirrors.
This adds <BootOrder>1</BootOrder> to the first drive, which is
necessary to make the drive bootable (or would be necessary, if oVirt
hadn't implemented their own workaround for the OVF we were previously
generating).
https://bugzilla.redhat.com/show_bug.cgi?id=1308535#c11
This is not a full solution to the problem. Really we should be
copying the boot order over from the source hypervisor, but that
requires extensions to libvirt as discussed here:
https://bugzilla.redhat.com/show_bug.cgi?id=1308535#c7
Thanks: Shmuel Melamud
e2fsck returns 1 in case of "file system errors corrected".
We treat it as success in normal e2fsck, but fail if e2fsck
is run by resize2fs.
Change 'manual' execution of e2fsck to dedicated function call.
On distros not running NetworkManager, use the nm-online -x parameter
to exit quickly. The network connection will be checked just after
anyway when attempting to connect to the conversion server using ssh.
Kiwi is the tool used by openSUSE / SLES to generate many sort of
disk images. Add a virt-p2v-make-kiwi tool and his documentation to
geneate the p2v appliance kiwi configuration.
Use the common denominator for SLES and openSUSE in the dependencies.
For example most of NetworkManager pieces and metacity aren't provided
on SLES.
ifconfig is in the net-tools-deprecated package in openSUSE Factory,
which means after openSUSE Leap 42.2. Older versions have it in the
net-tools package. Adding this complexity only to add ifconfig because
sysadmins prefer it isn't too good: iproute2 is doing the job.
The password prompt is actually generated by the openssh client which
is under our control, so we can regexp match on the specific prompt
printed by openssh. This also avoids mistaking any server-side
issue.net message for a password prompt (eg. if the server prints a
message like "all users must change their passwords today!") which
would have prevented virt-p2v from logging in.
This can fail because the username is wrong. Also don't unnecessarily
reveal irrelevant implementation details in the error message, just
say the login failed.
Thanks: Juquin Zhou
Previously cancelling the conversion only set a flag, which was
checked when the run dialog displayed new output from virt-v2v. When
virt-v2v was showing hundreds of debugging messages, this wasn't a
problem, but now that we are hiding those messages, cancelling the
conversion might mean a wait of seconds or minutes.
By polling (albeit infrequently) we can make the cancel button more
responsive.
For some failures, 30 lines was not sufficient and only part of the
error was shown. Increase the number of lines shown to 50.
This also colourizes the failure message, and prominently displays the
location of the full log. The following message is now shown:
*** virt-v2v command failed ***
The full log is available on the conversion server in:
/tmp/virt-p2v-20160620-sga9rhk7/virt-v2v-conversion-log.txt
Only the last 50 lines are shown below.
[followed by up to 50 lines of log]
Updates and fixes commit 7447fe2478.
PLD Linux got /etc/os-release only in the recent 3.0 release; since
older versions have only /etc/pld-release, check for it to identify
the guest and get its version.
Previously we displayed the complete output of virt-v2v in the run
dialog. This output included all the debugging messages, and was very
long and confusing for users (especially we had false bug reports
about "errors" appearing in the debug output).
Only display stdout in the run dialog. However make sure everything
(stdout and stderr) is still logged to the conversion log.
Instead of constructing and directly executing a long virt-v2v command
line, build a wrapper script with the same command line and send it to
the remote server (stored in /<remote_dir>/virt-v2v-wrapper.sh).
This will make it a bit easier to construct more complex virt-v2v
wrappers.
Note this commit on its own is a simple refactoring and does not
change any functionality.
This option (alternately spelled: --color, --colour, --colors, or
--colours) enables ANSI colour sequences output even if that would be
disabled becaues the output is not a TTY.
The debug() function is already sending these to stderr, but in a few
places we were using printf. Change those to eprintf, except for one
informational message which should have been using info().
The virt-p2v slow test tested the local virt-p2v. However it ran
against the installed virt-v2v (and in fact would have failed if
virt-v2v was not installed).
This commit sets the environment up to run the locally built virt-v2v.
This is unfortunately quite a lot more complex than it should be.
There is no simple "set this environment variable" option in
sshd_config.
It makes no sense. By setting one field to non-sensitive when the
other field is populated, we can avoid this happening, and also make
tabbing between the fields simpler.
As a consequence of making this change, I also got rid of the now
unnecessary explanatory text telling you to leave one field blank.
In particular if you hit the 'Test Connection' button when the server
name field was empty, you would still see a stopped spinner.
This updates commit 036a11f379.
By changing the logging mode of the p2v.service unit, we can make sure
that the progress of virt-p2v is visible in non-GUI mode at the
console.
Since the output is now always shown on the console, there's no need
to print the journalctl command for following that output when the
user logs in.
This simulates doing a non-GUI conversion with virt-p2v being
downloaded over PXE. Conveniently we already had a test that was
doing this, and this rule just runs the same test.
Instead of throwing away the ssh error and printing the generic
message "unexpected end of file waiting for password prompt", we
capture the ssh error and print it. The user will see the ssh
diagnostic, eg. "No route to host".
For some reason I don't quite understand, in
commit fd82bb12fd if it got the password
prompt unexpectedly a second time then it would send the password
again, continually until ssh closed the connection because of too many
retries.
This obviously makes no sense. If we get the password prompt back,
then we must have sent an incorrect password so abort the connection
and report the error at once.
OCaml 4.02 introduced the 'bytes' type, a mutable string intended to
replace the existing 'string' type for those cases where the byte
array can be mutated. In future the 'string' type will become
immutable. This is not the default now, but it can be forced using
the '-safe-string' compile option.
This commit changes the code so that it could be compiled using
'-safe-string' (but does not actually make that change).
If we detect OCaml < 4.02, we create a dummy 'Bytes' compatibility
module ((nearly) an alias for the 'String' module). The only
significant difference from upstream OCaml is that you must write the
'bytes' type as 'Bytes.t' in interfaces, apart from that everything
else should work.
Removes this warning:
File "common_utils.ml", line 826, characters 24-28:
Warning 40: F_OK was selected from type Unix.access_permission.
It is not visible in the current scope, and will not
be selected if the type becomes unknown.
In commit ae6f726ecc we started to use
the virt-customize code to replace various virt-sysprep operations.
This had the effect of adding many more possible operations to
virt-sysprep, but some of them (specifically --install) did not work
unless the appliance network is enabled. It was not enabled in
virt-sysprep, so these operations never worked.
This change does NOT enable the network by default. However it adds a
--network flag which may be used in conjunction with --install etc to
make those commands work.
In addition we now emit a warning for certain customize operations if
they fail and if the network is not enabled. It will print:
[ 4.5] Installing packages: tcpdump
Error: Cannot retrieve repository metadata (repomd.xml) for repository: base. Please verify its path and try again
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os&infra=stock error was
virt-sysprep: warning: the command may have failed because the network is
disabled. Try either removing '--no-network' or adding '--network' on the
command line.
virt-sysprep: error: yum -y install 'tcpdump': command exited with an error
(If the network is already enabled, or if the command is successful,
then the warning is not printed.)
Thanks: Xianghua Chen
You will now see an error such as:
$ virt-customize -a centos-6.img --truncate-recursive /home/foo
[ 0.0] Examining the guest ...
[ 16.5] Setting a random seed
[ 16.5] Recursively truncating: /home/foo
virt-customize: error: libguestfs error: find0: /home/foo: No such file or
directory
Thanks: Xianghua Chen
Commit 82df768514 breaks the regression
test for RHBZ#1232192 because g#part_list can return an error when
called on a completely blank disk.
By rewriting and simplifying the code using higher-order functions we
can avoid the need for the double-nested imperative loop and calling
g#part_list at all.
Fixes commit 82df768514.
The 'semodule' command, called by the guest tools uninstaller, will
run out of memory with the default of 500MB
(https://bugzilla.redhat.com/426775).
Allocate a large amount of RAM, that will be at least 800MB, but scale
correspondingly on other architectures like ppc64 that have larger
memory requirements.
Thanks: Pavel Butsykin
This is a mostly mechanical renaming of the internal pointer types
used in the libxml2 bindings. Also I have added a longer comment to
make the system clearer.
Guest tools for Windows in Parallels / Virtuozzo Server 6 contain a
paravirtualized video driver, which wants to take full control of the video
mode. To avoid excessive video mode switches in UEFI VMs it executes
"bcdedit /set {current} graphicsmodedisabled true".
When such a VM is imported into a QEMU/KVM-based hypervisor, the
standard video drivers in Windows do not set the video mode as they
expect it to have already been configured by the bootloader. As a result,
the VM runs with black screen.
So patch the BCD store (which is a Windows registry hive with a special
structure, located on the EFI system partition) to clear this setting.
Signed-off-by: Pavel Butsykin <pbutsykin@virtuozzo.com>
Store the list of EFI system partitions on the inspect object in order to be
able to tune their contents later in the process.
Signed-off-by: Pavel Butsykin <pbutsykin@virtuozzo.com>
Guest tools for Linux in Parallels / Virtuozzo Server 6 come with a script
that can be used to uninstall them.
Signed-off-by: Pavel Butsykin <pbutsykin@virtuozzo.com>
'tc_410_close_event.rb' was not being run. You could prove this by
simply inserting "exit 1" into that test.
The reason is unclear, but by renaming every class and method in the
tests to be unique, this ensures the tests are run.
Parallels proprietary hypervisor uses RDPMC as the hypercall
instruction. As this instruction is supported since early P6 family,
the drivers didn't even bother to check for the presence of the
corresponding feature in CPUID.
In QEMU/KVM, however, this instruction triggers #GP unless the VM is run
with PMU (performance monitoring unit) enabled, which is often not the
case (due to its impact on perfromance and migratability).
So, to prevent crashes upon conversion, stop respective drivers from
loading by disabling the corresponding services. Note that the services
are being disabled ("Start" value set to 4) rather than their node
removed, to avoid confusing the uninstaller which is scheduled to run
from a firstboot script.
In addition, prl_strg (storage filter driver) is unlinked from the
storage subsystem following the DelReg directive from its .inf file,
otherwise Windows crashes with BSOD 0x7b due to missing dependency of
the storage subsystem.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Some virtio-win drivers contain more files than just *.{cat,inf,sys}.
They are filtered out currently, which prevents the drivers from being
installed by PnP.
Stop filtering driver files by extension, and copy all of them instead.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Use this command as a starting point:
git shortlog -s -- p2v v2v | awk -F'\t' '{print $2}' | sort -f
However note the above will include people who made only incidental
code changes, and also misses out QA team members, so the actual list
still has to be curated by hand.
It's better to have this overview of how the code is arranged before
any details about modifying the code, since most people will be
reading code before writing.
Recent updates of Virtuozzo Server 6 (fka Parallels Cloud Server 6) have
"Virtuozzo tools" instead of "Parallels tools" as their display name in
the corresponding Uninstall registry key.
Recognize those, too, and schedule their uninstallation on first boot.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
On Linux 4.7.0 the old code was giving -EINVAL errors. This rewrites
the code based on online sources, and it works on at least RHEL 7,
Linux 4.4 and 4.7. Also checked with valgrind.
Because EXTRA_DIST included $(TESTS) this ended up distributing the
binary file v2v_unit_tests.
Unfortunately to fix this we need to list all the test scripts and
data files explicitly a second time, but there we are.
If this flag is not set (ie. we are doing local testing) then we
disable some dangerous features. Currently this is just the "Reboot"
button, but may include other features in future.
Currently virt-p2v requires Gtk 2. This commit changes virt-p2v so it
can be built with either Gtk 2 or 3.
By careful use of macros, this code should compile on both recent
Gtk 2 and Gtk 3.
With no other options, ./configure will now prefer Gtk 3 if it finds
it, or fall back to Gtk 2. But you can control this by setting
'./configure --with-gtk=2|3|check|no' where the options mean:
* --with-gtk=2 - Only test for Gtk 2
* --with-gtk=3 - Only test for Gtk 3
* --with-gtk=check - Check for Gtk 3 then Gtk 2 (default)
* --with-gtk=no - Don't build virt-p2v
In the ./configure output you will see something like this:
checking for --with-gtk option... 2
checking for GTK... yes
checking if we can build virt-p2v... yes, with Gtk 2
Gtk 3 deprecates the GDK thread synchronization locks (functions such
as gdk_threads_enter/gdk_threads_leave), and requires that you issue
all Gtk calls in the main thread. To do this you have to restructure
any code that calls Gtk from other threads so it is wrapped in an idle
task, and so is run from the main thread.
This commit transforms the code in this way.
I found through experimentation that libxcb crashes noisily if you
issue any X11 calls (and hence any significant Gtk calls) when the GDK
thread sychronization initializer has been removed, so if there are
any remaining calls that are missed by this patch we should find out
about them quickly.
Use, for example:
./run virt-p2v --test-disk=$(pwd)/test-data/phony-guests/windows.img
to test conversions using a file of test data instead of the real host
/dev/sda.
But allow the warning to be turned off using --no-warn-if-partition.
Ming Xie tried to create a bootable USB key using
'virt-p2v-make-disk -o /dev/sdX1'. That works, but doesn't create a
bootable key because it puts everything into the first partition.
Emit a warning if someone tries to do this to try to prevent user
error:
virt-builder: warning: output device (/dev/sdb1) is a partition. If you
are writing to a USB key or external drive then you probably need to write
to the whole device, not to a partition. If this warning is wrong then you
can disable it with --no-warn-if-partition
Returns true if the device is a host partition.
The only tedious bit of this patch is that now Common_utils depends on
Dev_t so we have to add extra objects every time something links with
Common_utils.
For virt-p2v-make-kickstart this doesn't do very much. But for
virt-p2v-make-disk it enables verbose mode in virt-builder which is
extremely useful for debugging problems.
Thanks: Ming Xie.
Run sudo with -n (non-interactive), so it will fail right away when not
configured to not require a password. This will avoid the connection to
time out.
You can run `xterm -display :0' at the virtual console in order to get
a shell in the graphical environment.
This is useful for debugging and doesn't have much overhead unlike
other terminals.
Fetch the full version from the remote virt-v2v, including the
--with-extra field, since that tells us which distro virt-v2v was
compiled for which is useful information.
Display this information in all debug output and dialogs.
Multiple cell alignment issues in the treeviews on the right of the
conversion dialog. This particularly showed up when you had multiple
heterogeneous disks or network cards because the description cells did
not align nicely.
Thanks: Ming Xie.
Use KVM, fallback to TCG.
Add more network devices so we can test the layout of that
dialog box.
Move the documentation into guestfs-hacking(1).
Updates commit f8cfdf274f.
The kickstart version of virt-p2v has SELinux disabled, so this is at
least consistent. Mainly it avoids the very long relabel when the
disk boots first time.
Since Void Linux provides only an /etc/os-release with no VERSION_ID
field, then special-case it to avoid that the os-release parsing ignore
it.
This provides basic distro identification, and icon.
When parsing lsb-release, allow a version number in the format "X" other
than "X.Y" for the DISTRIB_RELEASE key.
This fixes version parsing in Mageia 4 guests.
Add a simple helper to run a command from a sequence of arguments,
without using a shell: this should help reducing the amount of quoting
ineeded, since arguments are passed straight as such.
Make use of it in the places currently using shell_command, and which
don't assume they can run anything (so no shell redirections, `env`,
etc).
Move the make_dib_args helper function to Common_utils as
stringify_args, so it can be used also within Common_utils itself.
This is mostly code motion.
Some ova files like those produced by SUSE Studio have their ovf, mf
and other files inside a folder rather than at the root of the
tarball. Consider the paths relative to the ovf and mf files to cover
this case too.
To add this support, two things are needed:
* make the existing code searches for either the viostor
or the SUSE VMDP (Virtual Machine Driver Pack) files.
* add a firstboot script setting up VMDP.
Note that 2 firstboot scripts are intentionally added for the VMDP
setup. This is due to windows potentially rebooting after loading the
virtio block driver. It may happen that this reboot interrupts the VMDP
setup in the firstboot script, we thus make sure the setup is run a
second time in case it needs to finish the previous run.
Sync the windows firstboot script with its linux brother. Also change
the main redirection to append to the log rather than overwriting it.
With this change, the firstboot script will resist reboots in the
executed scripts.
Add a common debug function for printing debugging messages. It only
emits the debug message when the verbose (-v) flag is used on the
command line.
It sends the output to stderr, which is flushed immediately after the
message is printed (to help with debugging unexpected crashes). There
are good arguments for sending the debug to either stdout or stderr,
and almost all existing debug messages replaced by this change went to
stdout. However using stderr is consistent with libguestfs's own
debug messages which also go to stderr.
I only made simple changes to code of the form 'if verbose () then
printf ...'. There are more places which could be changed in future.
In a few places I removed gettext calls since we probably should not
translate debug messages.
It is still only emitted if we are debugging in order not to cause
unnecessary alarm. Note this code needs a better long term fix, this
is still a hack.
Now that UEFI is fully open source the UEFI firmware can be included
in Fedora. However the location is slightly different from the
location that was used by kraxel's out of distro firmware. This
commit searches for UEFI files in the new location, falling back to
the old location (until the end of this year when we will drop it
entirely).
Thanks: Cole Robinson, Gerd Hoffmann (kraxel) & Microsoft for
relicensing the FAT code.
Since Linux 4.7, the process umask is available in /proc/<pid>/status.
See:
3e42979e65
Use this value if available, else fall back to the existing codepath
for Linux <= 4.6 and other Unix.
Newer versions of e2fsprogs refuse modifying journal devices, causing
the test #1 of get_e2uuid to fail with:
libguestfs: error: set_e2uuid: Cannot modify a journal device.
Since the test case just needs to verify get_e2uuid runs without
failing, run it without expecting a certain UUID set earlier.
Followup of commit da4812ab6b.
Only in very recent OCaml versions (or in Fedora) the Arg module
supports the --foo=bar syntax for options with arguments. Do not
require it yet.
Fixes commit 64bb9edd52.
To make sure we can also find the initrd on openSUSE and SLES, we need two improvements:
* the initrd filename may not end with '.img'
* don't use the version + release from the RPM data, rather from the
/lib/modules/<version>/ path as we need to find it out anyway.
When possible, make the disk image format explicit when invoking tools
or using add-drive. This avoids warnings from qemu about the unspecified
format for the image, and also makes qemu slightly faster (skipping the
disk image probing).
Tests checking the image probing are not touched.
This changes also:
- old-style invocations of tools (`$tool $filename`) into new style
(`$tool -a $filename`)
- add-drive-ro/add-drive-with-if guestfish commands into add/add-drive
with explicit readonly/iface arguments
There should be no change in the tests results.
Turn the snippet reading user information from /etc/passwd in a slightly
more generic function, so there is no need to copy&paste for other
details.
Mostly code motion.
At least on openSUSE and SLES, the /etc/mtab file is already existing.
Skipping the symlink creation in init removes one error message during
the appliance boot.
SUSE Linux Entreprise Server doesn't have dhcpcd and the hivex package
is not in the default repositories. Better use dhcp-client and
libhivex0.
openSUSE needs ntfs-3g and ntfsprogs packages.
To allow saving space in the package shipping the windows virtio
drivers, we can use symlinks between the drivers folders. For example
SUSE VMDP drivers are the same for Win8.1 and Win2012r2, one folder
is a symlink to the other.
To take advantages of this, find must use the -L flag.
qemu feature detection takes about 95ms on my laptop. The overhead is
almost all due to the time taken by the glibc link loader opening the
170+ libraries that qemu is linked to (×2 because we need to run qemu
twice).
Fixing that is seriously hard work.
Therefore memoize the results of guestfs_int_test_qemu.
This is keyed on the size and mtime of the qemu binary, so if the user
changes the qemu binary (eg. setting LIBGUESTFS_HV) we discard the
memoized result and rerun the qemu commands. There is also a
generation number so we can bump the generation in future versions of
libguestfs to invalidate all previously cached data.
The memo is stored in the supermin cache directory (eg. /var/tmp/.guestfs-*)
in the files:
qemu.stat Result of 'stat(2)' of the qemu binary
qemu.help qemu -help output
qemu.devices qemu -devices ? output
Note these files are only stored when using the 'direct' backend. For
the libvirt backend, libvirt itself memoizes this data in its own
place.
This is code motion, but I have cleaned up and formalized the
interface between this module and other parts of the library.
Also this adds documentation to the interface.
The maxrss is described in getrusage(2) as "This is the maximum
resident set size used (in kilobytes).". This lets us find out how
much memory qemu really used, and is useful for determining if memory
minimization techniques such as DAX are having any effect.
When creating the dummy scratch disk for a /dev/null drive, force its
format as raw if it was not specified already. After all, raw was
already the only format allowed for this kind of drives, so it makes
sense to specify the format explicitly and avoid the autodetection in
qemu.
Use the version struct in inspect_fs to hold the version of a
filesystem, adapting the inspection code to that.
Also, move the parts of the version parsing to helper functions of the
version struct, so common bits like parsing "X.Y" or "X" version strings
is done only once.
Introduce a new struct version with few helper functions for it: this
allows to "atomically" represent a version number, without different
variables to be used and checked together.
Add a initialization function from a libvirt-style version number, and
apply it for the qemu and libvirt versions in the direct and libvirt
backends.
Previously I disabled ACPI because it was not necessary and was slow.
However measuring it again now, I can see no significant difference in
performance, and it is necessary if we are going to use DAX.
It's also useful to keep the appliance kernel state as similar to the
ordinary state, and setting acpi=off is a massive change to the way
that the kernel boots.
Therefore, reenable it.
Versions older than 4.3.1 output also the version string after the
output. Ignore such line, since it isn't relevant to what
btrfs_filesystem_show needs.
In /etc/fstab the UUID= or LABEL= field may be quoted. Augeas returns
the field including the quotes, and we passed this directly to
guestfs_findfs_uuid or guestfs_findfs_label. It happens that this
works on upstream findfs, although it doesn't work in RHEL 7.2. The
correct thing to do is to remove the quotes before passing the UUID or
label to these functions.
Thanks: Thom Carlin for reporting the bug, Karel Zak for identifying
the change in behaviour in util-linux.
This function is useful outside virt-v2v so move it into the core
utilities library.
The function has been rewritten from OCaml into C, but it should be
functionally identical.
I suspected data corruption (but didn't prove it). This commit just
adds some magic numbers to the structs so we will see data corruption
quickly if it happens again.
Previously we were running ldconfig to create /etc/ld.so.cache.
This is required, at least on Fedora, if we need to run any binary
that uses a library with a weird path. libiscsi (a dependency of
qemu-img, used by virt-dib) is an example of such a weird library,
since it puts its single library into /usr/lib64/iscsi for no readily
understandable reason, and drops a configuration file into
/etc/ld.so.conf.d/ so that this new directory gets picked up.
By copying the /etc/ld.so.cache from the host we get an already
configured cache which should contain every library on the host, so
there is no need to run ldconfig.
Running ldconfig adds about 100ms to the boot time. I would prefer
that we understood which libraries need ldconfig to be run, and fix
that. We could also consider running ldconfig in parallel, but since
it might be required by just about any binary that the init script
runs it's not clear what benefit that gives.
This reverts commit 66aa98265d.
If the guest uses SELinux, then make sure to run a relabel (or at least
schedule one) after the image build: this way the template is
relabelled, or at least it will do that at the next boot, without the
need for the user to ask for a relabel.
This just covers the case of building a new image with no additional
operations on it though.
Add unit tests for the following internal functions:
- guestfs_int_getumask
- guestfs_int_new_command and other src/command.c functions
- guestfs_int_qemu_escape_param
- guestfs_int_timeval_diff
- guestfs_int_match, match1, match2
- guestfs_int_add_string and other src/stringsbuf.c functions
Because we now need to use a libguestfs handle, we have to link the
unit tests to the library. But because we also need to access the
internal functions (to test them) we need to link the test statically
to the objects making up the library. So this requires a small change
to the linking of this test too.
Use /etc/hosts as alternative of /etc/fstab to detect whether a partition
represents the root of a Linux installation; the latter might not exist
in smaller/special installations like Docker images.
Improve the error messages produced by C-based tools in case of issues
with the command line options:
- explicitly mention to use -a/-d (and -A/-D in virt-diff)
- when extra arguments are found, mention the correct way to pass
options to certain command line switches (like --format)
- in virt-inspector, give a cleaner error message when neither -i nor
any -m is specified
In all the cases, keep the extra notice to use 'TOOL --help' to get more
help with it.
python/guestfs.py:136:37: E712 comparison to True should be 'if cond is True:' or 'if cond:'
python/t/tests_helper.py:42:8: E713 test for membership should be 'not in'
No functional changes, as the new versions follow the suggested Python
coding style to do the same things.
Reindent Python scripts to make sure lines are not longer than 80
columns.
Regarding autogenerated code (guestfs.py and bindtests.py): add an
helper function to make sure comma-separated lists are wrapped at the
wanted length.
This produces only differences in the indentation of long Python lines,
with no behaviour changes.
Add (after comma) or remove (before opening round bracket, and around
'=' in arguments) whitespaces according to the PEP 8 specification.
This is just code reformatting, with no behaviour changes; no content
changed beside whitespaces, so "git diff -w" gives an empty diff.
For Windows, we now print:
$ virt-customize -a ./test-data/phony-guests/windows.img --install MSSQL
[ 0.0] Examining the guest ...
[ 14.2] Setting a random seed
virt-customize: warning: random seed could not be set for this type of
guest
[ 14.2] Installing packages: MSSQL
virt-customize: error: cannot use '--install' because no package manager
has been detected for this guest OS.
If this guest OS is a common one with ordinary package management then this
may have been caused by a failure of libguestfs inspection.
For OSes such as Windows that lack package management, this is not
possible. Try using one of the '--firstboot*' flags instead (described in
the manual).
In cases where parsing the release file failed and so we have an
obviously incorrect major version number, don't try to infer the
package manager from the major version number.
In the bug report, parsing the /etc/redhat-release file of a CentOS
7.1 guest failed, so major version was set to 0, and the package
manager was inferred as "up2date". virt-customize then failed with a
peculiar error:
virt-customize: sorry, don't know how to use --install with the 'up2date' package manager
Instead this sets it to "unknown" which will cause virt-customize to
fail with:
virt-customize: --install is not supported for this guest operating system
which is (only very slightly) better.
Problem reported by novegin on IRC.
All developer documentation should go in guestfs-hacking(1) and the
"internal documentation" (ie. documentation about internal functions
and interfaces) belongs here, not in the coincidentally synonymous
guestfs-internals(1).
Since moving boot-benchmark-range.pl out from under a tests/
subdirectory, it is now being added to po/POTFILES. This causes the
following error:
/usr/bin/xgettext: Non-ASCII string at utils/boot-benchmark/boot-benchmark-range.pl:205.
Please specify the source encoding through --from-code.
Add --from-code parameter as instructed.
The Arg module of OCaml does not support hiding options from the --help
output: hence, mark --short-options and --long-options as internal
options, since we need them only for the bash completion scripts.
Create a new top level directory called 'utils' and move the
following programs there:
tests/qemu/boot-analysis -> utils/boot-analysis/
tests/qemu/boot-benchmark -> utils/boot-benchmark/
tests/qemu/qemu-boot -> utils/qemu-boot/
tests/qemu/qemu-speed-test -> utils/qemu-speed-test/
Also we only build the boot-analysis program on x86-64 and aarch64,
since it requires custom porting to each architecture.
Recently Debian switched to btrfs-progs, so almost all the distributions
(except openSUSE) have btrfs-progs which is then moved as common
package.
The old btrfs-tools name is left there, to support Debian Jessie and
older, and Ubuntu Xenial and older.
When (in --verbose mode) we dump the pass data, dump the times in
milliseconds (instead of nanoseconds) so they are consistent with
other output. Also dump the time difference from the previous event.
Useful for detailed debugging/analysis of problems.
When we are debugging use the verbose (debug messages enabled) version
of the AAVMF (UEFI) firmware. This is useful for diagnosing what is
causing failures to boot and what inside UEFI is causing slow booting.
Use an array of structs for the UEFI firmware instead of an array of
strings.
This change is hopefully just refactoring and there is no functional
difference.
It is safe to use /dev/urandom to seed the guest. In libvirt 1.3.4
this silly restriction has been lifted.
This restores commit b2c845333f.
See also commit 9423c16607.
Thanks: Cole Robinson for doing the libvirt implementation.
The old mapping code was directly copied from old virt-v2v, translated
from Perl to OCaml.
The new mapping code does a few things more accurately:
- Use the i_product_variant field (Windows InstallationType) if available.
- Simplify rules, so there is only one special case needed for RHEL 3/4.
- Don't assume Fedora == Desktop.
- Don't assume all later Windows variants are server.
- Works for Windows > 7.
Since commit b23e149774, we have been
strict about POD errors. An error in the Japanese translation caused
"L</guestfish" (ie. an unclosed L<>) to appear in the POD.
This commit fixes the problem, but when building you may need to do:
rm po-docs/ja/guestfish.pod
rm po-docs/podfiles; make -C po-docs update-po
in order to update the broken po-docs/ja/guestfish.pod file.
Unfortunately I have been unable to upload this change back to the
Zanata servers because of a variety of client problems.
PHP (5?) renamed the PHP_EXECUTABLE variable to TEST_PHP_EXECUTABLE.
As a result of that if you enabled debugging, the tests broke because
we no longer used our custom PHP wrapper to filter out debugging
environment variables before running the tests, so debug output was
mixed with the expected output.
This commit also updates an old comment telling you how to debug PHP
tests.
Prevents us from accidentally "losing" a disk during conversion. I
believe from code inspection that this assertion is always true for
the current code, so this should have no effect.
It's a large feature / function with its own set of nested functions,
so move the inspect_source function into its own module. It also lets
us specify and document the interface explicitly.
Also: Define a root_choice type in Types module. I turned it into a
non-polymorphic variant type for extra type safety.
This is just code motion.
- Add headings, further documentation to the Types module.
- Move two type definitions in Types module. Not a functional change.
- Add more comments and clean up comments throughout v2v/v2v.ml.
- Refactor guestcaps / conversion in v2v/v2v.ml. Not a functional
change, just hides local variables from the rest of the code.
- In --in-place mode, change inspection message to 'Inspecting the
source VM'. This matches the later message 'Closing the source VM'.
Enable a few POD options:
- Don't generate an errata section in the output.
- Send errors and warnings to stderr.
- Die if any errors or warnings are seen while generating the outputs.
The drive used for the appliance is a raw (sparse) disk: specify that
explicitly in its -drive qemu command line options, so qemu can skip the
autodetection of its format and save a tiny bit of time.
Allow kernel options such as p2v.o=libvirt to override internal
defaults, even for GUI configuration.
The main change is to split up the kernel conversion into two steps:
reading the kernel command line configuration, and performing the
conversion. The kernel command line can then be read by the main
program and used to initialize the config structure for either kernel
conversion or GUI conversion.
A small adjustment is required in the test because p2v.pre no longer
runs before kernel command line parsing. (The aim is to have
p2v.pre/post/fail still only run when doing a kernel conversion, not
in the GUI case.)
Since we started to use the parallel tests framework in automake,
'make check-slow' has been broken. This is because parallel tests
doesn't allow you to run 'make check TESTS=...' with a set of test
scripts which do not also appear in the static list of tests in the
Makefile.am. We would like to list and run only "fast" tests in the
Makefile.am, and have other scripts for slow tests.
The solution is to add the slow tests to Makefile.am, but condition
those tests on an environment variable SLOW=1 being set.
This commit fixes all the existing slow tests in this way, and updates
the documentation (guestfs-hacking(1)) to document how slow tests
should be written in future.
Valgrind has a weird hack where it invokes a glibc function called
__libc_freeres on exit. See:
http://valgrind.org/docs/manual/faq.html#faq.exit_errors
This is intended to free up memory that glibc won't normally free
(since glibc doesn't free everything on exit for efficiency reasons).
More importantly, valgrind runs __libc_freeres even if the process
calls _exit, resulting in this bug:
https://bugs.kde.org/show_bug.cgi?id=361810
(either a bug in valgrind, or in glibc, or in both, depending on your
point of view).
In any case we don't want this behaviour, so disable it.
Also we have to add suppressions for new "leaks" in glibc found by
valgrind because __libc_freeres no longer runs. In fact there is only
one such suppression needed, for TLS allocation in multithreaded
tests.
When we are valgrinding we don't really care about the child
processes, which might be qemu, libvirtd, etc. So disable tracing
into children (at least, as far as is possible with valgrind, which is
not entirely disabling it, but suppressing it).
2016-04-15 17:20:52 +01:00
2024 changed files with 2581510 additions and 1018361 deletions
Remove files in package libguestfs-bash-completion, these files are bash completion files, some of the virt tool completion are already implement in another file, so can remove its completion file
set-label can only set <=127 bytes for btrfs and <=126 bytes for ntfs filesystem which not meet the help message. Also for ntfs it should give a warning message when the length >128 bytes
set-label can only set <=63 bytes for btrfs and ntfs filesystem which not meet the help message. Also for btrfs and ntfs it should give a warning message when the length exceed the limited length
error(f_"no plan could be found for making a disk image with\nthe required size, format etc. This is a bug in libguestfs!\nPlease file a bug, giving the command line arguments you used.");
error(f_"no guest operating systems or multiboot OS found in this disk image\nThis is a failure of the source repository. Use -v for more information.")
in
Customize_run.rungrootcmdline.ops;
(* Collect some stats about the final output file.
* Notes:
* - These are virtual disk stats.
* - Never fail here.
*)
letstats=
ifnot(quiet())then(
try
(* Calculate the free space (in bytes) across all mounted
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
*(at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License along
* with this program; if not, write to the Free Software Foundation, Inc.,
* 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
*)
openCommon_gettext.Gettext
openCommon_utils
openUtils
openPrintf
typecsum_t=
|SHA256ofstring
|SHA512ofstring
letstring_of_csum_t=function
|SHA256_->"sha256"
|SHA512_->"sha512"
letstring_of_csum=function
|SHA256c->c
|SHA512c->c
letverify_checksumcsumfilename=
letprog,csum_ref=
matchcsumwith
|SHA256c->"sha256sum",c
|SHA512c->"sha512sum",c
in
letcmd=sprintf"%s %s"prog(quotefilename)in
ifverbose()thenprintf"%s\n%!"cmd;
letlines=external_commandcmdin
matchlineswith
|[]->
error(f_"%s did not return any output")prog
|line::_->
letcsum_actual=fst(String.split""line)in
ifcsum_ref<>csum_actualthen
error(f_"%s checksum of template did not match the expected checksum!\n found checksum: %s\n expected checksum: %s\nTry:\n - Use the '-v' option and look for earlier error messages.\n - Delete the cache: virt-builder --delete-cache\n - Check no one has tampered with the website or your network!")
elseifi=len1||j=len2thenfalse(* no match - different lengths *)
else(
letx1=getxdigitfp1.[i]andx2=getxdigitfp2.[j]in
matchx1,x2with
|Somex1,Somex2whenx1=x2->loop(i+1)(j+1)
|Somex1,Somex2->false(* no match - different content *)
|Some_,None->loopi(j+1)
|None,Some_->loop(i+1)j
|None,None->loop(i+1)(j+1)
)
in
loop00
andgetxdigit=function
|'0'..'9'asc->Some(Char.codec-Char.code'0')
|'a'..'f'asc->Some(Char.codec-Char.code'a')
|'A'..'F'asc->Some(Char.codec-Char.code'A')
|_->None
letverifying_signaturest=
t.check_signature
letrecverifytfilename=
ift.check_signaturethen(
letargs=quotefilenamein
do_verifytargs
)
andverify_detachedtfilenamesigfile=
ift.check_signaturethen(
matchsigfilewith
|None->
error(f_"there is no detached signature file\nThis probably means the index file is missing a sig=... line.\nYou can use --no-check-signature to ignore this error, but that means you are susceptible to man-in-the-middle attacks.")
error(f_"GPG failure: could not verify digital signature of file\nTry:\n - Use the '-v' option and look for earlier error messages.\n - Delete the cache: virt-builder --delete-cache\n - Check no one has tampered with the website or your network!");
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
