It looks like flex 2.6.1 changed [1] the return code for EOF in
yyinput. Therefore, use the right value depending on the version of
flex which generates the lexer.
[1] f863c9490e
Setup the volatile /run in the appliance also with the tmpfiles
configurations available. In particular, setting up correctly the lvm
bits allow lvmetad to run.
Commit 2e16e3e993 added lv_active=active
as additional condition when listing LVs, to ignore those with the
activationskip flag set. OTOH, this check is too broad, and matches
also other kind of LVs.
Change the condition to lv_skip_activation!=1, so matching precisely
what was meant, and only that.
Related to: RHBZ#1306666
Currently lvmetad is started in init, and thus using the system
(= appliance) configuration of lvm. Later on, in the daemon, a local
copy of the lvm configuration is setup, and set it for use using the
LVM_SYSTEM_DIR environment variable: this means only the programmes
executed by the daemon will use the local lvm configuration, and not
lvmetad.
Thus manually start lvmetad from the daemon, right after having setup
the local lvm configuration, and still without failing if it cannot be
executed.
Additionally, since lvmetad now respects the right configuration, make
sure to update its cache when rescanning the VGs by passing --cache to
vgscan.
When editing the lvm configuration to set the LVM filter, edit the
'global_filter' key in addition to 'filter': the latter is not used when
lvmetad is running, when only the former works.
Commit b91b39e06a changed the separator to
':', although this creates parsing issues when there are fields with
colons (for example lv_tags=imgbased:pool).
Change once more the separator, but this time using a non-printable
character such as '\r': while it will produce uglier debug logs, this
should greatly reduce the possibilities of conflicts with texts of
metadata.
Switch to guestfs_int_version_from_x_y_or_x to parse version numbers --
although, restrict the parsing to what could look like a valid version
number, to discard quickly version strings like "unknown".
This makes sure that Debian, Fedora, Mageia, and SLED ISOs have the
right version number.
More recent versions of libosinfo switched the internal directory with
the XML files of OSes to a different layout (still with the same XML
format), causing libguestfs to not read them anymore. Furthermore, the
internal directory is going to disappear soon, replaced by a public
osinfo database [1].
Revamp the way libguestfs reads the data: first try the upcoming osinfo
layout, falling back to the current libosinfo layout (which is the same
as osinfo), and then to the old flat layout.
[1] https://gitlab.com/libosinfo/libosinfo/blob/master/docs/database-layout.txt
-Wnull-dereference and -Wshift-overflow are new warnings in GCC 6, so do
not try to disable them with pragmas on older GCC versions.
Fixes commit a8e15ea924.
One -Wnull-dereference warning is real: we deliberately cause a
segfault in one of the tests.
There is a -Wshift-overflow bug in a Gtk 2 header.
The others are the result of shortcomings in GCC.
In all cases we have to add GCC diagnostic overrides to ignore
the warnings when compiling with ./configure --enable-werror.
The commit claimed that SELinux was disabled in the
virt-p2v-make-kickstart script, and so this was just making things
consistent. However this is not true. SELinux *is* enabled in the
kickstart version. There is some code to disable it, but it is
commented out (admittedly it's hard to tell because the code is buried
in a multi-line sed expression).
The commit message also claimed (this time correctly) that the
'--selinux-relabel' step caused an extra reboot each time virt-p2v is
run from the ISO. However we have recently fixed this so it can
usually do the relabelling during the build, not at boot time, so this
is no longer a problem.
This reverts commit 25ffcc7d43.
For OCaml tools this does essentially nothing useful because the
--help output is automatically generated from the options, and so
cannot be wrong. However for C tools this is a useful check.
It would be nice to generate C tools --help output, but there isn't
enough information in the getopt data to do that.
This commit also includes fixes to the --help output for a few tools.
This function allows to download file system data units (blocks) from
the given partition.
The API can be used to detect data hidden within filesystem bad blocks
or slack space.
Moreover for filesystems such as Ext3 and Ext4, this function is the
only way to retrieve deleted files. An example is given in the function
tests.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
Pass -q to virsh if virt-v2v was run with -q, to reduce its output in
quiet mode.
Unfortunately this does not currently work in virsh as it should, see
RHBZ#1358179, so only after that bug is fixed this change can actually
take effect.
ocamlmklib -g does not work in OCaml 4.01.0. It does work (and is
desirable) in OCaml 4.02.2. However since we would like to support
back to OCaml 3.11, we cannot use it.
In libguestfs 1.33.43 we managed to release a completely
non-functional virt-customize binary (it ran without error but did
nothing at all, whatever command line arguments you gave it).
Improve the test so this cannot happen again.
podcheck.pl is run as part of the tests to perform various checks on
the documentation and the tool.
Currently we check only that the documented options matches the
options that the tool implements and vice versa. This commit would
also allow us (in future) to check --help, --long-options,
--short-options, --version output.
This commit includes scripts to run the tests and various fixes to the
manual pages to ensure that the tests pass.
For guestfish, guestmount, remove '?' from short options. Currently
those tools don't process -?, so I believe these are erroneous:
$ guestfish -\?
Try `guestfish --help' for more information.
For virt-format, the -c, -d and -q options are removed. These options
just give errors because they appear in the short options list but not
in the case statement.
Store them directly in List_entries, an adding helper function to
convert from string.
Also use Getopt.Symbol for them, so there is no need to manually reject
invalid formats.
Instead of linking with individual objects, which is very tedious,
build a proper library and link virt-builder, virt-customize and
virt-sysprep to it.
This makes the binaries a tiny bit smaller because .cmxa/.a files
allow unused code to be removed by the linker, whereas explicitly
linking .cmx/.o files does not.
Instead of linking with individual objects, which is very tedious,
build a proper library and link the other tools with it.
This doesn't make the resulting binaries any larger.
This implements the --selinux-relabel option for virt-customize,
virt-builder and virt-sysprep. There is no need to autorelabel
functionality now.
Thanks: Stephen Smalley
Sadly, the dhclient-script shipped as part of isc-dhcp-client in Ubuntu
unconditionally reads from /etc/fstab without checking for its
existence. Since no package holds /etc/fstab, this file will not exist
in the appliance, cause dhclient to fail (actually keep looping calling
the failing dhclient-script) when the network is requested.
As a workaround, touch /etc/fstab just before enabling the network: if
that file exists nothing changes, while an empty file will be available
in the other case, making at least dhclient-script in Ubuntu working.
Since the function uses the StringMap module, I also had to
move this to mllib.
The function is renamed and the 'if verbose ()' part is moved into the
function, but apart from that, it's all just code motion.
Add a dummy description value to mark an option as "hidden", so it will
not be shown in the help text.
Mark few options as hidden:
- common: --short-options, --long-options, --debug-gc
- virt-sysprep: --dump-pod, --dump-pod-options
Since --debug-gc is now really considered internal, it is no more
documented.
Add a new Getopt module to mllib, to parse command line arguments with
handlers close to the ones used with Arg, but using getopt(3) (actually
getopt_long_only) to do the real parsing. This allow us to provide
options for OCaml tools with a syntax similar to the C tools, and use
the additional features getopt offers and Arg does not.
Getopt now handles every part of the command line handling, including
the output of short & long options.
Do a single-step conversion of Common_utils and all the OCaml tools to
the syntax of Getopt.
Move a couple of utility functions from Common_utils to Getopt, since
they fit better there (and Common_utils cannot be used in Getopt, as
the former already uses the latter).
As side-change due to the conversion, extra arguments for sysprep
operation can have more keys for the same argument.
In previous commit c5f12e47e4 we changed
the PATH environment variable to run the locally built virt-v2v, not
the system-installed virt-v2v.
However this meant that we were running the locally built virt-v2v
against the system-installed libguestfs.so.0. We also need to set
LD_LIBRARY_PATH.
Fixes commit c5f12e47e4.
In the case where we are building a Python dist (which can be
distributed separately), we generated calls to non-existent optargs
structures if an API was once no-optargs but had optargs added. A
typical error was this (library: 1.32.5, Python sdist: 1.33.42):
guestfs-py.c: In function ‘guestfs_int_py_glob_expand’:
guestfs-py.c:9648:40: error: storage size of ‘optargs_s’ isn’t known
struct guestfs_glob_expand_opts_argv optargs_s;
This is a simple solution to this, although it has the possible
disadvantage that such functions will not be bound at all.
Since commit 83e92b4a97, utils.c
includes "ignore-value.h". We copy utils.c into the python sdist
tarball, but it didn't not compile because of the missing header file.
Therefore we need to copy the header in too.
Fixes commit 83e92b4a97.
guestfs_set_uuid wants device as argument.
Moreover, btrfstune -U is unable to set uuid for subvolumes,
only unmounted partitions are supported.
Here we skip btrfs subvolumes.
Calling guestfs_is_lv on btrfs subvolume throws an error.
Here we workaround it by taking Mountable instead of Device
and returning 'false' for non-device mountables.
Access, modification, last status change and creation time in
Unix format as for statns.
Number of links pointing to a given entry.
If the entry is a symbolic link, report the its target path.
A new flag (DIRENT_COMPRESSED 0x04) indicating whether the file is
compressed using native filesystem compression support.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
This adds imperative list manipulation functions inspired by Perl.
The functions are passed list refs which get updated in place.
This allows us to replace some awkward pure functional code like:
let xs = ys in
let xs = if foo then xs @ zs else xs in
with:
let xs = ref ys in
if foo then append xs zs;
These are now passed using a curl configuration file, which is a
little bit safer than using command lines. virt-builder doesn't need
to pass usernames and passwords to curl, but if it ever does in future
this will be a lot safer.
Change the Curl module to use an ADT to store the name of the curl
binary and the arguments.
The callers in virt-v2v are changed accordingly.
This also adds a (currently unused) ?proxy argument to allow callers
to override the proxy. It also adds some safety arguments implicitly.
Declare most of the stringsbuf as CLEANUP_FREE_STRINGSBUF, so they are
freed completely on stack unwind: use take_stringsbuf() in return
places to take away from the stringsbuf its content, and remove all the
manual calls to free_stringslen (no more needed now).
This requires to not use free_stringslen anymore on failure in the
helper functions of stringsbuf, which now leave the content as-is (might
be still useful even on error).
This allows us to simplify the memory management of stringsbuf's, which
are not properly fully freed, fixing memory leaks in some error paths
(which were not calling free_stringslen).
If add_string_nodup fails free the passed string instead of leaking it,
as that string would have been owned by the stringbuf.
Adapt few places to this behaviour.
We have to download the old 32 bit explorer.exe in order to find the
icons as the 64 bit version doesn't contain any icons (where are they?).
Thus prefer the 32 bit (WoW64 subsystem) directory if found.
Fixes commit 7f16c346bb.
Thanks: Xiaoyun Hu
The current location doesn't exist unless you've installed GNOME,
which is not so common on Ubuntu. Unfortunately I couldn't find any
other location containing a clean, high quality logo.
This adds another low quality icon source, and also prevents any icon
being returned if the highquality flag was set (note this prevents
virt-manager from displaying an icon, but there's nothing we can do
about that, and it's no worse than the current situation).
Updates commit 1d0683964f.
Thanks: Xiaoyun Hu
This is now enabled by default and cannot be changed/disabled by the
user. The output is saved into /tmp/.../virt-v2v-conversion-log.txt
on the conversion server.
This makes sure that in the internal configure_kernel_modules function,
for virtio or SCSI block types:
a) the warnings about leftover Xen modules are printed
b) the changes in Augeas are saved
Fixes commit ee02191483.
First check for the existence of the directory /etc/modprobe.d, in case
using a file under it; this also skips all the other checks, since they
are not needed at all. Also /etc/modprobe.d exists on recent Linux
versions, so let's give priority to the more common methods.
When /etc/modprobe.d does not exist, check for the file to edit using a
single list of possible files, now in order of priority, where the first
find is used without checking further for the rest.
Also, make sure all the returned paths are absolute: they are used in
Augeas paths later on, so relative paths will not do anything useful.
We are now able to cancel the conversion instantly by sending ^C to
the remote virt-v2v process.
Also, this reverts:
"p2v: Poll to make Cancel Conversion button more responsive."
(commit 6da4941db7)
Previously we copied / creates files in the remote dir by running
complex shell commands like:
cat > file <<'EOF'
## the file was copied in here
EOF
This was a little hairy, but in particular it doesn't allow us to set
the ssh session to cooked mode (so we can send ^C to cancel a
conversion).
A cleaner way to do it is to use 'scp' to copy the files over.
The messages that come "through" from virt-v2v are already colourized.
However the other messages are not. This colourizes the ones
generated during kernel-mode conversions.
Note that because of the way journal/syslog works we have to write the
ansi_restore before the end of line, thus code like this:
ansi_magenta (stdout);
printf ("%s: %s", guestfs_int_program_name, data);
ansi_restore (stdout);
putchar ('\n');
This also adds a "Conversion finished successfully" message (in green).
Thanks: Ming Xie
Add powerpc64 (ppc64) and powerpc64le (ppc64le) cases to the
normalize_arch function. Also remove a few duplicate cases which are
already caught in the catch-all case at the end of the match.
In kernel conversion mode, we must ensure messages are flushed to the
journal as soon as they are received from virt-v2v, otherwise (now
that debugging is not the default) we won't see any messages at all
until the last moment.
Thanks: Ming Xie
This allows you to install extra packages in the disk/ISO. The
implementation of this option in virt-p2v-make-disk was particularly
simple and followed naturally from the previous commit.
- sanitize the logs, removing terminal color codes and carriage returns
- add an empty "properties" node
- put the log for skipped tests as "message" attribute, hoping it is
read from there
- do not blacklist the log of test-virt-rescue.pl, which should not
cause issues now
If an ssh server is completely unresponsive (ie. it doesn't even send
back ICMP errors), then eventually ssh times out after ConnectTimeout
seconds [ConnectTimeout is an ssh option].
Unfortunately the miniexpect timeout (default 60 seconds) was less
than ConnectTimeout so we never saw the ssh error. Instead we would
see the internal error:
remote server closed the connection unexpectedly,
waiting for: password prompt
This commit sets the ssh ConnectTimeout to an explicit value, and sets
the miniexpect timeout to be 20 seconds larger than ConnectTimeout, so
that in most cases we will see the ssh error message before miniexpect
times out.
Thanks: Ming Xie for finding and diagnosing the problem
In the GUI disks block, instead of using separate "Size" and "Model"
columns, combine that information into a markup column under "Device".
To make the GUI look consistent I also had to change the removables
"Device" column to be a markup column and embolden the device name
there too.
This is just refactoring, but it reveals and fixes a bug too. The
size_gb field was left as NULL when the --test-disk option was used.
Apparently passing NULL to gtk_list_store_set is fine, but just in
case I replaced it with "".
The error you would see is:
virt-v2v: error: insufficient free space in the conversion server temporary
directory /var/tmp (853.8M).
Either free up space in that directory, or set the LIBGUESTFS_CACHEDIR
environment variable to point to another directory with more than 1GB of
free space.
See also the virt-v2v(1) manual, section "Minimum free space check in the
host".
Also adds some documentation.
Thanks: Ming Xie and Xiaodai Wang
There is a tiny functional change in this patch, since overlay_dir is
now always evaluated once (eg. even in --inplace mode), whereas
previously it was evaluated twice but only in copying mode.
As described in the comment, this solves a number of problems with
non-standard remote configurations.
I tested this with:
- tcsh
- zsh
- ksh
and all behaved correctly.
Print a better error message when the non-root user on the conversion
server requires a password to use sudo, and p2v is told to use sudo.
See also RHZ#1340809.
Library's counterpart of the daemon's internal_filesystem_walk command.
It writes the daemon's command output on a temporary file and parses it,
deserialising the XDR formatted tsk_dirent structs.
It returns to the caller the list of tsk_dirent structs generated by the
internal_filesystem_walk command.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
The tests check whether the filesystem_walk command is able to retrieve
information regarding both existing and deleted files.
A NTFS image is used as Ext3+ filesystems deletion is more aggressive
in terms of metadata removal.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
- generator: Added tsk_dirent struct
The tsk_dirent struct contains the information gathered via TSK APIs.
The struct contains the following fields:
* tsk_inode: inode of a file
* tsk_type: type of file such as for dirwalk command
* tsk_size: file size in bytes
* tsk_name: path relative to its disk partition
* tsk_flags: bitfield containing extra information
* tsk_spare[1-5]: extra space for future usage
- configure: Added libtsk compile-time check
Ensure libtsk is available at compile time.
If not, daemon routines depending on it won't be available.
- API: internal_filesystem_walk
The internal_filesystem_walk command walks through the FS structures
of a disk partition and returns all the files or directories
which could be found.
The command is able to retrieve information regarding deleted
or unaccessible files where other commands such as stat or find
would fail.
The gathered list of tsk_dirent structs is serialised into XDR format
and written to a file by the appliance.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
The previous code treated floppy disks and CD-ROMs as the same kind of
thing, resulting in malformed libvirt XML. You would see the
following error when importing a guest into libvirt:
error: Failed to define domain from /tmp/v2vlibvirt063486.xml
error: internal error: Invalid floppy device name: hdb
because we incorrectly generated this bogus libvirt XML fragment:
<disk device='floppy' type='file'>
<driver name='qemu' type='raw'/>
<target dev='hdb' bus='ide'/>
</disk>
This commit models floppy devices as a distinct type, occupying their
own bus ("/dev/fdX"). When writing to libvirt, we generate correct
XML fragments, looking like this:
<disk device='floppy' type='file'>
<driver name='qemu' type='raw'/>
<target dev='fda'/>
</disk>
Miscellaneous other changes were required in the code. There is also
a regression test (see following commit).
Note this ignores floppy disks in '-o qemu' mode.
aarch64 image isn't buildable at the moment because of
Anaconda bug RHBZ#1348980.
ppc64 & ppc64le images are not buildable because of delays updating
the Fedora mirrors.
This adds <BootOrder>1</BootOrder> to the first drive, which is
necessary to make the drive bootable (or would be necessary, if oVirt
hadn't implemented their own workaround for the OVF we were previously
generating).
https://bugzilla.redhat.com/show_bug.cgi?id=1308535#c11
This is not a full solution to the problem. Really we should be
copying the boot order over from the source hypervisor, but that
requires extensions to libvirt as discussed here:
https://bugzilla.redhat.com/show_bug.cgi?id=1308535#c7
Thanks: Shmuel Melamud
e2fsck returns 1 in case of "file system errors corrected".
We treat it as success in normal e2fsck, but fail if e2fsck
is run by resize2fs.
Change 'manual' execution of e2fsck to dedicated function call.
On distros not running NetworkManager, use the nm-online -x parameter
to exit quickly. The network connection will be checked just after
anyway when attempting to connect to the conversion server using ssh.
Kiwi is the tool used by openSUSE / SLES to generate many sort of
disk images. Add a virt-p2v-make-kiwi tool and his documentation to
geneate the p2v appliance kiwi configuration.
Use the common denominator for SLES and openSUSE in the dependencies.
For example most of NetworkManager pieces and metacity aren't provided
on SLES.
ifconfig is in the net-tools-deprecated package in openSUSE Factory,
which means after openSUSE Leap 42.2. Older versions have it in the
net-tools package. Adding this complexity only to add ifconfig because
sysadmins prefer it isn't too good: iproute2 is doing the job.
The password prompt is actually generated by the openssh client which
is under our control, so we can regexp match on the specific prompt
printed by openssh. This also avoids mistaking any server-side
issue.net message for a password prompt (eg. if the server prints a
message like "all users must change their passwords today!") which
would have prevented virt-p2v from logging in.
This can fail because the username is wrong. Also don't unnecessarily
reveal irrelevant implementation details in the error message, just
say the login failed.
Thanks: Juquin Zhou
Previously cancelling the conversion only set a flag, which was
checked when the run dialog displayed new output from virt-v2v. When
virt-v2v was showing hundreds of debugging messages, this wasn't a
problem, but now that we are hiding those messages, cancelling the
conversion might mean a wait of seconds or minutes.
By polling (albeit infrequently) we can make the cancel button more
responsive.
For some failures, 30 lines was not sufficient and only part of the
error was shown. Increase the number of lines shown to 50.
This also colourizes the failure message, and prominently displays the
location of the full log. The following message is now shown:
*** virt-v2v command failed ***
The full log is available on the conversion server in:
/tmp/virt-p2v-20160620-sga9rhk7/virt-v2v-conversion-log.txt
Only the last 50 lines are shown below.
[followed by up to 50 lines of log]
Updates and fixes commit 7447fe2478.
PLD Linux got /etc/os-release only in the recent 3.0 release; since
older versions have only /etc/pld-release, check for it to identify
the guest and get its version.
Previously we displayed the complete output of virt-v2v in the run
dialog. This output included all the debugging messages, and was very
long and confusing for users (especially we had false bug reports
about "errors" appearing in the debug output).
Only display stdout in the run dialog. However make sure everything
(stdout and stderr) is still logged to the conversion log.
Instead of constructing and directly executing a long virt-v2v command
line, build a wrapper script with the same command line and send it to
the remote server (stored in /<remote_dir>/virt-v2v-wrapper.sh).
This will make it a bit easier to construct more complex virt-v2v
wrappers.
Note this commit on its own is a simple refactoring and does not
change any functionality.
This option (alternately spelled: --color, --colour, --colors, or
--colours) enables ANSI colour sequences output even if that would be
disabled becaues the output is not a TTY.
The debug() function is already sending these to stderr, but in a few
places we were using printf. Change those to eprintf, except for one
informational message which should have been using info().
The virt-p2v slow test tested the local virt-p2v. However it ran
against the installed virt-v2v (and in fact would have failed if
virt-v2v was not installed).
This commit sets the environment up to run the locally built virt-v2v.
This is unfortunately quite a lot more complex than it should be.
There is no simple "set this environment variable" option in
sshd_config.
It makes no sense. By setting one field to non-sensitive when the
other field is populated, we can avoid this happening, and also make
tabbing between the fields simpler.
As a consequence of making this change, I also got rid of the now
unnecessary explanatory text telling you to leave one field blank.
In particular if you hit the 'Test Connection' button when the server
name field was empty, you would still see a stopped spinner.
This updates commit 036a11f379.
By changing the logging mode of the p2v.service unit, we can make sure
that the progress of virt-p2v is visible in non-GUI mode at the
console.
Since the output is now always shown on the console, there's no need
to print the journalctl command for following that output when the
user logs in.
This simulates doing a non-GUI conversion with virt-p2v being
downloaded over PXE. Conveniently we already had a test that was
doing this, and this rule just runs the same test.
Instead of throwing away the ssh error and printing the generic
message "unexpected end of file waiting for password prompt", we
capture the ssh error and print it. The user will see the ssh
diagnostic, eg. "No route to host".
For some reason I don't quite understand, in
commit fd82bb12fd if it got the password
prompt unexpectedly a second time then it would send the password
again, continually until ssh closed the connection because of too many
retries.
This obviously makes no sense. If we get the password prompt back,
then we must have sent an incorrect password so abort the connection
and report the error at once.
OCaml 4.02 introduced the 'bytes' type, a mutable string intended to
replace the existing 'string' type for those cases where the byte
array can be mutated. In future the 'string' type will become
immutable. This is not the default now, but it can be forced using
the '-safe-string' compile option.
This commit changes the code so that it could be compiled using
'-safe-string' (but does not actually make that change).
If we detect OCaml < 4.02, we create a dummy 'Bytes' compatibility
module ((nearly) an alias for the 'String' module). The only
significant difference from upstream OCaml is that you must write the
'bytes' type as 'Bytes.t' in interfaces, apart from that everything
else should work.
Removes this warning:
File "common_utils.ml", line 826, characters 24-28:
Warning 40: F_OK was selected from type Unix.access_permission.
It is not visible in the current scope, and will not
be selected if the type becomes unknown.
In commit ae6f726ecc we started to use
the virt-customize code to replace various virt-sysprep operations.
This had the effect of adding many more possible operations to
virt-sysprep, but some of them (specifically --install) did not work
unless the appliance network is enabled. It was not enabled in
virt-sysprep, so these operations never worked.
This change does NOT enable the network by default. However it adds a
--network flag which may be used in conjunction with --install etc to
make those commands work.
In addition we now emit a warning for certain customize operations if
they fail and if the network is not enabled. It will print:
[ 4.5] Installing packages: tcpdump
Error: Cannot retrieve repository metadata (repomd.xml) for repository: base. Please verify its path and try again
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os&infra=stock error was
virt-sysprep: warning: the command may have failed because the network is
disabled. Try either removing '--no-network' or adding '--network' on the
command line.
virt-sysprep: error: yum -y install 'tcpdump': command exited with an error
(If the network is already enabled, or if the command is successful,
then the warning is not printed.)
Thanks: Xianghua Chen
You will now see an error such as:
$ virt-customize -a centos-6.img --truncate-recursive /home/foo
[ 0.0] Examining the guest ...
[ 16.5] Setting a random seed
[ 16.5] Recursively truncating: /home/foo
virt-customize: error: libguestfs error: find0: /home/foo: No such file or
directory
Thanks: Xianghua Chen
Commit 82df768514 breaks the regression
test for RHBZ#1232192 because g#part_list can return an error when
called on a completely blank disk.
By rewriting and simplifying the code using higher-order functions we
can avoid the need for the double-nested imperative loop and calling
g#part_list at all.
Fixes commit 82df768514.
The 'semodule' command, called by the guest tools uninstaller, will
run out of memory with the default of 500MB
(https://bugzilla.redhat.com/426775).
Allocate a large amount of RAM, that will be at least 800MB, but scale
correspondingly on other architectures like ppc64 that have larger
memory requirements.
Thanks: Pavel Butsykin
This is a mostly mechanical renaming of the internal pointer types
used in the libxml2 bindings. Also I have added a longer comment to
make the system clearer.
Guest tools for Windows in Parallels / Virtuozzo Server 6 contain a
paravirtualized video driver, which wants to take full control of the video
mode. To avoid excessive video mode switches in UEFI VMs it executes
"bcdedit /set {current} graphicsmodedisabled true".
When such a VM is imported into a QEMU/KVM-based hypervisor, the
standard video drivers in Windows do not set the video mode as they
expect it to have already been configured by the bootloader. As a result,
the VM runs with black screen.
So patch the BCD store (which is a Windows registry hive with a special
structure, located on the EFI system partition) to clear this setting.
Signed-off-by: Pavel Butsykin <pbutsykin@virtuozzo.com>
Store the list of EFI system partitions on the inspect object in order to be
able to tune their contents later in the process.
Signed-off-by: Pavel Butsykin <pbutsykin@virtuozzo.com>
Guest tools for Linux in Parallels / Virtuozzo Server 6 come with a script
that can be used to uninstall them.
Signed-off-by: Pavel Butsykin <pbutsykin@virtuozzo.com>
'tc_410_close_event.rb' was not being run. You could prove this by
simply inserting "exit 1" into that test.
The reason is unclear, but by renaming every class and method in the
tests to be unique, this ensures the tests are run.
Parallels proprietary hypervisor uses RDPMC as the hypercall
instruction. As this instruction is supported since early P6 family,
the drivers didn't even bother to check for the presence of the
corresponding feature in CPUID.
In QEMU/KVM, however, this instruction triggers #GP unless the VM is run
with PMU (performance monitoring unit) enabled, which is often not the
case (due to its impact on perfromance and migratability).
So, to prevent crashes upon conversion, stop respective drivers from
loading by disabling the corresponding services. Note that the services
are being disabled ("Start" value set to 4) rather than their node
removed, to avoid confusing the uninstaller which is scheduled to run
from a firstboot script.
In addition, prl_strg (storage filter driver) is unlinked from the
storage subsystem following the DelReg directive from its .inf file,
otherwise Windows crashes with BSOD 0x7b due to missing dependency of
the storage subsystem.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Some virtio-win drivers contain more files than just *.{cat,inf,sys}.
They are filtered out currently, which prevents the drivers from being
installed by PnP.
Stop filtering driver files by extension, and copy all of them instead.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Use this command as a starting point:
git shortlog -s -- p2v v2v | awk -F'\t' '{print $2}' | sort -f
However note the above will include people who made only incidental
code changes, and also misses out QA team members, so the actual list
still has to be curated by hand.
It's better to have this overview of how the code is arranged before
any details about modifying the code, since most people will be
reading code before writing.
Recent updates of Virtuozzo Server 6 (fka Parallels Cloud Server 6) have
"Virtuozzo tools" instead of "Parallels tools" as their display name in
the corresponding Uninstall registry key.
Recognize those, too, and schedule their uninstallation on first boot.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
On Linux 4.7.0 the old code was giving -EINVAL errors. This rewrites
the code based on online sources, and it works on at least RHEL 7,
Linux 4.4 and 4.7. Also checked with valgrind.
Because EXTRA_DIST included $(TESTS) this ended up distributing the
binary file v2v_unit_tests.
Unfortunately to fix this we need to list all the test scripts and
data files explicitly a second time, but there we are.
If this flag is not set (ie. we are doing local testing) then we
disable some dangerous features. Currently this is just the "Reboot"
button, but may include other features in future.
Currently virt-p2v requires Gtk 2. This commit changes virt-p2v so it
can be built with either Gtk 2 or 3.
By careful use of macros, this code should compile on both recent
Gtk 2 and Gtk 3.
With no other options, ./configure will now prefer Gtk 3 if it finds
it, or fall back to Gtk 2. But you can control this by setting
'./configure --with-gtk=2|3|check|no' where the options mean:
* --with-gtk=2 - Only test for Gtk 2
* --with-gtk=3 - Only test for Gtk 3
* --with-gtk=check - Check for Gtk 3 then Gtk 2 (default)
* --with-gtk=no - Don't build virt-p2v
In the ./configure output you will see something like this:
checking for --with-gtk option... 2
checking for GTK... yes
checking if we can build virt-p2v... yes, with Gtk 2
Gtk 3 deprecates the GDK thread synchronization locks (functions such
as gdk_threads_enter/gdk_threads_leave), and requires that you issue
all Gtk calls in the main thread. To do this you have to restructure
any code that calls Gtk from other threads so it is wrapped in an idle
task, and so is run from the main thread.
This commit transforms the code in this way.
I found through experimentation that libxcb crashes noisily if you
issue any X11 calls (and hence any significant Gtk calls) when the GDK
thread sychronization initializer has been removed, so if there are
any remaining calls that are missed by this patch we should find out
about them quickly.
Use, for example:
./run virt-p2v --test-disk=$(pwd)/test-data/phony-guests/windows.img
to test conversions using a file of test data instead of the real host
/dev/sda.
But allow the warning to be turned off using --no-warn-if-partition.
Ming Xie tried to create a bootable USB key using
'virt-p2v-make-disk -o /dev/sdX1'. That works, but doesn't create a
bootable key because it puts everything into the first partition.
Emit a warning if someone tries to do this to try to prevent user
error:
virt-builder: warning: output device (/dev/sdb1) is a partition. If you
are writing to a USB key or external drive then you probably need to write
to the whole device, not to a partition. If this warning is wrong then you
can disable it with --no-warn-if-partition
Returns true if the device is a host partition.
The only tedious bit of this patch is that now Common_utils depends on
Dev_t so we have to add extra objects every time something links with
Common_utils.
For virt-p2v-make-kickstart this doesn't do very much. But for
virt-p2v-make-disk it enables verbose mode in virt-builder which is
extremely useful for debugging problems.
Thanks: Ming Xie.
Run sudo with -n (non-interactive), so it will fail right away when not
configured to not require a password. This will avoid the connection to
time out.
You can run `xterm -display :0' at the virtual console in order to get
a shell in the graphical environment.
This is useful for debugging and doesn't have much overhead unlike
other terminals.
Fetch the full version from the remote virt-v2v, including the
--with-extra field, since that tells us which distro virt-v2v was
compiled for which is useful information.
Display this information in all debug output and dialogs.
Multiple cell alignment issues in the treeviews on the right of the
conversion dialog. This particularly showed up when you had multiple
heterogeneous disks or network cards because the description cells did
not align nicely.
Thanks: Ming Xie.
Use KVM, fallback to TCG.
Add more network devices so we can test the layout of that
dialog box.
Move the documentation into guestfs-hacking(1).
Updates commit f8cfdf274f.
The kickstart version of virt-p2v has SELinux disabled, so this is at
least consistent. Mainly it avoids the very long relabel when the
disk boots first time.
Since Void Linux provides only an /etc/os-release with no VERSION_ID
field, then special-case it to avoid that the os-release parsing ignore
it.
This provides basic distro identification, and icon.
When parsing lsb-release, allow a version number in the format "X" other
than "X.Y" for the DISTRIB_RELEASE key.
This fixes version parsing in Mageia 4 guests.
Add a simple helper to run a command from a sequence of arguments,
without using a shell: this should help reducing the amount of quoting
ineeded, since arguments are passed straight as such.
Make use of it in the places currently using shell_command, and which
don't assume they can run anything (so no shell redirections, `env`,
etc).
Move the make_dib_args helper function to Common_utils as
stringify_args, so it can be used also within Common_utils itself.
This is mostly code motion.
Some ova files like those produced by SUSE Studio have their ovf, mf
and other files inside a folder rather than at the root of the
tarball. Consider the paths relative to the ovf and mf files to cover
this case too.
To add this support, two things are needed:
* make the existing code searches for either the viostor
or the SUSE VMDP (Virtual Machine Driver Pack) files.
* add a firstboot script setting up VMDP.
Note that 2 firstboot scripts are intentionally added for the VMDP
setup. This is due to windows potentially rebooting after loading the
virtio block driver. It may happen that this reboot interrupts the VMDP
setup in the firstboot script, we thus make sure the setup is run a
second time in case it needs to finish the previous run.
Sync the windows firstboot script with its linux brother. Also change
the main redirection to append to the log rather than overwriting it.
With this change, the firstboot script will resist reboots in the
executed scripts.
Add a common debug function for printing debugging messages. It only
emits the debug message when the verbose (-v) flag is used on the
command line.
It sends the output to stderr, which is flushed immediately after the
message is printed (to help with debugging unexpected crashes). There
are good arguments for sending the debug to either stdout or stderr,
and almost all existing debug messages replaced by this change went to
stdout. However using stderr is consistent with libguestfs's own
debug messages which also go to stderr.
I only made simple changes to code of the form 'if verbose () then
printf ...'. There are more places which could be changed in future.
In a few places I removed gettext calls since we probably should not
translate debug messages.
It is still only emitted if we are debugging in order not to cause
unnecessary alarm. Note this code needs a better long term fix, this
is still a hack.
Now that UEFI is fully open source the UEFI firmware can be included
in Fedora. However the location is slightly different from the
location that was used by kraxel's out of distro firmware. This
commit searches for UEFI files in the new location, falling back to
the old location (until the end of this year when we will drop it
entirely).
Thanks: Cole Robinson, Gerd Hoffmann (kraxel) & Microsoft for
relicensing the FAT code.
Since Linux 4.7, the process umask is available in /proc/<pid>/status.
See:
3e42979e65
Use this value if available, else fall back to the existing codepath
for Linux <= 4.6 and other Unix.
Newer versions of e2fsprogs refuse modifying journal devices, causing
the test #1 of get_e2uuid to fail with:
libguestfs: error: set_e2uuid: Cannot modify a journal device.
Since the test case just needs to verify get_e2uuid runs without
failing, run it without expecting a certain UUID set earlier.
Followup of commit da4812ab6b.
Only in very recent OCaml versions (or in Fedora) the Arg module
supports the --foo=bar syntax for options with arguments. Do not
require it yet.
Fixes commit 64bb9edd52.
To make sure we can also find the initrd on openSUSE and SLES, we need two improvements:
* the initrd filename may not end with '.img'
* don't use the version + release from the RPM data, rather from the
/lib/modules/<version>/ path as we need to find it out anyway.
When possible, make the disk image format explicit when invoking tools
or using add-drive. This avoids warnings from qemu about the unspecified
format for the image, and also makes qemu slightly faster (skipping the
disk image probing).
Tests checking the image probing are not touched.
This changes also:
- old-style invocations of tools (`$tool $filename`) into new style
(`$tool -a $filename`)
- add-drive-ro/add-drive-with-if guestfish commands into add/add-drive
with explicit readonly/iface arguments
There should be no change in the tests results.
Turn the snippet reading user information from /etc/passwd in a slightly
more generic function, so there is no need to copy&paste for other
details.
Mostly code motion.
At least on openSUSE and SLES, the /etc/mtab file is already existing.
Skipping the symlink creation in init removes one error message during
the appliance boot.
SUSE Linux Entreprise Server doesn't have dhcpcd and the hivex package
is not in the default repositories. Better use dhcp-client and
libhivex0.
openSUSE needs ntfs-3g and ntfsprogs packages.
To allow saving space in the package shipping the windows virtio
drivers, we can use symlinks between the drivers folders. For example
SUSE VMDP drivers are the same for Win8.1 and Win2012r2, one folder
is a symlink to the other.
To take advantages of this, find must use the -L flag.
qemu feature detection takes about 95ms on my laptop. The overhead is
almost all due to the time taken by the glibc link loader opening the
170+ libraries that qemu is linked to (×2 because we need to run qemu
twice).
Fixing that is seriously hard work.
Therefore memoize the results of guestfs_int_test_qemu.
This is keyed on the size and mtime of the qemu binary, so if the user
changes the qemu binary (eg. setting LIBGUESTFS_HV) we discard the
memoized result and rerun the qemu commands. There is also a
generation number so we can bump the generation in future versions of
libguestfs to invalidate all previously cached data.
The memo is stored in the supermin cache directory (eg. /var/tmp/.guestfs-*)
in the files:
qemu.stat Result of 'stat(2)' of the qemu binary
qemu.help qemu -help output
qemu.devices qemu -devices ? output
Note these files are only stored when using the 'direct' backend. For
the libvirt backend, libvirt itself memoizes this data in its own
place.
This is code motion, but I have cleaned up and formalized the
interface between this module and other parts of the library.
Also this adds documentation to the interface.
The maxrss is described in getrusage(2) as "This is the maximum
resident set size used (in kilobytes).". This lets us find out how
much memory qemu really used, and is useful for determining if memory
minimization techniques such as DAX are having any effect.
When creating the dummy scratch disk for a /dev/null drive, force its
format as raw if it was not specified already. After all, raw was
already the only format allowed for this kind of drives, so it makes
sense to specify the format explicitly and avoid the autodetection in
qemu.
Use the version struct in inspect_fs to hold the version of a
filesystem, adapting the inspection code to that.
Also, move the parts of the version parsing to helper functions of the
version struct, so common bits like parsing "X.Y" or "X" version strings
is done only once.
Introduce a new struct version with few helper functions for it: this
allows to "atomically" represent a version number, without different
variables to be used and checked together.
Add a initialization function from a libvirt-style version number, and
apply it for the qemu and libvirt versions in the direct and libvirt
backends.
Previously I disabled ACPI because it was not necessary and was slow.
However measuring it again now, I can see no significant difference in
performance, and it is necessary if we are going to use DAX.
It's also useful to keep the appliance kernel state as similar to the
ordinary state, and setting acpi=off is a massive change to the way
that the kernel boots.
Therefore, reenable it.
Versions older than 4.3.1 output also the version string after the
output. Ignore such line, since it isn't relevant to what
btrfs_filesystem_show needs.
In /etc/fstab the UUID= or LABEL= field may be quoted. Augeas returns
the field including the quotes, and we passed this directly to
guestfs_findfs_uuid or guestfs_findfs_label. It happens that this
works on upstream findfs, although it doesn't work in RHEL 7.2. The
correct thing to do is to remove the quotes before passing the UUID or
label to these functions.
Thanks: Thom Carlin for reporting the bug, Karel Zak for identifying
the change in behaviour in util-linux.
This function is useful outside virt-v2v so move it into the core
utilities library.
The function has been rewritten from OCaml into C, but it should be
functionally identical.
I suspected data corruption (but didn't prove it). This commit just
adds some magic numbers to the structs so we will see data corruption
quickly if it happens again.
Previously we were running ldconfig to create /etc/ld.so.cache.
This is required, at least on Fedora, if we need to run any binary
that uses a library with a weird path. libiscsi (a dependency of
qemu-img, used by virt-dib) is an example of such a weird library,
since it puts its single library into /usr/lib64/iscsi for no readily
understandable reason, and drops a configuration file into
/etc/ld.so.conf.d/ so that this new directory gets picked up.
By copying the /etc/ld.so.cache from the host we get an already
configured cache which should contain every library on the host, so
there is no need to run ldconfig.
Running ldconfig adds about 100ms to the boot time. I would prefer
that we understood which libraries need ldconfig to be run, and fix
that. We could also consider running ldconfig in parallel, but since
it might be required by just about any binary that the init script
runs it's not clear what benefit that gives.
This reverts commit 66aa98265d.
If the guest uses SELinux, then make sure to run a relabel (or at least
schedule one) after the image build: this way the template is
relabelled, or at least it will do that at the next boot, without the
need for the user to ask for a relabel.
This just covers the case of building a new image with no additional
operations on it though.
Add unit tests for the following internal functions:
- guestfs_int_getumask
- guestfs_int_new_command and other src/command.c functions
- guestfs_int_qemu_escape_param
- guestfs_int_timeval_diff
- guestfs_int_match, match1, match2
- guestfs_int_add_string and other src/stringsbuf.c functions
Because we now need to use a libguestfs handle, we have to link the
unit tests to the library. But because we also need to access the
internal functions (to test them) we need to link the test statically
to the objects making up the library. So this requires a small change
to the linking of this test too.
Use /etc/hosts as alternative of /etc/fstab to detect whether a partition
represents the root of a Linux installation; the latter might not exist
in smaller/special installations like Docker images.
Improve the error messages produced by C-based tools in case of issues
with the command line options:
- explicitly mention to use -a/-d (and -A/-D in virt-diff)
- when extra arguments are found, mention the correct way to pass
options to certain command line switches (like --format)
- in virt-inspector, give a cleaner error message when neither -i nor
any -m is specified
In all the cases, keep the extra notice to use 'TOOL --help' to get more
help with it.
python/guestfs.py:136:37: E712 comparison to True should be 'if cond is True:' or 'if cond:'
python/t/tests_helper.py:42:8: E713 test for membership should be 'not in'
No functional changes, as the new versions follow the suggested Python
coding style to do the same things.
Reindent Python scripts to make sure lines are not longer than 80
columns.
Regarding autogenerated code (guestfs.py and bindtests.py): add an
helper function to make sure comma-separated lists are wrapped at the
wanted length.
This produces only differences in the indentation of long Python lines,
with no behaviour changes.
Add (after comma) or remove (before opening round bracket, and around
'=' in arguments) whitespaces according to the PEP 8 specification.
This is just code reformatting, with no behaviour changes; no content
changed beside whitespaces, so "git diff -w" gives an empty diff.
For Windows, we now print:
$ virt-customize -a ./test-data/phony-guests/windows.img --install MSSQL
[ 0.0] Examining the guest ...
[ 14.2] Setting a random seed
virt-customize: warning: random seed could not be set for this type of
guest
[ 14.2] Installing packages: MSSQL
virt-customize: error: cannot use '--install' because no package manager
has been detected for this guest OS.
If this guest OS is a common one with ordinary package management then this
may have been caused by a failure of libguestfs inspection.
For OSes such as Windows that lack package management, this is not
possible. Try using one of the '--firstboot*' flags instead (described in
the manual).
In cases where parsing the release file failed and so we have an
obviously incorrect major version number, don't try to infer the
package manager from the major version number.
In the bug report, parsing the /etc/redhat-release file of a CentOS
7.1 guest failed, so major version was set to 0, and the package
manager was inferred as "up2date". virt-customize then failed with a
peculiar error:
virt-customize: sorry, don't know how to use --install with the 'up2date' package manager
Instead this sets it to "unknown" which will cause virt-customize to
fail with:
virt-customize: --install is not supported for this guest operating system
which is (only very slightly) better.
Problem reported by novegin on IRC.
All developer documentation should go in guestfs-hacking(1) and the
"internal documentation" (ie. documentation about internal functions
and interfaces) belongs here, not in the coincidentally synonymous
guestfs-internals(1).
Since moving boot-benchmark-range.pl out from under a tests/
subdirectory, it is now being added to po/POTFILES. This causes the
following error:
/usr/bin/xgettext: Non-ASCII string at utils/boot-benchmark/boot-benchmark-range.pl:205.
Please specify the source encoding through --from-code.
Add --from-code parameter as instructed.
The Arg module of OCaml does not support hiding options from the --help
output: hence, mark --short-options and --long-options as internal
options, since we need them only for the bash completion scripts.
Create a new top level directory called 'utils' and move the
following programs there:
tests/qemu/boot-analysis -> utils/boot-analysis/
tests/qemu/boot-benchmark -> utils/boot-benchmark/
tests/qemu/qemu-boot -> utils/qemu-boot/
tests/qemu/qemu-speed-test -> utils/qemu-speed-test/
Also we only build the boot-analysis program on x86-64 and aarch64,
since it requires custom porting to each architecture.
Recently Debian switched to btrfs-progs, so almost all the distributions
(except openSUSE) have btrfs-progs which is then moved as common
package.
The old btrfs-tools name is left there, to support Debian Jessie and
older, and Ubuntu Xenial and older.
When (in --verbose mode) we dump the pass data, dump the times in
milliseconds (instead of nanoseconds) so they are consistent with
other output. Also dump the time difference from the previous event.
Useful for detailed debugging/analysis of problems.
When we are debugging use the verbose (debug messages enabled) version
of the AAVMF (UEFI) firmware. This is useful for diagnosing what is
causing failures to boot and what inside UEFI is causing slow booting.
Use an array of structs for the UEFI firmware instead of an array of
strings.
This change is hopefully just refactoring and there is no functional
difference.
It is safe to use /dev/urandom to seed the guest. In libvirt 1.3.4
this silly restriction has been lifted.
This restores commit b2c845333f.
See also commit 9423c16607.
Thanks: Cole Robinson for doing the libvirt implementation.
The old mapping code was directly copied from old virt-v2v, translated
from Perl to OCaml.
The new mapping code does a few things more accurately:
- Use the i_product_variant field (Windows InstallationType) if available.
- Simplify rules, so there is only one special case needed for RHEL 3/4.
- Don't assume Fedora == Desktop.
- Don't assume all later Windows variants are server.
- Works for Windows > 7.
Since commit b23e149774, we have been
strict about POD errors. An error in the Japanese translation caused
"L</guestfish" (ie. an unclosed L<>) to appear in the POD.
This commit fixes the problem, but when building you may need to do:
rm po-docs/ja/guestfish.pod
rm po-docs/podfiles; make -C po-docs update-po
in order to update the broken po-docs/ja/guestfish.pod file.
Unfortunately I have been unable to upload this change back to the
Zanata servers because of a variety of client problems.
PHP (5?) renamed the PHP_EXECUTABLE variable to TEST_PHP_EXECUTABLE.
As a result of that if you enabled debugging, the tests broke because
we no longer used our custom PHP wrapper to filter out debugging
environment variables before running the tests, so debug output was
mixed with the expected output.
This commit also updates an old comment telling you how to debug PHP
tests.
Prevents us from accidentally "losing" a disk during conversion. I
believe from code inspection that this assertion is always true for
the current code, so this should have no effect.
It's a large feature / function with its own set of nested functions,
so move the inspect_source function into its own module. It also lets
us specify and document the interface explicitly.
Also: Define a root_choice type in Types module. I turned it into a
non-polymorphic variant type for extra type safety.
This is just code motion.
- Add headings, further documentation to the Types module.
- Move two type definitions in Types module. Not a functional change.
- Add more comments and clean up comments throughout v2v/v2v.ml.
- Refactor guestcaps / conversion in v2v/v2v.ml. Not a functional
change, just hides local variables from the rest of the code.
- In --in-place mode, change inspection message to 'Inspecting the
source VM'. This matches the later message 'Closing the source VM'.
Enable a few POD options:
- Don't generate an errata section in the output.
- Send errors and warnings to stderr.
- Die if any errors or warnings are seen while generating the outputs.
The drive used for the appliance is a raw (sparse) disk: specify that
explicitly in its -drive qemu command line options, so qemu can skip the
autodetection of its format and save a tiny bit of time.
Allow kernel options such as p2v.o=libvirt to override internal
defaults, even for GUI configuration.
The main change is to split up the kernel conversion into two steps:
reading the kernel command line configuration, and performing the
conversion. The kernel command line can then be read by the main
program and used to initialize the config structure for either kernel
conversion or GUI conversion.
A small adjustment is required in the test because p2v.pre no longer
runs before kernel command line parsing. (The aim is to have
p2v.pre/post/fail still only run when doing a kernel conversion, not
in the GUI case.)
Since we started to use the parallel tests framework in automake,
'make check-slow' has been broken. This is because parallel tests
doesn't allow you to run 'make check TESTS=...' with a set of test
scripts which do not also appear in the static list of tests in the
Makefile.am. We would like to list and run only "fast" tests in the
Makefile.am, and have other scripts for slow tests.
The solution is to add the slow tests to Makefile.am, but condition
those tests on an environment variable SLOW=1 being set.
This commit fixes all the existing slow tests in this way, and updates
the documentation (guestfs-hacking(1)) to document how slow tests
should be written in future.
Valgrind has a weird hack where it invokes a glibc function called
__libc_freeres on exit. See:
http://valgrind.org/docs/manual/faq.html#faq.exit_errors
This is intended to free up memory that glibc won't normally free
(since glibc doesn't free everything on exit for efficiency reasons).
More importantly, valgrind runs __libc_freeres even if the process
calls _exit, resulting in this bug:
https://bugs.kde.org/show_bug.cgi?id=361810
(either a bug in valgrind, or in glibc, or in both, depending on your
point of view).
In any case we don't want this behaviour, so disable it.
Also we have to add suppressions for new "leaks" in glibc found by
valgrind because __libc_freeres no longer runs. In fact there is only
one such suppression needed, for TLS allocation in multithreaded
tests.
When we are valgrinding we don't really care about the child
processes, which might be qemu, libvirtd, etc. So disable tracing
into children (at least, as far as is possible with valgrind, which is
not entirely disabling it, but suppressing it).
This is convenient because it allows you to do:
rm -f docs/C_SOURCE_FILES
make docs/C_SOURCE_FILES
if you ever wanted to recreate that file (without doing 'make dist').
Note that the rules depend on configure.ac so that the files get
updated on each new version (otherwise the files would never get
updated).
On Linux this will load the whole file into the page cache. However
the output file is empty and zero sized just after it is opened, so
this has no effect. Note that the advice is not persistent, so this
really does nothing.
I considered adding the call back after the file has been written,
just before the close, but:
- If we do a virt-resize next then we will open and read the file mostly
sequentially, so readahead will deal with any missing pages.
- If we do a virt-customize next then we will only access a small
part of the disk image, so loading it all into the page cache adds
extra work.
- In any case, since we have just written the file it's likely to
still be in the page cache.
This commit adds guestfs_int_fadvise_normal, but it's not enabled
since nothing calls it.
It also documents what Linux actually does, which is a bit different
from what I thought these settings did. Note this is for Linux 4.6.0rc3
and may change in future.
This updates commit 83e92b4a97.
Setting POSIX_FADV_RANDOM makes no measurable difference, but at least
it's the right thing to do.
POSIX_FADV_WILLNEED makes no measurable difference either.
Moving the calls to posix_fadvise to just after the open() makes no
measurable difference, but does make the code a bit clearer.
Changing POSIX_FADV_NOREUSE on the input file descriptor to
POSIX_FADV_DONTNEED slows things down by about 10%.
Add wrappers around posix_fadvise and use them in places we were
calling posix_fadvise directly before.
Also in virt-builder we were doing this (and ignoring the result):
posix_fadvise (fd, 0, 0, POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED);
However the POSIX_FADV_* flags are _not_ bitmasks! In fact
POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED == POSIX_FADV_NOREUSE so we were
giving a completely different hint from what we thought we were
giving.
The current implementation of getumask involves writing a file with
mode 0777 and then testing what mode was created by the kernel. This
doesn't work properly if the user set a per-mount umask (or fmask/
dmask).
This alternative method was suggested by Josh Stone. By forking, we
can use the thread-unsafe method (calling umask) and pass the result
back over a pipe.
This change also fixes another problem: mode_t is unsigned, so cannot
be used to return an error indication (ie. -1). Return a plain int
instead.
Thanks: Josh Stone, Jiri Jaburek, Eric Blake.
Exclude more files from the documented sources in C_SOURCE_FILES:
- sources generated by XDR definitions
- sources generated by gperf
- errnostring.c (generated by the generator, and copied around from src
to daemon during build)
Also update C_SOURCE_FILES as well, so libguestfs builds again from
scratch.
Virtio-SCSI offers a number of advantages over virtio-blk, in
particular, it supports SCSI UNMAP (aka trim) which is crucial for
keeping the virtual drive from wasting host disk space.
This patch adds support for virtio-scsi as the virtual disk connection
type both on input and on output of v2v.
Virtio-blk remains the default, so for now virtio-scsi-based guests can
only be produced in --in-place mode.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Reviewed-by: "Richard W.M. Jones" <rjones@redhat.com>
There's no particular reason that we have to update these files in
every 'make'. It's slow, and it doesn't really matter if a single
file is accidentally omitted from translations on a developer's build
machine.
On the other hand generating it from 'make dist' has advantages:
- We only do it once per release.
- The maintainer can check the changes carefully.
- If a developer adds some random files in their local build tree,
these won't be picked up accidentally.
- Changes to POTFILES etc won't leak into general commits.
Add the real list of C source files to the Makefile, instead of trying
to calculate it. This (will, in the next commit) fix a problem with
the build on Koji where we created a subdirectory (for python3/) but
the find command picked up files from that directory for the
documentation (and then failed).
This fixes commit 04229c68d6.
As in libvirt, allow internal documentation comments in C files.
These are marked using '/**' comments. Unlike libvirt which has an
ill-defined and inconsistent markup, the markup we use is Perl POD.
These comments are added to guestfs-internals(1), but are most likely
to be read in-place.
This commit changes some existing comments in src/launch.c to
demonstrate how this can be used.
Refactor registry patching to share common parts between Win <= 7 and
Win >= 8. Also branch between the two based on the presence of the
DriverDatabase subtree rather than on the OS version, which allows not
to pass the inspect handle into the functions.
OTOH make the patching functions accept the driver name and device PCI
id as arguments; this will allow to easily expand the code to support
other drivers (e.g. vioscsi) later on.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Reviewed-by: "Richard W.M. Jones" <rjones@redhat.com>
It turned out (by trial and error) that for Windows >= 8 / 2012, even
though the PnP system uses names related to the original driver's .inf
file for certain entries under DriverDatabase registry subtree, this
doesn't have to hold for the storage driver to make it boot.
So use our own artificial names there and simplify the surrounding code.
This will also make sure the added entries don't confilct with those
generated by PnP proper once the driver is installed, and that the
entries are easy to look up in the eventual registry.
The name chosen to refer to the driver -- "guestor" -- is intentionally
different from the original "viostor" both to avoid collisions and to
naturally expand to other relevant drivers (e.g. "vioscsi") later on.
Tested on
Win 8 x32
Win 8 x64
Win 8.1 x32
Win 8.1 x64
Win 2012R2 x64
Win 10 x64
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Reviewed-by: "Richard W.M. Jones" <rjones@redhat.com>
The patch to the Windows registry allowing it to boot off a virtio-blk
drive was initially conceived by comparing the state with virtio-blk
driver properly installed, to that without.
However, we don't want to replicate the Windows PnP system; rather we
need to apply just enough edits to make the system boot, and then let
the Windows PnP manager figure out the rest.
It turned out (by trial and error) that for Windows >= 8 / 2012 the
"just enough" set includes only the service entry for the driver, and a
few entries under DriverDatabase.
Tested on
Win 8 x32
Win 8 x64
Win 8.1 x32
Win 8.1 x64
Win 2012R2 x64
Win 10 x64
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Reviewed-by: "Richard W.M. Jones" <rjones@redhat.com>
The patch to the Windows registry allowing it to boot off a virtio-blk
drive was initially conceived by comparing the state with virtio-blk
driver properly installed, to that without.
However, we don't want to replicate the Windows PnP system; rather we
need to apply just enough edits to make the system boot, and then let
the Windows PnP manager figure out the rest.
It turned out that for Windows <= 7 / 2008R2 the "just enough" set
includes only the service entry for the driver, and a
CriticalDeviceDatabase entry matching the PCI id of the device and
pointing at that service entry.
While at this, also drop CriticalDeviceDatabase entries for older and
no longer relevant PCI ids of the virtio-blk device.
Tested on
Win XP SP3 x32
Win XP SP2 x64
Win 2003R2 SP2 x32
Win 2003R2 SP2 x63
Win Vista SP2 x32
Win Vista SP2 x64
Win 2008 SP2 x32
Win 2008R2 SP1 x64
Win 7 SP1 x32
Win 7 SP1 x64
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Reviewed-by: "Richard W.M. Jones" <rjones@redhat.com>
Adapt to the API changes in the newer versions of the Zend framework, in
particular regarding:
- strings handling
- resources handling (used for the guestfs_h pointer)
- iterating in hash maps
- data types for function arguments
Introduce helper macros to reduce greatly the amount of #if's all around
the generated C code.
In the generated description of the guestfish commands, wrap and indent
the help text, so it is a multiline string instead of a very long single
one.
This has no behaviour changes, only makes cmds.c more readable (and
easier to diff when the description of actions change).
Like with the previous commit, this replaces instances of:
if (something_bad) {
fprintf (stderr, "%s: error message\n", guestfs_int_program_name);
exit (EXIT_FAILURE);
}
with:
if (something_bad)
error (EXIT_FAILURE, 0, "error message");
(except in a few cases were errno was incorrectly being ignored, in
which case I have fixed that).
It's slightly more complex than the previous commit because we must be
careful to:
- Remove the program name (since error(3) prints it).
- Remove any trailing \n character from the message.
Candidates for replacement were found using:
pcregrep --buffer-size 10M -M '\bfprintf\b.*\n.*\bexit\b' `git ls-files`
Wherever we had code which did:
if (something_bad) {
perror (...);
exit (EXIT_FAILURE);
}
replace this with use of the error(3) function:
if (something_bad)
error (EXIT_FAILURE, errno, ...);
The error(3) function is supplied by glibc, or by gnulib on platforms
which don't have it, and is much more flexible than perror(3). Since
we already use error(3), there seems to be no downside to mandating it
everywhere.
Note there is one nasty catch with error(3): error (EXIT_SUCCESS, ...)
does *not* exit! This is also the reason why error(3) cannot be
marked as __attribute__((noreturn)).
Because the examples can't use gnulib, I did not change them.
To search for multiline patterns of the above form, pcregrep -M turns
out to be very useful:
pcregrep --buffer-size 10M -M '\bperror\b.*\n.*\bexit\b' `git ls-files`
It was confusing to have a mix of seconds and milliseconds. For all
upstream testing (eg. with SeaBIOS) we are discussing everything in
milliseconds, so use those exclusively.
Add a new test program called 'boot-benchmark'. This is similar to
'boot-analysis' but it simply boots and shuts down the appliance
several times in a row and measures how long it takes, calculating
mean and standard deviation.
When a struct containing FChar fields is defined
in generator/structs.ml, the Python code generator will generate
code referencing a dirent struct instead of the correct one
thus breaking at compile time.
This bug was left unnoticed because the dirent struct is the only
one using FChar fields. Yet it prevents any other struct
from using such fields.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
Look for /lib/os-release in the /usr partition and try to use it, if
present, before using lsb-release later on. This should not change the
final result of the inspection, while using the os-release detection
method also for CoreOS.
Update the phony CoreOS image to use os-release instead, uploading the
version found in the current stable version.
The "icat" name comes from the employed command line tool which might be
replaced at any time with a different implementation.
The command name is a bit confusing because it's similar to "cat" but
act as "download".
download_inode is more clear and descriptive.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
Use the proper length modifier for asprintf, instead of a padding
modifier. This fixes the parsing of versions in VERSION_ID which are a
single number (i.e. "X" and not "X.Y", etc).
Bug introduced with commit 32d19e3289, but
uncovered recently with the switch away from VLAs, i.e.
commit 07c496c53c.
In order to support the new features I am renaming the file with a better name.
The file sleuthkit.c will contain the code depending on the sleuthkit package.
The original tsk.c file will contain the logic built using libtsk
which is the sleuthkit core library.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
Some of the systemd-tmpfiles snippets need the machine ID of the running
system; the current lack of this file produces warning messages during
the appliance boot like:
[/usr/lib/tmpfiles.d/systemd.conf:26] Failed to replace specifiers: /run/log/journal/%m
[/usr/lib/tmpfiles.d/systemd.conf:28] Failed to replace specifiers: /run/log/journal/%m
[/usr/lib/tmpfiles.d/systemd.conf:29] Failed to replace specifiers: /run/log/journal/%m
Thus create a new randomly-generated /etc/machine-id on boot.
These options allow you to control the appliance memory size, number
of vCPUs, and extra kernel options respectively.
Note that using --smp is not usually a good idea. Not only does it
slow down the appliance, but it tends to break the boot analysis
program because it makes runs (more) non-deterministic.
the description of the --selinux-relabel option suggests that it
perform an immediate relabel, when in fact it may (and probably will)
instead simply touch /.autorelabel on the image, which schedules a
relabel operation for the next time the image boots. This can be
surprising because it results both in an extended initial boot time
*and* results in an automatic reboot (on some distributions).
It seems that the documentation for --in-place mode of v2v wasn't clear
enough, so try to explain it better.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Signed-off-by: "Richard W.M. Jones" <rjones@redhat.com>
In in-place mode, the decisions on which interfaces to use are made and
the source VM configuration is created by the outside entity. So in
that case v2v needs to look it up in the source configuraion, and try to
follow.
For that, the source VM configuration is used to populate requested caps
object which is then passed to the convert routine.
It's assumed that the configuration has exactly one type of storage
devices and no more than one type of network and video adapters;
anything else is rejected.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Signed-off-by: "Richard W.M. Jones" <rjones@redhat.com>
Give the caller certain control over what kind of interface to use for
virtual disks, network and video cards upon conversion.
For that, make convert functions accept additional rcaps parameter
containing an object with optional capabilities similar to the ones
produced on output, with None indicating that the decision is left to
the convert function itself.
To facilicate review, this patch unconditionally passes rcaps with no
preferences; populating it with more sensible values is done in a
followup patch.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Signed-off-by: "Richard W.M. Jones" <rjones@redhat.com>
Introduce a type to contain the guestcaps that are to be put in effect
in the converted VM: options of the block type, net type, and video.
It'll be populated by the caller and passed into convert function to
affect its choice of devices and drivers.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Signed-off-by: "Richard W.M. Jones" <rjones@redhat.com>
Those will be useful when making decisions about what configuration to
set on output.
The data is also included in --print-source so the tests are adjusted
accordingly.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Signed-off-by: "Richard W.M. Jones" <rjones@redhat.com>
This tedious workaround avoids a 0.26 second pause when using sgabios
(the Serial Graphics Adapter). It's basically a workaround for buggy
code in sgabios, but much easier than fixing the assembler.
Since commit bb5d30ab2a, we don't
require any external programs like grep to parse the command line. We
only use bash intrinsics.
Therefore we can do it early (but after /proc is mounted).
This allows verbose mode to enable set -x early on, so we can trace
most things that the init script does.
This function was kind of like debug(), except that it didn't check
the g->verbose flag and it only worked in the library (it would crash
if used after fork).
It also wasn't very useful. The sort of boot analysis done by
[forthcoming program] tests/qemu/boot-analysis.c means that
timestamping (some) messages is even less interesting than before.
Remove it and replace calls with debug() instead.
Dr. David Gilbert pointed out to me that the first line of the
qemu -help output includes the qemu version, so we don't need
to run qemu -version at all.
This saves about 0.04s on the launch time when using the direct
backend.
Read the content of /proc/cmdline using bash features, and use its
[[ ... ]] expression to find texts in a variable.
This shaves off 5 grep invocations.
Instead of hardcoding "make" in run-php-tests.sh, pass the actual name
of make from the Makefile; the default is still "make", mostly to use
the script without having to set $MAKE.
sha1sum(1) does not exist everywhere, so wrap it in an own function so
the right implementation can be chosen on each OS. Also, wrapping it
avoid using awk everytime.
Move it to the test-data directory, giving it a more generic name so it
can be used also for more than just md5; adjust qemu tests accordingly.
This is just code motion, no behaviour change.
The quiet option suppresses kernel messages. On my laptop it improves
appliance boot times by about 40% (3.5s -> 2.5s).
The emulated UART is slow and has a fixed, small FIFO (16 bytes). But
it has the advantage of being a simple ISA device which is available
very early in boot, thus enabling us to diagnose early boot problems.
So the aim is to reduce our usage of this UART on fast paths.
Of course when we are in verbose mode, we should not add this flag
because we want to see all the messages.
This change is not entirely invisible:
(1) Progress messages use the "Linux version ..." string from kernel
output in order to determine part of where we are in the boot process.
This string will no longer be detected. We should probably use a BIOS
message or maybe drop this altogether. I have added a comment to the
code.
(2) It is possible for programs to be listening for
GUESTFS_EVENT_APPLIANCE events, and they will see fewer messages now
(although what kernel messages programs see is never defined).
In case /usr is a symlink to /usr/bin, then we cannot rely on /usr/bin
to exist, since /usr might be in a different partition. Thus, in case
/bin is a symlink, check it points to "usr/bin".
Provide new <define> for the following elements: "distro",
"package_format", "package_management", and "format", adding all the
values so far possible for each.
Update the system at the end of the installation, so the generated image
is up-to-date. This also removes the need to manually update dnf on
i686 and x86_64.
The list of filesystems that is printed when there was an error prints
the internal mountable string even for the btrfs subvolumes. Let's
printing a valid -m option value instead.
Add command line options to optionally disable the output of the
installed application, and the guest icon.
This makes the inspection slightly faster, producing a smaller XML to
parse/process in case there is no interest in these details.
GCC has two warnings related to large stack frames. We were already
using the -Wframe-larger-than warning, but this reduces the threshold
from 10000 to 5000 bytes.
However that warning only covers the static part of frames (not
alloca). So this change also enables -Wstack-usage=10000 which covers
both the static and dynamic usage (alloca and variable length arrays).
Multiple changes are made throughout the code to reduce frames to fit
within these new limits.
Note that stack allocation of large strings can be a security issue.
For example, we had code like:
size_t len = strlen (fs->windows_systemroot) + 64;
char software[len];
snprintf (software, len, "%s/system32/config/software",
fs->windows_systemroot);
where fs->windows_systemroot is guest controlled. It's not clear what
the effects might be of allowing the guest to allocate potentially
very large stack frames, but at best it allows the guest to cause
libguestfs to segfault. It turns out we are very lucky that
fs->windows_systemroot cannot be set arbitrarily large (see checks in
is_systemroot).
This commit changes those to large heap allocations instead.
Implement the helper function for guestfs_inspect_list_applications2 to
be able to parse the list of installed applications with the APK package
manager (used on Alpine Linux).
Update the check for the first line (the one containing the version) to
match also the message with OpenJDK; switch to awk so it is easier to
extract the version inside the double quotes.
Introduce and use a new inspect_mount_root function to mount all the
mountpoints of a root in the guest, replacing the same code doing that
in different tools.
inspect_mount_root_ro is inspect_mount_root with readonly mount option.
If there are no labels set for the disks, the directory with the
symlinks will not even exists, causing list-disk-labels to fail with
ENOENT. In this situation, act as if the directory was there, but
empty.
This makes a bit easier to diagnose failures on mkfs, without the need
to restart the filesystem creation with verbose output (which will
produce a lot more output).
Use PRIi64 as format string for int64_t, so it builds and works fine
also on 32bit. Also switch from asprintf_nowarn to asprintf, since no
custom formats (eg %Q, %R) are used.
Adding ntfscat_i command for downloading files based on their inode number.
This allows the dowload of files unaccessible otherwise from a NTFS guest disk image.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
Add a comment to appliance/Makefile.am describing the purpose of this
directory.
Add a 'make clean-supermin-appliance' rule which forces the supermin
appliance to be reprepared from scratch. See the discussion here:
https://www.redhat.com/archives/libguestfs/2016-February/msg00264.html
Document which feature, if any, is needed for a function; this should
help users in properly checking feature availability when using certain
functions.
Old virt-v2v changed the <graphics> to VNC if the guest was using the
Cirrus hardware and Spice if the guest used QXL.
In commit 3d1cb74b3e I got the logic
backwards, using Spice if the guest used Cirrus and VNC if the guest
used QXL, which obviously makes no sense.
In this commit, I preserve the original <graphics> type from the
source guest. This has the advantage that the user can use the same
method to access the guest after conversion. If the source guest had
no <graphics> element, then we force VNC (a safe choice), and if the
source guest is a local disk that we use SDL, but this should only be
used for testing.
Thanks: Xiaodai Wang for the original bug report here:
https://bugzilla.redhat.com/show_bug.cgi?id=1225789#c10
Set the (libguestfs handle) program name correctly when running
programs like virt-copy-in, so the program name is not "guestfish" but
"virt-copy-in".
Note this feature cannot be tested using the ./run script, since
libtool (buggily) resets the program name to "lt-guestfish". However
I tested it on the installed copy and it worked there.
When listing the subvolumes of a btrfs filesystem, ignore the default
subvolume: we get the content of it when mounting the whole device
(without specifying any particular subvolume), so avoid listing it
twice.
Extend the generator to generate a source (and the header for it) with
functions that print the content of a guestfs struct. The code is
modelled after the code for it in fish.ml, although made a bit more
generic (destination FILE*, line separator) so it can be used also in
the library, when tracing.
This just introduces the new code and builds it as part of the helper
libutils.la.
Since the daemon links to pcre and use regular expressions, and since
the COMPILE_REGEXP macro doesn't depend on any aspects of the
library-side code (eg. the guestfs_h handle etc), we can allow the
daemon to use the COMPILE_REGEXP macro. Move the macro to
"guestfs-internal-all.h" to permit this.
It seems the default behaviour of rake is to run tests sequentially, and
not in parallel (there are separate gems to achieve that behaviour).
Hence just invoke "rake test" to run all the available tests at once.
This reverts commit 8f30c3c3f8.
Since we enabled parallel tests, you can no longer run tests
under valgrind merely by doing:
TESTS_ENVIRONMENT = $(top_builddir)/run --test $(VG)
check-valgrind:
$(MAKE) check VG="@VG@"
The reason is that the parallel tests framework doesn't run
``$(TESTS_ENVIRONMENT) <test>''. It inserts some other processes in
between the environment and the test, so you end up valgrinding some
unrelated process (currently the 'env' program).
In run.in, remove out of date documentation for using $(VG).
In guestfs-hacking(1), document how to do it properly.
Instead of running all the tests manually, the unittest module has a
'discovery' mode to run tests by importing them from a directory: this
requires the tests to have different filenames, since they need to be
imported as modules now (hence an empty __init__.py is added), and the
current naming does not match the convention.
Using unittest as loader/runner brings another change: tests skipped as
whole cannot be done anymore with exit(77), since they are not run but
imported: thus introduce an helper module with decorators applied to the
test classes to skip them according to the current checks. This also
gets us nicer recordings in the unittest log.
Due to the relative imports (needed for the helper code), it is no more
possible to execute tests anymore by invoking them manually; although
it is possible to run single tests, still using unittest's runner:
$ cd python
python$ ../run python -m unittest discover -v t test010Load.py
This does not change anything in what the tests do/check.
If you ran 'make installcheck' when libguestfs-devel was not
installed, then the installcheck would fail in many places with:
gcc: error: ../../src/.libs/libguestfs.so: No such file or directory
We should stop the installcheck script immediately if the .so file was
not found.
Also this commit ensures that src/.libs/libguestfs.so* not existing
before installcheck is not fatal.
Make this error message useful and actionable by suggesting commands
that the end user can run to find available pools, etc.
I also had to extend the fixed size buffer we use for messages since
the message got truncated.
Introduce a new kind of bindings tests, 090-retvalues, to check all the
possible return values in bindings; start implementing them for
scripting languages such as GObject introspection, Perl, PHP, Python,
and Ruby, reusing existing implementations where existing.
Thanks Joshua Pincus for finding the bug.
I checked back with the original W2K8R2 guest which I had used for
getting these registry entries, and there was a transcription error
with two of the CompatibleIDs.
- use assertIsInstance, assertNotEqual, and assertIsNotNone as more
specific checks (will produce better logging)
- use assertRaises when expecting exceptions being thrown
- when testing internal_test_rhashtable, instead of checking type and
elements of the return values just check the return value as a whole
(easier and already getting all the work needed by unittest)
When listing logical volumes, ignore the ones which don't get
activated automatically. No /dev/VG/LV device node is created for
these ones which confuses APIs that attempt to do 'guestfs_lvs'
followed by opening the device node. Note that 'guestfs_lvs_full' is
unaffected by this change.
Rename the existing tests according to the naming/numbering described in
guestfs-hacking(1), and improve the current ones:
- guestfs_php_001.phpt: rename to guestfs_020_create.phpt
- guestfs_php_003.phpt: rename to guestfs_070_optargs.phpt
- guestfs_php_bindtests.phpt: rename to guestfs_090_bindtests.phpt
- guestfs_090_version.phpt: new, checks taken from the former
guestfs_php_002.phpt
- guestfs_100_launch.phpt: new, modelled after the equivalent in e.g.
OCaml/Perl/Python
- guestfs_php_002.phpt: remove, as what it did is now covered by
090_version and 100_launch
And that READ_ONLY access is insufficient. I believe this is a bug in
libvirt, but it requires further investigation. If it is resolved in
libvirt then we can update this documentation again.
Thanks: Ulhas Surse
Now that all the stuff related to Windows virtio drivers has been moved
into a dedicated module, it makes sense to move the definition of
virtio_win there, too, and stop passing it around as a parameter.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Move to lazy_make_tmpdir the logic for making world-readable (but only
for root) newly-created temporary directories, removing the non-fatal
code doing that in guestfs_impl_launch.
Followup of commit 772f649e59.
When running as root libvirt will launch qemu as qemu.qemu, which will
not be able to read inside the socket directory in case it is set as
XDG_RUNTIME_DIR under /run/user/0.
Since normal users don't need particular extra access permissions for
their sockdirs, restrict /tmp as only possible sockdir for root,
changing the permissions only for this user (and making this operation
fatal).
Fixes commit 55202a4d49 and
commit 7453952d24.
1 GB should be enough to create a btrfs filesystem, even with 64K page
size; hence, make the /dev/sda and /dev/sdb test devices smaller so
there is less space taken during the test run.
Followup of commit 8ffad75e5b and
commit 9e9b648770.
This patch updates the guestfs_inspect_get_drive_mappings API call to
also return drive letters for GPT paritions. Previously this worked
only for MBR partitions. This is achieved by matching the GPT partition
GUID with the info stored in the blob from
HKLM\SYSTEM\MountedDevices\DosDevices keys. For GPT partions this blob
contains a "DMIO:ID:" prefix followed by a 16 byte binary GUID.
Support globbing in paths passed to --delete, telling glob to not
return directories with leading slash.
This re-adds back globbing for --delete in virt-sysprep, which was
available before the integration with common code from virt-customize.
Add a new optional bool "directoryslash" to indicate whether the caller
wants trailing slashes in names of directories, defaulting to true (the
current behaviour); this helps with interoperability with other tools
(such as rm).
Related to RHBZ#1293271.
The current VDSM .meta file parser is incorrect:
https://code.engineering.redhat.com/gerrit/gitweb?p=vdsm.git;a=blob;f=vdsm/storage/fileVolume.py;h=20573bd2a3fe6909a8474bd7c53c8d0fea79091c;hb=c1289fd70edcde0a0b3bc8e37ecad1e2a5babe1a#l285
When parsing the key=value fields in the .meta file, it uses the
Python expression:
key, value = l.split("=")
which throws an exception "ValueError: too many values to unpack" if
the value part also contains an "=" character (because split returns
an n-tuple with n > 2). The correct way to split the string in this
situation is to use l.split("=", 1).
In commit 399e031c25 I changed the
DESCRIPTION field in .meta so that in RHEL it would look like this:
DESCRIPTION=generated by libguestfs 1.32.2rhel=7,release=1.el7,libvirt
thus containing an "=" character in the value and throwing an
exception in VDSM.
Although this is a bug in VDSM which needs to be fixed separately,
change any occurrence of "=" in the value to "_" to avoid this
problem.
Thanks: Derrick Ornelas for helping to diagnose this bug and finding
the offending code in VDSM.
One of the mkfs-btrfs tests used two 200 MB partitions. That isn't
enough to create a btrfs filesystem on aarch64 (with 64K page size).
Since we now have a 10 GB disk, we can make the test partition 10
times larger.
In btrfs-progs 4.4, the --leafsize parameter is deprecated. It's now
just an alias for --nodesize.
On aarch64, --nodesize 4096 does not work because it's smaller than
the page size (64K). Luckily we can test a 64K nodesize on any
platform, so use that instead.
Previously these were rather small - just 500 MB. This is too small
to create a btrfs device on aarch64 (where page size may be 64K), and
barely enough even on x86-64. This change makes both these devices
10 GB, and adjusts a few tests so they continue to pass.
Introduce a new read-only API to get a path where to store temporary
sockets: this is different from tmpdir, as we need short paths for
sockets (due to sockaddr_un::sun_path), and it is either
XDG_RUNTIME_DIR if set, or /tmp; adapt guestfs_int_create_socketname
to create sockets in that location.
Furthermore, print sockdir and XDG_RUNTIME_DIR in test-tool for
debugging.
Introduce an internal helper to create paths for sockets -- will be
useful for changing later the logic for placing sockets.
Futhermore, check that the length of sockets won't overflow the buffer
for their filenames.
Extract the bit of code for lazy creation of a temporary directory, so
it can be used for more directories as well.
This is just code motion, with no behaviour changes.
Move the paths of the sockets used, from the libvirt_xml_params struct
to backend_libvirt_data; this way, they will be usable also outside the
launch callback.
Simply code motion.
I find the code slightly easier to read if we get rid of the ~write
flag, and instead split the with_hive function into two functions, one
for readonly access, and one for write access.
This is just code refactoring.
Add a common module `Windows_virtio' and move all the code related to
finding and installing Windows virtio drivers to this module.
This is just code motion and code refactoring, there is no change in
functionality.
Instead of pass 0x3fffffff as mask value, pass a simplier 0xfff, which
is the value of the IN_ALL_EVENTS define. This will still catch all
the inotify events, and avoid a EINVAL error with Linux 4.4.
Actually these are all static functions, so they don't really need the
'guestfs_' prefix at all, but using a prefix avoids any possibility of
a collision with a standard C function.
This is a mechanical refactoring.
For functions linked to virt tools, make sure the name of the function
actually matches the virt tool. Because of the history of moving
functions across tools, some names no longer matched.
For mllib, use `guestfs_int_mllib_' prefix.
For OCaml bindings, use `guestfs_int_ocaml_' prefix.
This commit is by no means complete. There are many other C functions
in other language bindings which could do with being more consistently
named.
There is no functional change, this is just refactoring.
lvm2 2.02.107 adds the -S/--select option used in lvs to filter out only
public LVs (see RHBZ#1278878). To make this work again with versions
of lvm2 older than that, only on old versions filter out thin layouts
and compose the resulting device strings ourselves.
The filtering done is much simplier than what "-S lv_role=public" will
do, but should be good enough for our need.
In Fedora 23, dracut moved to /usr/bin. However we only looked for
/sbin/dracut, so we didn't find it. You would have seen a failure
like this:
virt-v2v: error: unable to rebuild initrd
(/boot/initramfs-4.2.3-300.fc23.x86_64.img) because mkinitrd or dracut was
not found in the guest
Look for dracut in /usr/bin as well.
Thanks: Arnaud Bougeard.
print "" is no more available in Python 3, and print(...) is available
since Python 2.7; as one of the tests was using print(), use this form
everywhere so the tests can run fine with Python 3.
The warning (see below) is fairly useless. This modification to the
code avoids it.
inotify.c: In function 'do_inotify_read':
inotify.c:219:32: error: logical 'or' of equal expressions [-Werror=logical-op]
if (errno == EWOULDBLOCK || errno == EAGAIN) /* End of list. */
^~
The btrfs implementation of vfs_minimum_size requires
btrfs-progs >= 4.2, erroring with ENOTSUP otherwise; thus mark the btrfs
test case of the vfs_minimum_size tests as TestRunOrUnsupported, so it
will not cause failures if the available btrfs-progs is not enough.
Create a new TestRunOrUnsupported test type, which represents a test
sequence where a failure with ENOTSUP in the last command only marks the
test as skipped. To be used mainly when testing features available only
with some versions of helper tools used in the appliance, for example.
Having to choose the os-version of the virt-p2v disk was confusing.
Users thought it had something to do with the physical machine being
converted.
In some cases, virt-p2v-make-disk can now choose a suitable os-version
for the user. Usually it's the latest numeric version of Fedora or
Debian, depending on the host distro. If we don't recognize the host
distro, we bail and ask the user to choose, but this is still an
improvement.
Actually the OCaml compiler rewrites all printf functions at compile
time to a series of primitive string operations, so the use of sprintf
here doesn't really matter. Still, it's a bit tidier not to use
sprintf where it's not needed.
Also this commit contains some code reformatting.
This is just refactoring.
When the user clicks on the second column of the list of network
interfaces, run 'ethtool --identify <if_name> 10', which (on supported
cards) flashes a light on the physical interface for 10 seconds,
allowing it to be identified by the operator.
Resolve the paths of the binaries we are looking in the guest to
identify the architecture. This way we can identify also busybox-based
guests such as Alpine Linux, where all the "base" binaries in /bin are
symlinks to busybox.
Apparent newer versions of that report everything on stdout, including
error messages; since we only print something on failure, fold stdout to
stderr so we can see everything on failure.
This flag cannot be disabled (yet) in V5 xfs filesystems; since 2 out
of the current 3 tests of xfs_admin check other results than that flag,
avoid setting it when not needed.
Apparently with newer btrfs-progs (seen with 4.4) 100M are not enough
for a btrfs filesystem; hence double the size of the partitions
created in the test of btrfs_image, so now 200M are enough for btrfs.
In commit af762e4342 (2015-03-05)
I stopped libguestfs from using the supermin -dtb parameter when
building appliances.
It has not been necessary for quite a long time to specify a DTB,
since qemu itself creates a device tree structure in the guest
(appliance) memory from the devices presented to the guest. Qemu does
this on the fly each time the guest starts.
As I promised in the commit mentioned above I would remove this code,
and this commit does that (for libguestfs only).
This allows you to build a 32 bit virt-p2v USB key, even on a 64 bit
host.
Because of problems with the Fedora build system it's unfortunately
not easy to build a 32 bit virt-p2v binary, so this patch punts on
this issue and assumes that the user can supply the binary themselves.
This action moves second(backup) GPT header to the end of the disk.
It is usable in in-place image expanding, since free space after
second GPT header is unusable. To use additional space, we have
to move second header. This is what sgdisk -e does.
However, sgdisk -e may perform additional actions if the partition
table has unexpected params (e.g. if we call sgdisk -e /dev/sda1,
it may fix partition table thus destroying the filesystem).
To prevent such cases, we do a dry-run at first and fail if
additional actions are scheduled.
If present, Parallels Tools may stand in the way of proper running the
windows guests in non-Parallels hypervisors, so we're better off
uninstalling them on the first boot into the new environment.
With this patch, the uninstall records for Parallels Tools are looked up
in the registry and, if found, corresponding firstboot actions are
registered, taking special care that those actions are run fully
unattended.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
This test would fail on machines with around 4 GB of free memory.
Adjust the test so it would now work on such machines (it will however
still fail on a machine with around 2 GB of free memory).
Some OSes (e.g. Windows Server 2012 R2) fail to boot if the disk
GPT GUID has changed. To preserve disk guid e.g. during virt-resize,
we need a way to get/set disk GUIDs.
Libvirt has a fixed 15 second timeout for qemu to exit. If qemu is
writing to a slow USB key, it can hang (in D state) for much longer
than this - many minutes usually.
The solution is to check specifically for the libvirt EBUSY error when
this happens, and retry the virDomainDestroyFlags operation
(indefinitely).
See also the description here:
https://www.redhat.com/archives/libvir-list/2016-January/msg00767.html
Similar to the following OpenStack Nova commit:
http://git.openstack.org/cgit/openstack/nova/commit/?id=3907867
Thanks: Kashyap Chamarthy and Daniel Berrange.
The err->domain field is essentially random from the point of view
of libguestfs.
The err->int1 field has some useful information in some circumstances:
it has the errno value from the failed system call.
Resolve in the guest the given path, so absolute symlinks can be listed
using appliance tools without resolution errors.
Also remove the note about the possibility to escape the sysroot using
ll and llz, since realpath won't return paths outside sysroot.
Fixes part of RHBZ#1293276.
Fedora/i686 does not have a package called 'kernel', so the command
'rpm -q kernel' fails. In any case the method used wasn't exactly
very robust.
The new method is based on how febootstrap used to do this.
This updates commit d723b352f8.
Sometimes the user wants to know minimum size
for dirty (e.g. mounted) filesystems. In this case,
resize2fs -P will require calling e2fsck -f, while
"in general, it is not safe to run e2fsck on mounted filesystems".
Since resize2fs -P does not modify filesystem, we force it
to display (probably approximate) minimum size.
Allow the user to specify a template in --list mode, which will be the
only result in the resulting output (instead of all the available
templates).
This makes it easier to find out the details of a specific template.
For every user command line tool (eg. guestfish or virt-cat) require
that the manual page has a DESCRIPTION section. This doesn't apply to
non-CLI man pages (eg. guestfs-perl, guestfs-faq).
Require qemu >= 1.3.0, the first version that supported
`qemu-img --output=json'.
This means we require yajl (for parsing the JSON output of qemu-img),
and that in turn has consequences elsewhere.
1064041 POST https://bugzilla.redhat.com/show_bug.cgi?id=1064041
virt-sparsify fails if a btrfs filesystem contains readonly snapshots
1099976 POST https://bugzilla.redhat.com/show_bug.cgi?id=1099976
virt-builder gives GPG warning message with gnupg2
1164708 POST https://bugzilla.redhat.com/show_bug.cgi?id=1164708
set-label can only set <=127 bytes for btrfs and <=126 bytes for ntfs filesystem which not meet the help message. Also for ntfs it should give a warning message when the length >128 bytes
1173695 POST https://bugzilla.redhat.com/show_bug.cgi?id=1173695
RFE: allow passing in a pre-opened libvirt connection from python
1174551 POST https://bugzilla.redhat.com/show_bug.cgi?id=1174551
"lstatnslist" and "lstatlist" don't give an error if the API is used wrongly
1176801 POST https://bugzilla.redhat.com/show_bug.cgi?id=1176801
File /etc/sysconfig/kernel isn't updated when convert XenPV guest with regular kernel installed
1180769 POST https://bugzilla.redhat.com/show_bug.cgi?id=1180769
Security context on image file gets reset
1225789 POST https://bugzilla.redhat.com/show_bug.cgi?id=1225789
Wrong video driver is installed for rhel5.11 guest after conversion to libvirt
1229119 POST https://bugzilla.redhat.com/show_bug.cgi?id=1229119
Unrelated info in fstab makes virt-v2v fail with unclear error info
1238053 POST https://bugzilla.redhat.com/show_bug.cgi?id=1238053
v2v:Duplicate disk target set when convert guest with cdrom attached
1242853 POST https://bugzilla.redhat.com/show_bug.cgi?id=1242853
mount-loop failed to setup loop device: No such file or directory
1260801 POST https://bugzilla.redhat.com/show_bug.cgi?id=1260801
virt-builder --ssh-inject doesn't set proper permissions on created files
1264835 POST https://bugzilla.redhat.com/show_bug.cgi?id=1264835
ppc64le: virt-customize --install fail to detect the guest arch
1267032 POST https://bugzilla.redhat.com/show_bug.cgi?id=1267032
guestfish copy-in command behaves oddly/unexpectedly with wildcards
1277074 POST https://bugzilla.redhat.com/show_bug.cgi?id=1277074
Virt-p2v client shouldn't present the vdsm option because it's not usable
1277122 POST https://bugzilla.redhat.com/show_bug.cgi?id=1277122
RFE: virt-sparsify: make '--in-place' sparsification safe to abort (gracefully or ungracefully)
1290755 POST https://bugzilla.redhat.com/show_bug.cgi?id=1290755
guestfish should be able to handle LVM thin layouts
1293527 POST https://bugzilla.redhat.com/show_bug.cgi?id=1293527
There should be a reminder to avoid user to edit a guest image by multiple tools at the same time in guestfish man page
1294956 POST https://bugzilla.redhat.com/show_bug.cgi?id=1294956
set-label returns wrong error message when set the ext3/ext4 filesystem label
1359652 POST https://bugzilla.redhat.com/show_bug.cgi?id=1359652
Remove files in package libguestfs-bash-completion, these files are bash completion files, some of the virt tool completion are already implement in another file, so can remove its completion file
set-label can only set <=63 bytes for btrfs and ntfs filesystem which not meet the help message. Also for btrfs and ntfs it should give a warning message when the length exceed the limited length
Remove files in package libguestfs-bash-completion, these files are bash completion files, some of the virt tool completion are already implement in another file, so can remove its completion file
set-label can only set <=127 bytes for btrfs and <=126 bytes for ntfs filesystem which not meet the help message. Also for ntfs it should give a warning message when the length >128 bytes
error(f_"cannot find os-version '%s' with architecture '%s'.\nUse --list to list available guest types.")
argcmdline.archin
letitem=selected_cli_itemcmdlineindexin
letarg=fstitemin
letentry=snditemin
letsigchecker=entry.Index.sigcheckerin
@@ -270,6 +280,15 @@ let main () =
(* --- If we get here, we want to create a guest. --- *)
(* Warn if the user might be writing to a partition on a USB key. *)
(matchcmdline.outputwith
|Somedevicewhenis_partitiondevice->
ifcmdline.warn_if_partitionthen
warning(f_"output device (%s) is a partition. If you are writing to a USB key or external drive then you probably need to write to the whole device, not to a partition. If this warning is wrong then you can disable it with --no-warn-if-partition")
device;
|Some_|None->()
);
(* Download the template, or it may be in the cache. *)
error(f_"no guest operating systems or multiboot OS found in this disk image\nThis is a failure of the source repository. Use -v for more information.")
error(f_"GPG failure: could not verify digital signature of file\nTry:\n - Use the '-v' option and look for earlier error messages.\n - Delete the cache: virt-builder --delete-cache\n - Check no one has tampered with the website or your network!");
virt-customize -a $output --install "https://kojipkgs.fedoraproject.org//packages/dnf/1.1.4/2.fc23/noarch/dnf-1.1.4-2.fc23.noarch.rpm,https://kojipkgs.fedoraproject.org//packages/dnf/1.1.4/2.fc23/noarch/dnf-conf-1.1.4-2.fc23.noarch.rpm,https://kojipkgs.fedoraproject.org//packages/dnf/1.1.4/2.fc23/noarch/dnf-yum-1.1.4-2.fc23.noarch.rpm,https://kojipkgs.fedoraproject.org//packages/dnf/1.1.4/2.fc23/noarch/python3-dnf-1.1.4-2.fc23.noarch.rpm"
virt-customize -a $output --install "https://kojipkgs.fedoraproject.org//packages/dnf/1.1.4/2.fc23/noarch/dnf-1.1.4-2.fc23.noarch.rpm,https://kojipkgs.fedoraproject.org//packages/dnf/1.1.4/2.fc23/noarch/dnf-conf-1.1.4-2.fc23.noarch.rpm,https://kojipkgs.fedoraproject.org//packages/dnf/1.1.4/2.fc23/noarch/dnf-yum-1.1.4-2.fc23.noarch.rpm,https://kojipkgs.fedoraproject.org//packages/dnf/1.1.4/2.fc23/noarch/python3-dnf-1.1.4-2.fc23.noarch.rpm"
error(f_"host cpu (%s) and guest arch (%s) are not compatible, so you cannot use command line options that involve running commands in the guest. Use --firstboot scripts instead.")
warning(f_"the command may have failed because the network is disabled. Try either removing '--no-network' or adding '--network' on the command line.");
prerr_newline()
);
error(f_"%s: command exited with an error")display
(* Windows has package_management == "unknown". *)
anderror_unknown_package_managerflag=
error(f_"cannot use '%s' because no package manager has been detected for this guest OS.\n\nIf this guest OS is a common one with ordinary package management then this may have been caused by a failure of libguestfs inspection.\n\nFor OSes such as Windows that lack package management, this is not possible. Try using one of the '--firstboot*' flags instead (described in the manual).")flag
anderror_unimplemented_package_managerflagpm=
error(f_"sorry, '%s' with the '%s' package manager has not been implemented yet.\n\nYou can work around this by using one of the '--run*' or '--firstboot*' options instead (described in the manual).")flagpm
error(f_"'%s' is missing. This file is required in order to install Windows firstboot scripts. You can get it by building rhsrvany (https://github.com/rwmjones/rhsrvany). Original error: %s")
error(f_"One of rhsrvany.exe or pvvxsvc.exe is missing in %s. One of them is required in order to install Windows firstboot scripts. You can get one by building rhsrvany (https://github.com/rwmjones/rhsrvany)")
virt_tools_data_dirin
(* Create a directory for firstboot files in the guest. *)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
