In the case where virt-v2v runs on the same server as the imageio
daemon that we are talking to, it may be possible to optimize access
using a Unix domain socket.
This is only an optimization. If it fails or if we're not running on
the same server it will fall back to the usual HTTPS over TCP
connection.
Thanks: Nir Soffer, Daniel Erez.
Previously we lazily requested the server options in the can_*
callbacks. The can_* callbacks are always called by nbdkit straight
after open, so this just adds complexity for no benefit. This change
simply makes the code always fetch the server options during the open
callback.
This is — functionally at least — mostly just refactoring. However I
also added a useful debug message so we can see what features the
imageio server is offering.
"http" and "transfer" variables were missing in emulate_zero, so the
code would fail with NameError. This can happen only when communicating
with old imageio versions not supporting the "zero" feature.
Testing with qemu-img 2.12 show that we never send emulated zero request
because of the highestwrite mechanism, but it can break with older
qemu-img-rhev used on RHEL.
Python manual warns[1]:
Note that you must have read the whole response before you can send
a new request to the server.
The reason for this warning is exposed only when the server is using
keep alive connections. When the response is not read, sending a new
request will fail with:
httplib.ResponseNotReady
Even if Content-Length was 0 or the request has no content. The failure
looks like this when using --verbose:
nbdkit: python[1]: debug: zero count=33554432 offset=0 may_trim=1 fua=0
nbdkit: python[1]: debug: zero count=33554432 offset=33554432 may_trim=1 fua=0
nbdkit: python[1]: error: /home/nsoffer/src/libguestfs/tmp/rhvupload.Au2B5I/rhv-upload-plugin.py: zero: error: Request-sent
nbdkit: python[1]: debug: sending error reply: Input/output error
qemu-img: error writing zeroes at offset 0: Input/output error
Change all requests to read the whole response.
Tested with imageio patch supporting keep alive connections:
https://gerrit.ovirt.org/#/c/92296/
[1] https://docs.python.org/3.8/library/http.client.html#http.client.HTTPConnection.getresponse
The new ‘-o rhv-upload’ output mode contains a '-' character in the
name, but the regular expression which matched the output of the
virt-v2v command did not recognize '-' as a valid character. It ended
up mapping this to just "rhv" meaning two "rhv" entries would appear
in the list of output drivers.
Thanks: Ming Xie.
Virt-v2v transfers to newer versions of ovirt-engine can fail with:
ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "failed to parse a given ovf configuration ovf error: [empty name]: cannot read '//*/disksection' with value: null". HTTP response code is 400.
This was caused by a change made to oVirt:
https://gerrit.ovirt.org/#/c/91902/
so that it now requires the <Disk ovf:capacity> attribute.
Thanks: Arik Hadas
When sending request with small or no payload, it is simpler and
possibly more efficient to use the high level HTTPSConnection.request(),
instead of the lower level APIs.
The only reason to use the lower level APIs is to avoid copying the
payload, or on python 2, to use a bigger buffer size when streaming a
file-like object.
The 'localmountpoint' variable in the handle is available only when
building with FUSE support, so guard it in a proper #ifdef block.
Fixes commit 296370fb86.
Thanks to: Corentin Labbe (both reporting, and testing)
Simple refactoring of the order of parameters to input objects
so they are always in the order:
input_foo ; class name
input_conn ; command line -iX parameters, alphabetically
input_password ;
... ;
vddk_options ; extra class parameters
... ;
guestname ; remaining command line parameters
Consistent with the option -op added in
commit a4e181137a, use -ip to pass
passwords for the input side.
This replaces --password-file, and like that option you have to pass a
filename.
This also changes the code so we pass around the filename instead of
the password between functions. For ‘-it vddk’ this simplifies
things: we can pass the filename through to nbdkit and we never need
to read it in virt-v2v. For other input methods we eventually need to
read the password from the file at some lower level place in the code.
This increases the inactivity timeout for transfers from the default
(60 seconds in oVirt < 4.2.3, 600 seconds in >= 4.2.3), up to 1 hour,
so that we should never hit it for ordinary transfers.
Note this requires oVirt >= 4.2.3. The corresponding oVirt fix was
https://bugzilla.redhat.com/1563278
I also replaced the deprecated ‘image’ parameter with ‘disk’.
Thanks: Nir Soffer, Daniel Erez.
Factor out the common code used to hexdump the protocol when
./configure --enable-packet-dump is used.
It's not quite a straight refactor because I had to fix some
signedness bugs in the code which were not revealed before because
this code is usually disabled.
Extended MBR partitions cannot hold filesystems - filter them out.
RWMJ:
* Simplify is_mbr_extended function.
* Fix regression test rhbz1285847:
The test assumed that list-filesystems would return an extended MBR
partition (as "unknown"). However such partitions only contain
logical partitions inside them, not filesystems.
* Some minor whitespace changes.
1. Now we use GPT partition type to filter out LDM partitions.
2. We also check for filesystems on LDM volumes because LDM partitions
doesn't contain filesystems (list_ldm_partitions is not called anymore).
3. Obvious repetitive comments are moved to a function description.
Test guestfish list-filesystems command finds file system on partitioned md device and doesn't take into account md device itself (similar to as physical devices are filtered out if they are partitioned).
Instead of using part_to_dev to find such devices we open the device's
directory under /sys/block/<dev_name> and look for entries starting with
<dev_name>, eg. /sys/block/sda/sda1.
I saw several Windows disk images which contains strange registry entry
for mapped drives:
"\\DosDevices\\Y:"=hex(3):00,00,00,00,00,00,00,00,00,00,00,00
Which is decoded something like diskID = 0x0, partition starts at 0
bytes offset from the start of the disk. In addition to a Windows disk
image, I have attached dummy disk and made xfs file system on a whole
device without partitioning it. I mount xfs file system to a "/" and
then mkdir and mount other found file systems inside (/fs1, /fs2 etc.).
When we decode drive mappings we are looking for a disk with ID 0x0 (it
is 4 bytes somewhere LBA0). It is appeared that dummy non-partitioned
disk with xfs file system has zeros by offset where diskID is expected
to be). So the disk is considered as a candidate to search for
partition at offset 0. part-list command (and "parted" which is used
under the hood) reports there is 1 partition on "dummy" disk which
starts exactly at offset 0. And thus dummy device name and partition
number are simply concatenated together and corresponding drive mapping
is returned: Y => /dev/sdX1. But /dev/sdX1 is not existing block
device.
No matter either it is a bug in "parted" (or it works this way
by-design), let's protect ourself from this situation: in addition we
look for msdos partition table on a disk before making any further
assumptions.
When the daemon starts up it creates a fresh (empty) LVM configuration
and starts up lvmetad (which depends on the LVM configuration).
However this appears to cause problems: Some types of PV seem to
require lvmetad and don't work without it
(https://bugzilla.redhat.com/show_bug.cgi?id=1581810). If we don't
start lvmetad earlier, the device nodes are not created.
Therefore move the whole initialization step into appliance/init.
Two further changes had to be made:
Now we are using lvmetad all the time, using vgchange is incorrect.
With lvmetad activated early we must use ‘pvscan --cache --activate ay’
to scan all disks for PVs and activate any VGs on them (although the
documentation is complex, confusing and contradictory so I'm not
completely sure about this).
The ‘lvm_system_dir’ local variable in ‘daemon/lvm-filter.c’
previously contained the path of the directory above $LVM_SYSTEM_DIR
(eg. $LVM_SYSTEM_DIR = "/etc/lvm", lvm_system_dir = "/etc"). As this
was highly confusing, I have changed it so the local variable and the
environment variable have identical contents. This involved removing
the ‘lvm/’ component from a couple of paths since it is now included
in the local variable.
Use the $(srcdir) variable where needed, to make sure it builds also
with srcdir != builddir.
Furthermore, make sure that the OCaml dependencies calculation depend on
the generated output_rhv_upload_*_source.ml files, otherwise there will
be incomplete OCaml rules for them in the generated .depend.
Fixes commit cc04573927.
The current detection code for Linux kernels assumes that a kernel
package contains everything in it, i.e. the kernel itself, its modules,
and its configuration. However, since recent Ubuntu versions (e.g.
starting from 18.04) modules & config (with few more files) are split in
an own package, thus not detecting the modpath from installed vmlinuz
files.
To overcome this situation, rework this detection a bit:
1) find the vmlinuz file as before, but then immediately make sure it
exists by stat'ing it
2) find the modules path from the package as before:
2a) if found, extract the version in the same step
2b) if not found, get the kernel version from the vmlinuz filename,
and use it to detect the modules path
3) check that the modules path exists
The detection done in (2b) is based on the current packaging scheme
found in the most important Linux distributions (Fedora, RHEL, CentOS,
Debian, Ubuntu, openSUSE, AltLinux, and possibly more). The notable
exception is Arch Linux.
As additional change, do not assume the config file is in the same
package as vmlinuz, but directly look into the filesystem using the
version we already have.
Commit 4699c7b6e1 converted the null
output to use the null-co qemu driver with a JSON URL syntax --
especially the latter is only available in newer versions of qemu.
Even if this output mode is mostly for testing, check at runtime whether
the null-co + JSON way is possible, falling back to the creation of
thrown-away temporary files as before.
