Commit Graph

10947 Commits

Author SHA1 Message Date
Pino Toscano
ec438b7b85 mltools: create a cmdline_options struct
Instead of returning directly a Getopt.t handle, now
Tools_utils.create_standard_options returns a struct, which at the
moment contains only the Getopt.t handle.  This way, it will be easy to
add more data needed for handling standard command line options.

This is mostly refactoring, with no functional changes.
2018-09-20 17:52:53 +02:00
Richard W.M. Jones
51a9c874d3 v2v: Add a slow test of Windows conversions.
As with the previous commit this requires that Windows guests have
been created first using the procedure from:

https://rwmj.wordpress.com/2018/09/13/creating-windows-templates-for-virt-builder/

For me:

PASS: test-v2v-conversion-of-windows-6.3-server.sh
PASS: test-v2v-conversion-of-windows-6.2-server.sh
PASS: test-v2v-conversion-of-windows-10.0-server.sh

If you don't have these templates in virt-builder then the tests will
skip.
2018-09-14 14:59:35 +01:00
Richard W.M. Jones
4b6341072c customize: tests: Add slow test of Windows firstboot.
This requires that Windows guests have been created using the
procedure outlined here:

https://rwmj.wordpress.com/2018/09/13/creating-windows-templates-for-virt-builder/

For me:

PASS: test-firstboot-windows-6.2-server.sh
PASS: test-firstboot-windows-6.3-server.sh
PASS: test-firstboot-windows-10.0-server.sh

An incidental change is that we dump the firstboot log from the guest
(even on success).  If the firstboot fails this is very useful for
determining the real cause.
2018-09-14 14:39:46 +01:00
Richard W.M. Jones
d54be028a2 customize: Use set -x for test-firstboot.sh slow test.
Aids in debugging failed tests.
2018-09-14 13:49:54 +01:00
Richard W.M. Jones
bbefbc8a48 v2v: -it vddk: Fix check for incorrect LD_LIBRARY_PATH.
In nbdkit >= 1.7.2, nbdkit vddk --dump-plugin will print the shared
library name in normal output, which breaks this test.

The actual error when LD_LIBRARY_PATH is not set includes this line:

  nbdkit: error: libvixDiskLib.so.6: cannot open shared object file: No such file or directory

so search instead for the error message "cannot open shared object file".
2018-09-14 12:01:53 +01:00
Richard W.M. Jones
089f0345f4 builder: templates: Add Windows template support.
Provides support for building:

 - Windows 7
 - Windows Server 2008 R2
 - Windows Server 2012
 - Windows Server 2012 R2
 - Windows Server 2016

Note that these images cannot be released to the public because of
obvious licensing issues.  But this documents how we build them for
internal consumption so that others can also build them.

Thanks: Christophe Fergeau, Tomáš Golembiovský.
2018-09-13 09:51:58 +01:00
Richard W.M. Jones
755834c797 builder: templates: Use virt-sparsify --in-place (instead of copying).
virt-sparsify in copying mode takes a huge amount of temporary space
and takes a very long time.  In any case it's not really necessary for
modern guests since in-place sparsification is fully supported now.
2018-09-13 09:17:21 +01:00
Nikolay Ivanets
eaa9c8cbc6 daemon: consider /etc/mdadm/mdadm.conf while inspecting mountpoints.
Inspection code checks /etc/mdadm.conf to map MD device paths listed in
mdadm.conf to MD device paths in the guestfs appliance. However on some
operating systems (e.g. Ubuntu) mdadm.conf has alternative location:
/etc/mdadm/mdadm.conf.

This patch consider an alternative location of mdadm.conf as well.
2018-09-13 09:17:21 +01:00
Richard W.M. Jones
e6895bd9d2 builder: templates: Specify --arch explicitly, even on x86_64. 2018-09-11 16:52:10 +01:00
Richard W.M. Jones
cc01166a24 builder: templates: Make UEFI code arch-independent.
Previously it only worked for aarch64.
2018-09-11 16:52:10 +01:00
Richard W.M. Jones
613c1782ea builder: templates: Refactor code for using UEFI.
Simple refactoring.
2018-09-11 16:52:10 +01:00
Richard W.M. Jones
00fff03485 builder: templates: Refactor code for adding --initrd-inject.
Initrd is a Linux thing, so this code should only be used for
Linux-based OSes.  Refactor to make it a bit cleaner.
2018-09-11 16:52:10 +01:00
Richard W.M. Jones
d0ff42dfa2 builder: templates: Add a can_sysprep_os function.
Does virt-sysprep know how to sysprep this disk image?  Add a function
to find out.

Just refactoring.
2018-09-11 16:52:10 +01:00
Richard W.M. Jones
112880e61b builder: templates: Rename make_kickstart function.
This function will soon be used to generate Windows unattended.xml
files (as well as Debian preseed) so just rename it back to
‘make_kickstart’ rather than attempting to explain in the name every
way it could be used.
2018-09-11 16:52:10 +01:00
Richard W.M. Jones
e9bd010f6a builder: templates: Add generic method for deleting files on exit.
Simple refactoring.
2018-09-11 16:52:10 +01:00
Richard W.M. Jones
2c349a00d2 Revert "build: Maintain a central list of skipped tests."
This reverts commit 79a3ac5d0e.
2018-09-04 11:25:56 +01:00
Richard W.M. Jones
46bb1428ec Revert "tests: Disable 255 disk test until we can find a way to make it reliable."
This reverts commit 9b911a2a85.
2018-09-04 11:25:56 +01:00
Richard W.M. Jones
969cacead9 v2v: Add -o openstack target, writes to OpenStack & Cinder using APIs. 2018-09-03 18:29:33 +01:00
Richard W.M. Jones
e4f4ed6188 mltools: Add with_timeout utility function.
Higher-order function which performs an operation repeatedly until it
succeeds or fails, or a timeout is reached.
2018-09-03 18:29:33 +01:00
Richard W.M. Jones
16c9dad598 v2v: -o glance: Refactor OpenStack image properties into a new module.
Straightforward refactoring.  Apart from small modifications to clean
up the code and reorder the properties more logically there should be
no functional change.
2018-09-03 18:29:33 +01:00
Richard W.M. Jones
e12c56176a v2v: windows: Fix rhev-apt command line (RHBZ#1624902).
See:
https://bugzilla.redhat.com/show_bug.cgi?id=1584678#c15

Fixes commit bcdbe6405c.  However this
bug was copied directly from old virt-v2v which did the same thing
(from lib/Sys/VirtConvert/Converter/Windows.pm):

  echo installing rhev-apt >>log.txt
  "rhev-apt.exe" /S /v /qn >>log.txt

Thanks: Lev Veyde
2018-09-03 18:29:33 +01:00
Richard W.M. Jones
741ef228cd v2v: docs: Describe support for SHA-2 certs for Windows 7 / 2008 R2 (RHBZ#1624878).
Thanks: Yan Vugenfirer.
2018-09-03 18:29:33 +01:00
Richard W.M. Jones
56f1e6cecd java: Link with gnulib to resolve missing hash_free symbol. 2018-08-28 22:24:46 +01:00
Richard W.M. Jones
9b911a2a85 tests: Disable 255 disk test until we can find a way to make it reliable. 2018-08-28 22:24:46 +01:00
Richard W.M. Jones
79a3ac5d0e build: Maintain a central list of skipped tests.
I was maintaining this list unofficially.  Let's maintain it in the
sources so it is versioned and branched along with the libguestfs
code.
2018-08-28 22:24:46 +01:00
Richard W.M. Jones
5b87b9f132 ppc64le: Remove bogus __powerpc64le__.
Not a thing - tested on P9.
2018-08-28 22:24:46 +01:00
Richard W.M. Jones
56318f0b5f ppc64le: Don't use -cpu parameter under any circumstances (RHBZ#1605071). 2018-08-28 22:24:46 +01:00
Richard W.M. Jones
598e879ad1 Version 1.39.9. v1.39.9 2018-08-28 22:17:02 +01:00
Richard W.M. Jones
af5b117ade mltools: Add test-machine-readable-script.sh to EXTRA_DIST.
Fixes commit afa8111b75.
2018-08-28 22:15:30 +01:00
Nir Soffer
8f250c00c8 v2v: rhv-upload-plugin: Use BrokenPipeError
With python 3, we have a nicer way to handle socket.error with errno set
to EPIPE (or ESHUTDOWN).

This is also more correct since in some cases (that I could not
reproduce yet with v2v), using e[0] with BrokenPipeError will fail with:

>>> OSError(errno.EPIPE, "Broken pipe")[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'BrokenPipeError' object is not subscriptable

For python 2 e[0] seems to work, but is leftover from historic python
version that used to raise a tuple instead of socket.error instance.
In python 2.7 library code e.args[0] is used. If we ever port this to
python 2 this is the best form.
2018-08-28 21:56:11 +01:00
Richard W.M. Jones
0d0b551130 v2v: Document how to adjust ESXi NFC memory limit (RHBZ#1614276). 2018-08-24 14:25:27 +01:00
Richard W.M. Jones
9f678c3bde v2v: Add --print-estimate option to print copy size estimate.
This option prints the estimated size of the data that will be copied
from the source disk.

Currently this overestimates by the size of the qcow2 header, but for
real disk images that doesn't matter much.

For example:

$ virt-builder fedora-27
$ virt-v2v -i disk fedora-27.img -o null --print-estimate
[...]
virt-v2v: This guest has virtio drivers installed.
[  44.0] Mapping filesystem data to avoid copying unused and blank areas
[  44.5] Closing the overlay
disk 1: 1047920640
total: 1047920640
2018-08-24 14:13:01 +01:00
Pino Toscano
10f9779c08 v2v: test-harness: use Xml.parse_file
Use Xml.parse_file instead of read_whole_file + Xml.parse_memory.
2018-08-24 13:11:09 +02:00
Pino Toscano
d7772fb78b builder: index_parser_tests: really use parameter
Really use the parameter of the "read_file" function, instead of
hardcoding "out".  This does not change the behaviour, since all the
callers already use "out" as the file name to read.

Fixes commit 7e3689bfe0.
2018-08-24 13:06:36 +02:00
Pino Toscano
ec345aca02 builder: use the new json_parser_tree_parse_file
No need to read the whole file manually.
2018-08-23 19:12:46 +02:00
Pino Toscano
bb00c482e8 mltools: JSON: add json_parser_tree_parse_file
Easy way to parse JSON from a file, without reading it all to string
first.
2018-08-23 19:06:22 +02:00
Sebastian Meyer
fd43730e2b Fix error with --uninstall option on SUSE
The `-l` option for some `zypper` subcommands is short for
`--auto-agree-with-licenses` and not available on the `zypper rm` command.
2018-08-23 18:05:04 +02:00
Pino Toscano
afa8111b75 OCaml tools: add output selection for --machine-readable
Add an optional argument for --machine-readable to select the output,
adding a new function to specifically write data to that output stream.
The possible choices are:
* --machine-readable: to stdout, like before
* --machine-readable=file:name-of-file: to the specified file
* --machine-readable=stream:stdout: explicitly to stdout
* --machine-readable=stream:stderr: explicitly to stderr

Adapt all the OCaml-based tools to use the new function, so the
--machine-readable choice is respected.
2018-08-23 18:03:39 +02:00
Pino Toscano
8863ccbd64 common/mltools: getopt: add Getopt.OptString
Introduce a new type of option with an optional string argument.
2018-08-23 16:04:27 +02:00
Pino Toscano
962e8b6255 lib: create: avoid one extra string allocation
When creating the qemu-img command, use the guestfs_int_cmd_add_arg &
guestfs_int_cmd_add_arg_format APIs to add the proper filename directly,
without creating a string for it.

This should cause no functional change.
2018-08-23 13:22:00 +02:00
Richard W.M. Jones
df11067329 mltools: JSON: unify JSON_parser type with JSON.json_t. 2018-08-23 09:56:51 +01:00
Richard W.M. Jones
f9994c7ffb mltools: JSON: Implement JSON.Null.
Implements the ‘null’ value.
2018-08-23 09:56:51 +01:00
Richard W.M. Jones
38f688b8b5 build: Rebuild docs/C_SOURCE_FILES and po/POTFILES.
Remove:

common/utils/libxml2-utils.c
common/utils/libxml2-utils.h

These were accidentally added by
commit a63d02f8f1 because of a bad
interactive rebase.
2018-08-22 15:28:50 +01:00
Richard W.M. Jones
279472f304 mltools: JSON: combine JSON.Int and JSON.Int64 into a single variant.
It was convenient to have these as separate variants when we were only
using this type to generate JSON.  However if we also use it to parse
JSON documents then integers in the document should only map to a
single variant.
2018-08-22 15:18:57 +01:00
Richard W.M. Jones
bd5c0b7c12 mltools: JSON: Implement JSON.Float.
This is not actually used anywhere, but it completes the JSON
implementation and it will be useful for unifying JSON with
JSON_parser.
2018-08-22 15:18:57 +01:00
Richard W.M. Jones
a63d02f8f1 mltools: JSON: Rename Yajl module as JSON_parser and move to common/mltools.
Commit bd1c5c9f4d changed all the code
to use Jansson instead of yajl.  However it didn't change the OCaml
module name (still Yajl).

This commit changes the module to a neutral name ("JSON_parser") and
moves it into common/mltools so it can be used by other tools.

This leaves us in a slightly awkward situation of having two JSON-ish
OCaml modules (JSON for creating trees and JSON_parser for parsing
them) with incompatible types.  That is left for future work to
resolve.  (It should be easier to do now that both modules live in the
same directory.)

This is just renaming and general refactoring.  There should be no
change in functionality.
2018-08-22 15:18:57 +01:00
Richard W.M. Jones
1b25fc393b build: Use LT_INIT in configure.ac.
Avoids the warning:
libtoolize: Remember to add 'LT_INIT' to configure.ac.

This is the new name for AC_PROG_LIBTOOL, so I removed that.

However to use this macro we must enable AC_USE_SYSTEM_EXTENSIONS.
(AC_GNU_SOURCE was removed back in 2011).
2018-08-22 11:36:36 +01:00
Bernhard M. Wiedemann
4d39faaa30 generator: Do not claim copyright for future years
This change helps to make libguestfs package build reproducible.
See https://reproducible-builds.org/ for why this is good.

Without this patch, building today's libguestfs in 2033, claims
 Copyright (C) 2009-2033 Red Hat Inc.
which cannot be correct.

This affected files like
/usr/include/guestfs-gobject.h
/usr/lib/perl5/vendor_perl/5.26.2/x86_64-linux-thread-multi/Sys/Guestfs.pm
/usr/lib64/python2.7/site-packages/guestfs.py
/usr/lib64/ocaml/guestfs/guestfs.mli

Commits like 212762c593
will take care of updating the year.
2018-08-22 11:29:29 +01:00
Nir Soffer
1368b1bb7d v2v: rhv-upload-plugin: Fix error formatting
Fix the error format so we actually format the arguments instead of
raising the format string and the arguments.

Here is an upload error formatted correctly with this change:

nbdkit: python[1]: error: /home/nsoffer/src/libguestfs/tmp/v2v.eC5yCl/rhv-upload-plugin.py:
pwrite: error: could not write sector offset 218911744 size 3584: 403 Forbidden:
b'{"explanation": "Access was denied to this resource.", "code": 403, "detail":
"Ticket u\'61ac0483-48e3-4984-84d6-438884ba8bb2\' expired", "title": "Forbidden"}'
2018-08-21 12:05:24 +01:00
Nir Soffer
a651169085 v2v: rhv-upload-plugin: Handle send failures
The oVirt server may fail a PUT request before reading all request body.
However before closing the connection, it writes a detailed error
response, that will make debugging issues like expired tickets much
easier to handle. If we don't get the response and log the error, the
error may be lost when the daemon log is rotated.

Change pwrite() and emulate_zero() to get the response after EPIPE,
failing with the error response from the oVirt server.

Here is an example error log when a ticket expires during import:

nbdkit: python[1]: error: /home/nsoffer/src/libguestfs/tmp/v2v.pRoyXm/rhv-upload-plugin.py:
pwrite: error: ('%s: %d %s: %r', 'could not write sector offset 1841154048 size 1024', 403,
'Forbidden', b'{"explanation": "Access was denied to this resource.", "code": 403, "detail":
"Ticket u\'6071e16f-ec60-4ff9-a594-10b0faae3617\' expired", "title": "Forbidden"}')
2018-08-21 12:05:24 +01:00