Commit Graph

10245 Commits

Author SHA1 Message Date
Richard W.M. Jones
61d4891ef4 mllib: Split ‘Common_utils’ into ‘Std_utils’ + ‘Common_utils’.
The new module ‘Std_utils’ contains only functions which are pure
OCaml and depend only on the OCaml stdlib.  Therefore these functions
may be used by the generator.

The new module is moved to ‘common/mlstdutils’.

This also removes the "<stdlib>" hack, and the code which copied the
library around.

Also ‘Guestfs_config’, ‘Libdir’ and ‘StringMap’ modules are moved
since these are essentially the same.

The bulk of this change is just updating files which use
‘open Common_utils’ to add ‘open Std_utils’ where necessary.
2017-07-10 17:01:59 +01:00
Richard W.M. Jones
989292c146 mllib: Move Xml (libxml2) OCaml bindings to common/mlxml.
The ‘Xml’ module is a self-contained library of bindings for libxml2,
with no other dependencies.

Move it to a separate ‘common/mlxml’ directory.

This is not pure refactoring.  For unclear reasons, the previous
version of ‘Xml.parse_file’ read the whole file into memory and then
called ‘xmlReadMemory’.  This was quite inefficient, and unnecessary
because we could use ‘xmlReadFile’ to read and parse the file
efficiently.  Changing the code to use ‘xmlReadFile’ also removes the
unnecessary dependency on ‘Common_utils.read_whole_file’.
2017-07-10 17:01:59 +01:00
Richard W.M. Jones
9254c8f152 mllib: Move Progress OCaml bindings to common/mlprogress.
The ‘Progress’ module is a self-contained library with the only
dependencies being:

 - the C ‘progress’ implementation

Move it to a separate ‘common/mlprogress’ directory.

This change is pure code refactoring.
2017-07-10 17:01:59 +01:00
Richard W.M. Jones
746a87cbf8 mllib: Move Visit OCaml bindings to common/mlvisit.
The ‘Visit’ module is a self-contained library with the only
dependencies being:

 - the C ‘visit’ implementation

 - the guestfs OCaml bindings

Move it to a separate ‘common/mlvisit’ directory.

This change is not entirely refactoring.  Two other fixes are made:

 - remove unsafe use of CLEANUP_FREE from a function which could
   raise an OCaml exception (cleanup handlers would not be called
   correctly if the exception is thrown)

 - don't link directly to common/visit/visit.c, but instead use
   the library (common/visit/libvisit.la)
2017-07-10 17:01:59 +01:00
Pino Toscano
28c375701d builder: templates: pass empty proxy to d-i when not set
When there is no proxy set, pass an empty string to the Debian
Installer.  This way, the installer will not ask for a proxy.
2017-07-10 17:56:13 +02:00
Pino Toscano
941545bde6 builder: templates: save NVRAM before removing guest
When installing guests that need NVRAM variables, the cleanup of the
guest with `virsh undefine` will remove that file too, which is not what
we want.  Instead, compress the NVRAM file right before the cleanup,
to ensure we have it.

Also, fix the filename for it, removing the double "-nvram" suffixes.
2017-07-10 17:56:13 +02:00
Pino Toscano
a67346b2fe builder: templates: improve debian.preseed
- set the mirror to deb.debian.org, which is the official redirector, so
  generated images will use the closest mirror depending on their
  location
- automake the task selection, using the choice we want
- do a full upgrade of the distro, so the template is already up-to-date
- install grub on the default device, which should also be the only one
  in the automated installation
2017-07-10 17:56:13 +02:00
Pino Toscano
eb830e89dc builder: templates: generalize debian location
Add an helper function to translate our architecture to the Debian
naming, and use it to get the location for all the architectures.
2017-07-10 17:56:13 +02:00
Pino Toscano
2ff6938935 builder: templates: update Debian preseed from Stretch
Update our debian.preseed file from a Stretch example file, leaving all
the customization done already.
2017-07-10 17:56:13 +02:00
Richard W.M. Jones
4cea914d24 builder: make-template: Combine both os_variant_of_os functions into one.
Cosmetic only change, no functional change.
2017-07-10 16:49:32 +01:00
Richard W.M. Jones
77d25587b8 v2v: docs: VDSM location of virt-v2v log file.
See this bug for background information:
https://bugzilla.redhat.com/show_bug.cgi?id=1350465

Thanks: Tomáš Golembiovský, Pino Toscano
2017-07-07 17:17:54 +01:00
Richard W.M. Jones
d96209ef07 lib: libvirt: Pass copyonread flag through to the libvirt XML (RHBZ#1466563).
We were dropping the add_drive copyonread flag when using the libvirt
backend.  This resulted in significant performance degradation (2x-3x
slower) when running virt-v2v against VMware servers.

Thanks: Kun Wei.
2017-07-01 11:40:12 +01:00
Richard W.M. Jones
38bf2a0f97 v2v: Allow -i libvirtxml to open network disks over http or https.
Currently -i libvirtxml mode only works for local files or NBD disks.
The purpose of NBD is to support virt-p2v.

This change adds support for network disks over http or https, ie:

    <disk type='network' device='disk'>
      <driver name='qemu' type='raw'/>
      <source protocol="http" name="/scratch/disk.img">
        <host name="server.example.com" port="80"/>
      </source>
      <target dev='hda' bus='ide'/>
    </disk>

This is just for testing.  It's especially useful for exercising curl
support in qemu without requiring VMware to be available.
2017-06-29 16:01:27 +01:00
Pavel Butsykin
40fcb3e4d2 launch: add support for autodetection of appliance image format
This feature allows you to use different image formats for the fixed
appliance. The raw format is used by default.

Signed-off-by: Pavel Butsykin <pbutsykin@virtuozzo.com>
2017-06-29 12:13:12 +01:00
Richard W.M. Jones
dd4e95c636 builder: Add debian-9 template. 2017-06-27 10:04:45 +01:00
Richard W.M. Jones
669739a16c builder: make-template: Remove trailing \n in Debian index-fragment.
Caused the following error when running make-template:

  syntax error at line 22: syntax error
  virt-index-validate: ‘debian-9.index-fragment’ could not be validated, see errors above
2017-06-27 10:04:45 +01:00
Richard W.M. Jones
0e7c0877be builder: Support Debian 9 ("Stretch") in make-template. 2017-06-27 10:04:45 +01:00
Richard W.M. Jones
cfab7e122b configure: Fix the release date so that it's impossible to forget about.
Add a RELEASES file which contains the release date for each version
of libguestfs.  When a new version is released, the configure script
checks that the date has not been omitted from the file.

This also fixes the release date for version 1.37.16 which I omitted
from commit 9455f21237.
2017-06-23 15:02:19 +01:00
Richard W.M. Jones
dc6bf1d847 m4: Add utimbuf.m4 to gitignore.
Updates commit 01a6d88660.
2017-06-22 19:14:02 +01:00
Richard W.M. Jones
01a6d88660 Update gnulib to latest. 2017-06-22 15:19:44 +01:00
Richard W.M. Jones
fbb0cd8582 builder: Allow large variable length arrays.
Gnulib added: -Wvla-larger-than=4031, get rid.
2017-06-22 15:19:44 +01:00
Richard W.M. Jones
d5467855e5 build: Force -Wformat-truncation=1.
Gnulib changed the default to -Wformat-truncation=2, but GCC 7 gives a
bunch of completely incorrect warnings.  Change it back.
2017-06-22 15:19:44 +01:00
Richard W.M. Jones
b8600a2a1c build: Force -Wimplicit-fallthrough=4.
New gnulib uses ‘-Wimplicit-fallthrough=5’ which warns (and hence
errors) on every switch statement with an implicit fallthrough.

We could use the new GCC attribute here:

  #ifndef FALLTHROUGH
  # if __GNUC__ < 7
  #  define FALLTHROUGH ((void) 0)
  # else
  #  define FALLTHROUGH __attribute__ ((__fallthrough__))
  # endif
  #endif

but instead this changes the level to 4 which allows us to use
/*FALLTHROUGH*/ comments (which we are already using) to mark cases
where we expect fallthrough.
2017-06-22 15:19:44 +01:00
Richard W.M. Jones
f861c13855 maintainer: Unset LC_ALL before running Python script to list bugs.
Gnulib installs two files: ‘GNUmakefile’ and ‘maint.mk’.  The first
one runs the second one, and the second one sets LC_ALL=C universally.

However this breaks the ‘bugzilla’ script (or Python) because some
bugs contain non-ASCII characters in their Summary.  The script prints
this error:

  Traceback (most recent call last):
    File "/usr/bin/bugzilla", line 1117, in <module>
      main()
    File "/usr/bin/bugzilla", line 1112, in main
      _format_output(bz, opt, buglist)
    File "/usr/bin/bugzilla", line 702, in _format_output
      print(format_field_re.sub(bug_field, opt.outputformat))
  UnicodeEncodeError: 'ascii' codec can't encode characters in position 108-109: ordinal not in range(128)

The simplest fix for this is to unset LC_ALL before running the
external script, so that the normal locale settings are used.
2017-06-22 14:32:26 +01:00
Richard W.M. Jones
d98b381ef6 v2v: Document vdsm-compat-option in machine-readable output.
Thanks: Ming Xie
2017-06-22 11:52:02 +01:00
Richard W.M. Jones
9455f21237 Version 1.37.16. v1.37.16 2017-06-21 22:43:37 +01:00
Richard W.M. Jones
13cc6faedb maintainer: New layout of libguestfs website. 2017-06-21 22:23:12 +01:00
Richard W.M. Jones
7e29760d28 Version 1.37.15. v1.37.15 2017-06-21 21:56:11 +01:00
Richard W.M. Jones
68a88a6b0e mllib: Handle empty files correctly in read_first_line_from_file.
The OCaml function ‘input_line’ throws an End_of_file exception if the
end of the file is read before any other input, ie. if the file here
is zero length.  Return an empty string instead.

This can produce peculiar errors such as:

$ virt-customize ... --root-password file:/dev/null
libguestfs: uncaught OCaml exception in getopt callback: End_of_file
2017-06-21 18:14:18 +01:00
Richard W.M. Jones
9fdc495602 appliance/init: Allow daemon stack traces to be captured by gdb.
This fragment, not enabled by default, allows stack traces to be
captured and displayed by gdb in case the daemon crashes.
2017-06-21 16:12:09 +01:00
Pino Toscano
1ca2a8b7ba resize: make sure the input disk is read-only
Fixes commit 8a2b0738d1.
2017-06-21 16:34:05 +02:00
Pino Toscano
713243b3cb python: PEP 8: remove extra spaces in example lines
Remove (before opening round bracket) whitespaces in the documentation
of the Python binding, according to the PEP 8 specification.

This is just code reformatting, with no behaviour changes; no content
changed beside whitespaces, so "git diff -w" gives an empty diff.
2017-06-21 13:46:58 +02:00
Pino Toscano
dec7a267d8 dib: use Common_utils.run_commands
Use the new helper for spawning parallel commands, and redirecting their
stdout to file.
2017-06-20 15:10:48 +01:00
Pino Toscano
4f02613b73 mllib: add new Common_utils.run_commands
Mostly modelled after a snippet implemented in dib, it is an helper
function to run multiple commands in parallel, waiting for all of them
at once, and returning all their exit codes.  It is possible to pass
custom descriptors for collecting stdout and stderr of each command.

Common_utils.run_command is adapted to use few helper methods used by
run_commands, so all the existing code using it keeps working; in
addition, it gets labelled parameters for stdout and stderr FDs.

Add a simple unit tests for them.
2017-06-20 15:10:48 +01:00
Richard W.M. Jones
19498b199d gitignore: Remove a few unused patterns from this file. 2017-06-19 14:50:18 +01:00
Richard W.M. Jones
db8fdfd8d6 lib: direct, uml: Refactor unblocking SIGTERM code.
A small refactoring of
commit 1f1cd11cf0.
2017-06-13 08:49:05 +01:00
Richard W.M. Jones
92995c5564 lib: Move close_file_descriptors into guestfs-internal.h.
This macro is only used by the library, so move it to
lib/guestfs-internal.h.
2017-06-13 08:42:29 +01:00
Richard W.M. Jones
8a98fe91b8 daemon: lvm: Pass --yes option to force pvresize (RHBZ#1460577).
LVM2 >= 2.02.171 requires the ‘--yes’ option to force pvresize to work
in various circumstances, eg. reducing the size of an existing PV.

Pass this flag unconditionally.

Note this does NOT break earlier versions which just ignore this flag.
2017-06-12 13:24:26 +01:00
Richard W.M. Jones
1f1cd11cf0 lib: direct, uml: Unblock SIGTERM in the hypervisor and recovery processes (RHBZ#1460338).
If SIGTERM is blocked in the main program, then it ends up still being
blocked in the subprocess after we fork.  This means that we cannot
kill qemu by sending SIGTERM to it.  This commit fixes the problem by
unblocking SIGTERM unconditionally after fork.

Thanks: wtfuzz on IRC for reporting and analysis.
2017-06-12 13:24:26 +01:00
Richard W.M. Jones
5856323e6f lib: create: Allow any [[:alnum:]]+ string as a backingfmt parameter (RHBZ#1459979).
If you use the libguestfs tools which open disk images read-only
(eg. virt-df), with formats such as 'vdi', then you will see an error:

  error: invalid value for backingformat parameter 'vdi'

This is because opening a disk image read-only will try to create a
qcow2 file with the original image as a backing file.  However the
list of permitted backing formats was very restrictive and did not
include 'vdi' (nor many other uncommon formats).

Instead of using a whitelist for backing formats, just validate that
the string is alphanumeric and short.

Thanks: Mike Goodwin for reporting the bug.
2017-06-09 12:57:03 +01:00
Richard W.M. Jones
0b8556695f daemon: ldm: Use CLEANUP_FREE on list of static strings.
This likely causes a double-free in the ‘ldmtool_scan_devices’ API.
2017-06-09 12:57:03 +01:00
Richard W.M. Jones
4e05ad9e1c docs: Document common/qemuopts subdirectory. 2017-06-08 19:18:06 +01:00
Richard W.M. Jones
ab1620b8d0 builder: Add cirros-0.3.5. 2017-06-04 09:39:39 +01:00
Richard W.M. Jones
d606edfba4 builder: templates: Ensure a stable sort when creating index file. 2017-06-04 09:39:39 +01:00
Richard W.M. Jones
3ca0110adc builder: Fix Unicode quotes in index file.
Fixes commit 126ef5d082.
2017-06-04 09:39:23 +01:00
Pino Toscano
76eeb0afc9 v2v: tell v2v the real root device to mkinitrd
Complementary fix of commit 2d25872df3.
On SLES 11 SP4 with kdump enabled mkinitrd calls mkdumprd which calls
mkinitrd, but mkdumprd doesn't have any clue of the root device.

Call mkinitrd with rootdev environment variable to tell them all
what device to use as root.

Tested-By: Cédric Bosdonnat <cbosdonnat@suse.com>
2017-06-02 14:59:21 +01:00
Richard W.M. Jones
8c89299d8a mllib: Export which in "stdlib mode".
As with the previous commit, this was already being compiled in, but
not exported, so there's no substantive change here.
2017-06-01 09:07:05 +01:00
Richard W.M. Jones
5e49e3ff53 mllib: Export prog, quiet, trace, verbose in "stdlib mode".
These were already included in the .ml implementation file, but just
weren't being exported in the interface, so there's no real change
here.  It just allows these functions to be used.
2017-06-01 09:07:05 +01:00
Pino Toscano
2c41f001de btrfs_replace: fix position of subcommand options
btrfs-progs 4.11 is more strict with option parsing, so subcommand
options must appear now before the subcommand arguments.
2017-05-30 13:15:55 +02:00
Pino Toscano
7e6c16f1e7 v2v: add crypto support (RHBZ#1451665)
Make use of the additional command line arguments, and API needed to
decrypt LUKS partitions.  This extends to v2v the work done in other
OCaml tools with commit 6b26a0cce4,
since it seems to be working fine after a basic testing.

Related to: RHBZ#1362649
2017-05-22 11:31:45 +02:00