BUGS
docs/C_SOURCE_FILES
po/POTFILES
po/POTFILES-ml
These files are normally updated by ‘make dist’ but as we've not had a
release in a while they had not been updated for some time.
Instead of generating the p2v kernel config using the OCaml generator,
create a Perl script to do this job, mostly at build time. This is done
to rely less on the generator for p2v, and because the generation of
these sources is quick enough that it can be done at build time (instead
of shipping the generated sources in dist tarballs).
The generate-p2v-config.pl mimics what generator/p2v_config.ml --
namings, and general structure are kept close to that for comparison.
The two C sources are created at build time by the script; however, the
p2v-config.h header is generated at configure time: this is done because
p2v-config.h is included by p2v.h (another header), which in turn is
included by all the p2v C sources -- automake is not able to properly
resolve the dependency, and thus it would not be generated properly.
Currently virt-v2v has few custom C-based functions for libvirt
operations, which are limited in what they do, and there is a lot of
duplicated code.
Instead, switch to ocaml-libvirt for all the libvirt interaction
currently done by the Libvirt_utils module. This has few advantages:
- each input & output module now opens a libvirt connection only once,
only when needed
- no need to pass URIs and passwords around, if not needed
- a wider range of libvirt APIs can now be used, with no need to create
bindings manually
The hierarchy of input_libvirt* classes is changed to take a Lazy object
with the libvirt connection, accessing it through a "proctected" method:
this way, the connection is opened only at the first access.
Also, the Libvirt_utils module now is just helpers around the Libvirt
module, to centralize error handling, and few common operations.
In some places when generating XML output in C code we use some clever
macros:
start_element ("memory") {
attribute ("unit", "MiB");
string_format ("%d", g->memsize);
} end_element ();
This commit which is mostly refactoring moves the repeated definitions
of these macros into a common header file.
I also took this opportunity to change / clean up the macros:
- The macros are now documented properly.
- comment() and empty_element() macros are now available everywhere.
- Error handling has been made generic.
- Added do..while(0) around some of the macros to make them safe to
use in all contexts.
Remove:
common/utils/libxml2-utils.c
common/utils/libxml2-utils.h
These were accidentally added by
commit a63d02f8f1 because of a bad
interactive rebase.
Commit bd1c5c9f4d changed all the code
to use Jansson instead of yajl. However it didn't change the OCaml
module name (still Yajl).
This commit changes the module to a neutral name ("JSON_parser") and
moves it into common/mltools so it can be used by other tools.
This leaves us in a slightly awkward situation of having two JSON-ish
OCaml modules (JSON for creating trees and JSON_parser for parsing
them) with incompatible types. That is left for future work to
resolve. (It should be easier to do now that both modules live in the
same directory.)
This is just renaming and general refactoring. There should be no
change in functionality.
common/mlutils: Unix_utils.StatVFS.statvfs: This commit implements a
full-featured binding for the statvfs(3) function.
We then use this to reimplement the daemon statvfs API in OCaml.
Note that the Gnulib fallback is dropped in this commit. It
previously referenced non-existent field names in the fs_usage struct
so it didn't work. Also it's not necessary as POSIX has supported
statvfs(3) since 2001, it's supported in *BSD, macOS > 10.4, and there
is already a Windows fallback.
This directory which previously contained random modules and functions
now has an official purpose: to be the place for any OCaml utility
needed by the OCaml virt tools.
This is just code movement, I didn't (yet) rename or move any of the
modules.
Move the last remaining function ‘guestfs_int_download_to_tmp’ to
lib/inspect-icon.c (the main, but not only user of this function).
Then remove lib/inspect.c entirely.
This is not quite code motion because I updated the comment for the
function to reflect what it does in reality.
hivex has a function hivex_value_string. We were not calling it under
the mistaken belief that because hivex implements this using iconv,
the function wouldn't work inside the daemon. Instead we
reimplemented the functionality in the library.
This commit deprecates hivex_value_utf8 and removes the library side
code. It replaces it with a plain wrapper around hivex_value_string.
Thanks: Pino Toscano
This commit bundles the ocaml-augeas library (upstream here:
http://git.annexia.org/?p=ocaml-augeas.git;a=summary). It's identical
to the upstream version and should remain so.
We can work towards using system ocaml-augeas, when it's more widely
available.
The following functions were previously reimplemented in OCaml. This
commit replaces them with calls to the C functions:
- is_root_device
- prog_exists
- udev_settle
plus the internal get_verbose_flag function.
However note that we cannot do this for every utility function. In
particular the C function must not call any reply* functions.
This also reimplements the lv_canonical function in OCaml. We cannot
call the original C function because it calls reply_with_perror which
would break the OCaml bindings.
Move the list_filesystems API into the daemon, reimplementing it in
OCaml. Since this API makes many other API calls, it runs a lot
faster in the daemon.
The previously library-side ‘file_architecture’ API is reimplemented
in the daemon, in OCaml.
There are some significant differences compared to the C
implementation:
- The C code used libmagic. That is replaced by calling the ‘file’
command (because that is simpler than using the library).
- The C code had extra cases to deal with compressed files. This is
not necessary since the ‘file’ command supports the ‘-z’ option
which transparently looks inside compressed content (this is a
consequence of the change above).
This commit demonstrates a number of techniques which will be useful
for moving inspection code to the daemon:
- Moving an API from the C library to the OCaml daemon.
- Calling from one OCaml API inside the daemon to another (from
‘Filearch.file_architecture’ to ‘File.file’). This can be done and
is done with C daemon APIs but correct reply_with_error handling is
more difficult in C.
- Use of Str for regular expression matching within the appliance.
This change allows parts of the daemon to be written in the OCaml
programming language. I am using the ‘Main Program in C’ method along
with ‘-output-obj’ to create an object file from the OCaml code /
runtime, as described here:
https://caml.inria.fr/pub/docs/manual-ocaml/intfc.html
Furthermore, change the generator to allow individual APIs to be
implemented in OCaml. This is picked by setting:
impl = OCaml <ocaml_function>;
The generator creates ‘do_function’ (the same one you would have to
write by hand in C), with the function calling the named
‘ocaml_function’ and dealing with marshalling/unmarshalling the OCaml
parameters.
After the previous refactoring, we are able to link the daemon to
common/utils, and also remove some of the "duplicate" functions that
the daemon carried ("duplicate" in quotes because they were often not
exact duplicates).
Also this removes the duplicate reimplementation of (most) cleanup
functions in the daemon, since those are provided by libutils now.
It also allows us in future (but not in this commit) to move utility
functions from the daemon into libutils.
Create a module ‘C_utils’ containing functions like ‘drive_name’ and
‘shell_unquote’ which come from the C utilities.
The new directory ‘common/mlutils’ also contains the ‘Unix_utils’
wrappers around POSIX functions missing from the OCaml stdlib.
This refactoring change just moves the cleanup functions around in the
common/utils directory.
libxml2 cleanups are moved to a separate object file, so that we can
still link to libutils even if the main program is not using libxml2
anywhere. Similarly gnulib cleanups.
cleanup.c is renamed to cleanups.c.
A new header file cleanups.h is introduced which will replace
guestfs-internal-frontend-cleanups.h (fully replaced in a later commit).
The new module ‘Std_utils’ contains only functions which are pure
OCaml and depend only on the OCaml stdlib. Therefore these functions
may be used by the generator.
The new module is moved to ‘common/mlstdutils’.
This also removes the "<stdlib>" hack, and the code which copied the
library around.
Also ‘Guestfs_config’, ‘Libdir’ and ‘StringMap’ modules are moved
since these are essentially the same.
The bulk of this change is just updating files which use
‘open Common_utils’ to add ‘open Std_utils’ where necessary.
The ‘Xml’ module is a self-contained library of bindings for libxml2,
with no other dependencies.
Move it to a separate ‘common/mlxml’ directory.
This is not pure refactoring. For unclear reasons, the previous
version of ‘Xml.parse_file’ read the whole file into memory and then
called ‘xmlReadMemory’. This was quite inefficient, and unnecessary
because we could use ‘xmlReadFile’ to read and parse the file
efficiently. Changing the code to use ‘xmlReadFile’ also removes the
unnecessary dependency on ‘Common_utils.read_whole_file’.
The ‘Progress’ module is a self-contained library with the only
dependencies being:
- the C ‘progress’ implementation
Move it to a separate ‘common/mlprogress’ directory.
This change is pure code refactoring.
The ‘Visit’ module is a self-contained library with the only
dependencies being:
- the C ‘visit’ implementation
- the guestfs OCaml bindings
Move it to a separate ‘common/mlvisit’ directory.
This change is not entirely refactoring. Two other fixes are made:
- remove unsafe use of CLEANUP_FREE from a function which could
raise an OCaml exception (cleanup handlers would not be called
correctly if the exception is thrown)
- don't link directly to common/visit/visit.c, but instead use
the library (common/visit/libvisit.la)
The yara_scan API parses the file generated by the daemon counterpart
function and returns the list of yara_detection structs to the user.
It writes the daemon's command output on a temporary file and parses it,
deserialising the XDR formatted yara_detection structs.
It returns to the caller the list of yara_detection structs generated by
the internal_yara_scan command.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
The yara_load API allows to load a set of Yara rules contained within a
file on the host.
Rules can be in binary format, as when compiled with yarac command, or
in source code format. In the latter case, the rules will be first
compiled and then loaded.
Subsequent calls of the yara_load API will result in the discard of the
previously loaded rules.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
