Commit d5b6f1df5f ("daemon: Allow parts of the daemon and APIs to be
written in OCaml.", 2017) contained a bug where in any OCaml function
that returns int64_t, the result was truncated to an int. This
particularly affected part_get_gpt_attributes as that returns large 64
bit numbers, but probably affects other functions too, undetected.
Fixes: commit d5b6f1df5f
(cherry picked from commit 882ef4d93a)
(cherry picked from commit 285b8fa92b)
Run this command across the source:
perl -pi.bak -e 's/(20[012][0-9])-20[12][012]/$1-2023/g' `git ls-files`
and remove changes to po{,-docs}/*.po{,t} (these will be regenerated
later when we run 'make dist').
In some places in the generator we were still generating "noalloc".
It was hidden from the previous regexp I used to replace these because
of string escaping.
Updates: commit a69cde79ca
This commit deprecates luks-open/luks-open-ro/luks-close for the more
generic sounding names cryptsetup-open/cryptsetup-close, which also
correspond directly to the cryptsetup commands.
The optional cryptsetup-open readonly flag is used to replace the
functionality of luks-open-ro.
The optional cryptsetup-open crypttype parameter can be used to select
the type (corresponding to cryptsetup open --type), which allows us to
open BitLocker-encrypted disks with no extra effort. As a convenience
the crypttype parameter may be omitted, and libguestfs will use a
heuristic (based on vfs-type output) to try to determine the correct
type to use.
The deprecated functions and the new functions are all (re-)written in
OCaml.
There is no new test here, unfortunately. It would be nice to test
Windows BitLocker support in this new API, however the Linux tools do
not support creating BitLocker disks, and while it is possible to
create one under Windows, the smallest compressed disk I could create
is 37M because of a mixture of the minimum support size for BitLocker
disks and the fact that encrypted parts of NTFS cannot be compressed.
Also synchronise with common module.
Use the cleanup handlers to free the structs (and list of structs) in
case their OCaml->C transformation fails for any reason; use calloc()
to not try to use uninitialized memory.
In case of lists, avoid allocating the memory for the array if there
are no elements, since the returned pointer in that case is either NULL,
or a free()-only pointer; also, set the list size only after the array
is allocated, to not confuse the XDR routines.
Add a way to generate OCaml interfaces for all the modules in the
daemon that implement APIs: this makes sure that for them the
interface of each function matches the actual API specified in the
generator.
We reimplemented some functions which can now be found in the OCaml
stdlib since 4.01 (or earlier). The functions I have dropped are:
- String.map
- |>
- iteri (replaced by List.iteri)
- mapi (replaced by List.mapi)
Note that our definition of iteri was slightly wrong: the type of the
function parameter was too wide, allowing (int -> 'a -> 'b) instead of
(int -> 'a -> unit).
I also added this new function to the Std_utils.String module as an
export from stdlib String:
- String.iteri
Thanks: Pino Toscano
If you have a struct containing ‘field’, eg:
type t = { field : int }
then previously to pattern-match on this type, eg. in function
parameters, you had to write:
let f { field = field } =
(* ... use field ... *)
In OCaml >= 3.12 it is possible to abbreviate cases where the field
being matched and the variable being bound have the same name, so now
you can just write:
let f { field } =
(* ... use field ... *)
(Similarly for a field prefixed by a Module name you can use
‘{ Module.field }’ instead of ‘{ Module.field = field }’).
This style is widely used inside the OCaml compiler sources, and is
briefer than the long form, so it makes sense to use it. Furthermore
there was one place in virt-dib where we are already using this new
style, so the old code did not compile on OCaml < 3.12.
See also:
https://forge.ocamlcore.org/docman/view.php/77/112/leroy-cug2010.pdf
This change allows parts of the daemon to be written in the OCaml
programming language. I am using the ‘Main Program in C’ method along
with ‘-output-obj’ to create an object file from the OCaml code /
runtime, as described here:
https://caml.inria.fr/pub/docs/manual-ocaml/intfc.html
Furthermore, change the generator to allow individual APIs to be
implemented in OCaml. This is picked by setting:
impl = OCaml <ocaml_function>;
The generator creates ‘do_function’ (the same one you would have to
write by hand in C), with the function calling the named
‘ocaml_function’ and dealing with marshalling/unmarshalling the OCaml
parameters.
After the previous refactoring, we are able to link the daemon to
common/utils, and also remove some of the "duplicate" functions that
the daemon carried ("duplicate" in quotes because they were often not
exact duplicates).
Also this removes the duplicate reimplementation of (most) cleanup
functions in the daemon, since those are provided by libutils now.
It also allows us in future (but not in this commit) to move utility
functions from the daemon into libutils.
The new module ‘Std_utils’ contains only functions which are pure
OCaml and depend only on the OCaml stdlib. Therefore these functions
may be used by the generator.
The new module is moved to ‘common/mlstdutils’.
This also removes the "<stdlib>" hack, and the code which copied the
library around.
Also ‘Guestfs_config’, ‘Libdir’ and ‘StringMap’ modules are moved
since these are essentially the same.
The bulk of this change is just updating files which use
‘open Common_utils’ to add ‘open Std_utils’ where necessary.
Previously the generator did not change any string returned from the
daemon. Thus guestfs_list_devices (for example) might return internal
device names like /dev/vda (if virtio-blk was in use).
This changes calls to the daemon so that returned strings are
annotated as plain strings, devices or mountables:
old ---> new
RString "uuid" RString (RPlainString "uuid")
RString "device" RString (RDevice "device")
RString "fs" RString (RMountable "fs")
For hash tables, keys and values must be annotated separately. For
example a hash table of mountables (keys) -> plain strings (values)
would be annotated like this:
old ---> new
RHashtable "fses" RHashtable (RMountable, RPlainString, "fses")
The daemon calls reverse_device_name_translation (currently a no-op)
for devices and mountables.
Note that this has no effect for calls which are handled on the
library side.
(cherry picked from commit 6b77cc196ecb8d7e1d73592ef65a189a7412c97c)
Previously we had lots of types like String, Device, StringList,
DeviceList, etc. where Device was just a String with magical
properties (but only inside the daemon), and DeviceList was just a
list of Device strings.
Replace these with some simple top-level types:
String
StringList
and move the magic into a subtype.
The change is mechanical, for example:
old ---> new
FileIn "filename" String (FileIn, "filename")
DeviceList "devices" StringList (Device, "devices")
Handling BufferIn is sufficiently different from a plain String
throughout all the bindings that it still uses a top-level type.
(Compare with FileIn/FileOut where the only difference is in the
protocol, but the bindings can uniformly treat it as a plain String.)
There is no semantic change, and the generated files are identical
except for a minor change in the (deprecated) Perl
%guestfs_introspection table.
In every instance where we used the ‘cancel_stmt’ parameter of
these macros:
ABS_PATH
NEED_ROOT
the value was only ever ‘cancel_receive ()’ or empty. We only use
‘cancel_receive’ for FileIn functions, so replace it with a simple
flag for whether the current function is a FileIn function.
This also removes some incorrect comments about macros that cannot be
used with FileIn functions when in fact they can.
As a simple consequence of the previous commit, every instance of the
‘fail_stmt’ parameter to one of the following macros:
RESOLVE_DEVICE
RESOLVE_MOUNTABLE
REQUIRE_ROOT_OR_RESOLVE_DEVICE
REQUIRE_ROOT_OR_RESOLVE_MOUNTABLE
is now ‘return’ and therefore we can replace it in the macro and drop
the parameter completely.
The generated code had:
...
done_no_free:
return;
}
There was no possible code between the label and the return statement,
so both the label and the return statement are redundant. The
instances of ‘goto done_no_free’ can simply be replaced by ‘return’.
The only small complication is that there is a label just before this
which contains some optional code. So we might have ended up with
generated code looking like:
done:
// if there was no optional code, this would be empty
}
which provokes a parse error in C. Therefore I had to add a semicolon
after the ‘done:’ label. This will be removed in a subsequent commit,
so it's just to make the current commit bisectable.
Sort the functions so the output is stable.
This changes the order in which the C API tests run. Previously we
ran the newest tests first, which was useful when we were frequently
adding new APIs. Now we run them in sorted order.
Run the following command over the source:
perl -pi.bak -e 's/(20[01][0-9])-2016/$1-2017/g' `git ls-files`
(Thanks Rich for the perl snippet, as used in past years.)
For a very long time we have maintained two sets of utility functions,
in mllib/common_utils.ml and generator/utils.ml. This changes things
so that the same set of utility functions can be shared with both
directories.
It's not possible to use common_utils.ml directly in the generator
because it provides several functions that use modules outside the
OCaml stdlib. Therefore we add some lightweight post-processing which
extracts the functions using only the stdlib:
(*<stdlib>*)
...
(*</stdlib>*)
and creates generator/common_utils.ml and generator/common_utils.mli
from that. The effect is we only need to write utility functions
once.
As with other tools, we still have generator-specific utility
functions in generator/utils.ml.
Also in this change:
- Use String.uppercase_ascii and String.lowercase_ascii in place
of deprecated String.uppercase/String.lowercase.
- Implement String.capitalize_ascii to replace deprecated
String.capitalize.
- Move isspace, isdigit, isxdigit functions to Char module.
This mostly mechanical change moves all of the libguestfs API
lists of functions into a struct in the Actions module.
It also adds filter functions to be used elsewhere to get
subsets of these functions.
Original code Replacement
all_functions actions
daemon_functions actions |> daemon_functions
non_daemon_functions actions |> non_daemon_functions
external_functions actions |> external_functions
internal_functions actions |> internal_functions
documented_functions actions |> documented_functions
fish_functions actions |> fish_functions
*_functions_sorted ... replacement as above ... |> sort
Commit b91b39e06a changed the separator to
':', although this creates parsing issues when there are fields with
colons (for example lv_tags=imgbased:pool).
Change once more the separator, but this time using a non-printable
character such as '\r': while it will produce uglier debug logs, this
should greatly reduce the possibilities of conflicts with texts of
metadata.
The separator character (currently comma) was picked arbitrarily to
allow us to parse the output of commands like 'lvs' for APIs such as
'lvs-full'. Unfortunately the choice of comma conflicts with the 'lvs
-o modules' column, since the list of modules is separated by commas,
breaking our parser.
Change the separator to another arbitrary character (colon) which
hopefully is not used by any column.
See also:
https://bugzilla.redhat.com/show_bug.cgi?id=1278878#c11
Updating gnulib has caused -Wformat-signedness to be enabled. This
has revealed many problems in C format strings. The fixes here fall
into the following main categories:
- Using %d with an unsigned parameter.
- %x and %o expect an unsigned argument.
- uid_t and gid_t are unsigned on Linux. The safe way to print these
is to cast them to uintmax_t and then print them using the %ju
modifier (see http://stackoverflow.com/a/1401581).
- Using %d to print an enum. Since enums may be either char or int,
I fixed this by casting the enum to int.
- strtol_error & lzma_ret are both unsigned types.
When dealing with DeviceList parameters, the generator produces code
similar to the following:
CLEANUP_FREE_STRING_LIST char **devices = NULL;
[...]
devices = malloc (sizeof (char *) * (args.devices.devices_len+1));
{
size_t i;
for (i = 0; i < args.devices.devices_len; ++i)
RESOLVE_DEVICE (args.devices.devices_val[i], devices[i],
, goto done);
devices[i] = NULL;
}
The block hidden within the RESOLVE_DEVICE macro is supposed to
assign something to devices[i]; on the other hand, the code in
RESOLVE_DEVICE can cause to just end (with an error) the current RPC,
which would cause the cleanup of the "devices" array... whose members
from the i-th to the (args.devices.devices_len-1)-th would be garbage
pointers, causing random memory to be free'd (and thus crashing the
daemon).
Avoid the access to garbage memory just by having a cleaned "devices"
array, so there will be always a NULL element after the initialized
members.
Add a test for vgcreate which passes a wrong device path causing the
situation above, to test that vgcreate would fail gracefully.
Handy macro to reply the right way for an unavailable feature.
While generally used so far in generated code, it can shorten that a
bit, and avoid copy&paste when wanting to do manual feature checking.
Support the possibility to have optional groups always enabled (e.g.
because they were present in the past, and they need to be kept for
users).
Add and use few helper optgroups-related functions to deal also with
them.
