90 Commits

Author SHA1 Message Date
Richard W.M. Jones
50486232db df: Allow parallel test to be skipped.
It currently fails because of
https://bugzilla.redhat.com/show_bug.cgi?id=1025942
2013-11-02 08:31:27 +00:00
Richard W.M. Jones
74d9fdf56f df: parallel: Add a debug message when work function returns an error.
Further attempts to find out why this hangs under Koji.
2013-10-08 12:15:52 +01:00
Richard W.M. Jones
5971fb2a70 virt-df: parallel: Send debugging messages to stderr.
Not stdout (ordinary program output) since that gets eaten by
the tests.

This fixes commit 67b9469754.
2013-09-12 09:16:31 +01:00
Richard W.M. Jones
67b9469754 virt-df: parallel: Compile debugging messages in always.
Enable debugging messages whenever LIBGUESTFS_DEBUG=1 / -v option, so
that we can track down possible race condition seen in Koji.
2013-09-03 14:58:15 +01:00
Richard W.M. Jones
4d7c14fdbb fish: Split off URI handling (for -a argument) from general options parsing.
This is so we will be able to reuse the same code in the OCaml tools.

This is just code motion.
2013-07-30 15:37:16 +01:00
Richard W.M. Jones
a3891430bc fish: Don't store xmlURIPtr directly in the drive struct.
Original drv_uri fields:

      xmlURIPtr uri;        /* URI */
      char *socket;         /* ?socket parameter from URI. */
      const char *format;   /* format (NULL == autodetect) */

New drv_uri fields:

      char *path;           /* disk path */
      char *protocol;       /* protocol (eg. "nbd") */
      char **server;        /* server(s) - can be NULL */
      char *username;       /* username - can be NULL */
      const char *format;   /* format (NULL == autodetect) */
      const char *orig_uri; /* original URI (for error messages etc.) */

This is just code motion.
2013-07-30 12:17:22 +01:00
Richard W.M. Jones
5c80f22b64 syntax-check: Remove unused ``#includes''. 2013-05-24 14:20:51 +01:00
Richard W.M. Jones
87690b10ff df: Move variable decl to top of function.
Since this code is no longer conditional, we don't need to have the
extra { ... } inside the case statement.

This tidies up commit 2f0a4d7c18.
2013-04-16 19:37:20 +01:00
Richard W.M. Jones
2f0a4d7c18 Require libxml2.
libxml2 is very commonly available on Linux distros and has also been
ported (and is widely available) on Mac OS X and Windows.  Therefore
simply require libxml2, and remove a lot of conditional code.
2013-04-16 16:46:01 +01:00
Richard W.M. Jones
349300af08 fish: Add -a URI (add remote storage) to options.
Add a remote drive by doing:

 guestfish -a ssh://example.com/path/to/disk.img

There are several different protocols supported, as explained in the
man page.

This affects all virt-* tools that use the common guestfish options
parsing code.
2013-04-16 15:28:31 +01:00
Richard W.M. Jones
142b874ce8 Remove use of gnulib progname module.
It's simpler to use the glibc 'program_invocation_short_name(3)'
feature, and fall back to a generic solution.  Also remove risky
assignments to argv[0].
2013-04-11 14:42:54 +01:00
Richard W.M. Jones
429ffda329 parallel: Propagate errors from worker threads and exit with error code.
In virt-df and virt-alignment-scan, ensure that errors that happen in
worker threads are propagated all the way up and result in
exit(EXIT_FAILURE).

Note that this makes the align/test-virt-alignment-scan-guests.sh test
fail (for a genuine reason).  This is fixed in the following commit.

This updates commit 8b90f55dc7.
2013-03-30 16:33:08 +00:00
Richard W.M. Jones
6c9a7fe561 Add --long-options option to most tools.
For example:

$ guestfish --long-options
--add
--cmd-help
--connect
--csh
--domain
--echo-keys
[etc.]

The idea of this is to make it easier to write a bash completion
script that accurately expands --<TAB> options for each command.
2013-03-28 14:46:20 +00:00
Richard W.M. Jones
3a895ba794 df/parallel.c: Fix if libvirt is not available at compile time. 2013-03-07 23:29:21 +00:00
Richard W.M. Jones
93feaa4ae8 parallel: Don't access the global handle from multiple threads.
libguestfs handles are not thread safe, and it's not safe even to read
settings from the handle from multiple threads (eg. guestfs_get_trace).
Stop doing this in the parallel library.  This caused fairly
reproducible segfaults when you enabled '-x' and/or '-v'.

This fixes commit 34e77af1bf.
2013-02-28 13:38:25 +00:00
Richard W.M. Jones
b6cbd980fb tests: Add tests of guests to virt-df and virt-alignment-scan. 2013-02-26 18:48:14 +00:00
Richard W.M. Jones
34e77af1bf Partial rewrite of virt-df to use parallel threads instead of single appliance.
Previously when you asked virt-df to show stats for all your libvirt
guests, it used a single appliance and tried to attach as many disks
as possible to it, even disks from different guests.

However this has been problematic: Not only is the code to do this
horrendously complex, but it's also insecure, and it doesn't interact
well with sVirt labels (see RHBZ#912499 comment 7).

In the meantime we discovered that running parallel appliances gives
you most of the performance of using a single appliance, but with a
lot less complexity and better guest isolation (see the documentation
in commit 680450f3b4).

Therefore this is a partial rewrite of virt-df so that in this case it
now uses parallel appliances.

Notes:

The '--one-per-guest' option is now the default and only way to do
things; this option now does nothing.

By default, the amount of parallelism to use is controlled by the
amount of free memory seen when virt-df starts up (subject to some
minima and maxima).  The user can control this through new command
line option '-P'.
2013-02-26 17:02:32 +00:00
Richard W.M. Jones
556e109765 tools: Introduce a mini-library for estimating max threads based on free memory.
This mini-library runs the 'free -m' command and greps the output to
estimate the max. number of libguestfs appliances we could run in
parallel in the remaining free memory.
2013-02-26 17:02:23 +00:00
Richard W.M. Jones
f8b92c0668 df: Always have TESTS= line even if appliance is disabled. 2013-02-26 17:02:23 +00:00
Richard W.M. Jones
04723b4dd1 lib: Create libutils convenience library.
The libutils convenience library is a place for code shared between
the main library, language bindings and virt tools.  Note that the
code is statically linked into both the library, each binding and each
tool, but this is an improvement because (a) the source is shared and
(b) libguestfs.so can export fewer private functions.

Currently it contains the cleanup functions, and the functions
guestfs___free_string_list function and guestfs___for_each_disk.

guestfs___for_each_disk has changed so that it no longer
unconditionally sets the error in the guestfs handle.  Instead callers
can control error handling.
2013-02-20 19:15:05 +00:00
Richard W.M. Jones
09c4f94c9d build: Separate out *_CPPFLAGS from *_CFLAGS.
This is pretty pointless.
2013-02-11 21:36:27 +00:00
Richard W.M. Jones
4255db65e5 tools: Use CLEANUP_* macros in several C tools. 2013-02-09 20:37:03 +00:00
Richard W.M. Jones
0d18a8b407 Update copyright dates for 2013. 2013-02-02 13:56:19 +00:00
Richard W.M. Jones
42bffcd00a Remove all occurrences of the bad_cast (lowercase) function.
Not to be confused with the libxml2 macro 'BAD_CAST' which converts
from 'signed char *' to 'unsigned char *'.

The 'bad_cast' function was defined and used all over the place as a
replacement for a '(char *)' cast.  I think it is better to make these
casts explicit, instead of hiding them in an obscure function.
2013-02-01 14:16:51 +00:00
Matthew Booth
27b995c841 Make internal-only functions and structures private
Certain functions are intended to be internal only, but we currently
export them anyway. This change moves them into a separate section of
guestfs.h protected by a GUESTFS_PRIVATE variable. This change also
enables private structs, but doesn't implement any.

This change only affects the C api. Language bindings aren't affected,
but probably should be in the future.
2013-01-30 17:27:01 +00:00
Richard W.M. Jones
62e775c350 Change the handling of private functions, safe_malloc etc.
Rename guestfs_safe_malloc et al to guestfs___safe_malloc etc.

To use the private functions, code now has to define
-DGUESTFS_PRIVATE_FUNCTIONS=1.  This will make it easier for us in
future to work out which programs are using these functions and to
minimize both the number of programs and the functions they are
calling.

Note that the Perl, Python, OCaml, Ruby and Java bindings use
guestfs_safe_* calls.  None of the other bindings do.  This is a bug
(in the bindings using those functions): these functions will call the
out of memory callback on failure.  This function defaults to abort(),
and since this happens from a language binding, there is no way to
change this default.
2012-12-15 19:41:29 +00:00
Richard W.M. Jones
5a2e320ec9 configure: Add --enable-code-profiling and --enable-code-coverage flags.
These configure flags enable code profiling (with gprof) and code
coverage (with gcov) respectively.

Although this is a nice idea, it's not currently very useful.

Libtool mangles filenames in such a way that gcov cannot locate its
datafiles.

Profiling is of dubious utility with libguestfs which is not CPU-bound
and relies extensively on running external programs (oprofile-like
system profiling that took into account libguestfs + qemu or
libguestfs + qemu + the appliance + filesystem tools *would* be
useful).

Also neither flag will help in capturing data from the appliance.
2012-12-04 14:11:14 +00:00
Richard W.M. Jones
380ce2eeaa fish: Make type == drv_N field private to guestfish.
The -N option / drv_N type only makes sense when we are compiling
guestfish, and just serves to complicate non-guestfish code.
2012-11-29 11:45:39 +00:00
Richard W.M. Jones
9e7daf2ee8 virt-df: Better output when multiple -a/-d options are specified (RHBZ#880801).
This is very well explained in the bugzilla comments:
https://bugzilla.redhat.com/show_bug.cgi?id=880801#c1

With this change, the disk name has a one or more plus signs ('+')
appended to show that there are additional disks:

  $ virt-df -a Win7x32TwoDisks-a -a Win7x32TwoDisks-b
  Filesystem                   1K-blocks    Used  Available  Use%
  Win7x32TwoDisks-a+:/dev/sda1    102396   24712      77684   25%
  Win7x32TwoDisks-a+:/dev/sda2  12478460 7403416    5075044   60%
  Win7x32TwoDisks-a+:/dev/sdb1    521212   55728     465484   11%
2012-11-29 11:45:39 +00:00
Richard W.M. Jones
3636c5fcff tests: Replace 'make extra-tests' with individual tests.
'make extra-tests' was a monolithic set of tests that did all sorts of
things: valgrind, tests over local guests, tests with upstream qemu,
tests with upstream libvirt, tests with the appliance attach method.

This made it hard to perform individual tests, eg. just valgrind
testing.  It was also hard to maintain because the tests were not
located in the same directories as the programs and sometimes
duplicated tests that were run elsewhere.

This commit splits up 'make extra-tests' into 5 separate targets:

 make check-valgrind              # run a subset of tests under valgrind
 make check-valgrind-local-guests # test under valgrind with local guests
 make check-with-appliance        # test with attach-method == appliance
 make check-with-upstream-qemu    # test with an alternate/upstream qemu
 make check-with-upstream-libvirt # test with an alternate/upstream libvirt

(You can also still run 'make extra-tests' which is now simply
a rule that runs the above 5 targets in order).

This replaces everything that was in the tests/extra directory,
so that has now gone.
2012-11-13 15:26:29 +00:00
Richard W.M. Jones
2d220f5da2 tools: Modify existing tools to use guestfs_{push,pop}_error_handler.
This is a shorter and more convenient way to disable errors
temporarily across calls.
2012-11-09 19:51:09 +00:00
Richard W.M. Jones
6272e49f51 syntax: Remove include <assert.h> where assert is not used.
Found by 'make syntax-check'.
2012-09-15 12:32:41 +01:00
Richard W.M. Jones
a1d981edea tests: Attach copyright and license (GPLv2+) notices to various test scripts.
For some reason these tests did not have license notices.
2012-08-31 19:12:27 +01:00
Richard W.M. Jones
2f97bf873b podwrapper: Add --license parameter, which is required.
This adds standard LICENSE and BUGS sections to all of the man pages
that are processed by podwrapper.

Modify all the calls to $(PODWRAPPER) to add the right --license
parameter according to the content.  Note that this relaxes the
license on some code example pages, making them effectively BSD-style
licensed.
2012-08-21 20:33:21 +01:00
Richard W.M. Jones
f1d98bbc79 man pages: Ensure consistent copyright/author sections, remove license
section.

Ensure each man page contains consistent COPYRIGHT and AUTHOR
sections.

Remove the LICENSE section.  We will add that back in podwrapper in a
later commit.
2012-08-21 20:16:29 +01:00
Richard W.M. Jones
6054051a9d fish: Rename fish/virt.c to fish/domain.c
This file handles the -d option for guestfish and other C command line
utilities.  Renaming this file makes it less confusing.
2012-07-19 16:11:26 +01:00
Richard W.M. Jones
f2ea617e22 build: Change calls to podwrapper.sh to use $(PODWRAPPER).
This will allow us to easily change the location of this
script in future.
2012-07-16 18:56:57 +01:00
Richard W.M. Jones
927ef14c58 run: Set MALLOC_PERTURB_ to a random value.
MALLOC_PERTURB_ is a glibc feature which causes malloc to wipe memory
before and after it is used, allowing both use-after-free and
uninitialized reads to be detected with relatively little performance
penalty:

  http://udrepper.livejournal.com/11429.html?nojs=1

Modify the ./run script so that it always sets this.

We were already using MALLOC_PERTURB_ in most tests.  Since ./run is
now setting this, we can remove it from individual Makefiles.  Most
TESTS_ENVIRONMENT will now simply look like this:

  TESTS_ENVIRONMENT = $(top_builddir)/run --test
2012-06-28 13:19:39 +01:00
Richard W.M. Jones
05d4e07918 tests: Add ./run --test option.
This option, when added via
  TESTS_ENVIRONMENT = [...] $(top_builddir)/run --test
allows us to run the tests and only print the full output (including
debugging etc) when the test fails.
2012-06-26 23:34:30 +01:00
Richard W.M. Jones
8735e92a1d virt-df: Use guestfs_max_disks instead of hard-coding limit of 25.
And comprehensively fix it so it works with > 26 disks.
2012-06-13 23:07:26 +01:00
Richard W.M. Jones
e0f36898cb virt-df: Don't fail immediately if a disk is not accessible (RHBZ#789504). 2012-05-03 15:38:08 +01:00
Richard W.M. Jones
9e2cdd309b virt-alignment-scan, virt-df: Add newlines after error messages. 2012-05-03 15:23:14 +01:00
Richard W.M. Jones
6aa95e87c1 Remove "convenience header" "gettext.h" and use <libintl.h> instead.
gettextize provides a local file called "gettext.h".  Remove this and
use <libintl.h> from glibc headers instead.

Most of this change is mechanical: #include <libintl.h> in every C
file which uses any gettext function.  But also we remove the
gettext.h file, and adjust the "_" macros.

Note that this effectively removes the ./configure --disable-nls
option, although we don't know if that ever worked.
2012-05-01 08:57:55 +01:00
Richard W.M. Jones
5d6a323b93 Check library still compiles without libvirt. 2012-04-18 14:25:15 +01:00
Richard W.M. Jones
f1d10672c7 df: Avoid a compilation error if libxml2 is not available.
virt_df-domains.o: In function `add_domain':
/home/feeliwood/Downloads/libguestfs-1.17.21/df/domains.c:274: undefined reference to `guestfs___for_each_disk'

guestfs___for_each_disk is only available when libvirt _and_ libxml2
dependencies are available at compile time.

Thanks Tho Huynh.
2012-03-25 19:35:43 +01:00
Hilko Bengen
b6e0552ee5 Do not run appliance-related checks if not building appliance 2012-01-23 09:08:33 +00:00
Hilko Bengen
7004fafc69 Replace setting of environment variables with usage of local run script
(Includes fix by RWMJ)
2012-01-23 09:08:21 +00:00
Richard W.M. Jones
08840bab44 Tempus fugit.
Update all copyright dates to 2012.
2012-01-18 22:05:02 +00:00
Richard W.M. Jones
dd0707be5f fish options parsing: Allow add_drives to be called multiple times.
Ensure that the drv structure is always zeroed on allocation.

Don't leak old drv->device when add_drives is called multiple times.
2012-01-18 16:28:10 +00:00
Richard W.M. Jones
cd077b8229 tests: Split images -> tests/data + tests/guests 2011-12-22 15:48:11 +00:00