While debugging some udev issues, I found use of 'udevadm monitor' to
be fairly useful. Add a suggested debugging command that might be
used, but comment it out.
Thanks: Alasdair Kergon
dhcpcd fails to update /etc/resolv.conf if is a dangling symlink, so remove it
if it is.
This happens on Arch Linux when systemd-resolved is enabled by symlinking
/etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf. The symlink is copied
into the supermin appliance, but the target file is not and this causes dhcpcd to
fail when it tries to update /etc/resolv.conf.
The handling of /etc/resolv.conf could be improved in dhcpcd but it's not their
job to decide when to remove the symlink. We do have the authority to decide
whether or not to use systemd-resolved or resolvconf in the appliance, so we can
remove it.
Since RHEL 7.4, the noop scheduler is no longer a thing. Trying to
set it results in the error:
+ echo noop
/init: line 108: echo: write error: Invalid argument
The current recommendation (https://access.redhat.com/solutions/5427)
is to use mq-deadline, but that's also the default so we don't have to
do anything.
A bigger reason to remove these lines is that kernel 6.11.0 has
introduced a hang where -- rarely -- the ext4 filesystem hangs if you
try to change the scheduler while handing a page fault, even if you're
setting a scheduler that doesn't exist. I couldn't get much detail
except for a couple of stack traces from different VMs:
crash> set 234
PID: 234
COMMAND: "modprobe"
TASK: ffff9f5ec3a22f40 [THREAD_INFO: ffff9f5ec3a22f40]
CPU: 0
STATE: TASK_UNINTERRUPTIBLE
crash> bt
PID: 234 TASK: ffff9f5ec3a22f40 CPU: 0 COMMAND: "modprobe"
#0 [ffffb21e002e7840] __schedule at ffffffffa718f6d0
#1 [ffffb21e002e78f8] schedule at ffffffffa7190a27
#2 [ffffb21e002e7908] __bio_queue_enter at ffffffffa67e121c
#3 [ffffb21e002e7968] blk_mq_submit_bio at ffffffffa67f358c
#4 [ffffb21e002e79f0] __submit_bio at ffffffffa67e1e3c
#5 [ffffb21e002e7a58] submit_bio_noacct_nocheck at ffffffffa67e2326
#6 [ffffb21e002e7ac0] ext4_mpage_readpages at ffffffffa65ceafc
#7 [ffffb21e002e7be0] read_pages at ffffffffa6381d17
#8 [ffffb21e002e7c40] page_cache_ra_unbounded at ffffffffa6381ff5
#9 [ffffb21e002e7ca8] filemap_fault at ffffffffa63761b5
#10 [ffffb21e002e7d48] __do_fault at ffffffffa63d1892
#11 [ffffb21e002e7d70] do_fault at ffffffffa63d2425
#12 [ffffb21e002e7da0] __handle_mm_fault at ffffffffa63d8c6b
#13 [ffffb21e002e7e88] handle_mm_fault at ffffffffa63d95c2
#14 [ffffb21e002e7ec8] do_user_addr_fault at ffffffffa60b34ea
#15 [ffffb21e002e7f28] exc_page_fault at ffffffffa7186e4e
#16 [ffffb21e002e7f50] asm_exc_page_fault at ffffffffa72012a6
RIP: 000055d16159f8d8 RSP: 00007ffdd4c1f340 RFLAGS: 00010206
RAX: 00000000000bec82 RBX: 00007ff2fd00dc82 RCX: 000055d1615b492a
RDX: 00007ffdd4c216b0 RSI: 00000000200bec82 RDI: 000055d185725960
RBP: 00007ffdd4c1f5a0 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000202 R12: 00000000200bec82
R13: 000055d185725960 R14: 00007ffdd4c216b0 R15: 000055d1615b9708
ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b
crash> set 230
PID: 230
COMMAND: "modprobe"
TASK: ffff98ce03ca3040 [THREAD_INFO: ffff98ce03ca3040]
CPU: 0
STATE: TASK_UNINTERRUPTIBLE
crash> bt
PID: 230 TASK: ffff98ce03ca3040 CPU: 0 COMMAND: "modprobe"
#0 [ffffaf9940307840] __schedule at ffffffff9618f6d0
#1 [ffffaf99403078f8] schedule at ffffffff96190a27
#2 [ffffaf9940307908] __bio_queue_enter at ffffffff957e121c
#3 [ffffaf9940307968] blk_mq_submit_bio at ffffffff957f358c
#4 [ffffaf99403079f0] __submit_bio at ffffffff957e1e3c
#5 [ffffaf9940307a58] submit_bio_noacct_nocheck at ffffffff957e2326
#6 [ffffaf9940307ac0] ext4_mpage_readpages at ffffffff955ceafc
#7 [ffffaf9940307be0] read_pages at ffffffff95381d1a
#8 [ffffaf9940307c40] page_cache_ra_unbounded at ffffffff95381ff5
#9 [ffffaf9940307ca8] filemap_fault at ffffffff953761b5
#10 [ffffaf9940307d48] __do_fault at ffffffff953d1895
#11 [ffffaf9940307d70] do_fault at ffffffff953d2425
#12 [ffffaf9940307da0] __handle_mm_fault at ffffffff953d8c6b
#13 [ffffaf9940307e88] handle_mm_fault at ffffffff953d95c2
#14 [ffffaf9940307ec8] do_user_addr_fault at ffffffff950b34ea
#15 [ffffaf9940307f28] exc_page_fault at ffffffff96186e4e
#16 [ffffaf9940307f50] asm_exc_page_fault at ffffffff962012a6
RIP: 0000556b7a7468d8 RSP: 00007ffde2ffb560 RFLAGS: 00000206
RAX: 00000000000bec82 RBX: 00007f5331a0dc82 RCX: 0000556b7a75b92a
RDX: 00007ffde2ffd8d0 RSI: 00000000200bec82 RDI: 0000556ba8edf960
RBP: 00007ffde2ffb7c0 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000202 R12: 00000000200bec82
R13: 0000556ba8edf960 R14: 00007ffde2ffd8d0 R15: 0000556b7a760708
ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2303267
In case network was requested, but the host lacks both dhclient and
dhcpcd, skip the loop which waits for a resolv.conf update.
This reduces boot time by 10 seconds.
Signed-off-by: Olaf Hering <olaf@aepfle.de>
Downstream patched openssl in Fedora 37+ broke unless
/etc/crypto-policies/back-ends/opensslcnf.conf is present. Files in
this directory are generated by %post rules that use scripting
languages so cannot easily be created by supermin.
Add a symlink to the DEFAULT policy file if the configuration file
doesn't exist.
A symptom of this problem is the error:
Requested hash sha256 is not supported.
Failed to set pbkdf parameters.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2133884
Updates: commit d6ba398825
In guestfs-tools commit 4fe8a03cd2d3 ('sysprep: remove lvm2's default
"system.devices" file', 2022-04-11), we disabled the use of LVM2's new
"devicesfile" feature, which could interfere with the cloning of virtual
machines.
We suspected in
https://bugzilla.redhat.com/show_bug.cgi?id=2072493#c6
that the same lvm2 feature could affect the libguestfs appliance itself,
but decided in
https://bugzilla.redhat.com/show_bug.cgi?id=2072493#c8https://bugzilla.redhat.com/show_bug.cgi?id=2072493#c10
that this would not be the case, because "appliance/init" already
constructed a pristine LVM_SYSTEM_DIR.
Unfortunately, that's not enough: due to the "use_devicesfile=1" default
(on RHEL9 anyway), some "lvm" invocation, possibly inside the
lvm-set-filter API, *creates* "$LVM_SYSTEM_DIR/devices/system.devices".
And then we get (minimally) warnings such as
> Please remove the lvm.conf global_filter, it is ignored with the devices
> file.
> Please remove the lvm.conf filter, it is ignored with the devices file.
when using the lvm-set-filter API.
Explicitly disable the "devices file" in "appliance/init", and also
whenever we rewrite "lvm.conf" -- that is, in set_filter()
[daemon/lvm-filter.c]. In the former, check for the feature by locating
the devicesfile-related utilities "lvmdevices" and "vgimportdevices". In
the C code, invoke the utilities with the "--help" option instead. (In
"appliance/init", I thought it was best not to call any lvm2 utilities
even with "--help", with our lvm2.conf still under construction there.) If
either utility is available, set "use_devicesfile = 0".
Cc: David Teigland <teigland@redhat.com>
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1965941
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20220530141027.16167-1-lersek@redhat.com>
Acked-by: Richard W.M. Jones <rjones@redhat.com>
[lersek@redhat.com: style fix: break "devicesfile_feature" in the function
definition to a new line]
bash can read input from a spawned process, and even provide input to
such process. This feature relies on /dev/fd/ being present. In the
past udev silently created this symlink, so this bash feature worked
more or less by accident. With recent systemd versions, such as 246
which is included in Leap 15.3, the symlink is not created anymore. As
a result scripts, such as /sbin/dhclient-script, fail to work
properly.
This symlink should have been created in version 1 of this variant of /init.
https://bugzilla.opensuse.org/show_bug.cgi?id=1190501
Signed-off-by: Olaf Hering <olaf@aepfle.de>
When the daemon starts up it creates a fresh (empty) LVM configuration
and starts up lvmetad (which depends on the LVM configuration).
However this appears to cause problems: Some types of PV seem to
require lvmetad and don't work without it
(https://bugzilla.redhat.com/show_bug.cgi?id=1581810). If we don't
start lvmetad earlier, the device nodes are not created.
Therefore move the whole initialization step into appliance/init.
Two further changes had to be made:
Now we are using lvmetad all the time, using vgchange is incorrect.
With lvmetad activated early we must use ‘pvscan --cache --activate ay’
to scan all disks for PVs and activate any VGs on them (although the
documentation is complex, confusing and contradictory so I'm not
completely sure about this).
The ‘lvm_system_dir’ local variable in ‘daemon/lvm-filter.c’
previously contained the path of the directory above $LVM_SYSTEM_DIR
(eg. $LVM_SYSTEM_DIR = "/etc/lvm", lvm_system_dir = "/etc"). As this
was highly confusing, I have changed it so the local variable and the
environment variable have identical contents. This involved removing
the ‘lvm/’ component from a couple of paths since it is now included
in the local variable.
The issue:
- raid1 will be in degraded state if one of its components is logical volume (LV)
- raid0 will be inoperable at all (inacessible from within appliance) if one of its component is LV
- raidN: you can expect the same issue for any raid level depends on how many components are inaccessible at the time mdadm is running and raid redundency.
It happens because mdadm is launched prior to lvm AND it is instructed to run found arrays immediately (--run flag) regardless of completeness of their components.
Later (when lvm activates found LVs) md signature on LV might be recognized BUT newly found raid components could't be inserted into already running (in degraded state)
or marked as inoperable raid arrays.
The patch fixes the issue in the following way:
1. Found arrays won't be run immediately unless ALL expected drives (components) are present. Here '--no-degraded' flag comes into a play. See mdadm(8).
2. Second mdadm call (after LVM is scanned) will scan UNUSED yet devices and make an attempt to run all found arrays (even they will be in degraded state).
There is no performance penalty because second pass scans UNUSED yet devices. Here is 'boot-benchmark' before and after patch:
: libvirt backend : direct backend
------------------------------------------------
master : 835.2ms ±1.1ms : 670.4ms ±0.3ms
master+patch : 837.7ms ±2.4ms : 671.8ms ±0.2ms
Commit a6330e9d3a enabled /run for
systemd-tmpfiles: while this works fine in most of the cases, there are
few tmpfiles configurations that still references /var/run instead of
/run. As result, include also /var/run in the systemd-tmpfiles
execution.
`virt-rescue -a disk -i' does the right thing.
`-m' was previously an alternate form of `--memsize'. By sniffing the
parameter we can make `-m MB' continue to work, while also allowing
`-m' to be used as a short form for the `--mount' option.
This also removes most of the description of `--suggest' from the man
page, since it is no longer needed.
Instead of using "direct mode" (which was basically a quick hack),
virt-rescue now launches the appliance with a running daemon.
The daemon doesn't do much -- there is still a bash shell which the
user interacts with. The daemon is there simply to provide the
initial GUESTFS_LAUNCH_FLAG message and to handle shutdown a bit more
gracefully.
To interact with the shell, and replacing direct mode, virt-rescue now
prints out log messages (the output of the shell), and sends input
typed by the user directly to the console socket. This uses the
guestfs_internal_get_console_socket API added previously. Most of the
complexity behind this is hidden in virt-rescue.
This fully fixes the handling of ^C (RHBZ#1152819). Also there were
earlier reports that full screen commands like 'vim' didn't work well,
(RHBZ#1171654), but in this version vim appears to work fine, albeit
only using 80x24 of the screen because of the serial console.
Tools could require the use of pseudo-terminals, so make sure we have
/dev/pts available in the appliance. The "command" API already
bind-mounts it when running commands, so this is the only bit needed.
Setup the volatile /run in the appliance also with the tmpfiles
configurations available. In particular, setting up correctly the lvm
bits allow lvmetad to run.
Currently lvmetad is started in init, and thus using the system
(= appliance) configuration of lvm. Later on, in the daemon, a local
copy of the lvm configuration is setup, and set it for use using the
LVM_SYSTEM_DIR environment variable: this means only the programmes
executed by the daemon will use the local lvm configuration, and not
lvmetad.
Thus manually start lvmetad from the daemon, right after having setup
the local lvm configuration, and still without failing if it cannot be
executed.
Additionally, since lvmetad now respects the right configuration, make
sure to update its cache when rescanning the VGs by passing --cache to
vgscan.
Sadly, the dhclient-script shipped as part of isc-dhcp-client in Ubuntu
unconditionally reads from /etc/fstab without checking for its
existence. Since no package holds /etc/fstab, this file will not exist
in the appliance, cause dhclient to fail (actually keep looping calling
the failing dhclient-script) when the network is requested.
As a workaround, touch /etc/fstab just before enabling the network: if
that file exists nothing changes, while an empty file will be available
in the other case, making at least dhclient-script in Ubuntu working.
At least on openSUSE and SLES, the /etc/mtab file is already existing.
Skipping the symlink creation in init removes one error message during
the appliance boot.
Running ldconfig adds about 100ms to the boot time. I would prefer
that we understood which libraries need ldconfig to be run, and fix
that. We could also consider running ldconfig in parallel, but since
it might be required by just about any binary that the init script
runs it's not clear what benefit that gives.
This reverts commit 66aa98265d.
Some of the systemd-tmpfiles snippets need the machine ID of the running
system; the current lack of this file produces warning messages during
the appliance boot like:
[/usr/lib/tmpfiles.d/systemd.conf:26] Failed to replace specifiers: /run/log/journal/%m
[/usr/lib/tmpfiles.d/systemd.conf:28] Failed to replace specifiers: /run/log/journal/%m
[/usr/lib/tmpfiles.d/systemd.conf:29] Failed to replace specifiers: /run/log/journal/%m
Thus create a new randomly-generated /etc/machine-id on boot.
Since commit bb5d30ab2a, we don't
require any external programs like grep to parse the command line. We
only use bash intrinsics.
Therefore we can do it early (but after /proc is mounted).
This allows verbose mode to enable set -x early on, so we can trace
most things that the init script does.
Read the content of /proc/cmdline using bash features, and use its
[[ ... ]] expression to find texts in a variable.
This shaves off 5 grep invocations.
If you've ever tried to use this option, you'll know that it didn't
work well. It broke random things (probably RHBZ#1020216, definitely
RHBZ#1023630), and caused random failures generally, while often not
actually failing when valgrind itself found problems.
dhcpcd requires an interface name as parameter to work. We are now
getting it from /proc/sys/net/ipv4/conf/ folder children. dhclient
on Debian also has the problem, thus use the guessed interface name
for it too.
Instead of parsing /proc/cmdline from the daemon, move all of that
parsing into the init script, and pass the argument via the daemon
command line.
For example, previously the daemon and init script both looked for
guestfs_network=1 in /proc/cmdline. Now the init script still looks
for it, and if found it runs `guestfsd --network'.
Currently if /tmp (on the host) is a symlink, then the symlink is
copied into the appliance, probably pointing to a non-existent
directory, and everything goes downhill from there.
Avoid this by making sure that /tmp and /var/tmp are real directories.
qemu in SLIRP mode offers DHCP services to the appliance. We don't
use them, but use a fixed IP address intead. This changes the
appliance to get its IP address using DHCP.
Note: This is only used when the network is enabled. dhclient is
somewhat slower, but the penalty (a few seconds) is only paid for
network users.
On SuSE or other distros, dhcpcd could be used if available.
