Disk Hotplugging broken on Debian Bookworm (qemu 7.2)

[rbott]

Disk Hotplugging is broken on Debian Bookworm. This is the node daemon log excerpt:

2023-10-31 19:35:05,020: ganeti-noded pid=28480 INFO 192.168.1.5:38520 POST /jobqueue_update HTTP/1.1 200
2023-10-31 19:35:05,099: ganeti-noded pid=28481 INFO 192.168.1.5:38530 POST /jobqueue_update HTTP/1.1 200
2023-10-31 19:35:05,142: ganeti-noded pid=28482 INFO 192.168.1.5:38546 POST /hotplug_supported HTTP/1.1 200
2023-10-31 19:35:05,210: ganeti-noded pid=28483 INFO 192.168.1.5:38554 POST /hooks_runner HTTP/1.1 200
2023-10-31 19:35:05,249: ganeti-noded pid=28484 INFO RunCmd pvs --noheadings --nosuffix --units=m --unbuffered '--separator=|' -opv_name,vg_name,pv_free,pv_attr,pv_size,pv_name
2023-10-31 19:35:05,287: ganeti-noded pid=28484 INFO RunCmd lvcreate -Wn -L1024m -n670b4d23-e4cc-4784-bfdc-4d652e5af938.disk3 -i1 gnt /dev/vdb
2023-10-31 19:35:05,359: ganeti-noded pid=28484 INFO RunCmd lvs --noheadings '--separator=|' --units=k --nosuffix -ovg_name,lv_name,lv_attr,lv_kernel_major,lv_kernel_minor,vg_extent_size,stripes,devices
2023-10-31 19:35:05,415: ganeti-noded pid=28484 INFO RunCmd lvchange -ay /dev/gnt/670b4d23-e4cc-4784-bfdc-4d652e5af938.disk3
2023-10-31 19:35:05,475: ganeti-noded pid=28484 INFO RunCmd lvs -o tags --noheadings --nosuffix /dev/gnt/670b4d23-e4cc-4784-bfdc-4d652e5af938.disk3
2023-10-31 19:35:05,527: ganeti-noded pid=28484 INFO RunCmd lvchange --addtag originstname+kvm-test-instance02.staging.ganeti.org /dev/gnt/670b4d23-e4cc-4784-bfdc-4d652e5af938.disk3
2023-10-31 19:35:05,583: ganeti-noded pid=28484 INFO 192.168.1.5:38566 POST /blockdev_create HTTP/1.1 200
2023-10-31 19:35:05,634: ganeti-noded pid=28499 INFO RunCmd lvs --noheadings '--separator=|' --units=k --nosuffix -ovg_name,lv_name,lv_attr,lv_kernel_major,lv_kernel_minor,vg_extent_size,stripes,devices
2023-10-31 19:35:05,673: ganeti-noded pid=28498 INFO 192.168.1.5:38576 POST /upload_file_single HTTP/1.1 200
2023-10-31 19:35:05,687: ganeti-noded pid=28499 INFO RunCmd lvchange -ay /dev/gnt/670b4d23-e4cc-4784-bfdc-4d652e5af938.disk3
2023-10-31 19:35:05,751: ganeti-noded pid=28499 INFO 192.168.1.5:38592 POST /blockdev_assemble HTTP/1.1 200
2023-10-31 19:35:05,829: ganeti-noded pid=28502 INFO RunCmd echo 'drive_add dummy file=/dev/fdset/0,if=none,id=disk-aad2ee95-6aa5-499f,format=raw,auto-read-only=off,aio=threads,cache=none' |  /usr/bin/socat STDIO 
UNIX-CONNECT:/var/run/ganeti/kvm-hypervisor/ctrl/kvm-test-instance02.staging.ganeti.org.monitor
2023-10-31 19:35:05,886: ganeti-noded pid=28502 ERROR Error in RPC call
Traceback (most recent call last):
  File "/usr/share/ganeti/3.1/ganeti/server/noded.py", line 189, in HandleRequest
    result = (True, method(serializer.LoadJson(req.request_body)))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ganeti/3.1/ganeti/server/noded.py", line 660, in perspective_hotplug_device
    return backend.HotplugDevice(instance, action, dev_type, device, extra, seq)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ganeti/3.1/ganeti/backend.py", line 3217, in HotplugDevice
    return fn(instance, dev_type, device, extra, seq)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ganeti/3.1/ganeti/hypervisor/hv_kvm/__init__.py", line 182, in wrapper
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ganeti/3.1/ganeti/hypervisor/hv_kvm/__init__.py", line 2312, in HotAddDevice
    self.qmp.HotAddDisk(device, kvm_devid, uri, drive_add_fn)
  File "/usr/share/ganeti/3.1/ganeti/hypervisor/hv_kvm/monitor.py", line 224, in wrapper
    ret = fn(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ganeti/3.1/ganeti/hypervisor/hv_kvm/monitor.py", line 603, in HotAddDisk
    self.Execute("device_add", arguments)
  File "/usr/share/ganeti/3.1/ganeti/hypervisor/hv_kvm/monitor.py", line 433, in Execute
    ret = self._GetResponse(command)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ganeti/3.1/ganeti/hypervisor/hv_kvm/monitor.py", line 456, in _GetResponse
    raise errors.HypervisorError("kvm: error executing the %s"
ganeti.errors.HypervisorError: kvm: error executing the device_add command: Property 'virtio-blk-device.drive' can't find value 'disk-aad2ee95-6aa5-499f' (GenericError):
2023-10-31 19:35:05,888: ganeti-noded pid=28502 INFO 192.168.1.5:38594 POST /hotplug_device HTTP/1.1 200
2023-10-31 19:35:05,929: ganeti-noded pid=28507 INFO 192.168.1.5:38598 POST /blockdev_getmirrorstatus HTTP/1.1 200
2023-10-31 19:35:06,010: ganeti-noded pid=28509 INFO 192.168.1.5:38608 POST /hooks_runner HTTP/1.1 200
2023-10-31 19:35:06,025: ganeti-noded pid=28508 INFO 192.168.1.5:38600 POST /upload_file_single HTTP/1.1 200
2023-10-31 19:35:06,083: ganeti-noded pid=28510 INFO 192.168.1.5:38612 POST /jobqueue_update HTTP/1.1 200

The problem does not appear with #1667, but that would probably not be a viable solution for the stable-3.0 branch which currently is the base for the Debian package in Bookworm.

//Edit:

I added some debug logging. It looks like the HMP command drive_add does not succeed. socat does not return an error (hence it is undetected), but the output of the command shows:

'file' driver requires '/dev/fdset/0' to be a regular file

According to the qemu 6.0 changelog:

The file block driver no longer permits use with block devices

The removed features page adds some more details here:

The ‘file’ driver for drives is no longer appropriate for character or host devices and will only accept regular files (S_IFREG). The correct driver for these file types is ‘host_cdrom’ or ‘host_device’ as appropriate.

That would mean we need to figure out if the disk is actually a block device and switch to host_device in that case (while using file for all other types). Any other ideas here?

[rbott]

Well actually drive_add seems to support only file and nothing else. I am a little confused here. We might have to accept the fact that #1667 will be the only way forward here.