Guest configured with -usbdevice cannot be started in Ganeti 3

[pboguslawski]

Ganeti 3 from Debian 11...

# dpkg -s qemu-system-x86 | grep Version
Version: 1:5.2+dfsg-11+deb11u1

# dpkg -s ganeti | grep Version
Version: 3.0.1-2

...throws an error on guest start

# gnt-instance start vm.test
Waiting for job 1384 for vm.test ...
Job 1384 for vm.test has failed: Failure: command execution error:
Could not start instance 'vm.test': Hypervisor error: Failed to start instance vm.test: exited with exit code 1 (kvm: -usbdevice host:02e1:5612: '-usbdevice' is deprecated, please use '-device usb-...' instead
)

This guest was imported from backup created in Ganeti 2.15 in Debian 9. USB was connected to guest in Debian 9 using

gnt-instance modify -H usb_devices=host:02e1:5612 vm.test

and worked fine there. After importing such guest to Ganeti 3 in Debian 11 with gnt-backup import:

# gnt-instance info vm.test
[...]
  Hypervisor: kvm
  Hypervisor parameters:
 [...]
    usb_devices: host:02e1:5612
 [...]

Workaround that works in Ganeti 3 in Debian 11:

gnt-instance modify -H usb_devices='' vm.test
gnt-instance modify --hypervisor-parameters 'kvm_extra=-device usb-host\,vendorid=0x02e1\,productid=0x5612' vm.test

Seems that kvm in Debian 11 requires syntax described on

https://qemu-stsquad.readthedocs.io/en/doc-updates/system/usb.html#using-host-usb-devices-on-a-linux-host

and Ganeti should not use -usbdevice host:02e1:5612 with this kvm version.

[rbott]

Hi @pboguslawski, thanks for reporting this and all the details provided. I will try to look into this and come up with a fix 🙂

[rbott]

Looking a bit at the documentation, the -usbdevice parameter has been dropped with Qemu 2.12.0 (see changelog).

Since our absolute minimum supported QEMU version is 2.12 anyways (or maybe even newer, I am not 100% sure right now), there is no need for any compatibility code. Instead I would suggest to simply move to the "new" -usbdevice parameter entirely.

What do you think @saschalucas, @apoikos?

[saschalucas]

Looking a bit at the documentation, the -usbdevice parameter has been dropped with Qemu 2.12.0 (see changelog).

Thanks for pointing out. It seems not entirely dropped. Just for host, serial, disk and net. Maybe mouse, tablet, braille is still possible? Anyway ...

Since our absolute minimum supported QEMU version is 2.12 anyways (or maybe even newer, I am not 100% sure right now), there is no need for any compatibility code. Instead I would suggest to simply move to the "new" -usbdevice parameter entirely.

That makes sense. The fix for this issue should be backported to Ganeti-3.0 and there we decided to support qemu-2.11 from Ubuntu Bionic, which I guess is safe for a switch to -device usb-... (qemu-system-x86_64 -device help 2>&1 | grep usb seems to indicate this).

What I remember is that #1546 pumps qemu to >= 2.12, which is not in stable-3.0.

[ylopes]

Ganeti 3 from Debian 11...

# dpkg -s qemu-system-x86 | grep Version
Version: 1:5.2+dfsg-11+deb11u1

# dpkg -s ganeti | grep Version
Version: 3.0.1-2

...throws an error on guest start

# gnt-instance start vm.test
Waiting for job 1384 for vm.test ...
Job 1384 for vm.test has failed: Failure: command execution error:
Could not start instance 'vm.test': Hypervisor error: Failed to start instance vm.test: exited with exit code 1 (kvm: -usbdevice host:02e1:5612: '-usbdevice' is deprecated, please use '-device usb-...' instead
)

This guest was imported from backup created in Ganeti 2.15 in Debian 9. USB was connected to guest in Debian 9 using

gnt-instance modify -H usb_devices=host:02e1:5612 vm.test

and worked fine there. After importing such guest to Ganeti 3 in Debian 11 with gnt-backup import:

# gnt-instance info vm.test
[...]
  Hypervisor: kvm
  Hypervisor parameters:
 [...]
    usb_devices: host:02e1:5612
 [...]

Workaround that works in Ganeti 3 in Debian 11:

gnt-instance modify -H usb_devices='' vm.test
gnt-instance modify --hypervisor-parameters 'kvm_extra=-device usb-host\,vendorid=0x02e1\,productid=0x5612' vm.test

Seems that kvm in Debian 11 requires syntax described on

https://qemu-stsquad.readthedocs.io/en/doc-updates/system/usb.html#using-host-usb-devices-on-a-linux-host

and Ganeti should not use -usbdevice host:02e1:5612 with this kvm version.

Hi @pboguslawski , I have the same problem but

gnt-instance modify -H usb_devices='' myvm

does not work as it continues to add -usbdevice tablet to the command line...

Maybe can you help me in this ?

[pboguslawski]

Just checked: no -usbdevice tablet in guest params in our ganeti debian 11 environment so its not default setting problably. Maybe you've set in on cluster level - see gnt-cluster info or other param enables it.

[saschalucas]

-usbdevice tablet comes from KVM hypervisor parameter usb_mouse=tablet. However this alone still works on Debian Bullseye, maybe with a deprecation warning to stdout/err. Often usb_mouse=tablet is combined with VNC console (vnc_bind_address), which is broken in Bullseye and fixed with #1612. The error shown by the gnt-instance command regarding deprecation of -usbdevice is misleading there (see also https://groups.google.com/g/ganeti/c/8BouDdnUpd0/m/dLNznGjgBAAJ)

[rbott]

Hi @pboguslawski, just for the sake of completeness, could you check whats in the instance's log on the node where Ganeti tried to start it?
You should be able to find it under /var/log/ganeti/kvm/<instance>.log and it might have a longer/more useful error message.

[pboguslawski]

You should be able to find it under /var/log/ganeti/kvm/.log and it might have a longer/more useful error message.

root@myhost:/var/log/ganeti/kvm# cat vm.test.log 
kvm: -usbdevice host:02e1:5612: could not add USB device 'host:02e1:5612'

[mweinelt]

Fixed for me after upgrading to Ganeti 3.0.2.

[rbott]

Thanks for the feedback @mweinelt! :-)

@pboguslawski how is your status on this issue?

[pboguslawski]

how is your status on this issue?

We use Ganeti from stable Debian 11 and workaround as described above. Will check it in the future during upgrade and let you know if problem will still exists.

[rbott]

3.0.2 has been uploaded to Bullseye Backports, if you want to give it a try :-)

[saschalucas]

3.0.2 has been uploaded to Bullseye Backports, if you want to give it a try :-)

The original issue here is regarding usb_devices WRT host pass through. I think there is nothing fixed in 3.0.2 therefore.

Only #1612 fixes VNC, which is often combined with usb_mouse=tablet. The later omits a deprecation warning, which is why one might think this is an error, but it's not (as already mentioned here #1633 (comment))