test: Add checks for the middle page after VM creation

[yunmingyang]

Check status changes during the VM creation.

[martinpitt]

@yunmingyang FYI: Moving to draft while you are still working on this (half of the tests are red). Please move back to "ready for review" when you're done. Also, what is a "middle page"?

Thanks!

[yunmingyang]

@martinpitt Thanks. The "middle page" is that, before the VM creation is success and there is a VM detail page, the VM status is "Creating VM", the page after we clicking the VM name will not be directed the VM detail page, but a page like

I called it the "middle page" as it seems that no official name for it. Do we have a official name for this page?

[martinpitt]

Ah, perhaps "Creating VM state"? Note that this is going to be very flaky unless you have a way to control the installation -- i. e. make it take a nontrivial amount of time and then do something to the system to finish it.

[yunmingyang]

Ah I think the test will have enough time to check if using "Download an OS" with creating running VM. And for "Create and edit", I have not met the flaky locally, not sure whether the failed job is related with it or not. But actully, "create and edit" is a key point for this PR, as there will be enough time to test manually by using "Create and running", but it is too fast with "create and edit" so that tests could not be done manually in this situation.

[martinpitt]

There are still lots of failures here:

wait_js_cond(ph_in_text("#vm-subVmTestCreate31-system-state","Creating VM")): Uncaught (in promise) Error: actual text: Shut off

This smells like the race condition discussed above.

[yunmingyang]

@martinpitt sorry for the late response, I checked the original failed, according the log and screenshot, I think the reason why it failed is that the VM is still left there after deletion when using "create and run". Then, I tried to resolve this by adding some extra steps, but I failed. So I add a sleep for debugging, and the case passed the original failed location as the VM deletion will not leave a shutoff VM. For the new failed case, I think the reason is that there are two deletion operation in checkMiddlePageWhenCreating, the first passed with the debug line, the second didn't pass as there is no sleep before deletion just as the debug line, and it seems that this failure doesn't happen every time as some of checks pass. I think maybe this problem is related with libvirt, I am not sure. I think we could add some time wait before the deletion to resolve that. But I think add a sleep there is not a good way. Do you have some another ways for adding the time wait?

[martinpitt]

Right, static sleeps are never the right way, but rather waiting for something specific -- like, an UI change, or maybe even polling some virsh command. I don't know why the test is failing, but if it's failing to delete that VM then it could be useful to enable debug logging what's going on under the hood:

b.eval_js('window.debugging = "machines"')

That helps especially if you cannot reproduce the issue locally, so that you can see the libvirt/dbus messages in the test output. However, how hard did you try? Try to run the test several times, and also if you run against a persistent bots/vm-run VM, then do add --no-network (to reproduce what the real CI does with run-test).

[yunmingyang]

Hi, @martinpitt. Thanks for your information, it is very helpful. After some investigation, I am not whether it is correct, but I think maybe the difference between failure and success is there will be doUsagePolling, "UPDATE_VM" action and some other API invoking before the Destroy event. You could find it in https://cockpit-logs.us-east-1.linodeobjects.com/pull-1421-f22c88e1-20240423-115708-arch/log.html(subVmTestCreate31). So I add a refresh before removing the VM, and it seems that the problem disappears even though removing the sleep. But it smells like an issue about race condition. What do you think of it?