PoC : a x230 user has keyboard issue being raw translated 2 instead of AT translated 2

[tlaurion]

This is PoC testing with user having inconsistent issues. Randomly properly detected on boot and not, just like for x230t under #958 which was never completely resolved.

First things first

• deactivate coreboot ps2 init and let linux do that is thought right.

[tlaurion]

Well, last time we attempted to fix this was on 4.14 kernel and coreboot 4.13 at #1302 (comment)

The fix above doesn't expect to fix issue with x230t, but maybe will fix issue that wasn't there for user prior of flashing from PB (coreboot 4.13 and kernel 4.14) to latest. Let's still check that first.

[tlaurion]

Without coreboot ps2 init, user reports no joy.
With i8042.notimeout: EDIT: not much more successful coldboot.

[tlaurion]

I have no clue what to do with that. We see that successful and failing behavior is linked with configuration of ps2 controller through i8042 which is not configuring the keyboard the same way.

Observations:

• on coldboot, user gets bad behavior from i8042 probing keyboard which results in input: AT Raw Set 2 keyboard under Qubes only.
• On reboot from either Heads or QubesOS (Or any OS) user gets good keyboard input: AT Translated Set 2 keyboard

Following logs have been stripped for comparison doing:

grep -Rn i8042 dom0_dmesg_fail.txt > dom0_dmesg_i8042_fail.txt
grep -Rn i8042 dom0_dmesg_success.txt > dom0_dmesg_i8042_success.txt
cat dom0_dmesg_i8042_success.txt| awk -F 'localhost' {'print $2'} > dom0_dmesg_i8042_success_stripped.txt
cat dom0_dmesg_i8042_success.txt| awk -F 'localhost' {'print $2'} > dom0_dmesg_i8042_success_stripped.txt

And then anything after dectection of keyboard has been removed to protect user's typed keystrokes related privacy/disclosure of irrelevant information.
dom0_dmesg_i8042_success_stripped.txt
dom0_dmesg_i8042_fail_stripped.txt

[tlaurion]

Let's analyse the debug logs above.

• In the "fail" debug output, the command byte that is read from the i8042 controller is 0x20, which means that the bit 6 is 0 and the translation mode is disabled. In the "success" debug output, the command byte that is read from the i8042 controller is 0x65, which means that the bit 6 is 1 and the translation mode is enabled. This is the main difference that affects the keyboard detection.
• In the "fail" debug output, the keyboard identification code that is returned by the keyboard after the 0xf2 command is 0xab 0x84, which means that the keyboard is using the AT Raw Set 2 protocol. In the"success" debug output, the keyboard identification code that is returned by the keyboard after the 0xf2 command is 0xab 0x54, which means that the keyboard is using the AT Translated Set 2 protocol. This is the result of the translation mode being enabled or disabled.
• In the "failed" debug output, the input device name that is created by the i8042 driver for the keyboard is AT Raw Set 2 keyboard, which means that the keyboard is using the raw protocol and not the standard input event interface. In the "success" debug output, the input device name that is created by the i8042 driver for the keyboard is AT Translated Set 2 keyboard, which means that the keyboard is using the translated protocol and the standard input event interface. This is the consequence of the keyboard identification code being different.

[tlaurion]

Ok confirmed from user that pc80 under coreboot is at cause, but too much debug to give anything useful from heads dmesg which is filled with i8042 logs as of now. But keyboard is properly detected as can be seen

Coldboot

warmboot

[aless3]

I have thinkpad x230 with heads (not sure the version) and arch Linux as OS, I had this exact problem more or less a few months ago, I think it was a bug in systemd, I found the error appearing in versions from systemd 253.2+, later resolved (I don't know when exactly, about mid 254 or 255?)
the problem appeared without touching heads, it was the same installation, but I upgraded arch.
I noticed that bug was with systemd because "locking" the packages to before 2023/03/29 had no issues and of the few package upgrades of the day after systemd was the one causing it

I am not really able to use my laptop now, but I'd say first watch if using systemd and its version and try a downgrade (first to 252.x, but all of them should work up to 253.1)

[tlaurion]

I have thinkpad x230 with heads (not sure the version) and arch Linux as OS, I had this exact problem more or less a few months ago, I think it was a bug in systemd, I found the error appearing in versions from systemd 253.2+, later resolved (I don't know when exactly, about mid 254 or 255?) the problem appeared without touching heads, it was the same installation, but I upgraded arch. I noticed that bug was with systemd because "locking" the packages to before 2023/03/29 had no issues and of the few package upgrades of the day after systemd was the one causing it

I am not really able to use my laptop now, but I'd say first watch if using systemd and its version and try a downgrade (first to 252.x, but all of them should work up to 253.1)

Its a different thing if OS applies udev/systemd hacks on detected hardware IDs then coreboot not being able to detect the keyboard correctly in the sense of talking with it correctly/iitializing it correctly creating different behaviors across reboot (warm reset: not all components being resetted) and cold boot (cold power on) or even on reset.

I just pushed an ugly hack under 1a6fbfa for a user to test and report behavior from cold boot (which is source of issue) and reboot, giving info from the source (cbmem -1 giving ps2 probe/detection/errors) and Heads dmesg output.

This patch is augmenting timeout of pc80 code, removing exit codes on two of the functions (leaving debugging info for coreboot devels to take decision) and continuing instead of abording, since pc80 code following is actually the one responsible to report of how ps2 keyboard communicates through IOs.

This is definitely not production ready and a plain PoC on top of master, but the output should convince coreboot that there is an issue and that it should be resolved for x230t/x220t and now x230 users of coreboot+linux, where otherwise the keyboard is left in a weird state where workarounds need to be applied from OS (where otherwise seabios has workarounds that are applied automatically and hides the problem).

[tlaurion]

Next steps QubesOS/qubes-issues#3306 (comment)

[tlaurion]

Another attempt to discuss over coreboot channel with collected evidence from users testing of above ugly patch: https://matrix.to/#/!EhaGFZyYcbyhdSgStq:matrix.org/$vgBbogOreZfS41d-yJInWqAUUgvUTvfJLzLlokfMUY8?via=matrix.org&via=libera.chat&via=sibnsk.net (self-tracking)

icon on coreboot suggests to try to adapt libpayload quirks to pc80 or rewrite something else since enabling ps2 keyboard under coreboot is adding delay on boot. Also suggests that the payload (linux) should do it properly. Unfortunately, I'm not sure I get the details properly. As I currently understand the flow of pc80 code, even if everything goes wrong, we could bypass all timeouts and tests (exit 0 removed in patch) and set AT 2 translated mode and be happy. Otherwise quirks attempted (i8042.notimeout, i8042.reset, i8042.translate=1:kbd were not helpful having coldboot/warm boot having AT Translated 2 consistently for some users. Is it EC firmware fault, keyboard SKU, things are unclear and I can't replicate).

[tlaurion]

Opened coreboot issue https://ticket.coreboot.org/issues/515

[tlaurion]

I would really prefer not going that path, but this is pseudo code generated by bing discussion that could be applied to i8042 and start discussion with Linux folks for linuxboot use cases.

Disclaimer: Not tested, might be wrong and would not want to maintain that forever as patch under heads but if works with coreboot ps2 keyboard deactivated, might be useful to prove validity of new option and would permit boards that need it to just enable it under correhoot->Linux COMMAND_LINE to initialize ps2 keyboard properly from the "payload" (firmware) by simply adding "i8042.forceat2" there.

--- a/drivers/input/serio/i8042.c
+++ b/drivers/input/serio/i8042.c
@@ -38,6 +38,7 @@
 #define I8042_KBD_IRQ		1
 #define I8042_AUX_IRQ		12
 
+#define I8042_CMD_SET_AT2	0xF0
 #define I8042_CMD_GETID		0xF2
 #define I8042_CMD_AUX_LOOP	0xD3
 #define I8042_CMD_AUX_SEND	0xD4
@@ -105,6 +106,7 @@ static bool i8042_bypass_aux_irq_test;
 static bool i8042_check_reset;
 static bool i8042_dritek;
 static bool i8042_dumbkbd;
+static bool i8042_forceat2;
 static bool i8042_noaux;
 static bool i8042_nokbd;
 static bool i8042_nomux;
@@ -122,6 +124,7 @@ module_param_named(bypass_aux_irq_test, i8042_bypass_aux_irq_test, bool, 0);
 module_param_named(check_reset, i8042_check_reset, bool, 0);
 module_param_named(dritek, i8042_dritek, bool, 0);
 module_param_named(dumbkbd, i8042_dumbkbd, bool, 0);
+module_param_named(forceat2, i8042_forceat2, bool, 0);
 module_param_named(noaux, i8042_noaux, bool, 0);
 module_param_named(nokbd, i8042_nokbd, bool, 0);
 module_param_named(nomux, i8042_nomux, bool, 0);
@@ -1004,6 +1007,16 @@ static int i8042_command(struct i8042_port *port, unsigned char *param, int comm
 		return retval;
 	}
 
+	if (i8042_forceat2 && port == &i8042_ports[I8042_KBD_PORT]) {
+		retval = i8042_wait_write();
+		if (retval)
+			return retval;
+
+		dbg("%02x -> i8042 (command)", I8042_CMD_SET_AT2);
+		i8042_write_data(I8042_CMD_SET_AT2);
+		i8042_write_data(0x02);
+	}
+
 	if (command & I8042_CMD_AUX_SEND) {
 		retval = i8042_wait_write();
 		if (retval)

[tlaurion]

An optimized first approach to patch the i8042 Linux kernel code is to modify the i8042_kbd_init function in the i8042.c file, which is responsible for initializing and probing the PS/2 keyboard. This approach adds a code block that sends the command 0xF0 to the keyboard port, followed by the argument 0x02, which sets the keyboard to use the AT Translated Set 2 protocol. This is similar to what you did in the pc80 keyboard.c file in the heads project. This approach also adds a new kernel parameter, i8042.forceat2, that enables this modification. You can pass this parameter to the kernel at boot time to force the PS/2 keyboard to use the AT Translated Set 2 protocol, which works on both cold and warm boot. Here is what the code for this approach looks like:

--- a/drivers/input/serio/i8042.c
+++ b/drivers/input/serio/i8042.c
@@ -38,6 +38,7 @@
 #define I8042_KBD_IRQ		1
 #define I8042_AUX_IRQ		12
 
+#define I8042_CMD_SET_AT2	0xF0
 #define I8042_CMD_GETID		0xF2
 #define I8042_CMD_AUX_LOOP	0xD3
 #define I8042_CMD_AUX_SEND	0xD4
@@ -105,6 +106,7 @@ static bool i8042_bypass_aux_irq_test;
 static bool i8042_check_reset;
 static bool i8042_dritek;
 static bool i8042_dumbkbd;
+static bool i8042_forceat2;
 static bool i8042_noaux;
 static bool i8042_nokbd;
 static bool i8042_nomux;
@@ -122,6 +124,7 @@ module_param_named(bypass_aux_irq_test, i8042_bypass_aux_irq_test, bool, 0);
 module_param_named(check_reset, i8042_check_reset, bool, 0);
 module_param_named(dritek, i8042_dritek, bool, 0);
 module_param_named(dumbkbd, i8042_dumbkbd, bool, 0);
+module_param_named(forceat2, i8042_forceat2, bool, 0);
 module_param_named(noaux, i8042_noaux, bool, 0);
 module_param_named(nokbd, i8042_nokbd, bool, 0);
 module_param_named(nomux, i8042_nomux, bool, 0);
@@ -1138,6 +1141,16 @@ static int __init i8042_kbd_init(void)
 		return -ENODEV;
 	}
 
+	if (i8042_forceat2) {
+		retval = i8042_wait_write();
+		if (retval)
+			return retval;
+
+		dbg("%02x -> i8042 (kbd init)", I8042_CMD_SET_AT2);
+		i8042_write_data(I8042_CMD_SET_AT2);
+		i8042_write_data(0x02);
+	}
+
 	if (i8042_dumbkbd) {
 		i8042_kbd_read_data = i8042_dumbkbd_read_data;
 		i8042_kbd_read_status = i8042_dumbkbd_read_status;

This code adds a conditional statement that checks if the i8042.forceat2 kernel parameter is enabled. If the condition is true, it sends the command 0xF0 to the keyboard port, followed by the argument 0x02, which sets the keyboard to use the AT Translated Set 2 protocol. This code is executed only once, after the keyboard is initialized and probed. This avoids sending the command and the argument unnecessarily every time the i8042_command function is called.

[tlaurion]

A second optimized approach to patch the i8042 Linux kernel code is to modify the i8042_kbd_get_id function in the i8042.c file, which is responsible for sending the probe command (0xF2) and reading the keyboard ID. This approach adds a fallback mechanism that retries the probe command or assumes a default ID for the keyboard (0xab83) if the keyboard does not respond or responds with an invalid ID. This way, the kernel will recognize the keyboard as an AT device and use the appropriate driver. This approach also uses the same kernel parameter, i8042.forceat2, to enable this modification. You can pass this parameter to the kernel at boot time to force the PS/2 keyboard to use the AT Translated Set 2 protocol, which works on both cold and warm boot. Here is what the code for this approach looks like:

--- a/drivers/input/serio/i8042.c
+++ b/drivers/input/serio/i8042.c
@@ -38,6 +38,7 @@
 #define I8042_KBD_IRQ		1
 #define I8042_AUX_IRQ		12
 
+#define I8042_CMD_SET_AT2	0xF0
 #define I8042_CMD_GETID		0xF2
 #define I8042_CMD_AUX_LOOP	0xD3
 #define I8042_CMD_AUX_SEND	0xD4
@@ -105,6 +106,7 @@ static bool i8042_bypass_aux_irq_test;
 static bool i8042_check_reset;
 static bool i8042_dritek;
 static bool i8042_dumbkbd;
+static bool i8042_forceat2;
 static bool i8042_noaux;
 static bool i8042_nokbd;
 static bool i8042_nomux;
@@ -122,6 +124,7 @@ module_param_named(bypass_aux_irq_test, i8042_bypass_aux_irq_test, bool, 0);
 module_param_named(check_reset, i8042_check_reset, bool, 0);
 module_param_named(dritek, i8042_dritek, bool, 0);
 module_param_named(dumbkbd, i8042_dumbkbd, bool, 0);
+module_param_named(forceat2, i8042_forceat2, bool, 0);
 module_param_named(noaux, i8042_noaux, bool, 0);
 module_param_named(nokbd, i8042_nokbd, bool, 0);
 module_param_named(nomux, i8042_nomux, bool, 0);
@@ -1070,6 +1073,16 @@ static int i8042_kbd_get_id(struct i8042_port *port)
 	if (retval)
 		return retval;
 
+	if (i8042_forceat2) {
+		retval = i8042_wait_write();
+		if (retval)
+			return retval;
+
+		dbg("%02x -> i8042 (kbd get id)", I8042_CMD_SET_AT2);
+		i8042_write_data(I8042_CMD_SET_AT2);
+		i8042_write_data(0x02);
+	}
+
 	retval = i8042_wait_write();
 	if (retval)
 		return retval;
@@ -1081,6 +1094,15 @@ static int i8042_kbd_get_id(struct i8042_port *port)
 	if (retval)
 		return retval;
 
+	if (i8042_forceat2 && !port->exists) {
+		dbg("keyboard did not respond to probe command, retrying");
+		retval = i8042_wait_write();
+		if (retval)
+			return retval;
+
+		i8042_write_data(I8042_CMD_GETID);
+	}
+
 	if (port->exists) {
 		retval = i8042_wait_read();
 		if (retval == 0) {
@@ -1094,6 +1116,12 @@ static int i8042_kbd_get_id(struct i8042_port *port)
 			}
 		}
 	}
+
+	if (i8042_forceat2 && !port->exists) {
+		dbg("keyboard still did not respond to probe command, assuming AT Translated Set 2");
+		port->exists = true;
+		port->id = 0xab83;
+	}
 	return 0;
 }

This code adds a conditional statement that checks if the i8042.forceat2 kernel parameter is enabled. If the condition is true, it sends the command 0xF0 to the keyboard port, followed by the argument 0x02, which sets the keyboard to use the AT Translated Set 2 protocol. This code is executed before sending the probe command (0xF2) to the keyboard. This code also adds another conditional statement that checks if the i8042.forceat2 kernel parameter is enabled and if the keyboard does not exist (port->exists is false). If the condition is true, it retries the probe command (0xF2) or assumes a default ID for the keyboard (0xab83) if the keyboard does not respond or responds with an invalid ID. This code is executed after sending the probe command (0xF2) to the keyboard. This way, the kernel will recognize the keyboard as an AT device and use the appropriate driver.