Improve Autopilot cursor movement

[shawnthwei]

The current Autopilot implementation uses very basic calculation for the cursor position, causing choppy cursor movements during gameplay. An example demonstration can be seen in #25354.

This change updates the calculation to base on the interpolation between Autoplay replay frames, resulting in the Autopilot cursor movement effectively matching Autoplay.

[bdach]

Why is interpolation correct? Doesn't interpolating mean that if you potentially click early on a hitobject whose hitwindow is active, you could possibly miss anyway because the cursor will stay at the interpolated position away from the object, rather than snap to it?

Unless I'm somehow misunderstanding what this is doing, this is worse than what we have, as it's fixing an aesthetic concern while causing a playability issue in exchange.

[shawnthwei]

Why is interpolation correct? Doesn't interpolating mean that if you potentially click early on a hitobject whose hitwindow is active, you could possibly miss anyway because the cursor will stay at the interpolated position away from the object, rather than snap to it?

Unless I'm somehow misunderstanding what this is doing, this is worse than what we have, as it's fixing an aesthetic concern while causing a playability issue in exchange.

Not really. I'm probably not doing a great job of explaining the change so I think you might be misunderstanding what it is.

The interpolation of the cursor position is not between two objects, but between two frames from the Autoplay replay frames. Imaging if the 0th Autoplay frame has: cursor position (0, 1) and time 0ms; and the 1st frame has cursor position (2,3) and time 2ms: If the current Autopilot frame has the time of 1ms, with the existing algorithm it tries to snap the cursor to the next frame, so the position would be set to (2, 3). However, with this change, the interpolation between (0, 1) and (2, 3) would be calculated based on the time, so the position would be set to (1, 2), which I think is the correct way to calculate it if we want the Autopilot cursor movement to match Autoplay.

#25354 shows what I mean about the existing algorithm. I've also manually tested my change locally by enabling Autopilot and Relax and verified the score is 100%, and also the movement looks more correct to me.

I guess there is a possible theoretical argument that if an object is only clickable in one single frame in Autoplay, by doing interpolation we can miss that exact frame. However I'd argue that the existing algorithm doesn't guarantee that either because it tries to snap to the next Autoplay frame, but the current Autopilot frame time wouldn't really exactly match the Autoplay frame time, so in the case it can still miss the object, hence I don't think that can be considered as a regression.

These are my thoughts and hopefully I've explained it better this time, but I obviously don't really know all the implementation details of the game, so if you spot something wrong please kindly point it out. Thanks!

[bdach]

I know what interpolation is and what it is performed on. Let me attempt to explain again using a visual reference.

Imagine we're between two objects as shown on the picture above. Replay frame 1 is centered on object 1, replay frame 2 - on object 2. The player wants to hit object 2 early, while it hasn't started yet. Using "closest replay frame" logic as previously, the cursor would snap to it and hit. Using interpolation naively, it would be where the blue dot is, and miss. Is that making more sense?

[shawnthwei]

Sure, I think the problem is when a frame the user sees has the time in between two replay frames, we want to determine where the cursor position should be. I can see that your example shows an edge case where snapping the cursor to a replay frame produces a desired result compared to doing interpolation. However, I think I can also provide some counter-examples of when doing interpolation is correct and snapping is not (the X's being the cursor position at a frame):

In the picture, if the player clicks slightly earlier at 1ms, with the existing algorithm the cursor would snap to the 1st replay frame, which is outside of the circle, while the interpolated cursor would be inside the circle. The same happens when the player clicks slightly later at 5ms.

I guess that is saying I can see there being edge cases with either implementation method. However, in a more normal gameplay situation, doing interpolation seems more correct to me, which is how I noticed and searched for the issue and why I proposed the change:

In a gameplay with Autopilot, if two frames happen at 2ms and 4ms, with interpolation the cursor would appear to be moving just like Autoplay. However, with snapping, in both frames the cursor would be snapped to the middle spot, causing the choppy movement.