Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve default WDLDrawRateTarget handling #2029

Merged
merged 7 commits into from
May 29, 2024

Conversation

Naphthalin
Copy link
Contributor

This fixes the following unintended behavior:

If we want to turn off the WDL sharpening when using WDLCalibrationElo by setting it to 0, the code falls back to directly using WDLDrawRateTarget. This however means that when using the correct WDLDrawRateReference in the config without also setting WDLDrawRateTarget to the same value, turning WDLCalibrationElo off doesn't actually retain the raw WDL values, since it internally works with the default 0.5 value.

By changing it to the default behavior "value 0 means no effect", it doesn't need to be set in the config anymore, and turning off WDLCalibrationElo also turns off any WDL sharpening.

…matically if not specified and WDLCalibrationElo: 0 is used
@borg323
Copy link
Member

borg323 commented May 19, 2024

Further testing shows that very small WDLDrawRateTarget values (e.g. 0.00000001) cause issues.

@Naphthalin
Copy link
Contributor Author

Further testing shows that very small WDLDrawRateTarget values (e.g. 0.00000001) cause issues.

I assume you're talking about numerical issues? Probably coming from 1 - eps being rounded to 1 numerically before taking the log.

Because from a chess point of view, already values like 0.02 (which were allowed in the old range) cause completely erratic evals, play etc. We could either sanitize for numerical issues only, or give a warning/exception/... when a too low value is used. One such value could be whatever the Elo model automatically assigns to a game between 0 (or 1000) rated players.

@borg323
Copy link
Member

borg323 commented May 20, 2024

After we decide on a safe minimum value, what is the preferred solution? Make smaller values zero or increase them to this safe minimum? What about WDLDrawRateReference, should this also have the same minimum? And if we decide on a minimum, what about the corresponding maximum?

@Naphthalin
Copy link
Contributor Author

After we decide on a safe minimum value, what is the preferred solution? Make smaller values zero or increase them to this safe minimum?

Since any "safe minimum" is arbitrary anyway, I would treat them as zero instead and thus turn off the WDL sharpening. I'd be happy with using the current minimum (0.001) as that value so we have consistent behavior for example, even if it allows one order of magnitude more than reasonable.

What about WDLDrawRateReference, should this also have the same minimum?

It has per design resp. definition, since the reference draw rate comes from the raw WDL of the net, and we don't have any nets with values outside the current value range, nor will we ever have unless on purpose.

And if we decide on a minimum, what about the corresponding maximum?

Mathematically, it's max = 1 - min, and the current max value of 0.999 already is 1 order of magnitude above what I personally consider to be reasonable.

@borg323 borg323 self-requested a review May 23, 2024 22:56
@borg323 borg323 merged commit f80dd7e into LeelaChessZero:master May 29, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants