cmake: preseed CMake cache on MSVC to speed up configuration

[madebr]

This preseeds the CMake cache for MSVC CMake projects:

On CI, CMake configuration time goes from:

• 3m19 -> 48s (x64)
• 2m58 -> 1m21s (x86)
• 3m28 -> 1m15s (clang-x64)
• 2m48 -> 28s (arm32)
• 2m58 -> 57s (arm64)

I disabled it for UWP (WINDOWS_STORE),

To disable this, configure with -DSDL_MSVC_PRESEED=OFF --fresh.
(--fresh performs a fresh configuration of the build tree, removing the CMakeCache.txt file)

Description

Existing Issue(s)

#9355

[madebr]

The cmake script was generated by this python script:

find-common-cmake-cache-variables.py

#!/usr/bin/env python

import argparse
import dataclasses
import os
from pathlib import Path


@dataclasses.dataclass
class CMakeVariable:
    name: str
    value: str
    type: str
    doc: str

    @property
    def bool_value(self) -> bool:
        assert self.value in ("", "1", "ON", "OFF"), f"{repr(self.value)} is not a bool"
        return self.value in ("1", "ON")

    def print_cmake_set_bool(self, indent: int, *, value:str=None):
        if value is None:
            value = self.value
        assert value in ("", "1", "ON", "OFF"), f"{repr(value)} is not a bool"
        value_str = f"\"{value}\""
        indent_str = " " * indent
        print(f"{indent_str}set({self.name:<48} {value_str:<5} CACHE {self.type:<8} \"{self.doc}\")")

    def __hash__(self):
        return hash((self.name, self.value))


def find_cmakecache_txt_files(folder: Path) -> list[Path]:
    result = []
    for root, _, files in os.walk(folder):
        if "CMakeCache.txt" in files:
            result.append(Path(root) / "CMakeCache.txt")
    return result


def read_cmakecache_txt(path: Path) -> dict[str, CMakeVariable]:
    result = {}
    doc = None
    with path.open() as f:
        for line in f.readlines():
            line = line.strip()
            if not line:
                continue
            if line.startswith("#"):
                continue
            if line.startswith("//"):
                doc = line[2:]
                continue
            key_type, value = line.split("=", 1)
            key, vtype = key_type.split(":", 1)
            assert key not in result, f"{key} must be unique"
            result[key] = CMakeVariable(name=key, value=value, type=vtype, doc=doc)
            doc = None
    return result


def filter_common_variables(vars1:  dict[str, CMakeVariable], vars2:  dict[str, CMakeVariable]) -> dict[str, CMakeVariable]:
    result = {}
    common_keys = set(vars1.keys()).intersection(set(vars2.keys()))
    for common_key in common_keys:
        if vars1[common_key].value == vars2[common_key].value:
            result[common_key] = vars1[common_key]
    return result


MSVC_TO_CL_VERSION = {
    2015: "19.0",
    2017: "19.1",
    2019: "19.2",
    2022: "19.3",
}

KNOWN_ARCHS = {
    "x86",
    "x64",
}


def main():
    parser = argparse.ArgumentParser(allow_abbrev=False)
    parser.add_argument("folder", type=Path)
    args = parser.parse_args()

    files = find_cmakecache_txt_files(args.folder)

    all_variables = {}
    constant_variables = {}
    version_variables = {version: None for version in MSVC_TO_CL_VERSION.keys()}
    arch_variables = {arch: None for arch in KNOWN_ARCHS}
    bool_variable_names = set()

    for file in files:
        rel = file.relative_to(args.folder).parent
        msvc_version = int(str(rel.parent).rsplit("-", 1)[1])
        assert msvc_version in MSVC_TO_CL_VERSION
        arch = str(rel.name)
        assert arch in KNOWN_ARCHS

        variables = read_cmakecache_txt(file)
        all_variables[str(rel)] = variables
        if not constant_variables:
            constant_variables = dict(variables)
        constant_variables = filter_common_variables(constant_variables, variables)

        if not version_variables[msvc_version]:
            version_variables[msvc_version] = dict(variables)
        version_variables[msvc_version] = filter_common_variables(version_variables[msvc_version], variables)

        if not arch_variables[arch]:
            arch_variables[arch] = dict(variables)
        arch_variables[arch] = filter_common_variables(arch_variables[arch], variables)

        for v in variables.values():
            if v.value in ("", "1", "0") and v.type in ("INTERNAL", "BOOL") and not "ADVANCED" in v.name and not "CMAKE" in v.name:
                bool_variable_names.add(v.name)

    def arch_dependent_filter(key: str) -> bool:
        return any(simd in key for simd in ("SSE", "AVX", "MMX", "ALTIVEC", "LSX", "LASX", "ARM"))

    def independent_filter(key: str) -> bool:
        return not arch_dependent_filter(key)  # and key.startswith("HAVE_") or key.startswith("LIBC_HAS") or "_IS_" in key or "_IN_" in key

    print("cmake_dependent_option(SDL_MSVC_PRESEED \"Preseed CMake cache for MSVC to speed up configuration\" ON \"MSVC;NOT WINDOWS_STORE\" OFF)")
    print("")
    print("if(SDL_MSVC_PRESEED)")

    remaining_variables = set(bool_variable_names)
    remaining_variables = set(filter(lambda v: not v.startswith("CHECK_CPU_ARCHITECTURE"), remaining_variables))

    for variable in sorted(list(remaining_variables)):
        if independent_filter(variable) and variable in constant_variables:
            constant_variables[variable].print_cmake_set_bool(indent=2)
            remaining_variables.remove(variable)

    bool_arch_variables = set(filter(arch_dependent_filter, remaining_variables))
    for arch in KNOWN_ARCHS:
        print()
        print(f"  if(CHECK_CPU_ARCHITECTURE_{arch.upper()})")
        for variable in sorted(list(arch_variables[arch])):
            if not arch_dependent_filter(variable) or variable not in remaining_variables:
                continue
            arch_variables[arch][variable].print_cmake_set_bool(indent=4)
        print(f"  endif()")
    remaining_variables.difference_update(bool_arch_variables)

    variables_min_msvc_version = dict()
    for var in remaining_variables:
        min_msvc_version = min(msvc_version for msvc_version, msvc_variables in version_variables.items() if msvc_variables[var].bool_value)
        assert all(msvc_variables[var].bool_value for msvc_version, msvc_variables in version_variables.items() if msvc_version >= min_msvc_version)
        variables_min_msvc_version.setdefault(min_msvc_version, []).append(var)

    for msvc_version, variables in variables_min_msvc_version.items():
        cl_version = MSVC_TO_CL_VERSION[msvc_version]
        print()
        print(f"  if(CMAKE_C_COMPILER_VERSION VERSION_GREATER_EQUAL \"{cl_version}\")")
        for variable in sorted(variables):
            version_variables[msvc_version][variable].print_cmake_set_bool(indent=4)
        print(f"  else()")
        for variable in sorted(variables):
            version_variables[msvc_version][variable].print_cmake_set_bool(indent=4, value="")
        print(f"  endif()")

    print("endif()")

    return 0


if __name__ == "__main__":
    raise SystemExit(main())

[icculus]

Oh man, can we do this for more platforms? There's really no reason we should be spending time asking macOS if it has a function called "malloc" in its C runtime, right?

[icculus]

(Maybe it's better said we should just assume platforms has a standard C runtime for non-controversial things, and the weird embedded platform can enable the symbols_to_check tests. This doesn't count things we know are different, like strlcpy not being standardized, etc)

[slouken]

Oh man, can we do this for more platforms? There's really no reason we should be spending time asking macOS if it has a function called "malloc" in its C runtime, right?

Yes please! :)

[slouken]

Anything holding this back? Did you want to implement it for more platforms?

[madebr]

No, because I don't know much about the apple toolchains, I'd prefer to not apply this to other platforms (yet).
Same with mingw/linux, the number of variants are too big to fix reliably.

I have a poc repo that can significantly speed things up, but is UB and does not work on clang.

[slouken]

Silly naming question... what's the difference between "preseed" and "seed" in this case? Is it fine to use the word "seed"?

[madebr]

Silly naming question... what's the difference between "preseed" and "seed" in this case? Is it fine to use the word "seed"?

I'm not sure. I only know about seeds in CS in the context of random generators or torrents.
Wikipedia: seed vs preseed

In our context, preseed looks like the correct term. wdyt?

[slouken]

In our context, preseed looks like the correct term. wdyt?

Sure, fair enough.

[slouken]

Is this off by default? I just did a fresh build under MSVC and it took a long time for the configure step.

[madebr]

It's enabled by default (only on SDL3).
CMake configuration when using the Visual Studio generators are always slow, and very slow when you do multiple tests.
On my Windows system, I always configure with the Ninja generator (you then need to configure and build in a vcvars environment).

When you don't see lots of tests for stdlib symbols, then the preseeding is active.

[slouken]

Ah, okay, it's slow, but not as slow as it was. :)

Thanks!