Refactor handling of embedded magic.mgc.

[teo-tsirpanis]

SC-25167
SC-47655
SC-47656
SC-47657
SC-47658

This PR overhauls the facilities to embed and load the magic.mgc file that is needed by libmagic:

• The most important change is the removal of magic_mgc_gzipped.bin.tar.bz2. This file contained a copy of magic.mgc that was compressed, converted to escaped C characters, packed and compressed again to take less space, and stored in source control, so that at build time to get unpacked and #included in mgc_dict.cc. Because this file was being prepared ahead of time by a manually invoked C++ program, this approach had the disadvantage that it tied the Core to a specific version of libmagic. This was made evident in Update libmagic to version 5.45 #4673, where just updating libmagic was not enough; we also had to update magic_mgc_gzipped.bin.tar.bz2.

  What we do now is rely on CMake to find magic.mgc and perform its entire preparation at build time. The C++ program was rewritten to be a CMake script, which makes it much simpler and enables it to run on cross-compilation scenarios. The script accepts the uncompressed magic.mgc file, compresses it and produces a header file of the following format:

  static const unsigned char magic_mgc_compressed_bytes[] = {
  0x28, 0xb5, 0x2f, 0xfd, …
  };
  constexpr size_t magic_mgc_compressed_size = sizeof(magic_mgc_compressed_bytes);
  // Editorial note: we used to prepend the decompressed size at the start of the
  // binary blob, but this was non-standard and could not be easily done by CMake.
  constexpr size_t magic_mgc_decompressed_size = 7041352;

• The algorithm to compress magic.mgc was changed from gzip to zstd, resulting in a 17.9% reduction of the compressed size (from 333067 το 273500 bytes).

• Tests for mgc_dict were also updated to use Catch2, and were wired to run along with the other standalone unit tests.

  • This necessitated to make an object library for mgc_dict, which was done as well.

Validated by successfully running unit_mgc_dict locally.

---

TYPE: BUILD
DESC: Improve embedding of magic.mgc and allow compiling with any libmagic version.

[teo-tsirpanis]

Suggested change

	DEPENDS "${libmagic_DICTIONARY}" "${PROJECT_SOURCE_DIR}/scripts/generate_embedded_data_header.cmake"
	DEPENDS "${libmagic_DICTIONARY}"

The Docker build apparently does not like the dependency to generate_embedded_data_header.cmake. I tried to reproduce it locally on WSL (with similar versions of CMake and make) but failed. I can just remove it, but magic_mgc.zst.h_ will not be regenerated in the (rare) case the script changes.

[teo-tsirpanis]

I managed to reproduce this locally on Docker, and spent lots of time trying to figure it out without success. I even docker cped the generated build tree and compared the makefiles to those on my plain WSL environment (it builds there for some reason), but could not find anything suspicious. 😕