CMake and Zlib #831
Comments
|
Other people linked to CMake and Zlib. @malytomas, @PiotrSikora, @avoidik, @jeking3, @JonasVautherin, @sav-ix, @01e9, @henryborchers, @zdenop, @rgpublic, @Slackadays, @bear24rw, @kgerheiser, @paulsapps, @danielalves, @rocammo, @bmarques1995, @rafael-santiago, @Andreas-Schniertshauer, @Togtja, @joeyparrish, @robiwano, @jeking3, @henryborchers, @aneeshpb-amazon, @kavionic, @cloudwindy |
|
There are too many cmake issues and pull requests for me to go through and disposition them, and I wouldn't know what's good or bad anyway. I very much like @Mizux 's suggestion to have a group render on, coordinate, and most importantly verify proposed cmake changes. My main concern are proposed changes that work in one case, but messes up a bunch of others. Ideally this group would be able to cover a good set of platforms. |
|
You might want to give interested parties Triage access to the repository to help manage all the GitHub issues. You could decide to merge in PRs based on the number of GitHub code review approvals. I’d say stick to only merging changes that are absolutely necessary or ones that fix problems that lead to many duplicate issues. |
|
It would help a lot if someone with the necessary rights could approve the CI for PRs. Mine has been pending for 1+ year with no test run. |
|
My |
|
@ClausKlein Neat workaround for the blocked CI. |
|
@ClausKlein: Nice :) I hope it will be a good beginning! @madler, can you look? |
|
@vszakats: Can you look improvements from @ClausKlein to adapt your PR? |
|
@Neustradamus Both PRs touch the buggy Can't see anything else I could adapt from the other PR, they are targeting different issues. UPDATE: to my RC comment above: missed a |
|
I see this diff on my branch to #667, should I apply it? diff --git a/CMakeLists.txt b/CMakeLists.txt
index d4eb7cb..19ac18b 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -121,7 +121,7 @@ set(ZLIB_SRCS
zutil.c
)
-if(NOT MINGW AND NOT MSYS)
+if(WIN32)
set(ZLIB_DLL_SRCS
win32/zlib1.rc # If present will override custom build rule below.
)
diff --git a/win32/Makefile.gcc b/win32/Makefile.gcc
index 081e391..35172fb 100644
--- a/win32/Makefile.gcc
+++ b/win32/Makefile.gcc
@@ -50,7 +50,7 @@ AR = $(PREFIX)ar
ARFLAGS = rcs
RC = $(PREFIX)windres
-RCFLAGS = --define GCC_WINDRES
+RCFLAGS =
STRIP = $(PREFIX)strip
diff --git a/win32/zlib1.rc b/win32/zlib1.rc
index ceb4ee5..5253b6a 100644
--- a/win32/zlib1.rc
+++ b/win32/zlib1.rc
@@ -1,11 +1,7 @@
#include <winver.h>
#include "../zlib.h"
-#ifdef GCC_WINDRES
VS_VERSION_INFO VERSIONINFO
-#else
-VS_VERSION_INFO VERSIONINFO MOVEABLE IMPURE LOADONCALL DISCARDABLE
-#endif
FILEVERSION ZLIB_VER_MAJOR,ZLIB_VER_MINOR,ZLIB_VER_REVISION,0
PRODUCTVERSION ZLIB_VER_MAJOR,ZLIB_VER_MINOR,ZLIB_VER_REVISION,0
FILEFLAGSMASK VS_FFI_FILEFLAGSMASK |
|
I think we are better keeping PRs granular and each fixing distinct problems, so I'd keep them separate. I'm also fine with rebasing if necessary. This makes it easier to later bisect where a problem was introduced or track down what changed and why. Our issue IMO is that none of the CMake patches are actually merged. (Unless we want to somehow offer one "megapatch" solving all issues at once, which is possible but needs a great amount of work and testing.) |
As far as I know, |
|
My patch (#677) fixes that by deleting all |
|
|
CMake documented it in 3.2 (2015-03-04) indeed, but zlib's use predates that: dc5a43e (2011-09-09) by almost 4 years. Even more interestingly, the first introduction of
|
MSYS2 still seems to define I agree that there is bugs in some versions of cmake when it comes to relying to correct variables getting defined. Some projects manually define these and not rely on cmake defining them automatically. |
|
Current zlib solution for RC compilation is a toolchain-specific hack, and broken. Question: Was it purposefully done that way to support ancient CMake versions? Was it an oversight? CMake does support Also, are the resource optimization flags targeting Windows 16-bit (3.11 from 1992-04-06) intentionally kept there or is it okay to delete them yet? It'd be useful to get answers or any kind of feedback instead of guessing and pushing PRs that are perhaps wrong or not desired by zlib. This would help getting out of the CMake-stall we experience. CMake being the only standard tool to build zlib with, this also seems critical. |
|
@mtl1979: mingw-64 (and old mingw32) (= IMO we should deal with those if an actual fallout is reported and not optimize for niche tool bugs upfront. I've personally found these macros working as expected. |
cmake uses preprocessor to detect the operating system as seen on the patch posted above. |
It seems correct to me. This is also the initial commit for this feature, so it doesn't tell the full story. Anyhow I don't to seem get where this discussion is heading, so closing up my PR and will continue to patch the zlib build I maintain. |
|
pip doesn't replace anything installed in I also don't understand what |
zlib is supposed to support 16-bit and 32-bit operating systems... In recent days the support for 64-bit operating systems (and larger compressed files) has gotten better and more complete. |
|
@mtl1979: Re-read what I wrote, or look at the patch. This concerned I'm back to the less obtrusive workaround for the bug I'm hitting. #347 would also help because |
|
Maybe some people can look for this ticket: |
|
I wonder when people will start drawing the obvious conclusions about CMake. Of course there are many justifications to its use like being locked in or having no power or say, but it is clearly requiring enormous amounts of time and energy investments to just use it properly (in all its edge cases). At some point the madness has to stop. |
CMake does have a lot of limitations but it's still somewhat easier to maintain than manually writing configure scripts... People have not used autoconf or automake in ages even though those were meant to help with creating configure scripts and Makefiles. |
Whichever is easier is to an extent a matter of opinion, but definitely a matter of a developer's experience. My criticism was directed towards CMake, not to suggest that the GNU Autotools are the best solution for zlib. But I can comment on that. There are objective reasons why the infamous GNU Autotools are easier, for example one can decide to what degree to use them (generated or static Makefiles, generated config.h or not, free choice of other template files). So, there is a gentle learning curve compared to GNU Makefiles. Also, it builds on useful or already present knowledge, like shell scripting and said GNU Makefiles. There is a reason that the GNU Autotools are still widely used, but I see projects that offer support for one or more additional build systems. You do realize that zlib is currently using something that is identical to the GNU Autotools, right? configure, Makefile template, header file templates. For minizip even Automake is used. CMake is a very different story as any developer with enough experience can confirm. However, there is no need to convince anyone -- regardless CMake's supposed strengths or benefits we can see that support for CMake is a problem for zlib, hence this very GitHub issue. |
|
My 2 cents on this is that CMake and Autotools are pretty much orthogonal to each other, and not supporting CMake would leave a very large community hanging. And it isn't really rocket science. |
Thanks. I once met a rocket scientist who claimed that rocket science is not rocket science :-P I would claim that CMake is at least more frustrating than rocket science. How would you describe the current situation? Is CMake currently supported? That is why I suggested 2 solutions:
For people in favor of CMake the first option is obviousy most attractive. For the second option to work well, zlib needs to cooperate. However, now we have neither. In the end, the current developers decide, but anyone is free to fork... |
|
After recently spending 1.5 years trying to understand CMake (and autotools) while fixing it up for two open-source projects (curl and libssh2), my conclusion it that CMake has a steep and frustrating learning curve, but it can be made useful to build a project reliably. It offers many ways for mis-use, but I could see no evidence that it'd "dictate the structure of project development". With autotools, the main problem is that it isn't portable, e.g. it doesn't support native Windows (and thus MSVC.) But zlib also has no autotools support, so it's not an option anyway. Also autotools' It seems safe to say that CMake is at the moment the only ubiquitous portable build tool. |
Saying that autotools (or autoconf) is same as using hand-written configure script and Makefile.in is like saying using pico or nano is same as using vi... The learning curve is completely different... |
Please correct me if I am wrong, but as far as I know building in-place is not supported. It should (must) be done out-of-source. Spaces in paths are not supported, so a requirement on the naming. Building different variants cannot be done from the same project. Building shared and static libraries cannot be done in one run.
As I explained zlib uses a configure script, a Makefile template (Makefile.in), a header file template (zconf.h.in) and a pkg-config file template (zlib.pc.in). The script configure looks generated, but I don't know for sure. Regardless, in practice this is what could also be done with GNU Autotools. The only difference would be the existence of configure.in. Another difference might be configure (if it is currently not generated). In other words, maybe zlib is currently using GNU Autotools. The main reason why GNU Autotools exist is portability, so saying that they are not portable is strange to say the least. Maybe you assume that all UNIX systems are the same? However, GNU Autotools run in any UNIX shell, including when those shells are ported to Windows. You may have heard of MSYS, a port of Bash that is used for MinGW. Or MSYS2, a command line environment used for MinGW-w64. Or of course Bash as part of Cygwin. Windows development is no longer synonymous with Microsoft Visual Studio. GCC or Clang as part of MinGW or MinGW-w64 generate native Win32 or Win64 binaries. I agree about Libtool -- it's an utterly redundant project that appears to be based on assumptions about a hypothetical world that just doesn't match with reality. However, as I tried to explain before is that with the GNU Autotools, it is not all or nothing. Libtool can be entirely ignored. The configurability of a project can be scaled, one template macro at a time.
I hope I could show that that is not true. |
If you know that it was hand-written then it is clear zlib itself does not currently use GNU Autotools. Thanks for clarifying. The analogy is faulty though. I do not know where the miscommunication comes from. If configure were generated from configure.in nothing would be different from a user's perspective compared to now. Note that I am not advocating for this, just explaining how the GNU Autotools could be used. |
|
Is this really the place to debate whether or not CMake is good? Those who don't want to use CMake don't have to explain it in details: just don't use it. I am personally happy about the fact that zlib has some degree of CMake support, and I do understand that it is difficult for the maintainer(s) to support every build system under the sun. @dbjh: you seem to be putting a lot of energy into this. Would you be willing to lead an effort to try to improve the state of CMake in zlib? As mentioned by @Mizux and @madler, it would probably make sense to try to make a list of people who actively use zlib (together with their platform), and ask them to review/test CMake-related PRs. Then you could slowly triage the PRs listed by @Neustradamus, find low-hanging fruits (i.e. simple PRs or PRs that are still up-to-date) and try it there. Criticizing CMake won't improve the CMake support in zlib. What do you think? |
The analogy is quite accurate in my opinion... I've used Autotools, written configure scripts from scratch and converted configure scripts to CMakeLists.txt used by cmake. With both Autotools and cmake you need to be familiar with the available functions just like you need to be familiar with all the different commands in vi. With pico and nano, or hand-written configure scripts, you can pretty much just write whatever you want and not familiarize any extra documentation. Obviously you still need to know the available shell commands and if the current distribution has pico or nano installed (or available as package). |
There are forks of zlib that have improved CMake support, which makes it very easy to backport any applicable improvement. Some of the contributors to the forks are also quite active in pushing Mark Adler to fix various shortcomings of zlib build system. As most of the forks use same license as zlib, there isn't any issue stopping from backporting. |
Then it's failed abysmally: it doesn't support one of the major environments out there.
Are you trying to contradict yourself? It's good that you recognise that Autotools aren't portable because and they don't support the standard toolchain on said major platform. Personally I find GNU Autotools obtuse, cryptic and apparently made to be difficult to debug when they cause random fails (yes, I'm debugging such an instance now after upgrading from Ubuntu 20.04 to 22.04 and switching LLVM 10 to LLVM 16). Cmake on the other hand works much better, although I won't claim it supports all of the niche platforms. Furthermore, it can generate project files for the major IDEs, which makes it infinitely more useful. Not all of us prefer gdb or lldb. Furthermore, CMake makes it easy to keep the development tree clean, by design. |
Thanks for the response. I guess I made the wrong impression :-) I noticed the stall in addressing the issues and PRs regarding CMake and can understand it from the perspective of someone who is not a fan of CMake. So, I would be the wrong person for improving CMake support. I did try to suggest what would help in moving forward. I ended up mainly explaining GNU Autotools and the similarity to the current system, not so much criticizing CMake. As @mtl1979 explained there already are zlib forks with improved CMake support, so others realized the same thing. And apparently that is good enough... I think a lot of words could have been avoided if that was clearer. I still wonder why those issues and PRs are not closed. It gives off an entirely different signal. |
zlib uses a hand-coded
MSYS2 and MSYS are not native Windows. Not all envs support a POSIX shell.
It may be so in theory, but most projects are relying on it nevertheless. TL;DR I think it would serve zlib users much better to offer a tidied-up CMake support, than having a half-broken one or no support for any widely-used and portable build tool at all. Some long-time pending PRs are aiming for low-hanging issues and even those are not merged. |
This is getting a bit silly. I guess you are referring to the Windows family of OSes? Then you are simply mistaken. Windows cannot natively execute a GNU Autotools generated configure script, just as it cannot natively execute Windows binaries produced by Visual Studio that are not statically linked. For the latter you need to install run-time binaries of the respective compiler. For the former you need to install a shell like MSYS, MSYS2 or some other port of Bash.
I do not understand where you see a contradiction. I was claiming they are portable. You just need the proper environment to execute them. Of course the GNU Autotools work seemlessly with GCC and Clang, but I do not see any hard block in using Microsoft's tools. If $CC=cl.exe a configure script would call the Microsoft compiler.
I have had bad experiences with CMake, but that is not exactly an argument. You claim something works better while giving the strong impression that you do not know what it is better than. Like I said it is getting a bit silly. |
|
@dbjh @Mr-Clam @vszakats @mtl1979 and others: maybe it's just my opinion, but I feel like you are hijacking @Neustradamus's issue, and I will unsubscribe after this message. Not that I am important, but if I unsubscribe, it means that maybe others (who may have been willing to help with the CMake situation) did/will, too. |
|
@JonasVautherin: If speaking for CMake hurts this, I'm not quite sure what would help it. edit: anyway, I am unsubscribing, as I've already pulled out with my CMake PR after losing hope earlier. |
|
CMake is not rocket science, but I can attest that being a rocket scientist does not help with understanding CMake. configure was not auto-generated — it is hand-written. zlib will continue to support both configure/make and cmake. At least to the extent that anyone considers the current CMakeLists.txt to be supporting cmake. No, this is not the place to debate whether 'tis better to cmake or not to cmake.
So who can be the judge of whether any given change is actually an improvement or not? It's not me. I don't just apply whatever commits willy nilly. Anyone can propose a change to CMakeLists.txt that fixes their local situation, but also screws up someone else's. Recently I reverted a CMakeLists.txt commit for that reason. In my opinion, we would need cmake testing across a wide variety of platforms, beyond what is available in github actions. |
|
@madler: I understand that. Judging contributions isn't an easy feat. Perhaps one way to bootstrap this it is to merge the most reasonably solid-looking PRs, and/or to request peer-reviews for the most important ones. Or to require adding CI tests before merge. This would also assume that CI tests are actually allowed to run (manually or by default), because at the moment this isn't the case. Adding CMake integration tests would also help, e.g. like this one. Another option is to ask for as granular PRs as possible, targeting one specific issue at a time. While also separating style/format changes from logic changes. I understand that all of this is work and not necessary a small amount, but zlib being probably the most exposed dependency out there, there is a good chance to distribute this among many interested contributors. |
|
@madler wrote:
Please forgive me if I've misunderstood you, but does this mean that it is your position as maintainer that CMakeLists.txt is frozen now and will not change? This is an important detail for those of us who have issues using the existing cmake support for one reason or another. |
I use three different CI providers myself, GitHub Actions is one of them. Besides Linux, MacOS and Windows, I can test pretty much any operating system that Google has CI image for, but mostly I just test different FreeBSD versions. |
No. Note the "At least" which you left out of your emphasis. |
@madler: It is possible to solve all problems and merge some PRs?
Thanks in advance.
Issues:
ZLIB::ZLIBfor CMake. zlib-ng/zlib-ng#1320PRs:
I have not finished:
Linked to:
@Mizux, @albrematt, @johnb003, @miurahr, @WildCard65, @wrowe, @ScatteredRay, @jnhyatt, @Jihadist, @hwhsu1231, @mtl1979, @ClausKlein, @sum01, @craigscott-crascit, @hjmallon, @mattparks, @puneetmatharu, @vszakats, @tbeu, @fredgan, @RobinGeffroy, @jfern2011, @kirankotari, @etiennearnal, @omegacoleman, @yyzworker, @yalcinerbora, @niparx, @navidR, @lperron, @vguen, @tr1cks, @AndrewAtAvenza, @gvollant, @hopeless-programmer-online, @glebm, @jheaff1, @SpaceIm, @AraHaan, @pzychotic, @siepkes, @schultetwin1, @Bagira80, @wantehchang, @dankegel, @chanphil, @MichaelGoulding, @stkruse, next after.
@nmoinvaz
The text was updated successfully, but these errors were encountered: