Menu
I'm trying to build zlib 1.2.8 for Windows using CMake, but I get build errors that I don't know how to fix. Here's my CMake GUI:
This generates without errors, but when I build the resulting solution. I get this error:
2>------ Build started: Project: zlib, Configuration: Release x64 ------
2> Creating library C:/Users/erik/Documents/zlib/1.2.8/project/zlib-1.2.8-vc10/Release/zlib.lib and object C:/Users/erik/Documents/zlib/1.2.8/project/zlib-1.2.8-vc10/Release/zlib.exp
2> inflate.obj : error LNK2019: unresolved external symbol inflate_fast referenced in function inflate
2>infback.obj : error LNK2001: unresolved external symbol inflate_fast
2>C:\Users\erik\Documents\zlib\1.2.8\project\zlib-1.2.8-vc10\Release\zlib.dll : fatal error LNK1120: 1 unresolved externals
I don't know how I can fix this, so I appreciate any help.
449
According to https://wiki.apache.org/httpd/Win64Compilation a very similar error means:
This means you have a typo in either -DASMV -DASMINF or your OBJ="inffasx64.obj gvmat64.obj inffas8664.obj" since inflate_fast is defined in inffas8664.c.
I was able to successfully build with a simple:
mkdir C:\Builds\zlib; cd C:\Builds\zlib
cmake -G "Visual Studio 12 2013" -A x64 D:\Downloads\zlib-1.2.8\
cmake --build .
I looked at my cmake cache and I see that AMD64 is set to false, unlike what your cmake-gui window shows. Setting it to true it results all kinds of build errors for me, though not the ones you show.
CMakeLists.txt says this option is to enable an AMD64 assembly implementation. Just doing without this seems to be the easiest solution.
114
You need contrib\masmx64\inffas8664.c included in visual studio project file.
This file contains inflate_fast function which calls corresponding asm functions.
42
contrib/masmx* directories were removed back in 2017 (at answer time I was working on the v1.2.11 .zip file which contains them), so everything below no longer applies (OOTB).
But their references were not removed (at least not from all places), so if they're enabled (from CMake), build will fail.
However, I submitted [GitHub]: madler/zlib - Re enable ASM speedups on Win, (rejected on 221007) so everything below does apply (again).
For possible ways to benefit from the patch (once / if it's accepted), check [SO]: How to change username of job in print queue using python & win32print (@CristiFati's answer) (at the end).
Applied the above patch to (current) master branch:
Built for:
064bit / 032bit
Release / Debug
With / without ASM speedups
Ran minigzip.exe (which links to zlib(d).dll):
On binary / text file
Methods: inflate / deflate
Levels: 1, 5, 9
Used following script to go through all configurations, run and aggregate results.
code00.py:
#!/usr/bin/env python
import hashlib as hl
import os
import shutil
import subprocess as sp
import sys
import time
from pprint import pprint as pp
ARCHS = (
"pc064",
"pc032",
)
def print_file_data(file, level=0):
st = os.stat(file)
header = " " * level
print("{:s}File: {:s}\n{:s} Size: {:d}, CTime: {:s}".format(header, file, header, st.st_size, time.ctime(st.st_ctime)))
def main(*argv):
verbose = False
build_dir = os.getcwd() #os.path.dirname(os.path.abspath(__file__))
if argv:
file = argv[0]
if "-v" in argv:
verbose = True
else:
file = "file.onnx"
#file = "bigfile.txt"
#file = "enwik8"
file_test = file + ".test"
file_gz = file_test + ".gz"
shutil.copy(file, file_test)
md5_src = hl.md5(open(file_test, mode="rb").read()).hexdigest()
print_file_data(file_test)
data = {}
for arch in ARCHS:
if verbose:
print("Arch: {:s}".format(arch))
ad = {}
for typ in ("plain", "masm"):
if verbose:
print(" Type: {:s}".format(typ))
mg = os.path.join(build_dir, "_build", arch, typ, "minigzip.exe")
for level in (1, 5, 9):
shutil.copy(file, file_test)
if verbose:
print("\n Compress (level {:d})".format(level))
proc = sp.Popen([mg, "-{:d}".format(level), file_test])
time_start = time.time()
proc.communicate()
elapsed = time.time() - time_start
if verbose:
print(" Took {:.3f} seconds".format(elapsed))
ad.setdefault("level {:d}".format(level), {}).setdefault("inflate", {})[typ] = elapsed
if verbose:
print_file_data(file_gz, level=2)
if verbose:
print(" Decompress")
proc = sp.Popen([mg, "-d", file_gz])
time_start = time.time()
proc.communicate()
elapsed = time.time() - time_start
if verbose:
print(" Took {:.3f} seconds".format(elapsed))
ad.setdefault("level {:d}".format(level), {}).setdefault("deflate", {})[typ] = elapsed
if verbose:
print_file_data(file_test, level=2)
if hl.md5(open(file_test, mode="rb").read()).hexdigest() != md5_src:
print("!!! File hashes differ !!!")
data[arch] = ad
pp(data, indent=2, sort_dicts=False)
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("\nDone.")
sys.exit(rc)
Output:
[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q029505121]> "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Scripts\python.exe" ./code00.py file.onnx Python 3.9.9 (tags/v3.9.'level 9':ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] 064bit on win32 File: file.onnx.test Size: 255890833, CTime: Sat Sep 3 02:03:05 2022 { 'pc064': { 'level 1': { 'inflate': { 'plain': 12.552296161651611, 'masm': 11.09960412979126}, 'deflate': { 'plain': 1.802419900894165, 'masm': 1.8380048274993896}}, 'level 5': { 'inflate': { 'plain': 13.694978713989258, 'masm': 12.098156213760376}, 'deflate': { 'plain': 1.756164312362671, 'masm': 1.7628483772277832}}, 'level 9': { 'inflate': { 'plain': 13.969024419784546, 'masm': 12.125015497207642}, 'deflate': { 'plain': 1.7450010776519775, 'masm': 1.756005048751831}}}, 'pc032': { 'level 1': { 'inflate': { 'plain': 13.748999118804932, 'masm': 11.81002926826477}, 'deflate': { 'plain': 1.9236936569213867, 'masm': 2.3493638038635254}}, 'level 5': { 'inflate': { 'plain': 15.036035299301147, 'masm': 12.898797512054443}, 'deflate': { 'plain': 1.8580067157745361, 'masm': 2.282176971435547}}, 'level 9': { 'inflate': { 'plain': 15.134005308151245, 'masm': 12.89007306098938}, 'deflate': { 'plain': 1.8709957599639893, 'masm': 2.2773334980010986}}}} Done. [cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q029505121]> "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Scripts\python.exe" ./code00.py enwik8.txt Python 3.9.9 (tags/v3.9.'level 9':ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] 064bit on win32 File: enwik8.txt.test Size: 100000000, CTime: Tue Sep 6 00:33:20 2022 { 'pc064': { 'level 1': { 'inflate': { 'plain': 1.9976372718811035, 'masm': 1.9259986877441406}, 'deflate': { 'plain': 0.7285704612731934, 'masm': 0.7076430320739746}}, 'level 5': { 'inflate': { 'plain': 4.5627357959747314, 'masm': 4.003000020980835}, 'deflate': { 'plain': 0.6933917999267578, 'masm': 0.6450159549713135}}, 'level 9': { 'inflate': { 'plain': 8.079626083374023, 'masm': 6.618978977203369}, 'deflate': { 'plain': 0.7049713134765625, 'masm': 0.6319396495819092}}}, 'pc032': { 'level 1': { 'inflate': { 'plain': 2.1649997234344482, 'masm': 2.1139981746673584}, 'deflate': { 'plain': 0.7583539485931396, 'masm': 0.8125534057617188}}, 'level 5': { 'inflate': { 'plain': 5.03799843788147, 'masm': 4.2109808921813965}, 'deflate': { 'plain': 0.8489999771118164, 'masm': 0.6870477199554443}}, 'level 9': { 'inflate': { 'plain': 7.9073097705841064, 'masm': 7.512087821960449}, 'deflate': { 'plain': 0.7378275394439697, 'masm': 0.7450006008148193}}}} Done. [cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q029505121]> cd dbg [cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q029505121\dbg]> "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Scripts\python.exe" ../code00.py ../file.onnx Python 3.9.9 (tags/v3.9.'level 9':ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] 064bit on win32 File: ../file.onnx.test Size: 255890833, CTime: Tue Sep 6 00:37:51 2022 { 'pc064': { 'level 1': { 'inflate': { 'plain': 25.337001085281372, 'masm': 22.544013023376465}, 'deflate': { 'plain': 3.915001153945923, 'masm': 2.3709957599639893}}, 'level 5': { 'inflate': { 'plain': 28.28699827194214, 'masm': 24.88018822669983}, 'deflate': { 'plain': 3.846531867980957, 'masm': 2.2239699363708496}}, 'level 9': { 'inflate': { 'plain': 28.81813645362854, 'masm': 23.6450355052948}, 'deflate': { 'plain': 3.9910058975219727, 'masm': 2.302088737487793}}}, 'pc032': { 'level 1': { 'inflate': { 'plain': 24.923137664794922, 'masm': 20.991183042526245}, 'deflate': { 'plain': 3.7310261726379395, 'masm': 2.6056015491485596}}, 'level 5': { 'inflate': { 'plain': 27.760021209716797, 'masm': 22.589048624038696}, 'deflate': { 'plain': 3.566000461578369, 'masm': 2.55342698097229}}, 'level 9': { 'inflate': { 'plain': 28.245535135269165, 'masm': 22.70799994468689}, 'deflate': { 'plain': 3.553999423980713, 'masm': 2.5700416564941406}}}} Done. [cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q029505121\dbg]> "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Scripts\python.exe" ../code00.py ../enwik8.txt Python 3.9.9 (tags/v3.9.'level 9':ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] 064bit on win32 File: ../enwik8.txt.test Size: 100000000, CTime: Tue Sep 6 00:39:59 2022 { 'pc064': { 'level 1': { 'inflate': { 'plain': 4.711355447769165, 'masm': 4.008531808853149}, 'deflate': { 'plain': 1.4210000038146973, 'masm': 0.9230430126190186}}, 'level 5': { 'inflate': { 'plain': 8.914000034332275, 'masm': 6.604032516479492}, 'deflate': { 'plain': 1.3359959125518799, 'masm': 0.8460018634796143}}, 'level 9': { 'inflate': { 'plain': 13.503999948501587, 'masm': 9.228030920028687}, 'deflate': { 'plain': 1.328040599822998, 'masm': 0.8240146636962891}}}, 'pc032': { 'level 1': { 'inflate': { 'plain': 4.435391664505005, 'masm': 3.933983087539673}, 'deflate': { 'plain': 1.3369977474212646, 'masm': 0.9399752616882324}}, 'level 5': { 'inflate': { 'plain': 8.48900055885315, 'masm': 6.599977731704712}, 'deflate': { 'plain': 1.2629964351654053, 'masm': 0.8410165309906006}}, 'level 9': { 'inflate': { 'plain': 12.677618026733398, 'masm': 9.191060781478882}, 'deflate': { 'plain': 1.251995325088501, 'masm': 0.8130028247833252}}}} Done.
As seen, for release builds (that most people are interested in), the speed gain using the speedups is not that great (sometimes it's even slower than the C code). This (combined with the lack of maintenance) was one of the reasons for disabling them.
While playing with assembler speedups, I discovered that the issue is reproducible on the (currently) latest version: v1.2.11 ([GitHub]: madler/zlib - ZLIB DATA COMPRESSION LIBRARY).
This error happens only (obviously, OS: Win, build toolchain: VStudio and assembly speedups enabled) for:
CMake build (works OK for "${ZLIB_SRC_DIR}/win32/Makefile.msc")
x64 (AMD64 (pc064)) architecture (works OK for x86 (pc032))
Below is a "callstack" (top -> down is equivalent to outer -> inner) during decompression.
Normal case:
inflate (inflate.c)
inflate_fast (inffast.c)
...
Assembler case:
inflate (inflate.c)
inflate_fast (contrib/masmx64/inffast8664.c)
inffas8664fnc (contrib/masmx64/inffasx64.asm)
...
#2.2. is missing ("${ZLIB_SRC_DIR}/CMakeLists.txt" doesn't know anything about inffast8664.c), so the chain is broken, leading to invalid (incomplete) data to build library.
Make CMakeLists.txt aware of that file, and that is by adding:
set(ZLIB_SRCS
${ZLIB_SRCS}
contrib/masmx64/inffas8664.c
)
at line ~#158 (enclosed by if(MSVC) and elseif (AMD64) conditionals).
Posting full changes as well.
zlib-1.2.11-msvc_x64_asm_speedups.diff:
--- CMakeLists.txt.orig 2017-01-15 08:29:40.000000000 +0200
+++ CMakeLists.txt 2018-09-03 13:41:00.314805100 +0300
@@ -79,10 +79,10 @@
endif()
set(ZLIB_PC ${CMAKE_CURRENT_BINARY_DIR}/zlib.pc)
-configure_file( ${CMAKE_CURRENT_SOURCE_DIR}/zlib.pc.cmakein
- ${ZLIB_PC} @ONLY)
-configure_file( ${CMAKE_CURRENT_SOURCE_DIR}/zconf.h.cmakein
- ${CMAKE_CURRENT_BINARY_DIR}/zconf.h @ONLY)
+configure_file(${CMAKE_CURRENT_SOURCE_DIR}/zlib.pc.cmakein
+ ${ZLIB_PC} @ONLY)
+configure_file(${CMAKE_CURRENT_SOURCE_DIR}/zconf.h.cmakein
+ ${CMAKE_CURRENT_BINARY_DIR}/zconf.h @ONLY)
include_directories(${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_SOURCE_DIR})
@@ -136,30 +136,34 @@
set(ZLIB_ASMS contrib/amd64/amd64-match.S)
endif ()
- if(ZLIB_ASMS)
- add_definitions(-DASMV)
- set_source_files_properties(${ZLIB_ASMS} PROPERTIES LANGUAGE C COMPILE_FLAGS -DNO_UNDERLINE)
- endif()
+ if(ZLIB_ASMS)
+ add_definitions(-DASMV)
+ set_source_files_properties(${ZLIB_ASMS} PROPERTIES LANGUAGE C COMPILE_FLAGS -DNO_UNDERLINE)
+ endif()
endif()
if(MSVC)
if(ASM686)
- ENABLE_LANGUAGE(ASM_MASM)
+ ENABLE_LANGUAGE(ASM_MASM)
set(ZLIB_ASMS
- contrib/masmx86/inffas32.asm
- contrib/masmx86/match686.asm
- )
+ contrib/masmx86/inffas32.asm
+ contrib/masmx86/match686.asm
+ )
elseif (AMD64)
- ENABLE_LANGUAGE(ASM_MASM)
+ ENABLE_LANGUAGE(ASM_MASM)
set(ZLIB_ASMS
- contrib/masmx64/gvmat64.asm
- contrib/masmx64/inffasx64.asm
- )
+ contrib/masmx64/gvmat64.asm
+ contrib/masmx64/inffasx64.asm
+ )
+ set(ZLIB_SRCS
+ ${ZLIB_SRCS}
+ contrib/masmx64/inffas8664.c
+ )
endif()
- if(ZLIB_ASMS)
- add_definitions(-DASMV -DASMINF)
- endif()
+ if(ZLIB_ASMS)
+ add_definitions(-DASMV -DASMINF)
+ endif()
endif()
# parse the full version number from zlib.h and include in ZLIB_FULL_VERSION
The above is a diff. See [SO]: Run / Debug a Django application's UnitTests from the mouse right click context menu in PyCharm Community Edition? (@CristiFati's answer) (Patching UTRunner section) for how to apply patches on Win (basically, every line that starts with one "+" sign goes in, and every line that starts with one "-" sign goes out).
I also submitted this patch: [GitHub]: madler/zlib - Ms VisualStudio - Assembler speedups on x64, but then I closed it, as it's contained in the one at the beginning.
Output:
e:\Work\Dev\StackOverflow\q029505121\build\x64>"c:\Install\Google\Android_SDK\cmake\3.6.4111459\bin\cmake.exe" -G "NMake Makefiles" -DAMD64=ON "e:\Work\Dev\StackOverflow\q029505121\src\zlib-1.2.11" -- The C compiler identification is MSVC 19.0.24215.1 -- Check for working C compiler: C:/Install/x86/Microsoft/Visual Studio Community/2015/VC/bin/amd64/cl.exe -- Check for working C compiler: C:/Install/x86/Microsoft/Visual Studio Community/2015/VC/bin/amd64/cl.exe -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Looking for sys/types.h -- Looking for sys/types.h - found -- Looking for stdint.h -- Looking for stdint.h - found -- Looking for stddef.h -- Looking for stddef.h - found -- Check size of off64_t -- Check size of off64_t - failed -- Looking for fseeko -- Looking for fseeko - not found -- Looking for unistd.h -- Looking for unistd.h - not found -- Renaming -- E:/Work/Dev/StackOverflow/q029505121/src/zlib-1.2.11/zconf.h -- to 'zconf.h.included' because this file is included with zlib -- but CMake generates it automatically in the build directory. -- The ASM_MASM compiler identification is MSVC -- Found assembler: C:/Install/x86/Microsoft/Visual Studio Community/2015/VC/bin/amd64/ml64.exe -- Configuring done -- Generating done -- Build files have been written to: E:/Work/Dev/StackOverflow/q029505121/build/x64 e:\Work\Dev\StackOverflow\q029505121\build\x64>"c:\Install\Google\Android_SDK\cmake\3.6.4111459\bin\cmake.exe" --build . --target zlibstatic Scanning dependencies of target zlibstatic [ 5%] Building C object CMakeFiles/zlibstatic.dir/adler32.obj adler32.c [ 10%] Building C object CMakeFiles/zlibstatic.dir/compress.obj compress.c [ 15%] Building C object CMakeFiles/zlibstatic.dir/crc32.obj crc32.c [ 21%] Building C object CMakeFiles/zlibstatic.dir/deflate.obj deflate.c Assembler code may have bugs -- use at your own risk [ 26%] Building C object CMakeFiles/zlibstatic.dir/gzclose.obj gzclose.c [ 31%] Building C object CMakeFiles/zlibstatic.dir/gzlib.obj gzlib.c [ 36%] Building C object CMakeFiles/zlibstatic.dir/gzread.obj gzread.c [ 42%] Building C object CMakeFiles/zlibstatic.dir/gzwrite.obj gzwrite.c [ 47%] Building C object CMakeFiles/zlibstatic.dir/inflate.obj inflate.c [ 52%] Building C object CMakeFiles/zlibstatic.dir/infback.obj infback.c [ 57%] Building C object CMakeFiles/zlibstatic.dir/inftrees.obj inftrees.c [ 63%] Building C object CMakeFiles/zlibstatic.dir/inffast.obj inffast.c Assembler code may have bugs -- use at your own risk [ 68%] Building C object CMakeFiles/zlibstatic.dir/trees.obj trees.c [ 73%] Building C object CMakeFiles/zlibstatic.dir/uncompr.obj uncompr.c [ 78%] Building C object CMakeFiles/zlibstatic.dir/zutil.obj zutil.c [ 84%] Building C object CMakeFiles/zlibstatic.dir/contrib/masmx64/inffas8664.obj inffas8664.c [ 89%] Building ASM_MASM object CMakeFiles/zlibstatic.dir/contrib/masmx64/gvmat64.obj Microsoft (R) Macro Assembler (x64) Version 14.00.24210.0 Copyright (C) Microsoft Corporation. All rights reserved. Assembling: E:\Work\Dev\StackOverflow\q029505121\src\zlib-1.2.11\contrib\masmx64\gvmat64.asm [ 94%] Building ASM_MASM object CMakeFiles/zlibstatic.dir/contrib/masmx64/inffasx64.obj Microsoft (R) Macro Assembler (x64) Version 14.00.24210.0 Copyright (C) Microsoft Corporation. All rights reserved. Assembling: E:\Work\Dev\StackOverflow\q029505121\src\zlib-1.2.11\contrib\masmx64\inffasx64.asm [100%] Linking C static library zlibstatic.lib [100%] Built target zlibstatic
Notes:
I am using VStudio 2015
Regarding the above output:
To keep the output as small as possible, I am only building the static version
inffas8664.c is being built (somewhere near the end)
You could also disable assembler speedups (by unchecking AMD64 in CMake-GUI), but that would be just a workaround
I did some rough tests (by far, I'm not claiming these results to be general), and the performance improvement of the assembler implementation over the standard one (Debug versions) was (a percent below is the ratio between times took to perform the same operation (with / without) speedups):
Compress: ~86%
Decompress: ~62%
[GitHub]: madler/zlib - ASM zlib build on Windows gives erroneous results (@madler's comment) states (emphasis is mine):
What assembly code is being used? There are a few in zlib's contrib directory. By the way, the stuff in the contrib directory is not part of zlib. It is just there as a convenience and is supported (or not) by those third-party contributors. What I will do is simply remove the offending code from the next distribution.
So does the compile warning (that everyone must have seen (and most likely ignored)):
Assembler code may have bugs -- use at your own risk
Apparently, assembler speedups and VStudio don't (didn't) get along very well. More, on x86 (pc032), there are some issues:
A nasty one is a SegFault (Access Violation) during decompression. For that, [GitHub]: madler/zlib - inffas32.asm struct/enum binding to zlib 1.2.9 (also included in the one at the beginning) is needed
Might no longer apply: one fix is proposed by [SO]: module unsafe for SAFESEH image C++ (@NayanaAdassuriya's answer) (although it's not directly related to the question). In short, inffas32.asm and linker's option [MS.Docs]: /SAFESEH (Image has Safe Exception Handlers) don't match. To get rid of it, either:
Disable the option (enabled by default on x86 Release)
Pass the /safeseh option to the assembler (ml.exe)
Declare one in asm
Since I use CMake to build for CmdLine, I found a workaround for this. After CMakeFiles generation (but before the build), I specify it:
In the flags.make file (ASM_MASM_FLAGS)
Since I build a static lib (and for Debug builds I need the symbols as well), I also change [MS.Docs]: /Z7, /Zi, /ZI (Debug Information Format), so I modify the same file (C_FLAGS)
I'm sure that CMake offers a way of doing the above in a proper manner, but I didn't find it (nor did I investigate thoroughly)
After fixing them, everything works fine, and the performance improvements are similar to pc064.
In case anyone needs them, I've built and placed binaries at [GitHub]: CristiFati/Prebuilt-Binaries - (master) Prebuilt-Binaries/ZLib (with / without speedups).
9
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
