FFMPEG on cygwin failed to compile libx264 error: unknown type name ‘HMODULE’

Question

I am trying to compile libx264 in ffmpeg under cygwin environment.

I have followed some directions from several sources from Koohiimaster's blog, FFMPEG compilation guide, SO post 1, SO post 2 but I always stuck at the same step which is the libx264 compilation (make) process.

As mentioned in the FFMPEG compilation guide these steps should be followed in order to make libx264 works

cd ~/ffmpeg_sources
wget http://download.videolan.org/pub/x264/snapshots/last_x264.tar.bz2
tar xjvf last_x264.tar.bz2
cd x264-snapshot*
PATH="$HOME/bin:$PATH" ./configure --prefix="$HOME/ffmpeg_build" bindir="$HOME/bin" --enable-static --disable-opencl
PATH="$HOME/bin:$PATH" make
make install

but when i enter this command PATH="$HOME/bin:$PATH" make the compiler always stop with the following errors:

In file included from input/avs.c:49:0:
./extras/avisynth_c.h:825:3: error: unknown type name ‘HMODULE’
HMODULE handle;
^

I was wondering whether this is libx264 source's bug, but after I tried several earlier source version, it produce the same error. Any thoughts to solve this problem?

Answer 1

Info

The type HMODULE is declared in WinDef.h. So you're missing to include this header. I think normally WinDef.h is included by windows.h.

Alternative 1: Without AVS

If you do not need avs support you might just disable it and see if it works then:

PATH="$HOME/bin:$PATH" ./configure --prefix="$HOME/ffmpeg_build" bindir="$HOME/bin" --enable-static --disable-opencl --disable-avs

Actually this is a good choice to just check out and see if it gets you more information on your problem.

Alternative 2: Check windows.h

Another thing is to check which windows.h file is used in your configuration.

In the libx264 sources input/input.h includes windows.h.

input.h itself is then included by avs.c. avs.c also includes extras/avisynth_c.h thereafter. So all in all when the line HMODULE handle appears for the compiler, windows.h should have already been included (where WinDef.h should be included).

So either your windows.h file is somehow "wrong" or something really goes wrong during preprocessing.

Answer 2

On my system (Cygwin 2.9.0(0.318/5/3)/Windows 7 64-bit SP1), HMODULE is defined in wtypes.h, which is not included by windows.h or any other included file.

Editing windows.h (at /usr/lib/w32api/windows.h) to add #include <wtypes.h> worked for me (I added it on the line after #define _INC_WINDOWS).

I also added --extra-cflags="-I/usr/include/w32api/" to the configure command, along with --extra-ldflags="-L:/usr/local/lib/" although I'm not sure whether the ldflags addition was related to this or libfdk-aac

Answer 3

EDIT: I looked at the README from the snapshots page.

$ wget -O - \
   https://download.videolan.org/x264/snapshots/x264-snapshot-20191218-README.txt \
    2>/dev/null
The snapshotting service is discontinued.

Please use https://code.videolan.org/videolan/x264/ to get the tarballs.

Because of this, I used the new repository for the download.

If you used the previous instructions, you'll probably want to clear out the previous x264 stuff as follows

cd $HOME/programs/ffmpeg/ffmpeg_sources/x264-snapshot-20191217-2245
make clean
make distclean

Then, if you ran make install for x264, you'll also need to do the following

cd $HOME/programs/ffmpeg

find . -type f -name "*x264*" -print0 | \
   xargs -I'{}' -0 \
     bash -c 'orig="{}"; echo; echo "Attempting to remove:"; '\
'echo "${orig}"; echo "       ... "; echo "         ... as file ..."; '\
'[ -e "${orig}" ] && rm "${orig}" && '\
'echo "                              ... success" || '\
'echo "                              ... FAILURE";'; \
find . -type d -name "*x264*" -print0 | \
   xargs -I'{}' -0 \
     bash -c 'orig="{}"; echo; echo "Attempting to remove:"; '\
'echo "${orig}"; echo "       ... "; echo "         ... as directory ..."; '\
'[ -d "${orig}" ] && rm -rf "${orig}" && '\
'echo "                              ... success" || '\
'echo "                              ... FAILURE";';

There's a shorter but less informative version of the second command. It will try to delete the files in directories which were already deleted, which might take a longer time. I'm not sure the best way to check if it always worked or not.

find . -name "*x265*" -print0 | \
   xargs -I'{}' -0 \
     bash -c 'orig="{}"; echo > tmpf; [ -e "${orig}" ] && rm "${orig}" 2>tmpf; '\
'grep -v "rm.*cannot.*remove.*directory" tmpf; rm tmpf; '\
'[ -d "${orig}" ] && rm -rf "${orig}";' 

Now, we go on with the newest way to take care of x264.

cd $HOME/programs/ffmpeg/ffmpeg_sources
git clone https://code.videolan.org/videolan/x264.git
cd x264

From here, I had to follow the same steps as I did with the tarball from the snapshots page, i.e. I just repeated the instructions starting with

PATH="$HOME/programs/ffmpeg/bin:$PATH" \
    ./configure --prefix="$HOME/programs/ffmpeg/ffmpeg_build" \
                --bindir="$HOME/programs/ffmpeg/bin" \
                --enable-static`

including the wtypes.h fix.

I've marked this location with ++Where to start with newer download++

---

I have a slightly different system. I was able to get things working thanks to the tips from both @Missy and @Jan and from other references - koohiimaster especially (since I want a Cygwin build), as well as this SuperUser answer for "How to compile the best version of FFmpeg for Windows" (a Windows build - further from what I want).

However, there were a few things that were quite different, and I'm sharing them for anyone who might be in my situation.

System Details

$ date && date +'%s'
Sat, May  2, 2020  2:41:12 PM
1588452072
$ uname -a
CYGWIN_NT-10.0 MY-MACHINE 3.1.4(0.340/5/3) 2020-02-19 08:49 x86_64 Cygwin
$ bash --version | head -n 1
GNU bash, version 4.4.12(3)-release (x86_64-unknown-cygwin)
$ gcc --version | head -n 1
gcc (GCC) 9.3.0
$ g++ --version | head -n 1
g++ (GCC) 9.3.0
$ make --version | head -n 2
GNU Make 4.3
Built for x86_64-pc-cygwin
$ systeminfo | sed -n 's#^OS\ *##p'
Name:                   Microsoft Windows 10 Home
Version:                10.0.18363 N/A Build 18363
Manufacturer:           Microsoft Corporation
Configuration:          Standalone Workstation
Build Type:             Multiprocessor Free

My directory structure is a bit different, but I hope close enough for everyone to follow along.

---

My Steps and (Error) Output

mkdir -p ~/programs/ffmpeg
cd ~/programs/ffmpeg
mkdir bin && mkdir ffmpeg_sources && mkdir ffmpeg_build
cd ffmpeg_build

Note that the instructions want you to download and extract using the following commands,

wget http://download.videolan.org/pub/x264/snapshots/last_x264.tar.bz2
tar xjvf last_x264.tar.bz2

However, that isn't the actual URL; they want you to download the latest x264 version from https://download.videolan.org/x264/snapshots/. I also got the README, because it keeps me out of trouble.

wget https://download.videolan.org/x264/snapshots/x264-snapshot-20191217-2245.tar.bz2  ## necessary
wget https://download.videolan.org/x264/snapshots/x264-snapshot-20191218-README.txt  ## optional, but helpful

Hopefully, you'll have a different date at the end of the filename if you see this answer in a couple months or so. To save some space in this answer, I'm using xjf flags instead of xjvf - I don't want verbose. After the untaring, I'll continue on, following the instructions.

$ tar xjf x264-snapshot-20191217-2245.tar.bz2
$ cd x264-snapshot-20191217-2245

Now, we're ready for the build. You should be in a directory similar to the following (though your date might be different).

$HOME/programs/ffmpeg/ffmpeg_sources/x264-snapshot-20191217-2245

++Where to start with newer download++

cd $HOME/programs/ffmpeg/ffmpeg_sources
git clone https://code.videolan.org/videolan/x264.git
cd x264

(If you are doing the newer download, your directory should be $HOME/programs/ffmpeg/ffmpeg_sources/x264)

$ PATH="$HOME/programs/ffmpeg/bin:$PATH" \
    ./configure --prefix="$HOME/programs/ffmpeg/ffmpeg_build" \
                --bindir="$HOME/programs/ffmpeg/bin" \
                --enable-static

## Suppressed output: Some nice info on the configuration. No error stuff.

You can run 'make' or 'make fprofiled' now.

Next command:

$ PATH="$HOME/programs/ffmpeg/bin:$PATH" make -j $(nproc)

## Suppressed output: Some good output - not even any warnings.

In file included from input/avs.c:49:
./extras/avisynth_c.h:825:3: error: unknown type name ‘HMODULE’
  825 |   HMODULE handle;

## Suppressed output: A bunch of warnings that we won't see once we fix this.

make: *** [Makefile:272: input/avs.o] Error 1

The Solution (and How to Get There)

Thanks to @Missy , I had an idea what to look for and what to do.

Depending on whether you're using the older snapshots or the newer git stuff, your current working directory (pwd) will be different.

$ pwd   ### (older snapshot)
/home/bballdave025/programs/ffmpeg/ffmpeg_sources/x264-snapshot-20191217-2245
$ pwd   ### (newer git stuff)
/home/bballdave025/programs/ffmpeg/ffmpeg_sources/x264

If you don't match, cd into the directory you need.

$ whereis w32api
w32api: /usr/lib/w32api /usr/include/w32api
grep -iIRHn "^typedef.*HMODULE[;]" /usr/include/w32api
/usr/include/w32api/wtypes.h:78:typedef void *HMODULE;
## For grep
## -i : ignore case
## -I : ignore binary files
## -R : recursive search
## -H : output the filename
## -n : output the line number

Note that some of you might have the mingw version or some other version somewhere like

/usr/i686-pc-cygwin/sys-root/usr/include/w32api/

but that won't help if we want to have a Cygwin build. You might need to do something like @Missy did and

[add] --extra-cflags="-I/usr/include/w32api/" to the configure command, along with --extra-ldflags="-L:/usr/local/lib/"

I didn't have to do that, however.

We could go to ./input/input.h and ./extras/avisynth_c.h (where, for me . is ~/programs/ffmpeg/ffmpeg_sources/x264-snapshot-20191217-2245) to add lines to include wtypes.h, but, when I did that, it didn't work. There is a problem with that header file that we'll need to fix first. I'll do a little exploring by passing grep the command to give ten (10) lines of Context before and after the found line: -C 10. I'll also put in line numbers with cat -n. The command would be as follows (notice that I no longer have the ^ anchor at the beginning of grep's regex due to the line numbers).

cat -n /usr/include/w32api/wtypes.h | grep -C 10 "typedef.*HMODULE[;]"

That allows me to find the problem and show it to you with less lines. We'll put a few before (-B <n_lines>) and a few after (-A <n_lines>).

$   ### before
$ cat -n /usr/include/w32api/wtypes.h | grep -B 6 -A 2 "typedef.*HMODULE[;]"
    72      byte data[1];
    73  } RemHBRUSH;
    74  #if 0
    75  typedef UINT_PTR WPARAM;
    76  typedef LONG_PTR LPARAM;
    77  typedef LONG_PTR LRESULT;
    78  typedef void *HMODULE;
    79  typedef void *HINSTANCE;
    80  typedef void *HTASK;

That #if 0 is going to be a problem - it's the equivalent of saying if false, so it will never be evaluated or executed. (If you want to see all the way to the #endif, use -A 49 with your grep).

The simplest solution, in my mind, is to take the typedef void *HMODULE: out of the if statement. I will heed the warning of the file comments,

$ head -n 5 /usr/include/w32api/wtypes.h   ### BEFORE
/*** Autogenerated by WIDL 1.6 from include/wtypes.idl - Do not edit ***/

#ifndef __REQUIRED_RPCNDR_H_VERSION__
#define __REQUIRED_RPCNDR_H_VERSION__ 475
#endif

and not edit. I figure it's best to leave the Win32 API alone. I'll copy the header file into our project. Because of the way the linker looks for header files, it will look for our ffmpeg_sources/wtypes.h first. (I tried renaming it to avoid possible namespace conflicts, but that didn't work).

Okay, I wasn't able to get the linker to find the new header (which I named wtypesx264.h) while it was in our project folder. After I make the edits, I'll move it back to /usr/include/w32api/ with the new name.

Once again, I don't want to mess with the Win32 API any more than necessary, so I'm not going to edit windows.h. Here's the copy, followed by the edit (which I've done in vim, but which you can do in whichever editor you want). To show you which changes need making, I've shown BEFORE versions - the head and grep from before - and AFTER versions, which will be coming).

Depending on whether you're using the older snapshots or the newer git stuff, your current working directory (pwd) will be different.

$ pwd   ### (older snapshot)
/home/bballdave025/programs/ffmpeg/ffmpeg_sources/x264-snapshot-20191217-2245
$ pwd   ### (newer git stuff)
/home/bballdave025/programs/ffmpeg/ffmpeg_sources/x264

If yours doesn't match, cd into the appropriate directory.

$ cp /usr/include/w32api/wtypes.h ./wtypes.h
$ vim wtypes.h
$ head -n 7 wtypes.h   ### AFTER
/*** Edited from the version Autogenerated by WIDL 1.6 from include/wtypes.idl
 *  - Edited 2020-05-02 by bballdave025
 ***/

#ifndef __REQUIRED_RPCNDR_H_VERSION__
#define __REQUIRED_RPCNDR_H_VERSION__ 475
#endif
$ cat -n wtypes.h | head -n 83 | tail -n 10   ### AFTER
    74      byte data[1];
    75  } RemHBRUSH;
    76  typedef void *HMODULE;/*[A]Moved here from inside `#if 0` DWB 20200502*/
    77  #if 0
    78  typedef UINT_PTR WPARAM;
    79  typedef LONG_PTR LPARAM;
    80  typedef LONG_PTR LRESULT;
    81  /*[A]Removed `typedef void *HMODULE;` from inside `#if 0` DWB 20200502*/
    82  typedef void *HINSTANCE;
    83  typedef void *HTASK;

Now, we'll include the header file where it needs to go. Once again, I'll show the before and after to show the edits that need to be made.

First edit: ./input/input.h

$ cat -n ./input/input.h | head -n 37 | tail -11   ### BEFORE
    27
    28  #ifndef X264_INPUT_H
    29  #define X264_INPUT_H
    30
    31  #include "x264cli.h"
    32
    33  #ifdef _WIN32
    34  #include <windows.h>
    35  #endif
    36
    37  /* options that are used by only some demuxers */
$ vim ./input/input.h
$ cat -n ./input/input.h | head -n 39 | tail -13   ### AFTER
    27
    28  #ifndef X264_INPUT_H
    29  #define X264_INPUT_H
    30
    31  #include "x264cli.h"
    32
    33  #ifdef _WIN32
    34  #include <windows.h>
    35  /*Next line added by bballdave025 2020-05-02*/
    36  #include "wtypes.h"
    37  #endif
    38
    39  /* options that are used by only some demuxers */

Second (and last) edit: ./extras/avisynth_c.h

$ cat -n extras/avisynth_c.h | head -n 47 | tail -12   ### BEFORE
    36
    37  #ifndef __AVISYNTH_C__
    38  #define __AVISYNTH_C__
    39
    40  #ifdef __cplusplus
    41  #  define EXTERN_C extern "C"
    42  #else
    43  #  define EXTERN_C
    44  #endif
    45
    46  #define AVSC_USE_STDCALL 1
    47
$ vim extras/avisynth_c.h
$ cat -n extras/avisynth_c.h | head -n 53 | tail -18   ### AFTER
    36
    37  #ifndef __AVISYNTH_C__
    38  #define __AVISYNTH_C__
    39
    40  #ifdef __cplusplus
    41  #  define EXTERN_C extern "C"
    42  #else
    43  #  define EXTERN_C
    44  #endif
    45
    46  /*Start of the part added by bballdave025 2020-05-02*/
    47  #ifdef _WIN32
    48  #include "wtypes.h"
    49  #endif
    50  /*  End of the part added by bballdave025 2020-05-02*/
    51
    52  #define AVSC_USE_STDCALL 1
    53

Trying Again

Depending on whether you're using the older snapshots or the newer git stuff, your current working directory (pwd) will be different.

$ pwd   ### (older snapshot)
/home/bballdave025/programs/ffmpeg/ffmpeg_sources/x264-snapshot-20191217-2245
$ pwd   ### (newer git stuff)
/home/bballdave025/programs/ffmpeg/ffmpeg_sources/x264

If yours doesn't match, cd into the appropriate directory.

$ make clean       # I won't show any of the output - there's a lot
$ make distclean   # I won't show any of the output - there's a lot

Let's check that our changes made it through that distclean

$ ls wt*.h
wtypes.h
$ grep -RHn "[#]include \"wtypes.h\"" .
./extras/avisynth_c.h:48:#include "wtypes.h"
./input/input.h:36:#include "wtypes.h"

Run the ./configure script; this is quick, so I don't worry about using multiple processors.

$ PATH="$HOME/programs/ffmpeg/bin:$PATH" \
    ./configure --prefix="$HOME/programs/ffmpeg/ffmpeg_build" \
                --bindir="$HOME/programs/ffmpeg/bin" \
                --enable-static   ### I won't show output

Here's our test to see if it worked - let's run the make, using all our processors.

$ PATH="$HOME/programs/ffmpeg/bin:$PATH" make -j $(nproc)

The output has some warnings, but no errors. I don't feel like going back and making sure everything is up to perfect-C standards, so I'm going on to the install part.

Depending on whether you're using the older snapshots or the newer git stuff, your current working directory (pwd) will be different.

$ pwd   ### (older snapshot)
/home/bballdave025/programs/ffmpeg/ffmpeg_sources/x264-snapshot-20191217-2245
$ pwd   ### (newer git stuff)
/home/bballdave025/programs/ffmpeg/ffmpeg_sources/x264

If yours doesn't match, cd into the appropriate directory.

$ make install   ### I won't show output
$ make distclean   ### I won't show output.

Now, we'll add the executable to our PATH using ~/.bashrc, but we'll back things up, first, and make sure we know the backup names in case we need to make a quick fix.

Backup

$ date && date +'%s'
Sun, May  3, 2020  3:07:48 PM
1588540068
$ cp ~/.bashrc ~/.bashrc.$(date +'%s').bak
$ echo "$PATH" > ~/.PATH.$(date +'%s').bak
$ find ~ -mindepth 1 -maxdepth 1 -type f -name ".bashrc*.bak" | sort | tail -1
/home/13852/.bashrc.1588540100.bak
$ find ~ -mindepth 1 -maxdepth 1 -type f -name ".PATH*.bak" | sort | tail -1
/home/13852/.PATH.1588540128.bak

Actual change

$ echo -e "\n## Make x264 available\nexport PATH=\"/home/bballdave025/programs/ffmpeg/bin:"'$PATH'"\"\n\n" >> ~/.bashrc
$ source ~/.bashrc

---

Testing Expected Output

Two or three rite high-jump video. (I don't think I can upload it, here, so try your own video).

$ # TEST 1
$ x264 --help | head -n 14
x264 core:157
Syntax: x264 [options] -o outfile infile

Infile can be raw (in which case resolution is required),
  or YUV4MPEG (*.y4m),
  or Avisynth if compiled with support (no).
  or libav* formats if compiled with lavf support (no) or ffms support (no).
Outfile type is selected by filename:
 .264 -> Raw bytestream
 .mkv -> Matroska
 .flv -> Flash Video
 .mp4 -> MP4 if compiled with GPAC or L-SMASH support (no)
Output bit depth: 8/10
$ ## That works, but it seems strange that there's no Avisynth after all
$ ## that trouble

$ # TEST 2
$ bballdave025@MY-MACHINE /cygdrive/c/Users/bballdave025/Desktop
$ x264 --input-res 480x360 --preset placebo --tune psnr -o highjump2.mkv FOSBUR-Dick_1968_Olympics_Mexico_City.mkv
raw [info]: 480x360p 0:0 @ 25/1 fps (cfr)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AX2
x264 [info]: profile Progressive High, level 3.1, 4:2:0, 8-bit
x264 [info]: frame I:1     Avg QP:28.40  size:147888
x264 [info]: frame P:1     Avg QP:29.52  size:137810
x264 [info]: mb I  I16..4:  0.0%  0.0% 100.0%
x264 [info]: mb P  I16..4: 95.7%  0.0%  4.3%  P16..4:  0.0%  0.0%  0.0%  0.0%  .0%    skip: 0.0%
x264 [info]: 8x8 transform intra:0.0%
x264 [info]: coded y,uvDC,uvAC intra: 98.8% 100.0% 100.0%
x264 [info]: i16 v,h,dc,p:  0%  0% 90% 10%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu:  6%  3% 37%  9% 10%  7% 10%  8% 10%
x264 [info]: i8c dc,h,v,p: 91%  0%  0%  8%
x264 [info]: Weighted P-Frames: Y:100.0% UV:100.0%
x264 [info]: kb/s:28569.80

encoded 2 frames, 3.83 fps, 28639.30 kb/s

bballdave025@MY-COMPUTER /cygdrive/c/Users/bballdave025/Desktop
$

We got an output video. It barely looks like a video, but I was expecting quality to go way down.

Since I want to use this for a Cygwin build of FFMPEG, I think I'm on the right track.