Compiling Httrack on MAC OS X

Question

I'm trying to compile httrack on my MAC. ./configure is successful. But while compiling the package i'm getting following error, and not able to resolve it.

In file included from htscore.c:40:
In file included from ./htscore.h:81:
In file included from ./htslib.h:67:
./htsbasenet.h:76:10: fatal error: 'openssl/ssl.h' file not found
#include <openssl/ssl.h>
      ^
2 warnings and 1 error generated.
make[2]: *** [libhttrack_la-htscore.lo] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

Also tried this solution, but no luck https://serverfault.com/questions/283271/how-to-get-httrack-to-work-with-ssl-on-mac-os-x-libssl-so-not-found

Openssl is located at /usr/include/openssl

Answer 1

If you can live without HTTPS then

./configure --enable-https=no

should be enough.

Answer 2

It sounds like you don't have OpenSSL on path or Httrack does not pick it up. It also appears Httrack does not have a configure option to specify an OpenSSL directory.

It also appears Httrack rejects anything less than OpenSSL 1.0.1g (Heartbleed). OS X provides OpenSSL 0.9.8 by default, so you should download and install the latest version of OpenSSL.

Once you have an updated version of OpenSSL, export the needed paths through CFLAGS and LDFLAGS. Httrack picks up LDLIBS on its own.

$ export CFLAGS="-I/usr/local/ssl/macosx-x64/include"
$ export LDFLAGS="-L/usr/local/ssl/macosx-x64/lib"

$ ./configure 
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
...
config.status: executing depfiles commands
config.status: executing libtool commands

$ cat ./config.log | grep -i openssl
| #define HTS_USEOPENSSL 1
| #define HTS_USEOPENSSL 1
OPENSSL_LIBS='-lcrypto -lssl'
#define HTS_USEOPENSSL 1

The above is from OS X 10.8.5 and Htrack 3.48.21

---

You can verify OpenSSL was linked with:

$ find . -name *.dylib
./libtest/.libs/libbaselinks.dylib
...
./src/.libs/libhtsjava.dylib
./src/.libs/libhttrack.dylib

And then:

$ otool -L ./src/.libs/libhttrack.dylib | grep -i ssl
    /usr/local/ssl/macosx-x64/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/local/ssl/macosx-x64/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)

---

You should be able to test the build before the install with a make check:

$ make
...

$ make check
...
PASS: 00_runnable.test
PASS: 01_engine-charset.test
PASS: 01_engine-entities.test
md5 selftest succeeded
hashtable summary: size=262144 (lg2=18) used=72619 stash-size=0 pool-size=6693195 pool-capacity=8388608 pool-used=6480537 writes=6600000 (new=175000) moved=44471 stashed=4 max-stash-size=1 avg-moved=0.25412 rehash=14 pool-compact=15 pool-realloc=30 memory=14680688
all hashtable tests were successful!
PASS: 01_engine-hashtable.test
PASS: 01_engine-idna.test
PASS: 01_engine-simplify.test
online tests are disabled
skipping online unit tests
SKIP: 10_crawl-simple.test
online tests are disabled
skipping online unit tests
SKIP: 11_crawl-cookies.test
online tests are disabled
skipping online unit tests
SKIP: 11_crawl-idna.test
online tests are disabled
skipping online unit tests
SKIP: 11_crawl-international.test
online tests are disabled
skipping online unit tests
SKIP: 11_crawl-longurl.test
online tests are disabled
skipping online unit tests
SKIP: 11_crawl-parsing.test
online tests are disabled
skipping online unit tests
SKIP: 12_crawl_https.test
======================
All 6 tests passed
(7 tests were not run)
======================
make[2]: Leaving directory '/Users/.../httrack-3.48.21/tests'
make[1]: Leaving directory '/Users/.../httrack-3.48.21/tests'
make[1]: Entering directory '/Users/.../httrack-3.48.21'
make[1]: Nothing to be done for 'check-am'.
make[1]: Leaving directory '/Users/.../httrack-3.48.21'

I don't know why the online tests were not performed since they are enabled by default:

$ ./configure --help | grep -i online
    --enable-online-unit-tests=[yes/no/auto]
                                    Enable online-unit-tests [default=yes]

Also see Issue 92: 'make check' and "uname: illegal option -- o" on OS X.

---

Their use of OpenSSL looks like it has room for improvement... You might consider opening src/htslib.c, scrolling down to line 5166:

// OpenSSL_add_all_algorithms();
openssl_ctx = SSL_CTX_new(SSLv23_client_method());
if (!openssl_ctx) {
  fprintf(stderr,
          "fatal: unable to initialize TLS: SSL_CTX_new(SSLv23_client_method)\n");
  abortLog("unable to initialize TLS: SSL_CTX_new(SSLv23_client_method)");
  assertf("unable to initialize TLS" == NULL);
}

And changing it to something like:

// OpenSSL_add_all_algorithms();
openssl_ctx = SSL_CTX_new(SSLv23_client_method());
if (!openssl_ctx) {
  fprintf(stderr, "fatal: unable to initialize TLS: SSL_CTX_new(SSLv23_client_method)\n");
  abortLog("unable to initialize TLS: SSL_CTX_new(SSLv23_client_method)");
  assertf("unable to initialize TLS" == NULL);
}

const long flags = SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3 | SSL_OP_NO_COMPRESSION;
SSL_CTX_set_options(openssl_ctx, flags);

const char PREFERRED_CIPHERS[] = "HIGH:!aNULL:!kRSA:!PSK:!SRP:!MD5:!RC4";
long res = SSL_CTX_set_cipher_list(openssl_ctx, PREFERRED_CIPHERS);
if (res != 1) {
  fprintf(stderr, "fatal: unable to initialize TLS: SSL_CTX_set_cipher_list(PREFERRED_CIPHERS)\n");
  abortLog("unable to initialize TLS: SSL_CTX_set_cipher_list(PREFERRED_CIPHERS)");
  assertf("unable to initialize TLS" == NULL);
}

There are some other things you can do. See SSL/TLS Client on the OpenSSL wiki.