Apache reverse proxy to Node - Connection refused: AH00957

Question

I am trying to get a brand new cloud based server working with a default version of 20.04 server ubuntu working with apache and node. The node server appears to be running without issues reporting 4006 port is open. However I believe my apache config is not. The request will hang for a very very long time. No errors are displayed in the node terminal. So the fault must lie in my apache config seeing as we are getting the below apache errors and no JS errors.

Request error after some time

502 proxy error

Apache Error Log

[Sun Oct 17 20:58:56.608793 2021] [proxy:error] [pid 1596878] (111)Connection refused: AH00957: HTTP: attempt to connect to [::1]:4006 (localhost) failed
[Sun Oct 17 20:58:56.608909 2021] [proxy_http:error] [pid 1596878] [client 207.46.13.93:27392] AH01114: HTTP: failed to make connection to backend: localhost

vhost

<VirtualHost IP_ADDRESS:80>
    ServerName api.aDomain.com
    Redirect permanent / https://api.aDomain.com/
</VirtualHost>
<IfModule mod_ssl.c>
    <VirtualHost IP_ADDRESS:443>
        ServerName api.aDomain.com

        ProxyRequests on
        LoadModule proxy_module /usr/lib/apache2/modules/mod_proxy.so
        LoadModule proxy_http_module /usr/lib/apache2/modules/mod_proxy_http.so

        ProxyPass / http://localhost:4006/
        ProxyPassReverse / http://localhost:4006/
    

        #certificates SSL
        SSLEngine on
        SSLCACertificateFile /etc/ssl/api.aDomain.com/apimini.ca
        SSLCertificateFile /etc/ssl/api.aDomain.com/apimini.crt
        SSLCertificateKeyFile /etc/ssl/api.aDomain.com/apimini.key

        ErrorLog ${APACHE_LOG_DIR}/error_api.aDomain.com.log
        CustomLog ${APACHE_LOG_DIR}/access_api.aDomain.com.log combined

    </VirtualHost>
</IfModule>

terminal output

[nodemon] 1.19.4
[nodemon] to restart at any time, enter `rs`
[nodemon] watching dir(s): *.*
[nodemon] watching extensions: js,mjs,json
[nodemon] starting `babel-node -r dotenv/config --inspect=9229 index.js`
Debugger listening on ws://127.0.0.1:9229/c1fcf271-aea8-47ff-910e-fe5a91fce6d2
For help, see: https://nodejs.org/en/docs/inspector
Browserslist: caniuse-lite is outdated. Please run next command `npm update`
🚀 Server ready at http://localhost:4006

Node server

import cors from 'cors'

import scrape from './src/api/routes/scrape'

const express = require('express')
const { ApolloServer, gql } = require('apollo-server-express')
const { postgraphile } = require('postgraphile')
const ConnectionFilterPlugin = require('postgraphile-plugin-connection-filter')

const dbHost = process.env.DB_HOST
const dbPort = process.env.DB_PORT
const dbName = process.env.DB_NAME
const dbUser = process.env.DB_USER
const dbPwd = process.env.DB_PWD
const dbUrl = dbPwd
  ? `postgres://${dbUser}:${dbPwd}@${dbHost}:${dbPort}/${dbName}`
  : `postgres://${dbHost}:${dbPort}/${dbName}`

var corsOptions = {
  origin: '*',
  optionsSuccessStatus: 200, // some legacy browsers (IE11, various SmartTVs) choke on 204
}

async function main() {
  // Construct a schema, using GraphQL schema language
  const typeDefs = gql`
    type Query {
      hello: String
    }
  `

  // Provide resolver functions for your schema fields
  const resolvers = {
    Query: {
      hello: () => 'Hello world!',
    },
  }

  const server = new ApolloServer({ typeDefs, resolvers })

  const app = express()
  app.use(cors(corsOptions))
  app.use(
    postgraphile(process.env.DATABASE_URL || dbUrl, 'public', {
      appendPlugins: [ConnectionFilterPlugin],
      watchPg: true,
      graphiql: true,
      enhanceGraphiql: true,
    })
  )
  server.applyMiddleware({ app })

  //Scraping Tools
  scrape(app)

  const port = 4006
  await app.listen({ port })
  console.log(`🚀 Server ready at http://localhost:${port}`)
}

main().catch(e => {
  console.error(e)
  process.exit(1)
})

Apache Mods Enabled

/etc/apache2/mods-enabled/proxy.conf /etc/apache2/mods-enabled/proxy.load /etc/apache2/mods-enabled/proxy_http.load

Updated Error Logs

[Thu Oct 21 10:59:22.560608 2021] [proxy_http:error] [pid 10273] (70007)The timeout specified has expired: [client 93.115.195.232:8963] AH01102: error reading status line from remote server 127.0.0.1:4006, referer: https://miniatureawards.com/
[Thu Oct 21 10:59:22.560691 2021] [proxy:error] [pid 10273] [client 93.115.195.232:8963] AH00898: Error reading from remote server returned by /graphql, referer: https://miniatureawards.com/

Answer 1

I cannot exactly predict what exactly happen it could be NodeJS app crushed and no longer running or there are misconfiguration Apache files. But I strongly believe this scenario will be solved from doing things back from the top.

This step would go through updating unbuntu packages, installing needed application, configuring Apache files and setting up reverse proxy with NodeJS and Apache.

Just don't touch your NodeJS files and other code related application and they will be safe. You may also backup just to make sure. Other running application on that ubuntu server example database application like MySQL as will be just fine and still be running.

---

1. First we need to update ubuntu packages and install Apache, and NodeJS

$ sudo apt update
$ sudo apt install apache2 npm

---

2. Run this command to enable us to use Apache as a reverse proxy server

sudo a2enmod proxy proxy_http rewrite headers expires

---

3. Create an Apache virtual host file.

This command would will let you use ubuntu terminal as your text editor follow the guide and prompt from the terminal to write.

NOTE:

Change the "yourSite.com" with the domain of your site. It isn't really important should be the name of the file. But I think its better to name it after your site domain so you can recognize it.

$ sudo nano /etc/apache2/sites-available/yourSite.com.conf

---

4. Use the nano editor is to write your Apache config file for your site.

Notice: This part is critical so please pay attention

Change your ServerName and ServerAlias with your site domain name.

The ProxyPass and the ProxyPassReverse this has two parameters.

The first one is a back-slash "/" This an absolute path where your NodeJS should be located and since its single back-slash that means its your home directory.

The second one is the url "http://127.0.0.1:3000/" of your NodeJS application. Pay attentions to its PORT "3000" you may need to replaced it with the PORT you use in your NodeJS app.

<VirtualHost *:80>
    ServerName example.com // replace this with site domain name without www at the beginning
    ServerAlias www.example.com // replace this with site domain name beginning with www. + yourdomainname + .com

    ProxyRequests Off
    ProxyPreserveHost On
    ProxyVia Full

    <Proxy *>
        Require all granted
    </Proxy>

    ProxyPass / http://127.0.0.1:3000/
    ProxyPassReverse / http://127.0.0.1:30000/
</VirtualHost>

---

5. disable the default Apache site and enable the new one.

$ sudo a2dissite 000-default
$ sudo a2ensite example.com.conf

---

6. Restart your Apache Server to apply the changes

sudo systemctl restart apache2

We could be ready at this point as we done setting up Apache as a reverse proxy, But we also need to install the npm package of your project and then run your NodeJS application.

---

7. The rest of the step is all related to NodeJS deployment. You may be already know this steps.

// install npm packages
npm install

// for a better experience using NodeJS in production install pm2 globally
npm install -g pm2

// Then run your NodeJS application using pm2 command

pm2 start // you should be at root of your NodeJS project folder when running this command

// run this another pm2 command to make sure your NodeJS app will re-run when it encounter downtime.

$ pm2 save
$ pm2 startup

Your Apache and NodeJS server is up and running now

Try to access your site by typing entering your site domain name in the browsers address bar

e.g http://yourSite.com

Answer 2

In major situations this is caused by selinux (when you have RHEL or CentOS):

# setsebool -P httpd_can_network_connect 1

link: https://unix.stackexchange.com/questions/8854/how-do-i-configure-selinux-to-allow-outbound-connections-from-a-cgi-script

Also check:

• connectivity between the machines
• back-end port is open
• Use static IP-address (IPv4) or use host-name that are in you /etc/hosts file

Answer 3

If you use a docker for your node server, then it might be set up incorrectly

Answer 4

I'm not an expert on this topic, but I have a similar setup; I use socket.io to serve WebSockets...

From your posts it seems you don't need to proxy WebSockets as well, the one shown in your logs seems to be only for debugging purposes (please correct me if I'm wrong).

Following the core of my Apache configuration:

    RewriteEngine On

    RewriteCond %{REQUEST_URI}  ^/socket.io            [NC]
    RewriteCond %{QUERY_STRING} transport=websocket    [NC]
    RewriteRule /(.*)           ws://127.0.0.1:4006/$1 [P,L]

    <Location />
        ProxyPass        http://127.0.0.1:4006/ retry=2
        ProxyPassReverse http://127.0.0.1:4006/
    </Location>

Another couple of suggestions.

Warning

Do not enable proxying with ProxyRequests until you have secured your server. Open proxy servers are dangerous both to your network and to the Internet at large.

Source: https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxyrequests

I don't know which is the IPV6 setup on your host, you could try to use 127.0.0.1 rather than localhost in you Apache configuration to try forcing Apache to use IPV4.