Nuxt Sites not getting crawled

Question

I have made a website using NUXT that needs SEO

When I use www.xml-sitemaps.com website to see if it can find all my pages, it only finds the home page, and none of the other routes. When I try other NUXT demo websites it finds them all.

My robots.txt file looks like:

User-agent: *
Disallow: /profile/
Sitemap: https://www.example.com/sitemap.xml

I am using @nuxtjs/sitemap to generate the sitemap.xml that ends up looking something like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
<url> <loc>https://www.example.com/about</loc> </url>
<url> <loc>https://www.example.com/</loc> </url>
</urlset>

And if this helps, my nuxt.config.js looks like:

module.exports = {
  /*
  ** Headers of the page
  */
  head: {
    title: 'Title',
    meta: [
      { charset: 'utf-8' },
      { name: 'viewport', content: 'width=device-width, initial-scale=1' },
      { hid: 'description', name: 'description', content: 'Title' }
    ],
    link: [
      { rel: 'icon', type: 'image/x-icon', href: '/favicon.ico' }
    ]
  },
  mode: 'spa',
  loading: { color: '#3B8070' },
  build: {
    /*
    ** Run ESLint on save
    */
    extend (config, { isDev, isClient }) {
      if (isDev && isClient) {
        config.module.rules.push({
          enforce: 'pre',
          test: /\.(js|vue)$/,
          loader: 'eslint-loader',
          exclude: /(node_modules)/
        })
      }
    }
  },
  css: [
    '~/assets/main.css'
  ],
  modules: [
    '@nuxtjs/pwa',
    [
      '@nuxtjs/sitemap', {
        generate: true,
        hostname: 'https://www.example.com',
        exclude: [
          '/profile'
        ]
      }
    ]
  ],
  plugins: [
    '~/plugins/uikit.js',
    '~/plugins/fireauth.js'
  ],
  manifest: {
    name: 'Title',
    lang: 'en'
  },
  router: {
    middleware: 'router-auth'
  },
  vendor: [
    'firebase',
    'uikit'
  ]
}

Answer 1

I'm the creator of the nuxt sitemap module.

Your sitemap-module configuration is set in the wrong section.

Please, update your nuxt.config.js:

modules: ['@nuxtjs/pwa', '@nuxtjs/sitemap'],
sitemap: {
  generate: true,
  hostname: 'https://www.example.com',
  exclude: [
    '/profile'
  ]
},
plugins: [

Then run npm run generate.

Finally check your generated sitemap.xml in the \dist\ folder.

(If you have an other issue or question, you may open an issue on github project: https://github.com/nuxt-community/sitemap-module/issues)

Answer 2

It's important to understand what's going on with different Nuxt.js modes. Read the explanation about server side rendering in the Nuxt.js Guide, where they explain the difference between the three modes the framework can be configured to work in:

• Universal (with server side rendering, so that when any page is rendered, that page will be served with all HTML rendered (SEO and crawler friendly mode)
• SPA (Single Page Application) which will serve up the HTML skeleton together with css and javascript bundles, which will only be unbundled to create the initial DOM in the browser. Cool for intranet apps, bad for SEO.
  • Static generation of all pages (pre-rendering) so that the site can be served up in any shared hosting as simple HTML.

Once the concepts are clear, you can try changing the "mode" property in your Nuxt.js configuration file from "SPA" to "Universal", together with the other suggestion regarding xml sitemap configuration in the same nuxt.config.js file.

Additionally, you can try out and learn about different configurations by either using:

• The Nuxt.js starter template discussed in the Installation Guide.
• Something like Create Nuxt App that, once installed via npm install -g create-nuxt-app allows you to see how many different configurations are automatically set up for you.