Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

GAE is not serving my robots.txt

I have the following defined in my app.yaml:

handlers:
- url: /favicon.ico
  static_files: img/favicon.ico
  upload: noop

- url: /apple-touch-icon.png
  static_files: img/apple-touch-icon.png
  upload: noop

- url: /images
  static_dir: img

- url: /robots.txt
  static_files: media/robots.txt
  upload: noop

- url: /humans.txt
  static_files: media/humans.txt
  upload: noop

There are other mappings after the declaration for /humans.txt but I'll remove them for brevity.

The noop directory is an empty directory.

However my browser gives me a 404 when I try to access these urls:

  1. http://myapp.appspot.com/humans.txt
  2. http://myapp.appspot.com/robots.txt

Why ?

like image 969
Frankie Ribery Avatar asked Apr 10 '11 07:04

Frankie Ribery


People also ask

How do I fix robots.txt error?

Luckily, there's a simple fix for this error. All you have to do is update your robots. txt file (example.com/robots.txt) and allow Googlebot (and others) to crawl your pages. You can test these changes using the Robots.

Can robots.txt be ignored?

You are able to choose whether you would like to ignore robots exclusions for all hosts within a specific seed (seed level rules) or all instances of a specific host within a collection (collection level rules).


1 Answers

Since you're using static files, upload should match the static_files location:

- url: /robots.txt
    static_files: media/robots.txt
    upload: media/robots.txt

- url: /humans.txt
    static_files: media/humans.txt
    upload: media/humans.txt
like image 143
hyperslug Avatar answered Nov 04 '22 16:11

hyperslug