How to ignore changes in some folders when developing locally?

Question

I'm using the Go SDK for the Google App Engine (it uses some parts of the Python SDK and calls into dev_appserver.py when running goapp serve). By default, it seems to watch all files and folders for changes. I have a crapload of bower dependencies in the static folder, and the SDK complains that it can't watch that many files.

How to set the dev server to ignore changes in a folder?

Edit. Reading devappserver2/watcher_common.py, it seems that it ignores directories starting with .. I suppose in the worst case scenario, I could prefix folders with ., but it's a hack. There should be a configuration option, and I can't seem to find it.

Answer 1

Using:

Google Cloud SDK 197.0.0
app-engine-python 1.9.68
app-engine-python-extras 1.9.63
bq 2.0.31
core 2018.04.06
gsutil 4.30

From the App Engine Docs, you can pass an argument, --watcher_ignore_re, to dev_appserver.py when running from the command line.

Example

dev_appserver.py app.yaml --watcher_ignore_re="(.*\.data|.*\.vscode)"

This will ignore changes to the .data and .vscode directories, relative to the directory from which the command is run (CWD).

---

Edit 7th June 2018.

Using:

Google Cloud SDK 204.0.0
app-engine-python 1.9.70
app-engine-python-extras 1.9.69
bq 2.0.34
core 2018.06.04
gsutil 4.31

The watcher_ignore_re seems to not be working anymore.

Rather, what worked for me is to add a skip_files directive in the app.yaml file as stated here in the docs. The dev_appserver.py command will respect the definitions in this section.

Example

Inside app.yaml of the project:

...
...
skip_files:
# default from GAE
- ^(.*/)?#.*#$
- ^(.*/)?.*~$
- ^(.*/)?.*\.py[co]$
- ^(.*/)?.*/RCS/.*$
- ^(.*/)?\..*$

# custom entries

# dev-related files
- ptvsd
- pydev_startup.py
- .pylintrc
- .data
- .vscode

# version control files
- .git
- .gitignore
- .gitattributes

# non-application files
- README.md
- ^(.*/)?.*\.mwb(.bak)?$ # workbench models

Now, run project with command:

dev_appserver.py app.yaml

Answer 2

You are correct that there is currently no command-line or configuration option to specify files/directories to be ignored by the watcher.

. (dot) hack

As you've discovered, files and directories beginning with . are ignored. This is a fairly standard convention for many *nix systems. It is not really scalable however with python projects with many local dependencies.

Modifying the watcher file

You can modify this file but this too is unfavorable as it makes for very brittle changes and is prone to breaking if updates overwrite the file.

Barring the above workarounds, there's no way to achieve this. I've also not seen any mention of this being implemented in the App Engine release notes. There is however an open feature request on the public issue tracker to have the watcher ignore files specified by the skip_files yaml directive. Feel free to star that issue to get updates regarding its progress.

EDIT: Jan 4, 2017

NPM 3

For users of NPM encountering this limitation, you may find it helpful to use NPM v3 as it resolves dependencies differently.

npm3 attempts this [mitigate deep trees and redundancy] by installing some secondary dependencies (dependencies of dependencies) in a flat way, in the same directory as the primary dependency that requires it.

This can be very effective for Node users encountering the file watching limitation.

Frankenserver

Khan Academy has developed a fork of the App Engine devserver called Frankenserver. While I've not explored this option myself, there seems to be a fair amount of support and recent updates to it. From it's readme:

frankenserver's biggest advantage over the vanilla SDK is in how it watches the files in your app for changes. It does this much more efficiently by 1) using a native FSEvents-based file watcher on Mac OS X and 2) respecting the skip_files directive in your app.yaml.

Though this is not an official Google solution, it may be a worthwhile workaround for time being if this limitation affects you direly.