How do I force a data refresh in GatsbyJS when I'm using a Source plugin that calls a REST API without restarting Gatsby?

Question

Is it possible and how can I force GatsbyJS to reload external data? Either by disabling caching or limiting TTL in the cache?

I have a GatsbyJS running gatsby develop and it will reload code changes immediately, but it's hooked up to an external REST API via fetch using a source plugin. The data in GraphQL doesn't update once the data is loaded at build time / first load.

I'm using staticQuery(...) and non-static queries, both behave the same. I'd expect static to not update, but the normal query fires once and is forever cached.

From gatsby-site/gatsby-node.js

exports.createPages = async ({graphql, actions}) => {
    const pages = await graphql(`
         ...
    `);

Questions I've been unable to find useful answers to...

• Can I make it not cache at all?
• Can I limit how long it keeps the data?
• Is it in-memory, or if there's a folder on the disk could I delete it?

All help and ideas valuable. I couldn't find any other questions related, and even the tutorials on gatsbyjs.org indicate a server restart on data changes.

Answer 1

This works but probably doesn't scale.

1. Start Gatsby with ENABLE_GATSBY_REFRESH_ENDPOINT=true gatsby develop or put the setting in the .env file
2. Use curl -X POST http://localhost:8000/__refresh to invoke a full refresh.
3. Alternatively, make it repeatedly call the hook to trigger a refresh periodically with watch -n5 "curl -X POST http://localhost:8000/__refresh" (every 5 seconds in this case, tune as needed) or put it in a crontab for unattended refresh with longer intervals.

Changes to source data will now be reflected, at the expense of hammering the source. I only needed this for development, and might not be a great idea for production use.

Answer 2

Proposal of solution (Gatsby automated rebuild)
The following example uses Strapi as a data source.

PROS and CONS of the solution are in the end of post.

1. Create webhook in Strapi (Settings>Webhooks>Add new webhook), which will be fired on create, edit or delete entry event. Set url with port eg. http://newhost:1111/webhook .
2. Dockerize your Strapi and Gatsby app. Gatsby's dockerfile does a trick as it is calling separated script which make build and serve.
3. Create node application working on 1111 port, which will be able to receive Strapi POST request on localhost:1111/webhook path. Then as a response node will run docker restart script, which in a couple of seconds refresh our Gatsby blog.

Cautions and code.
There are few breakpoints which you have to do carefully in order to make your services work well. First of all install locally your Strapi and Gatsby projects - nice tutorial is here. Don't forget to add all necessary plugins like graphql plugin in Strapi. Also make sure that services are connected - Gatsby is able to fetch data from Strapi. If everything seems fine then let's save it into docker image.

Strapi Dockerfile

// Strapi Dockerfile
FROM strapi/base

WORKDIR /project

COPY ./package.json ./
COPY ./yarn.lock ./

RUN yarn install

COPY . .

ENV NODE_ENV production

RUN yarn build

EXPOSE 1337

CMD ["yarn", "start"]

Gatsby Dockerfile

//Gatsby Dockerfile
FROM node:12.2.0-alpine

WORKDIR /project

RUN npm install -g gatsby-cli

COPY . .
RUN yarn install

EXPOSE 3000

RUN chmod +x gatsby.sh
CMD sh gatsby.sh

gatsby.sh

//gatsby.sh
gatsby clean
gatsby build
gatsby serve -p 3000 --host 0.0.0.0

Here we have little magic. Every RUN command are called during docker image build but CMD will be called only on image start. So when we restart gatsby image, new build is created. Why we need fresh build? Each new Gatsby build fetch fresh data from Strapi.

Eg. command for creating docker Strapi image: docker build --no-cache -t my-strapi:v1 . . You call it inside Strapi project, also here must be a Dockerfile. After building two separated images, one for Gatsby and one for Strapi you can list them with docker images.

Time to run images
docker run --add-host newhost:[machine ip] -p 1337:1337 strapi:v1
docker run --env API_URL="http://[machne ip]:1337" -p 3000:3000 gatsby:v1

• newhost is an alias url used in Strapi's Webhook. Remainder: http://newhost:1111/webhook is a node app address.
• API_URL is an environmental variable used by Gatsby to connect with Strapi. (checkout your gatsby-config.js, I assume that your gatsby-source-strapi options have apiURL: process.env.API_URL).

Node.js app
On POST request call script which restart the docker Gatsby container.

// app.js
...
router.post('/webhook', function (req, res) {
  shell.exec('./docker-restarter.sh');
  res.send('POST request finished')
});
...
app.listen(1111);

docker-restarter.sh

//docker-restarter.sh
echo Calling docker ps to find container id
gatsby=$(docker ps -aqf "name=projects_gatsby_1")
echo Calling docker restart
docker restart $gatsby

"name=projects_gatsby_1" switch projects_gatsby_1 to the name of your Gatsby container

Start node with node app.js

Now when everything is running, open two tabs: one with Gatsby site and second with Strapi. Add new post to Strapi. Checkout logs in Node.js console. When restart is done, make hard refresh on Gatsby's tab (for Chrome it is CTRL+SHIFT+R). Now you can see that new post is displaying in your Gatsby site.

There are pros and cons to this solution:

PROS:

• when blog post is added, deleted or edited in Strapi, Gatsby will refresh and show new data
• refresh process is fully automated

CONS:

• docker and node is in use
• each refresh demands restarting Gatsby container, because we are automatically creating new build (author of the question tried to avoid it)