Playwright E2E Tests Passing Locally but Failing in GitHub Actions with Next.js 14 App Router, TypeScript

Question

I’m running into an issue where my Playwright end-to-end tests work perfectly fine on my local machine but fail consistently when executed in GitHub Actions. My setup involves a Next.js 14 app using the App Router, TypeScript, Playwright for testing, and GitHub Actions for CI.

Local Environment:

• Node.js: v20.x
• Next.js: 14.x
• Playwright: 1.x
• OS: macOS/Linux

GitHub Actions Environment:

• ubuntu-latest
• Node.js: 20.x

Code Overview:

• setup.ts (Sets up authentication state for tests):

import { test, expect } from "@playwright/test";
import { mkdirSync } from "fs";
import { dirname } from "path";

const authFile = "playwright/.auth/user.json";

test("test", async ({ page }) => {
  mkdirSync(dirname(authFile), { recursive: true });

  await page.goto("http://localhost:3000/api/auth/signin?csrf=true");
  await page.getByRole("button", { name: "Sign in with Azure Active" }).click();
  await page.getByPlaceholder("Email, phone, or Skype").click();
  await page
    .getByPlaceholder("Email, phone, or Skype")
    .fill(process.env.PLAYWRIGHT_EMAIL ?? "");
  await page.getByRole("button", { name: "Next" }).click();
  await page.getByPlaceholder("Password").click();
  await page
    .getByPlaceholder("Password")
    .fill(process.env.PLAYWRIGHT_PASSWORD ?? "");
  await page.getByRole("button", { name: "Sign in" }).click();
  try {
    const yesButton = page.getByRole("button", { name: "Yes" });
    await yesButton.waitFor({ state: "visible", timeout: 10000 });
    await yesButton.click();
  } catch (error) {
    console.log(
      "The 'Yes' button was not found or not clickable within the time limit."
    );
  }
  await page.goto("http://localhost:3000/home");

  await page.context().storageState({ path: authFile });
});

• redirect.spec.ts (Tests routing logic):

import { test, expect } from "@playwright/test";

test("Authenticated root route redirects to home page", async ({ page }) => {
  await page.goto("http://localhost:3000/home");
  await expect(page).toHaveURL("http://localhost:3000/home");
  await page.goto("http://localhost:3000/");
  await expect(page).toHaveURL("http://localhost:3000/home");
});

test("Unauthenticated access redirects to login page for root and home routes", async ({ page }) => {
  await page.context().clearCookies();
  await page.goto("http://localhost:3000/");
  await expect(page).toHaveURL("http://localhost:3000/login");
  await page.goto("http://localhost:3000/home");
  await expect(page).toHaveURL("http://localhost:3000/login");
});

• playwright.config.ts (Playwright configuration):

import { defineConfig, devices } from "@playwright/test";
import dotenv from "dotenv";
import path from "path";

dotenv.config({ path: path.resolve(__dirname, ".env.local") });

export default defineConfig({
  testDir: "./e2e-tests",
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,
  reporter: "html",
  use: {
    trace: "on-first-retry",
  },
  projects: [
    { name: "setup", testMatch: "setup.ts" },
    {
      name: "chromium",
      use: {
        ...devices["Desktop Chrome"],
        storageState: "playwright/.auth/user.json",
      },
      dependencies: ["setup"],
    },
    {
      name: "firefox",
      use: {
        ...devices["Desktop Firefox"],
        storageState: "playwright/.auth/user.json",
      },
      dependencies: ["setup"],
    },
    {
      name: "webkit",
      use: {
        ...devices["Desktop Safari"],
        storageState: "playwright/.auth/user.json",
      },
      dependencies: ["setup"],
    },
  ],
  webServer: {
    command: "npm run dev",
    url: "http://127.0.0.1:3000",
    reuseExistingServer: !process.env.CI,
  },
});

• GitHub Actions Workflow (playwright.yml):

name: Playwright Tests

on:
  pull_request:

jobs:
  end-to-end-test:
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [20.x]

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}

      - name: Install dependencies
        run: npm install

      - name: Install Playwright browsers
        run: npx playwright install --with-deps

      - name: Run Playwright tests
        env:
          PLAYWRIGHT_EMAIL: ${{ secrets.PLAYWRIGHT_EMAIL }}
          PLAYWRIGHT_PASSWORD: ${{ secrets.PLAYWRIGHT_PASSWORD }}
          AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID}}
          AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET}}
          AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID}}
          NEXTAUTH_SECRET: ${{ secrets.NEXTAUTH_SECRET}}
          NEXTAUTH_URL: ${{ secrets.NEXTAUTH_URL}}
        run: npm run test:e2e
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 30

Issue:

The tests fail on GitHub Actions with the following error:

Error: Timed out 5000ms waiting for expect(locator).toHaveURL(expected)
Expected string: "***/home"
Received string: "***/login"

This suggests that the authentication step might not be working correctly in the CI environment. However, the same setup works perfectly fine locally.

Troubleshooting Steps Taken:

• Verified environment variables are correctly set in GitHub Actions.
• Ensured all dependencies and browser versions are up-to-date.
• Tried increasing timeouts but still encounter the same issue.

Question:

Why are my Playwright tests passing locally but failing in GitHub Actions? Is there something I’m missing in terms of configuration or environment setup that could cause this discrepancy? Any help or pointers would be greatly appreciated.

---

Answer 1

You have set trace: "on-first-retry", and retries: process.env.CI ? 2 : 0,. The easiest way to debug your problem is to download an HTML report saved in the last step in your CI.

If you don't know how to download GitHub artifacts, read this article https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-workflow-runs/downloading-workflow-artifacts

Then open the HTML report, find the failing test, and switch tabs to "Retry #1". In trace viewer, you will see what exactly happens. Here are articles about trace viewer https://playwright.dev/docs/trace-viewer

Probably your storage state file was created without session cookies. You may have to wait sometime after login eg. wait until you see the profile picture. Locally you can have a faster machine and it creates.