Kubernetes MySql image persistent volume is non empty during init

Question

I am working through the persistent disks tutorial found here while also creating it as a StatefulSet instead of a deployment.

When I run the yaml file into GKE the database fails to start, looking at the logs it has the following error.

[ERROR] --initialize specified but the data directory has files in it. Aborting.

Is it possible to inspect the volume created to see what is in the directory? Otherwise, what am I doing wrong that is causing the disk to be non empty?

Thanks

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: datalayer-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
---
apiVersion: v1
kind: Service
metadata:
  name: datalayer-svc
  labels:
    app: myapplication
spec:
  ports:
  - port: 80
    name: dbadmin
  clusterIP: None
  selector:
    app: database
---
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  name: datalayer
spec:
  selector:
    matchLabels:
      app: myapplication
  serviceName: "datalayer-svc"
  replicas: 1
  template:
    metadata:
      labels:
        app: myapplication
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: database
        image: mysql:5.7.22
        env:
        - name: "MYSQL_ROOT_PASSWORD"
          valueFrom:
            secretKeyRef:
              name: mysql-root-password
              key: password
        - name: "MYSQL_DATABASE"
          value: "appdatabase"
        - name: "MYSQL_USER"
          value: "app_user"
        - name: "MYSQL_PASSWORD"
          value: "app_password"
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: datalayer-pv
          mountPath: /var/lib/mysql
      volumes:
      - name: datalayer-pv
        persistentVolumeClaim:
          claimName: datalayer-pvc

Answer 1

This issue could be caused by the lost+found directory on the filesystem of the PersistentVolume.

I was able to verify this by adding a k8s.gcr.io/busybox container (in PVC set accessModes: [ReadWriteMany], OR comment out the database container):

- name: init
  image: "k8s.gcr.io/busybox"
  command: ["/bin/sh","-c","ls -l /var/lib/mysql"]
  volumeMounts:
  - name: database
    mountPath: "/var/lib/mysql"

There are a few potential workarounds...

Most preferable is to use a subPath on the volumeMounts object. This uses a subdirectory of the PersistentVolume, which should be empty at creation time, instead of the volume root:

volumeMounts:
- name: database
  mountPath: "/var/lib/mysql"
  subPath: mysql

Less preferable workarounds include:

• Use a one-time container to rm -rf /var/lib/mysql/lost+found (not a great solution, because the directory is managed by the filesystem and is likely to re-appear)
• Use mysql:5 image, and add args: ["--ignore-db-dir=lost+found"] to the container (this option was removed in mysql 8)
• Use mariadb image instead of mysql

More details might be available at docker-library/mysql issues: #69 and #186

Answer 2

You would usually see if your volumes were mounted with:

kubectl get pods # gets you all the pods on the default namespace
# and
kubectl describe pod <pod-created-by-your-statefulset>

Then you can these commands to check on your PVs and PVCs

kubectl get pv  # gets all the PVs on the default namespace
kubectl get pvc # same for PVCs
kubectl describe pv <pv-name>
kubectl describe pvc <pvc-name>

Then you can to the GCP console on disks and see if your disks got created: