Avoid multiple cron jobs running for one cron execution point in Kubernetes

Question

EDIT: Question is solved, it was my mistake, i simply used the wrong cron settings. I assumed "* 2 * * *" would only run once per day at 2, but in fact it runs every minute past the hour 2. So Kubernetes behaves correctly.

I keep having multiple jobs running at one cron execution point. But it seems only if those jobs have a very short runtime. Any idea why this happens and how I can prevent it? I use concurrencyPolicy: Forbid, backoffLimit: 0 and restartPolicy: Never.

Example for a cron job that is supposed to run once per day, but runs multiple times just after its scheduled run time:

job-1554346620                   1/1           11s        4h42m   
job-1554346680                   1/1           11s        4h41m                     
job-1554346740                   1/1           10s        4h40m 

Relevant config:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: job
spec:
  schedule: "* 2 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: job
              image: job_image:latest
              command: ["rake", "run_job"]
          restartPolicy: Never
          imagePullSecrets:
            - name: regcred
      backoffLimit: 0

Answer 1

The most common problem of running CronJobs on k8s is:

spawning to many pods which consume all cluster resources

It is very important to set proper CronJob limitations

If you are not sure what you need - just take this example as a template:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: my-first-conjob
  namespace: devenv-admitriev
spec:
  schedule: "*/10 * * * *" # MM HH DD MM WKD -- Minutes, Hour, Day, Month, Weekday (eg. Sun, Mon)
  successfulJobsHistoryLimit: 3 # how many completed jobs should be kept
  failedJobsHistoryLimit: 1 # how many failed jobs should be kept
  suspend: false # Here you can suspend cronjob without deliting it
  concurrencyPolicy: Forbid # Choose Forbid if you don't want concurrent executions of your Job

  # The amount of time that Kubernetes can miss and still start a job.
  # If Kubernetes missed too many job starts (100)
  # then Kubernetes logs an error and doesn’t start any future jobs.
  startingDeadlineSeconds: 300 # if a job hasn't started in this many seconds, skip
  jobTemplate:
    spec:
      parallelism: 1 # How many pods will be instantiated at once.
      completions: 1 # How many containers of the job are instantiated one after the other (sequentially) inside the pod.
      backoffLimit: 3 # Maximum pod restarts in case of failure
      activeDeadlineSeconds: 1800 # Limit the time for which a Job can continue to run
      template:
        spec:
          restartPolicy: Never # If you want to restart - use OnFailure
          terminationGracePeriodSeconds: 30
          containers:
          - name: my-first-conjob
            image: busybox
            command:
              - /bin/sh
            args:
              - -c
              - date; echo sleeping....; sleep 90s; echo exiting...;
            resources:
              requests:
                memory: '128Mi'
              limits:
                memory: '1Gi'

Answer 2

Hi it's not clear what you expected - looking into the question but if I understand correctly you mean not running all cronjobs at the same time:
1. First option - it's to change their schedule time,
2. Second option try to use in your spec template other options like - Parallel Jobs - described: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/

"For a work queue Job, you must leave .spec.completions unset, and set .spec.parallelism to a non-negative integer"

  jobTemplate:
    spec:
      parallelism: 1
      template:

To recreate this task please provide more details.

In addition for "Jobs History" by default successfulJobsHistoryLimit and failedJobsHistoryLimit are set to 3 and 1 respectively.
Please take at: https://kubernetes.io/docs/tasks/job/ If you are interested you can set-up limit in "spec" section:

successfulJobsHistoryLimit: 1

failedJobsHistoryLimit: 1

Hope this help.