Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Unable to Start Scheduler

I am new to Python and trying to install Airflow in my Mac, by following this tutorial

While these two commands work fine:

$ airflow initdb
$ airflow webserver -p 8080

The scheduler command (airflow scheduler) throws the following error:

 [2020-02-18 13:18:09,012] {scheduler_job.py:1382} ERROR - Exception when executing execute_helper Traceback (most recent call last):

  File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1380, in _execute
    self._execute_helper()

  File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1413, in _execute_helper
    self.processor_agent.start()

  File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 554, in start
    self._process.start()

  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)

  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)

  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)

  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)

  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)

  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)

  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)

AttributeError: Can't pickle local object 'SchedulerJob._execute.<locals>.processor_factory'

[2020-02-18 13:18:09,035] {helpers.py:322} INFO - Sending Signals.SIGTERM to GPID None

Traceback (most recent call last):    File "/Users/mac/Workspace/airflow/airflow_venv/bin/airflow", line 37, in <module>
    args.func(args)    File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/utils/cli.py", line 75, in wrapper
    return f(*args, **kwargs)    File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/bin/cli.py", line 1040, in scheduler
    job.run()    File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 221, in run
    self._execute()    File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1384, in _execute
    self.processor_agent.end()    File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 707, in end
    reap_process_group(self._process.pid, log=self.log)    File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/utils/helpers.py", line 324, in reap_process_group
    signal_procs(sig)    File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/utils/helpers.py", line 293, in signal_procs
    os.killpg(pgid, sig)

TypeError: an integer is required (got type NoneType)
like image 642
brijesh chowdary lavu Avatar asked Feb 18 '20 07:02

brijesh chowdary lavu


People also ask

How do I enable scheduler?

Click Start, click All Programs, click Accessories, click System Tools, and click Task Scheduler. In the Task Scheduler window, in the left pane, click to expand the Task Scheduler Library folder. Under the Task Scheduler Library folder, click to expand the folder where the task is located.

How do you fix Task Scheduler started and then stopped?

Type Services. msc in Start Search, open the app, scroll down to Task Scheduler service, click on it, make sure it is Running and Set to Automatic. Then click the Dependencies tab, make sure those services are running too.


1 Answers

EDIT: Python 3.8 is supported now https://github.com/apache/airflow#requirements. So this answer might not be relevant now.

This due to the Python version you are using. Airflow doesn't support Python 3.8 yet https://github.com/apache/airflow#stable-version-1109.

Downgrade your Python to 3.7 and check.

like image 190
arunvelsriram Avatar answered Oct 25 '22 21:10

arunvelsriram