Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Prefect Flow runs : How to send run logs to prefect server?

I'm currently trying to run some prefect Flows in my local machine (then in some Google cloud runs instances) and I would like to send runs logs to a distant prefect server.

Here's the example code :

@task
def my_task():
    logger = prefect.context.get("logger")

    logger.info("An info message.")
    logger.warning("A warning message.")

@task(log_stdout=True)
def say_hello(name):
    print("Hello, {}!".format(name))

with Flow(name="My First Flow") as flow:

    my_task()
    say_hello(Parameter('msg'))

flow.run(msg='You')

Also, I've put some configs in backend.toml :

backend = "server"

[server]
host = "SERVER_PUBLIC_IP"
port = "4200"
host_port = "4200"
endpoint = "${server.host}:${server.port}"

[server.graphql]
host = "SERVER_PUBLIC_IP"

[server.ui]
host = "SERVER_PUBLIC_IP"
port = "8080"
host_port = "8080"
endpoint = "${server.ui.host}:${server.ui.port}"
graphql_url = "SERVER_PUBLIC_IP:4200/graphql"

[logging]
# The logging level: NOTSET, DEBUG, INFO, WARNING, ERROR, or CRITICAL
level = "INFO"

# The log format
format = "[%(asctime)s] %(levelname)s - %(name)s | %(message)s"

# additional log attributes to extract from context
# e.g., log_attributes = "['context_var']"
log_attributes = "[]"

# the timestamp format
datefmt = "%Y-%m-%d %H:%M:%S"

log_to_cloud = true

So, here's the problem i'm encountering :

When I run this code, I have this error =>

  INTERNAL_SERVER_ERROR: Variable "$input" got invalid value null at
    "input.logs[0].flow_run_id"; Expected non-nullable type UUID! not to be null.

The GraphQL query was:

mutation($input: write_run_logs_input!) {
        write_run_logs(input: $input) {
            success
    }
}

Because the first line of my logs don't have flow_run_id.

Example: {"input": {"logs": [{"flow_run_id": null, "task_run_id": null, "timestamp": "2020-12-22T21:24:13.628595+00:00", "name": "prefect.FlowRunner", "message": "Beginning Flow run for 'My First Flow'", "level": "INFO"....

I'm wondering what did I forget or what am I doing wrong :(

Thanks for help

like image 474
AlaeddineB Avatar asked Oct 15 '25 22:10

AlaeddineB


1 Answers

There are two types of runs in the Prefect ecosystem:

  • interactive runs: this is the type of run you are attempting here; this is a run that you generate in your local process and are responsible for managing / monitoring. Interactive runs do not know how to talk to a Prefect API, nor do they attempt to. These runs are largely intended for local testing.

  • managed runs: these are runs that the Prefect API knows about, tracks and orchestrates - you can only create these runs via the Prefect API, either through a schedule on a registered Flow or via an ad-hoc created run (which is an API call).

Your code is attempting to mix these two; you are seeing that error because the run you created does not have the appropriate information to communicate with the backend. I suggest checking out some of the Prefect tutorials, maybe starting with this one

like image 199
chriswhite Avatar answered Oct 19 '25 12:10

chriswhite



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!