What is the use case of Airflow Params?

[hussein-awala]

I read a lot about Airflow Dag Run configurations (dag_run.conf) and Airflow parameters (params), and I think that there is something missing in the documentation or a something incomplete in the development.

Currently, I'm using dag_run.conf to add some runtime variables to my dags, where I can provide the conf dict when I create my run using the UI, using the Airflow CLI (-c <conf>) and as an argument for the method create_dagrun in the DAG model class. And I can access this dict as an attribute for the Dag Run instance available in the context variables.
But there is an annoying limit where I cannot define a default Dag Run conf dict for my dag.

I understood that the Dag params is developed to replace the Dag Run conf and to bypass its limits, where we can define default values for our params, and even add a validation step by typing them and add a range and conditions, in order to let Airflow shows a warning instead of creating the Dag Run if the user supplies values don’t pass validation.

But based on my deep search, I found that we cannot provide these params when we trigger the run (with the 3 methods I listed before), and I didn't find any way to update them during the runtime, so I find them useless until now. I don't know if there is something missing in the documentation, or this params feature is still under development.

[potiuk]

I think you should show example of what you tried. there are probably 1000s of other users who used it so far and you are the first to raise the concern, so I guess you made some grave mistake somewhere. But if you provide some examples of what you tried and what was wrong, we might be able to help you to explain what was the wrong assumption you've made. CC: @msumit - might likely help - but sharing examples from your side are absolutely necessary to be able to help to rectify a mistake you've done.

[hussein-awala]

In fact, I was preparing an example to explain my use case, but fortunately, I tested it in a new Airflow docker container, and I found the answer to my question.

Here is a Dag which use dag_run.conf:

with DAG(
    dag_id='airflow_run_conf',
    start_date=datetime(2022, 8, 14),
    default_args=default_args,
    schedule_interval=None,
) as dag:
    task = BashOperator(
        task_id='print_dagrun_conf',
        bash_command='echo "{{ dag_run.conf.get("var") }}"'
    )

And here is an example to how I can override the conf value using Airflow CLI:

airflow dags trigger airflow_run_conf -c '{"var": "val"}'

The second example is a Dag which user params instead of dag_run.conf:

with DAG(
    dag_id='airflow_params',
    start_date=datetime(2022, 8, 14),
    default_args=default_args,
    schedule_interval=None,
    params={
        "var": "val1"
    }
) as dag:
    task = BashOperator(
        task_id='print_dagrun_conf',
        bash_command='echo "{{ params.var }}"'
    )

When I called this command:

airflow dags trigger airflow_params -c '{"var": "val2"}'

the BashOperator task printed val2 instead of val1, so I seems that params is based on dagrun conf, it just validates the values and add default ones if there is something missing in the dict.

I will check why this didn't work in my main Airflow server, maybe I have core.dag_run_conf_overrides_params=False.

[potiuk]

Cool. Glad it is solved.

[Foundsheep]

Hi, sorry to bother you guys, if this question sounds a bit silly or too basic.

But as a newbie to use Airflow for the first time, can I assume that dag_run.conf is an outdated concept and params replace that?

I was looking for a way to provide arguments to my dag in the UI's trigger and codex suggested me to use dag_run.conf. I think I can safely use params instead, right?