If this helped you, please share!

A closer look at Airflow’s KubernetesPodOperator and XCom

Published July 11, 2019 in data - 0 Comments

The KubernetesPodOperator handles communicating XCom values differently than other operators. The basics are described in the operator documentation under the xcom_push parameter. I’ve written up a more detailed example that expands on that documentation.

An Airflow task instance described by the KubernetesPodOperator can write a dict to the file /airflow/xcom/return.json (always the same file) that will be read and turned into an XCom value by a airflow-xcom-sidecar sidecar container. This sidecar gets created when the KubernetesPodOperator parameter xcom_push is True.

For instance, if a KubernetesPodOperator task instance executes the following Python code in a Docker container example_image:

This is the KubernetesPodOperator task instance:

The XCom values will be made available to downstream task instances under the ‘return_value’ key:

The default for xcom_pull‘s key parameter is ‘return_value’, so key is an optional parameter in this example.

XCom values can also be pulled using Jinja templates in operator parameters that support templates, which are listed in operator documentation. Here is the list of parameters for the KubernetesPodOperator, and also for the PythonOperator.

This is how to get the value of ‘key1’ in a Jinja template:

A useful way to use XCom values with templates is to pass arguments to a Docker container:

No comments yet

Leave a Reply: