Tag Archives for " kubernetes "

TIL: How to pass extra gunicorn arguments to MLflow server

Published March 22, 2022 in today I learned - 0 Comments

The MLflow server exposes the –workers flag to change the number of gunicorn workers, but if you want to pass other arguments to gunicorn then you need the –gunicorn-opts flag: It looks like –waitress-opts serves the same purpose if running MLflow on Windows. If passing the –gunicorn-opts flag to a container running the server on […]

Tags: kubernetes , mlops

A closer look at Airflow’s KubernetesPodOperator and XCom

Published July 11, 2019 in data - 8 Comments

The KubernetesPodOperator handles communicating XCom values differently than other operators. The basics are described in the operator documentation under the xcom_push parameter. I’ve written up a more detailed example that expands on that documentation. An Airflow task instance described by the KubernetesPodOperator can write a dict to the file /airflow/xcom/return.json (always the same file) that […]

Useful Airflow on Kubernetes Features

Published June 7, 2019 in data , devops - 0 Comments

KubernetesExecutor The KubernetesExecutor sets up Airflow to run on a Kubernetes cluster. This executor runs task instances in pods created from the same Airflow Docker image used by the KubernetesExecutor itself, unless configured otherwise (more on that at the end). Getting Airflow deployed with the KubernetesExecutor to a cluster is not a trivial task. I […]