- Home>
- devops
Categories
KubernetesExecutor The KubernetesExecutor sets up Airflow to run on a Kubernetes cluster. This executor runs task instances in pods created from the same Airflow Docker image used by the KubernetesExecutor itself, unless configured otherwise (more on that at the end). Getting Airflow deployed with the KubernetesExecutor to a cluster is not a trivial task. I […]
The AWS web console is not always intuitive to navigate and the documentation can be opaque. I recently had to edit an EC2 launch template, which is only possible by creating a new version and then making the new version the default used by the autoscale group. Here are the steps to create a new […]
For integration tests with few external dependencies that don’t require much orchestration beyond networking Docker containers and setting up environment variables, Docker Compose is a simple and easy to manage solution for building, running and tearing down tests. This Flask application example is typical. The project’s multistage Dockerfile defines both the service and test images: […]
Here are my best reasons for using container solutions like Docker: 1. Simplify the deployment process! I used to struggle with software deployments. Traditional software package installers work fine for simple deployments on a single platform type, but require maintenance if deployment guidelines change between OS verisons. Deployment tools that need to target multiple platforms […]
All python code is Python 3.5+. Having an automatic way to build GitHub pull requests before merging saves a lot of time and trouble compared with pulling, building and testing a GitHub pull request locally. TeamCity makes it easy to set this up using branch specifications. The blog post refers to a much older version […]
Sometimes it’s necessary to query GitHub for repo information through an API; during a continuous integration (CI) build step for example. I’ve used GitHub’s REST API before, which is OK but dumps a lot of extra data that can be annoying to parse. Also, sometimes multiple queries are needed to get to the data I […]