Ayla Khan – Page 2 – Data Engineering, DevOps, Code

How to split a Pandas dataframe into training and test sets?

Published June 18, 2018 in data

This question came up recently on a project where Pandas data needed to be fed to a TensorFlow classifier. In this case, we wanted to divide the dataframe using a random sampling. Frameworks like scikit-learn may have utilities to split data sets into training, test and cross-validation sets. For example, sklearn.model_selection.train_test_split split numpy arrays or […]

Tags: data wrangling , pandas , python

Docker Compose for integration testing

Published June 17, 2018 in devops

For integration tests with few external dependencies that don’t require much orchestration beyond networking Docker containers and setting up environment variables, Docker Compose is a simple and easy to manage solution for building, running and tearing down tests. This Flask application example is typical. The project’s multistage Dockerfile defines both the service and test images: […]

Tags: ci , docker

Why do I use containers?

Published May 2, 2018 in devops

Here are my best reasons for using container solutions like Docker: 1. Simplify the deployment process! I used to struggle with software deployments. Traditional software package installers work fine for simple deployments on a single platform type, but require maintenance if deployment guidelines change between OS verisons. Deployment tools that need to target multiple platforms […]

Tags: docker , python

Pandas DataFrame axis basics (Part 2)

Published April 13, 2018 in data

Part 1 covered Pandas DataFrame basics. Pandas offers multiple options for accessing DataFrame values by axis labels using he DataFrame.loc function, or by integer indexes using the DataFrame.iloc function in one or two dimensions. If the DataFrame has a numerical index, calling the DataFrame.loc and DataFrame.iloc functions looks the same. Otherwise, use the appropriate axis […]

Tags: pandas

Pandas DataFrame axis basics (Part 1)

Published April 2, 2018 in data

By default, a Pandas DataFrame is 2 dimensional with 2 axes initialized as empty Index structures. Under the basic indexing scheme, the first axis is the ‘index’ axis, which by default is a numerical index starting from 0 (using np.arange) generated for each DataFrame row. The second axis is the ‘columns’ axis, which is the […]

Tags: pandas

GitHub GraphQL in CI

Published March 9, 2018 in devops

All python code is Python 3.5+. Having an automatic way to build GitHub pull requests before merging saves a lot of time and trouble compared with pulling, building and testing a GitHub pull request locally. TeamCity makes it easy to set this up using branch specifications. The blog post refers to a much older version […]

Tags: ci , git , graphql , python

Python type hints: alias those types!

Published February 21, 2018 in programming

All python code is Python 3.5+. PEP484 goes beyond built-in type annotations. Another feature of the Python type hinting libary is the ability to create type aliases. I’ve used type aliasing frequently in C++ (typedef, using) to improve code readability and for its other benefits. I’m happy to see that it’s available in Python too. […]

Tags: python

First look at GitHub’s GraphQL API

Published February 20, 2018 in devops

Sometimes it’s necessary to query GitHub for repo information through an API; during a continuous integration (CI) build step for example. I’ve used GitHub’s REST API before, which is OK but dumps a lot of extra data that can be annoying to parse. Also, sometimes multiple queries are needed to get to the data I […]

Tags: ci , git , graphql

Customize Pandas Histograms

Published January 18, 2018 in data , programming

All python code in this post is Python 3.5+. In my previous post, I described how I got usable Pandas dataframes from the Kaggle movies dataset. My next step was to start exploring the data with simple visualizations. The first feature I wanted explore was the distribution of movies by year in the movies_metadata data […]

Tags: pandas , python , visualization

Importing Stringified JSON Objects Into Pandas (Part 3)

Published December 14, 2017 in data , programming

All python code in this post is Python 3.5+. This post describes how I parsed movies_metadata.csv from the Kaggle movies dataset; a task I started in Part 1 and Part 2. After some digging into the Pandas documentation and Stack Overflow, I found that the best solution to my parsing problems was to explicitly set […]

Tags: data wrangling , pandas , python

« Previous 1 2 3 Next »