Ayla Khan – Page 3 – Data Engineering, DevOps, Code

First look at GitHub’s GraphQL API

Published February 20, 2018 in devops

Sometimes it’s necessary to query GitHub for repo information through an API; during a continuous integration (CI) build step for example. I’ve used GitHub’s REST API before, which is OK but dumps a lot of extra data that can be annoying to parse. Also, sometimes multiple queries are needed to get to the data I […]

Tags: ci , git , graphql

Customize Pandas Histograms

Published January 18, 2018 in data , programming

All python code in this post is Python 3.5+. In my previous post, I described how I got usable Pandas dataframes from the Kaggle movies dataset. My next step was to start exploring the data with simple visualizations. The first feature I wanted explore was the distribution of movies by year in the movies_metadata data […]

Tags: pandas , python , visualization

Importing Stringified JSON Objects Into Pandas (Part 3)

Published December 14, 2017 in data , programming

All python code in this post is Python 3.5+. This post describes how I parsed movies_metadata.csv from the Kaggle movies dataset; a task I started in Part 1 and Part 2. After some digging into the Pandas documentation and Stack Overflow, I found that the best solution to my parsing problems was to explicitly set […]

Tags: data wrangling , pandas , python

Python type hints: make code easier to read and check

Published December 1, 2017 in programming

All python code is Python 3.5+. In the Pandas data import posts, I’m using Python type hints. Type hinting is a fairly new feature in Python and has been provisionally accepted as a language feature. It is also a thought-provoking design feature for a dynamically typed language. Types are not enforced at runtime. The type […]

Tags: python

Importing Stringified JSON Objects Into Pandas (Part 2)

Published November 30, 2017 in data , programming

All python code in this post is Python 3.5+. Continuing from Part 1, I discovered that movies_metadata.csv contains malformed rows that have missing fields, which is what caused file import to fail. I tried experimenting with some of the more advanced Pandas.read_csv parameters to see if I could work around the malformed rows. def main(path: […]

Tags: data wrangling , pandas , python

Importing Stringified JSON Objects Into Pandas (Part 1)

Published November 24, 2017 in data , programming

All python code in this post is Python 3.5+. I’m continuing to work with the same Kaggle movies dataset as in the SQL import experiment. This time, I imported the data into Pandas DataFrames. The trickiest dataset to import was movies_metadata.csv. I first tried to use pandas.read_csv with the default settings. import argparse import pandas […]

Tags: data wrangling , pandas , python

JSONB arrays in PostgreSQL tables

Published November 13, 2017 in programming

All python code is Python 3.5+. PostgreSQL database version is 10. I started digging into the Kaggle movies dataset recently, which is a collection of CSV files. I was curious to see if the data could be inserted into a SQL database (PostgreSQL) for further exploration. The credits.csv file contains two columns (cast, crew) of […]

Tags: data wrangling , python , sql

Using the Python ijson parser

Published November 2, 2017 in programming

All python code is Python 3.5+. A few months ago, I had to extract a small amount of data from a large and deeply nested JSON file quickly and export to CSV. I was working in C++ and Python on this project, so my first attempts to extract the data were using the Python json […]

Tags: data wrangling , python

« Previous 1 2 3