- Home>
- programming
Categories
List comprehensions are an extremely useful and optimized idiomatic Python language feature for manipulating and returning data stored in lists (or any iterable type). The Python 3 docs describe how to use basic and nested list comprehensions. Advanced nested list comprehensions If I wanted to clean some text by removing stop words and generate lists […]
The Celery distributed task queue introduced retrying a failed task automatically for known exception types in version 4.0, and some very useful retry exponential backoff settings in version 4.2. Exponential backoff is beneficial because it spaces out retry requests in exponentially increasing intervals which can allow time to recover or restart. I turned off jitter […]
All python code is Python 3.5+. PEP484 goes beyond built-in type annotations. Another feature of the Python type hinting libary is the ability to create type aliases. I’ve used type aliasing frequently in C++ (typedef, using) to improve code readability and for its other benefits. I’m happy to see that it’s available in Python too. […]
All python code in this post is Python 3.5+. In my previous post, I described how I got usable Pandas dataframes from the Kaggle movies dataset. My next step was to start exploring the data with simple visualizations. The first feature I wanted explore was the distribution of movies by year in the movies_metadata data […]
All python code in this post is Python 3.5+. This post describes how I parsed movies_metadata.csv from the Kaggle movies dataset; a task I started in Part 1 and Part 2. After some digging into the Pandas documentation and Stack Overflow, I found that the best solution to my parsing problems was to explicitly set […]
All python code is Python 3.5+. In the Pandas data import posts, I’m using Python type hints. Type hinting is a fairly new feature in Python and has been provisionally accepted as a language feature. It is also a thought-provoking design feature for a dynamically typed language. Types are not enforced at runtime. The type […]
All python code in this post is Python 3.5+. Continuing from Part 1, I discovered that movies_metadata.csv contains malformed rows that have missing fields, which is what caused file import to fail. I tried experimenting with some of the more advanced Pandas.read_csv parameters to see if I could work around the malformed rows. def main(path: […]
All python code in this post is Python 3.5+. I’m continuing to work with the same Kaggle movies dataset as in the SQL import experiment. This time, I imported the data into Pandas DataFrames. The trickiest dataset to import was movies_metadata.csv. I first tried to use pandas.read_csv with the default settings. import argparse import pandas […]
All python code is Python 3.5+. PostgreSQL database version is 10. I started digging into the Kaggle movies dataset recently, which is a collection of CSV files. I was curious to see if the data could be inserted into a SQL database (PostgreSQL) for further exploration. The credits.csv file contains two columns (cast, crew) of […]
All python code is Python 3.5+. A few months ago, I had to extract a small amount of data from a large and deeply nested JSON file quickly and export to CSV. I was working in C++ and Python on this project, so my first attempts to extract the data were using the Python json […]