All posts in "programming"

Python comprehension fun!

Published November 15, 2018 in programming - 0 Comments

List comprehensions are an extremely useful and optimized idiomatic Python language feature for manipulating and returning data stored in lists (or any iterable type). The Python 3 docs describe how to use basic and nested list comprehensions. Advanced nested list comprehensions If I wanted to clean some text by removing stop words and generate lists […]

Tags: python

Robust automatic retries in Celery tasks

Published October 22, 2018 in programming - 0 Comments

The Celery distributed task queue introduced retrying a failed task automatically for known exception types in version 4.0, and some very useful retry exponential backoff settings in version 4.2. Exponential backoff is beneficial because it spaces out retry requests in exponentially increasing intervals which can allow time to recover or restart. I turned off jitter […]

Tags: celery , docker , python

Python type hints: alias those types!

Published February 21, 2018 in programming - 0 Comments

All python code is Python 3.5+. PEP484 goes beyond built-in type annotations. Another feature of the Python type hinting libary is the ability to create type aliases. I’ve used type aliasing frequently in C++ (typedef, using) to improve code readability and for its other benefits. I’m happy to see that it’s available in Python too. […]

Tags: python

Importing Stringified JSON Objects Into Pandas (Part 2)

Published November 30, 2017 in data , programming - 0 Comments

All python code in this post is Python 3.5+. Continuing from Part 1, I discovered that movies_metadata.csv contains malformed rows that have missing fields, which is what caused file import to fail. I tried experimenting with some of the more advanced Pandas.read_csv parameters to see if I could work around the malformed rows.

[…]

Importing Stringified JSON Objects Into Pandas (Part 1)

Published November 24, 2017 in data , programming - 0 Comments

All python code in this post is Python 3.5+. I’m continuing to work with the same Kaggle movies dataset as in the SQL import experiment. This time, I imported the data into Pandas DataFrames. The trickiest dataset to import was movies_metadata.csv. I first tried to use pandas.read_csv with the default settings.

I was able […]