Using the Python ijson parser

All python code is Python 3.5+.

A few months ago, I had to extract a small amount of data from a large and deeply nested JSON file quickly and export to CSV. I was working in C++ and Python on this project, so my first attempts to extract the data were using the Python json module and Pandas json reader. The file was too big and nested for either tool to work well. Solution? Use the ijson parser.

I can’t share the original so I’m using the JSON file sf-city-lots-json as an example. Structurally, the JSON is simpler, but the file is of similar size and has enough structure to be interesting to parse.

Taking a brute-force approach, I explored the file directly using the parse function, which generates event tuples of (prefix, event, value) constructed from their location in the JSON object tree and values.

import ijson


def parse_json(json_filename):
    with open(json_filename, 'rb') as input_file:
        # load json iteratively
        parser = ijson.parse(input_file)
        for prefix, event, value in parser:
            print('prefix={}, event={}, value={}'.format(prefix, event, value))


if __name__ == '__main__':
    parse_json('./data/citylots_small.json')

Prefixes describe location of the keys or names in the object tree. Events report value types, mark the start (start_array) and end of arrays (end_array) and objects (start_map, end_map) and also mark keys (map_key).

Ijson’s parser works efficiently with large JSON files by leveraging generator iterators and yield expressions to avoid loading the entire structure in memory at once. The object tree is also an efficient way to represent nested JSON.

This is the beginning of the JSON file:

{
"type": "FeatureCollection",
"features": [
{ "type": "Feature", "properties": { "MAPBLKLOT": "0001001", "BLKLOT": "0001001", "BLOCK_NUM": "0001", "LOT_NUM": "001", "FROM_ST": "0", "TO_ST": "0", "STREET": "UNKNOWN", "ST_
TYPE": null, "ODD_EVEN": "E" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -122.422003528252475, 37.808480096967251, 0.0 ], [ -122.422076013325281, 37.80883501981508
5, 0.0 ], [ -122.421102174348633, 37.808803534992904, 0.0 ], [ -122.421062569067274, 37.808601056818148, 0.0 ], [ -122.422003528252475, 37.808480096967251, 0.0 ] ] ] } }
...

The output from the script is pretty verbose. Breaking it down, the first few lines of the file are:

{
"type": "FeatureCollection",
"features": [

Which is parsed as:

prefix=, event=start_map, value=None
prefix=, event=map_key, value=type
prefix=type, event=string, value=FeatureCollection
prefix=, event=map_key, value=features
prefix=features, event=start_array, value=None

This is an example object from the “features” array:

{ "type": "Feature", "properties": { "MAPBLKLOT": "0001001", "BLKLOT": "0001001", "BLOCK_NUM": "0001", "LOT_NUM": "001", "FROM_ST": "0", "TO_ST": "0", "STREET": "UNKNOWN", "ST_TYPE": null, "ODD_EVEN": "E" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -122.422003528252475, 37.808480096967251, 0.0 ], [ -122.422076013325281, 37.808835019815085, 0.0 ], [ -122.421102174348633, 37.808803534992904, 0.0 ], [ -122.421062569067274, 37.808601056818148, 0.0 ], [ -122.422003528252475, 37.808480096967251, 0.0 ] ] ] } }

The parser output describes all the keys in the object hierarchy, their types and associated values.

prefix=features.item, event=start_map, value=None
prefix=features.item, event=map_key, value=type
prefix=features.item.type, event=string, value=Feature
prefix=features.item, event=map_key, value=properties
prefix=features.item.properties, event=start_map, value=None
prefix=features.item.properties, event=map_key, value=MAPBLKLOT
prefix=features.item.properties.MAPBLKLOT, event=string, value=0001001
prefix=features.item.properties, event=map_key, value=BLKLOT
prefix=features.item.properties.BLKLOT, event=string, value=0001001
prefix=features.item.properties, event=map_key, value=BLOCK_NUM
prefix=features.item.properties.BLOCK_NUM, event=string, value=0001
prefix=features.item.properties, event=map_key, value=LOT_NUM
prefix=features.item.properties.LOT_NUM, event=string, value=001
prefix=features.item.properties, event=map_key, value=FROM_ST
prefix=features.item.properties.FROM_ST, event=string, value=0
prefix=features.item.properties, event=map_key, value=TO_ST
prefix=features.item.properties.TO_ST, event=string, value=0
prefix=features.item.properties, event=map_key, value=STREET
prefix=features.item.properties.STREET, event=string, value=UNKNOWN
prefix=features.item.properties, event=map_key, value=ST_TYPE
prefix=features.item.properties.ST_TYPE, event=null, value=None
prefix=features.item.properties, event=map_key, value=ODD_EVEN
prefix=features.item.properties.ODD_EVEN, event=string, value=E
prefix=features.item.properties, event=end_map, value=None
prefix=features.item, event=map_key, value=geometry
prefix=features.item.geometry, event=start_map, value=None
prefix=features.item.geometry, event=map_key, value=type
prefix=features.item.geometry.type, event=string, value=Polygon
prefix=features.item.geometry, event=map_key, value=coordinates
prefix=features.item.geometry.coordinates, event=start_array, value=None
prefix=features.item.geometry.coordinates.item, event=start_array, value=None
prefix=features.item.geometry.coordinates.item.item, event=start_array, value=None
prefix=features.item.geometry.coordinates.item.item.item, event=number, value=-122.422003528252475
prefix=features.item.geometry.coordinates.item.item.item, event=number, value=37.808480096967251
prefix=features.item.geometry.coordinates.item.item.item, event=number, value=0
prefix=features.item.geometry.coordinates.item.item, event=end_array, value=None
prefix=features.item.geometry.coordinates.item.item, event=start_array, value=None
prefix=features.item.geometry.coordinates.item.item.item, event=number, value=-122.422076013325281
prefix=features.item.geometry.coordinates.item.item.item, event=number, value=37.808835019815085
prefix=features.item.geometry.coordinates.item.item.item, event=number, value=0
prefix=features.item.geometry.coordinates.item.item, event=end_array, value=None
prefix=features.item.geometry.coordinates.item.item, event=start_array, value=None
prefix=features.item.geometry.coordinates.item.item.item, event=number, value=-122.421102174348633
prefix=features.item.geometry.coordinates.item.item.item, event=number, value=37.808803534992904
prefix=features.item.geometry.coordinates.item.item.item, event=number, value=0
prefix=features.item.geometry.coordinates.item.item, event=end_array, value=None
prefix=features.item.geometry.coordinates.item.item, event=start_array, value=None
prefix=features.item.geometry.coordinates.item.item.item, event=number, value=-122.421062569067274
prefix=features.item.geometry.coordinates.item.item.item, event=number, value=37.808601056818148
prefix=features.item.geometry.coordinates.item.item.item, event=number, value=0
prefix=features.item.geometry.coordinates.item.item, event=end_array, value=None
prefix=features.item.geometry.coordinates.item.item, event=start_array, value=None
prefix=features.item.geometry.coordinates.item.item.item, event=number, value=-122.422003528252475
prefix=features.item.geometry.coordinates.item.item.item, event=number, value=37.808480096967251
prefix=features.item.geometry.coordinates.item.item.item, event=number, value=0
prefix=features.item.geometry.coordinates.item.item, event=end_array, value=None
prefix=features.item.geometry.coordinates.item, event=end_array, value=None
prefix=features.item.geometry.coordinates, event=end_array, value=None
prefix=features.item.geometry, event=end_map, value=None
prefix=features.item, event=end_map, value=None

The parsed output for the end of the file:

]
}

Closes up the “features” array and the top-level object:

prefix=features, event=end_array, value=None
prefix=, event=end_map, value=None

Now that I understand the JSON structure and how the prefixes are formatted, I can extract a specific item from the file. For example, if I wanted property lot numbers:

import ijson


def extract_lot_numbers(json_filename):
    with open(json_filename, 'rb') as input_file:
        lot_numbers = ijson.items(input_file, 'features.item.properties.LOT_NUM')
        for number in lot_numbers:
            print('Lot number: {}'.format(number))


if __name__ == '__main__':
    extract_lot_numbers('./data/citylots.json')

Which outputs:

Lot number: 001
Lot number: 001
Lot number: 002
Lot number: 001
...

I can also filter the objects generated by ijson using list comprehensions. For example, if I wanted all the properties with street type “ST”:

import ijson


def extract_properties(json_filename):
    with open(json_filename, 'rb') as input_file:
        objects = ijson.items(input_file, 'features.item.properties')
        street_properties = (o for o in objects if o['ST_TYPE'] == 'ST')
        for prop in street_properties:
            print('Property on a street (ST): {}'.format(prop))


if __name__ == '__main__':
    extract_properties('./data/citylots.json')

Which outputs:

Property on a street (ST): {'MAPBLKLOT': '0005001', 'BLKLOT': '0005001', 'BLOCK_NUM': '0005', 'LOT_NUM': '001', 'FROM_ST': '206', 'TO_ST': '286', 'STREET': 'JEFFERSON', 'ST_TYPE': 'ST', 'ODD_EVEN': 'E'}
Property on a street (ST): {'MAPBLKLOT': '0005001', 'BLKLOT': '0005001', 'BLOCK_NUM': '0005', 'LOT_NUM': '001', 'FROM_ST': '206', 'TO_ST': '286', 'STREET': 'JEFFERSON', 'ST_TYPE': 'ST', 'ODD_EVEN': 'E'}
Property on a street (ST): {'MAPBLKLOT': '0006001', 'BLKLOT': '0006001', 'BLOCK_NUM': '0006', 'LOT_NUM': '001', 'FROM_ST': '350', 'TO_ST': '366', 'STREET': 'JEFFERSON', 'ST_TYPE': 'ST', 'ODD_EVEN': 'E'}
Property on a street (ST): {'MAPBLKLOT': '0007001', 'BLKLOT': '0007001', 'BLOCK_NUM': '0007', 'LOT_NUM': '001', 'FROM_ST': '2936', 'TO_ST': '2936', 'STREET': 'HYDE', 'ST_TYPE': 'ST', 'ODD_EVEN': 'E'}
...