Mark Vasilkov
Mark Vasilkov

Follow

Mark Vasilkov

Follow
CSV field larger than field limit in Python

Photo by Daniel Sandoval on Unsplash

CSV field larger than field limit in Python

Mark Vasilkov's photo
Mark Vasilkov
·Apr 24, 2023·

1 min read

So I was working on something very simple involving a CSV file:

import csv

with open('file.csv', 'r') as f:
    reader = csv.reader(f)
    lines = [ln for ln in reader]

And as one would expect, the result of running this code was Error: field larger than field limit (131072)

The 128 KiB field size boundary is so beautifully arbitrary. Use the following workaround to fix that:

import sys

csv.field_size_limit(sys.maxsize)

Doesn't have to be specifically sys.maxsize, any sufficiently large number would do.

There's no moral to this story, just mild suffering.