Photo by Daniel Sandoval on Unsplash
CSV field larger than field limit in Python
·Apr 24, 2023·
1 min read
So I was working on something very simple involving a CSV file:
import csv with open('file.csv', 'r') as f: reader = csv.reader(f) lines = [ln for ln in reader]
And as one would expect, the result of running this code was
Error: field larger than field limit (131072)
The 128 KiB field size boundary is so beautifully arbitrary. Use the following workaround to fix that:
import sys csv.field_size_limit(sys.maxsize)
Doesn't have to be specifically
sys.maxsize, any sufficiently large number would do.
There's no moral to this story, just mild suffering.