Skip to main content

CSV field larger than field limit in Python

Updated 1 min readMarkdown
CSV field larger than field limit in Python

So I was working on something very simple involving a CSV file:

import csv

with open('file.csv', 'r') as f:
    reader = csv.reader(f)
    lines = [ln for ln in reader]

And as one would expect, the result of running this code was Error: field larger than field limit (131072)

The 128 KiB field size boundary is so beautifully arbitrary. Use the following workaround to fix that:

import sys

csv.field_size_limit(sys.maxsize)

Doesn't have to be specifically sys.maxsize, any sufficiently large number would do.

There's no moral to this story, just mild suffering.

1.9K views

Hello) next problem is "OverflowError: Python int too large to convert to C long"

Well, depending on what's in that int, you either reach for a C bigint library, or just truncate it I guess.