Skip to main content

Command Palette

Search for a command to run...

CSV field larger than field limit in Python

Updated
1 min read
CSV field larger than field limit in Python

So I was working on something very simple involving a CSV file:

import csv

with open('file.csv', 'r') as f:
    reader = csv.reader(f)
    lines = [ln for ln in reader]

And as one would expect, the result of running this code was Error: field larger than field limit (131072)

The 128 KiB field size boundary is so beautifully arbitrary. Use the following workaround to fix that:

import sys

csv.field_size_limit(sys.maxsize)

Doesn't have to be specifically sys.maxsize, any sufficiently large number would do.

There's no moral to this story, just mild suffering.

1.7K views
A

Hello) next problem is "OverflowError: Python int too large to convert to C long"

M

Well, depending on what's in that int, you either reach for a C bigint library, or just truncate it I guess.