UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbf
파일이 utf-8이 아니라 다른 방식으로 저장되어있다란 의미.
한국 공공데이터 csv는 대부분 cp949, euc-kr로 저장되지만 pandas는 기본이 utf-8이다.
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
/tmp/ipykernel_44167/539685071.py in <cell line: 0>()
1 from google.colab import drive
2 drive.mount('/content/drive')
----> 3 df = pd.read_csv(
4 "/content/drive/MyDrive/data/국가데이터처_나라통계_우편번호_20211110.csv"
5 )
/usr/local/lib/python3.12/dist-packages/pandas/io/parsers/readers.py in read_csv(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, date_format, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, encoding_errors, dialect, on_bad_lines, delim_whitespace, low_memory, memory_map, float_precision, storage_options, dtype_backend)
1024 kwds.update(kwds_defaults)
1025
-> 1026 return _read(filepath_or_buffer, kwds)
1027
1028
/usr/local/lib/python3.12/dist-packages/pandas/io/parsers/readers.py in _read(filepath_or_buffer, kwds)
618
619 # Create the parser.
--> 620 parser = TextFileReader(filepath_or_buffer, **kwds)
621
622 if chunksize or iterator:
/usr/local/lib/python3.12/dist-packages/pandas/io/parsers/readers.py in __init__(self, f, engine, **kwds)
1618
...
parsers.pyx in pandas._libs.parsers.raise_parser_error()
/usr/lib/python3.12/codecs.py in decode(self, input, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbf in position 0: invalid start byte
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...
encoding="cp949" 붙이기.
안되면 encoding="euc-kr"
import pandas as pd
df = pd.read_csv(
"/content/drive/MyDrive/data/국가데이터처_나라통계_우편번호_20211110.csv",
encoding="cp949"
)
df = pd.read_csv(
"/content/drive/MyDrive/data/국가데이터처_나라통계_우편번호_20211110.csv",
encoding="euc-kr"
)
인코딩을 자동으로 추측할 수 도 있다 .
import chardet
with open("/content/drive/MyDrive/data/국가데이터처_나라통계_우편번호_20211110.csv", "rb") as f:
print(chardet.detect(f.read(10000)))