Day38 - bad data in csv file

[alt3red]

Hello,

On Day 38 when loading the Thanksgiving data from the file, there is a error that occurs:

UnicodeDecodeError:

'utf-8' codec can't decode byte 0xed in position 3599: invalid continuation byte

The file gets processed until the record with 'RespondentID':'4335955206' then it errors out.

Also when opening the csv file in PyCharm I get "File was loaded in the wrong encoding: 'UTF-8'"

The error goes away when removing the line with RspondentID: '4335955152'

[bbelderbos]

Thanks @alt3red, must indeed be some weird character in RespondentID 4335955206:

import csv

filename = 'thanksgiving-2015-poll-data.csv'

row = None
try:
    with open(filename, 'r', encoding='utf-8') as fin:  # ISO-8859-1
        reader = csv.DictReader(fin)

        for row in reader:
            s = ''.join(row.values())
            s.encode('utf-8')
except:
    print(row['RespondentID’])

If I use ISO-8859-1 it works. See also: https://intellij-support.jetbrains.com/hc/en-us/community/posts/360000004284-File-was-loaded-in-the-wrong-encoding-UTF-8-

@mikeckennedy maybe something to add to the README?

[mikeckennedy]

Thanks for the heads up guys! I fixed the file and cloned the repo. There is a new link the readme.md now.

https://github.com/talkpython/100daysofcode-with-python-course/blob/master/days/37-39-csv-data-analsys/README.md

I also did a PR back to FiveThirtyEight fixing their data. We'll see if they accept it.

[mikeckennedy]

PR here:

fivethirtyeight/data#181