Debugging notes

I spent a couple of hours today wrestling with encodings.  Writing out the details here to help me remember, and perhaps save someone else from grief.

So!  I’d previously written some code which loaded a PDF from file and used pdfminer to convert the content to HTML.  I wanted to adapt the code to handle PDFs in memory as well.  However, I kept getting this error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)

I realized I needed to spend some time learning encoding basics. Luckily, a bit of googling turned up a guide from Nedbat, who is in my opinion unbeatable as a teacher of basic python topics.  (I’ve recommended his intro to unit testing to pretty much anyone who will listen.)

It seems my confusion was a matter of terminology. ‘String’ is a semantically overloaded word.  In python, ‘string’ means by default an unencoded text string, like the kind you produced by typing, say, example = “exampletext” or by casting something with str().  When you open a python file it gives you the contents as unencoded text string by default, and that’s what pdfminer expects.  However!  I was using io.StringIO to create my memory-stream equivalent to the loaded PDF file, and StringIO actually returns unicode strings, which are not the same thing as unencoded text strings at all.  What I should have been using was io.BytesIO.

I guess you could say that I got strung along. 😉