Can openers, Python imports, and other illusions

There’s a body of research literature showing that people overestimate how well they can explain phenomena.  Ask someone whether they understand how a can opener works, for example, and they’re likely to say they do.  Ask someoneto explain how a can opener works, though, and you’re likely to get confusion, frustration, and a confession that they don’t understand as well as they thought they did.

Once you’re aware of this phenomenon, you can see it everywhere.  I was reminded of it this morning when I ran into trouble with Python imports.

I have somehow managed to get through years of Python programming without ever building a complicated directory structure from scratch.  Everything’s been either single directory, a Django project (where imports are mostly handled by the framework), or someone else’s pre-existing work I’m contributing to or adapting.

It turns out that getting a file in one directory to use functions or classes defined in another directory is surprisingly difficult.  Here is what I learned, once I realized I needed to learn more:

The importer works by searching for the requested module in a series of directories determined by the system environmental variable ‘PYTHONPATH’.  The first place it looks is the current directory, which is why single-directory structures avoid this problem.  It then checks a series of default locations, none of which will be your project directories.  So you need to add to PYTHONPATH.

One way to do this is using sys.add.path(), but it can be quite annoying to have to insert one of these statements before every import.  I thought I was being clever by creating a ‘setup.py’ file where I used sys.add.path() once for every module, however I discovered that this method does not work _permanently._Once the setup.py file finished running, the directories disappeared from the path, which obviously was no good for my purposes.

Okay, I thought.  I’ll add to PYTHONPATH directly.  I edited my ~/.bashrc according to StackOverflow’s recommendations.  Print statements in my test file confirmed that the relevant files were added to the paths.  Yet I was still getting ImportErrrors!

It turned out that I was adding an incorrect path.  I assumed that I needed to add the path of the modules themselves.  Let’s say my project is in a folder named project with two subfolders, module1 and module2, each containing an init.py and some files.  Instead of setting the pythonpath to include /fullpath/project/module1 and /fullpath/project/module2, I only needed to specify /fullpath/project.

Thus ends the story of how I lost an entire morning to trying to understand something I thought I already knew.