Flaky, Breaky Internal Importing
Maybe you’ve been lucky so far and never experienced the flaky, breaky side of internal importing in Python – for example, mysterious ModuleNotFound
exceptions. But if internal importing has been a problem, typically when code is imported from a multi-folder code base, it is good to know there is at least one guaranteed solution.
If you want your internal imports to work reliably, the following rules will guarantee success. Other approaches may also work, at least some of the time, but the rules presented below will always work. Following these rules will also make it possible to run modules as scripts1 no matter where they occur in the code structure. This can be especially useful during development.
Tear-Free Rules2
- Be Absolute
- Anchor Well
- Folder Imports Bad!
- Don’t Just “Run” Code
#1 Be Absolute
Use absolute importing everywhere in your code package, not just where you think internal importing is broken
Python has both relative and absolute importing. Relative importing is relative to the file doing the importing and uses dots e.g. one for the existing folder, two for the parent folder, three for grandparent etc. How many dots you need, and what your import looks like, depends on where you are in the folder structure. Absolute importing starts from an anchor point which is the same wherever the file doing the importing is.
DO:
import charm.tables.freq
import charm.conf
import charm.utils.stats.parametric.ttest
DON’T:
from ..stats.parametric import ttest
from . import ttest ## importing same thing but doing it from another module
#2 Anchor Well
Anchor absolute imports from the code package folder. E.g. if we have a module in a location like: /home/pygrant/projects/charm/charm/tables/freq.py
we would import it like:
import charm.tables.freq
assuming the code package folder was the rightmost charm folder.
It is common to use the same name for the surrounding folder as the code package folder but we don’t have to and the following example might make the situation more clear. If we have a module in a location like: /home/pygrant/projects/charm_project/charm/tables/freq.py
we would import it from charm
(the code package folder) not charm_project
(the surrounding folder).
DO:
import charm.tables.freq
DON’T:
import tables.freq ## <===== missing the anchor i.e. the code package folder
#3 Folder Imports Bad!
Don’t import folders – instead import modules or attributes of modules.
DO:
import charm.tables.freq
DON’T:
import charm.tables ## <== a folder - so you might be importing
## the __init__.py under that folder if there is one
tables.freq.get_html()
#4 Don’t Just “Run” Code
One doesn’t simply run code. Code is always executed with a particular context, often implicitly. Use one of the ways that works i.e.
- that puts the surrounding folder in the
sys.path
, so Python can find your modules to actually import their contents - and resolves module names (in particular, folders used with the dot notation) – once again, so Python can find your modules
Either use -m
option
python -m code_package.folder.folder.script
<– without .py
extension
or
ensure your IDE has the surrounding folder, the folder surrounding the code_package, in its python path (sys.path
) (possibly defined in quirky, possibly unintuitive IDE-specific ways)
You can always run the following one-liner before the problem to see what is in your python path:
import sys; print('\n'.join(sys.path))
Final Comments
I have seen mysterious internal importing problems impact numerous Python developers. The Python import system is very flexible and extensible but it is far from simple. Flaky, breaky internal importing is definitely not only a problem for beginners.
Confusion is increased by such factors as:
- Python ignoring repeated imports of the same module (name caching). This is a perfectly logical behaviour but it means a faulty import in one piece of code might be ignored in favour of a working import in another piece of code. Or vice versa – a working import statement might be ignored because of an earlier faulty import. Remember in Rule #1 – “Use absolute importing everywhere in your code package”
- IDE quirks e.g. in VS Code I was advised by a friend that the following was necessary for success:
In.vscode/settings.json
add:"terminal.integrated.env.windows": {
"PYTHONPATH": "${workspaceFolder}"
}
where${workspaceFolder}
points “to the surrounding folder”, either relative to the workspace folder using the${...}
syntax, or as an absolute path. Also put this in.env
as an absolute path to the surrounding folder:PYTHONPATH=<path-to-surrounding folder>
Simple right? 😉
PyCharm seems to require using the correct content root and allowing the Content Roots to be added to the PYTHONPATH. If the project is created off the surrounding folder this is probably the default behaviour but if this doesn’t happen it is not obvious how to fix the problem.
You don’t necessarily have to follow the “rules” above to get your internal imports working, but why take the risk? Follow the rules and then you can turn your precious attention to other programming issues. Here they are again:
- Be Absolute
- Anchor Well
- Folder Imports Bad!
- Don’t Just “Run” Code