{"id":1560,"date":"2023-09-19T11:39:35","date_gmt":"2023-09-18T23:39:35","guid":{"rendered":"http:\/\/p-s.co.nz\/wordpress\/?p=1560"},"modified":"2023-09-24T10:01:08","modified_gmt":"2023-09-23T22:01:08","slug":"internal-python-imports-without-tears","status":"publish","type":"post","link":"http:\/\/p-s.co.nz\/wordpress\/internal-python-imports-without-tears\/","title":{"rendered":"Internal Python Imports Without Tears"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Flaky, Breaky Internal Importing<\/h2>\n\n\n\n<p>Maybe you&#8217;ve been lucky so far and never experienced the flaky, breaky side of internal importing in Python &#8211; for example, mysterious <code>ModuleNotFound<\/code> exceptions. But if internal importing has been a problem, typically when code is imported from a multi-folder code base, it is good to know there is at least one guaranteed solution.<\/p>\n\n\n\n<p>If you want your internal imports to work reliably, the following rules will guarantee success. Other approaches may also work, at least some of the time, but the rules presented below will always work. Following these rules will also make it possible to run modules as scripts<sup data-fn=\"489a18cc-88a2-4f86-97f9-84671c2b683c\" class=\"fn\"><a href=\"#489a18cc-88a2-4f86-97f9-84671c2b683c\" id=\"489a18cc-88a2-4f86-97f9-84671c2b683c-link\">1<\/a><\/sup> no matter where they occur in the code structure. This can be especially useful during development.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Tear-Free Rules<sup data-fn=\"0d1ce12b-aee3-4d47-ae0b-899e555f0b86\" class=\"fn\"><a href=\"#0d1ce12b-aee3-4d47-ae0b-899e555f0b86\" id=\"0d1ce12b-aee3-4d47-ae0b-899e555f0b86-link\">2<\/a><\/sup><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Be Absolute<\/li>\n\n\n\n<li>Anchor Well<\/li>\n\n\n\n<li>Folder Imports Bad!<\/li>\n\n\n\n<li>Don\u2019t Just \u201cRun\u201d Code<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">#1 Be Absolute<\/h3>\n\n\n\n<p>Use absolute importing everywhere in your code package, not just where you think internal importing is broken<\/p>\n\n\n\n<p>Python has both relative and absolute importing. Relative importing is relative to the file doing the importing and uses dots e.g. one for the existing folder, two for the parent folder, three for grandparent etc. How many dots you need, and what your import looks like, depends on where you are in the folder structure. Absolute importing starts from an anchor point which is the same wherever the file doing the importing is.<\/p>\n\n\n\n<p>DO:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import charm.tables.freq\nimport charm.conf\nimport charm.utils.stats.parametric.ttest<\/code><\/pre>\n\n\n\n<p>DON&#8217;T:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>from ..stats.parametric import ttest\n\nfrom . import ttest  ## importing same thing but doing it from another module<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">#2 Anchor Well<\/h3>\n\n\n\n<p>Anchor absolute imports from the code package folder. E.g. if we have a module in a location like: <code>\/home\/pygrant\/projects\/charm\/charm\/tables\/freq.py<\/code> we would import it like:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import charm.tables.freq<\/code><\/pre>\n\n\n\n<p>assuming the code package folder was the rightmost charm folder.<\/p>\n\n\n\n<p>It is common to use the same name for the surrounding folder as the code package folder but we don&#8217;t have to and the following example might make the situation more clear. If we have a module in a location like: <code>\/home\/pygrant\/projects\/charm_project\/charm\/tables\/freq.py<\/code> we would import it from <code>charm<\/code> (the code package folder) not <code>charm_project<\/code> (the surrounding folder).<\/p>\n\n\n\n<p>DO:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import charm.tables.freq<\/code><\/pre>\n\n\n\n<p>DON&#8217;T:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import tables.freq  ## &lt;===== missing the anchor i.e. the code package folder<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">#3 Folder Imports Bad!<\/h3>\n\n\n\n<p>Don&#8217;t import folders &#8211; instead import modules or attributes of modules.<\/p>\n\n\n\n<p>DO:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import charm.tables.freq<\/code><\/pre>\n\n\n\n<p>DON&#8217;T:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import charm.tables  ## &lt;== a folder - so you might be importing\n                     ## the __init__.py under that folder if there is one\ntables.freq.get_html()<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">#4 Don&#8217;t Just &#8220;Run&#8221; Code<\/h3>\n\n\n\n<p>One doesn&#8217;t simply run code. Code is always executed with a particular context, often implicitly. Use one of the ways that works i.e.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>that puts the surrounding folder in the <code>sys.path<\/code>, so Python can find your modules to actually import their contents<\/li>\n\n\n\n<li>and resolves module names (in particular, folders used with the dot notation) &#8211; once again, so Python can find your modules<\/li>\n<\/ul>\n\n\n\n<p>Either use <code>-m<\/code> option<\/p>\n\n\n\n<p><code>python -m code_package.folder.folder.script<\/code>&nbsp; &lt;&#8211; without <code>.py<\/code> extension<\/p>\n\n\n\n<p>or<\/p>\n\n\n\n<p>ensure your IDE has the surrounding folder, the folder surrounding the code_package, in its python path (<code>sys.path<\/code>) (possibly defined in quirky, possibly unintuitive IDE-specific ways)<\/p>\n\n\n\n<p>You can always run the following one-liner before the problem to see what is in your python path:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import sys; print('\\n'.join(sys.path))<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Final Comments<\/h2>\n\n\n\n<p>I have seen mysterious internal importing problems impact numerous Python developers. The Python import system is very flexible and extensible but it is far from simple. Flaky, breaky internal importing is definitely not only a problem for beginners.<\/p>\n\n\n\n<p>Confusion is increased by such factors as:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Python ignoring repeated imports of the same module (name caching). This is a perfectly logical behaviour but it means a faulty import in one piece of code might be ignored in favour of a working import in another piece of code. Or vice versa &#8211; a working import statement might be ignored because of an earlier faulty import. Remember in Rule #1 &#8211; &#8220;Use absolute importing <strong>everywhere<\/strong> in your code package&#8221;<\/li>\n\n\n\n<li>IDE quirks e.g. in VS Code I was advised by a friend that the following was necessary for success:<br><br>In <code>.vscode\/settings.json<\/code> add:<br><br><code>\"terminal.integrated.env.windows\": {<br>    \"PYTHONPATH\": \"${workspaceFolder}\"<br>}<\/code><br>where&nbsp;<code>${workspaceFolder}<\/code>&nbsp;points &#8220;to the surrounding folder&#8221;, either relative to the workspace folder using the <code>${...}<\/code> syntax, or as an absolute path. Also put this in <code>.env<\/code> as an absolute path to the surrounding folder:<br><code>PYTHONPATH=&lt;path-to-surrounding folder&gt;<\/code><br><br>Simple right? \ud83d\ude09<br><br>PyCharm seems to require using the correct content root and allowing the Content Roots to be added to the PYTHONPATH. If the project is created off the surrounding folder this is probably the default behaviour but if this doesn&#8217;t happen it is not obvious how to fix the problem.<\/li>\n<\/ul>\n\n\n\n<p>You don&#8217;t necessarily have to follow the &#8220;rules&#8221; above to get your internal imports working, but why take the risk? Follow the rules and then you can turn your precious attention to other programming issues. Here they are again:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Be Absolute<\/li>\n\n\n\n<li>Anchor Well<\/li>\n\n\n\n<li>Folder Imports Bad!<\/li>\n\n\n\n<li>Don&#8217;t Just &#8220;Run&#8221; Code<\/li>\n<\/ul>\n\n\n<ol class=\"wp-block-footnotes\"><li id=\"be297435-5e04-4608-80e4-949c19fa783f\">Running a script means actually doing something (e.g. writing a file, making an API call, etc)<br>rather than just defining something without running it (e.g. defining a function or a class). <a href=\"#be297435-5e04-4608-80e4-949c19fa783f-link\" aria-label=\"Jump to footnote reference 1\">\u21a9\ufe0e<\/a><\/li><li id=\"0d1ce12b-aee3-4d47-ae0b-899e555f0b86\">The &#8220;Tears&#8221; theme is a nod to the popular statistics book &#8220;Statistics Without Tears&#8221; <a href=\"#0d1ce12b-aee3-4d47-ae0b-899e555f0b86-link\" aria-label=\"Jump to footnote reference 2\">\u21a9\ufe0e<\/a><\/li><\/ol>","protected":false},"excerpt":{"rendered":"<p>Flaky, Breaky Internal Importing Maybe you&#8217;ve been lucky so far and never experienced the flaky, breaky side of internal importing in Python &#8211; for example, mysterious ModuleNotFound exceptions. But if internal importing has been a problem, typically when code is &hellip; <a href=\"http:\/\/p-s.co.nz\/wordpress\/internal-python-imports-without-tears\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"[{\"content\":\"Running a script means actually doing something (e.g. writing a file, making an API call, etc)<br>rather than just defining something without running it (e.g. defining a function or a class).\",\"id\":\"be297435-5e04-4608-80e4-949c19fa783f\"},{\"content\":\"The \\\"Tears\\\" theme is a nod to the popular statistics book \\\"Statistics Without Tears\\\"\",\"id\":\"0d1ce12b-aee3-4d47-ae0b-899e555f0b86\"}]"},"categories":[11,3],"tags":[21,18,15],"class_list":["post-1560","post","type-post","status-publish","format-standard","hentry","category-programming","category-python","tag-code","tag-programming","tag-python"],"_links":{"self":[{"href":"http:\/\/p-s.co.nz\/wordpress\/wp-json\/wp\/v2\/posts\/1560"}],"collection":[{"href":"http:\/\/p-s.co.nz\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/p-s.co.nz\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/p-s.co.nz\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/p-s.co.nz\/wordpress\/wp-json\/wp\/v2\/comments?post=1560"}],"version-history":[{"count":22,"href":"http:\/\/p-s.co.nz\/wordpress\/wp-json\/wp\/v2\/posts\/1560\/revisions"}],"predecessor-version":[{"id":1587,"href":"http:\/\/p-s.co.nz\/wordpress\/wp-json\/wp\/v2\/posts\/1560\/revisions\/1587"}],"wp:attachment":[{"href":"http:\/\/p-s.co.nz\/wordpress\/wp-json\/wp\/v2\/media?parent=1560"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/p-s.co.nz\/wordpress\/wp-json\/wp\/v2\/categories?post=1560"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/p-s.co.nz\/wordpress\/wp-json\/wp\/v2\/tags?post=1560"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}