Log in

entries friends calendar profile Elf Sternberg's Pendorwright Projects Previous Previous
Elf M. Sternberg

In the last post, I introduced the concepts of the module object, module, and package, concrete objects that exist within the Python runtime, as well as some basic ideas about packaging, finding, and loading.

In this post, I’ll go over the process of finding, what it means to find something, and what happens next.

A Clarifying point

I’ve been very careful to talk about finding vs. loading vs. listing in this series of posts. There’s a reason for that: in Python 2, the terms "Finder" and "Importer" were used interchangeably, leading to (at least on my part) massive confusion. In actual fact, finders, hooks, loaders, and listers are all individual objects, each with a single, unique method with a specific signature. The method name is different for each stage, so it is theoretically possible to define a single class that does all three for a given category of module object, and only in that case, I believe, should we talk about an "Importer."

In Python 2.6 and 2.7, the definitive Finder class is called pkgutil.ImpImporter, and the Loader is called pkgutil.ImpLoader; this was a source of much of my confusion. In Python 3, the term "Importer" is deprecated and "Finder" is used throughout importlib. I will be using "Finder" from now on.


When the import <fullname> command is called, a procedure is triggered. That procedure then:

  • attempts to find a corresponding python module

  • attempts to load that corresponding module into bytecode

  • Associates the bytecode with the name via sys.modules[fullname]

  • Exposes the bytecode to the calling scope.

  • Optionally: writes the bytecode to the filesystem for future use

Finding is the act of identifying a resource that corresponds to the import string and that can be compiled into a meaningful Python module. The import string is typically called the fullname.

Finding typically involves scanning a collection of resources against a collection of finders. Finding ends when finder A, given fullname B, reports that a corresponding module can be found in resource C, and that the resource can be loaded with loader D."


Finders come first, and MetaFinders come before all other kinds of finders.

Most finding is done in the context of sys.path; that is, Python’s primary means of organizing Python modules is to have them somewhere on the local filesystem. This makes sense. Sometimes, however, you want to get in front of that scan and impose your own logic: you want the root of an import string to mean something else. Maybe instead of directory.file, you want it to mean table.row.cell, or you want it to mean website.path.object, to take one terrifying example.

That’s what you do with a MetaFinder: A MetaFinder may choose to ignore the entire sys.path mechanism and do something that has nothing to do with the filesystem, or it may have its own filesystem notion completely separate from sys.path, or it may have its own take on what to do with sys.path.

A Finder (both MetaFinder and FileFinder) is any object with the following method:

[Loader|None] find_module([self|cls], fullname:string, path:[string|None])

The find_module method returns None if it cannot find a loader resource for the provided fullname. The path is optional; in the standard Python implementation, when the path is None it means “use `sys.path`”; when it’s set, it’s the path in which to look.

A MetaFinder is placed into the list sys.meta_path by whatever code needs the MetaFinder, and it persists for the duration of the runtime, unless it is later removed or replaced. Being a list, the search is ordered; first match wins. MetaFinders may be instantiated in any way the developer desires before being added into sys.meta_path.

PathHooks and PathFinders

PathHooks are how sys.path is scanned to determine the which Finder should be associated with a given directory path.

A PathHook is a function (or callable):

[Finder|None] <anonymous function>(path:string)

A PathHook takes a given directory path and, if the PathHook can identify a corresponding FileFinder for the modules in that directory path and return a constructed instance of that FileFinder, otherwise it returns None.

If no sys.meta_path finder returns a Loader, the full array of sys.paths ⨯ sys.path_hooks is compared until a PathHook says it can handle the path and the corresponding finder says it can handle the fullname. If no match happens, Python’s default FileFinder class is instantiated with the path.

This means that for each path in sys.paths, the list of sys.path_hooks is scanned; the first function to return an importer is handed responsibility for that path; if no function returns, the default FileFinder is returned; the default FileFinder returns only the default SourceFileLoader which (if you read to the end of part one) blocks our path toward heterogeneous packages.

PathHooks are placed into the list sys.path_hooks; like sys.meta_path, the list is ordered and first one wins.

The Takeaway

There’s some confusion over the difference between the two objects, so let’s clarify one last time.

Use a meta_finder (A Finder in sys.meta_path) when you want to redefine the meaning of the import string so it can search alternative paths that may have no reference to a filesystem path found in sys.path; an import string could be redefined as a location in an archive, an RDF triple of document/tag/content, or table/row_id/cell, or be interpreted as a URL to a remote resource.

Use a path_hook (A function in sys.path_hooks that returns a FileFinder) when you want to re-interpret the meaning of an import string that refers to a module object on or accessible by sys.path; PathHooks are important when you want to add directories to sys.path that contain something other than .py, .pyc/.pyo, and .so modules conforming to the Python ABI.

A MetaFinder is typically constructed when it is added to sys.meta_path; a PathHook instantiates a FileFinder when the PathHook function lays claim to the path. The developer instantiates a MetaFinder before adding it to sys.meta_path; it’s the PathHook function that instantiates a FileFinder.


Note that PathHooks are for paths containing something other than the traditional (and hard-coded) source file extensions. The purpose of a heterogeneous source file finder and loader is to enable finding in directories within sys.path that contain other source files syntaxes alongside those traditional sources. I need to eclipse (that is, get in front of) the default FileFinder with one that understands more suffixes than those listed in either imp.get_suffixes() (Python 2) or importlib._bootstrap.SOURCE_SUFFIXES (Python 3). I need one that will return the Python default loader if it encounters the Python default suffixes, but will invoke our own source file loader when encountering one of our suffixes.

We’ll talk about loading next.

Leave a comment
Yesterday afternoon I made my usual commute home, but it was not the pleasant ride I've come to expect. I walk from my office through downtown to the underground, where I hop on the light rail that runs from the University of Washington through downtown, then up the Martin Luther King corridor to the airport. It's usually a very pleasant ride, 35 minutes. Sometimes its crowded, and rarely crowded enough I have to stand.

There has been an effort by the local press, egged on by the always anti-transit conservative press, to document a spate of assaults on public transit in the past year. So that's also very much on my mind.

As the train exited the underground and started through the light industrial zone, a man got on with a bicycle. He was white, scruffy and dirty, dressed in overalls and carrying a bucket and squeegee. I was sitting on the bench next to the doors, and an attractive young woman in her mid-20s was sitting next to me.

He started talking at her. He got louder and louder. She started to look away uncomfortably, and other people on the train were getting equally uncomfortable. I decided to try something: I very deliberately pulled out my cell phone and took his photograph.

"Hey, you! You think you can just take my picture like that?" He turned his attention to me. He didn't move any closer, fortunately. But he then went on a rant. "You. I never forget a face. I will never forget you. If I see you twenty years from now, if some fool like you even lives that long, I will deliver my punishment on you. I will. I never forget. Never. You can't just take someone's picture like that without permission." On and on like that for the entire rest of the ride. I managed to deflect his attention from the woman, but that didn't make the rest of the ride at all pleasant. He went from demanding attention to making vague but abusive threats.

When I reached the commuter station, I was able to get off. He stayed on, with the woman. There was only one more stop left.

This is the United States. Where the fucking homeless are armed. In fucking Seattle. Where anyone and everyone could be a target because of our gun culture. And where anyone can become a poorly trained, adrenaline-activated "good guy with a gun" and actually create more injury. "Terrified" doesn't begin to cover what situations like this begin to be with that kind of awareness floating around just days after the Orlando massacre. Where I end up on a train looking at some scruffy lowlife who could end me and everyone else on this train if he got pissed off enough.

America is a horror show. So many things, enforced by guns at the low level, and by courts controlled by the powerful at the high, work to keep us cowed.

Current Mood: scared scared

6 comments or Leave a comment

A minor bug in the Hy programming language has led me down a rabbit hole of Python’s internals, and I seem to have absorbed an awful lot of Python’s import semantics. The main problem can best be described this way: In Python, you call the import function with a string; that string gets translated in some way into python code. So: what are the exact semantics of the python import command?

Over the next couple of posts, I’ll try to accurately describe what it means when you write:

import alpha.beta.gamma
from alpha import beta
from alpha.beta import gamma
from .delta import epsilon

In each case, python is attempting to resolve the collection of dotted names into a module object.

module object: A resource that is or can be compiled into a meaningful Python module. This resource could a file on a filesystem, a cell in a database, a remote web object, a stream of bytes in an object store, some content object in a compressed archive, or anything that can meaningfully be described as an array of bytes (Python 2) or characters (Python 3). It could even be dynamically generated!

module: The organizational unit of Python code. A namespace containing Python objects, including classes, functions, submodules, and immediately invoked code. Modules themselves may be collected into packages.

package: A python module which contains submodules or even subpackages. The most common packaging scheme is a directory folder; in this case the folder is a module if it contains an __init__.py file, and it is a package if it contains other modules. The name of the package is the folder name; the name of a submodule would be foldername.submodule. This is called regular packaging. An alternative method (which I might cover later) is known as namespace packaging.

Python has a baroque but generally flexible mechanism for defining how the dotted name is turned into a module object, which it calls module finding, and for how that module object is turned into a code object within the current Python session, called module loading.

Python also has a means for module listing. Listing is usually done on a list of paths, using an appropriate means for finding (identifying) the contents at the end of a path as Python modules.

The technical definition of a package is a module with a __path__, a list of paths that contain submodules for the package. Subpackages get their own__path__. A package can therefore accommodate . and .. prefixes in submodules, indicating relative paths to sibling modules. A package can also and to its own __path__ collection to enable access to submodules elsewhere.


The problem I am trying to solve:

Python module listing depends upon finder resolving a path to a container of modules, usually (but not necessarily) a package. The very last finder is the default one: after all alternatives provided by users have been exhausted, Python reverts to the default behavior of analyzing the filesystem, as one would expect. The default finder is hard-coded to use the Python builtin imp.get_suffixes() function, which in turn hard-codes the extensions recognized by the importer.

If one wants to supply alternative syntaxes for Python and have heterogenous packages (for examples, packages that contain some modules ending in .hy, and others .py, side-by-side)… well, that’s just not possible.


In the next post, I’ll discuss Python’s two different finder resolution mechanisms, the meta_path and the path_hook, and how they differ, and which one we’ll need to instantiate to solve the problem of heterogenous Python syntaxes. The actual solution will eventually involve eclipsing Python’s default source file handler with one that enables us to define new source file extensions at run-time, recognize the source file extensions, and supply the appropriate compilers for those source files, while falling back on the default behavior correctly when the extension is one Python can handle by itself.

My hope is that, once solved, this will further enable the development of Python alternative syntaxes. Folks bemoan the explosion of Javascript precompilers, but the truth is that it has in part led to a revival in industrial programming languages and a renaissance in programming language development in general.   Python, with its AST available and exposed at runtime, is eminently suitable as an alternative language research platform– except for this one little problem.

I am trying to solve that problem.

1 comment or Leave a comment
I ought to be a Sad Puppy.

No, I'm not a supporter of snot-nosed Vox Day's ridiculous and stupid campaign to overwrite the will of Hugo voters. I'm not going to say that he and his ilk deserve anything over than public opprobrium in the face of their self-serving and malign campaign to deprive the science fiction community of its next level up.

When it comes to what I like to read, well... I generally like what Baen has to offer. Sad, but true. I bounce off women writers more often than I do men, I like bad space opera and ridiculous tales of derring-do, and I cheer when the hero gets laid. One of my favorite writers is David Friggin Weber, which is about as silly as its sounds.

I'm a little more sensitive now that I was in my 20s. Admittedly, in my 20s I was living through the early Cyberpunk era and its post-New Wave backlash, which both embraced and attacked the mores and ethics of New Wave feminist SF 1970s. I wince if the story is too obviously written by a an old white guy who thinks a homogeneous, conservative version of Southern California in space!, where the men are men and the women are pliant, is paradise. But the truth is I'm an old white guy who likes reading stories written by other old white guys.

Which is why I really can't stand the Sad Puppy / Rabid Puppy thing at all. Look, boys, we're old. Like, really old. Kids these days are going to forge their own worlds, with their own stories, and there ain't a damn thing we can do about it. You can poison some awards processes and wreck a fine time you could have otherwise enjoyed. The kids will have this world long after we're gone, and the only thing we can do is try to hand off a world of faith, hope and caritas... or, in the case of the SP/RP crowd, a burning cinder, a hellscape fixed forever in a universe of pure hatred.

Nobody is stopping anyone from writing. No one is preventing you from buying John C. Wright's books. (I wouldn't encourage anyone to give Wright money, though; his hackneyed, ridiculous style would disqualify him long before we start discussing his manifest personal cruelty.)

Go ahead, write the swash and the buckle. Just remember that you're outnumbered. Your niche rose to prominence because of the privileges we white, English-speaking guys had in the early 20th century. But in the age of the Internet, you are outnumbered, you are a minority, and you are an embarrassment if you continue to write with the kind of deft wit and narrative grace that brought Ken Robeson or Victor Appleton fame.

Tags: ,
Current Mood: amused amused
Current Music: Hamilton, Farmer Refuted

Leave a comment

The weirdest interview I ever had was shortly after I got laid off from Isilon. One-fifth of Isilon had been laid off in one brutal morning, and I was among them. The economy was deep into the recession, and like everyone else who had been laid off recently I had put out my resume and my shingle, then settled into my basement to teach myself everything I hadn’t learned in the previous eight years.

A large hardware-based American computer company whose name I won’t repeat, but which was once headed by a former US presidential contender infamous for an ad featuring demon sheep, reached out through its recruitment efforts and asked if I would come in for an interview. They said my skills were a perfect fit for a new product they had in mind.

The interview was way the hell out in Bellevue, hours and miles from my house, and utterly inconvenient for me. When I got there, I learned that their “new product” was basically a direct competitor for Isilon’s; they were trying to get into the mass storage market, and were having a hard time of it.

I was scheduled to be there all day. But as the first hour wound down, I already knew the job wasn’t for me. I didn’t really have the skills they wanted; I wasn’t all that familiar with the C/C++ side of the business, after all. I knew more about the Isilon product from the configuration side than anyone outside of field support, but not much about the internals. More than that, the job didn’t sound interesting. Java and Swing weren’t technologies I was particularly interested in. The office environment seemed cold and unfriendly. The commute was murderous. And I’d just gotten cut from an infrastructure job; why would I want another one?

When the second interviewer came in, I said to him, “I’m sorry, but this job isn’t really for me. I think we should call it here.”

He looked at me, confused. “You mean you just want to… stop?”

I said, “I’m not going to take the job.”

“But…” He stopped, utterly still, for a few moments, then rose and left. A few minutes later, he came back and said, “The candidate isn’t allowed to end the interview process.”

I told him flatly, “I don’t think you can legally stop me from leaving the building.”

He stopped for a moment and looked very confused.  “No, that’s true,” he said. “Okay, I guess I’ll, um, escort you out.”

“Thank you.”

To this day, I have no idea what he meant by “The candidate isn’t allowed to end the interview process.”  I suspect that what he got from the higher ups he went to talk to was “There’s no process for the candidate ending the interview,” and went from there, but… weird.

2 comments or Leave a comment
What's the difference between esteem and respect? The dictionary gives it precisely:
a feeling of deep admiration for someone or something
a feeling of deep admiration for someone or something elicited by their abilities, qualities, or achievements.
I won't even bother to highlight the difference in the text.

It's been one of those things that bothers me when we talk about self-esteem versus self-respect. Criminals frequently score high on self-esteem, after all.

So here's another one: What's the different between accountablity and responsibility? Well, the definitions:
The obligation to account for one's activities and to disclose the results
The obligation to perform or complete a task
. Again, I'm not going to bother highlighting the difference in the text.

When I hear people talk about "accountability in government," I tend to cringe. Not because I don't think we should have it, but because I think it should be the second thing we talk about. Yet I never hear people talk about "responsibility in governance," never hear issues of whether or not we trust people to do their jobs responsibly. Whenever someone issues a statement about "accountability," they're starting from a position of faithlessness and mistrust.

Current Mood: amused amused

1 comment or Leave a comment
Conservative talk show host Dennis Prager has two questions for atheists:
1. Do you hope you are right or wrong?
2. Do you ever doubt your atheism?

I respect atheists who answer that they hope they are wrong. It tells me that they understand the terrible consequences of atheism: that all existence is random; that there is no ultimate meaning to life; that there is no objective morality — right and wrong are subjective personal or societal constructs; that when we die, there is nothing but eternal oblivion, meaning, among other things, that one is never reconnected with any loved ones; and there is no ultimate justice in the universe — murderers, torturers and their victims have identical fates: nothing.
Prager's "respect" is something I can heartily live without. I'm happy with the notion that there's no "ultimate" meaning to life, that the assumptions Prager's tribe have made about what is "meaning" are the best and only, and those who don't hew to them, who don't grant power to traditions cast by goat herders and those who mouth their platitudes, deserve punishment. Prager's "punishements" include never being reunited with loved ones, at least the ones who likewise didn't earn the punishments.

But more than that, I've always found the very notion of God to be fundamentally diabolical. Do it as a thought experiment: there exists an entity with immense power, who can reach in and raise your life or throw it down with a mere thought. That entity either (a) helps you reach its state, or (b) thwarts you in reaching its state. Everything in the God story points toward answer (b): the misdirection, the hiddeness, the arbitrariness, the entire kit and kaboodle of excuses for why "god" is hard to understand is just that: excuses.

Give me Buddhism Without Beliefs, a systemic appreciation for human suffering without supernatural reasons, victim blaming, or excuses. Give me Stoicism, a basic, eudaemonic approach to life that likewise doesn't blame victims, has no supernatural reasons, and makes no excuses for its own failings.
How is it that when you see a baby born or a spectacular sunset, or hear a Mozart symphony, or read about the infinite complexity of the human brain — none of these has ever prompted you to wonder whether there really might be a God?
How is it you see a child born with microencaphly and not wonder about the infinite cruelty of a being who could make that stop happening, but chooses not to? I'm comforted by the knowledge that there is nothing out there with power over and interest in my life, or yours, or Dennis Prager's.

Tags: ,
Current Mood: amused amused

5 comments or Leave a comment
Usually, JStor isn't a place where I have much truck, but Kate Bielamowicz's America's Workforce Runs On Narcotic Stimulants hits me where I live. She keeps using the term "narcotic," and I'm not sure what she means by it: "narcotic" has two meanings, the first of which means not prescribed, the second of which means opiate-related. Since she's describing prescription stimulants, I can only conclude that her use of the term narcotic is a perjorative: she's trying to link the readers' minds an unwarranted connection between stimulant overprescription and the current opiod addiction crisis, which is claiming a lot of lives.

I can't speak for people who take amphetamines recreationally, or for those who get it off-label.

But I take Dextroamphetamine IR for my ADHD. I have fought my entire freakin' life with this condition, and to say it doesn't occur in adults is to deny my lived experience. I take 5mg, once a day, with breakfast, and it actually allows me to function as a human being. Since I've started taking it, I have actually started to complete a number of important projects that have languished for years.

I have a prescription. It's hard to get that prescription filled; it's a Class-I substance, which means I have to drive to the doctor, have a check-in, get a hand-written prescription, drive to the pharmacy, and wait for them to get it in. I can't fax it, I can't have it called in. I have to be present at each and every step of the transaction and present ID for it.

The number one effect of it has been the alleviation of frustration. I'm not longer constantly mad at myself for being a failure, for being unable to finish things, for being so damned incapable of calming down my hyperactive, overly talkative, ooooh-shiny! brain.

So don't make me angry. I'm pretty pathetic when I'm angry. And Bielamowicz's article seems determined to try and stir up a new moral scare, one that will make me angry.

Tags: , ,
Current Mood: annoyed annoyed

4 comments or Leave a comment
A couple of months ago I discussed having idiopathic dysgeusia, a benign condition associated with middle age in which everything tastes bad and funny. One of the hypotheses floated originally was posioning, that somehow I had encountered something that damaged the inside of my mouth.

But then it got worse, and I saw another doctor who sent me to my dentist. He was able to say, "It look like desquamative gingivitis, which happens at your age. I'm sending you to a specialist."

So I saw the specialist today. By now, the condition has almost completely cleared up. Everything tastes normal, I can enjoy spicy foods again, I'm no longer suffering "mechanical damage" (i.e. chapping from constantly probing at the weirdness in my mouth). There's a slight hesitancy at the tip of my tounge, but that's about it. The specialist is back to thinking it was poisoning, but we can't pinpoint what.

He basically said if it comes back, take pictures, document the hell out of everything I ate, drank or tasted that week, and make another appointment.

Tags: , ,
Current Mood: annoyed annoyed

Leave a comment

I may have inadvertently contributed to the Coffeescriptification of Python. And in doing so, I have learned way too much about Python internals.

Coffeescript probably wasn’t the first programming language that transpiled to Javascript, but it was the first that caught on. Coffeescript and all its successors, like Clojurescript, Gorilla, and Earl Grey, exist because Javascript is a terrible language written on top of a truly universal runtime: the browser. Then some bright person took that runtime and put it on the server and called it Node, and then someone else took it and put it in the database and called it PLv8, and folks who’d been battle-hardened living in the browser could now be productive on the full stack. Transpilers exists because, rather than write in that terrible language, they allow people to write code in a language they like and it ends up all running in Javascript in the end.

Python doesn’t have a universe like this. Because Python is actually a fairly good dynamic language, and the syntax is pretty much the best of the pick of its competition from Perl, Ruby, Visual Basic, or obscurities like Pike or Io. But the Python runtime is not much loved; the global interpreter lock constrains Python’s performance in many ways.

There are a few, though, including:

  • Hy, a Lisp

  • Mochi, a Python derivative with Erlang actors

  • Dogelang, an ML-imitating… thing

With the coming Gilectomy, Python may finally move past some of its more egregious runtime limitations, and when it does, maybe alternative language developer will start to look at it more seriously, and develop languages that run on top of it the way Pure or Elm run on top of Javascript, yet manage to do so while honoring functional purity in a very real way.

But Python has one other limitation: the extension “.py” is hard coded into the import library that ships with python. Hy had a fairly good work-around that let it load its libraries and python side-by-side, and Doge seems to have a similar solution, but none of them were universal.

Then I learned that Hy doesn’t work with Django.

This all started out because I wanted to write a Django app. It had been awhile, and I had a specific desire to take what I’d learned writing mp_suggest and do something a little more complex with it, as a way of organizing the several thousands of CDs and their rips I have on my home NAS server.

I’m sure there are Django apps out there that already do this, but I wanted to write my own, and as you’ll see if you follow that like mp_suggest runs on Hy. Most of Django worked, but custom commands couldn’t be found by a Hy-based version of manage.hy, which annoyed me.

An itch that needed scratching! It took me two weeks; I probably put about 20 hours into untangling all of it. The real issue was that Hy was hooking into Python’s sys.meta_path, when what was needed was something even earlier in the import chain, a sys.path_hook. Path hooks are black magic. I’ll have a more technical entry up in a little bit.

In the meantime, here’s the patch. At first, it was just hooking up the path_hook, but as I was looking at the code, I realized that the hy-specific parts were actually fairly limited. It came down to two items: filename extensions, and a compiler. Everything else was just… boilerplate.

So I abstracted it out.

It’s now possible, by adding a few extra imports to your manage.py file, to build a Django app where:

  • Your views are written in Hy

  • Your models are written in Mochi

  • Your custom commands are written in Doge

And it all just works.

Leave a comment