Log in

entries friends calendar profile Elf Sternberg's Pendorwright Projects Previous Previous
Elf M. Sternberg

I may have inadvertently contributed to the Coffeescriptification of Python. And in doing so, I have learned way too much about Python internals.

Coffeescript probably wasn’t the first programming language that transpiled to Javascript, but it was the first that caught on. Coffeescript and all its successors, like Clojurescript, Gorilla, and Earl Grey, exist because Javascript is a terrible language written on top of a truly universal runtime: the browser. Then some bright person took that runtime and put it on the server and called it Node, and then someone else took it and put it in the database and called it PLv8, and folks who’d been battle-hardened living in the browser could now be productive on the full stack. Transpilers exists because, rather than write in that terrible language, they allow people to write code in a language they like and it ends up all running in Javascript in the end.

Python doesn’t have a universe like this. Because Python is actually a fairly good dynamic language, and the syntax is pretty much the best of the pick of its competition from Perl, Ruby, Visual Basic, or obscurities like Pike or Io. But the Python runtime is not much loved; the global interpreter lock constrains Python’s performance in many ways.

There are a few, though, including:

  • Hy, a Lisp

  • Mochi, a Python derivative with Erlang actors

  • Dogelang, an ML-imitating… thing

With the coming Gilectomy, Python may finally move past some of its more egregious runtime limitations, and when it does, maybe alternative language developer will start to look at it more seriously, and develop languages that run on top of it the way Pure or Elm run on top of Javascript, yet manage to do so while honoring functional purity in a very real way.

But Python has one other limitation: the extension “.py” is hard coded into the import library that ships with python. Hy had a fairly good work-around that let it load its libraries and python side-by-side, and Doge seems to have a similar solution, but none of them were universal.

Then I learned that Hy doesn’t work with Django.

This all started out because I wanted to write a Django app. It had been awhile, and I had a specific desire to take what I’d learned writing mp_suggest and do something a little more complex with it, as a way of organizing the several thousands of CDs and their rips I have on my home NAS server.

I’m sure there are Django apps out there that already do this, but I wanted to write my own, and as you’ll see if you follow that like mp_suggest runs on Hy. Most of Django worked, but custom commands couldn’t be found by a Hy-based version of manage.hy, which annoyed me.

An itch that needed scratching! It took me two weeks; I probably put about 20 hours into untangling all of it. The real issue was that Hy was hooking into Python’s sys.meta_path, when what was needed was something even earlier in the import chain, a sys.path_hook. Path hooks are black magic. I’ll have a more technical entry up in a little bit.

In the meantime, here’s the patch. At first, it was just hooking up the path_hook, but as I was looking at the code, I realized that the hy-specific parts were actually fairly limited. It came down to two items: filename extensions, and a compiler. Everything else was just… boilerplate.

So I abstracted it out.

It’s now possible, by adding a few extra imports to your manage.py file, to build a Django app where:

  • Your views are written in Hy

  • Your models are written in Mochi

  • Your custom commands are written in Doge

And it all just works.

Leave a comment
I've been reading David Weber's Safehold series. I can't recommend it unless you're an absolute Weber fan; it's pretty much the worst Weber fanservice I can imagine all rolled into a single, many-millions-of-words-long story. It's not really a work of modern fiction; it's more like an fable or apologue, a "moral story" in which the good guys are emphatically good, the bad guys are emphatically bad, and everyone in between is meant to highlight a point on the scale between grace and damnation.

The premise is ridiculous: the human race has been Wiped Out by an alien species that hates innovative competitors. In a classic "they hate us for our freedoms" parable, the first book has Captain Langhorne order the last human colony ships to race as far away from the battlefield as they can, find a habitable world, and create a new set of colonists brainwashed to believe in a singular, monotheistic religion with a very Catholic-like heirarchal setting, but designed with sins and punishments that theorically will prevent humans from ever innovating again, to avoid coming to the attention of the aliens. Everything is taught in terms of "preserving men's souls into heaven." Not only were the colonists brainwashed to believe in Langhorne's religion, but for the first century colonists interacted with Archangels who flew and had clearly divine powers and they all wrote down their testimonies. There is no competing religion, no alternative viewpoint, at all in this world.

Our hero, Nimue, wakes up a few centuries later to discover she's been embodies as an almost indestructible robot by rebels who want the human race to be freed of Langhorne's restrictions and take the fight back to the aliens. By now, the world is circa 16th century Earth, with sailing ships and all the rest, and it lets Weber do his Napoleonic Wars thing all over again.

It is very silly; we're expected to buy that Nimue's influence has the world going from the Battle of Cape Celidonia (1616) at the end of the first book to Appotmattox (1865) at the end of the 8th book in less than ten years.

But one thing really bothers me: to map our history to his setting, we have secular rulers who are secular rulers first. While it's unthinkable that anyone would defy The Church, and the absolute Truth of the Church is unshakable to 99% of the world, the kings and princes act as if they're just church believers like anyone else, and their duty to their kingdoms starts and stops with keeping the people alive and healthy. The church is separate from the state. There's even a republic in this world, with elections!

Which is ridiculous. Langhorne and his cronies, the "Archangels" who set up this world, knew better. In a society like this, the monarch's reign is acknowledged by the church, and would be believed "Blessed by God," and given the duty above all else of the corporate "preservation" of the nation, accountable not to The Church, but to God himself.

There would be no republic, because a republic requires republicans, leaders accountable to the people and their petty wants, rather than to God. Schism would be far harder. The excuse that our England-analogue is "on the other side of the planets, months away by sea" and therefore harder to oversee isn't credible; the Langhorne's Church not only failed in its duty in this generation, it failed in all its generations, from the first when the Archangels were around, to the present.

That's just one of many problems with the series, but it's the most subtle and yet, the most glaring.

Still, the series does swash and buckle nicely, if you're into that sort of thing.

Tags: , ,
Current Mood: amused amused

2 comments or Leave a comment

There have always been attitudes that I don’t understand. James Hague’s recent Death of a Language Dilettante is one of them. Now, most people know that I’m a language dilettante; that I love everything new and shiny coming out of the programming language space, even if it does seem that in some ways that space has been tapped out, with languages becoming ever more blunt on one side with Go and Python, and ever more esoteric on the other with Agda and Idris.

James issues a challenge: “Give a language with a poor reputation (JavaScript, Perl) to someone who knows it passably well and–this is the key–has a strong work ethic. Let the language dilettante use whatever he or she wants, something with the best type system, hygenic macros, you name it. Give them both a real-world task to accomplish. My money is on the first person by a wide margin.”

Well, sure. Tooling matters. Experience matters. In the ${DAY_JOB} I write a ton of Javascript, and I write it more than passably well. I also write a ton of Python, again more than passably well. Those are two languages at which I consider myself working at an expert level.

On the other hand, I also love esoteric languages and, more importantly, every time I’ve worked in one, it’s made me a better programmer in the languages I use professionally. Learning Haskell made me really appreciate your responsibility for separating TheWorld from the program, and taught me the power of higher-order functions; learning Lisp made me really appreciate the power of expressions qua expressions in a way Haskell tried hard to make so simple you didn’t notice them. When you work in a “real world” language, those matters are present in everyday details, and knowing how to work with them is a gift, not a disease.

Right now, my “esoteric” language is Hy. Hy is an implementation of Lisp that produces a Python AST and runs on the Python VM. Hy implements its Python importlib layer incorrectly, and in a way that means certain Django features are unavailable to Hy developers, and I intend to fix it.

Unfortunately, Python’s importlib is pretty esoteric in its own right, and implementing a new pather/finder/loader has been a beast. But I’m closing in on a solution, and I intend to have it working soon.

And then I can do what I set out to do, which is write a Django app. Only I can do it in Hy.

And along the way, maybe get a new Django command, newhyproject, out of it, as well as newhyapp and maybe newhymigrations. But those are stretch goals. Right now, all I want is for Hy to interoperate with Django correctly, and then I can think about future projects.

So I’ll take James’s challenge, but I’ll have to say: “Both is good.”

Leave a comment

There’s a comment I get from my peers from time to time that boils down to: “Your code is tight and expressive, and the names are well-chosen, but it’s ugly and it makes reviewers uncomfortable.” And that’s when I realized that functional programming, by its very nature, causes us to write visually unattractive code.

The human eye likes rhythm. Aesthetically pleasing visual arrangements often involve creating a sense of flow, a sense of time, and a sense of transition from one scene to the next. Repetition is a key element of beauty. Artists and musicians come up with motifs and set up the expectation of repetition, and then maintain, modify, or diverge from the repetition into a new landscape where a new repetitious motif must be established and maintained in order to sustain the impression of beauty.

This is utterly in contrast to DRY: Don’t Repeat Yourself.

Well-constructed, highly functional source code written in a traditional enterprise language like Javascript, Python or C++ has very little repetition. It’s not supposed to. The comforting rhythm of a long if/else tree is broken out into a lookup table. The comforting rhythm of long transformations is itself transformed into a brutally compact map/reduce. As we try to make the syntax of the language itself more succinct (the Javascript keyword function has been brutally shortened to =>, for one very recent example), familiar “hooks” onto which the eye might rest and understand become harder to find.

It’s not all this bad. Well-written Haskell is actually very pretty to look at, but it’s also daunting to realize that every word on the page is dense with meaning.  It’s also significant that Haskell’s organization within a source file isn’t sequential; the compiler hooks up all the definitions in the correct order regardless of their location.  This gives the developer greater freedom to organize code into meaningful units, but it can also mean that understanding the file isn’t as simple as reading it in order from top to bottom.

That sense of intimidation is even greater when you apply it to more traditional languages that can support a functional style. A language like Javascript can support strong functional idioms, but in doing so, you end up writing the code equivalent of Shostakovitch and the “difficult listening hour.”

Leave a comment
The United States will never grow up.

I don't mean that in the sense that we're primed to elect Donald Trump to the presidency, or that we're still way too goddamn squeamish about sex. I mean that in the sense that we in the USA will never get our maturity fast enough to treat ourselves fairly.

In every fully mature country in the world, countries like Sweden, Japan, Norway, and South Korea, poverty is understood to be a contingency: it happened not because there was something wrong with you, or even with your parents, but is a result of a great many things, most of them historical in nature. Where you were born, the quality of schooling available there, the quality of food available there, the prevalence of clean water, the wealth, knowledge, and habits of thought your parents inherited from their parents, all of them contributed to your ability to navigate the modern world effectively and to your advantage. Yes, innate ability is a part of it, but only a tiny part, a fifth of all the influnces at best.

Ijeoma Oluo captures American attitudes toward this reality perfectly. Americans internalize early on, young in life, that innate ability is the only thing that matters: everything else can be overcome if only you're good enough, strong enough, Darwinian enough. And if you aren't, you deserve the shit life pour on you. You don't deserve a single goddamn good thing in life. (There was another beautiful story about a young boy growing up in poverty whose brother saved every penny he could to buy the kid a Nintendo handheld game, but the kid could never bring himself to play it. He felt ashamed to have something so shiny when his family was so poor. I wish I could find it again.)

Oluo says that every pleasure poor people buy is turned into a form of self-loathing. She hated her mother for buying an indulgence, and now she regrets the hatred American culture heaped upon them both for their shame.

So when Timothy B. Lee points out, we're all in that same boat, what are we gonna do?

It's popular in lefty circles to claim that capitalism requires constant growth, which is a quadratic factor on consumption: that is, if growth continutes as a static percentage, consumption must increase exponentially. The Earth is a finite resource: exponential growth will ultimately overwhelm the carrying capacity of the Earth and doom us all.

We may already be there.

If we're not doomed, let's press on. It's popular in righty circles to argue that the lefty claim is wrong: what people want changes over time, and that can be expressed as "growth" that isn't really consumption. After all, we want more and more exciting electronics, but they consume surprisingly smaller amounts of electricity as they get more and more efficient at meeting our needs. We want better and more wonderful houses, but they're also marvels of efficiency and insulation. "Growth" and "Consumption as a factor of the Earth's carrying capacity" are separate and unrelated, according to this view of economics.

I would call bullshit on this stance, but doing so may actually be completely unnecessary. Something more significant is at play nowadays: The end of significant technological improvements.

(Yes, I'm perfectly aware of then-Commissioner of Patents Charles Duell's infamous "quote" in 1899 that "Everything that can be invented has been." Duell never said it. In fact, Duell wished he would live longer to see what the 20th century would bring forth. I wish Duell were still with us; I'd love to share a drink with the man.)

As economist Timothy B. Lee (no relation to "Sir" Timothy, I believe) points out, outside of IT, medicine, transportation, we are out of room for technological improvements. The low-hanging fruits of food distribution, clothing, and housing have all been super-saturated.

I'm not sure Lee's inclusion of transportation is even vaild. Transportation of people is actually a political problem.1. Intermodal cross-ocean transportation of heavy goods has other technologies that are only now starting to come on-line as shippers look for ways to reduce costs.

Medicine and IT still have a long way to go, but I believe they're the only spaces really left. Services may have a long way to go in the US, but that linked-to article isn't really far off (I'm already ⅓ of the way there myself). Medicine and IT are also, like IT and transportation (or IT and dating!), about to become one and the same.

But if, outside of medicine, all economics becomes signalling economics ("I'm richer than you are, na-na na-na boo-boo") (Gods, I love that there's a Wikipedia entry for that!), then what for the impoverished? Especially if you're disciplined enough, or simply not interested, that you have time to spare? Some guys may be busy fucking their sexbots, and others may just be lost in virtual realities, but short of the Introdus, there's going to be an awful lot of people who haven't had the luck, circumstance, or (circumstantially-given) will to acquire the resources or (if they have the resources) the interest to play the signalling game.

What becomes of them?

Jacobin has four possbilities. I would love for "Egalitarianism and Abundance," but in a world of utter abundance, those in power would chafe having to share their world with the likes of you and me.

Which is why I say America will never grow up. America is the perfect place for "Hierarchy and Abundance" to emerge, because those who know Egalitarianism is the right outcome would never admit it, and those who chafe under Hierarchy have spent their whole lives not merely accepting it, but promulgating it not only to others, but to themselves.

1 Transportation is such a solved problem that I have had some thoughts about how to exploit it.

Tags: ,
Current Mood: drunk drunk

2 comments or Leave a comment
Does anyone else grok that the Hyperloop is really Elon Musk's clever way of exploiting America's weird transportation politics for his own uses?

The Hyperloop is being sold to politicians and tech bros as a sweet, quick way to get from San Francisco to Los Angeles in the time it takes to get from the Financial District to Japantown by the San Francisco MTBA, when in fact it's a way to test out the viability of the Gigafactory's power supplies united with linear induction motors to power a Clarke Mass Driver.

Elon Musk never does anything without getting multiple bits of data out of it. Self driving cars are a way to test various machine learning techniques against geography until the best one presents itself. The barge landings involve a hard restart in re-entry conditions of a thin, violently fast atmosphere, the kind encountered when landing on Mars. Hyperloop is a way of testing the viability of linear induction motors for insane acceleration rates with heavy payloads.

Hyperloop is how Elon Musk has tricked investors into giving him the money to experiment with yet another way to lower the weight and cost of his first-stage rockets. They may, with some probability, get the trains their Atlas Shrugged-addled minds want. Musk will, with all probability, figure out if LIMs are a way to boost a launch vessel to a significant fraction of escape velocity without having to have the fuel on board the launch vehicle.

Current Mood: drunk drunk

Leave a comment
As an amateur futurologist (no, really!), it often falls to me to compare and contrast the news of the day in order to figure out if what's being said jibes with what I believe is coming in the next twenty years. I expect to live another twenty years, and I expect to be able to talk about whether or not my predictions will come true.

Most of the job of a futurologist isn't really to predict the future. It's to take other people's off-the-cuff comments and decide whether or not they're indicative of something more interesting. My favorite example: in 1982, Elaine Lee's Starstruck was published as a graphic novel. In one scene there's a background in which an advertisement is playing. Starstruck was absolutely prescient that video and holographic advertisements would be ubiquitous and annoying, but it was the content that got my attention.

Now, you have to understand that Starstruck was one of the seminal works of my own writing career. "Living Doll Cybernetics" and their production of non-sentient love robots, especially the way it dealt with emergent sentience and second-hand robotics, was super-significant to my own ideas about human/robot relations in a way C3PO could never be. But "Living Doll" specialized specifically in non-sentient toys because, as their slogan put it, Why fuck something with a mind of its own?

I read that and thought, "Welp, I know where the future is going."

I wasn't wrong. Here we are, 35 years after Starstruck, and Elaine Lee's prediction is spot-on. Widespread distribution of Internet pornography, in all its incarnations from the most feminist to the most abusively gonzo, has actually led to a reduction in sex assaults. The evidence is solid: pornography gives men who would commit sexual assault an alternative outlet that sometimes alleviates their criminal impulses. Violent men really would sometimes rather masturbate to images they can't even influence than go through the trouble of finding a victim.

At the same time, Men's Rights Activists predict that when apparently submissive, enthusiastic, lissome, docile and deferent sexbots appear on the market, women will be "sorry." No real men, MRAs argue, really wants to fuck a woman with a mind of her own. And they may be right.

Even more prominent: are you aware of the psychological phenomenon known as Presence?. When you read a book or watch a movie, you have to consciously be willing to suspend your disbelief. No such suspension is necessary in a sufficiently powerful virtual reality: in fact, it's hard to suspend disbelief. The Samsung Gear isn't good enough, but the Occulus Rift easily reaches this state.

Headsets will get lighter. Smaller. Not only will the sexbot you buy be physically fulfilling, but an augmented reality overlay will change her face and her body type so readily you won't need to buy more than two of them to literally have a harem of hundreds.

The fetishists of the future will be those who want to fuck their fellow flesh-and-bloods.

Tags: ,
Current Mood: drunk drunk

3 comments or Leave a comment

There was an article highlighted in a variety of nerdy news sites called “I’m A Good Engineer But I Suck At Building Stuff,” and while I was reading it, I felt the writer Lionel Barrow’s pain.

Because I’m a better engineer, but I still suck at building (some) stuff. When Barrow writes: “I find myself instantly criticizing my technique, to the point of paralysis. This function is hard to test; this object’s dependencies need to be injected rather than initialized internally; that module needs an integration test; and so on and so forth,” I don’t sympathize. Not only do I know how to do all those things, I know how to avoid them. A hard-to-test function should be broken down into public components; injected dependencies shouldn’t be a difficulty, they should absolutely be the starting point of your composition; integration tests are mastered by giving your interface the smallest possible surface area. When Barrow writes, “Even when writing spike or proof of concept code, I find myself revisiting the same lines over and over again, looking for the best, most natural expression of the ideas it contains — obsessing over my own construct, rather than on the thing my code does,” I remind my self that the very first thing my code does is communicate with other developers. Code is meant to be read first and compiled second. Yeah, maybe if you’re doing a prototype you can cut that corner once or twice, but get it out of your habit.

Where I work, my fellow engineers have a joke about “Elf’s de-cleverizing pass.” I work in Javascript but am a Lisp programmer at heart, and I start by writing everything in terms of transformers and partials wrapped inside and around map/filter/reduce, and then at the end I will remove the partials and break out the transformers into functions and make the code look like something my peers expect. Most modules I write end up as long, easily-followable lists of var expressions that end with a single side effect, because that’s what all code is: an expression that ultimately has some real-word consequence.

On the other hand, I feel Barrow’s pain because I, too, suck at building stuff. Modifying other people’s code is easy. Even if I’ve been tasked with building “something entirely new” for my employer, the skeleton of every application we’ve ever built is in our repository, and if I can find something that does 51% of what my new task calls for, the rest is more or less fait accompli.

But when it comes to my own projects, I get analysis paralysis. I wanted to write a simple compiler with a few ideas of my own, something that would make my life as a developer easier, at least for a few follow-on projects, and …

There’s way too much to know about compilers. Stephen Diehl’s wonderful (if woefully incomplete) articles on “Write You A Haskell,” James Longster’s expedition into his Outlet project, all the effort I put into Lisp In Small Pieces, parsing, hygiene and scope, type systems (oy! Type systems!), compiling with continuations… My eyes start to glaze over and I get this terror that I’m never going to be smart enough or have time enough to do anything at all. The size of my ambitions overwhelms me. I lose track of what I’m trying to do as yet another article on CPS Transforms and Nanopass Compilation comes to my attention.

The twin impulses of wanting something useful now and wanting to know everything now are, rather than helping each other, constantly at war with time itself. I’m sure it’s an ADHD problem that no amount of Adderall is going to handle, because it’s not about attention, it’s about desire. I never have this problem professionally, because when it comes to money both what must be done and what I desire to be done are more or less well-aligned. It’s about the projects I’m really passionate about; I don’t want to screw them up, and there’s no pressure to get them out the door. I really don’t know how to make these impulses play well together.

Leave a comment
So, I turned 50. Omaha and I have a rule: I'm not allowed to buy myself anything within a few weeks of any gift-giving holiday. I do that a lot. I do keep an Amazon wish list, but nobody pays attention to that so I only put on it things I'd like, not things I need.

But I decided to buy myself something today anyway. I bought myself a surge suppressor.


But it's really a treat. I bought it to replace the one under my bed, a Belkin branded thing which had sucked mightily. The sockets had zero grip; things would just fall out with the slightest provocation. The new one is GE-branded and comes with two USB ports built-in, meaning I now have two chargers freed up for other duties. The space next to my bed is slightly more organized, and my watch, phone and laptop will all be fully charged when I get up in the morning. At least twice in the past year the cat has knocked something loose going under the bed, meaning I woke up to my phone or watch at 25% battery and falling fast.

It's a 1% improvement. That's all it is. A few seconds out of my day that I don't have to waste anymore worrying about whether or not my personal charging station is working. (I do worry the thing is a bit of a vampire, since even when nothing's plugged in or turned on there are two LEDs on it telling me it's working correctly, but the old one had that too.)

But a hundred 1% improvements could end up doing a lot.

Current Mood: amused amused

1 comment or Leave a comment

While I can’t comment too strongly on the performance aspects of this article on Green Thread performance, there’s a paragraph at the beginning that caught my attention:

If one looks closely enough at green threads, such as Go’s ‘go’ statement, it’s not hard to realise that they are really a flow control mechanism similar to ‘if’, ‘for’ or ‘while’ statements. Where ‘if’ allows you to skip a block of code, green threads give you a way to easily switch between different points of execution. Very much like ‘if’ or ‘while’ they are glorified jump statements.

To which my initial reaction was, “Yes, that was covered in Chapter 3 of Lisp In Small Pieces.”

In fact, everything you need to know about control flow is covered in Chapter 3 of Christian Queinnec’s amazing, now 20-year-old book. In terms of control flow mechanisms for programs, we haven’t invented anything new since that book came out. If you’ve read Lisp in Small Pieces and Why Functional Programming Matters, you’ve read everything you need to create modern, composable, performant software. The biggest insight in LiSP is that all of these mechanisms for code parallelization, threads, communication modes, concurrency, and so forth are exactly the same as “if”; everything else is (important) details, like performance, but the underlying nature of programming is unchanged: there is one and only one instruction that matters: conditional jump. All else is commentary.

Necessary commentary. Abstractions necessary to constraining the development process in order to eliminate error. But still, commentary; when someone expresses “surprise” that Go’s go is semantically similar to if, it tells me that they’ve never taken, or somehow missed, the fundamentals taught in their CS program.

1 comment or Leave a comment