Tricks with Pseudorandom Number Generators


Tags: and

Pseudorandom number generators (PRNGs) are often treated like a compromise: their output isn’t as good as real random number generators, but they’re cheap and easy to use on computer hardware. But a special feature of PRNGs is that they’re reproducible sources of random-looking data:

import std.random;
import std.stdio;

void main()
    // Seed a PRNG and generate 10 pseudo-random numbers
    auto rng = Random(42);
    foreach (_; 0..10) write(uniform(0, 10, rng), ' ');
    // Reset the PRNG, and the same sequence is generated again
    rng = Random(42);
    foreach (_; 0..10) write(uniform(0, 10, rng), ' ');

    // Output:
    // 2 7 6 4 6 5 0 4 0 3
    // 2 7 6 4 6 5 0 4 0 3

This simple fact enables a few neat tricks.

PSA: Please Just Close Your Files


Tags: and

An article called “Contrarian view on closing files” is on the front page of Hacker News right now with over 60 upvotes. It starts off with a quote from the Google Python style guide:

Explicitly close files and sockets when done with them. Leaving files, sockets or other file-like objects open unnecessarily has many downsides […]

The article’s main complaint is that “this advice is applying a notably higher standard of premature optimization to file descriptors than to any other kind of resource”. It complains that it’s “depressingly commonplace” to see code like this:

with open("") as readme:
    long_description =

Sure, if it’s a one-off read of a doc file, you can almost certainly get away with just open("").read(), but I honestly have no idea what’s depressing about code that just works reliably.

Leaving files and sockets open is something that you can usually get away with, until weird stuff happens. And, no, it’s not just about running out of file descriptors (although that does happen, too). I’ve previously written about servers that mysteriously ran out of disk space because (spoiler) they were deleting files that hadn’t been closed properly. Regardless of how awesome your computer and network equipment are, the number of TCP connections you can make to a given host and port are limited by a 16 bit integer (and practically always limited more by your network settings), then you get network failures. Writing to files that you don’t explicitly close (or flush) is especially dicey — the write might actually happen straight away, then on another day or another environment it might get delayed.

Sure, these failure modes aren’t very common, but they definitely happen. I can think of three examples right now from the past few years of my day job. Closing files as a habit is easier than figuring out when it’ll go wrong.

The article has an example of a function that lazily loads JSON given a file path string. Sure, it doesn’t work properly if you close the file inside the function, but I’d say the problem is in the API design: the file handle resource straddles the interface.

The first good alternative is to keep the file handle completely inside: take a path argument, load the data eagerly, close the file and return the parsed data. The other good alternative is to keep the file handle completely outside: take a file handle as argument and return it wrapped in a lazy JSON parser. Either way makes it easier to see how and where the file should be closed.

The Google advice is pretty solid: when you’re done with a file or socket, just close it. I’ll add: sure, maybe it won’t always be obvious when you’re done with a handle, but perhaps that code design is making life more “exciting” than necessary. Production failures are more depressing than file closing code ever will be.

The Enterprise Content Management System


Tags: , , and

A few years ago I worked on the version 2 of some big enterprise’s internal website. A smaller company had the contract, and I’d been subcontracted to deal with deployment and any serverside/backend changes.

The enterprise side had a committee to figure out lists of requirements. Committees are famously bad at coming up with simple and clear specs, and prone to bikeshedding. Thankfully, the company I was contracting with had a project manager who had the job of engaging with the committee for hours each day so that the rest of us didn’t have to. However, we still got a constant stream of inane change requests. (One particular feature of the site changed name three times in about two months.)

It was pretty obvious early on what was happening, so I integrated the existing website backend with a content management system (CMS) that had an admin panel with a friendly WYSIWYG editor. New features got implemented as plugins to the CMS, and old features got migrated as needed. We couldn’t make everything customisable, but eventually we managed to push back on several change requests by saying, “You can customise that whenever you want through the admin panel.”

So, we got things done to satisfaction and delivered, but there was one complication: using the admin panel and WYSIWYG editor. The committee members wouldn’t use it because they were ideas people and didn’t implement anything. The company had IT staff who managed things like websites, but they were hired as technical staff, not for editing website content. On the other hand, they had staff hired for writing copy, but they weren’t hired as website administrators.

So here’s how they ended up using the CMS: CMS data would get rendered as HTML by the website backend, which would then be exported to PDF documents by IT staff. The PDF documents would be converted to Word documents and sent to the writers via email. The writers would edit the documents and send them back to the IT staff, who would do a side-by-side comparison with the originals and then manually enter the changes through the graphical editor in the admin panel. All of the stakeholders were delighted to have a shiny version 2 of the website that had a bunch of new features, was highly customisable, integrated well with their existing processes and was all within budget.

Nowadays, when I’m designing something and I think it’s obvious how it will be used, I remind myself about that CMS and its user-friendly, graphical editor.

Object-Oriented Programming and Essential State


Tags: and


Back in 2015, Brian Will wrote a provocative blog post: Object-Oriented Programming: A Disaster Story. He followed it up with a video called Object-Oriented Programming is Bad, which is much more detailed. I recommend taking the time to watch the video, but here’s my one-paragraph summary:

The Platonic ideal of OOP is a sea of decoupled objects that send stateless messages to one another. No one really makes software like that, and Brian points out that it doesn’t even make sense: objects need to know which other objects to send messages to, and that means they need to hold references to one another. Most of the video is about the pain that happens trying to couple objects for control flow, while pretending that they’re decoupled by design.

Overall his ideas resonate with my own experiences of OOP: objects can be okay, but I’ve just never been satisfied with object-orientation for modelling a program’s control flow, and trying to make code “properly” object-oriented always seems to create layers of unneccessary complexity.

There’s one thing I don’t think he explains fully. He says outright that “encapsulation does not work”, but follows it with the footnote “at fine-grained levels of code”, and goes on to acknowledge that objects can sometimes work, and that encapsulation can be okay at the level of, say, a library or file. But he doesn’t explain exactly why it sometimes works and sometimes doesn’t, and how/where to draw the line. Some people might say that makes his “OOP is bad” claim flawed, but I think his point stands, and that the line can be drawn between essential state and accidental state.

Data Still Dominates


Tags: , and

Translations: русский

Here’s a quote from Linus Torvalds in 2006:

I’m a huge proponent of designing your code around the data, rather than the other way around, and I think it’s one of the reasons git has been fairly successful… I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important. Bad programmers worry about the code. Good programmers worry about data structures and their relationships.

Which sounds a lot like Eric Raymond’s “Rule of Representation” from 2003:

Fold knowledge into data, so program logic can be stupid and robust.

Which was just his summary of ideas like this one from Rob Pike in 1989:

Data dominates. If you’ve chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.

Which cites Fred Brooks from 1975:

Representation is the Essence of Programming

Beyond craftmanship lies invention, and it is here that lean, spare, fast programs are born. Almost always these are the result of strategic breakthrough rather than tactical cleverness. Sometimes the strategic breakthrough will be a new algorithm, such as the Cooley-Tukey Fast Fourier Transform or the substitution of an n log n sort for an n2 set of comparisons.

Much more often, strategic breakthrough will come from redoing the representation of the data or tables. This is where the heart of your program lies. Show me your flowcharts and conceal your tables, and I shall be continued to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.

So, smart people have been saying this again and again for nearly half a century: focus on the data first. But sometimes it feels like the most famous piece of smart programming advice that everyone forgets.

Let me give some real examples.

Hello World Marketing (or, How I Find Good, Boring Software)


Tags: , , , and

Back in 2001 Joel Spolsky wrote his classic essay “Good Software Takes Ten Years. Get Used To it”. Nothing much has changed since then: software is still taking around a decade of development to get good, and the industry is still getting used to that fact. Unfortunately, the industry has investors who want to see hockey stick growth rates on software that’s a year old or less. The result is an antipattern I like to call “Hello World Marketing”. Once you start to notice it, you see it everywhere, and it’s a huge red flag when choosing software tools.

Unfortunately, Garbage Collection isn't Enough


Here’s a little story of some mysterious server failures I had to debug a year ago. The servers would run okay for a while, then eventually start crashing. After that, trying to run practically anything on the machines failed with “No space left on device” errors, but the filesystem only reported a few gigabytes of files on the ~20GB disks.

Using D Features to Reimplement Inheritance and Polymorphism


Tags: , and

Some months ago I showed how inheritance and polymorphism work in compiled languages by reimplementing them with basic structs and function pointers. I wrote that code in D, but it could be translated directly to plain old C. In this post I’ll show how to take advantage of D’s features to make DIY inheritance a bit more ergonomic to use.

Although I have used these tricks in real code, I’m honestly just writing this because I think it’s neat what D can do, and because it helps explain how high-level features of D can be implemented — using the language itself.

How Inheritance and Polymorphism Work


Tags: , , and

I’ve promised to write a blog post about the DIY polymorphic classes implementation in Xanthe, the experimental game I once wrote for bare-metal X86. But first, I decided to write a precursor post that explains how polymorphism and inheritance work in the first place.

Relational Databases Considered Incredibly Useful


Tags: , , , and

A year ago I worked on a web service that had Postgres and Elasticsearch as backends. Postgres was doing most of the work and was the primary source of truth about all data, but some documents were replicated in Elasticsearch for querying. Elasticsearch was easy to get started with, but had an ongoing maintenance cost: it was one more moving part to break down, it occasionally went out of sync with the main database, it was another thing for new developers to install, and it added complexity to the deployment, as well as the integration tests. But most of the features of Elasticsearch weren’t needed because the documents were semi-structured, and the search queries were heavily keyword-based. Dropping Elasticsearch and just using Postgres turned out to work okay. No, I’m not talking about brute-force string matching using LIKE expressions (as implemented in certain popular CMSs); I’m talking about using the featureful text search indexes in good modern databases. Text search with Postgres took more work to implement, and couldn’t do all the things Elasticsearch could, but it was easier to deploy, and since then it’s been zero maintenance. Overall, it’s considered a net win (I talked to some of the developers again just recently).