Robust and Race-free Server Logging using Named Pipes

Published

Tags: , , , and

If you do any server administration work, you’ll have worked with log files. And if your servers need to be reliable, you’ll know that log files are common source of problems, especially when you need to rotate or ship them (which is practically always). In particular, moving files around causes race conditions.

Thankfully, there are better ways. With named pipes, you can have a simple and robust logging framework, with no race conditions, and without patching your servers to support some network logging protocol.

D Declarations for C and C++ Programmers

Published

Tags: , , and

Because D was originally created by a C++ compiler writer, Walter Bright, it’s an easy language for C and C++ programmers to learn, but there are little differences in the way declarations work. I learned them piecemeal in different places, but I’m going to dump a bunch in this one post.

Leaving Google: Five Years On

Published

Tags: , , and

About five years ago now, I handed in my Google employee badge and walked out of the Sydney Google office to start a new life of self-employment. I figured I should write up this story because I got a lot out of reading Michael Lynch’s. As you can see, it’s still taken me a couple of years to get around to writing this post, but I finally told myself that if I don’t write it for the fifth anniversary, I never will.

This post is kind of long, but I hope it has something useful for new developers who are interested in working at a big tech company, or for big company employees who are wondering what it’s like to quit. I’ll talk about my story of getting into, working at and quitting Google, and what I’ve done since. Feel free to ask if you want more detail about something, though I already have a lot of blog posts to write, so I can’t promise anything in-depth straight away.

Also, at the risk of labouring the obvious: I haven’t worked at Google for five years, so don’t take this story as a literal description of Google today or what all Google employees experience. However, I think a lot of it’s still relevant to tech careers in general.

Tricks with Pseudorandom Number Generators

Published

Tags: and

Pseudorandom number generators (PRNGs) are often treated like a compromise: their output isn’t as good as real random number generators, but they’re cheap and easy to use on computer hardware. But a special feature of PRNGs is that they’re reproducible sources of random-looking data:

import std.random;
import std.stdio;

void main()
{
    // Seed a PRNG and generate 10 pseudo-random numbers
    auto rng = Random(42);
    foreach (_; 0..10) write(uniform(0, 10, rng), ' ');
    writeln();
    // Reset the PRNG, and the same sequence is generated again
    rng = Random(42);
    foreach (_; 0..10) write(uniform(0, 10, rng), ' ');
    writeln();

    // Output:
    // 2 7 6 4 6 5 0 4 0 3
    // 2 7 6 4 6 5 0 4 0 3
}

This simple fact enables a few neat tricks.

PSA: Please Just Close Your Files

Published

Tags: and

An article called “Contrarian view on closing files” is on the front page of Hacker News right now with over 60 upvotes. It starts off with a quote from the Google Python style guide:

Explicitly close files and sockets when done with them. Leaving files, sockets or other file-like objects open unnecessarily has many downsides […]

The article’s main complaint is that “this advice is applying a notably higher standard of premature optimization to file descriptors than to any other kind of resource”. It complains that it’s “depressingly commonplace” to see code like this:

with open("README.md") as readme:
    long_description = readme.read()

Sure, if it’s a one-off read of a doc file, you can almost certainly get away with just open("README.md").read(), but I honestly have no idea what’s depressing about code that just works reliably.

Leaving files and sockets open is something that you can usually get away with, until weird stuff happens. And, no, it’s not just about running out of file descriptors (although that does happen, too). I’ve previously written about servers that mysteriously ran out of disk space because (spoiler) they were deleting files that hadn’t been closed properly. Regardless of how awesome your computer and network equipment are, the number of TCP connections you can make to a given host and port are limited by a 16 bit integer (and practically always limited more by your network settings), then you get network failures. Writing to files that you don’t explicitly close (or flush) is especially dicey — the write might actually happen straight away, then on another day or another environment it might get delayed.

Sure, these failure modes aren’t very common, but they definitely happen. I can think of three examples right now from the past few years of my day job. Closing files as a habit is easier than figuring out when it’ll go wrong.

The article has an example of a function that lazily loads JSON given a file path string. Sure, it doesn’t work properly if you close the file inside the function, but I’d say the problem is in the API design: the file handle resource straddles the interface.

The first good alternative is to keep the file handle completely inside: take a path argument, load the data eagerly, close the file and return the parsed data. The other good alternative is to keep the file handle completely outside: take a file handle as argument and return it wrapped in a lazy JSON parser. Either way makes it easier to see how and where the file should be closed.

The Google advice is pretty solid: when you’re done with a file or socket, just close it. I’ll add: sure, maybe it won’t always be obvious when you’re done with a handle, but perhaps that code design is making life more “exciting” than necessary. Production failures are more depressing than file closing code ever will be.

Scaling a GraphQL Website

Published

Tags: , , and

For obvious reasons, I normally write abstractly about work I’ve done for other people, but I’ve been given permission to write about a website, Vocal, that I did some SRE work on last year. I actually gave a presentation at GraphQL Sydney back in February, but for various reasons it’s taken me this long to get it into a blog post.

Vocal is a GraphQL-based website that got traction and hit scaling problems that I got called in to fix. Here’s what I did. Obviously, you’ll find this post useful if you’re scaling another GraphQL website, but most of it’s representative of what you have to deal with when a site gets enough traffic to cause technical problems. If website scalability is a key interest of yours, you might want to read my recent post about scalability first.

Glico (Weighted Rock Paper Scissors)

Published

Tags: , , and

This still isn’t the blog post I said I was going to write about now, but I figured some game theory would make a good post at the moment, especially when a lot of people I know are working at home with kids who need entertaining. Here’s some stuff about a traditional Japanese kids’ game called Glico, a form of weighted Rock Paper Scissors (RPS).

What is a High Traffic Website?

Published

Tags: , and

Terms like “high traffic” are hazardous when designing online services because salespeople, business analysts and engineers all have different perspectives about what they mean. If we’re talking about, say, a high-stakes online poker room, then “high traffic” for the business side will be very low compared to what it is for the technical side. However, all these people will be in a meeting room together making decisions, using the same words to mean different things. It’s obvious how that can lead to bad (and sometimes expensive) choices.

A lot of my day job is talking to business stakeholders and figuring out the technical solutions they need, so this is a problem I have to deal with. So I’ve got my own purely technical way to think about traffic levels for online services.

Update

Published

Tags:

So, I haven’t published any posts since the end of January, but I’m still around. I’m currently in Sydney, where many shops and most offices are closed because of COVID-19. The pandemic has disrupted a lot of my plans for this year (including DConf), but I’m hoping to get back into blogging at least.

There’s a post I’ve been wanting to get done since February, but I’ve split off some of it into a separate post I’m publishing now. Hopefully I’ll get the main post out a few weeks later.

Best wishes wherever you are. Right now I’m feeling pretty lucky to be in the IT industry, even though a lot of us are losing jobs, too. If you happen to know anyone working in health, or in warehouses, or emergency services, or as drivers, or as cleaners, or in other services, let them know there are people out there who have respect and gratitude for the work they’re doing right now.

Some Useful Probability Facts for Systems Programming

Published

Tags: , and

Probability problems come up a lot in systems programming, and I’m using that term loosely to mean everything from operating systems programming and networking, to building large online services, to creating virtual worlds like in games. Here’s a bunch of rough-and-ready probability rules of thumb that are deeply related and have many practical applications when designing systems.