How Real-World Apps Lose Data

Published

Tags: , and

A great thing about modern app development is that there are cloud providers to worry about things like hardware failures or how to set up RAID. Decent cloud providers are extremely unlikely to lose your app’s data, so sometimes I get asked what backups are really for these days. Here are some real-world stories that show exactly what.

Reverse Engineering a Docker Image

Published

Tags: , and

This started with a consulting snafu: Government organisation A got government organisation B to develop a web application. Government organisation B subcontracted part of the work to somebody. Hosting and maintenance of the project was later contracted out to a private-sector company C. Company C discovered that the subcontracted somebody (who was long gone) had built a custom Docker image and made it a dependency of the build system, but without committing the original Dockerfile. That left company C with a contractual obligation to manage a Docker image they had no source code for. Company C calls me in once in a while to do various things, so doing something about this mystery meat Docker image became my job.

Fortunately, the Docker image format is a lot more transparent than it could be. A little detective work is needed, but a lot can be figured out just by pulling apart an image file. As an example, here’s a quick walkthrough of an image for the Prettier code formatter. (In fact, it’s so easy, there’s a tool for it. Thanks Ezequiel Gonzalez Rial.)

Extending Looped Music for Fun, Relaxation and Productivity

Published

Tags: , , and

Some work (like programming) takes a lot of concentration, and I use noise-cancelling headphones to help me work productively in silence. But for other work (like doing business paperwork), I prefer to have quiet music in the background to help me stay focussed. Quiet background music is good for meditation or dozing, too. If you can’t fall asleep or completely clear your mind, zoning out to some music is the next best thing.

The best music for that is simple and repetitive — something nice enough to listen too, but not distracting, and okay to tune out of when needed. Computer game music is like that, by design, so there’s plenty of good background music out there. The harder problem is finding samples that play for more than a few minutes.

So I made loopx, a tool that takes a sample of music that loops a few times, and repeats the loop to make a long piece of music.

When you’re listening to the same music loop for a long time, even slight distortion becomes distracting. Making quality extended music audio out of real-world samples (and doing it fast enough) takes a bit of maths and computer science. About ten years ago I was doing digital signal processing (DSP) programming for industrial metering equipment, so this side project got me digging up some old theory again.

Djinn: A Code Generator and Templating Language Inspired by Jinja2

Published

Tags: and

Code generators can be useful tools. I sometimes use the command line version of Jinja2 to generate highly redundant config files and other text files, but it’s feature-limited for transforming data. Obviously the author of Jinja2 thinks differently, but I wanted something like list comprehensions or D’s composable range algorithms.

I decided to make a tool that’s like Jinja2, but lets me generate complex files by transforming data with range algorithms. The idea was dead simple: a templating language that gets rewritten directly to D code. That way it supports everything D does, simply because it is D. I wanted a standalone code generator, but thanks to D’s mixin feature, the same templating language works as an embedded templating language (for HTML in a web app, for example). (For more on that trick, see this post about translating Brainfuck to D to machine code all at compile time using mixins.)

As usual, it’s on GitLab. The examples in this post can be found there, too.

Woothee (HTTP User Agent Parser)

Published

Tags: , , and

I’ve written a D implementation of the Project Woothee multi-language HTTP user agent parser. Here are some notes about what it’s useful for, and a few things special about the D implementation.

Pi from High School Maths

Published

Tags: and

Warning: I don’t think the stuff in this post has any direct practical application by itself (unless you’re a nuclear war survivor and need to reconstruct maths from scratch or something). Sometimes I like to go back to basics, though. Here’s a look at π \pi and areas of curved shapes without any calculus or transcendental functions.

Robust and Race-free Server Logging using Named Pipes

Published

Tags: , , , and

If you do any server administration work, you’ll have worked with log files. And if your servers need to be reliable, you’ll know that log files are common source of problems, especially when you need to rotate or ship them (which is practically always). In particular, moving files around causes race conditions.

Thankfully, there are better ways. With named pipes, you can have a simple and robust logging stack, with no race conditions, and without patching your servers to support some network logging protocol.

D Declarations for C and C++ Programmers

Published

Tags: , , and

Because D was originally created by a C++ compiler writer, Walter Bright, it’s an easy language for C and C++ programmers to learn, but there are little differences in the way declarations work. I learned them piecemeal in different places, but I’m going to dump a bunch in this one post.

Leaving Google: Five Years On

Published

Tags: , , and

About five years ago now, I handed in my Google employee badge and walked out of the Sydney Google office to start a new life of self-employment. I figured I should write up this story because I got a lot out of reading Michael Lynch’s. As you can see, it’s still taken me a couple of years to get around to writing this post, but I finally told myself that if I don’t write it for the fifth anniversary, I never will.

This post is kind of long, but I hope it has something useful for new developers who are interested in working at a big tech company, or for big company employees who are wondering what it’s like to quit. I’ll talk about my story of getting into, working at and quitting Google, and what I’ve done since. Feel free to ask if you want more detail about something, though I already have a lot of blog posts to write, so I can’t promise anything in-depth straight away.

Also, at the risk of labouring the obvious: I haven’t worked at Google for five years, so don’t take this story as a literal description of Google today or what all Google employees experience. However, I think a lot of it’s still relevant to tech careers in general.

Tricks with Pseudorandom Number Generators

Published

Tags: and

Pseudorandom number generators (PRNGs) are often treated like a compromise: their output isn’t as good as real random number generators, but they’re cheap and easy to use on computer hardware. But a special feature of PRNGs is that they’re reproducible sources of random-looking data:

import std.random;
import std.stdio;

void main()
{
    // Seed a PRNG and generate 10 pseudo-random numbers
    auto rng = Random(42);
    foreach (_; 0..10) write(uniform(0, 10, rng), ' ');
    writeln();
    // Reset the PRNG, and the same sequence is generated again
    rng = Random(42);
    foreach (_; 0..10) write(uniform(0, 10, rng), ' ');
    writeln();

    // Output:
    // 2 7 6 4 6 5 0 4 0 3
    // 2 7 6 4 6 5 0 4 0 3
}

This simple fact enables a few neat tricks.