Kestrel Development

Posting Images to Bluesky from TypeScript

Tue, 24 Dec 2024 05:44:56 GMT

Special thanks to Guillermo Esteves @allencompassingtrip.com, whose Rails photoblogging CMS features a Bluesky integration which was my inspiration to build this project.

Since the demise of Twitter, I’ve been posting to Bluesky. The communities that have formed there are lovely, including lots of fellow photographers, and I wanted to start sharing photos from my Photolog to Bluesky on a daily basis. Here’s what one of those posts looks like!

The Bluesky app works like Twitter – you can post small messages and attach images, videos, or a website embed. It’s different from Twitter in that it is based on the AT Protocol, which makes Bluesky more like an app that runs on top of an open database. So, instead of making a REST request to the Bluesky API, you “insert into the database” and your post shows up in the app.

My goal was to write a text post, attach an image, and run this code daily with a random photo. AT Protocol complicates things a little bit compared to a POST /posts/new REST API in that the AT Protocol is data-oriented rather than task-oriented. This means you have to perform the intermediate steps to construct the parts of a post before assembling them into a completed post.

Since I’m using Astro to post my photos, it was easy enough to update my template to add a photos.json file with content for daily posts.

How To

The Bluesky team has some documentation on posting with images, but these docs use Python and are not terribly complete, so I wanted to share my complete and working example of the Node.js TypeScript code I use to post to Bluesky.

Here are the steps from the above code described in detail:

Upload image data to Bluesky

This results in a blob, a bit of structured data which points to the uploaded image. You can’t upload an image of > 1 MB and you have to strip the image metadata yourself. At this time, these are both responsibilities of the client.

To upload content to Bluesky, you use the ATP Agent with a set of credentials (username & app-specific password). Under the hood, this is performing REST requests for you.

Build the image metadata

Construct an app.bsky.embed.images object by passing the image data blob. You have to provide the aspect ratio of your image in this part, or the image won’t be rendered properly in the official clients.

Build the post content

Assemble the rich text for the post. To add stuff like inline links, hashtags, or @ mentions, you provide the raw string alongside facets, bits of data which indicate a range of string characters should point to a target thing. Bluesky’s SDK provides a helper to turn raw text into rich text.

Finally, build an app.bsky.feed.post by combining the text, facets, image embed, and created at date. You can actually back/forward date stuff as part of how the protocol works. Some people use this to import their posts from other social media sites onto ATProto.

Post it

Use the same ATP Agent we used in the image upload step to upload this post to Bluesky, and you’re done!

Posting Daily

To run this task daily, I created a Netlify app with one scheduled function configured to run at 09:00 Mountain Time. I want this script to try not to repost images that have been posted recently, but I don’t want to maintain a database of recent posts. So I implemented the following:

Get the date of the most recently added image (the basis date)
Shuffle the list of images using the basis date as the RNG seed
Get the distance in days between the basis date and now
Pick the photo to post from the shuffled images at index [daysSince % imageCount]

With this scheme, the script will go through all images in order, only repeating after each image has been posted once. When I add new images – a relatively infrequent operation – the basis date changes, the images are reshuffled, and the script starts posting sequentially from that point. This is the only time that there’s a risk of posting the same image twice in a row, but I’m OK with the low chance of that happening in exchange me not having to run a database for this project.

How I Build Frontend Apps

Fri, 20 Dec 2024 21:27:34 GMT

I’ve written a bit about how I deliver software, and now I want to share some of the details around how I build the browser side of web applications when I’m in charge of the project.

Language: TypeScript

My main tool is TypeScript, which I use whenever it’s feasible. I think that writing your projects in TS is the obvious best decision for the vast majority of webapps for several reasons:

Just use one language. TypeScript and npm packages work great on both frontend and backend.
Great frontend/backend interop story. Frameworks like RedwoodJS provide type safety across your frontend clients and backend APIs, and they make it easy to define templates to be rendered at various points in your app lifecycle: pre-rendered static bits, dynamic on content load, etc. Full-stack TS frameworks do a much better job of updating partially dynamic content in statically rendered HTML without requiring the entire app to be a big blob of single-page JavaScript application.
Massive ecosystem. The new data format you’re trying to work with probably has an npm package. If you’re solving something that three or more developers have ever had to solve – structured logging, ANSI terminal colors, authentication, data validation – npm almost certainly has a package for you.

TypeScript does have a couple of drawbacks which occasionally make Go the better language to solve some problems:

Poor compute concurrency. Node.js runs CPU-intensive tasks in a single-threaded event loop. If you are doing a lot of parallel computation or managing big stacks of deferred tasks, Goroutines solve your problems better.
Complex runtime. Setting up a TypeScript project is non-trivial: you need to configure tsc, set up package.json scripts, decide between ESModules or CommonJS, set up linting rules and formatters… and then to run your project on a remote machine, you need Node.js installed. The Go language comes with those tools out of the box and compiles your program into a static binary with no dependency requirements.

Components: Astro & React

For my static sites, Astro is my framework of choice. It comes with the Astro component language, as well as a bunch of other helpful components which work best in Astro. For example, the <Image> and <Picture> components make it easy to drop your images into your repo and let the framework resize your images to the appropriate size for your target browsers. For work that doesn’t include a lot of frontend logic, Astro provides enough for you to build an adequate compmonent-based architecture.

When I need interactivity, I add React to my Astro apps. I first started using React in 2013 when it was released, and shortly after that, I discovered Vue v1 which I found was more effective. Vue provided “batteries included” features like state management and styling which React wasn’t really prepared to solve yet. But as React has grown, it’s become a real competitor: its integrations with state and styling libraries are now excellent. The main reason I like modern React is its functional components with hooks. With these tools, I find it very easy to reason about the data flow through a function which returns a JSX blob which ends up in my page.

State Management: Jotai

Sometimes passing state up and down from component to parent and back gets cluttered and unwieldy. Jotai is a minimal library which provides atoms, singletons that encapsulate a piece of stateful data. Atoms are defined globally and can be used throughout your app, allowing you to teleport important data wherever it needs to appear in the DOM. It also supports storage, so you can use the atomic paradigm to persist data such as session tokens between page loads. Most importantly, it works very well with React via the useAtom hook.

Styling: Tailwind

To make my UI usable, I use Tailwind, a CSS framework with great tooling based on utility classes. In the past, I’ve used Bulma to build my apps. It’s a great CSS framework with opinions included, but eventually I find that I want more flexibility over my design language, and Tailwind provides this for me.

Using Tailwind for your styling brings many advantages:

Comprehensive coverage. An extensive docs library includes every single CSS feature, from flexbox to grid to max-width containers with screen-sized breakpoints. With a well-selected library of named custom colors, you may not even have to open a hex picker.
Skip writing CSS. No, really – 99% of what you need to do in your app can be done by adding classes to an HTML element. You mostly don’t need to write CSS by hand to implement a desired design.
Editor integration and linting. VSCode can help you autocomplete valid Tailwind classes and warn you when you’re making mistakes like overwriting Y-axis padding values py-3 with py-4.
Theming and plugins. Tailwind lets you customize the class names available for use by configuring the theme. Your custom names are available across your class: creating fontFamily.funky automatically creates the text-funky class, even in your IDE’s autocomplete.
Performant builds. Tailwind automatically optimizes its production build to only include the class definitions which are directly relevant to your application. If you don’t use a feature, it isn’t included in the final CSS.

It also adds a few complications:

Component systems are required. Copying px-2 py-3 my-4 into all of your <Button>s is tedious at best and a source of mistakes at worst. If you’re using a system that is primarily doing server-side HTML templating and makes it hard for you to compose your app out of components, using only CSS utility classes may create more problems than it solves.
Build system integration. You need to configure Tailwind to work with your app’s build system. If you have to do this by hand, you might be in for pain. Luckily, Astro and RedwoodJS both include plugins for Tailwind support.

How I Ship It

Thu, 19 Dec 2024 07:05:21 GMT

I just migrated my blog from Ghost to Astro. I realized I haven’t posted something new in two years, so I thought I’d take this opportunity to share how I work on software in my personal time.

Preamble

Until recently, I was hosting my blog using self-hosted Ghost on my personal Kubernetes cluster. I like Ghost a lot, but the reasons I picked it originally were: it has a web editor, and it’s not Wordpress. I originally paid for hosted Ghost Pro just to get started, but eventually I set up a self-hosted MySQL database via the moco operator, which let me save some money on hosting.

Moco is high-quality software, it works reliably (even backup and restore), and the team writes high-quality documentation. But it’s pre-1.0 and I found myself going through difficult upgrade processes more than once, so I’ve been looking for an excuse to move back to a static site generator.

These days, most of my personal web work is done in Astro, so that’s where I migrated my blog content. Welcome to the new site!

My Principles

These days, I feel like I already spend too much time maintaining and thinking about my home servers. I want to reclaim some of my time by simplifying the way I develop and ship software, and these principles help me get there.

Automation saves time: If you plug the repo straight into CI, you can push to deploy. If you don’t automate this process, your deploy script will soon break, and then you’ll have to fix that instead of deploying your feature.
Shorten the minimum loop: I only program for fun when a project is already loaded into my brain. If I put down a high-friction project for a while, I might never pick it back up. Choosing simpler frameworks means the minimum loop – the amount of time it takes to pick up a project, write a minimal feature, and deploy to production – stays shorter.
Stay frugal: A project that consumes no resources produces no stress. Modern platform hosts have generous free tiers within which you can easily remain if you make the right technical decisions.
Use patterns: Prefer the same sets of tools for the same tasks. Pick the thing that works, not the different thing that’s shiny. If you solve a problem, solve it categorically. If you see something three or more times, it’s time to make it a template.

My Tech Stack

I have some redundancy in my tech stack, but not much. Let’s discuss:

Infrastructure as Code: Pulumi

I write a lot of Terraform in my day job and I don’t love it. The outcomes from managing infrastructure using code are excellent, but defining complicated configurations in HCL is a pain. Pulumi embraces the typed programming language and allows you to define your software stack in TypeScript. As a developer, this means you get high-quality linting in your systems before your apply operation fails. It’s also a natural fit for composing your stack together: it is trivial to extract an existing app’s hosting config (i.e. compute + ingress + DNS) into a builder function which acts as a template you can use to host all your other apps in the same spot.

Domains and DNS: Porkbun

After some poor experiences in the past with domain registrars that don’t respect their customers, I’ve moved all my domains to Porkbun where I’m very happy. Their prices are competitive, their DNS service is reliable, and they are my favorite no-nonsense place to buy a .com.

Code hosting and CI: GitHub

GitHub is the unrivaled home of open source, and it’s where my software lives. GitHub Actions is perhaps not the cheapest CI/CD solution, but it’s well-integrated and reasonably painless for my automated testing and deployment needs. I stay in the free tier.

Static and serverless hosting: Netlify

Netlify has long been a straightforward way to turn a Git repo into a hosted website, including any optional build step. But their offerings have advanced dramatically since they first launched, and now they are a formidable host for serverless apps. They repackage the reliable AWS Lambda at very reasonable per-request prices with a generous free tier.

Serverless platforms manage all of your compute infrastructure for you, charging you per request and per execution second rather than charging you to rent a server by the month. These days, if I can build my app into a serverless paradigm, I do – they are extremely cheap to free on the low end and nearly zero-maintenance. RedwoodJS and Astro are two frameworks that help me accomplish this, and they both deploy natively to Netlify.

Dynamic apps: RedwoodJS

Ruby on Rails has long been a powerful solution for booting up your new startup’s codebase, but what if you’re trying to avoid running persistent servers that need to be always-on to handle requests? And what if you’re looking to take advantage of more modern paradigms for integrating your frontend and backend code?

RedwoodJS is a framework that tackles the approximate scope of vanilla Rails or Django: it has relational database models and migrations, it renders HTML, and it ships JS to your browser. It also goes beyond: the frontend and backend aspects are tightly integrated via Cells, React components which handle the data lifecycle for you. And with RSC landing in the framework soon, it will be even easier for you to simply ask for the data you need and have it magically arrive in your UI.

I am excited about the future of Redwood. The leadership is focused on building features that help me be more productive. They have prioritized non-breaking upgrade paths on their journey from v1 to v8, something that has caused me no end of grief with Rails. And they are transparent about their roadmap and development priorities, which gives me confidence that the app I build today can be even better tomorrow.

Static sites: Astro

Not everything needs a SQL database. What if your project is, say, a blog? What if it’s a community hub? And what if you want to use tools like React to build a component hierarchy without requiring your users to run some JS just to see your text on a page?

Astro is a framework for building high-performance static sites that also supports dynamic behavior. By default, anything you build in Astro is rendered at compile time into static HTML – no JS is sent to the browser! But for interactive bits, you can add islands of interactivity, and you can even slot server-rendered content directly into an otherwise-static page.

With support for React, MDX, Tailwind, and RSS via first-party addons, the Astro team has shown their priority is to give you the tools you need to build the site you want. Astro is the best fit for most of my content today, and I am excited to continue using it as it grows.

Serverless-friendly DB: Neon

Postgres databases are expensive, especially if you’re not using the database 24/7. My small apps are well-served by Neon, a “serverless Postgres” platform. What’s a serverless database, exactly? This means that Neon is happy to serve individual requests from my Netlify function requests, and they manage the complexities of shared DB hosting for me, including the pgbouncer instance.

When a new request comes in, they quickly boot my database up and serve the request, and when it’s idle for 10 minutes, they suspend it and stop charging me. This makes Neon extremely cost-efficient for my side projects which may be only occasionally used by a few people, but should still be highly avilable. It’s a match made in heaven for dynamic apps hosted on Netlify.

Cloud services: AWS

There are some categories of services I am not interested in hosting right now. I don’t want to spend my time trudging toward a reliable, redundant setup for blob storage, hosted databases with recovery, or reliable email delivery. When I need these things, I am happy to pay AWS, the cloud provider that has historically been excellent for long-term support. I can be confident that the services I’m using today won’t be taken offline in six months.

General-purpose hosting: Kubernetes

Kubernetes is for everyone, including you! If your goal is to build a production-quality service, featuring rolling deployments, redundant compute hosting, and several nines of uptime, Kubernetes solves these problems in a way that’s accessible if not immediately friendly.

Managed Kubernetes and DNS: DigitalOcean

DigitalOcean’s managed Kubernetes offering is reliable and extremely affordable. You pay the sticker price for data plane Droplets in your cluster and the managed control plane is free.

I use Pulumi to provision my k8s cluster, connect it to DO public load balancers, set up DNS entries programmatically via DO networking, and request SSL certificates for my apps using Cert Manager. Now, putting a Docker image on the public internet is my happy path.

How to Ask a Good Question

Thu, 08 Sep 2022 21:24:28 GMT

When you have a problem with a software project or library, often the best way to ask for help is through text. Different projects offer support in different ways, such as email, chat rooms, or GitHub issues. Most of the time, the people helping you out are unpaid volunteers, so it's polite to respect their time by structuring your question well up front.

Here's an example of a bad question:

<newbiecoder123> why doesnt this work??

It's really hard to understand what's going on here! The user hasn't told us what they're trying to do or what the issue is, the screenshot is hard to read, and we don't have enough information to help them. It's going to be very hard to help this person figure out what's wrong.

Here's how to ask a good question:

Structure

What are you trying to do?

Start by summarizing your problem briefly. Keep this to one sentence. For example:

"I'm trying to make a POST request to a web server."
"I'm trying to run the application locally."
"I'm running into a bug and I'm not sure if it's my fault."

Often, we forget to give people the full context of what we're trying to do. Saying this up front helps a volunteer understand what you want to do.

What have you attempted?

If you're inexperienced with a tool, it's quite possible you're using it wrong. Complex tools often have multiple ways to use them, or subtle ways in which they can be misconfigured. State what you've tried so that the helpers know what they can skip past. For example:

"I used this example from the documentation:" (with a link to the docs and example in question)
"I ran this command in my terminal:" (with the command you used)
"Here is the code I am running:" (with a code snippet)

When you're asking for help with a programming language or library, make sure to include the code that is causing the problem. The secret benefit of including readable code is that it nerd snipes people into mentally engaging with your problem, so they'll probably help you out sooner rather than later.

If you include a code snippet, try to cut it down to the smallest block of code which demonstrates your issue. If you post multiple pages of code, people tend to ignore you; if you only post ten lines, it's much easier for someone to see what you're trying to do.

What happened?

Tell the volunteers what actually happened when you tried to use this tool. This could be something like:

an error message from your terminal
a stack trace from a program crash
a screenshot of the issue (only if you can't copy and paste the text)

Try to give people the full error message, but omit extraneous information that you don't think is related to the crash.

When possible, prefer copying and pasting text into a code block rather than attaching a screenshot. Screenshots are not accessible – they do not render properly on some mobile devices, and they cannot be easily read by screen readers.

What did you want to happen?

Now tell the volunteers your expected behavior. This is what you want to happen, and what you believe should reasonably be happening given the code you posted. This could be that:

you didn't expect an error to occur
you expected an action to complete
you expected a program to start sucecssfully

Often, this is obvious, but sometimes you might be forgetting to include context because you take the goal for granted. If a volunteer doesn't know what you're trying to do, it's hard for them to help you get there. So make sure to include this part.

Complete example

Here's what a well-asked question looks like:

Hello! I'm trying to start a Node web server using this code:
...snippet of TypeScript code...

I have tried to start it by running node index.js in my terminal.

However, when I run node index.js command to start the server, I get this error message:
...snippet of a stack trace...

What I expect to happen is that my webserver starts on port 8000 and answers requests.

Does anyone have experience with this error who could help me?

Etiquette

Don't demand help. You're using free, open-source software – you get the support level that you pay for. Most of the time, authors and experts want to help you have a good experience with their software. They're volunteering to help you – they don't owe you anything.
Don't be bothersome. Pings can be disruptive to peoples' focus or work days. Often, volunteers are helping people out in their spare time between other important things in their life. If someone doesn't get back to you for a while, they may be away from their computer, or busy, or they may have gone to sleep. In general, asynchronous, slow notifications such as email or a GitHub @-tag are OK; instant pings such as a Slack or Discord @-tag are extremely rude.
Be polite and appreciative. These are unpaid volunteers, and you're getting free help from them! To pay them back for the time and effort they're spending with you, speak politely and express gratitude. Hello, please, and thank you go a long way, and they help ensure other volunteers will want to help you in the future too.

Tips

Don't ask to ask. Just ask. A help channel is full of folks who are ready to help. Simply go ahead and ask your question, and if you're not in the right place, folks will redirect you to the right place.

My blog now lives on Gemini, too

Tue, 23 Aug 2022 02:35:15 GMT

Outdated: Since writing this post, I’ve migrated my blog off Ghost onto Astro, a static site framework. I haven’t updated my Gemini integration to point to the new blog, and this post was a bit mangled in the migration process. But this content might still be useful to you, and if you want to chat about Gemini, email me! Here is the source code for this post.

If you're already on Gemini, you can visit my blog on Gemini right now!

Project Gemini (Gemini link) is an exciting new project! It aims to bring together some of the best parts of Gopher and the web:

Content-focused: no JavaScript, no pop-ups, no images
Privacy-oriented: no cookies, no non-consensual tracking
Mandatory TLS, with client certificate support for identity management

The specification (Gemini link) is rather small for a web protocol at only ~5200 words. I found the spec easy to read and understand. It declares an extremely basic content format which sort of looks like a stripped-down version of Markdown.

I'm always excited for new global communication technologies, and Gemini seems like a friendlier version of the Internet – one where Facebook pixels aren't threatening to aggregate our activity at every turn. So I wanted to start hosting some interesting projects on Gemini, starting with this blog.

Approach

The approach I took was to host my content on Ghost, my favorite lightweight blog platform which runs on Node and features an excellent WYSIWYG CMS. Ghost provides a Content API which exposes all of my blog's content and its metadata. I wrote an application called Ghostini to read data from that API and serve it. I used the html2gemini library to convert from my Ghost HTML to Gemini text.

I host my Ghost blog inside my personal Kubernetes cluster, which runs on DigitalOcean. Traffic enters my cluster via a DigitalOcean Load Balancer, and I route it using Traefik to handle my Kubernetes ingress.

Routing

Most of my apps are HTTP/REST apps, which Traefik and Kubernetes handle as first-party citizens. This means they can natively and easily route requests to the right host by reading the Host HTTP header. However, Gemini doesn't use HTTP headers, or any headers at all. A Gemini request contains only the URL and looks like this:

gemini://kesdev.com/<CR><LF>

To use Gemini properly, the Gemini server must terminate the TLS connection. This allows it to read the client certificate, which it needs if it wants to do anything with the client's identity. This means that we can't MITM the TLS connection, by having Traefik terminate and restarting it, without losing the client identity. We have to do TLS passthrough, forwarding the encrypted connection directly to the Gemini server without reading it.

But if we can't read the message, we can't see the absolute hostname inside and know that this request is for kesdev.com. So we have to take advantage of the TLS extension called Server Name Indication (SNI). Compatible clients can send the name of their target host (the Server Name) at the TLS ClientHello step, which we're able to read before continuing with TLS negotiation. This lets our Traefik TCP router understand where this request should go and route it there without terminating TLS.

I'm lightly familiar with TLS, but learning enough of this to make this work with my k8s + Traefik setup took me three entire days of work. So I'd like to share a basic configuration that I wish I had had when I was starting this endeavor.

Configuration

My cluster uses Traefik 2.6.1. Below is a Kubernetes manifest which does the following:

starts an nginx deployment and service in your cluster
configures nginx to listen on port 443 and terminate TLS using a localhost certificate
configures Traefik to route requests on the websecure entrypoint for localhost and example.com to the nginx service using an IngressRouteTCP with TLS passthrough enabled

This should be enough to get you to a point where you can port-forward into your traefik service on the websecure port. Then you should be able to visit https://localhost:1443 to see a valid TLS connection (with a self-signed cert).

Verification

Here's how I tested this using kubectl and httpie:

kubectl apply -f nginx.yaml
kubectl port-forward service/traefik 1443:443
https get localhost:1443 --verify=no

You can verify that Traefik is routing TCP with passthrough, not offloading, by visiting in your browser and checking that the server certificate signature in your browser matches the SHA256 signature for the certificate in the configmap:

B1:71:61:6F:A9:62:44:4C:78:84:B0:A9:4D:6C:AB:51:4E:8B:EC:AB:06:A8:7C:F3:FC:C4:63:EE:71:1D:9E:A9

Kubernetes manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      volumes:
        - name: nginx-configs
          configMap:
            name: nginx-configs
        - name: nginx-certs
          configMap:
            name: nginx-certs
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - name: https
              containerPort: 443
          volumeMounts:
            - name: nginx-configs
              mountPath: /etc/nginx/conf.d
            - name: nginx-certs
              mountPath: /tmp/certs

apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 443
targetPort: 443

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
name: nginx
spec:
entryPoints:
- websecure
tls:
passthrough: true
routes:
- match: HostSNI(localhost)
services:
- name: nginx
port: 443

apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-certs
data:
openssl x509 -noout -fingerprint -sha256 -inform pem -in localhost.crt
SHA256 Fingerprint=B1:71:61:6F:A9:62:44:4C:78:84:B0:A9:4D:6C:AB:51:4E:8B:EC:AB:06:A8:7C:F3:FC:C4:63:EE:71:1D:9E:A9
localhost.crt: |
-----BEGIN CERTIFICATE-----
MIIBSzCB8qADAgECAhEAxReivp6Xurv1VFFia/KbrTAKBggqhkjOPQQDAjAUMRIw
EAYDVQQDEwlsb2NhbGhvc3QwHhcNMjIwMjAzMDUyODQ3WhcNMzIwMjAzMDUyODQ3
WjAUMRIwEAYDVQQDEwlsb2NhbGhvc3QwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNC
AARtJ3ZPKaPpmyUz4Lt5r7UgDUsa5vjiDKQeh3UX0DIlIKywO1S5k0IUrnFOlrdf
RLmBK4BqpEi8IMAHOGhwwQ5WoyUwIzAhBgNVHREEGjAYgglsb2NhbGhvc3SCCyou
bG9jYWxob3N0MAoGCCqGSM49BAMCA0gAMEUCIGTSuMSaShxZ4HQLnN7cQz+s/vG5
uyTmMI0WZL+MDLsoAiEA/TYIzjxzbFVPkU8+uD2TXlidlk1kib+eGcZ45DObPc0=
-----END CERTIFICATE-----
localhost.key: |
-----BEGIN PRIVATE KEY-----
MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQglzpLGw4kdm8tCuoe
LgmkatGPo7p0DXgy4szovoYEswChRANCAARtJ3ZPKaPpmyUz4Lt5r7UgDUsa5vji
DKQeh3UX0DIlIKywO1S5k0IUrnFOlrdfRLmBK4BqpEi8IMAHOGhwwQ5W
-----END PRIVATE KEY-----

apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-configs
data:
https-only.conf: |
server {
listen 443 ssl http2;
ssl_certificate /tmp/certs/localhost.crt;
ssl_certificate_key /tmp/certs/localhost.key;
    server_name  localhost;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}</code></pre><h1 id="conclusion">Conclusion</h1><p>I spent a long time trying to get this to work because I wanted the ability to host Gemini apps in my Kubernetes cluster. I'm very happy to have it working, and I hope that what I've learned saves you some time if you try this for yourself! I look forward to hosting more public Gemini apps.</p>

Building multi-architecture Docker images on CircleCI

Sat, 20 Aug 2022 22:35:42 GMT

I spent a couple of hours today working to build my Docker images for ARM and x64 machines in my hosted CI provider, CircleCI. Here is the solution I came up with. I hope this saves you some time if you have to do the same thing in the future.

This solution:

builds ARM and x64 images
runs on commits on the main branch of your repo
pushes your_app:latest and your_app:<1234567> image tags to Docker Hub (where <1234567> are the first 7 characters of the commit SHA)
if the commit is tagged, pushes your_app:<commit_tag> as well

version: 2.1 jobs: build: # We must use a machine image to get multi-arch support; # this doesn’t work inside a Docker image resource_class: medium machine: image: ubuntu-2204:2022.07.1 steps: - checkout - run: # Configure the “Docker Hub” for your organization, and set the DOCKER_LOGIN and DOCKER_PASSWORD env vars name: Sign in to Docker Hub command: docker login -u $DOCKER_LOGIN -p $DOCKER_PASSWORD - run: # required for multi-arch builds name: Create Docker builder command: docker buildx create —use - run: name: Build and push image command: bin/build-and-push-image environment: IMAGE_NAME: your_username/your_image_name

workflows: build: jobs: - build: context: - Docker Hub filters: branches: only: main

Contents of .circleci/config.yml

#!/bin/bash if [ -z “$CI” ]; then # protective magick echo “This script must be run inside a CI environment.” exit 0 fi export BUILD_CMD=“docker buildx build —progress plain —platform linux/amd64,linux/arm64 —push” BUILD_SHA=”$(echo “$CIRCLE_SHA1” | cut -c1-7)” export BUILD_SHA set -euxo pipefail $BUILD_CMD -t “$IMAGE_NAME:latest” . $BUILD_CMD -t “$IMAGE_NAME:$BUILD_SHA” .

set +u if [ -n “$CIRCLE_TAG” ]; then $BUILD_CMD -t “$IMAGE_NAME:$CIRCLE_TAG” . fi set -u

Contents of bin/build-and-push-image script

Designing Data-Intensive Applications Cheat Sheet

Thu, 04 Aug 2022 22:06:05 GMT

This is a summary I wrote of each chapter of the excellent Designing Data-Intensive Applications (affiliate link) book. I use this to prepare for remote job interviews in infrastructure and devops. Please enjoy!

Chapter 1

Reliability: Systems must work correctly even when faults occur
Scalability: Systems have strategies for maintaining performance, even when load increases
Response time percentiles can measure performance
Maintainability: Makes life better for eng/ops teams who work sith the system
Good abstractions reduce complexity and make the system easier to modify and adapt
Good operability means visibility into system health and having ways to manage it

Chapter 2

Document DBs target use cases where data is self-contained and rarely related
Graph DBs target use cases where all data is deeply interlinked
Schema can be explicit (enforced on write) or implicit (enforced on read)

Chapter 3

OLTP: Optimized for transaction processing

User-facing, large volume of requests
Small record count per query
Key-indexed lookup, dependent on disk seek time

Log-structured:

Append-only to files; delete obsolete files; do not update a written file
Random-access writes are turned into sequential writes on disk, enabling higher write throughput on HDD/SSD
LevelDB, Cassandra, HBase, Lucene
Update-in-place: disk is a set of fixed-size pages that can be overwritten
B-trees, used in all major relational DBs

OLAP: Optimized for analytics processing

Used by biz analysts, not end users
Lower query volume, but they demand millions of records scanned
Disk bandwidth is the bottleneck
One solution: column-oriented storage
Data warehouses: when your queries require sequential scans, indexes matter a lot less
It’s more important to encode data compactly to minimize the amount of data that a query must read from disk

Chapter 4

Rolling upgrades allow new versions of a service to be released without downtime
This promotes frequent small releases over rare big releases
This derisks deployments by allowing faulty releases to be rolled back before large user impact
This improves evolvability, the ease of making changes to an app
During rolling upgrades, different versions of the app are running at the same time
Encoding must be backward-compatible (new code reading old data) and forward-compatible (old code reading new data)
JSON, XML, CSV have optional schemas and are vague about datatypes (e.g. numbers)
Binary schema formats (Thrift, Protobuf, Avro, gRPC) provide compact, efficient encoding, with explicit forward- and backward-compatibility semantics, but are not human-readable

Chapter 5

High availability: Keep system running even if 1+ machines goes down
Disconnected operation: App keeps working even if network unavailable
Latency: Place data closer to users so they can use it faster
Scalability: Handle higher volume than any one machine could using read replicas

Replication approaches:

Single-leader: Clients send all writes to a leader which streams data change events to followers; reads from followers may be stale.
Easy to understand, no conflict resolution
Multi-leader: Clients send a write to any leader node; leaders stream events to each other and any followers
Leaderless replication: Clients send each write to several nodes and read from several nodes in parallel to detect stale data.
Can be more robust to faulty nodes, network outage, latency, but harder to reason about, weak consistency guarantees

Replication lag causes issues:

Read-after-write: Users must always see data they submitted
Monotonic reads: Users must always see data in chronological order (not see earlier point-in-time data)
Consistent prefix reads: Users must always see data in a state that makes causal sense: a question is followed by its reply
In multi-leader and leaderless schemes, conflicts may occur and must be resolved

Chapter 6

Partitioning is necessary when data cannot fit onto a single machine
Goal is to spread data and query load evenly among multiple machines, avoiding hot spots
Must choose a partition scheme that fits data, and rebalance partitions when nodes are added or removed

Approaches:

Key range partitioning: e.g. node owns keys from A through F
May cause hot spots if application frequently accesses keys that are close together in the sorted order
Hash partitioning: keys are assigned to a node corresponding to their hashed value
Distributes load evenly but destroys ordering of keys, so range queries are inefficient
Common approach: Create a fixed number of partitions in advance, assign several to a node, and move entire partitions when a node is added/removed

Secondary indices must also be partitioned:

Document-partitioned (local): secondary indices are stored in same partition as primary K/V
Only update a single partition on write; read of secondary index requires scatter/gather across all partitions
Term-partitioned (global): secondary indices are partitioned separately using indexed values
Entry in secondary index may include records from all partitions of the primary key
Write updates several partitions; reads from a single partition

Chapter 7

Tranasactions allow an app to pretend that some concurrency problems and SW/HW faults don’t exist – lots of errors become “transaction aborts”
Txns hugely reduce the number of potential error cases you need to worry about
Without txns, hardware errors (power outage, disk crash) cause various data inconsistencies – hard to reason about effects of concurrent access

Race conditions include:

Dirty reads: Client sees another client’s writes before they are committed. Solvable with read-committed isolation level.
Dirty writes: Client overwrites another client’s data that has been written but not committed. Solvable with snapshot isolation.
Read skew: Client sees different parts of the DB at different points in time
Lost updates: Two clients perform a concurrent read-modify-write (e.g. bank balance problem). Solvable with snapshot isolation.
Write skew: Txn reads something, makes decision, writes decision – by the time the write is made, the premise is no longer true. Solvable with serializable isolation.
Phantom reads: Txn reads objects matching a search condition; someone else writes data that modifies those search results. Write skew issues may require index-range locks.

Approaches to implementing serializable transactions:

Literally executing transactions in serial order on a single CPU core
Two-phase locking: standard approach; may have poor performance
Serializable snapshot isolation: optimistically allow txns to proceed without blocking; commits are aborted if not serializable

Chapter 8

Some kinds of partial failures in distributed systems:

Network: Packets may be lost or artificially delayed
Time: Node clock may jump forward or backward and be out of sync with other nodes
Pauses: GC may pause a process; other nodes declare it dead; it resumes and is unaware it was paused
Any software that interacts with other nodes may fail, go slow, or time out

Detecting faults:

Most systems con’t know if a node has failed
Most distributed algorithms rely on timeouts to detect node failure
But timeouts might be network issues, not node failures
A limping node might cause more issues than a dead one
Once a fault is detected, info must flow over unreliable network between nodes – we rely on quorum protocols to make decisions
Distributed systems enable scalability, fault tolerance, and low latency

Chapter 9

Linearizability makes a database behave like an atomic variable, but is slow, especially across high-latency networks
Causality imposes an ordering on events, based on cause and effect
Weaker consistency model
Some things can be concurrent – branching and merging
Less sensitive to network issues than linearizability; lacks the coordination overhead of linearizability

Consensus

Solves atomic problems in causal models, e.g. signing up for a username requires that username to not already be taken
All nodes must agree on what was decided, irrevocably

Consensus decision problems include:

Linearizable compare-and-set registers: set a register based on a parameter
Atomic transaction commit: commit or abort
Total order broadcast: decide on order to deliver messages
Locks and leases: only one client can grab a lock
Membership/coordination: decide which nodes are alive and dead
Uniqueness constraint: which txn is allowed and failed due to constraint violation

Single-leader failure resolutions:

Wait for leader to recover
Human does manual failover
Algorithm chooses new leader
Even if a leader can be chosen algorithmically, we still need consensus to select the new leader
Leaderless and multi-leader replication systems don’t use global consensus

My Tips for Traveling Light

Sun, 10 Jul 2022 09:38:23 GMT

Disclaimer: I use Amazon referral links in this post. Using links on this page to purchase items may result in me receiving an affiliate bonus. I have not been otherwise paid to write this post, and these are my personal views alone.

Now that I have the flexibility in my remote-work career and the income to buy plane tickets more often, I'm making the most of it. I love to travel, but I hate hauling all my shit from place to place just to live somewhere new for a bit. Here's what I've learned in my efforts to travel light while keeping my creature comforts (read: my computers) with me.

Essentials

Left to right: 24 oz ThermoFlask, 25000 mAh USB-C powerbank, hanging toiletry bag

As I've gone beyond my twenties, I've realized that my mom, with her big shoulder bags full of water bottles, wet wipes, and snacks, was right: the worst thing that can happen to you as a full-grown adult is to be caught lacking. So the core of my packing strategy is self-sufficiency. I should never find myself running around to find something I need which I could have brought with me. My big needs are water and power.

I love my insulated 24 oz ThermoFlask which can keep my water ice-cold overnight, and these days it doesn't leave my side – doubly so when I'm abroad. Any day that I need to pop into a shop to grab a bottle of water is a day I haven't prepared properly.

These days, between my boarding pass, my music, and my movies for the flight, my phone is critical to me too. Life is too short to live on Low-Power Mode, so I keep a 25000 mAh USB-C powerbank on me for longer trips. This thing has enough power to keep my phone and (M1 Air) laptop going for days, and I've never regretted hauling it with me. I'm religious about keeping all of my batteries topped up.

Toiletries

To hold my toiletries, I use a hanging bag with plenty of space for everything I could need, including medication and mouthwash. For shorter trips or one-bag travel, I use a smaller, lighter hanging bag.

Drugs

The drugs I typically bring with me are:

Melatonin: for solving jet lag by helping me get to sleep on schedule to arrive in a new time zone. Aligning my sleep schedule to avoid losing a day to travel is my favorite party trick, and it never fails to impress anyone over 30.
Benadryl: for occasional allergies, and it doubles as a much stronger (occasional) sleep aid than melatonin. Great for when you've been out later than you intended but you still need to catch a 7am train tomorrow morning.
Aleve: never leave home without painkillers.

Sleep

Left to right, top to bottom: Trtl pillow, SleepSloth eye mask, Laser Lite earplugs, Sony WH-1000XM3 noise-cancelling headphones

I'm a person who can fall asleep anywhere if I'm tired enough, but sometimes I still need a little help. So these are always in my carry-on:

Earplugs: My favorites are the Howard Leight Laser Lite disposable foam earplugs. They're extremely comfortable and cheap, they're reusable for a few days if you keep your ears clean of wax, and they block out a ton of sound if inserted properly.
Eye mask: I love this SleepSloth eye mask, which blocks out all light and doesn't put any pressure on my eyes. You can even comfortably open your eyes underneath it.
Trtl pillow: This adjustable neckscarf-style pillow provides support to keep you from "nodding on" as your head tips over sideways. It's also incredibly compact compared to a big fluffy neck pillow.I've slept dozens of hours with this strapped to my neck on long trips, and it only makes you look a little silly. (They're mostly just jealous of you.)
Noise-cancelling headphones: The Sony WH1000-XM series provide incredible noise cancelling, even if you're frugal like me and find a deal on an older generation. I used to be a die-hard Bose QuietComfort fan, but I had two pairs go out on me out of warranty and I no longer trust the brand's quality.

Clothes

Left to right, top to bottom: Under Armour technical shirt, tabi slippers, synthetic stretch chinos, Fun Toes merino wool socks, Timberland Mt. Maddsen hiking boots

Never skimp on anything that separates you from the ground. It's critical to bring a pair of shoes that won't let you down when you're away from home. Shoes are heavy and bulky, so if you bring one pair that will get you through everything, you won't have to fit a second pair into your luggage.

Look for something durable and hard-wearing. Outdoor hiking boots are a great option, and high-quality brands (with varying levels of everyday style) include Keen, Merrell, Red Wing, and Teva. I am extremely partial to the Timberland Mt. Maddsen boots, which look amazing and are fully waterproof – I've even brought these on canoe trips. I typically wear these with cheap merino wool socks, which are comfortable in hot and cold conditions as well as breathable to keep sweat off your skin.

When you're lounging indoors, you'll want some shoes to grip the dirty floors and protect your nice socks from them. Look into tabi-style slippers, which are lightweight, easy to wash, and fold flat into your luggage.

To stay comfortable when you're spending several hours sweating into your seat on a bus, an airplane, or in an airport chair, I recommend you pack only high-wicking synthetic technical shirts. They double as great shirts for hiking or walking around a city, they don't wrinkle, and they move the sweat off your skin. Anything will do, so find a brand that you find comfortable which fits you well. (I like these shirts by Under Armour.)

Pick shirts that are all around the same color – coordiating your outfits is much easier when all your shirts and sweaters match all your pants. If you're going somewhere sunny, look for long-sleeve technical shirts with built-in SPF protection.

Science has made incredible advances in stylish, wrinkle-free, everyday pants. Synthetic stretch chinos are so comfortable you'll forget you're stuck in a plastic chair for seven hours during your layover, and they're stylish enough to shame everyone else showing up to the pub. I love these so much that next time, I'll bring a second pair instead of my denim jeans.

Devices

Left to right: Fire Stick 4K Max, Peak Design Tech Pouch, Anker PowerExtend adapter

I spend a lot of my downtime on vacation catching up on my favorite TV shows and websites, so it's important to me that getting power isn't a pain in the ass. It's critical if I'm working remotely!

Power

An outlet extension strip is an incredible thing to have with you when power outlets are limited. No need to snipe an outlet or wait for one to free up – just piggyback their adapters onto yours and you can plug in too. This Anker PowerExtend adapter triples as an extension cord, a power strip, and a USB-C PD charger for my MacBook, which makes it totally indispensable to me.

If you're going somewhere with different plugs, don't forget to bring an outlet adapter. A 240V -> 120V adapter is often huge and heavy, so if you can avoid bringing one of these, don't bring one! Odds are your USB adapter (e.g. the Anker listed above) supports anywhere from 100 to 240 volts AC – you can find out for sure by reading the fine print on the back.

And of course, don't forget the USB power bank I mentioned under the Essentials section.

Cables

Triple-check to make sure you have the correct cables on you for all your devices before you leave. It's never a great feeling to leave on your trip only to have to hunt down an overpriced Lightning cable at the airport. And if you're planning on using your iPhone as a mobile tethering modem for your MacBook, make sure you bring that pesky 3ft USB-C to Lightning cable too.

If you carry as many cables and devices as I do, you'll want to organize them better than coiling them all up and tossing them into a big pocket together. The Peak Design Tech Pouch is the gold standard for organizing lots of little cables, but of course, it's just a pouch with pockets! There are plenty of $10 options which do the job just fine.

Media

I love catching up on my TV and movie backlog while I'm on vacation, but I hate showing up to a rental apartment and finding out their smart TV is all goofed up, or the HDMI cable is broken. So these days I bring my Fire Stick with me on trips, along with its power adapter and cables, in a little accessory bag. It stays signed into my streaming accounts and makes it easy to put on whatever I want when I arrive.

Amazon devices like the Fire Stick regularly go on sale during holidays and Prime Day, so don't buy one at full price.

Bags

A great trip ends when your luggage goes missing. The best way to prevent this is to never let your bags leave your side – that means no checked luggage! There are two main schools of thought:

1. One-bag

Left to right: Osprey Farpoint 40, Aer Travel Pack, Peak Design Travel Backpack

Pack all your stuff into a single bag.

Aim for a 40L backpack intended for carry-on travel. Bags of this volume will fit on airplanes, either below the seat or in the overhead compartment. (In my experience, they even fit onto Ryanair.) A bag listed as a travel pack will typically unzip all the way around so that you can pack it like a suitcase, rather than stuffing in from the top like a hiking pack. My favorites are the Osprey Farpoint 40, the Aer Travel Pack, and the Peak Design Travel Backpack.

If you only bring one bag, your one bag never leaves your sight (or your arms) so it's at reduced risk of being left behind or stolen. Airlines love you – many offer priority boarding for folks with no overhead carry-on, and you're never at risk of having to check your second bag when the plane is full. However, you will have less space for all your items.

Folks who are willing to pack light can absolutely do a trip of any length with a very minimal set of clothing:

3-4 shirts, sets of underwear, and pairs of socks
1 pair of versatile pants
1 pair of comfortable pants (or shorts) for sleeping and lounging
1 all-purpose down jacket
1 versatile sweater layer

This gives you enough clothing to wash your dirty laundry in a sink and hang dry every couple of days. You can bring travel detergent packets, or you can just put a bit of your own detergent into your travel shampoo bottles – it's the same stuff.

2. Backpack and carry-on

Left to right: Travelpro Maxlite 5, Briggs and Riley Domestic 56cm, Peak Design Everyday Backpack strapped to a carry-on handle

Pack your clothes into your carry-on bag and your devices into your backpack.

A one-bag travel pack can turn you into a tortoise, and they're not ideal for bringing your laptop to work from a coffee shop. If you choose to bring a carry-on, you can bring most of the clothes you want, and you can bring a backpack that's more ergonomic for daily use. Plus, you'll have more cargo space for bringing souvenirs for friends back home.

If you go this route, bring your favorite daily backpack and a good softshell carry-on. Mine is the Peak Design Everyday Backpack, which comes with a handy built-in strap that fits over a carry-on bag's handle. You do not need a hardshell carry-on – hardshells are most useful for protection against airline employees chucking your bag into the cargo hold, and softshells allow you to use more space and expand the bag's capacity with a zipper.

Pick a high-quality spinner (four universal wheels) carry-on bag. It should either come with a lifetime warranty (Briggs and Riley) or be cheap enough to replace when the wheels go out in a few years (Travelpro Maxlite 5). Don't forget that American brands have a "domestic" size (larger for American plane cabins) and an "international" size (smaller for European plane cabins), so size according to the airline you usually fly.

Other bag tips and tricks

Left to right: Gonex compression packing cubes, crossbody sling, 4Monster 16L packable daypack

I am deathly afraid of being made a target when on vacation, especially when I'm out drinking. I'm also horribly scatterbrained and liable to lose anything not physically attached to me. So I keep my passport, wallet, and cash glued to my body using a crossbody bag. (It has enough space for an N95 mask and my AirPods as well.) I like this bag enough that I've started using it for my dailies at home, rather than transferring all my items between my pairs of pants every morning.

Sometimes I just need a bag to hold my down jacket and water bottle, or a tote for groceries, and it's inappropriate to bring my nice laptop backpack. So I always bring a 16L packable daypack in my luggage. It packs down into a tiny bundle that I can shove into any free corner of space, and it's ultralight when empty.

Whichever bags you use, you can get more volume from it if you use packing cubes to store your clothes. These Gonex compression cubes come with a zipper for the contents and a second zipper to compress the air out of the clothes inside. This way you don't have to worry about carefully rolling your clothes together – just fold them flat, put as much as you can into the bag, and zip it down to compress.

Conclusion

These tips aren't intended for anyone in particular – they're just items and practices that I've found which help me make a good trip great. I hope some of these help you on your future trips, and I'd love to hear what you like to use when you travel.

SRE for Backpackers

Mon, 23 May 2022 16:35:18 GMT

Last week, I took some time away from work to travel back to Minnesota and camp in the Boundary Waters Canoe Area (BWCA) with my friends. When I was growing up in Wisconsin, my dad took my brothers and me to the BWCA with an outfitter guide. These days I feel confident going out on my own without a guide.

Our party was six people, all experienced backpackers. Everyone had been to the BWCA on a canoe camping trip before. Still, some things went wrong, and since it's my job to keep things from going wrong, I wanted to write about how the lessons I learned as a Site Reliability Engineer apply to the world outside computers.

Pack light

Each of us carried a hiking pack with our personal stuff – clothes, toiletries, sleeping bag and pad. We split the shared goods – 2-person tent, water filter, rain tarp – among bags evenly. But the spread of weight was drastically different from person to person. On the low end, James and I packed ultralight Osprey Atmos packs weighing in fully loaded around 30 lb (14 kg), while Dinh and Jared packed wider, larger "portage packs" clocking in around 50 lb (23 kg).

Carrying bigger, heavier packs means you can bring more. It also means you have to portage them to get to your campsite. Our campsite was separated from the initial put-in lake by three other lakes, each separated by land. We had to pull the canoes out three times, portage the canoes to the other side, haul the bags over, load the bags, and put the canoes back in. The process is lengthy and strenuous: our longest portage was about 0.7 mi (1.12 km) of hilly, rocky trail. We rented light Kevlar canoes, but a "light" canoe still weighs 50 lb (23 kg) – a significant load for your shoulders.

Our canoe – light packers – had the option to haul in the following configuration in one trip:

A: carry a pack and the canoe
B: carry a pack and help balance the nose of the canoe
C: carry two packs

The other canoe – heavy packers – did not have this option because their packs were so heavy, and had to make two trips between the entry and exit point for every portage. The discrepancy in active portage tiplanme made it harder to keep our party together because the "light" canoe would be in the water waiting while the "heavy" canoe was still moving down the trail.

When we unpacked at camp, I noticed lots of ways that our heavy campers could have saved weight:

Someone brought a blue hardware store tarp that weighed about 4 lb (2 kg). Modern ultralight tarps can come in as low as 8 oz (224g), pack smaller, and serve the same function.
Someone brought two Cliq camp chairs, each of which weighed 3.7 lb (1.7 kg). They're nice chairs, but we didn't strictly need chairs with legs. Every campsite had logs configured around the firepit grate as seating, and we always sat around the fire anyway. A great alternative would have been the folding foam canoe chairs – these doubled as secure, comfortable seats in the canoes when we paddled, and when we arrived at camp, we unclipped them from the canoes, brought them up to the campfire sitting logs, and used them as seats. Any equipment that pulls double duty is worth its weight!
Someone brought a bulky 32 oz (900 ml) plastic canteen with a built-in LifeStraw. The LifeStraw doesn't weigh much, but it adds volume to the water bottle, and it's redundant when we're already hauling water filters for the lake water. My hydration bladder fit nicely along the inside flat back of my hiking pack and took up much less volume.

In my work as an infrastructure engineer, I see lots of ways that we misuse tools and add extraneous redundancy and overhead to daily processes:

Approval and sign-off workflows in our shared repos demand code review and approval from people who are not only outside of the codebase, but outside of the domain – the mandatory reviewers don't understand the problem that's being solved in the pull request.
Overly-complex domain tools are chained together in redundant ways: using Terragrunt to template out Terraform charts that deploy using Helm charts under the hood makes it incredibly hard to understand how a parameter turns into a value, and makes the system inscrutable to new engineers. It doesn't add value, either: Terraform can do all of this without additional tooling.
Sometimes, less is more. Many of our shared infrastructure modules define categories of databases, storage systems, or workloads to be used in production environments. But sometimes this work to define domain problems as infra modules is premature, and users find it easier to simply write the verbose infrastructure by hand ("going full YAML") to get what they need. Rich Hickey's Simple Made Easy is a wonderful talk about how to understand your domain to write the correct abstraction for your users without overcomplicating.

Plan backups for your backups

Throughout the trip, we had a series of redundant systems to get where we needed to go without getting lost in the wilderness. We had two canoes, and one person brought a pair of FRS/GMRS radios so we could communicate between parties. We had two waterproof maps of the area, detailing the location of campsites and length of portages. And we had a GPS in each boat as a quick reference and backup to the maps.

On our last day, we had chosen a campsite a stone's throw away from our entry point. We camped on Alton Lake, one lake over from Sawbill Lake via an 0.1 mile (0.15 km) portage. We planned to get back to the parking lot one hour after we shoved off into the water. The heavy boat launched first while we finished loading the light boat. But as our boat headed south along the east shore toward the east portage landing, we saw the other boat head south along the west shore and blow straight past the portage. Worried, we landed our boat at the portage, moved to the exit point, and gave the other party five minutes to catch up. We tried to figure out what happened while we waited, but we only got the full story after we got back to the parking lot:

The person with the radios had decided to pack them both up, reasoning that it was a short, straightforward trip and we wouldn't need them.
The person in the other boat had folded their map up into their tent by mistake and couldn't access it when they loaded into the canoe.
No one in the other party had reviewed the course closely before shoving off.
Someone in the other party had mistaken the location of the portage, telling the party it was at the south end of the lake rather than halfway down the east bank.
The person in the other boat with the GPS didn't review the route as they paddled.
I had asked the other crew to keep our boats together in a single convoy, but they opted to set off immediately so they could get back to civilization faster.

To anyone who has ever participated in a high-pressure production outage that's gone poorly, this sequence of failures likely looks familiar. Disasters are usually not caused by a single event, but rather a series of systemic failures that chain to cause an unwanted outcome. An example of this is the Tenerife airport disaster, in which two 747 passenger jets collided on a runway, resulting in massive loss of life. Major factors in this incident included:

Pressure on one of the pilots to take off ASAP to remain in compliance with his airline's duty-time regulations
Pressure to take off immediately to avoid further delays from deterioration of weather conditions
Sudden fog which greatly limited visibility, preventing plane crews from seeing each other and tower crew members from seeing either plane
Radio interference which made it difficult for plane crews to understand tower instructions

When you build practices for daily operation – or incident management – into your organization, take time to think through the human factors. By working to answer these questions, you will have a better understanding of how your system will fail in ways you didn't design it to. You can use these answers to build stronger processes that mitigate broken links in the disaster chain.

Are your engineers typically working on code and production changes as a team, or are they independent cowboys shipping solo?
How do your team members typically communicate when something is going right – or wrong?
How do they behave when under stress, as opposed to during business as usual?
What conflicts exist in your organization map? What teams don't fully trust one another, and are likely to withhold information in a critical period?
Which teams are motivated to uphold the integrity of the system as a whole, and which teams are directly motivated to uphold the integrity of their deliverables above other teams' work?
How are engineers being pressured by executive leadership? Which teams are being overloaded into taking on more work than they can reasonably support? What kinds of tech debt will definitely result in future incidents if not addressed?

Rethink your knots

When camping in the northwoods of the midwestern United States, the most common concern is black bears. Black bears in Minnesota average from 154 lb (70 kg) to 275 lb (125 kg) and are known to be persistent and intelligent about acquiring food. Although they rarely approach campers and direct encounters are uncommon, they become more dangerous when acclimated to human food, learning that human campsites have food which can be foraged.

To minimize bear attacks, park staff have drilled into campers the utmost importance of making food totally inaccessible and removing it from campsites. Before you go to bed in your tent every night, you have to take the following precautions:

Remove all scent from your campsite. Store food and toiletries (e.g. deodorant sticks, toothpaste) in sealed dry bags. Scoop any loose food debris into the fire pit and either burn it or bury it in ash.
Wash dishes in a bucket. Bring the bucket of dirty water to the woods several hundred feet from your campsite (usually near the latrine) before dumping it. Brush your teeth and spit out here.
Hang a bear bag. Attach a rope between two trees at least 12 feet (4m) apart. Rig a pulley system with carabiners. Attach your food bag to the center of the rope between the trees and pull it up. Secure the working end of the pulley rope to the tree.
Watch out for common pitfalls. Bears can figure out how to shred an improperly-hung rope, dropping the bag right into their claws. Bears can stand on their hind legs and shred bags that hang lower than 12 feet above the ground (the dreaded "bear bag pinata").
Do this every night. Do not skimp. Do not skip one night because it's late, the sun is down, and you're cozy in your tent.

All of us in the group have hung bear bags before, with varying degrees of ease. The first night, we hung our bear bags between two trees in the simplest configuration: two ropes joined in the middle by a carabiner, hauled up by hand. We hung the bag successfully, but we had trouble lifting it: everyone's food and toiletries were reasonably heavy at the start of the trip, before we had eaten all the food, and it took three people to successfully pull the bag to hanging height. We also used a rope toggle to secure the free ends.

I had brought along a copy of The Little Book of Incredibly Useful Knots, a wonderful book that goes into detail on various sorts of knots, bends, hitches, and more. It gave me some great ideas on how we could minimize the amount of hardware we needed to use and improve the hanging setup using only the rope we brought with us.

The next night, I offered to improvise a hanging method that would be simpler, use fewer carabiners, and accomplish the same security for our bear bags:

Two ropes joined in the middle by a Zeppelin bend, eliminating the need for a carabiner
Use of the now-free carabiner to form a pulley, easing the burden of lifting the heavy bags through a 2:1 mechanical advantage
Securing the working ends of the ropes to the tree using a round turn and two half hitches to clean up the slack and minimize un/winding work
Hanging the bags from a butterfly loop to secure their position along the line between the trees

We had success with most of these techniques, but struggled with tossing our rope over the branches in the trees available to us. We ended up skipping the pulley and muscling the bags up to height – which wasn't so bad in the end, because by the last day, we had eaten most of the food and the bags were much lighter. But the techniques I got to practice made me much more comfortable with my own ability, and I'm now confident that I can nail down a two-tree pulleyed bear bag system on my own next time. (Outside of hanging the bear bags, I also learned some great knots for tying guylines to support tarps and tents!)

In systems engineering, especially in software, it's important to return to your assumptions and your working system frequently. Often, you'll find a live production system has evolved organically to support business needs, and developers have bolted functionality on like campers bolt carabiners to their backpacks. If you take a big step back and audit the system holistically, you can often find ways to rearchitect these legacy applications to simplify them and make them easier to work with.

For example, I've worked on two major "data mover" systems whose purpose was to move data from one big database to another on a regular basis. They effectively served as ETL applications: extract data from the source, transform it to fit the form the consumers want, load it into the destination. But their business logic code had grown organically without a design document or, seemingly, a plan, and both systems were quickly becoming unmaintainable.

I wrote a project proposal to completely overhaul each system from first principles. The proposal included the time to fully audit each system's business responsibilities and source code, diagram and document each system, build a drop-in replacement, migrate business logic to the new app, and perform a switchover of the daily job. "Big bang" rewrites are often seen as productivity poison by project managers because engineers love to spend time refactoring code that they see as "ugly" or "sub-optimal," but if you put the right plan into place beforehand, you can build your way toward success from the start. I was able to execute both of these "big bang" rewrites successfully by spending my time getting buy-in from my managers and team leads so that they understood how supporting this project would end up directly supporting their work.

In conclusion

The more time I spend inside large systems, the more I tend to think about the world around me as a large system. Even though I type glyphs into a screen all day for a living, I still find that the lessons I learn about reliability and resilience from the digital world apply to the analog one. By packing light, planning backups for my backups, and rethinking my knots, I find that my systems improve in all parts of my life. As a site reliability engineer, you're in an excellent place to help your fellow humans by sharing your favorite techniques, discussing novel approaches, and reworking systems that have been taken for granted. It's your duty to make the world a safer, more reliable place!

Required Viewing: Tech Talks

Thu, 16 Dec 2021 20:33:41 GMT

I recommend these tech talks to everyone I know in software. They've been deeply influential in my engineering practices, and I think about them often when I'm working to write high-quality code.

Thank you to everyone who contributed to these talks – I couldn't do it without you.

Rich Hickey: Simple Made Easy

How do we build systems that are ergonomic to work with without producing systems bloated with complexity?

Simple Made Easy

Rich Hickey emphasizes simplicity’s virtues over easiness’, showing that while many choose easiness they may end up with complexity, and the better way is to choose easiness along the simplicity path.

InfoQRich Hickey

Richard Feldman: Making Impossible States Impossible

Your language's type system is a powerful tool. Learn how to leverage it to reduce the number of runtime errors in your software by encoding your domain problem into compiler-checked types.

Brie Bunge: Adopting TypeScript at Scale

Learn how Airbnb migrated from JavaScript to TypeScript. My favorite fact from this video: 38% of Airbnb production incidents could have been prevented with TypeScript's compile-time validation.

On The User Interview

Wed, 03 Nov 2021 01:32:50 GMT

As engineers, we often have grand ideas of fantastic things we could build to solve everyone's problems. We spend a couple of months building and deploying a prototype that we're proud of, only to discover that everyone hates our solution. The tool we built is the wrong shape for the user, adds friction to their workflow, and makes their problems worse.

I often forget that I should start by asking people what they need before I build anything. If you ask the right questions, the users will tell you exactly what you need to build – even if they don't yet know what they want.

A structured user interview will give you the unbiased answers you need to build the right thing and make your people happy. Here are my tips on how to conduct user interviews successfully.

Don't over-specify your solution

Your manager has given you the time and space to build a product that solves someone's problem. You probably have an idea in your head of what a solution might be. But if you're not the user, your idea of a solution is probably very wrong.

If you start building out your idea immediately, you'll probably end up building something that:

Introduces too much friction and frustration
Interferes with an existing workflow
Isn't suited for your users' skill level
Solves a problem that exists, but isn't the one your users want you to solve

So it's important to structure your process around what the user wants, not what you think they want. Here's an example of an over-specified early solution to building a CI/CD tool:

We are building a deployment pipeline that takes a user's Docker container, turns it into a Kubernetes Deployment with Service and Ingress, and gets it online at a public URL.

To focus on the user's needs first, simplify:

"takes a user's Docker container:" What if the user uses LXC or .deb packages to deploy their software? Instead, say "takes a user's software."
"turns it into a Kubernetes Deployment with Service and Ingress:" What if they're deploying their app to a different system, like Fargate? Is supporting Fargate deployments within your team's scope? If not, do you have a plan to help the team into Kubernetes and make sure they understand how it works? Instead, say "deploys it to cloud compute."
"gets it online at a public URL:" Is this a public app? Does the user want a public URL to access their app? Are there security concerns if you take their internal/VPN apps and put them online? Instead, say "exposes it so the team can use it."

After de-specifying this solution, it sounds something like this:

We are building a deployment pipeline that takes a user's software and deploys it to cloud compute, then exposes it so the team can use it.

Great! Now you have an open-ended shape of a solution, rather than a concrete solution. As you perform your interviews, you have an idea of what kind of thing you're building, and the feedback from your users will help you refine this idea.

Write your questions up front

We all carry personal bias into everything we do. When we speak to people, the conversation changes based on the other person's appearance, mood, and perceived experience with the subject at hand.

When we perform user interviews, our bias can prevent us from getting to the important parts of a conversation we want to have. We might assume a backend developer knows all about load balancing, only to find out later that they've never heard of it. We might not ask a data scientist about a deployment pipeline because we assume one of their peers handles that for them. It might turn out that they're the only person maintaining the production deploys for the team.

To avoid your bias keeping you from discovering what you need to know, write your questions before a user interview starts. Some of my most successful user interview questions look like this:

Tell me about how you deploy software into production today.
How is your team's morale when it comes to deploying software? Why?
How does your team like the existing process?
What pain points do you encounter with this process?
If you could wave a magic wand and instantly fix three things in your system, what would they be?

I find that people are not often able to come up with a concrete solution to their problems and how to fix them. But most people do a great job of telling you what is wrong. When someone encounters a point of friction every day for a year, it starts to burn itself into their memory. They're usually ecstatic to find someone with an open ear, ready to write down their complaints. I always make sure to include open-ended questions about what a user finds annoying, frustrating, or painful.

A couple of other tips:

Keep your questions open-ended so that the user can tell you how they feel.
Avoid simple questions that lead to yes/no answers.
Don't write questions that focus on a specific piece of infrastructure or technology, or on a specific solution you have in mind.

Follow, don't lead, the subject

Finally, it's important to remember that a user interview is about the user, not your team's mandate or goals. You're here to discover what pains them so that you can empathize with them and build something to solve their pain. Remember this during your user interview. Try to stick to your question list when you need to move onto a new topic.

This shouldn't prevent you from following their lead. If a user has strong feelings about a piece of infrastructure, it's a great idea to ask them to elaborate. Ask them why and how often they feel that way. If you pick up on an unusual emotion—say, anger or despair—consider the context, and consider adding a question to your list to help you explore these feelings with future interview subjects.

As long as you're asking open-ended questions, starting with a common base to expose how users truly feel, and working to discover the thoughts of your users in an unbiased way, you're doing a great job.

Best of luck with your user interviews! I wish I had more resources to share on this topic – my experience with user interviewing is mostly practical in my work, alongside a couple of UX classes in college. If you have resources that you found helpful for conducting user interviews, I'd love for you to share them with me at matt@mplewis.com.

My Site Reliability Resources

Fri, 27 Aug 2021 19:48:09 GMT

This is my list of resources for folks who want to learn more about site reliability engineering, embracing risk, and building observable systems.

Last updated 2020-08-27.

Site Reliability Engineering: Embracing Risk: The Site Reliability Engineering book outlines the Google Way for keeping the lights on. It's an excellent book and you should read all of it if you're interested in building reliable software. This chapter is a great introduction to the idea of "one to zero:" understanding that components of a system are inherently unreliable, rather than assuming the building blocks you put together will always work.
Friday Deploy Freezes Are Exactly Like Murdering Puppies: Charity Majors is the CTO of Honeycomb, where they work hard to make your systems observable so you can make them reliable. (Follow her on Twitter.) This blog post discusses why deployments are the heartbeat of your company, and why good judgment matters more than prescriptive policies.
Deploys: It’s Not Actually About Fridays: Charity writes about how system reliability and rapid, frequent deployments are tightly linked, and how a culture of observability-driven development is key to the success of your systems and your organization.
Shipping Software Should Not Be Scary: Charity discusses ownership and agency surrounding code and systems, why your senior engineers must be able to deploy and debug their own code, and why nothing is production except production.
The Phoenix Project: In this novel, an IT manager is promoted to CTO just in time to take the flak for a big-bang system rewrite and catastrophic production deploy. This book is required reading for anyone looking to manage an engineering organization. It discusses the ways in which your project flow can be seen as an assembly line in a factory, and how disruptions to your flow make it difficult to see the true causes behind your group's failure to deliver.

Burner Phones for Activists

Sun, 17 Jan 2021 23:03:01 GMT

If you participate in street activism and attend actions, you probably understand the value of having a smartphone on you with an active data plan. The Signal secure messenger runs purely on data networks, and it can be useful to have a live mapping app in case you're unfamiliar with the area or need to find a way home.

Many activists recommend you do not bring your everyday phone with you to actions. If you do, you run the risk of law enforcement having a Stingray device on site. If the action is deemed unlawful, LEOs might cross-reference the list of serial numbers detected in the area with lists of known cell phone subscribers. This data might be used to start building a case against you.

Learn more about Stingray phone surveillance:

A burner phone is a device that is not tied to your identity that you may not plan to keep forever. Since this is for the purpose of modern activism, we'll look for a smartphone that runs Signal. Here's what you'll need to do:

Buy a prepaid phone SIM
Buy a smartphone
Activate your phone plan
Install Signal

Buy a prepaid phone SIM

When you buy stuff and you don't want to have it associated with your identity, the golden rule is use cash. You want to avoid a paper trail, so go to your favorite no-fee ATM, withdraw some cash, and only use cash for the following purchases.

I recommend you select Tracfone as your wireless carrier. They're inexpensive, run on a variety of wireless networks, and are almost definitely compatible with the smartphone you will buy. (My SIM card kit came with three different SIM cards for different networks.)

Tracfone SIM cards are sold as a BYOP (Bring Your Own Phone) kit. You can buy a smartphone/SIM card bundle, but I recommend against this as the phones are generally much more expensive when purchased through Tracfone. (Tracfone also sells odd models of Android that almost certainly will not receive security updates after too long.)

Go to your local tech superstore (e.g. Best Buy), walk over to the Tracfone section, and purchase a BYOP SIM card kit for $1. You're done.

Buy a smartphone

When buying a smartphone, consider the following:

Does this do what I want? The device should run an up-to-date operating system (iOS, Android). Google it and make sure it was made within the last five years.
Is it secure? Before you buy your device, make sure you can still update it. Android phones typically last two years before they stop receiving security updates from Google and the manufacturer. iPhones typically last four years as Apple puts a higher focus on keeping their devices running for longer. (This is why I recommend iPhones over Android devices.)

If you are a power user, you might be able to flash a clean updated ROM onto your Android; make sure it's compatible before you buy.
Is it in good shape? Sure, you won't use this thing every day, but you want the screen to be free of cracks, the battery to be in good condition, and all the components in working order. (And don't forget to buy a case to protect it from drops while you're on the move.)

Here are three places you could buy a used smartphone:

Locally on Craigslist. Craigslist people take cash, which is ideal, and you can inspect the phone in person before taking it home. But you might not be able to find a wide variety of devices in your area.
In a bundle with your SIM card. But odds are this is much more expensive than purchasing a similar phone in a personal transaction.
On eBay. This is not ideal, because you'll have to pay via PayPal. But the prices and selection will be better than shopping locally.

After you receive your phone, make sure to wipe it completely before setup, and update the OS as soon as possible.

Activate your phone plan

Your Tracfone plan has service days as well as minutes and data. Service days correspond to how long your phone number and service will remain activated. Minutes and data are how much of your prepaid quota you have available for use. When your plan runs low on any of these, you purchase Tracfone prepaid cards to top-up your plan.

There are two ways to top-up your plan without leaving a paper trail:

Buy a Tracfone prepaid card with cash
Buy a Visa/MasterCard debit gift card with cash, and use that to top-up on the Tracfone website

Tracfone cards are available at many gas stations, pharmacies, and grocery stores. They look like this:

The packaging for a Tracfone retail gift card

But if you choose to buy a Tracfone prepaid card with cash, you may run into some issues:

These cards usually come with service days and minutes, but not data. You need data to use Signal.
The "pick-your-own-size" cards (e.g. the card itself has barcodes for various denominations of "virtual top-up cards") must be activated at the register. The clerk will ask you for your phone number and name, and you don't want your name on record here.

I recommend the following process:

To activate your plan for the first time, purchase the cheapest Tracfone card you can find that includes plan minutes. (Again, use cash.)
To top-up your plan with data, purchase a $100 prepaid debit card. Then, go to the Tracfone website and refill your account using the debit card. Tracfone supports prepaid debit cards as a method of payment, and you don't have to provide your real name to complete the checkout process. (Your debit card may take some time to activate after you purchase it, so don't worry; just wait two hours and try again.)

At the time of writing, Tracfone offers the following refill on their site:

$15: 30 days service, 500 minutes, 500 texts, 500 MB data

If you refill with three of these, you'll get 90 days of service and 1.5 GB of data total for ~$50 after tax. This means you can get six months of service out of your $100 debit card. And as long as you set your phone up and install apps on wifi, and only use data for messenger apps, that data should last you a while.

Don't forget: you'll need to top up your phone to keep your service active, so pick up debit cards with cash before you need them.

Install Signal

Now that your phone is set up, it's time to install Signal so you can communicate with your comrades. If you're on Android, you can download the APK directly so that you don't have to sign into the Google Play Store with a real account.

If you're on iPhone, you will need an Apple account to download Signal. So don't use your everyday account here. Create a new email address for an alternate identity of yours through ProtonMail, a privacy-focused email provider with free plans available. Then use that email and identity to create an Apple account. Finally, sign into the App Store with this email and install Signal. Don't do anything with these accounts on any other device!

Conclusion

At this point, you have an activated smartphone, with a data plan and Signal, that are not tied to your true identity. The Intercept has a great article on true anonymity and privacy that you should read before you set up your device: Protesters, Here’s How to Set Up a Cheap Burner Phone. In particular, read the section titled A Note About Anonymity, as it raises important concerns about how you use your new device.

Be safe out there, and keep your comrades safe!

I'm Scared of Learning Hard Stuff

Tue, 01 May 2018 16:32:04 GMT

At Gusto, I work with engineers from diverse backgrounds of experience. Some of us came from the typical Silicon Valley companies like Google and Facebook, while others joined Gusto as their first startup. I started in a consulting shop in Minneapolis fresh out of college. It gave me the confidence to learn more on my own, but also taught me to learn in a way that I’m still trying to recover from. With the support of my colleagues and mentors, I’m learning how to learn in a way that will make me a better engineer and leader.

In my first job, I built mobile and web software for connected IoT devices that used Bluetooth. I worked on very small teams and had near-total ownership over my projects. This meant I got to learn in a way I really enjoyed: picking a greenfield approach, learning iOS or Android, and building with the shiniest new tools. In consulting, we weren’t often concerned with the legacy of our code – if the customer accepted delivery, we had finished our project and could move on.

My Approach Falls Apart

When I moved to Gusto, I entered a gauntlet of new and foreign technology. Gusto's engineers are full-stack, and my mentor asked me to start where I was weakest: a JavaScript frontend app built with React, Backbone, and Bootstrap. I did my best to adopt the organization's best practices around testing and code review. I became comfortable in the dev tools as well as the Rails console.

But as I moved into the thicker parts of the payroll domain, my team asked me to debug thornier issues. I had to figure out why a payment wasn't sent as expected, or why a feature I wrote was locking customers out of the app. This meant I had to dig into the five-year-old Gusto codebase. In contrast, I'd written the majority of my past codebases from scratch, so I knew about every dark corner harboring bugs. It was a shock to jump into Gusto and find out I knew nothing about the mechanisms I needed to fix. The codebase was complex and many of the authors had left the company. There was a lot I didn't know!

I coped with my inexperience by avoiding the problem. When we assigned tasks during sprint planning, I asked for the ones within my comfort zone. When fires happened in production or I needed to make important design decisions, I deferred to anyone who was confident about tackling the problem.

Pictured: My reaction when asked to make hard decisions and work with scary code.

My coworkers noticed that I often spent time avoiding hard problems by working on other problems I enjoyed more. They saw me as someone who wrote excellent tests but became distracted when it came to building entire features. In my performance review, my PE (People Empowerer -- what we call a manager at Gusto) and I discussed my peer feedback:

“You delegated a key decision on this Payroll Engineering story to an engineer that is your junior.”

“You spent a couple of days optimizing the build process, but you didn’t finish your stories for the week.”

“Your teammates don’t feel you’re focusing on the work you’re given, and they get the impression you may not be pulling your weight.”

When everything added up, I found myself in the middle of the pack at Gusto. I wasn't yet reliable enough to be a senior-level engineer.

I wanted to do better. I thought about how working in a codebase with history and complexity made me feel. I discussed my feelings about code and engineering with my PE frankly and openly. I found that I'm scared of learning hard things.

Figuring Out What’s Wrong

At my old gig, I could almost always build a project from scratch. Because I built them myself, I understood everything about how they worked. Gusto presented me with intense problem domains, dozens of engineers committing at once, and a complex series of interlocked systems with side effects. It was petrifying to work with this codebase. What if I made a mistake? I could annoy my coworkers, upset our customers, and cost the company real money.

I found it easier to change only the parts I knew. But that caused me to leave projects without a deep understanding of their domain. I couldn't apply that knowledge to new projects I joined, even ones adjacent to my past work. And I felt less effective coming back to fix bugs in code I had written before.

My trepidation over working in an established codebase had led to a phobia. Our primary repo has over five hundred thousands of lines of Ruby code, interfaces with dozens of state and federal agencies, and has thousands of edge cases. I needed to make conscious decisions to get over my fear of the unknown and build my knowledge of the scary, cobwebbed corners of our payroll codebase. So my PE and I came up with actions I could consciously take to drive my independent study and help me learn through my work.

Pictured: I investigate tactical approaches to independent study and the growth-oriented mindset.

Changing My Approach

I aimed for the hard stuff on purpose. For example, I preferred stories that dealt with our automated tax filing system, rather than stories around building the frontend components I had grown comfortable with. During sprint planning meetings, I chose to take the tasks I knew nothing about. I hoped to learn more about the unknowns and gain a better understanding of our systems as a whole.

I spent more time investigating before asking for help. I used to spend about ten minutes investigating a difficult problem before asking for guidance. To change that, I came up with my personal Two-Hour Rule: I had to work at a problem for two hours before deciding it was time to ask someone for help.

I chose to avoid asking for help until I could describe my problem in detail. I used to ask for help after a brief period of not understanding the code I was looking at. To remedy this, I spent more time observing and thinking, then describing my understanding of the codebase on paper. I jotted down diagrams in a small notebook to help organize my thoughts and my mental models. For example, when I worked on a feature that interacted with our complex Historical Payroll model, I wrote pages of notes and diagrams that I eventually synthesized into a page of documentation for our company wiki.

To me, this was the road less traveled, but I found it made an immediate difference in the quality of my work. When I chose to stick with a difficult bug instead of asking for help, I often found I learned a lot about how the bug happened and how to fix it. In the end, I realized I didn’t need help after all. When I did need to ask for help, I could describe the problem completely, accurately, and point at the exact bits of code I didn't understand. When I left a project, I was able to write clear, concise documentation about what the code did and why – even if I hadn't written it originally.

Pictured: My new approach to diving deep and documenting my work yields results.

Looking Forward

As I move forward, I want to learn and grow in my abilities. It’s only been a short time since I adopted my new strategies, but I’ve already seen improvement in my ability to grok complex problems in my projects. I want to become a great engineer at Gusto, I want to grow as a skilled engineer in general, and I want to learn how to learn so I can continue my trajectory. I look forward to seeing how the skills I’m teaching myself will help me become a better leader at Gusto and wherever I go.

Gusto has been an incredible place for me to learn the skills around working with a large team on complex problems. I've received nothing but support as I grow in my abilities and develop my strengths. Come work with me to create a world where work empowers a better life! Check out our open engineering positions in San Francisco and Denver.

An Opinionated Guide to Fixie Commuters

Fri, 04 Nov 2016 04:52:03 GMT

Two months ago, I picked up a basic little road bike and turned it into a fixie commuter.

What's so great about a fixie?

They're simple

Fixed-gear bikes are way simpler than geared bikes. Reducing the number of moving parts on your bike means you have less stuff to maintain. Less stuff is liable to break.

When stuff does break, you have less cascading failures to fix before you can get back on the road. The last time I tuned up a hybrid bike, I had to clean the bottom bracket, replace and jerry-rig a new front chainring, and recalibrate the rear derailleur to get it working again. The last time I worked on my fixie, I replaced the rear cog and lubed the chain.

When you turn a road bike into a fixie, here's what you'll replace:

Rear cassette → Single cog
Rear derailleur → Nothing!
Three front chainrings → One front chainring
Rear brake → Nothing! (Aesthetics! If you want.)
Front brake → Nothing! (Don't do this. You should really have a brake.)

They're romantic

I still feel that variable gears are only for people over forty-five. Isn't it better to triumph by the strength of your muscles than by the artifice of a derailer?

There is an almost mystical connection between a fixed-gear cyclist and bicycle: it feels like an extension of your body to a greater extent than does a freewheel-equipped machine.

Riding a fixie is a different experience. Pedal faster to go faster. You'll feel if your rear wheel slips on gravel because your feet will slip too. Get ready to spin your legs when you hit the downhill!

How to Convert a Road Bike

The easiest way to start riding a fixie is to buy an already-built fixie. The second easiest way is to buy a cheap vintage road bike and take off most of the parts.

You'll be looking for a bike with horizontal or mostly-horizontal dropouts. That's because you're removing the derailleur, which tensions the chain, so the only way to tension the chain is to move the rear wheel forward and back in the frame.

Here are the types of dropouts that will work and won't work to tension your chain:

From Sheldon Brown's site

Strip the not-fixie parts

There are a lot of parts to strip! You'll need to remove these:

Rear wheel (the entire thing)
Front and rear derailleur
Shifter levers
Shifter cables

And optionally:

Rear brake (if you want less stuff on your bike)
Front brake (again, please don't. Keep the front brake.)

Take off these parts, one by one, and put them in a bin to donate to your favorite community bike shop. I gave mine to Bikes Together, an awesome Denver non-profit.

Resize the chain

Your old chain is too long for your fixie. You need to shorten it so that your chain is tight between your chainring and rear cog.

If you'd like to replace the chain, now is a great time. If you choose to do this, here's how to tell what width chain you have.

Replace one or both wheels

Your geared road bike's rear wheel has a cassette. This lets you change gears using the derailleur. It slides onto a freehub which lets you coast. You want neither of these things – you don't want your fixie to coast!

Instead, you want a rear wheel with a fixed hub and a single cog on each side. You'll often see these wheels called "flip-flop," "fixed-free", or "fixed-fixed." Flip-flop and fixed-free mean the same thing: one side has a fixed cog that moves the pedals with the wheel, and one side has a cog that lets you coast. Fixed-fixed wheels have a fixed cog on both sides.

If your bike has its original parts, it might still have 27" wheels. It's a lot harder to find modern tubes and tires for a 27" wheel than a standard road bike 700c wheel. So now is a great time to upgrade to a 700c rear wheel. And as long as you're at it, you might as well replace the front wheel as well.

Don't forget long-reach brake calipers: The radius of a 700c wheel is 4mm smaller than that of a 27" wheel. This means your brake pads will have to reach 4mm farther to make proper contact with the rims.

Your existing calipers might not have 4mm of wiggle room to work with your new wheels. Make sure to check before you buy. If you need more reach, there are several options available for long-reach brake calipers.

Get new tires and tubes

Hooray! Your new wheels are here. Now you get to experience the candy shop that is tire selection.

There are a couple of factors that all trade off against each other when you're buying tires:

Flat protection: Heavier, more armored tires can roll over sharper stuff without damaging the tube inside.
Rolling resistance: Heavier tires have more rolling resistance. This saps all your precious forward momentum and means you spend more energy going the same distance.
Agility: Some tires are more nimble around corners and in slick conditions than others.
Weight: More weight generally correlates to more durable and longer-lasting. It also makes you slower.
Price: You get what you pay for. Cheap tires won't be good at anything listed above. But even expensive tires will wear out down the line.

To find a tire that's best for you, check out Rolling Resistance reviews. They cover all the tradeoffs of every road bike tire you should consider.

Replace the ancient brake levers and bar tape

Vintage brake levers and cables look like this:

If your bike comes with these, they are probably original and they probably really suck at translating hand pull into braking action. And if you're like me, you want to minimize the exposed cables on your clean frame.

I like it when my cables pop out from under the bar tape:

Consider buying road bike brake levers so you can safely ride on the hoods, and a crosstop lever if you ride on the tops a lot.

Once you have those, you need to install the levers, then wrap the bars over the brake cables. Although it can be nervewracking, this is actually a lot of fun. Find a color you like at your LBS or online and buy twice as many rolls as you need. That way, if you screw up, you can just try again.

Retent yo foots

From Cycling Basics: How to Cycle with Clipless Pedals

Foot retention does two important things:

Keeps your feet glued to the pedals. You can't coast on a fixie. Your pedals move when your wheels do. If your feet slip off the pedals while you're moving fast, it will be really hard to get your feet back onto the pedals. That's a recipe for a crash.

Lets you pull up for more power. You only have one gear now – time to make it count. When your feet are bound to the pedals, you can pull up with one foot while you press down with the other. This lets you apply more force and means you don't need a higher gear as much.

Going up a hill? Focus on the pull of your back leg – your body will naturally place weight on your front foot. It's like a magic sixth gear you never knew you had.

Pulling up and cranking hard with both feet also helps you start from a dead stop. It's very convenient when you're in the left lane at a light that just turned green and you want to keep the drivers behind you happy.

What's a clipless pedal?

Confusingly, a clipless pedal uses a metal clip mounted to your shoe to hold your foot to the pedal. The pedals are shaped so that they latch onto the clip in your shoe.

To clip in, you simply step into the pedal with the clip in your shoe. Slide the toe side in, then step down until you hear the click. To clip out, rotate your ankle outward away from your bike.

There are two main types of clipless systems.

Off-road systems use small clips that make it easy to clip out. Lots of casual shoes offer recessed mounts that let you walk around without crunching your clips against asphalt.

Shimano SPD is the most popular off-road system. Other competitors are Crank Brothers, with the egg-beater design, and Speedlite with the hockey puck pedals.

Road systems use large clips. They are intended to retain your feet with more force. Their larger contact area lets you transfer power more efficiently to the pedals for long rides and races. Popular road systems include SPD-SL, Look, and Time.

If you don't know what you need, start with the Shimano SPD M520 pedals. Then search for some classy, walkable SPD shoes and go from there.

What are my other options?

Toe clips are metal or plastic cages that surround your toes. You can use normal shoes in toe clips. You just slip the front of your shoes in and pedal normally.

Toe straps are velcro straps that fit around the front of your pedals. Slide your feet through them, pull them tight, and velcro them down. They're more secure than toe clips, but they're harder to bail out of in a crash or at a stop light.

Platform pedals are what you find on most peoples' bikes. They're flat on both sides. They don't come with straps, clips, or cages on their own. I don't recommend you use platform pedals without additional foot retention.

Enjoy your new old bike!

Practice safely! Learn to clip in, clip out, and spin before you take this on your daily commute. Everyone has the embarassing sideways slow fall when they come to a full stop but forget to unclip in time. Try to get that out of the way at a park before you end up doing it at a stoplight.

How to get stuff from Craigslist at ridiculous prices

Mon, 29 Aug 2016 03:31:44 GMT

Here's my strategy to getting the used stuff I want at insane prices. I've used this to buy bicycles, motorcycles, home furniture, and kitchen equipment. If the thing you're looking for isn't one-of-a-kind, I think you'll have luck with this approach.

As a brief introduction, here's a story about something I bought recently.

I love two-wheeled vehicles. Ever since I moved to Denver, I've been getting to work on my dual-sport motorcycle, and more recently, my cheap mountain bike. While it's fun to spin my legs at 120 RPM and slam my suspension over curbs, my single-speed, knobby-tire mountain bike isn't quite the right tool for a commute.

I was keeping my eye out for a cheap, steel-frame bicycle for a couple of weeks. Last week, I found this beauty on Craigslist and snapped her up. Meet my new (1989-ish) Nishiki Performance Equipe Sport:

I found this two-wheeled friend listed for about 70% less than comparable bikes on Craigslist. Everything is in great shape. Wheels are true, rust is minimal, and the previous owner upgraded the suicide shifters to indexed shifters on the bars! In short, I love my bike and I got it for a steal.

Enough about me – here's how you can find what you want and spend less than you planned.

Get obsessed

If you know me, you know I get obsessed about something I want.

Two years ago, it was scuba diving.
One year ago, it was motorcycling.
This year, it was my road bike.
My entire life, I'm going to be obsessed with planes and gliders.

If you're obsessed with the thing you want, you'll be spending all your time doing research. Search for what you want on Craigslist, on Reddit, and on thing-specific forums. There are awesome communities out there with FAQs and glossaries of all the technical terms you'll need to know when you buy something used.

Here are a few of the search terms I found useful during my hunt:

how to buy used road bike
vintage bikes subreddit
how wheel sizing works
steel vs aluminum bike frame

You won't know everything by the time you buy, but you'll keep building on your working knowledge as you go. Just make sure you keep learning and keep an open mind.

Watch the market

If you're like me, you've already had dreams about bicycles at this point. You should be fully invested and want to buy off some rando on Cragislist RIGHT NOW!! But don't do it.

It's time for you to channel your obsession. You're going to watch the market with the goal of figuring out what you want and what it's generally priced at.

My first step was to go to Craigslist and search for road bike. Turns out there are a lot of these. So I started looking through the results like a kid in a candy shop and crossing off the bikes that I didn't want. Either they didn't fit me, or they were too pricey, or they were from a cheap low-quality manufacturer.

After about a week of drooling over used bikes, I started seeing a pattern. You will too. You'll notice that seeing something specific will make you reject the listing completely, or you'll associate a certain brand with a specific price point and quality.

Set your standards

Once you start seeing a pattern, you've started to know what you want instead of just guessing. Now you're going to define exactly what you want.

Set your standards high and be specific. This is where you save your money!

You know what the average price for your thing is – your max should be 75% of that or less. Tons of people on Craigslist want to get rid of their perfectly good stuff immediately, and they're willing to cut a deep discount to do so.

You know what makes the thing you want quality. In my case, specific Japanese brands – Nishiki, Panasonic, Univega – made awesome bikes in the 80s and 90s that often get overlooked today. This means their prices go way down.

Here's what I thought I wanted going into my search:

Road bike. Thin wheels. Frame that fits me. Working.

Here's what I knew I wanted after I watched listings on Craigslist:

Road bike. Thin wheels, reasonably slick, preferably 28mm. 50-54cm vintage Japanese frame, cromoly steel. Drop bars. Minimal rust. Non-suicide shifters. Working. $150 or less.

Get notified

Now you know exactly what you want. You know you want your high-quality thing at a low price. Unfortunately, this is exactly what other people want too. You'll have to beat them to the punch.

Start by building your Craigslist search. Go to your Craigslist site, enter your search query, and run it. Here are some tips that served me well:

Use the OR | and NOT - operators to specify multiple keywords you want and filter ones you don't. Here's an example Craigslist query for Nishiki, Panasonic, and Univega bicycles that aren't mountain bikes:
nishiki|panasonic|univega -mountain
Set a minimum price limit. Jokesters list stuff as $1 to ensure it appears in searches with upper price bounds. You know no one will just give your thing away for $20 – so set your minimum to $20 to filter the noise.
Set the maximum price limit to that price you set above. It's important for you to stick to your guns on this one, even if you have FOMO for search results. You'll find a deal eventually.

Great! You've got your search dialed in. Now, copy the Craigslist URL for your search. Use it to set up an IFTTT recipe to email you when a new item pops up on Craigslist for a specific search. You can also use push notifications or SMS – they're a bit more urgent.

Jump on it

Why is the notifier important? Because if you've found a good deal, you need to jump on that immediately. Good deals on Craigslist don't last three days. They rarely last one or two days. When you see that notification, excuse yourself from your meeting, lunch, or therapy session, and call the lister immediately.

Craigslist people tend to have a specific code of honor. If you're the first person to call them and set up a meeting, they'll usually tell other interested parties that they have to wait until you swing by and take a look. The quicker you contact, the more likely you are to get the thing. But to take advantage of this, you need to show up that day, so swing by the seller right after work and leave your kids late at day care. (Kidding. Probably.)

Profit!

If you're doing this right, you might get the thing of your dreams in one to two days. You should find something you love in two weeks, unless the market you're searching is unusually small. And you know that you'll love it because you've been looking for something very specific at a specific price. Congratulations!

What about negotiating?

I'm gonna be honest. I suck at Craigslist negotiating. If I'm saving less than $100, negotiating makes me feel uncomfortable enough that I don't bother. And at the end of my search, I've usually found an item that's way, way under market value, so I feel good enough about the price already.

If you're interested in driving a hard bargain, there are lots of other resources to teach you better than I can.

Good luck!

I hope this guide helps you get the thing you want at a price you love. If any of these tips help you, or if you have other methods you use to find great deals, drop me a line or leave comment on this post. I love hearing from readers!

Finding a Career You Love: Lessons Learned From My Job Hunt

Tue, 19 Jul 2016 21:58:58 GMT

I'm writing this article for someone who works in tech and is looking for a new job, but wants to be somewhat selective and build stuff they believe in. If that describes you, read on!

On June 3, I left my job at Punch Through, a hardware consulting firm in Minneapolis and San Francisco. My wife and I moved to Denver, CO. She's attending pharmacy school in the fall and I was looking for something new in my career.

Before I left Minneapolis, I started looking for a new job in Denver. Here's what I learned and what I would do differently.

Overview

What I Wanted: An overview of my target career and why I was so picky.
How I Searched: The boards and networks I used to find my dream job in Colorado.
How I Applied: How I got my foot in the door and how I kept the conversation going.
General Tips: Everything else that I think will help people searching for an awesome place to work.
How It Turned Out: The results. How many jobs I applied for, how many companies turned me down, and how many offers I received.

What I Wanted

I was in the market for

a product manager role,
or if that wasn't available, an engineering role with room to grow into management,
with a team that cared about quality
and worked on a product I believed in.

I loved my coworkers and the projects at Punch Through. I worked there for three years during and after college as a software developer. My primary work was on iOS, Android, and web software for LightBlue Bean, a Bluetooth Low Energy development board for makers and product designers.

Our team was small – about 16 people when I left – and we all had lots of ownership over the things we built. I enjoyed the responsibility of ownership and liked managing the Scrum backlog. I made it my short-term career goal to get into a position where I was formally responsible for a product's direction. Since my experience was in engineering, my friends and coworkers advised me to consider taking a dev role with the possibility of growth into management.

When I searched for companies to apply to, the specific product and industry were very important to me. If the best company in the world were hiring, but they weren't building a product I could be proud of, I would never apply for a job there. I avoided companies doing marketing and building social networks, and I refused to work in advertising tech.

How I Searched

I searched for leads through a few channels:

Personal network and friends of friends
Job sites: Glassdoor, Stack Overflow Careers, LinkedIn
Community: Denver Devs, TechFriends Front Range, Built In Brews
Recruiters: Dice

Personal Network

I reached out to my coworkers from Punch Through and friends living in Colorado and asked if anyone knew anyone building anything cool. Everyone was happy to pass along what they knew. I got very few leads through my personal network, but the ones I did get were all high quality – no fussing about, let's meet up for coffee and chat.

Job Sites

I searched job sites for the keywords Product Manager, Developer and Engineer.

Glassdoor has excellent data on job satisfaction which helped me get a quick idea of whether a team's culture was healthy. When most people are complaining about management not listening, not communicating, and hiring in managers from outside, that's a big set of red flags.

Stack Overflow Careers has less jobs listed, but most of them are high quality. People posting jobs on SO are encouraged to list the job's salary range, which is a breath of fresh air.

LinkedIn has everything but it's incredibly hard to find relevant postings or info. This is a last resort. What you should be using LinkedIn for is to find the name of a hiring manager at Acme Corp, figure out their email, and send them a nice cover letter directly. More on that later.

Community

All three of the communities I listed were excellent leads for awesome companies. The Denver Devs Slack channel has a weekly Gigs Day: employers post about open jobs and individuals post if they're looking for a job. It's a great way to skip the recruiting site and ask some quick questions to see if you're a good fit.

Built In Brews is a fantastic networking event. Show up at an awesome office, have some food and drinks, and chat with other awesome people. I got in touch with a couple of excellent recruiters who set me up with great companies. Their parent site, Built in Colorado, also has a job posting board with lots of great leads.

Recruiters

The easiest way to get recruiters to do the legwork for you is to put your resume on Dice and list your relevant skills. You'll get calls and emails for the next month with stuff that is – surprisingly – mostly relevant! The quality is mixed but there are a couple of diamonds in the rough. For example, many of the recruiters that reached out were filling Contract or Contract to Hire roles, even though I was seeking full-time exclusively. But the calls I got for full-time roles were mostly real jobs where my experience was relevant.

One recruiter was a complete asshole. They called me once to see if I was interested in the role and the industry. I was, so they sent over the job details. I liked them, so I scheduled a Skype call. Here's how that call went once we got past small talk:

Them: "How much did you make at your last job?"
Me: "Sorry, I prefer not to say."
Them: "I really need to know."
Me: "Sorry, I don't want to cap my next salary by stating that. But I can tell you what I'm targeting."
Them: "Let me worry about that. What did you make at your last job?"
Me: "...I really can't give you that number for several reasons."
Them: "I understand. Sorry this didn't work out. Have a nice day." Skype hangup

I don't think I could fix that recruiter's broken behavior. There are so many things wrong with what they said and did. Between the blatant disrespect, lack of professionalism, and the Skype equivalent of slamming the phone down, I'm amazed that Asshole Recruiter moves forward with anyone in the hiring process.

Pro tip – don't say Let me worry about [your salary], because I will worry about my salary. Thanks.

How I Applied

OK, I have a big pile of job listings. How do I turn those into interviews?

Here are the approaches I used, from most to least successful:

Get a personal introduction from a friend of a friend and send them a nice cover letter
Find the hiring manager's work email and send them a nice cover letter
Apply for the job online with a nice cover letter

Notice anything in common?

The Cover Letter

This is the cover letter skeleton I used for 90% of my job applications. I think it struck a balance of giving them information about me, letting them know what I wanted, telling them why I'd be a great pick, and asking to move forward in the process.

Hello! My name is $NAME. [Why I'm looking for a job right now.] My last position was as a $ROLE at $COMPANY, a $TYPE_OF_COMPANY in $PLACE. I wanted to get in touch with you because I’m looking for a position building $TYPE_OF_PRODUCT, and I really like the sound of $COMPANY. I think your company is a great place to $PROFESSONAL_GOAL.

[Brief history of my time at my last company. Any of my relevant experience with impressive KPIs attached.]

I want to work with $COMPANY because $COMPANY_CULTURE. [Short blurb about stuff I saw on their site or read about the company that would make me happy to work there.]

I feel my experience in $RELEVANT_EXPERIENCE will fit right in at $COMPANY. [Sentence about how my personality fits with the company culture. Sentence about how my experience is relevant. Sentence about how I can start adding value as soon as I'm hired.]

I’d like to chat with you sometime. I want to learn more about $COMPANY's needs and let you find out if there’s a place I would fit in.

Thank you for your time!

$NAME
$WEBSITE
$PHONE_NUMBER

Please feel free to steal this template and use it yourself. This letter is brief, to the point, and professional. If your paragraphs start looking longish, consider shortening them. It's a cover letter, and the goal is to get the hiring manager on the phone so they can ask about what they really care about.

It's important that this letter has your voice in it. Think business casual, not T-shirt and shorts. Some excitement (!) and some talk about your personal reasons for looking for a new job is OK. Emoji is not.

Always ask for the job! It's important to ask for the job throughout the process. It's not aggressive – it's professional. Asking for the job makes it clear that you're interested and want to be put at the top of their list.

After an email, ask when they're available for a call. At the end of a phone call, ask what the next steps are. After an interview, ask when you should expect to hear back from them on a decision.

Great – you have a cover letter. How should you get it to the hiring manager?

Personal Introduction

If you can get someone to CC you into a conversation with the person responsible for hiring, you have just saved yourself weeks of waiting for them to check the inbound resumes. Ask your friends if they would be comfortable with introducing you to hiring at their company, and nudge them until they send the email.

Email the Hiring Manager

There are a couple of tricks to this one. You can often leverage LinkedIn to help you find the hiring manager. Once you know who they are, you can usually guess their email based on the email addresses of other people at the company. Here's a good set of tips.

Just Apply Online

When you apply online, assume your resume will go to the glorious digital circle file and never be seen again. I consider it a happy accident when someone actually responds to my online application. Don't forget to write a cover letter for online apps – it goes a long way.

General Tips

Here's stuff I wish I had known when I started my job hunt.

Plan For A Long Hunt

My job hunt took ten weeks total. When you start your search, plan to be looking for at least a few weeks.

A few weeks?!

Yes, a few weeks. The longer you hunt, the more you apply, and the later you wait, the better your odds of finding the job you really want. Unfortunately, that gets a bit stressful when you don't have steady income coming in.

I wish I had had a job lined up by the time I moved to Denver. If I were to do this all again, I'd start hunting a month earlier.

Set Clear Expectations

In my first few interviews, I told the interviewer I wanted a position in management even though I was applying for a dev role. I told them that I would be happy to grow into a new role. This spooked my interviewers – they were worried I would be dissatisfied with my job and leave if I didn't get a promotion to management quickly.

I honed my approach as I went through more interviews. Instead of saying "I want to move to a mangement role sometime," I asked about the company roles and culture. I asked if the company was big enough to hire product managers, and if they tended to hire from inside. I asked questions like "If I were to work here and I liked it a lot, is there room to grow?"

Those questions helped me understand what roles I could move into without making them think I wouldn't be happy with an engineering role.

Be So, So, So Organized

Use iCal. Use Google Calendar. Use Fantastical, my favorite Mac and iOS calendar app. It doesn't matter what tool you use – just use some tool. When you apply for five to ten jobs a day, you will need a calendar to tell you who you're talking to and when.

Show up early. 30 minutes early is acceptable. 15 minutes early is ideal. If you arrive on time, you are late. When you're trying to find parking in downtown Denver, you'll pass the lot you want twice. Then you won't be able to find the office. (Not that I know from experience.)

If you screw up, completely flub an interview time, and don't show up, you might be screwed. But you can usually salvage it if you are very apologetic and polite and extremely thankful that they gave you another shot.

Always Say Thank You

Any time you finish a phone call, a meeting over coffee, or an in-person interview, send that person a thank you note the next day. It doesn't have to be anything fancy. Just email them saying you appreciated their time and mention what you liked about the last interaction. For example:

I was happy to learn about the role and my day-to-day responsibilities.

I liked meeting Sam and getting a feel for the engineering culture at E Corp.

I enjoyed learning about the roles you have available and I'd like to learn more.

Sending the thank you note the next day keeps your interaction fresh in their mind. Don't forget to ask for the job!

Be Politely Pushy

You've gotten in touch with hiring. Now you're sitting and waiting for them to get back to you. What's the etiquette on bugging someone you hardly know after they don't email you for weeks?

My rule is: Wait three to five business days and ping them again. You need a job and being too polite will hurt your chances. If someone hasn't returned your email or scheduled a call with you, send them one of these:

Hey $NAME,

Just wanted to check in and make sure I'm still on your radar. I'd like to move forward at $COMPANY. Are you available for a call this week?

Thanks,
Matt

It's a polite reminder that you're still interested but do need a response soon. You can send a few of these in a row if you wait a few days and still don't hear back. If you send three of these and don't get anything back, assume they ghosted you and aren't interested.

Set A (Fake) Deadline

I started applying for jobs on May 2. I accepted a job on July 16. This was way longer than I thought I would spend job searching.

Once I got my first offer from a Boulder dev shop, I asked them for just over a week to consider the offer. Then I told all the other companies I was still interviewing with that I had an offer on the table. Once I told them I had a deadline, the managers hustled to get me through the process. One mid-sized company flew me to SF in the morning and back at night the day before my deadline, while another got me a last-minute on-site appointment to finish up the process with two engineers and a hiring manager.

The lesson? Most places can lift heaven and earth to move you through the process – but they aren't incentivized to unless you tell them they have to. The next time I seek a job, I'm setting a deadline to encourage companies to move it.

How It Turned Out

I applied to 26 companies. This doesn't include recruiter gigs that fell through or positions I wasn't seriously interested in. Here's how those turned out:

9 companies dropped out because I couldn't get in touch with them.

4: No response. I applied online and no one got back to me.
5: Ghosted. I started talking with a recruiter or hiring manager and they refused to respond after I pinged them several times.

13 companies didn't give me an offer for one reason or another.

2: Situation mismatch. They were looking for something totally different. One manager wanted to hire someone like me but couldn't get a budget in time. One team found a lead developer in Pennsylvania and wanted me to relocate.
3: I declined to continue the process. I didn't have time to complete their code test, or they got back to me too late in the process and I had to accept an offer.
8: Job not offered. I wasn't a good fit for the position or they found someone they preferred.

4 companies got back to me with an offer. I turned down three offers and accepted one.

Conclusion

I feel OK about the job hunt. Four offers out of 23 serious applications sounds low, but at each step along the way I learned more about the companies and the roles I was applying to. It was occasionally disappointing to be turned away, but I'm glad I didn't end up working somewhere where I would be a poor fit for the role.

I'm happy where I ended up: a Project Management position at a connected device company. That's exactly what I was looking for when I started my job hunt, and I'm glad I found a growing company that was looking for someone like me.

I don't look forward to searching for a job, but I'm confident I will be able to navigate the interviews and hiring process much more easily next time.

You got LaTeX in my Markdown!

Tue, 21 Oct 2014 06:51:05 GMT

Working title: You got Markdown in my LaTeX!

I hate writing documents in Microsoft Word and Apple Pages. I spend way too much time fiddling with formatting, page layout, and typesetting before everything looks the way I want it. And writing math equations, even with Word's GUI equation editor, is tedious and takes too much clicking. I'm a programmer—just let me use LaTeX's equation format instead!

tl;dr If you want to write documents in Markdown, embed LaTeX, and generate PDFs, use Pandoc.
Skip down to Kung Fu Pandoc to get started!

So just use LaTeX, they said. It'll be fun, they said.

I like typesetting tools, and I like LaTeX... with reservations. Some of my complaints:

The boilerplate needed to get a TeX document up and running is pretty massive.
LaTeX is a pain in the ass to write. There is so much character overhead behind basic formatting.
- Bold? \textbf{bolded text}.
- Inline code? \texttt{for (int i = 0; i < 42; i++)}
- Basic tables? Here's a short reference.
Holy reserved characters, Batman! Here are things you can't write in LaTeX without escaping:

# $ % ^ & _ { } ~ \

It's not like I'd ever want to use a dollar sign in my budget report or an underscore in my C code.

If you don't escape a character properly, you'll probably get a syntax error. LaTeX's syntax errors are cryptic and terrifying:
- Missing $ inserted: could mean just about anything, such as: you put too many lines inside an equation block, or you forgot to close a curly bracket.
- No line here to end: means that you tried to continue a line when LaTeX "didn't expect it". When does LaTeX expect it? God only knows.
- Not confusing enough for you? There are more errors where those came from.

Wouldn't it be nice if...

This semester, I tried to write an assignment for class and found out my LaTeX build chain in Sublime broke. Instead of trying to troubleshoot it, I started thinking about what I'd like to use instead.

Here's why I love LaTeX:

It lets me write equations quickly. Its equation syntax makes sense to me, and it's easy to ~~steal~~ import equations from Wikipedia pages.
It's a typesetting program, not a WYSIWYG. No more screwing around with weird margin sliders and page layout tools and header settings: simply throw in your text, edit the settings once, and watch all your text reflow into place.
It's easy to generate a PDF from a LaTeX file.

Here's why I prefer Markdown as a language to write formatted text:

It's SUPER lightweight. **bold**, *italic*, ```inline code` ``.
GitHub-flavored Markdown is super cool. It adds support for syntax-highlighted code blocks and tables! I like it so much I built a tool to generate Markdown tables from CSV files.
You probably already know its syntax if you use GitHub or Reddit frequently.

I asked myself: "Self, wouldn't it be great if there were a typesetting tool that let you write documents in Markdown, embed LaTeX where you needed it, and generate PDFs?" And it turns out that already exists.

Kung-Fu Pandoc

If you need to convert files from one markup format into another, pandoc is your swiss-army knife.

—John MacFarlane, author of pandoc

We'll use this Swiss Army knife called Pandoc to compile and export our Markdown-LaTeX docs. Let's try it out!

Install Pandoc and LaTeX

You'll need Pandoc and pdfTeX installed. If you can run pandoc and pdflatex from a terminal, you're probably good to go.

If you're on a Mac, install Pandoc using Homebrew (brew install pandoc) and install MacTeX using their .pkg installer.

Get Your Source Doc

Here's a sample Markdown-plus-LaTeX document you can try to compile. Save that file somewhere as example.md.

Compile Your Doc

Here's the command I used to compile example.md —> example.pdf:

pandoc -o example.pdf example.md

See? It's that easy. Now open your PDF and you should see something like this:

Awesome. You're one step closer to well-documented world domination.

Automate That Ish with Sublime Text's Build Systems

Great! Now you can build Pandoc Markdown docs manually from the command line.

If you use Sublime Text like me, you might use the Build command (F7 or Cmd+B) to build and run your scripts. When I used LaTeXTools, I got really used to checking how my LaTex looked when it was rendered by hitting Cmd+B. The LaTeX build command rendered my source document and opened a PDF viewer with the results.

I duplicated this in Sublime Text. Here's how you can too!

Create a new Sublime-Build file

Sublime-Build files tell Sublime Text how you want to build a document when you hit Ctrl-B.

Create a new Sublime-Build file by selecting Tools —> Build System —> New Build System...

You'll get a document named untitled.sublime-build that looks like this:

{
    "shell_cmd": "make"
}

Write Your Pandoc Build Command

Replace the contents of your new document with the following:

{
    "shell_cmd": "pandoc -o \"$file.pdf\" \"$file\" && open -a Preview \"$file.pdf\"",
    "selector": "text.html.markdown",
    "path": "/usr/texbin:$PATH"
}

Then change a few things:

If you're not on a Mac, remove the `open -a` command.

open -a Preview \"$file.pdf\"" is just for Macs—it opens the rendered file in Preview.app.
If you're not on a Mac, strip off the && characters and everything after them. Your shell_cmd will look like this:

{
    "shell_cmd": "pandoc -o \"$file.pdf\" \"$file\"",
    ...

Remove the `path` setting if you don't want it. Easier: just leave it in.

I found my Sublime install couldn't find pdflatex unless I added this directory to the build system's path directly. You can probably leave it in—it shouldn't do any harm as long as Sublime can find pdflatex.

Save Your New Build Command

The default save location will be Sublime Text 3/Packages/User. Save the file you just created into that directory as Markdown to PDF.sublime-build.

Set "Build System" to "Automatic"

Select Tools —> Build System —> Automatic to ensure Sublime picks your new Markdown build system.

Build a Markdown File!

Open the example.md file you downloaded earlier in Sublime, then hit F7 or Cmd-B. The Sublime Text console should open as it starts compiling your document, then print something like this when it's done:

[Finished in 2.4s]

Check the directory holding example.md and look for a brand-new example.md.pdf file!

You did it! I am very proud of you.

Bonus Round: Custom LaTeX Templates

You're a cool guy who knows cool stuff. Want to make your LaTeX docs even cooler? Add your own custom template.

Say I hate the default LaTeX monospace font and I want to add \usepackage{inconsolata} to fancy up my code blocks. Here's how I can do that:

Save a LaTeX template somewhere on my computer
Edit that template and add the LaTeX packages I want
Ask Pandoc to use that template every time I compile my documents

Get a LaTeX template

I used this default.latex template provided by jgm. Seems to work great!

I saved mine as ~/.pandoc/default.latex because I think that's where Pandoc templates usually go.

Edit that template

At line 134, I added some text to remind me where I can safely put \usepackage commands, as well as my custom command:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% ADD YOUR \usepackage{...} COMMANDS HERE
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{inconsolata}

Make Pandoc use that template

I edited my shell_cmd inside my .sublime-build file from the following:

"shell_cmd": "pandoc -o \"$file.pdf\" \"$file\" && open -a Preview \"$file.pdf\"",

I added a command-line argument pointing Pandoc to my default template as shown:

"shell_cmd": "pandoc --template=\"/Users/mplewis/.pandoc/default.latex\" -o \"$file.pdf\" \"$file\" && open -a Preview \"$file.pdf\"",

Now Pandoc uses my new template and makes my typewriter text not suck. Hooray!

Building an iOS Cocoapod that uses the Dropbox Sync API

Fri, 08 Aug 2014 23:29:03 GMT

One of the frameworks we're building at Punch Through Design links to Dropbox to help users access their files from both iOS and their computer. Dropbox provides a very useful Sync SDK for iOS that makes it easy to access Dropbox accounts.

Installing the Dropbox SDK can be complicated, so I simply used Cocoapods to install the Dropbox iOS Sync SDK for use with our framework.

I wanted the framework to be easy to install into any project, so I made efforts to distribute it as a Cocoapod. This caused problems: I needed to write a Podspec that included the Dropbox SDK as a framework dependency, and the Podspec documentation is sorely lacking. I couldn't figure out how to write a podspec that included the Dropbox SDK as a pod dependency AND linked the Dropbox framework so Xcode could find all its headers.

We ended up stealing some lines of code from the ParcelKit podspec—they also list the iOS Dropbox SDK as a Cocoapod dependency, and they seem to have it figured out.

Here are the important lines of code from our Podspec file:

Pod::Spec.new do |s|
  # ...your code goes here
  s.frameworks   = 'Dropbox'
  s.dependency 'Dropbox-Sync-API-SDK', '~> 3.0.2'
  s.xcconfig     = { 'FRAMEWORK_SEARCH_PATHS' => '"${PODS_ROOT}/Dropbox-Sync-API-SDK/dropbox-ios-sync-sdk-3.0.2"' }
end

I believe you'll need to change the framework search path if the Dropbox SDK version changes. This code helped me successfully link our Podfile with Dropbox, minimizing the amount of damage we had to do to our .xcodeproj file.

Submitting a Python CLI Tool to PyPI

Mon, 21 Jul 2014 00:19:15 GMT

Last week, I built a command-line tool, csvtomd, to help me write documentation for my personal and work projects. Building Markdown tables by hand can be tedious and often involves lots of manual spacing tweaks to make the source code look just as good as the rendered table. With csvtomd, you can build a table in Excel or Numbers, export it as CSV, and convert the CSV to a Markdown table.

I wrote this tool in Python 3 and wanted to make it available to other developers. The best way to distribute Python 3 packages is to upload your package to the Python Package Index, or PyPI. Once a package is on PyPI, other developers can install your package with a simple command:

pip3 install csvtomd

Helpful Help

Here are a few of the articles that helped me put my package online:

The most useful tutorial I found was Sharing Your Labor of Love: PyPI Quick And Dirty by Hynek Schlawack. It's a great, simple introduction to getting your Python code ready for PyPI and uploading it as a standalone package.

PyPI and CLI Scripts

One of the hiccups I discovered while trying to install my CLI app was that all the tutorials I found involved creating packages for use in other Python scripts. I had to figure out how to set the entry point to my application so that I could run it properly from the command line.

My solution was to add this entry_points line to my setup.py:

setup(
    ...
    entry_points={
        'console_scripts': [
            'csvtomd = csvtomd:main'
        ]
    },
    ...
)

To make this entry point work, I had to modify my csvtomd.py script. Originally, my script's main module code—parsing arguments, running the conversion—was located outside a function and ran in the body of the main script. I had to move that code into a main() function.

For convenience, I added a runner for executing the script standalone:

if __name__ == '__main__':
    main()

PyPI and Markdown

Will McKenzie came up with a nifty solution for using Markdown READMEs in PyPI packages. Here's the problem:

PyPI uses ReStructuredText to render READMEs
GitHub uses Markdown to render READMEs

Will's solution was to do the following:

Create a wrapper called register.py
Have register.py convert README.md (Markdown) to README.txt (ReStructuredText)
Have setup.py read the README.txt file as its long_description property on PyPI

I had problems with the pyandoc library, so I took Will's idea and rebuilt it slightly differently into setup_wrap.py:

import sys
import os
import subprocess
import pypandoc
with open(‘README.rst’, ‘w’) as dest:
long_description = pypandoc.convert(‘README.md’, ‘rst’)
dest.write(long_description)
args = [‘python3’, ‘setup.py’] + sys.argv[1:]
subprocess.call(args)
os.remove(‘README.rst’)

setup_wrap.py does the following:

Converts README.md to README.rst using pypandoc
Runs python3 setup.py and passes along its own arguments
Deletes README.rst after setup.py is done

Additionally, I added the following to setup.py:

with open('README.rst') as f:
    long_description = f.read()
setup(
…
long_description=long_description,
…
)

So now, instead of running python3 setup.py sdist, just run python3 setup_wrap.py sdist and you'll have a very pretty README on both GitHub and PyPI.

Week One with iOS

Sat, 14 Jun 2014 06:43:58 GMT

It's really not that bad. I think I might prefer it to Android soon.

Interface Builder and Storyboards are nicer than Android's XML layouts, but you have the learning curve of the message-passing component linking GUI. I was really annoyed with iOS for using IBOutlets and delegates to programatically link code and UI until I realized that Android does the exact same thing under a different name.
It's much easier to build a good-looking iOS app out of the box thanks to their great UI elements and thought put into design. Android apps look like trash out of the box and they're hard to make pretty, especially if you try them on more than one phone.
I didn't like how I had to subclass UIViewController for every single individual screen until I realized that I have to do literally the same thing in Java. Android does have Activities though, and those were really nice—it was clear how to pass parameters from one view to another. There's probably just as good of a way to do this in the view constructor.
I was really afraid of memory management but it turns out iOS has ARC which is pretty neat.
I still don't know what half the keywords in Obj-C mean. Here's an example: @property (nonatomic, retain) IBOutlet UIButton *myButton;—I know retain has something to do with not letting ARC discard it, but how does IBOutlet "decorate" it? What's nonatomic mean? What does an @ symbol specify, because I've been using it for like ten things already today.
Why are there three ways to register something into Interface Builder, and why does the most common one involve setting a return type that gets thrown away and left unused?
It took me 30 minutes today to realize that my Configs/app_ids.plist file was being statically linked into the root when I ran my app, keeping me from importing it from the Configs folder I specified. Aaagh. Why?
This week, Objective-C has felt like a mesh between the readability of Haskell, the parens of Lisp, and the elegant naming styles of Java.

I will probably keep developing for iOS and Android, but I think I like iOS a bit more now—it lets me work my chops as a designer a bit more than Android, and it's not that different after banging your head into the system for a week.

LED crowd visualizations suck, and here's how to make them better

Wed, 21 May 2014 04:16:03 GMT

This post is a WIP outline. Feel free to read it if you'd like.

People love attending fireworks events and laser light shows. It's awe-inspiring to see coordinated events happen on a scale so much larger than a single person.

It's even more incredible to become a part of the performance yourself. In 2007, Graffiti Research Lab launched a project called L.A.S.E.R Tag that let users spray large-scale digital graffiti on the sides of massive buildings using a laser and a high-power projector.

Giving people a chance to be a part of the show grants them a feeling of empowerment. It feels good to be part of something bigger, especially when you can personally see hundreds or thousands of other people invested in the same idea.

Crowd visualization is exciting.

In 2012, Coldplay gave out LED wristbands called Xylobands to people attending their Moto Xyloto tour concerts. This was the first time Xylobands were used in a large-scale performance.

The Xylobands were given out for free to concert attendees. They sport LEDs that light up in sync with the performance. [FIXME] It's really cool to see the entire crowd flash with the beat of the music.

In June 2012, Disney rolled out their Glow with the Show ears at select Disney parks.

While the Xylobands flash, blink, and glow in a single color determined by the plastic casing, the Disney ears are translucent white and have RGB LEDs built into the body, allowing them to glow in any color. In addition, the Disney ears also support more sophisticated patterns than the Xylobands.

Some of the Disney ears patterns shown in the video include:

blending from one color to another,
fading on and off smoothly, and
displaying independent colors and fades on each ear.

We can do better.

Crowds aren't blobs
They're people
Screens (2D pixel arrays) are cool
Displaying coherent patterns on free-form blobs of pixels creates an incredible effect

Tech Specs

Here's a quick overview of

Glow with the Show ears

Translucent plastic enclosure
IR blaster at front of stage sends signal to Vishay TSMP6000 IR receiver
Each device repeats IR from receiver on front to emitter on back
TI MSP430G2553 microcontroller contros RGB LEDs in ears
Runs on 3 AAA batteries
All devices flash the same patte (!!)rn during the show
$25 (!!) at park kiosks

Coldplay Xyloband

Opaque colored plastic enclosure mounted on nylon wristband strap
Each device has its own color
Atmel ATmega48PA (according to Hackaday) or Silicon Labs 8-bit µC (according to CarlsTechShed) is probably the microcontroller being used
Operator can select to pulse all bands or select colors independently from one another
SI4313 RF receiver (PDF datasheet)

Don't forget to check out the teardown of the Xyloband—thanks, CarlsTechShed!

Why do they suck?

Single setting for the entire crowd
No way to treat the crowd like a screen
Devices aren't interactive—users don't give back to the show

Device Requirements

Devices need to:

Be cool
Be light
Be wearable
Have an RGB LED
Receive data from a base station
Be individually addressable—this is the one thing these devices should do but don't
Be cheap enough that people will buy them at a gift shop or that the price can be factored into a show ticket ($10)

Devices could:

Have an accelerometer
Translate live accelerometer data into visualizations
Return accelerometer data to the base station
Bounce signals back to other devices
Be controlled by DMX via an adapter at the base station for use in shows with DMX setups

Analyzing 8 years of my music history with SQL

Thu, 10 Apr 2014 05:34:20 GMT

Ever heard an album or a song for the first time and loved it so much you played it on repeat for three days straight? I wanted to dig into my listening history to find out which songs I loved, got sick of, and forgot about for years.

Good thing I use Last.fm obsessively. Last.fm is a service that "scrobbles"—keeps track of—every song you listen to past the halfway mark. I've been using Last.fm since 2006 and have accumulated a seriously extensive scrobble log.

Last.fm provides users with an API for accessing their scrobble history. I figured that if I scraped my entire listening history into a SQL database, I could better analyze my listening history.

Scraping the Last.fm API with Python

I wrote some Python to claw my data out of the internals of Last.fm, one page at a time. It uses the Requests and Dataset packages. I used iPython Notebook to assemble a docs-and-code mashup of doom.

Check out my writeup on scraping Last.fm data here!

(If you have iPython Notebook, you can download the notebook file here and run it yourself.)

Whatchu know 'bout me?

Once my data was scraped into my DB, I ran some queries to search for songs I played obsessively after I discovered them, and I found several of my music-listening phases, such as:

that period during which I listened exclusively to "Weird Al" Yankovic,
the landmark date in 2006 when I listened to Daft Punk's Discovery for the first time, and
that time I saw Wicked with my high school band and listened to the soundtrack for a week straight.

This is awesome. I'm a stats nerd and I love the idea of a quantified self. Thank you, 14-year-old Matt, for signing up for Last.fm 8 years ago.

There's plenty more to find out. Like, this has been my favorite song so far this week and this is my favorite newly-discovered album. And this is probably my favorite new artist by far.

OK, this is getting kinda creepy

But don't take my word for it—I've put my entire scrobble log online for you. Want to know on which day I fell asleep listening to Ellie Goulding in 2012? Want to dig up that evidence that I'm actually a closet Ke$ha fan? Download my SQLite database and learn what you will about 8 years of Matt's musical taste.

As always, email me at matt@mplewis.com or tweet me at @mplewis if you do something cool with this! I always love hearing when someone takes something I helped build and makes it even more interesting.

And if you do find out I like Ke$ha, please keep it to yourself maybe?

Throwback Thursday: My First Sorting Algorithm

Thu, 03 Apr 2014 09:31:12 GMT

I just rediscovered some code I wrote for the University of Minnesota's "Structure of Computer Programming I", a course I took three years ago. CSCI 1901 focused mostly on learning Scheme with everyone's favorite Scheme textbook, the venerable SICP from MIT.

To get better at writing Scheme, I practiced by writing two versions of the bubble sort algorithm. One sorted a list of integers into increasing order, while another sorted a list of Cartesian tuples (x, y coordinates) into increasing distance from the origin. I think these sorts are even stable!

Building a bubblesort algorithm from scratch is one of the most rewarding things I've ever done as a programmer. To this day I'm still proud of these programs.

I even commented (one of) them. Enjoy:

(Released under the ISC License. Go crazy.)

So You Want To Make Your Images Responsive

Wed, 11 Dec 2013 00:19:29 GMT

Your website looks like it's from '06. Your images look blurry when those kids pull up your site on their fancy new Retina MacBooks and iPad Airs. What do?

Wait for srcset to come out
Implement srcset-polyfill with code that isn't production-ready
Send your users images at 2X resolution, even if they're on mobile
Use BBC News' imager.js

You should be using imager.js. Here's why:

It's production-ready—the BBC uses this on their sites
It's documented very, very well.
It's smart—it lazy loads specific images based on the viewport properties, saving your mobile users time and bandwidth
Your users will love you for having a fast-loading site (with pretty Retina images, if they're into that).

Go check out some examples and get started!

Appendix: OK, I'm using imager.js. Now what?

Start generating your images automatically from source. Ain't nobody got time for doing that by hand.

grunt-responsive-images will do the heavy lifting for you. Yay for automation!

How To Make An EE Cry With One Animated GIF

Mon, 11 Nov 2013 03:02:44 GMT

Auto route, yo. (Check out the HTML5 video here.)

Deploying an Octopress blog from a VPS to NearlyFreeSpeech.net

Fri, 08 Nov 2013 19:10:36 GMT

Author's note: This was migrated from an Octopress blog. I think the data is still relevant to anyone trying to run Octopress and deploy to NearlyFreeSpeech, so I've kept the post online in its old form. Enjoy!

Hello! I'm Matthew Lewis, and this is my Octopress blog.

I'd like to use this blog for projects, and I suppose the first project I should discuss is this blog itself.

Octopress is a blogging framework. It's based on jekyll, a static site generator.

Frankly, Octopress is amazing. It's easy to install and modify themes and layouts, and I can run a single rake command to deploy my blog from my VPS to my NearlyFreeSpeech site.

In this post, I'll detail how to generate and deploy your Octopress blog straight to your NearlyFreeSpeech.net hosting account in one command.

Deployment Stack

I'm running on a Debian 32-bit VPS hosted on BigScoots and hosting www.mplewis.com on NearlyFreeSpeech.net.

Why use a VPS and a web host?

I want to host my blog on NFS.net because:

NFS can handle much more traffic than my dev VPS (my VPS is uber cheap)
NFS is seriously cheap for static sites, which makes using Octopress on NFS very attractive

I have to use a VPS to deploy because:

NFS doesn't have the proper Ruby or bundles or anything I need to generate my Octopress site using Jekyll

So, how's the deploy work?

The process behind deploying an Octopress blog is as follows:

Run rake generate to generate the static files behind the blog posts
Run rake deploy to rsync the static files to a web server

That's it. The config is documented in the Octopress docs.

And if you're lazy like me, use the combined rake command:

Run rake gen_deploy to generate the blog and deploy it to your server. Hooray!

And that's it? That easy?

Yes... and no.

rake deploy wants a public key to connect to your web server.

NearlyFreeSpeech.net will let you use a public key to authenticate. However, you have to file a free assistance ticket and wait a couple of hours for the key to be set up on your account.

More information can be found here on NFS.net's FAQ, but it's a very simple process:

Click the link titled "assistance request" to submit a free assistance request.
Copy your id_rsa.pub and paste it in the text field, and ask them to add that pub key to your account.
Wait two hours.

And you're done! Your dev server will now be able to deploy straight to any of your NFS sites without a password.

I hope this information helps anyone looking for info on how to deploy to NearlyFreeSpeech.net via an external server.

I'll be watching you: Stalk me in real time via the web

Fri, 08 Nov 2013 19:07:21 GMT

Author's note: Google took down Google Latitude and Locality is no longer online. But my code is still around, and I'll leave this up in case anyone finds anything of use!

I've always been fascinated by the idea of real-time location data. The first Google Glass video showed a person checking where his friend was with a real-time location service. I always liked the idea of the Weasley family clock from Harry Potter, which the mother of the large Weasley family used to keep tabs on her kids and husband. So I tried to build a Weasley clock of my own with HTML, CSS, and a little JavaScript magic.

I named my web application Locality. Check it out right now, and read on to find out how it works.

locality

Noun

The position or site of something.

An area or neighborhood, esp. as regarded as a place occupied by certain people or as the scene of particular activities.

Step one is figuring out where I am at any given time. The easiest way to get my own real-time location data is with an Android smartphone that I already have. It's got GPS and 4G, so it'll try to sync my location to Google Latitude whenever I move around.

Getting the data out of Google Latitude is slightly tricky. Latitude offers "site badges", HTML drop-ins that put a little map and an icon on the map to show people where you are. But I want the coordinates, and Latitude offers a JSON feed to get that data out too. The URL for any given user's JSON feed is https://latitude.google.com/latitude/apps/badge/api?user=YOUR_USER_ID_HERE&type=json.

If you've ever worked with JSON before, you know that making a JSON request from one domain to another is likely to fail because of HTTP access control (CORS). So I used Yahoo's YQL to import a JSON feed, read the data, and spit it back out as JSONP -- which isn't governed by CORS.

// URL for Latitude data in JSON form
var latitudeUrl = 'https://latitude.google.com/latitude/apps/badge/api?user=MY_USER_ID&type=json';
// use Yahoo YQL to proxy the JSON into JSONP
var yahooYqlUrl = 'http://query.yahooapis.com/v1/public/yql?callback=?'
$.getJSON(yahooYqlUrl, {
q: ‘select * from json where url=”’ + latitudeUrl + ’”’,
format: ‘json’
}, function(response) {
var data = response.query.results.json;
// do something with data here
});

It's a bit of a hacky solution but it works very well. Credit where credit is due to Stack Overflow user hippietrail. Apparently there's also a cool little service called jsonp.ru that does the same thing, but I trust Yahoo more with my data.

Once I've got the Matt coordinates, I need to figure out where Matt is. I started by constructing a list of location objects with data on each location of which I wanted to keep track. I grabbed the coordinates from Google Maps by using the "right-click to drop coordinates" Labs addon and manually typing the coordinates for each location into my code.

var locations = [
    {
        name: "Stadium View",
        coordinates: {
            latitude: 44.971599,
            longitude: -93.221807
        }
    },
    {
        name: "Kenneth H. Keller Hall",
        coordinates: {
            latitude: 44.974650,
            longitude: -93.232460
        }
    },
    // ... put more locations here
]

Okay, so now I have the coordinates of locations and the coordinates for myself. I need to be able to tell how far away I am from each location. It turns out this is a reasonably well-solved problem -- I found some fantastic scripts from Chris Veness at movable-type.co.uk that do just what I want: simple, accurate calculation of distances between two latitude/longitude pairs.

I wrote a little function that uses LatLon.js to calculate the distance in miles between two pairs of coordinates:

function distMiLatLon(lat1, lon1, lat2, lon2) {
    var rMi = 3963.1676; // earth's radius in miles
    var p1 = new LatLon(lat1, lon1, rMi);
    var p2 = new LatLon(lat2, lon2, rMi);
    return p1.distanceTo(p2); // in miles
}

For my application, I decided if I was within 500 feet of a known location, I should be checked in. 500 feet == .094697 miles, so I set var maxDistToValidateLoc = .094697 in my code. After Locality gets the coordinates for locations and for me, it iterates down the list of locations and calculates a distance to each one. If a distance is less than .094697 miles, it registers me as being present at that location. Otherwise, it puts me in the "no man's land" category of "somewhere else".

That's the JavaScript and data handling side covered. I'll write more about the HTML/CSS layout in another post. For now, check out the app at mplewis.com/locality and tell me what you think!

Downloading Your Google Location History With The Google API Python Client Library

Fri, 08 Nov 2013 19:06:34 GMT

tl;dr If you want to get started with your Latitude data right away, check out my gist at the bottom.

I've started a new project! I'm investigating ways to share my life with others electronically, with little to no conscious effort on my part.

One of the things I'd like to share is my location history. I'm not sure how much or how little information I want to give to a total stranger, but I believe that if I want to share a significant amount of my life with everyone on the Internet, context is key -- and location makes a huge deal when it comes to taking events in my life into context. Right now I live in San Francisco, but two months ago I lived in Minneapolis, and somewhere in between I spent time with my family in Wausau. Knowing this is key to understanding what I want to share with the world.

I'm being intentionally vague about the actual project I'm working on, but that's okay -- this post is about one of the key elements, and I'll post more later when I start assembling other parts.

Read on to find out how to get your super-interesting real-time location data out of everyone's favorite location-tracking Google app.

How to Geolog 101

What's the best, most convenient, cheapest way to tell people where I've been in the past? There are plenty of ways to log my location, and most of them involve the use of GPS to lock onto my location, and some kind of data logging device to put the data into a format I can use.

I could build a device from scratch involving a GPS module, a microcontroller and a data logger, or I could just use my Android phone!

Android, as we all know, is developed by Google, and Google puts out tons of other cool web products. One of those products is Google Latitude.

From Google's website:

{% blockquote %}
Google Latitude is a location-aware mobile app developed by Google as a successor to its earlier SMS-based service Dodgeball. Latitude allows a mobile phone user to allow certain people to view their current location. Via their own Google Account, the user's cell phone location is mapped on Google Maps.
{% endblockquote %}

Perfect! Plus -- if I use my cell phone, it uses Skyhook to approximate my location even better with the use of wifi hotspots and cell phone towers.

So, Google Latitude will log my location over time in Location History. All I need to do is carry my cell phone on me and keep it turned on. Easy.

My next goal, then, is to get my data out of Location History and into a form I can use.

Prerequisites (Whatchu Need)

To be able to follow along and extract your own Location History data, you'll need the following:

Location History data in your Google account
A Python 2.7+ installation
The Google API Python Client -- I used pip install google-api-python-client to install this.
A Google account.
A Google API project with access to the Latitude API. See Creating a project for information on creating a new API project.

After you create a project, you'll want to go to the API Access tab and create an OAuth 2.0 Client ID. Make sure you set the application type to Installed application and the installed application type to Other.

Accessing Your Data Using A Poorly-Documented (But Well-Written) API Library

Authentication

The Google Location History API runs on OAuth 2. It's finnicky because of that. OAuth 2 is finnicky. Avoid writing an OAuth 2 client from scratch if you can -- you'll regret it. Use the tools you're given by the developers who know what they're doing.

It took me about four hours to figure out the basics of the API because the documentation is a bit lacking. You should still check out the docs though -- here's Google's only Latitude API sample, and here's some samples for other applications that might help you out.

Hopefully I can save you some time by explaining how the script I wrote to authenticate with the Google OAuth endpoint via the Python API works and why it works that way.

Here's the code you'll need to authenticate with Google:

import httplib2
from apiclient.discovery import build
from oauth2client.file import Storage
from oauth2client.client import AccessTokenRefreshError
from oauth2client.client import OAuth2WebServerFlow
from oauth2client.tools import run
CLIENT_ID = ‘YOUR_CLIENT_ID’
CLIENT_SECRET = ‘YOUR_CLIENT_SECRET’
REQUESTED_SCOPE = ‘https://www.googleapis.com/auth/latitude.all.best’
flow = OAuth2WebServerFlow(CLIENT_ID, CLIENT_SECRET, REQUESTED_SCOPE)
storage = Storage(‘credentials.dat’)
credentials = storage.get()
if credentials == None or credentials.invalid:
credentials = run(flow, storage)
http = httplib2.Http()
http = credentials.authorize(http)

In the above example, you need to replace CLIENT_ID and CLIENT_SECRET with real values. Fill those in with the values found in the application you created above in the Google API Console.

When you run this code, Python will open your web browser to an authorization page. If you're not logged into a Google account, you'll be asked to log in. You'll then be asked to allow access to your location history.

After you grant access to your application to access your location history, you can close your browser window. The Python script will continue by saving your login credentials in credentials.dat as shown in the script above -- if you want to change that, just provide a different path to Storage(). Once you have credentials.dat saved, you won't have to log in again until your OAuth token expires.

Great! Now you have authorization to access your Location History data. How do you actually access it?

Accessing Your Data

This uses some of the code Google provides in their API acess examples. This code follows immediately after the authentication above and assumes you have a valid http instance available from credentials.authorize().

latitude = build('latitude', 'v1', http=http)

try: # access Location History data via the “latitude” object; put your code here! except AccessTokenRefreshError: print ‘Access token refresh error. Please restart the application to reauthorize.’

The latitude object indicates you want to access the Latitude API, version 1, using the credentials.authorize() http object to make the requests.

Inside that try section you'll need to use the latitude object to access your data. Here's the syntax you need:

response = latitude.location().list(granularity='best').execute()

By default, Google assumes that you want the latest results. Specifying the keyword argument granularity='best' indicates you want the best available location granularity as well. You're allowed to do this because earlier, when you asked the user for permission, you told Google you wanted permission to use the best-available location data (REQUESTED_SCOPE = 'https://www.googleapis.com/auth/latitude.all.best').

When you request this data, you'll get the 100 latest data points. You can access them by looping through the object response['items']. If the response object has no key items, then the Google API returned no results.

Since Latitude likes to take one point a minute, you almost certainly won't get all your data with one query. So, how do you cycle through this data?

Iterating Through Your Data

You can specify other arguments to the latitude object inside the location().list() function as well. One option you can specify is max_time, which indicates that you want no objects with the milliseconds timestamp later than the one you provide in the max_time argument.

Since the latitude.location().list().execute() function returns the 100 latest location items that match your query, you can cycle through the list by doing the following:

Run a latitude.location().list(granularity='best').execute() and save that data.
Take the least-recent data point in that set of data, look for the key timestampMs, and subtract 1 millisecond.
Take that new lower timestamp and use it in the query as follows: latitude.location().list(granularity='best', max_time=LOWER_TIMESTAMP_HERE).execute().

Since you asked the API to return no results with a timestamp greater than the one you provided, you'll get a new page of the next-100-most-recent Latitude results. Yay!

If you keep doing that, someday you'll get a response back that doesn't have an items key. This means you don't have any results left. You could either check for an items key with if items in response, or you could just try to read response[items] and catch the Exception with except KeyError:. The second one is "more Pythonic" if you want to get super pedantic and PEP-8ic.

Working Sample

Here's a working sample. I'm using this with my application's OAuth2 ID and secret and it saves all my data as a big pile of JSON files. Hopefully this gets you up and running!

{% gist 5993157 google_latitude_api_access.py %}

Alternatives to Latitude for Geologging

If you don't want to use Google Latitude to log your GPS location constantly (which is probably a good idea, because the API is shutting down in less than a month), then you can log the data on your phone with an app instead. Ultra GPS Logger is the best one I've found in the Play Store, and at $5ish it's much cheaper than buying a dedicated hardware device.

Summary

In this post, I detailed how to use the Google API library for Python with the Google Latitude API to get your personal location history data out of Google Latitude and onto your computer in a versatile data format.

In my next post, I'll be writing about the purpose for which I'm using this location history data, and why it's so important to have accurate location data!

Installing Kaleidoscope Integration (KSdiff build 89) With Kaleidoscope 2.0.0

Fri, 08 Nov 2013 19:05:57 GMT

The other day I was using the fantastic file diff application Kaleidoscope, and I wanted to integrate the diff functionality with my SVN client. Kaleidoscope provides a terminal tool called KSdiff that integrates the GUI Kaleidoscope application with other programs.

However, after installing KSdiff from the Kaleidoscope website as directed, KSdiff build 111 told me that my version of Kaleidoscope was too low and that I needed to upgrade my application to use Kaleidoscope's integration features. In other words, the version of KSdiff I had installed was too high for my current installation of the application.

I didn't want to upgrade Kaleidoscope right away, so I Googled around for an older version of KSdiff that would work with my out-of-date Kaleidoscope.

I found ksdiff-89.zip on the Kaleidoscope CDN. That file did what I was looking for!

If you're having the same problem I am and want to integrate Kaleidoscope without upgrading from version 2.0.0, download and install ksdiff-89.zip and you'll have integration with Kaleidoscope in your favorite version control clients.