The HTTPS-only experience

EFF recently announced that We’re Halfway to Encrypting the Entire Web.” As a celebration of this achievement I’ve started an experiment: as of yesterday, no unencrypted HTTP traffic reaches this machine*.


Even though securing your web site is easier than ever it will take some time before everybody encrypts. But there are a bunch of sites which support both secure and insecure HTTP transfer, and there are a few tricks to tilt the scales and make the current experience better:

  • The HTTPS Everywhere browser extension ensures that you use HTTPS whenever possible on thousands of sites.
  • Editing the URL to add “https://” at the start works for some sites. If you get the dreaded certificate error page make sure to report them, and only add an exception if … well, that subject is too big to get into here. Just don’t.

Many sites have excellent HTTPS support, and have enabled HTTP Strict Transport Security (HSTS). In short, if your browser knows that the domain supports HTTPS (by visiting it or the browser being installed with it) you can simply type the domain name in the URL field and the browser will default to fetching the secure version of the site. On the other end of the spectrum I can still visit sites which have no HTTPS support at all if I really need to by using Tor, which provides privacy but not integrity or authenticity.

pacman stopped working after setting this up. It turns out the package database is fetched using unencrypted HTTP by default, but it was easy to generate a new list of only HTTPS mirrors.

Some sites have a strange HTTPS setup. The BBC only support HTTPS for the front page, which is just weird. Why go to all that trouble for 1% of the gain? Other sites require authentication to access using HTTPS, possibly not realising that setting up HTTPS for everyone would be easier.

My home router runs DD-WRT, and the web interface for it is only accessible by HTTP by default. This is easy to configure though.

OCSP uses HTTP (at least in Firefox), since the returned file signature has to be checked separately anyway. So if I go to about:config, change security.OCSP.require to true, and visit a site I haven’t seen for a while, I get an error message like this:

An error occurred during a connection to The OCSP server experienced an internal error. Error code: SEC_ERROR_OCSP_SERVER_ERROR

The solution is to either allow OCSP queries specifically or to allow HTTP to specific hosts. Let’s see what can be done…

The Steam client uses insecure HTTP for both game updates and the store pages. There doesn’t seem to be any way to force it to use HTTPS, so I have submitted suggestion to Valve using the official channel.

These have been the only major hassles so far. The only other sites I really can’t get to work over HTTPS are various hold-outs like Wikia and BBC.


The change was a simple addition to my Puppet manifest:

firewall { '100 drop insecure outgoing HTTP traffic':
  chain  => 'OUTPUT',
  dport  => 80,
  proto  => tcp,
  action => reject,

The resulting rule:

$ sudo iptables --list-rules OUTPUT | grep ^-A
-A OUTPUT -p tcp -m multiport --dports 80 -m comment --comment "100 drop insecure outgoing HTTP traffic" -j REJECT --reject-with icmp-port-unreachable

* Technical readers will of course notice that the configuration simply blocks port 80, while HTTP can of course be served on any port. The configuration wasn’t meant as a safeguard against absolutely every way unencrypted HTTP content could be fetched, but rather focused on the >99.9% of the web which serves unencrypted content (if any) on port 80. I would be interested in any easy solutions for blocking unencrypted HTTP across the board.

When MFA is not enough

I hope you’ll excuse the format of this post. Coffee does strange things to my brain.

In the beginning, there was the username. And it was deemed good enough, because everybody on the machine was a trusted colleague. Then came machines shared by strangers, and with them the password. And it was deemed good enough, because the resources to crack it were beyond most. Then came the Internet, and the world flourished and grew. But the world contains more villains than the password, in its myriad implementations, could withstand. And thus came multi-factor authentication. And it was deemed good enough, because only someone in possession of the authenticated device could gain access.

But they were all deceived, because the service providers foretold that their users would lose their authenticated device, and lock themselves out of their accounts, and would blame them rather than their own misfortune. And so it was that, fearing their users’ wroth, the service providers gave their support personnel unfettered access to override MFA at the behest of a phone call.

Some service providers were wiser, and gave out backup codes after authenticating, thus allowing users to regain access by themselves, and promised not to give access to anybody with a honeyed voice, and reminded their users ever and anon that they should be mindful of their backup codes, and even gave them the option to be told when their account was logged into. And their users were at peace, trusting that their service providers knew what the fuck they were doing, that their files were safe, and being forever loyal to them.

Full disk encryption with Arch Linux footnotes

Pavel Kogan has an excellent guide to install Arch Linux with full disk encryption. I’ve taken the liberty of copying the instructions, adding a couple tweaks:

  1. Boot the Arch Linux installation medium.
  2. Run these commands (You may want to use different sizes for swap and root volumes):
    parted -s /dev/sda mklabel msdos
    parted -s /dev/sda mkpart primary 2048s 100%
    parted -s /dev/sda set 1 boot on
    cryptsetup luksFormat /dev/sda1
    cryptsetup luksOpen /dev/sda1 lvm
    pvcreate /dev/mapper/lvm
    vgcreate vg /dev/mapper/lvm
    lvcreate -L 4G vg -n swap
    lvcreate -L 15G vg -n root
    lvcreate -l +100%FREE vg -n home
    mkswap -L swap /dev/mapper/vg-swap
    mkfs.ext4 /dev/mapper/vg-root
    mkfs.ext4 /dev/mapper/vg-home
    mount /dev/mapper/vg-root /mnt
    mkdir /mnt/home
    mount /dev/mapper/vg-home /mnt/home
  3. Go through the software installation steps of the installation guide, skipping the Initramfs and Boot loader steps.
  4. Install GRUB: pacman --sync --noconfirm grub
  5. In /etc/mkinitcpio.conf:
    • Change the line starting with FILES= to FILES="/crypto_keyfile.bin"
    • On the line starting with HOOKS= add lvm2 encrypt just before filesystems.
  6. Find the UUID of /dev/sda1 by running basename "$(find -L /dev/disk/by-uuid -samefile /dev/sda1)"
  7. In /etc/default/grub:
    • Change the line starting with GRUB_CMDLINE_LINUX= to GRUB_CMDLINE_LINUX="cryptdevice=UUID=[UUID]:lvm", replacing [UUID] with your own.
    • Add a line with GRUB_ENABLE_CRYPTODISK=y
  8. Run these commands:
    dd bs=512 count=4 if=/dev/urandom of=/crypto_keyfile.bin
    cryptsetup luksAddKey /dev/sda1 /crypto_keyfile.bin
    chmod 000 /crypto_keyfile.bin
    chmod -R 700 /boot
    mkinitcpio -p linux
    grub-mkconfig -o /boot/grub/grub.cfg
    grub-install --target=i386-pc /dev/sda
  9. If necessary, set up your BIOS to allow booting in CSM mode.

It also required me to enter the password using a QWERTY keymap. The instructions to add an alternative keymap to GRUB are rather involved, but I’ll try to write them up if I go through with it.

Shell scripting dos and don’ts

Shell scripting is like a room full of power tools: handy but dangerous.


  1. Build complex systems. There are just too many ways that external state can affect any piece of shell code. Do you know what your script will do if you change IFS before running it?  What if you give it a file name starting with a dash or containing a newline? How do you recover the state of the system if the script crashed somewhere in the middle? Complex shell script environments invariably end up looking like Rube Goldberg machines of chainsaws and power drills. Use languages and frameworks appropriate for the task.
  2. Expose them to the Internet. Safe input handling is just too damn hard. Unless you’re GreyCat or Stéphane Chazelas.
  3. Use eval. Don’t be evil. There are safer ways to do whatever you’re trying to do.
  4. Write portable code. (By this I mean code which works in multiple shells without change, as opposed to code which can easily be ported to other shells.) Writing portable code means limiting what language features you use and adding complexity to make sure it works the same way in all the supported shells. Because of this, the end result will be more complex and less flexible than the simplest script that supports the shell you have.
  5. Minimise the number of characters. The next maintainer will hate you.
  6. Create interactive menus. Very few shell tools like less and top only make sense interactively. Use command-line arguments instead, so that your tool will be useable both standalone and with other tools.


  1. Test everything automatically. This gives you and others the confidence that your script actually works. Bonus: Allows you to modify your code without having to test everything manually. Extra bonus: Experiencing how difficult it is to test shell scripts exhaustively will convince you to never use them for anything complex.
  2. Provide --long-names for every -s -h -o -r -t option. And if you can bear the screams of dogmatic developers, don’t support short options at all. As long as the names make sense this allows people to write readable scripts. Bonus: No wondering whether -n5f0 is two, three or four options.
  3. Use guard statements like the POSIX set -o errexit -o noclobber -o nounset. While there are some caveats to how these work, they can save a whole lot of headache. Bonus: Use -o xtrace to see what the script does in detail.
  4. Add an auto-complete script. The users will be grateful. Bonus: Gives an incentive to keep the structure of your options sane.

Save your bookmarks on Pinboard!

Why? Check this out:

  • Simplicity
    • List interface:2016-02-24-145849_1366x768_scrot
    • Bookmarking interface:2016-02-24-140756_1366x768_scrot
    • No ads
  • Price
  • Safety
    • Rock solid since I started using it in 2012
    • Backups in multiple portable formats
  • Privacy & security
  • Access via basically any means: browser extensions, JavaScript widgets, apps and API
  • Extensive FAQ
  • Likely to survive for long because of a simple, sustainable business model

Other than being a happy customer I am in no way affiliated with Pinboard.

Why I still contribute to Stack Overflow

This is in response to Michael T. Richter’s excellent critique of Stack Overflow. While I share some of the concerns for the problems mentioned there, I don’t believe they are quite as detrimental to the quality of the site as he appears to.

From the article:

I’m not a Java programmer. I’ve only ever briefly programmed in Java professionally. I hated the experience and I hate the language. I certainly don’t consider myself a Java expert. Yet I managed to get the bulk of my points from Java. How is this possible?

It’s possible because I did what many of the people whose questions I answered (and got points for) should have done for themselves: I saw a simple Java question, hit Google, read briefly, then synthesized an original answer.

This is what I read:

I know Java well enough to have used it professionally. Considering the recent explosion of access to computers and the Internet, the vast numbers of students and hobbyists who only ever touch on Java very briefly to solve a highly specific problem, the low level of entry relative to most other languages, the tiny amount of people who end up specialising in Java rather than any other languages in existence, and the fact that the language hasn’t been around long enough for there to be large amounts of people with many decades of experience, that single point probably makes me more knowledgeable than 90% of Stack Overflow users in Java.

I freely admit that I do pay attention to the points. They are the best way I have to figure out if I am actually getting better at submitting useful questions and answers. People’s motivation for answering doesn’t matter one jot to me if they write interesting or useful on-topic content. How could it? Such content is completely off topic, and would be edited out like the plague.

There’s room for improvement (and I wouldn’t even say “obviously”, like for so many other sites), but SO/SE is still miles ahead of anything we’ve ever had. It’s the Google of Q&A sites: Terrible for super specific issues (which are likely to be closed since they are not interesting to anyone but the author), but awesome for the rest.

Much of the rest of the article seems to be simply a complete lack of faith in other human beings. For example:

  • “If you’re going for points (and that’s the entire raison d’être for gamification!), are you going to waste time like that for 60 points when you could fit in a dozen 460-point answers? Of course not! You’re going to go wherever the points are, And the points are the low-hanging fruit of trivial questions from popular languages.” I personally think the Stack Overflow answer rate is plenty proof that lots of that people are willing to help even when the answer is likely to get very few votes.
  • “There’s an old cliché in English: give a man a fish, he eats for a day; teach a man to fish, he eats for a lifetime. StackOverflow is filled to the brim with people giving fishes.” After using other web sites which were much more geared to fish donations (almost every linear forum, IRC channel and mailing list I ever used) I find it refreshing that adding links to more resources (even on other people’s posts), where people can learn how to fish, is extremely easy on Stack Overflow. I use this feature extensively, having far too many times googled a problem only to find a tip in a random forum reply with absolutely no rationale or link to more information. This is something which could possibly be pushed even more by the SO system, but it’s already very good.

And finally, the end paragraph: “How about learning?” Great! “You know, that thing that puts information in your head that you can apply later at need.”“Use Google.” About 30 times a day, and usually the best result is on Stack Overflow or Stack Exchange. “Use Wikipedia (if you must).” Except it’s pretty terrible to learn about practical software development or actual development issues. “Use RosettaCode for code examples.” Impressive collection, but I don’t think it’s very relevant for real-world development or easily searchable. And the MediaWiki editor is a pain compared to SO. “Engage with other users of the tools you use in the form of user groups, mailing lists, web forums, etc.” Vastly inferior technology to solve the exact same problem. No thanks. “Learn foundational principles instead of answers to immediate questions.” Stack Overflow is a Q&A site, not a substitute for work experience or college degrees.

UK technology fail

Since moving to the UK after living in Norway, France and Switzerland, it seems that this country, more than any other I know of in the Western world, is stuck in the 20th century. Here are some examples encountered in the last few months:

  • Many banks support no other authentication than usernames and passwords. No one-time pads, SMS codes, smart cards or client certificates, at least one of which was supported at all of my other banks. My bank forces you to enter your password at the counter, in front of other customers.
  • Don’t talk to me about Internet banking. Compared to every other bank I’ve used, my UK bank has by far the worst usability. If something fails, it doesn’t give any hint to what happened, and just sends you to the front page. Retrying means finding the same form again and filling in everything. It doesn’t play nicely with any online shops. It takes several days to add every single payment recipient. And its “company search” doesn’t find big UK companies.
  • Usually three or four buses arrive within two minutes, and then none for 20 minutes. Sometimes these clumps even have more than one of the same bus line.
  • Automated check-outs at supermarkets are sometimes OK: They take cards and cash, weigh your items properly, and calculate the sum properly when there’s a discount. However, none of them do all of these, and some of them seem to do none of them.
  • The phone only has a signal when the stars align just right. At home 1-2 bars, at work 0-1 bars.
  • The roads are too narrow for buses to pass, but there’s still parking on both sides.
  • I just ordered something online (via bank transfer since the card doesn’t work for some reason). The confirmation page showed the bank details. The confirmation email showed the final sum. Neither showed both.

Sometimes it’s so surprisingly backward that it’s amusing. Mostly I just wonder whether people care about service at all.