mlindgren.ca

Puzzle Panel Postmortem: Introduction

2010-05-30T08:26:00Z

Welcome, dear reader!

Although this is the inaugural post for this website, I won't bother with any introduction here. If that sort of thing interests you, I refer you to the "about" page. For this entry, I'd like to dive right into the sort of content that this site is to be host to. My iPhone developer program membership expires today, and I won't be renewing it—not immediately, at least. That being the case, I think that this is a good time to reflect on my year as a so-called "iPhone developer."

There's lots to be said about the iPhone, the app store, and Apple in general; I'm sure I'll write on those topics profusely. For the time being, though, I'm going to limit myself to a discussion of the product of my labour: Puzzle Panel. I've twice given a presentation on the development of Puzzle Panel as part of the University of Alberta Computers & Games course, but while I was grateful for the opportunity to discuss my game, the format of the presentation was such that I was never quite able to express everything that I wanted to. (I doubt that most of those in attendance would have been interested, anyway.) It is my hope that this site will give me a platform to communicate with greater fidelity.

If you're not familiar with Puzzle Panel, there's a trailer available here, or you can check out the project page or website for more information. Unfortunately, the expiry of my developer program membership means that Puzzle Panel will no longer be available on the iPhone app store for the time being. I'd like to remedy this; I want the game to be available indefinitely and to as many people as possible. However, for reasons I'll likely discuss at a later date, I'm hesitant to pay the $100 membership fee again, so at this point I'm... investigating my options, let's say.

Anyway, returning to the topic at hand, I purchased my iPhone with the specific intent to develop an app. The iPhone is often cited as a great gaming device, and I think that would be true if not for the lack of physical input options: the phone's hardware is powerful and its operating system is pleasantly efficient, but the touch screen just doesn't work for most games. For puzzle games, though, it works pretty well; the tap-and-swipe paradigm of the iPhone lends itself naturally to puzzle-style gameplay. (The good ol' D-pad is still faster and more accurate, though.) Hence, Puzzle Panel.

The game was developed over five months. I don't have a good sense of how many hours it actually took, but if I'd estimate somewhere between 200 and 400. I worked on it in the evenings after work and on my days off - sometimes obsessively, devoting all of my free time to it, sometimes barely touching it for weeks at a time. Not including the frameworks and libraries I used, the game consists of roughly 8,500 lines of Objective C code across 40 or so classes. The most time-consuming part of the project, though, was the artwork. I'm anything but a practised artist, so creating the game backgrounds and other art assets took dozens and dozens of hours. While I was impressed at what my Wacom tablet allowed me to accomplish, I was never entirely satisfied with the results. Still, I think doing the artwork myself was a wise decision.

That's the basic high-level project overview. In the next post, I'll discuss the Cocos2D-iPhone framework and why I chose to use it, and then I'll go through a "postmortem" in subsequent posts. I'll endeavour to include more technical information—code snippets and the like—in future posts.

My iPhone debacle

2010-06-16T06:07:00Z

...or "Another Reason I'm Considering Switching to Android."

Yesterday evening, as I was leaving work, I was carrying on a conversation via text message with a friend of mine. Upon arriving home, I realized that she'd apparently stopped responding, so I pulled my phone out of my pocket to check if I'd missed a message. Instead of my usual lock screen, though, I was met a very unwelcome "Connect to iTunes" prompt. I'd been running the beta of OS 4, and it turns out that if your developer licence expires while you still have the beta installed, Apple will remotely lock you out of your device without so much as a warning. I guess I should have expected as much.

At this point I was mildly annoyed. Because it's such a slow an arduous process to sync an iPhone, I hadn't backed mine up since... probably about two weeks ago, when my developer license actually expired. "Go figure," I thought; "if only they'd locked me out of my device when my licence expired, I might not have lost as much data." Still, I don't keep much important information on my phone, so a two-week setback, while irksome, was not disastrous.

...Or at least it wouldn't have been if not for what happened next. In their infinite wisdom, Apple decided not to provide an officially supported way to downgrade from a beta operating system release to a stable release, despite their aforementioned policy of locking "expired" installations. Luckily, there exists a workaround, and if all goes well it's relatively uncomplicated. Unfortunately for me, all did not go well. The workaround relies on the libusb library in order to communicate with the iPhone. Its Windows port, libusb-win32, is ostensibly compatible with both 32 and 64-bit Windows Vista and Windows 7. The only problem is that there's a known issue which causes all of your USB drivers to stop working if the library isn't installed in Windows XP compatbility mode under those operating systems, and it's not mentioned anywhere in the installer or in any of the downgrade tutorials I saw.

So, yes... in an attempt to downgrade my iPhone, I managed to completely disable my computer's USB drivers. libusb breaks one's computer in particularly pernicious way, too. I'm anything but an expert on precisely how Windows handles drivers, so forgive me if the terminology here is incorrect, but I believe this is how it happens: since a number of USB devices are inevitably in use when it's installed, libusb's changes don't affect those drivers until the system is restarted. I didn't have my iPhone plugged in when I installed libusb, though, so when I did plug it in afterwards it failed to connect. My mouse and keyboard were still working at this point. I figured that perhaps I needed to reboot my computer. Upon rebooting, of course, I found that none of my USB devices worked at all. I couldn't so much as log in, let alone uninstall libusb.

I don't own any PS/2 devices, so switching to a different mouse and keyboard wasn't an option. I spent probably about an hour trying to find a way to restore my USB drivers from the Windows recovery console, but came up empty handed. Deleting the libusb sys and dll files did nothing, despite forum posts to the contrary. I couldn't find any registry keys which obviously pertained to the library, nor could I find any system files which appeared to have been modified recently (excluding those I'd already deleted, of course.) Ultimately I gave up, and since my operating system is installed on an SSD drive which is otherwise empty, I decided that reformatting would be the simplest solution. I had to reformat my computer because Apple locked my phone.

I did eventually manage to restore my iPhone, using my development Mac Mini. I probably should have done so in the first place, but since my iPhone is synced to my Windows PC (which has much larger hard drives and is therefore better suited to storing my music collection), I figured that it would be more convenient to do the restore process using the PC. Oh, how wrong I was! That said, I don't fault Windows for this; I'm sure that installing a bad device driver on a Mac would have much the same effect.

This is what I see as a typical case of Apple's "security" policies interfering with the usability of their products. Granted, I wasn't nearly as prudent as I should have been, so I can't blame all of my troubles on them. But really, if they're going to put out operating systems which simply "expire" under certain circumstances, could they not at least provide an officially supported method to downgrade to an older version?

Puzzle Panel Postmortem: Framework

2010-07-13T07:33:00Z

I'm currently reading Coders at Work by Peter Seibel. The book is a collection of interviews with respected and knowledgeable programmers. While the interview format can be somewhat difficult to read at times, the book is excellent. Seibel is himself an experienced programmer, and asks the sort of insightful questions no ordinary interviewer could, making for fascinating discussions. One could mine the book for weeks for interesting quotes. I'm going to try to resist that temptation, but because it relates to the topic at hand, I will permit myself to share with you one statement made by Joe Armstrong:

Being a young programmer today must be awful—you can choose 20 different programming languages, dozens of framework[s] and operating systems and you're paralysed by choice.

I think Armstrong is mistaken; it's anything but awful to be a young programmer today. However, it's certainly true that the number of choices one must make in starting a project can be daunting. The choices you make at the start of a project can have far-reaching implications. Choose wisely and your endeavour might be easy and successful, but choose poorly and you'll almost certainly regret it later. For iPhone game developers, perhaps the most important choice that must be made early on in a project is which graphics libraries to use.

There are essentially two graphics libraries for iPhone game developers to choose between. Apple's Quartz and Core Animation libraries provide a fairly robust set of classes and methods for 2D drawing, and they're quite easy to use and well-integrated with regular iPhone UI elements. Of course, this simplicity comes at a price; the performance of Quartz leaves something to be desired for high-complexity scenes. The other option is, of course, OpenGL. OpenGL offers much better performance (and is of course capable of 3D, which Quartz is not), but it's also more difficult to use.

For Puzzle Panel, I wanted to be able to squeeze absolutely every drop of performance possible out of the iPhone to ensure the best experience for users. A simple 2D puzzle game might not seem like a high-performance application, but given the limited fillrate of the iPhone 3G GPU and the fidelity of effects I was hoping to achieve, I wanted to ensure that I wouldn't be limited by my choice of rendering platform. This made OpenGL the obvious choice, but given that I was also learning Objective C as I wrote the game, I was concerned about the amount of time it would have taken me to write the abstraction layers necessary to allow me to focus on gameplay.

Luckily, another option presented itself to me in the form of Cocos2D for iPhone. Cocos2D is a framework for 2D game creation, originally written in Python, but ported to Objective C for the iPhone by Ricardo Quesada. It provides a rich hierarchy of classes specifically designed for 2D game development, and handles all of the low-level OpenGL calls for you. Its features include texture loading, text rendering, sprite atlases, particle systems, timed actions, and physics simulation (through integration with Box2D and/or Chipmunk). Best of all, it's distributed under the GNU LGPL license, so it's open source and completely free for both commercial and non-commercial projects.

There are a multitude of excellent Cocos2D tutorials on the Internet, so I'm not (at this time) going to go into much detail about how it's used. For illustrative purposes, though, I will provide a specific example of how a scene is organized in Puzzle Panel:

The above is a side-view of a typical game scene in Puzzle Panel. Rendered objects in Cocos2D are always subclasses of CCNode, labeled "Node" in the diagram above. Each node (except for the top-level node, the Scene) has a parent and may have any number of children; Cocos2D uses a simple 2D scene graph in which nodes are rendered breadth-first. CCLayers, or "Layers" above, are simple Node subclasses which are typically used for scene organization: one might have a background layer and a foreground layer, and add sprites to each as necessary.

Essentially, creating a game is as simple as subclassing renderable classes (such as CCSprites/"Sprites") and adding your own behaviours. Once you've coded all of the behaviours you need, you throw a few Nodes in your scene and you're done! Cocos2D makes excellent use of object-oriented paradigms, and if you leverage that properly in your own code, you'll end up with a very flexible and powerful game engine.

Using Cocos2D probably saved me hundreds of hours of development time. In contrast to Puzzle Panel's mere 8,500 lines of source code, Cocos2D is comprised of more than 23,000 lines. As an amateur game developer, I found using a framework to be particularly helpful because it allowed me to maintain a higher-level perspective on the design of the game. By ignoring the complexities of OpenGL, I was able to spend more time thinking about how best to expand my game and make it as fun, polished and playable as possible. Using a pre-written framework also eliminates a lot of worrying about feature creep - by starting with a robust set of tools, it's less likely that one will spend unnecessary time writing lower-level features that are never ultimately used.

I don't wish to disparage the process of writing one's own OpenGL code "from the ground up." I'd love to write my own game engine "from scratch" for my next big project. Furthermore, a skilled OpenGL programmer could likely gain some extra performance by foregoing unnecessary features. I really do think, though, that Cocos2D is an absolutely excellent choice if you really want to focus on game design, while still having the flexibility to get "close to the metal" where necessary. If you're considering developing a 2D game for the iPhone, check it out.

Where are all the Computer Scientists?

2010-08-28T02:01:00Z

It's common knowledge that enrollment in computer science programs has declined significantly over the past decade. Although the trend seems to be reversing itself with enrollment increasing over the past few years, computer science educators shouldn't get complacent just yet. There's still much work to be done in order to ensure that computer science departments continue to grow and are able to meet the increasing demand for technically skilled graduates.

I've always felt that a lack of exposure is one of the primary detriments to computer science enrollment. While I don't have any hard data, my own experience combined with an informal surveyal of friends and colleagues leads me to believe that a large percentage of North American high schools—possibly even a majority—offer little in the way of comprehensive computing classes. While I don't dispute the importance of math, English and science, I can't help but wonder why computing isn't given an equal footing in the educational sphere, given its ubiquity today.

I've encountered a great number of intelligent and educated people who have no idea whatsoever about what the field of computer science is, or even how computers and software work. I understand that most people don't necessarily need that information. However, it's odd to me that basic computer literacy isn't required of college graduates (let alone high school students) in the same way that basic writing and math skills are. In the interest of meeting ever-increasing economic demand for technically skilled graduates, shouldn't we do more to expose students to computing?

To illustrate my point, here are a few examples:

I used to work retail, and small-talk at the till would frequently turn to my university career. Almost invariably when I told a customer that I was enrolled in a computer science program, he or she would reply "Oh, I always have so much trouble getting computers to work. I think mine has a virus." This was usually followed by "But it's good that you're learning about them; we'll always need people to fix them!"
While discussing my current job with a friend of mine who is in the third year of her degree program, I described my workflow to her. I told her that a bug report or feature request will come in from a customer, I'll gather requirements and implement a solution, and then make adjustments as necessary until the customer is satisfied. She asked, "How many do you get done in a day?" I couldn't help but laugh; there are very few tasks in client-focused software development which are both worth mentioning and completable in one day. Perhaps I'm being unfair here, as that question probably seems reasonable to someone with little knowledge of software development. However, no one would ask of a an architect, "How many buildings do you design in a day?" Why is software different? I'd argue that it's because most people are never exposed to software development.
In my third year of university I was required to take an English course to fulfill my breadth requirements. While discussing one of my papers with her, the professor complimented me on my writing. She said that I had a strong, persuasive style, and asked if I was a law student. She was shocked when I told her that, as a matter of fact, I was in computer science. She then told me that she was glad I'd taken her course (on Renaissance and early modern literature), as it would ensure that I was somewhat cultured and not simply a "computer nerd." I have a great deal of respect for that professor and I don't believe that her comment had any ill intent, but it again illustrates the fact that computer science is frequently misunderstood and/or not taken seriously.

I'm not entirely sure if these examples will properly communicate the point I'd hoped they would. I wouldn't want anyone to misunderstand the purpose of this post, either; I don't mean to complain about how computer science is perceived. I don't particularly care how people feel about my field of study.

My point is this: it seems to me that a large percentage of people, no matter how well educated they may otherwise be, have no idea how computers work or how software is developed. I feel that the primary reason for this is, in most cases, that they were never exposed to computing in a serious way. Until this is remedied, computer science enrollment will continue to be mediocre and the field of computer science will not reach its full potential. After all, who wants to go into a field he or she knows nothing about?

I've mentioned that I'm surprised at the lack of computer literacy I see in some college graduates. Colleges are not my primary concern, though. If computer science enrollment is to be increased, exposure to computing needs to start much earlier. At the very least, I'd like to see is comprehensive computing courses—which teach skills up to and including basic programming—made available in a majority of high schools. Ideally, these classes should not simply be electives; one computing class should be a required part of any high school curriculum. (I don't mean a course in touch typing.) Of course, given the real-world economic and temporal constraints that educators face, I don't expect that any of these things will happen any time soon. One can always hope, though!

On a related note, you may be amused to find out what a browser is.

Android First Impressions

2010-09-07T03:19:00Z

Like a number of disgruntled users and developers, I recently decided to trade my iPhone in for an Android device. My distaste for Apple's app store policies, design philosophy and corporate culture had been pushing me in that direction for some time. The final straw, though, was iOS 4, which rendered my iPhone 3G almost unusably slow while failing to grant any of the most noteworthy new features. The inordinate difficulty of downgrading iOS exacerbates the problem for users who are still stuck with the iPhone 3G. To be fair to Apple, Wednesday will see iOS 4.1 released, and they've claimed that it will fix the iPhone 3G performance issues. For me, though, 4.1 is too little, too late.

The phone I switched to is the HTC Desire on Telus:

With a 1Ghz CPU and 576MB of RAM, the Desire packs some powerful hardware. So far, I've been quite impressed by the hardware, but that's all I'll say about it for now. I'm more interested in relating my first impressions of the Android platform. In no particular order:

The openness of the Android market and of the platform itself is immediately apparent as an ex-iPhone user. I love it. I love being able to download emulators directly from the app store. I love being able to install interactive programming environments on my phone. I love having a native Latitude app. I love that Google Voice is available, even though it's currently US-only. Most of all, I love that neither Apple nor any other company can tell me what I can and cannot install on my phone.
Widgets provide an extremely convenient way to access information without having to switch between apps. While there aren't as many widgets available as I wish there were (or perhaps they're just difficult to find - more on this later), the built-in widgets alone have made the switch worthwhile for me. I frequently check my phone for Twitter updates, new e-mail from work, news, and what have you; with Android, I can do all of this without ever leaving the home screen. It's delightful. Beyond widgets, Android offers supreme customizability with themes, home screen replacement apps, and even (if you're adventurous) custom device ROMs.
Although openness and customizability are boons to the Android platform, they also result in serious fragmentation which could hurt the platform in the long run. I've already encountered widgets and apps which don't quite display properly on my device, presumably because of HTC's custom Sense UI. What's worse is how sluggish some manufacturers are to release critical operating system updates for their devices. I'm picking on Sony because they don't have a single device running Android 2.x, but even among other manufacturers, it doesn't seem unusual for ROM updates to lag behind Google releases by several months. Worse still is that carrier locks can prevent these updates from being pushed to devices. Android 2.2 "Froyo" has been available for the HTC Desire since late July, but Telus customers (who haven't unlocked their phones) still haven't received the update. Supporting custom UIs, old operating system versions and varied hardware will force Android developers to commit more resources to testing and fewer to innovation, and the platform as a whole will suffer for it.
Android is fast. Remarkably fast. I don't presently have enough information to objectively compare it to iOS, but I can say that I've been very impressed. Five years ago I would have doubted that such performance would ever be possible with Java running on a mobile device. (I'm aware that most of the core of Android is written in C, but the apps and UI are still Java. See correction below.[^java]) Having just purchased my Desire three days ago, I haven't had a chance to install Froyo, either, but benchmarks show that it increases performance by up to 700%.[^benchmarks]
Navigation and searching in the Android Market is absolutely terrible. I find it hard to believe that the Market is Google's doing, because searching the market never seems to bring up what I'd consider to be the "best" results, unless you type in the exact name of a specific app you're looking for. The aren't nearly enough categories, either.
On a related note, Android does lack some of the polish that iOS offers. There's not as much consistency in terms of UI elements and general design patterns. Customizing home screens and managing multiple apps which run simultaneously may not be something that typical phone users will care to do. For power users, though, it's not a problem. (Some sort of built-in task manager would be nice, though.)
Android is just a much better communication platform than iOS ever has been, and perhaps ever will be. Apple has finally grudgingly implemented something akin to multi-tasking (if you own an iPhone 3GS or better), but there's still no good way for iPhone apps to notify users of important events. When participating in a conversation using an instant messenger app (or any similar activity), it's crucial that you're notified of incoming messages, but notifications shouldn't be obtrusive and interrupt whatever else you may be doing. Android's notification bar is perfect for this. iOS doesn't even have what I'd call a serious attempt at a solution. And with Google Talk a core platform feature, Android devices can actually compete with the ease and convenience of BlackBerry Messenger. Meanwhile, Apple's just introduced... Ping. Snore.

I fully intend to do some serious Android development, so I'll surely write more about the platform as I become more familiar with it. For now, though, the bottom line is this: I love Android. Switching from iOS was an excellent decision and I doubt I'll ever look back. I'd urge anyone thinking about a phone upgrade to consider Android as well.

[^java]: A reader on Reddit brought to my attention the fact that the Android Dalvik virtual machine and its bytecode are significantly different from their Java equivalents, so although Android apps are written in Java, compiled Android apps are not really Java. (Apparently Oracle disagrees, mind you.)

[^benchmarks]: Another Redditor pointed out that the 700% figure may be something of an exaggeration. I've heard varying figures - anywhere from 2x to 10x - but 700% was based on the specific video I linked to. However, it's true that real-world performance will likely differ significantly from the results of synthetic benchmarks.

Android Development Device Setup in Ubuntu 10.04

2010-09-13T00:02:00Z

I spent part of last night setting up my HTC Desire as a development device on my Ubuntu laptop. This was a somewhat frustrating process, because a lot of the information I came across seem to be outdated and didn't work for me. Even the official documentation refers to a version of Ubuntu which is more than two years old. I did eventually piece together the correct process for Ubuntu 10.04, so I'm documenting that process here in the hope that it will help out other developers in the future. This guide assumes that you've already installed and configured the Android SDK.

Connect your phone to your computer and open a terminal window. Run adb devices to confirm that adb can "see" your phone. You should see something like this:
```
$ adb devices
List of devices attached
???????????? no permissions
```
Type lsusb to get a list of attached USB devices. Look for an entry with your phone's manufacturer; for instance, since I have an HTC device, I get:
```
$ lsusb
...
Bus 002 Device 003: ID 0bb4:0c87 High Tech Computer Corp. 
...
```
Note the ID - in my case, 0bb4:0c87. The part before the colon is the vendor ID. The part after it is the product ID. Keep these in mind.
You'll now need to create a udev rules file using your preferred text editor. This guide lists a number of different file names per-device, but based on my reading of the udev man page, I don't believe the file name is actually important. The official documentation recommends 51-android.rules, which is what I went with. You'll need to use sudo to get permission to create the file:
```
$ sudo vi /etc/udev/rules.d/51-android.rules
```

The contents of the file should be as follows:

SUBSYSTEMS=="usb", ATTRS{idVendor}=="[Vendor ID]", ATTRS{idProduct}=="[Product ID]", MODE="0666"

...where the bracketed bits are replaced with the IDs you got earlier from lsusb. For my HTC Desire, I entered:

SUBSYSTEMS=="usb", ATTRS{idVendor}=="0bb4", ATTRS{idProduct}=="0c87", MODE="0666"

Save the file. Most of the documentation I've come across says that restarting adb and udev is sufficient to give adb access to your device, but in my case I had to completely restart my computer before adb properly recognized my phone. Nevertheless, it's probably best to adb and udev first:
```
$ sudo restart udev
$ adb kill-server
$ adb start-server
```
Run adb devices again to see if your phone is recognized. If you still see ???????????? no permissions, you'll probably need to restart your computer.
You should now have full access to your phone from adb and Eclipse.

I hope this proves helpful. If you have any corrections or if this method didn't work for you, leave a comment.

Making Samba shares accessible to Apache on OS X

2010-10-04T02:29:00Z

As a small weekend project, I've been working on a simple Python HTTP "file server" of sorts. The basic idea is that authorized users can log in to a web interface and browse files in specific locations on my home network. The Python script is accessed through Apache on my Mac Mini, but some of the files I need access to are on Windows boxes. Unfortunately this presented something of a problem, as the Apache user on OS X (_www as of Snow Leopard) doesn't have permission to access Windows shares mounted by users.

Sounds simple enough, right? Just chmod or chown the Windows share and go to town. Unfortunately, this doesn't seem to work; neither chown or chmod have any effect on mounted Windows shares. Oddly, they don't output any errors; they just don't do anything. The only way around this seems to be to mount the shares as the Apache user (or whichever user you need to have access). Here's how to do so in Snow Leopard, with instructions modified slightly from a Stack Overflow question on the subject:

Create the mountpoint
mkdir /Volumes/Mount_Name

Set the permissions on the mount point

sudo chown _www:_www /Volumes/Mount_Name
sudo chmod 755 /Volumes/Mount_Name

If necessary, make a symlink from your Apache documents folder to the mount point
ln -s /Volumes/Mount_Name /Library/WebServer/Documents/Mount_Name
Mount the share as the Apache user
sudo -u _www mount_smbfs //DOMAIN;User:Password@Host/Share /Volumes/Mount_Name

I doubt a great number of people will need this information as it's a strange sort of setup, but it's my hope that these instructions will save someone some time.

Non-Myths About Programming

2011-02-04T04:53:00Z

While browsing Reddit today, I came across a short essay by Mordechai Ben-Ari titled "Non-Myths About Programming" (PDF warning). It's only six pages long, and I found it refreshingly realistic and insightful. I would urge anyone considering a career in computer science to read it. Given its brevity (and the fact the title is reasonably descriptive), I won't bother summarizing the article—instead, I'll merely say that you should click the link and check it out for yourself.

Android NDK Stack Trace Script

2011-04-09T23:30:00Z

I've finally started doing some Android development in earnest, and because I have a stubborn dislike of Java, I've decided to use the Native Development Kit. The native development kit allows you to write "performance-critical" parts of your application in C or C++ code, which is then compiled to platform-native ARM or MIPS instructions. The NDK is something of an unfriendly beast; it makes interaction with most regular Android libraries very difficult, and debugging native code on Android is none too easy. Although Google provides a script which can attach gdb to your native code, I've not yet figured out how to use it properly; every time I execute it, gdb complains about missing debugging symbols and libraries and seems to be unable to do anything useful. (I don't imagine it's too difficult, but I haven't spent much time on it—I've not needed to do much debugging yet.)

When your native code crashes due to a segmentation fault or some other similar problem, though, you do get a nice stack trace in the Android debugging log, which you can view with adb logcat. Unfortunately, the stack trace only gives you memory addresses, and you need to use another script to convert those memory addresses to lines of code. That script is already included with the NDK, but it's a bit of a pain to use. To that end, I've put together a simple shell script which glues everything together. I should mention that this is designed to be used with Linux or Mac OS X; if you'd like to do something similar on Windows, I'm sure it won't be hard, but you're on your own. Here's how to use it:

Create a folder in your project called "debug" (or whatever name you'd like)
Create a file called debug.sh, consisting of the following:
:::bash #!/usr/bin/env bash NDK_ROOT/toolchains/arm-eabi-4.4.0/prebuilt/linux-x86/bin/arm-eabi-objdump -S PATH_TO_SOLIB > SOLIB_NAME.arm adb logcat -d > logcat.txt python parse_stack.py SOLIB_NAME.arm logcat.txt
chmod +x debug.sh so that it's executable
Run the script with ./debug.sh each time you need to see a stack trace

Obviously there are a few things you'll need to replace in the above code. NDK_ROOT should be the path to your Android NDK installation. PATH_TO_SOLIB should be the path to your compiled shared object library. Usually this will be in your project folder under obj/local/armeabi. SOLIB_NAME can really be whatever name you want, but I liked to name the .arm file after the corresponding .so file for consistency.

You should repeat line 2 for each shared object library you need to debug in your project, and repeat line 4 for each corresponding .arm file. However, note that the arm-eabi-objdump process is quite slow, so if you have any large .so files which don't change frequently, you can skip dumping them to improve speed. Just remember that if they do change, you'll need to dump them again to get accurate results.

Android UI/UX gripes

2011-04-17T05:08:00Z

The week before last, whilst browsing Reddit I came across a blog by the name of "Android Gripes." The name is perhaps a bit misleading, as it's not actually a blog by an Android user hoping to spur Google into making UI improvements, as I would have expected. It is, rather, a blog by a self-proclaimed Apple fanboy (though he's recently edited his about page to remove that admission) complaining about Android for the sake of... well, complaining about Android.

That might be a little unfair. Perhaps I should let the author speak for himself:

I have constantly heard many gripes about Android. Recently I happened to have access to some Android phones and use them in a daily base. The more I use Android, the more I feel that Android is indeed a half-baked OS, its UX/UI is horrible and various hardwares are poorly designed. It’s true that those phones are working like smartphones, better than feature phones from old school, but they ignore a lot of the details. Just for Google’s own benefit, they helped handset manufactures brutally dumped a huge amount of unpolished devices to the market. They made people think this is how smartphones should work. This is so unfair to consumers. IMHO, it is no different from committing a crime. That’s why I made the cartoon of "Android Gripes" as a Monopoly "Go Directly To Jail" card showing an Android on it.

As much as I believe in Apple, I think everybody deserves great user experience, especially when it comes to phones. I think it is my mission to let more people know that Android does not provide you that. Android only provides inferior user experience. You either admit it and bear with it, or go for better alternatives. That’s why I started writing this blog.

(Emphasis added.) English is not the author's native language, so you'll have to excuse the occasionally odd grammar. It's immediately apparent that the author's stated opinion of Android is patently absurd; bad UI/UX can have many negative consequences, but it is certainly not criminal. Redditors have also pointed out that a number of comparisons the Android Gripes author makes are somewhat unfair. For instance, he compares old, oudated Android apps against apps which were recently updated on iOS, and misleadingly remarks on "extra" (meaning blank) space in the Facebook app. The blank space in question does contain content when one is signed into a legitimate profile.

With all of that said, I'm not writing this post to refute the Android Gripes author. I actually agree with an unfortunate number of his points. While there's a lot that can be said about where the iPhone's UI/UX fails (I've previously mentioned their complete lack of a workable notification system), excuses and rationalizations won't do anything to improve the state of Android apps. Even if you believe that Android's UI/UX is better than Apple's (and I do feel that that's a hard case to make), I think it's undeniable that there's room for improvement. It's certainly worth mentioning that some of the Android Gripe's authors examples are misleading, but what's more important is that we—that is to say, Android developers—ask ourselves where we're failing users, why it's happening, and what we can do to improve the platform. In a future post, I'll offer my perspective on where Android could be improved.

I'm considering switching back to iOS. Here's why.

2011-06-13T05:13:00Z

In September of last year I ditched my iPhone 3G and switched to the Android-powered HTC Desire (which I later sold in order to purchase a Desire Z, which sports a physical keyboard.) I'm sad to report that I'm now considering switching back to iOS, after less than a year on the Android platform. The following is an excerpt from an e-mail I wrote to a friend of mine whom I'd describe as a fervent Android evangelist, detailing some of the frustrations I have with the operating system and why I feel that I might be better served by iOS. I've edited it slightly to make it more readable for anyone who might come across this blog.

I really want to like the Android platform, and writing this gave me no joy. There are two reasons I'm posting it here, though. The first is simply that the purpose of this website is for me to post about my programming and my experiences with technology (as well as whatever else catches my fancy.) The second is that I hope that sharing my honest opinion about Android might be helpful to anyone considering a phone purchase.

As mentioned on Twitter, my number one complaint is having to use custom ROMs. There are always problems with custom ROMs. CyanogenMod 7 caused my phone to reboot at random. I also had problems with the Contacts app crashing. All of these issues persisted throughout multiple versions of CM7 up to and including final.

With the ROM I'm using now (a custom Sense ROM, can't remember the exact name), I can't set the SSID/password on my WiFi hotspot because Settings crashes when I try to. I seriously haven't used a single custom ROM that hasn't had some issues with it.
You mentioned that Google is promising quicker updates, but I'll believe it when I see it. Are they actually forcing manufacturers to do this in order to use Android branding or get Google apps, or are they just asking nicely? The latter won't work. And as far as I know, all that the Android update committee has promised is 18 months of updates following the launch of a new device. So does that mean if I release a device and then push a point release to consumers 17 months later, I've fulfilled that? All I see here is flowery language, and frankly I don't think anything is going to improve.

The fact is that it's not profitable for these manufacturers to update existing devices when they could spend the same development effort on releasing another six phones, so unless Google does something to change that, nothing will happen.

The rest are in no particular order:

No good Exchange client available. Even Touchdown, which is expensive, is pretty bad. The HTC client is okay, but it ~~doesn't support important features like threading.~~ [I later realized that this is untrue; the HTC client does support threading. I still find it cumbersome to use, and I've previously had problems with it failing to send messages without any notification of the failure. That's a pretty big problem when you're using it for work e-mail.]
Launcher redraw, agghhhhhh. Sense is horrendous for this, but I much prefer the built-in Sense apps to their Google alternatives, so I feel like I'm stuck using it. Even on stock Android, it seems like I can't really use live wallpapers or I have to wait for the launcher to reload all the time.
It's annoying that when I read an e-mail in Gmail on my computer, the notification for it stays in my notification bar. Also, my Gmail push seems to have stopped working of late. To be fair, it remains to be seen how the new iPhone notification system handles these things.
I've noticed that with multiple IM clients (specifically, AIM and eBuddy) if I leave them running in the background while I'm doing other things like browsing the Internet, I'll eventually get signed out or the app will close. I don't know if this is just because they're not written correctly, but it's annoying; IM is the only thing I really use background applications for and it doesn't seem to work consistently.
I often have trouble sending/receiving MMS. It's frustrating when someone sends me an MMS and I have to ask them to e-mail it to me instead because it won't download.
Camera quality on the Desire Z is pretty bad. I'm certain the iPhone 4 is better in this department. I feel like my Desire Z's camera is actually worse than my Desire's camera for still photos (although it's better for video.) This isn't a strike against Android in particular, but the problem of lower-quality hardware is endemic to the platform in a certain sense.
A lot of apps in the Android market are just very poor quality. The iPhone app store has its fair share of bad apps, but I still feel like the average quality is much higher there. In particular, any app that's available for both platforms is almost always better on iPhone. A prime example is Facebook. The Android client was horribly broken for several weeks, and regardless of that, it just doesn't support the same feature set as the iPhone version (e.g. watching videos.)
I complained about this even before I bought an Android phone, but the Android operating system is sometimes just not very responsive. Scrolling is sometimes smooth, but rarely as smooth as on iPhone/Windows Phone 7. Minor annoyance, but an annoyance nonetheless.
I'm not sure if you're familiar with the life cycle of an app in Android, but basically every different screen you can go to in an app is a separate "activity"—essentially a separate application. What were they thinking in designing it like this? It causes so many problems. Any time the "state" of the phone changes (e.g. you rotate the device, the phone leaves/enters roaming, etc.) the activity is destroyed and re-created unless developers manually override this behavior. This is why, for instance, the keyboard often disappears if you have it open and then rotate the phone. (Highly polished apps like Twitter handle rotation properly, but don't be fooled; it's not the default behavior. Many other apps do not handle rotation very well.) Other examples of activity recreation problems include pop-up dialog boxes disappearing and fields losing focus.
Editing text sucks; I find that my phone never seems to do what I want it to when I'm trying to highlight or copy and paste some text.
I've had problems on both of my Android phones with the music player pausing in the middle of a song at random. This has happened in multiple different media players. It's as though the phone thinks the headphones are being removed when they are in fact still in place.
I like being able to set alarm/ringer/media volume separately (although the iPhone can also do this), but I don't like that the defaults on Android aren't very sensible. For instance, if I have my ringer silenced, it probably means that I also want my media silenced. But that's not what happens if I just use the volume button; I have to actually go into settings and change the media volume separately.
Sometimes Google Talk just flat out refuses to start for me until I reboot my phone. This happened on multiple ROMs. On the Desire I noticed that it seemed to refuse to start when I was low on internal storage. I'm not sure why it does it on my Desire Z. This should be a core platform feature on par with BBM, but it seems to be very poorly supported.
From a developer perspective, I hate paying a development fee to Apple to be able to install my apps on a non-jailbroken device. I hate that once you've written an iPhone app, they basically blackmail you into continuing to pay them because otherwise they take down your app, which means that it may not be accessible to paying customers if they need to wipe their phone and re-install their apps. I hate their greedy policies with regards to content subscriptions. I hate their underhanded payment system which is, as a matter of statistical certainty, allowing them to make interest on millions of dollars that are nominally owed to developers but will never actually be paid. In general, I hate that they treat their developers like serfs just because they can, even though those same developers made their platform what it is today.

I hope that properly illustrates how much I hate Apple's policies, and I hope that it underscores the significance of what I'm about to say: from a pragmatic perspective, I'd still rather be developing for iOS because Google's development tools are complete garbage. I write all my code in vim, so I'm really not that picky, but I can only take so much of using Eclipse and having it literally delete my art assets from the file system. (Not just removing them from the project - it literally deletes the files, seemingly at random. I've never lost any work because of this, but it's extremely frustrating nevertheless.) Not to mention the Android emulator: it's practically unusable for regular apps, and for games, forget about it. You're lucky if you get 5 FPS.

Even if I buy an iPhone, I do still want to finish my game for Android. The code is cross-platform, so if I could compile and test it using Xcode in the meantime... even just using the iPhone simulator, since it can actually run games at a reasonable framerate... it would save me a lot of headaches. Either way I'll have to compile my app for Windows or iOS to do any marketing; taking screenshots of an Android app is difficult, and taking video is, as far as I can tell, impossible unless you have a phone with HDMI out and a capture card. Ridiculous.

The behavior of the back button is inconsistent, and developers use it as a crutch rather than thinking about and carefully designing a proper workflow through their applications. I was arguing about this behavior with someone on Reddit a while ago, and I found a good comment on Hacker News about it:

I just got an Android with that great Virgin Mobile deal, and I was surprised at just how much the Back button is used in almost all contexts. It's severely overloaded: want to get out of an on-screen keyboard? Back. Want to move back a screen? Back, except when it's Google Reader that opened a Browser window, in which case it takes you back to the previous page that Browser had. Back through a menu? Back. Back to a previous page in Browser? Back.
It's as if people just substituted the button for any instance in human language where "back" was used."

...(and another)...

"Back" to some random place most of the time.
1. receive email notification for two messages.
2. click notification bar
3. go to gmail app.
4. open first message and click a link.
5. you are now in the browser. (clicking back will close the window, but you want to read it later!)
6. you long press HOME button (hoping to select gmail and read the other message)
7. gmail is not in the open app list (WTH?!)\
8. you curse, click home, find the gmail app, click it.
9. now you are on the message with the link
10. press BACK
11. now you are on the message list (aha! not you home screen you came from!)
12. click the 2nd message and read it

In fairness, I'm of two minds about this because the back button does save a lot of screen real-estate by removing the necessity for on-screen navigation controls. However, I'm not sure how much of an advantage that is in the face of the confusion it sometimes causes.

[This wasn't included in my e-mail to my friend, because I didn't know about it at the time. I'm adding it now because it's fairly important.] Android can actually lose SMS messages. That's completely unacceptable on a phone.

What are lambdas good for, anyway?

2011-06-30T05:27:00Z

I had a discussion today about what lambda functions are used for, how they differ from regular functions, and why they're important. I thought I'd reproduce it here, as it seems that lambdas are often thought of as confusing and obscure. The truth is that they're very simple, and also quite powerful. While they can reduce readability if used incorrectly, they do serve an important purpose, and those who understand them should have nothing to fear from them. This discussion pertains specifically to lambda functions as they're implemented in Python, but they're very similar in other languages where they're available.

I think the best way to explain lambdas is with a simple example. Say you want a switch statement. Python doesn’t have them, but what you can do is set up a dictionary and get a value out of it, which is sort of like a switch. However, you can't do this...

my_dict = {
            1 : my_func1(x, y),
            2 : my_func2(x, y, z)
          }

...because the functions will be evaluated when the dictionary is initialized, so if they modify any state (or if they just take a while to execute) you’ll get bad behaviour. Functions are first class objects in Python, so you could insert the functions into the dictionary like so…

my_dict = {
            1 : my_func1,
            2 : my_func2
          }

...but that doesn't work either, because you don’t know how many (or which) arguments you need to pass to the function when you pull it out of the dictionary. So, what to do?

my_dict = {
            1 : lambda x, y, z: my_func1(x, y),
            2 : lambda x, y, z: my_func2(x, y, z)
          }

Lambdas solve the problem because they're evaluated lazily. By being wrapped in a lambda, the functions won’t be evaluated until the lambda itself is called. Furthermore, you can use the lambda to normalize the parameters of each function, so that you can always call it using my_dict[my_var](x, y, z), even though the first function being called doesn’t actually make use of the z parameter. There is no other way to accomplish this in Python, besides a big long ugly list of ifs. (Obviously you’d normally reserve this technique for situations where you have more than two options.)

You can also use the laziness aspect of lambdas to refer to variables which haven't been declared yet, although I'd recommend against that because it can become quite confusing. Anything within the lambda is evaluated only when the lambda is called.

And there you have it! Those are the basics of lambdas in Python. Of course, the above is only a trivial example of what can be accomplished with lambdas. If you'd like to learn more, there's a great write-up available here.

In Defense of Clones

2011-07-15T05:23:00Z

[2015-10-10]: I made some minor edits to this post to remove some content that I now feel is superfluous. These edits did not change the substance of the post.

With the phenomenal and growing successes of the iPhone App Store, the Android Market, and the Xbox Live Indie Games marketplace, gamers have seen an unprecedented number of titles which could be described as "clones" of more successful games—or rip-offs, if you're feeling less kind. Along with these titles has come a veritable tsunami of hatred for their developers, from game critics and anonymous Internet commenters alike. The commercial success of some of these titles, such as Angry Birds,[^birds] has shown that average gamers are happy to patronize any developer who can provide a fun experience, regardless of originality. However, any mention of a new clone on a gaming blog or podcast inevitably brings about a stunning amount of vitriol from gamers who, presumably, are upset that their favorite megacorporations aren't getting their due.

I don't understand this, but my best guess it that it stems from a misunderstanding of the motivations of game developers, and of the development process itself. I hope that by illuminating those things I can do something to stem the tide of rage and indignation. First, though, I'd like to briefly discuss the legality of game clones. The patentability and copyrightability of game mechanics and "look and feel" varies from country to country. Notably, the European Patent Convention explicitly states that "schemes, rules and methods for performing mental acts, playing games or doing business, and programs for computers" are not patentable, whereas no such restriction exists in the United States. There are several notable examples of game-related patents in the United States, but the industry is young and there are relatively few legal precedents on their enforceability. I am not a lawyer, so it would be unwise of me to assert too much here, but I will state that throwing around terms like "copyright infringement" is neither productive nor terribly meaningful when discussing game clones.

I'd also like to make clear that by "clone," I mean a legitimately developed game with similar gameplay mechanics and/or art style to an established title. I am emphatically not referring to situations in which actual code or art assets have been used without permission; under no circumstances do I condone actual copyright infringement. Finally, in the interest of full disclosure, I should also inform you that I have a dog in this race, so to speak: my game, Puzzle Panel, borrows gameplay mechanics from Tetris Attack and takes artistic cues from World of Warcraft.

A comparison between Puzzle Panel, left, and Tetris Attack.

Tobar for iOS is a recent example of the kinds of titles I'm talking about. The game is a fairly obvious mimicry of Super Meat Boy, and while Meat Boy co-creator Edmund McMillian is relatively unperturbed by it, many of his fans are less forgiving. I'm going to pick on the Destructoid community here for no other reason than that Destructoid is one of my favorite gaming publications, but a quick Google search reveals no shortage of hatred for Tobar.[^hatred] Destructoid user Noir Trilby writes:

What a shameless rip off. It's good that Ed is cool about it and all, but if I was him I'd be raging. Both Apple and Android need someone doing some quality control on their aming services to screen out blatant rip offs like this.

As I've already mentioned, I don't think Tobar, nor other games like it, deserve the hate they receive. There are a number of reasons why I feel this way. First, all art is iterative, and video games are no exception. Creativity is not binary, but is a matter of degree. Second, when indie developers borrow from established properties, there's little evidence that it's detrimental to the owners of those properties, whereas it may be extremely beneficial to consumers. Finally, I feel that there's a misconception that games like Tobar exist because lazy, greedy developers want to make a quick buck and can't be bothered to do the work to come up with an original title. That perception is wrong on many levels, which I'll discuss in further detail in the following paragraphs.

The apparent trend of smaller developers "ripping off" well known games is probably an illusion. It's likely that these are simply the instances of liberal borrowing that we hear about, due to the popularity of the "borrowee." Take Angry Birds, for example. Most people would point to it as an example of a successful, original title by a studio that was relatively small at the time of its release. The truth is that although its art style is unique, its gameplay mechanics are borrowed heavily from an older flash game called Crush the Castle. Or how about Super Meat Boy? It surely owes some of its success to predecessors such as Jumper Two, I Wanna Be The Guy, and N+.

A comparison between Super Meat Boy, N+, Jumper Two, and I Wanna Be The Guy.

Granted, each of these games have their own original elements, whereas Tobar at least appears to be a fairly direct attempt at reproducing Super Meat Boy. But every great game stands on the shoulders of giants, and every developer has to start somewhere. Games like Tobar should not be condemned as soulless exercises in profiteering, but recognized for what they are: a means for inexperienced developers to hone their skills while contributing to new markets, much as a musician might start out by covering popular songs in venues where the original artist would never perform.

Words like stealing and theft are often thrown around in discussions of game clones. But one of the defining characteristics of theft is that it deprives the owner of his or her property. What harm is the existence of Tobar doing to Team Meat? Given that Team Meat's Tommy Refenes has publicly stated that he has "no intentions on doing anything for iPhone or iPad ever...I hate the [iPhone and iPad] platforms and I hate the [app] stores," Tobar is in no sense a competitor to Super Meat Boy. I don't know the developer of Tobar, but I'd wager that his intent was simply to bring gameplay he loves to a new platform that he's passionate about.

Passion is exactly why I created Puzzle Panel; I wasn't looking to make quick money by ripping off someone else's creativity and putting it on a new platform. Rather, I wanted to share a game that I loved with an audience who might have never seen it before, on a platform where it was unlikely to ever see the light of day otherwise. I am not a veteran game critic who values unique mechanics above all else because he's seen most gameplay elements a thousand times before, nor am I a would-be artiste desperately attempting to create a legacy for himself. For these reasons, I value fun over originality. If people enjoy my game, that is enough for me. If a nervous child at the doctor's office finds solace in something I've created, my work has paid off immeasurably. If I can contribute just a little more happiness to someone's life, I am satisfied.

"If the motivations of indie developers are purely in the interest of gamers," you might ask, "why charge money for these clones?" Well, the reasons are simple. First, we're human, and we need to eat; it's nice to get paid for one's work, even when said work is not completely novel. Second, even discounting the dozens or hundreds of hours of effort required to make the simplest of games, there are significant up-front costs to developers in creating games. Developer licenses, development software, test hardware, and licensed or commissioned music, sound effects and graphics add up very quickly.

I did all of the programming and artwork myself for Puzzle Panel, as well as much of the sound, and it still cost me over $1,000 to make. To a student, that is a great deal of money. I charged a dollar for the game, but in the end I recouped none of the cost of creating it. I don't regret a thing, but this illustrates the point that game development is anything but a quick and surefire way to get rich. For every developer making money on their games, there are literally tens of thousands who sink in countless hours and significant money, only to receive nothing in return. That being the case, I don't give much credence to the argument that developers such as Tobar's are just in it for the money.

PinchMedia's statistics show that the top 10% of paid apps receive more than 80% of all downloads; most developers are left in the cold.

Thus far I've been writing specifically about indie developers. I'd be remiss to not mention that established, big-budget studios often clone each other's games, or worse, the games of indie developers. I'm much less sympathetic to this; particularly in cases like Capcom's obvious copy of indie hit 'Splosion Man. But although I personally find this distasteful, I still don't see a great need for alarm. A well-made, original title which has found an audience has little to fear from pretenders.

Now, please don't misunderstand me. I am by no means saying that every developer should just go out and rip off the biggest franchise he can because there's nothing wrong with unoriginality. On the contrary, innovation and creativity should be encouraged and rewarded, and while I think it's fine for hobbyist programmers to start out by mimicking their favorite titles, we should definitely turn a more critical eye to established studios who have the time, budget and experience to be original but are content with releasing an endless stream of sequels, clones and remakes. But please, let's cut the indie developers some slack. Cloning a popular game to make a quick buck is simply not a winning proposition. Regardless of their originality, regardless of their financial success or failure, I can assure you that indies do it for the love.

[^birds]: Although it has its own unique art style, I count Angry Birds among the clones for its striking similarities to Armor Games' Crush the Castle.
[^hatred]: Ironically, the "negative" publicity Tobar is generating will probably help its sales more than anything else could have.

How to reinstall OS X from a disc image using only one computer

2011-09-21T07:17:00Z

My internship recently ended, and as such, I had to move from Calgary back to Edmonton, where I'll be completing the fourth year of my Computing Science degree at the University of Alberta. I hope to be able to write more about my internship later, but at the moment I want to share a handy trick I discovered for reinstalling OS X. The reason I bring up the move is that after I arrived in Edmonton, I found myself wanting to reformat my Mac Mini and re-install Snow Leopard, but my Snow Leopard disc was nowhere to be found. Luckily, as a semi-paranoid computer scientist with an abundance of disk space, I keep images of important discs on my network-attached storage device for just such occasions.

Unfortunately, however, the Snow Leopard disc image is large enough that it can only be burned to a dual-layer DVD. I don't tend to keep writeable dual-layer DVDs lying around, so I needed a solution which required only the disc image itself. There are a number of methods available for installing OS X over the network, but none of them seemed to work for me—perhaps owing to the fact that I had only one Mac. Luckily, I discovered that you can install OS X from a separate partition on the same disk. Here's how:

First, you'll need to re-partition your disk. Open Disk Utility and select the disk you want to use. Be sure to select the disk itself, rather than one of its partitions, which are indented beneath the disk.
Click the + button to add a new partition, and drag the new partition's section in the disk layout graph upwards until it's large enough to store your operating system disc image. Apply the changes. It may take some time for the disk to be re-partitioned.
Now, select the new partition from the list on the left. Go to the Restore tab, and restore your operating system disc image to the new partition.
Once the restore is complete, restart your computer, holding Option (Alt) at start-up. This should bring up a list of available start-up disks. Choose your new partition.
You should now be at the installer, and you can re-install to your primary partition. Once the installation is complete, delete your installation disc partition to and resize the primary partition to reclaim the extra space.

That's it! In retrospect, it's completely obvious, but I haven't seen these instructions anywhere else on the web. This technique might be particularly useful for owners of Macs without disc drives.

How to link GLUT in Qt Creator on OS X

2011-09-21T07:34:00Z

This tutorial is for an extremely specific and possibly unusual set of circumstances, but I'm hoping that it will prove useful to someone. For one of my courses this term, we're building OpenGL apps using Nokia's open source Qt framework. Qt provides some very nice scaffolding, and the Qt Creator IDE is among the best I've used, although I imagine it lacks flexibility for general-purpose applications—that is, you probably wouldn't use it to build non-Qt applications. Qt also allows one to easily write cross-platform C++ GUI applications. That said, linking procedures can be a bit different on different platforms, and as a beginner it's not always immediately obvious how instructions for Windows development translate into other environments.

In particular, I found myself having difficulty linking the OpenGL Utility Toolkit (GLUT) with my project on Mac OS X. After much Googling and some frustration, I came up with the correct way to go about it, which is as follows:

In Qt Creator, open your project file (.pro) in the text editor. Right click on the text editor pane and select "Add Library..."
Chose "System Library."
Locate the GLUT framework. On my Mac, running Lion, I used /Developer/SDKs/MacOSX10.7.sdk/System/Library/Frameworks/GLUT.framework. However, it seems there are multiple different copies on the disk that you could use. If that path doesn't work for you, just search for glut.h in Finder and you'll find a copy of the framework. The path you enter into Qt Creator should point to a .framework directory.
Qt Creator will generate code in your project file which will link to the correct framework or library depending on which platform the code is being compiled on.
Finally, because Apple loves to be different, the glut.h header path is different on OS X. You'll need to #include <GLUT/glut.h>, instead of <gl/glut.h> is usually the case. You can use preprocessor conditional statements to make this inclusion cross-platform:

:::c #if defined(APPLE) || defined(MACOSX) #include <GLUT/glut.h> #else #include <gl/glut.h> #endif

That's it! If you have any trouble, leave a comment.

Ethical Game Design from Gamasutra

2011-12-31T00:28:00Z

I came across a couple excellent articles on Gamasutra which I want to share here. The first, "Contrivance and Extortion: In-App Purchases & Microtransactions" is by Adam Saltsman, creator of Canabalt, among other games. He argues that in-app purchases and so-called microtransactions are being abused by some game developers, "deliberately contriving their designs in the worst way in order to extort money from players, which is unethical and unacceptable design practice." I'll lay my cards on the table and say that I am in complete agreement with Adam. As he does in his explanatory follow-up, I acknowledge that there are benefits to the "freemium" model, but the techniques commonly associated with that model are often used in a cynical, manipulative and profit-driven manner which I feel is harmful to consumers and the industry. Whether or not you agree, Adam's thoughts on the subject are certainly worth reading.

"Radical Plagiarism: The Ethical Lessons of the Gamenauts Controversy," by Evan Jones, is equally thought-provoking. Evan recounts the cloning of the indie game Radical Fishing by the more established indie studio Gamenauts, and adds his perspective on the cultural impact of the rampant cloning that takes place on mobile app stores. Returning readers may recall that I've previously defended the practice of cloning games. My defense excluded the cloning of smaller titles by established studios, which I've always viewed as morally questionable at best, but Evan's article nevertheless gave me pause. He provides a strong counterpoint to my own thinking on the matter, and while I'm not yet ready to reverse my position, his argument has undeniable weight. If you're at all interested in game development, both articles are well worth your time.

2010 and 2011 in Review

2011-12-31T23:59:00Z

If you'll forgive me for "metablogging," I've been ruminating lately on the purpose of this website. Up until now, I've been trying to focus exclusively on technology, programming and game design, and to minimize the subjective aspects of my connection to those fields. Put more simply, I've been avoiding writing about who I am and what I do in favour of producing a series of mostly impersonal tutorials and critiques of technology. There are a number of reasons for this—the word "professionalism" comes to mind as chief among them. I'd like this website to be my public face on the Internet, and as every saavy Internet denizen should know by now, there are certain risks associated with maintaining a public persona. I greatly admire Zed Shaw, for his personality as well as his technical accomplishments, but I have no desire to make myself unemployable as he claims to have done (albeit temporarily).

That said, trying to take a dry, detached and objective approach to creating a personal website presents a number of problems itself. First and foremost, it's an inherent contradiction; if I don't write about myself in any meaningful capacity, then there is nothing aside from the superficial about this website which makes it mine. I can't imagine it makes for a very interesting read, either. Realistically, how many people will come here looking for advice on how to link GLUT? Readership numbers are of little to no concern to me, mind you, but I can at least endeavour to give this website a purposeful existence. To that end, I want to start writing more about personal topics, although for the sake of cohesiveness I'm going to continue to focus primarily on those aspects of my life that will be of interest to a technically-minded audience.

As a start, I think that this is a good time to reflect on what I've done since I started this blog in May of 2010. Early in May, I moved to Calgary, Alberta, to start a sixteen-month internship with Pason Systems Corporation. Pason is the market-leading provider of oil rig instrumentation and service, with a wide variety of tools and technology available to facilitate safety and improve efficiency on oil rigs. All of Pason's rig-site hardware and software is designed in house, but my work was on an internal web application designed to support Pason's most valuable employees: the field technicians who directly provide services and equipment to customers.

For obvious reasons, I can't go into great detail about the work I did at Pason. I will say, though, that it was extremely rewarding. I had the opportunity to work alongside some of the smartest developers I've ever met, and I was able to do work that I felt had a tangible, significant and lasting impact on my team's product. That's a rare honor for an intern! From a technical perspective, I got to work with some interesting and unfamiliar technologies; I did extensive development in Python, created rich user experiences with HTML5 and JavaScript, and dabbled with C# when the need arose. The whole experience was hugely instructive, and I will always fondly remember my time at Pason. Of my entire education, my internship is without a doubt the part that I value most.

In September of this year, I moved back to Edmonton to finish my Computing Science degree at the University of Alberta. It being my final year and with most of my specific course requirements out of the way, I loaded up on computing science courses, which led to a fairly engaging but difficult semester. Among the courses I took were Advanced Games Programming in C++, Compiler Design, and Introduction to Computer Graphics.

As a term project for the Advanced Games Programming course, I undertook a modification of the ORTS game engine to allow user interface scripting with Lua, à la World of Warcraft. I feel that the outcome of this project was a little underwhelming, as limited time and the complexity of ORTS forced me to settle for less comprehensive scripting capabilities than I would have liked. Nevertheless, I learned a great deal about Lua and the process of interfacing with Lua from C++. I think I now have an inkling of what's involved in creating a robust client-side scripting system as in World of Warcraft, or even a custom development platform like Corona. Given the popularity of Lua as a scripting language in game development and my interest in developing games, I'm sure it will be valuable experience.

Although I believe I would be within my academic rights to do so, for the time being I don't foresee myself releasing the source code for my Advanced Games Programming project. ORTS has not seen active development in a number of years, so it is outdated and difficult to compile and run, let alone modify. Additionally, my own work—being a school project as it is—would probably not prove to be a good demonstration of Lua/C++ integration best practices, and I'm always wary of releasing code from which others might learn the wrong way of doing things. I do have a report on the project which I'd like to reproduce here at some point, though.

For University of Alberta Computing Science students, I have some advice which I'd like to share regarding the Advanced Games Programming course. If you already know C++, as I did, I would have to recommend against taking it. Much to my chagrin, the majority of the lectures were actually about C++ in and of itself, rather than algorithms and techniques relevant to games. It certainly would be helpful for anyone who wants to pick up C++, but if you're looking for a thorough examination of game programming, look elsewhere.

Compilers was perhaps my favorite course of the semester, although it was also by far the most difficult. I would describe it as a strangely paradoxical course. The theory behind formal languages, parsing and code generation is not extremely complex, but it can be initially hard to grasp, as it is very heavy on jargon and has numerous rules and axioms to memorize. The process of building a simple compiler for a simple language is, in retrospect, fairly mechanical and manageable, but it seems anything but when approached without the benefit of hindsight. Overall, the course is just a staggering amount of work, but it is also among the most rewarding courses I've taken.

Finally, Introduction to Computer Graphics, I got to expand upon the OpenGL knowledge I had built during the development of my iOS game, while also learning the fundamental aspects of computer graphics. Like Compilers, Graphics is a challenging but rewarding course, one of the most difficult aspects of it being the time commitment required. (Of course, it is also fairly math-heavy.) As someone who appreciates the beauty of a well-designed, simple algorithm, I found that learning about such achievements as the midpoint circle algorithm was one of the most enjoyable aspects of the course. I feel that there's an almost Zen-like quality to a clever but simple algorithm which strips away preconceptions of complexity to accomplish a task in the most efficient manner possible. If you're of a similar mind, graphics is full of examples of such algorithms. Of course, working in graphics is also rewarding for its ability to produce visually interesting results, which is a rare treat in academic computing science:

A demonstration of shadow volumes from an Introduction to Computer Graphics assignment.

2010 and 2011 were exciting years for me, and although I'm not ready to discuss it yet, I have something huge lined up for after I graduate in April of 2012. In the meantime, I'll continue to post here as often as I can manage. Thanks for reading, and happy new year!

Some thoughts on used games and online passes

2012-02-04T07:37:00Z

Online passes are becoming more and more ubiquitous in the games industry. For those who aren't familiar with the terminology, an online pass is essentially a form of digital rights management designed to discourage and devalue used game sales by tying some game content to unique, one-use codes packaged with new copies of games. Online passes are a contentious issue. Some feel that they're a perfectly legitimate way for developers and publishers to ensure that they get their cut of each sale, and to reward those who purchase games new. Others see them as a violation of consumer rights or contend that they inconvenience customers who do buy new (by requiring them to type in long access codes on game controllers) while being of only dubious value to publishers and developers.

Most recently, there's been a bit of an uproar over the inclusion of an online pass in Kingdoms of Amalur: Reckoning, a single-player RPG with no online component. The pass confers seven extra quests on those who purchase the game new, or deprives those who purchased used of them, depending on your perspective. Before I lay my cards on the table, I'll let Curt Schilling, the founder of the studio behind the game, explain what he sees the purpose of the online pass as:

That being said, this next part is likely to piss people off, but it's a truth and it's how I feel. You can argue the merits and effectiveness of it, but right now it's how it's done and as someone that's as invested as I am in this company, I stand by what has happened.

DAY 1 DLC, to be extremely and VIVIDLY clear, is FREE, 100% totally FREE, to anyone that buys a new copy of Reckoning, ANYONE.

If you don't buy new games you buy them used, and in that case you will have to pay for the Day 1 free DLC content the new copy buyers got for free.

It's clear the intent right? To promote early adopters and MUCH MORE IMPORTANT TO ME, REWARD fans and gamers who commit to us with their time and money when it benefits the company.

You can read the rest of his post on the Kingdoms of Amalur forum. I think Curt's position is reasonable and I am sympathetic to the frustration he's clearly feeling; he believes that he's trying to protect his business interests and in the process rewarding loyal fans with free bonus content, and receiving nothing but scorn in return.

I don't think Kingdoms of Amalur's online pass is an egregious abuse by any means, but in general, I do count myself among those who see online passes as an unsettling trend. I think consumers have good reason to be suspicious of companies that choose to use online passes. There are good arguments for and against online passes, but the most popular arguments in favor of online passes often seem to rely on strawmen and dubious assumptions with no supporting data. Jerry Holkins (Tycho) of Penny Arcade writes:

I've been reading a lot his weekend about Fat Cats and how fat they are and how they want your money, but the only choice you get in this matter (aside from the wholly valid "not buying it" choice, of course) is which supposed Fat Cat to enrich. You can enrich the people who made the game you are enjoying, or you can enrich people who had nothing to do with the game. Policies like this are designed to incentivize new purchases: that is to say, sales. We call those sales.

In other words, you're either supporting the developers who produce the games you love, or you're supporting retailers like Gamestop. Naturally most consumers would prefer the former, all else being equal; developers are creative entities, focused—or so we can hope—on artistic expression before profit. Retailers, by contrast, are often viewed as greedy and incidental vestiges of the rapidly dying era of physical media. Their services are at best unnecessary, and at worst abusive of their customers to an extent that would make an ardent objectivist blush.

This is a false dichotomy for two reasons. First, traditional retailers are not the only avenues for the trading and selling of used games. One could trade games directly with friends or colleagues, or buy and sell them at garage sales and flea markets. There are even a number of online services that have been created for the explicit purpose of allowing consumers to trade and sell used games at reasonable prices. (Of course, those services take a cut too, but I think it's safe to assume that their margins are nothing like Gamestop's, and they are much more consumer-centric in general.)

More importantly, there is an implicit assumption in Holkins' argument that each used sale deprives the publisher of a new sale. This is the same false equivalency that the MPAA and RIAA have been using for years to overstate the negative effects of piracy, and it is no less pernicious when applied to secondhand markets. To the best of my knowledge there is absolutely no data to support this assertion, and it doesn't really hold up to logical scrutiny. Sure, a sale of a used game does not directly support the game's publisher and developer, but there's no guarantee that the buyer would otherwise have bought the game new. More importantly, the person who traded in that game for store credit is likely to use that credit toward the purchase of a new game.

"Triple-A" retail games cost $60 in the US and Canada these days, and in countries like Australia prices can reach AUS $100 and above. That is, frankly, an absurd amount of money and out of the reach of a great many people. Without a healthy used market to mitigate these prices, many consumers would simply be forced to purchase fewer games. Others would likely turn to piracy. Nobody wins in that scenario; not developers, not publishers, and certainly not gamers.

My arguments thus far have been based on assumptions and logic rather than empirical data, just as the arguments I am attacking are. What I am postulating is that my analysis is at least as plausible as the alternative (in my opinion, moreso), and that in the absence of actual scientific data the sane solution for publishers is to try to compete with used sales by offering better service and pricing incentives rather than trying to find technological solutions which, even when well-meaning, embitter those who can't afford to buy every game they are interested in at $60 a pop.

Now, you might be thinking that because I'm not in the industry, my perspective is limited. Maybe publishers do have data which support the conclusion that used game sales are a net loss. It's possible; it certainly makes sense that entities which exist solely to maximize game sales would want to have that data. That said, given the ineptitude displayed by media giants in fighting piracy, I am not convinced that it should be taken for granted that they take a scientific approach to anything. Take this exchange between Kristen Bornemann and I on Twitter:

@kristen_: @cwgabriel When people pick up used games it reduces the chance of publishers investing more money into the game (sequel, etc).
@kristen_: @cwgabriel Therefore it directly hurts us as devs… especially 3rd party devs.
@lindgrenM: @kristen_ Do you have statistics that demonstrate that, or are you just assuming so by the false equivalency of 1 used sale = 1 lost sale?
@kristen_: @lindgrenM I worked in publishing. The fact is if a publisher hears that used sales are high = death for game. They don't care about stats.
@lindgrenM: @kristen_ Then perhaps the community should focus on fixing that, rather than limiting consumer choice?
@lindgrenM: @kristen_ Because if the publishers "don't care about stats," what are their business decisions based on? Vindictiveness?
@kristen_: @lindgrenM hearsay and gut/instinct… why do you think I left? ;)

I'm aware that this short Twitter exchange is by no means a definitive example of how every publisher everywhere conducts business. Again, I'm simply saying that until I see data which actually show a net loss for publishers and developers from used sales, I see no reason to take as gospel the assertion that they are a net loss. That said, even if conclusive data were available, I still don't think I could support online passes at the conceptual level. Amalur is innocuous, but it's the exception rather than the rule; most online passes lock used buyers out of multiplayer until they've shelled out another ten to fifteen dollars. That is blatantly vindictive anti-consumer behaviour on the part of publishers, and it epitomizes the troubling trend of media companies punishing legitimate consumers in a misguided attempt to artificially control the marketplace and perpetuate outdated business models. For an example of exactly how far this thinking goes, consider this post by Jameson Durall of Volition:

There’s another big rumor about the next Xbox console that could really start to shake things up... it won’t play used games at all! Personally I think this would be a fantastic change for our business and even though the consumers would be up in arms about it at first... they will grow to understand why and that it won’t kill them.

To say that entirely removing the used market "won't kill" consumers strikes me as extremely ironic. Used games have existed for as long as games have existed, and have evidently not killed the industry, and yet that is exactly what developers and publishers are complaining will happen if we continue with the status quo. Durall himself says that used game sales are "making it more and more difficult for us to continue making higher quality products." My feeling is that the rhetoric is a lot more emotionally charged on the part of online pass supporters; there is a strong insinuation that used buyers are costing people their jobs and are in some sense morally inferior to those who have the luxury of being able to buy only new games.

But I digress. I don't give too much credence to the rumor Durall refers to, but the idea that anyone would consider it a good course of action is mind boggling to me. It's another step in the march towards the complete erosion of consumer rights, and along the way the products we purchase are further crippled and devalued. It hurts consumers and it hurts content producers because this devaluation of legitimate products is what drives growth in piracy.

I must clarify exactly what I mean when I say that product devaluation drives growth in piracy. It is a popular belief among technology writers on the Internet that services which compete on convenience, such as iTunes and Steam, can actually reduce piracy. The data that I have seen do not support that conclusion. However, history has demonstrated that a legislative or technical solution is likely impossible because pirates will always find ways to break digital locks and make enforcement of anti-piracy legislation effectively impossible. Any legislative solution broad enough to admit the possibility of significantly decreasing piracy by inconveniencing would-be pirates would likely have negative repercussions outweighing the benefits, and the assumption that a nuking-the-anthill style solution à la SOPA would be effective for more than a few weeks is extremely dubious anyway.

So what are content producers to do? Pandora's Box has been opened and the clock cannot be turned back. There will always be people with no compunction about downloading works without even attempting to compensate the artist, and dishearteningly, the research I have read indicates that once someone is turned on to piracy he is unlikely to stop—even if content producers then offer better incentives to compete with pirates. It has been demonstrated, however, that removing options increases piracy. Devaluing products and services with arbitrary restrictions and inconveniencing legitimate consumers will have the same effect. Copying is the fundamental operation on which the principle of computing is built. It cannot be stopped. The only thing content producers can do is to prevent people from turning to piracy in the first place by offering the best service possible.

Returning to the topic at hand, online passes go hand in hand with the consumer-as-enemy mindset that plagues media giants. They arise from the same attitude that justifies $30 BluRay movies with minutes and minutes of unskippable commercials, and always-on DRM which prevents legitimate purchasers from playing single-player games when the publisher's servers are down. Through legislation and technology, these companies are trying to make ownership rights exclusive to corporations and prevent consumers from interacting with the media they purchase in any manner that is not explicitly pre-approved. They are declaring a war on their customers, and it is a war they cannot win. Make no mistake, though: as these behemoths run themselves into the ground because they are incapable of adapting, there will be casualties among the artists, musicians and developers who create the works of culture we love.

That is why I cannot support online passes. I truly believe that in the long run they are detrimental to all involved. I truly believe that the best way for a company to protect its interests is to do its utmost to reward its fans and give them the best experience possible—all of them, even those who do not directly contribute to the bottom line.

Social networks and content ownership

2012-09-14T07:45:00Z

I'm always wary about pre-announcing things I'm working on; I have more ambition than spare time, which leads to a tendency to start projects and not finish them. But if there's to be a point in maintaining this website, I need to actually write things once in a while, so I may as well write about what I've been hacking on lately. At the moment, I'm building a web-based photo album in Python. My primary reason for doing this is that I'd like to own more of my information. Apps and social networks have some significant advantages, particularly for the less technologically inclined, but I'm ambivalent about the trend away from an open Internet and towards a set of non-interoperable, monolithic walled-garden services.

When Google+ debuted, my enthusiasm for the service followed a sort of bell curve. I was already using Twitter and Facebook, and I didn't want to have to post everything to yet another service. At the urging of friends, I joined anyway, and warmed to it considerably. It was simply better software than Facebook; the interface was cleaner, the privacy controls more usable, the communication options more diverse, the means of sharing and interacting with media more engaging and deeply integrated. Best of all, there wasn't nearly as much app spam. Much of this is probably still true, although the impression I get from Google+ pages I've stumbled across recently is that they've been trying to make it look more like Facebook, to its detriment.

Realizing that it was a better product in almost every regard, I started using Google+ almost exclusively and tried to recruit additional friends away from Facebook. I stopped uploading new photos to Facebook and even purchased additional storage capacity on Google+ so I could upload all of my photos there at the original quality. Problems quickly became apparent, though. Despite my best efforts, nobody was using the service; even among those of my friends who were enthusiastic early adopters, actual social activity among my circles dropped off quickly, and I soon found myself wondering why I even bothered to check for updates. I finally deleted my account when I logged in one day and realized that the newest post in my news feed was from several months prior.

And with my Google+ account, there went all of the photos and other content I had uploaded. I still have the files, of course, but the effort I expended in selecting a handful of good photos to upload to each gallery from the hundreds of shots I take on most outings, not to mention the comments, captions, and other metadata I had curated, vanished into the ether with not a trace left behind. The moral of the story is that private social networks are impermanent, and thus anything uploaded to them necessarily has an unknown but very much finite period of accessibility and relevance. Whether or not Google ever admits it, Google+ never quite took off, but it could have supplanted Facebook just as Facebook did MySpace. If and when some other service comes along and succeeds where Google+ failed, every user on Facebook will be forced to start over from scratch on whatever's in vogue next.

I'm tired of playing that game. I want to keep control over the content I create so that I can truly decide how it's shared and with whom, and so that if I need to migrate my data to a new system, I have the means to do so.[^export] The ultimate realization of this ideal is in decentralized services based on open protocols, such as Tent. I'm very enthusiastic about such attempts, but realistically, I'm skeptical that any of them will ever gain the necessary traction to make a real difference. Even if Tent or some analogue gains millions of users, it will be a drop in the bucket compared to Facebook, and entrenched players such as Facebook will have no incentive to interoperate because decentralized services do nothing but threaten their business model.

I will do as much as is in my power to help services like Tent succeed, but for the time being, that is not much. That kind of leaves me without a good means of pushing my content to the people who care about it. Nevertheless, I can still take ownership of it by hosting it on my own server using my own tools. That is the purpose of developing my own photo album software. I'm making good progress so far, and hopefully soon I'll have more to share; once the code is in a reasonably complete state with most of the major features I want for my own purposes, I intend to make the app open source and share the code on my Github account. And now that I've announced my intent publicly, I expect anyone who reads this to hold me to my plan to ensure that I get it finished!

[^export]: Both Facebook and Google+ provide a means of exporting the data you've shared. I believe both will export everything belonging to you personally, but I don't think you necessarily get to keep all of the social data associated with your content, e.g. comments, nor the relationships between different pieces of content. Last I checked, the data was merely dumped into a bunch of flat files; nothing is kept in a structured format that makes it easy to work with, and even if it were, other privates services don't usually offer the option to import your data. While it may still not be easy, if you own your own data and manage it using open source tools, it's at least possible to migrate everything to another system with the greatest fidelity supported by that system.

The state of Python imaging

2012-09-14T10:36:00Z

(With apologies to Eevee, of whom I am a great fan.)

So, I'm working on a photo album app written in Python. I'm using Python because, well, I love it. I think it's the perfect language for web development; it's simple and very readable, and ample syntactic sugar and just the right mix of procedural and functional features almost completely eliminate tedious boilerplate. As a trivial example for those who have never used Python, consider:

db_tags = dict([(tag.name, tag) for tag in
                db.session.query(db.MetadataTag).filter_by(source = 'exif')])

What I'm doing here is grabbing all of the EXIF tags I know about out of my database with SQLAlchemy (also awesome, by the way), and creating a dictionary which maps each tag's name to the corresponding ORM object; I can find the ORM object simply with db_tags[tag_name], which lets me easily and efficiently insert new tag values as I read EXIF data from a photo. This isn't particularly difficult to do in any other language; as I said, it's a trivial example. But consider how much more verbose the code would be: PHP, which is still the most popular language for web applications despite being an incorrigible pile of garbage, doesn't have list comprehensions. I don't even want to think about how many lines of code this would take in Java; additionally, while I'm unfamiliar with them in general, I suspect Java ORMs require the use of generics to a painful extent (i.e. more than not at all). In Python it only takes one SLOC, without sacrificing any readability.[^1][^2]

So, Python's pretty great. But like all languages, it does have a few problems. The one I have had the most difficulty with in my current project is that third party library support is somewhat lacking in some important areas; specifically, there aren't many good options for reading and manipulating images. The most popular library seems to be the Python Imaging Library, PIL. It's easy enough to install and provides a passable set of core features, but doesn't go much beyond that. For instance, none of the various camera raw formats seem to be supported, which I consider a fairly important feature for my photo album; I'd very much like be able to upload original raws and have the application automatically convert them to JPEGs for me. Additionally, it doesn't provide much in the way of convenience methods for cropping and resizing images. ImageMagick has a very nice means of specifying cropping and resizing geometry with various aspect ratio-preserving (or not) behaviour, and I wanted to be able to leverage something similar for my app so that I could quickly prototype with various size options.

Given that ImageMagick provides most of the functionality I want that PIL lacks, it seemed like a good option, but it's a C library and Python doesn't yet have mature bindings for it. It's getting there, though! I decided to go with Wand, which is a fork of python-magickwand. It has a nice high-level API, but also provides direct access to most of the low-level MagickWand APIs, making it fairly flexible. When I first found it, it did not have support for the nice geometry specifications I wanted to use, but being an open source project I was able to patch that in. Version 0.2.2 will include my patch, which provides the Image.transform method for easy cropping and resizing. Eevee is also working on a Python imaging library, which is promising, but it's still a work in progress.

Unfortunately Wand doesn't yet support retrieving EXIF, IPTC or XMP metadata from images, which is another requirement for me. ImageMagick does support EXIF at least, and that's on the roadmap for Wand, but implementing a high-level interface for EXIF tags is a bigger task than I have time for at the moment. That meant that I had to turn to another library to retrieve metadata. pyexiv2, a Python binding for the exiv2 C++ library, initially looked promising. Unfortunately I discovered that it's not installable via any Python package manager; it must be built from source (via a build system I've never even heard of, and I'm familiar with more than a few build systems), unless a binary is provided for your platform by the project maintainers (i.e. you are on Debian, Ubuntu, Fedora or Windows).

My project already has more dependencies than I'm entirely comfortable with; one of the reasons that Python only has modest adoption as a web application platform despite being a great language for the task is that it doesn't have a kitchen sink-style standard library, and third-party libraries require separate installation. In general I consider that a good thing; I'd rather have a well-designed standard library than a gigantic mishmash of inconsistently named global functions. However, it does make deployment a little more complex. So far I've managed to stick to libraries which can be installed in one step using well-known package managers, so the setup procedure for my app should not be too arduous. Adding pyexiv2 as a dependency and thereby including an entirely different build system in the process would likely have complicated the installation procedure to the point that no actual ordinary humans would ever use my application. That's something I want to avoid, if possible.

~~For now I have settled for exif-py.~~ It only handles EXIF tags, not IPTC or XMP, which is disappointing because I really wanted to support multiple metadata formats. It's a pure Python library in a single file, though, so I can just include with my distribution. It's easy to use and gets the job done, so it will have to do for now. I'm hoping that pyexiv2 will show up in pip (a Python package manager) at some point, though. [Updated September 16, 2012: I've discovered that exif-py is broken in at least a couple significant ways, so I'm kind of back at square one for handling metadata. exif-py seems to be unable to detect EXIF data in some files that have it. Additionally, the strings that it translates orientation codes to are not consistent with each other, so they're basically wrong. I've informed the maintainers of these issues, but I don't have time to fix them myself.]

The process of trying to find a good imaging package for Python and ultimately settling for two separate libraries which still don't quite do everything I want has made it clear to me that third party library support for Python still has a way to go. Hopefully the situation will improve as adoption of the language increases. I'm happy to do my part, both through contributions to open source libraries such as Wand, and by building apps using Python to bolster the ecosystem. I hope that by demonstrating what's possible with Python as a web application platform, I can help convince new web developers to build their applications using Python as well... which provides yet more reason for me to be diligent about finishing my photo album app!

[^1]: Okay, the dict function is perhaps not the most explicit. Check the docs once, though, and it's very easy to understand. [^2]: Ruby would doubtless be similarly concise, and it seems like a nice language in my limited experience, but I find its syntax and distinction between "symbols" and strings to not be to my taste.

strlen without conditionals

2012-09-19T07:21:00Z

I'm not usually much enamoured with interview-style programming puzzles because I find that a lot of them are actually more akin to math problems, trivial to implement once you figure out the salient mathematical property. I think I have a decent intuition for math, and I certainly took enough math courses in high school and college to give me a solid foundation in the fundamentals of algebra, geometry, statistics, calculus, etc., but I'm not confident enough in my math skills to be entirely comfortable being judged by my ability to exercise them.

There are some programming puzzles I really enjoy, though. This evening I happened across one such puzzle, via Eevee's Twitter: "implement a strlen() function in C that, when compiled, would not contain any conditional branches." (The page contains solutions, so don't read the orange text if you want to try this yourself.) This is exactly the kind of puzzle I like; it's fun to think about, reasonably challenging, and requires knowledge of language features combined with creative thinking.

My solution is below, but I'd recommend that you go give this a try yourself before you read on.

Alright, ready? Here's my solution:

#include <stdio.h>

int mystrlen(char *str, int count);
int mystrlenret(char *str, int count);

int (*funcs[2])(char *, int) = {mystrlen, mystrlenret};

int mystrlen(char *str, int count)
{
  return funcs[(int)(!*str)](++str, ++count);
}

int mystrlenret(char *str, int count)
{
  return count - 1;
}

int main(int argc, char *argv[])
{
  printf("%i\n", mystrlen(argv[argc - 1], 0));
}

As you can see, the solution is actually very simple, but there are a couple of tricks necessary to make it work. First of all, you obviously can't use a loop to count the characters because a loop requires a conditional jump to either terminate or continue. You could still use a loop-like construct such as goto, but that only reframes the problem; you still need to make a decision about when to jump out.

So, the problem can basically be reduced to this: how can you make a decision without using a conditional? Well, you can index into an array of possibilities using some property that you can derive from your input. In this case, the most important property is that the characters can be grouped into two sets: terminating characters (i.e. '\0') and non-terminating characters (all others). On that basis, our array of possibilites can consist of two function pointers, one which continues recursively and the other which terminates.

The only problem remaining is how to group the characters, again without using conditionals. My first thought here was actually to divide the character by itself, in which case all non-terminating characters would be 1, and the null terminator would be... oh, a divide-by-zero error. I quickly dismissed that thought and realized that binary not is a simpler way to map the characters. However, Eevee came up with what I think is a very clever and quite unorthodox solution involving division by handling the SIGFPE generated by the division by zero and using the handler to change an unconditional jump address.

I'd recommend also checking out the solutions provided in the original post; all three of them are a bit more elegant than mine, if perhaps a bit more difficult to understand. (In particular, the count parameter that I use is superfluous, although I don't consider the difference particularly important since it can easily be hidden in my solution using a macro or helper function.)

Since my solution isn't significantly different from those presented in the original post, I wanted to go a bit further and take a look at the performance implications of not using conditionals. Yeah, yeah, premature optimization is the root of all evil and all that—I'm just doing this out of curiosity; regardless of the results, I would never go this far out of my way to avoid a conditional jump in a real program, nor would I recommend doing so to anyone else. I decided to compare my solution above against a naïve implementation with a while loop; comparing it against strlen() from string.h would be pretty meaningless because it would be linked against a precompiled library which, for all I know, could be hand-optimized. So here's the code I'm using instead:

int mystrlen(char *c)
{
  int len = 0;
  while(*c++ != 0) len++;
  return len;
}

And here's how I'm timing the functions...

struct timeval start;
struct timeval end;
volatile int n;

gettimeofday(&start, NULL);
for(int i = 0; i < 100000; ++i)
  n = mystrlen(argv[argc - 1]);
gettimeofday(&end, NULL);

printf("Elapsed: %ld sec %ld usec\n",
       (long) end.tv_sec - start.tv_sec,
       (long) end.tv_usec - start.tv_usec);

I'm assigning the result to a volatile int to ensure that even when the code is optimized, the compiler won't completely optimize out the function call. So, first, predictions: anyone who has taken a computer architecture course will tell you that conditional jumps can be very expensive because of the recovery the CPU has to do if the branch is mispredicted. However, branch predictors typically have very high accuracy, and there's also a significant overhead involved in putting the parameters and return address on the stack when a function is called. Therefore, I predict that in that with no compiler optimization, the while loop will be faster.

...And, it is. Running each function in a loop 100,000 times on a 23-character string, the average over five trials was 28,699 µsec for the non-conditional version and 9,012 µsec for the while loop. Using a fixed string will cause the branch predictor to have near-perfect accuracy in the while loop version, so there might be a slight difference if I used a large array of strings of randomized lengths, but I doubt it would be significant.

But what if we let the compiler (llvm-gcc in this case) optimize it? I don't know enough about compiler optimization in general or llvm-gcc in particular to predict what will happen here. The non-conditional strlen() is tail recursive so the compiler will optimize out the extra function calls at -O2 and above, but what will be optimized beyond that I really don't know.

As it turns out, compiling at -O3 and running 100,000 times on the same 23-character string, the non-conditional function took an average of 11,182 µsec over five trials. That's a significant improvement, but it's still slower than the unoptimized while loop, so it obviously won't beat that. And indeed, the optimized while loop takes only 3,072 µsec.

Switching to Octopress

2012-09-24T06:22:00Z

I've been using Wordpress on various sites that I've maintained for probably the better part of a decade now. By most metrics, it's still very good blogging software; I'm not aware of any self-hostable alternative that can match Wordpress in terms of the union of ease of use, feature set and flexibility that it provides. That said, lately I've been feeling that perhaps Wordpress is no longer the right choice for my specific needs. These are my problems with it:

It's not very good for sharing source code. There are a multitude of plugins available for syntax highlighting; the best of them seems to be Alex Gorbatchev's Syntax Highlighter, which is used by Wordpress.com and is also what I was using on my blog. But I've had problems with the Wordpress converting some of the characters in my code to HTML entities (e.g. & for & ) which are then printed literally by the plugin.
I find that more often than not Wordpress' visual editor is insufficient for the formatting I want in my posts. That forces me to use the raw HTML editor. That would be fine, except that Wordpress has a strange sort of off-spec way of storing post HTML. Paragraph and line break tags are omitted in the editor and the database, and are added when the page is actually rendered to the client. This is okay, I guess, and I think Wordpress is smart enough to not insert extra paragraph tags if you do wrap your own paragraphs with them. Still, it leads to some unpredictability with regard to how a post with hand-written HTML will actually be rendered.
A similar but worse problem occurs when you try to insert extra line breaks. Even if you're using the raw HTML editor to insert <br /> tags, Wordpress will just get rid of them unless you insert additional non-breaking spaces ( ) so that it thinks the lines are non-empty. But! If you switch back to the visual editor for any reason, even your  <br /> lines will be wiped out. Even for "average" users I can't think of any situation in which this "feature" would be desirable. If I put a bunch of extra newlines into something I'm writing, it is because I want exactly that many newlines. I tend to not have much patience for software that thinks it's smarter than I am; that is why I still write most of my code in Vim and use hand-written makefiles in an age of IDEs.
Never-ending updates. It seems like every time I log in there's a new update to download. Wordpress has updates down to a one-click install process, so this wouldn't be much of a problem except for the fact that every time I have to update Wordpress warns me that I should back up my database first. Why? Is there a significant possibility that the update might break everything? It's probably just a COA, and I back up my database daily anyway, but this whole aspect of Wordpress makes me uneasy and I don't want to have to deal with it any more. Even though I'm covered for backups, recovering from a botched update would undoubtedly be a nightmare.
Vulnerabilities. One of the big reasons that Wordpress is updated so frequently is that it has a long history of security vulnerabilities, and more are still being discovered and patched. This also means that if I don't log in frequently enough to download new security patches, I'm at risk of having my site compromised, loaded up with malware and blacklisted by Google.
It's written in PHP. PHP sucks. (I can't link to that enough.) I don't want to be dogmatic about technology choices; good software is good software regardless of what language it's written in, and my primary desire is always to use whatever is best for the task at hand. But lately I've been pretty vocal, both here and in person with friends and colleagues, about how bad PHP is. Continuing to rely on it undermines my credibility on the issue and makes me a bit of a hypocrite.

And while this might seem trivial or pedantic, it's actually one of the root causes of almost all of the issues above; the weird HTML handling, inappropriate entity conversion, and security issues are all due, at least in part, to bad design patterns popularized and perpetuated by PHP. Wordpress still uses raw non-parameterized SQL queries; blogging software built on a modern web framework with a proper ORM would not be plagued by SQL injection exploits as Wordpress has been historically and probably continues to be.
Wordpress supports themes so that you can customize the apperance of your site, but editing them kind of sucks, because they're all written in PHP instead of a proper templating language. Years ago I actually wrote a lot of PHP, but I still find Wordpress templates pretty ugly and difficult to understand; they seem more verbose than should really be necessary for what they accomplish, and have, on average, more <?php ?> sections than a keyboard has keys. (That analogy sucked. Sorry, it's late. What I'm trying to say is that there are a lot of them.)

So recently I got it into my head to switch over to Octopress. Octopress bills itself as "a blogging framework for hackers," which is an attractive description given my predilections. It uses Jekyll to generate static pages from a set of templates and content written in Markdown.[^markdown] It is, without a doubt, harder to use than to Wordpress, but it has the following advantages:

Rendering static pages once and serving dynamic content via AJAX is much more efficient than rendering entire pages dynamically for each visitor. This isn't really a concern for me because this is a very low-traffic site, and I have an entire quad-core VPS doing basically nothing besides serving this site. That said, it's nice to know that when I start doing other stuff on my VPS, I won't be contending for cycles with PHP and MySQL, so the site will always be very responsive. Plus, wow does the site load fast now (once you have that huge header image cached, anyway. I'll fix that soon.) Also not a huge concern for a blog, but I didn't expect the difference to even be noticeable, and it definitely is.
Static pages significantly decrease the attack surface of the site; there is no database, so no SQL injection to worry about, and no scripts are being executed, so PHP vulnerabilities aren't a concern.
Static pages means I get a nicely organized, easily navigable directory structure instead of God-awful rewrite rules.
The default Octopress theme is very well designed and well-suited to customization. I was able to modify it to make it look almost exactly like my old custom Wordpress theme did, while retaining the fully fluid layout of the Octopress theme—meaning that the layout will resize to look beautiful at any screen resolution, including on mobile devices.
Octopress has pretty nice built in Github and Twitter widgets.
Comments are handled by Disqus, which provides a nicer comment system than Wordpress' built in comments. I was using Disqus anyway via a plugin, but now I don't have to worry about keeping the Disqus comments synced with the Wordpress database. Incidentally, Disqus provides a very nice system to migrate comments to new URLs by uploading a CSV file mapping old URLs to new URLs. They've really thought of everything, so props to them for providing a great service.
All of the blog content is stored on my local machine, which means I can write posts in a disconnected environment using Vim, version control them with Git, and then push them to my server when they're finished. I have a nice deployment setup where everything is stored in a Git repository on the server and a post-commit hook copies the updated content to the web root for the blog subdomain. All I have to do is git push and the blog is instantly updated.
Since Wordpress uses hand-written SQL queries, it is tightly coupled with MySQL and doesn't support any other database. PHP is similarly tied to Apache; while it's possible to run it with other servers, it's not particularly easy. By ditching Wordpress, I can also get rid of Apache, PHP and MySQL. This is nice since I was planning to use nginx and Postgres for my photo album project anyway.

Those are, give or take a few minor considerations, my reasons for switching blogging platforms. I'd be remiss to not mention that Octopress does have some noteworthy disadvantages even for "power users" to whom the above advantages might sound attractive. This post does a good job cataloguing a few of them. On the balance, though, I feel like it's a good choice for me, and although the switch took me considerably longer than I expected (the better part of an weekend where I'd expected an evening or so of work), I'm pretty happy that I did it.

In an upcoming post, I'll discuss the actual process of reconfiguring my server and importing all of my content (which is what I had planned to do before this turned into a 1,600 word diatribe against Wordpress.) For now, if I've inspired you and you're thinking about switching your own Wordpress site to Octopress, you can take a look at the ETL I wrote to ease the import process.

[^markdown]: Did you know that Markdown was created by John Gruber? Until very recently, I didn't. And here I thought all he did was play apologist for Apple.

Photo App Progress Update

2012-11-15T06:58:00Z

In September I wrote about my intention to develop a web application to share my photos. I made good progress on it throughout September and early October, but for various reasons I haven't been able to work on it much in the past month or so. Last weekend, though, I was able to solve some blocking issues which were preventing me from doing an alpha release. Those being dealt with, I've now got a few of my albums up at photos.mlindgren.ca; take a look and leave a comment if you have any feedback.

I haven't released the source code yet as there's still a great deal of work to be done, and I'm generally of the belief that dumping a bunch of unfinished, messy code on Github with the hope that the community will sort it out is of little benefit to anyone. My goal at this point is to do a beta release in a two to three months or so and release the code at that point. That timeline is anything but firm, though. This project is turning out to be much more work than I expected (as projects are wont to do), and I expect to be fairly busy over the next couple months.

Anyway, with the alpha version up and running, this seems like a good opportunity to reflect on some of the issues I've faced that complicated the project, and on what remains to be done.

Wand and the OOM Killer

I mentioned in my post on Python imaging libraries that I'm using Wand to resize and transform uploaded photos. It's very convenient and easy to use, and from a maintainability and ease of use perspective I much prefer using Python bindings for ImageMagick to shelling out to the ImageMagick executable. One of my goals for this project is that other people should be able to install it on their own servers with relative ease, and I suspect relying on ImageMagick executables for image processing would lead to cross-platform and configuration issues.

But, Wand is not yet a mature library and it has some significant issues of its own. First and foremost, when I started using it it had some major memory leaks. I found and fixed more than 10 leaks and submitted a pull request which was merged into the main repository—however, there hasn't yet been an official release including my fixes yet, so pip installs and downloads from the Wand website will still include those leaks.

I'm not sure if I got all of the leaks or if more still remain. The ones I fixed were discovered mostly through ad-hoc testing and reading the code. I've been running my app for several days now and the memory consumption when it's idle has stayed flat, so I believe I've removed at least the most common leaks, but I haven't had a chance to sit down with Valgrind yet and run the Wand unit tests through it to make sure that I got all of the leaks. I'll have to do that before I do any sort of public release of my software; I don't want to rely on a leaky library.

Beyond the leaks, I ran into another memory issue which proved problematic: resizing photographs can use a great deal of memory, which is extremely limited on a reasonably-priced VPS. I have limited knowledge about image processing in general and ImageMagick in particular, but I'm not aware of any resizing algorithms which can operate directly on compressed image data, and if any such algorithms exist, ImageMagick doesn't use them. That being the case, to resize an image, the entire image has to be loaded into memory decompressed. For an 8 megapixel photograph at 24 bits per pixel, that means allocating approximately 183MB of memory.

On a 512MB VPS, this means that trying to resize three images concurrently—or even sequentially in some cases, since Python isn't guaranteed to immediately relinquish allocated memory to the operating system when objects resident in that memory are "deleted"—practically guarantees that the process will be killed by the OOM killer.

This goes hand in hand with another complication, which is that resizing images is too slow to be done in the same server thread that's handling requests and generating responses. For each image that's uploaded, I save several different sizes for different purposes, and due to the aforementioned memory constraints I can't even resize them concurrently. Resizing each image several times in the responding thread drastically reduces throughput for the client since it incurs a significant wait time for the server to process each image after it's uploaded.

Enter Subprocess

So here's the solution I've come up with for the time being, and I'm very unsure whether or not it's actually a good solution, so please do let me know in the comments if there's a better way to do this. When a new image is uploaded, the responding thread saves the image in the appropriate directory and adds it to the database, but does not resize it. For each of the sizes that the image needs to be resized to, it adds a task to a synchronous queue. The queue is consumed by a different thread which runs throughout the lifetime of the application. That thread maintains a (thread-local) queue of subprocesses with a configurable maximum length. When a new resizing task is consumed, if there is room in the subprocess queue, a new subprocess is spawned to resize the image. Otherwise, the thread joins the subprocess at the front of the queue so as to block until there's room to spawn a new subprocess. The exit code of completed processes is checked to ensure that the resize was successful; in case of failure, the task is re-added to the back of the task queue.

This guarantees that the application server itself will never be killed by the OOM killer since it uses minimal memory. It never has to load images into memory; all of that is done in the resizing subprocesses. By configuring the maximum number of concurrent subprocesses, one can scale this solution according to available resources: if you have plenty of RAM, you can resize many images concurrently and get through the queue faster, or if you have very little, you can limit the queue to one or two processes to minimize the chance that anything will be OOM killed.

The subprocess spawning notwithstanding, this is a pretty standard task queue model, so you might be wondering why I didn't use something like Celery in conjunction with a real message queue. It comes down to minimizing the number of external dependencies and maximizing ease of use. My project already depends on a number of libraries which will have to be installed by users. To the greatest extent that I can do so without compromising on features, I want to avoid complicating the installation further by adding more dependencies. Celery is particularly difficult to install as it requires installing and configuring a broker and managing processes for both the broker and Celery itself. Were I writing a large-scale service and maintaining a single backend for hundreds of users, it would probably be the right choice, but as I am writing software for users to install on their own servers, it is probably not.

This method works fairly well and it's the best I've come up with so far, but something about it makes it feel more like a hack than a well thought out and robust solution. I also wonder if it defeats the purpose of using Wand in the first place; I use it for very little other than resizing, and since I'm going to the trouble of spawning subprocesses to do that, perhaps it would actually be better to just directly invoke the ImageMagick binaries. As mentioned above, I have concerns about what impact that would have in different environments, but I haven't really validated those concerns; they're just hunches.

The Road Ahead

So I've got a workable, if not perfect, solution to one of the biggest problems I encountered... but as I mentioned previously, there's still much work to be done. Here are a few (but not all) of the things I'd like to do before I release anything publicly:

Lots of UI work. Right now the app works very poorly on phones and iPads. I also need to figure out how to make certain features more discoverable, and although I'm happy with how albums look right now, there are still some extra touches that I want to add which I haven't yet been able to (because CSS and HTML suck.)
Better administrative tools. Uploading works great, but I have very barebones interfaces for editing albums right now. There is currently no means of rotating a photo from the editing interface (although doing so should almost never be necessary since the app automatically rotates images which have EXIF orientation data.)
Mirroring and proxying to cloud storage services such as Azure and S3. VPS resources are very expensive; extra disk space on some popular hosts runs around a dollar per gigabyte per month. That's ten times the cost of disk space on the cloud storage services I've looked at. Cloud-hosted images might also improve load speeds for visitors.
Comments. This one is pretty big, obviously, but I haven't added it yet because I can't decide on how best to handle it. I'd like to avoid writing my own comment system and requiring potential commenters to sign up for yet another account. (I'd use Persona for commenter identification and authentication just as I do for the admin login, but very few people currently have Persona accounts or even know what it is.) I like Disqus, but I'm not sure it would integrate well with the minimalist interface I've designed.
Geotagging and maps. I'm already reading GPS data from EXIF tags where available, but now I need to do something with it. I'd like to add the ability to manually add locations to photos and albums, and to have a map that shows where photos were taken.

Those five items represent maybe a quarter to a third of the work I still have planned, so to repeat myself again, there's a lot of work to do. But I'm very happy with the progress I've made so far, and I feel confident that I can get this finished. It's just a matter of when.

The Subaru Saga

2013-06-10T05:22:00Z

Or "buying used cars is the worst and I will never do it ever again."

I recently purchased a new-to-me 2011 Subaru WRX STI sedan. I had been thinking about buying a new car for a long time—since I graduated, more or less. Until I bought my Subaru, I owned an ageing Honda Civic (1997, to be precise). While it was a very capable and extremely reliable car throughout the many years I owned it for, it was old enough that it lacked some of the conveniences one expects in a car these days, such as power locks and windows, and most importantly, air conditioning. The lack of air conditioning in particular made it less than ideal for summer road trips, and more importantly, it was in need of some moderately expensive maintenance, and it didn't seem to make sense to me to spend much money on it given that the overall value of the car was fairly low.

I didn't go on many test drives before I purchased my Subaru, but I didn't feel the need to; I had already spent months watching reviews, reading forums, comparing specs and window shopping on AutoTrader. I knew I wanted a sporty car with plenty of horsepower for under $40,000, and all-wheel drive was a must as I go to the mountains nearly every weekend in the winter and almost as often in the summer. Besides the STI, the Evo, MazdaSpeed3, and the Golf R were the best contenders, but each had significant drawbacks with regard to my personal preferences. The Evo has a notoriously lacklustre interior (not that the STI is a lot better, to be fair); I don't like the way the MazdaSpeed3 looks, and the Golf comes with the reputation of higher maintenance costs for German cars.

I also really like the Subaru community. /r/subaru is the largest manufacturer-specific subreddit, and in addition to forums like NASIOC, it comprises a vibrant community of enthusiastic Subaru owners. Of course other brands have significant followings as well—particularly VW—but my impression of the Subaru community is that it's less elitist and, in contrast to the others, puts as much emphasis on what one can do with the cars as on the cars themselves. To "go Subaruing" is a popular term for going out into the mud or the snow or the sand or the bush and getting the car wet and dirty, and having a ton of fun in the process.

The STI, then, was the clear winner in my eyes. Truth be told, a WRX probably would have been more than sufficient for my needs, but I generally prefer to put down the extra money and be sure that I'm getting the best product. And hey, one only has so long to exercise the opportunity to be young and irresponsible!

Finding My Car

So, finally I decided to bite the bullet, and I found a used STI that looked really nice on Craigslist. On March 9th, 2013, I went with a friend to Walker's Renton Subaru to look at the vehicle and take it for a test drive. Walker's is sort of two separate dealerships on the same lot; there's the Subaru dealership and there's also Walker's Renton Mazda. Obviously they're both under the same ownership, and it seems that salespeople on one lot can sell cars on the other lot as well; the car I went to look at was listed as being at the Subaru dealership, but I ultimately ended up dealing primarily with the Mazda dealership because that is where the salesman I first contacted worked.

Ken Palmer was the salesperson I dealt with. Both he and the dealership itself have very high ratings on the site that link points to (DealerRater.com), which is surprising to me because dealing with them (particularly the dealership; I have less animosity for Ken) has been by far the worst experience I've ever had as a consumer. The reason for that will become clear shortly, but bear in mind that even before I discovered the major issue which prompted this post, I would not have said that my buying experience with Walker's was good by any means. Incidentally, their Yelp ratings are considerably lower, although there aren't as many reviews on Yelp.

We went to the Mazda dealership and I took a look at the car. It looked great; the paint job was (and is) in great shape for the most part, although there were a number of small nicks on the rear doors where the body flares out into the wide wheel arches, as well as some on the front bumper pieces where the fog lights would go (this car doesn't have fog lights). I didn't really notice those at the time, but they're minor and I doubt they would have influenced my purchasing decision anyway. There was also some minor cosmetic damage to the interior: a bit of chipping on the painted bezel around the satellite navigation unit and centre vents, a white mark on the driver's side between the centre console and the steering wheel, another on the glove box, and a minor nick in the steering wheel leather. There is some minor curb rash on wheels which I somehow failed to notice before I bought the car, but it's not noticeable from a distance of more than a few feet. Overall the car appeared to be in excellent condition.

We went for a test drive and I really liked the way it handled. Traffic was moderately heavy and I didn't get much of a chance to open up the taps and really test our the car's power on the route we took. Truth be told, I wasn't even doing a particularly good job driving it, as I'd driven a manual car a grand total of two times before that test drive, including once on the way to the dealership—I drove my friend's car there to get some additional practice[^manual]. Even so, I could tell then that I would grow to love the car more and more as I learned how to drive it better, and in retrospect I was completely correct about that.

During the test drive I noticed that the exhaust was very loud—not irritatingly so, and I loved the way it sounded, but I thought it couldn't possibly be stock so I asked Ken. He confirmed that it was stock, which at the time I naïvely accepted. I later found out that this was not true. I believe that the downpipe, midpipe and catalytic converter are stock, but the tail pipe seems to have been replaced with a Nameless Performance AxelBack Muffler Delete. That's strike number one against Ken. I like the exhaust, and I don't mind that it's aftermarket, but he should have done his research on the vehicle before trying to sell it to me. If he wasn't sure whether or not the exhaust was stock, he could have answered honestly and we could have looked it up to find the correct answer. It irritates me that I was given false information, and while everyone makes mistakes, given what I now know I can't help but ascribe this to him valuing presentation over honesty.

This Is Where It Gets Unpleasant

We got back to the dealership and I decided that I liked the car enough that I was prepared to sit down and negotiate on the price. The list price on the Craigslist ad I'd responded to was around $32,000, but I told Ken that I was not prepared to pay that much and that I was looking for $29,000. He said there was no way he could go that low, as the price they were offering was already low in his opinion. For the record, the Blue Book value for the car as I bought it is currently $32,900 when purchased through a dealership, but that assumes excellent condition, which I don't think the car quite qualifies for given the minor cosmetic damage to the interior and the fact that some of the tires are at less than 75% tread depth. The private party Blue Book value given those facts is currently $30,000. Ultimately I paid $31,000, which given dealership markup seemed reasonable at the time.

I spent quite a long time negotiating with Ken, so I want to take some time to describe his personality and style as a salesperson. I can't help but view him as a reasonably nice guy. He seemed to be friendly and in possession of a good sense of humour, and was very patient with my less than stellar manual driving. That said, he was subtly pushing me to spend more than I wanted to, and he gave me misinformation and made statements of dubious veracity in several instances (I hesitate to say "lied"). For instance, when I told him I didn't need a new car and that I could just as easily walk away, he repeatedly emphasized that the STI was "a hot item" and that if I left and came back tomorrow, it might not still be on the lot. This is kind of true under the right circumstances, but I had been looking at Craigslist and AutoTrader for months at this point, and I knew that this car in particular had been sitting on their lot for at least a month—maybe more like two months. That fact was also ascertainable from some of the paperwork Ken gave me on the vehicle, so it was a pretty poor attempt to mislead me.

I also happened to notice that the price on Craigslist had dropped something like $50 between the night before and the day I was there, so I called him on that as well. He changed his story at that point and told me how the STI is a performance car for a particular type of buyer, and not just anyone would be looking for one, to which I replied: yeah, that's my point! It's a sought-after car but it's also not something just anyone is going to drop in and buy, so I know you can't sell it as quickly as you say you can.

While Ken repeatedly insisted that he was on my side and wanted to work with me to get me the best deal possible, and that he wasn't a typical pushy sales guy, I'd also describe his tactics as emotionally manipulative. During the negotiation whenever I requested something and he went and talked to his manager and came back with a not-quite-good-enough offer, he'd walk back to the desk languidly with his head hung in faux-exhaustion, and say with an exasperated sigh that he just couldn't make it work or something along those lines. The friend I'd come with told me at one point that he was using tricks specifically tailored to work on my personality type, which I think is probably an accurate assessment because even though I knew what he was doing, I think there was a subconscious element of feeling bad for the guy and wanting to work with him for that reason.

After the initial negotiation we ended up agreeing on $30,800, conditional on me being able to take the car to an independent mechanic for an inspection. I am about as mechanically adept as Jeremy Clarkson, but without the benefit of his encyclopedic knowledge of cars. For that reason, I hadn't even bothered to look under the hood, and it was important to me to get the car checked out independently to verify its condition.

The process got a little strange at this point. I wanted to write above that we verbally agreed on $30,800, but that's not quite accurate. Ken and I would agree on something and he'd write down the offer on paper and make a crude "sign here" line and ask me to initial it. Even though I knew his handwritten offer couldn't possibly be legally binding, and he assured me of as much, I was a bit hesitant to do this as it just gave me a bad vibe. But he refused to proceed unless I'd initial his paper, so I did. He took that to his manager and finally we were able to come to an agreement.

I got a temporary permit to drive the car and took it to Community Automotive Service in Burien, which, strangely, seems to be one of very few mechanics in the area south of Seattle. I paid around $80 for the mechanic there to do a quick examination of the car. He pulled the ODBII codes, checked the brakes, fluid levels, etc, and took the car for a short test drive, among other things. When I came back to pick it up he told me that overall it was in great condition, but it seemed that the 30,000 mile maintenance had never been done. That was pretty alarming to me as the car was at approximately 33,500 miles at that point, meaning it was 3,500 miles overdue for some fairly important maintenance. I expected the dealership to have done that maintenance as part of the trade in process and factored the cost into the price they paid for the car; as far as I'm concerned, one shouldn't even be taking the car on test drives when it's 3,500 miles overdue for maintenance. I got antsy when my Civic was a couple hundred kilometres overdue for an oil change, and an STI has considerably more power with which to rip itself apart if it's not kept in good condition. In retrospect I think that alone should have been enough to make me go look elsewhere, but I had already spent probably 4-5 hours at this point between the test drive and the negotiating. I didn't want all of that time to be for nothing.

To be specific, the mechanic at Community Automotive said that a number of the car's fluids needed to be flushed, that the air and fuel filters needed to be replaced, that the brake pads would need to be replaced within a few months, and that it would need new tires in the not-too-distant future. His statement about the tires turned out to be questionable as I later found out that they do all still have more than 50% tread depth remaining, but the rest of his findings seem to have been accurate. He also gave me a printout of the ODBII history. There were a few error codes (abnormal clutch signal, power supply voltage failure, excessive variation from steering position sensor, and right-hand curtain airbag control module failure), but they were from the past and not reproducible so he said he didn't feel that they were anything to worry about. Overall, he told me that he liked the car and thought it was in great shape except for the maintenance that needed to be done. He thought the price I'd agreed on was a bit high, but I don't think he was aware of how high the Blue Book value for the vehicle was, because he seemed surprised when I told him that as well.

I went back to the dealership and began hours more of painful negotiation. Given the issues that the mechanic had identified I was not willing to pay $30,800 unless the dealership would fix some of them. They were very reluctant to budge on this and in the end I gave in more than I probably should have. I talked to both Ken and his manager, and both were incredulous about the report that the mechanic had given me; they didn't feel that most of the items were real issues. I very clearly recall Ken's manager, Jan Rac, giving me a spiel about how Walker's Renton Subaru had well-qualified technicians, how they had a reputation to uphold, and how they had the only "STI tech" in the area and that nobody else is allowed to work on STIs. I thought at the time that much of this was probably a huge load of bullshit, and in retrospect it almost certainly is. As far as I can tell there is no such thing as an "STI tech." Nissan has something like that for GT-Rs, but that's a $100,000 car. I cannot find any reference, anywhere, to any such thing as an "STI tech." I do not like being lied to.

Ken, who had previously been absolutely fine with me taking the car to an independent mechanic and was convinced that I wouldn't find any issues, exasperatedly told me how independent mechanics never say anything good about cars; they're only looking for problems (yes, that's exactly the point) and implied that the guy I'd taken it to was just looking to cash in... even after my wife and I[^wife] repeatedly told him that we had no affiliation with that mechanic and that the mechanic knew that we would not be taking the car back to him for repairs. We negotiated more and finally agreed to them replacing the brake pads and doing some of the fluid flushes in exchange for bumping the sale price up to $31,000 (because the brake pad replacement alone cost almost $600 since the STI uses Brembo pads). It terms of the maintenance they offered, it wasn't that great a deal for me, and as I mentioned earlier, in retrospect I really should have just walked at this point. In fact I almost did over a miscommunication about the brake pad replacement: I wanted all of them replaced, but Ken had assumed I meant only one pair and had neglected to ask which pair. I had been at the dealership literally all day at this point; I was there for at least ten hours, and I really wanted to close the deal just so it wouldn't be a huge waste of time. Nevertheless I told them it was all of the brake pads or I was leaving, and after some pageantry about confirming the offer with the sales manager (who had already left at this point), we finally reached an agreement.

I added on some extras as well, including 3M protection for the front of the vehicle for around $600, and an extended warranty for around $2000. Including taxes, the total cost was around $38,000.

Foreshadowing

Here's where things get really shady. I left the car at the dealership to have the service done so I wouldn't have to drive back to Renton for it (which is a 30 minute to one hour drive from where I live in Redmond, depending on traffic). When I came back to pick up the car, I went to the service desk to get the keys and request a copy of the service history and schedule so that I would have it for my records (and to verify that they actually did the work). The service advisor I talked to, whose name I think is Kay, asked me if I was aware that the car had been in an accident and had had about $4,000 worth of bodywork done. My heart leapt into my throat at this revelation, and I said that no, I did not know that, and in fact I had been told that the car had not been in any accidents. She got an "oh shit" kind of look on her face and then said she'd double check, and after a bit of typing told me that actually she'd just typed the VIN in wrong, and that this car was fine.

This is what those of a literary predilection might call foreshadowing.

I was too relieved to really think too much of it at the time, and anyway the contract had already been signed at this point, but knowing what I do now this is obviously hugely suspicious. Kay strikes me as a nice person, and I mean genuinely nice rather than in a salesperson sort of way where niceness is necessitated as a means of being persuasive. I really want to believe that she actually did just type in the wrong VIN and wasn't lying to me. But... well, let's do some math.

A VIN is 17 alphanumeric characters long, and each character has certain restrictions on what it can be. That means that there are 6.5273512 × 10²⁵ possible VINs. That is a very, very large number. VINs have been in use since 1954, and the number of cars in the world just surpassed one billion in 2011, but I think the source I got that from was only referring to cars currently on the road. So I think it's generous to assume that there have been 10 billion cars with VINs since 1954. Now, I was never very good at statistics so this calculation is probably inaccurate for a host of reasons, but by my math that means that if you type an arbitrary wrong digit when entering a VIN, you have a 1.532015 × 10^-16 chance of getting a vehicle that actually exists. By way of comparison, that is 8 orders of magnitude more unlikely than getting six of six numbers on the 6/49[^lottery].

I don't know all the details of how VINs really work; they're definitely not uniformly distributed, and as I said, I was never very good at statistics anyway. So let's make this a little simpler. The last six digits of a VIN are sequential, and to the best of my knowledge, if all the other digits are fixed then then all of the possibilities afforded by those last six digits should refer to the same model of car, for the same model year, made by the same plant. In North America the last five digits must be numeric, but since STIs are manufactured in Japan, I think that gives us 33⁶ possibilities (because some characters are excluded), which is 1,291,467,969. So again, the possibility of typing a wrong digit in an arbitrary position and getting a car that actually exists is very low. But even assuming that she managed to do that, I'm fairly certain their service records only show cars that have actually come to that dealership for service.

So how the fuck could she have possibly happened upon the same car, that came to their dealership and got work done for an accident which probably would have cost a similar amount to fix to the one my car was in, by typing in the wrong VIN? If anyone has a plausible explanation for this please do let me know, but I just cannot believe that that really happened unless I have incredible bad luck.

As I mentioned earlier, I didn't think too much of what Kay said at the time. I got the car; the brakes had been replaced; things seemed fine. Ken told me that the difference with the new brakes was really noticeable and that he was glad they got that done for me. I couldn't help but think "Oh, you mean the same brake pads you and your manager were telling me didn't need to be replaced when I bought the car from you?"

The 3M service hadn't been done yet as it had to be scheduled in advance. I ended up waiting until the 8th of April to get that done. When I dropped the car off for that, I asked Kay to check if they had actually done the fluid flushes they were supposed to, as the service record I'd received from the last time I picked the car up only indicated the brake replacement. I hadn't noticed that at the time because my mind was on the supposed accident damage that she told me about. She checked and sure enough they had not done the fluid flushes, which they were contractually obligated to do. She said that they'd get it done with the 3M service, and indeed they did (or at least they gave me a service record indicating such—I don't really have the expertise to check myself). I wondered if they neglected to do this the first time on purpose, hoping that I just wouldn't notice. When I picked up the car the next time, Kay told me she was really glad that they had done the rear differential fluid flush, as the fluid had been very overdue for a change. I think the word she used to describe it was "pudding" or "sludge."

Recall that this is the maintenance they told me their world-class technicians hadn't felt it necessary to do. I'm sure you can understand that my attitude at this point was fuck these guys.

The Short-lived Honeymoon

Finally, I had the car and all the service was done, and I was overjoyed. It was a ton of fun to drive, and I quickly (in my opinion, anyway) became more adept at driving a manual. I bought floor mats and Rally Armor mudflaps, which I installed myself. My new car put a smile on my face every day; even the short commute to work was a joy.

But I noticed a couple problems. First I noticed that the heated seat bottom on the driver's side didn't work. The back worked, but the bottom didn't get warm at all. I was annoyed that I hadn't noticed it during the test drive, but it would be covered by the warranty, so I didn't consider it a big issue. Then, on the 27th of April, my wife and I went to central Washington to escape the rain and go hiking. In contrast to the Seattle area, it was very warm in central Washington that day, and the interior of the car was boiling from sitting in the sun when we got back from our first hike of the day. I turned on the air conditioning and... it was clearly not as cold as it should be. It was no colder than the regular air coming in from the exterior. This was even more bothersome than the heated seat issue, but again I figured that it could be fixed under warranty, so I didn't worry about it too much.

During the next week I also replaced the air filter (by myself, which I was pretty proud of, even though it's about the easiest thing to change on a car) and noticed that the air box had a large crack in it, and that the air intake duct which goes into the air box had been broken in half height-wise and shoddily glued back together with some kind of sealant. I still didn't think much of it at this point as I figured someone had damaged them while replacing the air filter previously.

Finally, I booked a service appointment for May 3rd to get the heated seat and air conditioning fixed at Eastside Subaru in Kirkland, which is a lot closer to Redmond than Renton is. I dropped the car off and they called me back later in the day to inform me that they verified both of the issues, but that while the heated seat problem could be fixed under warranty, the air conditioner couldn't because the condenser had been damaged in an accident.

Ugh.

I went back to Eastside Subaru and asked them to point out how they knew it was accident damage, since Walker's had told me the car had not been in an accident. The technician who serviced the car came out and pointed out several things to me. The tubes from the AC condenser were bent in such a way that it was obvious to him that there had been an impact, and they leaked badly. Moreover, the condenser itself was quite obviously bent. There are tool marks on the bolts that hold on both of the front quarter panels, indicating that they've been removed to be repaired or replaced. There's an AC recharge sticker on the inside of the hood which appears to have been glued on, not very well, so it's peeling off; I believe the technician also said that he thought the whole hood had likely been replaced. The power steering fluid reservoir attaches to the body of the vehicle in two places and one of those was cracked. The radiator support bar is also bent. It's slight enough that the untrained eye would probably not notice it, but the technician at Eastside told me, and I am very confident that these were his exact words[^legal], that Walker's "had to have known" that the vehicle was in an accident.

I drove back down to Renton and spent probably about half an hour yelling at various people. Ken tried to placate me in various ways; he told me that they didn't know and that they couldn't have known; he mentioned that "my guy" didn't notice the problem either; he tried to tell me that this wouldn't actually decrease the resale value of the car (bullshit) and that since it's not on the CarFax or title no one needs to know anyway. He made repeated use of euphemisms implying that maybe there wasn't actually any damage and the car hadn't actually been in an accident, even after I showed him the damaged parts. He seemed to be acting almost as though if I simply willed it hard enough I could bring about some alternate reality wherein my car was pristine and undamaged. (Believe me, I tried.) This, to me, emphasizes the problem with people like Ken and his manager and, as I'll soon describe, his manager's manager—that is to say, salespeople—they have an atypical understanding of the concepts of facts and truth and act as though reality is open to interpretation.

I told Ken that I refuse to lie to someone when I sell the car (and let this post be a testament to that, because it is now public knowledge, whether or not it ever shows up on the CarFax) and that I felt that I was probably out thousands of dollars in resale value, not to mention the fact that I have no idea what kind of mechanical issues this could result in down the road. My confidence in my extended warranty has also gone down the drain, because if anything fails and can be shown to have signs of accident damage, even if it's very minor, then it won't be covered.

I made it clear that I wanted them to take the car back and refund me the entire purchase price including the extended warranty, 3M treatment, and taxes I paid. That didn't even account for the hundreds of dollars I had already put into the car (mud flaps, floor mats, air filter, etc); I didn't care, I just wanted to cancel the transaction and be done with it. We argued for a while, but it was fairly late on Friday afternoon and he told me that there was no one at the dealership who could make that decision at that time. He said that he'd get the general manger to call me back the next day before noon. I asked him to tell the GM that I would be talking to an attorney if I didn't get the answer I wanted, and left. From this point until the beginning of June, I completely stopped driving the car, as I wanted to ensure the strength of my case by minimizing the number of miles I put on the car.

That Saturday I had plans to go hiking, and I didn't want to cancel them just for a phone call that probably wouldn't go anywhere, so I called the dealership early in the morning. Ken wasn't there yet, but I left a message for him saying that I may not be available to answer the phone but that he could leave me a message. Nobody called me on Saturday, and in fact I haven't spoken to Ken at all since that Friday, the 3rd of May. On Monday, the 7th of May, I called back and asked to speak to the general manager, James Capestany. I ended up having more or less the same conversation with him that I had with Ken. He was and is adamant that the dealership didn't know that the car was in an accident, and further, that it's not their responsibility to know. He pointed out that the salespeople do not have mechanical qualifications. I asked him why his technicians failed to thoroughly inspect a car which costs upwards of $30,000 when they took it as a trade-in. He gave me a completely unsatisfactory answer which I don't quite recall. I told him that I had read the sale contract, including the provision for trade-ins which states that the party trading in the car warrants that it has not had any undisclosed damage requiring bodywork, and asked why he was screwing me instead of taking the car back and going after the person who clearly violated that contract when he traded it in. I don't recall his answer to this question either, except insofar as it also being unsatisfactory.

As Ken did, the James tried to convince me that the accident wouldn't affect the resale value of the car that badly. I said that if he were confident that that was the case, he should have no problem with me getting it appraised by another Subaru dealership and then having Walker's compensate me for the difference in value. He replied, somewhat angrily I think, "No, I'm not gonna do that." I told him that I held the dealership responsible for their failure to thoroughly inspect the car, and that I would be talking to a lawyer if they wouldn't refund me or at least compensate me for the depreciation. He stated that I could talk to a lawyer if I wanted to, but that the RCW (i.e. state law) on the matter states that they are only liable if they don't disclose damage that they know about, and he reiterated that they did not know.

Even to this date, I am unsure of whether or I believe that. I am a firm believer in Hanlon's razor ("Never attribute to malice that which is adequately explained by stupidity"), but the odds I calculated above make me incredulous that Kay didn't actually have some record of the accident, in which case they obviously would have to have known. The dealership has shown me, as "proof" that they didn't know, copies of reports run in December of 2012 which indicated that the car had a clean record. But still, I'm incredulous that nobody noticed the damage before I took the car to Eastside Subaru. The technicians at Walker's should have done a thorough inspection as Ken and Jan assured me they did, so either someone lied about the damage, or their technicians neglected to do their duty, or they are incompetent. And what of the other customers who must have looked at the vehicle before I did? When I was selling my Civic after I bought the STI, two of the three people who test drove it had some mechanical experience and identified some potential issues with the vehicle that I wasn't aware of. Did nobody more knowledgeable than I look at this STI and notice the damaged parts?

Either way, I told James that I would be talking to a lawyer. He told me that he'd try to get in touch with the previous owner of the vehicle, and that he'd pull up the repair order from when it was first traded in to see if they had any indication that the car may have been in an accident. Ultimately he wasn't able to get in touch with the previous owner, but I'm not sure what difference it would have made. If that guy had any common sense whatsoever, he would have lied about the accident since knowingly trading the car in without disclosing the accident is a violation of the trade in contract. And even if he had told the truth, what then? Would Walker's have sued him if he refused to compensate me? I doubt it.

Later that same day, James called back. He asked me to explain again what Eastside Subaru told me about the air conditioning. When I mentioned that they said they couldn't warranty the part because of accident damage, he said that that's not true and he didn't know why they'd say that. He said that they just changed management and probably didn't know what they were doing, and that Walker's has been in business for 25 years. He told me if I brought the car down he would replace the AC condenser under warranty. I told him that that was not my primary concern and that it wouldn't be good enough. He accepted that, but pointed out that I still needed to get the AC fixed and that he could have it done under warranty.

Obviously what James told me about the condenser being replaceable under warranty was not true. I looked it up after I got off that call and this is what the Subaru warranty says:

These warranties do not cover any part which malfunctions, fails or is damaged due to objects striking the vehicle, road hazards, whether on or off the road, accident, fire, neglect, abuse or any other cause beyond the control of [Subaru of America].

(Emphasis mine.) This is obviously exactly what I expected since no warranty ever in the history of automobiles, as far as I can fathom, has covered accident damage. Otherwise there would be no reason for any type of insurance other than liability insurance. When I talked to him next I asked James to explain this discrepancy, and he said that while "the letter of" the warranty agreement said one thing, it was open to interpretation and it was a matter of customer service and reputation to do warranty replacements in situations like this.

Maybe I'm splitting hairs at this point, but what James was offering was not a warranty replacement, and I know that for a fact because as part of the eventual resolution of this nightmare, I did get Walker's to replace the AC condenser, and the amount charged is listed as "Internal" rather than "Warranty" on the service record. I do think the distinction is important because it means that the dealership, rather than Subaru of America, is paying for the new part, and it's not a cheap part. Although offering to replace the condenser is not an admission of legal liability, he obviously would not offer it if he truly felt that his shop was guiltless in this, because the repair comes out of his bottom line. So again, he's directly lying to me, and throwing Eastside Subaru under the bus at the same time.

I readily admit to being a bit naïve in some respects, but it's still shocking to me that there are people who will do that.

Lawyer Up

I wasn't bluffing about talking to a lawyer. I have a group legal plan through work which covers all my legal fees for applicable issues, which include consumer protection cases. Unfortunately there is also a downside to the group legal plan: it somewhat limits one's choice of lawyer, and even among the limited selection of in-network lawyers within a reasonable distance, many are very busy. Thus, there's no guarantee that you'll get a good lawyer.

I do not think that I got a good lawyer.

I had to call several lawyers before I was able to find one who would take on the case. One of the lawyers who called me back but couldn't take the case was kind enough to provide me with some basic information. He said that he thought I had a good case under the Washington state consumer protection law. He said that the dealership's failure to thoroughly inspect the car before selling it and claiming it had not been in an accident could be considered an unlawful "deceptive act" under RCW 19.86.020. He also pointed out that the Consumer Protection Act allows for triple damages up to a maximum of $25,000, which might have been useful leverage in my discussions with James, but I never actually mentioned it to him because the lawyer who eventually took on the case didn't think much of its applicability.

In fact, the lawyer I did eventually end up hiring didn't initially think that I had a case at all (but didn't tell me this until I had already hired her). She was essentially in agreement with James that the dealership would only be liable if they knew about the accident before selling the car to me, and since we would likely not be able to prove that even if it were true[^arbitration], we would have no case.

I did some research of my own and discovered a few other laws which might be applicable. I won't bother mentioning all of them, but I felt that the strongest was the Implied Warranty of Merchantability. My understanding is that this state law applies to consumer goods in general, but as it relates to cars in particular, it states that vehicles sold by dealerships "must be fit for ordinary driving purposes, free of major defects, reasonably safe, and of the average quality of similar vehicles available for sale in that price range." (Emphasis mine.) In my opinion, given the depreciation and damaged components, my car obviously did not meet those standards, but my lawyer still felt that it would be an uphill battle. And of course, even if I prevailed in arbitration or court, there would be no guarantee that the dealership would be required to offer anything beyond repairing the AC condenser.

My lawyer agreed to write a letter to the dealership on my behalf, and that if they did not respond favourably, we could take them to arbitration. I sent her all of the documents necessary for the letter on May 15th, and followed up on May 20th to check on her progress. She told me that she would be working on the letter soon. I sent her another email on the 24th, but didn't hear back from her until the 7th of June when I let her know that I would no longer need her services. Good help is hard to find.

Resolution

Luckily lawyers are not the only means of gaining leverage against corporations. I also contacted Jesse Jones of King 5 News, and filed complaints with Subaru of America, the Better Business Bureau, and the Washington State Attorney General's office. I didn't get a call back from Jesse, but the other three seem to have had an effect. I exchanged several emails with a representative from Subaru who contacted the dealership on my behalf to try to mediate a resolution. The representative didn't seem to have much actual power over the dealership, but she was courteous and timely, seemed sympathetic to my concerns, and made a real effort to do what she could, so I certainly appreciate that.

The complaint to the Attorney General's office seemed to have more of an effect. Not long after Walker's received it, James called me back and offered to write me a cheque to cover some of the depreciation in addition to fixing the air conditioner, as long as I would sign a document releasing the dealership from all liability relating to the matter. Given that I had been fighting with the dealership for a month and that my lawyer had gone AWOL, I readily accepted this offer.

I took the car back to Eastside Subaru to determine what the depreciation would actually be. Given the lack of mechanical damage, the absence of records of the accident, the fact that the damage was almost completely repaired, and the rarity of the STI sedan, I was told that the depreciation would probably not be more than $1,000 to $1,500. This seemed low to me, but I discussed it with a friend who is very knowledgeable about cars and he felt that it was not unreasonable. If you disagree with this, please do not tell me, because at this point I don't want to know. (Ignorance is bliss.) I took the car back to Walker's on the 4th of June, and they ended up fixing the AC condenser and air intake duct and giving me a cheque for $1,000.

In the interest of fairness I will say that I did have a fairly positive experience at the Walker's Renton Subaru service department when I took my vehicle in for the repairs. The service advisor I dealt with, Brad Hallock, was very courteous and helpful. Of course, that doesn't come close to excusing the rest of my experience with the dealership, particularly given that it was the Walker's Renton Subaru service technicians who ostensibly inspected the vehicle when they bought it. While I was picking up my car after the service was finished, I overheard another service advisor talking about the "world class inspection" that they were going to do for another customer. I was tempted to interrupt and tell the customer about my experience with their "world class inspection," but to my shame, I'm too non-confrontational to do something like that. It's a flaw that I need to work on.

I am fairly satisfied with the compensation I received, and overjoyed to be able to put this ordeal behind me. At this point I consider the matter to be closed. The Attorney General's office has closed their file on the case, and as soon as Walker's responds to the Better Business Bureau complaint, I will close that as well. I am looking forward to never having any contact with them again.

That said, I am absolutely not satisfied with how the dealership handled the issue in the month between my discovering the problem and them finally giving me some compensation. (Otherwise I would not be writing this, of course.) I spent far too much time fighting with them. At every juncture I encountered dishonesty, self-interest, and a complete refusal to accept responsibility for the failure of their technicians to thoroughly inspect the car. Between the time I spent playing telephone tag with the general manager, the time to find and hire a lawyer, the time spent on filing complaints with third parties, and the disruption of my schedule resulting from the loss of use of my vehicle for nearly a month[^time], I probably wasted more than a dozen hours on this, and it certainly impacted my productivity at work as well.

When I was buying the car, Ken and Jan repeatedly assured me of the quality of Walker's inspections and servicing, and the qualifications of their technicians. When it turned out that those technicians had missed significant accident damage which was still present on the vehicle and should have been easily discoverable by a qualified technician, nobody at the dealership would stand behind those claims of excellence; they repeatedly insisted that they could not have known and were not responsible. I find this appalling, though I guess I shouldn't find it shocking. When I asked why they would take such an expensive vehicle as a trade-in without doing a thorough inspection, I was told that since the CarFax didn't indicate any problems, they had no reason to suspect that there would be anything wrong with the vehicle.

I don't understand why I had to fight with the dealership about this at all. They should have just refunded the sale completely as soon as I raised the issue with them, and gone after the original owner to recoup their losses, who was clearly in violation of his sale contract.

Lessons Learned

I hope this doesn't sound histrionic, because it is completely genuine, but I was also emotionally affected by all of this. I tend to be very trusting and give people the benefit of the doubt with respect to their motivations. I am ill-equipped to deal with salespeople because I am not naturally inclined to assume that someone will mislead me and take advantage of me out of self-interest. I expect cold, detached, exploitative behaviour from large corporations, but Walker's really isn't that large, and these were real people with whom I had face-to-face conversations, who had the power to do the right thing but were unwilling to. The whole experience has made me more cynical. Moreover, the feeling I hate more than any other is powerlessness, and I found it simultaneously frustrating and discouraging to be in a situation where for a long time I seemed to have practically no recourse against the dealership for the losses I'd suffered due to their incompetence.

I'll never again enjoy my car as much I did for the first two months I owned it, and although I've been reassured that Subarus are built like tanks, I'm not sure I'll be able to completely get past the paranoia that every tiny creak or shudder might be forewarning of some impending catastrophic failure. But on the bright side, I do still love the car. I was worried that I'd hate it after all of this because of the negative associations with all the time I wasted and frustration I experienced, but that isn't the case. It's still a ton of fun to drive. I'm looking forward to finally taking it on its inaugural road trip, to modding it and making it my own, and to spending many hours "going Subaruing."

And, with every negative experience comes the silver lining of life lessons to be learned, and this is no different. Here are my takeaways from this:

A clean record from CarFax and other similar services is completely meaningless. Anything not reported to insurance or the police will not show up on a CarFax report, and in my case, I have reason to believe that an insurance claim and police report were made for the accident my vehicle was in, and CarFax still shows nothing[^CarFax].
Mechanics can be similarly unreliable. Ideally one should have a trusted mechanic whom they've dealt with previously. I haven't lived in Washginton long enough to know any mechanics well, so I should have asked friends who are knowledgeable about cars for recommendations instead of just picking one based on Yelp reviews.
Even if you don't know much about cars, there are certain things you can look for under the hood when buying a used car which may indicate that the vehicle was in an accident or had bodywork done. Check for tool marks (i.e. missing paint) on the bolts under the hood. Check for damaged plastic components. Some vehicles (although I don't think mine is among them, unfortunately) have the VIN stamped on major components; if any of the VINs are missing or don't match, something is up.
I'm not sure if this is a fair generalization based on this one experience, but if I were to buy another used car I would be very suspicious of vehicles which are overdue for important maintenance, as it may indicate that the seller did not thoroughly inspect the vehicle or does not care to maintain it himself.
I am by no means an expert negotiator so take this with a grain of salt as well, but I suspect that immediate face-to-face negotiation is not the best way to go. It suggests a desire to buy a vehicle quickly, and the more time one spends negotiating, the less willing one will be to walk away due to the lost investment of time. It is probably better to tell the dealership what you are willing to pay and immediately leave if they won't meet that demand. Either they'll call you back with a good offer, or you can continue to look elsewhere.
Buying a new car means suffering immediate depreciation, but it's also much less likely to result in nasty surprises like this. Many states have lemon laws to protect new car buyers from troublesome cars, but these rarely apply to used cars. Consider whether this tradeoff is worthwhile. Personally, I'll definitely be looking at new cars next time I buy a vehicle.
And most importantly, when legal options don't pan out, the best way to get what you want from a corporation may simply be to be persistent and annoying enough that it becomes worth their money to solve your issue just to make you go away.

[^manual]: The STI is only available with a six-speed manual transmission, but I was specifically looking for a manual car anyway. One might wonder why I'd seek out a manual even when I hardly knew how to drive them. The answer is that I'd always been intrigued by manual transmissions, and after taking a quick lesson I was set on buying a manual because I found it to be a lot of fun. Plus, without simply jumping in and buying a manual car, one has limited opportunity to learn—you can only spend so much time borrowing friends' vehicles, and lessons are expensive. (For those of you outside of North America, manuals are fairly rare here, accounting for less than 7% of yearly sales. They can't be rented at all because most people, particularly younger people, don't know how to drive them, so the rental companies would be constantly replacing burnt-out clutches and ground up gears if they did offer manual vehicles.) Unfortunately, being inexperienced at driving manual may have influenced me to make the purchase faster than I otherwise might have, as it's embarrassing to go on a test drive and stall the car several times, and I was worried that if I couldn't borrow a car to practice before subsequent test drives I'd do an even worse job of it.

[^wife]: Between the first round of negotiation and the second, my friend had to leave due to a commitment—not to mention the excruciating tedium of sitting their listening to me negotiate the price of a car. My wife joined me when I went back to the dealership.

[^lottery]: The 6/49 is a national lottery in Canada, for those of you from other countries.

[^legal]: NB for legal purposes: his words, not mine.

[^arbitration]: The sale contract contains an arbitration clause, and there is no discovery process in arbitration, so even if the dealership has evidence of the accident we would likely not have been able to get it.

[^time]: This was admittedly perhaps an unnecessary step on my part, but at the time I felt that it was the safest thing to do, and it would never have been a problem had the dealership not sold me a damaged vehicle in the first place.

[^CarFax]: With my luck it will probably show up on the report around the time I go to sell the car... not that it makes much of a difference, since as I mentioned previously, I refuse to lie prospective buyers about my knowledge of the accident.

Road Trip! Part I

2013-07-30T09:58:00Z

I love visiting new places. I've lived in the United States for just over a year now, and one of my favourite things about this country is how vast it is and how many opportunities it presents for exploration. Canada is just slightly larger than the United States, but it's also very sparsely populated north of, say, Edmonton, so practically speaking I think one can explore much more of the States while still maintaining access to the comforts of civilization.

To that end, I try to go on at least one long road trip every year. I'm also one of those people who actually enjoy driving, even for absurdly long stretches of time; anything up to 12 hours per day on the road is fairly comfortable for me, and even beyond that I don't find long trips all that tiring. I'd much, much rather drive 12 hours in my car than take a two hour plane trip.

My new car was also begging to be broken in on a long road trip. Although I had a bit of a rocky start with it, all of the issues have now been sorted out (or at least as sorted out as they'll ever be), so it offered another compelling reason to take a long trip. And so it was that I booked the 5th to 12th of July off of work and embarked on a 3,378 mile (5,436km) drive from Redmond, Washington through Oregon, Idaho, and Nevada to Utah, and then back home through Arizona, Nevada, California and Oregon.

Here's a map of my travels:

View Larger Map

The first couple days of my trip were uneventful and consisted mostly of a lot of driving. I took the scenic route through Washington and stopped to check out the I-90 Vantage Bridge over the Columbia River near the Ginkgo State Petrified Forest, as well as Palouse Falls in the southeast. I didn't actually get off the freeway at the Petrified Forest, but from what I could see, it didn't seem to live up to its name. I didn't see any trees, or indeed any wood, petrified or not. I'm not sure how a bunch of sand and scrub grass qualify as a forest. On the bright side, the view from the rest area on the I-90 east of the river was pretty spectacular.

Palouse Falls was a lot more impressive. Depending on conditions, the falls are up to 200 feet (61 metres) high. Some crazy person named Tyler Bradt recently survived going over the falls in a kayak—uninjured, even! The view from the state park campground isn't ideal, but I didn't have time to find a way down to the river and back up, so I continued on to Haines, Oregon, where I caught some decent Fourth of July fireworks, and then spent the night in Ontario, Oregon.

On the second day I drove through Idaho, Nevada and Utah. The tag line on Idaho's license plates is "Scenic Idaho," and while I don't doubt that it's true of some parts of the state, it's probably not referring to the southwestern bit. Other than some cool gorges, there wasn't much to see. I stopped at Malad Gorge State Park to take a few photos, and then continued on to Wendover, Utah, where the Bonneville Salt Flats are located.

I was looking forward to driving on the salt flats and trying to hit my car's top speed of 158mph (254km/h), but unfortunately it had been raining before I arrived and the flats were covered in a thin layer of water. I drove around a little bit anyway to put my car's AWD system to the test, but I didn't go very fast or venture very far from the paved road—it's not recommended to drive on the salt flats when they're flooded as it becomes very easy to get stuck, apparently in some cases to the point that your vehicle becomes unrecoverable. I'll have to go back to the salt flats another time when they're dry, but at any rate, they were still very pretty and it was fun to get my car a bit dirty.

Continuing on from the Salt Flats, I drove through Salt Lake City, which is absolutely beautiful. As I drove in from the west, the sunset painted the mountains to the east of the city brilliant red, and a thunderstorm in Provo to the south periodically lit the sky with flashes of lighting. Sadly, I didn't have time to find a vantage point to get any pictures of this, as it was starting to get dark and I still had a few hours of driving ahead of me. I stopped for dinner in Salt Lake City and then continued on to Price, Utah, where I spent the night.

The third day of my trip was Saturday, the 6th of July, and I arrived in Moab, Utah around noon, where it was already over 100°F (38°C). I checked into my campsite and explored the town briefly. I was reluctant to go hiking in the oppressive heat, but it showed no signs of letting up before dusk, so I bought several bottles of water and Powerade and headed to Arches National Park.

Arches is remarkably beautiful. Of all the places I visited on my trip, it was second only to Zion National Park. Even if I had gone to Zion first, Arches still would have been amazing, but as I hadn't yet been to Zion it was truly awe inspiring. Arches is also a very accessible national park; the scenic drive through the park alone is impressive, and many of the most impressive arches require only short hikes to see up close. That said, for someone from a northern climate such as myself, a two mile hike in 100-degree heat is a lot harder than one might expect.

I started with Landscape Arch, which was suitably picturesque and monumental for my first arch. From there, I tried to continue on to Double O Arch, which is further along the same trail. But between the extreme heat and a harsh wind that coated my face, eyes, mouth and camera with orange-red sand, I decided to turn back before I reached Double O Arch; with so many other arches to see, I didn't want to exhaust myself visiting the first few arches. I stopped at Pine Tree Arch and Tunnel Arch on the way back. Tunnel Arch is aptly named because it's really more of a hole through a large fin of rock, on its way to becoming an arch but not quite there yet. Pine Tree Arch was nice, but difficult to photograph without a wide angle lens.

I then made the short hikes to Skyline Arch and Sand Dune Arch, the latter of which is one of the most picturesque arches in the park, in my opinion. Being in a canyon, the soft sand around the arch is also delightfully cool on a hot day. Were it not so busy, it would be the perfect place for a nap. Behind the arch, there's a crevice that one can easily walk into for a change of perspective, and it's quite easy to climb onto the top of the arch as well.

Finally, with sunset coming on, I hiked up to Delicate Arch. I waited until the sun had gone down to try to get some long exposures, but I was mostly unsuccessful, partly for lack of skill and partly because I didn't have all of the equipment I needed. Most importantly, I didn't have a remote shutter trigger, which meant I could only take long exposures on a timer, limiting me to a maximum exposure length of 30 seconds. I resolved to come back again to try to get some night time and sunrise shots after buying a remote shutter trigger.

Back at my tent as I was getting ready for bed that evening, I left the door flap open for probably about a minute or two while I moved some of my gear from the car into the tent. Upon getting in the tent and turning on my flashlight, I discovered that that minute or two had been long enough for a solfugid to invite itself in. My dad is an entomologist so I'm not particularly frightened of insects or arachnids, though I'm certainly not as welcoming of them as he is, nor, obviously, as knowledgeable about them. The solfugid was pretty alarming because at the time I didn't actually know what it was; I only knew that it was very fast and had very large mandibles, and that Utah (in contrast to everywhere I've ever lived) does actually have venomous arthropods.

After a lot of swearing and awkward attempts to keep my eye on its rapid confused scurrying while blindly reaching for something that I could safely trap it in, I managed to scoop it up in a cooking pot and throw it outside. I then searched the Internet for something like "insect with 10 legs with 2 huge front legs" and found out what it was, and that they're relatively harmless. (When I talked to my dad about it later, he was quick to point out that solfugids are arachnids, not insects, and that the two front "legs" aren't actually legs at all. He also said that he was jealous of my encounter as he's never seen one in the wild.) I still wouldn't want to get bitten by those big mandibles, though.

Sunday was just as crushingly hot as Saturday had been (and, indeed, as the entire next week would be). To cool down during the hottest part of the day, I took a trip up the Colorado River with Moab Jett. I didn't find the scenery all that remarkable, but the trip was nevertheless a lot of fun and very refreshingly cool. Because it is a jet boat with no external propeller, by maintaining a high rate of speed the boat is able to skim over water only a few inches deep, and at various points on the tour the captain puts the boat into a spin, soaking everyone on board with refreshing river water. In retrospect, I could easily have brought my phone along in a couple ZipLoc bags for photos, but at the time I wasn't sure just how wet the tour would be; hence, there are no photos.

After the jet boat tour I decided to check out Dead Horse Point State Park. I should mention at this point that I had been using my car's built-in GPS to navigate throughout the trip. It has a couple nice features such as the ability to input multiple waypoints on the way to a destination, but other than that, it's pretty bad and is one of my few significant complaints about the car. It does a poor job of choosing optimal routes relative to software like Bing, Google or Apple Maps, and is abysmal at estimating how long a drive will take. And, as I discovered, it also has a pretty broad definition of what qualifies as a road.

Mapping a route from Moab to Dead Horse Point, I was presented with two options, and I chose the "alternate" route which I thought would be a more scenic drive. And indeed it was—but it also turned out to traverse Long Canyon Road, a somewhat steep and rocky 4x4 trail from the Colorado River basin up to the Dead Horse Point mesa. There are settings in the GPS to enable or disable routing through major highways and toll roads, so I'm surprised that there's no way to filter out dirt roads (and that such an option isn't turned on by default), but I wasn't actually bothered about it taking me on a 4x4 trail; I figured it would be another good chance to try out my Subaru AWD.

The clearance of my car is too low for serious offroading, and my "performance" street tires don't have the grip necessary for steep sandy slopes, but nevertheless my car performed admirably. I saw a few trucks and SUVs on the trail, but I didn't encounter any other cars. I drove up to just beyond the giant fallen rock before Pucker Pass, at which point the road was fairly steep and covered in extremely soft sand which afforded my tires almost no traction. I think I probably could have made it through and continued to the end of the trail, but as I was alone, dozens of miles from the nearest city, and with no cellular reception, I didn't want to push my luck, so I turned around and went back down the canyon and back to the interstate. (I feel like I'm letting other Subaru owners down by admitting that, but better safe than sorry!) The road probably doesn't look that bad in my photos, and it wouldn't be for a vehicle with proper tires and more clearance, but keep in mind that the places I stopped for pictures were the least steep and bumpy parts.

As an aside, on my way back, right by the intersection of Long Canyon Road and the highway, I saw a large matte black Ford truck with a huge stabilized camera rig on the back, as well as a Mustang with fake bullet holes and a missing taillight on a flatbed tow truck. Clearly filming for some movie was going on in the area. Fast & The Furious 7? Transformers 4? Who knows.

Finally, more than an hour later than I had expected to, I arrived at Dead Horse Point. The view of the Colorado River from Dead Horse Point is well worth the drive, but once you've seen it, there's not much else to do in the state park, so I didn't spend too long there. As I was leaving, a beautiful but ephemeral rainbow presented itself to the east, punctuated by flashes of lightning from a storm that had rolled in to the east of Moab. Sadly I couldn't get any decent pictures of this.

Earlier in the day I had picked up a remote shutter trigger for my camera from a small photography store in Moab, so with my resolution to return to Delicate Arch for sunrise in mind, I went to bed as early as the heat would allow. I woke up around 3:30am on Monday and drove back out to Arches. Equipped with a flashlight, my camera and tripod, and my new shutter trigger, I hiked up to the arch in complete darkness and waited for the sun to rise.

I actually arrived much earlier than I needed to for sunrise, because I had been hoping to get some photos of the arch with stars behind it. Unfortunately this is where I discovered another piece of equipment my photography kit is lacking. Obviously auto-focus doesn't work in low light, let alone complete darkness, but I discovered that it's also nearly impossible to correctly manually focus on an object for a long exposure when you can't actually see the object. I thought I might be able to accomplish this by trial and error, but doing so turned out to be time consuming and unreliable.

I later did some research and discovered that one of the preferred solutions to this problem is to carry a laser pointer with one's photographic equipment. By shining the laser on the object you want to focus on, you'll have a small, bright point, and you'll know that your image is in focus when that point is readily visible through the viewfinder. While I'm on the subject of photography, I also learned that on lenses with optical image stabilization, the stabilization must be turned off for long exposures, or fine movements in the stabilization mechanism will result in a blurry image.

Anyway, once the sun came up I was able to get some reasonably nice photos, although I was somewhat compositionally limited by the swarm of people who had arrived around 6am. Now that I was able to see where I was going, I took the opportunity to explore the natural stone amphitheatre below the arch, as well as the jutting outcropping of rock to the south. Here I found a number of lizards warming themselves in the morning sun.

Returning from Arches, I made a brief stop in Canyonlands National Park to see the Mesa Arch. Canyonlands is a very big park, which makes it quite time consuming to explore. I was eager to move on to Bryce Canyon and Zion, so after seeing the Mesa Arch (which is very impressive and well worth going to Canyonlands for), I decided to start driving to Panguitch, where I'd check out Bryce Canyon and spend the night. I'll save the rest for part II, as Bryce Canyon and Zion National Parks were even more spectacular than Arches and Canyonlands, and are more than deserving of their own post.

Road Trip! Part II

2013-12-25T06:07:00Z

Five months ago, I left off my last post having just left Canyonlands National Park, heading further south to visit Bryce Canyon and Zion National Parks. I had booked a motel for the evening in Panguitch, Utah, which is a three or four hour drive from Moab. The drive there was uneventful, with relatively little in the way of scenery. I mostly followed the interstate, until about an hour outside of Panguitch, where I exited onto state route 89.

Route 89 winds through scrub desert, farmland, and several very small towns nestled between modest mountains. I arrived in Panguitch and discovered that it, too, is very small. In contrast to Moab, though, it didn't seem to have much of a tourist draw despite its proximity to Bryce Canyon. As a foreigner, I have to admit that I felt some apprehension at this. Although everyone I'd met in Utah so far had been very friendly and welcoming, a small, isolated town like Panguitch is the kind of place where I imagined that I could encounter a Top Gear-esque redneck mob. Granted, I wasn't driving around with a purposefully offensive slogan on my car, nor do I really have a discernible accent, but I still felt like a pretty obvious outsider with my Washington plates and conspicuous blue car with absurd spoiler. I was relieved, then, when I checked in at the New Western Motel and was greeted at the front desk by an older Indian man (who I presume is also the owner); it was good to see that the population of Panguitch isn't completely homogeneous.

If you read the previous post you might recall that I had only slept for a few hours the previous night, as I was up around 3:30am to get pictures of Delicate Arch at night. That being the case, I was exhausted by the time I checked into my room, so I decided to nap for a few hours before heading to Bryce Canyon. It might have just been because I was so tired, but I distinctly recall thinking that the bed in my motel room was among the most comfortable I had ever slept in.

I woke up around five or six in the evening and went to grab dinner from "The Pizza Place" in the nearby town of Tropic. Yes, it's literally just called "The Pizza Place." It's a nice little family-run business, and the pizza was pretty good. I couldn't help but notice that they had several copies of the Book of Mormon available for guests to peruse. That didn't surprise me, given that I was in Utah, but I did find it strange that most if not all of the books seemed to be written in Danish.

I made it to the Bryce Amphitheater in Bryce Canyon National Park just in time for sunset, and took the opportunity to walk the Rim Trail and take some pictures. Unfortunately, as it was quickly becoming dark, I didn't have time to go on any of the hiking trails or explore the rest of the park. I contemplated spending another day in Panguitch and going on a horseback tour of Bryce Canyon, but ultimately opted to leave for Zion National Park the next morning so that I'd have more time there.

The next morning I made the 70-mile drive to Zion National Park. When I arrived, I immediately knew that I'd made the right choice in prioritizing Zion over another day at Bryce Canyon. The descent into the canyon from the eastern entrance is nothing short of spectacular. The Checkerboard Mesa is the first sight to greet visitors entering from the east, and it alone is a pretty remarkable formation, but it offers only the faintest hint at what awaits.

Walls of red stone rise up on either side as the road continues its descent into the Zion-Mt. Carmel tunnel. The long, narrow tunnel was carved right into the side of the canyon between during the years of 1927 to 1930. Occasional openings in the side of the tunnel offer glimpses of the valley below, but the sight when one finally emerges on the other side is truly breathtaking. It reminded me of the descent into Yosemite Valley, but with towering walls of red sandstone instead of granite. As impressive as the drive is, though, it doesn't even pass through Zion Canyon proper.

I checked into my campground, set up my tent, and planned for my first hike: Angels Landing. It was still early in the afternoon, and far too hot for a 5 mile (8km) hike with around 1500 feet (460m) of elevation gain. Luckily, my campground was also an RV park and inn, so it had amenities such as a pool and showers. I lounged around the pool until later in the afternoon when the temperature started to drop, and then, carrying my tripod, camera, and an ample supply of water, hopped on the canyon bus.

Due to the limited area of the canyon, and to reduce environmental impact, visitors aren't allowed to drive or park in Zion Canyon itself. Instead, the park operates a free shuttle bus up and down the canyon, complete with narration describing the history of the canyon and the significance of each of location it stops at. While informative, the narration gets a bit tiring after the third or fourth time you hear it, which makes one feel sorry for the park staff who are subjected to it daily. On the first bus I took, a woman who I assumed to be an off-duty park employee entertained us by reciting word for word, but in a very sarcastic tone, several minutes of the narration.

The climb up to Angels Landing starts off with a deceptively gentle walk along the Virgin River at the bottom of the canyon, but then quickly climbs upward along a series of increasingly steep switchbacks. It was still hot enough out to make the upward trek thoroughly exhausting. Thankfully, the aptly named Refrigerator Canyon offered some respite. Encountered about half way through the hike, the small box canyon is consistently several degrees cooler than ambient temperatures.

At the end of Refrigerator Canyon, the trail ascends Walter's Wiggles, a long set of tight, steep switchbacks, and then emerges atop a spine of cliff that culminates in Angels Landing itself. This is where the trail really gets fun—or terrifying, depending on your perspective. I don't consider myself to be particularly afraid of heights, but the view down to the valley floor is truly dizzying, even before one reaches the top of Angels Landing. And between the top of the of the Wiggles and the Landing, there are still hundreds of vertical feet to climb along a twisting, rocky, extremely narrow trail where a slip to either side means certain death. So steep is this section of trail that much of it is spent pulling oneself upward along sections of chain that the National Park Service has anchored into the rock. It's remarkable that only a handful of people have died at Angels Landing. Did I mention that I was carrying a tripod and several pounds of camera equipment the whole way up?

Arriving at the summit, though, the perilous hike proved to be more than worthwhile as I was afforded a beautiful view of Zion Canyon. The brilliant green foliage along the valley floor gives way to majestic orange cliffs, which turn fiery red as the sun sinks in the distance. Words can't do it justice, and frankly, neither can my photography, but I tried nevertheless:

Later that evening, I drove back along the Zion-Mount Carmel highway, stopping a few miles from the east entrance to try to get a few pictures of the stars above the canyon. That didn't work out as well as I had hoped, but I did see another scorpion!

The next day, I set off to hike The Narrows, a 16-mile (25km) long, 20-30 foot (6-9m) wide, 2000 foot (610m) deep gorge carved by the Virgin River. What makes The Narrows unique is that there is no hiking trail at all: the hike is actually undertaken in the river. As such, it can vary greatly in difficulty depending on the season; during months where the water level is high, it can be all but impassable, whereas in the summer it tends to be relatively easy with water ranging from ankle-deep to waist-deep. As I was in Zion during July, the water level was quite low.

Even so, such an atypical hike requires extra preparation, so I walked across the road from my campground to the Zion Adventure Company and picked up the required gear. This consisted of a pair of canyoneering shoes and neoprene socks, a walking stick, and a waterproof bag for my camera equipment. I ended up finding the walking stick to be more of a burden than a boon, and the waterproof bag wasn't really necessary either given the low water levels, but it's better to be safe than sorry with expensive camera equipment.

A word of advice for anyone planning to hike The Narrows, though: if you rent canyoneering shoes, make sure they fit with plenty of room to spare. They'll shrink in the water, so shoes that are a comfortable fit at the beginning of the day can become very, very uncomfortable after several hours in the river. I learned this the hard way: by the end of my hike, I could barely walk, and much of the toenail on my big toe on my left foot ended up turning black and falling off. Five months later, it's finally almost completely grown back.

Aside from that slightly painful mistake on my part, though, The Narrows proved to be a truly amazing hike, unlike anything I've ever done before. Looking thousands of feet up the sheer canyon wall offers a new perspective on the enormity of Zion Canyon, and there's something new to see in every corner of the canyon: foliage sprouting out of the ancient stone, birds circling overhead, gurgling springs pouring from cracks in the walls, and all around, the ever-changing Virgin River.

There are two ways to hike The Narrows. Starting from the top down, it's a 16-mile hike, which the National Park Service estimates will take about 12 hours. Unfortunately, this route requires two permits, one of which is very sought-after and therefore difficult to get. Not having either permit, I was left with the bottom-up hike, which is a 10-mile (16km) round trip if you go all the way to the mandatory turn-around point at Big Spring, which I did. I also took the optional side-trip into Orderville Canyon, which added another mile to the hike for a total of around 11 miles (18km).

The first few miles of the hike were very busy, probably owing both to how spectacular the views on the hike are, and how refreshing the cool waters of the Virgin River are on hot days in Zion. Luckily, most people seem to stop and turn back around Orderville Canyon, which is about 2.5 miles into the hike. After the Orderville fork, the main canyon becomes very narrow in spots—hence the name. Of course, this also means that the river is much deeper in a few areas. Even with the low water levels, I had to wade through several waist-deep pools with my camera bag held above my head. There was even one pool deep enough that I would have had to swim across it, but I managed to find an alternate route by sneaking behind a colossal boulder in the river.

After five miles of walking upstream, I reached Big Spring, which is exactly what it sounds like: a large spring pouring into the river from the canyon wall. With the water low, it's also a perfect spot to stop and relax. There's a nice sandbar to sit on, and the canyon is wide enough that the afternoon sun bakes the sandbar to a pleasantly warm temperature. There's also a fairly deep pool in the river opposite the sandbar, adjacent to the spring itself; this I found to be perfect for diving and swimming, and a nice way to cool off when the sun became too warm.

I met a group of students and their football coach from a Mormon school in Provo (if I recall correctly) at the spring. They were all very friendly, and I encountered them again several times on my way back down the canyon, so we ended up chatting about various subjects. Learning that I'm from Canada but working in the US, they were interested to hear how Canadians can acquire US work visas. This is something I've been asked about several times; it seems that some aspects of the US immigration system are almost universally unknown. I don't mean that to sound judgmental, though—it's merely an (I think) interesting observation. Most people have little reason to know, and I can't claim to understand in detail how immigration works in Canada.

They also mentioned that they knew someone who was currently on a mission in Edmonton, which is where I went to school. I was a little surprised to learn that Mormons from the US actually go on missions in Canada; I guess I had pictured missions primarily taking place in poorer countries where the missionaries can help the locals with construction, education, health care, and so forth, in addition to their evangelism. I didn't really ask about it, though; mostly we talked about how mind-numbingly cold winters in Edmonton can be.

On my way back down the river, I stopped briefly in Orderville Canyon. It's a short detour, and well worth it. Orderville is narrower than the main canyon, and a bit more challenging to hike in some spots. After one of the first several bends, there's a long straight stretch with a clear view to a jutting peak above the canyon, which offers a phenomenal sense of the scale of the stone around you. Unfortunately, I couldn't effectively photograph this. At the turn-around point, Orderville rewards those who take the time to hike it with a small waterfall.

Sadly, I only had a couple days to spend in Zion, so the next day I packed up my tent and started the long drive back to Redmond. The fastest route would have been to back north through Idaho, essentially following the same route I'd arrived on. I wanted the drive to be interesting, though, so instead I headed southwest to St. George, Utah, then through the northwest corner of Arizona and on to Las Vegas. (There's a map in my first post.) From there, I drove northwest along the California-Nevada border. The GPS in my car can be extremely uncooperative at times, particularly where seasonal roads are involved, so I ended up following an unnecessarily circuitous route, crossing the state border no fewer than 5 times. I did find some beautiful roads, though; in particular, the Robert M. Jackson Memorial Highway (California State Route 89) near Leviathan Peak. I reached the peak just as the sun was setting, and the view was gorgeous.

I stayed the night in South Lake Tahoe. It was already night when I arrived, and I didn't have much time to explore the next day given how much driving I still had left to do. Still, driving around the lake re-emphasized the beauty of the area; Lake Tahoe now has a high spot on my list of places that I have to visit.

From Tahoe, I made another detour back into Nevada, through Carson City, and then back through California up to Oregon. Northern California reminds me quite a bit of the southern interior of British Columbia, but other than that, there wasn't much to see. As for the drive through Oregon, the only thing I want to say about it is that south-central Oregon doesn't seem to be very interesting (except for Crater Lake, which I didn't get a chance to stop at), and that speed limits in the state are far too low. 55mph on the interstate makes for a long, boring drive.

So, that's it! I hope you've enjoyed reading about my adventures. I try to do at least one long road trip each year, so if you have suggestions about where I should go next year, let me know.

Game Review: The Elder Scrolls Online

2014-05-19T04:02:00Z

[Updated March 17, 2015: The Elder Scrolls Online: Tamriel Unlimited was officially released today, meaning that ESO no longer requires a subscription fee to play. In addition, there have been six major content updates since the game's original release, with dozens upon dozens of improvements. I'd encourage you to read my new post on the subject or just go straight to Zenimax Online Studios' own announcement.]

It has been about a month and a half now since The Elder Scrolls Online was released, and it's received generally mixed reviews. Despite some undeniable technical issues and a handful of design flaws, it's a great game and more than a bit underrated in my opinion; if you're a fan of The Elder Scrolls or an MMO aficionado, you owe it to yourself to give ESO a try. I'll explain why.

First, by way of qualifications, I should mention my history with the Elder Scrolls series. I didn't play Arena or Daggerfall, so I haven't been following the series since its inception, but Morrowind captured hundreds of hours worth of my attention and made a huge impression on my taste in games—it was unlike anything I'd ever played at the time, and not even other Elder Scrolls games have quite managed to recapture that magic for me. Oblivion was, at first, quite awe-inspiring, but once the captivating new graphics wore thin, it ultimately seemed shallow and uninspired compared to Morrowind. Skyrim was a return to form, with better and more varied environments, a more interesting story, and improvements to game mechanics all around—in addition to, once again, remarkable visuals. In short, I've been an Elder Scrolls fan for more than 12 years now, so although I don't go all the way back, I have a good basis for comparison.

The Elder Scrolls Online came as a surprise, being announced less than a year after Todd Howard indicated that Bethesda had no interest in an Elder Scrolls MMO. In fact, at that point ESO had been in development for four years, but Howard may not have known and certainly would not have been at liberty to reveal that. At any rate, my initial reaction was extreme skepticism; I doubted that the best aspects of The Elder Scrolls could be preserved when introduced to the MMO formula. As more details were revealed about the game's design and mechanics, I grew more interested, but I after playing the beta I was again left with lukewarm impressions. Still, something about it kept nagging at me, and I couldn't stop watching Twitch streams of ESO as launch drew nearer. Eventually, I decided to buy it just in time for early launch. Since then, I've put in nearly 150 hours, and that number would doubtless be much higher were it not for the obligations of the real world.

There are a few things that the Elder Scrolls Online actually does much better than any of the previous Elder Scrolls games, and if you've any experience with other MMOs, they'll likely come as a big surprise. One is the quests: they're better written, more engaging, and more meaningful than those in previous Elder Scrolls games, and in fact most games in general. The aggressive use of phasing (an MMO technique which allows many players to co-exist in different versions of the same space) allows your actions to actually have a lasting, visible effect on the world, to an even greater extent than I can recall seeing in previous Elder Scrolls games.

When you're given a quest to liberate a city from the clutches of the enemy du jour, for instance, you'll actually liberate the city, cleansing the streets of marauding monsters and opening up new shops and questgivers. Phasing is not a novel concept, but I've never seen it used so extensively or to such great effect, and it really makes quests feel much more worthwhile than the typical MMO tropes of slaughter 10 pigs or collect 8 doodads. Even when the underlying quest mechanic is no more complex than one of those, the presentation in ESO makes it feel much more interesting, and to a large extent, that's what counts.

There are also actual moral choices in The Elder Scrolls Online. The game doesn't yet offer the opportunity to steal from or kill NPCs and possibly face the consequences as previous Elder Scrolls games have (though Zenimax promises that they are planning such a feature), but many quests offer compelling moral dilemmas. Will you allow the willing sacrifice of a loyal servant to cure a Duke infected with lycanthropy? Will you show mercy to a traitor, knowing that your choice could come back to haunt you? Will you use a powerful Necromantic artifact to bolster the strength of your faction's army and possibly save the lives of many soldiers, or will you destroy it lest it fall into the wrong hands and lead to many more deaths?

The outcomes of these choices are never Earth-shattering (Nirn-shattering?), and their effects on gameplay are varied, but they're more believable and relatable than the choices in most games touted for their "morality systems." There is no omnipotent force here tallying up your Karmic balance and changing the colour of your robes depending on whether you've been good or bad. Instead, your choices may determine whether certain NPCs live or die, or simply how they perceive you. It sounds minor, but as some of the NPCs are recurring throughout the storyline, the results of choices you make early in the game are often reflected later on. More importantly, the moral choices in ESO are often actually difficult because both options have costs associated with them, or because making the upstanding choice means more work or less reward. Such is the morality of real life; it's complicated and difficult because there's isn't always a good option, and virtuous choices often involve sacrifice. Very few games, even among those praised for excellent narratives, get this right, so to see it done so well in an MMO (a genre not known for good storytelling) is a pleasant surprise.

ESO has largely have the same cast of voice actors that have been utilized in previous games, and overuse of some actors is a problem as always, but on the whole I'm impressed at how much content Zenimax was able to include while still providing full voiceovers for all of the NPC dialogue. This is one of the things that I didn't know I wanted until I actually experienced it, but it really makes a huge difference in how interested I am in the story and the quests. I've played several MMOs, including World of Warcraft for more than 7 years, and I've never before bothered to actually read the quest text. In The Elder Scrolls Online, I am not only listening to the quest dialogue, but I'm also taking the time to listen to supplemental NPC dialogue. It's well written and mostly pretty well-acted, and there's a lot of good humour to be found as well. Excellent performances by big names like Michael Gambon, John Cleese and Jennifer Hale make the main quest lines a real pleasure, too.

Of course, all of this would be for naught without solid game mechanics, and fortunately ESO excels there, too. Combat mechanics have always been something of a weak point for The Elder Scrolls series. Each new iteration improves on the last, but conversely, each time one goes back to an older Elder Scrolls game one wonders how he or she ever put up with it. Even as each new game brings significant improvements, Bethesda has never quite captured a real sense of weight and solidity in their combat mechanics. In contrast to (for instance) Dark Souls, where the impact of each slash and thrust can almost be felt through the controller, swordfighting in The Elder Scrolls often feels like swinging a tennis racket through empty air.

Being an MMO, ESO's combat can't compete with the likes of Dark Souls, either... but it feels surprisingly good. It's more fast-paced and dynamic than combat ever has been in The Elder Scrolls, requiring a strategic approach and an attentive eye to take on groups of enemies, particularly in the Veteran Ranks. Weapon attacks feel good, with real differentiation between various type of weapons, and the available abilities are varied enough to keep things interesting.

The skill system in ESO is a nice change from MMO orthodoxy, being a blend of the prescribed, XP-based systems seen in other games and the more open-ended approach that The Elder Scrolls is known for, where skills are levelled up by using them. Basically, there are a few skill lines unique to each of the game's four classes, but there are many more that can be unlocked by any character. Skill points are unlocked in a number of ways, including through levelling up with XP, but how you can invest them is dictated by how far you've advanced each of your skill trees (which happens primarily by using the skills in the tree). It might sound complex, but it's actually fairly intuitive in practice, and it works quite well overall.

Of course, combat has never really been the focus of The Elder Scrolls, so many will wonder whether ESO lives up to the standard of boundless exploration set by previous games in the series. The answer is no; while I find the term a bit distasteful, ESO could fairly be called a "themepark" MMO, and although there's lots of world to explore, character progression through different zones is very linear. In that respect, fans who enjoy The Elder Scrolls primarily for the open world aspect may be disappointed. But there are a few ways in which I think ESO makes up for it.

The first is delves, or public dungeons. These are much the same as the dungeons, caves and ruins one can explore in other Elder Scrolls games. They are littered across the overworld, and although they often offer quests, no quest or NPC interaction is typically required to access them—they're best found through exploration. In MMO terms, they're similar to instanced dungeons, with a miniboss at the end that may drop higher-quality loot, but they're open to everyone and designed to be completed solo. They're surprisingly effective at breaking up the routine of solo levelling, and they do a lot to make ESO feel like a real Elder Scrolls game.

Second, and more importantly in my opinion, is the diversity of environments in ESO. This is another thing Bethesda has never been very good at: they pick a single province of Tamriel with a single type of terrain and geography, and while they do a marvelous job of realizing it, their single-minded focus can lead to game worlds which become stale over time. Morrowind and Skyrim had enough variation that this wasn't a huge problem, but Oblivion was terrible for it; everything looked the same, to the extent that exploring eventually felt pointless.

While not all of Tamriel is yet playable in ESO, the game offers the opportunity to visit many different provinces, including several that haven't been featured since the days of Arena and Daggerfall. Along the way, you'll explore grasslands, forests, rocky spires, deserts, swamps, tundra, and more. Exploring new zones is a big aspect of what appeals to me about MMOs, and ESO does a much better job of offering many different types of environments to explore than previous Elder Scrolls games have. (That said, it's not perfect; in particular, I wish the Daggerfall Covenant zones had more variation.)

Having said so many positive things about the game, it should be obvious that I am a huge fan of The Elder Scrolls online, and would highly recommend it to MMO, RPG and TES fans. There are many more good things I could write that I simply don't have time to. However, I would be remiss to not also mention the problems with the game, for there are several, and some of them are significant.

Bugs and performance/latency issues are at the top of the list, unfortunately. Since the game came out, there have been many, many bugged quests, and although I can't recall seeing one recently, patch notes and forum threads make it clear that many still abound. Many class skills are currently bugged; although I don't play one, I'm told that this is particularly bad for Nightblades. Similarly, there is an issue with passives not applying properly after your character dies and resurrects, which is especially annoying in PvP.

ESO offers a weapon switching mechanic, but it's currently quite unreliable because the amount of time it takes to execute is very inconsistent. This often leads to accidental double-switches, or failed switches, which in turn frequently leads to frustrating deaths. And in general, even without a high ping, I've experienced occasional latency issues which, when they occur, make combat unpredictable and irritating. This happens most often in large PvP battles for me, but I understand that latency issues in general are quite common for European players.

PvP balance is another sore spot. Early on, the game was dominated by vampires, and although they've now received a well-deserved nerf, I still feel that their balance could use more attention - not necessarily outright nerfs, but changes, at least. Werewolves, in contrast, are so weak as to be almost pointless. (Both lycanthropy and vampirism are "diseases" which any player can contract.) Class balance is also less than stellar, although I won't go into specifics on that, as it's a more controversial topic. Of course, this is no surprise in a brand new MMO. Even World of Warcraft had remarkably poor PvP balance until at least the launch of the Burning Crusade expansion, a full two years after the main game launched.

Certain game systems, while good in theory, were not particularly well executed. For instance, when you hit level 50 in ESO, you begin levelling through the "Veteran Ranks," and are given an opportunity to experience the other factions' content and quests with levelled up (and generally more challenging) enemies. Think of it as the MMO version of New Game+. As a recovering alt-a-holic, I like that this gives me an incentive to experience more of the game's content without having to roll a new character. But at the same time, because there are few realistic alternative means of reaching max level, I dislike that it almost precludes me from trying another class without having to play content I've already completed. (Experience from monsters is severely reduced in Veteran Rank content, meaning that grinding is not viable. Hardcore players hate this.) The fact that you can't choose which enemy faction to visit first is more than a little bit annoying, too.

There are a host of other, less severe issues, which affect different players to varying degrees. I'm sure many players have "pet" issues which I've failed to mention here. I can't say that these issues don't affect the game experience; in some cases, they can be quite significant. But, like it or not, this is par for the course for a new MMO. I am confident that Zenimax is carefully monitoring the game and taking customer feedback into account. Their progress on bug fixes so far has been acceptable if not stellar (although I'm sure not everyone would agree), but I think it's clear that the game is stabilizing fairly quickly. The upcoming content patch should include a number of major fixes that players have been demanding, which I think will help a great deal.

The Elder Scrolls Online is not for everyone. It's not The Elder Scrolls VI, nor is it the next World of Warcraft. If you expect either of those things, you will likely be disappointed. What it is is an adept blend of those aspects that make The Elder Scrolls great with all of the social, competitive and cooperative elements that underpin good MMOs. Despite its flaws, I think it's an absolutely excellent game overall, and one which will surely appeal to most of its target audience if approached with an open mind. Even if the idea of a $15/month subscription fee is offputting to you, you can easily experience enough content in the first month to make the game well worth its $60 box price. I can't recommend it enough.

Hope to see you in Tamriel!

Book Review: Dataclysm

2014-10-09T05:51:00Z

If you've ever read OkCupid's popular OkTrends blog, you're already familiar with the work of Christian Rudder. As OkCupid's resident mathematician and one of its founders, he has been writing for years about statistical trends related to dating and attraction, leveraging OkCupid's vast database of traits about, and interactions between, its millions and millions of users. OkTrends produced numerous interesting and surprising results, and it developed a significant following before going quiet for the better part of three years beginning in 2011.

Now, Rudder is back with Dataclysm: Who We Are When We Think No One's Looking, which expands on OkTrends by drawing from a wider variety of sources (including Facebook, Google, and Twitter, in addition to OkCupid and other dating sites) and discussing a more diverse set of topics. Organized into three themed sections which focus on, respectively, unity, division, and individuality, the book explores what data can tell us about gender, sexuality, race, identity, community, and other sociopolitical questions. The discussion is consistently balanced and neutral, which is important when dealing with such potentially charged subjects, but Rudder also manages to inject enough of his personality and characteristic style to keep the book enjoyable throughout.

Indeed, the OkTrends blog was popular not only for the fascinating observations it produced, but also for Rudder's accessible and entertaining writing style, and he continues this to great effect in Dataclysm. No mathematical or statistical knowledge whatsoever is required to understand the topics discussed in the book; data are presented in the form of simple graphs and explained using plain English with a minimum of jargon.

Unfortunately, this focus on accessibility also gives rise to my biggest criticism of the book: on the spectrum of "pop science" writing, Dataclysm is decidedly more pop than science. In and of itself, that isn't a problem per se, but in many instances I felt that a more methodical approach could have led to a broader and more productive discussion. Rudder is not nearly thorough enough in describing the methodology by which he arrived at his conclusions, nor in presenting (or at least making available) salient facts about the data he used, nor in exploring alternative hypotheses.

For example, consider the following graph of the "perception of male vs. female attractiveness" from the introduction. On the x-axis is the average of all ratings (on a 1-5 star scale) received by a user, and on the y-axis is the percentage of all users that received that average rating:

Rudder explains:

Our real-world data [of women's attractiveness as rated by men] diverges only slightly (6 percent) from this formulaic ideal [of a symmetric beta distribution] ... it implies that the individual men who did the scoring are likewise predictable, centered, and above all, unbiased... It's practically common sense that men should have unrealistic expectations of women's looks, and yet here we see it's just not true.

The red chart [of men's attractiveness as rated by women] is cenetered barely a quarter of the way up the scale; only one in six guys is "above average" in an absolute sense... [as an analogy,] translate this plot to IQ, and you have a world where women think 58% of men are brain damaged.

This is certainly interesting data in its own right, but I don't think Rudder goes far enough in exploring what it means or why it might be the case. By comparing against a similar random sample from a social network, he determines that it is not because men on OkCupid are unusually ugly—this pattern persists beyond dating sites. His hypothesis is simply that "men and women perform a different sexual calculus," and, quoting Harper's Magazine, he notes that "women are inclinded to regret the sex they had, and men the sex they didn't."

This is certainly a plausible explanation, and it's intuitively appealing, but it also seems like a bit of a cop out based stereotypical beliefs about gender differences. I'd like to see data about how straight men and women rate the attractiveness of people of their own gender—although not directly analogous since it would be divorced from sexual desire, I suspect that the results might raise questions about Rudder's hypothesis here.

In the interest of fairness, I must restate that his discussion takes place in the book's introduction, so it's not representative of the length at which other findings in the book are discussed. Most likely, it's just meant to pique the reader's interest and serve as an example of the kind of data the book will examine. Nevertheless, similar issues persist in later chapters of the book. Consider this map of responses to the question "Should flag burning be illegal?" among OkCupid users:

Rudder claims that this map demonstrates that "this is truly a nation defined by its principles, or, as you can see, two nations: Urban and Rural." But to my eyes, the colors overlap too much to make a meaningful distinction; this really just looks like a population map. What's more, it appears to include Canadian responses even though I can only assume that Rudder is referring the US when he writes "a nation," singular. And again, there is no discussion of important methodological questions or possible confounding factors, such as whether OkCupid's user base is disproportionately urban (which doesn't seem unlikely to me), and whether or not that was corrected for if so.

Rudder anticipates criticism of his decision to eschew "statistical esoterica," and defends it as follows:

I don't mention confidence intervals, sample sizes, p values, and similar devices in Dataclysm because the book is above all a popularization of data and data science... But like the spars and crossbeams of a house, the rigor is no less present for being unseen. Many of the findings of the book are drawn from academic, peer-reviewed sources. I applied the same standards to the research I did myself...

I fully accept that there is a place for and a value to "popularization" of scientific fields, and that it's sensible to exclude complex statistical measures which would be meaningless to most readers... from the main body of the text. But I don't see why Rudder didn't even see fit to include these as footnotes or endnotes, which the book does have. And with respect to the academic research used, Rudder relegates all of his citations to endnotes, which usually link back to the text referring to them, but are not linked to from that text, which can make it very hard to match up specific statements with the studies supporting them.

All of this criticism notwithstanding, I really enjoyed reading Dataclysm, and I would heartily recommend it to anyone interested in gaining insight into the social and political identities of (primarily American) Internet users. Anyone interested in the burgeoning field of data science would also likely enjoy this book. It's just worth noting that some of Rudder's conclusions should be taken with a healthy dose of skepticism until they can be replicated with rigor that is seen.

Book Review: Free Will

2014-10-27T00:00:00Z

Sam Harris is a controversial figure, to say the least. Personally, I'm a fan of his work, but as one of the "four horsemen" of so-called New Atheism, he has met with much ire since the publication of his first book. This comes not only from religious adherents, but also from secular liberals, many of whom seem to willfully (and sometimes maybe even gleefully) misinterpret him. Although Free Will, published in 2012, courts less political controversy than Harris' other works, it too will raise the eyebrows of some readers. In fact, earlier this year, it met with a prickly reception from fellow "horseman" and philosophy professor Dan Dennett.

In Free Will, Harris lays out a case against the philosophical concept of the same name. There are many facets to his argument, but I think his primary claims can be enumerated as follows:

Scientific research shows that "determinism, in every sense relevant to human behavior, is true. Unconscious neural events determine our thoughts and actions—and are themselves determined by prior causes of which we are subjectively unaware."
Determinism cannot be reconciled with (what Harris believes is) the popular conception of free will. (In philosophical terms, this means that Harris takes an incompatibilist stance on the relationship between determinism and free will. He spends much of the book arguing against compatibilism, which is a philosophy that tries to reconcile the concepts.)
Even if determinism were not true (due to, for example, randomness in quantum events), free will would still not be possible, as in that case "[our wills would be] the product of chance and we [would not be] responsible for them."
Upon introspection, "free will doesn't even correspond to any subjective fact about us ... Seeming acts of volition merely arise spontaneously (whether caused, uncaused, or probabilistically inclined, it makes no difference) and cannot be traced to a point of origin in our conscious minds."

Harris begins with the scientific evidence for the determined nature of human behavior. He writes:

The physiologist Benjamin Libet famously used EEG to show that activity in the brain's motor cortex can be detected some 300 milliseconds before a person feels that he has decided to move. Another lab extended this work using functional magnetic resonance imaging (fMRI): Subjects were asked to press one of two buttons while watching a "clock" composed of a random sequence of letters appearing on a screen. They reported which letter was visible at the moment they decided to press one button or the other. The experimenters found two brain regions that contained information about which buttons subjects would press a full 7 to 10 seconds before the decision was consciously made. More recently, direct recordings from the cortex showed that the activity of merely 256 neurons was sufficient to predict with 80 percent accuracy a person's decision to move 700 milliseconds before he became aware of it.

(Harris provides sources for each of these claims in the book, but I am not reproducing the citations here for the sake of brevity.) These findings are pretty remarkable in their own right, and will probably be quite unsettling to some readers. This is also likely the only part of the book that will seem truly novel to those already highly schooled in philosophy; the debate about free will has been ongoing for centuries, and it seems unlikely to me that Harris' short book (at less than 100 pages long) contains any truly novel philosophical arguments. Even so, I'm not sure the discussion of these studies will actually work to Harris' advantage. The core of his argument is philosophical, but some readers are certain to get caught up in overthinking the scientific claims, even though, as Harris himself states, nothing actually depends on them.

In fact, Harris goes so far as to say that even if one believes that consciousness originates in a "soul" which is separate from and unconstrained by our physical brains, that still would not make free will feasible: a soul would not change the fact that we do not have full awareness of, let alone control over, the origins of our thoughts and intentions. To use one of Harris' examples:

Why did I order beer instead of wine? Because I prefer beer. Why do I prefer it? I don't know, but I generally have no need to ask. Knowing that I like beer more than wine is all I need to know to function in a restaurant. Whatever the reason, I prefer one taste to the other. Is there freedom in this? None whatsoever. Would I magically reclaim my freedom if I decided to spite my preference and order wine instead? No, because the roots of this intention would be as obscure as the preference itself.

Even with that example, I suspect this claim is likely to confuse many readers, as a naïve reading of it will make it seem flatly absurd: you probably can explain most of your thoughts and intentions with reference to your values, preferences, duties, etc. But Harris' point is that if you follow this chain of causation back as far as possible, you will inevitably arrive at something you either can't explain or have no control over. You might say that you prefer beer over wine (or vice versa) because beer is more pleasing to your taste buds. But why is it more pleasing? Probably, ultimately, because of your genes. Obviously, you have no hand in that. (I hope I'm not muddying the water by adding that whatever you think your reasons for a given decision are, studies have shown that many of the decisions we make are actually rationalized after the fact.)

Neither does the possibility of indeterminism offer any refuge for free will, claims Harris. I think he is being accurate when he states that arguing against determinism is no longer "philosophically respectable," but nevertheless there are some who persist in doing so. In fact, often as a way to salvage traditional notions of free will, "some scientists and philosophers hope that chance or quantum uncertainty can make room for [it]." This argument doesn't stand up to much scrutiny, though, and Harris dispatches it succinctly, observing that "[c]hance occurrences are by definition ones for which I can claim no responsibility. And if certain of my behaviors are truly the result of chance, they should be surprising even to me. How would neurological ambushes of this kind make me free?"

Harris reserves a couple chapters at the end of the book to discuss the practical consequences of giving up one's belief in free will. I found these chapters somewhat perfunctory and unconvincing. In fairness, they are tangential to the actual purpose of the book, but given how deeply the assumption of free will is rooted in human psychology and society, I would have liked to see a more thorough discussion here. Moreover, I think Harris could have made room for that by eschewing some of his examples of how we feel free but really aren't, many of which are reducible to the same observations and therefore feel repetitive despite the book's short length.

On the impact of disbelief in free will on our behavior, Harris writes:

Many people worry that free will is a necessary illusion—and that without it we will fail to live creative and fulfilling lives. This concern isn't entirely unjustified. One study found that having subjects read an argument against the existence of free will made them more likely to cheat on a subsequent exam. Another found such subjects to be less helpful and more aggressive. It is surely conceivable that knowing (or emphasizing) certain truths about the human mind could have unfortunate psychological and/or cultural consequences. However, I'm not especially worried about degrading the morality of my readers by publishing this book.

(Again, Harris provides citations for the studies he references.) Harris goes on to justify the last sentence with reference to his personal experience of losing his sense of free will, stating that it has "[increased his] feelings of compassion and forgiveness, and [diminished his] sense of entitlement to the fruits of my own good luck." I suppose his experience shows that it is possible for losing one's belief in free will to have positive (or at least neutral) psychological consequences, but given that research shows that this is not the typical result, I don't think that a personal anecdote should be enough to set us at ease. Of course, it should go without saying that this has no bearing on the existence or non-existence of free will.

Harris' discussion of morality is a bit more satisfying. He observes that, without a belief in free will, retributive justice becomes difficult if not impossible to justify: "Once we recognize that even the most terrifying predators are, in a very real sense, unlucky to be who they are, the logic of hating (as opposed to fearing) them begins to unravel." This is eminently compatible with modern liberal understandings of how justice should operate. Happily, the logic behind rehabilitation, incarceration for the safety of society, and punishment as a means of deterrence, remains intact: "It may seem paradoxical to hold people responsible for what happens in their corner of the universe, but once we break the spell of free will, we can do this precisely to the degree that it is useful."

Compatibilism

Astute readers may have noticed that I mentioned that Harris argues against comaptibilism, but that thus far I have neglected to actually address those arguments. That is because Harris' incompatibilist position is both somewhat complex, and certainly the most controversial part of Free Will, at least among philosophers. Indeed, it is Harris' dismissal of compatibilism that compelled professional philosopher Dan Dennett to write a criticism of the book that is, as Harris observed, "nearly as long as the book itself." I'm going to refer to both Dennett's criticism and Harris' response to it throughout this section, which is unorthodox for a review, but in this case I think those works make an extremely valuable addendum to the book itself, as they expose (to a greater degree than the book does) the core of the disagreement between Dennett's compatibilism and Harris' incompatibilism. (Where I quote Harris, it will be from the text of Free Will, unless otherwise noted.)

"Consider what it would actually take to have free will," Harris writes. "You would need to be aware of all the factors that determine your thoughts and actions, and you would need to have complete control over those factors." Harris takes this as, I think, basically axiomatic, but compatibilists disagree. Dennett writes that "[s]uch 'perfect' freedom is, of course, an incoherent idea," which is true, but I think this is actually Harris' point: this is the free will that people suppose they have, but when one thinks about the implications of that belief, it is shown to be nonsensical.

Here we arrive at another disagreement between Dennett and Harris: Dennett argues that Harris doesn't actually know what people in general think about free will, and that "suitably expressed questionnaires might well prove him wrong." In fact, he provides a study which he purports does exactly that: polls conducted but Nahmias et al. appear to show that people with no philosophical background actually do have compatibilist notions of free will.

In his response to Dennett, Harris disputes the validity of this study, noting that "Nahmias and his coauthors repeatedly worry that their experimental subjects didn't really understand the implications of determinism—and on my reading, they had good reason to be concerned. ... More than anything, this research suggests that people find the idea of libertarian free will so intuitively compelling that it is very difficult to get them to think clearly about determinism." I have to say that I am in agreement with Harris here. Having read the paper, it is not at all clear to me that Nahmias et al accurately assessed their subjects' compatibilist or incompatibilist beliefs. They write:

Despite our efforts to induce participants to make their judgments based on a conditional acceptance of the scenario, it is likely that some did not do so, as evidenced by those who missed the manipulation check on the back of the surveys, which tested whether they recognized the deterministic nature of the scenario. Some of these participants may have even been expressing incompatibilist intuitions, e.g., by asserting that Jeremy acts of his own free will and then (mis)responding that the scenario therefore could not have been describing a fully deterministic situation.

I agree with Harris that this is a very valid concern, and I'm not convinced that the "manipulation check" (which isn't described in much detail) would have been sufficient to account for it. Nahmias et al note that "many" participants found the deterministic scenarios they used "implausible or impossible," and that in one case, "almost all participants" believed the scenario was impossible. While participants' stated reasons for this belief varied, it seems entirely plausible to me, and perhaps even likely, that a significant number of participants could not effectively reason about the situations being presented because of strong attachments to beliefs that contradict the scenarios' premises.

In any case, Dennett argues that even if people do in fact have a natural tendency towards libertarian incompatibilism (i.e. they believe in free will but do not believe that it is compatible with determinism, and, perhaps consequently, disbelieve determinism), this might just show "that most people are suffering from a sort of illusion that could be replaced by wisdom." He seems to believe that the illusion in this "folk conception of free will" is based on Cartesian dualism, and that the goal of compatibilism is to correct replace this "nonsense" with something more "scientifically sound."

Dennett wonders why Harris objects to this project and suggests that Harris is arguing for Cartesian dualism, but this is a misconstrual: Harris is simply drawing a distinction between what we consciously experience and all of the unconscious processes our brains are engaged in at every moment, of which we have no (subjective) knowledge. This is an important distinction, because Dennett's compatibilist view of free will is predicated on the fact that "we are coterminous with everything that goes on inside our bodies, whether we are conscious of it or not" (Harris' words). But although this is true, our subjective, conscious experiences are actually only proxies for the underlying neural states in our brains; one simply does not have conscious access to the totality of his or her being. This is not simply a misconception that can be corrected: it does seem to be, as Harris argues, a "psychological fact," and I would argue that it is probably also a biological fact. I am sure this is not what Dennett hopes to replace with compatibilism, but I find myself confused about the nature of the misunderstanding he feels he can correct.

I am reluctant to dismiss Dennett's arguments outright without first reading his own books on the subject, but intuitively I find myself agreeing with Harris that compatibilism seems like a cop out. I have spent quite some time trying to get at the root of this intuitive feeling, and I've found it difficult to do so even with reference to Harris' arguments, which makes me suspicious of its validity. But I think this paragraph by Dennett at least gets close to the core of my skepticism about compatibilism:

Contrast your competence at this moment [of deciding which card to play in bridge] with the "competence" of a robotic bridge-playing doll that always plays its highest card in the suit, no matter what the circumstances. It wasn't free to choose the six, because it would play the ace whatever the circumstances were whereas if it occurred to you to play the six, you could do it, depending on the circumstances. Freedom involves the ability to have one's choices influenced by changes in the world that matter under the circumstances. Not a perfect ability, but a reliable ability. If you are such a terrible bridge player that you can never see the virtue in ducking a trick, playing less than the highest card in your hand, then your free will at the bridge table is seriously abridged: you are missing the opportunities that make bridge an interesting game. If determinism is true, are these real opportunities? Yes, as real as an opportunity could be: thanks to your perceptual apparatus, your memory, and the well-lit environment, you are caused/determined to evaluate the situation as one that calls for playing the six, and you play the six.

If we instead contrast a human chess player with a computer such as Deep Blue, we see that the latter has all of the faculties that Dennett claims give a person free will: "perceptual apparatus," memory, etc. Granted, the human player could purposefully make a bad move, or quit the game, which the computer is not free to do given its programming. But its programming could easily be changed to permit those options as well. The point is that the computer behaves in a determined manner according to the combination of its algorithm and the inputs to that algorithm. Dennett seems to admit that we humans do exactly the same, if with a more complex algorithm and broader range of inputs, but claims that this is freedom nevertheless. But this does not seem to me to be a meaningful use of the word freedom, and I think Harris is correct that it is not the sort of freedom that most people suppose they have.

Conclusion

Despite the length at which I've discussed Free Will, there is much that I did not cover. If you found this discussion even moderately interesting, then you'll certainly enjoy the book, and I do recommend reading both Dennett's criticism and Harris' response to it as well. Philosophical concepts can be extremely difficult to grapple with, and reading about them second-hand is rarely beneficial to understanding.

I really enjoyed reading Free Will, and I found myself agreeing with pretty much all of it, but I hesitate to call it convincing: I was already skeptical of the idea of free will given our scientific understanding of determinism, so I really didn't need much persuasion. For those who feel a deep sense of attachment to the idea of free will, I am not sure Harris' arguments will be sufficient to convince you otherwise. As the debate between Dennett and Harris demonstrates, the concept itself can be so nebulous as to make it difficult to even discuss. This may also prove an obstacle for those who lack exposure to philosophical reasoning, as the arguments can sometimes seem to devolve into semantic disputes. Conversely, those with a significant philosophical background may find Free Will uninteresting as they're likely to have read similar arguments before. For everyone else, though, it comes highly recommended: this is the rare sort of book that could really have a profound impact on a person, and it can be read in a sitting, so there's really not much to lose.

Well, except your sense of autonomy, I suppose.

Book Review: Lying

2014-11-03T02:09:00Z

Originally published as a "Kindle single" in 2011, Sam Harris' Lying was republished in a revised and expanded format in 2013, but it remains more of an essay than a book. The entirety of the book weighs in at 108 pages, but the main content takes up only about a third of that. The remaining two thirds consist of a transcript of a conversation between Harris and professor Ronald A. Howard, whom Harris credits as the inspiration for the book, and an FAQ based on feedback received since the original publication. These latter two parts make a worthwhile addendum to the original essay, but they are by no means indispensable: the centerpiece here is Harris' analysis of the act of lying and all the ways in which it can lead to bad outcomes.

In Lying, Harris attempts to make the case that dishonesty is almost always unethical, and that simply by being truthful, one can avoid "endless forms of suffering and embarrassment." This might seem like a fairly obvious statement, but Harris explicitly includes (and, in fact, primarily focuses on) "white lies"—those small lies we tell not for our own benefit, but, we think, to spare others from hurt feelings or unpalatable truths. Though white lies are often told by good people for what they believe to be are good reasons, Harris argues that the cost of these lies, in the long term, outweighs their benefit:

What could be wrong with truly "white" lies? First, they are still lies. And in telling them, we incur all the problems of being less than straightforward in our dealings with other people. Sincerity, authenticity, integrity, mutual understanding—these and other sources of moral wealth are destroyed the moment we deliberately misrepresent our beliefs, whether or not our lies are ever discovered.

And although we imagine that we tell certain lies out of compassion for others, it is rarely difficult to spot the damage we do in the process. By lying, we deny our friends access to reality—and their resulting ignorance often harms them in ways we did not anticipate. Our friends may act on our falsehoods, or fail to solve problems that could have been solved only on the basis of good information. Rather often, to lie is to infringe on the freedom of those we care about.

...

When we presume to lie for the benefit of others, we have decided that we are the best judges of how much they should understand about their own lives—about how they appear, their reputation, or their prospects in the world. This is an extraordinary stance to adopt toward other human beings, and it requires justification. Unless someone is suicidal or otherwise on the brink, deciding how much he should know about himself seems the quintessence of arrogance. What attitude could be more disrespectful of those we care about?

Harris provides a number of examples of how white lies can backfire: by dishonestly telling a friend that he doesn't need to lose weight, you might be denying him the motivation he needs to live a healthier lifestyle. By hiding a serious medical diagnosis from loved ones, you foreshorten their opportunity to come to terms with it, and miss a chance to deepen your relationship. By providing false encouragement about a friend's career prospects, you lead her to waste time and energy that might be spent more productively with a change of course.

Although he does make a case for honesty in situations in which many of us would be tempted to lie, Harris does not seem to be advocating for "Radical Honesty". Radical Honesty, as I understand it, requires not only complete honesty, but also complete transparency, and what some might call a stunning lack of tact. Harris, by contrast, allows that it may sometimes be necessary to "[hold] one's tongue, or [steer] a conversation toward topics of relative safety" in order to maintain civility. He also sees no conflict in principle between being honest and keeping secrets: "If someone asks how much money you have in your bank account, you are under no ethical obligation to tell him. The truth could well be 'I'd rather not say.'"

Harris' position is certainly easier to embrace than "Radical Honesty," which I imagine could easily ruin your life if you don't have extremely understanding friends and family. That said, it's still not easy, in practice, to always be honest. It is hard to tell people things that you know they won't want to hear. It takes willpower and emotional energy to always make the virtuous choice rather than looking for the easy way out. And there are edge cases, rare but by no means impossible, where the ethical value of honesty really does seem questionable. For instance, consider this example from Harris' conversation with professor Howard:

Harris: ... Other examples of this [ethical difficulty] have come to me from people who are caring for parents with dementia. Your mother wakes up every morning wondering where your father is, but your father has been dead for fifteen years. Every time you explain this, your mother has to relive the bereavement process all over again, only to wake up the next morning looking for her husband. Let's assume that when you lie, saying something like "Oh, he's away on a business trip," your mother very quickly forgets about your father's absence and her grief doesn't get reactivated.

Howard: That's an interesting one. I would be tempted to say something like "Well, he's where he usually is at this time of day." The fact that he's buried in the ground somewhere doesn't add anything to this person's knowledge of what's going on. And as you pointed out, you would just be putting her through pain all over again. As you stated the case, why would you want to do that?

This answer doesn't seem entirely satisfactory. First of all, what if you were asked for a more specific answer than the one Howard gave? In that case, I think you really would be forced to lie, and from a consequentialist point of view that really does seem like the right thing to do given the situation; Howard almost admits as much. And even if that's not the case, Howard's answer still seems deliberately designed to be deceptive, even if it is not strictly false. He goes on to call this "'skillful truth-telling' as opposed to 'evasion,'" but that seems like a euphemism to me.

This exemplifies the problem I have with Lying. I'm in complete agreement with Harris that lying is almost never the right choice, for both ethical and practical reasons. But believing that in principle is much different from adhering to it in practice. How does we resist the almost reflexive impulse to tell people only what we think they want to hear? Where do we draw the line between reasonably concealing of private information and dishonestly withholding facts that people have a right to know? And how can we train ourselves to reliably decide when situations really do warrant "skillful truth-telling" if not outright deception? These questions are not always easy to answer, and I don't think that Lying does enough to address them.

If you find yourself to really be in need of convincing that white lies can be harmful and should generally not be ethically permissible, then Lying is probably a worthwhile read. But if you already accept that proposition, this book is hard to recommend.

Book Review: I Am Malala

2014-12-26T09:57:00Z

I expect most people will already have some familiarity with Malala Yousafzai. In case you've been living under a rock, though, Malala is a young Pakistani woman who, at the age of 15, was shot in the head by the Taliban for her outspoken support of the right to education of children, including (especially) girls, who are often denied the right due to their gender. Miraculously, she survived the assassination attempt, and following her recovery, has continued to campaign in support of education. Now, at age 17, she has an impressive list of accomplishments and accolades to her name, including most recently becoming the youngest Nobel laureate in history after jointly winning the 2014 Peace Prize with Kailash Satyarthi, another activist for children's rights.

I Am Malala: The Girl Who Stood Up for Education and Was Shot by the Taliban, written jointly with award-winning foreign correspondent Christina Lamb, is Malala's autobiography. It chronicles her experience growing up in Pakistan's Swat Valley, the invasion of the Taliban into the area and her life under their rule, their attempt on her life, and her recovery and continuing efforts in support of education.

Before I continue, I have a caveat about this review. Reviews always contain elements of subjectivity, and there's nothing wrong with that per se: art, after all, is in the eye of the beholder. But a reviewer must endeavor to keep his own biases in check, and in particular, to separate his opinion of the creator from his judgment of the creation. In that respect, this review is difficult to write. I believe that Malala Yousafzai is the most courageous and heroic person alive today, and that her campaign has the potential to be among the most important activist movements of the 21st century. The emancipation and education of oppressed, powerless and impoverished women and girls in developing countries is vitally important to global stability and progress, and Malala has become not only an internationally-recognized figurehead for this struggle, but also, through her foundation and her speeches, one of the most important leaders in effecting a solution.

Naturally, then, I find I have a hard time criticizing Malala's work, and you should bear that in mind as you read this review. Despite any effort to the contrary, I will likely be more forgiving to this book than I would be to a book written by someone for whom I did not have such great respect.

It is unfortunate, then, that I must admit that this book took me almost a year to read, and not due to its length: even including a short glossary and timeline, it's barely more than 300 pages. The problem was that I just didn't find the first third of the book very interesting, so I started reading it very slowly, and then eventually put it down for a long time, only returning to it recently.

The first chapters of I Am Malala mainly serve as context for the rest of the book. Malala first describes the history of Pakistan and of her family, including much of her father's life before she was born. Her father, Ziauddin Yousafzai, is shown to be an ambitious and courageous person in his own right: his own dream was to be an educator, and as a young college graduate, he quickly progressed from teaching English in a private college to setting up his own school. He then tenaciously oversaw his school's development into a sizable and successful institution, despite many setbacks including considerable financial hardship and a natural disaster which all but destroyed the school building.

Malala also describes her father's respect for women, which no doubt played a formative role in her iron-willed personality, and which she contrasts with the rest of Pashtun culture:

For most Pashtuns it's a gloomy day when a daughter is born. My father's cousin Jehan Sher Khan Yousafzai was one of the few who came to celebrate my birth and even gave a handsome gift of money. Yet, he brought with him a vast family tree of our clan, the Dalokhel Yousafzai, going right back to my great-great-grandfather and showing only the male line. My father, Ziauddin, is different from most Pashtun men. He took the tree, drew a line like a lollipop from his name and at the end of it he wrote, "Malala." His cousin laughed in astonishment. My father didn't care.

Much of the rest of the opening of the book describes Malala's life in Mingora, from broad details about the general character of life in Swat (which will probably seem to most Western readers to be a peculiar mix of semi-modern technology, patchwork infrastructure, and ancient cultural traditions), to specifics such as her reputation in school as a talented and dedicated student. All of this is necessary background, particularly for Westerners such as myself who haven't the slightest clue what life in Pakistan is like, so I don't fault Yousafzai and Lamb for including it in the book—but, well, it's just not very interesting to read. Perhaps it's the linear, matter-of-fact way in which it's written, or perhaps it's the marked similarity of Malala's life before the Taliban occupation to that of a typical schoolchild, but I found myself struggling to get through this section.

And yet, peculiarly, this section also seems to be short on some basic factual details; for instance, until I looked it up for this review, I had pictured Mingora as a small, rural city. In fact, though, its population was 175,000 in 1998—sixteen years ago, so it's likely grown considerably since then. The city I grew up in has less than half that population today. Thus, my perception as I read the book was greatly skewed in this respect, but it needn't have been given all of the other details provided about Swat and Mingora.

The pace of the book picks up as the Talibanization of the Swat Valley begins in 2007, when Malala is ten years old. Maulana Fuzlullah, now a highly ranked militant leader in the Pakistani Taliban, started a radio broadcast in which he professed his interpretation of Islam. Fazlullah's broadcasts started innocently enough:

He used his station to encourage people to adopt good habits and abandon practices he said were bad. He said men should keep their beards but give up smoking and using the tobacco they liked to chew. He said people should stop using heroin, and chars, which is our word for hashish.

Over time, though, Fazlullah became increasingly radical and fundamentalist: claiming that sinful actions such as "listening to music, watching movies and dancing" had caused a recent earthquake; instructing women to never leave their homes except in emergencies, and to always be veiled while in public; and referring to the Pakistani government as infidels, stating that "his men would 'enforce [Sharia law] and tear [government officials] to pieces.'" By 2007 Fazlullah had become a powerful and influential figure within the Pakistani Taliban, and on the 12th of July, following the Red Mosque siege, he declared war on the Pakistani government.

The Taliban continued to spread throughout the Swat valley. By 2009, the Talibanization had become so widespread that even local politicians, including the area's deputy commissioner, were involved: "When the highest authority in your district joins the Taliban, then Talibanization becomes normal." As they gained more and more political power, they also began enforcing their strict interpretation of Islamic doctrine through threats and outright violence. They issued an edict banning the education of girls, and meted out brutal punishments for "immoral" acts such as dancing or wearing clothes "incorrectly." There were bombings, executions, and public whippings.

It was during this time that Malala began writing, at the behest of the BBC, "a diary about life under the Taliban." This diary was published online as part of the BBC's coverage of the situation in Swat, but Malala used the pseudonym "Gul Makai" due to the great danger inherent in criticizing the Taliban. The diary only lasted until a few months, though. The Pakistani army had been trying for some time, noncommittally and mostly ineffectively, to push back the Taliban, but after a brief ceasefire, in May of 2009 they launched a full-scale operation, "Operation True Path, to drive the Taliban out of Swat." This resulted in an exodus of civilians from the valley, and thus Malala and her family became "Internally Displaced Persons," forced to leave their home for nearly three months while the army effected its campaign to take back Swat.

The campaign ended with the army claiming victory in July of 2009, and Malala and her family were able to return home. But, predictably, the army's successful campaign could not completely or permanently rid the valley of Taliban:

In Swat we began to see more signs that the Taliban had never really left. Two more schools were blown up and three foreign aid workers from a Christian group were kidnapped as they returned to their base in Mingora and then murdered. We received other shocking news. My father's friend Dr. Mohammad Farooq, the vice chancellor of Swat University, had been killed by two gunmen who burst into his office. Dr. Farooq was an Islamic scholar and former member of the Jamaat-e-Islami party, and as one of the biggest voices against Talibanization he had even issued a fatwa against suicide attacks.

...

Once again people started worrying that the Taliban were creeping back. But whereas in 2008-9 there were many threats to all sorts of people, this time the threats were specific to those who spoke against militants or the high-handed behavior of the army.

"The Taliban is not an organized force like we imagine," said my father's friend Hidayatullah when they discussed it. "It's a mentality, and this mentality is everywhere in Pakistan. Someone who is against America, against the Pakistan establishment, against English law, he has been infected by the Taliban."

Despite this, Malala continued speaking at events in support of education and peace, and continued to gain national and international attention, including winning Pakistan's National Peace Prize—a brand new award for which Malala was the first recipient ever. But as we now know, this stubborn refusal to be silenced would come at great cost to her.

On Tuesday, October 9th, 2012, Malala was shot in the head by a young Taliban gunman as she rode the school bus home. This chapter, in contrast to the terrible event it recounts, is probably the best written in the book. Malala describes every minute detail of the moments leading up to the assassination attempt: the sounds from the shops along the road, the smell of the air, and then the sudden, ominous absence of people in the street. The style here precisely communicates the feeling of a memory of a terrible event: the way time seems to slow leading up to it, and how vividly we recall the mundane circumstances and sensations it becomes associated with. Even though this is the part of Malala's story that everyone already knows, it's pulse-quickening to read in her own words, and at this point the book becomes very hard to put down.

The rest of the book, which comprises about a fifth of its total length, describes Malala's treatment and recovery, and briefly illustrates her new life in Birmingham, England. Living in England has given Malala the opportunity to return to school, and to continue to speak in support of education for children, without being in constant danger. Nevertheless, she misses her home in Swat, and already has aspirations to return to Pakistan:

I know I will go back to Pakistan, but whenever I tell my father I want to go home, he finds excuses. "No, [dear one], your treatment is not complete," he says, or, "These schools are good. You should stay here and gather knowledge so you can use your words powerfully."

He is right. I want to learn and be trained well with the weapon of knowledge. Then I will be able to fight more effectively for my cause.

Here again, though, I wish I Am Malala provided more detail. Malala's cause is clear, as is her fearless determination in pursuit of it. Still only seventeen years old, she already has the recognition and support of the international community, and a non-profit foundation in her name working towards her dream of safe, free education for all children. But achieving this will be no easy task, and I wish the book provided a more complete picture of Malala's vision. Besides the overt threat of terrorist groups such as the Taliban, Malala and her foundation will have to contend with poverty, war, cultural impediments to learning, suspicion of the West and its influence, and innumerble other challenges. She has a long fight ahead of her. What is her plan?

Perhaps this is a lot to expect from a seventeen-year-old. But then, Malala has very clearly demonstrated that she is no ordinary seventeen-year-old. I'm sure she has given some thought to these issues, and I wish her book had provided more insight in that respect. For instance, she briefly mentions a now-infamous letter she received from Taliban commander Adnan Rashid, who claims she was shot for espousing what the Taliban view as a pro-Western, anti-Islamic ideology. She barely dedicates a paragraph in response to this, but even here in the West there are people who take Rashid's words at face value, as if his stance is perfectly legitimate and reasonable. This is a problem that Malala will likely have to contend with for as long as she continues to fight for her cause. What will she do to solve it? I can only hope that details such as these will be the subject of another book.

Much as I'd like to, I can't say that I Am Malala is among the best books I've ever read. It is linear and factual to a fault, some of it written almost like a narrated CV, and is at times ponderously paced. It seems to me to be more about what Malala has done than who she is; for an autobiography, it feels curiously impersonal. That said, the latter two-thirds of the book are still an enjoyable read, at least for those who want to know more of Malala's story. Most importantly, whatever quibbles I have with the way I Am Malala is written, it may well be one of the most important books I've ever read, because like Malala herself, it is a testament to what can be accomplished through hope, dedication, and perseverance.

I hope, and expect, that we will only hear more and more about Malala Yousafzai as the years go on.

Book Highlights: Halo: The Fall of Reach

2015-01-03T04:03:24Z

As the first book set in the Halo universe, Eric Nylund's Halo: The Fall of Reach describes the training and induction of the first-ever Spartans, including John-117, the Master Chief. It follows the Spartans through some of their earliest missions, depicts humanity's first contact with the Covenant, and culminates in the battle for planet Reach, a human colony. I don't often read fiction, and I am not sure I've ever before read "expanded universe" fiction for a game or a movie, so Halo: The Fall of Reach was a bit of an unusual read for me. However, since I really enjoy the Halo games, universe, and characters, and since this book came highly recommended by other fans of the series, I figured it couldn't hurt to give it a try.

Unfortunately, per my review policy, I can't actually review this book: the Halo series, and indeed the copyright to the book, is owned by Microsoft, by whom I am employed by. Instead, I want to highlight a few things that I found interesting in the book. Be warned that this post will contain some spoilers for the book and other parts of the Halo series.

I've never been involved in professional game development, but my understanding is that game mechanics and environments are typically built first, with story added later in the development process; this necessarily means that the plot is shoehorned into a set of scenarios and environments that might not actually be an optimal fit for it. This in turn is one of the reasons video game writing is notoriously mediocre. To the extent that this is true, it is especially true of the first game in a new franchise. However, Halo: The Fall of Reach was actually originally published on October 30th, 2001, about two weeks before Halo: Combat Evolved was released. This shows that Bungie had an impressively thorough understanding of the universe they were building at that relatively early stage in the franchise's life.
On the other hand, much of that universe might still have been conceived in a short amount of time. In an afterword to the Halo: The Fall of Reach, Frank O'Connor writes that "[t]he book went from conception to final print in a staggering four months." That is certainly impressive for 352-page book!
According to the book, Spartans in MJOLNIR armor are able to sprint at superhuman speeds: "The Chief and his team sprinted up the half-kilometer sandstone slope in thirty-two seconds flat." That is a speed of 35 miles per hour (56.2km/h) on an upward slope. This is somewhat at odds with the relatively sluggish pace of movement in the first three Halo games. However, sprinting was fairly unpopular when it was added to the game series, as many people felt that it changed multiplayer balance for the worse. (I don't personally have an opinion on that, but obviously, gameplay balance is more important than being accurate to lore.)
On the subject of gameplay being at odds with lore, the game Halo: Reach also depicts the fall of Reach, but from the perspective of a group of Spartans on the ground rather than from the Master Chief's perspective. It also infamously retcons some significant portions of the plot laid out in The Fall of Reach. I haven't actually played Halo: Reach in several years, but going back and reading the plot summary on Wikipedia, I can immediately see a number of differences from the events depicted in the book.
Final warning: spoilers begin here! At the beginning of Halo: Combat Evolved, the UNSC ship Pillar of Autumn is supposed to have made a random slipspace jump to retreat from a Covenant fleet—it's not mentioned in the game that the ship is retreating from the lost battle for Reach. This is how the humanity first happens upon the Halo rings (and, in fact, how the Covenant find them as well). I always had trouble suspending disbelief that a jump to some random point in the universe would result in the discovery of one of the handful of Halo rings that exist. But it turns out that the jump wasn't actually random: Cortana chose the jump coordinates based on ancient navigation symbols found on an artifact that the Master Chief had recovered from the Covenant after they invaded the human colony world of Sigma Octanus IV.
Similarly, I had trouble suspending disbelief about the concept of AI rampancy, at least as described in the game, when it was introduced in Halo 4. Frankly, I thought the whole thing was a fairly stupid plot device. But, actually, rampancy wasn't introduced in Halo 4; it was a known phenomenon in the Halo universe right from the start, and is mentioned at the same time as Cortana is introduced. I'm still not sure I'm totally on board with the concept, but it does feel less idiosyncratic now that I know that it was established long before being reintroduced in Halo 4. Originally, I viewed it as something 343 added to introduce tension to the romantic subplot between Cortana and the Master Chief, but it turns out they were actually strictly following existing lore the whole time:

Cortana was a "smart" AI, an advanced artificial construct. Actually, the terms smart and dumb as applied to AIs, were misleading; all AIs were extraordinarily intelligent. But Cortana was special.

So-called dumb AIs within the set limits of their dynamic memory-processing matrix were brilliant in their fields but were lacking in "creativity." Déjà, for example, was a "dumb" AI—incredibly useful, but limited.

Smart AIs like Cortana, however, had no limits on their dynamic memory-processor matrix. Knowledge and creativity could grow unchecked.

She would pay a price for her genius, however. Such growth eventually led to self-interference. Cortana would one day literally start thinking too much at the expense of her normal functions. It was as if a human were to think with so much of his brain that he stopped sending impulses to his heart and lungs.

Like all other smart AIs that Dr. Halsey had worked with over the years, Cortana would effectively "die" after an operational life of seven years.

How to manually send a Pingback

2015-01-18T05:04:29Z

A couple years ago I stopped using WordPress and switched to Octopress, which is a derivative of Jekyll. I continue to be pretty happy with this decision, but there are some downsides to using a static site generator. One issue is the lack of support for Pingbacks, which are a method supported by a number of dynamic blogging platforms to notify another blog when you link to its content. Pingbacks provide a good way to respond to blog posts using your own blog rather than via commenting; thus, they facilitate "conversation" between blogs. Unfortunately, if you're using Octopress or Jekyll or another framework to generate static HTML pages, you won't natively have the ability to send or receive Pingbacks.

Luckily, Pingbacks can be sent manually with relative ease. Receiving Pingbacks still won't be possible, though; that's a much harder problem to solve, for a variety of reasons which aren't relevant to this post.

Before I proceed, I want to note that full credit for this information goes to Aaron Parecki; he explains the technique in this GitHub Gist. The reason I'm reproducing it here is because it was surprisingly hard to find Aaron's Gist, or anything else of relevance, when I was searching for information on how to send a Pingback. Hopefully, by posting about it and adding some extra context, I can make the information more searchable.

First, you'll need to create a copy of this XML file on your local machine:

:::xml pingback.ping http://source/url/here http://target/url/here

Replace the two URLs as appropriate - the first one is the source (i.e. your blog post), and the second is the target (the post you are linking to).

Now, you just need to POST the request to the target server. You'll need to know the target's XML-RPC endpoint URL for this. Usually, it will be /xmlrpc, or for WordPress blogs, /xmlrpc.php. You can find out for sure by inspecting the site's source for a <link rel="pingback" href="..."/> tag.

You can use cURL to post your request. It's built into most Unix-like operating systems, but Windows users will have to download it, and if you're on Mac OS X you probably need to have Apple's developer tools installed. Just run:

curl -X POST -d @pingback.xml http://example.com/xmlrpc.php

...and that's it! cURL should print out an XML response from this server with a message indicating that your Pingback was registered. Of course, to be truly sure it worked, you should check the target URL and see if your Pingback was added to the page. (Note that some sites disable Pingbacks.)

The care and feeding of software engineers

2015-01-26T08:48:10Z

Via a friend's Facebook feed comes this excellent article about "the psychology of software engineers and what makes us the way we are." I'm in almost complete agreement with the article, although I do somewhat take issue with this paragraph:

Part of the problem [of software engineers consistently giving overoptimistic estimates and then failing to meet them] is also our fragile egos. We get afraid that if we give an estimate that is "too long", that people will think less of us. "Good engineers" should be able to work faster, they say, and so we acquiesce.

I don't really think software engineers in general have "fragile egos," or at least not to a greater extent than people in any other profession. We are notoriously bad at giving realistic estimates, but in my experience, this is often because we're actually incentivized to do so: deadlines are set before the amount of work required for a project is actually determined, or without regard to it, and then we're made to fit our estimates into too-short iterations while still trying to accomplish all we set out to do. There's a common sentiment that it's better to be under-optimistic and out-perform your goals than the converse, and while that seems true in theory, in practice there don't seem to be many discouragements to failing to live up to estimates that everyone already knows are unrealistic and often meaningless.

By contrast, I think the fear that people will think less of us for giving "too long" estimates is completely valid in a culture where it's commonplace to set unrealistic goals and then break our backs trying to achieve them by any means necessary. I think this especially true at the feature team level when one is working on a large piece of software. What do you expect to happen to a team that estimates up front that its feature cannot be completed within the release window? (Remember, adding more engineers might just make the problem worse.) I think it's safe to say that it won't involve promotions and pay raises.

Finally, as my wife pointed out when we discussed this article, the planning fallacy is by no means unique to software engineers.

All of that said, I think the rest of the article is very good; this part in particular struck a chord for me (emphasis mine):

So, without enough information, changing requirements, not enough knowledge to do the job, and people constantly second guessing us, we trudge into work every day. Being creative people, we put up with all of this because we know that one day people will use our work. That's really what drives software engineers more than anything else: the idea that people we don't even know will be affected by our work. Whether you’re working on a web site visited by millions each day or you’re working on a point-of-sale system for restaurants, the knowledge that we’re affecting people's lives is a powerful driver.

Configuring DPI in Lubuntu/LXDE

2015-02-22T04:36:28Z

Updated 2015-10-10: Paul suggests that this can also be accomplished by editing or creating /usr/share/lightdm/lightdm.conf/50xserver-command.conf and adding the command xerver-command=X -core -dpi 150. I haven't tried this myself. Alternatively, you can just use Xubuntu instead, which seems to run just as well, and has a handy UI for adjusting DPI. It also handles multi-monitor setups much better.

If you've tried to use Lubuntu (or LXDE on another distribution) with a high-DPI display, you've probably noticed that fonts and other UI elements are so tiny as to be illegible without a magnifying glass. You've probably also noticed that there is no GUI with which to adjust the UI scaling factor. Happily, it is possible to change the DPI settings in LXDE, but this being Linux, it requires editing obscure configuration files. Here's what you'll need to do:

In your home directory, create a new text file named .Xresources
In this file, enter your desired DPI in the following format: Xft.dpi: 150
Restart the X server. You can do this by pressing Ctrl + Alt + F1 to enter single-user mode, then running sudo service lightdm stop, and then sudo service lightdm start. (Note that it may be a different service if you are not running Lubuntu. Alternatively, you can just reboot your machine.)

This will scale UI elements in most, but not all, applications. For instance, it doesn't resize the desktop panel, so you'll likely want to do that as well. Luckily you can do so easily by right clicking on an empty space on the panel and selecting Panel Settings; from there, just change the height of the panel in pixels to a suitable value.

This is yet another tip that I'm posting mainly becuase it took me an inordinate amount of time to figure out how to do it. I found many suggestions on how to enable scaling, but none of them worked until I stumbled across the above instructions on the blog of bebabi34. His blog is in Italian, so naturally it's not very searchable for English speakers; hopefully by reproducing his instructions here, I can save others some time.

1989

2015-03-02T07:57:06Z

I've been working on my next book review for a few days now, but I'm finding that when I try to work on it, words don't come quickly or easily to me. So, what the hell—might as well try something different. Music is immensely important to me. I've got a fairly large collection spanning over a thousand artists in dozens of different genres; I spent two years working in a record store (back when those still existed) and I consider music crucial to my ability to handle difficult emotions, be productive, and maintain sanity.

However, I've always been reluctant to write about it, because I'm basically musically illiterate. I don't play any instruments; I can't tell a C from an F; I can barely pick out the individual instruments in a song; I don't even know the right terminology to use when discussing what makes music good or not. I hope that eventually I'll be able to take some music lessons and correct this to some extent, because I also believe that you can't fully appreciate something you don't understand. But for now, I remain ignorant of the intricacies of musical composition. Thus, I'm not going to be so presumptuous as to call this a review, but... well, keep reading and you'll see what I'm on about.

I go through periods of obsession with musical artists where I'll discover someone new and become absolutely, completely captivated by them, usually buying several if not all of their records and listening to them over and over again until I've thoroughly "worn out" every single song. Then, if I'm lucky, I find someone new to become obsessed with, and the cycle begins anew.

Last month it was Lana Del Rey: her albums Born to Die and Paradise are absolutely phenomenal, and she herself is just an incredibly talented artist. One thing I find particularly interesting about Lana is how her sound—her voice, in particular—seems to evoke the 1950s; according to Wikipedia, "her music has been noted for its cinematic sound and its references to various aspects of pop culture, particularly that of 1950s and '60s Americana." But Del Rey's music doesn't sound dated—lyrically, she manages to capture a timeless quality while also epitomizing modernity and the hollowness of American decadence. It's a combination unlike anything else I have ever heard, and I still find it almost intoxicating.

Del Rey has announced a fourth album, Honeymoon, which is billed as being more like Born to Die and Paradise than her third album, Ultraviolence, which I have to admit that I didn't care for as much. Naturally, I'm really looking forward to that. Enough about Lana, though. This month, I'm hooked on Taylor Swift's latest album, 1989.

Of course, I didn't just now discover Taylor Swift; you'd have to be living under a rock to be unaware of her. Eight years after her debut, she's one of the most successful musical artists of all time, and with good reason. But while I've always had some respect for her music, I didn't actively listen to it or consider myself a fan of hers, because country—which her first three albums were firmly rooted in— is the one genre that I just can't get into (with very few exceptions). But 1989 is purely a pop album; it's her "first documented official pop album." And it's really, really damn good. Actually, "good" doesn't really cover it. 1989 is sublime.

Taylor Swift was born in 1989 (and, incidentally, so was I), but what the title is really meant to communicate is the album's musical influences: 1989 is Swift and her co-producer's homage to sounds of the 1980s—especially 80s synthpop. To borrow a quote from Rolling Stone (via Wikipedia), Swift said of the album:

It [the 1980s] was a very experimental time in pop music ... People realized songs didn't have to be this standard drums-guitar-bass-whatever. We can make a song with synths and a drum pad. We can do group vocals the entire song. We can do so many different things. And I think what you saw happening with music was also happening in our culture, where people were just wearing whatever crazy colors they wanted to, because why not? There just seemed to be this energy about endless opportunities, endless possibilities, endless ways you could live your life. And so with this record, I thought, "There are no rules to this. I don't need to use the same musicians I've used, or the same band, or the same producers, or the same formula. I can make whatever record I want."

1989 is spectacularly successful at evoking this sound and mood, and it's also a testament to Swift's versatility. (I feel compelled to note here that the ninth track, "Wildest Dreams," reminds me a bit of Lana Del Rey.) It's fascinating and instructive to go back and contrast any of the songs on 1989 with Swift's older work. Besides the new musical style, lyrically, her writing has matured a great deal; it retains her trademark themes of love and heartbreak, but on a more refined, introspective and abstract level, and is therefore much more relatable. (I guess this is unsurprising given that Swift was only 16 when her first album was released.) And yet, as different as her new record is, the fundamentals of her style remain intact; it's clear that she's had the capacity to "make whatever record [she wanted]" from the very beginning.

The most remarkable thing about 1989, I think, is that there's not a single bad or even mediocre track on it. There are highs and lows, of course, but the lows are still very good in their own right. It's very rare to find an album that is so consistently excellent from start to finish.

I look forward to seeing which direction Taylor Swift's next album takes. It seems clear that she will only continue to get better with time.

[Addendum, March 2nd, 2015] Vox has a great analysis of both the musical content and the cinematography of one of the singles from 1989, "Style," which I've included below. This kind of analysis fascinates me, since as I mentioned above, I'm not really capable of doing it myself.

Fun with the Microsoft Band SDK Preview

2015-03-14T09:07:34Z

First of all, happy Pi Day! Using the standard United States date format, today is 3/14/15, which makes it the most significant Pi Day this century.

Anyway, I've spent the past few hours tinkering with the Microsoft Band SDK Preview, and although I think it's unfortunate that there's no SDK for writing apps which actually run on the Band itself, you can still accomplish some pretty cool things with the phone SDK. Unfortunately, this being a preview release, the documentation is a bit sparse, and I had some trouble figuring out how to get things working. Thus, I wanted to share some of what I've learned. If you prefer to dive right into the code, I've got a small C# sample application on GitHub.

Standard disclaimer: although I am a Microsoft employee, everything I am posting here and on GitHub is my own personal work done on my own time. None of it should be considered as officially representative of Microsoft in any respect. While I will do my best to answer any questions I can related to the Band SDK Preview, I cannot provide official support for this or any other Microsoft product.

Let's get started. If you're following the documentation, the first few steps describe how to connect to the Band, as one would expect. The sample code provided creates a connection within a using block. This ensures that the connection is disposed of as soon as control leaves the block, which is sensible since maintaing a connection to the Band, especially when subscribed to multiple sensors, can severely impact its battery life. However, this approach can also be a bit misleading, because if you do want to collect data from the Band over some interval of time, you will only be able to do so during the lifetime of the IBandClient returned by ConnectAsync. As soon as the IBandClient is disposed of, your connection will be closed and you will no longer receive sensor data.

Since you'll typically only be connected to one Band at a time, the easiest way to solve this problem is to store the IBandClient as a static member variable of your Application class. (Note: I don't guarantee that this is the optimal solution. I am by no means an expert here.) This also makes it easy to access your Band client from different pages within your application, and to unsubscribe from sensor data when you no longer need it so that you won't drain the Band's battery.

[Updated March 17, 2015: In my original post, I was missing a call to Dispose() before setting the IBandClient member to null. As Phil pointed out in the comments, this is a bad practice, because garbage collection of the IBandClient may not happen immediately, in which case the connection to the Band would be left open for some time. During that time, other applications would be prevented from connecting to the Band. I have corrected this error in my code below, and in my project on GitHub. If you are only connecting to the Band for a short time, you can use a using block instead, as discussed above, which will ensure that Dispose() is called for you. Thanks to Phil for the correction.]

:::csharp // App.xaml.cs using Microsoft.Band;

namespace BandDemo { public sealed partial class App : Application { public static IBandClient BandClient { get; set; }

// ...

private async void OnSuspending(object sender, SuspendingEventArgs e) { var deferral = e.SuspendingOperation.GetDeferral();

if(App.BandClient != null) { // Unsubscribe from sensor data so that we don't drain the Band's battery when // the app is not in use. await App.BandClient.SensorManager.Accelerometer.StopReadingsAsync(); await App.BandClient.SensorManager.Gyroscope.StopReadingsAsync(); await App.BandClient.SensorManager.HeartRate.StopReadingsAsync();

// Call Dispose to close the connect to the Band App.BandClient.Dispose();

// We are done with this client, so assign null to the member variable so the // IBandClient can be garbage collected App.BandClient = null; }

deferral.Complete(); } } }

:::csharp // MainPage.xaml.cs using Microsoft.Band; using Microsoft.Band.Sensors;

namespace BandDemo { public sealed partial class MainPage : Page {

// ...

protected async override void OnNavigatedTo(NavigationEventArgs e) { IBandInfo[] pairedBands = await BandClientManager.Instance.GetBandsAsync();

try { App.BandClient = await BandClientManager.Instance.ConnectAsync(pairedBands[0]);

// Do work after successful connect System.Diagnostics.Debug.WriteLine("Connected!"); } catch(Exception) { System.Diagnostics.Debug.WriteLine("Connection failed!"); } } } }

The next thing you'll notice in the documentation is that it recommends that you query the device for available reporting intervals for each sensor. It's fairly easy to miss that setting the reporting interval is completely optional; you can feel free to skip this step unless you really need fine-grained control over how frequently you get updates.

Finally, the documentation describes how to subscribe to sensor data by adding your callback to a sensor's ReadingChanged property. When your callback is executed, you might want to update your app's UI to indicate the new reading (indeed, the documentation itself suggests this). However, this is another gotcha because in Windows (Phone) Store apps, "all UI components share the same [UI] thread," (reference), and methods on this thread cannot be directly invoked by background threads. If you try to do so, an exception will be raised. This will of course be no surprise to those of you already familiar with Store apps (or even with .NET circa 2006), but for the rest of us, it can be a bit confusing. Luckily, the solution is simple: you just need to use Dispatcher.RunAsync to do your UI updates. So, putting it all together, here's how you subscribe to sensor data and update the UI when it changes:

:::csharp App.BandClient.SensorManager.Accelerometer.ReadingChanged += async (sender, args) => { await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () => { AccelerometerXValue.Text = args.SensorReading.AccelerationX.ToString("###00.00"); AccelerometerYValue.Text = args.SensorReading.AccelerationY.ToString("###00.00"); AccelerometerZValue.Text = args.SensorReading.AccelerationZ.ToString("###00.00");

System.Diagnostics.Debug.WriteLine("Got accelerometer event!"); }); };

await App.BandClient.SensorManager.Accelerometer.StartReadingsAsync();

That's it for now! If you have any questions, leave a comment and I'll do my best to answer them.

Windows Hello, Microsoft Passport, and Tamriel Unlimited

2015-03-18T06:28:10Z

Lots of exciting news today! First off, Microsoft has made a couple exciting announcements about Windows 10. Full disclosure: I am a Microsoft employee, so I am posting these for self-interested reasons, but I'm also genuinely really excited about what they mean for Windows.

The two new features are called Windows Hello and Microsoft Passport. Windows Hello is a software feature which, with supported hardware, offers "biometric authentication which can provide instant access to your Windows 10 devices." But Windows Hello is not just the same old support for fingerprint readers that Windows has had for many years now; it "enables you to authenticate applications, enterprise content, and even certain online experiences without a password being stored on your device or in a network server at all."

To accomplish that, Windows Hello works in conjunction with Microsoft Passport, which enables you to "securely authenticate to applications, websites and networks [...] without sending up a password. Thus, there is no shared password stored on their servers for a hacker to potentially compromise." You can read the linked article for the full details, but if you just want the overview, watch this video:

Second, as of today, The Elder Scrolls Online: Tamriel Unlimited has launched, meaning that ESO no longer requires a subscription to play. You'll still have to buy the base game, but after that, no subscription is required to log in. I previously wrote a very favorable review of The Elder Scrolls Online, and while I haven't had much time to play it over the past few months, I do still log in occasionally, and I still enjoy the game. I'm hoping this change to the payment model will bring in new players, because at this point the game's biggest problem seems to be long dungeon queue times; I can only assume this is due to a population that has dwindled since launch almost a year ago.

Truth be told, I have mixed feelings about Tamriel Unlimited. I think there are significant merits to the subscription-based model that many players don't recognize, and the progress that Zenimax Online Studios has made since ESO's release is a testament to that: they've released six significant content patches with features including a new zone, multiple new dungeons and Trials (think raids), new facial and combat animations, a new endgame character advancement system and drastic rebalancing, a justice system, and dozens more improvements. I worry that the focus will now shift from gameplay enhancements to purchasable cosmetic items, because the latter may be more of a cash cow for the company. I also have more general concerns about the continuing trend of people expecting everything on the Internet to be free... but that's a topic for another post.

That said, I'm willing to give Tamriel Unlimited the benefit of the doubt. It could be that this payment model change is just what the game needs to bring it the market success I've always felt it deserved. If you played ESO previously but have since quit, it's well worth picking up again; if you've never played before, I would sincerely urge you to give it a try. Either way, take a look at the launch trailer below, which shows off some of the enhancements that have been made since the game's release:

Game Review: Bloodborne

2015-05-22T08:27:44Z

[I completed this review after playing Bloodborne for approximately 30 hours. I have finished the game, but I have not discovered 100% of the content. I played the entire game solo, not using any summons to help with boss fights. I have not yet participated in any PvP.]

Bloodborne is the spiritual successor to From Software's Demons Souls and Dark Souls series, and its pedigree is clear right out of the gate. Upon starting the game, you'll enter the character creation menu and be asked to choose from a number of classes with different stats, but will be given almost no explanation of what the stats mean or how they affect gameplay. If you're a Dark Souls veteran, you can make some educated guesses, but new players will be completely on their own. After a short introductory sequence, you're thrown into an open world replete with buildings of impossible scale, which stretches vast, seemingly endless distances in each direction. You're completely on your own to make your path through this world, and there's no obvious "correct" route; in fact, a friend and I found different bosses first, because we had taken different routes from the start.

If this experience sounds like it might be frustrating for new players, that's because it absolutely can be. Even for veterans of the series, the complete lack of guidance can feel punishing at times. And yet, I can't stress how much I absolutely love it. The freedom not only to explore, but to truly discover the workings of the world, and to make mistakes, is what made the Souls series what it was, and Bloodborne continues this legacy. The game pays homage to times past, when games didn't hold the player's hand throughout the entire experience, and it does so with such expertise and craftsmanship that none of its contemporaries can compare.

Like its predecessors, Bloodborne adeptly balances challenging and sometimes frustrating gameplay with frequent rewards and other powerful motivators: the prospect of discovering something new, of overcoming the next challenging boss, of finding the next area, or of glimpsing another thread of the world's deep backstory. These enticements work so well because of the depth and imagination with which the world is crafted. There's a constant sense of cataclysmic urgency, but you won't truly understand it until you take the time to piece together all of the subtle clues placed throughout the world. Bloodborne gives you very little to go on, but it trusts you to have the fortitude and curiosity to work things out for yourself.

Adding to the game's excellent ambiance is its soundtrack, which is absolutely sublime. The Souls series has always had excellent music, but Bloodborne sets a new standard in terms of how evocative the music is, and how harmonious it is with the world and gameplay. A few prime examples are the tracks "Micolash, Nightmare Host", and "Lullaby for Mergo". They're excellent pieces in their own right, but if you've not yet completed the game, I would recommend against listening to them out of context. Both are so perfectly unsettling and disconcerting; they're exact thematic matches for the scenes in which they're used. And then there's my favorite track, "The First Hunter," which is beautiful and melancholy, and paradoxically manages to create the perfect mood by being just so incongruous with its place in the game.

There's a lot to like about Bloodborne. Unfortunately, there are also a few respects in which I feel it falls short of the mark set by Dark Souls and Dark Souls II. First and foremost, the fact that it's exclusive to the PS4 is a huge disappointment. I should disclose, as always, that I work for Microsoft, so my opinion here is obviously colored by that. But console exclusives in general are bad for consumers; they force a choice between spending significant additional cash, or missing out on some of the best games the industry has to offer. Some people will never play Bloodborne simply because the monetary bar for entry is very high if the PS4 is not already their console of choice, and that's a shame.

Perhaps of more concern to those players who already have a PS4, though, is that Bloodborne doesn't even run particularly well on the system. It looks great, to be sure, and the performance issues are by no means intolerable or game-breaking—but they're there. Some areas and graphical effects consistently cause serious stuttering; there are occasional hitches in the sound; and some textures and models are surprisingly low-definition for a game that should be a showcase for Sony's next-gen platform.

Even worse, load times were absolutely atrocious when I first started playing, sometimes taking upwards of 40 seconds. The most punishing aspect of each death became not the potential loss of blood echoes (souls in Dark Souls parlance) or need to backtrack, but the tedium of having to sit through another 30-plus seconds of load screen. The load screens completely killed any sense of flow, and were a serious detriment to my overall enjoyment of the game. The inconvenience they cause is exacerbated by the fact that you can no longer heal at lanterns (bonfires) without traveling to the Hunter's Dream and incurring yet another load screen. Recent patches seemed to have reduced load times to more tolerable levels, but it could also be that I've just become used to the long load times. They're still longer than ideal, to say the least.

In short, Bloodborne should have been available on PC. Despite From Software's initial reluctance to work with the platform and the shoddy port that Dark Souls received, the PC versions of Dark Souls and especially Dark Souls II were the definitive editions, with higher and more consistent framerates, support for multiple screen resolutions, and free online gameplay. The same could no doubt have been said of Bloodborne had it been released on PC, but alas, it looks like this one is bound to stay a Sony exclusive.

As I mentioned previously, the world of Bloodborne is crafted with a level of care, imagination, and depth that most other games can't even approach. That said, I get the sense that the extra fidelity that the Sony's new platform allowed came at a cost in terms of environmental variety. This is disappointing, because exploring From Software's meticulously constructed and darkly beautiful worlds has always been one of my favorite aspects of the Souls series. Dark Souls had decaying, overgrown castles, moonlit forests, fantastic medieval cityscapes, caverns of shining crystals, fiery magma pits, poisonous swamps, and labyrinthine walkways and hovels spanning impossibly large chasms. Dark Souls II followed this up with a similarly impressive set of locales, especially if you count the DLC packages, all three of which were excellent. Bloodborne, by contrast, has only a handful of environment types. The vast majority of the game takes place in a twisted and gruesome caricature of Victorian England. There are maybe half a dozen other environments, and all of them are are extremely well imagined and rendered, but I found myself wishing for the more of the variety that the Souls games offered.

This complaint also extends to the available weaponry. Dark Souls and Dark Souls II had dozens upon dozens of different weapons across roughly two dozen different weapon classes. Each class of weapon had its own characteristics, animations and gameplay style. During a single playthrough of each game, one was bound to discover many of these weapons, and miss still many more which were hidden or rare. The huge number of distinct weapons with unique characteristics made it very likely that you'd switch out your weapon at least a few times during a playthrough as you discovered new ones that worked better with your playstyle or attribute allocation. In Bloodborne, I discovered only a handful of different weapons, and none of them gave me a compelling reason to give up the weapon I'd started the game with. I'm know there are others that I have not yet discovered, but given that I finished the game and completed a number of optional areas, it's disappointing that I encountered so little in the way of unique weaponry.

In general, choices and variety seem much more limited in Bloodborne. Another example of this is the covenant system; whereas Dark Souls II had maybe a dozen or so covenants, Bloodborne has only three, and they're all more or less the same. Perhaps the best example, though, are the new procedurally generated "chalice dungeons." The first few chalice dungeons you explore will be novel and exciting. They were especially so for me as a software developer, because it was interesting to think about how From Software might have gone about designing the system, which produces a good variety of layouts while doing a respectable job of maintaining the illusion that there might have been some reason or purpose in each dungeon's design. But after those first few dungeons, the experience begins to wear thin: although there are practically infinite unique layouts to explore, as far as I can tell (having tried two of the four different chalices), there's really only one tile-set. Thus, the dungeons quickly begin to feel samey and boring. There doesn't seem to be a great incentive to explore them from a story or gameplay perspective either: they are purely optional and don't appear to provide much in the way of unique loot, so what's the point?

The idea with chalice dungeons, I think, was to add an extra layer of replayability to Bloodborne. But the effort seems to have backfired. The Souls games have always had very strong speedrun and PvP communities, which kept the games alive years after their original releases. I expect speedrunners will largely if not entirely ignore the chalice dungeons, as they're not mandatory, don't offer especially good loot, and being pseudo-random, can't be made part of a route or considered objectives for any competitive runs. The fact that Bloodborne is shorter overall doesn't help either. That's not to say the game is lacking in content, as it most certainly is not. But whereas it took almost a year for the world record Dark Souls II speedrun time to drop below 50 minutes, casual speedrunners are already matching that time in Bloodborne. I don't think that bodes well for the game's longevity. Similarly, while I'm not much of a Souls PvP player, I think the PvP community will be hurt by the relative dearth of weaponry and the homogenization of playstyles relative to Dark Souls II.

Finally, the difficulty curve in Bloodborne seems suboptimal. For me, the early game felt much more difficult the later half. To some extent, this is probably a natural consequence of me learning the new game mechanics and getting better overall as I played more, but I also felt that some of the early boss fights were more challenging to learn and less forgiving than some of the later fights. There were at least a couple late-game fights that I beat on the first try, which I don't think ever happened to me before in a Souls game. (I'll be honest, I'm not especially good.)

There are clear improvements to be found as well, though. The addition of a dedicated button for healing is huge; never again will you miss a crucial heal because you're fumbling around with the item selector. This is just one of a host of changes, including new animations and a simplified armor system, designed to make combat quicker and award aggression more. Some of these changes will be positive or negative depending on your specific tastes and playstyle, but personally I love pretty much all of them. While there's an undeniable satisfaction to mastering the slower-paced, more methodical and strategic combat of the Souls games, Bloodborne's emphasis on agility distinguishes it from its predecessors even though it is actually very similar to them overall. I don't think that the Dark Souls formula was getting stale, but Bloodborne makes everything fresh again, and the value in that can't be overstated in an age of yearly franchises which continue to be churned out like clockwork, long past their primes.

There are also some changes to cooperative and competitive play that will be welcomed by the community. The much-maligned Soul Memory system from Dark Souls II is gone, which will make it easier to cooperate with friends and find PvP opponents. At least for the time being, healing is allowed by invaders, which should finally put an end to the heretofore constant moaning about dishonorable "chuggers." It does seem a bit unfair to PvE players who get invaded, though; without the advantage of being exclusively able to heal, they'll be even easier targets for dedicated PvPers. (Side note: I didn't get invaded a single time during my playthrough, even though I was online the whole time. I have no idea why this was the case.)

Finally, although I previously complained about its presumed impact on environmental variety, the fidelity with which the world was imagined and realized is remarkable to behold. This is another thing that Souls titles have always been excellent at, but in the past they were built for outdated hardware and running on underpowered game engines. Bloodborne takes advantage of newer technology to bring forth a new level of artistry. Every surface is packed with intricate detail, but not to the extent of being distracting or overwhelming; every object and building is thoughtfully designed to evoke a Gothic-Victorian horrorscape. And horrifying it is: nobody does grotesque, oppressive creepiness like From Software.

I spent a lot of text discussing the shortcomings that I feel Bloodborne has relative to its predecessors. I think those statements are reasonable and accurate; I do feel that some aspects of the game are a step backwards. Bloodborne isn't quite the follow-up to Dark Souls II that I had dreamed of. Whereas Dark Souls and Dark Souls II were damn near perfect, Bloodborne is a fine work of art, but not quite a masterpiece.

But not everything has to be a masterpiece. I want to make it very clear that despite its flaws, I absolutely loved Bloodborne, and I can't wait to play through it again. It's a hell of an experience and will easily be a contender for the best game of 2015, despite how early in the year it was released. If there's one thing you should take away from this review, it's this: I spent more than $450 just to play Bloodborne, and I'd do it again in a heartbeat.

Windows 10!

2015-07-29T07:00:00Z

Windows 10 is out today! This is the first release of Windows that I have worked on as a developer , and I'm proud of what my team and Microsoft as a whole have accomplished in the past year-and-a-half or so. Several tech websites have already posted reviews, and so far the trend is very positive. A colleague of mine posted some excerpts, which I'm going to borrow:

Windows 10 represents the obvious future of PC operating systems. It makes Mac OS X feel old-fashioned, stuck in a time where The Desktop was a thing that mattered and the only way to access the Internet was through a browser.

David Pierce Wired

So the question finally presents itself: should you upgrade your machine to Windows 10? Yes, you should.

Brad Sams Neowin

The best part of Windows 10 is that it ends the cycle of good and bad in favor of something great.

Tom Warren The Verge

It's nice, for once, to be able to recommend a new version of Windows without any hesitation.

Devindra Hardawar Engadget

If you are on a PC at all, Windows 10 is where you want to go. That it is free—for now—only makes the point simpler.

Alex Wilhelm Techcrunch

If you haven't already done so, you can follow the instructions here to get your free upgrade. It's very exciting to have played a role in the creation of the latest and truly greatest version of Windows; I think this is a huge achievement for Microsoft, and I'm happy to have been a part of it. The real reward, though, is seeing people use and enjoy Windows 10—and that starts today!

Observations from Three Years in America

2015-09-07T22:44:23Z

Some of this post is adapted from a Facebook comment I wrote earlier in the year in response to this CBC article. Many of the statements in this post are conjecture based on my own experiences and perceptions; I do not claim that everything in it is broadly applicable.

May of 2015 marked the beginning of my fourth year in the United States. Now that I have a green card and don't have to worry quite as much about saying the wrong thing and getting kicked out of the country, I wanted to take the opportunity to discuss what it's like to live here from my perspective. I was going to write "from a Canadian perspective," but of course, I don't speak for all Canadian immigrants.

Before I started my career, I had never really planned to come to the US. I was only part of the way through my senior year at the University of Alberta when I happened upon a Microsoft recruiting booth at a career fair. Microsoft was the only real technology company present; most of the others were oil and gas companies, that being what Alberta's economy is primarily based on. I left a resume with them, not really expecting much to come of it.

As it turned out, I did get a call back (which I almost missed—funny how such small events can result in drastic changes in one's life). A few weeks and several highly stressful interviews later, I had signed a job offer. Shortly after I graduated, my wife and I were on a plane to Seattle. Even at that point, I didn't really expect to live in the States for more than a year or so. My goal was primarily to get a "marquee company" on my resume (a piece of advice a friend and former colleague gave me) and then use that as a bargaining chip to get a better job back home. But now it's been more than three years, and not only am I still here, but I no longer have plans to leave in the foreseeable future.

Immigration

I guess it would make sense for me to start with immigration. This could easily be the subject of its own post, but I always find some of the rhetoric in the H-1B debate perplexing. Suffice it to say it is neither easy nor cheap to bring a foreign worker into the US on an H-1B visa. The assumption, then, that companies are using low-paid H-1B workers to replace American-born workers doesn't make economic sense to me. I won't claim that no companies ever do this, but I very much doubt that it's as widespread as it's sometimes made out to be. Getting an employment-based green card is more difficult still.

I think it's probably a minority of Americans who really have a problem with immigration, but they are a vocal minority. And honestly, I don't begrudge anyone their desire to be selective about who is allowed to enter their home. I think immigration raises some difficult questions even if one is not a nativist (and I am not). People on both sides speak as though they have absolute certainty about what effects immigration will have, but in my experience, it's hard to find solid, unbiased empirical data.

Nevertheless, my belief based on the data I have seen is that protectionist policies will not allow the States to continue to be competitive in the global economy. I don't think it's a stretch to say that the US economy benefits massively from the "brain drain" that other countries experience: students get a government-subsidized education at home and then move to the US, depriving their home countries of cognitive capital, entrepreneurship, and tax revenue. (Obviously, I am part of this problem.) It seems self-evident to say that without this influx of talent, the US would not be as successful as it is, especially given that educational achievement here falls below the OECD average and has been trending downwards.

I live in a special sort of bubble in Redmond—there are probably more immigrants here than non-immigrants. Even among the non-immigrants, most are "transplants" from other states. There's sort of a unique character to a community which most people have joined by choice rather than by accident of birth—a real understanding of the tradeoffs of living in this particular place, and a worldliness you don't always get in other communities. Of course, much of this is probably due to people here generally being financially comfortable and well-educated, so I hesitate to draw too many conclusions. Whatever it is, it's a good thing; I've met so many people here who are so much wiser and more knowledgeable than I am, and that can only help me improve.

Healthcare

I've seen more of the American healthcare system than I would have cared to over the past three years, but this has given me a fairly good basis to compare against my experience with healthcare in Canada. Non-emergency care is definitely much faster here, if you can afford it. Of course, that's a big if; many Americans simply can't. If I had to choose which country to live in on this basis alone without knowing whether I'd be poor or rich, I'd definitely choose Canada; it has a much better social safety net. However, I have to admit (somewhat ashamedly) that the quality of the system in general all seems very hypothetical when you're sick and you want to see a doctor as soon as possible. In that respect, the American system is enticing.

It's often said that the United States has "the best hospitals in the world," at least with respect to the treatment of rare and/or serious illnesses. I don't know if this is true. Thankfully, I don't have any experience in those matters. It certainly does seem to be true that the United States is a world leader in medical research and pharmaceutical development, though, and that's nothing to sneeze at.

I want to make sure I'm clear here, though: I'm still not endorsing privatized healthcare over public. Data are always more important than personal perception, and the data here are very clear. Other developed countries have similar or better outcomes and are able to achieve those outcomes for less money. Still, my experience with the American system has been very good so far.

Gun Violence

There are days where I shake my head and wonder why I choose to live in the US. Usually, these are days on which I read about yet another mass shooting. The amount of gun violence in this country, while not something I worry about on a daily basis, is truly astounding. As of today, there has been an average of more than one mass shooting per day in 2015 (defined as four or more people shot in one event). 2013 fell just short of an average of one per day with 363 mass shootings throughout the year. Of course, these mass shootings represent a small percentage of overall violent crime in the United States, but they're especially disturbing because the victims are usually innocent people with no connection to the shooter. There seems to be little political will to make any serious attempt at fixing this problem. To quote The Economist, "Those who live in America, or visit it, might do best to regard [mass shootings] the way one regards air pollution in China: an endemic local health hazard which, for deep-rooted cultural, social, economic and political reasons, the country is incapable of addressing. This may, however, be a bit unfair. China seems to be making progress on pollution."

To be honest, I'm not entirely sure what to make of the gun control debate myself. On the one hand, research seems to suggest that more guns lead to more homicide, and that gun control measures have a strong negative correlation with gun deaths (even stronger than other factors such as mental illness which are often blamed). On the other hand, at least one analysis indicates that more guns lead to less crime overall. I haven't yet read all of the research myself, and I'm not sure I have the statistical background to make a competent determination of the quality of each study. That said, it seems blindingly obvious to me that more guns would lead to more homicide even if they did in fact reduce the rate of other types of crime. That tradeoff does not seem worthwhile to me.

In any case, as 3D printing technology continues to improve and become cheaper, it seems that it may only be a matter of time before gun control laws become significantly more difficult to enforce—not just in the US, but in every country. If that happens, the whole question may be moot, although cultural factors could still play an important role in moderating gun violence. Whatever the cause is, the US is an outlier among developed countries when it comes to gun violence, and I don't foresee that changing any time soon.

Economics

Many things here are cheaper than they are in Canada. Gas is a big one. People in the States complain about gas prices just like everyone else does, but gas here is much cheaper than it is in most other countries. Even when the Canadian dollar was at parity with the US dollar, I was able to get a tank of premium here for less than the cost of a tank of regular back in Canada.

There are a lot of services available here that either aren't available in Canada or aren't as good—for example, Amazon Prime, Amazon Fresh, and Netflix. Canada has some of these services, but for Netflix in particular, the content isn't nearly as good. In general, it's much cheaper and faster to get things delivered here; I think this is both due to greater population density and better transportation infrastructure. The interstate system is awesome; I didn't realize until I lived in the US how laughably bad some parts of the Trans-Canada highway are considering that it's Canada's only cross-country highway.

Taxes vary between states and provinces, so I'm not sure I can definitively say that taxes are lower here than in Canada. Washington has no state income tax, and the federal tax rates are lower, so I'm definitely paying less here than I would be in Canada. Of course, all of the economic benefits of living in the States have to be balanced against the fact that there are greater levels of wealth and income inequality here. This has a lot of negative consequences (including higher levels of crime).

Politics

I'm not sure if Americans are more politically engaged than Canadians or if it just seems that way because politics here is more polarized, leading to angrier, louder arguments. In one sense it's hard not to be politically engaged when you're in a never-ending election cycle, as the US seems to be. The presidential election won't be held until November of 2016, and yet the Republican and Democratic primaries already dominate the news. And when there's no presidential election on the horizon, there's always a mid-term election no more than two years away, not to mention state and local elections. By contrast, this year's Canadian election campaign, which started in early August and is expected to run until mid-October, will be the longest campaign since 1872.

The candidates for the 2016 presidential election don't inspire confidence. Things could still change before the election is held, but right now it looks like it's going to be one war-loving, civil liberties-hating neo-con (Hillary) versus another (whoever ends up winning the Republican nomination). I'm not even going to mention you-know-who. It seems to be a common sentiment among Americans that government in the country is badly broken. This is not new, but even so, I suspect the current election cycle is profoundly embarrassing to liberals, moderates, and sensible Republicans alike.

That said, Canada's political system has its own faults. Like the US, it uses a horribly non-representational "first past the post" voting system, but the problem is exacerbated by the fact that there are actually multiple viable parties. In the last election, the Conservative party ended up with a majority government despite 62% of the populace voting for more left-leaning parties. I'm hopeful that the NDP or Liberals will win this election; both have promised to revise the electoral system to make it more representative. (Plus, I don't like the Harper government or their disastrous policies on the environment, science, and the economy.) It's a very close race at this point, though, and the Liberals are apparently not open to forming a coalition with the NDP should the Conservatives win again.

There are also certain aspects of the American political and legal system that I prefer. I like the idea of states as "laboratories of democracy" with broader control over local legislation. There are downsides to this as well—it can lead to negative outcomes vis-à-vis civil rights, and when state and federal laws contradict each other it can result in confusing and volatile situations. Nevertheless, I think it's generally a good thing that states have some ability to catalyze progress in the country by demonstrating the viability of legislative changes that the rest of the country isn't yet on board with.

I also really appreciate the commitment to freedom of speech in the United States. The first amendment, which protects all kinds of speech (including "hate speech"), is venerated here. I think this is a wonderful; in fact, it's possibly one of my favorite things about the country. The US has, to my knowledge, the strongest legal protection for free speech of any country on Earth. Canada, meanwhile, has "human rights tribunals" (perhaps better known as kangaroo courts) which have been known to fine comedians for making offensive jokes.

Culture

It's pretty much the same as in Canada.

Okay, not entirely. One of the first things I learned about living in America is that, culturally, the country isn't actually all that unified. Sure, there's a federal government and a shared constitution and whatnot, but there are some pretty marked differences in attitudes and customs between states. Texas is far more different from Washington state than Washington is from British Columbia. California is different from both Texas and Washington. Michigan is different still—you get the picture. In some respects, visiting a new state is almost like going to another country.

One thing that's always struck me as paradoxical about the United States is the country's religiosity. To quote Richard Dawkins, it is surprising that the US, "founded in secularism, is so much more religious than those western European countries that have an official state religion, like Scandinavia and Britain." The constitution states that "no religious Test shall ever be required as a Qualification to any Office or public Trust under the United States." The First Amendment further specifies that "Congress shall make no law respecting an establishment of religion, or prohibiting the free exercise thereof..." And yet, in practice, religion and politics are more intertwined in the US than they are in most other developed countries. As of 2012, only 54% of Americans said they would vote for a well-qualified presidential candidate who happened to be an atheist. "In God We Trust" is on the currency (since 1956), and politicians are constantly talking about God; a significant minority would doubtless be offended if a major politician were to end a speech without saying "God bless America." In Canada, to my recollection, politicians simply don't mention religion most of the time.

There are other paradoxes, as well. Politics, as I mentioned, seems to be all-encompassing, and yet is a taboo subject in polite company. Free speech is a foundational value, but American colleges seem as if they're becoming world leaders in trying to coddle students and protect them from "dangerous" ideas. The "war on drugs" has been more thorough and brutal here than in most developed countries, and yet America is one of only a handful of countries which have legalized marijuana (albeit only in some states).

Individualism is considered much more important in the States than in other developed countries, which tend to be more socialist. This, too, has downsides; the virtue of personal responsibility is preached, often hypocritically, with a callous disregard for the immense inertia the lower classes have to overcome to be successful. Nevertheless, I'm glad the ideal exists. Other countries often go too far in trying to protect citizens from everything that could possibly harm them, regardless of whether or not anyone consents to the "protection." The United States has its own overreaches of this sort, of course, but it's still better than most. For instance, it would be almost unthinkable difficult and unpopular[^censorship] to censor broad swaths of the Internet here, as is done in the UK. I also think the American attitude toward individual freedoms (along with the autonomy of states that I mentioned above) can do a lot of good; I suspect it's part of the reason the country is slowly moving toward more progressive drug policies.

Global Influence

The United States has reshaped the world many times over in the past few centuries. Needless to say, this hasn't always been for the better, but it's worth reflecting on all of the remarkable innovations that have come out of the States, and how different the world would be without them. This is especially pertinent to me, because my entire field was essentially invented here. I don't want to downplay the important contributions of other countries, and many American innovations are actually innovations by American immigrants. Still, the transistor, the microprocessor, the graphical user interface, Unix, most successful programming languages, and countless other advancements were made in America. This, I think, is a large part of the reason that the United States still dominates the information technology market today.

As I mentioned above, my immigration to the US was largely not planned, so saying that I was drawn here by some particular thing would likely just be rationalizing the decision after the fact. But I think the culture of innovation is what has kept me here. I want to work on something with global reach—something that might, perhaps, have an impact on people's lives. Especially for a technologist, there is no better place to do that than the United States.

It took me a long time to get around to writing this post, so now I'm already a third of the way into my fourth year in the States. The US isn't perfect, but I've been incredibly fortunate to have had the opportunities I've been given, and I'm grateful to the country and particularly to my friends and colleagues in Redmond for making me feel welcome here. I'm happy to be here.

[^censorship]: I originally wrote that it would be "almost unthinkable" to censor the Internet in the US, but a friend pointed out that continuous attempts to pass legislation like SOPA contradicts this. Copyright and patent laws are one of my least favorite things about the US, and one of the spheres in which I think the US has a significant negative influence on other countries. That's a subject for another post, though.

TV Series Review: Mr. Robot, Season One

2015-09-25T08:19:00Z

I don't watch a whole lot of TV. In fact, I don't even have cable. The only shows I end up watching, therefore, are those I'm introduced to or I read about. Mr. Robot is the latest such show; I read David Auerbach's piece on it and was intrigued. In Auerbach's words:

Real misfits—disturbing, unappealing, unlovable misfits—make for poor Hollywood protagonists. It takes some faith to inject a genuinely alienating hero into a film or TV show. But ever since the broader world of tech has gained a whiff of cool over the past decade or so, Hollywood's portrayals of hackers, programmers, and other screen-strapped geeks have tended to sugarcoat the frequently unvarnished and even unappealing aspects of a culture that prides itself on not caring about the mainstream. I'm not bemoaning the lack of shy nerds with hearts of gold here, but of awkward and even abrasive characters who talk too loudly, can't follow conversational cues, transgress boundaries obliviously, and harbor conspiracy theories or other unappealing ideas—like a lot of actual hackers out there. We forget that people detest savant types for reasons beyond their smarts, and a lack of social graces is one of them.

Elliot, the protagonist of the new USA Network drama Mr. Robot, comes far closer to this type than an American television show has dared for a while. (Elliot isn't the titular Mr. Robot—that's Christian Slater.) As played by Rami Malek, Elliot doesn't have the charisma of Benedict Cumberbatch's Sherlock or the anti-charisma of Hugh Laurie's Dr. House. Unlike Tony Shalhoub's Adrian Monk, he's not a collection of quivering nerves and endearing eccentricities. These characters—all nerds of one sort or another—are fundamentally capable of functioning in society, and Elliot really isn't. He and the world just don't hook up in the right way. Taking copious drugs in order to function, he carries a vacant and flattened affect, neither making eye contact with anyone nor turning away when they look at him. With his emaciated face and sunken eyes, Malek gives an impression of emptiness, paranoia, and above all disconnection.

This is probably a polarizing description: some people will be immediately interested while others will probably question why anyone would want to watch it. It was very much the former for me, possibly because Auerbach's description of Elliot hits a bit close to home. I watched the first episode and was instantly hooked. The show immediately draws you into its world: a sort of dystopian present, a mirror image of our own reality, but viewed through the eyes of a brilliant, cynical, emotionally troubled young man who struggles not only with an unfeeling universe but also with drug addiction and mental illness.

Atypically for movies and TV, Elliot's mental illness is depicted respectfully and at least somewhat accurately, too. He's not an over-the-top Hollywood psycho, but merely a vulnerable person suffering from paranoia and delusions. And being a software developer, I also really appreciated that the show nails the technology. It's not completely realistic, but that's to be expected; the reality of programming and hacking is that they're slow, analytical processes, often consisting of long stretches of thought punctuated by short bursts of activity. In other words, they're boring to watch, and sometimes to undertake, too. Mr. Robot strips that away, but it keeps the interesting aspects intact without insulting the audience's intelligence with impossible technology, meaningless invented jargon, or awful 3D GUIs.

The overarching social commentary in Mr. Robot is also captivating. It's not especially original, nor remarkably insightful, but it's executed marvelously; because everything is presented from Elliot's point of view, it's very hard not to take an empathetic attitude to his perception. This is also true of the mood of specific scenes: the intimacy between the viewer and Elliot is an extremely effective tool for creating suspense, or sadness, or whatever else the show wants you to feel, because you're so invested in him.

It helps that the show's production quality is excellent as well. The cinematography is excellent, the music is superb and perfectly matched to each scene, and the acting is stellar—not only from star Rami Malek, but also from his co-stars, who include Christian Slater, Portia Doubleday, Carly Chaikin, and others. Not all of them have the name recognition of Slater, but every one is excellent in his or her role, which is far more important. (Besides, it's nice to see some new faces.)

But unfortunately, it's not all good news. The first seven or so episodes of the series are all-around excellent, but after that... well, first a warning: major spoilers begin below here. Scroll down to continue reading.

...after that, shit gets crazy. Literally.

From the beginning of the series, Elliot is clearly an unreliable narrator. The presentation of the show is not "objective" in the sense that we see the majority of it through the filter of Elliot's biases, but beyond that, the extent of his mental illness isn't really clear. Is he just socially anxious and a bit paranoi,d or is he actually suffering from hallucinations? What else is he hiding from us? Up through episode seven, we don't really know.

But then, in episode eight, Elliot has a complete psychotic break. It is revealed that Mr. Robot never actually existed, at least not as we know him. Rather, he is a dissociated personality of Elliot's, and not only that, his appearance and behavior are projected from repressed memories of Elliot's dead father. I'm not a psychologist, but this is implausible to me to the extent that it makes suspension of disbelief becomes very difficult.

I wrote above that the show's portrayal of Elliot's mental illness was respectful and realistic in contrast to what Hollywood typically does. That was what I thought until episode eight. Post-episode eight, Elliot is the archetype of ridiculous Hollywood insanity. His dissociative identity disorder is straight out of Fight Club (and so, for that matter, are certain elements of Mr. Robot's plot); I'm not sure how many times that specific trope has been used, but it already feels tired and overdone.

There's a scene in episode ten where Elliot gets in a fight with Mr. Robot, and the camera cuts between what he's hallucinating and what's really happening; in the latter, we see him choking himself up against a wall while people look on in shock and confusion. The scene is just ridiculous—I honestly wasn't sure whether it was supposed to be sad, or dramatic, or comical.

To be fair, although I wasn't impressed with the over-the-top portrayl of Elliot's hallucinations, the show does a good job of making it plausible that Mr. Robot never actually existed separately from Elliot. If you watch closely, you notice that other characters never address Elliot and Mr. Robot at the same time, and when the two of them are talking, nobody else ever takes notice. (Some viewers caught on to this from the beginning and thus saw the twist coming. I didn't, so I had to go back and double-check.) And although the ability that Elliot apparently develops to summon Mr. Robot at will is also absurd, I'm glad that Slater's character won't be disappearing. He does an excellent job in the role, and he's a big part of what makes the show so as good as it is.

Some scenes, though, require serious mental gymnastics once we realize that Mr. Robot isn't real. For instance, one wonders why Darlene was so slow to recognize Elliot's psychosis given how differently he must have been acting from day to day. And then there are the scenes in episode five, in which Mr. Robot, Romero, and Mobley all talk to Elliot via radio as he infiltrates Steel Mountain. They don't all talk at once, and the other two don't directly acknowledge Mr. Robot's presence, so presumably we're to believe that Elliot is also just imagining Mr. Robot talking to him in this case. But this feels like a cop out. If almost everything we're being shown is just Elliot's wild hallucinations—even when Elliot isn't in the scene and cannot possibly know what is occuring—it feels like the rug has been pulled out from under us and there's no basis on which to trust the authenticity of anything. That makes the entire plot feel hollow.

As much as I was disappointed by the direction Mr. Robot took towards the end of the first season, it's still a very good show. All of the positive things I wrote before the cut still apply. And, critically, it's managing to advance the plot while maintaing suspense. I never watched Lost, but from everything I've heard it eventually became a disastrous mess of questions piled upon questions, all unanswered, ultimately leaving viewers feeling as if they'd been jerked around by writers who were flying by the seats of their pants. Mr. Robot doesn't do that. The destruction of Evil Corp's records that the whole season was building up to did happen, and in the next season we'll see the reprecussions of that. But there are still many other threads waiting to be explored—will we see more of Fernando Vera? What is Evil Corp's motive in hiring Angela? What happened to Wellick? Why does Whiterose know Price?

I'm looking forward to season two. I just hope it's a bit more grounded—more like the early episodes of season one than the latter half. I'm worried about whether or not Sam Esmail can pull that off given bizarre history he now has to accomodate, but I enjoyed the show enough that I'll give him the benefit of the doubt. If nothing else, the first few episodes of Mr. Robot will always stand out to me as an instant classic.

And we've finally got a successful TV show written by someone who understands computers as more than voodoo boxes capable of literal magic if you can type gibberish fast enough (perhaps by enlisting another pair of hands on the same keyboard). I'll take my victories where I can get them.

Switching to Pelican

2015-10-12T00:00:00Z

I keep forgetting to mention it, but this May was the fifth anniversary of this blog. Thus, it's time again for some housekeeping. Three years ago I stopped using Wordpress for my blog and moved it to Octopress. Octopress is a great blogging framework, but its current incarnation has some major shortcomings. Those will be fixed with Octopress 3.0, but 3.0 has been a long time coming, and due to the changing distribution model the work to move from 2.0 to 3.0 may be substantial. Plus, Octopress isn't the best fit for me because it's written in Ruby, which I don't know and don't have much interest in learning; I much prefer Python.

I recently learned of Pelican, which is a Python-based blogging tool. Luckily, Maurizio Sambati has already done the hard work of porting the Octopress theme to Pelican. Since my blog's theme is the Octopress theme with a few tweaks, it wasn't an inordinate amount of work to port my own theme to Pelican. Jake Vanderplas also did some great work in porting some of Octopress' Liquid tags to Pelican, and he also has a great blog post which walks through the process (and describes some of his reasons for switching, which mirror my own). In the end, it did take me several hours of work to move everything over and clean up some of the posts where I was making use of various Octopress plug-ins, but I think it will be a boon in the long run. (Please let me know if anything appears to be broken, as it's possible that I missed something.)

In addition to being easier for me to use and contribute to, I'm really excited about Pelican's support for tags. Tagging will help organize my blog, especially since I've limited myself to having only a few categories since I'm committed to having a unique, easily-distinguishable color for each one. I'm also excited that Pelican has built-in support for translating articles, which I plan to take advantage of to practice the languages I'm learning.

Check back in three years to see which new blogging platform I switch to next!

Game Review: Fallout 4

2015-12-20T00:00:00Z

[I completed this review after playing Fallout 4 for a total of 25 hours. I completed the main quest line during my play through.]

Fallout 4 is the latest in Bethesda's revival of Black Isle Studios' beloved post-apocalyptic role playing series, and like its predecessors it has met with much praise and market success. But I find myself unable to quite grasp the popularity of the game, because also like its predecessors, Fallout 4's impressive scale and ambitious design are hamstrung by some major flaws in the execution.

I've really come to dislike the aesthetic of the Fallout series, and I'm disappointed that it hasn't really evolved at all over the years; the D.C. of Fallout 3 is not very distinct from New Vegas, and even less so from Boston in Fallout 4. Does everything have to be so drab and gray and dirty, so dead and devoid of plant life? I get that the world is supposed to be a wasteland shattered by nuclear warfare, but I don't think this design choice was made for the sake of realism (and I don't think it is realistic anyhow). More likely, Bethesda is taking design cues from the original series. But when so much of the game has evolved, the dull environment hardly seems worth hanging onto. Furthermore, I can't help but feel that Bethesda has used this theme of dilapidation as an excuse to cut some corners when designing the world. Little care has been taken to prevent objects from clipping through each other and through the world geometry, and this frequently results in getting stuck on the strewn garbage, or being unable to jump over or crouch under objects that look like they should be navigable.

These are hardly the worst of the bugs you'll encounter, though. About two-thirds of the way through the main quest, I encountered a game-breaking bug where a door didn't unlock as it should have, leaving me stuck in a room I couldn't leave. Luckily, I was able to get around this by turning off clipping, but had I been playing on console I probably would have just been screwed. The PC version isn't without its own problems, though. Performance varies a great deal between different areas, and some scenes—simple indoor scenes, even—can bring relatively powerful hardware to its knees. And, as usual, the menu system was clearly designed for consoles and not changed at all when the game was ported, which makes navigating the menus painfully slow—a big pain point in a game which is so menu-driven.

The plot doesn't fair much better. The game's introduction is boring and clichéd, and things hardly improve from there. The introduction attempts to build emotional attachment and set up the protagonist's motivations, but its rushed nature and predictability make it fall completely flat. Beyond its failure to have the intended emotional effect, the attempt to establish a meaningful backstory for the protagonist is a questionable decision, even in theory, in this sort of game. The appeal of the series is, to a large extent, that it's open ended and non-linear. Some players will follow the main quest line to its conclusion, but many more won't, and for them the one-size-fits-all introduction could hamper immersion by robbing them of the chance to fully construct their own characters.

The tedious introduction is crucial to the plot of the main quest, so I'd be more forgiving if the main quest were really spectacular, but it's not. There are some high points, where mystery and suspense propel the story forward, but the questions raised are always followed by disappointing answers, and the whole thing is constantly hampered by mediocre voice acting. Events that should be heart-wrenching are portrayed with about the level of emotion you'd expect from someone who missed her bus and had to wait 30 minutes for the next one, and life-or-death situations seem to always heed a sort of detached overconfidence, as though the characters all know that it's just a video game. I don't entirely blame the actors, though. Good acting requires a sense of a character's personality and motivations, and most of the characters in Fallout 4 lack both of those aspects. There are a handful of exceptions, such as Nick Valentine, but they can't save the rest of the cast.

It doesn't help that the character models have a very limited set of expressions and animations to visualize their emotions with. Granted, not every game can be The Last of Us, and perhaps it's kind of unfair to even expect that sort of fidelity in an open-world game; it's like complaining that a choose-your-own-adventure book doesn't have the depth of a novel. But Rockstar does a pretty good job with Grand Theft Auto, whereas Bethesda seems to be constantly behind the state-of-the-art, despite everyone always complaining about this in every review of a Bethesda game. At any rate, the combination of shallow, often nonsensical writing and tepid acting result in an uninspiring story, which I expect I'll have forgotten entirely in a matter of months.

On the bright side, combat has improved since Fallout 3. The addition of ironsights is a welcome change; it allows for more precise aiming while also adding a sense of authenticity. It also ties in well with the impressive weapon modification system, which allows you to select from a variety of different sights and scopes, and craft mods that change various characteristics such as fire rate, damage, and accuracy. However, the guns still don't have the feeling of weight and impact that they do in pure FPS games such as Battlefield and Call of Duty. I think this is partly an issue inherent to the RPG-FPS genre—no gun feels powerful when it takes 37 headshots to kill a high-level enemy—but it's also a matter of animation and sound design, and neither are stellar here.

The most significant addition to the game, and in my opinion also the best, is "settlements," which are areas of the map in which you can build your own little oases of safety by placing structures and resources. Once you have set up a settlement, NPCs will travel to it, and you can assign them to produce resources or act as guards, as you see fit. You can scavenge building materials from scrap you find in the settlement area or from junk items found elsewhere in the wasteland, which is a nice touch because it gives these items some real value rather than just having them be "vendor trash" as they are in most RPGs. There are a variety of fixtures that can be constructed with the right components—furniture, water pumps, food crops, generators, turrets, and more. Switches and computer terminals can be used to control the flow of electricity or set behaviors on certain objects, which allows for some interesting creative applications. As your settlement grows, it will attract more friendly NPCs, and you can construct shops and set up trade routes. But prosperous settlements also attract the attention of raiders, necessitating the aforementioned defenses—or, if you're around, you can help defend the settlement yourself.

The settlement system is the most novel aspect of Fallout 4, and also among the most fun. But even so, it feels somewhat aimless and lacks a sense of necessity or challenge. You set up a few beds, plant some food, add turrets for defense, and... then what? My settlements never got raided, or if they did, I didn't notice. There was little incentive for me to build further once I'd established the basics. It's a cool little addition, but if base-building is what you're looking for, there are countless other games that do it better, and nothing unique about Fallout 4's implementation.

Despite my complaints, Fallout 4 isn't a bad game. It has a ton of content‐a whole world to explore, with interesting little surprises all over the place. The many game systems provide numerous ways to play, meaning everyone is likely to find something to like. But folk wisdom about trying to please everyone applies here: it's impossible, and in attempting to do it you typically end up doing many things decently but never doing anything excellently. I'm a bit disappointed in the Fallout series' repeated failure to live up to its potential. These games could be amazing; they have a foot in the door to greatness already. Unfortunately, Bethesda has never quite been able to step across the threshold.

Steam Early Access Roundup

2015-12-27T00:00:00Z

Today I'm previewing three early access titles now available on Steam: Ghost in the Shell Stand Alone Complex First Assault Online, Rust, and Hurtworld.

Ghost in the Shell Stand Alone Complex First Assault Online

...Tactical Combat Video Game Entertainment Simulation System. Okay, I made that last part up, but somebody in Nexon America's marketing department really should have reigned in that title a little bit. It can't even be made into a convenient acronym. "Hey, want to play GITSSACFAO?" asked no one, ever.

Anyhow, I was drawn to this title because I love Ghost in the Shell: Stand Alone Complex (as well as the Ghost in the Shell films). If you're not familiar with the series, it follows a special forces team led by Major Motoko Kusanagi in a sci-fi 2030s Japan where everyone who can afford it is a cyborg. (The overoptimistic timeline is worthy of Back to the Future!) The setting is leveraged to deliver some surprisingly sophisticated philosophical and social commentary—for an anime, anyway. But more to the point, there's also a lot of over-the-top action and a hint of gun-fetishism, which makes the series perfect for a first person shooter. It's surprising, then, that nobody has tried to make one until now.

Unfortunately, the results are mixed, at least at this early stage. I've seen some significant stuttering during some matches, which is a real problem in such a fast-paced shooter. The gunplay also is also ever-so-slightly off. It's got a satisfying, weighty feel to it, but I can't quite get used to the aiming and recoil. Adjusting your aim while looking down the sights is so incredibly slow compared to the pace of player movement that the best strategy seems to be to just never use iron sights at all except at extreme range. It feels like they're taking cues from Counter-Strike, but they haven't quite nailed the experience yet.

I trust that some of these issues will be fixed as the game is developed further, but what's more worrying is the character progression mechanic. Although early access to the game costs money, this is pretty clearly intended to be a free-to-play game, and as such there are already a plethora of unlockable weapon and ability upgrades. There are so many different things to customize that it's almost overwhelming at first. That would be a good thing if not for the fact that the pace of unlocks seems very slow (as if to encourage people to spend real money—imagine that!), and the unlockables don't seem very balanced. A fully upgraded character has clear advantages over a new character, which leads to a frustrating sense of unfairness.

I think this one is worth keeping an eye on, but I don't see it becoming my go-to shooter unless I end up being dead wrong about the free-to-play monetization scheme.

Rust

Have you ever wanted a reason to revile, fear and distrust humanity more than you perhaps already do? Rust is the game for you! I'm incredibly late to the party with this game, and with the survival genre as a whole; for a long time I felt, for no particular reason, that it just wasn't for me. I suppose Fallout 4 made me curious about what the genre had to offer. It turns out that the answer is pain and misery.

Don't get me wrong, though. Rust is actually an awesome game. Despite still being in early access, it's very polished, has an excellent core gameplay loop of gathering, building, and defending oneself, and is a lot of fun to play. Although I only played Rust over the course of a few days, I managed to rack up 14 hours on it, which is more time than I devote to most games these days. But therein lies the problem as well: it's a time-sink of MMO-esque proportions. Except that it's even worse than most MMOs, because when you're offline your sleeping body remains where ever it was when you logged out, leaving you vulnerable to being killed and having all of your stuff stolen. So not only does it take a lot of time to progress, but you can easily lose all of your progress, and if you're not constantly playing it becomes more and more likely that that will happen. Look, I made an equation to describe the effect:

$$ \lim_{t \to several\ days} P(being\ murdered) = 1.0 $$

It doesn't help that most of the people who play Rust are assholes. During my 14 hours, I once met some friendly players who gave me some items to help me out. Almost every other time I came across someone, if they had the means to kill me, they would, even when I was clearly not a threat and had nothing of value to them. If Rust is any indication of what a real apocalyptic disaster would be like, I'm amazed that humanity made it out of the jungle in the first place.

Still, I think Rust is pretty awesome. The construction system is neat. There are lots of cool items to craft. The maps are huge and fun to explore. The sense of tension is unmatched in any other modern game. If you have the time to devote to it, it's a great game. But I'm probably never going to play it again, because the psychological cost of wondering whether or not anything I've built will still be there next time I log on is just too high.

Hurtworld

A newer and lesser-known entry into the survival genre, Hurtworld is Rust but with slightly more forgiving mechanics and cel-shaded graphics. Oh, and no nudity. Did I mention that Rust has nudity? It does, but before you get too excited, all of the character models are male.

Anyway, back to Hurtworld: it describes itself as having "a focus on deep survival progression that doesn't become trivial once you establish some basic needs," presumably as opposed to its competitors. So far, what this seems to mean is that certain resources are more difficult to acquire; you can't go from a basic bow-and-arrow to a shotgun as quickly as you can in Rust. You can also craft cars, which is pretty awesome. Of course, I never got that far, because Hurtworld is also pretty time-consuming.

Hurtworld is clearly the earliest of these three early access games, and I ran into many technical issues while playing it. Lag was a persistent problem, with frequent "de-syncs" where the server thought something was in a different position than my machine did. Some servers seemed to have serious stability issues, with frequent disconnections. At other times, creature AI would stop working entirely. Most importantly, the interface and combat mechanics are very rough around the edges right now.

It's too early to say anything definitive about Hurtworld, but it could be a great alternative to Rust for casuals like me who have jobs and other boring adult responsibilites.

Computer Science Career Advice

2017-10-05T00:00:00Z

A few months ago, a family member of a friend of mine sent me an email asking for advice on pursuing a career in computer science. I enjoyed answering his questions, and I thought other aspiring computer scientists might have similar questions, so with his permission, I decided to reproduce my answers here. I've made a few edits here and there, mostly to remove personal information.

I want to be clear up front that I've only been doing this professional for about six years now, and I don't consider myself any sort of expert on how to have a successful career. The path I've chosen has worked out pretty well for me so far, but it's not the right path for everyone, and in an industry that changes as quickly as this one does, it's always hard to know what students should focus on when they're still several years from graduation.

What position do you hold at Microsoft and what is the scope of your work?

I'm a Software Development Engineer working in Windows OS security. Specifically, my team and I have spent the last several years working on Windows Hello. We don't do the actual template generation or matching for biometrics (face/fingerprint authentication), but we own pretty much the whole stack below that, including basically all of the crypto and some parts of the UI and UX. The marketing for Windows Hello mostly focuses on the biometric aspect, but the feature is actually a lot more than that.

Under the covers, Windows Hello provides key and/or X509 certificate-based login to domains and certain web services such as Microsoft accounts. In short, Windows Hello is more secure than using passwords, and you can get that benefit even if you don't have the hardware to do biometrics or don't want to use biometrics. We're working with industry standards bodies to specify and implement a set of web authentication APIs which will allow the use of key-based authentication technologies like Windows Hello with regular websites, which will hopefully eventually mean you won't have to have dozens of different passwords for each website you visit, or reuse the same password everywhere and risk having all your accounts get compromised.

My day-to-day work involves designing and implementing features and improvements to make Windows Hello more secure and easier to use, and of course fixing bugs as they're found. Developers are also responsible for writing and maintaining automated tests related to their features. Personally my favorite part of the job is writing code, so I try to spend as much time as possible doing that. Product managers (PMs) are responsible for mapping out the requirements, but in practice it's a very collaborative process between PMs and developers, so there are often a lot of meetings to hash things out. Meetings aren't my favorite way to spend my time, but I am glad that developers get to have input on the way features should work. The balance between design and coding varies a lot depending on where we are in the release cycle; Microsoft still basically uses waterfall-style development.

All of the code I write is in C++. On my team we make fairly heavy use of C++11 and C++14 (and soon C++17) features, which have really modernized the language and made it much nicer to deal with. Plus, trying to keep up with the latest additions to the C++ standard means I'm always learning new techniques and tricks, which is fun.

What kind of education, work experience, and or skills did you cultivate to obtain your current job?

I started writing code when I was probably around 10 years old in Visual Basic 6.0. I wasn't super dedicated to writing code when I was younger; I'd come up with projects and work on them for a few weeks or months at a time, and then eventually "finish" or abandon them and not write any code for several months. As a teenager, I started writing PHP and occasionally tinkered with other languages such as Java. I tried out C a few times during this period, but I generally found it to be over my head until I picked it up in university, where I was finally able to grasp some of the trickier concepts like stack versus heap allocation and how pointers work.

I spent two years in the CS program at the University of Northern British Columbia, which was mostly taught using Java. I then decided to transfer to the University of Alberta. UNBC has a decent CS program, but it's very small and has limited funding, so they simply can't offer the variety of courses that bigger schools can offer. For instance, I took courses at U of A in computer vision and "Advanced Game Programming in C++." To my knowledge, UNBC doesn't offer anything like that. That said, neither of those courses are very relevant to the work I do now, and the game programming one was actually pretty disappointing. It ended up being more like "basic C++ programming and a few concepts that tangentially relate to games." The best thing U of A offered was a very good internship program, through which I landed a 16-month internship at Pason Systems in Calgary.

I think that internship was the single most important part of my education, both formal and informal. This is not necessarily due to the technical knowledge I gained from it, but more due to the hands-on industry experience I got. I worked with some very skilled developers who gave me a lot of good career advice and connections within the industry, and I also consider many of them to be great personal friends to this day. It's honestly hard for me to completely enumerate all the things I gained from working at Pason, but career-wise, when you're looking for your first job out of university it's a huge plus if you already have some industry experience on your resume. If you only take one thing away from reading this post, it should be that the most important thing you can do for your career is to take opportunities like this when they come up, even if you're not sure you're ready for them.

The next best thing to have on your resume is a personal project (or projects). In my case, I developed an iPhone game during my second and third years of university. I didn't really do this with my career in mind, to be honest. At the time, the app store was fairly new and there was still sort of a "gold rush" for iOS app developers, although it was already starting to die out by the time I released my game. I was kind of hoping that I'd make some money off of it, but I ended up making $0 because the game never sold enough copies to get to the threshold at which Apple actually pays out the earnings. It was worth it for the experience and for the enjoyment I got out of the project itself, though, and it helped me land both the internship I mentioned above and probably also my job at Microsoft.

In terms of specific skills to cultivate, that largely depends on what you want to work on. I've done a bunch of web programming, both back-end and front-end, but personally I've always found client programming more interesting. There are endless job opportunities for programming enterprise asset management apps, but to a large extent, if you've worked on one CRUD app you've worked on all of them, and they're generally not very technically interesting. For that reason, it was important to me to become proficient in C++.

That said, having strong fundamentals is more important than what particular languages you know. It's easy to pick up a new language if you have a strong understanding of the core concepts underlying all software development. For example, memory allocation, threads, page tables, user mode versus kernel mode, indirection (pointers, passing by value vs by reference), compilers, recursion, debugging, different language paradigms (imperative vs functional), etc. Pretty much any CS program will teach you these things, but to really understand them in a hands-on manner, it helps to have practical experience. How much of that you get kind of depends on what courses you take. Even if you plan to focus on higher level languages, I'd recommend doing at least a little programming in C (not C++) and assembly, because those are the best ways to get close to the metal and really understand what a chunk of code in a high-level language translates to at the machine level.

Do you like your job and would you recommend the field of work to others?

Generally, yes, I like my job, but not always. And I would recommend the field to others, though not to everyone. This question reminds me of an article titled "Non-myths about programming," which I think is worth reading if you're wondering whether or not CS is the right career path for you. The gist of it is that some of the stereotypes that exist about computer science exist for a reason, and depending on your personality and lifestyle preferences these may or may not be an issue for you.

Specifically, the most obvious "downside" to CS is that you will spend most of your time staring at a computer screen and not interacting with other people. I put "downside" in quotation marks because if you're an introvert like me, then that's actually an upside. I've had customer-facing jobs in the past, and while I enjoyed some aspects of those jobs, ultimately I much prefer self-directed work where I generally don't have to worry about being the public face of the company. Of course, most programming jobs do still involve collaborating with a team and hashing things out in meetings, and if you're more of a people person then there are always project management and people management roles available. But I do have friends who went into CS and found that they disliked the relative lack of human interaction, so for some people that can be an issue.

The other caveat is that at least for me, how much enjoyment I get out of programming largely depends on what it is I'm actually making. I view programming as a creative exercise, but unless you have the ambition, energy, and financial resources to create your own startup, the parameters of that creativity will be constrained by whatever project your employer has you working on. Business don't measure the output of programmers in terms of code, but rather in terms of "business value," i.e. how much money you make for them.

When I was younger, I wanted to be a game programmer, and a big part of me still wants to do that, but I've never seriously pursued it. I've learned that the game development industry is extremely hard to get into and also actually not that desirable. Because it's seen as such a cool place to work, the number of qualified candidates vastly exceeds the number of available positions, and as a result, companies get away with offering relatively low pay despite the jobs being extremely technically demanding and infamous for having long hours. And even for those who do get into the game industry, for every awesome game like Overwatch or Halo, there are a hundred shitty pay-to-win mobile games that will suck out your soul.

CS in general doesn't suffer from an overabundance of qualified candidates; it's generally accepted that there's a labor shortage, and as a result the job market is very friendly to employees, and wages are high. (That said, there is some dispute over how much of a shortage there really is.) But, as with games, for every cool product or interesting problem you might be working on, there are a hundred shitty Enterprise JavaBeans line-of-business apps that need some poor soul to maintain them.

The point of all this is that you probably won't always be fully engaged in what you're working on. That has certainly been the case for me. While I do think that what I work on now is pretty cool, I ended up in security more or less by chance, and it has only been through working on it that I've come to develop an interest in it. But it's technically challenging and impactful enough that it has held my interest for five years now, and more generally, I enjoy programming enough that I'd probably be doing it anyway even if I had to work on a product I found really boring.

Of course, one also has to weigh in pragmatic aspects such as job security and monetary compensation. I think CS scores very favorably in this regard as well. There are lots of well-paying jobs available and the field only seems to be continuing to grow. I don't think I'd recommend that anyone should go into CS purely for the money; if one is only concerned about money then it would probably make more sense to be a dentist or something like that. But all things considered, I can't think of any realistic career option I'd rather be pursuing. (Sadly, I think I'm getting a bit old to pursue a professional snowboarding career...)

What would you tell your younger self or someone starting out in computer science today?

Besides what I wrote above, I think I would just point out that like other fields, CS has certain fads that come and go, and it's best not to get too caught up in these. Node.js and Ruby on Rails strike me as two frameworks which might fall into this category; they both became very popular very quickly, and they seem to me like buzzwords that a lot of recruiters are looking for. That said, I have to admit that I'm biased here because I don't personally like either of those technologies very much.

Don't get me wrong—there's nothing wrong with learning Ruby on Rails, or Node.js, or any other specific technology, whether or not it's a "fad." Being skilled in the latest thing that recruiters are looking for is probably a good career move in most cases. I just want to reiterate what I stated earlier, which is that it's more important to have a strong grasp of the fundamental concepts underlying all of CS. If you have strong fundamentals, you can learn any specific language or framework with relative ease. But if you only learn one language and don't ever think about how concepts in that language map to fundamental CS concepts, then you're going to find it very difficult to transition to something new.

Also, for this reason, I am very skeptical of "coding bootcamps." I find it very hard to believe that anyone can be taught to be a competent programmer in just a few months. However, I have no empirical data on how these programs perform, and part of my skepticism might just be subconscious elitism and desire to preserve the value of my degree. I do think that there's a lot of room for improvement in how computer science is taught at universities that do traditional four-year degree programs, and there are also serious issues with credentialism in general, but that's another topic altogether.

Where do you see the future of computer science in five years and how are you preparing to meet those challenges?

The two big trends that seem most likely to me to have a big impact are quantum computing and machine learning.

I know about as much about quantum computing as a layperson does about computer science in gneeral. In other words, it's basically magic, as far as I'm concerned. However, I can say that when we get a truly general-purpose quantum computer which can run Shor's Algorithm, with enough qubits to work efficiently on 2048-bit and larger RSA keys, that will have huge implications for CS in general and especially for security. I don't really expect this to happen in the next five years, but again, my knowledge here is quite limited. If it does happen that soon, it's hard to predict the magnitude of the consequences. Banking now relies heavily on our ability to perform transactions over the Internet with cryptographic security guarantees, but current software uses cryptographic techniques which will be vulnerable to quantum factorization, and changing it to use new cryptographic techniques will not be fast or easy. At worst, I can imagine this having a huge negative impact on the global economy as digital shopping and banking grind to a halt.

And that's not to mention all the other aspects of computing that rely on cryptographic security. When you download a software update, your computer knows that it's safe to install because it can validate the digital signature on the package to confirm that it came from the software publisher. Without that, it would be almost trivial to trick any current operating system into installing malware.

Incidentally, Microsoft just announced "a new quantum computing programming language, with full Visual Studio integration, along with a quantum computing simulator." They haven't been released yet, but that's slated for later this year, and personally I intend to play around with this toolkit to see if I can at least wrap my head around what programming for quantum computers might look like, and what the practical implications could be for me as a software developer.

Machine learning is something that we already have, but it is becoming more and more powerful, and we're finding more and more ways to leverage it. For example, I do expect that fully autonomous vehicles (i.e. level 4 or 5 according to SAE classifications) will probably be commercially available within the next 5-10 years, at least in certain areas where conditions are favorable. This will have a huge economic impact as well, as a significant percentage of the workforce will be made redundant in countries that adopt this technology. (In the US, driving makes up something like 4 million jobs; truck driving alone is the single most common occupation in a majority of states.)

In the case of self-driving cars, and for machine learning in general, I think the impact on the broader economy and society in general will be more consequential than the impact on CS specifically. Programming will probably be one of the last occupations to be automated, as it's not really feasible to have a machine program itself until we have real artificial general intelligence, which is probably decades away at least. Until then, you will at least need humans to define the inputs and outputs for the machine learning system. But I do think programmers in many different sub-disciplines will increasingly be expected to be familiar with machine learning techniques and how they can be leveraged in an increasingly wide variety of applications. There are already many categories of problems which are more feasible to solve using machine learning than through human intelligence (computer vision is one example), and that set of problems will only expand as our tools and techniques get more and more powerful.

Fitbit Ionic SDK impressions

2017-10-17T00:00:00Z

I got the new Fitbit Ionic a few weeks ago, and I'm generally pretty happy with the device. I'm in the process of writing a full review of the Ionic, but I realized partway through that I had written quite a lot about the SDK, which many people probably won't be interested in. Hence, I decided to make that into its own post. In the meantime, if you're looking for a review, I would recommend reading DC Rainmaker's. He's a much more serious athlete than I am, and he reviews fitness gear semi-professionally, so he's able to put the devices through much more rigorous tests and go into detail about things like heart rate and GPS accuracy, which I won't be doing.

Anyway, the Ionic being Fitbit's first real smartwatch, I was eager to check out the SDK and dive into app development. And, well, I've tried to do so, but so far the results haven't been great. I think this is partly just because the platform is very new, and partly because of some questionable technology decisions that Fitbit made. Although it's too early to really evaluate the success or failure of the Ionic, or of FitOS an app platform, I think it can still be instructive to look at some of the choices that Fitbit made in designing the device and the OS, what effects they're having on developers now, and why I think they may prove to have been suboptimal choices.

Battery life was clearly the foremost concern for Fitbit when they designed the Ionic, and the technology decisions reflect that. Whereas the Apple Watch series 3 has a dual core CPU with a maximum clock speed of at least 780 Mhz, 3D graphics acceleration, and a rumored 768MB of RAM, the Ionic contains the humble Toshiba TZ1201XBG, with a meager maximum clock speed of 120Mhz[^cpu]. The battery life of the two devices reflects these hardware differences: the Ionic lasts 4-5 days on a charge, whereas the Apple Watch lasts a maximum of 2 days. Personally, I think this was a great choice by Fitbit. Although the Apple Watch is inarguably a more capable "smartwatch," its two-day maximum battery life is an absolute deal-breaker for me, which is one of the primary reasons I bought the Ionic instead.

With the hardware itself being so thoughtfully designed to maximize battery life, it strikes me as a bit odd that Fitbit chose JavaScript as the sole language for app development on their platform. JavaScript is not known for being a high-performance, memory-efficient programming language; historically, it has been quite the opposite. Of course, it's not the language itself that matters so much as the interpreter, and modern JavaScript engines like V8 and Chakra have made extremely significant performance improvements compared to how slow JavaScript interpreters used to be. The Ionic, for its part, uses an engine called "JerryScript", which is designed specifically for IoT devices and allows JavaScript to be precompiled to bytecode; it boasts that it can run in environments with less than 64KB of RAM and less than 200KB of ROM. I'm not sure where the name comes from; contrary to what you might expect, the project's main contributor does not appear to be named Jerry. For me, the name just conjures images of Jerry from Rick and Morty:

Uh, anyway, JavaScript is still orders of magnitude slower than C and C++ for most workloads, and when you're concerned about energy efficiency and performance on a device with a 120Mhz single-core CPU, that difference matters. I freely admit that I am biased in favor of languages like C and C++, which are as close to the metal as you can get in a high-level language. Some would consider this preference anachronistic, but it's undeniable that there are still many places where maximizing performance really does matter. Embedded applications are one of those places, and I'm fairly confident that on the Ionic, using native code could have offered even better battery life, or perhaps improved performance, which is sometimes suboptimal even in the built-in apps.

Similarly, all UI on the Ionic is done using Scalable Vector Graphics (SVG). SVG is a lot easier to use than requiring developers to render their own UI, but it's also a lot less flexible than just blitting pixels to the screen. The minimum benchmark that any modern computing device should be able to pass is whether or not it can run DOOM, but between the poor performance of JavaScript and the inflexibility of SVG, I don't see that happening on the Ionic any time soon.

In fairness, Fitbit had a difficult problem to solve in developing an SDK for a smartwatch which supports multiple "companion" devices (i.e. iOS, Android, and Windows devices which communicate with the watch). First, there's the technical problem to solve of enabling the device to communicate with those three platforms, each of which has different restrictions on what apps can do. But perhaps equally important is the challenge of making the SDK easy and convenient enough that people will actually want to develop for it. Having an app platform is meaningless if nobody actually writes apps for it, and as Microsoft found out with Windows Phone, it can be very hard to attract developers to your platform if you're not already in a dominant (or at least strong) market position. Fitbit has been very successful with its cheaper wearables, but this is their first real smartwatch, and between Apple, Garmin, Suunto, Samsung, and dozens of others, there's a lot of competition.

With that in mind, it's tempting to think that a platform can be differentiated by making it very easy to develop for. I assume this was one of the primary factors that weighed into Fitbit's decision to use JavaScript: it's generally considered to be a pretty easy language to learn and use, and since it can be run within the official Fitbit app on the phone or PC, it allows developers to write "companion" modules for their watch apps in the same language that the watch app itself is written in. (This is often cited as an advantage of node.js—you can write your code in the same language on the server and the client. Personally, I've never understood why one would want to write JavaScript on the server, but that's just me.) The ability to easily create companion modules is important, since the watch can't directly communicate with the Internet; all Internet access must use the companion app as a proxy. (The Ionic does have WiFi, but it seems to only be used for syncing must and downloading firmware updates. As I understand it, this is because WiFi consumes too much battery.)

The problem is that I don't think it's actually true that you can attract developers to your platform on the basis of it being easy to develop for—at least, not solely or primarily on that basis, and not the kind of developers you actually want to attract (that is, skilled developers who know what they're doing). The whole open source community is a testament to the insane amount of work that some software developers are willing to do for free, but everyone needs to eat. Developers will come to a platform if it provides a money-making opportunity, even if it's difficult to develop for; conversely, the easiest development platform in the world won't help if it has no users. With that in mind, it's also concerning that the FitOS platform has no built in way for app developers to charge for their applications.

And in any case, the biggest problem is that currently, the Ionic isn't actually easy to develop for, because the SDK is seriously short on features and extremely buggy. The web-based toolchain can also be pretty painful to use. In no particular order, here are some of the issues I've run into in the process of trying to develop a simple app, along with a few other problems that other developers have mentioned in the forums or in Discord:

The DOM for an app's interface cannot be modified at runtime, so the only way to change your UI dynamically is to hide and show pre-existing elements. (Limited support for being able to spawn copies of existing elements is coming.)
There is currently no built-in support for multi-page applications. Again, you just have to show and hide different elements to achieve the effect of multiple pages.
Although the watch CPU appears to support hardware accelerated AES and SHA256, this isn't exposed through any API, so cryptographic operations require the use of third-party JavaScript libraries and end up being unusably slow.
I cannot get the OAuthButton component to work in my application settings, which means I can't authenticate to basically any web service. This might be iOS-specific; one of the Fitbit developers mentioned that their QA had reported something similar, so hopefully they're looking into it. Another developer mentioned having trouble getting OAuthButton to work with Fitbit's own REST APIs! Unfortunately, there are currently no examples of how this component is supposed to work. [Edited 2017-10-22] This appears to be fixed now.
There is no emulator available, and no way to test the companion app or Settings page in the browser. This is an annoyance in general, and it seriously exacerbates the OAuthButton issue makes it really hard to debug any issue with REST APIs, because it also seems to be nigh-impossible to capture decrypted HTTPS traffic from an iOS device. (Granted, that last bit isn't Fitbit's fault at all, but it also wouldn't be an issue if the OAuthButton just worked.)
PNGs are automatically converted to a proprietary hardware-accelerated format, but JPGs are not hardware accelerated so they take forever to load and can actually cause crashes or hangs if they're too big. This isn't a big deal for static images within the app, but what if you want to show content from the Internet?
Communication between the watch and the "companion" app running on the phone seems to have some reliability issues. Sometimes the companion seems to just not launch at all.
There is no way to "push" data from the companion to the watch. The watch app has to request the data, which means you need to poll for it if you don't know when the data will be available. (In fairness, this might be an iOS limitation.)
Some APIs having confusing asymmetries. For instance, apparently in order to send a file from the companion app to the device, you need to manually CBOR-encode the contents before sending it. But when reading the file on the other side, it is decoded for you automatically.
Documentation is fairly limited; for example, a lot of APIs have parameters that aren't well-explained, so you kind of have to guess what they do and then figure it out by trial and error.
There is no local toolchain that you can install on your PC or Mac; there is only a web-based IDE which builds and deploys your app for you. This means no Git (unless you download copies of your code every time you make changes), no inline diffs, no using your own text editor with your own keyboard shortcuts, and no working on your app if you're offline. Some people have also reported connectivity issues when trying to get the watch to talk to the IDE.
The IDE does not support breakpoints, performance profiling, or other such tools that you'd expect from a modern development environment. I hope you enjoy debugging using console.log(...)! By the way, there's a (relatively small) limit to length of string you can log.

I could go on, but I'm sure you get the point. Fitbit is aware of many (hopefully most) of these issues and is already working on many of them, and I have to commend their developers for being very responsive and involved in the community. There's an unofficial Discord channel where a few of the developers are very active in providing advice and investigating issues raised by the community. It's nice to have such a direct line to the development team; when interacting with a bigger company, you usually have to go through six layers of customer service and escalation engineers before you get to talk to an actual developer. Unfortunately, the downside is that the handful of Fitbit developers who are going above and beyond by fielding SDK questions during and outside of work hours can't possibly hope to keep up with the volume of requests they're getting, especially given how many issues there are right now. A public bug tracker would be nice so that we have a definitive source for which issues are being tracked...

It's frustrating that the SDK wasn't more complete at release. I'm sympathetic to the need to get the device out the door, especially given that Fitbit is facing stiff competition from bigger companies with vastly more resources. However, that fact only compounds my feeling that building a brand new JavaScript-based app platform was probably the wrong decision. In general, I'm of the opinion that any new app platform should start by exposing APIs that can be called from native code, and providing a toolchain that allows developers to compile C and/or C++ into the platform's native instruction set. Not only is this more performant, but it's easier to do: since you need to at least partly implement your APIs in native code so that they can talk to onboard devices, adding a scripting interface is just extra work. Once the (public!) native APIs are working, stable, and reasonably complete, you can build a managed code SDK on top of it… or just sit back and wait for someone else to do it for you.

This appears to be what Pebble did, as they provided a C API to create "watchapps and watchfaces," and a JavaScript API which could only be used for watchfaces. Given that Fitbit acquired Pebble last year, reportedly primarily for the purpose of using their code and expertise to build FitOS, I don't know why Fitbit didn't just literally rename all the Pebble APIs and call it a day. I haven't actually done any Pebble development, so maybe I'm missing some major issues, but from a quick perusal of their documentation, their APIs look pretty solid, and they're obviously a lot more mature than Fitbit's new platform is. Pebble's users certainly seemed to be happy with the platform (to the point that many of them were very vocally disapointed when the company was acquired by Fitbit). Plus, Pebble had Android and iOS SDKs to allow third party apps on those platforms to communicate with the watch, in addition to a JavaScript-based SDK that ran as part of Pebble's own Android/iOS app. This approach appears much more flexible and powerful to me. Perhaps eventually Fitbit will follow suit, but at this point I'm not holding my breath.

So, what's the takeaway of all this? It's not that the Ionic is a bad device, or that its app platform is doomed to failure, or that Fitbit has bad engineers. I don't think any of those things are true. What I would like to communicate—and this will probably be a running theme in all of the CS-related writing I ever do—is that technology decisions really matter. Programmers are notorious for getting into religious wars about their favorite languages, and evangelizing your preferred technology too hard makes you come across as an asshole. I'm probably already guilty of this. But on the other hand, I feel that there's a strange sort of postmodern stream of thought in CS that it doesn't really matter what languages or technologies you choose to use in any given case, because they're all more or less equally capable, and that's… just not true.

Yes, there is a sort of isomorphism between any two Turing-complete languages, but in practice you can't simply gloss over the performance implications of using, say, JavaScript (or even Java) instead of C or C++. This might sound blindingly obvious, but remember that even huge companies are subject to this mentality, sometimes with disastrous results, as was the case with Windows Longhorn. (It also runs rampant in college CS programs, which is why it's notoriously difficult to hire college graduates who are comfortable with C or C++.)

Lest I come off as too much of a native code zealot, I'm not saying that native code is always the right choice. If you're writing a web application, or anything else that involves a lot of text manipulation, database interactions, or (de-)serialization of dynamic objects, you're probably crazy if your tool of choice is C++. Even just writing a client for REST APIs in C++ is not very fun (I know this from experience). But, particularly if you're building an app platform or a modding system or something like that, it's better to offer third-party developers as much flexibility and power as you can, rather than painting yourself and everyone else into a corner to start with. Remember that you can always just export C APIs and provide bindings to call them with your favorite scripting language. It's quite a bit more difficult to do the other way around.

[^cpu]: The first personal computer I ever used was a Macintosh LCII, which had a clock speed of 16 MHz. That was 20-some years ago. If it doesn't blow your mind that we can now fit a CPU an order of magnitude more powerful than that in a watch, and make it run for several days on a tiny battery while powering several other sensors... well, it should!

Review: Fitbit Ionic

2017-11-05T00:00:00Z

Conflict of interest disclosure: I own 20 shares of Fitbit stock, which I purchased halfway through writing this review, on the hope that apparently strong Ionic sales would give the stock a boost. So far, that hasn't been the case. At any rate, the value of the stock is less than the cost of the device itself, so I don't think it compromises my objectivity, but you have been warned.

Fitbit recently released their first true smartwatch, the Ionic. After going through a phase of slight obsession in the lead-up to its release, during which I spent many hours watching preview videos and reading comparisons between the Ionic, the Apple Watch Series 3, and the Garmin Vivoactive 3, I ultimately decided to buy the Ionic. I've now had it for about a month, so this seems like a good time to review the device and discuss what I think it does well, and not so well.

If you're looking for an in-depth review of the Fitbit Ionic with a particular emphasis on how it performs as a fitness tracker, I'd recommend reading DC Rainmaker's review. I'm not that serious an athlete, so I won't be going into depth about the GPS and heart rate accuracy, and really there's no need for me to do so, since DC Rainmaker has already covered those aspects of the device. This review is merely my own non-expert opinion of the device based on my own experience, with a bit more emphasis on the smartwatch features of the Ionic and how suitable the watch is for various types of users.

Fitness features

Even though I don't consider myself a serious athlete, I did buy the Fitbit Ionic primarily as a fitness tracker, so I'd be remiss to not mention how the device fares in that respect. Indeed, I think it's fair to say that the Ionic was designed primarily as a fitness tracker, with the smartwatch functionality being somewhat secondary. In my mind, this puts the Ionic in a slightly different category than the Apple Watch—plus, the Ionic supports Android and Windows devices, whereas the Apple Watch does not. Unfortunately for Fitbit, competition in this market is extremely fierce, even if you only consider all the Android and Windows users that the Apple Watch can't reach.

So, how does the Ionic perform as a fitness tracker? I think it's pretty good for the most part, but there are some noteworthy shortcomings that Fitbit absolutely needs to address in future software updates if they want to stay competitive. But let's start with the good parts. First of all, I've been impressed with the GPS. It connects quickly and consistently, and in my experience, the accuracy is pretty good for a wrist-mounted device. I have found that it tends to cut corners if you make sharp turns, which can result in runs appearing slightly shorter than they actually are, but the deviation is only about 50-100 meters over a 5km run. This is on par with the accuracy of other devices, and it won't be cause for concern for most people. Besides, in some cases that error will be averaged out by other noise.

Likewise, the heart rate monitor is quite reliable. Like most wearable devices, it records continuously throughout the day at a lower frequency, and at a higher frequency during activities. I have found it to be fairly accurate, only occasionally getting confused during strenuous exercise, and pretty much always appearing correct at rest—even if I wear it on my tattooed arm. I say "appearing" because my testing of the heart rate sensor's accuracy has been thoroughly unscientific; I've just counted beats manually and compared it to the reading on the Fitbit. Optical heart rate sensors are never as accurate as chest straps, but they're much more convenient. I've owned a chest strap for years now, but I can never be bothered to actually use it.

Arguably the most important aspect of a wearable device is its battery life, and happily Fitbit did not disappoint with the Ionic. They advertise 4+ days for most users, and in my experience that is accurate. If you don't use the GPS much, I think you could easily get 5 days on a single charge, but even with fairly frequent GPS use, you should have no problem lasting at least 3.5-4 days between charges. With GPS on constantly, you should get 10 hours of battery life, which is nice as it would allow you to track a longish hike without having to bring a battery pack to recharge at the top of the mountain. Ultramarathon runners or multi-day backpackers might find 10 hours of GPS time insufficient, though.

I have had a couple days where the battery rapidly dropped from 40% to zero over the course of only a few hours, part way through the fourth day of use. I assume this is an issue with the calibration of the battery guage which can be corrected by a software update; at any rate, it has only happened to me a couple times.

Garmin, Suunto, and some lesser-known brands do have Fitbit beat in terms of battery life, but the important thing for me is that I can use the Ionic all day for several days without having to think about when I last charged it, or when I'll next have to charge it. That was not the case with the Microsoft Band 2[^microsoft], which I previously owned. At only two days to a charge, I had to be mindful of when to put the Band on the charger if I wanted to be able to consistently track my sleep, and if I happened to forget, I'd just have to forego sleep tracking that night. Incidentally, this is the main reason I didn't get an Apple Watch; the short life is a deal breaker for me.

(Apparently the Apple Watch doesn't natively do sleep tracking, anyway. One could argue that sleep tracking isn't actually very useful, but personally I enjoy having the data and being able to compare what the device measured to how I feel the next day. It also seems odd to me that the Apple Watch doesn't natively support sleep tracking given that the Apple Health app feature Sleep as one of four prominent tiles on the Health Data page.)

To my mind, there are three major shortcomings on the fitness front. First, you're currently limited to a maximum of seven "exercise shortcuts" stored on the device at a time. This means that if you want to track an activity that is not one of the seven you've chosen, you need to use your phone to reconfigure your shortcuts before you can track your exercise on the device. Doing that is a fairly tedious process when you just want to start exercising, and even as someone with a limited repertoire of athletic activities, I've already had to swap out activities a few times because I can't get everything I want on the device. I can't imagine that there's any real technical limitation behind this arbitrary maximum number of activities. If I had to guess, it's more of a UI issue, because swiping through activities on the device might be cumbersome if there were dozens of activities available. Whatever the reason, I really hope this is fixed in a future software update.

Second, the list of available activities has some notable omissions. The available options are:

Run
Walk
Swim (pool swimming)
Treadmill
Weights
Hike
Bootcamp
Yoga
Interval Workout
Stairclimber
Martial Arts
Elliptical
Tennis
Bike
Spinning
Golf
Kickboxing
Pilates
Circuit training
Generic "workout"

This isn't a bad start, but I'd really like to see options for rock climbing (including bouldering and top roping/lead climbing), skiing, snowboarding, open-water swimming, standup paddleboarding, kayaking, and canoeing. There are dozens of other activities that can be logged on Fitbit's website but don't have an equivalent on the device, which begs the question: why not? Again, this is the kind of thing that could be easily corrected in a software update, but the extent to which Fitbit will provide updates and support for their new device and operating system remains to be seen. Frankly, looking through Fitbit's feature suggestion forum doesn't make me extremely optimistic about the level of post-launch support we're likely to see on the Ionic. There are over 9,000 suggestions on that forum, across all Fitbit products, but only a handful of them have been implemented so far. And I can't help but wonder why more exercise options weren't available at launch, given Fitbit's emphasis on, you know, fitness. It's especially irritating when you consider that the Apple Watch does support a lot of these activities.

Third, the Ionic cannot be connected to external sensors like chest strap heart rate monitors or power meters for bicycles. This is not a huge issue for me personally, because I don't use any of those devices. As mentioned previously, I do actually own a chest strap heart rate monitor, but I find it too much of a nuisance to wear it when I run or work out, which is part of the reason I like having a fitness watch in the first place. But other devices do support external sensors—including the Garmin Vivoactive 3 and, again, the Apple Watch. So this too is a big shortcoming for serious fitness enthusiasts, on a device that really should be capitalizing on its fitness features to make up for its shortcomings as a smartwatch. (More on that later.)

Aesthetics and comfort

Aesthetics are obviously inherently subjective, but personally I think the Ionic looks great. I've never been a fan of the Apple Watch's flat, rounded rectangular design, and although I haven't worn one, they've always appeared unpleasantly thick to me. On the other hand, I'm also not a huge fan of the circular screens sported by many other smartwatches. As a software developer, I can't help but think about what a pain in the ass it must be to develop UI for a round display, and the although I understand that many people have an aesthetic preference for round watch faces, it seems a bit backwards to me to constrain modern technology in a form factor that was designed for mechanical devices.

A lot of people have complained about the large black Fitbit logo bezel at the bottom of the Ionic's screen. I agree that it does give the front of the watch a somewhat dated look if you really focus on it—in fact, combined with the curvature of the screen, the Ionic's display almost looks like a tiny little CRT. Bezels are very much out of style in 2017, and it certainly would have been nice to just have more screen. I will say, though, that the close up marketing images somewhat exaggerate the bezel's prominence. It's really not that noticeable on the watch itself when it's on your wrist, at least to my eye.

Actually, although to some it may look dated, overall I really like the subtle curve of the Ionic. It's hardly noticeable from most angles, but it still manages to make the watch feel like it's hugging my wrist all the way around, rather than just sitting flat against it. As someone with small wrists, this is really nice. In contrast to a lot of other watches I've worn, including mechanical watches, the Ionic doesn't look absurdly bulky on my wrist. And more importantly, it's light and comfortable enough that I'm not constantly aware of it on my wrist, although I can't say that it ever entirely disappears. That said, the stock wrist band, which I'm still using, is definitely not great. It's a little too stiff, and it doesn't breathe at all. I'll definitely pick up a sport band at some point. There are already a good variety of third-party bands available on Amazon, although a lot of them are pretty low quality.

Smartwatch features

Unfortunately, the Ionic is pretty underwhelming as a smartwatch, at least in its current state. I've never really understood the desire to read news articles or compose tweets on a watch, but I assume there are some cool things a smartwatch can be used for. If so, though, the Ionic isn't a very good showcase of it, and it remains to be seen whether or not the app platform will be successful at all.

The Ionic has the following built-in apps:

Exercise: is used to start and stop workouts. Currently, it supports 20 different activities, which I listed above. Annoyingly, although perhaps unsurprisingly given the Ionic's specs, the exercise app cannot run in the background. Once you're in a workout, you can't also check your steps or start a timer.
Today: shows you your stats for the day (steps, hourly step goals, current and resting heart rate, distance, calories, floors climbed, and active minutes).
Timers: has timer and stopwatch functionality.
Alarms: allows you to set silent alarms. You can have multiple alarms active at the same time, and you can choose which days alarms repeat on. Whereas most other Fitbit devices only allow alarms to be configured from the phone app (since many of them have no screens or tiny screens), on the Ionic you can only edit your alarms on the device.
Weather: provides weather at your current location and other locations of your choosing. This is an app I could see myself actually making a lot of use of, but I'm very disappointed in the current implementation. It seems to sync at random—perhaps whenever the Ionic syncs exercise data, but not immediately when the weather app itself is opened. As a result, it always seems to be 30 minutes out of date for me. That might not sound like a big deal, but the weather in the Seattle area is notoriously fickle, so 30 minutes can be the difference between sunny skies and torrential downpour.
Pandora: allows Pandora Plus subscribers to sync a few of their Pandora stations to the Ionic. The Ionic can't stream music from the Internet, so a selection of songs are synced over WiFi while the device is charging. Unfortunately, this is only available to US customers. Personally, I use Spotify rather than Pandora, so I haven't used this. Fitbit is reportedly trying to get Spotify to support the Ionic as well, but that is easier said than done.
Music: allows you to sync playlists from your PC or Mac to the Ionic. You cannot use your phone to sync music (which is probably an iOS limitation, but Fitbit doesn't support it on Android or Windows Phone either, probably just to keep the apps consistent). I haven't actually used this either since I basically always have my phone with me. From what I've read elsewhere, music playback on the device works pretty well, but syncing songs is very slow and you must create playlists for the songs you want to sync (i.e. you can't just sync a folder of music).
Coach: offers guided bodyweight workouts with short animations that show you how to perform each move. The workouts are pretty fast-paced, and the goal is to do as many reps as you can during the time frame you're given for each exercise, so it should be an effective workout regardless of your fitness level. So far there are only three workouts available, but Fitbit has promised that more will be coming (via a paid subscription service), and future workouts should be tailored to your ratings of how difficult your past workouts were.
Strava: shows you stats for your previous running and cycling sessions. In fact, this is the only way to look at your previous workouts on the device. Your workouts have to sync to Fitbit's website, and then to Strava, and then back to the watch via the Strava app before you can view them, so this app also seems to always be out of date. Also, it takes so long to load that sometimes it gets killed by the OS before it starts.
Relax: provides guided breathing sessions and gives you feedback about how well your breathing was synchronized with the prompts and how your heart rate changed during the session. It's a nice idea, but I think traditional mindfulness meditation is more effective (albeit more time consuming) if you really want to relax.
Wallet: stores your credit cards for use with Fitbit Pay. Fitbit Pay can be used anywhere that NFC payments are supported, which in my experience means it is available pretty much everywhere Apple Pay is, and possibly even at some places where Apple Pay isn't. There are a limited number of banks supported right now, but I was able to add both of my credit cards even though one of the issuing banks isn't actually listed as being supported. I've used Fitbit Pay several times and it has worked flawlessly almost every time. I did have one or two times where the terminal didn't seem to detect the Ionic, but that may have been an issue with the terminal.
Settings: lets you change brightness, vibration, all-day heart rate, and notification settings. On the subject of notifications, the Ionic can display any notification shown on the smartphone, but there's no way to interact with the notifications (i.e. you can't reply to a text message from the watch), and currently notifications on the watch are not perfectly reliable. Also, there is no option to keep the screen always on, which a lot of people seem to want. Personally I find the rotate-wrist-to-wake functionality sufficient.

Of course, built-in apps will only get you so far. Whether or not the Ionic proves to be a capable smartwatch will largely depend on the strength of its app platform. That is impossible to evaluate currently, because the third-party app store isn't available yet. (It should be coming before the end of fall, according to Fitbit.) However, I took a crack at developing my own app for the Ionic, and so far I have not been impressed with the SDK. If you're interested in that, I wrote a whole other post about it, but the gist of it is that the SDK is currently short on features, poorly documented, and extremely buggy. Let's hope that the next firmware update comes with some big changes.

Miscellaneous

The Ionic also contains a blood oxygen saturation sensor. This kind of sensor can be used in the diagnosis of sleep apnea and other medical conditions, although medical sensors are usually clipped onto the end of the finger; I've never seen a wrist-mounted one. Fitbit is reportedly experimenting with detecting sleep apnea using their sensor, but it's unclear whether that will ever actually see the light of day. I don't mind being a guinea pig for features that might not come out for several years, but I do wonder how much of the cost of the device went into that sensor and the R&D for it. If the Ionic could have been released at, say, $285 without the SpO2 sensor, I think that would have made it a much more competitive offering compared to the Apple Watch or Garmin Vivoactive 3. $15 might not seem like a lot, but I think there's a significant psychological difference between $280-ish and $300-ish.

Even if Fitbit finds an accurate way to detect sleep apnea using the sensor, they would require FDA approval to surface that data to US customers, which could take years to get. Fitbit claims to be working with the FDA to develop an efficient regulatory pathway for wearables, but that's going to be an uphill battle. Other countries may have fewer regulatory hurdles for features like this, though.

The Ionic also has a temperature sensor on board, which, as I understand it, is used to calibrate the barometric altimeter so that it can accurately measure elevation. This sensor is not currently exposed in the UI or the API at all. I'm not sure if there would be any benefit in exposing it to the API since its temperature reading is probably somewhere between the ambient temperature and the wearer's skin temperature, but it would be cool if Fitbit found a way to correct for the skin temperature and expose ambient temperature through the API. I wouldn't hold my breath on that happening, though.

One thing I haven't mentioned so far, which is a big advantage to the Fitbit ecosystem, is the community. The Fitbit app and website allow you to connect with friends and compete for the highest weekly step count or participate in challenges together. Of course, whether or not this will matter to you depends on whether or not your friends and family are also using Fitbit devices. But if they are, the significance of this feature as a motivational tool should not be underestimated. With my Microsoft Band, I never particularly cared about reaching my daily step goal, because the community aspect of the Microsoft Health platform was very limited. Even as a Microsoft employee, I only had three or four friends who participated, and I'm pretty sure none of them ever checked the rankings. With Fitbit, my whole family is on there, and I know they'll see it if I'm lazy, so I feel a lot more motivated to actually hit my goals.

Conclusion

So, do I recommend the Fitbit Ionic? Well, it depends. The main thing that my research into fitness wearables made clear to me was that there's no one device that will meet everyone's needs perfectly, so which one is right for you depends a lot on what you're planning to use it for. The Ionic is a great fitness device, but not a great smartwatch, and if you're a serious runner or cyclist, or a triathlete, there are better options out there. Bearing in mind that I haven't personally tested any of the devices on this list other than the Ionic, I can't really make specific purchase recommendations, but here are the devices I would recommend considering based on your needs:

The Ionic is a great choice if you're somewhat serious about running or cycling and don't care that much about fancy smartwatch features. If you have friends or family who also use Fitbit devices, then I think the Ionic is a great choice for the community aspect.
If you want a smartwatch first and foremost, and don't care as much about fitness, then the Apple Watch will probably be your best choice. If you're not an iPhone user, check out Samsung's Galaxy Gear lineup.
If you're a serious runner, cyclist, or especially if you're a triathlete, you should be looking at Garmin and Suunto devices. Again, I'd recommend looking at DC Rainmaker's blog for reviews of these devices. (How did you end up here instead of there in the first place?)
If your main sports are skiing, snowboarding, or surfing, take a look at the Nixon Mission, which is currently the only smartwatch to integrate with Trace. However, be wary of its limited battery life.

[^microsoft]: Disclaimer: I work for Microsoft, but the Microsoft Band has been discontinued, so Microsoft is no longer a competitor in this space.