dotfiles vs. the invisible bit

When trying to engage people in a text you're writing, a common approach is to introduce the concept you're discussing using a narrative, a short story vaguely related to the topic at hand. Joel Spolsky is a master at this. He used to serve in the army, so he often introduces articles by starting with an army story. Hey, I can do that, too, I also did military service! ¹

I used to serve as a medic and driver. This is probably the most laid-back thing you can do in the army. In a combat situation, medics, especially those who are also drivers, are typically not risking their lives, so there's no need to drill them until they put the word of their commander above their need for self-preservation. Doing service as a medic is akin to working in a highly professional, focused company; hierarchy doesn't count for too much.

As it turns out, we did not have enough lieutenants in our corps. Strangely, we ended up getting a lieutenant from the infantry. Needless to say, he thought his infantry approach to leading worked everywhere, and promptly started his first day by yelling at us, which was greeted by blank stares, followed by roaring laughter. At first, the poor fellow was rather confused by our insubordination, but he eventually figured out that talking to people worked a lot better than yelling at them.

The point of all this is that just because something seems to be the same doesn't mean that it is.

Most operating systems support the concept of "invisible files." Basically, an invisible file is a file that the user can't see using the standard file browser. But the similarities end there.

Unix

On many Unix-like systems, files whose name starts with a dot are not shown to the user. Obviously, users can easily add dots to names and make files invisible. Just as easily, they show invisible files again. Typing

ls -a

in the Terminal will list all files except for . (the current directory) and .. (the parent directory). Other than not being shown to the user by default, files starting with a dot typically behave just like every other file.

In other words, this is a way for the user to get rid of clutter; to make files she doesn't normally want to see invisible. It's a way to trim down long file listings, so only relevant files are shown.

Mac OS 9 and earlier

Apple also had invisible files in their pre-OS X systems. Whether a file was invisible was defined by its meta data. To make a file invisible, a bit could be set on the file. That would make the file invisible in all views; in Mac OS 9, there is no built-in way to see invisible files, or to make them visible.

In other words, invisible files in Mac OS 9 were a way for the system to hide files from the user. Typically, these were files that the system needed; files which the user was not supposed to see or tinker with. The invisible bit was a flag which allowed the system to keep files away from users.

Mac OS X

So when Apple bought NeXT and started working on OS X, they came upon a problem: On the one hand, OS X was supposed to be a Unix-like system. It has the BSD subsystem, it has a Terminal, it has ls -a.

On the other hand, it also had to support many concepts from Mac OS 9. HFS, Apple's file system, wasn't going anywhere. The invisible bit was still there.

So, like the infantry lieutenant yelling at medics because that's all he ever knew, Apple decided that Unix dot files should behave like Mac OS 9's invisible files. And just like that, files starting with dots were not visible anymore.

Unfortunately, this is a bit of a problem. While Apple took the concept of hiding files whose name starts with a dot from Unix, they left the semantics from Mac OS 9 intact. In other words, files which the user made invisible behave as if the system didn't want the user to touch them. In fact, Mac OS X actually prevents you from starting a file with a dot:

And if you do manage to start a file name with a dot, you'll never see the file again using Apple's default software. This is just plain user-unfriendly. It's my system, and I should be able to use whatever name I could possibly want.

Apple took the Unix syntax for invisible files, but kept the Mac OS 9 semantics.

The Problem

Why is this a problem? Because dotfiles and invisible files are not the same. Here's an example:

Due to reasons I might some day discuss in another article, Mac OS X stores some meta data in files named .DS_Store.² Many folders in Mac OS X contain such a .DS_Store file. .DS_Store files sometimes contain names of files stored in the folder.

As you can see from the name, Apple intended .DS_Store files to be invisible. These files are not supposed to be seen or opened by users. So Apple put a dot in front their name. Unfortunately, they originally neglected to also set the invisible bit. This becomes an issue if you do something like using an FTP client to mirror a local folder to a network. The FTP client might not know that the .DS_Store file is supposed to be invisible, thus happily copying it to a public web server, where it can then be opened and viewed by anyone. Obviously, it's typically not a good idea to publish file listings on the Internet. Soon after the issue became public, Apple set the invisible bit, and developers creating FTP clients changed their code so dotfiles were ignored during mirroring.

Which, of course, isn't the right thing to do, either.

Often, server configurations are stored in dotfiles. These are files that are supposed to be editable by administrators, and they are supposed to be mirrored to servers, but they start with a dot because you typically only edit them once, and then leave them be. An example of such a file is ".htaccess", which contains directives telling the web server Apache such things as which directories are password-protected.

Changing FTP clients to ignore dotfiles means these files are not mirrored to the server anymore. Which means that if an administrator decides to upload a password-protected directory, the directory may be password-protected on his local installation, but mirroring it to the server will only mirror the directory's content, and leave it unprotected on the server.

As has become obvious, many issues stem from Apple's decision to treat dotfiles and files with the invisible bit set the same, when in reality, they are different concepts.

dotfiles are files the user wants to be invisible. Files with the invisible bit set are files Apple wants to be invisible. One is a way for users to manage their files. The other is a way for Apple to prevent users from accessing files. Mixing the two concepts only causes problems and confusion.

The proper solution would be to treat files starting with a dot just like every other file. Let users start files with dots. Don't make them invisible. If a file needs to be invisible, set the invisible bit. Like the infantry lieutenant who had to stop treating medics like infantry soldiers, Apple should accept that forcing everyone to adhere to their vision may end up being detrimental.