It took me a long time to admit to myself that the venerable Unix command line interface is stuck in the past and in need of a refresh, but it was a formative moment in my development as a programmer when I finally did. Coming from that perspective, I am very glad that there is a new wave of enthusiasm (coming especially from the Rust community) to build new tools that are fixing some of the problems with this very old and established user-interface.

The Role of the Unix CLI Interface#

To describe the Unix command line interface, “venerable” is definitely the right word: many programmers (including myself at some points of my life) have an awe of Unix and its role in computing history that has sometimes bordered on veneration.

Since the Unix operating system began development at Bell Labs in 1969, it has gone viral. That’s probably an understatement: Most modern operating systems descend from this original Unix, either directly through gradual code change (macOS and iOS are descended it from it through BSD), or through Linux (the kernel behind most servers and behind Android and ChromeOS) and its accompanying usermode software (much of which was part of the GNU project), which were designed to work like Unix due its familiarity for users and programmers.

Unix was and is billed not just as an operating system, but a philosophy. Among other things, its command line interface has been held up time and time again as an example of good design practices and an ideal realization of this philosophy, with its developer- and administrator-friendly orientation towards plain text files and with its modularity, especially as embodied in the concept of pipelining.

And as a result, when people say they know “the command line,” it’s almost certainly the Unix command-line interface that they’re talking about. And what’s more, many of us were taught it from texts that gushed about how great it is. But even the Unix command line interface, though part of a well-established standard, the topic of many books, and used by and intimately familiar to millions of programmers and admins across generations, is, in the end, just another computer interface for users and developers. And it has its flaws.

A Disappointing Ambiguity#

As I alluded to before, when I was a much younger programmer, I had an awe-struck veneration for Unix. One of my colleagues at an early job in my career referred to me as our company’s “Unix philosopher.” While I wasn’t sure whether he meant it as a compliment, at the time, I took it as one.

The first flaw that really got my attention in the Unix command line had to do with the mv command. I’m going to take some time explaining this flaw in detail, as it’s somewhat subtle, and as discovering it was a formative moment for me in my development as a programmer.

mv, as many of you know, is short for “move.” And while its job indeed includes moving files from one place to another, due to idiosyncracies of the Unix file system (if they can be called idiosyncracies when most file systems followed Unix’s lead on this), moving files and renaming files are closely related operations under the hood, causing the mv command to be both the “move” command and the “rename” command:

# Assume a file called 'draft-file'
# Assume a directory called 'final-docs'

# Rename 'draft-file' to 'final-file' and put it in 'final-docs'
mv draft-file final-file # rename 'draft-file' to 'final-file'
mv final-file final-docs # move 'final-file' into 'final-docs' directory

# Alternatively, one step:
mv draft-file final-docs/final-file

As you can see, there is no distinction between these operations. There is no option that you must enable to get the “moving” feature as opposed to the “renaming” feature. And this can result in surprises, which are bad in software development.

Consider this command again:

mv draft-file final-file

What does it do? It changes the name of the file from draft-file to final-file, keeping it in the same directory, right? Well, probably, and that’s almost certainly what the user intended, but what if someone, accidentally or intentionally, had created a directory called final-file? That command would be interpreted instead as moving draft-file into the final-file directory:

$ # Rename operation
$ touch draft-file
$ ls
draft-file
$ mv draft-file final-file
$ ls
final-file
$ ls final-file
final-file
$ rm final-file
$
$ # Move operation
$ mkdir final-file # Imagine someone else did this, or it was done by accident
$ touch draft-file
$ mv draft-file final-file
$ ls
final-file
$ ls final-file
draft-file
$ rm final-file
rm: cannot remove 'final-file': Is a directory
$ rm -rf final-file

Notice that if there is no color-coding enabled, a simple ls command doesn’t even distinguish the two situations, so you can’t tell which one happened without issuing a more specific command, as ls also has a dual role: it can either show you the names of the files you specify, if they are present, or it can show you the files in a directory you specify. The -d option disambiguates that you want the names and not the contents, but the default is still ambiguous.

In the case of the mv command, this potentially could even be a security vulnerability in a shell script (which is admittedly not a very secure platform). It is in any case an unnecessary complication.

The GNU version of mv has a -t option to indicate that the destination is not to be interpreted as a directory to put things in, and a -T option to show unambiguous intent for a target directory to be used. But these are extensions; the POSIX standard manual page for mv doesn’t mention them.

And while this GNU extension is helpful, especially in scripts that you know will only be run with the GNU version of mv (that is, not on macOS), I don’t think it goes far enough. Most people don’t know about them, and the possibility of surprise is still there.

Disillusioned#

When I realized this, it created a huge hole in my previous (admittedly unreasonable) esteem for the Unix command line interface. I realized that the ideal solution was something impractical, almost unthinkable to the younger version of me: mv should be deprecated in favor of two commands, one to do renaming, and one to do targeted directory-dropping.

This glitch in the mv command is just a gotcha to be aware of, one of many minor flaws to dance around when shell scripting. But I remember it strongly, because rather than being warned about it in a book, I discovered it myself, and therefore it was the distinct moment I realized that the command line interface would need to be improved at some point. And once the metaphorical levee was broken, I started noticing many inconveniences and problems in the traditional Unix CLI tools, often more relevant to my day-to-day workflow than this minor gotcha.

I ultimately came to read more critical sources about Unix, such as the famous UNIX-HATERS Handbook, and similar sources that emphasized the problems. And I’m very glad I went through this process, because before this, I was a naive CLI user and shell-scripter, trusting the system way more than I should, leaving myself open to serious problems.

Many Unix commands have gotchas and inconveniences, some I knew about before this revelation and brushed aside, others that I found out about later. tar has its idiosyncratic traditional syntax that many, many scripts (and people) still use, and inconsistency between platforms on whether you need -z to unpack a compressed archive. The way the shell itself worked also contained gotchas: What happens if you have files whose names start with a -? (Answer: Their names get misinterpreted as options, even if you didn’t type them but simply included them accidentally in a wildcard expansion.)

Among the more practical issues that particularly effect me, I want to emphasize two in particular: Why is find’s syntax so gnarly, so that you have to type out --name and explicitly specify the current directory? Why is it so hard to get grep to not display the pages-long lines of minimized Javascript or similar files when I want to only display the shorter lines from actual source files?

The Future#

Luckily, improvement is on its way. For the last two cherry-picked examples, there are new re-conceptions of find and grep -r that fix them (with new names, of course, so they’re not beholden to interface-compatibility), and I recommend them (dare I say such blasphemy?) over the traditional equivalents:

Don’t let their long names dissuade you; they are commonly installed as fd and rg, respectively, and come with such modern features as:

  • Normal command line syntax (fd)
  • Integration with git, the de facto standard version control system, by ignoring .git and .gitignore’d files by default (both)
  • Line length maximums (rg)
  • Modern leveraging of multithreading (both)
  • Better performance than their traditional counterparts

These are the only new Rust-based commands I’ve tried, but they’ve already vastly improved my workflow, so that I miss having them (fd especially) when SSH’d into relatively minimalist embedded devices. And I have reason to hope there’s more gems out there as part of this explosive movement to implement new Rust-based commands.

Whether people are doing this to improve their Rust chops, or because they’ve felt a need for a long time and Rust is just their PL of choice, it’s good to see some actual evolution in my day-to-day experience as a Unix CLI user. It hasn’t fixed mv – yet – but it’s good to see it evolving.

On the implementation side of things, I am also very happy to see a Rust project to reimplement the standard coreutils. The C implementations undoubtedly leave some performance and stability on the table, and a new implementation is long over-due. A fresh implementation of these utilities will hopefully also spark improvements to the interfaces.

And Meanwhile, in git-land#

On a related positive note, I learned very recently (in 2022) that git has (in 2019) fixed a problem similar to mvs: git checkout, ambiguous in a similar way, has been rendered unnecessary by the less ambiguous git switch and git restore.