yesthattom | OS X tip of the day

The “ditto” command has a --nocache option that thrills me.

The “ditto” command copies files, directories, etc. in an “Apple safe” way. It copies the resource forks and even generates the “._filename” files if copying to a file system that do not natively support resource forks. According to the man page:

--nocache

do not perform copies using the Mac OS X Unified Buffer Cache. Files read and written will not be cached, although if the file is already present in the cache, the cached information will be used.

That is... if you do a backup using “ditto --nocache” this operation won’t pollute your disk cache.

That is... the problem with copying large numbers of files is that the OS normally tries to cache any file it reads since there is a good chance you will want to read it again soon. However, when making a backup, you are reading many many (if not all) files and none (or nearly none) will be read a second time. Thus, you cache is now filled with useless junk. Meanwhile, before you began this whole operation the cache was most likely filled with good stuff: commonly read files, read only executable blocks, directories that are commonly read, etc.

One of the innovations from 1990’s Unix is the concept of a “unified buffer cache”. Previously there was a buffer cache for disk blocks, one for inodes, one for executable pages, and so on. The problem was that no matter how hard you tried one of those buffers was always too small (slowing performance), and another was always too big (wasting memory). Finally someone realized that if there was one big “unified buffer cache” then there would be better utilization of memory.

Since the beginning of multitasking systems computers ran slowly while backups were being run. However, at least when there were separate buffer caches at least backups only made disk access slow. Unified buffer cache systems now had a bigger performance problem during backups, but it was worth it. Backups were only a once-a-day thing, right?

New Unix sysadmins may not realize how unified buffer cache systems changed the industry. Prior, a large amount of system administrator time was spent monitoring memory utilization and making careful adjustments to the sizes of caches. Each adjustment required a reboot, if not a kernel rebuild. Unified buffer cache meant that the system was “self-tuning” and relieved system administrators from hours of work each week so they could work on something else. (Oddly enough, I always thought that 99% of the Unix systems ran with the defaults and most system administrators were ignorant of how bad their system ran. This made a small industry of “unix tuning experts” who would be hired for a week or two to get huge performance increases from a few simple adjustments that looked like magic. First these consultants were an endangered species when vendors came up with the brilliant idea to ship with better defaults. Then the unified buffer cache came and put a nail in that coffin. Of course, that only eliminated the consultants that didn’t move with the times. There are plenty of badly tuned systems out there. Just not as much low-hanging fruit.)

Anyway...

Backups are not the only time large amounts of data is copied. Now a days I’m often copying gigabytes of files just in the course of managing my machines. Finding that ditto has a “--nocache” option should be a big win.

Of course, I’m writing this on a train, disconnected from the ability to test this until I get home.