CLIX False Beginners (2)

Looking at ls.

Get It Try It

ls is the Unix command (program) used to inspect a file system. The key to ls - and the key to the Unix file system itself if not the operating system Unix itself - can be found in the command line arguments for this program.

Copy out the manual page for ls and paste it into a text editor such as Rixedit. Note the first two sections only for now.

[Forgot how to copy out a Unix manual page with CLIX? Go back to part one of the series.]

Looking at ls

NAME
     ls -- list directory contents

SYNOPSIS
     ls [-ABCFGHLOPRSTUW@abcdefghiklmnopqrstuwx1] [file ...]

That's a lot of switches!

The key (as always) with 'res IT' is to be able to find the forest for the trees. There are a lot of switches there you won't need; understanding how Unix and ls work will not force you to remember them all. Or even most of them. But a few are central to understanding the system.

-@      Display extended attribute keys and sizes in long (-l) output.

This is an Apple addition. Extended attributes are a powerful addition, especially to object oriented user systems.

-a      Include directory entries whose names begin with a dot (.).

Another important one. Nothing is really 'hidden' on Unix - it's merely kept out of the way. Files with names beginning with a dot are often hidden (ignored) by many Unix commands unless expressly told otherwise by the user.

-c      Use time when file status was last changed for sorting (-t) or long printing (-l).

Very important in the long run as you can see when important file status information is changed. This data field is something that can't be got to programmatically. It's a fine determinant of what's actually happened in a file system.

-d      Directories are listed as plain files (not searched recursively).

Used when you want to list the directory itself and not its contents.

-e      Print the Access Control List (ACL) associated with the file, if present, in long (-l) output.

Another important one as of 10.6 Snow Leopard: access control lists are further gunk that can be associated with files. They can override most other file controls. They're often redundant or just too risky to use. But you want to see when someone or something has put access control lists on your files.

-i      For each file, print the file's file serial number (inode number).

Showing the file inode (serial number) can in many cases be an important way to learn a lot about a file.

-l      (The lowercase letter ``ell''.) List in long format.

This is where you open the system to show all the data you wouldn't otherwise see. This is the most important switch for the command ls.

-O      Include the file flags in a long (-l) output.

Another important one as these flags can further control files in ways you'd not otherwise see. [That's the uppercase letter 'O' and not the number '0'.]

-R      Recursively list subdirectories encountered.

Used sparingly: it will list everything recursively. And that can be a lot!

-r      Reverse the order of the sort to get reverse order.

Reverse sort. Can occasionally be useful.

-T      When used with the -l (lowercase letter ``ell'') option, display complete time
        information for the file, including month, day, hour, minute, second, and year.

Is this the 'turbo' version of ls -l? It would seem so. At least partly.

Do yourself a favour and save the ls manual page somewhere. (You can always get it back easily with CLIX.)

The Unix File System

The following applies to all 'Unix' file systems, whether they be systems that 'support' Unix or real Unix file systems originally designed for use with Unix. Apple's HFS+ is a file system that's been significantly retooled for Unix compatibility.

Some introductory principles.

• All file systems have some sort of 'volume control block'. The VCB contains all the information on the 'logical device' (the volume). It's normally (preferably) a contiguous disk area near the logical start of the volume.
• The volume control block on Unix systems is called the ilist.
• An entry in the 'ilist' is called an iblock. The iblocks in the ilist are all of the same size.
• All information about a file (or directory) is in the iblock (except the name). This information includes where the file can be found on the volume; when it was created, last accessed, last modified, and when the iblock information was last modified; the iblock's 'index' (inode) in the ilist; the user and group the file belongs to; its size in bytes; its generic Unix file type; and so forth and so on. The iblock has everything except the name (path) and this data is found nowhere else.
• This means that all Unix directories need are names and inodes. This is the way things started back when Unix file names were up to 14 bytes in length. Directories of today are slightly more complex but not overly so. Additional data is for navigational purposes only.
• This means that you can have several file 'names' pointing to the same physical file (on the same volume). And that's where inspection of inodes comes in.
• All the low level routines have to do to find an iblock is to multiply the inode by the size of the iblock and offset that into the ilist. They open the ilist for reading (or writing) and then 'seek' to the offset they've calculated. Easy peasy.

The information in an iblock is comprehensive; there's little to be found outside it. (Access control lists and extended attributes can be exceptions as they're Apple 'extensions' of ordinary Unix file system semantics; ls takes care of ACLs; another program is needed to access the XAs.) But ls will tell you most anything you want to know about a file.

Reading the Long List

The key is to be able to interpret ls output. Take a fully packed ls command line and see what you get.



Copy out the output and paste it into a text editor.

total 36885
      2 drwxrwxr-t  31 root     admin  -          1122 Jun  8 23:41:34 2010 .
      2 drwxrwxr-t  31 root     admin  -          1122 Jun  8 23:41:34 2010 ..
     21 drwxrwx---   4 rixstep  admin  -           136 Sep 25 16:39:17 2009 .Spotlight-V100
     20 drwxrwx---   2 rixstep  admin  -            68 Jan 23 04:02:23 2010 .Trashes
 289250 -rw-r--r--   1 root     admin  -             0 Mar  4 06:37:16 2010 .com.apple.timemachine.supported
 195504 drwxrwx---   2 rixstep  admin  -            68 Sep 14 19:28:25 2009 .fseventsd
 289254 -rw-------   1 root     wheel  hidden   196608 Feb 17 15:51:07 2010 .hotfiles.btree
  26074 drwxr-xr-x   2 root     wheel  -            68 Sep 15 21:30:40 2009 .vol
    264 drwxrwxr-x+ 26 root     admin  -           884 May 21 02:52:52 2010 Applications
 203020 drwxrwxr-x  13 root     admin  -           442 Apr 23 21:52:03 2010 Developer
    364 drwxrwxr-t+ 53 root     admin  -          1802 Jan 22 07:31:41 2010 Library
  80501 drwxr-xr-x   2 root     wheel  -            68 Sep 15 21:32:50 2009 Network
    188 drwxr-xr-x   3 root     wheel  -           102 Mar 30 00:01:04 2010 System
  37235 drwxr-xr-x   4 root     admin  -           136 Jun 20 09:20:29 2010 Users
  26075 drwxrwxrwt   6 root     admin  -           204 Jun 20 06:41:34 2010 Volumes
  24031 drwxr-xr-x  39 root     wheel  -          1326 Mar 29 23:56:53 2010 bin
  80503 drwxrwxr-t   2 root     admin  -            68 Sep 15 21:33:05 2009 cores
 290786 -rw-------   1 rixstep  admin  -          1479 Apr 14 10:33:28 2010 css1
 290787 -rw-------   1 rixstep  admin  -          1479 Apr 14 10:33:28 2010 css2
    300 dr-xr-xr-x   3 root     wheel  hidden     4127 Jun 20 06:37:01 2010 dev
  24743 lrwxr-xr-x   1 root     wheel  -            11 Sep 15 21:33:15 2009 etc -> private/etc
      5 dr-xr-xr-x   2 root     wheel  hidden        1 Jun 20 06:37:29 2010 home
2694188 -rw-r--r--   1 root     wheel  -      18659362 Mar 30 00:32:01 2010 mach_kernel
      3 dr-xr-xr-x   2 root     wheel  hidden        1 Jun 20 06:37:29 2010 net
    170 drwxrwxrwx   6 root     wheel  -           204 Dec 22 02:44:50 2009 private
   4469 drwxr-xr-x  64 root     wheel  -          2176 Mar 29 23:56:55 2010 sbin
  24783 lrwxr-xr-x   1 root     wheel  -            11 Sep 15 21:33:18 2009 tmp -> private/tmp
    259 drwxr-xr-x  12 root     wheel  -           408 Apr 22 17:39:29 2010 usr
  24784 lrwxr-xr-x   1 root     wheel  -            11 Sep 15 21:33:22 2009 var -> private/var

A very interesting thing to note immediately is that all uppercase names are sorted before lowercase names. This is a Unix thing. ls has the switch '-f' to output data unsorted, which when running HFS+ means you get a different result entirely.

total 36885
      2 drwxrwxr-t  31 root     admin  -          1122 Jun  8 23:41:34 2010 .
      2 drwxrwxr-t  31 root     admin  -          1122 Jun  8 23:41:34 2010 ..
 289250 -rw-r--r--   1 root     admin  -             0 Mar  4 06:37:16 2010 .com.apple.timemachine.supported
 195504 drwxrwx---   2 rixstep  admin  -            68 Sep 14 19:28:25 2009 .fseventsd
 289254 -rw-------   1 root     wheel  hidden   196608 Feb 17 15:51:07 2010 .hotfiles.btree
     21 drwxrwx---   4 rixstep  admin  -           136 Sep 25 16:39:17 2009 .Spotlight-V100
     20 drwxrwx---   2 rixstep  admin  -            68 Jan 23 04:02:23 2010 .Trashes
  26074 drwxr-xr-x   2 root     wheel  -            68 Sep 15 21:30:40 2009 .vol
    264 drwxrwxr-x+ 26 root     admin  -           884 May 21 02:52:52 2010 Applications
  24031 drwxr-xr-x  39 root     wheel  -          1326 Mar 29 23:56:53 2010 bin
  80503 drwxrwxr-t   2 root     admin  -            68 Sep 15 21:33:05 2009 cores
 290786 -rw-------   1 rixstep  admin  -          1479 Apr 14 10:33:28 2010 css1
 290787 -rw-------   1 rixstep  admin  -          1479 Apr 14 10:33:28 2010 css2
    300 dr-xr-xr-x   3 root     wheel  hidden     4127 Jun 20 06:37:01 2010 dev
 203020 drwxrwxr-x  13 root     admin  -           442 Apr 23 21:52:03 2010 Developer
  24743 lrwxr-xr-x   1 root     wheel  -            11 Sep 15 21:33:15 2009 etc -> private/etc
      5 dr-xr-xr-x   2 root     wheel  hidden        1 Jun 20 06:37:29 2010 home
    364 drwxrwxr-t+ 53 root     admin  -          1802 Jan 22 07:31:41 2010 Library
2694188 -rw-r--r--   1 root     wheel  -      18659362 Mar 30 00:32:01 2010 mach_kernel
      3 dr-xr-xr-x   2 root     wheel  hidden        1 Jun 20 06:37:29 2010 net
  80501 drwxr-xr-x   2 root     wheel  -            68 Sep 15 21:32:50 2009 Network
    170 drwxrwxrwx   6 root     wheel  -           204 Dec 22 02:44:50 2009 private
   4469 drwxr-xr-x  64 root     wheel  -          2176 Mar 29 23:56:55 2010 sbin
    188 drwxr-xr-x   3 root     wheel  -           102 Mar 30 00:01:04 2010 System
  24783 lrwxr-xr-x   1 root     wheel  -            11 Sep 15 21:33:18 2009 tmp -> private/tmp
  37235 drwxr-xr-x   4 root     admin  -           136 Jun 20 09:20:29 2010 Users
    259 drwxr-xr-x  12 root     wheel  -           408 Apr 22 17:39:29 2010 usr
  24784 lrwxr-xr-x   1 root     wheel  -            11 Sep 15 21:33:22 2009 var -> private/var
  26075 drwxrwxrwt   6 root     admin  -           204 Jun 20 06:41:34 2010 Volumes

This isn't ideal either (see net and Network) as HFS+ ignores case when sorting - that's not the way a dictionary would do it. But perhaps it's easier to work with.

There are nine (9) columns of data in the above output.

1. Inode. The file's index in the ilist.
2. Mode. This includes the file type, the file's permissions, and other funky stuff.
3. Number of links. This is for either the number of references to a file or the number of references in a directory.
4. User. The user account the file belongs to. The file's 'owner'.
5. Group. The group account the file belongs to.
6. Special flags. One is used in the above listing: 'hidden'.
7. Size. In bytes.
8. Date/Time. For this example ('-c') it's when the file's iblock was last changed.
9. Name. Self-explanatory.

The 2nd field ('mode') is initially the most intriguing. Take a look at the listing for .fseventsd. What can be known about it?

195504 drwxrwx---   2 rixstep  admin  -            68 Sep 14 19:28:25 2009 .fseventsd

1. It's inode is 195504.
2. It's mode is 'drwxrwx---' (whatever that means).
3. It has two (2) links.
4. It's owned by user account rixstep and group account admin.
5. It has no special flags.
6. It's 68 bytes in size.
7. Its iblock was last modified on 14 September at 19:28:25 (local time).

So what does that mode 'drwxrwx---' mean?

√ There are normally 10, sometimes 11 characters in that field. The first seems to be either a 'd' or a '-'. The rest appear in triads of 'rwx' (or '-' characters). Sometimes there's a '+' appended.

√ The 'd' at the beginning means 'directory'. You can therefore know that .fseventsd is a directory. This explains both the size (68 bytes for 'empty' directories) and the number of links (2 for empty directories as they still have '.' and '..').

√ The nine characters after the 'd' are the file permissions for user, group, and 'other' in that order. You can do three things with a Unix file: read it, write to it, execute it. This applies to directories as well. ('Execute' implies something different for directories.)

The permissions for .fseventsd allow user rixstep to both read, write, and 'execute' the directory .fseventsd. They also allow any member of the group admin to do the same. But 'other' - meaning 'anyone else' - can't do anything at all. The 'r', 'w', and 'x' stand namely for 'Read', 'Write', 'eXecute'. (To 'execute' a Unix directory is to 'enter' it - to use it as your current working directory.)

The occasional '+' at the tail end is explained in the ls manual page.

If the file or directory has extended security information (such as an access control list) the permissions field printed by the -l option is followed by a '+' character.

It's possible to show this information as well by adding the '-e' switch.

total 36885
      2 drwxrwxr-t  31 root     admin  -          1122 Jun  8 23:41:34 2010 .
      2 drwxrwxr-t  31 root     admin  -          1122 Jun  8 23:41:34 2010 ..
 289250 -rw-r--r--   1 root     admin  -             0 Mar  4 06:37:16 2010 .com.apple.timemachine.supported
 195504 drwxrwx---   2 rixstep  admin  -            68 Sep 14 19:28:25 2009 .fseventsd
 289254 -rw-------   1 root     wheel  hidden   196608 Feb 17 15:51:07 2010 .hotfiles.btree
     21 drwxrwx---   4 rixstep  admin  -           136 Sep 25 16:39:17 2009 .Spotlight-V100
     20 drwxrwx---   2 rixstep  admin  -            68 Jan 23 04:02:23 2010 .Trashes
  26074 drwxr-xr-x   2 root     wheel  -            68 Sep 15 21:30:40 2009 .vol
    264 drwxrwxr-x+ 26 root     admin  -           884 May 21 02:52:52 2010 Applications
 0: group:everyone deny delete
  24031 drwxr-xr-x  39 root     wheel  -          1326 Mar 29 23:56:53 2010 bin
  80503 drwxrwxr-t   2 root     admin  -            68 Sep 15 21:33:05 2009 cores
 290786 -rw-------   1 rixstep  admin  -          1479 Apr 14 10:33:28 2010 css1
 290787 -rw-------   1 rixstep  admin  -          1479 Apr 14 10:33:28 2010 css2
    300 dr-xr-xr-x   3 root     wheel  hidden     4127 Jun 20 06:37:01 2010 dev
 203020 drwxrwxr-x  13 root     admin  -           442 Apr 23 21:52:03 2010 Developer
  24743 lrwxr-xr-x   1 root     wheel  -            11 Sep 15 21:33:15 2009 etc -> private/etc
      5 dr-xr-xr-x   2 root     wheel  hidden        1 Jun 20 06:37:29 2010 home
    364 drwxrwxr-t+ 53 root     admin  -          1802 Jan 22 07:31:41 2010 Library
 0: group:everyone deny delete
2694188 -rw-r--r--   1 root     wheel  -      18659362 Mar 30 00:32:01 2010 mach_kernel
      3 dr-xr-xr-x   2 root     wheel  hidden        1 Jun 20 06:37:29 2010 net
  80501 drwxr-xr-x   2 root     wheel  -            68 Sep 15 21:32:50 2009 Network
    170 drwxrwxrwx   6 root     wheel  -           204 Dec 22 02:44:50 2009 private
   4469 drwxr-xr-x  64 root     wheel  -          2176 Mar 29 23:56:55 2010 sbin
    188 drwxr-xr-x   3 root     wheel  -           102 Mar 30 00:01:04 2010 System
  24783 lrwxr-xr-x   1 root     wheel  -            11 Sep 15 21:33:18 2009 tmp -> private/tmp
  37235 drwxr-xr-x   4 root     admin  -           136 Jun 20 09:20:29 2010 Users
    259 drwxr-xr-x  12 root     wheel  -           408 Apr 22 17:39:29 2010 usr
  24784 lrwxr-xr-x   1 root     wheel  -            11 Sep 15 21:33:22 2009 var -> private/var
  26075 drwxrwxrwt   6 root     admin  -           204 Jun 20 06:41:34 2010 Volumes

Now suddenly there are two more lines interspersed with the others, belonging to the lines immediately above.

    264 drwxrwxr-x+ 26 root     admin  -           884 May 21 02:52:52 2010 Applications
 0: group:everyone deny delete
    364 drwxrwxr-t+ 53 root     admin  -          1802 Jan 22 07:31:41 2010 Library
 0: group:everyone deny delete

They're access control lists on the two directories Applications and Library. They each have one (access control) entry (the ACEs both have index 0) and they each say 'everyone' is denied the ability to delete the target - which for those directories is a good thing.

Homework

Now the fun begins. You're to create a file - call it what you want - that has commands to successfully perform the following operations. [You might need to consult that manual page again - remember how to invoke it in CLIX?] Be sure your commands can be run successfully no matter where you save your command file.

1. Perform a simple listing of your home directory (just file names).
2. Perform a long listing of your home directory.
3. Perform a simple listing of your home directory showing all files.
4. Perform a long listing of your home directory showing all files.
5. Perform a simple listing of your home directory showing only the file names and inodes.
6. Perform a long listing of your home directory showing special flags (such as 'hidden').
7. Perform a simple recursie listing of your home directory.
8. Perform a long listing of your home directory showing all files and with both inodes and flags.
9. Perform the same listing as above but with user and group IDs instead of the user/group names.
10. Perform the same listing as above but now recursively.
11. Perform a simple listing of your home directory in reverse order.
12. Perform a listing of your root directory showing extended attributes.
13. Perform a listing of your root directory showing times for 'status' (iblock) changes.
14. Perform a simple unsorted listing of your root directory.
15. Perform a listing of your root directory with access control lists.
16. Perform a 'descriptive' listing of your root directory with special characters denoting file types.
17. Perform the same sort as above but with unsorted output.
18. Perform a 'streamed' listing of your home directory with comma-separated file names.
19. Perform a listing of your home directory sorted (ascending) according to size.
20. Perform a listing of your home directory sorted (descending) according to size.
21. Perform a simple listing of your root directory with sizes given as 512-byte blocks.
22. Perform a long listing of your root directory with sizes given as 512-byte blocks.
23. Perform a long listing of root with XAs, ACLs, inodes, flags, sizes in 512-byte blocks, and showing all files.
24. Perform a long listing as above but unsorted and adding descriptive characters for file types.

More Homework

For the intellectually eager.

• ls is located in /bin. Use the whereis command to verify this.
• Perform a listing of /bin. Can you find any files that share the same inode? Are their sizes identical as well? How about their permissions? How about their date/time stamps? Check as many date/time stamps as you can.
• /usr/bin is another Unix command directory. Can you see which files - excluding directories - have the most links?
• Perform a listing of /dev and see if you can determine what's going on there and how to interpret the 'mode' fields for those files.

More Homework Still

25. Your system's virtual memory (swap files) are located in /private/var/vm. Create a command to list these swap files along with their sizes. How much swap are you currently using? Did you find anything else in that directory?
26. Can you make any sense out of what you find in /private/tmp? Did you find any new file types?
27. What happens when you attempt to list /private/etc recursively? Make a note of any paths that gave you trouble.
28. What's in /private/var/folders? Did you run into any further trouble?
29. Do an identical scan of /private/var/log. What if anything gave you trouble here?

See Also
Learning Curve: CLIX False Beginners (1)
Learning Curve: CLIX False Beginners (2)
Learning Curve: CLIX False Beginners (3)

CLIX: Learn how to fish
Rixstep FTP: Download CLIX