Nameless Temporary Files

Linux 3.11 rc2 

Here's an interesting snippet from Linus's announcement post regarding Linux 3.11 rc2:

 (a) the O_TMPFILE flag that is new to 3.11 has been going through a
few ABI/API cleanups (and a few fixes to the implementation too), but
I think we're done now. So if you're interested in the concept of
unnamed temporary files, go ahead and test it out. The lack of name
not only gets rid of races/complications with filename generation, it
can make the whole thing more efficient since you don't have the
directory operations that can cause serializing IO etc.

Interesting idea!  Temporary files that aren't burdened with having to have filenames.

It will be some time before sysadmins see this feature in production, but in this case I think it's best to get the word out early, especially since this new feature could cause "mystery drive space exhaustion".

Right now, the only discrepancy between "df" and "du" numbers is due to deleted files with still-opened file descriptors.  With this new feature, it appears that nameless temporary files will join the ranks of hard-to-spot possible root causes of space exhaustion.

It's not clear how these new files will be identifiable / distinquished, for example, in the output of "lsof".  

As I learn more about this new feature, I'll be sure to write about it.

Why don't we have a way in Linux to know when a particular file was created?

Linux DOES have a way!  

The various filesystems have what they have, no more no less - pondering why they are as they are isn't productive.  They don't track creation-time metadata properly, and that's that.  The GREAT news is that the design of a workable solution isn't complex at all.  It's pretty straightforward.

The inotify kernel subsystem has been part of Linux since 2005 but it's still relatively unknown.  You can use it to learn when new files are added to directories, among other things.

So, if you choose to solve this problem, you'll build a daemon to monitor new files as they're created, and put their create times into a dataset you can later query.

Start with logic which recurses over whatever directory trees you wish to track, creating "inotify watches" on each directory.

Use a loop which calls "select" across that large array of file descriptors, one-per-directory, and reads the inotify events from the individual fds as they happen.  IN_CREATE events are the ones you'll be looking for - those indicate new files were created.  

Capture the ctime of the file as soon as you have received the IN_CREATE event indicating it was created, and, viola, you have it's "cr_time".

Next.  Implement in whatever way you prefer a persistent associative array of filenames -> creation timestamps.  

You might also implement the inverse, mapping creation timestamps to the file or files which were created at that time, to whatever granularity you prefer.

You can then query the creation time for a given file quite straightforwardly, and if you've implemented the inverse as I mentioned, you can query which files were created between two timestamps as well.

If you named it "pfcmd", short for "Paul's File Creation Monitor Daemon", I wouldn't mind one bit.  :-)

How to be almost-root

Today, most of us own our own Linux computers, or at least, our employers do, but they're signed out to us and dedicated for our use.  

If you have 'root' access on the computer, and consider it basically 'yours' - quite possibly you'll want to be able to look around at ALL of the files on the system, without first having to escalate to 'root'.

It's safer this way, by the way.  You should be able to look at ALL the files without having to escalate to root privilege.  How to do that?

If your filesystem(s) support ACLs, any regular user can be given this level of access.   For linux, the command 'setfacl' can be used to do this:

setfacl -R -m u:whoever:r /

The above recursively modifies access for the user whoever, to include "r".  It applies to ALL files and ALL directories.

setfacl -d -R -m u:whoever:r /

The above recursively modifies the DEFAULT acls for all directories such that they'll give the user whoever read access on any NEW files created in the future. (that's a REALLY REALLY REALLY cool feature!)

Now... the issue gets more complex.  "execute access" means different things for directories than it does for files.  Execute permission on a directory allows the user to list what files are in the directory.  Most people would lump that in with "reading" it.

find / -type d -exec setfacl -m u:whoever:rx {} \;

The above gives both read and execute permission for the user whoever to all directories.

Note, together these aren't perfect regarding NEW content.  The DEFAULT acls concept doesn't differentiate between new files in a directory and new subdirectories in that directory.  So, with the above, any NEW directories created after the "find" is run will have "r" permissions, not "rx" permissions, for the user whoever.

You might setup a nightly cron job to repeat the "find" command above - that'll take care of new directories and ensure you have "x" on them the next day.

If you have questions or concerns about ACLs just let me know and I'll be happy to help as best I can.

Finding Duplicate Folders

There isn't one specific linux utility that does this.  As is very often the case with Linux, a combination of tools must be used together to meet your goal.

The first challenge here is, What's your definition of duplicated?  

Almost certainly, the two folders don't have the same name.  Almost certainly, the two folders weren't created at exactly the same time.  The files within the two will most probably have different time stamps - access times, create times - as well.  What about ownership?  Permissions?  As you can see, there's a LOT about a directory that might differ.

I'll focus on whether the content of the folders appears to be duplicated - i.e. same number of files, with same filenames, all duplicates.  

You can achieve this as follows.

First, in a shell, change directory to the top of whatever tree you want to search for duplicated directories.  Then run the following.

for d in `find $CWD -type d -print`; do \
echo `cd $d;ls -1ARs .|cksum|sed 's/ /_/'` $d;done| \
awk '{c[$1]++; s[$1]=s[$1] " " $2} END {for (i in c) {if (c[i]>1) print s[i]}}'

Pretty, isn't it?  ...pretty ugly! Let's break it down.

for d in ` find $CWD -type d -print`;  do something; done

find gets run with $CWD as an argument, and asked to tell us every directory within the current directory.  Because we used $CWD - a fully-qualified pathname - we get back fully qualified pathnames, rather than relative pathnames.  This makes it possible for us to tell our script to change directory into each directory in turn - which is needed for a reason that'll become evident shortly.

With the above, in each iteration of the for loop, "$d" will have a directory name.

Let's look at the "something" that gets done with each directory name.

echo `cd $d;ls -1ARs .|cksum|sed 's/ /_/'` $d

That says to output a line, with the results of a command pipeline, followed by a space character, then the directory name.

cd $d;ls -1ARs .|cksum|sed 's/ /_/'

We "cd" into $d  - not caring where we were before, relative to $d, which is only possible because the directories are fully qualified.  I.e. we're not saying "cd ../otherdir" - we're saying "cd /home/joe/stuff/otherdir"

Now that we're in that directory, we can refer to it as "." - thus, working around the fact that our two otherwise "identical" directories probably don't have the same name.

Once we're in that directory, we do an "ls -1ARs ." - which will produce an identical listing on two directories which have the same files, taking the same number of blocks.

We don't really care what the output has to say - we care whether it's the same as the output from some other directory.  'cksum' to the rescue.  

cksum calculates CRC checksums, and outputs that checksum, with a byte count.  It's quite happy to checksum "stdin" which is what we do - we send the output from "ls" into cksum.    We then change the space between the resulting checksum and the byte count to an _ making it a single "word".

So... that "echo" creates output that looks kind of like this:

237894789783_1823 ./directory_one
367892367829_2290 ./directory_two
378927829789_4430 ./directory_three
367892367829_2290 ./duplicate_of_directory_two

Awk comes in REALLY handy for the next step - figuring out, across all of those checksums, which directories are duplicates.

awk '{c[$1]++; s[$1]=s[$1] " " $2} END {for (i in c) {if (c[i]>1) print s[i]}}'

We create two associate arrays - "c" and "s".  They get created automagically just by being referenced, so that works out great.

For each line of input, we run the first {block}:  c[$1]++; s[$1]=s[$1] " " $2

"c" (count) at index $1 (the checksum_size string) gets incremented.  It defaults to zero, so that works great.

s[$1] gets the directoryname added to whatever it currently holds.  It defaults to empty string, so that works great as well.

Then, once we're out of input lines... we run the "END" {block} - for (i in c) {if (c[i]>1) print s[i]}

That loops over every entry in the c array, looking for ones with more than one for a count, then prints out the corresponding entry from the s array.

The result, as you might expect if you've been following this all the way through (in which case... congratulations!) will be something like this:

% for d in `find $CWD -type d -print`; do echo `cd $d;ls -1ARs .|cksum|sed 's/ /_/'` $d;done|awk '{c[$1]++; s[$1]=s[$1] " " $2} END {for (i in c) {if (c[i]>1) print s[i]}}'
 ./directory_two ./duplicate_of_directory_two

... each line of output will identify two or more directories that have the exact same content.  almost.

I say "almost" because, in fact, the "size" option on "ls -1ARs" is in blocks. 

A file could have some extra content compared to it's other copy, and this would miss that until the extra content was sufficient in size to cause the file to grow in blocks.  

That means the above approach might have "false positives" - it might identify two directories as being duplicates, when there's actually a slight difference in one or more files within those directories.

Many people would consider this a pretty awesome feature, rather than a problem.  Indeed... this can identify "almost duplicate" directories.

The 100% accurate way would be to cksum each file.  This is HUGELY costly. It works, but it takes FOREVER to run.  If you have "forever" to wait, here's the 100% accurate approach:

for d in `find $CWD -type d -print`; do echo `cd $d;cksum $(find . -type f -print)|cksum|sed 's/ /_/'` $d;done|awk '{c[$1]++; s[$1]=s[$1] " " $2} END {for (i in c) {if (c[i]>1) print s[i]}}'

Instead of using "ls" we cksum all of the files in ".", then cksum the OUTPUT of all those cksums.

Since that involves reading every single byte of every single file in every single directory being scanned... it's very expensive to run.  That's why I presented an alternative that might possibly have a few "false positives" but takes much less processing power first.

Wrapping this up (as if it wasn't already way too long)... you could indeed use the first approach, grab the output listing all the directories that might be duplicates, then feed THAT list to a routine that used checksums to be 100% sure before identifying the directories as duplicates.  I'll leave that as an exercise in leveraging all of the techniques I've shared above.