Misplaced Pages

Revision as of 21:46, 17 February 2012

xargs is a command on Unix and most Unix-like operating systems used to build and execute command lines from standard input. Under the Linux kernel before version 2.6.23, arbitrarily long lists of parameters could not be passed to a command, so xargs breaks the list of arguments into sublists small enough to be acceptable.

For example, commands like:

 rm /path/*

or

 rm `find /path -type f`

will fail with an error message of "Argument list too long" if there are too many files in /path.

However the version below (functionally equivalent to rm `find /path -type f`) will not fail:

 find /path -type f -print0 | xargs -0 rm

In the above example, the find utility feeds the input of xargs with a long list of file names. xargs then splits this list into sublists and calls rm once for every sublist.

The previous example is more efficient than this functionally equivalent version which calls rm once for every single file:

 find /path -type f -exec rm '{}' \;

Note however that with modern versions of find, the following variant does the same thing as the xargs version:

 find /path -type f -exec rm '{}' +

xargs often covers the same functionality as the backquote (`) feature of many shells, but is more flexible and often also safer, especially if there are blanks or special characters in the input. It is a good companion for commands that output long lists of files like find, locate and grep, but only if you use -0, since xargs without -0 deals badly with file names containing ', " and space. GNU Parallel is the perfect companion to find, locate and grep if file names may contain ', " and space (newline still requires -0).

Examples

 find . -name "*.foo" | xargs grep bar

The above is equivalent to:

 grep bar `find . -name "*.foo"`

Note that the above command uses backticks (`), not single quotes ('). It searches all files in the current directory and its subdirectories which end in .foo for occurrences of the string bar. These commands will not work as expected if there are whitespace characters, including newlines, in the filenames. In order to avoid this limitation one may use:

 find . -name "*.foo" -print0 | xargs -0 grep bar

The above command uses GNU specific extensions to find and xargs to separate filenames using the null character;

 find . -name "*.foo" -print0 | xargs -0 -t -r vi

The above command is similar to the former one, but launches the vi editor for each of the files. The -t prints the command to stderr before issuing it. The -r is a GNU extension that tells xargs not to run the command if no input was received.

 find . -name "*.foo" -print0 | xargs -0 -I {} mv {} /tmp/trash

The above command uses -I to tell xargs to replace {} with the argument list. Note that not all versions of xargs supports the {} syntax. In those cases you may specify a string after -I that will be replaced, e.g.

 find . -name "*.foo" -print0 | xargs -0 -I xxx mv xxx /tmp/trash

The above command uses string xxx instead of {} as the argument list marker.

 find . -maxdepth 1 -type f -name "*.ogg" -print0 | xargs -0 -r cp -v -p --target-directory=/home/media

The command above does the same as:

 cp -v -p *.ogg /home/media

however, the former command which uses find/xargs/cp is more resource efficient and will not halt with an error if the number of files is too large for the cp command to handle. Another way to do it (choosing where to put your arguments) is:

 find . -maxdepth 1 -type f -name "*.ogg" -print0 | xargs -0 -I MYFILES cp MYFILES /home/media

The -I in the above command tells xargs what replacement string you want to use (otherwise it adds the arguments to the end of the command). You can also use -L to limit the number of arguments. If you do that, the command will be run repeatedly until it is out of arguments. Thus, -L1 runs the command once for each argument (needed for tools like tar and such).

The separator problem

Many UNIX utilities are line oriented. These may work with xargs as long as the lines do not contain ', " or space. Some of the UNIX utilities can use NULL as record separator (e.g. perl (requires -0 and \0 instead of \n), locate (requires using -0), find (requires using -print0), grep (requires -z or -Z), sort (requires using -z)). Using -0 for xargs deals with the problem, but many UNIX utilities cannot use NULL as separator (e.g. head, tail, ls, echo, sed, tar -v, wc, which).

But often people forget this and assume xargs is also line oriented.

The separator problem is illustrated here:

 touch important_file
 touch 'not important_file'
 find -name not\* | tail | xargs rm
 mkdir -p '12" records'
 find \! -name . -type d | tail | xargs rmdir

Running the above will cause important_file to be removed and will remove neither the directory called 12" records, nor the file called not important_file.

The proper fix is to use find -print0, but tail (and other tools) do not support NULL terminated strings:

 touch important_file
 touch 'not important_file'
 find -name not\* -print0 | xargs -0 rm
 mkdir -p '12" records'
 find \! -name . -print0 | xargs -0 rmdir

When using the syntax find -print0, entries are separated by a null character instead of a end-of-line. This is equivalent to the more verbose command:

 find -name not\* | tr \\n \\0 | xargs -0 rm

GNU Parallel is an alternative to xargs that is designed to have the same options, but be line oriented. Thus, using GNU Parallel instead, the above would work as expected.

For Unix environments where xargs does not support the -0 option (e.g. Solaris), the following can not be used as it does not deal with ' and " (GNU Parallel would work on Solaris, though):

 find -name not\* | sed 's/ /\\ /g' | xargs rm

References

1. GNU Core Utilities FAQ
2. Google search showing people forgetting -0
3. http://www.gnu.org/software/parallel/

External links

• xargs: construct argument lists and invoke utility – Shell and Utilities Reference, The Single UNIX Specification, Version 4 from The Open Group

Manual pages

• xargs(1) – GNU Findutils reference
• xargs(1): construct argument list(s) and execute utility – FreeBSD General Commands Manual
• xargs(1): construct argument list(s) and execute utility – NetBSD General Commands Manual
• xargs(1): construct argument list(s) and execute utility – OpenBSD General Commands Manual
• xargs(1): construct argument lists and invoke utility – Solaris 11.4 User Commands Reference Manual

• Unix text processing utilities
• Unix SUS2008 utilities