Dealing with 'Argument list too long' Problems

The solution to this problem is to the “Argument list too long“ error when trying to archive a large number of files is the “-T“ option of the “tar” command to pass in a list files generated by a “find” command:

  1. Create a list of the files to be archived using the “find“ command:

    $ find . -name="*.tre" > filelist.txt
  2. Use the “-T” option of the “tar” command to pass in this list of filenames:

    $ tar cvjf archive.tbz -T filelist.txt

If you want to delete a long list of files, however, this approach will not work, as “rm” does not support the very convenient “-T”/”--files-from” flag or the equivalent (so convenient, in fact, that I have started adding this to virtually every file-processing script or program that I write). Luckily, however, “finddoes support a “-delete” flag, so to recursively delete all files and directories:

find path/to/dir -delete

You can use a “-type f” argument to limit the operation only to files, and the “-depth 1” argument to limit the operation only to the current directory, so that:

find path/to/dir -type f -depth 1 -delete

will delete files in the specified directory, without touching subdirectories or the files within them. Note that using “find” in conjunction with the “-delete” flag is probably faster than any other approach, including using the “-exec rm \;” argument to “find” or looping over the files in a shell script. However, if you want to get rid of an entire directory and all sub-directories, then simply issuing an “rm -r” is, of course, a better performer.