Ok, so there’s a bit of geekery below the cut for those of you who like looking through shell script source code. All others should move on…<\/p>\n
Still there? Ok, you have been warned.<\/p>\n
Here’s the background: during the summer, my new job tends to have a few slack periods when people aren’t bringing in new work. Rather than sit around twiddling our thumbs, we go back through our old work and see what needs improvement. There’s a rather large database of images which includes a “quality control” feedback form. Images that score low on quality get flagged for review and repair.<\/p>\n
The trouble is, we’ve got literally tens of thousands of images in hundreds of directories. Gathering up the bad ones takes hours. So I wrote a script to do that for us. It was able to go through about 10,000 images and find 120 or so of the ones that we needed in about seven minutes.<\/p>\n
Code is below the break…<\/p>\n
<\/p>\n
##<\/p>\n # This script will page through a list of file names and then<\/p>\n # perform a recursive search of all files and directories in a<\/p>\n # user specified \"source\" path, copying any matching files to a<\/p>\n # second, user specified \"destination\" directory in their local<\/p>\n # machine's home directory.<\/p>\n ##<\/p>\n ## Has the script been invoked correctly<\/p>\n if [ -z \"$3\" ]; then<\/p>\n echo command $0 needs source, destination and matchlist<\/p>\n exit<\/p>\n fi<\/p>\n ## Assign command-line arguments to variables<\/p>\n source=$1<\/p>\n destination=~\/$2<\/p>\n matchlist=$3<\/p>\n ##<\/p>\n # NOTE! The ~\/ in the destination assignment forces the script<\/p>\n # to copy files to the user's home directory. This should help<\/p>\n # avoid a potential recursive condition where the destination<\/p>\n # directory is one of the ones being searched...<\/p>\n ##<\/p>\n ## Check if the source, destination and matchlist actually exist<\/p>\n if [ ! -d $source ]; then<\/p>\n echo $source is not a directory<\/p>\n exit<\/p>\n fi<\/p>\n if [ -e $destination ]; then<\/p>\n echo $destination already exists<\/p>\n exit<\/p>\n fi<\/p>\n if [ ! -e $matchlist ]; then<\/p>\n echo $matchlist does not exist<\/p>\n exit<\/p>\n fi<\/p>\n ## Create destination directory<\/p>\n mkdir $destination<\/p>\n check=$?<\/p>\n if [ ! $check -eq 0 ]; then<\/p>\n echo error creating destination directory<\/p>\n exit<\/p>\n fi<\/p>\n ## Create empty log files<\/p>\n touch $destination\/not_found.txt<\/p>\n touch $destination\/logfile.txt<\/p>\n ## Create lookup file from source directories<\/p>\n echo Building lookup table...<\/p>\n ls -R $source > $destination\/files_searched.txt<\/p>\n ## Prepare to loop through files in source<\/p>\n line_count=$(wc -l < \"$matchlist\")\n\n\n\necho there are $line_count lines in the search file\n\n\n\ncount=0\n\n\n\n##\n\n# Now to go through the specified matchlist file line by line.\n\n# First, 'grep' checks to see if 'find' would be fruitful -\n\n# this saves a lot of time when there are large numbers of\n\n# files to search through. The results of the lookup check are\n\n# logged, and the script continues with the 'find' command to\n\n# get the full directory path, and to make sure that the hit\n\n# isn't for a directory instead of a file. If 'find' fails, it\n\n# logs the query to the not_found.txt file. On success, it\n\n# copies the file and logs the action.\n\n##\n\n\n\nwhile [ \"$count\" -le $line_count ]\n\ndo\n\n read query\n\n echo looking for $query\n\n\n\n## Check the lookup file first - it's faster\n\n\n\n grep -i \"$query\" $destination\/files_searched.txt\n\n check=$?\n\n if [ ! $check -eq 0 ]; then\n\n echo $query not found in lookup file\n\n echo \"$query\" not found in lookup file >> $destination\/not_found.txt<\/p>\n else<\/p>\n ## If it's in the lookup file, get exact location and copy<\/p>\n result=`find $source -iname $query -type f`<\/p>\n if [ -z \"$result\" ]; then<\/p>\n echo NOTICE - $query not found after file search - it may be a directory<\/p>\n echo NOTICE - \"$query\" not found after file search - it may be a directory\\<\/p>\n >> $destination\/not_found.txt<\/p>\n else<\/p>\n cp $result $destination<\/p>\n echo copied \"$source\"\/\"$query\" to \"$destination\"\/\"$query\"\\<\/p>\n >> $destination\/logfile.txt<\/p>\n fi<\/p>\n fi<\/p>\n let \"count += 1\"<\/p>\n done <\"$matchlist\"\n\n\n\nexit 0\n\n<\/code><\/p>\n","protected":false},"excerpt":{"rendered":" Ok, so there’s a bit of geekery below the cut for those of you who like looking through shell script source code. All others should move on… Still there? Ok, you have been warned. Here’s the background: during the summer, … Continue reading #!\/bin\/bash<\/p>\n