Bash Script Automatic Exit on Command Failure

Stick this at the top of your multi-command BASH job scripts: set -e -o pipefail Exits on failure (== non-0 return code from any commands in the script) instead of cheerfully executing all subsequent lines in what is bound to result in a cascade of errors (or worse). More information in the excellent post here.

Read more

Scheduling Jobs with Systemd

In a previous post, I discussed some of the different options available to you on most POSIX system to automate the running of jobs on schedule. Here is a brief recipe on how to actually use the modern/trendy and actually functionally beneficial “systemd” to do this as a normal (non-root). In addition to the job file or script itself, you will need to create two files: a timer file and a service file (also referred to as timer unit and service unit).

Read more

Scheduling Jobs: Cron vs. Anacron vs. Systemd

Today, you generally have three choices for a time-based system scheduler: “Cron”, “anacron”, and “systemd”. There is a lot of information out on the web about these, so here I shall just summarize the basics from a practical standpoint to answer the question “which scheduler/timing system should I use?” “Cron” is the traditional timer. It is available everywhere, and tried, tested, loved, and it just works most of the time for most people.

Read more

Supercharging Your Mouse Ergonomics in Linux, Part 1: Scrolling with Mouse Movements

Sometimes I use a touchpad, and at other times an optical mouse, but my main workstation pointer device is a trackball, in particular, the Logitech M570. This model comes with two little index finger buttons, labeled “Forward Button” and “Back Button”, in the image below: which by default send you forward and backward through your web page visit chain in your browser. I find this mapping a phenomenal waste of two very useful buttons.

Read more

How to Install R on an HPC: A Comedy in T̶w̶o̶ -- NO -- THREE Acts (a.k.a. 'The Longest Day')

TL;DR: Just look at the Gist. Summary: Act I, in which I try and fail. Act II, in which I think I succeed but actually failed without knowing it till I tried to use it. Act III, in which I return to my beginning, ponder the universe, dive deep into the depths of the abyss, and come back with the magic bean that makes everything work.

Read more

Estimate Time for Job Completion (With Progress Updates) When Tar'ing Huge Directories

For the sake of future me, I am recording this here, the coolest shell trick I’ve learned this year: (Linux): tar cf - /folder-with-big-files -P | pv -s $(du -sb /folder-with-big-files | awk '') | gzip > big-files.tar.gz (OSX): tar cf - /folder-with-big-files -P | pv -s $(($(du -sk /folder-with-big-files | awk '') * 1024)) | gzip > big-files.tar.gz with output looking like: 4.69GB 0:04:50 [16.3MB/s] [==========================> ] 78% ETA 0:01:21 Requires ‘pv’: https://github.

Read more

Setting up the Text Editor in My Computing Ecosystem

Image from WikiMedia Commons Basic Setup of Shell to Support My Text Editor Preferences By “text editor”, I mean Vim, of course. There are pseudo-operating systems that include rudimentary text-editing capabilities (e.g. Emacs), and integrated development environments that allow for editing of text, but there really is only one text editor that deserves the title of “text editor“: Vim, that magical mind-reading mustang that carries out textual mogrifications with surgical precision and zen-like elegance.

Read more

'xargs' - Handling Filenames With Spaces or Other Special Characters

xargs is a great little utility to perform batch operations on a large set of files. Typically, the results of a find operation are piped to the xargs command: find . -iname "*.pdf" | xargs -I mv ~/collections/pdf/ The -I tells xargs to substitute “ in the statement to be executed with the entries being piped through. If these entries have spaces or other special characters, though, things will go awry.

Read more

Useful diff Aliases

Add the following aliases to your ‘~/.bashrc’ for some diff goodness: alias diff-side-by-side='diff --side-by-side -W"`tput cols`"' alias diff-side-by-side-changes='diff --side-by-side --suppress-common-lines -W"`tput cols`"' You can, of course, use shorter alias names in good old UNIX tradition, e.g. ‘ssdiff’ and ‘sscdiff’. You might be wondering why (a) I did not do so, and (b) what is the point, conversely, of having aliases that are almost as long as the commands that they are aliasing.

Read more

Supplementary Command-History Logging in Bash: Tracking Working Directory, Dates, Times, etc.

Introduction Here is a way to create a secondary shell history log (i.e., one that supplements the primary “~/.bash_history”) that tracks a range of other information, such as the working directory, hostname, time and date etc. Using the “HISTTIMEFORMAT” variable, it is in fact possible to store the time and date with the primary history, but the storing of the other information is not as readibly do-able. Here, I present an approach based on this excellent post on StackOverflow.

Read more