Merge Multiple Files With Header Rows

AWK / BASH one-liner to merge multiple files skipping the first row of each file except the first file, skipping blank lines (useful for concatenating table/data files where the first row is a header): awk 'NF==0 || (FNR==1 && NR!=1){next;}{print}' file1.txt file2.txt .

Read more

'Found existing installation', 'Not uninstalling', and 'Can[not] uninstall' Error Messages When Using 'Pip' to Install from a Local Filesystem

When running “pip” to install a package from a local file system, do you get messages like “Found existing installation package_name”, “Not uninstalling package_name”, “Can’t uninstall ‘package_name’“, “No files were found to uninstall”? This could be because you are trying to install the package from within the package root directory, and “pip” is confusing the (ephemeral/coincidental) package availability from the current working directory with the package being installed and available globally.

Read more

Bash Script Automatic Exit on Command Failure

Stick this at the top of your multi-command BASH job scripts: set -e -o pipefail Exits on failure (== non-0 return code from any commands in the script) instead of cheerfully executing all subsequent lines in what is bound to result in a cascade of errors (or worse). More information in the excellent post here.

Read more

Configuring that Vim Build

Because I always forget and need to look it up all the time, ./configure --with-features=huge \ --enable-multibyte \ --enable-rubyinterp=yes \ --enable-pythoninterp=yes \ --enable-python3interp=yes \ --enable-perlinterp=yes \ --enable-luainterp=yes \ --enable-gui=gtk2 \ --enable-cscope \ --prefix=$HOME/Environment/local \

Read more

You Are Only as Good as Your Next Resurrection (Part 3)

(NOTE: This is the third of a three part series on setting up a cloud-based backup system, and describes how to set up the backup system to run on schedule automatically. The first describes the rationale behind the various technologies selected for the system, while the second describes how to setup and use the backup system) From a Backup to a Backup System Ok, with all the above we just have the building blocks of a backup system.

Read more

You Are Only as Good as Your Next Resurrection (Part 2)

(NOTE: This is the second of a three part series on setting up a cloud-based backup system, and describes how to use the various technologies discussed previously. The first describes criteria and reasoning for the various technologies selected for use, while the third describes setting up the backups to run automatically on schedule.) Setting Up “restic” “restic” is the the backup technology we will use for its performance, deduplication, and native support for writing to various different remote backends.

Read more

You Are Only as Good as Your Next Resurrection (Part 1)

(NOTE: This is the first of a three part series on setting up a cloud-based backup system, and describes the rationale behind the various technologies selected for the system. The second describes how to setup and use the backup system, while the third describes setting up the backups to run automatically on schedule.) Introduction This is a series of three posts that describe how to set up a modern, performant, efficient, and economical cloud-based backup system on your machine.

Read more

Cortical Stacks, Backups, and the (Lack of) Perisistence of Memory Between Backup Cycles

In the world of Richard Morgan’s Takeshi Kovacs series of books, individuals record their psyches to “cortical stacks” integrated into their bodies, which allow for them to be resurrected into alternate bodies should their current bodies fail or be destroyed. This can only be done if the cortical stacks themselves survive the body’s destruction. The very rich and privileged backup their cortical stack information to remote digital storage farms so that they can be restored even if their cortical stacks get destroyed.

Read more

Scheduling Jobs with Systemd

In a previous post, I discussed some of the different options available to you on most POSIX system to automate the running of jobs on schedule. Here is a brief recipe on how to actually use the modern/trendy and actually functionally beneficial “systemd” to do this as a normal (non-root). In addition to the job file or script itself, you will need to create two files: a timer file and a service file (also referred to as timer unit and service unit).

Read more

Scheduling Jobs: Cron vs. Anacron vs. Systemd

Today, you generally have three choices for a time-based system scheduler: “Cron”, “anacron”, and “systemd”. There is a lot of information out on the web about these, so here I shall just summarize the basics from a practical standpoint to answer the question “which scheduler/timing system should I use?” “Cron” is the traditional timer. It is available everywhere, and tried, tested, loved, and it just works most of the time for most people.

Read more