Goodbye Walled Cult Garden, Hello (Again) Beloved Muddy Farmyard

End of an Era

The great experiment is over:

After almost a decade of OSX, I have now returned to Linux:

My foray into the world of OSX/macOS was good for most of its run, and, at its height, it was simply brilliant. When I first adopted OSX (after half a decade of running Linux), it was the ultimate developer experience. Not just the perfect balance between developer power/functionality and user bell-and-whistles, but actually as good as the best at both. Over the years, Apple has eroded the former (and arguably, dumbed down the latter). Where they used to offer solid, robust, total “good citizen” POSIX environment, where all the libraries were where you (and everyone else) would expect them to be and everyone got along, one by one things were moved around, removed, or changed to the point of total non-compatibility. The hoops and rabbit holes you needed to get through to build not just utilities, but fundamental build tools, became more and more ridiculous.

SciPy, matplotlib, etc. used to be challenging, but, ironically, as these have become easier now, it’s things like gcc or even basic Python (3.7 and the ssl libs come to mind) that have become impossible. Or, if possible, not something I want to fight and tinker and spend days down in the weeds trying to get on my system.

“Just use conda or homebrew, etc.” people say. Nope. I like these pre-built environments, and they definitely have a place. But as a developer I not just want, but should want, to have full control over my development environment, and the ability to build this from scratch. But it is ridiculous, I think, that we have to rely on the (wonderful) efforts of a third-party community to break holes in the increasingly thick walls Apple is building around its domain. Over successive generations of OS’s, this has become more and more difficult, with many frustrating hours. Recently, it has broken my tolerance threshold of the amount of time I want to commit to this task.

And then come all the other ridiculosities: e.g., the root user issue, the “/usr/local” issue etc. etc.

The final straw for me was the hoops I had to go through with respect to requirements for building Python 3.7. I did it allright, and, compared to what I had to go through for some of the other things I’ve had to do, it was not so much. But coming on top of all the other things, I’ve had to do, it was too much.

So, backed up my data, stuck in a USB loaded with Ubuntu, and jumped off the cliff …

I did NOT land in Arcadia.

What I Started Off With

A late 2013 15” MacbookPro with 16 GB memory and a 1TB SSD.

What I Was Going to Do

Wipe the drive and install Ubuntu 2018.04 LTS Bionic Beaver whole kit-and-caboodle, then stick on a virtualization system (either VMware or VirtualBox) and install Windows 10 to run Photoshop and Lightroom.

What I Expected

With a 2018 release of Ubuntu on a 2013 Intel-chipped Macbook?

Pretty much a seamless transtion into Linuxland requiring little more than entering computer and user names, password, date/times, etc. And, once there, apart from the aesthetics, it would be akin to my early halcyon days on the Mac: the perfect development environment in a beautiful, elegant, everything-just-works home.

Now, while back in my first fling with Linux days, I had all the craziness of, e.g., having to modprobe my wifi drivers on every boot etc. and bunches of other things I had to do to get from one day to the next, I recall that most of these cleared up toward the end (especially as releases caught up with my increasingly aging hardware as far as drivers etc. went). Given that, and given the broad adoption of Linux desktops worldwide (up to almost 1.96% now!), I expected a much slicker bells-and-whistles experience, which would allow me to focus on work more than tinkering/getting-things-to-work.

I was wrong on all counts.

What I Got

Admittedly, the parts that worked worked great. And admittedly, this was the majority of the system. And, admittedly, I loved what I got, everything felt natural and just right, so all this was enough for me to commit to stick with returning to Linux going forward …

BUT all these things I loved was also much according to what I expected, so not really cause for exceptional song and joy. It would have been if it all worked great. But it did not. And while there were not many parts that did not work, some of those that did not were really, really, really, really majorly frustrating to downright horrible to deal with. And while the remainder was either “just” underwhelming or irritating, it was time-consuming and tedious to deal with when I just wanted to get on with work.

I put down a lot of the worst issues to hardware: my old Macbook was not struggling wth the task of running the latest Ubuntu desktop and there might have been hardware driver issues as well.

The Surprisingly Good

The things that not only worked so right, but better than I remembered, better than I expected, and/or better than their equivalents ever did under macOS:

  • Workspaces: love the ergonomics (super PgUp/PgDn)
  • Windows manipulation short cuts: love the ergonomics (super Left/Right/Up/Down, the ability to easily resize and move windows using the keyboard, etc.)
  • Many other thoughtful user experience details, e.g., the dock with the little red dots to show multiple instance, and so on.

The As-Expected Good

  • apt“: as good as it ever was, and that is very, very, very, very good.
  • The general development environment experience: shell, library headers, building stuff, etc. (NOTE: this right here, is the raison d’etre for my move back to Linux. This single line outweighs ALL the negatives, whining, complaints, and salt in the rest of this post).

The Bad

Many of these are idiosyncratic, I think, i.e., attributable to my old and probably not-so-compatible hardware salvaged from the “walled garden”. But others are not …

  • [Machine-specific] Sluggish. Noticeably and significantly so than native OSX. This is a 2013 machine, and I guess the “close-to-the-metal” OSX could squeeze out a lot more performance than the generic Ubuntu drivers, but still.
  • [Machine-specific] A lot more CPU/resource, and, hence, battery usage than OSX. As above for the disclaimer.
  • [Machine-specific] Meaningless battery percentage remaining/time to empty/time to full charge. I get figures like: 55% battery remaining, 106 hours till empty, etc.
  • [Machine-specific] Very obviously struggles to drive my 4K monitor (not just when, e.g., watching full screen YouTube, but also refreshes/redraws of full screen windows when minimizing/maximizing/changing workspaces.
  • No WIFI working out of the box. Need to manually set the repo source to the USB key, then search for and manually install two packages. Ugh. With everything so dependent on the internet, this could have been horrible if we did not have multiple devices scattered around to allow us to troubleshoot via the Great Hive Mind that is the internet.
  • Most of my other complaints in this category can be attributed to the GUI app experience. This does not bother me as much as it might some other folks, because, to me, the GUI is very much a secondary experience, especially given the amount of time I spend there, and, as far as work goes, a non-essential one. Nonetheless, secondary or not, I do use the GUI, and in doing so I did face some issues.

    • The GUI package manager.
      • Slow, sluggish, poor information when it works, and when it doesn’t, with frequent errors/problems, it has TERRIBLE to non-existant error messages (this latter is recurring theme; it is, by far, my biggest gripe with this experience; better error messages would have gone a LONG way to mitigating and making less frustrating ALL the issues I faced).
      • Searching for packages sometimes bring up multiple hits, without it being clear what the differences are (yes, I have multiple source repos selected; this should be indicated if this is the difference). This can result in mutiple installs of identical packages (e.g. I installed another “Eye of Gnome”, not realizing it was already installed).
      • Luckily, as far as I can see, there is absolutely no reason to use this when the excellent “apt” exists.
    • The GUI app launcher:
      • Sure, nice to call up an application. But good luck trying to find out where the application is (you cannot; you have to go to “~/.local/share/applications” or “/usr/share/applications” and find the “.desktop” file and open it) or running it as root.
      • As with the GUI package manager, terrible to non-existant (fails silently) error messages.
    • The default GUI app assignment:
      • Here is a case example of a little thing that just should not be happening. Double-clicking an image opens it up in the default image viewer, “Eye of Gnome” (which I accidentally installed twice). This was a large photograph, so it was very, very, very sluggish (that’s another strike agains the entire experience, but that’s a different story). OK, so search for another image viewer, and it seems that GPicView is a good, fast one. Install it, and trying it out on some samples shows that sure enough, it is. Right click on my photograph again to select to open it with Gpicview instead of Eye of Gnome, and guess what, Gpicview advertises itself as “image viewer” as do the two other installs of “eye of gnome”. So I have three “image viewer” applications to choose from, with no idea which is which.
    • The way GUI apps handle errors:
      • In general, I find that GUI apps fail much more often than I even encountered on native OSX/macOS, and when they do, they often do silently with no errors or a misleading ones. THIS TRULY SUCKS.
      • Many other GUI apps spew out a bunch of warnings regarding missing/invalid libraries or even errors into my terminal session, which is unpleasant and not even helpful (e.g. first thing I do is to make sure the declared libraries are installed, but they often are or the errors/warnings persist even after I install them; a lot of Googling reveals the issues are something else that is related, but not directly inferrable at first glance from the messages, e.g. it requires a 32-bit architecture library instead of the default 64-bit one).
      • If I had to pick just ONE of these “bads” to be fixed, it would be this one. It is a recurring theme not just in the “bads”, but in the “uglies” below, that things fail without error message, leaving me to do forensics in “/var/log/syslog”. This fine and all for a sys admin or developer, but it really should not be the case in an end-user desktop system.
    • The rendering of GUI apps:
      • Font-scaling is unevenly applied. My 4K monitor needs to have 200% font scaling to be used correctly (though 150% would be better, but fractional font sizes are not supported but that’s another story), and this what Ubuntu selects by default (though this caused catastrophic failures during install; see the first “ugly” below). Unfortunately, not all applications respect this and they open with unreadable microdot size interfaces (e.g., “dirdiff”)
    • Zeitgeist integration:
      • I had disabled Zeitgeist because (1) I did not care for it; (2) it seemed very invasive; and (3) I suspected (wrongly, it turns out), that it might have been responsible for my startup/hangup woes. Turns out that many applications do not work with this and fail if it is not enabled. But instead of letting me know this, instead they fail with that stupid, stupid, stupid silence. Which leads me to want to say this again and again and again: the GUI application error/failure messaging system is TERRIBLE. Developers: USEFUL ERROR MESSAGES ARE USEFUL!!!
    • snap

      • It’s buggy and pollutes the home directory, and many of the programs that installed by it are also buggy compared to their apt counterparts. E.g. the calculator is installed by snap by default, and it simply does not launch when assigned hot key from the system settings. Replacing it with its excellent-as-ever apt version:

        sudo snap remove gnome-calculator
        sudo apt install gnome-calculator

        brings everything back to copaceticity again.

The Ugly

  • Installation failed with cryptic (actually, totally misleading) error message … AFTER partitioning/formatting drive; multiple reboots/variations led nowhere. Yeah, scary. No going back, and no clear way forward. Finally found out it was a bug in the installer during file copy, and I had to patch the Live USB distro to fix it. This for a distribution downloaded on Jun 2 for a bug reported on Apr 4 (and fixed soon after). Luckily someone discovered a simpler solution: just set the display scaling to 100%. Yep, that was all it was — the GUI crashing during a critical phase of installation when the machine was in limbo because it did not know how to deal with rescaling some fonts. Sheeeeeesh.
  • After the initial installation bumps, things went OK for a while. I started the task of provisioning the system on a as-need-or-as-it-occurs-to-me basis. Again, things worked fine for a while. Then it did not. It began consistently hanging on shutdown, requiring a pillow-in-face-suffocation-hold-the-button-down-till-it-dies shutdown. And that was not all: it started hanging on boot if there was an external monitor plugged in. All this this was especially frustrating because there was no error message, no complaints, no indication of what went wrong. After trying a bunch of stuff, finally went digging through the logs.

    • In ‘/var/log/syslog’, I found a bunch of errors, including:

      Jun  3 21:25:19 pluto systemd-udevd[3276]: inotify_add_watch(9, /dev/sdd, 10) failed: No space left on device
      Jun  3 21:25:19 pluto dbus-daemon[856]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.94' (uid=1000 pid=1791 comm="/usr/bin/gnome-shell " label="unconfined")
      Jun  3 21:25:19 pluto systemd[1]: systemd-hostnamed.service: Failed to add inotify watch descriptor for control group /system.slice/systemd-hostnamed.service: No space left on device
      Jun  3 21:25:19 pluto systemd-udevd[3276]: inotify_add_watch(9, /dev/sdd1, 10) failed: No space left on device
      Jun  3 21:25:20 pluto systemd-udevd[3294]: inotify_add_watch(9, /dev/sdd2, 10) failed: No space left on device

      From what I can make out to be the result of too many file monitoring programs running (e.g., “tail), and, in addition, I had earlier in the day just installed CrashPlan befor the problems began, so I suspect that these might be related. As such, I uninstalled CrashPlan and increased my inotify limit to over a million (from the default 8000):

      $ echo fs.inotify.max_user_watches=1048576 | sudo tee -a /etc/sysctl.conf
      $ sudo sysctl -p

      This seems to have fixed the issue so far. Of course, this means that I am currently without a backup service. I might try reenabling CrashPlan to see if things work fine now that I have increased the inotify limit (if indeed, that was the issue), or might just adopt a different backup approach.

      UPDATE 2018-06-06: Primed by the above experience, I was able to sharpen my search terms and that got me to this article, which seems to imply that: (a) my suspicions as to the cause of the inotify node exhaustion errors are correct; and (b) the solution is also correct. This means that if I am also (hopefully) correct that the inotify node exhaustion errors were the reason for the freezes/hangs on reboot and shutdown, and the limit I set was high enough, I should be able to reinstall CrashPlan and get back to backing up …. ?

      UPDATE 2018-06-09: Yep, it was CrashPlan exhausting the the inotify resources. Problem has entirely disappeared now. While I should be OK to use CrashPlan again with the higher inotify limit, I have decided that I do not want such a resource-hungry program constantly running in the background of my already challenged machine. With a little bit of tinkering, a variety of other backup solutions exist that are just as good, much more economical (especially when dealing with multiple machines), and all with the added benefit of no vendor lock-in.

  • Other system freezes on occassion

    • After all the above initial woes, it seemed like things had stabilized for a while. But after some routine usage, I would periodically get an almost hard freeze: nothing worked, but I could still move the mouse cursor around. It would take a hard boot to restore things, after which things would work fine for a while before it happened again.
    • UPDATE 2018-06-09: Turns out that this problem was due to my secondary monitor resolution. Acting on a hunch, I swapped out the external 3960 x 2160 display for an 1980 x 1080 one. And the freezes just went away. That’s all it was. The association with Chrome was just an artifact: Chrome is pretty much the main full-screen GUI app I use most of the time, and it is the one with the most taxing redraws (everything else, e.g. Vim and the shell, span much smaller regions and, further, are text only). As such, it was the one that stressed the GPU the hardest, and caused the failures. Not sure whether it was driver failures or the GNOME/GTK libraries messing up here when stressed like this, but I am pretty sure that removing that stress has solved that problem. I am going to have to live with less real estate until I get a machine with a more powerful GPU or with a GPU that can be drive more efficiently with the Linux drivers or both.
    • (Previously …

      • After a while, I did note that it was often while I was using Google Chrome when these occurred. Sometimes it was doing something heavy (like a major redraw or opening a new tab), and at other times not so heavy (e.g., scrolling down a page). But it always seemed to be involved.
      • There were a couple of occassions when I was not actually using Chrome, but there was some major re-rendering of windows going on.
      • These pointed at two major suspects: (1) Google Chrome; or (2) the GPU (or, to be more precise, the OS handling or drivers of the GPU).
      • Finally getting back into “/var/log/syslog” showed a TON of “RENDER WARNING: texture bound to texture unit 0 is not renderable. It maybe non-power-of-2 and have incompatible texture filtering” errors from “org.gnome.Shell.desktop”. I mean a TON.
      • Googling for this indeed confirmed that Google Chrome was at least possibly the culprit, though neither of the fixes proposed seemed relevant.
      • For now, what I’ve done is remove the Google Chrome I installed from their site and installed a native fully-free build:

        $ sudo apt remove google-chrome-stable
        $ sudo apt install chromium-browser
      • Time will tell if this fixes the issue, but in either case I prefer Chromium to Chrome on this system for its performance.

      • If the problem persists, I am going to dig into installing (or reinstalling) NVIDIA drivers, either the latest ones or maybe rolling back to older ones if the former does not hold up)*

  • Lots of other maybe-errors/issues in “/var/log/syslog”

    • I did see many other errors in “/var/log/syslog”, and, in fact, continue to see large numbers of them. But not sure whether they are business-as-usual or something-to-be-worried-about.
    • And while I am not seeing these flagged as errors, I find my logs are filling up — as in literally hundreds of thousands of lines — with “‘ureadahead’ ignoring relative path” messages.

Final Thoughts

I love Linux.

I like Ubuntu.

I am sticking with this going forward into the future.

But it was no slick utopian metropolis to which I returned when finally leaving the walled prison garden of the cult of Apple after a decade. Instead, it was like going back to a muddy farmyard, where not only do I sometimes need to break out my tools and fix up a fence, a bit of leaking plumbing, or a patch of missing barn wall/floor/roof, but here and there I also find myself stepping into a bit of do-do. All this is part of the cost of doing business now and probably into the future with this wonderful technology (and all its woes have to do with the desktop GUI and/or hardware issues; the core technology is fundamentally solid, robust, excellent, and elegant).

And you know what?

I do not just love the farmyard, but I need it.

If my work consisted of just email, (word processor) documents, presentations, or spreadsheets, then sure, the walled garden/condo living would have served me well enough. But if I want to raise some cows or goats or pigs or donkeys or horses, then that walled garden/condo is simply not going to cut it.

For all its maintenance needs, you see, the Linux desktop still offers the best developer experience/environment that I have ever had, one that was once rivaled, but never surpassed by, and now, with this rapid crawl into the cult of the lowest common denominator of let-us-relieve-you-of-that-confusion-of-knowing-things-and-controlling-things-dear-touch/swipe-based-user, not even contended by, Apple.

I am finally back at home, leaky plumbing, scraggly fences, uneven floorboards, cracked walls, muddy grounds, and, yes, the occasional not-actually-mud-but-something-else patch of ground and all.