15 June, 2012

A Tale of Two Stupidities

UPDATE from 21-Jun-2012 below.

Last week (on Wed., 06-Jun) I went to a regional retailer, Ollie's Bargain Outlet, for some miscellaneous items--socks, black landscaping cloth, maybe some grass seed.  This place has many "regular" items, but they sort of specialize (mostly) in buying out overstock of other retailers, and being an outlet for manufacturers' reconditioned/refurbished items.  One of the latter items is a Pandigital Nova, which they call a "Media Tablet" (and I call an Android tablet).  It's only been a week, and already it has urged me to spend insane amounts of time researching about, and hacking on, this thing.  In one instance I was up until about 0430 hours local time (Eastern Time).  But on Thursday this week I executed two stupidities.

If you recall biff(1), it "makes noise" when new mail arrives.  One myth (and it is an untrue myth) is that one Berkeley student's dog, named Biff, barked at the postal carrier.  I wrote my own to watch $MAIL because the standard biff behavior is to "bark" only once.  If I'm not around to hear it, it could be many hours before I even know I have new mail to read, let alone actually read it.  So my version checks $MAIL at (approximately) the top of every minute from 0800 to 2200 hours, and it barks every minute until the file's "last read" time is later than its "last write" time.  (There are a number of noises which could be similar to my original biff occuring at the top of some minutes, such as the top and bottom of hour markers broadcast on WBEN-AM, so it actually targets one second after.)  So this is one of the first things I usually hear in the morning, because there is usually at least one cron job which emails its results to me which ran overnight.

Previous to Thursday morning, I had set up email fetching on the tablet, and had seen it work.  So imagine my surprise when I get the tablet's swirl animation which putting "Syncing" on the screen, and up pops on the screen that the OS had to force close the app.  I tried it a few times to find out if this was some quirk, and it was insanity (doing something over and over and expecting a different result).  OK, so putting on my computer engineer's thinking cap for a second, the usual first thing to consider when something was previously working, and now is not working, is what changed.  The major change was installing the excellent (although Chinese and therefore difficult to understand at times) GO Launcher EX (available on Play, Amazon Appstore for Android, and where I got it, the Opera Mobile Store).  So recalling there are a lot of rather nasty folks overseas, I kind of felt I was duped into compromising my tablet, so I uninstalled GO Launcher EX.  That was no help.  I rebooted (powercycled) the tablet.  That wasn't any help either.  This was the first stupidity, assuming the newfangled (to me) tablet had to be at fault just because of that being the current hacking target, and its newness to me.  I even went through a factory defaulting wipe of the system, and a re-setup.  This of course, as we shall see, was no help at all either.

By now biff had barked at me about 20 or 25 times, and I was getting miffed that I could not pick up my email on the tablet, and I'd have to revert to using the SSH client on my Palm T|X.  So I pulled that out, logged in, and pulled up Mutt (my favorite ncurses-based MUA).

It turns out, as usual, the only thing waiting was the report of the spam messages cleaned up daily (retains 7 days for inspection and possible "rescue.").  But then after dismissing that, I started looking at the messages and maillog logs.  Much to my amazement, the  spawned IMAP daemons were not only exiting with SIGSEGV, but before they did exit, they were consuming all idle CPU cycles according to top(1).  And that, as we shall find out, was one of the keys to unraveling the mystery.

I thought, "that's kind of odd."  Although I didn't change anything relevant on the IMAP server side of things, and the uptime was only 7 days, I thought that every once in a blue moon things do go screwey on Linux, and so despite not wanting to lose my place on several open browser pages, I shut down and rebooted my workstation.  And that of course was no help at all.  This was the second of the two stupidities.

By this time I was a little worried that I had a fancy paperweight which also happens to be capable of playing Angry Birds intead of having an Android tablet.  One of the main reasons I got this thing is not to have to deal with basic email and such on a tiny T|X screen.

The key to solving the issue was to start dropping assumptions, and this is really plan B (and sometimes plan C) when the usual suspects in computer and general problem solving aren't getting oneself anywhere.  And admittedly, all too frequently plan B is rebooting.  That should really be plan C at soonest, and more appropriately plan D or E.

The fatal misunderstanding was that Dovecot automatically stores and subsequently retrieves a user's email "folders" rooted at the user's $HOME/Mail/.  Wrong!  That's not necessarily the case.  I had always assumed that since I had set up so many other IMAP clients before (Mutt, Thunderbird, maybe a few others) and leaving the optional server side parts at the MUAs' default that it was Dovecot doing that.  However, I think it's probably closer to the truth that at one time an MUA I used, probably ElM, set up ~/Mail and it had somehow been found ever since.  And I still don't know what the heck happend, why all the sudden the Gingerbread default "Email" app didn't work like that.

But looking at this in hindsight, I will show you my best guess at what was throwing a monkeywrench into the entire works:

20:30:04 rchandra@sal9000:~ 0> du -sk
11457296            .
20:30:14 rchandra@sal9000:~ 0> du -sh
11G          .
20:30:19 rchandra@sal9000:~ 0>

My best guess is that both the IMAP daemon and the poor, limited RAM tablet were trying to deal with 11GB of "mail" in my home directory.  They were both likely choking rather mightily on the multi-gigabyte virtual machine hard disk image which is in the QEmu subdirectory, among other reasons.

This was an extraordinarily easy fix.  I went into the tablet's incoming server settings for sal9000, specified "Mail" as "IMAP prefix path," and the IMAP client worked fine.  So take a slightly painful lesson from me, and work with fewer assumptions instead of more when computer problem solving.

UPDATE from 21-Jun-2012:  I really didn't want to do without my folders for filing away my email, and none were showing up by specifying the prefix.  So I decided I would snoop in on an IMAP session to see why imapd was chewing CPU and K-9 Mail was crashing.  Actually, I found out it's decidedly worse.

At one time, I was experimenting with delivery to maildirs.  As those in the know know, there are three subdirectories in that format: new, cur, and tmp.  I didn't want a whole new tree, so I decided to make some symlinks, all to "."  So...imapd told K-9 (in essence) "there appears to be a folder here, not with email, but with a bunch of subfolders, named 'new.'"  K-9 faithfully said, "OK, whatcha got?"  Imapd faithfully replied, "well, seems inside 'new' there are more folders, at least one of them which is named 'new.'"  And K-9 asked, "well, what's inside new/new then?" wherein imapd replied, "seems there's a new/new/new there."  And so on went the conversation, basically ending when K-9 filled up my poor tablet's RAM, and imapd probably filled up nearly 2.5GB of RAM and 1GB of swap, and probably also eventually died...but not before really messing with my computer.  Yesterday, it locked up hard, requiring not the three finger salute, but a holding of the power button.  As best as I can figure, this unusual taxing of the system was its demise.

Direct all comments to Google+, preferably under the post about this blog entry.

English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.

"Jeopardy!" replies and randomcaps really suck!

Please join one of the fastest growing social networks, Google+!