16 August, 2015

It's Been Interesting Times This Morning With Some Severe Thunderstorms

15-Aug-2015:

(dated here at the top in case I don't finish this by midnight...all the "this morning" or similar references refer to the 15th.)

The supposedly Chinese saying goes, "may you live in interesting times." This morning I would say qualifies, although not in the curse sense.  It was just a weird slice of life with some drama for me.  As I and Garrison Keillor have said many times, it could be worse, and I got away comparatively really well...this time.

What an early morning.  I had slept for a couple of hours, when for unknown reasons I awoke to heavy rain and a thunderstorm at about 1:45 EDT or so.  A few minutes after listening to this, there was a REALLY CLOSE lightning strike, no discernible delay between the really bright flash and a sound which seemed like it would be loud enough to at least crack if not shatter a window (but all mine seem OK).  Thinking only more of this is going to continue, making it next to impossible to get back to sleep with all this thunder going on, and the fact that I like watching lightning, I decided to stay up.  It was quite a light show.

As I was wandering over near my kitchen sink, I heard the sound of water pouring at an incredible rate.  I knew what this was; I have experienced times before when water was coming into the drain tiling so fast that it made that definite sound of water pouring into other water, such as pouring water out of a pail into a swimming pool.  However, this was different.  It seemed more intense, maybe much more intense, than I had previously experienced.  That underscored just how hard it was raining outside, and that it had been doing that for long enough to soak through 2 to 3 meters of earth.

It occurred to me this could be quite a problem.  One day a few years ago, I was standing in my basement at my workbench during a thunderstorm when we also had a close lightning strike.  This was a bit further away back then because there was a perceptible, although short, delay between flash and thunder boom.  But close to the time of the flash, I also heard the sound of clicking, which I quickly deduced was the sound of (more than one) circuit breaker tripping.  I was also quite concerned that day because I smelled that same smell when you've built up quite a bit of friction between blade and wood when sawing, that smell of wood slightly roasting.  The thought was, "something got hot enough to do that to one of my floor joists?  ouch!"  I never did find any evidence of electrical fault or roasted wood.

So that previous experience got me to thinking, what if the same thing happened in this really close lightning strike, but this time to the breaker protecting/controlling my sump pump?  I went into my basement post-haste to look.  I could hear the sump pump motor humming away, so no, at least that breaker was OK.  Nonetheless, I went to the load center to look at all the breaker levers.  The RCD (GFCI) breaker for the garage was tripped, so I reset that.  All the others looked to be in the ON position, so I was fortunate there.

I thought I had fixed the concrete block mortar so that water would not leak in.  I was proven wrong this morning.  The Susan Lane Basement Creek was flowing again, as usual from south wall to sump at the north end.  When I had first noticed it this morning, it was not even to my workshop wall yet, but almost.  Judging how water was falling out of the sky to beat the band, I figured it had to make it to the sump, eventually.  Ever since the first incident after moving in, I have been careful not to leave anything on the basement floor which did not have the tolerance for a couple of millimeters of water...so plastic bases on computers, glides on the legs of wooden tables, and so on.  The poor workshop wall still suffers, but oh, well; this sort of thing hasn't happened in a long, long time so it's not really worth trying to mitigate.

But that seemed the least of worries at the moment.  My attention turned to the sump, and the heretofore unseen volume per unit time entering through the drain tile outlets.  It had to be on the order of a liter per second.  After a pumping out cycle, it had to be somewhere around only 10 or at most 15 seconds to fill it to the pump trip point again.  And during the pumping cycle, it really seemed like the pumping rate was not all that much faster than the filling rate.  I began to think flooding was a real possibility, so I started "pulling for" the pump.  Then it occurred to me, I would be in some exceptionally deep doo-doo if, due to the thunderstorm, we lost mains power.  That would mean the sump would overflow, and who knows if the floor drain about 4 or 5 meters to the east could handle the requisite rate?  I also started contemplating what, if anything, was subject to water damage between those two points (turns out, not a lot).  I had to have been nervously standing there for at least 20 minutes.  I sort of assured myself after that much observation that it was rather unlikely the flow rate would increase significantly enough to overwhelm the pump.  That was somewhat diminished by the thought that even if the rain stopped immediately, there were still many, many liters remaining in the soil which would make their way into my sump.

As I was ascending the staircase out of the basement, as I have a number of electroluminescent night lights, making much of my house glow a dull green, I noticed the power was interrupted for a few hundred milliseconds.  As I passed through the kitchen, I noticed the microwave had held time, but the oven did not.  That's weird, it's either so brief they both hold or long enough they both lose it.  It's very rare indeed that one holds but the other loses.  I knew that since at least one lost it, it was likely at least one, probably several, of my computers had just been forcibly rebooted. I only have so much capacity in my UPS, so I have decided only the  barest of essentials...which includes the router, the managed switch, the video monitor, and little else... will be connected to it, not everything.

Sure enough, as I got to the systems room, the screen was showing the RAM test for sal9000 just ending, and beginning to execute the Adaptec SCSI host adapter POST.  I watched it to see that it wouldn't hang on, say, a USB HDD (I have still yet to figure out why a USB drive left plugged in, even one with GrUB installed onto it, freezes POST).  So at least this one is alive and well.  Next my attention turned to my MythTV, because the boot was hanging at the point where sal9000 was supposed to NFS mount some shares from it.  Ruh-roh.

That was a whole lot less hopeful and more worrisome.  That is the one system I access as a general purpose computer the least, so I have it set up not to have screen blanking.  One of its purposes is to be able to switch to it to warm up the CRT (this one takes on the order of 15 seconds to show a picture).  So it was quite disconcerting when the monitor would not lock onto a picture, and its LED went amber indicating power saving mode.  Tap, tap on some keys to try to wake it up; there was no response.  OK, fair enough, I'll just activate an ACPI shutdown by pressing the power button.   Ummm...ummmm....ummmm.....that does nothing.  OK, so it's very much not as preferred, but I'll hold in the power button to make the ATX PSU shut down hard.  Ummmmm....how long have I been holding it?   1...2...3...4...5...6...7...uhhhhh, shouldn't it have hard shut down by now?  Isn't that supposed to be 5 seconds?  At this point, I'm thinking my poor MythTV has had it, I'll probably never see anything through it again, my (yearly) SchedulesDirect subscription renewal in June will have gone mostly for naught.  Hmmm....what to do...I went around to the back, yanked the power cord from the PSU, waited for the Ethernet link lights to go dark, and reinserted it.  I went back around the desk to the front, and with much hope pressed the power button.  Yay!  The power LED came on.  The CRT warmed up, and I saw RAM being tested/counted.  512 MB OK; this is good.  It booted...until the part with the filesystem check (fsck).  Well, I knew this was going to take the better part of 10 minutes or so; this is "normal" PATA, on a 1GHz Pentium /// machine, 400 or so gigabytes-worth.  So I turned my attention to the rest of the network.

Rootin, being on the UPS, seemed just fine.  In fact, it looked to be getting out to both the IPv4 and IPv6 Internet just fine.  By that time, the ONT had plenty of time to reboot, or had not been taken down at all.

The next thing to cross my mind was the rain gauge in my back yard.  Had we had so much rain it had overflowed?  It was roughly 0330 hrs by now. I put on some boots and a raincoat and went out to look.  Even in the dim light of night I could see about a third of my yard and half of southern neighbor Megan's yard had visibly ponding water.  The rain gauge capacity is 110 mm, give or take.  It was to 96.2, of course using the scientific method of estimating the last digit between the markings.

Not too long after that, I saw what looked like a truck from our volunteer fire department (not an engine, something a lot more regular and rectangular) go by on Huth Road, the street which connects to my dead end street.  I thought that was kind of odd, and I wonder what they were doing. The best I could guess was they were looking around the neighborhood for a fire started by the lightning strike.  A few minutes later, I watched them slowly go down my street too; southbound.  I went out to my street to try to get a better look at what they were doing; they were maybe 175 m away by that time (my street I believe is about 400 m long).  It's then i got the idea that I wanted to see my sump discharge, as I knew it would be likely doing so about every 10 to 15 seconds.

I noticed then that the grate cap was nowhere in sight.  I walked down the street a little ways, figuring it floated away (it's made of plastic).  Darn.  I should probably go to Home Depot later today for a replacement, I thought.  Aw, heck, it's got to be somewhere on the street or close to it, right?  So I went back into my house and got a flashlight.  As I returned to the outside, I noticed Fire Rescue 7 had turned around, and now was going, much faster this time, north.  Whatever they were  looking for, either they found it, or decided it wasn't on Susan Lane (or Hemenway).

I kept looking.  I thought it was unlikely for the cap to disappear.  It wasn't small enough to fit down the storm drains.  It didn't lodge against the tires of the cars parked in the street.  Eventually, about 10 minutes later, I found it...maybe 120 m "downstream," and a meter and a half away from the side of the street (we don't have curbs).  I thought, score! don't have to go to Home Depot anymore!

When I got back to the house, after I put the cap back on, I thought, maybe the NWS would be interested in how much rain I measured.  I checked the latest chart of the last 72 hours of measurements, and they had recorded roughly 60mm since midnight.  Even though they are on the order of only 5km away, there still could be significant differences in things like rainfall.  So I Tweeted to them (note: this requires JavaScript to be allowed from Twitter to view completely):

Apparently, they liked this very much:


I was displeased that they "imperialized" it instead of leaving it in SI units, but what the fsck, I realize they're publishing to a primarily US audience, and the US doesn't deal with SI very well at all.

Things seemed to be calming down. The downpour had reduced to just drizzle, my sump pump was undoubtedly keeping up with inflow (which was reduced), The Internet was accessible and all my systems seemed to come away from this unharmed...or were they?

Every once in a while, I'll go down my list of systems in the ConnectBot SSH/Telnet client app on my Nexus 7 and log into each one, just so the date/time last accessed will say less than a day.  Maybe I should have just put the tablet down and went to sleep.  But I hit the "sal9000 con" entry, one that establishes a "telnet" session to my Chase IOLAN serial terminal server on TCP port 5003.  (The port number I chose is a holdover from me working with Bay Networks' Annex 3 terminal servers, which port numbers are not configurable, they're 5000 + serial port number).  This in turn connects to serial port 3, which is connected to the ttyS0 port on sal9000.  And...it wouldn't even handshake.  I tried the rootin entry (port 2/5002).  Similarly, no dice.  Sigh...all was not well with the computing world.  So I went and traced the power cord, yanked the 120VAC from the IOLAN PSU, replug, saw the LAN lights going nuts as it TFTP downloaded updated firmware from rootin, and again a few seconds later with the updated configuration (the firmware for the most part will operate just fine, but i wanted the latest available used.  Similarly, it has nonvolatile storage for the config, but I figured what the heck, it's good to have a live backup on a server too.)  So eventually, sometime after 0400, yes, things as far as I could tell were right with the computing and networking world.

I stayed up longer, hitting refresh on the 72 hour history page, to get the 3:54 precip figure.  It wasn't posted until about 4:15.  It added some millimeters, but the total still didn't get up to my measurement by a fair margin.  I had a snack and some orange drink, and finally settled down to "Coast to Coast AM" with David Schrader on sleep timer.

Direct all comments to Google+, preferably under the post about this blog entry.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.

"Jeopardy!" replies and randomcaps really suck!

Please join one of the fastest growing social networks, Google+!
posted by Joe Philipps at 00:30