This should be a quick one, possibly with an update after the initial post.
This past Monday evening, 16-Dec-2019, I went to T-S-K Electronics in North Tonawanda, NY, and took the tests for FCC Technician Class, then General Class, Amateur Radio license. Unfortunately, the Volunteer Examiners (VEs) are not permitted to show you which questions were answered wrong, but they will tell you how many you missed (2 for Tech, 5 for General in my case). When talking about license grants, my Technician course instructor (Kevin Kedzierski, WA2FKV) said the VEs will send out the documents that night, and should start to be processed by the Postal Service on Tuesday. Kevin's prediction was that I would have my call sign by Friday (20-Dec). Well...not to be; although I have proven my authorization to operate, it's not official until a call sign shows up on the FCC's licensing site. (That's actually one of the questions in the Technician Class question pool, how soon after passing your exams are you allowed to get on the air, and the (correct) answer is, when your license grant appears in the ULS database.)
Before someone takes the test(s), they may create an FCC Registration Number (FRN), in lieu of using one's Social Security Number (which I have done). It is that account which you may log into to download a PDF of your license for printing. I've been logging in virtually all week just to see if perchance my application has been processed. Although...when you pass, you get a document called a Certificate of Successful Completion of Examination (CSCE). One of the blanks you can fill in on that form is an email address, which I think the FCC will use once they have made your license grant. Every time I hear that I have new email, I keep hoping that it's the FCC, but at the moment I'm a little skeptical, because one, it's hand-printed on the form (and people screw up my surname, Philipps, all the time, too many Ls and not enough Ps, which is part of the address), and two, because I suspect they wouldn't do anything on weekends (it's Saturday as I write this).
So...I wait for things to happen, over which I have little control obviously. But still, I'm thinking almost like The Chipmonks, with their "hurry, Christmas, hurry past."
UPDATE: 27-Dec-2019, the FCC processed my license application. Hello, world, from KD2TFB.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
Hey, you EU folks: there's some cookie laws which stipulate Alphabet/Google (owners of Blogger and Blogspot) needs to disclose some stuff to you, which, unfortunately might not show up if you have JavaScript disabled. It's not my fault they're that stupid, that they'd require JS to be on to display that notice, but whatever. You have been forewarned there may be some cookie monkey business going on which you might not like, and Google informs me it's my responsibility to tell you about that.
23 December, 2019
17 December, 2019
Knowing the Details Is a Lot Better Than Not
I apologize in advance for a bit of rambling, but there's just no good way I know of to write everything I want.
It's one of those unfortunate things that's money-driven (of the many, I know, I know...). In order to display ODB II, you have to have, you guessed it, a display. For almost all passenger vehicles, the only display available is plugging a scan tool into the ODB II port. Boy, I with that wasn't so.
Let's flash back for a moment to when I owned a Plymouth Acclaim. This thing was a beauty with respect to ODB II. Turn the key to the run position three times within 5 seconds of being off, and the malfunction indicator lamp (MIL), a.k.a. "'check engine' light," would start flashing 2 digit codes, and end with the pseudocode 55. Of course, unless you have a ready reference (thank goodness these days for DuckDuckGo, Bing, Google, Yahoo!, etc., but not back then, as well as for cellular data) you'd still be in the dark as to what the two digit codes mean. But at least you can memorize a few of either the more dire or the more benign ones.
Today I had a little bit of a scare. At first I noticed a light was on on my dash which ordinarily isn't on. I don't know why, but for no particular reason, my mind first interpreted it as a brake warning light. But then on closer inspection a few seconds later, I realized it was the MIL. The first thought after that was that the motor could refuse to run at any time, and I'd be stuck. I happened to be on my way to my Amateur Radio FCC license exams (hey! Got two wrong on Technician and five on General, so I am effectively a General and just awaiting my call sign) so was a tad worried about how I would get home (although to a large extent, it's less of a worry because I'm an AAA Plus member). Would it even start after my tests? Would it conk out on the way home?
Turns out, no, it didn't, although my trip home was longer because I decided to avoid a highway (I-290 east for locals to Amherst, NY). Ironically, before going to my exams, I talked, in person, to the manager of the aforementioned repair shop about coming in on the 19th for other work (brakes, battery, and oil change). We had also talked about the possible causes of poor fuel economy, oh the irony (stay tuned :-)). I was thinking, should I just drive the car to the shop and walk home? Geez, Louise, who knows what's wrong? I may not even make it that far. But then after I got home and settled a while, I remembered...a long, long time ago, I can still remember how I--made a trip to Harbor Freight. And I knew if I had a chance, to use that ODB II reader tool I bought....OK, OK, enough with the unrhymed "American Pie" parody :-). I had to think for a second, where I might have put it, lo those many years ago.
Eventually the memory cells kicked in long and hard enough so that I could retrieve said ODB II scan tool, and I did hie to the waiting Elantra with it. Drat. I bought the tool when I had a Lumina sedan; I didn't know eactly where Hyundai chose to hide their connector. But eventually I found it. But alas...it'd also been so long ago that I did not remember how to use the darned thing! Back into the house I did go, once again to consult first the product documentation, and then Google University. Hint: it doesn't quite work right unless the key is in the run position. (Hey! Shout out to you Cen-Tech manual writers! How's about bolding some "quick start on using our product" instructions on that one? I gave up after trying to read your "how to use this tool" section). One nasty neck kink later trying to look up under my dash for the ODB II connector, and I was in like Flynn.
Hmmmm....the display shows "A/T". Oh, no...my transmission is toast? Well, no...that's just a discovered "query-able" endpoint as I might call it (not sure what the formal term would be; I tend to think in more general IT/client/serer terms as that is my profession). Thankfully there was nothing to read under "A/T". "Engine" on the other hand...that was a P0455. OK I'll spare you the extra Googling and just tell you it's a large evaporative system leak.
So no, it was highly unlikely the engine would just quit. It would cause a decrease in fuel economy, because those fuel vapors are no longer being collected in the canister for that purpose, they'd just go, unburned, wasted, into the air. If I did give into my fears, I would have walked home in the mid December cold for nothing.
Alas, the reason information is withheld is no doubt economic, possibly in two ways. First, it's more expensive to incorporate a display into a car, at least one from the 2008 era when this car was made. Not only that, but then there would need to be extra time for programmers to make that display useful, much as Plymouth had to have done. And people's time costs money. Second, you think some segments of the car repair industry wants such empowering information? I hate to be cynnical, but Idunno, there could be big repair sort of like there is big oil and big pharma.
So...armed with the knowledge of what the implications of a P0455 are, I can more confidently go on and do some errands, and not...welll....cower in fear? Maybe that's a little harsh. But nonetheless, once again, knowledge is power.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
It's one of those unfortunate things that's money-driven (of the many, I know, I know...). In order to display ODB II, you have to have, you guessed it, a display. For almost all passenger vehicles, the only display available is plugging a scan tool into the ODB II port. Boy, I with that wasn't so.
Let's flash back for a moment to when I owned a Plymouth Acclaim. This thing was a beauty with respect to ODB II. Turn the key to the run position three times within 5 seconds of being off, and the malfunction indicator lamp (MIL), a.k.a. "'check engine' light," would start flashing 2 digit codes, and end with the pseudocode 55. Of course, unless you have a ready reference (thank goodness these days for DuckDuckGo, Bing, Google, Yahoo!, etc., but not back then, as well as for cellular data) you'd still be in the dark as to what the two digit codes mean. But at least you can memorize a few of either the more dire or the more benign ones.
Today I had a little bit of a scare. At first I noticed a light was on on my dash which ordinarily isn't on. I don't know why, but for no particular reason, my mind first interpreted it as a brake warning light. But then on closer inspection a few seconds later, I realized it was the MIL. The first thought after that was that the motor could refuse to run at any time, and I'd be stuck. I happened to be on my way to my Amateur Radio FCC license exams (hey! Got two wrong on Technician and five on General, so I am effectively a General and just awaiting my call sign) so was a tad worried about how I would get home (although to a large extent, it's less of a worry because I'm an AAA Plus member). Would it even start after my tests? Would it conk out on the way home?
Turns out, no, it didn't, although my trip home was longer because I decided to avoid a highway (I-290 east for locals to Amherst, NY). Ironically, before going to my exams, I talked, in person, to the manager of the aforementioned repair shop about coming in on the 19th for other work (brakes, battery, and oil change). We had also talked about the possible causes of poor fuel economy, oh the irony (stay tuned :-)). I was thinking, should I just drive the car to the shop and walk home? Geez, Louise, who knows what's wrong? I may not even make it that far. But then after I got home and settled a while, I remembered...a long, long time ago, I can still remember how I--made a trip to Harbor Freight. And I knew if I had a chance, to use that ODB II reader tool I bought....OK, OK, enough with the unrhymed "American Pie" parody :-). I had to think for a second, where I might have put it, lo those many years ago.
Eventually the memory cells kicked in long and hard enough so that I could retrieve said ODB II scan tool, and I did hie to the waiting Elantra with it. Drat. I bought the tool when I had a Lumina sedan; I didn't know eactly where Hyundai chose to hide their connector. But eventually I found it. But alas...it'd also been so long ago that I did not remember how to use the darned thing! Back into the house I did go, once again to consult first the product documentation, and then Google University. Hint: it doesn't quite work right unless the key is in the run position. (Hey! Shout out to you Cen-Tech manual writers! How's about bolding some "quick start on using our product" instructions on that one? I gave up after trying to read your "how to use this tool" section). One nasty neck kink later trying to look up under my dash for the ODB II connector, and I was in like Flynn.
Hmmmm....the display shows "A/T". Oh, no...my transmission is toast? Well, no...that's just a discovered "query-able" endpoint as I might call it (not sure what the formal term would be; I tend to think in more general IT/client/serer terms as that is my profession). Thankfully there was nothing to read under "A/T". "Engine" on the other hand...that was a P0455. OK I'll spare you the extra Googling and just tell you it's a large evaporative system leak.
So no, it was highly unlikely the engine would just quit. It would cause a decrease in fuel economy, because those fuel vapors are no longer being collected in the canister for that purpose, they'd just go, unburned, wasted, into the air. If I did give into my fears, I would have walked home in the mid December cold for nothing.
Alas, the reason information is withheld is no doubt economic, possibly in two ways. First, it's more expensive to incorporate a display into a car, at least one from the 2008 era when this car was made. Not only that, but then there would need to be extra time for programmers to make that display useful, much as Plymouth had to have done. And people's time costs money. Second, you think some segments of the car repair industry wants such empowering information? I hate to be cynnical, but Idunno, there could be big repair sort of like there is big oil and big pharma.
So...armed with the knowledge of what the implications of a P0455 are, I can more confidently go on and do some errands, and not...welll....cower in fear? Maybe that's a little harsh. But nonetheless, once again, knowledge is power.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
27 September, 2019
Amazon Is Getting Really Smart: Amazon Locker
About a year or so ago, while shopping on Amazon, I first saw a delivery option of "Amazon Locker," or for the rest of this post, "AL" for short. If you've ever heard of the term "porch pirate," you'll know that this no-additional-cost service was designed to combat just that. I don't think I have to worry about this personally because of the character of the neighborhood I live in, but I imagine there are quite a few people to which this happens all too frequently (thank you, oh so much, Mark Rober). (OK, OK...I'll admit, crime can happen anywhere, so no neighborhood is totally safe or free of it.) There's quite a few things that I have ordered that make me worry about someone coming along and just helping themselves to it, not to mention leaving cardboard boxes out in the rain (which is another problem AL solves).
With Amazon's acquisition of Whole Foods, that was a natural choice for one of my area's first ALs. It is really nice and fairly well secured (6 digit code required for retrieval). I have retrieved many an item from there over the past months, but going to Whole Foods is kind of out of my way. So imagine my surprise when I was wandering through Rite Aid on my way home from my employer's office looking for something to treat my nascent cold and I see an Amazon logo.
What's more, there was a label "Aldo" on it. Aldo. Why "Aldo" of all things? This is the really smart part: it looks like newer ALs have names. Amazon is thinking I'll do exactly what I did, seeing a new (to me) AL, go home and search for it on Amazon's site. When I did, I could correlate the name seen in person on the Locker with the one on the Web site. And that's a good thing too. I would have guessed Rite Aid's address to be on Sheridan Dr. Since it turns out it's actually on Evans St., I would have been looking for "Sheridan" in the listing of search results for ZIP code 14221. But there it is in the list: "Amazon Hub Locker - Aldo". Also in the listing is Karriem, Langu, Minerva, and Aliva. I don't know what the variety of names that they use is, but if each has a unique name, all the better. As well, consider that for my part of the world, these names are not common, but they look enough like names for them to be at least a little memorable. You're not wondering, was the name I saw John? Nancy? Bob? Sally? Greg? Diane? Mike? Darlene? No, at least for me it's esoteric, so possibly more memorable.
But alas, as you might imagine, there are some minor downsides. For one, there are limitations on the size of things that will ship there, which you will find at checkout time. Certain classes of goods (hazardous materials, shipped from other countries, etc.) cannot be shipped to one. Sometimes at the time of your order, it will be predicted that there will be no more locker space left. You're automatically limited by the hours of the retailer, but in Rite Aid's case, that's 0800 to 2100, quite a wide range indeed. You have 3 days to pick up, after which time your order goes back. At least at my Whole Foods, I've never gotten the barcode on an iPhone or 8" Android tablet to work. But all these niggles pale in comparison to the assurance that stuff will not be setting in my driveway, open to anyone who might see it and be inclined to help themselves, as well as generally being guaranteed the boxes will not be soggy. (well...you never know due to package leaks, flooding, overhead plumbing gone wrong, etc.)
I hope as many businesses as possible strike deals with Amazon for these alternate pickup arrangements (looks like they also have "Amazon Hub Counter," which as it sounds, you pick up from a person instead of just electronically unlocking a locker). Then it won't matter much where I'm going, there will be one close by. The businesses can benefit too. Like Rite Aid, put it far from the entrance, and you now have potential customers seeing your wares as they go to pick up their Amazon stuff. There's still those impulse buys that will happen, even if it's cheaper on Amazon :-).
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
With Amazon's acquisition of Whole Foods, that was a natural choice for one of my area's first ALs. It is really nice and fairly well secured (6 digit code required for retrieval). I have retrieved many an item from there over the past months, but going to Whole Foods is kind of out of my way. So imagine my surprise when I was wandering through Rite Aid on my way home from my employer's office looking for something to treat my nascent cold and I see an Amazon logo.
What's more, there was a label "Aldo" on it. Aldo. Why "Aldo" of all things? This is the really smart part: it looks like newer ALs have names. Amazon is thinking I'll do exactly what I did, seeing a new (to me) AL, go home and search for it on Amazon's site. When I did, I could correlate the name seen in person on the Locker with the one on the Web site. And that's a good thing too. I would have guessed Rite Aid's address to be on Sheridan Dr. Since it turns out it's actually on Evans St., I would have been looking for "Sheridan" in the listing of search results for ZIP code 14221. But there it is in the list: "Amazon Hub Locker - Aldo". Also in the listing is Karriem, Langu, Minerva, and Aliva. I don't know what the variety of names that they use is, but if each has a unique name, all the better. As well, consider that for my part of the world, these names are not common, but they look enough like names for them to be at least a little memorable. You're not wondering, was the name I saw John? Nancy? Bob? Sally? Greg? Diane? Mike? Darlene? No, at least for me it's esoteric, so possibly more memorable.
But alas, as you might imagine, there are some minor downsides. For one, there are limitations on the size of things that will ship there, which you will find at checkout time. Certain classes of goods (hazardous materials, shipped from other countries, etc.) cannot be shipped to one. Sometimes at the time of your order, it will be predicted that there will be no more locker space left. You're automatically limited by the hours of the retailer, but in Rite Aid's case, that's 0800 to 2100, quite a wide range indeed. You have 3 days to pick up, after which time your order goes back. At least at my Whole Foods, I've never gotten the barcode on an iPhone or 8" Android tablet to work. But all these niggles pale in comparison to the assurance that stuff will not be setting in my driveway, open to anyone who might see it and be inclined to help themselves, as well as generally being guaranteed the boxes will not be soggy. (well...you never know due to package leaks, flooding, overhead plumbing gone wrong, etc.)
I hope as many businesses as possible strike deals with Amazon for these alternate pickup arrangements (looks like they also have "Amazon Hub Counter," which as it sounds, you pick up from a person instead of just electronically unlocking a locker). Then it won't matter much where I'm going, there will be one close by. The businesses can benefit too. Like Rite Aid, put it far from the entrance, and you now have potential customers seeing your wares as they go to pick up their Amazon stuff. There's still those impulse buys that will happen, even if it's cheaper on Amazon :-).
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
20 December, 2018
Chrome: More From the "My Way or the Highway" Department
Google, in their infinite "wisdom," decided a few months ago that when you sign into Google, you also sign into Chrome. I in particular do not really want my Google credentials being cached in a whole lot of software, and in particular, not within Chrome (or Chromium for that matter). When Chrom(e|ium) exits, I don't want either to stay logged in.
A number of security- and privacy-focused folks pointed this out, the fact that they didn't take too kindly to logging into the Google services meant logging into their browser. So Google said, OK, folks, don't worry, if you really, really want to separate these two functions out, all you have to is go into your flags and disable identity consistency. That way, if you want to sign out of Gmail (or other services), you can, and still have your Chrome signed in and syncing settings, extensions, or whatever else you've configured to sync. Or conversely, you can be signed out of Chrome, but still be able to log in and pick up your Gmail.
Now you know, this is the company which by default has a "remember me" style checkbox on their credentials dialog, which defaults to "on." Similarly, if you've enabled RFC6238 TOTP multifactor authentication, there is a by-default-checked option so that you don't have to enter a TOTP for 30 days. After all, they're trying to make using Google and its services as convenient and frictionless as possible...why authenticate when you can be remembered? But of course, I can't tell you for sure, but the odds are pretty good the reason Google wants you to stay signed in is so that you can be tracked by them and other sites. After all, that's extremely valuable data; they've made an entire very successful business of collecting, curating, and somewhat interpreting that data.
But here's the thing...in a subsequent Chrome release, that flag has no effect. No, instead of being hidden away in some internal browser configuration page, it has "graduated" to the normal settings UI page, as "Allow Chrome sign-in". Great,then! Fixed, right? Well...no, not really.
The "identity consistency flag" allowed nearly complete separation of in browser and Chrome "logged in status". You didn't have to go into chrome://flags and toggle it on or off in order to log in or log out of Chrome; you could log into Gmail and not log into Chrome, or vice-versa. If you did adjust that flag, you'd have to restart Chrome for it to take effect. But no, this new toggle simply allows logging into Chrome, or disables the ability to log into Chrome and all its syncing goodness. This is at first subtle, but really is profound in the implications of its implementation. No longer can I just log into Chrome without logging into Gmail, if I log into one, I am logged into the other. Sure, if I don't want to be logged into Chrome, I can go back into the settings ( == friction) and pull the slide switch the other way. But then when I do want to log in, to get a sync going, I have to go into settings and slide the switch again.
And again...I understand the dual implication: they want me logged in/identified as much as humanly and inhumanly possible for their business, and basically their cover story is that they want it to be convenient and as frictionless as possible for the end user. But to the security and privacy minded, conscientious end user, it is less convenient and more friction.
So...it's Sundar's way or the highway. Sure...I suppose you could download the Chromium source, slice out these nasty bits, and build it yourself, but who wants that badly to take on that maintenance responsibility?
Idunno...I'm actually tempted to do this, because I'm sick and tired of all the goddamn stoopid animations...like you can't even open the main menu without a stoopid bloom of the 3 dots, and you can't visit a subsection without the page being slid all around, either horizontally or vertically. This is DESPITE many requests to remove UI animations, usually from folks accessing computers with Chrome on them remotely (and the slow update times that entails sometimes).
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
A number of security- and privacy-focused folks pointed this out, the fact that they didn't take too kindly to logging into the Google services meant logging into their browser. So Google said, OK, folks, don't worry, if you really, really want to separate these two functions out, all you have to is go into your flags and disable identity consistency. That way, if you want to sign out of Gmail (or other services), you can, and still have your Chrome signed in and syncing settings, extensions, or whatever else you've configured to sync. Or conversely, you can be signed out of Chrome, but still be able to log in and pick up your Gmail.
Now you know, this is the company which by default has a "remember me" style checkbox on their credentials dialog, which defaults to "on." Similarly, if you've enabled RFC6238 TOTP multifactor authentication, there is a by-default-checked option so that you don't have to enter a TOTP for 30 days. After all, they're trying to make using Google and its services as convenient and frictionless as possible...why authenticate when you can be remembered? But of course, I can't tell you for sure, but the odds are pretty good the reason Google wants you to stay signed in is so that you can be tracked by them and other sites. After all, that's extremely valuable data; they've made an entire very successful business of collecting, curating, and somewhat interpreting that data.
But here's the thing...in a subsequent Chrome release, that flag has no effect. No, instead of being hidden away in some internal browser configuration page, it has "graduated" to the normal settings UI page, as "Allow Chrome sign-in". Great,then! Fixed, right? Well...no, not really.
The "identity consistency flag" allowed nearly complete separation of in browser and Chrome "logged in status". You didn't have to go into chrome://flags and toggle it on or off in order to log in or log out of Chrome; you could log into Gmail and not log into Chrome, or vice-versa. If you did adjust that flag, you'd have to restart Chrome for it to take effect. But no, this new toggle simply allows logging into Chrome, or disables the ability to log into Chrome and all its syncing goodness. This is at first subtle, but really is profound in the implications of its implementation. No longer can I just log into Chrome without logging into Gmail, if I log into one, I am logged into the other. Sure, if I don't want to be logged into Chrome, I can go back into the settings ( == friction) and pull the slide switch the other way. But then when I do want to log in, to get a sync going, I have to go into settings and slide the switch again.
And again...I understand the dual implication: they want me logged in/identified as much as humanly and inhumanly possible for their business, and basically their cover story is that they want it to be convenient and as frictionless as possible for the end user. But to the security and privacy minded, conscientious end user, it is less convenient and more friction.
So...it's Sundar's way or the highway. Sure...I suppose you could download the Chromium source, slice out these nasty bits, and build it yourself, but who wants that badly to take on that maintenance responsibility?
Idunno...I'm actually tempted to do this, because I'm sick and tired of all the goddamn stoopid animations...like you can't even open the main menu without a stoopid bloom of the 3 dots, and you can't visit a subsection without the page being slid all around, either horizontally or vertically. This is DESPITE many requests to remove UI animations, usually from folks accessing computers with Chrome on them remotely (and the slow update times that entails sometimes).
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
19 December, 2018
Everything Old is New Again Department, Part 2 for Today
My, it certainly is true that there are no new novels, no new movies, no new TV programs. It's the same thing retold in different ways, which I guess is what keeps us watching and listening, trying to figure out what that "new" way is.
One of my favorite movies is "Sneakers." I have listened to the soundtrack CD many, many times. One of my favorites from that is "Cosmo… Old Friend," particularly in the one part with a strings crescendo I think in a major key, followed by a decrescendo in a minor key, followed by a piano (meaning soft) replaying of the major key, and this 3 (or 4) measure figure is repeated. All that is with a really great bass behind it (not sure what instrument; maybe a bunch of double basses arco). I was kind of wondering what scene in the movie was this put behind. And I found a "Movieclips" YouTube video of at least part of the scene. That part I'm talking about is where Marty/Robert Redford says, "...small countries?" at about 1:40.
Then Cosmo/Ben Kingsley nods smilingly and says, "I might even be able to crash the whole damn system...destroy all records of ownership. Think of it, Marty: no more rich people, no more poor people, everybody's the same. Isn't that what we said we always wanted?"
Upon seeing that again, I thought, gee...isn't that one of the story arcs in "Mr. Robot," fsociety taking down E Corp? The aim of fsociety was to compromise totally all the world's financial systems, so everything is "level" again.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
One of my favorite movies is "Sneakers." I have listened to the soundtrack CD many, many times. One of my favorites from that is "Cosmo… Old Friend," particularly in the one part with a strings crescendo I think in a major key, followed by a decrescendo in a minor key, followed by a piano (meaning soft) replaying of the major key, and this 3 (or 4) measure figure is repeated. All that is with a really great bass behind it (not sure what instrument; maybe a bunch of double basses arco). I was kind of wondering what scene in the movie was this put behind. And I found a "Movieclips" YouTube video of at least part of the scene. That part I'm talking about is where Marty/Robert Redford says, "...small countries?" at about 1:40.
Then Cosmo/Ben Kingsley nods smilingly and says, "I might even be able to crash the whole damn system...destroy all records of ownership. Think of it, Marty: no more rich people, no more poor people, everybody's the same. Isn't that what we said we always wanted?"
Upon seeing that again, I thought, gee...isn't that one of the story arcs in "Mr. Robot," fsociety taking down E Corp? The aim of fsociety was to compromise totally all the world's financial systems, so everything is "level" again.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
From the Everything Old is New Again Department
A local talk show host, WBEN-AM's Tom Bauerle, had a good point: we are going back to the time of the ancient Egyptians, who used hieroglyphs to communicate. The modern day equivalent is emoji! In fact, Matt Gray and Tom Scott launched (and not too long afterward took down) a Web site dedicated to communicating with no text, only emoji. (OK, that was gratuitous use of linking using Blogspot to link to 3 different YouTube videos.)
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
09 December, 2018
Don't Trust Your Disk Enclosures to Assess Disk Health
For the second time, I have removed a disk from its purchased enclosure (the first was IEEE1394/FireWire, yesterday's was USB 2). The result both times is that the disk was exhibiting wonky behavior in the enclosure, either outright throwing errors or hearing the read/write heads being repositioned a lot of times (you experienced computing folks know exactly what I heard), and after being extracted, work just fine on its own. The lamentable part is both of them were intended as backup disks, and the next-to-last thing you want to go wonky is your backups (with your first thing you don't want to go wonky is the computer itself). A case-in-point follows.
Almost ever since I got my 2 TB Seagate FreeAgent, it would have that tell-tale "I'm having trouble reading the platters" thonking of the heads. However, Seagate will not take warranty claims unless their utility (ugh...Windows only, .NET 4 requiring) tests it and the utility pronounces it defective (or put another way, it sounded like if you tried sending it in for a warranty claim without their utility finding it defective, you'd be economically responsible, not Seagate). At the time, I should have taken that as a hint that lots of folks found these disks dodgy, probably with the same almost unmistakable "help! I'm having read problems!" head thrashing, and had their warranty claims shot down. After all, why would Seagate even have to caution people about that on their site? But I digress. Recently I pressed it into service as the storage for MythTV recordings attached to a Raspberry Pi. Finally, this past week I had had enough of sitting there during recordings, and at times during viewings, of hearing the thwacka...thwacka...thwacka of repositionings/retries. I bought a 2 TB Toshiba USB disk (essentially it's a laptop drive with a case and a USB to SATA converter). While copying the MythTV videos from one disk to the other, there were plenty of times I heard that head banging. I then put the Toshiba drive in Myth service, with seemingly the only detriment being that if you plug it into the running Pi, it makes the voltage dip below threshold (making the red LED go dark for half a second or so) during the time the disk motor spins up. (After all, the Pi is only USB 2, the Toshiba is a USB 3.x unit, therefore has those higher allowable current draws.) I then proceeded to tear apart the Seagate case.
After putting the Seagate drive on a Sabrent USB 3 disk converter (turns out it's a Barracuda LP at heart), I did the same rsync copy where I heard the clackity-clacks before, but this time, there was no such noise. Soooo.....did it remap sectors and now it was getting a clean read? Was it overheating in the enclosure? Was the Seagate USB converter board wonky? Was the power supply unstable? Without some professional diagnostic tools, and maybe a clean room, it will be VERY difficult to tell for sure, but I'm guessing it's unstable power to the drive, like the 160 GB LaCie FireWire drive that preceeded it.
That's OK...the FreeAgent power suppy is being used for a different purpose, on some audio gear, where it doesn't seem to make a difference. Whatever glitches it might have had do not seem to be audible. My money's on the traces on the USB to SATA converter board just weren't up to scratch, and didn't provide enough stable current for the disk.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
Almost ever since I got my 2 TB Seagate FreeAgent, it would have that tell-tale "I'm having trouble reading the platters" thonking of the heads. However, Seagate will not take warranty claims unless their utility (ugh...Windows only, .NET 4 requiring) tests it and the utility pronounces it defective (or put another way, it sounded like if you tried sending it in for a warranty claim without their utility finding it defective, you'd be economically responsible, not Seagate). At the time, I should have taken that as a hint that lots of folks found these disks dodgy, probably with the same almost unmistakable "help! I'm having read problems!" head thrashing, and had their warranty claims shot down. After all, why would Seagate even have to caution people about that on their site? But I digress. Recently I pressed it into service as the storage for MythTV recordings attached to a Raspberry Pi. Finally, this past week I had had enough of sitting there during recordings, and at times during viewings, of hearing the thwacka...thwacka...thwacka of repositionings/retries. I bought a 2 TB Toshiba USB disk (essentially it's a laptop drive with a case and a USB to SATA converter). While copying the MythTV videos from one disk to the other, there were plenty of times I heard that head banging. I then put the Toshiba drive in Myth service, with seemingly the only detriment being that if you plug it into the running Pi, it makes the voltage dip below threshold (making the red LED go dark for half a second or so) during the time the disk motor spins up. (After all, the Pi is only USB 2, the Toshiba is a USB 3.x unit, therefore has those higher allowable current draws.) I then proceeded to tear apart the Seagate case.
After putting the Seagate drive on a Sabrent USB 3 disk converter (turns out it's a Barracuda LP at heart), I did the same rsync copy where I heard the clackity-clacks before, but this time, there was no such noise. Soooo.....did it remap sectors and now it was getting a clean read? Was it overheating in the enclosure? Was the Seagate USB converter board wonky? Was the power supply unstable? Without some professional diagnostic tools, and maybe a clean room, it will be VERY difficult to tell for sure, but I'm guessing it's unstable power to the drive, like the 160 GB LaCie FireWire drive that preceeded it.
That's OK...the FreeAgent power suppy is being used for a different purpose, on some audio gear, where it doesn't seem to make a difference. Whatever glitches it might have had do not seem to be audible. My money's on the traces on the USB to SATA converter board just weren't up to scratch, and didn't provide enough stable current for the disk.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
14 November, 2018
Recent systemd Update May Make Your Service Units Not Start Properly
I had set up a MythTV on a Raspberry Pi using the mythtv-light repository. I even used a lot of the suggestions from the MythTV wiki advising how to formulate a systemd service unit file. It seemed to be working fine, I was watching programs, pulling content from a SiliconDust HDHomeRun Quatro and dutifully recording streams from it to a USB HDD. Then TV enjoyment disaster struck.
Today I decided to set up another Pi to do the commercial flagging and any transcoding (although currently not doing it, although I may start due to things like Roku not supporting the native format, which may be either its own Nupple format or MPEG-TS, not sure). The plan is also to include MariaDB replication to the new Pi so that the database is fault tolerant too. I'd like to a make note here that I do like to do my own OS updates rather than have them done automatically; that way it's far simpler to relate something that starts breaking to a recently performed update rather than having to go back into some log files and figure out what changed. This would be for Raspbian Stretch (FYI, Raspian releases follow Debian releases, and Stretch is the current stable release as of writing this.) And once again this policy decision proved quite useful. I remember that there was a systemd update very recently, probably between the last start of my mythbackend and now. Having a User= directive in my unit file worked just fine, up until today when I restarted MariaDB, then the Myth backend (don't know if it tolerates the DB being restarted too well).
Suddenly mythfrontend was complaining that it could not connect to the backend. huh....That's odd. Does systemctl show that it's running? Indeed, it's running, and it's not exiting and respawning, because its PID is not changing (the unit file specifies it is to be restarted after 3 seconds if it exits for any reason other than systemd telling it to do so). But was there a listening socket? Darn, netstat -tln told me that no, there was not. There was nothing much to go on in the log file or systemcl status output, just something about not having data in a files cache in order to process expirations. A weird thing was that the HDD activity LED indicated frequent disk access, yet there was very little/no activity indicated on the network switch LED for the HDHomeRun. I had no idea what else it would be trying to do with the disk other than recording a program.
I thought, OK, slow way, way down, this is just too freaky. Even stopping the backend was taking a really long time. I got impatient and just used killall to try to stop it directly. That seemed to "help." So I wanted to see if something would be written to stdout if I ran the backend manually, from an XTerm command line. And of course, while doing that, I would not use the loglevel clause. But an odd thing happened: it ran normally. OK, might this have been a temporary anomaly? I tried once again to start via systemctl , and no, it was consistent. Listening sockets were never being opened. watch netstat -tln confirmed that. Starting via the command line (which worked) and watching for the sockets to open showed it was only a few seconds.
I must say, I have had a lot of experience in things running differently depending on whether they have been started from an interactive login versus by the system (from init). It's all in the execution environment. Unix/POSIX/Linux has so many process properties, but more often than not it's environment variables (LD_LIBRARY_PATH and PATH are two of the most common which are different between system and interactive invocation and therefore cause things to fail). So the next thing to try was removing User=mythtv (with "#") and adding /bin/su - mythtv -c to the command specifying how to start this unit. Bingo, there you go; it started, stayed running, and even more importantly opened the socket listeners. So hmmmm....what else does the system do for interactive logins? Why, not only does it set HOME (which was already being done in the unit file) but it also sets your current working directory (cwd) to that value! So hmmm....does a unit file have any directive like that? Yes, yes it does, WorkingDirectory=. So I set that to /home/mythtv, and it worked! For some really oddball reason, mythbackend will not open its sockets/operate normally unless the cwd is set like that. I have to wonder what systemd will choose for a service's cwd if you don't specify it, maybe the root. Moreover, I don't know if User= previously changed the cwd to that listed for the user (usually in /etc/passwd) or not.
Hopefully this story will help someone whose service daemons have stopped working.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
Google+ Shuts Down, so This Will Have To Do for "Microblogging"
grrr...Oh, well, I'm glad for how long it lasted, but as of next year sometime, G+ will be going away. So therefore, I see no reason to continue posting there, even though it is very much more convenient than posting here. That's OK. I didn't post particularly frequently over there anyway.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
07 August, 2017
Descriptive, Not Prescriptive: Why I Agree With Much of the Google Memo
Wow, there is an awful lot being said about the Google memo leaked last week. (UPDATE 08-Aug-2017: Motherboard (motherboard.vice.com) has pieced together a PDF of this memo which is much easier to read because of its formatting. I see Motherboard might be lefty too, because they refer to this as an anti-diversity memo, which if anything it is pro-diversity if one reads it critically and fully.) I think when an awful lot of people read it, they read into it what they want. The memo says as much:
For example, if you dare to point out that there are differences between men and women (and despite explicitly stating that a lot of those characteristics are only generalizations and there is a lot of variation with respect to individuals), the subject person is excoriated and told they are wrong and a horrible person, for essentially pointing out the truth. In other words, some people read into it that all people conform to these tendencies, instead of taking the proposition at face value, that they're just tendencies. I'm reminded of Eric S. Raymond's blog post, which if I read it correctly, advocates just looking at the code, and never mind the identities attached to that code. Yes, eventually give credit where credit is due, but only after the code is judged for its merit, and consider nothing about from whom it comes.
While a seemingly laudible goal, to see roughly equal representation of gender, all races, all faiths, all <insert characteristic here>, is fairly unlikely for the foreseeable future to see that realized. A stance of total equality (of outcome) denies the points in the memo, that it's just the way things are. That doesn't mean that's the way things always have to be, it just means it's often easy to see that's how we are now as a society. The memo's author even states:
In the last large corporate IT environment in which I worked, we were about 75% men and 25% women. I saw no hint whatsoever that women were oppressed in any way. I could detect no way in which the company had favoritism for anyone, other than for what they had a demonstrated aptitude (some better at backups/archiving, some better at server adminstration, some better at databases, and so on). For the most part, we all got along very well indeed. We saw each other first as people, as IT pros, and secondarily as men and women. Except for only one case in the late 1990s, it didn't matter that my supervisor was female, my supervisors were all the same professionally to me. And in that exception, I thought she was fine as a person, just wasn't as good as I would hope at her job. (That also applies just about equally to two men I had as supervisors at that same company, they were really agreeable guys, just less-than-stellar supervisors.)
Come on...all I'm asking for is equal treatment. This includes that I don't want favoritism towards others just for being female, or black, or anything else. It's about equality of opportunity, not equality of outcome. It's in no way negative that an IT staff is mostly men. It's just the way IT society is right now. I certainly hope more women do enter into and succeed in the field. But it must not be because they are women, it must be because they're good IT people.
Direct all comments to Google+, preferably under the post about this blog entry.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
"We all have biases and use motivated reasoning to dismiss ideas that run counter to our internal values."When someone dares to point out any discrepancies which contradict the reader's beliefs and practices, no matter how well reasoned, and sometimes no matter how filled with caveats, they will denounce the piece as being racist, sexist, discriminatory, etc.
For example, if you dare to point out that there are differences between men and women (and despite explicitly stating that a lot of those characteristics are only generalizations and there is a lot of variation with respect to individuals), the subject person is excoriated and told they are wrong and a horrible person, for essentially pointing out the truth. In other words, some people read into it that all people conform to these tendencies, instead of taking the proposition at face value, that they're just tendencies. I'm reminded of Eric S. Raymond's blog post, which if I read it correctly, advocates just looking at the code, and never mind the identities attached to that code. Yes, eventually give credit where credit is due, but only after the code is judged for its merit, and consider nothing about from whom it comes.
While a seemingly laudible goal, to see roughly equal representation of gender, all races, all faiths, all <insert characteristic here>, is fairly unlikely for the foreseeable future to see that realized. A stance of total equality (of outcome) denies the points in the memo, that it's just the way things are. That doesn't mean that's the way things always have to be, it just means it's often easy to see that's how we are now as a society. The memo's author even states:
"I hope it’s clear that I’m not saying that diversity is bad, that Google or society is 100% fair, that we shouldn’t try to correct for existing biases, or that minorities have the same experience of those in the majority. My larger point is that we have an intolerance for ideas and evidence that don’t fit a certain ideology. I’m also not saying that we should restrict people to certain gender roles; I’m advocating for quite the opposite: treat people as individuals, not as just another member of their group (tribalism)."But of course, this will get glossed over and dismissed. Again, someone needs to be judged on their merits and actions, not their gender, not their race, not their religion, not to whom they're attracted, not their class of any sort. This includes not being given any favoritism. In other words, like the memo author, since I am male, I would feel a certain amount of frustration and anger if a company hires or promotes a woman because they seem to be low on their female quota, thus passing over me.
In the last large corporate IT environment in which I worked, we were about 75% men and 25% women. I saw no hint whatsoever that women were oppressed in any way. I could detect no way in which the company had favoritism for anyone, other than for what they had a demonstrated aptitude (some better at backups/archiving, some better at server adminstration, some better at databases, and so on). For the most part, we all got along very well indeed. We saw each other first as people, as IT pros, and secondarily as men and women. Except for only one case in the late 1990s, it didn't matter that my supervisor was female, my supervisors were all the same professionally to me. And in that exception, I thought she was fine as a person, just wasn't as good as I would hope at her job. (That also applies just about equally to two men I had as supervisors at that same company, they were really agreeable guys, just less-than-stellar supervisors.)
Come on...all I'm asking for is equal treatment. This includes that I don't want favoritism towards others just for being female, or black, or anything else. It's about equality of opportunity, not equality of outcome. It's in no way negative that an IT staff is mostly men. It's just the way IT society is right now. I certainly hope more women do enter into and succeed in the field. But it must not be because they are women, it must be because they're good IT people.
ADDITION 08-Aug-2017: There was something pointed out on "The Glenn Beck Program" this morning, which is quite relevant: The fix for discrimination is not discrimination. In other words, the fix for discriminating against women is not to discriminate against men, it's to remove discrimination.
Another thing on this morning's program which was in the back of my mind while writing yesterday, but I didn't put it in here, is this: When, if ever, is Google's lefty bias going to bleed over into search results? Will the indexer look at the page contents and decide it's not the sort of page it wants me to see, because of these biases? What we really need is a search engine superclass, one that goes out to Oath, Bing, Google, maybe others, and combines the results somehow. The challenge may be to find search engines maintained by companies who aren't lefites. The point is, diversity in search engines would be good too.
Their bias is sometimes manifest in what they choose to Doodle. There are a few times where the radio personalities I enjoy point out that Google will bother to Doodle something which it knows is served to Americans that is some overseas event, but when it comes to recognizing something in our culture, pffft, forget that. It's not even necessarily a seeming lack of patriotism. Looking back at the Google Doodle archive, there is no Doodle for Easter, no Doodle for Memorial Day, and there's one for Fourth of July, which....come on....everyone around the world has 04-Jul...I have to wonder why it's not labeled US Independence Day or something like that. Oddly enough, there is a Doodle for Veterans Day 2016.
Direct all comments to Google+, preferably under the post about this blog entry.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
06 March, 2017
Xorg Voodoo, and It Really Pays to Practice Your Backups and Restores
I wish Xorg/Wayland/Weston was not so much black magic voodoo juju.
For some experimentation and hacking fun, I installed gdm3 on my Xubuntu desktop system. I set up gdm as my default DM (dpkg-reconfigure lightdm and select gdm from the list), and then I ran systemctl stop lightdm and systemctl start gdm. That was somewhat of a visual shock, because I had never run gdm3 before, but nonetheless, it was usable. I logged in as my normal user.
I had some "normal" logins, where GDM started up my "normal" Xfce session. Then I decided, I wanted to see if the "Weston" option worked, as it had not under LightDM. Shazam, whatever GDM does that LightDM doesn't, I don't know, but that entry worked. Likewise I fiddled a little with "GNOME on Wayland", which was interesting. It's the first time I've ever used (I think it's called) the Lens. Meh. It's OK I guess, but I miss my menus of applications and such. I don't like the Lens so much.
One of the first things I noticed is, "log out" was not part of the dialog like it was under Xfwm/LightDM. There was only poweroff, reboot, and suspend. Huh? That seems kind of weird. Eventually, I found out (don't remember where) that there was a separate logout option. Then I hopped on over to tty1, and did systemctl stop gdm (might have had a 3 too). Then systemctl start gdm. Wow, that's really weird. GDM didn't start, but it looked like several times per second, it was trying. It was even difficult to type systemctl stop gdm because a couple of times a second, input was being stolen by the process trying to start GDM (or Xorg, not sure which). In fact, I don't know what the deuce was going on, but I could start neither gdm nor lightdm.
At this point I reasoned, I had seen the systemd file for lightdm and remembered it had a test for the default display manager. I would have figured dpkg-reconfigure would just use systemctl enable and systemctl disable because it "knows" the list of DMs, but it writes /etc/X11/default-display-manager anyway. Okeydokey, I did anoher dpkg-reconfigure lightdm and selected lightdm. That still wouldn't start either. Well, neither would gdm, so I rebooted.
The first surprise came when LightDM started really soon. I had set up lightdm to be disabled, because I want all the stuff which happens at boot to settle down first, then start the display server. I do this in rc.local by backgrounding a shell script which sleeps 20 seconds then does the appropriate thing for the service. It used to use an appropriate Upstart command, but of course when upgrading Xubuntu LTS 14 to 16, it had to be updated to use systemctl(8) instead. But it seems the dpkg-reconfigure had undone any enable or disable, since I had not even gotten the prompt on tty1 before the VT was changed to tty7 to start the X server. Meh, OK, I recognized this and just disabled lightdm.
I had fun experimenting with starting Weston (like loading different modules in the [core] section). One thing that didn't work too well was using drm-backend.so for Weston. That not only killed Weston, but also whacked Xorg too. That got a little whacky in that I had problems after that switching VTs. I had to log into another host on my network, SSH to the workstation, and systemctl reboot. After all, a computer isn't particularly useful if you can't type at it, if that's the way you normally give input to it.
I got tired of going through dpkg-reconfigure to switch DMs, so I just edited /etc/X11/default-display-manager directly. That seemed to be OK, but eventually, I got to a point where GDM wouldn't start, and LightDM wouldn't start either. Huh, that's weird. So I restarted the whole system.
Then there was the chilling realization that systemctl start lightdm did not do a whole lot except throw errors I could not understand in to the systemctl status lightdm and journalctl outputs, like stuff about some assertion failing. I'm sure if I wanted to take the time to download the ENTIRE SOURCE package for Xorg, I might see what that assertion does, and why its failure was happening, but I was not about to take all that time to futz around with that. What I thought might have helped is, I have an Xorg "prestart" script which sets the screen saver timeout and DPMS, changes the root window background color so that I know Xorg is running but before LightDM can initialize, and use some xrandr commands to set up the resolutions and refresh rates of the two framebuffers/monitors. (Xorg cannot read EDID information because the switches through which both monitors are connected mangle EDID, so it uses defaults...and that's just really ugly.) While I was writing that prestart script, I redirected stderr to an unused tty. All I got on that VT was messages about "can't open display." In retrospect, what I really should have peeked at was /var/log/Xorg.0.log for clues, but it would not necessarily have revealed anything I could understand.
I tried another dpkg-reconfigure to make sure whatever needs to be done to switch DMs is done, figuring it might be more than just rewriting the /etc/X11/default-display-manager file. That, unfortunately, was no help whatsoever. Restarting the system did not help either. I remove/purge'd the gdm3 package; no help. I reinstalled the lightdm package; that wasn't any help either. Sigh. It was going to be a really bad day if the only thing which is going to get my dailly driver back is a Xubuntu reinstallation and reconfiguration. At least the vast, vast majority of my personal settings and data is on a separate /home logical volume. I could very likely keep all the logical volumes, filesystems (but remade filesystems, except for /home of course) and stuff, so it wouldn't be like a blank disk installation. I have to imagine there will unfortunately be a somewhat large portion of *buntu users where that would be their only option because they're just not that experienced or learned in operations at this level. Most folks don't need it because their systems just work, they get their work done, and the amount of experimentation, especially at the system level, is minimal.
Next I did something I do very rarely, which is select the entry for system recovery at system boot. I figured I needed as little as possible running for what I was about to do next. Ugh. That is really ugly because of the nomodeset option. I am really, really used to the VTs coming up 1920x1080 (or 240x67). So, I restarted and edited the default entry instead, adding "single" to the end of the kernel command line. I figured pretty much all the configuration is held in /etc, so I figured out which disk and logical volume I used for backups last (which was right around midnight Sunday, started it up and went to bed) and mounted it. Then I did rsync -av --delete /mnt/bkup/thishost/etc/. /etc/. to get the /etc directory back to how it was. That went really quickly, as you can imagine. Then I just hit Ctrl-Alt-Del. That's of course going to umount the LV on the USB disk, deactivate all the USB LVs, everything buttoned up and ready to restart.
Except that didn't help either. I even tried unplugging my computer for a while figuring it was some really weird juju with how the video controller was being initialized..hoping letting the capacitors discharge would unstick this lack of Xorg starting. No, as I could have predicted, that really wasn't it either.
The semi-weird thing is, while logged in as the superuser on tty1, I could run Xorg :0 just fine. Of course, that's not particularly useful, but at least it proved it was not a hardware denial, or corrupted driver .so'es, or something like that. The X server itself would start, it's just that lightdm couldn't start it and use it. Well...come to think of it, the screen was initialized to all black, not the gray dot pattern it usually does, and no big X cursor appeared. Not sure what was up with that. At least it didn't go, as it sometimes will when it's failing, to VT 7 and do nothing but leave the blinking text mode cursor there.
I was getting really discouraged (and a little panicked to be honest) at this point. I thought it was going to be hours before being up and running again. I was starting to think of, how am I going to fetch the ISO to do another installation? Can I get one effectively with one of my other systems, likely with Lynx? I mean, as IT disasters go, this is pretty mild because at least there is a "known way out" (namelly OS reinstallation) which is nearly guaranteed to get the blasted thing working again. It's just the thought of the long, long time it was going to take to make that happen, with all the work that would need to be done in terms of installing the packages I like which basically has to happen after the standard installation was finished. It could be a lot worse; it could be the CPU itself which doesn't work, and I'd have to go back to a LOT slower machine (from a Core 2 Duo to a Pentium IV).
Sigh. OK, I wasn't too sure about using my complete backup. I do a number of --exclude= directives when I do the backups. But I'm never quite sure if I am excepting enough. For example, it'd probably be less than a good result if the LVM information was overwritten (so actually, that's already excluded). And sometimes the presence of files can make a difference, so of course you're going to have to use the --delete directive. I'm thinking, if this obliterates the wrong things, it's going to be a long, arduous reinstallation process, but hey, it's at least worth a try to do a full restore. After all, like YouTuber AvE often says, if it's broken, how can it hurt a whole lot to break it some more? Worst thing that happens is, my restore methodology overwrites zeroes over everything, and I have to reinstall everything anyway. Surely it will take not a whole lot of time to MUNG things to the point where OS reinstallation becomes a certainty.
So with some trepidation, in single user mode again, I mounted up the last backup, but I was still unsure of what I was about to mangle, so I added the --dry-run option to rsync. And boy am I glad I did. When you go about deleting things like lost+found, and bad things™ happen, even worse things tend to happen when fsck is trying to set things right and it can't write to lost+found because it's not there. It's also not particularly useful to go mucking about in /sys or /proc. I definitely didn't want to get into a loop trying to do untoward things with /mnt/bkup/thishost so I knew enough to mount the backup read-only, but still figured out what I really wanted to do is exclude everything under /mnt. I also chose to exclude everything under my $HOME but it would still be possible that some of the session files under there could screw with logging in under Xfce (or who knows, one time I got auto-logged into Weston when I didn't mean to, it must have stuck as the last thing I tried in the greeter).
So eventually I settled on a pretty significant set of --exclude's and let it rip. As I had been experimenting with --dry-run a number of times, there already was significant information in the block cache that really, it was only a few minutes later that rsync said it was finished. I restarted, unplugged the backup disk's PSU while the BIOS screen was showing (yep, it's that old, not UEFI), and let GrUB do its thing. And...
Success!!
I killed my little delayed DM starter script, did systemctl start lightdm, and the system once again looked normal. Of course, since the system is on a conventional SATA disk (not an SSD), it took agonizingly long to initialize, but I knew things were likely going to work OK because I got my normal prompt from ssh-agent to enter in the passphrase for my private keys in an XTerm.
What I'd really, really like to know is, what caused LightDM not to be able to start Xorg? That's the voodoo juju part of all this. You'd really hope that something particularly helpful woud be in the journalctl output, or systemctl status. But alas, no help was forthcoming. These days, if you don't have a working graphics environment where you can run a browser with JavaScript capabilities, lamentably you're at quite a disadvantage in researching possible causes and remedies. The usual copying/pasting of an error message into a Google search is going to be quite difficult.
Long and the short of it is, it's really a particularly good idea to practice restores every now and again. It will point out deficiencies in either your backup or your restore methodology, or maybe both. In any case, with such practice, it shoud speed up recovery from being in a jam.
Direct all comments to Google+, preferably under the post about this blog entry.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
Please join one of the fastest growing social networks, Google+!
For some experimentation and hacking fun, I installed gdm3 on my Xubuntu desktop system. I set up gdm as my default DM (dpkg-reconfigure lightdm and select gdm from the list), and then I ran systemctl stop lightdm and systemctl start gdm. That was somewhat of a visual shock, because I had never run gdm3 before, but nonetheless, it was usable. I logged in as my normal user.
I had some "normal" logins, where GDM started up my "normal" Xfce session. Then I decided, I wanted to see if the "Weston" option worked, as it had not under LightDM. Shazam, whatever GDM does that LightDM doesn't, I don't know, but that entry worked. Likewise I fiddled a little with "GNOME on Wayland", which was interesting. It's the first time I've ever used (I think it's called) the Lens. Meh. It's OK I guess, but I miss my menus of applications and such. I don't like the Lens so much.
One of the first things I noticed is, "log out" was not part of the dialog like it was under Xfwm/LightDM. There was only poweroff, reboot, and suspend. Huh? That seems kind of weird. Eventually, I found out (don't remember where) that there was a separate logout option. Then I hopped on over to tty1, and did systemctl stop gdm (might have had a 3 too). Then systemctl start gdm. Wow, that's really weird. GDM didn't start, but it looked like several times per second, it was trying. It was even difficult to type systemctl stop gdm because a couple of times a second, input was being stolen by the process trying to start GDM (or Xorg, not sure which). In fact, I don't know what the deuce was going on, but I could start neither gdm nor lightdm.
At this point I reasoned, I had seen the systemd file for lightdm and remembered it had a test for the default display manager. I would have figured dpkg-reconfigure would just use systemctl enable and systemctl disable because it "knows" the list of DMs, but it writes /etc/X11/default-display-manager anyway. Okeydokey, I did anoher dpkg-reconfigure lightdm and selected lightdm. That still wouldn't start either. Well, neither would gdm, so I rebooted.
The first surprise came when LightDM started really soon. I had set up lightdm to be disabled, because I want all the stuff which happens at boot to settle down first, then start the display server. I do this in rc.local by backgrounding a shell script which sleeps 20 seconds then does the appropriate thing for the service. It used to use an appropriate Upstart command, but of course when upgrading Xubuntu LTS 14 to 16, it had to be updated to use systemctl(8) instead. But it seems the dpkg-reconfigure had undone any enable or disable, since I had not even gotten the prompt on tty1 before the VT was changed to tty7 to start the X server. Meh, OK, I recognized this and just disabled lightdm.
I had fun experimenting with starting Weston (like loading different modules in the [core] section). One thing that didn't work too well was using drm-backend.so for Weston. That not only killed Weston, but also whacked Xorg too. That got a little whacky in that I had problems after that switching VTs. I had to log into another host on my network, SSH to the workstation, and systemctl reboot. After all, a computer isn't particularly useful if you can't type at it, if that's the way you normally give input to it.
I got tired of going through dpkg-reconfigure to switch DMs, so I just edited /etc/X11/default-display-manager directly. That seemed to be OK, but eventually, I got to a point where GDM wouldn't start, and LightDM wouldn't start either. Huh, that's weird. So I restarted the whole system.
Then there was the chilling realization that systemctl start lightdm did not do a whole lot except throw errors I could not understand in to the systemctl status lightdm and journalctl outputs, like stuff about some assertion failing. I'm sure if I wanted to take the time to download the ENTIRE SOURCE package for Xorg, I might see what that assertion does, and why its failure was happening, but I was not about to take all that time to futz around with that. What I thought might have helped is, I have an Xorg "prestart" script which sets the screen saver timeout and DPMS, changes the root window background color so that I know Xorg is running but before LightDM can initialize, and use some xrandr commands to set up the resolutions and refresh rates of the two framebuffers/monitors. (Xorg cannot read EDID information because the switches through which both monitors are connected mangle EDID, so it uses defaults...and that's just really ugly.) While I was writing that prestart script, I redirected stderr to an unused tty. All I got on that VT was messages about "can't open display." In retrospect, what I really should have peeked at was /var/log/Xorg.0.log for clues, but it would not necessarily have revealed anything I could understand.
I tried another dpkg-reconfigure to make sure whatever needs to be done to switch DMs is done, figuring it might be more than just rewriting the /etc/X11/default-display-manager file. That, unfortunately, was no help whatsoever. Restarting the system did not help either. I remove/purge'd the gdm3 package; no help. I reinstalled the lightdm package; that wasn't any help either. Sigh. It was going to be a really bad day if the only thing which is going to get my dailly driver back is a Xubuntu reinstallation and reconfiguration. At least the vast, vast majority of my personal settings and data is on a separate /home logical volume. I could very likely keep all the logical volumes, filesystems (but remade filesystems, except for /home of course) and stuff, so it wouldn't be like a blank disk installation. I have to imagine there will unfortunately be a somewhat large portion of *buntu users where that would be their only option because they're just not that experienced or learned in operations at this level. Most folks don't need it because their systems just work, they get their work done, and the amount of experimentation, especially at the system level, is minimal.
Next I did something I do very rarely, which is select the entry for system recovery at system boot. I figured I needed as little as possible running for what I was about to do next. Ugh. That is really ugly because of the nomodeset option. I am really, really used to the VTs coming up 1920x1080 (or 240x67). So, I restarted and edited the default entry instead, adding "single" to the end of the kernel command line. I figured pretty much all the configuration is held in /etc, so I figured out which disk and logical volume I used for backups last (which was right around midnight Sunday, started it up and went to bed) and mounted it. Then I did rsync -av --delete /mnt/bkup/thishost/etc/. /etc/. to get the /etc directory back to how it was. That went really quickly, as you can imagine. Then I just hit Ctrl-Alt-Del. That's of course going to umount the LV on the USB disk, deactivate all the USB LVs, everything buttoned up and ready to restart.
Except that didn't help either. I even tried unplugging my computer for a while figuring it was some really weird juju with how the video controller was being initialized..hoping letting the capacitors discharge would unstick this lack of Xorg starting. No, as I could have predicted, that really wasn't it either.
The semi-weird thing is, while logged in as the superuser on tty1, I could run Xorg :0 just fine. Of course, that's not particularly useful, but at least it proved it was not a hardware denial, or corrupted driver .so'es, or something like that. The X server itself would start, it's just that lightdm couldn't start it and use it. Well...come to think of it, the screen was initialized to all black, not the gray dot pattern it usually does, and no big X cursor appeared. Not sure what was up with that. At least it didn't go, as it sometimes will when it's failing, to VT 7 and do nothing but leave the blinking text mode cursor there.
I was getting really discouraged (and a little panicked to be honest) at this point. I thought it was going to be hours before being up and running again. I was starting to think of, how am I going to fetch the ISO to do another installation? Can I get one effectively with one of my other systems, likely with Lynx? I mean, as IT disasters go, this is pretty mild because at least there is a "known way out" (namelly OS reinstallation) which is nearly guaranteed to get the blasted thing working again. It's just the thought of the long, long time it was going to take to make that happen, with all the work that would need to be done in terms of installing the packages I like which basically has to happen after the standard installation was finished. It could be a lot worse; it could be the CPU itself which doesn't work, and I'd have to go back to a LOT slower machine (from a Core 2 Duo to a Pentium IV).
Sigh. OK, I wasn't too sure about using my complete backup. I do a number of --exclude= directives when I do the backups. But I'm never quite sure if I am excepting enough. For example, it'd probably be less than a good result if the LVM information was overwritten (so actually, that's already excluded). And sometimes the presence of files can make a difference, so of course you're going to have to use the --delete directive. I'm thinking, if this obliterates the wrong things, it's going to be a long, arduous reinstallation process, but hey, it's at least worth a try to do a full restore. After all, like YouTuber AvE often says, if it's broken, how can it hurt a whole lot to break it some more? Worst thing that happens is, my restore methodology overwrites zeroes over everything, and I have to reinstall everything anyway. Surely it will take not a whole lot of time to MUNG things to the point where OS reinstallation becomes a certainty.
So with some trepidation, in single user mode again, I mounted up the last backup, but I was still unsure of what I was about to mangle, so I added the --dry-run option to rsync. And boy am I glad I did. When you go about deleting things like lost+found, and bad things™ happen, even worse things tend to happen when fsck is trying to set things right and it can't write to lost+found because it's not there. It's also not particularly useful to go mucking about in /sys or /proc. I definitely didn't want to get into a loop trying to do untoward things with /mnt/bkup/thishost so I knew enough to mount the backup read-only, but still figured out what I really wanted to do is exclude everything under /mnt. I also chose to exclude everything under my $HOME but it would still be possible that some of the session files under there could screw with logging in under Xfce (or who knows, one time I got auto-logged into Weston when I didn't mean to, it must have stuck as the last thing I tried in the greeter).
So eventually I settled on a pretty significant set of --exclude's and let it rip. As I had been experimenting with --dry-run a number of times, there already was significant information in the block cache that really, it was only a few minutes later that rsync said it was finished. I restarted, unplugged the backup disk's PSU while the BIOS screen was showing (yep, it's that old, not UEFI), and let GrUB do its thing. And...
Success!!
I killed my little delayed DM starter script, did systemctl start lightdm, and the system once again looked normal. Of course, since the system is on a conventional SATA disk (not an SSD), it took agonizingly long to initialize, but I knew things were likely going to work OK because I got my normal prompt from ssh-agent to enter in the passphrase for my private keys in an XTerm.
What I'd really, really like to know is, what caused LightDM not to be able to start Xorg? That's the voodoo juju part of all this. You'd really hope that something particularly helpful woud be in the journalctl output, or systemctl status. But alas, no help was forthcoming. These days, if you don't have a working graphics environment where you can run a browser with JavaScript capabilities, lamentably you're at quite a disadvantage in researching possible causes and remedies. The usual copying/pasting of an error message into a Google search is going to be quite difficult.
Long and the short of it is, it's really a particularly good idea to practice restores every now and again. It will point out deficiencies in either your backup or your restore methodology, or maybe both. In any case, with such practice, it shoud speed up recovery from being in a jam.
Direct all comments to Google+, preferably under the post about this blog entry.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
Please join one of the fastest growing social networks, Google+!
02 March, 2017
Yesterday, I knocked myself off the Internet
Yesterday, I spent some time poking about the Actiontec MI424WR (Rev I) that Verizon supposedly provided "for free" as an incentive to subscribe to FiOS. Supposedly, in order to be (fully) supported, you need a Verizon approved router, including one of these. I'm not sure their site allows one to complete the online (service) order without agreeing to rent, purchase, or otherwise prove you have (or will have) one of Verizon's routers. You may be able to finish the order these days, but as I recall, two years ago when I was ordering, their maze of forms and JavaScript wouldn't allow a submission without one. Anyhow...I read a recent thread on DSL Reports with regards to residential class accounts being able to have static IP addresses (they don't allow that) and the workaround of using dynamic DNS services prompted me to start poking around to see what services (dyndns.org, noip.com, etc.) that the Verizon router supports directly.
I have VLANs set up on my switch, one for TWC/Spectrum WAN (although I don't subscribe to any of their services presently), one for my VOIP LAN, one for most of the rest of my LAN, one for FiOS WAN, and one for the FiOS LAN. The nexus for everything is a PC running Linux functioning as a router. I knew there was the possibility of an address "conflict" if I plugged in the WAN port on the Actiontec (because Verizon only allows one DHCP lease at a time) so initially I powered up the Actiontec with the WAN cable unplugged.
After puttering about with a lot of its settings (ugh, I hate the Actiontec Web interface), I'm not sure what possessed me, but I thought, hey, my Linux router has a DHCP lease, and since Verizon's systems will only allow one lease at a time, I thought plugging in the Actiontec WAN cable should be no problem. If it tries to obtain a lease it will just be denied, whether by DHCPNAK or just timing out.
Emmmm....wrong! Very shortly after plugging in the WAN cable, the "Internet" LED came on. First I thought, "wait, what?" That was shortly followed by "oh, crap!" Sure enough, I logged onto my "production" router, tried the usual "ping 8.8.8.8", and there were no replies whatsoever. There isn't anything of consequence connected to the LAN ports of the Actiontec; it was pretty much just connected so that I could get in to configure it, and possibly switch things up a bit if a Verizon TSR demanded to have it online. So either the lease which Linux had obtained was somehow "transferred" to the Actiontec, or the lease Linux had was invalidated, and at any rate, in that state the Linux router was of no (WAN) use. (It still routed just fine between all the LANs.) I basically knocked myself off the Internet, because nothing on my network is set up to operate through the Actiontec. I thought, you idiot, you should have logged onto the switch and issued "shutdown" to the interface for the Actiontec WAN port first.
As you may gather from some of my previous postings, here on the I Heart Libertarianism blog or on Google+, I get pretty anxious about not having Internet connectivity, so to lift a line from Dickens, this was not the best of times. I think this is mostly because I have the family's email server here, not to mention virtually all the important notifications I have would go to a philipps.us or joe.philipps.us address. It's also the DNS master for a number of my domains, including philipps.us. I know, I know...the TTLs on the SOA records themselves should make them valid for two weeks, so even without Internet for an extended-ish time, things should not fall apart entirely.
Email servers very typically keep retrying for several days, maybe even as much as a week, so that should not be so terrible. As a further mitigation of any failure of my email server here, it just so happens I was one of the people who got in on the "ground floor" when Google was beta testing Google Apps (the Web services, not the usual meaning these days of the apps to access Google on Android). As a consequence, I have a "no cost" G Suite configuration as a less preferred MX. Therefore, it would be somewhat messy from an email history standpoint, but a catchall account on G Suite would have any email which my setup cannot suck in. Still...I think it's the thought that without Internet, even that backup setup is no good because I can't get to it. I would have to "borrow" someone else's Internet access even to see what's over at my G Suite account.
This would be compounded by the fact that these days, many of my access passwords are utter gibberish, thanks to KeePass and KeePassX. The database is in my Google Drive, but also backed up on my local computer. The implication is, it's another one of those "bootstrap" problems, without Internet, I don't have access to the master KeePass database, and even if I work from the copy, say from a computer at the Erie County library, it's going to be a LOT of tedious typing because the library's computers are likely not going to be able to run the KeePass software. I'd be working with revealing the decrypted passwords on KeePassDroid on my Nexus 7. For any Web services which will accept it, I turn on basically everything printable except space for the KeePass generator, and typically 20 characters. So yeah....lots of tedious typing if I have to use another computer.
Despite the minor panic I was in, I thought, come on, this shouldn't be that difficult, you really should have a way out of this. You can try ifdown on the Internet interface (happens to be eth3), followed by ifup. Nope, that didn't really do anything. Just calm down a little, and work the problem. If you get back on the Actiontec, you should be able to pick and prod your way around it, and find the "release DHCP lease" button, which you know is in there somewhere. That would at least mollify Verizon's backend(s) (or the ONT) into letting Linux get a usable address again. That was in fact the key. After hitting "release" on the Actiontec, I was able to ifdown/ifup one more time, and Linux got an address/lease. However...it was not the IPv4 address I had before. Rats.
As mentioned, the whole exercise started with wondering about Actiontec's implementation of dynamic DNS. This is precisely what I needed to do. This happens so infrequently that I have a Google Keep checklist for IPv4 address changes (which I have exported to a Google Doc for linking in this blog entry). I have that accessible on my Nexus 7, so all I have to do is find it on there, and I'm good to go. I copied the list, renamed it with "1-Mar-2017" in the title, and went about executing its items.
There were items on the checklist that I still had to figure out on the spot. For example, for some of the items, I did not know the pathnames of what needed changing, or what item in the relevant file. So in a sense, it's good this happened, because it has made me refine the process and therefore improve it. Still, it's a pain whenever my address changes. Some of it could probably be scripted or automated, but it's one of those things that happens so infrequently, I have to wonder how much utility there is in writing anything.
Anyhow...obviously, I'm back online, or I couldn't be posting this. Hopefully I'll be better prepared for the next time my address changes.
Direct all comments to Google+, preferably under the post about this blog entry.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
Please join one of the fastest growing social networks, Google+!
I have VLANs set up on my switch, one for TWC/Spectrum WAN (although I don't subscribe to any of their services presently), one for my VOIP LAN, one for most of the rest of my LAN, one for FiOS WAN, and one for the FiOS LAN. The nexus for everything is a PC running Linux functioning as a router. I knew there was the possibility of an address "conflict" if I plugged in the WAN port on the Actiontec (because Verizon only allows one DHCP lease at a time) so initially I powered up the Actiontec with the WAN cable unplugged.
After puttering about with a lot of its settings (ugh, I hate the Actiontec Web interface), I'm not sure what possessed me, but I thought, hey, my Linux router has a DHCP lease, and since Verizon's systems will only allow one lease at a time, I thought plugging in the Actiontec WAN cable should be no problem. If it tries to obtain a lease it will just be denied, whether by DHCPNAK or just timing out.
Emmmm....wrong! Very shortly after plugging in the WAN cable, the "Internet" LED came on. First I thought, "wait, what?" That was shortly followed by "oh, crap!" Sure enough, I logged onto my "production" router, tried the usual "ping 8.8.8.8", and there were no replies whatsoever. There isn't anything of consequence connected to the LAN ports of the Actiontec; it was pretty much just connected so that I could get in to configure it, and possibly switch things up a bit if a Verizon TSR demanded to have it online. So either the lease which Linux had obtained was somehow "transferred" to the Actiontec, or the lease Linux had was invalidated, and at any rate, in that state the Linux router was of no (WAN) use. (It still routed just fine between all the LANs.) I basically knocked myself off the Internet, because nothing on my network is set up to operate through the Actiontec. I thought, you idiot, you should have logged onto the switch and issued "shutdown" to the interface for the Actiontec WAN port first.
As you may gather from some of my previous postings, here on the I Heart Libertarianism blog or on Google+, I get pretty anxious about not having Internet connectivity, so to lift a line from Dickens, this was not the best of times. I think this is mostly because I have the family's email server here, not to mention virtually all the important notifications I have would go to a philipps.us or joe.philipps.us address. It's also the DNS master for a number of my domains, including philipps.us. I know, I know...the TTLs on the SOA records themselves should make them valid for two weeks, so even without Internet for an extended-ish time, things should not fall apart entirely.
Email servers very typically keep retrying for several days, maybe even as much as a week, so that should not be so terrible. As a further mitigation of any failure of my email server here, it just so happens I was one of the people who got in on the "ground floor" when Google was beta testing Google Apps (the Web services, not the usual meaning these days of the apps to access Google on Android). As a consequence, I have a "no cost" G Suite configuration as a less preferred MX. Therefore, it would be somewhat messy from an email history standpoint, but a catchall account on G Suite would have any email which my setup cannot suck in. Still...I think it's the thought that without Internet, even that backup setup is no good because I can't get to it. I would have to "borrow" someone else's Internet access even to see what's over at my G Suite account.
This would be compounded by the fact that these days, many of my access passwords are utter gibberish, thanks to KeePass and KeePassX. The database is in my Google Drive, but also backed up on my local computer. The implication is, it's another one of those "bootstrap" problems, without Internet, I don't have access to the master KeePass database, and even if I work from the copy, say from a computer at the Erie County library, it's going to be a LOT of tedious typing because the library's computers are likely not going to be able to run the KeePass software. I'd be working with revealing the decrypted passwords on KeePassDroid on my Nexus 7. For any Web services which will accept it, I turn on basically everything printable except space for the KeePass generator, and typically 20 characters. So yeah....lots of tedious typing if I have to use another computer.
Despite the minor panic I was in, I thought, come on, this shouldn't be that difficult, you really should have a way out of this. You can try ifdown on the Internet interface (happens to be eth3), followed by ifup. Nope, that didn't really do anything. Just calm down a little, and work the problem. If you get back on the Actiontec, you should be able to pick and prod your way around it, and find the "release DHCP lease" button, which you know is in there somewhere. That would at least mollify Verizon's backend(s) (or the ONT) into letting Linux get a usable address again. That was in fact the key. After hitting "release" on the Actiontec, I was able to ifdown/ifup one more time, and Linux got an address/lease. However...it was not the IPv4 address I had before. Rats.
As mentioned, the whole exercise started with wondering about Actiontec's implementation of dynamic DNS. This is precisely what I needed to do. This happens so infrequently that I have a Google Keep checklist for IPv4 address changes (which I have exported to a Google Doc for linking in this blog entry). I have that accessible on my Nexus 7, so all I have to do is find it on there, and I'm good to go. I copied the list, renamed it with "1-Mar-2017" in the title, and went about executing its items.
There were items on the checklist that I still had to figure out on the spot. For example, for some of the items, I did not know the pathnames of what needed changing, or what item in the relevant file. So in a sense, it's good this happened, because it has made me refine the process and therefore improve it. Still, it's a pain whenever my address changes. Some of it could probably be scripted or automated, but it's one of those things that happens so infrequently, I have to wonder how much utility there is in writing anything.
Anyhow...obviously, I'm back online, or I couldn't be posting this. Hopefully I'll be better prepared for the next time my address changes.
Direct all comments to Google+, preferably under the post about this blog entry.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
Please join one of the fastest growing social networks, Google+!
24 July, 2016
Some Company Practices Are Just So Nonsensical
Sometimes I just have no understanding of company policies. I think I need to remind the reader that there's no such thing as a free lunch (TNSTAAFL). Things that appear to be free are never genuinely free, there is always a price to be paid for anything, especially when dealing with a company.
I don't know, maybe there is some law which says billing has to be like this, but I'm dumbfounded by my recent experience with Time Warner Cable (can we call them Charter or Spectrum? or not yet?).
I just couldn't justify paying O($25) per month for a service of which I rarely partake. (I'll use this quasi- "big O notation" throughout this entry to mean "on the order of." It has a similar but not quite the same meaning in mathematics and computer science.) I find a lot of entertainment in YouTube videos for example, and something like Netflix, Vudu, Amazon Video/Amazon Prime, Hulu, etc. would be half the cost or less. So I used the expiration of my SchedulesDirect subscription as a prompt to tell them, "sorry, disconnect me, please." They even offered to knock around $10/mo. off, but still, I don't watch that much on there. After all, how many times can you watch the same episodes of M*A*S*H on WBBZ? I must have more than 100 hours of recordings waiting to be watched on my MythTV.
A few days after I called in the disconnect order, I realized I never asked the CSR what my final bill should be, since it would be prorated. It's also worthy to note that billing is explicitly ahead of time; that is to say, your payment is for the month to come, not for the month that ends on your billing date. I hopped on their Web chat, and they gave me a figure for the partial month's service. I thought I had disabled automatic payment, but they advised me not to pay anything. However, I pointed out to them if I didn't pay for my partial month, and just wait for another bill, I'd be paying late, which I would rather not do for ethical reasons, plus a possible hit to my credit rating. So I logged into TWC's site, scheduled a payment for that reduced amount, and thought that would set things square. But no, that was wrong of me to think that.
For starters, there is ridiculous lag with those folks. It turns out if you want to cancel automatic payments, you should do that before your bill arrives, otherwise an automatic payment will be scheduled anyway. It doesn't matter that the amount to be billed might change between the bill being issued and the payment due date, which is weeks away. I should also note here that Discover's program of Freeze does not apply to certain classes of payments, including automatic charges from utilities (exactly like TWC). I would have liked to prevent TWC from making any further charges to my payment card, but when I asked Discover about this, their answer was that indeed Freeze would not apply to TWC, and charges cannot be disputed before they actually post. You'd hope, wish, and think that the amount charged on payment due time would be at most any difference between any payments already collected and the amount of the previous bill. Sorry, there will be no such luck.
When the original payment date arrived, I logged onto my Discover account and noticed a pending charge for the full month amount. So I called up TWC and asked, what gives? Oh, yes, we see your payment for the partial month here, and we also see that the automatic payment went through, we'll just refund that full month's amount to your payment card. Keep in mind TNSTAAFL, because toll-free number (TFN) per-minute charges had to apply, maybe as well as telecomm charges on top of that (depending on how their toll-free routing is accomplished and their deal with their telecomms provider). Plus the agent to whom I spoke had to be paid, the workstation he used had to be paid for somehow, the network to which that workstation was attached had to be paid for, the IVR which routed my call had to have been purchased and maintained, and so on. Even less insignificant than these minor overheads, I happen to know from working for a small business which accepts credit cards that there is a per transaction charge plus a percentage of the purchase fee for accepting a credit card payment, typically around $0.30 plus 2%. So that means that would be at LEAST O($0.60), $0.30 for my manual partial month payment and another $0.30 for their erroneous charge. I honestly don't how much the transaction charge is for refunds.
Well enough; a few days later, I saw both the charge and the refund were finalized on my payment card account. But that wasn't the end of it. No, that'd be too logical.
About two weeks before posting this, TWC sent a bill by mail. It turns out, bottom line, the chat rep. got the amount total wrong, I still owed another $1.04. Despite not being a customer anymore, they also sent along a pamphlet about their billing practices, "instructions on how to use your TV service," pricing, digital cable, remote controls, parental controls, and on and on. Of course, they also sent an envelope, assuming I'd be writing a check and mailing it back. Of course, again, TNSTAAFL. They printed two "letter" size pages, one with the relevant billing information, another with ads for even more expensive services (O($90)/mo.), plus the mentioned pamphlet, the return payment envelope, had to mail it in an envelope for the whole thing, and paid postage to send it to me.
So...I went to log onto TWC's site again to make my $1.04 payment, of which I knew a significant fraction would be eaten up by transaction fees. So be it; I don't want to deny them payment, even of that little amount, and I wouldn't want the potential damage to my credit rating. Except...that was not possible, because TWC decided to shut down Web access. So what's a debtor to do? Why, make a call to a TFN to authorize another charge card payment of course. I realize at this point I could have written a check and mailed it to them, which perhaps wouldn't cost them as much. However, I have scant few of those left, and I don't want to "waste" them when other payment methods are available, lest I have to spend several dollars to have many checks printed which I'll not likely use (especially consdiering they should be printed as Key checks instead of First Niagara checks). So there's more per-minute TFN charges, more agent time, and so on, plus another payment card transaction fee of course. This is quickly eating way into the $1.04 that I owed. And then you'd think that'd be the end of it, but I'd be wrong again.
A few of days ago, TWC sent another bill, this time showing $1.04 as a previous balance, but nothing in payments, for a bottom line of a $1.04 balance. And once again, a return envelope was included, another one page ad for services, yet one more copy of the policies pamphlet, an outer envelope, postage to send it, just ridiculous in the grand scheme of things. Noticing that there was no credit for my payment of a few days prior, I called them again to ask, what gives? So of course, there had to be more TFN charges, more agent time, and so on. The CSR assured me the second bill had been printed and mailed just prior to me arranging my payment, and indeed my balance was zero.
I have little doubt that at least one more mailing will be made showing that my $1.04 has been paid, my balance is zero, another sheet telling me that I can have a great triple play experience for four or more times what I was paying (despite telling them on the disconnect call what I was paying was too much); who knows, probably another policies pamphlet, and perhaps even though I don't owe any money, another envelope to return my (non)payment.
Let's take a moment to compare and contrast this with the IRS and some other companies, namely USA Datanet, Ink Bird, Monoprice, and Newegg.
First, I'll say I can't believe I'm using this as an example of being better, but the IRS is more sensible, at least in one, limited way. I know it's just outside the stated limit, but if you owe less than a dollar to the IRS, the feds are willing to call it even, and you don't have to pay.
I used to be a customer of USA Datanet. Their business model was charging a cheaper per-minute rate for long distance calls, only $0.10/minute, and transporting the calls with VOIP instead of conventional telephony infrastructure. Plus as a promotion to join, they gave away 60 minutes "free." The trouble is, I'm pretty sure they never profited from having me as a customer. I live in New York and have siblings in Texas, but at the time, we were both people kind of busy with our jobs, so we would rarely talk. For my first call, I think I talked for about 70 minutes. So that was my free 60 minutes, plus a dollar. They sent a bill, as you can imagine including an envelope for returning a payment, but I paid with a MasterCard. So that was a printed page, plus the cost of two envelopes, plus postage, plus the transaction charges for MasterCard. The next month, they sent another statement, showing that I owed a dollar and paid a dollar, so that was a printed page, an envelope, and postage. Over the next couple of years, every few months I would make a call and leave a message on her answering machine (this was well before voicemail), which would generally only be a minute or less. So I would be sent a bill for $0.10 or $0.20, I would pay with MasterCard over the Web, and the next month get a statement saying I paid what I owed. This was stupidity similar to TWC; spend more to bill and send statements than there was revenue.
What I fail to understand is that USA Datanet called me up one day wondering why I had not used their service in many months. After all, the way you would make calls is to call a PoP and enter in your account codes, and then the desired destination. Therefore they could tell when I would call, but more importantly. there was no obligation to use their service because it was not billed from a specific line/phone number. I just told them there was no reason, I would still use their service if I really needed to talk to my sister, just that I hadn't in quite a while. You would have thought they would have done some analysis deeper than "Joe hasn't used our service in a while" and figured out this is not exactly a profitable customer to have. Or...I suppose they were trying to entice me to make some longer calls, like the first longer-than-an-hour one that I made, so I would be a profitable customer.
We now see the result. USA Datanet is out of business. Maybe their business model just doesn't have the appeal that it once did, or maybe they spent too much on providing service for more than the revenues they were receiving.
A while ago I got annoyed at the unreliability of the mechanical thermostat in my main floor refrigerator (I have another in my basement). So through Amazon I bought an Ink Bird electronic thermostat. About 6-8 months in, the thermostat was working OK, but it would not accept any commands to change or display anything. It is stuck at 1.9° ± 2°. When it powers up, it shows "1.8" which I can only assume is some sort of diagnostic code. I wrote an email using Ink Bird's Web site asking if there was anything I could do, such as reflowing solder. Even though they are based in China, I'm going to guess they were reluctant to tell me to do much with this thing, figuring I'd do something to it which would cause me to be shocked. I also sent them a very short video of their unit being plugged in, and the "1.8." Eventually they asked for my Amazon order number, and sent another thermostat. I explicitly asked if I would have to return the defective one, and they said no. They realized that it would cost O($8-$10) to ship it back, which is about half the cost of a new one. They just figured this as a cost of doing business. The thing is, I bought two more, figuring I eventually want to replace the basement's mechanical thermostat, as well as having a fully working one for the main floor. Plus I have the one they sent as a spare, should one of the two active ones fail. So for the price of 3, I basically have one for free. And the one still works, it's just not adjustable, and is set for an awfully chilly temp.
A couple of months ago, I was frustrated at not being able to connect up all my audio gear simultaneously, always wanting for this cable or that cable, so I ordered a bunch from Monoprice ("MP"). The pickers must have been having a bad day because instead of getting the 5 Y cables of 3.5mm plug to left and right RCA jacks, I got 5 cables of RCA plug to two RCA jacks. (In warehouse terms, there is "pick," where items are taken off shelves or similar, "pack" which is readying them in packages in preparation for shipping, and "ship." A lot of operations have people do the picking, hence "pickers.") Both varieties of cable retail, from them, for O($0.80) each. I contacted MP, they redid that part of the order, and they said just dispose of, or use, the erroneously sent cables. They realized that the lost value was O($5), and that it'd be around the same dollar amount for me to ship the erroneous cables back. They're cognizant of the price of good customer relations and the economics of the goof.
Finally, I will relate a tale of Newegg. I ordered a number of items, one of which was a 3 m HDMI cable and an HDMI to DVI-D adapter, sold as a single item. When the combination arrived, it appeared as if cable had been picked but not the adapter. I will admit, I made a goof; as I opened the bubble pack they came in, the adapter fell out and out of sight. What Newegg ended up doing was refunding that item of the order, and telling me to use the money towards the purchase of the "missing" adapter. When I realized my error a few days later, and found the adapter, I called them up again, asking them to charge me for the combination item. They said something like, tell you what, for this one we appreciate your honesty, but we're just going to call it even. This really doesn't "hurt" us. Part of the issue was that Newegg has a strict policy that it does not take any orders over the phone. I'm sure they have experience where this is their most cost-effective way of doing business, either because they've gotten burned by phone orders before, or their considered analysis is that maintaining Web servers and networks is far cheaper than agent time. While in the short term this was a "minor" loss for them, I subsequently made more purchases from them instead of Amazon specifically because of their willingness to give me the benefit of the doubt on that one set of orders.
I can only guess at the economics of of the IRS and their banking services. I will guess that handling amounts of only a dollar might cost them more in processing fees or labor (especially if I were to mail in a check) than they would gain.
I can understand MP being wise and realizing the perspective of most customers, that most customers wouldn't want to pay the value of returned items in shipping alone for something they goofed in doing. On the other hand, they trusted me an awful lot that I was telling the truth and not trying to scam them out of goods. At least I'm reasonably sure since they ship in bulk, they do not pay the same rates as a "retail" shipping customer like me. Still...there was an extra small box, more pickers' and packers' time, and so on. All I can hope is there was enough margin in the other items I bought that day that at least they broke even, or came close. What made me a little nervous about the long-term viability of that company is that it took over a week for the "RMA" to go through, and when I asked about it, they said their RMA people were backed up more than usual.
Ink Bird likewise had a lot of faith in me (I could have sent them a doctored video showing them the "bootup" of their thermostat and the brief showing of "1.8"), and likewise thought they were willing to take a similar hit in a second thermostat, a second box, more shipping, etc. for improved reputation. I don't know what the warranty is supposed to be on those ITC-1000 thermostats, but somehow I don't think I was in the warranty period.
But sometimes businesses seem really stupid, as I've related with USA Datanet and Time Warner Cable (TWC).
Direct all comments to Google+, preferably under the post about this blog entry.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
Please join one of the fastest growing social networks, Google+!
I don't know, maybe there is some law which says billing has to be like this, but I'm dumbfounded by my recent experience with Time Warner Cable (can we call them Charter or Spectrum? or not yet?).
I just couldn't justify paying O($25) per month for a service of which I rarely partake. (I'll use this quasi- "big O notation" throughout this entry to mean "on the order of." It has a similar but not quite the same meaning in mathematics and computer science.) I find a lot of entertainment in YouTube videos for example, and something like Netflix, Vudu, Amazon Video/Amazon Prime, Hulu, etc. would be half the cost or less. So I used the expiration of my SchedulesDirect subscription as a prompt to tell them, "sorry, disconnect me, please." They even offered to knock around $10/mo. off, but still, I don't watch that much on there. After all, how many times can you watch the same episodes of M*A*S*H on WBBZ? I must have more than 100 hours of recordings waiting to be watched on my MythTV.
A few days after I called in the disconnect order, I realized I never asked the CSR what my final bill should be, since it would be prorated. It's also worthy to note that billing is explicitly ahead of time; that is to say, your payment is for the month to come, not for the month that ends on your billing date. I hopped on their Web chat, and they gave me a figure for the partial month's service. I thought I had disabled automatic payment, but they advised me not to pay anything. However, I pointed out to them if I didn't pay for my partial month, and just wait for another bill, I'd be paying late, which I would rather not do for ethical reasons, plus a possible hit to my credit rating. So I logged into TWC's site, scheduled a payment for that reduced amount, and thought that would set things square. But no, that was wrong of me to think that.
For starters, there is ridiculous lag with those folks. It turns out if you want to cancel automatic payments, you should do that before your bill arrives, otherwise an automatic payment will be scheduled anyway. It doesn't matter that the amount to be billed might change between the bill being issued and the payment due date, which is weeks away. I should also note here that Discover's program of Freeze does not apply to certain classes of payments, including automatic charges from utilities (exactly like TWC). I would have liked to prevent TWC from making any further charges to my payment card, but when I asked Discover about this, their answer was that indeed Freeze would not apply to TWC, and charges cannot be disputed before they actually post. You'd hope, wish, and think that the amount charged on payment due time would be at most any difference between any payments already collected and the amount of the previous bill. Sorry, there will be no such luck.
When the original payment date arrived, I logged onto my Discover account and noticed a pending charge for the full month amount. So I called up TWC and asked, what gives? Oh, yes, we see your payment for the partial month here, and we also see that the automatic payment went through, we'll just refund that full month's amount to your payment card. Keep in mind TNSTAAFL, because toll-free number (TFN) per-minute charges had to apply, maybe as well as telecomm charges on top of that (depending on how their toll-free routing is accomplished and their deal with their telecomms provider). Plus the agent to whom I spoke had to be paid, the workstation he used had to be paid for somehow, the network to which that workstation was attached had to be paid for, the IVR which routed my call had to have been purchased and maintained, and so on. Even less insignificant than these minor overheads, I happen to know from working for a small business which accepts credit cards that there is a per transaction charge plus a percentage of the purchase fee for accepting a credit card payment, typically around $0.30 plus 2%. So that means that would be at LEAST O($0.60), $0.30 for my manual partial month payment and another $0.30 for their erroneous charge. I honestly don't how much the transaction charge is for refunds.
An aside: I used to get my hair cut at Fantastic Sam's all the time. I thoroughly realized why they had a sign that read something to the effect of, "minimum credit card payment is $10." Their analysis likely figured out that paying the average shop rent, their stylists, etc. had not much margin, and that the credit card processing costs would cut significantly into that margin, if you'll pardon the pun. As I recall, haircuts were $7 at the time, so no, I couldn't use a card, had to pay cash.
Well enough; a few days later, I saw both the charge and the refund were finalized on my payment card account. But that wasn't the end of it. No, that'd be too logical.
About two weeks before posting this, TWC sent a bill by mail. It turns out, bottom line, the chat rep. got the amount total wrong, I still owed another $1.04. Despite not being a customer anymore, they also sent along a pamphlet about their billing practices, "instructions on how to use your TV service," pricing, digital cable, remote controls, parental controls, and on and on. Of course, they also sent an envelope, assuming I'd be writing a check and mailing it back. Of course, again, TNSTAAFL. They printed two "letter" size pages, one with the relevant billing information, another with ads for even more expensive services (O($90)/mo.), plus the mentioned pamphlet, the return payment envelope, had to mail it in an envelope for the whole thing, and paid postage to send it to me.
So...I went to log onto TWC's site again to make my $1.04 payment, of which I knew a significant fraction would be eaten up by transaction fees. So be it; I don't want to deny them payment, even of that little amount, and I wouldn't want the potential damage to my credit rating. Except...that was not possible, because TWC decided to shut down Web access. So what's a debtor to do? Why, make a call to a TFN to authorize another charge card payment of course. I realize at this point I could have written a check and mailed it to them, which perhaps wouldn't cost them as much. However, I have scant few of those left, and I don't want to "waste" them when other payment methods are available, lest I have to spend several dollars to have many checks printed which I'll not likely use (especially consdiering they should be printed as Key checks instead of First Niagara checks). So there's more per-minute TFN charges, more agent time, and so on, plus another payment card transaction fee of course. This is quickly eating way into the $1.04 that I owed. And then you'd think that'd be the end of it, but I'd be wrong again.
A few of days ago, TWC sent another bill, this time showing $1.04 as a previous balance, but nothing in payments, for a bottom line of a $1.04 balance. And once again, a return envelope was included, another one page ad for services, yet one more copy of the policies pamphlet, an outer envelope, postage to send it, just ridiculous in the grand scheme of things. Noticing that there was no credit for my payment of a few days prior, I called them again to ask, what gives? So of course, there had to be more TFN charges, more agent time, and so on. The CSR assured me the second bill had been printed and mailed just prior to me arranging my payment, and indeed my balance was zero.
I have little doubt that at least one more mailing will be made showing that my $1.04 has been paid, my balance is zero, another sheet telling me that I can have a great triple play experience for four or more times what I was paying (despite telling them on the disconnect call what I was paying was too much); who knows, probably another policies pamphlet, and perhaps even though I don't owe any money, another envelope to return my (non)payment.
Let's take a moment to compare and contrast this with the IRS and some other companies, namely USA Datanet, Ink Bird, Monoprice, and Newegg.
First, I'll say I can't believe I'm using this as an example of being better, but the IRS is more sensible, at least in one, limited way. I know it's just outside the stated limit, but if you owe less than a dollar to the IRS, the feds are willing to call it even, and you don't have to pay.
I used to be a customer of USA Datanet. Their business model was charging a cheaper per-minute rate for long distance calls, only $0.10/minute, and transporting the calls with VOIP instead of conventional telephony infrastructure. Plus as a promotion to join, they gave away 60 minutes "free." The trouble is, I'm pretty sure they never profited from having me as a customer. I live in New York and have siblings in Texas, but at the time, we were both people kind of busy with our jobs, so we would rarely talk. For my first call, I think I talked for about 70 minutes. So that was my free 60 minutes, plus a dollar. They sent a bill, as you can imagine including an envelope for returning a payment, but I paid with a MasterCard. So that was a printed page, plus the cost of two envelopes, plus postage, plus the transaction charges for MasterCard. The next month, they sent another statement, showing that I owed a dollar and paid a dollar, so that was a printed page, an envelope, and postage. Over the next couple of years, every few months I would make a call and leave a message on her answering machine (this was well before voicemail), which would generally only be a minute or less. So I would be sent a bill for $0.10 or $0.20, I would pay with MasterCard over the Web, and the next month get a statement saying I paid what I owed. This was stupidity similar to TWC; spend more to bill and send statements than there was revenue.
What I fail to understand is that USA Datanet called me up one day wondering why I had not used their service in many months. After all, the way you would make calls is to call a PoP and enter in your account codes, and then the desired destination. Therefore they could tell when I would call, but more importantly. there was no obligation to use their service because it was not billed from a specific line/phone number. I just told them there was no reason, I would still use their service if I really needed to talk to my sister, just that I hadn't in quite a while. You would have thought they would have done some analysis deeper than "Joe hasn't used our service in a while" and figured out this is not exactly a profitable customer to have. Or...I suppose they were trying to entice me to make some longer calls, like the first longer-than-an-hour one that I made, so I would be a profitable customer.
We now see the result. USA Datanet is out of business. Maybe their business model just doesn't have the appeal that it once did, or maybe they spent too much on providing service for more than the revenues they were receiving.
A while ago I got annoyed at the unreliability of the mechanical thermostat in my main floor refrigerator (I have another in my basement). So through Amazon I bought an Ink Bird electronic thermostat. About 6-8 months in, the thermostat was working OK, but it would not accept any commands to change or display anything. It is stuck at 1.9° ± 2°. When it powers up, it shows "1.8" which I can only assume is some sort of diagnostic code. I wrote an email using Ink Bird's Web site asking if there was anything I could do, such as reflowing solder. Even though they are based in China, I'm going to guess they were reluctant to tell me to do much with this thing, figuring I'd do something to it which would cause me to be shocked. I also sent them a very short video of their unit being plugged in, and the "1.8." Eventually they asked for my Amazon order number, and sent another thermostat. I explicitly asked if I would have to return the defective one, and they said no. They realized that it would cost O($8-$10) to ship it back, which is about half the cost of a new one. They just figured this as a cost of doing business. The thing is, I bought two more, figuring I eventually want to replace the basement's mechanical thermostat, as well as having a fully working one for the main floor. Plus I have the one they sent as a spare, should one of the two active ones fail. So for the price of 3, I basically have one for free. And the one still works, it's just not adjustable, and is set for an awfully chilly temp.
A couple of months ago, I was frustrated at not being able to connect up all my audio gear simultaneously, always wanting for this cable or that cable, so I ordered a bunch from Monoprice ("MP"). The pickers must have been having a bad day because instead of getting the 5 Y cables of 3.5mm plug to left and right RCA jacks, I got 5 cables of RCA plug to two RCA jacks. (In warehouse terms, there is "pick," where items are taken off shelves or similar, "pack" which is readying them in packages in preparation for shipping, and "ship." A lot of operations have people do the picking, hence "pickers.") Both varieties of cable retail, from them, for O($0.80) each. I contacted MP, they redid that part of the order, and they said just dispose of, or use, the erroneously sent cables. They realized that the lost value was O($5), and that it'd be around the same dollar amount for me to ship the erroneous cables back. They're cognizant of the price of good customer relations and the economics of the goof.
Finally, I will relate a tale of Newegg. I ordered a number of items, one of which was a 3 m HDMI cable and an HDMI to DVI-D adapter, sold as a single item. When the combination arrived, it appeared as if cable had been picked but not the adapter. I will admit, I made a goof; as I opened the bubble pack they came in, the adapter fell out and out of sight. What Newegg ended up doing was refunding that item of the order, and telling me to use the money towards the purchase of the "missing" adapter. When I realized my error a few days later, and found the adapter, I called them up again, asking them to charge me for the combination item. They said something like, tell you what, for this one we appreciate your honesty, but we're just going to call it even. This really doesn't "hurt" us. Part of the issue was that Newegg has a strict policy that it does not take any orders over the phone. I'm sure they have experience where this is their most cost-effective way of doing business, either because they've gotten burned by phone orders before, or their considered analysis is that maintaining Web servers and networks is far cheaper than agent time. While in the short term this was a "minor" loss for them, I subsequently made more purchases from them instead of Amazon specifically because of their willingness to give me the benefit of the doubt on that one set of orders.
I can only guess at the economics of of the IRS and their banking services. I will guess that handling amounts of only a dollar might cost them more in processing fees or labor (especially if I were to mail in a check) than they would gain.
I can understand MP being wise and realizing the perspective of most customers, that most customers wouldn't want to pay the value of returned items in shipping alone for something they goofed in doing. On the other hand, they trusted me an awful lot that I was telling the truth and not trying to scam them out of goods. At least I'm reasonably sure since they ship in bulk, they do not pay the same rates as a "retail" shipping customer like me. Still...there was an extra small box, more pickers' and packers' time, and so on. All I can hope is there was enough margin in the other items I bought that day that at least they broke even, or came close. What made me a little nervous about the long-term viability of that company is that it took over a week for the "RMA" to go through, and when I asked about it, they said their RMA people were backed up more than usual.
Ink Bird likewise had a lot of faith in me (I could have sent them a doctored video showing them the "bootup" of their thermostat and the brief showing of "1.8"), and likewise thought they were willing to take a similar hit in a second thermostat, a second box, more shipping, etc. for improved reputation. I don't know what the warranty is supposed to be on those ITC-1000 thermostats, but somehow I don't think I was in the warranty period.
But sometimes businesses seem really stupid, as I've related with USA Datanet and Time Warner Cable (TWC).
Direct all comments to Google+, preferably under the post about this blog entry.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
Please join one of the fastest growing social networks, Google+!
22 July, 2016
An Annoying Week With Allwinner
Many months ago, I got an Allwinner A33 Q8H tablet, a BrightTab, from Ollie's Bargain Outlet. I do not know why, but one day a couple of months ago, it would not authenticate with Google no matter what I did (wipe all the tablet's data, etc.). I'm guessing the firmware that was on it got either obsoleted or blacklisted (it would fail more or less immediately).
Earlier this week, since my Nexus 7 has developed distorted audio, I decided that I'm going to try to revive the Google functionality on this BrightTab since it's not particularly useful to me w/o Google cloud services. In an attempt to correct this authentication failure, I copied some .apk and .odex files over from my Nexus 7. Except I had one fatal procedural flaw: I did not back up one of the BrightTab's original .apk's in /system/priv-app . And since the ones for the Nex7 did not work at all with this Android, it got into a bootloop, basically an unrecoverable one (since I did not have any of the original firmware anywhere), with essential processes dying over and over again.
Unfortunately, I was in for a rude awakening about Allwinner. They seem to have implemented fastboot, but ultimately, it's not all that useful. Most Androids, you can take partition files and flash them from fastboot. Buuuuuuut.....Allwinner decided to go their own way, and create a whole different way of packaging firmware, one giant image lump (which apparently contains boot0, uboot, the bootloader, recovery, and system, judging from the messages which their flashing tool puts out on the stdout/stderr).
The first hurdle is, of course, that the majority of readily extant tools is written to be run on Microsoft (ugh). Not knowing what effect these things would have on WinXP, I decided to run my WinXP VirtualBox, making a snapshot before installing anything "foreign." This was good for about a day of hacking around, because unfortunately, this goofball of a tablet is not recognized by USB drivers, even ones that are supposed to be "universal." Some of the Allwinner-specific tools even have drivers packaged with them, but this VM didn't seem to recognize the tablet when physically then logically connected (would always pop up the "new hardware found, please install a driver for it" dialogs). In retrospect, one of the problems was that the tablet wasn't even being put into the correct mode (I guess similar to Samsung's download mode), possibly due to me not knowing how to do that/poor documentation. It seemed like the instructions were wanting to do their deeds through ADB, which I knew was active because i could see the processes which were dying with logcat. Eventually I came across "PhoenixCard," which is a utility to prepare microSD cards for firmware flashing.
Cleverly, Allwinner does seem to have some sort of provision for booting from microSD cards, a feature lacking on a lot of other tablets. That's particularly handy when you're in a boot loop and all the firmware writing tools you can find can't seem to find your tablet on USB. But it is a really tedious way to go through several iterations of trying things.
The second hurdle was that BrightTab does not publish their firmware anywhere. At least for the Pandigital I had, a .zip file which I could flash from recovery was available for download from Pandigital's site, so that any time my tablet's Android got corrupted (and it did, many times, before it died for good) I could reboot into recovery and reflash the stock firmware. Also, the great pleasure of Nexus devices is that the factory images are readily available from Google's site. There are several Allwinner boards with different ARM CPUs each, each with dozens and dozens of firmware variations for each processor (in my case, an A33). Thankfully, Google searching eventually led me to a blogger who has collected links to a lot of them. These pages may be in Spanish, but heck, with my 4 years of high school classes I knew most of the words used, and Google Translate helped with the rest.
The third hurdle was, given these dozens of firmware files, which one would work for my particular tablet? The first blob I downloaded actually was pointed to by XDA-Developers, a trusted Android site. I stuck it on a microSD card with PhoenixCard, it was accepted and flashed by the tablet, but after booting, the touch coordinates were way off, and I could not even activate the "continue" button after sliding the chooser to US English. So, I started going down the huge-ish list of links to ROMs, looking for ones which indicated the proper size (in my case, 800x480).
The next breakthrough was finding Linux tools on Allwinner's wiki and Git repository. This is in the form of source code for a USB kernel module and their flashing utility, LiveSuit. (Seriously...it's "LiveSuit," like clothing. It's not missing an "e;" I would have called it "LiveSuite," but apparently Allwinner wouldn't.) And the breakthrough right after that was in the very explicit on-screen instructions for LiveSuit: you have to disconnect and turn off your tablet, hold a key other than power as you plug USB in, and then press power many times (they suggest 10,, but it seems the tablet may come alive to LiveSuit in fewer than 10 presses). This apparently wakes up the boot code in the tablet to go into a special mode to accept ROMs. Now that I think of it, it's something like Apple's DFU mode, except the screen stays off. Yay! There was no more need to keep Windows XP in VirtualBox booted!
Another hurdle to endure was that a lot of these blobs are hosted on freemium download hosting sites. Sure, give them your credit card number, and for varying amounts for various terms (one month, six months, a year, etc.) you can log in and have "gold member" access, which amounts to faster data rates and no wait for the link to appear. Me being the cheap (and currently unemployed) type, I had to endure one minute countdowns, CAPTCHAs, and approximately DS-1 rates (roughly 180KBytes/sec). Since these files are considerable (maybe at least 300M), they took plenty of time to come across, and the site had stream counting too (so you couldn't download more than one file at a time...unless of course you are a gold member).
At the point of getting native Linux flashing working, I had tried a couple of ROMs using PhoenixCard. They ranged from not flashing (apparently a firmware image file either inconpatible with PhoenixCard or incompatible with my tablet) to flashing OK but being unusable. But then, with the Linux kernel module loaded and better instructions on how to prepare the tablet for another ROM image download, also without needing to go through VirtualBox, the pace of trials picked up considerably.
One of the ROMs looked (and sounded) while booting almost exactly how it did after I got it from Ollie's...except the touchscreen didn't work. bummer. A couple of tries later, it booted OK, the touchscreen worked, but it was all in (as best I could tell) Arabic. Actually, I found a YouTube video which explains the settings entries to look for to change any Android's language, and managed to switch it to English. But then I thought...any time you go into recovery and wipe all data, you're going to have to muddle through changing to English. So although the ROM seemed to be suitable, I plodded on to more ROMs.
Finally, I found an image that, when loaded and booted, it behaves much like the original firmware, except of course the Google stuff works. I also spent about half the day today perfecting the tablet, such as installing SuperSU, copying over some of the shell environment from my Nexus 7, installing some applications and generally customizing it.
As usual though, it wasn't necessarily all the annoyances of getting where I want to be, it was somewhat the fun of the challenge of the hacks needed to get there. So it wasn't the adventure of the destination, but the adventure of how to get to the destination.
Direct all comments to Google+, preferably under the post about this blog entry.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
Please join one of the fastest growing social networks, Google+!
Earlier this week, since my Nexus 7 has developed distorted audio, I decided that I'm going to try to revive the Google functionality on this BrightTab since it's not particularly useful to me w/o Google cloud services. In an attempt to correct this authentication failure, I copied some .apk and .odex files over from my Nexus 7. Except I had one fatal procedural flaw: I did not back up one of the BrightTab's original .apk's in /system/priv-app . And since the ones for the Nex7 did not work at all with this Android, it got into a bootloop, basically an unrecoverable one (since I did not have any of the original firmware anywhere), with essential processes dying over and over again.
Unfortunately, I was in for a rude awakening about Allwinner. They seem to have implemented fastboot, but ultimately, it's not all that useful. Most Androids, you can take partition files and flash them from fastboot. Buuuuuuut.....Allwinner decided to go their own way, and create a whole different way of packaging firmware, one giant image lump (which apparently contains boot0, uboot, the bootloader, recovery, and system, judging from the messages which their flashing tool puts out on the stdout/stderr).
The first hurdle is, of course, that the majority of readily extant tools is written to be run on Microsoft (ugh). Not knowing what effect these things would have on WinXP, I decided to run my WinXP VirtualBox, making a snapshot before installing anything "foreign." This was good for about a day of hacking around, because unfortunately, this goofball of a tablet is not recognized by USB drivers, even ones that are supposed to be "universal." Some of the Allwinner-specific tools even have drivers packaged with them, but this VM didn't seem to recognize the tablet when physically then logically connected (would always pop up the "new hardware found, please install a driver for it" dialogs). In retrospect, one of the problems was that the tablet wasn't even being put into the correct mode (I guess similar to Samsung's download mode), possibly due to me not knowing how to do that/poor documentation. It seemed like the instructions were wanting to do their deeds through ADB, which I knew was active because i could see the processes which were dying with logcat. Eventually I came across "PhoenixCard," which is a utility to prepare microSD cards for firmware flashing.
Cleverly, Allwinner does seem to have some sort of provision for booting from microSD cards, a feature lacking on a lot of other tablets. That's particularly handy when you're in a boot loop and all the firmware writing tools you can find can't seem to find your tablet on USB. But it is a really tedious way to go through several iterations of trying things.
The second hurdle was that BrightTab does not publish their firmware anywhere. At least for the Pandigital I had, a .zip file which I could flash from recovery was available for download from Pandigital's site, so that any time my tablet's Android got corrupted (and it did, many times, before it died for good) I could reboot into recovery and reflash the stock firmware. Also, the great pleasure of Nexus devices is that the factory images are readily available from Google's site. There are several Allwinner boards with different ARM CPUs each, each with dozens and dozens of firmware variations for each processor (in my case, an A33). Thankfully, Google searching eventually led me to a blogger who has collected links to a lot of them. These pages may be in Spanish, but heck, with my 4 years of high school classes I knew most of the words used, and Google Translate helped with the rest.
The third hurdle was, given these dozens of firmware files, which one would work for my particular tablet? The first blob I downloaded actually was pointed to by XDA-Developers, a trusted Android site. I stuck it on a microSD card with PhoenixCard, it was accepted and flashed by the tablet, but after booting, the touch coordinates were way off, and I could not even activate the "continue" button after sliding the chooser to US English. So, I started going down the huge-ish list of links to ROMs, looking for ones which indicated the proper size (in my case, 800x480).
The next breakthrough was finding Linux tools on Allwinner's wiki and Git repository. This is in the form of source code for a USB kernel module and their flashing utility, LiveSuit. (Seriously...it's "LiveSuit," like clothing. It's not missing an "e;" I would have called it "LiveSuite," but apparently Allwinner wouldn't.) And the breakthrough right after that was in the very explicit on-screen instructions for LiveSuit: you have to disconnect and turn off your tablet, hold a key other than power as you plug USB in, and then press power many times (they suggest 10,, but it seems the tablet may come alive to LiveSuit in fewer than 10 presses). This apparently wakes up the boot code in the tablet to go into a special mode to accept ROMs. Now that I think of it, it's something like Apple's DFU mode, except the screen stays off. Yay! There was no more need to keep Windows XP in VirtualBox booted!
Another hurdle to endure was that a lot of these blobs are hosted on freemium download hosting sites. Sure, give them your credit card number, and for varying amounts for various terms (one month, six months, a year, etc.) you can log in and have "gold member" access, which amounts to faster data rates and no wait for the link to appear. Me being the cheap (and currently unemployed) type, I had to endure one minute countdowns, CAPTCHAs, and approximately DS-1 rates (roughly 180KBytes/sec). Since these files are considerable (maybe at least 300M), they took plenty of time to come across, and the site had stream counting too (so you couldn't download more than one file at a time...unless of course you are a gold member).
At the point of getting native Linux flashing working, I had tried a couple of ROMs using PhoenixCard. They ranged from not flashing (apparently a firmware image file either inconpatible with PhoenixCard or incompatible with my tablet) to flashing OK but being unusable. But then, with the Linux kernel module loaded and better instructions on how to prepare the tablet for another ROM image download, also without needing to go through VirtualBox, the pace of trials picked up considerably.
One of the ROMs looked (and sounded) while booting almost exactly how it did after I got it from Ollie's...except the touchscreen didn't work. bummer. A couple of tries later, it booted OK, the touchscreen worked, but it was all in (as best I could tell) Arabic. Actually, I found a YouTube video which explains the settings entries to look for to change any Android's language, and managed to switch it to English. But then I thought...any time you go into recovery and wipe all data, you're going to have to muddle through changing to English. So although the ROM seemed to be suitable, I plodded on to more ROMs.
Finally, I found an image that, when loaded and booted, it behaves much like the original firmware, except of course the Google stuff works. I also spent about half the day today perfecting the tablet, such as installing SuperSU, copying over some of the shell environment from my Nexus 7, installing some applications and generally customizing it.
As usual though, it wasn't necessarily all the annoyances of getting where I want to be, it was somewhat the fun of the challenge of the hacks needed to get there. So it wasn't the adventure of the destination, but the adventure of how to get to the destination.
Direct all comments to Google+, preferably under the post about this blog entry.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
Please join one of the fastest growing social networks, Google+!
16 November, 2015
A DDoS Attack Can Really Ruin Your Day
I hope I don't have to go through too many more days like this past Saturday, 14-Nov.
The insidius thing about DDoS attacks is there is little that can be
done about them, except a few things. Ordinarily (the DoS attack) you
identify a host or netblock which is sending the nuisance traffic and
you add some rules in your router to drop those packets. But add that
extra "D", and it's like being surrounded by a swarm of bees, you can't
possibly swat them all dead.
The few things which can be done (which I know about) are:
The first thing I did of course was log onto the (Linux) router and do a tcpdump of the "WIC" (wide area network (WAN) interface card), and discovered a lot of TCP/25 traffic. So I pulled up the maillog, and discovered a lot of instances of "host such-and-such did not issue MAIL/EXPN/VRFY/ETRN" and messages about refusing connections because the configured number of (MTA) children has been reached (happens to be 7 in my case). By far my biggest concern is the attackers will be sending something which trips up my MTA and makes it spam other people, possibly getting me knocked off my ISP. So I did what I usually do, look at a representative sample of recent maillog entries and add some iptables(8) rules and DROP traffic coming from these hosts (or maybe netblocks) which are causing the "did not issue MAIL/EXPN/VRFY/ETRN" entries to be generated (basically, connecting and doing a whole lot of nothing, just tying up a socket and pushing me over my child process limit).
There came a point where I realized I needed more than just a line or three of rules, so I decided to grep(1) for "did not issue" in the logs, take the latest 150 such entries with tail(1), extract the IP addresses with sed(1), and use sort(1) to get a unique set of them. As I recall, I came up with 30 or so addresses, and used a for loop to append DROPs of them to the PREROUTING chain in the mangle table. (The idea is to cut off traffic as soon as the router takes packets in, so a PREROUTING chain.) Unfortunately, it became apparent that a handful of addresses wasn't going to be effective, because the log messages kept on a-comin'. So I decided maybe a little PTR record and whois(1) investigation (to block entire netblocks instead of individual addresses) was in order. A disturbing trend started to emerge. The IP addresses were generally purportedly to be Russian and other Eastern Bloc countries, who, at least to me, are notorious for being the origin of a lot of DDoS attacks and spam.
I really did not want to shut off services entirely, but I saw little choice. I did not want my MTA compromised to be yet another source of spam. I put in an iptables(8) rule which dropped all TCP/25 traffic arriving over the WIC. I noticed this did not stop the popping of message into maillog and realized the attackers were also attempting connections to TCP/587 and TCP/465, so I put in rules for those too. For the moment, IPv6 is, in a sense, only in its infancy (for being a twenty or so year-old infant!), so there was no apparent reason to add any ip6tables(8) rules. And in fact, email still flowed in over IPv6, most notably from Google (thanks for being a leader in the adoption of IPv6!).
It was at this point I was very glad that (according to Wikipedia) Dana Valerie Lank, D. Green, Paul Vixie, Meng Weng Wong, and so many others collaborated on initiating the SPF standard. I began to think, from whom was it particularly critical that I receive email? I could go to those domains' SPF records and insert ACCEPT rules for those addresses or netblocks. Google was already "covered," as noted above about IPv6. Oddly enough, the first domain I thought of was aol.com, because my volleyball team captain might want to tell me something about the soon-upcoming match on Monday. The SPF for them looked knarly, with some include: directives. I settled for looking at Received: headers in previous emails from Matt and identifying the netblock AOL had been using previously (turns out it could be summarized with a single /24). Next I thought of Discover, then of First Niagara, then Verizon (for them informing me of impending ejection from their network for violation of their ToS). I also thought that although it's not all that critical, I receive an awful lot of email from nyalert.gov, especially considering we had that extended rain and wind storm, and the extension of small craft advisories and so on. All in all, I made exceptions for a handful of mailers.
Then to evaluate the extent of the problem, I used watch(1) to list the mangle PREROUTING table every 60 seconds, to see the packet counts on the DROP rules. I'd say they averaged deltas of around 20, or one attempt every three seconds, and the peak I saw once was 51, or nearly an attempt per second. I know as DDoS attacks go, this is extremely mild. If it were a seriously large botnet controlled by a determined entity, they could likely saturate my 25 Mbit/s link, making doing anything Internet-related extremely challenging or impossible. Always in the back of my mind was that this was unsustainable in the long run, and hoped that the botnet was dumb enough to "think" that if not even ICMPs for connection refusal were coming back, that it would assume they in one way acheived their one possible objective, which was knocking my entire MTA host offline.
I then contemplated the work which would be involved in logging onto various Web sites (power, gas, DSLReports/BroadbandReports, First Niagara, Amazon, Newegg, and on and on) and updating my email address to be GMail. I also started trying to search for email hosting providers who would basically handle this entire variety of mess on my behalf, and forward email for me, whereby I could block all IP addresses except for those of said provider. Or maybe I could rejuvenate fetchmail(1) or similar to go get my email from them over IMAP and reinject it locally, as I used to do decades ago with my dialup ISP's email. To my amazement at the low prices, it looks as if, for example, ZoneEdit will handle email forwarding for only on the order of $1/month/domain (so $3 or maybe $4 in my case, because there is philipps.us, joe.philipps.us, philippsfamily.org, and philipps-family.org). This is in contrast (as far as I know) to Google's $5/month/user, and I have a ton of custom addresses (which might be separate "users"). (Basically, it's one of the perks of having your own domain and your own email server. Every single sender gets a separate address, and if they turn into a not-so-savory sender, their specific address ceases to work.) The search terms needed some tweaking because trying things like "mail exchangers" (thinking in terms of MX records) turned up lots of hits for Microsoft Exchange hosting.
A friend of mine runs a small Internet services business and has some Linux VPSes which I can leverage. He already lets me run DNS and a little "snorkel" where I can send email through a GRE tunnel, and not appear to be sending email from a "residential-class IP address." So I called him up and thankfully he answered right away. I got his permission to use one of the IPv4 addresses (normally used for Apache/mod_ssl) for inbound email, in an attempt to see if these attackers are more interested in my machine (the specific IP(v4) address) or my domain(s). If I add an additional address to accept email, and the attacks do not migrate to that address, I then know that it's far more likely that this botnet came across my address at random or by scanning the Verizon address space. So, I picked an address on one VPS, added a NAT rule to hit my end of the GRE tunnel, had to debug a routing table issue, redid the MTA configuration to listen on the tunnel's address, all-in all about an hour and a half's worth of work to implement and do preliminary testing.
It was at this time I realized in my watch window that the packet counts were no longer increasing, even over several samples (minutes). Even after I added the A record to supplement the one for my Verizon address, I noticed there was basically no activity in maillog. So as besst as I can tell, it was as I suspected, the botnet was really likely only interested in my specific address/host. And thankfully it "saw" the lack of any TCP response as an indicator that the host had gone offline, and to cease directing any resources to the attack on me. I hate to give these worms any ideas, but you could also try other things, like ICMP echo, to determine if that host is still alive. Then again, if your sole objective is compromising an MTA, maybe that doesn't matter.
Eventually, I inserted a rule to accept TCP/25 traffic again, thinking if attacks resumed, I could easily kick that rule out and spring the shutoff trap again. Or even better, I could replace it with a rule including the limit match, so only something like 10 or 15 connection attempts could be made per minute. At least the MTA would not be wasting time/CPU on these useless connections, and the kernel would not have to keep track of state beyond the token bucket filter. I almost hit the panic button when I saw some more "did not issue" messages, as well as a little later a notice of refusing connections because of child limit. But I reasoned, just wait a few minutes, and see if it's persistent. Howver, I had a lot of anxiety that it was not over yet, and that it was the domain and not the host the attackers wanted compromised, because some of that unwanted activity was through the newly set up IP address.
In retrospect, I question whether I want to continue doing this myself.
UPDATE Mon., 16-Nov: There are some more folks at it again. What I've decided to do for now is to put in some iptables(8) rules which use the limit match to allow only a certain number of connections per minute. So far this has been fairly effective at keeping the chaos to a minimum, but it's not ideal. I really think I'm going to have to look at an email hosting service, at least until I can put up a more robust email server, or maybe permanently.
Direct all comments to Google+, preferably under the post about this blog entry.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
Please join one of the fastest growing social networks, Google+!
The few things which can be done (which I know about) are:
- direct traffic somewhere else. There are companies which specialize in this sort of mitigation, and sometimes sink/absorb multiple gigabits per second of spurious traffic. Although I've never approached anyone who does this, I'm guessing that doesn't come cheap.
- insert some sort of rules to limit the rate at which these packets are handled. This still diminishes service, but hopefully it's not shut down completely. At least processing is occuring on the packet level, and it's not an application wasting time handling meaningless connections. Also the kernel doesn't waste time maintaining socket state and such.
- coordinate with your upstream Internet provider to block or reduce the rate of ingress of these packets. This is more useful than acting on one's own due to the limits of your upstream link...i.e., the attackers are simply filling your pipe with useless bits. Considering my link is what you might call residential-class service, this is impractical. (Besides, shhhhhhhhhhh! their ToS say I'm not supposed to be running a server. But let's think about this a second. If they were truly serious about this, all they would have to do is block the well-known proto/port used, just like they do with outbound TCP/25.)
- shut off service entirely, and hope that the attacker botnet eventually "loses interest" in in your host, or gets shut down by others' actions.
The first thing I did of course was log onto the (Linux) router and do a tcpdump of the "WIC" (wide area network (WAN) interface card), and discovered a lot of TCP/25 traffic. So I pulled up the maillog, and discovered a lot of instances of "host such-and-such did not issue MAIL/EXPN/VRFY/ETRN" and messages about refusing connections because the configured number of (MTA) children has been reached (happens to be 7 in my case). By far my biggest concern is the attackers will be sending something which trips up my MTA and makes it spam other people, possibly getting me knocked off my ISP. So I did what I usually do, look at a representative sample of recent maillog entries and add some iptables(8) rules and DROP traffic coming from these hosts (or maybe netblocks) which are causing the "did not issue MAIL/EXPN/VRFY/ETRN" entries to be generated (basically, connecting and doing a whole lot of nothing, just tying up a socket and pushing me over my child process limit).
There came a point where I realized I needed more than just a line or three of rules, so I decided to grep(1) for "did not issue" in the logs, take the latest 150 such entries with tail(1), extract the IP addresses with sed(1), and use sort(1) to get a unique set of them. As I recall, I came up with 30 or so addresses, and used a for loop to append DROPs of them to the PREROUTING chain in the mangle table. (The idea is to cut off traffic as soon as the router takes packets in, so a PREROUTING chain.) Unfortunately, it became apparent that a handful of addresses wasn't going to be effective, because the log messages kept on a-comin'. So I decided maybe a little PTR record and whois(1) investigation (to block entire netblocks instead of individual addresses) was in order. A disturbing trend started to emerge. The IP addresses were generally purportedly to be Russian and other Eastern Bloc countries, who, at least to me, are notorious for being the origin of a lot of DDoS attacks and spam.
I really did not want to shut off services entirely, but I saw little choice. I did not want my MTA compromised to be yet another source of spam. I put in an iptables(8) rule which dropped all TCP/25 traffic arriving over the WIC. I noticed this did not stop the popping of message into maillog and realized the attackers were also attempting connections to TCP/587 and TCP/465, so I put in rules for those too. For the moment, IPv6 is, in a sense, only in its infancy (for being a twenty or so year-old infant!), so there was no apparent reason to add any ip6tables(8) rules. And in fact, email still flowed in over IPv6, most notably from Google (thanks for being a leader in the adoption of IPv6!).
It was at this point I was very glad that (according to Wikipedia) Dana Valerie Lank, D. Green, Paul Vixie, Meng Weng Wong, and so many others collaborated on initiating the SPF standard. I began to think, from whom was it particularly critical that I receive email? I could go to those domains' SPF records and insert ACCEPT rules for those addresses or netblocks. Google was already "covered," as noted above about IPv6. Oddly enough, the first domain I thought of was aol.com, because my volleyball team captain might want to tell me something about the soon-upcoming match on Monday. The SPF for them looked knarly, with some include: directives. I settled for looking at Received: headers in previous emails from Matt and identifying the netblock AOL had been using previously (turns out it could be summarized with a single /24). Next I thought of Discover, then of First Niagara, then Verizon (for them informing me of impending ejection from their network for violation of their ToS). I also thought that although it's not all that critical, I receive an awful lot of email from nyalert.gov, especially considering we had that extended rain and wind storm, and the extension of small craft advisories and so on. All in all, I made exceptions for a handful of mailers.
Then to evaluate the extent of the problem, I used watch(1) to list the mangle PREROUTING table every 60 seconds, to see the packet counts on the DROP rules. I'd say they averaged deltas of around 20, or one attempt every three seconds, and the peak I saw once was 51, or nearly an attempt per second. I know as DDoS attacks go, this is extremely mild. If it were a seriously large botnet controlled by a determined entity, they could likely saturate my 25 Mbit/s link, making doing anything Internet-related extremely challenging or impossible. Always in the back of my mind was that this was unsustainable in the long run, and hoped that the botnet was dumb enough to "think" that if not even ICMPs for connection refusal were coming back, that it would assume they in one way acheived their one possible objective, which was knocking my entire MTA host offline.
I then contemplated the work which would be involved in logging onto various Web sites (power, gas, DSLReports/BroadbandReports, First Niagara, Amazon, Newegg, and on and on) and updating my email address to be GMail. I also started trying to search for email hosting providers who would basically handle this entire variety of mess on my behalf, and forward email for me, whereby I could block all IP addresses except for those of said provider. Or maybe I could rejuvenate fetchmail(1) or similar to go get my email from them over IMAP and reinject it locally, as I used to do decades ago with my dialup ISP's email. To my amazement at the low prices, it looks as if, for example, ZoneEdit will handle email forwarding for only on the order of $1/month/domain (so $3 or maybe $4 in my case, because there is philipps.us, joe.philipps.us, philippsfamily.org, and philipps-family.org). This is in contrast (as far as I know) to Google's $5/month/user, and I have a ton of custom addresses (which might be separate "users"). (Basically, it's one of the perks of having your own domain and your own email server. Every single sender gets a separate address, and if they turn into a not-so-savory sender, their specific address ceases to work.) The search terms needed some tweaking because trying things like "mail exchangers" (thinking in terms of MX records) turned up lots of hits for Microsoft Exchange hosting.
A friend of mine runs a small Internet services business and has some Linux VPSes which I can leverage. He already lets me run DNS and a little "snorkel" where I can send email through a GRE tunnel, and not appear to be sending email from a "residential-class IP address." So I called him up and thankfully he answered right away. I got his permission to use one of the IPv4 addresses (normally used for Apache/mod_ssl) for inbound email, in an attempt to see if these attackers are more interested in my machine (the specific IP(v4) address) or my domain(s). If I add an additional address to accept email, and the attacks do not migrate to that address, I then know that it's far more likely that this botnet came across my address at random or by scanning the Verizon address space. So, I picked an address on one VPS, added a NAT rule to hit my end of the GRE tunnel, had to debug a routing table issue, redid the MTA configuration to listen on the tunnel's address, all-in all about an hour and a half's worth of work to implement and do preliminary testing.
It was at this time I realized in my watch window that the packet counts were no longer increasing, even over several samples (minutes). Even after I added the A record to supplement the one for my Verizon address, I noticed there was basically no activity in maillog. So as besst as I can tell, it was as I suspected, the botnet was really likely only interested in my specific address/host. And thankfully it "saw" the lack of any TCP response as an indicator that the host had gone offline, and to cease directing any resources to the attack on me. I hate to give these worms any ideas, but you could also try other things, like ICMP echo, to determine if that host is still alive. Then again, if your sole objective is compromising an MTA, maybe that doesn't matter.
Eventually, I inserted a rule to accept TCP/25 traffic again, thinking if attacks resumed, I could easily kick that rule out and spring the shutoff trap again. Or even better, I could replace it with a rule including the limit match, so only something like 10 or 15 connection attempts could be made per minute. At least the MTA would not be wasting time/CPU on these useless connections, and the kernel would not have to keep track of state beyond the token bucket filter. I almost hit the panic button when I saw some more "did not issue" messages, as well as a little later a notice of refusing connections because of child limit. But I reasoned, just wait a few minutes, and see if it's persistent. Howver, I had a lot of anxiety that it was not over yet, and that it was the domain and not the host the attackers wanted compromised, because some of that unwanted activity was through the newly set up IP address.
In retrospect, I question whether I want to continue doing this myself.
- It's against the ISP ToS, and they could blast me off their service at any time. I'd likely have to pay their early termination fee and I'd have to go slinking back to Time Warner (and their slow upstream speed).
- What happens the next time this occurs? Will it be more persistent?
- What if a future attack is attacking the domain, and any IP address which I use get similarly bombarded?
- What if next time it's not around an attempt per second but instead hundreds or more attempts per second?
- I should have traced traffic on my GRE "snorkel" tunnel to see if they managed to compromise my MTA, and were actually sending email surreptitiously.
UPDATE Mon., 16-Nov: There are some more folks at it again. What I've decided to do for now is to put in some iptables(8) rules which use the limit match to allow only a certain number of connections per minute. So far this has been fairly effective at keeping the chaos to a minimum, but it's not ideal. I really think I'm going to have to look at an email hosting service, at least until I can put up a more robust email server, or maybe permanently.
Direct all comments to Google+, preferably under the post about this blog entry.
English is a difficult enough language to interpret correctly when its rules are followed, let alone when the speaker or writer chooses not to follow those rules.
"Jeopardy!" replies and randomcaps really suck!
Please join one of the fastest growing social networks, Google+!
Subscribe to:
Posts (Atom)