Patrick's playground

How not to treat your customers

Patrick — 2012-05-07T07:15:20+02:00

Here's a funny game - let's try to use a LSI controller from the command line in Linux.

First difficulty: Since a short while ago all downloads are behind a registration wall. Ok, no biggie, let's register then (grumble, grumble, moan.)

The registration demands a password containing at least seven characters, including at least one non-alphanumeric. Yey!

Then there's the Captcha, which looks quite amusing, but ... erf. After three or four guesses you might even get it right.

Fast forward to the driver download page. The specific model (according to lspci) doesn't even exist. Oh, haha, u so funny! But no worry, all the MegaRAID models share the same driver.

Now, here it gets a bit esoteric:

$ file 8.02.21_MegaCLI.zip 
8.02.21_MegaCLI.zip: Zip archive data, at least v2.0 to extract
$ unzip 8.02.21_MegaCLI.zip 
Archive:  8.02.21_MegaCLI.zip
  inflating: 8.02.21_MegaCLI.txt     
  inflating: 8.02.21_Windows_MegaCLI/MegaCli.exe  
  inflating: 8.02.21_Windows_MegaCLI/MegaCli64.exe  
  inflating: 8.02.21_Windows_MegaCLI/Readme.txt  
  inflating: 8.02.21_VMware_MegaCLI/MegaCli  
 extracting: 8.02.21_VMware_MegaCLI/MegaCli.zip  
  inflating: 8.02.21_Solaris_MegaCLI/MegaCli  
  inflating: 8.02.21_Solaris_MegaCLI/MegaCli.pkg  
  inflating: 8.02.21_Solaris_MegaCLI/readme.txt  
 extracting: 8.02.21_Linux_MegaCLI/MegaCliLin.zip  
  inflating: 8.02.21_Linux_MegaCLI/readme.txt  
  inflating: 8.02.21_FreeBSD_MegaCLI/MegaCli  
  inflating: 8.02.21_FreeBSD_MegaCLI/MegaCli64  
  inflating: 8.02.21_DOS_MegaCLI/LICENSE_DOS32A.txt  
  inflating: 8.02.21_DOS_MegaCLI/MegaCli.exe

... wow, we got a Solaris driver for free, and a linux ... zipfile? eh whut?

$ unzip 8.02.21_Linux_MegaCLI/MegaCliLin.zip
Archive:  8.02.21_Linux_MegaCLI/MegaCliLin.zip
  inflating: MegaCli-8.02.21-1.noarch.rpm  
  inflating: Lib_Utils-1.00-09.noarch.rpm

Oh, right, not really a zipfile but an rpm. My bad! haha! oh ... uhm ... :(

$ rpm2tar MegaCli-8.02.21-1.noarch.rpm
$ tar --list -f MegaCli-8.02.21-1.noarch.tar 
./
./opt/
./opt/MegaRAID/
./opt/MegaRAID/MegaCli/
./opt/MegaRAID/MegaCli/MegaCli                                                                                                                                                                                                               
./opt/MegaRAID/MegaCli/MegaCli64
$ file ./opt/MegaRAID/MegaCli/MegaCli64 
./opt/MegaRAID/MegaCli/MegaCli64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.4.0, stripped

zipception! it's a single binary, in an rpm, in a zip, in a zip, on a protected website! And look, "for GNU/Linux 2.4.0" - it's built to run everywhere. Even on Really Ancient Computers.

So, uhm, taddah:

$ ./opt/MegaRAID/MegaCli/MegaCli64 
Fatal error - Command Tool invoked with wrong parameters
Exit Code: 0x01

And now you know why so many sysadmins are alcoholics.

(Bonus feature: We're not allowed to mirror any of these highly proprietary files, so every single person trying to interact with a LSI "Raid" controller on Linux gets to experience that)

o_O

Patrick — 2012-02-22T16:29:48+02:00

Total: 2409 packages (33 upgrades, 2269 new, 7 in new slots, 100 reinstalls, 1 uninstall), Size of downloads: 
6,073,164 kB
Fetch Restriction: 8 packages (8 unsatisfied)
Conflict: 4 blocks

Would you like to merge these packages? [Yes/No]

Bleeding edge, in retrospect

Patrick — 2011-10-29T22:50:40+02:00

Found this gem while cleaning up:

Booting Gentoo - from init to console

Patrick — 2011-10-22T18:11:25+02:00

I've spent some time with OpenRC and sysvinit trying to understand a few things (for example how to integrate CGroups support), and along the way I've learned a few things about the boot process that are not that well documented.
So why not document it for posteriority ...

In the beginning, through some magic, the kernel is booted. How that happens is another story and not our concern at the moment. At some point the kernel has initialized, figured out where the rootfs is (for example through the root= kernel parameter), mounted it ... and now what?
At this point we need to start userspace process #1, traditionally known as "init". The kernel has some hardcoded defaults that by default will try /sbin/init, but it's easy enough to override that with the init= kernel parameter if you want to have something else run (like /bin/bash to get just a rescue shell)
init comes from the sysvinit package and is small enough to be read in an afternoon. There are some surprisingly elegant bits in it, but it's still just doing one job well - starting the rest of the userland processes. It takes its info from /etc/inittab, which just lists different runlevels and what to start. Now in the case of Gentoo this is a bit unusual as it mostly just calls "rc", which is part of the OpenRC package, with a parameter like "rc single". This is the name of the runlevel then - "default" by default. We have sane defaults!
Init is also triggered to change runlevels, this is usually done through the "init" or "telinit" commands.

Now OpenRC needs to figure out what to start. Before the "default" runlevel is started we need to start the "sysinit" and "boot" runlevels (for details have a look at /etc/runlevels ). This starts a few things like udev and mounts local filesystems, then starts all the daemons you requested. The bookkeeping for that (what has started, is starting, has failed etc.) can be found in /lib/rc/init.d/ - just another simple directory with self-explaining filenames. And running "rc" without arguments will just try to get us back to the current runlevel defaults - start what is stopped, and stop everything not defined in /etc/runlevels. Running "rc" in cron is a nice way to keep things like sshd running even through accidents like "killall sshd" :)

How OpenRC figures out the dependencies is quite "magic", but if you trace it you find runscript (an executable) running runscript.sh with a parameter like "depend", which sources the init scripts and just outputs the value of the DEPEND line. (Read /lib/rc/sh/runscript.sh to get an idea, or if you get bored read the source of runscript). And that information is cached in /lib/rc/init.d/deptree to avoid having to re-source the init scripts as this is a "slow" process (maybe 50msec per init script, but if you have 100 scripts that's still 5 seconds you lose just parsing the init scripts instead of starting stuff)

So OpenRC starts all the things from /etc/runlevel and is now done, it returns the control to init, which now notices that it has a few lines like this in its config (/etc/inittab):

# TERMINALS
c1:12345:respawn:/sbin/agetty -c 38400 tty1 linux

So what it does now is very simple - it runs agetty, which configures the (pseudo-)terminals (tty1 here) and starts a login program (in this case /bin/login, the default). This asks us for username and password (another interesting story for a different time), and when this is done runs the login shell specified for that user.
And here we are, booted up and ready to serve our human overlords ;)

OpenRC, agetty and terminal blanking

Patrick — 2011-10-19T15:11:02+02:00

Some people might have noticed a naughty interaction with one of the last sys-apps/util-linux updates and OpenRC.
The symptoms are described in bug 381401 - what you usually notice is that after boot the console blanks / resets and all you see is the boring login prompt.

The cause is a small change in agetty defaults, the manpage now mentions:

       -c, --noreset
              Don't reset terminal cflags (control modes). See termios(3) for more details.

So, if you are bothered by this change you need to edit /etc/inittab, which defines where and how the agetty processes are started. Change:

c1:12345:respawn:/sbin/agetty 38400 tty1 linux

c1:12345:respawn:/sbin/agetty -c 38400 tty1 linux

And from now on you should still have the old behaviour on reboot. And maybe we can convince Vapier that that would be a sane default so we don't have to change it on every system.

Until then you should still be able to see the boot messages in /var/log/rc.log - by default /etc/rc.conf has rc_logger="YES" set, so that should "just work"(tm)

CGroups support for OpenRC

Patrick — 2011-10-16T17:19:36+02:00

During a train ride I spent some time implementing a prototype for CGroups support for OpenRC - the big awesome feature for SysTemD that people seem to desire the most.
Here's the most amusing bit:

$ git diff 3ad849c5d6a24ef66152004eb3149d2cff973b1c..082c04e0a1c31115417af9fd348ae83ee8ecc397 --stat
 sh/init.sh.Linux.in |   10 +++++++++
 sh/runscript.sh.in  |   54 +++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 62 insertions(+), 2 deletions(-)

So I've managed to stay below my initial estimate of "under 100 lines of shell" - I don't quite get what the big deal is. Of course the initial patch is a bit rough, there are some things that just don't feel clean to me - but it works.

To explain the patch a bit - sh/init.sh.Linux.in:

+# Setup Kernel Support for CGroups
+if grep -qs cgroup /proc/filesystems; then
+       ebegin "Mounting CGroup filesystem"
+       mkdir -p /dev/cgroup
+       mount -n -t cgroup -o nodev,noexec,nosuid \
+               OpenRC-CGroup /dev/cgroup
+       eend $?
+fi

We need to mount a CGroup pseudofilesystem somewhere, I figured out that we better do it as early as possible (although procfs and sysfs init scripts aren't wrong per se). So we mount a cgroupfs named "OpenRC-CGroup" - that's just a nice marker so you see who did it. So much for setup.
Now when a service starts we just need to take the parent process (which is conveniently the runscript.sh process that runs the init script) and push it into its own group. Although, maybe, it makes sense to let some init scripts be in the same group, right?

+# Attach to CGroup - dir existing is enough for us
+if [ -d /dev/cgroup ]; then
+       # use the svcname unless overriden in conf.d
+       SVC_CGROUP=${CGROUP:-$RC_SVCNAME}
+       mkdir -p /dev/cgroup/${SVC_CGROUP}
+       # now attach self to cgroup - any children of this process will inherit this
+       echo $$ > /dev/cgroup/${SVC_CGROUP}/tasks
+       # TODO: set res limits from conf.d
+fi

Look, ma, no hands!
We allow users to set CGROUP in conf.d files, but if it's not set we just use RC_SVCNAME, which is the name of the init script usually. Then we create a subdirectory (but we don't care if it already exists) and throw ourselves into it.
Tadaah!

There's one little downside, we create directories but don't remove them. So let's add a little cleanup at the end of runscript.sh:

+# CGroup cleanup
+if [ -d /dev/cgroup ]; then
+        # use the svcname unless overriden in conf.d
+        SVC_CGROUP=${CGROUP:-$RC_SVCNAME}
+       # reattach ourself to root cgroup 
+       echo $$ > /dev/cgroup/tasks
+       # remove cgroup if empty, will fail if any task attached
+        rmdir  /dev/cgroup/${SVC_CGROUP} 2>/dev/null   
+fi
+# need to exit cleanly
+exit 0

The exit 0 is a bit naughty, but I think it's the right way to terminate. Otherwise the rmdir might give us a bad return value which makes it look like things fail even when they don't ...
There's a neat trick in it - we are still in that cgroup, so we take ourselves and move us back to the root cgroup - otherwise the cgroup will never be empty. Duh! :)

And here's a more, let's say, controversial part. It's a bit exquisitely rude, so I guess this will have to be adapted for public use - but it works too well. See, there's this leeeetle problem that OpenRC tends to not kill processes that well. So if, for example, the apache init script doesn't manage to kill all indians you may have a problem - for example re-starting might fail because there's still the last mohican protecting port 80.
Since we know that the processes end up in a specific cgroup we can just take all those, put them against the wall and shoot them dead. This has some potential side-effects, for example if there's multiple init scripts using one cgroup or if there's a service where it's expected that a process survives (although that's funkay). Anyway ...

+# Kill everything in the CGroup with maximum prejudice until it is dead
+# This is very naughty if multiple init scripts were to share a CGroup ...
+terminate()
+{
+        if ! [ -d /dev/cgroup ]; then
+                return 0
+        else
+                SVC_CGROUP=${CGROUP:-$RC_SVCNAME}
+                # we want to survive and thus must not be killed dead
+                echo $$ > /dev/cgroup/tasks
+                for signal in TERM KILL; do
+                        for i in `cat /dev/cgroup/${SVC_CGROUP}/tasks`; do
+                                kill -s $signal $i
+                        done
+                done
+        fi
+        # if anyone survived here we could try task_freezer; SIGTERM; unfreeze
+        # to be sure everyone really got the message that they are dead
+}

This works well even for uncooperative init scripts, but might not always be what you want. But at least it kills everyone dead :)
And there's some funky resource limits possible - I'm still trying to figure out how to make use of them properly, but ...

+       # cpuset support try 1
+       if [ -v CGROUP_CPUS ] && [ -e /dev/cgroup/${SVC_CGROUP}/cpuset.cpus ]; then
+               echo $CGROUP_CPUS > /dev/cgroup/${SVC_CGROUP}/cpuset.cpus
+       fi

... this nails everything from one init script to a cpuset, so you can hammer apache onto CPUs 1-7 and leave CPU 0 free for "everything else", if you want. It also allows memory limits and a few other advanced gadgets, but I haven't played with that yet.

And so in one afternoon we managed to grow a pretty decent bonus feature ..

Fighting Stupid

Patrick — 2011-08-15T14:09:39+02:00

The mission for the day: Activate a new email account and XMPP/Jabber

Sounds easy, right? So ... let's find an XMPP client. ayttm ? Sounds nice, but ... wow ...
If it were a waiter in a restaurant it'd ask you what you want to order, then empty a glass of water over your head and bring you some raw broccoli.

Absolutely no will to live, and ignores users.
Uhm, pidgin? Certain entities recommended that. One does wonder why, because it's a deafmute that communicates by throwing things at you. Username? Pah, who needs that! And why didn't you give me a password when I didn't ask? Either way, XML Parsing Error, so I can't communicate with XMPP anyway. And you're ugly!

Good thing emerge -C works so well. Now, some other entities mentioned telepathy, and the kde telepathy bits even do something. I'm unsure if that's really the expected mode of oprtation, but it seems to have connected to ... something.

Next mission: Setting up an email account in Thunderbird. Wow. Omg. It has gotten even more pessimized since I tried the last time ...

First of all it tries to guess the communication parameters. foo@bar.com ? Mailserver must be mail.bar.com ... or grmpflzrgh.bar.com ... or any of the other 35 wrong things it tries to connect to. SMTP? Let's do the guessing game once again. Hint: On the left side there's a button "manual configuration" which aborts this guessing game. Then you can add your own entries ...
Just to notice that some idiotic moron made it impossible to *use* your own settings. Because you get a wrong popup that claims that you can continue with these untested settings (wooooh, I'm scaaared!), but ... the button? It's greyed out. What the bleep. In a movie it would be hilarious, but ... WTF. SRSLY. URTARD.

Now then, let's go back, hit the mandatory "test settings" button (why is there a "continue button" ... oh nevermind), then it changes the settings to something wrong, you hit continue, wait until fetching emails fails, then go to settings and change it to the correct values again.

Which makes no difference because it still can't connect, so my plans to get rid of ThunderFail have been immensely accelerated.

About 2h wasted again with software that doesn't have a will to live, but at least I get some good ideas for policies and testing ...

Apt-gentoo? Gentoo-apt! Hah!

Patrick — 2011-08-13T11:00:20+02:00

I was marginally amused when I saw that some funny person had written an apt-gentoo wrapper that, err, scrolls build logs around slowly.

I guess that's a debian user's idea of an interpretation how package building works. It would have been a lot more amusing to turn debian into a proper build-from-source environment, but ... that's actually hard.

So in return I'll mention how to badly emulate apt on gentoo:
*) Create or use a binhost - this is pretty easy to set up, often FEATURES="buildpkg" is all you need to configure on the server side
*) Make the packages available, for example through http: cd /usr/portage/packages; python -m SimpleHTTPServer 80
*) set the PORTAGE_BINHOST variable to point at your newly created server
*) add "-G" to EMERGE_DEFAULT_OPTS
*) Tadaah. Now you have a proper binary Gentoo that behaves almost, but not completely unlike apt.

Next lesson: How to badly emulate system startup with OpenRC? ;)
(badly because it actually works, which is not part of the requirements)

Reboot

Patrick — 2011-07-28T23:07:08+02:00

After spending quite a bit more time than expected in Berlin I've finally returned back "home". Having access to more than a suitcase of stuff can be convenient (although it appears to be an optional luxury now).

Due to some unfortunate hardware failures the fastest working CPU I have locally is a single-core Athlon64. It's quite fascinating to see how bloated everything has become, what used to be a droolworthy CPU not so long ago is now the lower end for a simple desktop. Especially memory consumption is insane - Thunderbird easily absorbs 5GB RAM if you push it a bit. Firefox seems to grow at a rate of ~200MB/day and needs to be regularly restarted. So much sadness.

I'm slowly catching up with gentoo things, and it appears that I've found two new minions to recruit. Just as I had realized that I have some time and motivation - I like it. More people means less work per person, so less burnout, less abandoned packages and so on. More better happy.

Thanks to the work of sochotnicky our Ohloh Statistics have finally been updated. There are some interesting results - for example the amount of committers has been roughly constant for the last 2-3 years, which means that recruiting is at least absorbing the normal attrition. But I think we need to do one better and get things growing ... how else are we supposed to keep everything in good shape?
Which also makes me think about bugs and how to squish them most effectively. There are so many bugs open that I find it hard to get an overview what is "most urgent" or what are trivial bugs that might just take 5 minutes of work to fix. So we should definitely revive the BugDays and make it easier for people to get involved and provide us with fixes. Right now I have no motivation (and not enough hardware) to do any tinderboxing as I can easily divert all processing power I have into bugfix testing. And that's going to make people happier, on average, than finding even more bugs ;) (although we need to improve on both ends - and we need more metrics so we know where we are and where we are going).

And on it goes, the infinite hamster wheel of progress - who wants to help?

MDMA

Patrick — 2011-07-01T19:44:26+02:00

The Monitoring-Driven Master Administration

For software dev we have Test-Driven Development (TDD), unittests and all that machinery. The goal of all those methods is to catch errors, best before they can hurt anyone.
Detecting problems early saves you lots of time and frustration and makes changing and improving things easier.

For admins, we now have monitoring-driven administration:

(1) Set up monitoring. Watch it fail and notify you

(2) Set up service

(3) Watch all monitoring switch to greenlight

There are some simple rules to be followed:
No service can be deployed without monitoring. If there is any critical warning from the monitoring it needs to be fixed. If there are warnings they should be fixed, either by tackling the problems or increasing the monitoring threshold.

The default state is all green - no warnings, no errors (except during the test/integration phase of new services). Any warnings or errors should trigger you into fix-this-stuff mode.

Rationale:
When you enable a service in the greenlight state you never figure out if you monitor the right bits. Maybe the check for free disk is running locally instead of remotely? Will always look good, even if the actual service is in a failed state.

Having any warnings means that something is in a state you consider not-good. Either fix the service or the monitoring thresholds.

Results:
You'll be able to sleep a lot better if you get your daily status email and you know that everything is working fine. Then you can focus on improving things instead of playing infinite fireman.

If this sounds like stating the obvious, well, most good ideas are ...