Patrick's playground

Recently I aquired a nice Lenovo notebook. I'm quite happy so far with it (and need to document how to setup EVERYTHING), but I did have some small troubles getting the integrated GSM/UMTS modem to work as expected.
[btw, if anyone has some bluetooth gadgets to send me I'd appreciate it since I have no way of testing if bluetooth works or is configured right]
Now this notebook had a preinstalled XP (which booted once so I was sure the machine would boot) and some Vista recovery DVD I decided to use these magic recovery DVDs (three all in all) to install Windows so I could test the UMTS part. I needed to wipe the disk anyway for a fully encrypted setup, so nothing was lost :)
I might be spoiled from using Linux, but it took over 3 hours until the install was done. During that time it rebooted nine times. Imagine that. To recover from a simple failure ... 3 hours? On quite nice hardware? What! I am seriously annoyed at the level of fail exposed there. A binary distro like Ubuntu takes about that time to download, install, update. Heck, emerge -e world doesn't take that long.
Plus Vista lags. I mean, it does have some basic effects like transparency and does compiz-like "omg I haz effects" thingies when minimizing or maximizing windows, but it looks quite primitive and dated. So how can it be THAT slow?
On a normal desktop showing two or three windows I see no acceptable excuse for making a focus switch take noticeable time. If I click on a window I don't see why it should take half a second for the window to get focus. And starting apps shouldn't take that long. Also, what the bleeeeeep happened to the Control Panel? That looks like a nice game now, but it doesn't even let me configure my wired network interface in any easy and accessible way. Seriously not nice. Bah, I'm glad it's gone. It's a waste of time and nerves, and it's not viable as a desktop operating system. KDE4 or death!
Which leads me to the next disappointment - KDE4.5 trunk has not broken in unexpected ways. It's so boring! I expected a commit flood (which happened) and stuff breaking (which hasn't happened yet). Looks like those kids got the integration problems under control and things will only get more boring and predictable for us packagers. Isn't it nice?

So today has been quite productive.
On the XEN front I've killed the 2.6.20 and 2.6.21 kernels as they were lacking a will to live. Instead I finally added (masked) the shiny new kernels from the gentoo-xen-kernel project. That allowed me to close about 20 stale bugs ... and give y'all a chance to file a dozen new ones.
On the postgres front things are looking good, I finally fixed 7.4.27, made the init scripts posix sh compliant and fixed this annoying integer-datetime switcheroo. If all goes well we'll have the "new" split ebuilds stable and I can finally exterminate the oooold libpq and postgresql bugs. Aaaah :)
If you haven't noticed there's an election for a vacant council seat running, so stop slacking and vote. And/or live with the results.
KDE 4.5 branched. So I'm one login away from not using 4.4 anymore. Oooooooold :D
The whole bugwrangling thing is getting on my nerves a bit, I've been wrangling something over 30 bugs a day for a while now. And as soon as I don't do it for a day or two the bugwrangler queue grows over 100 bugs and I'm sad.

And that's how it goes. Another day ends, another pile of bugs sneaks up in the middle of the night. In the morning things look sad again and we restart our sysiphean task once more.

The recent updates in radeon and mesa have been quite nice, it leads to this:

OpenGL vendor string: Advanced Micro Devices, Inc.                             
OpenGL renderer string: Mesa DRI R600 (RV730 9498) 20090101  TCL               
OpenGL version string: 2.0 Mesa 7.8-devel                                      
OpenGL shading language version string: 1.10 

which means that more things work better now.
The upside (things like doom3 not crashing but actually starting) comes at a price for now: Some applications are just stupid slow. For example UT2003 is so slow that the menu is barely usable, but UT2004 runs very nicely. And the things that run fast seem to have gotten faster since last time I tried (when OpenGL 1.2 was all you could get from radeon).

Now I'll only have to wait a while until the nice people that are working on it fix the performance issues - that shouldn't take long, I hope. Open source drivers are fun!

Oh well. I don't really celebrate Christmas (once in a decade is enough). And it's a bit early to be in the christmas spirit (which most years seems to be a mix of being stressed and being rude to others).
Still there's a few things every geek can use, and maybe there's a magic elf out there who can make a wish come true, eh?
Now if you, by chance, feel the need to help me out there's a few little things I could use. My current buildbox is quite nice, but I'm hitting a ceiling. It's not so much the CPU (quadcore ftw), but the IO that's hurting. And I'm getting a bit cramped with space - 65GB distfiles alone!

So here's the things I would really appreciate:

SSD, SATA-II, 32GB or larger
harddisk, SATA-II, 750G or larger, at least two (RAID-1)
SATA Controller, PCI or PCI-E, "something reliable"
Memory, 4x4GB, for great justice

Having those bits would help a lot - let's see:
an SSD is really nice to compile in. Tremendously helps performance. I currently have "only" 8GB, which limits what I can build (boost with FEATURES="test" needs more!) and how much I can build in parallel.
Harddisks - all the distfiles, binpkgs and other things add up. At current prices a small disk makes little sense, and I'd want some reliability, so I'd want at least two disks (raid1) or more (raid5).
Since I have a "desktop" mainboard there's only four sata ports. This means I can currently add 0 extra disks. Which sucks a bit :) So I'd need a sata controller to really enjoy any goodies.
And lastly even with the current 8GB RAM I'm hitting a saturation point. It's insane, but with 4 or more chroots doing stuff it can get painfully tight. Again, desktop mainboard, so I'd need 4x4GB to have an advantage (or get a new mainboard with all the complications that involves)

All that just to compile faster. Aren't we a decadent bunch of hackers? :)

Now with 3D working quite well on my radeon (HD4650) I've tested a few games. Somewhere in the last 3 days the performance got a good kick, the improvements are noticeable.
Quake3 by default is limited to 1600x1200 and pretty much maxes out at 90fps all the time. It feels really smooth, and the resolution is quite nice too :)
UT2004 runs at 1920x1200, looks awesome, but is a bit too slow. Around 1280x1024 the performance is high enough to play well.
Nexuiz at 1920x1200 has a rather wobbly performance. Sometimes it goes down to about one frame per second, then spikes back to 30+. Around 1280x1024 it too becomes nicely playable.
That's quite awesome for an open driver, and it's about the first time I've heard the graphics card fan kick in like that.

And people were right, you really only need the -9999 versions of mesa + libdrm + xf86-video-ati.

About the portage options and all that, darkside has blogged about some in the past. It's nice to have some numbers to put next to the --jobs magic :)
And KingTaco had found the CPU hotplugging madness quite some time before me. Plus ca change, plus c'est la meme chose.

I've been slacking a bit, now I'm feeling almost guilty for letting bugs rot. It's interesting to see how motivation works. Sometimes I even wonder why I spend any time on such things - as soon as I fix a bug upstream does a new release, a security issue is found or some other form of breakage. It's a bit frustrating to see this endless stream of "work" coming in ...
But I'm quite happy to see more and more involvement from users. It's good to see people trying to help. Most seem to lack confidence, what they lack in skills they make up for in learning at an insane pace. So as long as there is someone to guide them a bit they are totally awesome. Which means that if I can get some more people motivated I can finally resume infinite slacking because I've been made redundant. I think that should be the goal of every package maintainer :)

Here's something awesome I found mostly by accident:
In recent kernels the support for hotswapping CPUs works on x86/amd64 architectures. I stumbled over it in the 2.6.32 menuconfig and couldn't wonder if it actually works. So I had a look and found this gem:

# cat /proc/interrupts | grep CPU
            CPU0       CPU1       CPU2       CPU3

Very boring, 4 processors.

echo 0 > /sys/devices/system/cpu/cpu3/online

And we just knocked out one!
We see that in dmesg:

kvm: disabling virtualization on CPU3                                                                                                                       
CPU 3 is now offline

Hmm, are you thinking what I'm thinking?

kvm: disabling virtualization on CPU2                                                                                                                       
CPU 2 is now offline                                                                                                                                        
kvm: disabling virtualization on CPU1                                                                                                                       
CPU 1 is now offline                                                                                                                                        
SMP alternatives: switching to UP code

Wheeee. I just castrated it to a single core! I actually didn't check if the kernel lets me take CPU0 offline. That would be hilarious. Anyway ...

echo  > /sys/devices/system/cpu/cpu1/online

And we just gained a CPU:

SMP alternatives: switching to SMP code                                                                                                                     
Booting processor 1 APIC 0x1 ip 0x6000                                                                                                                      
Initializing CPU#1                                                                                                                                          
Calibrating delay using timer specific routine.. 5200.20 BogoMIPS (lpj=10400418)                                                                            
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)                                                                                           
CPU: L2 Cache: 512K (64 bytes/line)                                                                                                                         
CPU 1/0x1 -> Node 0                                                                                                                                         
CPU: Physical Processor ID: 0                                                                                                                               
CPU: Processor Core ID: 1                                                                                                                                   
CPU1: AMD Phenom(tm) 9950 Quad-Core Processor stepping 03                                                                                                   
checking TSC synchronization [CPU#0 -> CPU#1]: passed.                                                                                                      
kvm: enabling virtualization on CPU1 

This is seriously wicked. Now I just need to figure out how to bolt that onto powermanagement so that the machine knocks out cores when idle and powersaves. Linux never gets boring ...

As documented in this bug (which mirrors an upstream bug here there's a bug in glibc 2.10 and 2.11, and this one seems to be easy to hit. Multithreaded apps "randomly" crash with "Invalid free" and other confusing errors. A hackaround is to unset or empty the environment variable "MALLOC_CHECK_". For me setting MALLOC_CHECK_="" before starting some of the affected packages seems to completely hide the error, now we can only hope that the gentoo glibc gets this patch soon.

For the rest of this post I'll only consider portage 2.2. Most options are in portage 2.1 already, but I'm a lazy bum, so I don't compare to see what's what.

You can set PORTAGE_DEFAULT_OPS in /etc/make.conf, but if you add --ask you will have trouble running emerge from a script. --ignore-default-opts disables those defaults so you can run emerge --sync in a cronjob again.

Sets are really great, --list-sets shows you which are available. Just have a look, there are some nice ones - "security", "installed", "unavailable" ... they can help streamline some tasks. I find their names quite self-explanatory.

If you want to put something into the world file without rebuilding it use --noreplace, and if you want to remove it again use --deselect.

--nospinner disables that funny rotating spinner thingy so you can save precious bandwidth when connected remotely, and --quiet hides most of the output, which can be nice if you don't want to be hypnotized by scrolling compile output. For the OCD crowd --quiet-build might be nice as it doesn't show the compile output on console, but redirects to logfiles. --changelog is neat for seeing the log messages for that update, this often shows fixed bugs or other issues you might care about. --color with a parameter y or n toggles colorized output. And of course --alphabetical. The horror of unsorted output!

Sometimes people are confused that emerge -e world tries to update packages that emerge -uND world misses. That is usually caused by build-only dependencies. --with-bdeps=y and --complete-graph are good options to modify portage behaviour.

If you're on a fast machine and in a hurry you can try to set --jobs X with a reasonable value of X. Think about memory needs and such before setting it to infinity minus one! With --keep-going it gets really easy to not have the whole process stopped on the first failed package. This is not without issues, but it avoids the --resume --skipfirst in a loop tricks. If --jobs seems to hard to calibrate to you --load-average=LOAD may help to limit it.

For the scripters --columns might be nice, it tweaks the output to be more script friendly.

Support for binary packages has grown considerably, there's support for local (-k / -K) and remote (-g / -G ) binpkg repositories. And you can --buildpkg and --buildpkgonly to create them (they are stored in PKGDIR). There's --binpkg-respect-use to only install the packages that have useflags set the same as the current configuration - it's a very powerful mechanism if you need to support Gentoo on multiple machines and don't want to compile that much.

I hope y'all enjoyed this little lesson in RTFM, there's plenty of other options to discover. Don't be afraid of the documentation, it doesn't bite and makes your life easier :)

Few people take the time to actually read through the documentation, but if you have some time to spare "man make.conf" is a great read.
For example you can pre-set some CLI options like --ask or --verbose in EMERGE_DEFAULT_OPTS so you never have to type them again. Especially the FEATURES variable has some interesting bits:

buildpkg builds packages of everything
buildsyspkg builds only packages of the system set, which is awesome for recovery and doesn't take much space.

keepwork keeps the $WORKDIR and can be quite useful for debugging purposes (but not for general use)
noclean leaves even more there.
fail-clean is the opposite, it always wipes the build directories. Useful if you build on a small (but fast) disk or tmpfs.
installsources installs all the package sources to /usr/src/debug/, which can be used for debugging, but eats lots of space. Together with splitdebug it offers some really great debugging convenience.
test-fail-continue helps when you just want to have the tests run for logging purposes, but don't want the package to not be installed if tests fail. Most people won't need this.

split-elog and split-log features are quite interesting if you do logging.

Logging can be very nice to have, and portage has lots of configuration options for it.
PORTAGE_ELOG_SYSTEM defines how the log data is sent, be it through syslog, email or just to a file. Or completely custom?
And you can do combinations like PORTAGE_ELOG_SYSTEM="mail:warn,error syslog:* save".
PORTAGE_ELOG_CLASSES defines what you want to log - warnings, errors, qa warnings, everything ... it's your choice.

Of course there are lots of other configuration options:
PORTAGE_NICENESS can be useful when you don't want portage to interfere with anything else.
PORTAGE_IONICE_COMMAND needs ionice (or an equivalent tool) and can be used to make the disk activity of portage a bit less distracting. Both features may increase the time needed to install things, but will make portage more benign so you can still do things while it runs.

Also you can change almost all directories - PORTDIR, DISTDIR, PKGDIR and so on. This allows you to make portage behave a lot more like you want it (unless the defaults satisfy you already ...)

Finally I stopped slacking for long enough to fix a few bits of my desktop, and the results are grrrrrrreat.
Now I have (OpenGL!) full effects in KDE4 as opposed to the slightly less bouncy XRender-accelerated thingies before. Performance is pretty awesome (but then the HD4650 shouldn't even notice those few effects).
What you need:

~arch install (I'm not going to care to find out what minimal versions you need)
x11 overlay
a really recent kernel

And with really recent I mean "at least 2.6.32". At the time of writing that hasn't been released, so a 2.6.32-rc6 git-sources has to substitute for me. From what I've read you might have to disable framebuffer for things to work well, but as I'm usually seeing text mode for ~30 seconds every month I don't care enough to find out. I'm lazy!
In the kernel config you need to enable DRM and especially the radeon bits. Device Drivers -> Graphics -> Direct Rendering Manager is the "most important" bit there.
The following packages were suggested in a few places, I have no idea if that is the minimal set. But you'll have to unmask:

>=x11-libs/libdrm-9999
>=media-libs/mesa-9999
>=x11-base/xorg-server-9999
>=x11-proto/fixesproto-9999
>=x11-proto/xextproto-9999
>=x11-proto/xf86vidmodeproto-9999
>=x11-proto/renderproto-9999
>=x11-proto/recordproto-9999
>=x11-proto/inputproto-9999
>=x11-proto/xineramaproto-9999
>=x11-proto/bigreqsproto-9999
>=x11-proto/xf86driproto-9999
>=x11-proto/xf86dgaproto-9999
>=x11-proto/xcmiscproto-9999
>=x11-base/xorg-drivers-9999
>=x11-libs/libXext-9999
>=x11-libs/libXi-9999
>=x11-proto/xproto-9999
>=x11-libs/libX11-9999
>=x11-libs/libxcb-9999
>=x11-proto/xcb-proto-9999

Now go forth and rebuild all your shiny new packages.
If you managed to build that and reboot your new kernel things should look pretty much as before. The only "obvious" hints I've found to test are the output of glxinfo (has changed quite a bit) and that KDE4 allows me to use OpenGL now. And maybe the wobbly windows effect was a giveaway :)

I'm positively surprised that things have progressed this far, and I'm happy to finally be able to use more of my graphics card :)

EDIT: Seems that this is not the minimal set of packages and configuration needed. Some people suggested -9999 packages of mesa + libdrm + xf86-video-ati only. If that works even better :)