Sat Apr 5 12:29:12 CEST 2014

"smart" software

1) Grab webbwrowser
2) Enter URL
3) Figure out that webbrowser doesn't want to use HTTP because ... saturday? I don't know, but ass'u'me'ing that some URLs are ftp is just, well stupid, because your heuristic is whack.

Or, even more beautiful:
$ clementine
18:02:59.662 WARN  unknown                          libpng warning: iCCP: known incorrect sRGB profile 
Bus error


I have no idea what this means, so I'll be explicitly writing http:// at the beginning of all URL I offer to Firefox. And Clementine just got a free travel to behind the barn, where it'll get properly retired - after all it doesn't do the simple job it was hired to do. Ok, before it randomly didn't play "some" music files because gstreamer, which makes no sense either, but open rebellion will not have happy results.

I guess the moral of the story is: Don't misengineer things, clementine should output music and not be a bus driver. Firefox should not interpret-dance the URLS offered to it, but since it's still less retarded than the competition it'll be allowed to stay a little bit longer.

Sigh. Doesn't anyone engineer things anymore?

Posted by Patrick | Permalink

Fri Feb 28 08:37:26 CET 2014

INSTALL_MASK'ing for a better future

So today I was pointed at a funny one:
/etc/systemd/system/ntpdate.service.d/00gentoo.conf
Now instead of being wrongly installed in /usr/lib (whuarghllaaaaaaaawwreghhh!?!$?) there's some config files for systemd bleeding into /etc.

Apart from being inconsistent with itself this eludes all previous ways to avoid useless files from being installed. The proper response thus looks like this now:
INSTALL_MASK="/lib/systemd /lib32/systemd /lib64/systemd /usr/lib/systemd /usr/lib32/systemd /usr/lib64/systemd /etc/systemd"
And on the upside this will break udev unless you carefully move config to /etc (lolwat ur no haz EUNICHS system operation?) - which just motivated me to shift everything I can to eudev.

Reading recommendation: FHS

Posted by Patrick | Permalink

Thu Feb 20 09:32:05 CET 2014

gentoo-x86 to git, round two

After my not-so-good experiments with cvs2git I was pointed at cvsps. The currently masked 3.13 release (plus the lastest ~arch version of cvs) seems to do the trick quite well. It throws a handful of warnings about timestamps that appear to be harmless to me.
What I haven't figured out yet is how to "fix" the email addresses, but that's a minor thing.
Take the raw cvs repo as in the first blogpost, then:
$ time cvsps --root :local:/var/tmp/git-test/gentoo-x86-raw/ --fast-export gentoo-x86 > git-fast-export-stream
cvsps: NOTICE: used alternate strip path /var/tmp/git-test/gentoo-x86-raw/gentoo-x86/
cvsps: broken revision date: 2003-02-18 13:46:55 +0000 -> 2003-02-18 13:46:55 file: dev-php/PEAR-Date/PEAR-HTML_Common-1.0.ebuild, repairing.

[SNIP]

real    212m56.219s
user    12m11.170s
sys     6m59.110s
So this step takes near 3h walltime, and consumes ~10GB RAM. It generates about 17GB of temporary data.
To get performance up you'd need a machine with 32GB+ RAM so that you can do that in TMPFS (and don't forget to make /tmp a tmpfs too, because tmpfile() creates lots and lots of temporary files there) - and the tmpfs needs to be >18GB

In theory you can pipe that directly into git-fast-import. To make testing easier I didn't do that..
Throwing everything into git takes "a while" (forgot to time it, about 20 minutes I think):
Alloc'd objects:    9680000
Total objects:      9675121 (    190979 duplicates                  )
      blobs  :      3020032 (    158366 duplicates    1389088 deltas of    2989578 attempts)
      trees  :      5150778 (     32613 duplicates    4633675 deltas of    4709477 attempts)
      commits:      1504311 (         0 duplicates          0 deltas of          0 attempts)
      tags   :            0 (         0 duplicates          0 deltas of          0 attempts)
Total branches:           8 (         3 loads     )
      marks:     1073741824 (   4682709 unique    )
      atoms:         431658
Memory total:        516969 KiB
       pools:         63219 KiB
     objects:        453750 KiB

pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize = 1073741824
pack_report: core.packedGitLimit      = 8589934592
pack_report: pack_used_ctr            =    7139457
pack_report: pack_mmap_calls          =    1976288
pack_report: pack_open_windows        =          3 /          9
pack_report: pack_mapped              = 2545679911 / 8589934592
And then run git gc (warning: Another mem-hungry operation peaking at ~8GB).
The result is about 7.2GB git repository and appears to have full history.

Files to play around with:
Raw copy of the CVS repo (~440MB)
The git-fast-importable stream created by cvsps (biiig)
The mangled compressed git repository that results from it (~6GB)
Edit:
The same repo recompressed (~1.7GB)
"git repack -a -d -f --max-pack-size=10g --depth=100 --window=250" takes ~3 CPU-hours and collapses the size nicely. Thanks, Mr.Klausmann!

Posted by Patrick | Permalink

Wed Feb 19 07:43:21 CET 2014

Thunderbird - double sending is better sending

So here's something brilliant I've found while debugging some PGP-issues:
0q2CYNVFEz6wXHAGYArfO/F/faOL5L6fQw9f93FurZgx7Y+iR1J7Civaa7LHxQ8h
FzstP7BYEhCx2HmEZuDf18htDsTBZAlNVGsI0DMb2wFKudCaI7hXhMHpYBQF/rdZ
=3Dw1hZ
-- --  END PGP MESSAGE     


--  -- --  --  -- 070107010101000406000609
Content Type: text/html; charset=ISO 8859 1
Content Transfer Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=ISO 8859 1"
      http equiv="Content Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <br>
         BEGIN PGP MESSAGE      <br>
    Charset: ISO 8859 1 <br>
    Version: GnuPG v2.0.22 (MingW32) <br>
    Comment: Using GnuPG with Thunderbird   <a class="moz txt link freetext" href="http://www.enigmail.net/">http://www.enigmail.net/</a> <br>
     <br>
    hQIMA0dhXCfgRaeBAQ/+P2NCYSVE7vxW742D9eYJmJ/7g7xHSvPFuYvGSZk2gRaJ <br>
    JoZ98x+TPjSlvYVWuS+Y2Fz04ydhi4vNcK+QAqImVO0nO6dFvxUfmZiERBcYGs4C <br>
    Lhe+B/I0P/hEDl+Zu/QJ/v+SEcFoXKv2iclrXwWF6RyLlO97iu8UsLYUjLIZ7Y+r <br>
    YGqphoIdJLfVZ9bb05RIb0ZKnYX5dzunpqu6V6zRpwckWCkos7qBOZ9hfBjaFkvD <br>
    ZQAoJM78qQ0//vV6qyxSpXXFEFbDZuJjPjjDfIF+qyNbcW657bDHQH2ctcyvdcTf <br>
(Modulo some dashes, but you get the idea)

So, uhm, there's a multipart-mime mail, with a PGP-encrypted attachment, and then there's a properly quoted HTML attachment, CONTAINING the same PGP attachment BASE64 encoded. Or something. The funny thing is that Thunderbird itself fails to display the body directly, but displays it in the editor window when you reply.
In vino veritas, and tonight I will need lots of veritas to unremember this madness.

Posted by Patrick | Permalink

Tue Feb 18 06:14:37 CET 2014

Converting gentoo-x86 to git, first attempt

A first attempt at cvs-to-git conversion of gentoo-x86; not yet complete. Needs: ~4GB storage for cvs repo, a few GB for temporary files, and a few GB for the git repo

Where possible using tmpfs is recommended as this whole operation is very IO-heavy.
Aquire a complete (server-side) copy of the CVS repo:
mkdir cvs; cd cvs
mkdir CVSROOT
rsync anoncvs.gentoo.org::vcs-public-cvsroot/gentoo-x86 . -r --stats
WIP: Use ferringb's modifications to cvs2git to transform the repo
git clone git://pkgcore.org/git-conversion-tools
I haven't figured out why it fails for me yet, but that would make the whole thing a lot easier.

Naive cvs2git run on one category to demonstrate that it works in theory:
cvs2git --encoding=utf_8 --fallback-encoding=ascii 
        --trunk-only --blobfile=./blob --dumpfile=./dump 
        --username=derp cvs/gentoo-x86/app-emulation/
This does work, but it's really slow and doesn't do things like rewrite committer names etc.etc.
cvs2svn Statistics:

Total CVS Files:              6569
Total CVS Revisions:         37696
Total CVS Branches:              0
Total CVS Tags:                  0
Total Unique Tags:               0
Total Unique Branches:           0
CVS Repos Size in KB:        37135
Total SVN Commits:           11385
First Revision Date:    Thu Oct 26 15:02:06 2000
Last Revision Date:     Mon Feb 10 06:58:17 2014

Timings (seconds):

   6   pass1    CollectRevsPass
   0   pass2    CleanMetadataPass
   0   pass3    CollateSymbolsPass
1100   pass4    FilterSymbolsPass
   0   pass5    SortRevisionsPass
   0   pass6    SortSymbolsPass
   2   pass7    InitializeChangesetsPass
   2   pass8    BreakRevisionChangesetCyclesPass
   2   pass9    RevisionTopologicalSortPass
   0   pass10   BreakSymbolChangesetCyclesPass
   2   pass11   BreakAllChangesetCyclesPass
   1   pass12   TopologicalSortPass
   3   pass13   CreateRevsPass
   0   pass14   SortSymbolOpeningsClosingsPass
   0   pass15   IndexSymbolsPass
   3   pass16   OutputPass
1121   total
This creates some temporary files which we feed to git fast-import:
$ git init --bare git-test; cd git-test
$ git fast-import --export-marks=../cvs2git-tmp/git-marks.dat <../blob                    
[snip]
$ git fast-import --import-marks=../cvs2git-tmp/git-marks.dat <../dump 
git-fast-import statistics:

Alloc'd objects:      70000
Total objects:        40469 (       253 duplicates                  )
      blobs  :            0 (         0 duplicates          0 deltas of          0 attempts)
      trees  :        29085 (       253 duplicates      26014 deltas of      26667 attempts)
      commits:        11384 (         0 duplicates          0 deltas of          0 attempts)
      tags   :            0 (         0 duplicates          0 deltas of          0 attempts)
Total branches:           1 (         1 loads     )
      marks:     1073741824 (     43450 unique    )
      atoms:           5742
Memory total:          5454 KiB
       pools:          2173 KiB
     objects:          3281 KiB

pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize = 1073741824
pack_report: core.packedGitLimit      = 8589934592
pack_report: pack_used_ctr            =      28072
pack_report: pack_mmap_calls          =          2
pack_report: pack_open_windows        =          2 /          2
pack_report: pack_mapped              =   19836762 /   19836762
And there's our converted category. Runtime of the cvs2git step is ~1200sec = 20min, the git fast-import steps both take ~5 seconds.
There's still a lot left to figure out, but this should be enough information to allow others to attempt to do this reliably.

Posted by Patrick | Permalink

Fri Jan 24 07:00:53 CET 2014

NTP: Working around bad hardware since 1842

After my recent troubles with NTP and excessive time drift things have settled down.

For reasons unknown to me the time drift on the problem server changed from -330ppm to +2.3ppm. I'm not quite sure how to interpret that.
Comparing some other machines I have access to:
Old P4:         -292.238
Dell R510:        12.428
Another R510:     13.232
Random amd64:    -28.438
Another amd64:   -23.323
A newish Xeon:    -7.296
So the general trend seems to be older = more time drift. And machines with "same" hardware appear to have similar drift factors.
The stability of <30ppm means a drift of about 0.1sec/day. Without extra correction that's tolerable (a few seconds a month), but still unsatisfactory.

The old P4 drifting at 300ppm means it'll be getting close to 3 minutes a week away from "real time" - that's enough to cause problems if you rely on it.

I think the lesson in this is "manufacturers use the cheapest they can get away with", so every computer should have a time correction mechanism (NTP, DCF-77, GPS - doesn't matter as long as you correct it). And there's a reasonable assumption that environmental factors (heat, hardware aging, change in the provided voltage, ...) will randomly change the time drift.

And I thought timekeeping was a problem solved two centuries ago ...

Posted by Patrick | Permalink

Fri Jan 17 04:24:40 CET 2014

Unexpected fun with NTP

This morning I had to fix an unexpected dovecot "failure" by restarting it. Apparently it only tolerates time jumps of less than seven seconds.
The trigger of this oopsie is NTP:
Jan 16 23:52:53 stupidserver ntpd[27668]: synchronized to 202.112.10.36, stratum 3
Jan 16 23:52:45 stupidserver ntpd[27668]: time reset -7.732856 s
Riiight. That's not nice, but why does it jump around so much? Looks like the time behaviour worsened over the last days:
Jan 15 19:34:18 stupidserver ntpd[27668]: no servers reachable
Jan 15 19:59:56 stupidserver ntpd[27668]: synchronized to 202.112.10.36, stratum 2
Jan 15 20:06:22 stupidserver ntpd[27668]: time reset +0.533773 s
...
Jan 16 11:47:33 stupidserver ntpd[27668]: synchronized to 202.112.10.36, stratum 2
Jan 16 11:47:30 stupidserver ntpd[27668]: time reset -2.966137 s
...
Jan 16 18:14:28 stupidserver ntpd[27668]: synchronized to 202.112.10.36, stratum 2
Jan 16 18:15:27 stupidserver ntpd[27668]: time reset -4.223295 s
...
Jan 16 23:52:53 stupidserver ntpd[27668]: synchronized to 202.112.10.36, stratum 3
Jan 16 23:52:45 stupidserver ntpd[27668]: time reset -7.732856 s
That's an offset of more than 1sec/h, and that's with ntpd correcting at around 330 PPM. The docs say: "The capture range of the loop is 500 PPM at an interval of 64s decreasing by a factor of two for each doubling of interval." (PPM = parts-per-million)
In other words, if the drift is above 500 PPM it may force a clock reset because it can't drift fast enough. And it looks like this situation was either a failing mainboard RTC clock, or a screwed up ntp server (since it always sync'ed to the same one).

I've tried two things to avoid this time skipping:
1) Change the ntp servers used to something more "local" - the global pool.ntp.org may not be as reliable as servers geographically close you
2) Remove the drift file to force the system to re-learn

The results, at first glance, look promising:
Jan 17 10:48:37 stupidserver ntpd[3059]: kernel time sync status 0040
Jan 17 10:52:55 stupidserver ntpd[3059]: synchronized to 202.120.2.101, stratum 3
Jan 17 10:52:50 stupidserver ntpd[3059]: time reset -5.023639 s
Jan 17 10:57:54 stupidserver ntpd[3059]: synchronized to 202.120.2.101, stratum 3
Jan 17 11:01:08 stupidserver ntpd[3059]: synchronized to 202.73.36.32, stratum 1
Jan 17 11:05:34 stupidserver ntpd[3059]: kernel time sync enabled 0001
So after an initial 5-second skip it managed to sync twice without abnormal drift. Let's hope that it's going to stay sane ...

Posted by Patrick | Permalink

Thu Jan 16 07:36:03 CET 2014

EAPI usage in tree

Total number of ebuilds: 37807

EAPI 0:  5959  15.78%
EAPI 1:   370   0.98%
EAPI 2:  3335   8.82%
EAPI 3:  3005   7.95%
EAPI 4: 12385  32.76%
EAPI 5: 12746  33.72%
That looks quite good: EAPI5 has grown very well, EAPI1 is almost gone.

EAPI0 is still needlessly common, and EAPI 2+3 should be deprecated.

Update: Now running as a cronjerb, Output here, History here

Posted by Patrick | Permalink

Sun Dec 15 12:39:35 CET 2013

Standardization: How did things go so wrong with clothes?

In the last weeks I've been trying to adapt to the Cold Season by aquiring new clothes. What used to be annoying has now gone into full retardation: It's become impossible.

So, here's the confusing thing: Clothes come in different Sizes that are supposed to be at least somehow mildly Standardized. Thus I'd be on the lower end of the M classification, with an occasional S size fitting better.
Pants come in imperial units: Length and waist circumference. This makes it trivial for me to predict that a 32/32 will fit me as I am just about that size.

Soooooooo. Uhm. The last pants I bought were a 34/32, and they are *SMALLER* than my older 32/32. So they are too small by around 10%. Bonus: They are "wide", which means they are skintight - apparently pants are now made for stick people. I have no idea how a 32/32 is supposed to fit anyone, you'd need to be extremely thin and have a huge gut for that size to add up to something reasonable with the way they are cut.

Right, so what about other things? I now have a long-sleeve shirt size XL that is extremely tight. It is smaller than my old M shirts. By that logic I am now size XXL+, which means most shops don't even carry my size - I'm neither tall nor heavy, it appears that stores have shifted to servicing dwarves and fairies. (For reference: My BMI is barely over 20, and at 175cm length I'm pretty much "average" size)

Seriously? What the bleep!
There's no logic to sizes, there's no precision in precise units.
The female clothes sizes are even smaller, so as a female I'd be size XXXL or a bit more. I seriously wonder if that's part of some psychological warfare ...
A few years I used to be extremely thin - to the point where it was difficult to find pants. I was a 28/32 size (which doesn't usually exist in german clothes shops). And the pants for females have that as UPPER size limit, with a 23/32 being the lower end. I can't even figure out how extremely thin and/or malnourished humans would have to be to fit in that.

A big bonus of this random sizing is that I can't predict what size I'll be today, so I can't order things online. Instead I'm supposed to go into retail shops and spend lots of time trying on clothes in the hope of finding things that fit me.
Last time I tried that in 5 shops there were no pants cut wide enough for my legs, they all refused to be pulled above my knees. So my motivation to do that dance again is very small...

It'd be, like, really awesome, if clothes retailers could care more about their audience and produce clothes in human sizes. I'd appreciate being considered normal again. A huge bonus (which would be extra awesome) are Jeans that are made for legs, so that I could actually buy Jeans that don't have to be re-tailored completely to approximate my human body shape. Like, I have a Gluteus Maximus that is attached to me, so maybe have pants acknowledge that.

Oh, also. Please. I'd appreciate shoes in feet-shapes. I haven't bought many shoes in the last years because I haven't found any that fit my feet in ways that are anatomically correct - last purchase was a EU size 46 that is actually too long, but at least it doesn't try to rearrange my toes into a triangular shape.
(Funny thing there: ASICS has a "2E" series of extra wide shoes. The old ones, size 46.5, just barely fit me well. The new ones, ahem, are cut more narrow, which slightly makes the label absurd because now they are more like ... moderately wide. Even there the shrinkeration is going on at a nice pace, soon I'll reach size 52 because things become smaller, and then I'll just completely give up and have shoes hand-made.)

So ... standards, good thing; but without enforcement it ends up as an unpredictable mess that makes life more difficult. Fixed units: Supposed to be Fixed and not dynamic. Sanity optional.

Posted by Patrick | Permalink

Wed Dec 4 07:07:33 CET 2013

Histograms of commit activity

Commit histograms for all gentoo devs, rough first draft.
It doesn't handle renames/dev moves yet (fauli/opfer are two histograms) and since it contains manifest commits it's a bit more than it should be, but as a comparative graph it should still amuse.
Thanks to autoscaling every graph should have a max width of ~80 stars, so please don't try a direct comparison.

EDIT:
Commit time-o-grams showing the frequency for each hour of the day

Posted by Patrick | Permalink