movement is currently certified at Master level.

Name: John Levon
Member since: 2000-04-14 14:10:55
Last Login: 2011-12-23 13:10:45

FOAF RDF Share This


See the project descriptions below, and my homepage, for a description of what open source stuff I'm involved in.

movement @,


Recent blog entries by movement

Syndication: RSS 2.0

PCID support on Illumos

I joined Joyent at the start of the year while Meltdown was breaking news; it was certainly an "interesting" time to start a new job. Luckily by my first week, Alex and Robert had pretty much figured out how the changes should look and made good inroads on the implementation. So I began working with Alex on his KPTI trampoline code (mainly involving breaking it with my old friend KMDB). I also picked up the PCID work which I describe here.

As you can probably tell from Alex's blog post, Meltdown is unusual for a security issue: aside from the usual operational pains of any security patch, the fix itself involved some pretty significant code changes to the low-level core of the kernel.

There's also another potential impact, and that's performance. While the actual overhead is heavily workload-dependent - and some of the reports out there seem pretty alarmist - having to switch page tables (i.e. reloading %cr3) on every kernel entry and exit has a non-trivial impact on system call cost. Nor can we keep the kernel state in the TLB. Previously, we would set PT_GLOBAL on kernel mappings so they're not flushed across a %cr3 reload, but as the CPU would happily use these TLB entries to speculate into the kernel, we must flush them.

The good news is that there's a CPU feature on reasonably recent Intel CPUs called Process Context IDs. This lets you load the lower bits of %cr3 with a small integer value. This ID is used as a tag in any TLB lookups or fills. This feature is somewhat similar to ASIDs seen on other architectures, with one notable difference. The PCID applies to TLB state implicitly, that is, there's no way to say "load from memory using this ID" in ddi_copyin() and the like.

One way of using PCIDs is to associate an ID with a struct as: that is, each time we load a process's address space into the HAT, we will use a specific PCID for it, and avoid having to flush the mappings for the previous processes. This isn't really a viable option for Illumos, though: if nothing else we suspect that the additional shootdown flushes needed (since we'd maintain TLB entries even after switching away from a process's struct as) would counteract any performance gain.

Instead we define two fixed PCID values. PCID_KERNEL, defined as 0 mainly to keep the boot process simple, is used for the kernel %cr3. Thus, all TLB loads while in the kernel will be tagged with this value. PCID_USER is used when in userspace. Now, when we switch %cr3 on kernel entry or exit, we can do a non-flushing load. This lets us keep both the kernel and the userspace mappings around across kernel/user transitions.

When we do need to invalidate TLB entries, though, things are now slightly more complicated. We are by definition in the kernel (and hence using PCID_KERNEL), but we have to account for memory addresses below USERLIMIT. In this case, we have to flush both PCID_USER (for anything that ran in user mode) and PCID_KERNEL (for any accesses the kernel may have made such as with ddi_copyin()). hat_switch() is also a little more complicated. As the %cr3 load there is non-invalidating, we have to explicitly flush everything if we're switching away from a non-kas HAT, to clear out now-stale user-space mappings. (Note that this has always been done eagerly on Illumos, even when switching to a kas HAT).

The INVPCID instruction is what enable us to flush PCID_USER while in the kernel. Unfortunately, support for INVPCID came quite some time after PCID itself. On such systems, we have to emulate, and the only way Intel gives us to do this is to load the ID into %cr3 before invalidating the TLB entries. We don't want to "pollute" PCID_USER with any extraneous kernel mappings, so this means we need to switch to the user page tables when loading PCID_USER. But, remember, KPTI requires us not to have kernel text (or stack!) mapped into these page tables. So we have to first make sure we're in the trampoline text before doing the invalidations: see tr_mmu_flush_user_range.

For those interested, Alex posted a draft webrev of the PCID changes.

Syndicated 2018-02-26 11:32:00 (Updated 2018-02-26 11:32:13) from John Levon

Converting HTML mail via procmail

All the procmail recipes I found on a quick search failed to handle quoted-printable HTML encodings, regularly used everywhere. And those that had quoted-printable examples used tools no longer maintained - such as mimencode.

The solution is to use Perl directly:

* ^Content-Type: text/html;
* ^Content-Transfer-Encoding: *quoted-printable
| perl -pe 'use MIME::QuotedPrint; $_=MIME::QuotedPrint::decode($_);'
| lynx -dump -force_html -stdin
| formail -i "Content-Type: text/plain; charset=us-ascii"

Syndicated 2016-11-03 20:27:00 (Updated 2016-11-03 20:33:04) from John Levon

10 Jan 2014 (updated 29 Oct 2017 at 02:22 UTC) »

Ripping vinyl on Linux

 I've been ripping a lot of stuff from vinyl to FLAC recently. Here's how I do it.

I have an Alesis I/O 2, which works well and seems fairly decent quality.

First, most important, step, is to stop trying to use Audacity. It's incredibly broken and unreliable. Go get ocenaudio instead. It's fairly new, but it works reliably.

After monitoring your levels, record the whole thing into ocenaudio.

First trim any obviously loud clicks such as when landing the needle. ocenaudio doesn't seem to have a "draw sample" function yet, the only thing I miss from Audacity, but deleting just a few samples is usually fine.

Normalise everything.

Then select a whole track using Shift-arrows (and Control to go faster). Press Control-K to convert it into a region, and name it if you like.
You'll see references to using zero-crossing finders to split tracks. This is always a bad idea - it's simply not reliable enough, especially with an old crackly record, isopropyl'd or not.

Zoom all the way out again, make sure the number of tracks is right.

Then File->Export Audio From Regions, making sure that the "separate files" checkbox is set.

Now it's tagging time: run "kid3 yourdirwithflacs". First import from discogs, presuming it has the release (it usually will) File->Import From Discogs. Then click 'Tag 2' in the Format Up part, along with the format you need. Save all those, then use Tools->Rename Directory to rename the containing directory. You're done.  

Syndicated 2014-01-10 11:53:00 (Updated 2016-11-24 13:01:38) from John Levon

31 May 2012 (updated 16 Sep 2015 at 16:07 UTC) »

Recording on Linux with Alesis io|2

A little note for myself: to get low-latency monitoring, and more importantly, record at the right rate, you need to set the Configuration-Profile to "Digital Stereo Input" in pavucontrol!

Update: you also need this in ~/.pulse/daemon.conf :


Another update: PA/ALSA often seems to forget the sensible default devices, and ocenaudio starts
trying to record from the monitor devices. Solution seems to be to run pavucontrol, start ocenaudio recording, and change the drop down box to select io|2 Digital Stereo.

Syndicated 2012-05-31 15:39:00 (Updated 2015-09-16 15:26:44) from John Levon

PayPal idiocy

This is unbelievably stupid of Paypal. I just got this email from them:

vinyl tap records would like you to use PayPal - the safer, easier way to pay and get paid online.
To send vinyl tap records your payment and see the details of this invoice, copy and paste this link into your web browser:

So much for "never click a URL in email". Even worse, if you log in separately, the request is not visible anywhere. Morons.

Syndicated 2011-02-22 11:17:00 (Updated 2011-02-22 11:18:56) from John Levon

290 older entries...


movement certified others as follows:

  • movement certified davej as Master
  • movement certified redi as Apprentice
  • movement certified riel as Master
  • movement certified prumpf as Master
  • movement certified quintela as Journeyer
  • movement certified JMarc as Master
  • movement certified tim as Journeyer
  • movement certified fenrus as Journeyer
  • movement certified Excalibor as Apprentice
  • movement certified JALH as Apprentice
  • movement certified baruch as Journeyer
  • movement certified daniels as Apprentice
  • movement certified nzkoz as Apprentice
  • movement certified jdike as Journeyer
  • movement certified akpm as Master
  • movement certified dwheeler as Journeyer
  • movement certified hch as Master
  • movement certified wli as Master
  • movement certified zwane as Journeyer
  • movement certified erikm as Journeyer
  • movement certified DV as Master
  • movement certified xmldoc as Journeyer
  • movement certified psg as Journeyer
  • movement certified jfreak as Apprentice
  • movement certified dak as Journeyer
  • movement certified kerz as Apprentice
  • movement certified KaiL as Apprentice
  • movement certified infodragon as Apprentice
  • movement certified ndw as Master
  • movement certified braden as Journeyer
  • movement certified graydon as Journeyer
  • movement certified LotR as Journeyer
  • movement certified phe as Journeyer
  • movement certified fscked as Apprentice
  • movement certified jeffalo as Apprentice
  • movement certified madhatter as Apprentice
  • movement certified rcastro as Journeyer
  • movement certified ahu as Journeyer
  • movement certified Mmarquee as Journeyer
  • movement certified lerdsuwa as Journeyer
  • movement certified movement as Journeyer
  • movement certified lukeg as Journeyer
  • movement certified mulix as Journeyer
  • movement certified amnesia as Apprentice
  • movement certified hald as Apprentice
  • movement certified hussein as Apprentice
  • movement certified dberlin as Master
  • movement certified returnoftheredi as Apprentice
  • movement certified linuxata as Master
  • movement certified bcully as Journeyer
  • movement certified richlowe as Journeyer

Others have certified movement as follows:

  • rbrady certified movement as Apprentice
  • davej certified movement as Master
  • redi certified movement as Master
  • JMarc certified movement as Master
  • Excalibor certified movement as Journeyer
  • JALH certified movement as Apprentice
  • baruch certified movement as Master
  • daniels certified movement as Journeyer
  • fxn certified movement as Journeyer
  • hch certified movement as Journeyer
  • mulix certified movement as Master
  • erikm certified movement as Journeyer
  • zwane certified movement as Journeyer
  • minami certified movement as Journeyer
  • sits certified movement as Journeyer
  • braden certified movement as Journeyer
  • deepfire certified movement as Master
  • ladypine certified movement as Journeyer
  • phe certified movement as Journeyer
  • prla certified movement as Master
  • fscked certified movement as Master
  • madhatter certified movement as Journeyer
  • rcastro certified movement as Journeyer
  • AlanShutko certified movement as Journeyer
  • shlomif certified movement as Master
  • pasky certified movement as Journeyer
  • richdawe certified movement as Master
  • asanchez certified movement as Master
  • lypanov certified movement as Master
  • amits certified movement as Master
  • alejandro certified movement as Journeyer
  • albert certified movement as Master
  • error27 certified movement as Master
  • lerdsuwa certified movement as Master
  • movement certified movement as Journeyer
  • hussein certified movement as Master
  • amnesia certified movement as Master
  • miconof certified movement as Apprentice
  • returnoftheredi certified movement as Journeyer
  • nconway certified movement as Master
  • linuxata certified movement as Master
  • ncm certified movement as Journeyer
  • graydon certified movement as Master
  • bcully certified movement as Journeyer
  • murajov certified movement as Master
  • ittner certified movement as Master

[ Certification disabled because you're not logged in. ]

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

Share this page