DabbleE

Repost: Freshman, be ambitious!

2020-03-27T19:00:01-07:00

Another repost in the same vein as the previous one. This post came from April of 2013, just as I was about to finish my undergraduate degree. Foolish me would not begin to understand the frustrations that awaited me in the coming years as I prepared to start my Ph.D.

I would still hold to my advice at the time though. I did not fully know the full value of the social interactions I was recommending at the time, and in retrospect I would argue they are more critical than the standalone ambition the title implies, but the message is no less valuable than it was 8 years ago.

In May of this year, I will graduate with my undergraduate degree in Electrical Engineering. After four years I feel that I’m far wiser than I was four years ago, but I’m painfully aware of how much more I need to learn about the world. That being said, I now realize how many opportunities were available to me that I simply did not know existed.

As such, the simplest way I can put everything, is this: freshman, be ambitious.

Do not underestimate the value of getting involved with a group that does things you cannot yet comprehend your freshman year. My freshman year I tried to get involved with the universities local Linux Users Group and IEEE out of the gate. Currently, I am participating with neither group, but I wouldn’t discount the interactions I gained from starting to look into these to organizations for a minute.

It was due to my interactions with the LUG that I met the advisers and officers of our local Amateur Radio Club. As it turns out W7YH is one of the oldest ARC’s in the nation with a history spanning just over 100 years! At the time amateur radio just seemed like a passing interest, but it is because of my activity in that club that many of the opportunities I have available to me now became available. My interest in communications got me started with microcontrollers and actually building circuits by hand a year earlier than the first of my classmates. This in turn would land me my first engineering job/internship with Digilent Inc.

Two years after I first was introduced into the WSU ARC, I found myself forced into an officer position to keep the club alive. Two years later I am proud to say that I presided over W7YH for it’s 100th and 101st years. My involvement with the club introduced me to enumerable ideas and concepts related electrical engineering: from circuit design, filter design, signals analysis, antenna analysis, RF safety–the list goes on. While personally I always felt that the introduction.

From here the actual value of your experience with a club should emerge. While I personally felt that my knowledge of these subjects was inadequate for the type of work I wanted to do, what I quickly found was that my knowledge would far surpass what a typical undergraduate would ever learn. On three occasions I’ve found myself teaching organized classes on the material I learned, and I ended up personally mentoring dozens of students in courses related to the material.

don’t want this to sound like a bland exposition of my college experience, but the point I’d like to stress is that everything that I’ve found myself doing today, was as a congruence of a single meeting with two individuals three and a half years ago. Had I not met K7LL and KE7URN, I could not imagine where I would be today. The thing to remember, is that what you gain will be dictated by who you meet, and how you interact with them. Little of what I learned came directly from these two individuals, but was through my correspondence and interaction with them that I found myself pursuing the things that would drive me to where I am today.

My advice for Freshman a few months away from their first year at college is this:

Find a club, group, or organization related to your interests and stay involved. This usually doesn’t mean actually doing anything, but take advantage of any social interaction you can with this group. The true value you gain from a group is rarely from working directly in a project they have organized, but rather it is being comfortable enough interacting with them to use them as a resource to pursue what you want to do. Even if they cannot provide much help for you, if you interact with a larger group in this way, you may be an aid to someone else.

The second thing that you’ll find from this, is it opens opportunities to garner help from students outside of your club. The senior members of your club are typically familiar with the leaders or senior member of other clubs. The WSU ARC, LUG, and Aerospace clubs have this familiarity to a certain extent. Often our work will draw us outside of our comfort zone, so but our personal relations with the members of other clubs makes it easier to bring other students into contact with resources that they will need to pursue their own goals.

Do not underestimate the value of the relations you form, for it is through them that your future will truly be shaped.

Repost: Understanding your knowledge

2020-03-27T18:00:01-07:00

I started wading through some of the digital cruft of my life today, and stumbled on this bit of writing back from March 2012. I was a Junior in college at the time; full of ambition, hope, and enjoying the challenges and new ideas I was being exposed to. Now at 30, it strikes me how much 21 year old Luke already had figured out, even though he may have not realized it. As I started reading I expected to find myself cringing at the thoughts of my younger self, yet I instead found myself surprised and being proud of younger self.

Re-reading this post I found that my opinion has largely remained unchanged. My personal biases have indeed changed, and I’ve had the misfortune of seeing exactly what I described in this post grind up engineers. Some of the opinions I had when I wrote the piece I would disagree with today, but the message rings true to me in spite of the opinions a younger me once had.

The post below is unaltered from it’s original writing 9 years ago sans updating the link in the original post to an internet archive mirror.

Enjoy.

Understanding your knowledge

It is very common for those of us who learn on a certain tool, or who are taught a certain methodology to become very attached to the nuances and the specifics of that tool or method. Obvious examples come in the form of programing language preferences, where a software developer might swear by C# or C++ when a more lightweight language such as ANSII C might fill the role. There is a rather well-known discussion thread from the Linux kernel development community along just these lines, in which Linus Torvalds chews out a prospective developer who was entrenched in the ideas used in most C++ development.

More often than not, this mindset is worsened by a poor understanding of alternatives or a lack of appropriate experience to effectively judge a new tool. There is nothing unusual or inexcusable about this mindset. Indeed I should expect this type of view to develop quite naturally among engineers and hackers; it is only to be expected. As we develop projects we work with a certain conviction towards its completion, and if we are haunted by the notion that our project is inherently flawed we will lose interest and become unsure of our work.

Problems begin to arise when one fails to understand the limitations of his or her knowledge, when presented with a possible alternative. Engineering as well the vast majority of design, requires decisions to be made on incomplete and imperfect information; but with this power to act on assumption, we accept the responsibility that we might have to reevaluate decisions made not just on a project, but on the knowledge and skills we have pursued in our lifetime.

Consider something as common place as a keyboard layout; QWERTY is the de-facto standard keyboard layout for nearly every latin text input device on the planet, yet various internet communities are continually badgered by users who insist that alternative layouts such as Dvorak are superior for one reason or another. It is easy to find such an argument as quite silly; unlike a software language or considering a brand of some general purpose devices (such as AVR or a Microchip brand micro controller), a keyboard layout introduces no fundamental difference to how we interact with a computer. This argument persists not because one layout is superior to another (a statement I feel I can safely make being a typist of Dvorak for more than two years), but rather because typically both parties are unwilling to consider that a fundamental tool they have been using, i.e. the keyboard, might have been better developed had they taken a different starting approach to it.

Remember you when design anything that you may have taken a flawed approach. Do not fear this flaw but do not let your convictions make you cling to this flaw. Simply note that you may have made a mistake, and that you might end up repeating some of your work over again in the future because of it.

On a purely personal note: I should note that while I’ve been typing in Dvorak for more than two years now, I have had no great life changing events because of it. I use Dvorak on my primary computer and Qwerty when I’m using someone else’s computer. Contrary to common claims, Dvorak does not increase your typing speed, it does not change your world view, and it does not grant you some sort of amazing skill set. It’s best use is as a neat trick to show your friends or a conversation starter, little else.

My Experience snapping

2020-02-21T00:00:01-08:00

poking at packaging

After hearing about the exciting new poke project a few months back, I got deep into the weeds of packaging again. At the time of writing the authors of the poke project have just tagged poke 0.90 as a pre-release of the tool.

For the past few months I’ve wanted to work with poke, but it has some extremely recent dependencies that have made building and using it difficult for me. I don’t really want to just dump newer versions of binaries into my filesystem and hope everything keeps working (though it might be okay to do so), but I still wanted to experiment with the Poke language and see what it is. Briefly though, poke is hot shit for decomposing and working with binary structured data files. I’m going to leave it at that, and say if you’ve ever been frustrated modifying any binary file manually with some sort of hex editor and some manually hacked C or Python, you should take a look at the magic of poke. The Applied Pokology blog is a good start, as are some of the cool talks (at Kernel Recipes 2019 and GNU Tools Cauldron 2019) the project’s lead author has given of late.

Why Snaps?

Canonical’s snaps have interested me precisely because I hoped they would let me sidestep the biggest pain points I have with standard packaging. I hate the magic build scripts that make up the debian packaging system, and I’m routinely confounded by attempts to dig out the documentation as to what they do and why they fail to reproduce packages that are written by others.¹

Secondly, I wanted a way to isolate my build system and the runtime requirements from my host system. Having tried and failed to build legacy versions of Corebird back when it required newer versions of GTK than those that shipped with Ubuntu, I don’t want to maintain parallel build trees and library paths ever again.²

Snaps advertize themselves as a solution to that problem. The tutorial suggests the build environment is fully contained, and the build commands are automated provided you don’t need any dramatic deviations from the typical “build from source” commands (e.g. ./configure, make, sudo make install). More to the point, a few Canonical employee’s have been preaching the glory of snaps for simple little projects on a few podcasts I picked up during COVID.

So here I am, I’ve got a fancy new project, that requires some fancy build deps, and hopefully another reason to look at snaps and see if they can let me use the tool.

Results and Observations

Simply put, I nearly got it. The poke snap has rudimentary functionality, but getting here was far from painless.

My biggest gripe about trying to learn how to build a deb is the magic build tools scripts that convert the magic debian metadata directory into the archived bundle. Snaps are somewhat better in this regard, but you wouldn’t realize that unless you wandered off to the Multipass web site to learn about the multipass tool. Multipass is a qemu VM manager focused on Ubuntu images. When you call snapcraft on a project, mutlipass builds a VM behind the scenes and runs through a series of build phases as specified by your snapcraft.yaml file.

Without knowing this the process of trying to build a snap is one of confounding trial and error when addressing anything remotely nonstandard in the build process. Worse still, until very recently every trial and error attempt would start back over from scratch. If you found you had some minor flag problem in the last phase of the build, you could spend hours rebuilding the entire project and pulling down source trees from the internet. Thankfully this is no longer the case with the core20 base image. All that is to say, major pain points in the build process existed in the past, but are much better today.

Recommendations for would-be snapcrafters

Dig into the build process details.
- Snaps are built of parts, and the parts are each built in isolated directories inside of the VM performing the build.
Wade through documentation like Wikipedia
- The documentation on snapcraft.io totally sidesteps the complexities of the build tree, and hides a few magic nuggets (like being able to call snapcraft --shell to inspect a snap build process) in seemly random places. The best solution to this is unfortunately to pour through links, and never close a tab you came from.
Inspect the github
- It’s here. All of the pain of not following the staging can be (slowly) dissected from the various build tools. It’s a far cry from formal documentation, but the various tooling scripts spit out bash commands at the bottom layer. Everything you battle in the build process usually comes down to one or two python scripts in the snapcraft repo.
- As an example, recently the autotools plugin gained support for ./bootstrap commands. It does not support the build flags I needed for poke, but I now know exactly what modifications are expected.
Standard Maintainer build rules apply.
- For example, if you’re building a library, install it into /usr rather than /usr/local. The later will build the tool but not let you use it as an internal dependency for other parts of the build. Both the bison and gettext dependancies in my poke snap are simple examples.
Confinement is painful.
- Most of my time in finishing the poke snap has been fighting confinement. I want all of the normal $HOME/.<tool>rc and $HOME/.config/<tool> resources to be exposed to the snap. That linkage is broken by default. Confined processes internally see $HOME as a common user location and files must be manually linked.

Final Thoughts

I found myself constantly asking basic questions about the build process: “Where does the multipass configuration live? Why do the various VMs appear to stomp on each other in the build process? Can I give a VM extra RAM or CPU horsepower? Can I reconfigure a VM post spin-up? Can I rename a VM?” Some of these I still don’t have a clue about. These paper cuts will continue to impede adoption of snaps even if they’re a great solution to a common set of problems. That said, snaps are quite close to being ready for primetime. The documentation leaves a lot to be desired, but the paper cuts these days are just that, paper cuts. They aren’t debilitating wounds.

Having never tried to bundle a Flatpak or Appimage I don’t have full insight, but I don’t think these particular pain points (or anything else I’ve mentioned) are unique to snaps. Flatpack has confinement problems like snaps, it has a confounded build system, and I suspect it has similarly sparse documentation.

I’m left in resignation that packaging will always suck to learn, but on the upside, soon a really cool binary hacking tool will be accessible to users of likely quite old Ubuntu releases.

SOAPBOX: has anyone who isn’t a kernel maintainer ever tried to rebuild the kernel package? How many build failures did you go through before you finished? Do you have any faith in the final output being “correct”? ↩
See Cawbird for the project fork since Corebird’s demise. ↩

Waiting for processes to exit

2018-06-02T05:00:01-07:00

TL;DR: waitdone source.

For quite a while when using Linux I found myself forgetting to chain commands when I wanted to. Using && is simple enough, but I found when I was first getting started with Linux it really didn’t help me very much. I rarely thought about commands that I wanted to run in sequence, and most commands were so fast that I would be spending more time typing them than they would run for anyway. What was to be gained by spending time thinking about it.

As I use Linux more and more, and more importantly found myself using it in ways that pushed the hardware I was using, the value started to be more apparent than an abstract trick. When extracting multiple archives fragmentation and file disk seeking is reduced by running two tar commands in sequence rather than in parallel, or running a second instance of a command that’s resource intensive (for example ffmpeg) is easier to handle after a previous instance was running. The problem of course being that if I started one of these commands, and forgot about the next command, to chain them together required me to stop the first to modify the command.

This started as a frustration, but after enough instances I grew tired of either stopping the original command, or randomly waiting to start the second by hand, and wrote a little bash script I called waitdone. The principle of it’s use is simple enough, suppose I have two ffmpeg calls and I want the second to start after the first has finished. In terminal session A I first make my call to ffmpeg to get it started quickly.

$ ffmpeg -i <input file 1> <some crap> <output file 1>

Now in a second session, I call waitdone on ffmpeg in place of chaining to ffmpeg calls, then chain waitdone with ffmpeg.

$ waitdone ffmpeg && ffmpeg -i <input file 2> <some crap> <output file 2>

The first terminal will run normally, while the second will display a sleep message until all running instances of ffmpeg exit. At that point it will the waitdone will exit cleanly, passing flow onto the second ffmpeg call.

An alternate handy way to use this is in conjunction with something like tmux and the notify-send alias in Ubuntu systems. If ffmpeg (or some other long process) is left in the background, waitdone can be chained into notify-send to visually alert the user when the process exits.

Hopefully someone will get as much use out of this as I have. Source code is below for the interested. -Luke

GIST

Building in-tree Kernel Modules in Ubuntu

2018-04-28T15:30:00-07:00

As I noted earlier this year, I’m currently fighting with the LucidSound LS30 headset and some specific oddities that it has. The whole process has lead me into a mountain of learning about the Linux kernel, it’s development and the deployment of software in a broader context. The personal problem of course being this will do little to help me professionally if at all—I can code, but I don’t want to make it my primary job. Even so, it has lead me to some rather humorous comments with in the Linux kernel source, such as this nugget:

linux-4.13.0/sound/usb/card.c:289

/*
 * Firmware writers cannot count to three.  So to find
 * the IAD on the NuForce UDH-100, also check the next
 * interface.
 */

Firstly let me take a brief moment to describe some of the details I’ve learned related to my specific device, then I’ll get into the core trick I finally figured out to making debugging a bit easier.

LS30 Oddities

The core problem I can discern in the LS30 lies in the specifics of how it reports it’s functional behavior, and the specifics of how the Linux USB Sound subsystem parses this description. At this point root causes are largely speculative, and until recently I had not nailed down a way to quickly rebuild modules to help decipher what the kernel was doing. Short of hopping to a Linux distribution distributed with a nearly vanilla kernel, I was somewhat bound by the documentation currently available.

My suspicion is the majority of USB sound card devices (e.g. adapters, DACs, and USB headsets) make use of a single interface descriptor containing a set of hardware control descriptors. The control descriptors are a series of simple structures that indicate which interfaces (in the software sense) on a USB device correspond to various physical functions. They provide, among other things, a category based qualitative description of the specific interface’s function (i.e. a headset, headphones, speakers, etc.), audio flow direction information (input/output), channel count, and some more technical information as well. Some descriptors also describe software interfaced controls such as volume and mute.

The LS30, being a “feature rich” device, contains a multitude of audio interfaces. These various interfaces are meant to allow a user to split game chat and game audio to the headset. This would allow them to balance game volume against the rabid yelling of the other players. The design idea not problematic on it’s own, but the specific implementation in the device description splits the interface descriptor into two sets (found below for the curious). This is where the Linux kernel’s current idiosyncrasies come into play:

linux-4.13.0/sound/usb/card.c:606

/*
 * For devices with more than one control interface, we assume the
 * first contains the audio controls. We might need a more specific
 * check here in the future.
 */

Well good sir, future is here. And it looks like we’re going to have to write it.

The problem observed is only some of the endpoints actually show up. As indicated by the comment above, the Linux kernel only investigates the grouping of interfaces described in the first interface descriptor. More over, it doesn’t even appear to validate that there are any control descriptors in the first descriptor. The groupings in the LS30 are organized such that the headset’s stereo audio output is in the second grouping of descriptors. This results in the kernel only passing the audio endpoints in the first set, rendering the headset’s functionality somewhat limited.

The fix is likely a one of validating that no deeper problem exists, then expanding the code to inspect and configure interface descriptors. A simple approach in description, likely somewhat painful for an engineer with no history of contributing to a project the scale of the Linux kernel. However, for simple validation I can likely assume index 3 (rather than zero) is the correct point to start parsing interfaces, and verify the kernel’s response.

Building in-tree modules

The key bit of work that will finally let me do my testing is deciphering the Debian/Ubuntu kernel build system, to let me quickly rebuild modules with the proper vermagic string into the kernel. In short, when building kernel modules, the stock build system inserts a signature into modules indicating which kernel version they are designed to operate against. This is fine in the context of preventing users from mixing and matching unrelated binaries¹, but it creates a few headaches for those of us tied to the Ubuntu/Debian kernel numbering schemes. On my current Ubuntu 17.10 based system, the vermagic string should be 4.13.0-39-generic SMP mod_unload, but if I were to blindly pull the current kernel source, let the build system apply it’s various patches, a plain build would spit out 4.13.16 SMP mod_unload. A subtle difference, but enough to prevent insmod from letting me load a modified module without forcing an override.

A slow but valid approach (from the Debian documentation) would be to rebuild the entire kernel every time I made a small change to a source file, and potentially wait hours on end. Being who I am, I have little patience for something being slow for no good damn reason. At the same time, I’m also torn by neglecting the hidden magic that I might be missing by manually inserting changes into the source tree Makefile to produce the correct vermagic string.

As it turns out, the trick to produce fast builds with the correct string is relatively simple. The Debian build system will build into a sub-directory for each kernel image it generates. This is readily observable when watching where results are placed when running a full build. What is less obvious, is the build scripts modify the stock Makefile and insert it into these build directories, linking back to the top level source. Simply moving into this directory lets us reuse the traditional kernel build commands.

Reproducible Steps

First fetch the source of the project, and insert your existing system’s configuration into the build tree.²

$ apt source linux
$ apt build-dep linux
$ cd linux-*
$ cp /boot/config-$(uname -r) ./.config

Next, we call the Debian workhorse scripts that do the preliminary configuration and kick of a build of the kernel image. In the example I’m assuming the kernel you want to build is the *-generic variant rather than something like the *-lowlatency variant.

$ fakeroot make -f debian/rules clean build-generic

From here we can actually kill the full build process. The Debian rules scripts will start to build everything, but early in the build process it finishes configuring a dependent build environment we can tinker with. I’ve attempted to nail down the specifics of the dependency flow, but in the package maintainers desire for robustness and flexibility we have lost obvious coherency³.

From here we can hop on over to the dependant target, and build just the modules we want to work with.

$ cd debian/build/build-generic
$ make M=sound/usb

Check the specific module and…

$ modinfo ./sound/usb/snd-usb-audio.ko | grep magic
vermagic:       4.13.0-39-generic SMP mod_unload

Payday! Now, any changes to source files are properly recognized, and calls to make will recompile only the subset of object files required to generate an appropriately tagged kernel module.

In retrospect I’m quite bothered that it took me so long to decipher what turned out to be somewhat simple procedure, but to date I haven’t found anyone with a similar problem related to either the build process or similar device problems, so I’m similarly unsurprised wandering in the dark was fruitless for so long.

Enjoy.

Appendix

Below is an trimmed and annotated tshark dump of a USB transaction against the LucidSound LS30 headset in it’s final state as referenced above. All items described in the interface descriptor indexed 3.0 are unavailable to the system as of writing.

Frame 6: 473 bytes on wire (3784 bits), 473 bytes captured (3784 bits)
USB URB
CONFIGURATION DESCRIPTOR
👉INTERFACE DESCRIPTOR (0.0): class Audio👈
    Class-specific Audio Ctl. Intf. Desc.: Header Descriptor
    Class-specific Audio Ctl. Intf. Desc.: Input terminal descriptor
    Class-specific Audio Ctl. Intf. Desc.: Feature unit descriptor
    Class-specific Audio Ctl. Intf. Desc.: Output terminal descriptor
    Class-specific Audio Ctl. Intf. Desc.: Input terminal descriptor
    Class-specific Audio Ctl. Intf. Desc.: Feature unit descriptor
    Class-specific Audio Ctl. Intf. Desc.: Output terminal descriptor
INTERFACE DESCRIPTOR (1.0): class Audio
INTERFACE DESCRIPTOR (1.1): class Audio
    Class-specific Audio Strm. Intf. Desc.: General AS Descriptor
    Class-specific Audio Strm. Intf. Desc.: Format type descriptor
ENDPOINT DESCRIPTOR
    Class-specific Audio Strm. Endpt. Desc.
INTERFACE DESCRIPTOR (2.0): class Audio
INTERFACE DESCRIPTOR (2.1): class Audio
    Class-specific Audio Strm. Intf. Desc.: General AS Descriptor
    Class-specific Audio Strm. Intf. Desc.: Format type descriptor
ENDPOINT DESCRIPTOR
    Class-specific Audio Strm. Endpt. Desc.
👉INTERFACE DESCRIPTOR (3.0): class Audio👈
    Class-specific Audio Ctl. Intf. Desc.: Header Descriptor
    Class-specific Audio Ctl. Intf. Desc.: Input terminal descriptor
    Class-specific Audio Ctl. Intf. Desc.: Feature unit descriptor
    Class-specific Audio Ctl. Intf. Desc.: Output terminal descriptor
INTERFACE DESCRIPTOR (4.0): class Audio
INTERFACE DESCRIPTOR (4.1): class Audio
    Class-specific Audio Strm. Intf. Desc.: General AS Descriptor
    Class-specific Audio Strm. Intf. Desc.: Format type descriptor
ENDPOINT DESCRIPTOR
    Class-specific Audio Strm. Endpt. Desc.
INTERFACE DESCRIPTOR (4.2): class Audio
    Class-specific Audio Strm. Intf. Desc.: General AS Descriptor
    Class-specific Audio Strm. Intf. Desc.: Format type descriptor
ENDPOINT DESCRIPTOR
    Class-specific Audio Strm. Endpt. Desc.
INTERFACE DESCRIPTOR (5.0): class HID
HID DESCRIPTOR
ENDPOINT DESCRIPTOR

i.e. doing really stupid things ↩
Listing edited 2018-06-02. I forgot to include the command to install build dependencies. ↩
If you, dear reader, know out how to fix this without killing or finishing the full build, let me know! ↩

Generating cookies.txt in Firefox

2018-04-15T13:10:00-07:00

The browser team at Mozilla have been working their buts off the last year in particular bringing what I would generally regarded as a better browsing experience. If you haven’t taken a look at Firefox in the last year or so (or you left Firefox because it was slow and Chrome was once upon a time the speedy, new, hotness everyone was after), I strongly recommend you give it another look.

As part of the work in modernizing Firefox, a major rewrite of the entire rendering and parsing system broke all legacy extensions. The replacement (web extensions) is probably a smarter way to go for compatibility and robustness, but a number of smaller unmaintained extensions have been left in the dust. The multitude of extensions to produce a cookies.txt file are what concern me in particular. The cook

When legacy extensions finally bit the dust sometime last year, I quickly wrote up a little bit of python to replicate the functionality on Linux systems. I was willing to let this little nugget of Python live in my personal bucket of tools until I started seeing a few bug reports for tools like youtube-dl where developers were complaining about being unable to generate a cookies.txt file. The particular value for these developers is extracting the Cloudflare DDOS prevention cookies for scripts and tools that may need them.

Basic Overview

The script (available here, and further down the page) is pretty straightforward. We do some arbitrary parsing to search for a particular Firefox profile in the .mozilla/firefox folder of a user’s home directory. This is where your profiles (for most people only one) reside. A simple wildcard glob selects all profiles that end with the name default, containing a cookies sqlite database. Provided everything checks out, all cookies matching a few particular hosts are extracted and dumped to a cookies.txt in the directory the script is executed in.

While it isn’t extremely pretty, and it could readily be expanded with a proper getopt front end to ease selecting profiles and such, it’s structurally sound enough to tinker for anyone impatiently Googling to figure out “HOW DO I GENERATE COOKIES.TXT” without getting hung up on the details.

Enjoy.

GIST

Debugging in the Linux Kernel

2018-01-15T16:10:00-08:00

I recently came into possession of a LucidSound LS30 wireless headset. The headset came well reviewed by the Wirecutter, and given the descriptions and the moderate documentation on the LucidSound web site, I was optimistic that the headset would have reasonable Linux compatibility.

Everything I’ve found out about the headset specifically will likely wait for another day, but this whole process of trying to debug the behavior of a misbehaving USB device has given me more insight into debugging and interfacing with the Linux kernel and hardware drivers.

Starting Point

The vast majority of Linux users should be familiar with the dmesg command. This command spits out the various debug messages sent from the linux kernel, and these days is full of color and is relatively readable. Staring at this will usually yield some basic information, but detailed debug messages are often not sent here as the user really doesn’t care on a day to day basis. As an extreme example, dmesg will usually spit out some basic information every time you insert a USB device, but you wouldn’t want this log spewing messages every time you triggered an event with your mouse or keyboard. If you want to view that kind of information, you’re probably not looking for kernel debugging information but rather should look into usbmon.

That aside, I haven’t dug into the specifics of how buffering is handled in the kernel interface, but my understanding suggests that any call that sends a message to dmesg is actually flushed right away. This is in contrast to most messages sent to stdout waiting to flush until the system is opportunistically ready to handle the buffer, or some specific line control character (typically \n) is placed into the buffer. One consequence of flushing regularly is you will incur a serious performance hit if a large volume of messages is being sent with high regularity.

Getting Debugging Messages

In my case I knew that the problem revolved around specifics of the snd-usb-audio.ko kernel driver, and digging into the source of that driver even shows a number of built in debug messages that I wasn’t seeing.

Getting access to these messages is actually pretty straightforward. A few years back a control mechanism for these messages was introduced into debugfs to control exactly what messages are displayed when you need to probe and debug your hardware. An LWN article describing the interface, along with the core kernel documentation are readily available, but given the difficulty I had finding anyone who knew that this thing existed, I gather it is not widely known to anyone who doesn’t dive into the kernel space from time to time.

With that in mind, here is a basic overview of the quick/dirty way to get the debug interface to spill it’s secrets:

The core configuration is handled by inspecting the debugfs dynamic debugging control file. Assuming sysfs is mounted at /sys, this file is at /sys/kernel/debug/dynamic_debug/control. If you want to enable messages that are at higher debug levels (e.g. dev_dbg() kernel calls and such), we simply send a message to this interface (via echo) specifying the file we want to enable message printing for. This takes the form of:

echo file <somefile.c> +p > /sys/kernel/debug/dynamic_debug/control

The obvious question being when starting out: how the hell do I figure out what file I actually need to look at? As in my case, there is a chance you already have a vague idea of where the problem exists, and simply want to pinpoint a certain behavior. Either that or you have a specific module in mind, but not much else. Conveniently, you can actually inspect the control file directly to pull out what lines of code are generating existing messages.

grep \.c: /sys/kernel/debug/dynamic_debug/control | cut -d: -f1 | sort | uniq

Most of the files for a bundled kernel module will have some relative path, but note that if you end up building and inserting a module (see insmod) as you tinker and hack at the kernel, that module may end up using an absolute path rather than the standard one.

To wrap it up, bellow is an example listing showing how to remove a stock kernel module in my system, followed by the insertion and enabling a custom module built in my home directory.

set -e; set -u; echo "$_______" # don't copy this directly into your terminal!
echo file sound/usb/stream.c +p > /sys/kernel/debug/dynamic_debug/control
dmesg -w # watch the output
# do some stuff, test your hardware, etc. ^C
# unplug hardware
rmmod snd-usb-audio
cd ${KERNEL_SOURCE_PATH}
sudo insmod ./debian/linux-image-4.13.0-25-generic-dbgsym/usr/lib/debug/lib/modules/4.13.0-25-generic/kernel/sound/usb/snd-usb-audio.ko
echo file ${KERNEL_SOURCE_PATH}/sound/usb/stream.c +p > /sys/kernel/debug/dynamic_debug/control
dmesg -w
# do more hardware stuff ^C

A note about your terminal suddenly closing

One final note, I’ve put that first line up there to discourage copy/paste from random web sites into your terminal. It should fail and stop your terminal session if you copy and paste it directly in. Ignore it, it’s to make you read this paragraph. Be honest, you found me through a random google search trying to quickly solve a problem. If you’re a bash user, use ^x ^e (Ctrl-X, Ctrl-E) to open your default editor in a terminal, to paste and inspect the commands. When you save the output and close the editor, the resulting file is run automatically.