Planet Linux Plumbers Conf

• Replace unionfs with aufs which has less users yelling at it and more developers.
• Use a filesystem like btrfs which has snapshots. This changes the model and makes api's like "what changed?" hard to implement.
• Move to a pure userspace model using git. The problem here is that git as currently written is meant for users not transactions.
  
• Use combination of copy, bind mount, and rsync.
• Use a database for configuration. This is easier for general queries but is the most work. Conversion from existing format would be a pain.
  

Now that linux.conf.au is over, there has been a bunch of information running around about the status of kdbus and the integration of it with systemd. So, here’s a short summary of what’s going on at the moment.

Lennart Poettering gave a talk about kdbus at linux.conf.au. The talk can be viewed here, and the slides are here. Go read the slides and watch the talk, odds are, most of your questions will be answered there already.

For those who don’t want to take the time watching the talk, lwn.net wrote up a great summary of the talk, and that article is here. For those of you without a lwn.net subscription, what are you waiting for? You’ll have to wait two weeks before it comes out from behind the paid section of the website before reading it, sorry.

There will be a systemd hack-fest a few days before FOSDEM, where we should hopefully pound out the remaining rough edges on the codebase and get it ready to be merged. Lennart will also be giving his kdbus talk again at FOSDEM if anyone wants to see it in person.

The kdbus code can be found in two places, both on google code, and on github, depending on where you like to browse things. In a few weeks we’ll probably be creating some patches and submitting it for inclusion in the main kernel, but more testing with the latest systemd code needs to be done first.

If you want more information about the kdbus interface, and how it works, please see the kdbus.txt file for details.

Binder vs. kdbus

A lot of people have asked about replacing Android’s binder code with kdbus. I originally thought this could be done, but as time has gone by, I’ve come to the conclusion that this will not happen with the first version of kdbus, and possibly can never happen.

First off, go read that link describing binder that I pointed to above, especially all of the links to different resources from that page. That should give you more than you ever wanted to know about binder.

Short answer

Binder is bound to the CPU, D-Bus (and hence kdbus), is bound to RAM.

Long answer

Binder

Binder is an interface that Android uses to provide synchronous calling (CPU) from one task to a thread of another task. There is no queueing involved in these calls, other than the caller process is suspended until the answering process returns. RAM is not interesting besides the fact that it is used to share the data between the different callers. The fact that the caller process gives up its CPU slice to the answering process is key for how Android works with the binder library.

This is just like a syscall, and it behaves a lot like a mutex. The communicating processes are directly connected to each other. There is an upper limit of how many different processes can be using binder at once, and I think it’s around 16 for most systems.

D-Bus

D-Bus is asynchronous, it queues (RAM) messages, keeps the messages in order, and the receiver dequeues the messages. The CPU does not matter at all other than it is used to do the asynchronous work of passing the RAM around between the different processes.

This is a lot like network communication protocols. It is a very “disconnected” communication method between processes. The upper limit of message sizes and numbers is usually around 8Mb per connection and a normal message is around 200-800 bytes.

Binder

The model of Binder was created for a microkernel-like device (side note, go read this wonderful article about the history of Danger written by one of the engineers at that company for a glimpse into where the Android internals came from, binder included.) The model of binder is very limited, inflexible in its use-cases, but very powerful and extremely low-overhead and fast. Binder ensures that the same CPU timeslice will go from the calling process into the called process’s thread, and then come back into the caller when finished. There is almost no scheduling involved, and is much like a syscall into the kernel that does work for the calling process. This interface is very well suited for cheap devices with almost no RAM and very low CPU resources.

So, for systems like Android, binder makes total sense, especially given the history of it and where it was designed to be used.

D-Bus

D-Bus is a create-store-forward, compose reply and then create-store-forward messaging model which is more complex than binder, but because of that, it is extremely flexible, versatile, network transparent, much easier to manage, and very easy to let fully untrusted peers take part of the communication model (hint, never let this happen with binder, or bad things will happen…) D-Bus can scale up to huge amounts of data, and with the implementation of kdbus it is possible to pass gigabytes of buffers to every connection on the bus if you really wanted to. CPU-wise, it is not as efficient as binder, but is a much better general-purpose solution for general-purpose machines and workloads.

CPU vs. RAM

Yes, it’s an over simplification of a different set of complex IPC methods, but these 3 words should help you explain the differences between binder and D-Bus and why kdbus isn’t going to be able to easily replace binder anytime soon.

Never say never

Ok, before you start to object to the above statements, yes, we could add functionality to kdbus to have some blocking ioctl calls that implement something like: write question -> block for reply and read reply one answer for the request side, and then on the server side do: write answer -> block in read That would get kdbus a tiny bit closer to the binder model, by queueing stuff in RAM instead of relying on a thread pool.

That might work, but would require a lot of work on the binder library side in Android, and as a very limited number of people have write access to that code (they all can be counted on one hand), and it’s a non-trivial amount of work for a core function of Android that is working very well today, I don’t know if it will ever happen.

But anything is possible, it’s just software you know…

Thanks

Many thanks to Kay Sievers who came up with the CPU vs. RAM description of binder and D-Bus and whose email I pretty much just copied into this post. Also thanks to Kay and Lennart for taking the time and energy to put up with my silly statements about how kdbus could replace binder, and totally proving me wrong, sorry for having you spend so much time on this, but I now know you are right.

Also thanks to Daniel Mack and Kay for doing so much work on the kdbus kernel code, that I don’t think any of my original implementation is even present anymore, which is probably a good thing. Also thanks to Tejun Heo for help with the memfd implementation and cgroups help in kdbus.

CONFIG_SMP=y
CONFIG_NR_CPUS=16
CONFIG_PREEMPT_NONE=n
CONFIG_PREEMPT_VOLUNTARY=n
CONFIG_PREEMPT=y
CONFIG_HZ_PERIODIC=n
CONFIG_NO_HZ_IDLE=y
CONFIG_NO_HZ_FULL=n
CONFIG_RCU_FAST_NO_HZ=n
CONFIG_RCU_TRACE=n
CONFIG_HOTPLUG_CPU=n
CONFIG_SUSPEND=n
CONFIG_HIBERNATION=n
CONFIG_RCU_FANOUT=3
CONFIG_RCU_FANOUT_EXACT=y
CONFIG_RCU_FANOUT_LEAF=2
CONFIG_RCU_NOCB_CPU=y
CONFIG_RCU_NOCB_CPU_ALL=y
CONFIG_DEBUG_LOCK_ALLOC=n
CONFIG_PROVE_RCU_DELAY=n
CONFIG_RCU_CPU_STALL_INFO=n
CONFIG_RCU_CPU_STALL_VERBOSE=n
CONFIG_RCU_BOOST=n
CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
CONFIG_PRINTK_TIME=y

One reason for my surprise was that I had been running similar rcutorture scenarios for some years. In sharp (and happy) contrast to previous efforts featuring but a handful of failures in a given ten-hour run, the effect was not subtle. In fact, TREE08 often failed more than 20 times per minute.

My first thought was that I had blown a recent commit, but testing earlier kernel versions gave similar failure rates.

Additional testing and analysis showed that the bug was quite sensitive to the exact values of the Kconfig parameters. For example, setting CONFIG_RCU_FANOUT=2 or CONFIG_RCU_FANOUT=4 resulted in rock-solid behavior, despite CONFIG_RCU_FANOUT=3 failing miserably. Similarly, setting CONFIG_RCU_FANOUT_EXACT=n (which is thankfully the default) also resulted in rock-solid behavior. This fortunately provides not only a workaround for the bug, but, even better, a workaround that is enabled by default. This enabled-by-default workaround did much to explain the total lack of bug reports.

This sensitivity to a few Kconfig parameters implicated the boot-time code that sets up the rcu_node structures. I therefore ran this boot-time setup code in usermode scaffolding, resulting in the following output:

NR_CPUS = 16, CONFIG_RCU_FANOUT = 3, CONFIG_RCU_FANOUT_LEAF = 2, MAX_RCU_LVLS = 4
rcu_fanout_leaf = 16, nr_cpu_ids = 3
NR_CPUS: 16, RCU_NUM_LVLS: 3, rcu_num_lvls: 3, rcu_num_nodes: 12
num_rcu_lvl[0]: 1
num_rcu_lvl[1]: 3
num_rcu_lvl[2]: 8
num_rcu_lvl[3]: 16
num_rcu_lvl[4]: 0
rsp->levelspread[0]: 2
rsp->levelspread[1]: 3
rsp->levelspread[2]: 3
CPU   0: rdp=0x804c420 rnp=0x804b180 (0-2) rnp=0x804b0a8 (0-8) rnp=0x804b060 (0-15)
CPU   1: rdp=0x804c4cc rnp=0x804b180 (0-2) rnp=0x804b0a8 (0-8) rnp=0x804b060 (0-15)
CPU   2: rdp=0x804c578 rnp=0x804b180 (0-2) rnp=0x804b0a8 (0-8) rnp=0x804b060 (0-15)
CPU   3: rdp=0x804c624 rnp=0x804b1c8 (3-5) rnp=0x804b0a8 (0-8) rnp=0x804b060 (0-15)
CPU   4: rdp=0x804c6d0 rnp=0x804b1c8 (3-5) rnp=0x804b0a8 (0-8) rnp=0x804b060 (0-15)
CPU   5: rdp=0x804c77c rnp=0x804b1c8 (3-5) rnp=0x804b0a8 (0-8) rnp=0x804b060 (0-15)
CPU   6: rdp=0x804c828 rnp=0x804b210 (6-8) rnp=0x804b0a8 (0-8) rnp=0x804b060 (0-15)
CPU   7: rdp=0x804c8d4 rnp=0x804b210 (6-8) rnp=0x804b0a8 (0-8) rnp=0x804b060 (0-15)
CPU   8: rdp=0x804c980 rnp=0x804b210 (6-8) rnp=0x804b0a8 (0-8) rnp=0x804b060 (0-15)
CPU   9: rdp=0x804ca2c rnp=0x804b258 (9-11) rnp=0x804b0f0 (9-15) rnp=0x804b060 (0-15)
CPU  10: rdp=0x804cad8 rnp=0x804b258 (9-11) rnp=0x804b0f0 (9-15) rnp=0x804b060 (0-15)
CPU  11: rdp=0x804cb84 rnp=0x804b258 (9-11) rnp=0x804b0f0 (9-15) rnp=0x804b060 (0-15)
CPU  12: rdp=0x804cc30 rnp=0x804b2a0 (12-14) rnp=0x804b0f0 (9-15) rnp=0x804b060 (0-15)
CPU  13: rdp=0x804ccdc rnp=0x804b2a0 (12-14) rnp=0x804b0f0 (9-15) rnp=0x804b060 (0-15)
CPU  14: rdp=0x804cd88 rnp=0x804b2a0 (12-14) rnp=0x804b0f0 (9-15) rnp=0x804b060 (0-15)
CPU  15: rdp=0x804ce34 rnp=0x804b2e8 (15-15) rnp=0x804b0f0 (9-15) rnp=0x804b060 (0-15)

The rdp= numbers show each CPU's rcu_data structure addresses, and the sequence of rnp= numbers show the rcu_node structures when traversing upwards from the corresponding CPU to the root of the rcu_node tree. The numbers in parentheses show which CPUs are handled by each rcu_node structure. The initial lines simply show the values of a few key boot-time macros, variables, and arrays.

Had I been paying attention, I would have noticed that this output shows two very odd things, with one of the called out twice. However, some of my assumptions blinded me to these warning signs.

I therefore enabled RCU event tracing, with the usual false start where I failed to build the kernel with CONFIG_RCU_TRACE=y. This resulted in a 20MB trace, roughly 15MB of which was required to show the entire grace-period dance starting from the callback being registered and ending with rcutorture detecting RCU bug. The following entries caught my eye:

rcu_pree-7       0d..2 5852674733: rcu_grace_period_init: rcu_preempt 18446744073709551553 0 0 15 3
rcu_pree-7       0d..2 5852674946: rcu_grace_period_init: rcu_preempt 18446744073709551553 1 0 8 7
rcu_pree-7       0d..2 5852675138: rcu_grace_period_init: rcu_preempt 18446744073709551553 1 9 15 7
rcu_pree-7       0d..2 5852675313: rcu_grace_period_init: rcu_preempt 18446744073709551553 1 18 15 0
rcu_pree-7       0d..2 5852675631: rcu_grace_period: rcu_preempt 18446744073709551553 cpustart
rcu_pree-7       0d..2 5852675802: rcu_grace_period_init: rcu_preempt 18446744073709551553 2 0 2 7
rcu_pree-7       0d..2 5852675974: rcu_grace_period_init: rcu_preempt 18446744073709551553 2 3 5 7
rcu_pree-7       0d..2 5852676148: rcu_grace_period_init: rcu_preempt 18446744073709551553 2 6 8 7
rcu_pree-7       0d..2 5852676321: rcu_grace_period_init: rcu_preempt 18446744073709551553 2 9 11 7
rcu_pree-7       0d..2 5852676519: rcu_grace_period_init: rcu_preempt 18446744073709551553 2 12 14 7
rcu_pree-7       0d..2 5852676691: rcu_grace_period_init: rcu_preempt 18446744073709551553 2 15 15 1
rcu_pree-7       0d..2 5852676862: rcu_grace_period_init: rcu_preempt 18446744073709551553 2 18 15 0
rcu_pree-7       0d..2 5852677034: rcu_grace_period_init: rcu_preempt 18446744073709551553 2 21 15 0

The smoking gun is the fourth line, the one containing the “18 15”. This indicates that the rcu_node structure corresponding to this trace event covers all CPUs whose numbers (as listed by smp_processor_id()) are greater or equal to 18 and also less than or equal to 15. There are of course no such CPUs. The last two lines show similar brokenness.

These trace events narrowed the bug down to the following code fragment in kernel/rcutree.c:

 1 #ifdef CONFIG_RCU_FANOUT_EXACT
 2 static void __init rcu_init_levelspread(struct rcu_state *rsp)
 3 {
 4   int i;
 5 
 6   for (i = rcu_num_lvls - 1; i > 0; i--)
 7     rsp->levelspread[i] = CONFIG_RCU_FANOUT;
 8   rsp->levelspread[0] = rcu_fanout_leaf;
 9 }
10 #else /* #ifdef CONFIG_RCU_FANOUT_EXACT */
11 static void __init rcu_init_levelspread(struct rcu_state *rsp)
12 {
13   int ccur;
14   int cprv;
15   int i;
16 
17   cprv = nr_cpu_ids;
18   for (i = rcu_num_lvls - 1; i >= 0; i--) { 
19     ccur = rsp->levelcnt[i];
20     rsp->levelspread[i] = (cprv + ccur - 1) / ccur;
21     cprv = ccur;
22   }
23 }
24 #endif /* #else #ifdef CONFIG_RCU_FANOUT_EXACT */

Can you spot the problem and the fix?

Sometimes you don’t have access to vim’s wonderful xxd tool, and you need to use it to generate some .c code based on a binary file. This happened to me recently when packaging up the EFI signing tools for Gentoo. Adding a build requirement of vim for a single autogenerated file was not an option for some users, so I created a perl version of the xxd -i command line tool.

This works because everyone has perl in their build systems, whether they like it or not. Instead of burying it in the efitools package, here’s a copy of it for others to use if they want/need it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

#!/usr/bin/env perl
#
# xxdi.pl - perl implementation of 'xxd -i' mode
#
# Copyright 2013 Greg Kroah-Hartman <gregkh@linuxfoundation.org>
# Copyright 2013 Linux Foundation
#
# Released under the GPLv2.
#
# Implements the "basic" functionality of 'xxd -i' in perl to keep build
# systems from having to build/install/rely on vim-core, which not all
# distros want to do.  But everyone has perl, so use it instead.

use strict;
use warnings;
use File::Slurp qw(slurp);

my $indata = slurp(@ARGV ? $ARGV[0] : \*STDIN);
my $len_data = length($indata);
my $num_digits_per_line = 12;
my $var_name;
my $outdata;

# Use the variable name of the file we read from, converting '/' and '.
# to '_', or, if this is stdin, just use "stdin" as the name.
if (@ARGV) {
        $var_name = $ARGV[0];
        $var_name =~ s/\//_/g;
        $var_name =~ s/\./_/g;
} else {
        $var_name = "stdin";
}

$outdata .= "unsigned char $var_name\[] = {";

# trailing ',' is acceptable, so instead of duplicating the logic for
# just the last character, live with the extra ','.
for (my $key= 0; $key < $len_data; $key++) {
        if ($key % $num_digits_per_line == 0) {
                $outdata .= "\n\t";
        }
        $outdata .= sprintf("0x%.2x, ", ord(substr($indata, $key, 1)));
}

$outdata .= "\n};\nunsigned int $var_name\_len = $len_data;\n";

binmode STDOUT;
print {*STDOUT} $outdata;

Yes, I know I write perl code like a C programmer, that’s not an insult to me.

This documents my personal flow for downloading and installing a Linux kernel with my xHCI and USB 3.0 code. Until the code is in the upstream kernel and shipping in Linux distributions, you'll have to follow these directions to get Linux USB 3.0 support.

Read more »

After spending a few days manning the Plumbers booth at OSCON, I thought I’d post the answer to the question that everyone seemed to want to know – What is a Linux Plumbers Conference? We came up with the word “Plumbing” to describe the low level infrastructure of a Linux System. This includes the Kernel, desktop infrastructure like X and graphics libraries, system utilities like udev and hal, as well as essential libraries like glibc and friends. These components interface with each other at times – some better than others. We hope to provide a forum for people from these types of projects to get together and try to solve problems that are system wide or cross multiple project boundaries.

In addition to the topics to be discussed in the microconfs and the general talks (see http://linuxplumbersconf.org/program/schedule/), we will have “unconference” style talks. We have several smaller rooms available for people to get together and work out specifics, talk about something they didn’t get on the schedule, or have a group hug. These rooms can be reserved at the start of the conference.

August 18th the registration fee for Plumbers will increase to $300. If you haven’t already registered, what are you waiting for?

The 2012 Linux Plumbers Conference (LPC) will be held on August 29-31 in the Sheraton San Diego, and we hope to see you there!

To that end, the LPC Planning Committee is pleased to announce a call for microconferences. These microconferences are working sessions that are roughly a half day in length, each focused on a specific aspect of the “plumbing” in the Linux system. The Linux system’s plumbing includes kernel subsystems, core libraries, windowing systems, media creation/playback, and so on. For reference, last year’s LPC had tracks on Audio, Bufferbloat and Networking, Cloud, Containers and Cgroups, Desktop, Development Tools, Early Boot and Init Systems, File and Storage Systems, Mobile, Power Management, Scaling, Tracing, Unified Memory Management, and Virtualization.

Please note that submissions to a given microconference should not normally cover finished work. The best submissions are instead problems, proposals, or proof-of-concept solutions that require face-to-face discussions and debate among people from different areas of the Linux plumbing. In other words, the best microconferences are working sessions that turn problems into patches representing solutions.

Leading an LPC microconference can be a fun, exciting, and rewarding activity, but please see here for the responsibilities of a microconference working session leader. If you have an idea for a good LPC microconference, and especially if you would like to lead up a particular microconference, please add it to the LPC wiki here.

Last week I found two tools that make my life better and make me look cool in front of my friends (j/k). So I thought I would share them.

Keeping bookmarks sync'd and accessible Back in the day I used to use a shareware tool to dump my IE bookmarks to html, then upload them via FTP, and then download them again and re-sync. But times have changed and del.icio.us is the new way to bookmark.

For those not in the know del.icio.us is a "social bookmarking" website. The first consequence is that your bookmarks are stored on a globally accessible webserver with an easy to remember URL like http://del.icio.us/philips. The second and more fun aspect is that when you make a bookmark (with one of the great del.icio.us bookmarklets) you can see who else has bookmarked the same page and what other sites may be related and of interest. From this feature I have found some great websites, including my new favorite techno radio station Radio ABF France.

But the coolest part is a plugin for Firefox called Foxlicious that allows you to sync your bookmarks from del.icio.us into a folder, organized by tags. It is great I can bookmark at home, and sync at work, then bookmark at work and sync at home, then; well you get the idea. Zebra Tele-scopic As you may already know I carry with me at most times an analog notebook (you know the paper kind). But I have never been able to find an inexpensive pen that is compact enough to keep in my pocket. Until my faithful run to the store last week where I found it! "It" being the Zebra Tele-scopic pen which is small enough to put in a jean pocket but telescopes into a regular sized and balanced ball point pen. Not only that but they are far cheaper than the Fisher Space Pen. At ~$5.49 US for two tiny telescoping pens with two refills these pens are a great deal!

Hello everyone,

Thanks for making this year’s plumbers conference such an enjoyable event.  Next year, we’re planning to co-locate Plumbers with the Kernel Summit and LinuxCon in San Diego from 29-31 August.  The current plan is that Plumbers and LinuxCon would run as parallel but separate events.  To accommodate the parallelism, we’re still planning on keeping the numbers for Plumbers down to 300 and having a separate registration from LinuxCon.  We’re also planning to move the refereed presentations track into LinuxCon itself as a hard core technical track which would still be selected by the Plumbers Programme committe (both Plumbers and LinuxCon attendees would be able to go to this). We plan to keep the two microconference tracks for plumbers only, but also add a third unconference type track, where people could plan meetings and split into discussion groups in a style very similar to Ubuntu Developer Summit (only Plumbers registered attendees would be able to go to this).

If you have any feedback about this plan, please sent it to the current programme committee at lpc2011@virtuousgeek.org

Of course, we’re also looking to recruit another organising and programme committee for 2013, so if you want to volunteer, please read this web page and then send your bid to the plumbers conference steering committee (who are also the Linux Foundation Technical Advisory Board) at tech-board@lists.linux-foundation.org

I’m doing a lot of things this year – and for the hundredth time, I find myself discarding my previous life and blogs and starting anew.

Last October, in a chocolate-induced haze of post Halloween self-satisfaction, I somehow thought it would be a good idea to volunteer to run the Linux Plumbers Conference in 2009. Portland continues to host the event, which makes it possible for me to help out, along with an extraordinarily strong local Linux development and business ecosystem. Last year’s crew did a heck of a job creating the event for the first time. We’re coasting on their toil and troubles, this year, frankly.

Despite the continual incoming dripdripdrip of somber economic news, tightening budgets, market collapses, layoffs, disappearing finances and individual anxieties, we are somehow crafting together what will be a rather interesting and productive conference.

The outstanding news today was we got a lot closer to signing up another big name for our keynote! It won’t get announced anytime soon, unfortunately, but it will be fantastic if we can get them.  We have already lined up Keith Packard, X Window genius and all round great guy to give one of the keynote addresses.

We will also have Linus giving an advanced tutorial on git.  It pains me to impose on Linus, but I’m personally very grateful that James Bottomley did the heroic arm-twisting for us.  After losing the video of Linus’s git tutorial in 2008, we badly wanted a chance to reassemble our dignity and geek cred.

If you’re a Linux developer in Portland, OR (or for that matter, anywhere else), what are you waiting for? Register already!

Linux Plumbers Conference will run from Sept 23-25 in 2009 at the Downtown Portland Marriott.

A useful tool: slock is a tiny c program that locks your screen like xlock. But, with only 147 lines of very straightforward code it would be very difficult to introduce vulnerabilities :)

I'm pleased to announce that the eXtensible Host Controller Interface (xHCI) 1.0 specification is now publicly available on intel.com. This is the specification for the PC hardware that talks to all your USB 3.0, USB 2.0, and USB 1.1 devices. (Yes, there are no more companion controllers, xHCI is the one host controller to rule them all).

Open, public documentation is always important to the open source community. Now that the spec is open, anyone can fully understand my Linux xHCI driver (although it's currently only compliant to the 0.96 xHCI spec; anyone want to help fix that?). This also means the BSD developers can implement their own xHCI driver.

Curious what a TRB or a Set TR Deq Ptr command is? Want to know how device contexts or endpoint rings work? Go read the spec!

If you’re ever in the state of Oregon, take the time to visit the  Rice Mineral Museum. I took a trip there today, and it was eye-opening, staggering, and simply wonderful. This is a world-class museum, a Smithsonian-level collection hiding out in the middle of nowhere, also known as the north end of Shute Road in Hillsboro. Their website and photo gallery simply do not do them justice.

Most of the pieces were so staggeringly beautiful that they far outdid commercial art that’s sold for megabucks. Amongst the very cool things, a slice of the collection comes from Pashan, Pune, one of the several places on this planet I call home.

for real.

I couldn’t have been happier with how the Linux Plumbers Conference went last week. I went back and looked at the original proposal that we had Arjan, Greg, and Randy present to the Linux Foundation, and we seem to have hit all our original goals. From conception we wanted this to be a “working” conference – and from the conversations in the hallways that I overheard, to the discussions in the microconfs that went on, I could see that people were indeed getting together, discussing issues and solving problems. Conferences require a lot of time, effort, and money to do right, and it’s gratifying to feel that something useful will come out of this.

I think that now I can go back to blogging about duck poo and vegetables.