Planet Linux Plumbers Conf

July 03, 2009

Valerie Aurora

Soft updates explained

My latest article for LWN explains (!) soft updates. The "(!)" is because soft updates are notoriously difficult to understand. If you go to a file systems conference and get people drunk, they will eventually confide to you that they don't really understand soft updates either.

Soft updates, hard problems

This is a free link; if you like the article, please consider subscribing to LWN. You'll still need an account if you want to make snide comments on the article. :)

July 03, 2009 04:34 PM

July 02, 2009

Twitter

I'm at TIS Graphics finalizing the ads for the Linux plumbers conference. Http://linuxplumbersconf.org

I'm at TIS Graphics finalizing the ads for the Linux plumbers conference. Http://linuxplumbersconf.org

by brandsteve (Steven Miller) at July 02, 2009 10:32 PM

June 29, 2009

Valerie Aurora

Common knowledge and the Cold War

I recently visited the National Museum of Nuclear Science and History - or, as I knew it growing up in Albuquerque, the Atomic Museum. The museum has a brand new full size building, with enough room to display most of their catalog for the first time, but not quite enough money to do so professionally. The result is a brief, magic window in which rare artifacts are finally out on display, but you can touch them and bang on them and crawl around in them. Many of the larger items, including a disassembled B-52 bomber and many rocket engines, are simply dumped in rows in a dirt courtyard in back.

Somehow, I expected that I would traipse through the museum, looking at old photographs and brushing up on my nuclear weapons trivia, with perhaps some solemn moments of reflection in front of the reproductions of Fat Man and Little Boy. Instead, I found myself oscillating between uncontrollable sobbing and open-mouthed technological awe. It went something like this: "Wow, a cyclotron! Holy crap, the Potsdam declaration. (Muffled sob.) A real nose-cone from an ICBM, cool! Whoa, photos of ground zero at Hiroshima. (Fountain of tears.)" I went back the next day to take some original photographs with the intention of writing a thoughtful, well-researched article on my personal experience.

Unfortunately, I have discovered that I seem to know almost nothing about the history of nuclear arms testing and development - and this is from someone whose parents worked on the Strategic Defense Initiative (Reagan's "Star Wars"), who read Richard Rhodes' "The Making of the Atomic Bomb" AND "The Making of the Hydrogen Bomb", who grew up in New Mexico, home of the Manhattan Project. More accurately, I knew some of the relevant facts, but in a vague sort of manner devoid of any connection with everyday life. They were numbers of megatons in a reference book, fictional movie plots involving lost nuclear weapons, and contrived acronyms for arms reduction treaties.

But walking through the museum, I saw brass Nazi goggles and notebooks, the car that carried the Trinity bomb to the test site, a copy of the Potsdam Declaration, movies of ordinary Japanese citizens clearing rubble with hand baskets in Hiroshima and Nagasaki, and the dented shells of nuclear missiles that were, for reals, lost in a midair collision over Spain and recovered after a multi-million dollar search. (Far more were lost and never found, in or over the ocean.) I saw, and touched, and yet still almost could not believe in, the outer shell of a "Davy Crockett" miniature tactical nuke - a literal "backpack nuke," small enough that I could encircle it in my arms. I thought backpack nukes were only a theoretical possibility, and yet they were manufactured, assembly line style. I was particularly struck by how heavily the shoulder straps of the backpack were padded - a consideration so practical and down-to-earth in the face of the incomprehensible horror of the weapon itself.

And then I really got myself in trouble: I bought a copy of Michael Light's 100 Suns from the book shop. It is a collection of 100 photographs of nuclear explosions from the U.S. nuclear testing program, during the time when nuclear tests were conducted above ground. I knew, intellectually, that Enewetak and Bikini Atolls had been practically obliterated by thermonuclear bomb tests, but seeing a 20"x26" color photograph of the fireball of a 11 megaton explosion is... entirely different. And entirely different than seeing it on the computer screen - the image below has nothing like the power of that in the book.


Castle Romeo test, Bikini Atoll, 1954, 11 megatons


Each photograph in this book symbolizes and encapsulates the conflicting and overpowering feelings I had in the museum: awe, excitement, and deep grief. My favorite photos are ones of the people watching the tests - most of them are bored, or matter-of-fact, but a few of the faces show the same awe and awareness that I feel when I look at the photos of the explosions, decades after the fact. The photos are accompanied by short footnotes at the end of the book, describing the technical and political circumstances and fallout (literal and figurative) of each test.

And here, yet again, I learned how little I knew: that several of the thermonuclear bombs accidentally exceeded expected yield by several megatons and accidentally sickened people (How!? can something as complex as a thermonuclear bomb go wrong - and result in even greater power?? That's not how computers work!), that we actually exploded nuclear weapons above the atmosphere and were surprised by the resultant EMP (I thought physicists predicted it, not that we knocked out Hawaii's power grid by accident and worked backwards from there), that the largest nuclear explosion ever was a 50-megaton test by the Soviets in the Arctic ("test" - it was entirely for political effect), that we exploded thermonuclear bombs in the continental U.S., that U.S. soldiers were put in trenches close to bomb tests in Nevada so that they could conduct maneuvers within a few hundred feet of the smoking, radioactive craters immediately afterwards.

It never even occurred to me that thousands (hundreds of thousands?) of people had witnessed nuclear tests and that I could go talk to one of these people and ask them what it was like. And I never would have guessed that I would be jealous of them, because more than likely, no human will ever witness a nuclear explosion first-hand ever again.

I don't know what to do now. Maybe most people already know these things, in which case it will be difficult to communicate my awe. Maybe they don't know these things, but I won't be able to cross the boundary between intellectual knowledge, like what I knew before I went to the museum, and the intense visceral awareness that the physical objects and photos gave me. Maybe I can't do better than Michael Light's magnificent book and I should just write him a positive Amazon review. Maybe I can do better, if I use all the resources available to me on this here World Wide Web.

Questions for you, dear reader:
  • Which of the above facts surprised you? What is the most shocking thing you know about nuclear weapons and the Cold War?
  • Do you know anyone who saw a nuclear explosion in real life? Have you asked them about it? Are they willing to talk about it?
  • What is the movie/book/web site/whatever about nuclear weapons that you would recommend the most?
  • Any advice for me on what (if anything) to do with this project?
Thank you for reading all the way through this.

June 29, 2009 06:05 AM

June 23, 2009

Linux Plumbers Conf

Submissions closed - Thank you!

Submissions to Linux Plumbers Conference 09 are now closed.

Thank you to everybody who submitted - we had an overwhelming response! Author notification will go out July 15th. Whether you’re presenting or not, we hope to see you at the conference - our success is as dependent on the participants as the presenters - our goal is to move code forward!

Submissions to the Linux Plumbers Conference are now closed.

by Nivedita Singhvi at June 23, 2009 05:43 PM

Twitter

June 15, 2009

Sri Ramkrishna

Iran and the elections aftermath

I wanted to express my best wishes and hope for the safety of those
who are in danger.

Especially to the GNOME Farsi community in Iran, we are all hoping for your
continued safety during this upheaval. Please be safe.

re:Twitter -
The wonderful thing about twitter is that they can communicate their
stories and experience in such a profound way. I find myself glued
to the screen to see how things are unfolding in Iran. I’ve been
following the news on Andrew Sullivan’s blog site which gives up
to date information of what is happening in Iran.

edit: fixed broken link.

by sri at June 15, 2009 03:49 AM

June 12, 2009

Sri Ramkrishna

Linux Plumbers Conference…

So I’ve been meaning to put a post on this but I haven’t had a chance thanks to school and what not.  Having finished my exams and projects, I thought I would put in a word or two regarding the other thing I’ve been busy with and that is the Linux Plumbers Conference.  This is my second year in a row in being part of the organizing committee of Linux Plumbers Conference!

Linux Plumbers Conference is one of those small conferences organized by developers for developers and thus have a highly technical focus.  As a bonus, it’s held the same week as Linuxcon so you can get your taste of the O’Reilly type conference as well if laundry is an issue. :-)  The goal is to improve the Linux platform, targetting the weaknesses and improving them.  Last year we had the fast boot by Intel which started a distro war over boot time. and we had a piece of X put into the kernel, so things happen at this conference.

We are currently looking for paper submussions.  Have an idea for the desktop but you need kernel developer support?  This is your chance!  Thanks to Jim Gettys to being our runner for the UI track.  So if you have ideas that will help advance the Linux platform/ecosystem.  Please submit your ideas! The conference is in Portland, OR Sept 23-25, 2009. I hope to see a lot of you there. As many can attest, Portland is an awesome city with some of the coolest people on the planet living here.

by sri at June 12, 2009 12:35 AM

June 11, 2009

Linux Plumbers Conf

Deadline for proposal submissions is almost upon us!

UPDATE: Deadline submission extended to midnight, June 22nd, PT.

If you haven’t submitted yet, you’ll need to get cracking!  Submissions close at midnight, June 22nd, PT.

The latest update cross posted from LWN:
Please submit your abstracts using the “Submit a proposal” button on the right.

We now have runners for additional tracks:

1. Inter-Distributor Cooperation: James Bottomley
2. Kernel/Userspace/User Interfaces: Jim Gettys
3. Networking: Steve Hemminger
4. Storage: Matthew Wilcox

In addition, we have the runners announced earlier:

1. Security: James Morris and Paul Moore
2. Boot and Init: Dave Jones
3. X Window System: Keith Packard
4. Embedded Systems: Greg Kroah-Hartman and David Woodhouse
5. Audio: Lennart Poettering

Other topics are of course welcome in the General track. The perfect Plumbers topic would feature a real problem whose solution requires the relevant community members to get together face to face, preferably producing a solution during the conference itself. This of course requires buy-in from maintainers and other high-profile people active in the area.

We have an exciting program shaping up! Tutorials include an advanced git tutorial from Linus Torvalds. Keynotes include Vivek Kundra, Federal CIO, live via video with real time Q&A (invitee, to be confirmed), and Keith Packard, Intel, X Window guru.

So, to be part of the excitement, please submit your abstracts by Monday June 22nd!!!

by Nivedita Singhvi at June 11, 2009 05:07 AM

June 09, 2009

Sarah A Sharp

Installing a custom kernel with USB 3.0 support

This documents my personal flow for downloading and installing a Linux kernel with my xHCI and USB 3.0 code. Until the code is in the upstream kernel and shipping in Linux distributions, you'll have to follow these directions to get Linux USB 3.0 support.

Read more »

June 09, 2009 08:39 PM

June 08, 2009

Paul E. McKenney

RCU and unloadable modules

The rcu_barrier() function was described some time back in an article on Linux Weekly News. This rcu_barrier() function solves the problem where a given module invokes call_rcu() using a function in that module, but the module is removed before the corresponding grace period elapses, or at least before the callback can be invoked. This results in an attempt to call a function whose code has been removed from the Linux kernel. Oops!!!

Since the above article was written, rcu_barrier_bh() and rcu_barrier_sched() have been accepted into the Linux kernel, for use with call_rcu_bh() and call_rcu_sched(), respectively. These functions have seen relatively little use, which is no surprise, given that they are quite specialized. However, Jesper Dangaard recently discovered that they need to be used a bit more heavily. This lead to the question of exactly when they needed to be used, to which I responded as follows:

Unless there is some other mechanism to ensure that all the RCU callbacks have been invoked before the module exit, there needs to be code in the module-exit function that does the following:

  1. Prevents any new RCU callbacks from being posted. In other words, make sure that no future call_rcu() invocations happen from this module unless those call_rcu() invocations touch only functions and data that outlive this module.
  2. Invokes rcu_barrier().
  3. Of course, if the module uses call_rcu_sched() instead of call_rcu(), then it should invoke rcu_barrier_sched() instead of rcu_barrier(). Similarly, if it uses call_rcu_bh() instead of call_rcu(), then it should invoke rcu_barrier_bh() instead of rcu_barrier(). If the module uses more than one of call_rcu(), call_rcu_sched(), and call_rcu_bh(), then it must invoke more than one of rcu_barrier(), rcu_barrier_sched(), and rcu_barrier_bh().
What other mechanism could be used? I cannot think of one that it safe. For example, a module that tried to count the number of RCU callbacks in flight would be vulnerable to races as follows:

  1. CPU 0: RCU callback decrements the counter.
  2. CPU 1: module-exit function notices that the counter is zero, so removes the module.
  3. CPU 0: attempts to execute the code returning from the RCU callback, and dies horribly due to that code no longer being in memory.

If there was an easy solution (or even a hard solution) to this problem, then I do not believe that Nikita Danilov would have asked Dipankar Sarma for rcu_barrier(). Therefore, I do not expect anyone to be able to come up with an alternative to rcu_barrier() and friends. Always happy to learn something by being proven wrong, of course!!!

So unless someone can show me some other safe mechanism, every unloadable module that uses call_rcu(), call_rcu_sched(), or call_rcu_bh() must use rcu_barrier(), rcu_barrier_sched(), and/or rcu_barrier_bh() in its module-exit function.


So if you have a module that uses one of the call_rcu() functions, please use the corresponding rcu_barrier() function in the module-exit code!

Update: Peter Zijlstra rightly points out that the issue is not whether your module invokes call_rcu(), but rather whether the corresponding RCU callback invokes a function is in a module. So, if there is a call_rcu(), call_rcu_sched(), or call_rcu_bh() anywhere in the kernel whose RCU callback either directly or indirectly invokes a function in your module, then your module's exit function needs to invoke code>rcu_barrier()</code>, rcu_barrier_sched(), and/or rcu_barrier_bh(). Thanks to Peter for pointing this out!

June 08, 2009 09:13 PM

Sarah A Sharp

USB 3.0 support: coming soon to a Linux kernel near you!

The xHCI (USB 3.0) host controller driver and initial support for USB 3.0 devices is now publicly available on my kernel.org git tree. Greg K-H has queued the patches for 2.6.31, so Linux users should have official USB 3.0 support around September 2009. This is impeccable timing, since NEC recently announced they'll be producing 1 million xHCI PCI express add-in cards in September.

This means that Linux will be the first operating system with official USB 3.0 support. I'm working with Keve Gabbert (the OSV person in my group at Intel) to make sure that Linux distributions like Ubuntu and Red Hat pick up the xHCI driver. Advanced users can always compile their own kernel on a standard distro install.

I hope that some USB 3.0 vendors who have prototypes will test with my driver. Instructions on how to compile a kernel using my git tree will follow.

This is a giant project that I've been working on for the past year and a half. It's gratifying to see the code finally released, and exciting to know that hardware is on its way.

June 08, 2009 06:13 AM

June 05, 2009

Stephen Hemminger

Networking at Linux Plumbers Conference

Hey kernel developers, more proposals related to networking submitted for the Linux Plumbers Conference. This is the chance to have in-person discussions about future proposals like receive packet steering, RCU netfilter optimization, unified flow cache, and all those other topics that need need more brainstorming and discussion.

The Netconf 2009 is also being planned to occur before LPC.

by Linux Network Plumber (noreply@blogger.com) at June 05, 2009 10:00 PM

June 04, 2009

Greg KH

Linux Driver Project Status Report as of June 2009

This is a status report for the Linux Driver Project as of June 2009, describing what has happened in the past year of work. It was originally posted on the Linux Driver Project developer mailing list.

See more ...

June 04, 2009 08:14 PM

June 03, 2009

Darrick Wong

Picspam!

/me rounded up a bunch of (old) panoramas and put them into the high-definition panorama viewer. Be sure to check out the (huge spike in memory cache when you load the) panorama previewer (click the "See All" button).

June 03, 2009 02:27 AM

May 29, 2009

Paul E. McKenney

When is it a good idea to use a reader-writer lock?

Reader-writer locks can certainly cause problems if used carelessly, the problems including cache thrashing, write-side contention, and poor scaling. But it is a rare tool that does not have some jobs that it is good for, so it is reasonable to ask what types of jobs favor reader-writer locking.

First, it is important to note that if the reader-writer lock is used sufficiently infrequently, its overhead cannot possibly cause too much trouble. On most systems, if the critical sections are short and if the lock is not acquired more than a few tens of thousands of times per second, there should be no problem. On the other hand, if the lock must be acquired millions of times per second, you are likely to need to do something else. This “something else” might be partitioning the data protected by the reader-writer lock (thus reducing the per-lock acquisition frequency), or it might be some other tool. (Yes, yes, I might be expected to suggest RCU, and I might well do so, but reference counting, exclusive locks, and atomic operations are also possibilities.)

If the reader-writer lock is primarily read-acquired, and if it is read-held for a significant period of time, then reader-writer locking can again work quite well. However, the number of acquisitions per unit time is still important, and this number often rises with increasing numbers of CPUs. This effect can be seen in the following graph:



This graph was generated on a 64-core Power 5 system with two hardware threads per core for 128 hardware threads total. Each thread repeatedly:

  1. read-acquired a pthread_rwlock_t,
  2. took a fixed number of passes through a tight loop, and
  3. released the rwlock_t.

In the “1K” trace, this fixed number of passes was 1,000, in the “10K” it was 10,000, and so on up to the “100M” trace, where the fixed number of passes was 100 million. Each point is computed by taking the ratio of the acquisitions per unit time for N threads divided by N times the acquisitions per unit time for a single thread. Ideal scaling would give 1.0.

As you can see, adding CPUs eventually causes performance to suffer, and the shorter the critical section, the greater the degree of suffering inflicted.

So, reader-writer locking can be the right tool for read-mostly jobs, at least when acquired sufficiently infrequently or if the read-side critical sections are sufficiently long. Just keep in mind that the value of “sufficiently” varies with the number of CPUs!

May 29, 2009 05:55 AM

May 15, 2009

Nivedita Singhvi

LPC keynotes announced!


So here’s the news – we’ve invited Vivek Kundra, Federal CIO to keynote at LPC, and we hope he can make it! This should be really interesting. If he makes it, it will be via a remote, interactive, real time video session.  If you have any suggestions or questions for Vivek Kundra, or how the LPC team should set this up, feel free to post comments here or on our LPC blog!

by nivedita at May 15, 2009 05:40 AM

May 14, 2009

Brandon Philips

Greasemonkey makes real estate fun!

Nisha and I started seriously looking at getting a house recently. It being the 21st Century means large amounts of our house hunting time will be spent on the Internet; specifically rmlsweb.com a regional real estate listing service.

A huge annoyance with the site is that it uses a pretty lame mapping service called “GeoJet” which generates horrible looking maps.

Naturally, I started Googling for Greasemonkey scripts to find out if someone had fixed this already and sure enough my friend Kees Cook had written a user script in 2006!

With a little modification I got it working great with the new RMLS site. You can find it at git://ifup.org/philips/rmls-mapper.git or install the latest version now!

by Brandon Philips at May 14, 2009 06:10 AM

May 13, 2009

Nivedita Singhvi

Chairing LPC 2009


I’m doing a lot of things this year – and for the hundredth time, I find myself discarding my previous life and blogs and starting anew.

Last October, in a chocolate-induced haze of post Halloween self-satisfaction, I somehow thought it would be a good idea to volunteer to run the Linux Plumbers Conference in 2009. Portland continues to host the event, which makes it possible for me to help out, along with an extraordinarily strong local Linux development and business ecosystem. Last year’s crew did a heck of a job creating the event for the first time. We’re coasting on their toil and troubles, this year, frankly.

Despite the continual incoming dripdripdrip of somber economic news, tightening budgets, market collapses, layoffs, disappearing finances and individual anxieties, we are somehow crafting together what will be a rather interesting and productive conference.

The outstanding news today was we got a lot closer to signing up another big name for our keynote! It won’t get announced anytime soon, unfortunately, but it will be fantastic if we can get them.  We have already lined up Keith Packard, X Window genius and all round great guy to give one of the keynote addresses.

We will also have Linus giving an advanced tutorial on git.  It pains me to impose on Linus, but I’m personally very grateful that James Bottomley did the heroic arm-twisting for us.  After losing the video of Linus’s git tutorial in 2008, we badly wanted a chance to reassemble our dignity and geek cred.

If you’re a Linux developer in Portland, OR (or for that matter, anywhere else), what are you waiting for? Register already!

Linux Plumbers Conference will run from Sept 23-25 in 2009 at the Downtown Portland Marriott.

by nivedita at May 13, 2009 06:52 AM

May 01, 2009

Darrick Wong

How'd they get there?

Check out this Google Street View of Arastradero Park:



Happy May Day!

May 01, 2009 08:30 AM

March 25, 2009

Andy Grover

Hacker space, sounds good!

Open source conference prerequisite #1: Space for hacking - Open Source Bridge

I just found out about the conference and its hackerspace element via Reddit. How did I miss this up until now?

The hackerspace idea is obviously one I’m interested in. I have my own proposal for one, but tailored less of a 24-hour space for random hacking encounters, and more as an un-office to give remote-working people an in-person community of peers.

Definitely we have all realized that as great as online interaction can be, being in a room still cannot be duplicated.

Originally published at Groveronline. Please leave any comments there.

March 25, 2009 10:55 PM

March 24, 2009

Greg KH

Loving the 'droid

The 2.6.29 kernel release is now out, and if you look closely, there are a number of drivers for the Android platform merged in, starting with this patch.

Yes, there's still a lot to go, like cleaning up the user/kernel interfaces, and prodding the core Android developers to start working upstream more, which they have already started to do. It's a great start, and one that I hope will succeed overall, as it really is a nice system to develop for, and one the phone manufacturers have been asking for from the Linux community, for years.

And besides, how can you not like such a cute robot:

android and penguins

March 24, 2009 11:57 PM

March 05, 2009

Andy Grover

February 18, 2009

Stephen Hemminger

Parallelizing netfilter

The Linux networking receive performance has been mostly single threaded until the advent of MSI-X and multiqueue receive hardware. Now with many cards, it is possible to be processing packets on multiple CPU's and cores at once. All this is great, and improves performance for the simple case.

But most users don't just use simple networking. They use useful features like netfilter to do firewalling, NAT, connection tracking and all other forms of wierd and wonderful things. The netfilter code has been tuned over the years, but there are still several hot locks in the receive path. Most of these are reader-writer locks which are actually the worst kind, much worse than a simple spin lock. The problem with locks on modern CPU's is that even for the uncontested case, a lock operation means a full-stop cache miss.

With the help of Eric Duzmet, Rick Jones, Martin Josefsson and others, it looks like there is a solution to most of these. I am excited to see how it all pans out but it could mean a big performance increase for any kind of netfilter packet intensive processing. Stay tuned.

by Linux Network Plumber (noreply@blogger.com) at February 18, 2009 05:51 AM

February 17, 2009

Brandon Philips

Linux Plumbers Conference Call For Topics

Linux Plumbers 2009 coming soon

Linux Plumbers 2009 coming soon

The Linux Plumbers Conference focuses on a collection 2.5 hour microconferences that gather experts together to discuss and develop solutions to emerging problems that affect the Kernel, libraries and supporting system daemons that make Linux Linux. Last year there were six of these focused microconference on topics including: audio infrastructure, power management, and boot.

Many of the most successful microconference topics were suggested by community members not directly involved in planning the conference.

So, I need your help for 2009!

Head over to the Call for Topics and suggest an emerging topic in the Linux plumbing that we should discuss in 2009. Topics ideas can be submitted in the comments section of the LWN.net article.

by Brandon Philips at February 17, 2009 06:33 PM

September 22, 2008

Kristen Accardi

Mission Accomplished

for real.

I couldn’t have been happier with how the Linux Plumbers Conference went last week. I went back and looked at the original proposal that we had Arjan, Greg, and Randy present to the Linux Foundation, and we seem to have hit all our original goals. From conception we wanted this to be a “working” conference – and from the conversations in the hallways that I overheard, to the discussions in the microconfs that went on, I could see that people were indeed getting together, discussing issues and solving problems. Conferences require a lot of time, effort, and money to do right, and it’s gratifying to feel that something useful will come out of this.

I think that now I can go back to blogging about duck poo and vegetables.

by Kristen at September 22, 2008 09:50 PM

July 28, 2008

Kristen Accardi

What is a Plumbers Conference?

After spending a few days manning the Plumbers booth at OSCON, I thought I’d post the answer to the question that everyone seemed to want to know – What is a Linux Plumbers Conference? We came up with the word “Plumbing” to describe the low level infrastructure of a Linux System. This includes the Kernel, desktop infrastructure like X and graphics libraries, system utilities like udev and hal, as well as essential libraries like glibc and friends. These components interface with each other at times – some better than others. We hope to provide a forum for people from these types of projects to get together and try to solve problems that are system wide or cross multiple project boundaries.

In addition to the topics to be discussed in the microconfs and the general talks (see http://linuxplumbersconf.org/program/schedule/), we will have “unconference” style talks. We have several smaller rooms available for people to get together and work out specifics, talk about something they didn’t get on the schedule, or have a group hug. These rooms can be reserved at the start of the conference.

August 18th the registration fee for Plumbers will increase to $300. If you haven’t already registered, what are you waiting for?

by Kristen at July 28, 2008 05:12 PM