Ivan Krstic: XO Crippled by Crack-Smoking Hobos, Not Sugar


It seems that the ever opinionated Ivan Krstić has his own story to tell around the Sugar Learning Platform, and it isn't pretty.

First, he admits that he was against Sugar as a new GUI with the fairly reasonable argument that OLPC did not have the resources to build a new GUI platform, so they should have built upon the decades of windowing user interfaces in mainstream Linux.

But its after he explains that the GUI was separate from the Linux Fedora OS, even if the whole "omlette" was called Sugar, that we start to learn the messy details of XO development. Details that make me surprised the XO software even worked when it launched:

In truth, the XO ships a pretty shitty operating system, and this fact has very little to do with Sugar the GUI. It has a lot to do with the choice of incompetent hardware vendors that provided half-assedly built, unsupported and unsupportable components with broken closed-source firmware blobs that OLPC could neither examine nor fix.

So we wound up with a keyboard whose keys get stuck. A dual-mode touchpad, capacitive and resistive, where one mode doesn't work at all, and the other makes the cursor spontaneously jump around and sometimes shuts off the touchpad altogether, prompting OLPC kernel developers to beg for saner hardware in the next round. We had board engineering issues that made power management practically impossible. We had a custom display controller chip that was incomplete in some regards, and completely broken in others.

We had an embedded controller that blocks keyboard events and stops machine suspend, and to which we - after a long battle - received the source, under strict NDA, only to find a jungle of nested if statements, twelve levels deep, and no code history. (The company that wrote the code doesn't use version control, see. They put dates into code comments when they make changes, and the developers mail each other zip files with new versions.) And we had a wireless chip that is so far beyond fucked, it's just about funny.

(Each of those words is a different link. Click them all, I dare you.)

Thinking back, there's a hardware incident I remember particularly fondly: one of our vendors sent us a kernel driver patch which enhanced support for their component in our machine. They chose to implement the enhancement by setting up a hole which allowed any unprivileged user to take over the kernel, prompting our kernel guy to send a private e-mail to the OLPC tech team demanding that, in the future, we avoid buying hardware from companies whose programmers are, direct quote, "crack-smoking hobos".

Wow. I think I may actually be speechless.

Related Entries


Now… THAT’S an interesting post!
It basically says that the highly praised (by all tech media outlets) XO hardware, is a piece of crap!!! From screen, to networking, to power management, to controller, to touchpad to keyboard, to…
I wonder how is going to play out with this same media outlets :-)

I guess some will say that the MIT wis-kids engineers (MLJ’s team) just f***ed-up and had no idea what they are doing
Some others will say that the fact that this thing works is a testament to the abilities of the wis-kids that defined their design goals and implemented then even with crappy vendors.
Some will contest the accuracy of the claims or if they are really above and beyond the usual problems faced by any new system built.
Will see… but it should be fun :-)

At the end of the day though it should be clear that the Sugar UI was never a serious problem in the XO software development (despite some opposing opinions http://lists.laptop.org/pipermail/devel/2009-July/025165.html ), the OS was.
So how to call this thing? RedHat, Fedora, Sugar and the involved engineers want nothing of it…
I guess will call it NNinux!

I cannot help but wonder what the story would be for a Mac Book Pro :-)

Yet this is coming from the same guy who still brags (on his "blurb" page) about getting industry awards for the never-quite-implemented, never-quite-working security system.

Watching Krstic's video from his Google talk ( http://video.google.com/videoplay?docid=-4285568518538296189 ) I realized that besides the UI and the Apps, almost the entire OS was re-written is Python. Not that is related to anything discussed here, but out of curiosity who's idea/decision was it to use Python for practically everything?
Any authoritarian info on that?

I don't know who made the Python decision, but I know why it was made.

Python was seen as the most accessible software for non-developers - the children that OLPC hoped would view source and be ale to build applications themselves.

Personally, I doubted that use case - I just don't see a ground swell of programmers coming out of XO deployments. Yes, there may be more than with other software and much more than a non-computer group, but enough to justify re-writing all in Python? I don't think so.

mavrothal, what are you talking about? Where you'll find Python in Sugar is specifically the UI and many of components/interfaces required for activites, this is quite different to a "OS rewrite in Python"!

And for what it's worth I think that using Python wasn't the worst of ideas. Though I'm a CS student I'm not really great at coding but Python and Sugar are making it reasonably easy for me to understand approximately what's going behind the scenes. So I do believe that it will allow a broader circle of people to correctly diagnose (but not necessarily fix) issues that they encounter in the field.

Relax guys...this is what actually Krstic says @ 24:30 when he starts with Python and the (fast passing) slides show that 'GUI+Window manager, presence and communications,init deamon-system boot, security platform and crypto services,"filesystem" (journal), search, all of user userspace' is/will be developed in Python. OK, is not an OS but a good chunk of it than just GUI and Applications
And yes I know the arguments for Python (and some against it) but I do not imply anything. I just want to know who (if any) was the driving force behind this decision.
Looks like a big job for 12-14 people (as Krstic says in the video @ 58:20)
So does anyone know who was behind the choice or not?

mavrothal said "So does anyone know who was behind the choice or not?"

In one word: no. But if you're really interested in how this decision came about then I'd suggest getting in touch with folks like Ivan, Walter or others who worked at OLPC at the time.

One last note: The video you're referring to was taken in early 2007 when quite a bit of OLPC's software infrastructure was more fairy dust than bits and bytes;-)

"One last note: The video you're referring to was taken in early 2007"

And actually shows the current approaches at this crucial time. The time in question...

"when quite a bit of OLPC's software infrastructure was more fairy dust than bits and bytes;-)"

So which one of the aforementioned elements is NOT in python you said?...

No wonder he works with the fruit company now! They share the same "over-opinionated I know it all" traits!

Ivan is hardly guilt-free in the creation of the Sugar omelette. Bitfrost is nice on paper, but hasn't been adopted by any distribution. It, as much as a new GUI, required the cooperation of every application on the laptop plus a seriously customized kernel (a later, limited implementation used existing kernel security mechanisms). And this was but one example.

Anyway, his software/security talents don't qualify him to criticize the hardware development. With decades of experience, I can say that XO-1 hardware debugging wasn't any worse than the typical product trying to do something "different" from mainstream. It's just not seen by most people.

Unlike any other hardware company I know of, our full bug tracking database is public, so you along with Ivan can track the incremental improvement of the XO-1 as problems were discovered. The keyboards have been thickened, the troublesome touchpads replaced, the suspend/resume problem understood and fixed before production started. The only "unknown" software in the system is the WLAN firmware. OLPC could have direct access but used outside contractors instead.

With XO-1.5, OLPC is continuing to improve the laptop, addressing some repair and maintenance issues along with the changes needed to stay low cost. We've tried to minimize the closed source firmware, but given that most peripherals have embedded microprocessors in them they are fact of life. We encourage open sourcing of code, insisting on it in some cases. In some cases, we support clean-room efforts like OpenEC.

Keep slogging away! You guys have done some amazing work. The fact that we can now go out and buy a $250 netbook is in large part due to the waves that XO 1.0 created.

...Even if those netbooks don't have displays that can be read in direct sunlight, batteries that last half as long, aren't as durable, etc. -Keep pushing the innovation envelope in hardware and software.

Sugar (a constructivist learning platform) is a long-term investment which has the potential to be even more revolutionary.

As a complete outsider I think that things are simple (!) As usually success has many fathers and failure is an orphan.

Negroponte says that the XO-specific OS development process was the big mistake. He calls it Sugar because a) this was the name at the time and b) (my hypothesis) the people that where at the core of the Sugar GUI development were also developing the OS-it is wrong obviously wrong. Sugar is a GUI. He claims that a more traditional OS development would be more successful.

Krstic basically agrees that the OS development had problems (and how could he deny it, we all saw the final product) but says that was not the OS development process the problem, was the crappy hardware choice and its vendors that hampered the development process.

Bender does not say anything other than Sugar is a GUI and has nothing to do with the troubles of the OS development (he is acknowledged though as a draft reader in Krstic’s post). He is obviously right is saying that today Sugar is (a nice) GUI and not an OS.
I do not really have any relevant expertise to say how much of the OS development was dictated by the Sugar GUI needs. Today Sugar runs in a number of distros but not all the underline features are implemented in these distros yet. So this tells me that it probably did have something to do with the XO OS development too.

Finally Jespen so far says, “go fish”. I did what I had to do, it was great and now I’m trying to do something better.

Obviously the truth is somewhere in the middle. Everyone has its portion in the success or failure of the OLPC/XO. Some have more and others less but the size is not important!... :-))) What is important is to go ahead and provide better and cheaper hardware that runs on an optimized OS, it has a feature rich Sugar GUI and make it easy to develop of MANY appropriate applications. The last one is the really important part. That’s what the vast majority uses the XO or any other hardware/OS for.

As I have said before “who is in fault?” never solved a problem. “What is the problem?” did.
So if the OLPC can develop a vanilla Linux that optimally utilizes all the hardware features and SugarLabs can run Sugar with all its features on this vanilla Linux and standardize Activity development, _lets get on with it_.

Lol it all makes sense now