Jason Whitmire, General Manager of Wind River’s mobile business, devoted more than 1,300 words in the official Wind River blog yesterday to Symbian.
Entitled “Open Source Symbian and the Inescapable Truth of Product Lifecycles“, his piece contains a lot of insight about the difficulties of open source projects, and also about potential drawbacks of legacy architectures. There’s some imaginative writing in the article. Indeed, I would say there’s too much imagination – and the conclusions are by no means “inescapable”.
Jason’s basic thesis is that Symbian has had its day, and that the Symbian platform isn’t capable of addressing the challenges of next generation mobile devices and mobile services. More interestingly, he argues that open source, far from helping Symbian, will actually hasten our downfall. Specifically, he argues
that its demise as it nears an end to a natural product lifecycle will only be accelerated when it is made available under an open source license next year.
Jason structures his argument around seven observations. I’ll follow the same structure. In each case, I draw a different conclusion to Jason.
(1) Shifting of costs to device makers (WRONG)
A recent tour of OxMs in Japan pointed to the possibility that, although Symbian will not charge royalties per unit, development, support, upgrade, and maintenance costs in the Symbian program will likely be shifted to each respective manufacturer as the Symbian Foundation finds that funding a staff of hundreds of Symbian engineers is unlikely to be offset by membership fees.
Yes, open source projects need to consider the costs of development (integration), support, upgrade, and maintenance – over and above costs of licensing. That’s a good point. Users of platform software need to weigh up the total secondary costs of working with a platform, as well as the licence fee. The former can often be larger than the latter.
However, let’s be clear: it has never been the plan that the Symbian Foundation will have a staff of “hundreds of Symbian engineers” who work on customer integration, support, upgrade, and maintenance – funded by membership fees. There’s been no plan for that working model, since that working model is not required:
- Package owners for each of the different areas of functionality are committed to both upgrade and maintenance;
- These package owners are spread throughout the community of member companies: they’re not Symbian Foundation employees. In the initial phase of the Symbian Foundation, the substantial majority of these packages are owned within Nokia (including by engineers from the former Symbian Ltd, that was acquired by Nokia), but these will become spread through the community in the months and years ahead;
- Assistance with customer integration and support will be available to OxMs in the same way as before – via a rich ecosystem of professional services companies (some are listed here).
The more interesting observation in this part of Jason’s article is this:
Indeed, by all indications an Open Source Symbian will require considerably more sweat equity and costly out-sourcing by OxMs to keep the platform viable in the face of accelerated competition and technology change.
Part of this is sheer imagination. There is no indication that the Symbian platform will fail to keep up with accelerated competition and technology change.
However, it is true that there is one temporary drawback to Symbian’s transition from closed source to open source. As explained on the “Platform Completeness” page of the Symbian Developer website:
Symbian is going to be completely open source… However, some technologies which were historically included in Symbian OS / S60 platform releases were distributed under specific commercial licenses and hence could not be included in the initial Symbian Foundation codebase. Essentially Nokia have contributed everything they can to the Symbian Foundation but some technologies are licensed from other companies and hence can’t be included in the platform until discussions with those companies have been completed.
Many of these items have since been resolved and solutions are now present in platform releases. The rest continue to be a focus of activity for Symbian staff and companies in the community – this is a key part of the completeness story. The issues we are trying to resolve are mainly Third Party IP (TPIP) issues
In concrete terms – as has already been communicated when we announced our Release Plan:
Symbian^2 is the first release for which the source code is available on the Mercurial repositories on the Symbian developer website…
Device manufacturers who are relatively new to the Symbian world are expected to use Symbian^3 in their first commercial shipments…
Symbian^3 will be part of a complete ecosystem offering of all the software needed to quickly create a wide range of winning mobile devices – including state-of-the-app mobile phones, and lots more.
That is, for some manufacturers, the secondary costs of working with Symbian^2 may exceed those of working with previous (closed source) versions of the Symbian platform. But by Symbian^3, these secondary costs should be substantially reduced. More information on progress with the completeness of the Symbian platform is available via our developer website.
(2) Developers are flocking elsewhere (MISLEADING)
Although the best of its generation, developers still generally consider Symbian an inherently buggy software…
I’ll be interested in quantitative data in the comparative “bugginess” of different software platforms. Here’s one point I can share: the release of Symbian OS that was made to device manufacturers and development partners, by Symbian Ltd, towards the end of last year – Symbian OS v9.5 – broke all internal records in terms of low defect count, keeping the agreed release schedule, and delivering the agreed release scope.
developers are choosing other open source platforms three times as often…
The link is to some data about the numbers of open source projects arising from different mobile platforms. Since the period covered is 2008, it’s not surprising that Symbian fares comparatively poorly. The interesting data won’t emerge till later. The corresponding statistics for 2010 will be telling.
(3) No signs of a future proof architecture (NO EVIDENCE)
If we move beyond smartphones, Symbian simply is not architected for next generation larger screen mobile devices and runs neither easier nor more cheaply
This is sheer conjecture. I conjecture differently: the step-by-step re-architecting of Symbian that has taken place over the years (such as the Screenplay and Freeway architectures, and the support for SMP) means it should make a great job of running new types of devices.
To make matters more complicated, Symbian will attempt combine MOAP, S60 and UIQ to create an all-in-one platform…
This part of the challenge has already been met. The Symbian platform has its APIs based squarely on S60, though aspects of the other UI systems are also present.
Bridging the gap between Symbian today and Symbian tomorrow will require the kind of careful architectural planning that is rare in a new open source consortium
I agree that rare skills are required. But there are some very fine architects who have been overseeing this process – both inside Nokia and in the Symbian Architecture Council. Again I mention: a lot of the planning and development has already taken place.
(4) Symbian is on a decreasing volume precipice (DEBATABLE)
In addition to reporting a 12% year-on-year decrease in volumes in the quarter the Symbian Foundation was established, Symbian now shows a whopping 15 percentage points of lost market share versus other high-end OSs.
A temporary decline in sales could be a prelude to many different outcomes. Yes, if you’re inclined to see it that way, it could be imagined as the start of a precipice. But it could also be seen as a sign of transition. In the period of time that our new engine of innovation is being put in place, we’re not unduly alarmed by current market figures.
Here’s a possible analogy – sales of Palm-powered handsets declined for a while. But now that Palm has come out with a new software platform, we can expect a change in fortune. I believe it may well be the same pattern for Symbian. The real question is: how good will the new Symbian innovation engine be?
I believe the highly competitive market will make it more difficult for [Nokia] to indefinitely [support] the Symbian Foundation for other manufacturing players who will have access to source code under a zero royalty license
Nokia has made its intentions clear with regard to supporting the Symbian Foundation: they expect that, by opening the platform and making it free of charge, it will attract many contributors as well as many adopters. This will provide business units inside Nokia with a wealth of choice for software to add into their devices.
(5) Barriers to Symbian roadmap assurance for all members (DEBATABLE)
it remains to be seen if the Symbian Foundation platform will offer a roadmap that allows for continued feature convergence, service delivery, innovation, and differentiation…
What will help this to be “seen”, one way or the other, is the happy fact that the Symbian Developer website increasingly contains more information about future platform releases.
Indeed, the cost of keeping a stack and platform current with ever-changing technology standards is immense, and it is unclear who will invest in this process in the Open Source Symbian platform.
Again, this information will be published in advance on our website, for the whole world to review and debate. For example, here’s the root page for our Feature and Roadmap Council.
(6) How long will Helsinki hang on to Symbian? (MISLEADING)
Rumors that Qt technology from the Trolltech acquisition is being architected in as part of a post-Symbian new platform generation
Er, this is hardly a “rumour”. The adoption of Qt technology is something that’s the subject of open discussion on the Symbian Council forums. But it’s not a “post-Symbian new platform generation” – it’s part of an evolution of the user interface layer.
For further discussion on this point, see my earlier posting, “Insurance misunderstood“.
(7) The devil in the Symbian Foundation governance details (OPEN QUESTION)
Finally, there remain a thicket of open questions around the operation of the Symbian Foundation and open source, not the least of which is what the exact support burden manufacturers will need to take on in both creating new designs and supporting legacy platforms using Symbian.
Agreed, there are lots of important open questions about the best operations of mobile open source foundations – whether the foundation in question is LiMo, the OHA, the Symbian Foundation, or any other. Anyone who claims to know all the answers is probably a bit naive.
However, at Symbian, we take very seriously the value of getting the governance right – encouraging and accepting input from as wide a range of participants as possible. We’ll keep publishing our latest best ideas on how this should work. We’re open to suggestions for anything that should be changed!
Incidentally, one thing that is clear is that support for “legacy platforms” (prior to Symbian^2) will be handled by Nokia (as the owner of Symbian Ltd). Support for versions from Symbian^2 onwards will be organised via the Symbian Foundation (in which Nokia is but one of several board members). As I mentioned earlier, companies who commit packages of software into the Symbian Foundation are expected to be responsible for the maintenance of these packages, subject to overall rules agreed by the Symbian Release Council.
Conclusion: a sense of urgency
Even though I’ve taken issue with some of the analysis from Jason Whitmire, I want to avoid any impression of complacency. I notice that some comments to the recent article by blogger Matt Asay (in which Matt said that I gave the appearance of being “calm”) suggest that:
Mr. Wood is doing a good impression of Nero, fiddling while Rome burns … perhaps Mr. Wood and his colleages are too relaxed
Being calm is one thing. Being complacent is another. My view, as it happens, is that there is no time to lose. The mobile marketplace is highly competitive and highly complex. Even though some parts of the Symbian schedule are already mapped out and are most unlikely to change (such as the release dates and principal contents for Symbian^2 and Symbian^3), the Symbian community needs to double down to be sure to accomplish our essential priorities as quickly as possible.
So I’ll end by echoing some words from the fine book “A sense of urgency” by Harvard Business School Professor John Kotter. The biggest reason why significant change initiatives fail, in Kotter’s considered view, is because of a lack of:
a real sense of urgency – a distinctive attitude and gut-level feeling that lead people to grab opportunities and avoid hazards, to make something important happen today, and constantly shed low-priority activities to move faster and smarter, now.
The curious thing to me is that people who continuously take a jab at Symbian for being somehow old or out of date are almost always promoting a Linux-based alternative. While I’m an admirer of that operating system, I can’t help but note:
1) Linux is of a very similar age to Symbian and based on an even older paradigm (as far as OS kernel design goes).
2) Linux was not designed for battery powered devices and the design has not generally been evolving in that direction.
3) Nokia has significant internal Linux product experience and capability, plus they aren’t entirely stupid. If they believed that Linux would provide a better alternative in the near term then they wouldn’t have invested so much in buying Symbian, opening the source and improving UI, they would have simply dropped it.
Indeed, Nokia’s director of open source operations, a very strong proponent of Linux, publicly stated that Linux wasn’t the right platform to deliver phones on in the near future and that Symbian was superior for that purpose.
IMO, if the Symbian Foundation doesn’t deliver the expected increase in co-operation and innovation around the platform then a Linux-based alternative could overtake it in the next 3-5 years. That is a very long way from a certainty though.
if ( iWindRiver.Owner() == iIntel)
{
iFUDArticle.WriteL();
}
else
{
iWindRiver.DevelopOS();
}
if (Os != iWindRiver.OS() ) // Linux, VxWorks
{
iFUDArticle.WriteL();
}
@Mark Wilcox: I’ll answer to your point.
1) Ehm, the Linux community of main-stream developers were never scared of throwing in the bin any sort of retro compatibility in the name of “evolution” and “innovation”. That makes Linux, based on the “old” monolitic+modules pattern way more “modern” and “up to date”.
Compare the Memory Management, Scheduling and Virtual FS of the 2 OS before to say that “they are equally old”.
2) Ehm, really? Linux never evolved in the direction of improving “battery life”? Are you sure about that? Did you ask people like Intel, Google or… Windriver?
3) “Nokia… they are not entirely stupid”. True. But having been successful for years and years could have made “the nokia folks” too much relaxed on their dominance.
I’ll give you an example: N700 and N800. Those device, linux based, were the first attempts of Nokia to commercialize mobile devices with Linux. If you did a “diff” of the Vanilla Linux kernel and the Nokia version, you would have seen very (very) little work in terms of “battery life”, “scheduling optimization” and “realtime”.
Do the same with the Android Kernel against the Vanilla one… and have fun.
My conclusion?
This “challenge” is first technological, then economical. Renew Symbian for good. Focus on Orbit. Get rid of AVKON. Cut any retro-compatibility. Throw in the bin PlatSec. MAKE A NEW SYMBIAN, putting in all the superb knowledge that years of market dominance gave to the Engineers.
Then focus on “selling it”.
The Google philosophy is an example: make it good, build it new, put in bleeding-edge technologies, release often and make ANYONE able to build and run.
The market and the crowd will come.
OK, I agree that it’s better to chuck out compatibility sometimes in the name of innovation or evolution. However, IMO, Linux goes way too far in the opposite direction. Try to develop a single binary that targets any decent subset of Linux platforms… you can’t. That’s not so important in the desktop space but it is fairly critical for embedded consumer devices.
Of course people have been working on improving battery life on Linux, particularly recently. However, that isn’t the core focus of the main developers and many features are not implemented with that in mind. What about the Linux approach to SMP vs the Symbian approach?
The Nokia internet tablets deliberately avoided those weak areas on Linux by not including phone functionality. The battery life on those devices is absolutely fine, you can use them for days without a recharge – why would Nokia need to work on it (and they are of course now that they’re adding some telephony features to Maemo 5).
I agree the time has come to renew the Symbian UI layer and that’s exactly what is being worked on. Including getting rid of Avkon – have you read the design proposals. Platsec will not get thrown in the bin, for the same reason the native SDK on Android is unlikely to be opened up by OEMs – the operators and OEMs are completely paranoid about this, they want a security solution. They’re even a bit unhappy about how open the Java environment is on Android from what I’ve heard.
I agree with much of what you say about the Google philosophy but there’s a very critical point here – you can’t “build it new” every year. You do need to worry about compatibility and carefully control breaks or all the developers you attract will get fed up and move off to the next new thing.
But isn’t this what the SF is doing, more or less?
(fast-forward to Symbian^4 of course)
P.S. Genuinely interested here and would like to be enlightened:
Memory Management – Symbian has lots of different memory models, are you talking about one in particular? By contrast, what’s so great about the memory management on Linux. I was never amazingly impressed when working with it…
Scheduling – really – what’s so good about that? The nano-kernel scheduler on Symbian looks extremely well designed for purpose to me. In contrast the standard Linux scheduler doesn’t do real-time and even soft real-time is tough with the real-time patches. I’m not just making this up, I’ve been actively involved in trying to push high definition video across a network from a Linux set-top box.
Virtual FS – OK, you definitely win on this one, the Symbian file server is antiquated and lacking here. The only problem we found with this, is that if you want to do something ‘clever’ on Linux in the file system space you’re caught by the GPL, of course there’s FUSE to implement your file system in user space, but then you get caught by a big performance hit. I think this is where Symbian has a huge commercial advantage in general with the EPL.
So I’m still completely unconvinced by your argument, but I’m happy to be enlightened to the contrary.
I start from the last comment and catch up back.
Memory Management: Symbian doesn’t even implement a real Virtual Memory. It’s a “Virtual Addressing”. Good ideas are in the “demand paging” but still it’s something implemented on the top of old stuff. A nightmare for the engineers making it (and I was there).
Scheduling: Symbian “real-time” guarantee is a myth for me. Explained well during courses (Internals) but… where exactly are we supposed to see this real-time scheduler difference? The fact that is “theoretically” realtime… doesn’t make it so in reality. It’s a Priority-Based + Round robin algorithm. Nothing extremely new.
Virtual FS: What you mean that “you can’t do something clever on Linux in the FS area”? That’s not clear for me. I can mention, anyway, that Android provides support for a new FS designed for mobile (don’t recall the name now). It didn’t require voodoo art to do.
Binary Compatibility:
I think this is irrelevant for this discussion. “binary incompatibility” between different Linux Distribution is caused only by different FileSystem Layouts and different libraries available on those distribution. When you build a Mobile OS over Linux, of course you choose your Layout and your Libs and go for it. No?
SMP:
I don’t know your background, but if you have the possibility to speak with Engineers in Symbian that worked (and are working) on SMP, your will hear a lot about how a system mainly based on the HORRIFYING concept of “Active Object” turned out in a “slot machine” the day they introduced SMP.
And, again, for the 1.5 year I spent in Symbian I followed this stuff as much as I could: challenges always amused me.
In other words: the entire OS is designed around concepts that made SMP a nightmare.
Linux supports SMP since I was a student in University: probably was not as good as now, but for sure no one had to rewrite major software because “SMP was breaking it”.
New UI:
I’m following the design proposal of the “new UI” and I REALLY HOPE that who is supporting to maintain binary compatibility is not given credit.
For my personal experience, binary compatibility was a “service” that symbian provided to the licensees. When it reaches the final developers, the code compiled for Nokia N95 does always work on Nokia N95 8GB. And that is the biggest example of a constellation of SDKs delivered by Nokia.
Sometimes I feel like there is 1 SDK per phone…
So, let’s just get rid of binary compatibility for good
PlatSec:
PlatSec is an abortion because, in the name of a phantomatic security, Innovation has been stopped in it’s own home: 3rd party developers.
I was sceptic in the beginning of the Android SDK launch: then I saw what they meant and… I just love it.
Developer Signature is the only warranty needed. And the Java Sandboxing, in this case, it’s just the perfect companion.
Final bit:
Ah, I never said that “a new platform should be built every year”.
I’m saying that you can’t keep building over a platform this old forever.
The time for doing the “Big Bang” again (like when PlatSec was introduced) it’s arrived… long time ago…
PS Sorry if the comments is a bit messy, but it’s complicated to discuss so much in a blog
@Kensai: They are “discussing” if they should do it. That’s what concerns me. They are “not sure yet”.
Well, I don’t have a deep technical knowledge of the various platforms to support or counter your arguments, but from what you write I understand that there is a need for a leap of faith, even at the point of breaking backwards compatibility.
Most of us expect that “Big Bang” to happen with Symbian^4 in a year or so. Will it come too little, too late? I don’t know… Nevertheless, I tend to admire products that succeed in reinventing themselves!
> The only problem we found with this, is that if you want to do
> something ‘clever’ on Linux in the file system space you’re caught by
>the GPL
How so?
The caught by the GPL thing – if you want to build a new file system (and that’s just an example of something you might want to do) then you have to do it kernel side and thus it’s subject to the GPL. Some people argue this isn’t so, but there is enough uncertainty that companies are extremely uncomfortable with it. When you’re forced to publish source code via the GPL it’s extremely difficult to monetise R&D effort unless 1 large customer is prepared to pay for the whole thing.
HORRIFYING concept of “Active Object” turned out in a “slot machine” the day they introduced SMP.
Perhaps I’m just odd, but I beleive active objects are actually a very good design for event driven programming. Lets not create more FUD here with references to inside information. Active objects are only scheduled within a thread, Symbian also uses separate threads and processes and these are the things that get scheduled across cores by SMP. Of course there were complications in the implementation, but I haven’t been involved in a single complex software project where there weren’t horror stories, and it’s human nature to like to tell them to engineers not working on the project.
Virtual Memory: Try to create code that’s robust to memory allocation failures on Linux, the problem is, you can’t because memory allocation (almost) always succeeds and gives you a virtual address – it’s only when you come to use it and there isn’t enough physical memory that the error occurs. Unless of course you have paging out to disk – which is generally avoided on embedded systems, so you’re left with the old scheme of trying to find the maximum memory use of the system… which is pretty tough when you’re open to 3rd party apps (obviously while it’s only Java apps, you can handle that pretty well). The Symbian demand paging stuff preserves the ability to manage the actual physical memory.
BC in Linux – I’d agree with your point if there were any real attempt at a single mobile platform across manufacturers. Unfortunately there are dozens of them and little agreement on which ones to use. Perhaps if LiMo and the OHA were to merge and open up a bit more it would form a strong enough centre…?
Sometimes I feel like there is 1 SDK per phone…
That’s just completely unfair! Nokia has been nowhere near as disciplined with BC as Symbian in the past, but it’s not a fraction as bad as you’re making out.
PlatSec:
It has been a disaster for 3rd party developers so far, particularly small/independent ones. At the time it was created there was a real fear of mobile malware – the alternative was that there would be no native app installation. There a lot of work going on to improve it now, but I believe throwing it away completely would be a mistake. The mobile malware threat is not entirely “phantom” although it has been blown out of proportion in media.
New UI and Big Bang and compatibility going forward:
Yes, we agree here generally – I’m on the side of throwing out compatibility. I’d just like to see more of a migration path (hopefully before Symbian^4, and via firmware upgrades to older devices where possible), so we don’t completely wipe out the app ecosystem. The last “Big Bang” very nearly retained source compatibility – this one won’t in any way. After this we should inherit the (extremely well balanced IMO) policy of only breaking BC on a major version change from Qt.
Thanks for the interesting discussion! You’re right, blogs aren’t very good for it. If you want to discuss such topics further it’s probably best to do it on the Symbian developer forums.
This is just my opinion but I think that Symbian based phones will get less popular if user can not update the operating system to new version. We have seen this happening with Windows Mobile (like Samsung offers free update to 6.5 from 6.1), iPhone and with Android devices.
Tupe: Completely agreed, this has been raised several times already. Fundamentally the platform already supports this and it’s up to OEMs and operators to provide and approve the updates. Historically there used to be a cost involved to licensees to move to a later version of Symbian OS, but this has of course been removed now.
@Mark: I think we agree that those are personal position based on our personal perspective and experience. So, no need to argue more.
But I want to comment one piece:
Seriously, I believe there is a massive difference between the incredible work that Symbian did (does?) internally to maintain BC… respect to the final result of Nokia.
In my working projects we had to deal with all sort of differences between N95, N95 8GB, N96.
And when 5800 arrived, supposedly Binary Compatible, even more odd things happened.
And, of course, when we moved to use the S60 5th Ed, all the effort done to work across N95/6 was lost.
BC is maintained in terms of “code still running”. The problem is HOW the code works. One example: Drawing/Rendering completely “f**ked up”.
“I don’t know your background, but if you have the possibility to speak with Engineers in Symbian that worked (and are working) on SMP, your will hear a lot about how a system mainly based on the HORRIFYING concept of “Active Object” turned out in a “slot machine” the day they introduced SMP.”
This is simply not true.
Active Objects (and their associated Active Scheduler) are single threaded so in fact Active Objects are a very good base-framework for SMP.
@Michael: Ok, I’ll give you a scenario.
In a System where were generally preferred ActiveObjects over Threads, having only ONE CORE is pretty simple and straightforward. Actually, ActiveObjects make the “asynchronousity” a “cheap bless”.
When you have MultipleCore or, even better, SMP, the fact that the ActiveObjects are “caged” in a Single Thread make it “work” but… the hole concepts of SMP looks pretty “useless”. Practically, ActiveObjects became a “quick win” and make any “SMP power” pretty useless.
It’s why some stuff, like “ActiveObjects used into BT drivers” or other stuff was starting to be restructured to use real Threads.
And that’s when the crap come out: the ActiveScheduler can get pretty predictable if the priority are played “unfairly”, and this makes code “theoretically concurrent”, just simple “serial”. Introducing Threads instead of ActiveObjects just breaks everyone of this “programmin assumptions”.
And, adding up all the work done with ActiveObjects along the years, made the amount of code having this “problem” quite considerable.
Am I clear now?
Perhaps the fact that multi-core ARM processors for smartphones are just around the corner might speed up the necessity for a paradigm shift in how Symbian OS deals efficiently with threads.
Technology updating needs are good “excuses” for deep innovative changes. Even at the cost of breaking compatibility…
@Kensai: Absolutely. I couldn’t agree more.
But, again, the point here was the execution, the executor and the executability.
SMP Arm is been developed for and with Symbian: after all this operating system was the most used in the world for a long (long) period. The point is in the fact that the switch to SMP has turned into a nightmare “because of the way” Symbian OS is/was.
Ivan, I think you are a bit confused about what you’ve heard on this subject, or at least aren’t explaining it very well. Let me try to explain the issue and see if it makes sense to you?
First, although active objects are used in preference to threads throughout much of the system, this doesn’t prevent SMP from providing more power at all, that’s just nonsense. The reason is that at any time there are lots of different processes running and these can be scheduled on different cores. In the simplest abstract scenario, an asynchronous request to a server, the server is in a separate process, so the application can carry on using one core, while the request is handled on the other core. No different to the standard threading scenario. Add to this that the Symbian SMP solution only turns another core on when there’s an extremely high load on the system and you can see that the extra power is not lost. If the whole system were running in a single thread using one active scheduler than that would be true, but this is so far from true as to be laughable.
Of course extremely heavily loaded servers that were using AO’s as the most efficient multi-tasking paradigm can now get better performance by being re-implemented with threads.
The problems arise when engineers assume that there’s only one core and that the relative priorities of two threads/processes will mean that one thing is guaranteed to complete before another. Exactly the same issue exists when a multi-threaded code base transistions to SMP, the difference is that with multi-threading you’re more likely to have protected shared data with synchronisation objects anyway (mutex/semaphore/critical section – multi-threading forces you to think about that stuff) whereas with AO’s you don’t often have to do this.
As I heard it (also from engineers and consultants that worked with SMP on Symbian), most of the bugs of this nature were in test code, although there were quite a few in the main code too. The SMP engineers condensed their experience on this into an SMP-safe course, which I was lucky enough to attend a condensed version of for free at the last Smartphone Show. They also created a “crazy scheduler” tool that randomizes thread execution order on the emulator to find these assumptions in your code.
And finally, to come back to the comparison with Linux. The Linux kernel may have supported SMP for years, but a lot of the code that runs on Linux embedded systems has never been tested with SMP. A certain silicon manufacturer that produced set-top-box chips had a complete set of drivers that weren’t SMP-safe. Their initial solution to the move to SMP was simply to force the entire system onto one core, with the other only used for a small set of pre-defined tasks! Much less efficient than the solution that every Symbian device that uses SMP will have.
If the purpose of Symbian SMP was to run multiple processes and not multiple threads you would have been right.
As I said already, the complications arise when running multiple threads and, even more evil, more threads within the same process.
I’m just being lazy with language. A process is a unit of memory protection with one or more threads running inside. Two separate processes are (at least) two separate threads.
The situation I described above about priorities is the standard situation with multiple threads inside a process. Typically, when the threads run in separate processes, they’re using some form of shared memory or kernel mediated messaging to communicate and were already using the appropriate mutex/semaphore/critical section where necessary.
There is nothing worse, or “more evil” than what I described above. Or, if you believe there is, feel to explain a specific case that isn’t covered by the general issue I outlined (implicit engineering assumptions about execution order of threads).
Moving to SMP is difficult whatever system you are working with. It takes broader experience with different systems undergoing that transition to get that perspective.
Are you assuming that I’m talking about this stuff without knowing them?
Man, let’s not get into “skills measurements”.
I don’t know other way to make you understand Mark: with ActiveObjects engineers at Symbian (or in company developing components/drivers for it) did assumption on the impossibility of two “AO” to run in real concurrency.
When they moved from AOs to Threads on SMP this “disallineated” the assumptions.
An example: AO1 set’s a lock on Interrupts, does stuff, removes the lock. AO2 does other stuff using Interrupts or other IO related resources.
When you turn them into Threads, the party starts: Thread1 locks the Interrupts and, while he is doing stuff, Thread2 requires the Interrupts to be enabled.
My last attempt here: ActiveObject is evil in the sense that made people write code assuming a lot of “seriality” where they were not supposed to. If Threads would have been used all around, this kind of scenario would have been solved way more easly.
Multiple SMP-fixing time with all the work-refactoring that Symbian needs… and you will see.
I can see that humbleness is not in your deck of cards.
OK, sorry, lets please not argue further. I think we both understand the issue and just disagree on how bad it is. I know you worked for Symbian and actually I’ve seen your contributions elsewhere and am pretty impressed.
I’ll try one last clarification of my position…
Two AOs in the same thread could never and still can’t run concurrently. Two AOs in different threads didn’t used to be able to run concurrently but now they can. However, this is exactly the same as if you remove AOs from the equation completely. The only difference from a developers perspective is that you get used to not having to think about concurrency with AOs (which is the nice thing about them but also a danger). The good thing with AOs is that there are fewer threads, so fewer places for this problem to occur.
However, having seen the same transition in an embedded Linux system I can tell you confidently (although I’ll try to be a bit more humble
) that programmers made exactly the same assumptions about code running in tasklets and from work queues in the kernel (for drivers and custom file system modules, obviously the kernel itself had already been through this process), and even high-priority threads in user-space. In a single core system it is a valid assumption (although it turns out not a good one) that a high priority thread will complete what it’s doing before a low priority one gets to run. With two cores that’s not valid. Fixing this was so much work that the first products with two cores were actually reasonably likely to ship massively under-utilizing the second core.
I think our disagreement is fundamentally over this point:
If Threads would have been used all around, this kind of scenario would have been solved way more easly.
I don’t actually believe that if threads had been used all round programmers wouldn’t have made exactly the same assumptions because I’ve seen plenty of evidence to the contrary on a system without any AOs. Of course we can never know for sure, so our dispute can never be resolved.
Actually, my last comment in the previous post wasn’t aimed at you, but more saying that internally in Symbian there may have been a lot of finger pointing from engineers in the direction of AOs and much wailing and gnashing of teeth. However, unless some of those same engineers had done the same transition on another OS, then they weren’t in a position to make a comparison.
To try to play the humble card, I’m very aware that my experience is very broad but also very shallow in most places (there are a few exceptions but not that many). I’ve seen a lot of systems in a lot of different companies. If I didn’t believe from that perspective that Symbian was still the best platform for the industry I wouldn’t be working for the Symbian Foundation right now. I certainly had the choice to be doing embedded Linux work instead.
“If Threads would have been used all around, this kind of scenario would have been solved way more easly.”
@Ivan, I disagree.
Thread creation has a performance overhead over active objects (so i’m lead to believe).
But more importantly, if we didn’t have AOs it would make the code alot more complicated as a great deal more synchronization would be required. Which certainly isn’t “easy” in complex code.
Its like Mark said, there are plenty of active schedulers not one as you inferred in one of your previous answers.
Also your example about interrupts doesn’t make sense, as if these two AOs are on the same active scheduler then the behavior is unchanged on SMP.
CActive is, without question, as developer-hostile an approach to event-driven and async programming as you’ll find anywhere. From a usability perspective, there’s no workable defense. But show me any system with an event loop and I’ll show you the same kind of assumptions under discussion here being made as a consequence.
Mark’s right, there’s no such thing as a free lunch when it comes to the exploitation of SMP hardware. Rarely do you get useful emergent behaviour simply through having threads lying around. You have to design your approach carefully. And from experience, given the choice between (a) SMP-optimizing a component that currently works reliably but serially, and (b) SMP-optimizing and debugging a component that is heavily-threaded but was never designed for or tested on SMP hardware, I’d choose (a) almost every time.
Thanks for the support guys…
Thread creation has a performance overhead over active objects (so i’m lead to believe).
Technically it’s the context switch which causes the increased performance overhead for threads, and this argument still applies, although as CPUs and memory get faster and the amount of processing going on between context switches increases, it becomes less relevant. When you’ve got SMP you can afford to do longer running tasks in a separate thread without worrying about the performance impact (no more background processing with AO’s).
CActive is, without question, as developer-hostile an approach to event-driven and async programming as you’ll find anywhere.
I do take issue with this. I guess it depends what you come to it from, but having worked on feature phones with simple RTOSes before Symbian devices, I’ve seen enough components with multiple message queues for different priority messages, or hacks that take messages off a queue until a certain type of message is found and then put them all back again, that I have to disagree. I actually found the AO approach quite an elegant way of doing a priority based event loop.
I’m sure they could have worked out a way of implementing it such that developers wouldn’t have to write so much boiler-plate code, or remember to do so many things in the right places. If that’s what you mean by developer-hostile then I’m agreed, but would argue that providing a system that isn’t actually flexible enough for all the problems you need to solve is even more developer-hostile, and there are plenty of those.
@Michael:
I disagree. Would have been more complicated but way more correct “semantically”. AO makes developer lazy.
And is a “kind of programming” to which only Symbian developers are accustomed.
If they were done with Threads since the beginning, with proper “synchronization mechanism” in place, SMP switch would have been easier since the beginning for Symbian.
After all that’s what I’m trying to say here since the beginning!
Said as you say of course it’s non-sense: indeed I said that “WHEN YOU SWTICH AOs with Threads…”.
Why anyone should switch them? Because if you don’t try to use Threads more, why you started to work to support SMP in the first place?
I leave with a thought: why do you think so few “single developer” actually did develop stuff for passion on Symbian? Compared with platform like, I don’t know, iPhone?
And, after 50.000 apps, I don’t think anyone can question any more the size of the success of that device.
Symbian is surrounded by some sort of “elite” society of ActiveObjects Wizard and Descriptors Guru that defend the awkward paradigms on which this OS is based. Paradigm that were designed for Hardware and Memory Sizes that are now history.
Guys, this OS needs to be refreshed.
Innovation doesn’t happen only “giving flexibility and full control of their code”. Innovation happens when there are good, stable, well-known instruments, based on COMMON STANDARDS (stuff you can learn at University), on which other people build innovative ideas.
I can’t still be writing the full skeleton of a Client-Server architecture over and over again. And it’s unacceptable any more that if I forget to put an “ActiveL” somewhere, I waste a day figuring out why.
Innovation needs enablers. And a commonly known base, made of shared knowledge, is a mandatory condition to make does enablers.
Side comment:
The (as yet un-released) v2 of EUserHL contains framework code that significantly reduces both these difficulties you mention.
// David W.
OK Ivan, now you’re moving onto slightly more solid ground at the end there.
I don’t think descriptors are really worth defending – far too much complexity. Similarly, far too much boiler-plate code needs writing for a lot of things in Symbian C++. What you were attacking before was the fundamental design, rather than developer usability – yes, many of the paradigms were designed for a much more resource constrained environment that we have now and the programming overheads for app developers are no longer justified.
Nokia has decided Qt is the solution and I think it’s an excellent one.
For the system programmers, I think that if people can’t get their heads around active objects and descriptors, they should probably leave the system code alone! Linus Torvalds has a similar attitude with the Linux kernel, having famously been against kernel debuggers, since they make programmers lazy.
Absolutely my last comment on this thread. I leave you with a quote from Bjarne Stroustrup:
‘Legacy code’ often differs from its suggested alternative by actually working and scaling
> If they were done with threads from the beginning
In terms of context switching for ARM processors prior to ARMV6 this was extremely expensive, and in fact the design of the older Symbian memory models handled this hardware limitation fairly well.
Even if Doc Brown came back from the future to tell you needed to thread your AO frameworks it wouldn’t have been particularly clever to heavily thread your software.
Regardless of AO and Symbian. If you do some OS archaelogy your argument doesn’t make sense because the use of threads depended upon whether good SMP support was required ‘at the time’
Most OSs have their roots in a single CPU architectures and SMP support came later on. It shouldn’t need stating that the Linux kernel, filesystems and networking didn’t start off as miraculously SMP ready because they got it right first time. It required a lot of work to multithread these systems properly when it was seen that SMP systems were becoming more common place and such threading was desirable.
Talking about active objects vs threads as a root flaw in the OS is pretty pointless really.
@Ivan: I’m sure plenty of people share your frustrations with the relative usability of Symbian OS. It’s just your theories on the subject of SMP, and the conflating of the two concerns, that come across as a little bit naive.
@Mark: Event queues, even simple ones that you have to build layers over, are at least conceptually simple, general, and debuggable. Disconnecting signaling from data delivery, then request from readiness to receive, and so inventing stray signal conditions… less so.
As with descriptors, you can respect the higher-level goals while still despairing at the user-unfriendly execution.
@PP: I wish I could bring some guys from CommsFW in here to comment on SMP and all the problem the switch gave them.
I don’t disapprove AOs in old hardware: Symbian… I should say Epoc/Psion was solving the problem of limitation of the hardware in a great, efficient way.
I disapprove AOs TODAY. Hardware with more memory and better CPU was out way before Apple putted their mind on phones.
Symbian should have adopted common “coding standards” before.
But I know, then we start the discussion about BC and so on… so I’ll just stop here.
Re: point 2 “developers are flocking elsewhere”
If they are flocking elsewhere, I don’t think that quality issues in Symbian OS are to blame. In fact, I rarely get to use Symbian APIs other than descriptors and the like. Quality issues in Series 60 or specific phones are another matter altogether, as is the very high level of variability between devices.
If you’re to succeed, then I think you need to take a holistic view of the phone platform, as there is a large chasm between where Symbian OS 9.5 ends and the problems of application developers begin.
@IvanDM Perhaps you might ask yourself if a new, complicated, architecture in comms framework, that was very diffierent from the existing Symbian OS architectures, might be partly to blame for your views?
You might also like to consider the views of some of the people who work with Symbian OS as a whole system, rather than just dealing with one subsystem – some of us actually see how the extensive use of client-server designs have the potential to scale quite well to SMP platforms. It is quite definitely not the case that you need to thread the hell out of the system to improve performance; indeed that approach can actually reduce performance in many cases. The mantra of “more threads is better” is nonsense when applied repeatedly.
Perhaps you could restrain yourself until you actually see and use Symbian OS running on a multicore system? The perhaps you could speak with some authority on the matter.
Nice, now the Symbian Blog does apply censorship to comments?
Yesteday I posted an answer to Iain and… now is gone.
Great work!
Hi IvanDM,
Definitely not censorship – unfortunately your comment was the victim of our sensitive spam checker, which picked out some language it didn’t like and removed your comment. Please post your comment again, as we’d like to be able to address your comments in full.
Best,
Ashlee
Seems like a spam filter didn’t like my “colourful” language, so… here we go again.
@iain: So, practically you are saying, that to comment I need to know the entire system, and the causes behind things.
So, let’s make every application developer of Symbian study the whole source code so they understand how to code for the system.
And while we do this, why don’t we ask to do the same to every Windows, Mac, Linux developer: everyone will spend ages learning the reasons behind awkward APIs, and do better applications.
It’s a very effective solution to the problem.
But, I have an alternative proposal: what about avoiding position based on “I know more then you, so shut up?”. This could be very useful for an OS that is making of the Open Source it’s new mantra.
I never said to use Threads everywhere, BUT they are more natural to program then Active Objects. And that current hardware is not as bad as in the past, and multi-threading should be seen as “more doable”.
An example of “well done Active Objects” is the “QTimer” within the Qt framework: nothing absurdly complex like the Active Objects, but few methods that just do the job. And, funny enough, this class on Qt does embed an Active Object!!! That’s how bad Active Objects are.
It’s WRONG to think that “C++ is a not-for-kids” language: the MAIN TARGET of a framework should be FIRSTLY to make complex things easier, faster, leaner to do. This can only grow the amount of developers!
Qt is the magnificent example that is finally making me enjoy Symbian Application programming.
Another example? iPhone! It’s programmed using the almost unknown Objective-C… but there are hundreds of thousands Apps out there. Why? Because, once got a grip on the grammar, Objective-C is enriched by Apple with Cocoa, a superb framework. Plus, of course, plenty of nice and powerful tools.
There are literally kids building for it. And even though their work is not going to make millions itself, they are going to turn, eventually, into professional Developers, with real company and real money behind, and the expertise built in their dorm-room is going to be translated into successful million-making applications.
About “waiting to see Symbian OS on SMP before I judge”: I think we waited enough. The marketing about SMP started way (WAY) before I left Symbian… and still, we are here waiting for seeing any hardware using it effectivelly.
You are promoting threads on the one hand but dismissing SMP on the other.
Does this not sound a little at-odds to you IvanDM?
@Michael Cox: I probably mis-explained myself here: I don’t dismiss SMP! SMP is a great thing and a solution the need of more power without necessarily improving frequency.
SMP were introduced in the discussion because of the fact that the intensive usage of Active Objects in the building-blocks of Symbian made the introduction of SMP a “problem”. A lot of linear-code became non linear because of the SMP. My point in this was: “if everything would have being done with threads, the concurrency problem would have been addressed in the first place and introducing SMP would have been less dramatic”.
My final “bit” was to say: “see how long ago Symbian started to publicize the introduction of SMP… and how far down the line of time we got, still without SMP phones”.
Did I explained myself now?