Of course, I mean this figuratively – multimedia should be both seen and heard, preferably on high resolution displays with rich audio!
But the problem I’m highlighting (“I” being Martin Webb, Symbian’s Technology Manager for Multimedia) is that multimedia is a means to an end, rather than the end itself. Applications and users aren’t bothered about containers and codec profiles – they just want media to play, immediately will be fine thank you, but certainly not with any delays. Multimedia should “just work”.
Given Symbian’s transition into an open source project, I thought a good starting point for my multimedia blog would be to take stock of where we are. What are the key challenges facing the multimedia domain? I open with three:
- Playability Is King: Multimedia is a tool, and tools should work reliably. This isn’t yet the case – applications call “Play This”, and sometimes find the media isn’t playable. We need to ensure containers, codecs, stream sources etc work together reliably, failing only though a lack of device resources. This isn’t differentiating, it’s a hygiene factor – users assume that if a use-case is possible, it will always work, and so not working “negatively differentiates”. Sharing the burden through collaborative development is going to be an enormous help.
- Service Integration: More and more emphasis going forward will be placed on branded services – see the BBC’s iPlayer app on Nokia’s N97 for example. Such applications may want to manage video themselves, for example switching bit rates depending on network conditions. The platform needs to provide flexibility to these applications to allow them to differentiate.
- Picture Quality is Coming: devices are starting to appear that use tricks from TV land, such as dimming backlights, to improve picture quality. As displays get larger, this will increasingly become a differentiating issue. Ensuring the platform can work with these differentiating features will be key.
In short, the goal for the multimedia technology domain is to build a rock-solid platform onto which exciting new rich media services can be quickly deployed. Collaborative development is a big advantage here, as it frees up resources for focusing on new, differentiating multimedia use-cases.
Excellent stuff, Martin. Don’t forget music, as well. Current Symbian devices do not record playcounts or star ratings, which means when you synchronize them with your desktop music player, be it iTunes, Windows Media Player, Winamp, MediaMonkey, or other, neither of those important details are being sync’d over.
The result is a completely disjointed music experience, specifically when compared to the iPod+iTunes experience.
Would utilising a fellow open source project assist in navigating the container mine field? Yes, I’m looking over at ffmpeg. Most digital video that users own and watch – both legitimate and otherwise obtained – tends to be Divx/Xvid encoded .avi and the Divx mobile player isn’t available for anything >= Symbian^1 unfortunately.
Being able to play the more mobile friendly .mp4 is a definite positive step as both Sony (PSP) and Apple (iPod) use it. The downside to this is the messing around with converting. Being able to natively play media without the extra steps would be a great win. Also what about supporting the open standards of .ogg (both Vorbis and Theora)?
Symbian^2 currently doesn’t have any codecs at all.
You haven’t mentioned that as a “Key Challenge facing Multimedia”, so does that mean you already have agreements to fill this gap?
It’s 2009 and I still can not play any of the files I download off the internet. Transcoding is never going to take off. Just add XviD and DivX support, or at least make it easy for others to plug their codecs in.
Its funny, the Samsung i8910 should be able to play divx and xvids out of the box.
The N96 is more compatible with iPod formatted videos than the N95 and N97.
This bums me out as an N97 owner, I was expecting a better experience.
One issue with codec support is the the fact that there are patents held for most of them – all the MPEG family so that includes mp3 & mp4, Divx and many others. I believe that the i8910 as other Samsung devices are able to play Divx out of the box because Samsung has paid the patent holder to be able to redistribute the codec. This is an issue that is not easily resolvable unless software patents are dissolved (which isn’t going to happen any time soon
)
Ogg has the backing of all major OSS vendors both OS and application, as well as being the ratified standard for HTML5 for the distribution of media over the web. Why Nokia was scared of submarine patents, and as such not back the format I don’t know.
I think key to resolving this mine field of an issue, is for the Symbian Foundation to work with the major patent holders or even licensees that are able to redistribute. Help them to bring their products to market even if as an add on. Cooperating should provide the best option to satisfy most users/vendors.
Firstly, can I say I’m very glad to see such active discussion resulting from this post! There are many comments to reply to – if anyone feels I haven’t fully adressed something, post back.
@ricky – it certainly sounds as though improvements can be made here. I’ll raise it with our feature & roadmap council, and the package owner, but of course we would need to find someone willing to contribute these improvements…
The subject of codecs is worthy of a blog in its own right, and so I’ll give this some thought. To try to summarise Symbian’s current position – the most immediate priority is to a useful platform. It will take time to move everything over to EPL. In the meantime, this is likely to mean binary codecs available under an R&D license.
Moving forward, I would like to move towards a more modular multimedia framework, so that codecs, demuxers etc can be dropped in. This has several advantages:
1: cleaner boundary between the platform and device adaptation layers, and thus easier to port the platform to use hardware codecs
2: creates an easier path for contributions of OSS codecs (of course, the platform can’t exclusively consider OSS codecs for reasons of interoperability with services – I expect many IPTV-type services will go for H.264 or VC-1, for example)
3: cleaner boundary between “black box” components (most notably codecs) and the components that would benefit most for collaborative development (demuxers, stream sources etc). By this I mean that codec debugging, whilst by no means easy, is a boundable problem thanks to reference implementations etc. For me, the biggest hurdle to multimedia “just working” is the interaction between all parts of the system, and this is an area where we can all benefit from working together.
Hopefully this clarifies things, but keep posting questions and I’ll incorporate those answers into a more detailed blog soon.
What about Helix intergration? This was a key part (and one of the biggest changes in S603.2).
I think helix/nokia provideded some basic arm software codecs which is ok for testing or for supporting more esoteric formats. However, the ‘real’ accelerated codecs are the preserve of the semicos and subject to commerial licencing. They also require tweaking for various h/w architectures, memory speed and choice of dsp.
You need access to some propriatory code in order to optimise the handset for multimedia use cases.
If i wanted to make a phone from scratch with symbian OS. How much would it typically cost to license all the codes and pay for support?
similarly with opengl drivers. I would be very interested to know if symbian are encouraging developement of open versions of costly s/w assets? E.g openglses2.0 drivers for imagination’s latest offering.
Thanks for opening this discussion (and blog) Martin! I work on some of the Symbian Multimedia stuff so it’s very interesting to see people’s thoughts and wishes in this area.
Hi Martin, I don’t think you’ll get many arguments against a more modular multimedia framework. I’d be interested to know how far you think the planned/in progress move to support OpenMAX is taking us in that direction?
Indeed, Playability Is King and for that to happen as many formats as possible should be supported. The default music player for S60 is missing a lot there, where is the support for Ogg Vorbis, FLAC and so on? Being forced to use a 3rd party player is kinda lame.