Hello and happy 2010 from Chicago, USA. My name is Chris Karr and I’m the founder and chief developer at Audacious Software. My company is explicitly focused on creating the next generation of ubiquitous applications on platforms ranging from digital alarm clocks to mobile phones to entire physical environments. To learn more about how we approach ubiquitous computing, please see our company philosophy.

Haydn invited me to write about a Symbian product that I am currently constructing.

Back in September, my former graduate advisor approached me with a new consulting project that would combine some of my previous academic research in context-aware computing with an existing cognitive behavioral therapy system built by the local medical school. In graduate school, I worked with Dr. Darren Gergle to create Pennyworth, a Mac application that could sense and predict a user’s context using a variety of sensors combined with machine learners.

The new project was to augment and extend Dr. David Mohr’s moodManager therapeutic web application with Pennyworth’s sensing architecture on a mobile platform to enable ubiquitous access to a more proactive cognitive behavioral therapy (CBT). For those unfamiliar with CBT, it’s a psychotherapeutic method for helping patients deal with dysfunctional emotions using a structured goal-oriented set of reflective interactions.

The overall goal of our project was to create a mobile application that would employ environmental cues to suggest to the user when a CBT intervention may be helpful and to provide context-aware reminders when the user had previously scheduled an activity as part of their existing treatment. While the implementation is still underway (we hope to wrap up heavy development in the next month or so), we’re far enough along that I can share some details about the overall system.

First of all, after a lengthy discussion, we selected the Nokia XpressMusic 5800 as the hardware platform that we would be distributing to users in our pilot tests. We chose the 5800 over competing devices (iPhone, Android phones, other Symbian devices) for several reasons:

1. We could acquire a sufficient number of devices for our pilot tests for a reasonable price without incurring the obligation of the dreaded 2-year contract. For this project, being able to purchase an unlocked smartphone for around $300 was a major advantage.
2. The 5800’s hardware and software design was sufficiently robust, consistent, and user-friendly that we could be confident that supporting our test users would not become a major resource sink. We rejected the Android devices and the N97 because the hardware and software arrangements had not (yet) sufficiently matured for our comfort.
3. We needed background applications for implementing always-on sensors and we required low-level access to features like call logs and the Bluetooth hardware. These requirements eliminated the iPhone from consideration.

With the 5800 chosen as the hardware platform, we began developing the native application(s) for the device. We initially began programming with the base Symbian software toolkit, but quickly abandoned it for Qt when we found that Qt was a more productive development environment and that we could “drop down” into Symbian C++ as needed. For the first couple of months, the Qt code was written in Carbide, but we were happy to leave that behind when Nokia released S60 support for Qt Creator in the Qt 4.6 release. After resolving a few application signing issues (with the assistance of some very helpful Nokia & Symbian staff), we were productively developing in VMWare virtual machines and testing directly on the phone.

Figure 1: The Mobilyze macro-architecture

Our system employs the following architecture. On the phone, we run two native applications. The “UI” application provides a rich tree-based node interface for end-user interactions with the rest of the system. From this application, users may access the CBT training materials as well as provide training labels for the machine learner. Periodically, we prompt the users to enter information about their location, activity, and general mood. The UI application passes these labels to the “Core” application, which manages both the sensor suites and communication with the outside world.

Figure 2: Core application screenshots

We initially implemented both S60 components as a single application, but found that mixing the UI threads with the sensing threads resulted in a sluggish interface. By splitting the components into two applications, both parts became more responsive. The two applications act as one by communicating over an interprocess communication (IPC) channel implemented using the Qt publish/subscribe features.

Figure 3: UI application screenshots

The UI application is designed to be running only when there’s an interaction with the user. It can be shut down to conserve memory when the user is doing other things. Conversely, we’ve built the Core application to run continously. The Core component has three major tasks:

1. Manage approximately 20 sensors to collect information about the user’s location, ambient audio environment, running applications, accelerometer state, and so forth.
2. Periodically communicate (every minute, or so) the sensor readings to the cloud-based components for logging and mood predictions. Also, it relays any training information (with the most recent sensor snapshots) to the cloud components.
3. Listen for and relay any relevant data from the cloud to the user. This data includes mood predictions, notifications, and instructions to update the application.

Similar to our real-time IPC channel between the UI and Core applications, we also implemented an asynchronous IPC channel between the Core application and the cloud-based portions of the system. Since the mobile phones lack the resources to host our machine learners, we decided to host that technology on a more powerful system in the cloud that is addressable via XMPP. The Core application communicates with the learners through our master backend component using a modified version of QXmpp), which routes and logs messages between components in the system.  The existing web-based system (i.e. moodSense) can send real-time messages to the phone through the backend as well (e.g. schedule a notification to remind the user to review a CBT lesson).

Using XMPP as a common language between components allows us to augment the capabilities of the mobile device using technologies that would otherwise be too “bulky” for the device. In our Oracle component, we integrated the Weka machine learning system so that our mobile components would not be constrained in terms of predictive power by the mobile devices resource limits. Cloud-based processors also allow us to implement more computationally intensive sensors. Currently, we are building an audio sensor that extracts information about the user’s aural environment such as volume, whether people are present, and the nature of the local environment (train, home, classroom, etc.). We could not do this level of processing on the phone itself, but using our XMPP-based infrastructure, we can audio samples to a dedicated component that extracts the desired features. (Nuance and Google use a similar approach in their mobile voice recognition systems.)

The downsides to using the cloud in this manner are that network communication requires frequent transmissions (thus depleting batteries) and that any cloud-based functionality disappears when the phone is out of range of a wireless IP network. In the case of generating mood predictions, we’ve designed the Core application to download a simple model (a decision tree) from the Oracle when connection is present and to use that simplified model to generate mood predictions locally when connectivity is unavailable. Similarly, the XMPP portions of the Core app are designed to hold any messages intended for the cloud and transmit them when the network becomes available (a variation of the classic store & forward architecture).  Since we are still currently in the implementation phase, I expect that we will run into other limited connectivity challenges in testing and deployment, but our phone components are designed in such a way that we can add native implementations of cloud features as needed.

In the interests of brevity, I’ve omitted some of the more concrete details about the overall system, but I would be happy to answer any questions in the comments below.