Sync software for managing contacts, events, etc across devices can be very helpful when it works, and very frustrating when it doesn’t. In my experience, when an error occurs it’s often unclear how to rectify it; and even when there is no error, it’s hard to judge whether it worked because many updates may have been made to my data and it’s unclear whether the updates were correct until I run into an instance of not being able to find a contact, or finding that I have multiple instances of the same reminder.

I was really motivated to find a solution, and recently I had the opportunity because my employer chose to offer a sync service by using the open-source software shepherded by Funambol.com. This article describes how to use Funambol’s software to allow users to sync with a pre-existing store of their event and task data.

Funambol provides a very capable sync platform that gets one past the fundamental technical challenges and will allow you to focus on the usability issues that all current sync experiences have.

What Funambol Provides

The core piece of Funambol’s software suite is the Data Synchronization Server (DSS), comprised of Tomcat 5.x running a webapp that handles the syncML xml-over-http protocol plus a dbms (mysql, postgres, or hypersonic) for state persistence. There is also a webapp (“webdemo”) providing an html interface for managing events and contacts, to be stored in the same dbms, but this article is about using pre-existing storage, so you won’t want to use this webapp or its db tables.

Another major part of the suite is the large stable of plugins for mobile devices. Most smartphones and PDAs (even the iPod) have calendar and contact functionality, where the data is stored in a local file or dbms. Many of these devices also have factory-installed (“native”) sync software, which works by sending a “begin sync” request encoded in syncML wirelessly(*) to a server that knows how to coordinate sync attempts. (*Strictly speaking, a device doesn’t have to have a wireless connection; instead, like the iPod, it might require a cable to a desktop computer, which itself has to be connected to a network unless you’re syncing only with the desktop itself.) For devices which do have contacts or events storage but no native sync client, one can usually find a “plugin” from Funambol. Unfortunately, finding out whether your device already has a sync client can be difficult, since the app might be hiding several folders deep from your main menu.

If you don’t have a sync client, but do have the ability to install software directly on your device, you can look for the plugin at https://www.forge.funambol.org/download/. If you don’t have the ability to install directly, but your device can receive binary SMS messages, then you can signup for a free user-oriented account at http://my.funambol.com, where you can indicate your device type and trigger an SMS containing the plugin software — all you have to do is click on an install link in the message.

There are some surprises in the devices that Funambol does support and those it doesn’t. It does support Microsoft Outlook (via a plugin), allowing data there to be synced with your pre-existing storage even without an Exchange server being involved. But it doesn’t support OS X’s Sync application. It does support the iPhone for syncing contacts only, but not events (because, I’m told, the iPhone SDK doesn’t support access to the device’s calendar storage). It does support the iPod, but only if you can connect it to a Windows desktop (via the iPod’s USB cable, which talks to a plugin from Funambol that you must install); there is no support for using an iPod via a Mac (and since the iPod has no wireless capability, it’s dependent on being tethered to some desktop). Another wrinkle is that some carriers disable the native sync clients on phones they sell, or they block Funambol’s plugins from using the wireless connection unless Funambol has been certified. With new phones arriving all the time, this makes for a moving target.

Two more major pieces of Funambol’s suite are the “sync portal” and the “PIM listener”. These are not available in the free, open Funambol software but can be gotten in the “carrier edition” through agreement with Funambol. The portal is another webapp using the same dbms, providing an http-based API, which can be used to build a website allowing users to indicate their device type and trigger an SMS containing an installable plugin (similar to what my.funambol.com provides).

The PIM listener is useful for users who have several devices that must stay in synch; it notices when one of a user’s devices has been synced, and triggers an SMS to all the user’s other devices (which are known to support such SMSs) instructing them to automatically initiate their own sync to pick up the changes. Obviously, this won’t help with Outlook or an iPod (because they can’t receive SMS notifications), but is a good way to satisfy one’s most-engaged users, who tend to have lots of other toys. I believe that if one updates device A, makes a separate update to device B, and syncs A, then not only will B be auto-synced, but the result of that sync will trigger an auto-sync of A to pickup the new data from B. However, as the number of a user’s devices goes up, the number of auto-syncs may thus increase factorially/exponentially; I know of no tests to see if this leads to significant battery drain or application latency. For a demo, check out the video interview with Funambol’s CEO at TalkTv.

As a last note, although the sync portal allows plugins to be pre-configured to connect with one’s own DSS, I don’t know if it’s easy to configure such plugins for locales other than US English. Also, I believe there is no publicly-available list of all error messages that a user might encounter, for use in one’s own Help page.

How Sync with DSS Works

Once a user has located the native sync client on their device, or installed a Funambol plugin (actually, any syncML client from any software provider should work, since Funambol’s DSS is syncML-compliant), and has gotten a login from the data provider (e.g. mail.aol.com or my.funambol.com) to enter into the client, they are ready to start. Actually, there’s one more step: Make sure the data provider url is correct in the sync client.

When the “synchronize” button is pressed, a “begin sync” syncML message is sent to the data provider’s url along with the user’s login, a deviceId (unique to the device compared to all other devices in the world), and an indicator of what kind of data to sync (i.e., contacts, events, or tasks). The data type is actually mapped to a “sourceUri” in the device — for example, some devices allow syncing events using either sourceUri “scal” (indicating the SIF Event format) or sourceUri “event” (indicating the VCalendar format). The sourceUri values are arbitrary and depend on how the DSS has been configured, but the values given here are Funambol’s plugin defaults. I recommend using SIF formats if possible, since they appear to allow for a wider variety of event data, and are common on Microsoft devices and thus have a large user base and the implicit testing that comes with that.

If the login is approved, the DSS combines the userId and deviceId into a “principal id” and checks its fnbl_last_sync table to see if this principalId+sourceUri has synced with it before; if not, it’s treated as a “slow” (aka full, complete) sync; if it has synced before, it’s treated as a “fast” (aka incremental, partial) sync. For slow syncs, DSS responds in syncML by asking the device to send an id for each data item of the requested type; for fast syncs, DSS asks the device to send only id’s for the data added, updated, or deleted since the last sync (as indicated by the date in fnbl_last_sync). DSS makes a similar slow-or-fast request to the data provider end.

Who or what is this data provider? If you use the DSS’s “webdemo” webapp for managing contacts and events, it’s a set of tables in DSS’s dbms. But in our case, it’s a “calendar server” that serves many other client applications and which we access via the network. More on this later.

For a slow sync, DSS asks the device for the full data behind each data item id, and passes that to the data provider telling it to add it. The data provider must reply with an id for the new item. Similarly, DSS asks the data provider for the full data behind every id, and passes that to the device telling it to add it. The device must also reply with an id for the new item. DSS stores the pairs of {device event id, data provider event id} in db table fnbl_client_mapping, keyed by principalId and sourceUri. (Note: if you reinstall a plugin, the deviceId is likely to change, so the principalId would change, which means that syncing, then reinstalling and syncing again, is likely to lead to duplicates on both the device and data provider ends, because there is no longer a matching principalId in fnbl_last_sync and thus a slow sync is done the second time.)

For a fast sync, DSS asks the device and data provider ends what has been added since the time of the last sync, sends the new items to the other end, and updates the mapping table with the new id’s. It then makes a similar request to both ends for items updated since the last sync, and then for items deleted since the last time.

On the data provider end, DSS makes all its requests through “modules” which must implement a predefined API. But before a sync can occur, the userId and password that a user entered in the device must be authenticated, and each module is responsible for indicating what Java class should do the auth check. Funambol calls such an auth class an “Officer”, and provides a default one that checks the co-installed dbms for a matching login. Since we are focused on using pre-existing data providers, you will want to write your own Officer to connect with your existing auth service for user accounts.

The module API requires implementation of these signatures:

• beginSync – Called by DSS if the officer indicated successful auth; the principalId and sourceUri are passed in via a context parameter
• getAllSyncItemKeys – DSS uses this to get data item id’s when starting a slow sync
• getNewSyncItemKeys – DSS uses this to get data item id’s for fast syncs, to get items added since last time
• getUpdatedSyncItemKeys – DSS uses this to get data item id’s for fast syncs, to get items updated since last time
• getDeletedSyncItemKeys – DSS uses this to get data item id’s for fast syncs, to get items deleted since last time
• getSyncItemById – If DSS wants the device to add or update something, it gets it from the data provider this way
• getSyncItemKeysFromTwin – The user might have added similar items on both the device and data provider ends; this call gives the data provider a chance to report items it thinks are similar to the given device item, to avoid adding duplicates at both ends.
• addSyncItem – If the device has an item the data provider should have, DSS uses this to put it there
• updateSyncItem – If the device has an item that should replace one the data provider has, DSS uses this to put it there
• removeSyncItem – If an item has been deleted on the device, and a similar one should be deleted from the data provider, DSS uses this to delete it
• mergeSyncItems – If the module has been designed to resolve conflicting similar items from the device and data provider ends by merging them, DSS uses this to get that merged version as a step before sending the update to the device and data provider ends. (This method is available only if your PIMCalendarSyncSource extends MergeableSyncSource.)
• commitSync – This is the next-to-last call DSS makes into a module for a sync. It calls it even if a SyncSourceException has been thrown by a previous call, which IMO is a design flaw since it doesn’t follow the traditional semantics of a ‘commit’; for example, if one plans to do all updates and deletions to the data provider via a batch, one would want to do that here — but that allows for loss of data integrity because a related update to the device might have failed.
• endSync – Always the last call DSS makes into a module

If one wants to provide support for events and contacts using the SIF, VCalendar, and VCard formats, there are several ways to split up the work across modules. One could make a separate module for each combination, responsible for only one format of one kind of data, or have a single module that handles all types and formats. The major constraint on modules is that each can have only one auth Officer. While this constraint didn’t limit our design choices, we did happen to implement contacts and events in separate modules in separate source trees but where both modules would use the same officer code. Having separate trees had the unfortunate consequence of needing to copy/paste the officer code from one tree to the other, putting it in a different package, and configuring the two modules to use these different officer packages. Clearly, relying on copy/paste opens the door to maintenance problems; I recommend using a single source tree and single officer codebase if at all possible.

Putting a Module Together

In this section, we walk through the creation of a module to handle multiple data types and formats, where the data provider is a remote host rather than Funambol’s “webdemo” default.

First, find the source for DSS at Funambol’s downloads site, or via objectweb (e.g. http://cvs.forge.objectweb.org/cgi-bin/viewcvs.cgi/sync4j/funambol/modules/foundation/connector/src/main/java/com/funambol/foundation/engine/source/Attic/PIMCalendarSyncSource.java?hidecvsroot=0&search=None&hideattic=1&sortby=file&logsort=date&rev=1.18&content-type=text%2Fvnd.viewcvs-markup&diff_format=h). You’re looking for these files:

• PIMCalendarSyncSource.java (and parent PIMSyncSource.java)
• PIMCalendarManager.java (and parent PIMEntityManager.java)
• init_schema.sql
• SIFTaskSource.xml
• VCalendarSource.xml
• Funambol.xml
• PersistentStoreManager.xml

Notice that PIMCalendarSyncSource implements the module API mentioned earlier by making calls into PIMCalendarManager, and PIMCalendarManager manipulates the co-installed dbms on behalf of the “webdemo” default webapp. You should hollow-out the method implementations of PIMCalendarManager and code them so they work for your existing data provider. I don’t have more to say about that part of the job.

The remainder of the work is configuring the module. Look in init_schema.sql at how the fnbl_sync_source table maps an incoming sourceUri (e.g. stask) to a bean file (e.g. SIFTaskSource.xml) that describes how to configure PIMCalendarSyncSource for that sourceUri. The key insight here is that every sync attempt triggers the creation of a new PIMCalendarSyncSource object, and that object manages state between the beginSync and endSync calls. By creating an object for every attempt, we can use the same Java class to handle different data types (events or tasks, even contacts) for different formats. Differences across data types and formats are largely hidden from you by the convenience methods in PIMCalendarSyncSource for marshalling and unmarshalling formats into Funambol data objects (e.g. Event, Task, and their common parent com.funambol.common.pim.calendar.CalendarContent). If you need to know the data type within your module code, you can use PIMCalendarSyncSource’s entityType member (or add your own member field and an entry in the bean files to populate it).

A module can support multiple data types and formats by adding an entry for each to the fnbl_sync_source table.

Be sure to make these changes:

• Change the package and class name of the ‘object’ element in each bean file to match yours
• Do the same for the ‘class’ attribute of the fnbl_sync_source_type table in init_schema.sql
• Note that the values of the ‘config’ attribute of the fnbl_sync_source table aren’t filepaths in your source tree; instead, they reflect where the files will be unpacked from a zip after running bin/install-modules.sh. You need to check build.xml to make sure these files are pulled from the right place during the build.
• Since you will be using your own solution for user acct management, you need to disable Funambol’s by providing a stub. In init_schema.sql, all entries in fnbl_sync_source_type should reference the stub class for the ‘admin_class’ attribute.

Next, look in Funambol.xml and notice the officer, store, and serverURI properties. The ‘serverURI’ value is probably ‘{serverURI}’, which indicates a placeholder that will be filled using install.properties when install-modules.sh is run; using placeholders is a pretty convenient way to get config into your module; you can define your own bean java and xml files, add your values to install.properties, and update install-module.xml to map those properties to placeholders in your bean xml file. You can also use a placeholder for the entire ‘officer’ property and put the xml in install.properties (which is helpful if you’re using different officer classes in different environments).

The ‘store’ property is important because it points to PersistentStoreManager.xml, which you will edit for your module. If your module will support fast/incremental sync, it needs its own db storage to keep track of the item id’s involved in the previous sync (aka “anchors”). For example, if the data provider provides a way of asking for all items changed since a certain time, but it doesn’t distinguish between which are new and which are updated, and doesn’t indicate any of the deleted ones, then you need to compare those id’s with the anchors: anchors that aren’t in the list from the data provider should be considered “deleted”; items from the provider that aren’t among the anchors should be considered “new”; all the rest can be considered “updated”.

To store and read anchors, you should create files create_schema.sql and drop_schema.sql (referenced in install-modules.xml) for creating and dropping the table(s) you need; if your module is listed in the modules-to-install property of install.properties when you run install-modules.sh, you will be prompted whether to recreate a table; answering ‘y’ will run drop_table.sql and then create_table.sql (so, you generally want to answer ‘n’ when installing unless you want to force all users to do slow syncs next time). To allow the module to read from and update the anchors, create java and xml bean code where the xml defines the SQL for inserting, updating, querying, and deleting. Add the bean xml filename to the list of such beans in PersistentStoreManager.xml.

Some of the files I’ve mentioned may not be in Funambol’s public source, and I’m sorry for that oversight. But they are helpful folks and I’m sure they would send you some samples.

Lessons Learned

Other than issues I’ve already mentioned, here are some issues you should be aware of up-front:

1. There is no automated update process for getting the latest info about the world of devices. The sync portal needs this so it can offer support for the latest phones, and so it can keep up with changes in support for existing ones. You need to arrange with Funambol to be sent regular “phone pack” updates.
2. There is no support for sending a message from the server to be displayed in the client. For errors, the best one can do is set a message in a thrown SyncSourceException, since the message will appear in the client log if it uses log level “error”.
3. There is no way for a module to trigger a change from fast to slow sync in case, say, an error is found with anchor storage.
4. The module API requires addItem to return the id that your storage will use for the item. This makes it hard if not impossible to include add’s with update’s and delete’s in a single batch during commitSync(); instead, you may need a synchronous network roundtrip for every call to addItem().
5. It appears that the SIF Event content type used by Microsoft uses UTC-based times (e.g. 20080704T010000Z for 9amPT on July 4) for all values unless it’s an all-day event, in which case it uses YYY-MM-DD with no timezone indicator.  Funambol’s convert() in PIMCalendarSyncSource leaves these all-day values as-is instead of changing them to UTC format. I didn’t find any guiding principle about when UTC versus local time was used.
6. The default dbms in the open source version of DSS is hypersonic. If you plan to use mysql, be sure to indicate your engine is InnoDB in your DDL scripts since Funambol’s dev wiki mentions there’s a risk of memory leaks without that.
7. Since you’re using your own storage, make sure to remove the “webdemo” webapp (and associated db tables) so users don’t stumble across it by accident.
8. When dealing with com.funambol.common.pim.calendar.RecurrencePattern, note that recurrence types with “NTH” in their name are for use when instance!=0, not when interval>0.
9. Funambol’s Outlook plugin has a bug where if it is given an event that repeats every N years, it will change it to repeating yearly.
10. Outlook has a bug where if it is given an event that repeats monthly, that’s timed/not-allday, and whose start time crosses a day boundary when converted to UTC (e.g. 4pm Pacific in summer becomes 1am UTC the next day), then Outlook will move it a day late (e.g. from Friday to Saturday). There is a similar problem when Outlook provides such an event, instead of consuming it.
11. If an event is timed/not-allday but it has a duration of 24 hours, then the Outlook plugin will change it to an allday event, thereby losing start and end times.
12. When a repeating event is created, synced, and then some individual dates on each end are modified or deleted (they must be different dates on each end), then on the next sync they will show as dupes. This is because Funambol’s default merge method has a bug…such changed dates are represented as an “exclusions list” in the RecurrencePattern obj, and merge should take the union of the device’s list and the data provider’s.
13. Most devices don’t support multiple calendars, so Funambol has no special support for them, but your data provider may have many users who have these. As long as your module uses a concatenation of calendar id and event id as the full id given to DSS (to guarantee each item id that DSS has is unique within a user’s set of id’s), there should be no problem.
14. It’s not feasible to host, say, a contacts module in one DSS and a calendar module in another DSS, and rely on sniffing to route http requests to the correct server. The problem is that the sourceUri is the one piece of info that would allow such decisions, but it’s embedded in the request payload, not the http headers.
15. If a user did a manual sync at the same time that one of his devices happened to do an auto-sync, I believe there is no mechanism in DSS to detect this and block one of the requests, to avoid loss of data integrity for the user. For example, imagine there are similar events on device A, data provider B, and auto-syncing device C; while A and B are merging, C might update the event on B; when the merge of A and B is done and the update happens on B, it wipes out the info from C.
16. If you use the carrier edition, be aware that your init_schema.sql isn’t the only file controlling what sourceUri’s the DSS thinks it supports. There are also portal/database/cared-coredb-mysql.sql and portal/database/cared-mysql.sql. Be sure to remove INSERT calls from these for sourceUri’s you won’t support, to ensure the user sees an error saying something like “Remote name not recognized” rather than a generic error.
17. If you see error “Fatal error creating the PersistentStore object” in your log, a possible cause is having the wrong line endings in the config files of your s4j file. For example, if you build the s4j on a Windows machine and try to run it on a *nix machine, you are likely to see this error.
18. Trivia item: The second param to the ICalendarParser constructor, a String, represents the device’s charset, not its timezone.

Community Mailing Lists

• Announcements: https://lists.sourceforge.net/lists/listinfo/sync4j-announce
  • archive: http://sourceforge.net/mailarchive/forum.php?forum=sync4j-announce
• General discussion on SourceForge: https://lists.sourceforge.net/lists/listinfo/sync4j-users
  • archive: http://sourceforge.net/mailarchive/forum.php?forum=sync4j-users
• General discussion on YahooGroups: http://groups.yahoo.com/group/Sync4j/
  • archive: http://groups.yahoo.com/group/Sync4j/messagesearch?query=Sync4j@groups.yahoo.com

Good luck!