Category Archives: PIM

Why Palm’s webOS is the future of Android (and desktop computing)

Do you connect these dots in the same way I do?

  1. The current practice in OSs and browsers of asking the user at install time whether to proceed with the install, as a way of avoiding security threats, just doesn’t work. Users do not have the right kind of information at that time to decide.
  2. The threat of compromised systems and data loss is severe enough that consumer and enterprise OSs will have to be designed in a different way to manage installation risks. The widespread acceptance of smartphone apps indicates that smartphones will need such protection, too.
  3. Google’s NativeClient project is a good way of handling the risk because it provides a sandbox, and it’s better than alternatives like Java and Flash because it allows apps to run faster (because the apps are compiled natively rather than into bytecode).
  4. Palm’s webOS for its new smartphones has a very similar design to NativeClient (and since NativeClient is open source, could be built on top of it, for all I know). Specifically, webOS’ plugin development kit (PDK) will allow allow apps written in C and C++, two languages which by themselves allow altering memory contents almost anywhere in RAM and thus open to abuse by malicious app coders, but the PDK will sandbox apps, apparently in much the same way that NativeClient does. WebOS’ other interface, the Mojo SDK, allows apps written in Javascript to access data on the phone in much the same way that NativeClient’s browser plugin design would allow.
  5. Thus, webOS seems to provide a glimpse into what smartphone and desktop OSs will be like in coming years, if they deal with security threats in the inspired way detailed in the NativeClient design.

And there’s another force pushing Google’s Android smartphone OS in the same direction as webOS:

  1. Google always seems to prefer keeping its apps as platform-agnostic as it can by leveraging browsers when it can. The exceptions are Google Earth, GTalk, etc which must be installed either for performance reasons or to gain access to “hooks” in the OS that browsers can’t offer.
  2. Google’s apps for Android are Java-based (i.e., not browser-based) for apparently no strong reason. In fact, it seems that if Google had had Palm’s insights about how a web-oriented OS could be made back when Android was being designed, then Android would be very much like webOS so that Google wouldn’t have to split its app-building competence and resources across so many platforms (of course, the iPhone and Blackberry platforms would still make their own demands). Google’s efforts to build ChromeOS is another strong bit of evidence of its desire that there be fewer platforms and that they resemble browsers more.
  3. Eric Schmidt has said that Android and ChromeOS will eventually merge. I’m not sure if he came to this conclusion before or after learning about the design of Palm’s webOS, but webOS seems like a good hint of what such a merge would result in.

Am I pulling too hard on thin threads, or does this paint the same strong picture for you that Palm’s webOS really is a glimpse of the future? It sure is a fun way for me to stretch my thinking about what smartphones can do and be.

If this is an accurate prediction, then two consequences come to mind:

  1. The current unspoken practice of web engineers looking into the Javascript source of their competitors, learning new tricks, and helping the craft of web engineering to improve will suffer because companies will want to shift their presentation and business logic out of Javascript and into compiled native code for greater performance and out of a misguided attempt to protect their intellectual property.
  2. Having Google compete in the same idea space will help inspire both toward even better ideas. Of course, Google won’t buy Palm (why would it need to?), and it’s unlikely that having similar platform designs will affect the market share of either of them. As long as Palm can capture a significant share of the growing global demand for smartphones, it should be able to survive. And it’s likely to always have an advantage over Android in the beauty of its UI, given the DNA of the two companies.

UPDATE: Google released an “NDK” for Android way back in June 2009, which sounds like webOS’ planned PDK and also sounds like it was built on NativeClient. So, my prediction above that webOS is the future of Android has things a bit turned around.

Also, although the NDK seems to have a very similar design to NativeClient, and might have been built on NaCl, I’m somewhat doubtful because NaCl relies heavily on a feature known as “segmented memory” in the 386 chip architecture, and I wonder if that same feature is present in mobile CPUs such as ARM.

UPDATE: Other devs are worried that we might lose the ability to view html source and thus lose one of the primary learning and innovation paths for web app devs.

    Print Friendly, PDF & Email

    How to sync Windows Mobile with both Exchange and Google

    For Windows Mobile phones, Google suggests that you use the ActiveSync program on the phone to connect to one of their servers. This is problematic in principle for me, since ActiveSync allows only one connection with a server to be configured, and I need that to stay synced with work email. It also didn’t work even when I experimented, first with ActiveSync, and later with Funambol’s WinMo client.

    What did work was the free GMobileSync client. The 1.3.7 beta version supports multiple Google calendars, but one cannot select to sync just a subset of them, nor are secondary calendar events shown any differently than events from one’s default calendar. (Suggestion to GMobileSync: Allow defining a 2-letter uppercase code per calendar in device settings; strip these when syncing back to the server so they don’t appear to have changed.) It doesn’t sync contacts, but that feature is said to be coming. And syncs can’t yet be scheduled to happen automatically (but you can get CT Scheduler Lite free and tell it to launch the sync, but then you still have to press the Sync button — there is no parameter to automate that part yet; to install, download to a machine where you have admin privileges, or email them for the CAB file).

    As part of your move away from Microsoft, at least on the desktop, you may want to use Mozilla Thunderbird+Lightning to manage your Google Calendar.

    You may also want to try FinchSync with GCalDaemon.

    If you want to try it the way Google suggests, you might need help from this and this (from these matches) and use Funambol client for WM with these settings:

    Server URL: https://m.google.com:443/syncml

    User: [yourname]@gmail.com

    Password:[your gmail password]

    If you really want to use ActiveSync with two servers, it’s said to be possible, but one must first find a way to edit the device’s registry:

    So, to sync two exchange servers, put this in the device’s registry:

    [HKEY_CURRENT_USERSoftwareMicrosoftActiveSyncEnableNonLocalCrossPollination]
    = (DWORD) 1

    This key will most probably be there already, but if it is, its value
    will be 0. If so, change it to 1.

    Soft reset, then, setup activesync ON THE DEVICE (If you try setting up
    on the PC, it will still fail).

    There are also paid services that are geared to people who want more features or who want to avoid doing all this IT work themselves:

    • http://www.scheduleworld.com/
    • http://www.goosync.com/
    Print Friendly, PDF & Email

    Going portable as an alternative to using a remote desktop

    This tip is intended for people like me who:

    • Often need to work outside the office but don’t want to carry a laptop
    • Happen not to use any Linux or Mac machines, just Windows
    • Can’t use Window’s Remote Desktop Connection app. (Perhaps your IT dept won’t open the port in their firewall; but if the problem is just that you’re using a Home version of Windows, you could switch to a Business version or upgrade to an Ultimate version.)
    • Can’t use the similar VNC app because your client computer has User Account Control turned on and you want to keep it that way for security reasons.
    • Or, your IT dept won’t give you admin privileges on your machine, so you can’t install apps at will
    • If you use SVN, then either your SVN repo has a publicly-accessible IP address, or that you can access it via VPN. (That is, if you need to use SVN but it’s kept behind a firewall, then these instructions won’t help you access it…you’re stuck working non-portably.)

    The next best alternative I’ve found is to use a thumbdrive to keep your documents and applications (plus application state such as licenses, passwords, bookmarks, files currently being edited, email and contacts, etc).

    If all the desktops you’ll use the usb drive with are XP, then you could put MojoPac on the usb; it emulates an OS and provides a desktop view of your usb that runs as a window in XP. It’s not clear if there will ever be a version that works with Vista or Windows 7.

    Encryption

    Before copying any files to your thumbdrive, or installing any portable apps, consider whether you’d be hurt if the thumbdrive were lost or stolen and someone got access to its contents. If that’s at all important to to you, there are four options:

    • If you expect to have admin rights on any computer you might use, then you could install TrueCrypt on your thumbdrive and also create a TrueCrypt file container there.
    • If you don’t expect to have admin rights, but can convince your IT dept to install TrueCrypt for you, then check out the instruction in the next paragraph.
    • Or, if you won’t need more than 1G of space encrypted on your thumbdrive, you can try Rohos Mini Drive. (I haven’t)
    • Otherwise, you need to use Remora, where you will have to manually unencrypt each file you would want to use, then manually re-encrypt it after saving. (I haven’t tried this either)

    If you can go with TrueCrypt, then install it on your harddrive. Use it to encrypt the thumbdrive. You can configure the encryption so you are prompted for the password as soon as you plug in the thumbdrive anywhere; then, you will be able to access files and run apps from the drive as though it weren’t encypted but all your edits and adds will be encrypted. (Things will run slower due to the on-the-fly encryption, although perhaps not noticeably so.) You might want to encrypt only a folder of documents, but I opted to encrypt everything because apps like Thunderbird store one’s data (e.g. all one’s email, if you have opted to keep local copies) in generally unpredictable places. After installing TrueCrypt, do this (basically following TrueCrypt’s beginner tutorial):

    1. Do the following from a machine where you have admin privileges
    2. Start the TrueCrypt application from your desktop
    3. Click the Create Volume button
    4. Select “Create an encrypted file container”. (I tried “Encrypt a non-system partition/drive”, but when I plugged the drive into another machine where TrueCrypt wasn’t installed, I was prompted to format the usb drive.)
    5. Select “Standard TrueCrypt volume”
    6. Click “Select file”
    7. In the file selector popup, select your thumbdrive in the left pane, and for “File name:” provide a name for your container. Mine is “TrueCryptContainer”, but the paranoid might want to use “junk”. Then hit Save.
    8. In the Encryption Options view, just hit Next.
    9. When prompted for file size, use the full capacity available (e.g., 3680 MB for a “4GB” usb drive). Then Next.
    10. Choose a good password. Next.
    11. For Volume Format, set Filesystem to NTFS (or, if you can’t get admin privileges anywhere, choose FAT). Move your mouse around over the window several times to generate a good random seed for the encryption algorithm. Then click Format.
    12. When formatting’s done, close that dialog window.
    13. Back in the main TrueCrypt dialog, the one showing a list of unused drive letters, choose a letter you want your new container mapped to.
    14. Click the “Select File” button and choose the container file you just created.
    15. Click the “Mount” button. You’ll be prompted for the password you assigned. Sometimes this doesn’t work for me and I have to cancel and then hit Mount again before it works.
    16. Once it works, you’ll see your container file listed alongside the drive letter you chose in TrueCrypt’s main dialog. And the drive letter should appear in WindowsExplorer under “My Computer” along with your other drives. You should be able to open from and save to this encrypted container using any application’s File Open and File Save commands.
    17. Be sure to use the Dismount button before trying “safely remove hardware” and before removing the thumb drive.

    Sync with desktop or server/cloud

    After encryption, this is probably your highest priority. You can try a portable application like Toucan, but I think it’s not full-featured enough. For example, one needs to type or paste in paths when defining items to skip, instead of selecting them through a browse button. And my rules for skipping items were ignored anyway. Instead, rather than using a portable app to do the syncing, you probably just need to sync to one primary desktop, and Microsoft’s SyncToy running from that desktop works well. I configured it to sync docs/projects and apps separately, and I set my desktop-to-cloud sync service Mozy to sync just the docs/projects from the desktop (because I want double protection for things I can’t just reinstall and reconfigure). As a further step, I use Windows Task Scheduler (See “Help | Learn how to schedule SyncToy” within SyncToy) to kick off these SyncToy tasks near the end of every workday. Setting SyncToy to run at the end of a workday assumes your backup desktop is your work desktop; to sync to a desktop at home, you probably want the trigger event to be the insertion of your usb drive. TaskScheduler doesn’t natively support “mounting of usb drive” as a trigger, but you can buy MyTrigger for US$24 which enables TaskScheduler to launch SyncToy for such an event.

    Default programs

    Once you start reading email from Thunderbird on your thumbdrive, when you click links in msgs you’d want them to open with the browser also on your thumbdrive (especially if you might bookmark the link or enter a password). This doesn’t happen automatically; instead, you’ll get whatever app has been set as the default handler of the kind of file you want to open (where “kind” is determined by the file’s extension — the part after the dot). There is no good solution in XP nor Vista to this general problem of wanting to set usb-hosted apps as default handlers (but Windows 7 appears to support it).. However, just for the case of handling urls when they appear in apps other than the browser, one can make a desktop-hosted Firefox the default handler and then use the Foxmarks extension in that installation and all one’s other Firefox installs (including the portable one), since the extension syncs bookmarks and passwords across machines. (However, there does not appear to be any Firefox extension that syncs one’s open tabs, aka session.)

    Auto-start when inserting drive

    Many people like certain apps on their drive to launch as soon as it’s plugged in. PStart is a portable application with a small window (aka “panel”) where one can list other apps hosted on the same drive, and set some of them to launch when the drive is mounted. To make this work, however, one needs to configure each desktop OS to “autoplay” usb drives whenever they are inserted. XP and Windows 7 will prompt you if you want this done when you plug in your first usb drive, but Vista requires extra work:

    1. Open your run box (Start | Run) and type regedit and click OK.
    2. Go to HKLMSOFTWAREMicrosoftWindowsCurrentVersionPoliciesExplorer.
    3. You should see a key called NoDriveTypeAutoRun (see picture below). Double click it and set the Value Data to 91 (hexadecimal).
    4. Restart your computer and it should be fixed.

    Now install PStart to your usb drive.

    Once it’s installed, open its panel and go to Setup | Create autorun file. Select the drive letter that’s mapped to your TrueCrypt container. (You might want to tweak the autorun.inf file even further.)

    You can also tell PStart to launch any programs listed in its panel when PStart launches. To do so:

    1. Right-click on the item in the panel (or add it by right-clicking in an empty part of the panel and selecting “Add file”)
    2. In the dialog that appears, click the Advanced tab
    3. In this tab, set Autorun to “on startup”

    Reminder to take your drive when you log-off

    This is useful, but the Quiet version requires that you log-off instead of using Safely Remove Hardware.

    Force apps to release thumbdrive

    If you often have the problem of Safely Remove Hardware failing to dismount the drive, you might consider this workaround. However, there seems a pretty fair chance of data loss. (If you use PStart, the culprit might be that in Settings you don’t have “when closed” set to “exit application”.)

    I’ve heard that Windows 7 will actually tell you what application is holding onto the drive (but not what file it’s using).

    Reward if returned

    You probably want to create a text file at the root level of your drive called REWARD IF RETURNED.txt providing your email address, and make sure the file remains unencrypted. Or, you may want to make the name of the drive your email address or phone number.

    Mozilla Firefox browser

    I prefer Google’s Chrome browser, and there is a portable version (steps available below) but I haven’t found any way to export bookmarks once one starts using the portable version; that’s a critical flaw, because when I’ve installed updated portable versions, I’ve had to lose any bookmarks accumulated since I first started using the portable version. So, I’m using portable Firefox instead. (Update: I’m using Foxmarks because portable Firefox is slow, particularly when scrolling.) For portable Firefox, I recommend the following addons:

    • Xmarks – Keep your bookmarks in sync across machines and drives
    • Download Statusbar – I find it annoying that FF uses a popup to acknowledge every download attempt, and the designers of this addon felt the same
    • Undo Closed Tabs Button – If you were too quick to close a tab and want it back, this feature will help you
    • Tabs Open Relative – When you right-click to open a link in a new tab, it should appear right next to your current tab, not way down at the right. This feature fixes that.
    • Firebug – Useful for designing/debugging web pages
    • Zotero – Useful for managing a library of e-documents, such as you may have on your usb drive

    And if you really want to use portable Google Chrome, here’s how:

    1. The portable version is available from a German developer, and you’ll have to get a translation of his blog page (plus the download link) from Lifehacker.
    2. To copy over your default tabs and settings, (for Vista) copy your C:UsersyournameAppDataLocalGoogleChromeUserDataDefault contents to Portable_Google_Chrome_0.2.149.30Profil
    3. Migrating bookmarks from desktop Chrome to portable Chrome takes several steps
      1. Download and install Mark Clouden’s chrome bookmark exporter
      2. Run it and hit the Export Bookmarks button. Make sure it refers to the chrome installation on your harddrive rather than the one on your thumbdrive.
      3. We’re going to use Firefox to import the bookmarks, so we can tell Chrome on your thumbdrive to import bookmarks from it. So run an instance of Firefox where you don’t mind emptying all existing bookmarks first.
      4. In Firefox’s menu bar, go to Bookmarks | Organize Bookmarks, and delete all bookmarks in the Bookmarks Toolbar and Bookmarks Menu.
      5. Click the Import and Backup button at the top. Choose Import HTML.
      6. Select “From an HTML file, then select the bookmarks.html file you created with Clouden’s exporter.
      7. It may take awhile to import; wait for the bookmarks you expect to appear. Then quit Firefox.
      8. Start Chrome from your thumbdrive, then go to (Wrench icon in upper right) | Import bookmarks & settings.
      9. Set From to Firefox and click Import.
      10. You’ll find your bookmarks if you click “Other bookmarks” in the upper right, and then “Imported from Firefox”. You can drag items out of “Bookmarks bar” in this view right onto Chrome’s bookmarks bar. And you can drag the other bookmarks and folders onto “Other bookmarks” itself, and then right-click on empty folders to delete them.
    4. You may want to change the default download location under (Wrench icon in upper right) | Options | Minor Tweaks
    5. You may also want to tell Chrome to reopen the same pages when you restart it. Go to (Wrench icon) | Options | Basics | Startup | Restore the pages that were open last

    Notepad++ text editor

    PortableApps.com has a portable version of this very powerful and popular text editor. I recommend renaming the .exe file and all other folders and files that contain “++” to “NotepadPPPortable” because some sync/backup tools like Toucan have a problem with +’s in filenames.

    Mozilla Thunderbird email client

    PortableApps.com has a portable version of this email client and address book application. But there’s no way to connect to your office Exchange server unless they’ve enabled IMAP or POP support. (But if you have a mobile phone running Windows Mobile, its email client does support Exchange.) Also, I recommend adding the following extensions:

    AntiVirus

    PortableApps.com offers the ClamWin antivirus checker.  Note that this is only useful when you suspect a there is a problem, probably in a specific file. It is not a schedulable scanner. You would probably use it only after disconnecting your thumbdrive from a computer you have borrowed.

    WinAmp media player

    Just copy C:Program FilesWinamp to your thumbdrive. But disable Winamp Agent, or it will prevent unmounting the thumbdrive.

    Unzip / file compression

    I used to use the version of 7-zip available from PortableApps.com, but I find its UI very nonintuitive and am now very happy with IZArc, which has a portable version.

    OpenOffice for docs, drawings, spreadsheets

    PortableApps.com offers OpenOffice. OO’s doc writer is a great replacement for MSWord, and its drawing app is far better than Visio in my opinion.

    PDF-XChange viewer

    This PDF viewer is free, portable, and supports highlighting, comments, and typewriter features. The typewriter is great for filling out forms.

    I’m providing a link for the PDFXChange download, because the creator’s web site is so poorly designed and confusing. But the software itself is really good, and I bought a license for the extra features.

    Application launcher

    Launchy is a very popular choice, and its online PDF help file explains how to run it in portable mode. An alternative that bundles encryption and autorun scheduling is GeekMenu.

    Skype messenger and internet phone

    I haven’t tried it portably, but here’s a tip.

    Eclipse IDE

    (A programming environment for Java, C, and other languages like Prolog)

    These instructions are adapted from a forum post on PortableApps.com:

    1. If you’re going to use SVN as source control, make sure your SVN repository has a publicly-accessible IP address, or that you can access it via VPN.
    2. Run all updates in the Installed Software tab.
    3. Select Core SVNKit Library under http://eclipse.svnkit.com/1.2.x/Revision Graph, Subclipse
    4. Select SVNKit Adapter under http://subclipse.tigris.org/update_1.4.x
    5. Disable SVNKit library version 1.2.1.5297
    6. Once you launch Eclipse and select a workspace, the full filepath to that workspace — including drive letter — will be saved to eclipseconfiguration.settingsorg.eclipse.ui.ide.prefs. Because the drive letter assigned to the thumbdrive can differ across machines (and even across uses of the same machine), you probably want to edit this file so that the final line has no drive letter. For example, mine is RECENT_WORKSPACES=ProjectsEclipseWorkspace
    7. Eclipse seems to change eclipse.ini each time it runs by updating the ‘-vm’ value, and for me it puts an absolute path there including drive letter. So you may want to make eclipse.ini read-only to avoid this bug.

    Unfortunately if you are a Prolog coder wanting to use Eclipse, there is no way to work truly portably. There is just one version of Prolog that runs in Eclipse that I know of, Amzi (which works very well), but it depends on environment variables in your OS and there is no way to provide these vars via an Amzi config file (yet). However, if Amzi is installed on your desktop, then an Eclipse running from your usb drive will be able to support Prolog. I have only tried this when I’ve installed Amzi on both the desktop and in the Eclipse on my usb. Here’s how:

    1. Go to the Amzi Logic Server download page and scroll down to section “3. Existing Amzi! and/or Eclipse Users”. Follow that.
    2. When prompted for a Destination Folder, be sure you select a folder on your thumbdrive (I selected “F:AppsAmziProlog”).
    3. When that’s done, you can enable Prolog support in Eclipse by following Amzi’s install instructions in the section on Existing Eclipse Users.
    4. After restarting Eclipse, go to Windows | Open Perspective | Other | Prolog.
    5. Then do the same to open the Debug perspective.

    Task manager / Todo list

    I’m not quite obsessive enough yet to need a multi-level task manager, but ToDoList offers that capability and can run portably.

    Wishlist

    • An app that monitored the health of the usb drive and warned me when it’s time to copy its contents to a new drive. (I already try to auto-sync the drive to my laptop at home whenever I remember/have time to plug it in, and that laptop auto-syncs in turn with Mozy.)
    • A desktop-hosted antivirus that scanned any usb inserted before allowing it to autoplay.  This may just require me to hunt more; I currently use Kaspersky, which seems to work very well, but it’s not clear if it does this and their tech support has ignored my questions about it.
    Print Friendly, PDF & Email

    Prevent HTC phone from switching to Ulaanbaatar time from Singapore/Malaysia time

    My HTC Hermes (AT&T 8525) keeps switching from the time and zone I set in Settings — in particular, from Singapore/Malaysia’s timezone to Ulaanbaatar’s. I found just one Google match about this problem: http://www.ppcsg.com/lofiversion/index.php/t93410.html, and it indicates the problem also affects the HTC Touch. The solution is to go into the Phone app, Menu, Options, find the Time Zones tab on the far lower right, and uncheck “Automatic change time zone and clock”.

    Print Friendly, PDF & Email

    Implementing Calendar Sync via Funambol

    Sync software for managing contacts, events, etc across devices can be very helpful when it works, and very frustrating when it doesn’t. In my experience, when an error occurs it’s often unclear how to rectify it; and even when there is no error, it’s hard to judge whether it worked because many updates may have been made to my data and it’s unclear whether the updates were correct until I run into an instance of not being able to find a contact, or finding that I have multiple instances of the same reminder.

    I was really motivated to find a solution, and recently I had the opportunity because my employer chose to offer a sync service by using the open-source software shepherded by Funambol.com. This article describes how to use Funambol’s software to allow users to sync with a pre-existing store of their event and task data.

    Funambol provides a very capable sync platform that gets one past the fundamental technical challenges and will allow you to focus on the usability issues that all current sync experiences have.

    What Funambol Provides

    The core piece of Funambol’s software suite is the Data Synchronization Server (DSS), comprised of Tomcat 5.x running a webapp that handles the syncML xml-over-http protocol plus a dbms (mysql, postgres, or hypersonic) for state persistence. There is also a webapp (“webdemo”) providing an html interface for managing events and contacts, to be stored in the same dbms, but this article is about using pre-existing storage, so you won’t want to use this webapp or its db tables.

    Another major part of the suite is the large stable of plugins for mobile devices. Most smartphones and PDAs (even the iPod) have calendar and contact functionality, where the data is stored in a local file or dbms. Many of these devices also have factory-installed (“native”) sync software, which works by sending a “begin sync” request encoded in syncML wirelessly(*) to a server that knows how to coordinate sync attempts. (*Strictly speaking, a device doesn’t have to have a wireless connection; instead, like the iPod, it might require a cable to a desktop computer, which itself has to be connected to a network unless you’re syncing only with the desktop itself.) For devices which do have contacts or events storage but no native sync client, one can usually find a “plugin” from Funambol. Unfortunately, finding out whether your device already has a sync client can be difficult, since the app might be hiding several folders deep from your main menu.

    If you don’t have a sync client, but do have the ability to install software directly on your device, you can look for the plugin at https://www.forge.funambol.org/download/. If you don’t have the ability to install directly, but your device can receive binary SMS messages, then you can signup for a free user-oriented account at http://my.funambol.com, where you can indicate your device type and trigger an SMS containing the plugin software — all you have to do is click on an install link in the message.

    There are some surprises in the devices that Funambol does support and those it doesn’t. It does support Microsoft Outlook (via a plugin), allowing data there to be synced with your pre-existing storage even without an Exchange server being involved. But it doesn’t support OS X’s Sync application. It does support the iPhone for syncing contacts only, but not events (because, I’m told, the iPhone SDK doesn’t support access to the device’s calendar storage). It does support the iPod, but only if you can connect it to a Windows desktop (via the iPod’s USB cable, which talks to a plugin from Funambol that you must install); there is no support for using an iPod via a Mac (and since the iPod has no wireless capability, it’s dependent on being tethered to some desktop). Another wrinkle is that some carriers disable the native sync clients on phones they sell, or they block Funambol’s plugins from using the wireless connection unless Funambol has been certified. With new phones arriving all the time, this makes for a moving target.

    Two more major pieces of Funambol’s suite are the “sync portal” and the “PIM listener”. These are not available in the free, open Funambol software but can be gotten in the “carrier edition” through agreement with Funambol. The portal is another webapp using the same dbms, providing an http-based API, which can be used to build a website allowing users to indicate their device type and trigger an SMS containing an installable plugin (similar to what my.funambol.com provides).

    The PIM listener is useful for users who have several devices that must stay in synch; it notices when one of a user’s devices has been synced, and triggers an SMS to all the user’s other devices (which are known to support such SMSs) instructing them to automatically initiate their own sync to pick up the changes. Obviously, this won’t help with Outlook or an iPod (because they can’t receive SMS notifications), but is a good way to satisfy one’s most-engaged users, who tend to have lots of other toys. I believe that if one updates device A, makes a separate update to device B, and syncs A, then not only will B be auto-synced, but the result of that sync will trigger an auto-sync of A to pickup the new data from B. However, as the number of a user’s devices goes up, the number of auto-syncs may thus increase factorially/exponentially; I know of no tests to see if this leads to significant battery drain or application latency. For a demo, check out the video interview with Funambol’s CEO at TalkTv.

    As a last note, although the sync portal allows plugins to be pre-configured to connect with one’s own DSS, I don’t know if it’s easy to configure such plugins for locales other than US English. Also, I believe there is no publicly-available list of all error messages that a user might encounter, for use in one’s own Help page.

    How Sync with DSS Works

    Once a user has located the native sync client on their device, or installed a Funambol plugin (actually, any syncML client from any software provider should work, since Funambol’s DSS is syncML-compliant), and has gotten a login from the data provider (e.g. mail.aol.com or my.funambol.com) to enter into the client, they are ready to start. Actually, there’s one more step: Make sure the data provider url is correct in the sync client.

    When the “synchronize” button is pressed, a “begin sync” syncML message is sent to the data provider’s url along with the user’s login, a deviceId (unique to the device compared to all other devices in the world), and an indicator of what kind of data to sync (i.e., contacts, events, or tasks). The data type is actually mapped to a “sourceUri” in the device — for example, some devices allow syncing events using either sourceUri “scal” (indicating the SIF Event format) or sourceUri “event” (indicating the VCalendar format). The sourceUri values are arbitrary and depend on how the DSS has been configured, but the values given here are Funambol’s plugin defaults. I recommend using SIF formats if possible, since they appear to allow for a wider variety of event data, and are common on Microsoft devices and thus have a large user base and the implicit testing that comes with that.

    If the login is approved, the DSS combines the userId and deviceId into a “principal id” and checks its fnbl_last_sync table to see if this principalId+sourceUri has synced with it before; if not, it’s treated as a “slow” (aka full, complete) sync; if it has synced before, it’s treated as a “fast” (aka incremental, partial) sync. For slow syncs, DSS responds in syncML by asking the device to send an id for each data item of the requested type; for fast syncs, DSS asks the device to send only id’s for the data added, updated, or deleted since the last sync (as indicated by the date in fnbl_last_sync). DSS makes a similar slow-or-fast request to the data provider end.

    Who or what is this data provider? If you use the DSS’s “webdemo” webapp for managing contacts and events, it’s a set of tables in DSS’s dbms. But in our case, it’s a “calendar server” that serves many other client applications and which we access via the network. More on this later.

    For a slow sync, DSS asks the device for the full data behind each data item id, and passes that to the data provider telling it to add it. The data provider must reply with an id for the new item. Similarly, DSS asks the data provider for the full data behind every id, and passes that to the device telling it to add it. The device must also reply with an id for the new item. DSS stores the pairs of {device event id, data provider event id} in db table fnbl_client_mapping, keyed by principalId and sourceUri. (Note: if you reinstall a plugin, the deviceId is likely to change, so the principalId would change, which means that syncing, then reinstalling and syncing again, is likely to lead to duplicates on both the device and data provider ends, because there is no longer a matching principalId in fnbl_last_sync and thus a slow sync is done the second time.)

    For a fast sync, DSS asks the device and data provider ends what has been added since the time of the last sync, sends the new items to the other end, and updates the mapping table with the new id’s. It then makes a similar request to both ends for items updated since the last sync, and then for items deleted since the last time.

    On the data provider end, DSS makes all its requests through “modules” which must implement a predefined API. But before a sync can occur, the userId and password that a user entered in the device must be authenticated, and each module is responsible for indicating what Java class should do the auth check. Funambol calls such an auth class an “Officer”, and provides a default one that checks the co-installed dbms for a matching login. Since we are focused on using pre-existing data providers, you will want to write your own Officer to connect with your existing auth service for user accounts.

    The module API requires implementation of these signatures:

    • beginSync – Called by DSS if the officer indicated successful auth; the principalId and sourceUri are passed in via a context parameter
    • getAllSyncItemKeys – DSS uses this to get data item id’s when starting a slow sync
    • getNewSyncItemKeys – DSS uses this to get data item id’s for fast syncs, to get items added since last time
    • getUpdatedSyncItemKeys – DSS uses this to get data item id’s for fast syncs, to get items updated since last time
    • getDeletedSyncItemKeys – DSS uses this to get data item id’s for fast syncs, to get items deleted since last time
    • getSyncItemById – If DSS wants the device to add or update something, it gets it from the data provider this way
    • getSyncItemKeysFromTwin – The user might have added similar items on both the device and data provider ends; this call gives the data provider a chance to report items it thinks are similar to the given device item, to avoid adding duplicates at both ends.
    • addSyncItem – If the device has an item the data provider should have, DSS uses this to put it there
    • updateSyncItem – If the device has an item that should replace one the data provider has, DSS uses this to put it there
    • removeSyncItem – If an item has been deleted on the device, and a similar one should be deleted from the data provider, DSS uses this to delete it
    • mergeSyncItems – If the module has been designed to resolve conflicting similar items from the device and data provider ends by merging them, DSS uses this to get that merged version as a step before sending the update to the device and data provider ends. (This method is available only if your PIMCalendarSyncSource extends MergeableSyncSource.)
    • commitSync – This is the next-to-last call DSS makes into a module for a sync. It calls it even if a SyncSourceException has been thrown by a previous call, which IMO is a design flaw since it doesn’t follow the traditional semantics of a ‘commit’; for example, if one plans to do all updates and deletions to the data provider via a batch, one would want to do that here — but that allows for loss of data integrity because a related update to the device might have failed.
    • endSync – Always the last call DSS makes into a module

    If one wants to provide support for events and contacts using the SIF, VCalendar, and VCard formats, there are several ways to split up the work across modules. One could make a separate module for each combination, responsible for only one format of one kind of data, or have a single module that handles all types and formats. The major constraint on modules is that each can have only one auth Officer. While this constraint didn’t limit our design choices, we did happen to implement contacts and events in separate modules in separate source trees but where both modules would use the same officer code. Having separate trees had the unfortunate consequence of needing to copy/paste the officer code from one tree to the other, putting it in a different package, and configuring the two modules to use these different officer packages. Clearly, relying on copy/paste opens the door to maintenance problems; I recommend using a single source tree and single officer codebase if at all possible.

    Putting a Module Together

    In this section, we walk through the creation of a module to handle multiple data types and formats, where the data provider is a remote host rather than Funambol’s “webdemo” default.

    First, find the source for DSS at Funambol’s downloads site, or via objectweb (e.g. http://cvs.forge.objectweb.org/cgi-bin/viewcvs.cgi/sync4j/funambol/modules/foundation/connector/src/main/java/com/funambol/foundation/engine/source/Attic/PIMCalendarSyncSource.java?hidecvsroot=0&search=None&hideattic=1&sortby=file&logsort=date&rev=1.18&content-type=text%2Fvnd.viewcvs-markup&diff_format=h). You’re looking for these files:

    • PIMCalendarSyncSource.java (and parent PIMSyncSource.java)
    • PIMCalendarManager.java (and parent PIMEntityManager.java)
    • init_schema.sql
    • SIFTaskSource.xml
    • VCalendarSource.xml
    • Funambol.xml
    • PersistentStoreManager.xml

    Notice that PIMCalendarSyncSource implements the module API mentioned earlier by making calls into PIMCalendarManager, and PIMCalendarManager manipulates the co-installed dbms on behalf of the “webdemo” default webapp. You should hollow-out the method implementations of PIMCalendarManager and code them so they work for your existing data provider. I don’t have more to say about that part of the job.

    The remainder of the work is configuring the module. Look in init_schema.sql at how the fnbl_sync_source table maps an incoming sourceUri (e.g. stask) to a bean file (e.g. SIFTaskSource.xml) that describes how to configure PIMCalendarSyncSource for that sourceUri. The key insight here is that every sync attempt triggers the creation of a new PIMCalendarSyncSource object, and that object manages state between the beginSync and endSync calls. By creating an object for every attempt, we can use the same Java class to handle different data types (events or tasks, even contacts) for different formats. Differences across data types and formats are largely hidden from you by the convenience methods in PIMCalendarSyncSource for marshalling and unmarshalling formats into Funambol data objects (e.g. Event, Task, and their common parent com.funambol.common.pim.calendar.CalendarContent). If you need to know the data type within your module code, you can use PIMCalendarSyncSource’s entityType member (or add your own member field and an entry in the bean files to populate it).

    A module can support multiple data types and formats by adding an entry for each to the fnbl_sync_source table.

    Be sure to make these changes:

    • Change the package and class name of the ‘object’ element in each bean file to match yours
    • Do the same for the ‘class’ attribute of the fnbl_sync_source_type table in init_schema.sql
    • Note that the values of the ‘config’ attribute of the fnbl_sync_source table aren’t filepaths in your source tree; instead, they reflect where the files will be unpacked from a zip after running bin/install-modules.sh. You need to check build.xml to make sure these files are pulled from the right place during the build.
    • Since you will be using your own solution for user acct management, you need to disable Funambol’s by providing a stub. In init_schema.sql, all entries in fnbl_sync_source_type should reference the stub class for the ‘admin_class’ attribute.

    Next, look in Funambol.xml and notice the officer, store, and serverURI properties. The ‘serverURI’ value is probably ‘{serverURI}’, which indicates a placeholder that will be filled using install.properties when install-modules.sh is run; using placeholders is a pretty convenient way to get config into your module; you can define your own bean java and xml files, add your values to install.properties, and update install-module.xml to map those properties to placeholders in your bean xml file. You can also use a placeholder for the entire ‘officer’ property and put the xml in install.properties (which is helpful if you’re using different officer classes in different environments).

    The ‘store’ property is important because it points to PersistentStoreManager.xml, which you will edit for your module. If your module will support fast/incremental sync, it needs its own db storage to keep track of the item id’s involved in the previous sync (aka “anchors”). For example, if the data provider provides a way of asking for all items changed since a certain time, but it doesn’t distinguish between which are new and which are updated, and doesn’t indicate any of the deleted ones, then you need to compare those id’s with the anchors: anchors that aren’t in the list from the data provider should be considered “deleted”; items from the provider that aren’t among the anchors should be considered “new”; all the rest can be considered “updated”.

    To store and read anchors, you should create files create_schema.sql and drop_schema.sql (referenced in install-modules.xml) for creating and dropping the table(s) you need; if your module is listed in the modules-to-install property of install.properties when you run install-modules.sh, you will be prompted whether to recreate a table; answering ‘y’ will run drop_table.sql and then create_table.sql (so, you generally want to answer ‘n’ when installing unless you want to force all users to do slow syncs next time). To allow the module to read from and update the anchors, create java and xml bean code where the xml defines the SQL for inserting, updating, querying, and deleting. Add the bean xml filename to the list of such beans in PersistentStoreManager.xml.

    Some of the files I’ve mentioned may not be in Funambol’s public source, and I’m sorry for that oversight. But they are helpful folks and I’m sure they would send you some samples.

    Lessons Learned

    Other than issues I’ve already mentioned, here are some issues you should be aware of up-front:

    1. There is no automated update process for getting the latest info about the world of devices. The sync portal needs this so it can offer support for the latest phones, and so it can keep up with changes in support for existing ones. You need to arrange with Funambol to be sent regular “phone pack” updates.
    2. There is no support for sending a message from the server to be displayed in the client. For errors, the best one can do is set a message in a thrown SyncSourceException, since the message will appear in the client log if it uses log level “error”.
    3. There is no way for a module to trigger a change from fast to slow sync in case, say, an error is found with anchor storage.
    4. The module API requires addItem to return the id that your storage will use for the item. This makes it hard if not impossible to include add’s with update’s and delete’s in a single batch during commitSync(); instead, you may need a synchronous network roundtrip for every call to addItem().
    5. It appears that the SIF Event content type used by Microsoft uses UTC-based times (e.g. 20080704T010000Z for 9amPT on July 4) for all values unless it’s an all-day event, in which case it uses YYY-MM-DD with no timezone indicator.  Funambol’s convert() in PIMCalendarSyncSource leaves these all-day values as-is instead of changing them to UTC format. I didn’t find any guiding principle about when UTC versus local time was used.
    6. The default dbms in the open source version of DSS is hypersonic. If you plan to use mysql, be sure to indicate your engine is InnoDB in your DDL scripts since Funambol’s dev wiki mentions there’s a risk of memory leaks without that.
    7. Since you’re using your own storage, make sure to remove the “webdemo” webapp (and associated db tables) so users don’t stumble across it by accident.
    8. When dealing with com.funambol.common.pim.calendar.RecurrencePattern, note that recurrence types with “NTH” in their name are for use when instance!=0, not when interval>0.
    9. Funambol’s Outlook plugin has a bug where if it is given an event that repeats every N years, it will change it to repeating yearly.
    10. Outlook has a bug where if it is given an event that repeats monthly, that’s timed/not-allday, and whose start time crosses a day boundary when converted to UTC (e.g. 4pm Pacific in summer becomes 1am UTC the next day), then Outlook will move it a day late (e.g. from Friday to Saturday). There is a similar problem when Outlook provides such an event, instead of consuming it.
    11. If an event is timed/not-allday but it has a duration of 24 hours, then the Outlook plugin will change it to an allday event, thereby losing start and end times.
    12. When a repeating event is created, synced, and then some individual dates on each end are modified or deleted (they must be different dates on each end), then on the next sync they will show as dupes. This is because Funambol’s default merge method has a bug…such changed dates are represented as an “exclusions list” in the RecurrencePattern obj, and merge should take the union of the device’s list and the data provider’s.
    13. Most devices don’t support multiple calendars, so Funambol has no special support for them, but your data provider may have many users who have these. As long as your module uses a concatenation of calendar id and event id as the full id given to DSS (to guarantee each item id that DSS has is unique within a user’s set of id’s), there should be no problem.
    14. It’s not feasible to host, say, a contacts module in one DSS and a calendar module in another DSS, and rely on sniffing to route http requests to the correct server. The problem is that the sourceUri is the one piece of info that would allow such decisions, but it’s embedded in the request payload, not the http headers.
    15. If a user did a manual sync at the same time that one of his devices happened to do an auto-sync, I believe there is no mechanism in DSS to detect this and block one of the requests, to avoid loss of data integrity for the user. For example, imagine there are similar events on device A, data provider B, and auto-syncing device C; while A and B are merging, C might update the event on B; when the merge of A and B is done and the update happens on B, it wipes out the info from C.
    16. If you use the carrier edition, be aware that your init_schema.sql isn’t the only file controlling what sourceUri’s the DSS thinks it supports. There are also portal/database/cared-coredb-mysql.sql and portal/database/cared-mysql.sql. Be sure to remove INSERT calls from these for sourceUri’s you won’t support, to ensure the user sees an error saying something like “Remote name not recognized” rather than a generic error.
    17. If you see error “Fatal error creating the PersistentStore object” in your log, a possible cause is having the wrong line endings in the config files of your s4j file. For example, if you build the s4j on a Windows machine and try to run it on a *nix machine, you are likely to see this error.
    18. Trivia item: The second param to the ICalendarParser constructor, a String, represents the device’s charset, not its timezone.

    Community Mailing Lists

    Good luck!

    Print Friendly, PDF & Email

    Designing for Mobility, Collaboration, and Information Use by Blue-Collar Workers

    A comment on: Designing for Mobility, Collaboration, and Information Use by Blue-Collar Workers

    The focus of this study was mobile blue-collar workers like copy repairmen, and how they solve problems using wireless equipment like cellphones and laptops.

    The general findings were:

    • The tasks most in need of better support are
      • scheduling later Fact-To-Face mtgs, which is complicated by the need to find mutually available times and locations, record these decisions while having only one or no hands free, and where the other person may not even be able to talk with you at the moment
      • delivering docs to complete an interaction. For example, during a cellphone conversation, the other participant might request that a contract be sent in advance of a mtg; since cellphones generally can’t be used to access or transmit files, the promiser has to remember to send the doc later, perhaps hours later, when he has access to a computer or fax.
    • Some companies need to provide phone services to their mobile employees but are so concerned about cell minutes that they opt for phone cards instead of actual phones. Finding a phone is often a hassle, and these phones are often not close enough to the worksite to allow guidance over the phone.
    • Mobile workers need a good way of indicating a ‘busy’ status, to prevent interruptions. (The article seems to suggest a software solution, but I’m not sure why taking one’s phone off-hook wouldn’t be sufficient, and more reliable than a new type of software.)
    • Workers need a hands-free, out-of-car communication solution.
    Print Friendly, PDF & Email

    “Taking Email to Task” – PARC HCI research

    A comment on: Taking Email to Task

    This paper notes that email has morphed from simple asynchronous msg exchange to a means of managing tasks. For example, users have been observed sending themselves email as reminders, since they know that they scan their inbox several times a day. Similarly, users forward msgs to themselves to keep the msg within the visible area of the inbox. The inbox is also used as a file store — users will send themselves an email with an attachment as a way of quickly accessing the file during the run of the task.

    Based on these observations, a prototype was developed that:

    – Allows senders to indicate a deadline, and all recipients of this msg see a progress bar in the inbox showing how much time remains for this item

    – The inbox and outbox are shown in one view, since the two together represent current tasks

    – Attachments appear as their own item in the inbox, which sidesteps the problem of att’s being hidden within msgs

    These features (and others) were implemented as an Outlook plugin, and a study was done across a variety of users in diff locations. The added features were found to be generally very helpful, although most users stopped using it due to the plugin not supporting the full range of Outlook features they had become dependent on.

    The problem examples suggest some features that the authors didn’t discuss:

    – Allow recipients to create a summary meaningful to them that would be shown in place of the subject text (in a diff color)…useful for archiving how-to info and such

    – Allow senders to mark some recipients as “responsible”. The inbox would also be split into 4 bins: MyTasks, OthersTasks, Unread, and Read. Any msg where I’m marked responsible would go to MyTasks from Unread once I read it; similarly for msgs where only others are responsible, for OthersTasks. The bins help avoid important msgs falling out of the visible area (and MyTasks/OthersTasks/Unread should expand to fit new msgs until full, and then pulse).

    – Items with responsible parties should also have a status: {assigned, accepted, declined, finished}…this is a lightweight form of bugtracking/task resolution.

    – Changes to deadlines, those responsible, or status should cause the display of that item in everyone’s display to change

    This paper fired my already keen interest in Personal Information Management tools (“PIMs”) even further.

    Print Friendly, PDF & Email