omer bar-or · The Plan Askew

I have two major updates for you today. The first is a rather modified version of my test Mac OS X Address Book program. It is called ABColumns because it prints out information about all people in tab-delimited columns. Two notes: first, although this version is specifically designed for people, an earlier version of the program could also easily handle groups, until it became apparent that a "group" in the Mac OS X Address Book is really just a holder of people (which makes sense, given the name) and has all but no other properties to it. So, although it would be pretty nifty to be able to choose a particular group as your data source or to switch to a particular group in OpenOffice.org, it isn't very high priority. Second: this version, in addition to printing columns, can also handle UTF8, rather than simple ASCII. Although I don't know of a way to write UTF8 to a file so that it can be imported into OpenOffice.org (or Excel, or any other program, for that matter), this version of the program, because it is tab-delimited, can be imported into either Excel or OpenOffice.org. I've tested Excel, but have not yet test OpenOffice.org, in order to move to the second, more exciting update.

The second update is that I have begun the process of actually integrating the Mac OS X Address Book into OpenOffice.org. Based on the suggestions of some people at the dev@dba mailing list, I am basing it heavily on the structure of the KDE Address Book code. And, I am including four files from the ABColumns program (everything starting with MacabRecord), though I changed them to handle OUStrings instead of STL strings. Over the past few days, I managed to get the code to compile and now have the following two screenshots. The second, though an error, is exciting because it is an error that comes from my code, which means that my code is being run:

Tomorrow, I will be traveling to Oregon, where I will (finally) once more have a decent Internet connection and be able to lurk on IRC again (my apologies for having disappeared recently), and that's where I'll be for most of the remainder of the summer.

File: abcolumns.tar

As an update to this post on working with the Mac OS X Address Book, I now have another deliverable: a program that prints all of the properties available on the address book. It's still only printing ASCII, and it's not in the correct format to be imported into OOo as a text file, but these issues are hopefully not hard to fix, and since they don't actually involve the Mac OS X Address Book API, I'm not sure that I'll actually take those final steps. I'm also not yet handling groups (the Mac OS X Address Book is split into two databases: groups and people), which is a more important lack and one that I will likely spend some time exploring soon. Anyway, the tarball for the program is available here. A brief description follows.

Basically: the the most important parts of the program are two method, one called printInformationForRecord and the other called getStringRepresentationOfPropertyValue. The former (by using the latter) can print any property in the address book (at least any one that I've been able to create so far). It doesn't assume that it is working with people, though I haven't tested it with groups yet. The latter can turn any type of value into a string (with the exception of "data" values, which are basically byte arrays and thus could be storing any kind of data), where the types are any of the ones listed in Apple's list of possible types. Although these methods only print a property to STDOUT, they are general enough that only a few manipulations would be enough to get them to do more complex things. This might even become the skeleton for handling the Mac OS X Address Book in OOo, though the complexity of OOo code will likely prevent that.

P.S. Thanks to Florian Heckl for correcting errors in the original program that helped me make this new one!

File: abpeople.tar

So, after all of my digging, I've finally managed to get a few meager results out of the Mozilla Address Book (or, more specifically, out of Mozilla's Thunderbird Address Book) inside of OpenOffice.org. The process described in my previous post on the subject let me to change the mozab lines in the "pre" files (see that post for details) and remove the "!defined(MACOSX) &&" in each block. Unfortunately, this solution only led me slightly forward: selecting Thunderbird or Mozilla gave me the error that "No Thunderbird Addressbook Directories Exist" -- see it here. After some more digging, I discovered that OOo was looking for the wrong profile for Thunderbird (looking for a directory called either ".mozilla" or ".thunderbird" in "~/Library/Application Support/" while the correct folder is called "Thunderbird" and is in "~/Library/" -- with this fix and another few tests, I finally have something worthwhile to show you. Here you go! A (somewhat) successful integration of the Thunderbird Address Book into OOo:

The original Thunderbird Address Book:

The Steps to add it to OpenOffice.org:

And, the result (after view->data sources) :

There are still quite a few bugs to figure out. (OpenOffice.org has been freezing a lot more lately, and I'm not sure if it's my new code or not... And, I still get the "No Thunderbird Addressbook Directories Exist" error when I open OOo outside of the debugger...) I'll investigate these more in the future and hopefully soon get started working on the OS X Address Book integration itself. But, this is definitely a happy step in the right direction!

Another development from over the past few days is my first application that interacts with the Mac OS X Address Book. It is available here. Basically, all it does is tell you how many people are in your Mac OS X address book and lists their names. It currently can only handle ASCII values (just so that it can be easily printed to the terminal), and it's an entirely terminal-based application, but it is a first step for interacting with the Address Book. And, perhaps more importantly, it is a first step using the Address Book Carbon API (Carbon is what OOo uses) rather than the Cocoa API. Meaning: it should be possible to integrate the Address Book using Carbon and C (both supported by OOo) rather than Cocoa and Objective-C (which are not). Enjoy!

File: abnames.tar

So, that title is probably a bit more grand than I can actually deliver, but I have spent the past week attempting to figure out how to add services to OpenOffice.org (such as integration with the Mozilla Address Book, which [you might remember] isn't working properly in the Aqua build of OOo. Though I have not yet figured out why it isn't working, I have made a good deal of progress. I now have the Mozilla Address Book drivers installed and breaking, which is (I suppose) a step up from uninstalled, though my current status has them breaking the entirety of OOo, which is (one might say) not an ideal solution.

That aside, however, the point of this post is to walk through the steps that OOo takes to compile the correct services, link to them, and eventually run them.

Before we begin: what is a service? Well, that's actually pretty complicated... by which I mean, "I don't know." But, the short answer is that a service is something that is stored in a file called "services.rdb" that comes with the OOo package (in OS X, it is stored in the application package along with the executable that runs OOo: soffice.bin). The file holds information about all of the other external files that OOo will need to do all of he complex things it can do, or at least a large subset of them. The subset of which we are concerned are the driver libraries for address books, namely the Mozilla Address Book driver libraries, which weren't being added to the file. The files in question are: libmozab2.dylib and libmozabdrv2.dylib

That said, here is the process that I have seen:

1. The files that hold almost all of the information that will eventually go into services.rdb have an extension of "scp" and are stored (mostly, if not entirely) in a set of directories in scp2/source/. The ones that concern us are: scp2/source/ooo/file_library_ooo.scp and scp2/source/ooo/shortcut_ooo.scp. (And, I'm not yet sure that the latter is important.) The former holds information about address book (and other) driver libraries, and the latter holds information about shortcut files that are basically links to the libraries. Inside the former, we find the information about the two drivers that interest us.
2. The files all have directives that look a lot like preprocessing directives. They mostly (as far as I can tell) determine what gets pushed to the next level and what doesn't. When scp2 is built, the files go through a program called pre2par and the results are placed in files with a "par" extension in one of three folders in scp2/unxmacxi.pro/par/. Here, already, the Mozilla Address Book information was not being passed because it failed one of the directives. If it had, it would now be in scp2/unxmacxi.pro/par/osl/file_library_ooo.par and scp2/unxmacxi.pro/par/osl/shortcut_ooo.par. At the same time, scp2 also makes a complete file, with information from all files in the osl directory (I believe) called "setup_osl.ins"
3. When scp2 is delivered, a file called "d.lst," located in "scp2/prj/" tells the deliver script to move the par files and setp_osl.ins to "solver/680/unxmacxi.pro/bin/osl/" (Note: a d.lst file exists in a prj folder in all of the OOo modules I've seen, and I suspect that it always tells the deliver script where to move the important files in a module so that they can be used by the modules that depend upon them and also so that they can be put into the final package.)<li>The par files are used throughout the build process to determine which libraries are needed in the final package and which are not. (So, before I edited file_library_ooo.pre, the mozab drivers were not showing up in the "OpenOffice.org 2.3.app" package.)

4. On the other hand, the "setup_osl.ins" file is used by a script called "make_installer.pl" when the solver module is built. "make_installer.pl" is the script that creates services.rdb. make_installer.pl uses a whole bunch of modules located in "solenv/bin/modules/installer/" - the script and all of the modules all make use of the same global module (globals.pm) that basically holds all of the variables that everything shares. These variables are called with <x>installer::globals::<variablename>, where <x> is the Perl variable type ($ for scalars, @ for arrays, &c.) and <variablename> is the name of the variable (e.g., $installer::globals::debug is a boolean for whether we are debugging or not).

   Note (only for those curious about make_installer.pl): make_installer.pl is basically a list of function calls to external functions in the Perl modules in "solenv/bin/modules/installer/" It sets several global variables from the environment variables and then starts collecting information from files (such as setup_osl.ins) and setting more variables based on them. By line 788, when it runs the function to create services.rdb, only two other lines has actually modified files other than log files. And, these were in the previous few lines, in which the script unzipped files marked "ARCHIVE" in the setup_osl.ins file, and copied files marked "SCP_REPLACE" to a temporary location and replaced variables in the copied files (denoted by ${<variablename>} with the variables values as held by the script. Otherwise, the main goal of make_installer.pl has so far been the creation of variables based on the contents setup_osl.ins, and manipulating of those variables based on other variables in the script (e.g., replacing strings with ${<variablename>} with a variable in a hashmap that goes by that name). Then, to create the services.rdb file, the script uses another external program called regcomp. I don't know exactly how this program works, but it seems to somehow store the contents of the variable $filesinproductlanguageresolvedarrayref into a binary file...

   In order to understand how make_installer.pl works, I kept a running list of all of the variables that it uses until it creates services.rdb (on line 788). It is located here. That file also contains the exact call to make_installer.pl that was made during the build of solver. Some of the variables were too long to manageably include in the file directly, so there are some lines that read: "SEE: <somefile>" - the files that you will need are: d2.ins, files.setup_osl.ins, scpactions.setup_osl.ins, links.setup_osl.ins, profiles.setup_osl.ins, profileitems.setup_osl.ins, m2.ins, d2.ins, and f6.ins. (My naming scheme for files followed a progression: 1) <type>.setup_osl.ins, 2 trough n) <first_letter_of_type><number>.ins. So, a file having to do with files would follow: files.setup_osl.ins, f2.ins, f3.ins, f4.ins, &c.) The files went through so many stages because certain variables, especially those having to do with setup_osl.ins, are manipulated quite a bit. If you are interested, I built Perlscripts to go from each file to the next: fix_directories.pl, fix_files.pl, fix_files2.pl, fix_files3.pl, fix_files4.pl, and fix_modules.pl. The files that the particular script expects and the one that it outputs are included at the beginning of the script. Some also expect the contents of variables stored in the main variables file and are also noted in the beginning. I made these scripts so that I could experiment with what make_installer.pl did without actually doing any installation. Finally, if you are interested, here is a tarball containing all of the files and scripts (including intermediary files).

5. Once services.rdb is created, everything else is built, and the program is ready to run.
6. Upon running, the contents of services.rdb are placed in an XMultiServiceFactory (if I remember correctly, since, for some reason, I can't find right now). And, they wait there for the various services in OpenOffice.org to need them.
7. When we start the "Address Data Source" wizard (File->Wizards->Address Data Source...), OOo populates the list of your options based on certain preprocessing variables (e.g., is the Mozilla address book enabled?). See: the function OAddressBookSourcePilot::implCreateDataSource in extensions/source/abpilot/abspilot.cxx
8. When we select an option from this list, a URL is returned as a unique identifier linking the choice to the corresponding driver. (implCreateDataSource here calls functions located in extensions/source/abpilot/datasourcehandling.cxx). If we select Thunderbird, for example, the function createNewThunderbird is called and gives us the URL "sdbc:address:thunderbird"
9. A request is made for all Base drivers (including MySQL, Evolution, &c.), and each one is queried to see if its URL matches the one requested. All of the drivers are held in an OPoolCollection (see: connectivity/source/cpool/OPoolCollection.cxx), and the function to get a particular driver is "getDriverByURL." If successful, it returns the driver we requested.
10. The first time we do this, the pool has yet to be filled, so a new OPoolCollection is created and is bootstrapped (i.e., filled). It is given an XMultiServiceFactory in its constructor and creates an XDriverManager and and XDriverAccess based on the string "com.sun.star.sdbc.DriverManager" - the XDriverAccess is, in the future, used for every URL query. The factory holds all of the services (and we have seen it before), and when asked to create a manager for "com.sun.star.sdbc.DriverManager," it returns a driver manager that can access all drivers with sdbc URLs (e.g., sdbc:address:thunderbird).
11. Once we have the driver, it handles all interaction with the external source (e.g., Thunderbird). As to how it does this, well, that is where I have gotten stuck now. More updates to come!

So, after two days of one issue or another (see below), I finally got a successful build of OpenOffice.org for Aqua on Wednesday morning. Upon seeing the build, after doing a mental jump for joy, I promptly decided to rebuild yet again, and not because I wanted another five hours of not really being able to work. It turns out that two of my configuration options (I stole them almost word-for-word from the configuration described on the OOo's Aqua Build wiki page) were actually disabling the Mozilla address book, which is what my project is supposed to emulate. The configuration options are (or seem to me to be, as far as I understand how the program works) : --disable-mozab and --disable-mozilla (disable Mozilla Address Book and Disable Mozilla, respectively). I gave the parenthetical caveat because I'm not sure that --disable-mozilla is actually relevant based on OOo's description of configuration options, yet as far as I can tell the Mozilla Address Book is tied to Mozilla itself being enabled. I will investigate this more as the summer continues to see if I can better understand what the different configure options do. Note: after removing --disable-mozilla, the build failed. The build then required two things (only one of which is explicitly mentioned):

• mozilla-source-1.7.5.tar.bz2, to be placed in moz/download (available here, as described by the file mozilla-source.txt, which is also in moz/download)
• libidl2 -- I got this using fink (syntax: fink install libidl2)

The other main peculiarity that came up while I was trying to build is that, as far as I can tell, the OpenOffice.org code will not successfully compile using Java 6, only Java 5. I call this a peculiarity because Java 6 is supposed to be backwards compatible with Java 5, but apparently the Mac OS X version, since it is fairly new, might not be.

After my final build (to date), with the --disable-mozab and --disable-mozilla options removed, I was able to see a few options in the "Address Data Source" wizard (File->Wizards->Address Data Source...). The options available are, "Mozilla / Netscape," "Thunderbird," "LDAP address data," and "Other external data source." See it for yourself! Unfortunately, from here, things weren't quite so straight-forward. Although I removed the --disable-mozab and --disable-mozilla configuration options, when attempting to access the Mozilla or Thunderbird address books, I receive the following error:

I decided, before asking on the IRC channel or either of my mentors what this error means, to learn some more about it, under the assumption that investigating this error would lead me to the right areas of the OpenOffice.org code for integrating the OS X Address Book. It was a lucky guess because it led my almost directly to four directories that look quite promising:

1. extensions/source/abpilot --> handles the address book wizard
2. dbaccess/source/core/dataaccess --> handles data access using the driver provided by cpool
3. connectivity/source/cpool --> stores a pool of drivers and their proxies, fails if a driver does not exist
4. connectivity/source/drivers -> holds all of the drivers available for OOo (or at least for OOo database integration)

I've spent the remaining time since building trying to figure out what the various files/methods/classes in each of these directories is doing, mostly by following what happens from when I click on "next" using "Mozilla / Netscape" until I get the error shown above. I will report my progress as I have it!

Finally, during any off-time I had with OpenOffice.org, I started looking up some OS X Address Book API references. The most helpful of these references has been "A Look Inside Address Book" by Michael Bearn. Based on that article, as far as I can tell, the API works through Cocoa (rather than Carbon), which led me to "Getting Started with Cocoa" and, more specifically, the "What is Cocoa?" tutorial to which it links. Since "Getting Started with Cocoa" suggests reading up on Cocoa "before you write a line of code," and I am still rather early in the summer, I figured that I would do my best to do a lot of reading before starting to code, which is opposite my usual strategy. I will, as always, let you know my progress.