Enabling Postfix on OSX as a Relay: Revisited

Postfix on OSX: Revisited

A few years back, I had written a post on enabling the Postfix MTA as a relay server on OSX, which was quite well received. The article was originally written for OS X Lion, though it remained valid for OSX Mountain Lion, and more recently on
OSX Mavericks as well.

However, things have changed on the OS since the last two versions, which warrants a revisit of the instructions, and a refresh of some of the steps. While the old set of instructions still work, there are changes needed that will keep the configuration future-proof.

The key changes since the article was posted (in February 2012) are:

  1. The launchd configuration has changed somewhat, and a few elements of the launchctl plist file are now deprecated. The old configuration in the org.postfix.master.plist needs to be refreshed
  2. The old configuration depended on local mail-drop in the Postfix queues for the MTA daemon to be started. While this is quite correct (this is Unix, after all), there is another common scenario where the daemon is launched when a client connects to the SMTP ports (default port 25) on the
    local machine.

The Updated launchd Configuration

We would like to retain running the daemon on a mail-drop in the queue, but also enable running the MTA when a connection is made to the SMTP port (default 25). The updated org.postfix.master.plist configuration file listed below does this for us.

In addition, the redundant and deprecated plist tags have also been removed.

You can replace the existing copy with this version as a drop-in replacement at /System/Library/LaunchDaemons/ by using the following shell commands in Terminal.app (assuming you have downloaded the new version at ~/Desktop):

$ cd /System/Library/LaunchDaemons/
$ sudo cp org.postfix.master.plist org.postfix.master.plist.old
$ sudo cp ~/Desktop/org.postfix.master.plist .

The actual configuration file:

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
        <key>Label</key>
        <string>org.postfix.master</string>
        <key>ProgramArguments</key>
        <array>
            <string>/usr/libexec/postfix/master</string>
            <string>-e</string>
            <string>60</string>
        </array>
        <key>Sockets</key>
        <dict>
            <key>Listeners</key>
            <dict>
                <key>SockServiceName</key>
                <string>smtp</string>
                <key>SockType</key>
                <string>stream</string>
            </dict>
        </dict>
        <key>QueueDirectories</key>
        <array>
            <string>/var/spool/postfix/maildrop</string>
        </array>
        <key>AbandonProcessGroup</key>
        <true/>
    </dict>
    </plist>

Once the file has been copied into the correct location (perhaps by using sudo), we need to reload the configuration into launchd:

$ cd /System/Library/LaunchDaemons/
$ sudo launchctl unload org.postfix.master.plist
$ sudo launchctl load -w org.postfix.master.plist

This should be sufficient to reload the new configuration.

Testing out the configuration

We can test the changes via the shell itself by trying to connect via telnet to the smtp port and trying out a few basic SMTP commands (HELO for checking a response, and QUIT to logout of the SMTP session):

$ cd ~                                  # Lets get back to $HOME directory
$ mail &lt;your_id&gt;                        # Send yourself an email to check mail-drop
$ telnet localhost smtp                 # Now lets telnet into the server
Trying ::1...
Connected to localhost.
Escape character is '^]'.
220 localhost.local ESMTP Postfix
HELO
501 Syntax: HELO hostname
QUIT
221 2.0.0 Bye
Connection closed by foreign host.

In the sample SMTP session above, the third line (starting with “220” response) indicates that the Postfix MTA is running, and is ready for accepting mail drops via the SMTP port (as opposed to the mail-drop via the mail command).


OS X Mountain Lion: Need to reinstall Xcode command line tools

So I upgraded to OS X Mountain Lion yesterday. The install was pretty smooth (though had a tough time with the redeem code for the upgrade, worked on the third code that Apple sent across).

Most of the software is working fine. However, I found that my existing Xcode installation needed an upgrade (via a Xcode install from the App Store).

The problem is that any new Xcode install seems to nuke the command line development tools (think make, autoconf, etc.). The solution is pretty simple though. Once Xcode has been installed, run Xcode and then open the preferences. There is a section on downloads, which lists installing the command line tools as an option.

downloads-2012-07-27-11-51.png

Voila! Issue resolved.


Enabling postfix for outbound relay via Gmail on OS X Lion

Update on Dec 21, 2013: I have posted an update to the launchd setup for postfix. You should still read through this post, as most of the setup remains common to both posts.

The background

Mac OSX comes with the postfix MTA, which is a fully featured SMTP server. Under normal circumstances, there is usually no need to enable or configure this software, as most email access is usually done via GUI clients such as the Mail.app – which uses the POP/IMAP and SMTP settings to connect with the email service provider.

However, there are certain circumstances in which having a local SMTP server is very useful, such as:

  1. Allowing the batch logs and output from the cron daemon or other scripts to be sent via Internet email (this is otherwise delivered locally)
  2. Testing email based code; which requires a local sendmail like SMTP server to be present

For such use cases, the postfix server is ideal, as it provides all the features needed (and much more), and is also a nice drop-in replacement for the sendmail program.

While postfix can be used as a full-fledged SMTP server that connects directly to the mail-servers on the Internet, for the use cases above, it is usually better to redirect (i.e., relay) the emails via an authenticated and known server (such as Gmail), as this helps avoid a lot of constraints around open-relays, which are mostly blocked these days to prevent email spam.

Note that configuration of postfix does require dropping down to the command-line, and fiddling with system files. While not complicated, it is definitely not for faint of the heart (though much easier than configuring sendmail).

What you need to know (pre-requisites)

Some of the basic pre-requisites are:

  1. Understanding of the shell prompt and the Terminal.app program
  2. Usage of the sudo program (all the configuration files are owned by root, and hence usage of sudo is essential)
  3. Usage of any command line editor such as vim, Emacs, nano, or any other editor of your choice, that can be invoked with super-user rights (usually via sudo)
  4. A basic understanding of the Apple launchd service manager
  5. The configuration files
  6. A Gmail email ID (actually, any SMTP server credentials will do)

While this article will go step-by-step with the configuration process, knowledge of the above will allow a deeper understanding of the “why” for the changes done.

In the steps below. the $ character before any command represents the shell prompt. Also, I will assume usage of the vim editor in the steps below.

The configuration Files

The configuration files that will be changed are:

 
Name Location Purpose
org.postfix.master.plist /System/Library/LaunchDaemons launchd Configuration for postfix
main.cf /etc/postfix The main postfix configuration
aliases /etc/postfix Local recipient aliases
generic /etc/postfix Sender aliases (for external mail)
passwd /etc/postfix/sasl Relay host authentication

Note that the “/etc/postfix/sasl” directory might not exist, in which case, we will need to create it from the shell prompt:

$ sudo mkdir /etc/postfix/sasl

Step 1: Update the launchd configuration

Update Dec 21, 2013 : While this setup still works, you might want to also see an alternate configuration of postfix’s launchd setup, which I have documented in a follow-up article. The new configuration also allows postfix to be launched when network activity happens on the local SMTP port 25.

The org.postfix.master.plist file located at /System/Library/LaunchDaemons/ is used to start or stop the postfix program on demand, as and when any email is submitted to the mail system for processing. The basic Apple setup is fine, but may need a little tweaking (in my case, the file had a couple of tags which prevented postfix from being started.)

We need to edit the file (as a super user) to match the following content:

$ sudo vim /System/Library/LaunchDaemons/org.postfix.master.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>Label</key>
        <string>org.postfix.master</string>
        <key>Program</key>
        <string>/usr/libexec/postfix/master</string>
        <key>ProgramArguments</key>
        <array>
                <string>master</string>
                <string>-e</string>
                <string>60</string>
        </array>
        <key>QueueDirectories</key>
        <array>
                <string>/var/spool/postfix/maildrop</string>
        </array>
        <key>AbandonProcessGroup</key>
        <true/>
        <key>OnDemand</key>
        <true/>
</dict>
</plist>

Step 2: Edit the /etc/postfix/main.cf file

The next step is to edit the main configuration file for postfix. Do make a backup of the current file before editing.

$ cd /etc/postfix
$ sudo cp main.cf main.cf.orig
$ sudo vim main.cf

Note that the main.cf file is a pretty large one, and has a lot of commented out sections, which should be left as is. Please add the following lines at end of the file.

# Set the relayhost to the Gmail Server.  Replace with your SMTP server as needed
relayhost = [smtp.gmail.com]:587
# Postfix 2.2 uses the generic(5) address mapping to replace local fantasy email
# addresses by valid Internet addresses. This mapping happens ONLY when mail
# leaves the machine; not when you send mail between users on the same machine.
smtp_generic_maps = hash:/etc/postfix/generic

# These settings (along with the relayhost setting above) will make
# postfix relay all outbound non-local email via Gmail using an
# authenticated TLS/SASL session.
smtp_tls_loglevel=1
smtp_tls_security_level=encrypt
smtp_sasl_auth_enable=yes
smtp_sasl_password_maps=hash:/etc/postfix/sasl/passwd
smtp_sasl_security_options = noanonymous

Step 3: Edit the /etc/postfix/aliases file

We need to make a minor edit here, to allow mails sent to the root ID to your local user mailbox.

$ cd /etc/postfix
$ whoami                # This will provide your local user name
$ sudo cp aliases aliases.orig
$ sudo vim aliases
$ sudo newaliases

Find the line in the file which is:

\#root:              you

and replace the “you” with the username provided by the whoami command above.  Also, remove the “#” from beginning of the line.

Remember to run the newaliases command (the last command above), or else changes will not take effect!

Step 4: Edit the /etc/postfix/generic file

This file maps the local user address (usually of the form yourid@machine.local) to a valid Internet email address you would like to use when sending mails to the outside world. In our case, it would basically map your Unix user name to the Gmail ID.

$ cd /etc/postfix
$ whoami                # This will provide your local user name
$ hostname              # This will provide your machine name
$ sudo cp generic generic.orig
$ sudo vim generic
$ sudo postmap generic

In the file, add the following lines at the end of the file (replacing the <username> with the output of the whoami command, and <machinename> with output of the hostname command):

# Translate my primary email address to the Gmail address
# This is ONLY for the outbound email, and does not apply to
# local email.
<yourusername>@<machinename>  <your gmail ID, e.g. user@gmail.com>
@<machinename>                <your gmail ID, e.g. user@gmail.com>

Remember to run the last command (postmap) as otherwise the changes will not be picked up!

Step 5: Edit/Create the /etc/postfix/sasl/passwd file

In this step, we store the SMTP authentication (user ID and password) for Gmail, so that postfix can connect as any other SMTP client to Gmail via an authenticated session.

Note that the file may not exist prior to this step, in which case we will create it.

$ sudo mkdir -p /etc/postfix/sasl    # In case the directory does not exist
$ cd /etc/postfix/sasl
$ sudo vim passwd
$ sudo postmap passwd

Create the following file, replacing <gmailusername> with the ID you use for Gmail (with the “@gmail.com” added at the end), and <gmailpassword> with the password you use to login to Gmail.

[smtp.gmail.com]:587    <gmailusername>:<gmailpassword>

Note that if you use two-factor authenication with Google, then the password to use will be a new application specific password generated via Google’s account settings.

Final Step: Test the settings

We are now good to go. Lets test our settings from the terminal:

$ cd /System/Library/LaunchDaemons
$ sudo launchctl load -w org.postfix.master.plist
$ cd ~                             # Just to be safe, move to your home directory
$ mail <your_id>  # Output of the `whoami' command
# Type in a test email and hit Control-D on a new line
$ mail
# Check whether the email has arrived. Hit 'q' on the '?' prompt to quit

$ mail <your gmail ID>       # Lets now try to send an external mail.
# Type in a test email and hit Control-D on a new line

After the second step above, check your Gmail account for the test mail. If it has arrived, then we have a good configuration.

Summary

Setting up the postfix system on OSX is not particularly hard, but does require some steps. Also, this is just the basic setup to get things up and running. Postfix is an industrial strength mail server has a lot of features (and a corresponding number of configurations). Thankfully, the documentation at http://www.postfix.org/documentation.html is pretty good.

For more details on this specific setup, additional documentation is available at http://www.postfix.org/SOHOREADME.html.

[Updated on 19th Feb 2012]: Corrected a typo.  Thanks to jamrok for pointing it out.


The Missing iSync in OS X Lion (and what to do about it)

The short answer to someone looking for a solution: just copy over iSync from your Snow Leopard installation back to the /Applications folder in Lion, and also copy the phone plugin you need back into /Library/PhonePlugins folder. Everything will be as it was. Enjoy.

The longer story requires more explanation …

The Problem

I have been looking forward to upgrading to the latest version of OSX (Lion) ever since the announcements were made in the WWDC earlier this year.

However, in the middle of the whole slew of new functions and features (“250+”), Lion is also leaving behind a few things – notably Rosetta (the PowerPC translation layer) – and the iSync and FrontRow applications.

The demise of PowerPC emulation has been long in coming, and was not really a surprise, given that it has been 6 years since Intel became the official CPU for Apple machines. The main impact really seems to be for Quicken users, which might actually be a blessing in disguise, given how pathetic Quicken really was on the Mac.
isync-2011-07-21-23-25.png
iSync has been removed presumably because it has been one of the “low usage” applications on the platform. iTunes is the center of most of the Mac based synchronization these days, along with MobileMe for the cloud side of things.

However, iSync has been a key feature of the OS for me, as I carry a non-iOS phone (a Nokia E71 actually), and being able to seamlessly synchronize my contacts and calendars with the phone using iSync made it a key part of my workflow (I even have some Automator scripts to make this easier). And iSync has been able to synchronize over Bluetooth, unlike the tethered experience with the iOS devices so far (though OTA sync is on the way with iOS 5 later this fall).

Alas, OS X Lion removes the iSync application from the hard disk once it is installed, and for a time I thought that this was end of the line for the convenience I had with the E71, and would need to go back to the darks days of manual data entry for the addresses (manual entry of calendar entries is too much of a hassle on the phone, and would need to be dropped all together).

The Solution

Before installation of Lion, I had taken a full disk image of my previous Snow Leopard installation using the excellent SuperDuper! Disk cloner.

This was more from a backup and recovery perspective, but allowed me an unexpected solution to the iSync quandary – on a whim, I attached the Snow Leopard disk to the Mac running Lion, and clicked on the iSync application. Voila! iSync works exactly as it should!

In hindsight, this is not really surprising, since iSync really has been a front end for the underlying sync services (which are still around in Lion), and for managing phone and device specific plugins.

All that was needed was to just copy over iSync back into the “/Applications” folder, and also copy the phone plugins (just one in my case) to the “/Library/PhonePlugins” folder.

phoneplugins-2011-07-21-23-25.png

In summary, while Apple in its immense wisdom took out a feature that was useful to some (albeit a minority), getting it to work again was surprisingly easy. Will need to check out the situation with FrontRow next time.

[Update on July 22, 2011]

Looks like a similar recovery process exists for FrontRow as well.  See this article at Macworld.


How I Learned to stop using Firefox and love the Chrome

One word: SPEED.

Seriously. I have been a long proponent and user of Firefox, having been lured into it by its relative elegance and the extension framework many moons back. Also, the web development support has always been far better than its competition, with Firebug and Web Developer to name just two great reasons for it being the developer’s browser of choice.

firefoxbrowserfreewaystocustomizeyourinternet1.png

The sheer number of Firefox add-ons and extensions (about 13,000 in the last count) is staggering – and list absolute essentials such as Adblock Plus, XMarks and DownThemAll! This combined with the themes (I suggest GrApple Yummy on the Mac) has been making the web browsing experience a far better one for me than Safari.

But the problem with Firefox is … it is SLOW.

With just seven add-ons (Adblock Plus, XMarks, DownThemAll!, 1Password, LastPass, FlashBlock and Firefox PDF Plugin for Mac OS X) it takes about 3-4 seconds to launch the application opening a blank home page on an OS X 10.6.3 MacBook. Another 2-3 seconds before any reasonable page is fully rendered. This becomes excruciatingly slow when I am busily opening tabs from a RSS reader or another application – and frustrating when it has to launch the first time I click on a link in another application.

Also, while a custom theme does look pretty – it sometimes does expose artifacts in the chrome (no pun intended) when rendering new pages – especially in the “awesome bar”.

All in all, while the experience is nice, it certainly is not perfect. Speed of launch and rendering are the main gripes.

googlechrome-getafastnewbrowser-forpcmacandlinux.png

I have been toying with Google Chrome ever since the beta for OS X came out. I was initially put off by the inverted tabs as well as lack of extensions (hey, a 21st century browser with no extensions, come on!) Also, the single URL bar/search bar UI seemed … odd. So while the beta version did stay on the HDD, it did not see much use, and Firefox remained the work horse for daily use.

However, with the recent launch of the stable OS X version, I became interested again. And this time Chrome did have a pretty mature extensions ecosystem, some of which seemed to be reasonable replacements for the Firefox equivalents. Time for a spin!

The first thing which struck me was the speed of launch as well as page renders, and the UI feels much more “fluid”. The Inverted tabs still look odd and out of place, but I understand the need to squeeze the additional 20-30 pixels for actual page use.

Actual page rendering in terms of quality is more or less at par with Firefox, though a few oddball sites (especially the work related sites) sometimes get weird effects. I blame it on the IE centric development though. :-)

The unified bar is also starting to make sense, as it actually helps in not having to remember one extra key short cut for searching. It has good support for Firefox like keyword searches as well (example, ‘wk’ for Wikipedia searches) provided you set them up.

I also found more or less feature equivalent extensions:

Xmarks is available for Chrome
Lastpass is available for Chrome
AdThwart in replacement of AdBlock Plus

I found that FlashBlock does exist for Chrome, but I don’t really need it anymore.

The one big hole in the extensions/add-on replacement is DownThem All! There are quite a few download managers, but none can match the Firefox one in terms of features (I am still looking).

The extension manager is also pretty nice, and arguably better than the Firefox one (at least for FF 3.6.3). However, the actual extensions gallery on Google is not quite as user friendly as the Firefox one. The extensions are not categorized completely, which makes it somewhat of a pain to search and find the right one.

extensions1.png

All in all, the Chrome experience has been a refreshing one so far, and Firefox has not seen much use of late – except where I needed to use DownThem All! (simultaneously downloading all chapters of the free audiobooks from www.librivox.org is one example). If anyone has recommendation for a good replacement, let me know.

So there you have it. My infatuation with Chrome has already lasted more than a week, and I still find it a pleasure to use. Have not really dabbled much with the extensions (and themes – Chrome does have support for these as well) – but am finding that I don’t really need to.


Converting from TaskPaper to Emacs Org-Mode

Why TaskPaper and Org-Mode?

TaskPaper is a simple and elegant task management software for the OSX platform. It combines the simplicity of a text micro-format to mark the tasks, and the elegance of a Mac UI. It also provides a quick launch time and a nice system-wide quick entry window that is accessible with a single shortcut key.

tasks.png

taskpapercapturewindow.png

I have been a heavy user of Emacs’ Org-Mode for some years now, and love the power and flexibility it offers for tracking not just outlines and tasks, but any text based item, including notes and calendar entries. In fact Org-Mode has become one of the primary software that I use regularly, every day.

So where does the link between TaskPaper and Org-Mode come in? Both are text based, and have their own light-weight formats to define outlines and tasks. The underlying files are plain text with the meaning readable even when not viewed by the appropriate software. Org-Mode is obviously much more feature rich than TaskPaper, which by design keeps things simple.

However, there is one big difference that has led me to looking at integration: Org-Mode is Emacs based and hence takes ages to launch. TaskPaper on the other hand launches in under a second, and also offers a nice quick entry form that is available system-wide via a a global short-cut key.

Emacs aficionados will protest now – after all – Emacs is meant to be launched and never shutdown! This is true, but in my usage patterns, Emacs does get closed once in a while and having to launch it just to make a few quick Org-Mode entries (even with the excellent remember mode) becomes a pain.

My usage has now become more of the following:

  1. Use TaskPaper as an initial capture mechanism (sort of as a pre-Inbox store). I make heavy use of the quick entry window here
  2. Use a script to collect the TaskPaper entries and reformat them into a Org-Mode compatible file
  3. Append the converted entries into my primary Org-Mode Inbox whenever I have Emacs open

This (for me) provides best of both worlds – quick and ubiquitous data capture, and the power and flexibility of Org-Mode.

The Ruby Conversion Script

Without further ado, the script to convert from TaskPaper to Org-mode is:

#!/usr/bin/env ruby
#
# Converts Taskpaper files to Emacs org-mode files.
#
# Author: Anupam Sengupta, 2010
#
# Distributed under the BSD license (<a href="http://www.opensource.org/licenses/bsd-license.php">http://www.opensource.org/licenses/bsd-license.php</a>)
#
# Usage: From the command line, enter the command:
#
#  ./tpaper2org.rb &lt;taskpaperfilename&gt;
#
# The output is on STDOUT, which can be redirected to an Org-mode file.
# Whether the generated org-mode file should use odd-level prefix stars
# See <a href="http://orgmode.org/manual/Clean-view.html">http://orgmode.org/manual/Clean-view.html</a> for details.
ORG_USES_ODD_LEVELS = false

LINE_PATTERN = /^(\t*)          # Leading tabs
               -                # Followed by a dash (the taskpaper task identifier)
               (.*?)            # The task description
               ((@\w+\s*)*)     # The tags, if any
               $/x

all_tags = Hash.new(0)

Shiftlvl = ORG_USES_ODD_LEVELS ? 2 : 1 # Determine the number of stars to use in Org-mode entries

while (line = gets()):
  line.chomp!
   md = LINE_PATTERN.match(line)          # Match and extract each line
  if md then                                                # ................ A Task line
    tags = md[3].split(/ +/).reject {|tag| “@done” == tag } # get the tags, except @done tags
    tags = [‘’, tags, ‘’].flatten unless tags.empty?
    puts ‘*’ * (1 + Shiftlvl * (md[1].length + 1)) + (line =~ /@done/ ? “ DONE” : “ TODO”) + md[2] + tags.join(‘:’)
    tags.each { |tag| all_tags[tag] += 1} if tags # Keep a list of all tags
  elsif line =~/:$/                               # ................ A project line
    print “* “
    puts line.chomp(“:”)
  else                          # ................ Any other line
    puts line
  end
end

# Lets do a summary of the tags used.
puts &lt;&lt;END
# The tags used till now.
#+TAGS:#{all_tags.keys.sort.join(‘ ‘)}
END

Note that Org-mode supports multiple prefix styles with ‘*’. In particular, the odd-levels versus the odd-even levels is interesting and useful. The script has a ‘ORG_USES_ODD_LEVELS’ global variable that can be set to true if this is the desired export format.

In addition, the script also adds the tags used in the TaskPaper file as a ‘#+TAGS’ entry in the exported org-mode file. You can comment this if this is not required.

The TaskPaper format

The TaskPaper format is simple, and the file (which by default ends with the extension ‘.taskpaper’) is essentially a plain text file that can be opened and edited in any text editor.

The format can be summarized as (from the TaskPaper User’s Guide):

A project is a line ending with a colon:

      A Project:

A task is a line starting with a dash followed by a space:

    - My First Task

A Note is any line that is NOT a project or a task (i.e., does not start with a dash or end with a colon):

    Notes for a task

A tag is any word prefixed with the @ symbol. The tag can optionally have a value in parentheses after the tag name:

      - My First Task @atag @another_tag(1)

Outlining is done by indenting the tasks with tabs:

     - My First Task @atag
         - My sub-level task

Usage

Using the script is simple. Assuming that TaskPaper’s file is named tasks.taskpaper, from the OS X terminal, run the following command:

        $ tpaper2org.rb tasks.taskpaper >> tinbox.org

Where tinbox.org is the destination org-mode file.

This shell command can be put into a cron job or invoked from within Emacs to pull in the tasks as required. You may also want to delete the TaskPaper file (or empty its contents) after this is done, to prevent duplicate entries being imported the next time the Ruby Script is run.

Links

taskpaper.el is an Emacs mode for emulating the TaskPaper interface with support for projects and tasks. The tags support seems to be missing.

Discussion on the Org-Mode mailing list about Taskpaper and Org-Mode.


Firefox 3.6 finally allows true Fullscreen mode on Mac OSX

Updated to Firefox 3.6 today. As usual, a very solid update, and was pleasantly surprised by the full-screen mode (finally!) that works exactly as you would want it. This was one of the few areas where Firefox on OSX was lagging behind as compared to other platforms.

There were a few plugins that claimed to enable this in prior versions, but never worked very well.

The option is available under the “View”menu and has a standard Apple-Shift-F shortcut as well.

firefox1.png


gnuplot with AquaTerm on OSX Snow Leopard

The gnuplot graphing utility has always had excellent support for multiple terminal types. While the X11 terminal is a satisfactory GUI view for the graphs, I prefer to use the AquaTerm terminal on OSX as it is more ‘Mac-like’ and feels more natural.

Also, I do prefer to compile gnuplot by myself on OSX rather than downloading the pre-packaged binaries – as this gives me more control over the compilation (including getting around the stupid Apple readline bug – where Apple has essentially shipped a broken readline by using libedit to emulate the non-existent libreadline).

This local compile requires that AquaTerm be installed so that the library dependencies for aquaterm exists in:

terminal.png

and the corresponding headers are available at:

1____terminal.png
In addition, the AquaTerm.app itself resides in /Applications.

However, on OS X Snow Leopard, there is a catch – the version of AquaTerm is 32 bit, whereas the default compilation of gnuport results in a 64-bit version – which is not able to load the 32-bit libaquaterm dynamic libraries.

littlesnapper.png

In such a case, the gnuplot compilation does succeed – however, the default terminal becomes the X11 version – which is back to square-one.

A darwinports port does exist for gnuplot – however, as mentioned in an earlier post, this port seems to depend on half of the port repository (i.e., a ton of stuff you do NOT want gets installed as well).

However, there is a easier way to get around this situation. Here’s how.

  1. First install the default binary for AquaTerm from SourceForge and install normally. This step is to basically setup the right folders and symlinks so that you do not have to muck with these later
  2. Now install AquaTerm again from Darwinports – this port has the correct patches needed – and more importantly – builds a 64 bit version by default. This will also install the application under /Applications/MacPorts/

2____terminal.png

  1. Now comes the fun part. We will replace two folders from the darwinports version to the previously installed AquaTerm.
    • Step 1: Replace /Library/Frameworks/AquaTerm.framework with /opt/local/Library/Frameworks/AquaTerm.framework. This will ensure that the correct 64 bit AquaTerm libraries get referenced by the gnuplot compilation
    • Step 2: Replace /Applications/AquaTerm.app with /Applications/MacPorts/AquaTerm.app. This will ensure that the correct 64-bit AquaTerm binary is in the correct location
    • Step 3 (Optional): You can now uninstall the darwinports version by running sudo port uninstall aquaterm from a terminal window
  2. Download the source code for gnuplot and extract the same.
  3. Run ./configure (using a command line parameter to ignore the broken Apple readline) and then make and make install (install will happen in /usr/local)

3____terminal.png

That’s it! The compilation should now succeed and gnuplot will be linked with the correct 64-bit aquaterm dynamic library. Enjoy!

4____terminal.png
figure0.png


Snow Leopard – First Impressions

I finally bit the bullet and installed Snow Leopard over last weekend. I did a clean install (i.e., reformat and install afresh) – considering the significant under-the-hood changes that this OS revision has.

The basic steps I followed are:

  1. Cleaned up the existing Leopard install to remove applications, preference panes and documents I did not need
  2. Reviewed the preferences under ~/Library/Preferences as well as the application support items under ~/Library/Application Support and removed the items which were out of date or detritus from old installs
  3. Took an image backup using SuperDuper! of the Leopard install
  4. Took a backup of the disk using Time Machine to allow easier data restore during the installation
  5. De-authorized my iTunes to prevent increment of the install counter after reinstall of the OS
  6. Popped in the Snow Leopard DVD and started the install
  7. During installation, I used the Disk Utility to reformat the Mac OS partition
  8. I also did a custom install to prevent the printer drivers (over 2 GB) being installed. In addition, I selected the optional Rosetta and Quicktime 7 components (seriously Apple, these take less than 10 MB – why the optional tag?)
  9. Waited for around 20 minutes to have the base image installed, and then used the migration assistant to restore data from my Time Machine backup

The install was relatively smooth, and restoring the data from my Time Machine backup also went without any issues. However, post-install, some of the glitches started appearing:

  1. The default font for Safari as well as Firefox went crazy – with large bold fonts appearing. It turned out to be a font substitution issue with the Arial font, which for some reason was missing the “Regular” type. Restored the font from the old image backup and every thing was back to normal. It appears a lot of people are facing this issue
  2. Adobe’s Photoshop Elements 6 refused to start, stating expiry of the license. This also is a known issue, and a restore of the /Library/Application Support/FLEXnet Publisher/ folder from the image backup made things normal again
  3. The key chain was not allowing storage of any new passwords, and was also not allowing viewing of the stored passwords. This is a known issue and my solution was to simply change the login keychain password once and reset it back again to the original password
  4. Emacs 23 was not compiling under Snow Leopard. There is already a patch available in CVS. However, there are incompatibilities with the 64 bit GCC – hence you need to pass a CC=“gcc -arch i386” as part of the ./configure invocation. The compilation succeeded with this. Note that a recompile of some of the user installed lisp packages was required (notably Icicles)

Other than the issues listed. rest of the experience seems OK so far. The look and feel is not too different and most of the user visible changes are subtle – except for the new QuickTime Player – which is a dumbed down and dressed up version of the venerable QT Player.

In summary, this is an OS upgrade that does have the potential to break some of the old applications, though most of the issues do seem to have work arounds – it is also going to become a required update pretty soon – given the internals changes that Apple has made. No need to rush out right now (mea culpa!) to upgrade – but do be prepared to do the update in the near future.


Importing contacts from OSX Addressbook to Emacs BBDB

A major aspect of using Emacs is that it is more than a text editor – it is more of an application platform – some would call it an Operating System – that allows a host of applications to run within the same context and session – and vastly enrich the user experience. This is in fact a primary reason why many Emacs users spend their entire computer interaction via Emacs.

This unique aspect of Emacs is made possible by the availability of Emacs Lisp as a first class scripting/programming interface for the editor – and has made possible application packages such as W3 (a web browser), Gnus (the NNTP and mail reader), Eshell (a lisp based shell), Wanderlust (a very capable email client), and the Insidious Big Brother Database – aka BBDB – which is a contact manager and the subject of this post.

(BTW, the list of application is only a very small fraction of what is actually available on Emacs – see www.emacswiki.org for a ton of other packages).

As EmacsWiki points out, BBDB is:

“A complete address book for nearly every email and news client written for Emacs, and its functionality supports (by default) making entries in the address book automatically when reading email or posts by people, and easy lookup of addresses while composing messages.”

In essence, it is an always available address book that works with other packages such as mail, Gnus and org-mode to provide a easy to access repository of contact information. The repository itself is a plain text lisp file which by default is stored as ~/.bbdb.

However, as with any contact manager, the problems are usually with:

  1. Initial data import, and
  2. Synchronization with other repositories and devices

For my platform (Apple OSX), the first problem above translates to data import from my primary OS level address book – the AddressBook.app. The second problem is not much of an issue for me since I use AddressBook.app as the primary contact list elsewhere and sync my devices to that. For keeping BBDB in sync, I have a cron job to automate the scripts below.

The contacts import to BBDB requires a data conversion of the address from custom format in which AddressBook.app stores the contacts to the BBDB lisp format. The steps required are as follows:

  1. Extract the records from AddressBook.app
  2. Parse and reformat the relevant fields that BBDB requires, and
  3. Output the reformatted records in the BBDB lisp format

Luckily, for step # 1, an open source command line utility already exists – appropriately called contacts. It is available from http://gnufoo.org/contacts/ and is a svelte 194KB download. This utility allows querying of the OSX Address book and extraction of specific fields and records. The output is a simple CSV text file.

For step # 2 and #3, additional scripting is required. I have written a simple Ruby script called contacts2bbdb.rb to run the contacts utility and output the BBDB repository file:

#!/usr/bin/env ruby -w
#
# Filename: contacts2bbdb.rb
# Description: Converts addresses from OSX Addressbook.app to BBDB format
# Author: Anupam Sengupta (anupamsg ... A-T ... G - M - A - I - L )
# Maintainer: Anupam Sengupta
#
# (c) 2007-2016 Anupam Sengupta.
#
# Created: Fri May 25 15:53:22 2007
# Version: 1.0
# Last-Updated:
#           By: Anupam Sengupta
#     Update #: 733
# URL: http://slashusr.wordpress.com
# Keywords: `BBDB’, ‘OSX’, ‘contacts’, ‘convert’
# Compatibility: GNU Emacs 21 and above
#
#--------------------------------------------------------------------
#
# Commentary:
#
#  Converts the addresses and contacts from Apple OSX’s system addressbook
#  format (Addressbook.app) to Emacs’ BBDB format.
#
#  Requires the following additional software:
#
#  1. Ruby V1.8 and above (http://www.ruby-lang.org/)
#  2. Big Brother Database (BBDB) package for Emacs (http://bbdb.sourceforge.net/)
#  3. The ‘contacts’ program to read Addressbook’s contacts
#     (http://gnufoo.org/contacts/)
#
# Usage:
#
#  1. Install Ruby, BBDB and contacts, if not already present
#  2. Backup the OSX Address book (export the addresses as addressbook archive)
#  3. Run this Ruby script to generate the converted records in BBDB format in the STDOUT
#  4. Save the STDOUT output of this script to bbdb.new
#
#     $ ruby contacts2bbdb.rb &gt; bbdb.new
#
#  5. Replace your .bbdb file with bbdb.new
#
#--------------------------------------------------------------------
#
# Change log:
#
#
#--------------------------------------------------------------------
#
# License: GNU GPL V2.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; see the file COPYING.  If not, write to
# the Free Software Foundation, Inc., 51 Franklin Street, Fifth
# Floor, Boston, MA 02110-1301, USA.
#--------------------------------------------------------------------
#
Delim = “\t”                    # Default delimiter
ContactsProg = ‘/usr/bin/env contacts’ # The contacts program
#
ContactsProgParams = ‘-H -l’
#
# A map between the LDIF field names and the actual field name
ContactFields = {
:fn =&gt; ‘firstName’,
:ln =&gt; ‘lastName’,
:c  =&gt; ‘company’,
:nn =&gt; ‘nickName’,
:he =&gt; ‘homeEmail’,
:we =&gt; ‘workEmail’,
: oe =&gt; ‘otherEmail’,
:hp =&gt; ‘Home’,
:mp =&gt; ‘Mobile’,
:Mp =&gt; ‘Main’,
:wp =&gt; ‘Work’,
}
#
ContactFormatOpts = ContactFields.keys #  Options to pass to the contacts program
ContactsProgFormat =  “‘%” + ContactFormatOpts.inject { |s, f| s.to_s + “#{Delim}%#{f.to_s}” } + “‘“
ContactsFullExec = “#{ContactsProg} #{ContactsProgParams} -f #{ContactsProgFormat}”
#
output = `#{ContactsFullExec}`.split(/\n/) # Read the output of contacts program
#
# Start parsing the contacts output
records = output.map do |line|
record = Hash.new(nil)
line.chomp.split(Delim).each_with_index { |f, i| record[ContactFields[ContactFormatOpts[i]]] = f.strip unless f =~ /^\s*$/ }
record
end
#
# Start outputing the details to STDOUT
puts &lt;&lt;END
;; -*-coding: utf-8-emacs;-*-
;;; file-version: 6
END
#
for r in records do
r[‘nickName’] = nil           # No need for the nick names.
outs = %w{ firstName lastName nickName company }.inject(“[“) { |s, f| s + (r[f] ? “\”#{r[f]}\” “ : “nil “) }
outs = %w{ Home Main Mobile Work}.inject(outs + “(“) { |s, f| r[f] ? s + “[\”#{f}\” \”#{r[f].strip}\”] “ : s } + “) “
outs = %w{ homeEmail workEmail otherEmail }.inject(outs + “ nil (“) { |s, f| r[f] ? s + “\”#{r[f]}\” “ : s } + “) “
outs += “((creation-date . \”2009-02-08\”) (timestamp . \”2009-02-08\”)) nil]”
puts outs
end
# End of contacts2bbdb.rb script

A companion shell script to automate the execution of the Ruby program and also backup the previous BBDB file is also provided. Before usage, please edit the BBDB folder and file names:

#!/bin/sh
#
# Filename: contacts2bbdb.sh
# Description: Converts addresses from OSX Addressbook.app to BBDB format
# Author: Anupam Sengupta (anupamsg@gmail.com)
# Maintainer: Anupam Sengupta
#
# (c) 2007-2016 Anupam Sengupta.
#
# Created: Fri May 25 15:53:22 2007
# Version: 1.0
# Last-Updated:
#           By: Anupam Sengupta
#     Update #: 740
# URL: http://slashusr.wordpress.com
# Keywords: `BBDB’, ‘OSX’, ‘contacts’, ‘convert’
# Compatibility: GNU Emacs 21 and above
#
#--------------------------------------------------------------------
#
# Commentary:
#
# Runs the export of OS X Address Book contacts to the Emacs BBDB file.
#
# Requires:
#   1. The contacts2bbdb.rb Ruby script
#      (http://slashusr.wordpress.com/)
#   2. The Ruby interpreter (Version 1.8 and above)
#      (http://www.ruby-lang.org/)
#   3. The Big Brother Database (BBDB) Emacs Package
#      (http://bbdb.sourceforge.net/)
#   4. The contacts command line program to extract the records
#      (http://gnufoo.org/contacts/)
#
# Usage:
#   1. Backup the Addressbook.app’s addresses
#   2. Install BBDB and ‘contacts’, if not already present
#   3. Edit the BBDBDIR and RUBYPROG variables below
#   4. Open Terminal.app and run this script
#   5. The script will also create a backup of the previous bbdb file
#
#
#
# TODO: Edit these variables to match the file locations
BBDBDIR=$HOME/.emacs.d/data     # Edit this to match location of the bbdb file
BBDBFILE=bbdb                   # Edit this to match your bbdb file name
RUBYPROG=$HOME/bin/contacts2bbdb.rb # Name and loction of the Ruby Script
#
#######################################  Do not edit beyond this point !
#
RUBYEXEC=`which ruby`
#
if [ -x ${RUBYEXEC} ]; then
cp ${BBDBDIR}/$BBDBFILE ${BBDBDIR}/bbdb.bak
${RUBYEXEC} ${RUBYPROG} &gt; $BBDBDIR/$BBDBFILE
fi
echo “BBDB Export and conversion completed.”
#
# End of contacts2bbdb.sh

Do remember to take a backup of your existing BBDB file as well as a backup of the OSX address-book before running the scripts!

Enjoy!

Update [17th Aug 2009]: Emacs-fu has a nice bbdb tutorial at emacs-fu: managing e-mail addresses with bbdb


Follow

Get every new post delivered to your Inbox.

Join 67 other followers