The only problem I had was the way these expressions are indented in Emacs. For example, a simple table will be indented as follows:

(:table :id "layout"
        (:tr
         (:td :class "id"
              (:div :class "box"
                    "Content"))))

This is quite annoying for large pages where the content of the page is quickly far too much indented.

Fortunately, with the slime-cl-indent module provided with Slime, it’s possible to customize indentation rules.

I use this small function to modify indentation rules:

(defun nm-cl-indent (symbol indent)
  "Set the indentation of SYMBOL to INDENT."
  (put symbol 'common-lisp-indent-function
       (if (symbolp indent)
           (get indent 'common-lisp-indent-function)
         indent)))

Then I change the indentation rules for all HTML elements:

(defvar *nm-cl-html-symbols*
  (list :a :abbr :acronym :address :applet :area :article :aside :audio :b
        :base :basefont :bdi :bdo :big :blockquote :body :br :button :canvas
        :caption :center :cite :code :col :colgroup :command :datalist :dd
        :del :details :dfn :dir :div :dl :dt :em :embed :fieldset :figcaption
        :figure :font :footer :form :frame :frameset :h1 :h2 :h3 :h4 :h5 :h6
        :head :header :hgroup :hr :html :i :iframe :img :input :ins :keygen
        :kbd :label :legend :li :link :map :mark :menu :meta :meter :nav
        :noframes :noscript :object :ol :optgroup :option :output :p :param
        :pre :progress :q :rp :rt :ruby :s :samp :script :section :select
        :small :source :span :strike :strong :style :sub :summary :sup :table
        :tbody :td :textarea :tfoot :th :thead :time :title :tr :track :tt :u
        :ul :var :video :wbr))

(dolist (symbol *nm-cl-html-symbols*)
  (nm-cl-indent symbol '(&body)))

HTML S-expressions are now more compact:

(:table :id "layout"
  (:tr
    (:td :class "id"
      (:div :class "box"
        "Content"))))

Can't locate module: SWANK-IO-PACKAGE::SWANK-MEDIA

It’s not a problem if the Swank sources are on the machine the application runs on, but if it’s not the case, it’s possible to load the Swank modules in the core.

In the function which generates the core, I simply call the swank-require function of the swank system:

(swank:swank-require '(swank-repl swank-asdf swank-fuzzy swank-indentation
                       swank-media))

This way I don’t have to deploy any source file with my core.

I used to simply call an external process, the image viewer feh, but the following method is more interesting.

You first need to load the slime-media module which is provided with slime, for example by adding it to the list of modules to load with slime:

(slime-setup '(slime-repl
               slime-asdf
               slime-fuzzy
               slime-banner
               slime-indentation
               slime-media))

You then have to authorize Slime to evaluate arbitrary lisp forms coming from the Common Lisp process:

(setq slime-enable-evaluate-in-emacs t)

Let’s display an image:

It looks pretty good, but I really wanted to have the return value printed below the image, and not on its right.

The following function is a modification of slime-media-insert-image; an end-of-line character is inserted just after the image. This way, the result value is printed below the image. Furthermore, when more than one image is displayed during the same call, it won’t be crammed on the same line.

(defun nm-slime-media-insert-image (image string)
  (with-current-buffer (slime-output-buffer)
    (let ((marker (slime-output-target-marker :repl-result)))
      (goto-char marker)
      (slime-propertize-region `(face slime-repl-result-face
                                      rear-nonsticky (face))
        (insert-image image string)
        (insert ?\n))
      ;; Move the input-start marker after the REPL result.
      (set-marker marker (point)))
    (slime-repl-show-maximum-output)))

The code is ugly, inefficient, and there are highlighting errors, but it makes tcpdump logs far easier to read.

Screenshots

Source code

Download

#!/usr/bin/perl

=pod
tcpdump-colorize 1.0
Nicolas Martyanoff <khaelin@gmail.com>
This script is in the public domain.
=cut

use strict;
use warnings;

my $hex = qr/[0-9a-f]/;

my $fqdn = qr/(?:[A-Za-z\d\-\.]+\.[a-z]+)/;
my $ipv4_addr = qr/(?:(?:\d{1,3}\.){3}\d{1,3})/;
my $ipv6_grp = qr/$hex{1,4}/;
my $ipv6_addr = qr/(?:::)?(?:${ipv6_grp}::?)+${ipv6_grp}(?:::)?/;
my $ipv6_rev = qr/(?:(?:$hex\.){31}$hex)/;
   $ipv6_addr = qr/(?:$ipv6_addr|$ipv6_rev)/;
my $port = qr/(?:\d{1,5}|[a-z\d\-\.]+)/;

while (<STDIN>) {
    if (m/^((?:[\d\-]+\s)?[\d:\.]+ )?([A-Z0-9]{2,}(?: \d+\.[\da-z]+(?=,))?)([ ,].*)/) {
        my $timestamp = $1;
        my $protocol = $2;
        $_ = "$3\n";

        print "\e[32m$timestamp\e[0m " if $timestamp;
        print "\e[33m$protocol\e[0m";
    }

    # Numeric IPV6 address and port
    s/ $ipv6_addr\.\K$port(?=[ :,])?/\e[36m$&\e[0m/g;
    s/(?<!seq) \K$ipv6_addr(?=[ :,])?/\e[34m$&\e[0m/g;

    # Numeric IPV4 address and port
    s/ $ipv4_addr\.\K$port(?=[ :,])?/\e[36m$&\e[0m/g;
    s/(?<= )$ipv4_addr(?=[ :,])?/\e[34m$&\e[0m/g;

    # FQDN
    s/ $fqdn\.\K$port(?=[ :,])?/\e[36m$&\e[0m/g;
    s/(?<= )$fqdn(?=[ :,])?/\e[34m$&\e[0m/g;

    # Bridge
    s/(?<= )($port)\.($ipv4_addr|$ipv6_addr)\.($port)(?=[ :,])/\e[36m$1\e[0m.\e[34m$2\e[0m.\e[36m$3\e[0m/g;
    s/(?<= )($port)\.($ipv4_addr|$ipv6_addr)(?=[ :,])/\e[36m$1\e[0m.\e[34m$2\e[0m/g;

    # Packet data (tcpdump -x)
    s/\s+0x$hex+(?=:)/\e[35m$&\e[0m/;

    # Warnings
    s/\[bad udp cksum[^\]]*\]/\e[31m$&\e[0m/;

    print;
}

Osicat provides access to various POSIX functions, including mmap(). In the following example, we will use Osicat and mmap() to read the ID3 version of an audio file.

Mapping a file is done by opening it with open(), reading its size with fstat(), then calling mmap(). Once the file has been mapped we can close the file descriptor, it’s not needed anymore.

(defun mmap-file (path)
  (let ((fd (osicat-posix:open path (logior osicat-posix:o-rdonly))))
    (unwind-protect
         (let* ((size (osicat-posix:stat-size (osicat-posix:fstat fd)))
                (addr (osicat-posix:mmap (cffi:null-pointer) size
                                         (logior osicat-posix:prot-read)
                                         (logior osicat-posix:map-private)
                                         fd 0)))
           (values addr size))
      (osicat-posix:close fd))))

The mmap-file function returns two values: the address of the memory mapping and its size.

Unmapping this chunk of memory is done with munmap():

(defun munmap-file (addr size)
  (osicat-posix:munmap addr size))

Let’s add a macro to safely map and unmap files:

(defmacro with-mmapped-file ((file addr size) &body body)
  (let ((original-addr (gensym "ADDR-"))
        (original-size (gensym "SIZE-")))
    `(multiple-value-bind (,addr ,size)
         (mmap-file ,file)
       (let ((,original-addr ,addr)
             (,original-size ,size))
         (unwind-protect
              (progn ,@body)
           (munmap-file ,original-addr ,original-size))))))

It’s now easy to read data in memory using CFFI functions. An ID3 header starts with 3 bytes containing the ASCII characters "ID3", followed by 2 bytes indicating the ID3 version used:

(defun file-id3-version (path)
  (with-mmapped-file (path addr size)
    (when (< size 5)
      (error "~A doesn't contain an ID3 header." path))
    (let ((id (cffi:foreign-string-to-lisp addr :count 3)))
      (unless (string= id "ID3")
        (error "~A doesn't contain an ID3 identifier." path)))
    (cffi:incf-pointer addr 3)
    (let ((version (cffi:mem-aref addr :ushort)))
      (case version
        (2 :id3v2-2.2.0)
        (3 :id3v2-2.3.0)
        (4 :id3v2-2.4.0)
        (t :unknown)))))

Thanks to Peder O. Klingenberg for pointing out safety issues in the with-mmapped-file macro.

Git doesn’t have any specific rule to name hunks for Common Lisp files, but we can provide a method ourselves.

First add a .gitattributes files at the root of your repository with the following content:

*.lisp diff=lisp

This tells git to look at the diff.lisp section of the configuration file to know how to handle lisp files for diffs.

Then edit your Git configuration file (for example ~/.gitconfig), and add the following section:

[diff "lisp"]
    xfuncname="^(\\((def|test).*)$"

With this regexp, the title of each hunk will indicate the corresponding top-level form, which most of the time starts with the def prefix. I added test which is used to define 5am tests, and is handy for diffs in test suites.

Max Mikhanosha pointed out that we can have a global attribute file since git 1.7.4, which is really nice to avoid modifying the attributes of each repository you work with. Just indicate the path to the attribute file in you git configuration file:

[core]
     attributesfile = ~/.gitattributes

In the zeromq C library, each socket is supposed to be used in a single thread. While this is reasonable given the message-oriented paradigm of zeromq, it is an issue in a language such as Common Lisp where we have a REPL to interact with the application. In Slime, the Common Lisp backend can evaluate each expression is a separate thread. Therefore calling zmq functions from the REPL is definitely not a good idea.

With the new version of lisp-zmq (1.3.0), you can use the :thread-safe key argument when creating a socket:

(zmq:socket context :pub :thread-safe t)

In that case, operations on this socket will be protected by a mutex.

While this mutex makes socket functions safe, you may need to manually lock sequences of operations, for example if you want to receive a whole multipart message in a thread. For this reason, the macro with-socket-locked has been added.

The bordeaux-threads library is used for portable mutexes, it is a new dependency of lisp-zmq.

Note that sockets are now objects, and not foreign pointers to the underlying zeromq sockets. socket-%socket and socket-lock return respectively the foreign pointer to the zeromq socket and the mutex of a socket.

I also added a new function, process-name, which returns the task name of the current process. Oh, and it’s setf-able, which is helpful when you have several threads and want to easily tell them apart.

The patch is available in my Osicat fork on Github.

Some months ago, Nikodemus Siivola wrote sb-daemon, which can be used to daemonize an SBCL process. I find this solution cleaner and simpler, and now use it for all my Common Lisp daemons.

Since I regularly need to create background processes to run various pieces of code, I wrote a Common Lisp script that makes a daemon from the current process, start a swank server, and makes sure to redirect output to files.

Pid files are stored in /var/run/sbcl-daemon/$name where $name is the name of the daemon, and log files are saved in /var/log/sbcl-daemon/$name.

(require :asdf)

(asdf:load-system :sb-daemon)
(asdf:load-system :swank)

(defvar *swank-server*)

(defun signal-handler (signal)
  (format t "~A received~%" signal)
  (sb-ext:quit))

(when (< (length sb-ext:*posix-argv*) 3)
  (error "Missing command line arguments."))

(destructuring-bind (argv0 name port) sb-ext:*posix-argv*
  (let* ((name-directory (concatenate 'string name "/"))
         (log-path (merge-pathnames name-directory #p"/var/log/sbcl-daemon/"))
         (run-path (merge-pathnames name-directory #p"/var/run/sbcl-daemon/"))
         (pid-file (concatenate 'string name ".pid")))
    (sb-daemon:daemonize :exit-parent t
                         :output (merge-pathnames "stdout.log" log-path)
                         :error (merge-pathnames "stderr.log" log-path)
                         :pidfile (merge-pathnames pid-file run-path)
                         :sigterm 'signal-handler
                         :sigabrt 'signal-handler
                         :sigint 'signal-handler))
  (setf *swank-server*
        (swank:create-server :port (parse-integer port)
                             :coding-system "utf-8-unix"
                             :dont-close t))
  (loop
    (sleep 10)))

I then use the following shell script to easily create new daemons:

#!/bin/sh -e

if [ $# -lt 2 ]; then
    echo "Usage: $0 <name> <port>"
    exit 1
fi

name=$1
port=$2

logpath="/var/log/sbcl-daemon"
if [ ! -d $logpath ]; then
    echo "$logpath not found"
    exit 1
fi
runpath="/var/run/sbcl-daemon"
if [ ! -d $runpath ]; then
    echo "$runpath not found"
    exit 1
fi

logpath=$logpath/$name
if [ ! -d $logpath ]; then
    echo "creating $logpath"
    mkdir $logpath
fi
runpath=$runpath/$name
if [ ! -d $runpath ]; then
    echo "creating $runpath"
    mkdir $runpath
fi

sbcl --script $HOME/bin/sbcl-daemon.lisp $name $port

With this code, creating a new background process is as simple as running sbcl-daemon <name> <port>.

I finally configured the slime-banner package, which displays a line at the top of the REPL buffer, and made it shows the hostname of the machine the Lisp process is running on.

(eval-after-load 'slime
  '(progn
     (slime-setup '(slime-repl 'slime-banner))
     (slime-banner-init)

     (setq slime-repl-banner-function
           (lambda ()
             (setq header-line-format
                   (format "%s@%s:%d"
                           (slime-lisp-implementation-type)
                           (slime-eval '(cl:machine-instance))
                           (process-contact (slime-connection) :service))))))

We add 'slime-banner to the list of packages to load with slime, then initialize it with slime-banner-init. slime-banner calls a function which sets the header-line-format variable. We then use slime-lisp-implementation-type to get the name of the Common Lisp implementation (such as SBCL), and slime-port to get the port number of the swank server we are connected to read the port used by the process associated to the current slime connection. Getting the hostname of the remote machine is a bit more tricky, since we need to evaluate some code on this remote machine. We use slime-eval to execute a Common Lisp form on the remote machine. The standard function cl:machine-instance returns the hostname of the machine.

2012-03-24: the code has been modified to correctly display the port. slime-port is the default value used by slime-connect, not the port used by the current connection. The correct way is to get the emacs process associated with the connection, and read the port (:service) it uses.

Reading org-agenda.el tells us than org-agenda-get-restriction-and-command displays the list of agenda commands, and calls delete-other-windows.

A simple hack to avoid this behaviour is to advise this function:

(defadvice org-agenda-get-restriction-and-command
  (around nm-org-agenda-get-restriction-and-command activate)
  (flet ((delete-other-windows () nil))
    ad-do-it))

A proper fix would be to add an option to org-mode indicating whether other windows has to be deleted or not, but this will do for the moment.