Light it up! Pygments for Emacs Lisp.

The Challenge

More than 2 years ago, the formidable @bbatsov of Emacs Redux had this to say:

After so many years pygments (a popular syntax highlighting library used by GitHub & others) still lacks proper support for Emacs Lisp #fail

— Bozhidar Batsov (@bbatsov) September 15, 2012

Well, let's turn that #fail-frown upside down!

The Python

A quick search brought me to this page: Write your own lexer -- Pygments. Turns out that the Pygments development takes place on Bitbucket, so I had to start an account there. I shortly cloned the repository:

hg clone https://[email protected]/birkenfeld/pygments-main

Then I quickly copy-pasted some starting code:

__all__ = ['SchemeLexer', 'CommonLispLexer',
           'HyLexer', 'RacketLexer',
           'NewLispLexer', 'EmacsLispLexer']

class EmacsLispLexer(RegexLexer):
    """
    An ELisp lexer, parsing a stream and outputting the tokens
    needed to highlight elisp code.
    """
    name = 'ELisp'
    aliases = ['emacs', 'elisp']
    filenames = ['*.el']
    mimetypes = ['text/x-elisp']

    flags = re.MULTILINE

    # the rest of the code was copied from CommonLispLexer for now

Apparently, infrastructure-wise, I only need to know two commands. The first one needs to be run just once, so that Pygments is aware of the new lexer:

$ cd ~/git/pygments-main && make mapfiles

The second command is to (re-)generate /tmp/example.html:

$ cp ~/git/emacs/lisp/vc/ediff.el \
  ~/git/pygments-main/tests/examplefiles/
$ ./pygmentize -O full -f html -o /tmp/example.html \
  tests/examplefiles/ediff.el

I would repeat the last line with each update to the code, and then refresh the page in Firefox to see the result.

The Elisp

To finalize the lexer, the following tasks ensued:

• get a list of built-in macros
• get a list of special forms
• get a list of built-in functions

In the process, I've added two more lists:

• a list of built-in functions that are highlighted with font-lock-keyword-face:

  'defvaralias', 'provide', 'require',
  'with-no-warnings', 'define-widget', 'with-electric-help',
  'throw', 'defalias', 'featurep'
  

• a list of built-in functions and macros that are highlighted with font-lock-warning-face:

  'cl-assert', 'cl-check-type', 'error', 'signal',
  'user-error', 'warn'
  

To generate the other three lists, I started off writing things in *scratch*, but after a while my compulsion to C-x C-s kicked in and I've saved the work to research.el. At least, thanks to @bbatsov, I'm not C-x C-s-ing that much since I've added this:

(defun save-and-switch-buffer ()
  (interactive)
  (when (and (buffer-file-name)
             (not (bound-and-true-p archive-subfile-mode)))
    (save-buffer))
  (ido-switch-buffer))
(global-set-key "η" 'save-and-switch-buffer)

But it's time for the student to one-up the master, so here's a tip to improve even further:

(defun oleh-ido-setup-hook ()
  (define-key ido-buffer-completion-map "η" 'ido-next-match))

This way I can cycle the buffers with the same shortcut that invokes save-and-switch-buffer. The defaults are C-s and C-r, in case you didn't know.

The C

Getting the list of built-in C functions and special forms, obviously involved browsing the C source code. In case you don't (yet) have the Emacs sources, they're here:

$ git clone git://git.savannah.gnu.org/emacs.git

I switched to the ./src directory and called M-x find-name-dired with *.c to build a list of all the sources. Then I ran the following code from research.el:

(defvar foo-c-functions nil)
(defvar foo-c-special-forms nil)

(defun c-research ()
  (let ((files (dired-get-marked-files))
        (i 0))
    (dolist (file files)
      (message "%d" (incf i))
      (with-current-buffer (find-file-noselect file)
        (goto-char (point-min))
        (while (re-search-forward "^DEFUN (" nil t)
          (backward-char 1)
          (let ((beg (point))
                (end (save-excursion
                       (forward-list)
                       (point)))
                str)
            (forward-char 2)
            (search-forward "\"" nil t)
            (setq str (read (buffer-substring-no-properties
                             (+ beg 2) (1- (point)))))
            (if (re-search-forward "UNEVALLED" end t)
                (push str foo-c-special-forms)
              (push str foo-c-functions))))))))

This was beautiful, by the way, to just generate this sort of documentation from such well-formatted and documented C sources. Free Software FTW.

If you're interested, there are 1294 built-in functions. Here's a list of 23 special forms that I found:

and catch cond condition-case defconst
defvar function if interactive let let*
or prog1 prog2 progn quote
save-current-buffer save-excursion
save-restriction setq setq-default
unwind-protect while

You can read up on the special forms in the SICP. There's no node for them, so just use isearch.

The Result

You can see it here: ediff.html, as well as on the rest of the site, since I've switched it on everywhere.

The Impact

Unfortunately this won't have impact on the Github source code highlighter, since Github dropped Pygments recently.

But people that use the static blog generator Jekyll or the LaTeX package minted (that's the package that org-mode's PDF Export uses by default) will be able to get better Elisp highlighting. In fact, this blog is already using the new highlighter.

See the rest of projects that use Pygments here

The Bitbucket

So now, to share the new lexer with the world I just have to learn how to:

• stage and commit in Mercurial
• push Mercurial to Bitbucket
• open a pull request on Bitbucket

I don't want to become a hipster, these things just happen.

upcase-word, you so silly

Do you know what the most frequently used Emacs commands are? I can confirm by my own experience that they are next-line and previous-line (in that order). So why do M-u - upcase-word, M-l - downcase-word, and M-c - capitalize-word have such terrible synergy with Emacs's best commands?

No, upcase-word, this is not what I had in mind:

Learn basic keystroke commands
Overview of Emacs features at gnu.org

Learn basic keySTROKE commands
Overview of Emacs features at gnu.org

If you say:

But how did you get the cursor in such a crazy position in the first place? You should have used M-b/M-f.

Well, I got there with previous-line - one of the best Emacs commands!

Resolve the *-word malarkey with defadvice

Here are some simple advice commands that I've just rolled:

(defadvice upcase-word (before upcase-word-advice activate)
  (unless (looking-back "\\b")
    (backward-word)))

(defadvice downcase-word (before downcase-word-advice activate)
  (unless (looking-back "\\b")
    (backward-word)))

(defadvice capitalize-word (before capitalize-word-advice activate)
  (unless (looking-back "\\b")
    (backward-word)))

Small explanation to the Elisp novices:

1. before upcase-word is called, execute the body of upcase-word-advice
2. unless we are at the beginning of the word
3. backward-word once to move to the beginning of the word

I'm intentionally not using the newest advice system here, since not everyone has yet upgraded to Emacs 24.4. In fact, I saw this gem today at Stack Overflow:

I am using Emacs 23 and cedet 1.0.1 ...

Sometimes things break

I was very surprised to find the lispy build broken after I pushed some minor update, like a change to README.md. I mean, how in the world would a few words in README.md break the Elisp tests? Upon investigation, it turned out that only one test was broken ¹. This one:

(ert-deftest clojure-thread-macro ()
  (require 'cider)
  (should
   (string=
    (lispy-with
     "|(map sqr (filter odd? [1 2 3 4 5]))" "2(->>]<]<]wwlM")
    "(->> [1 2 3 4 5]\n  (map sqr)\n  (filter odd?))|")))

The culprit was an update in clojure-mode's indentation. The previous behavior:

(->> [1 2 3 4 5]
   (map sqr)
   (filter odd?))

is now replaced with:

(->> [1 2 3 4 5]
     (map sqr)
     (filter odd?))

Thankfully, the Travis CI in combination with cask is keeping me up to date. Apparently, there were some heated discussions accompanying the change, and there was some reverting going on. Anyway, it looks to me that both approaches have merit: the first one is more logical, since ->> is an operation akin to Elisp's with-current-buffer, where the first argument is different from the others, while the second one is more aesthetically pleasing. Fine with me either way, I'm not complaining:)

Also, the key sequence in the test is pretty ancient. These days I'd probably use: 2(->>C-fd<j<skwAM. I've recently done a more complex Elisp refactoring screencast, check it out here. Later on, I plan to do more Emacs-related screencasts (not just lispy-related) on my channel.

If you haven't tried lispy yet, you're missing out - doing this refactor operation feels like you're doing the 15-number puzzle:

And that's fun in my book. But let me get back to the short overview of the Emacs testing tools that lead me to this post, mainly cask.

cask: what does it do?

According to its own documentation:

Cask is a project management tool for Emacs Lisp to automate the package development cycle; development, dependencies, testing, building, packaging and more.

Yes, please, I'd like to do that! But after the exciting intro sentence, there's very little followup documentation-wise. It took me ages to figure out how cask can actually give me some tangible benefits, since I thought that package.el is enough to maintain my own config (it still is).

tangible benefits of cask

I'd like to be sure that my packages work across recent Emacs versions. I'm using the bleeding edge myself, but people who download my packages from MELPA might be using something older, like emacs-24.3.

So I want to run my tests on both versions. Also, even for just one version, the tests need to be run in a minimum environment, i.e. with only the dependencies loaded, so that my personal configuration does not interfere with the tests.

This is where cask actually shines: it can bootstrap a whole new .emacs.d, separate from your own, just for running tests. It can do it on your machine as well as on Travis CI.

Here's my Cask file for lispy:

(source gnu)
(source melpa)

(package-file "lispy.el")

(files "*.el" (:exclude "init.el" "lispy-test.el"))

(development
 (depends-on "helm")
 (depends-on "ace-jump-mode")
 (depends-on "noflet")
 (depends-on "iedit")
 (depends-on "multiple-cursors")
 (depends-on "cider")
 (depends-on "slime")
 (depends-on "geiser")
 (depends-on "projectile")
 (depends-on "s")
 (depends-on "highlight"))

And here's the Makefile:

EMACS = emacs
# EMACS = emacs-24.3

CASK = ~/.cask/bin/cask
CASKEMACS = $(CASK) exec $(EMACS)
LOAD = -l lispy-inline.el -l lispy.el -l lispy-test.el

all: test

cask:
    $(shell EMACS=$(EMACS) $(CASK))

compile:
    $(CASKEMACS) -q  $(LOAD) lispy.el \
    --eval "(progn (mapc #'byte-compile-file '(\"lispy.el\" \"lispy-inline.el\" \"le-clojure.el\" \"le-scheme.el\" \"le-lisp.el\")) (switch-to-buffer \"*Compile-Log*\") (ert t))"

test:
    $(CASKEMACS) -batch $(LOAD) -f ert-run-tests-batch-and-exit

clean:
    rm -f *.elc

As you can see, the Makefile has two separate testing targets: an interactive one (compile) and a non-interactive one (test). There's actually some validity to this, since it happened once that the same tests we failing in non-interactive mode, but passing in interactive mode. Also, compile obviously compiles, testing for compilation warnings/errors. I can change the Emacs version at the top, although I don't have to do it too often.

Finally, here's .travis.yml:

language: emacs-lisp
env:
  matrix:
    - EMACS=emacs24

before_install:
  - sudo add-apt-repository -y ppa:cassou/emacs
  - sudo apt-get update -qq
  - sudo apt-get install -qq $EMACS
  - curl -fsSkL --max-time 10 --retry 10 --retry-delay 10 https://raw.github.com/cask/cask/master/go | python

script:
  - make cask
  - make test

So each time I push a change to github, Travis CI will

• install emacs24
• install cask
• install the packages from MELPA:
  • helm
  • ace-jump-mode
  • noflet
  • iedit
  • multiple-cursors
  • cider
  • slime
  • geiser
  • projectile
  • s
  • highlight
• load Emacs with these packages
• load lispy-test.el and run it
• show up green if make test returned 0

Seems a bit wasteful, but it's the Cloud - what can you do?

Easy helm improvement

When you press DEL (also known as backspace) in a helm buffer, and there isn't any input to delete, it only errors at you with:

Text is read only

Why not make it do something useful instead, for instance close helm?

Easy:

(require 'helm)
(defun helm-backspace ()
  "Forward to `backward-delete-char'.
On error (read-only), quit without selecting."
  (interactive)
  (condition-case nil
      (backward-delete-char 1)
    (error
     (helm-keyboard-quit))))

(define-key helm-map (kbd "DEL") 'helm-backspace)

Poyekhali!

Welcome to (or emacs!

My name is Oleh and I've been using Emacs for about 3 years now. I think that it's an awesome editor, and I've accumulated some know-how to make it even better (at least for me and people who think like me). Sharing is caring, so here we go.

ace-window update

On this weekend I've made a major update to my package ace-window that allows it to be used as a library. Luckily the change went smoothly, as there are no complaints in the github issues so far. In case you don't know what the package does in the first place, a short blurb follows.

ace-window's "what and why"

I'm sure you're aware of the other-window command. While it's great for two windows, it quickly loses its value when there are more windows: you need to call it many times, and since it's not easily predictable, you have to check each time if you're in the window that you wanted.

Another approach is to use windmove-left, windmove-up etc. These are fast and predictable. Their disadvantage is that they need 4 key bindings. The default ones are shift+arrows, which are hard to reach.

This package aims to take the speed and predictability of windmove and pack it into a single key binding, similar to other-window. To achieve this, I'm using the excellent ace-jump-mode.

Here's how the package looks in action: ace-window.gif

Since switching windows is a frequently used operation, I recommend binding ace-window to something short, like M-p.

By default, three actions are available:

• M-p - select window
• C-u M-p - swap the current window with the selected window
• C-u C-u M-p - delete the selected window

finally, the library part

So now, what if you want to select a window to maximize with ace-window? After the change that I've mentioned, the code to do this is dirt simple:

(defun ace-maximize-window ()
  "Ace maximize window."
  (interactive)
  (setq aw--current-op
        (lambda (aj)
          (let ((wnd (aj-position-window aj)))
            (select-window wnd)
            (delete-other-windows))))
  (aw--doit " Ace - Maximize Window"))

(global-set-key (kbd "C-M-o") 'ace-maximize-window)