(or emacs irrelevant)

Quickly ediff files from dired

ediff.el --- a comprehensive visual interface to diff & patch

I wrote about ediff years ago. Today, I'll just reference a useful ediff snippet from my config that I've added some time ago and refined only recently.

The premise is quite simple: press e in dired-mode to immediately ediff two marked files, no questions asked:

(define-key dired-mode-map "e" 'ora-ediff-files)

And here's the code, with a few bells and whistles:

;; -*- lexical-binding: t -*-
(defun ora-ediff-files ()
  (interactive)
  (let ((files (dired-get-marked-files))
        (wnd (current-window-configuration)))
    (if (<= (length files) 2)
        (let ((file1 (car files))
              (file2 (if (cdr files)
                         (cadr files)
                       (read-file-name
                        "file: "
                        (dired-dwim-target-directory)))))
          (if (file-newer-than-file-p file1 file2)
              (ediff-files file2 file1)
            (ediff-files file1 file2))
          (add-hook 'ediff-after-quit-hook-internal
                    (lambda ()
                      (setq ediff-after-quit-hook-internal nil)
                      (set-window-configuration wnd))))
      (error "no more than 2 files should be marked"))))

Some notes on how the extra code adds convenience:

  1. In case no files are marked, the file at point is used as the first file, and read-file-name is used for the second file. Since I have the magic (setq dired-dwim-target t) in my config, in case a second dired buffer is open, dired-dwim-target-directory will offer it as the starting directory during completion. Very useful to compare two files in two different directories.

  2. Depending on the order of the arguments to ediff-files, the changes will appear either as added or removed; file-newer-than-file-p tries to put the arguments in a logical order by looking at the files' last change times.

  3. ediff-after-quit-hook-internal is used to restore the previous window configuration after I quit ediff with q.

That's about it. Hopefully, it's useful. Happy hacking.

Make it so: file1 -> Makefile -> file2

Intro

make-it-so is an old package of mine that I haven't yet highlighted on the blog. This package helps you manage a collection of makefiles that are used to generate new files from existing files using shell commands.

You can think of these makefiles as a directory of shell functions, arranged by the extension of the files that they operate on:

$ cd make-it-so && find recipes -name Makefile
recipes/ipynb/to-md/Makefile
recipes/ogv/crop/Makefile
recipes/ogv/trim/Makefile
recipes/ogv/to-gif/Makefile
recipes/pdf/to-txt/Makefile
recipes/md/to-org/Makefile
recipes/md/to-html/Makefile
recipes/cue/split/Makefile
recipes/dot/to-png/Makefile
recipes/m4a/to-mp3/Makefile
recipes/flac/to-mp3/Makefile
recipes/gif/gifsicle/Makefile
recipes/svg/to-png/Makefile
recipes/chm/to-pdf/Makefile
recipes/txt/encode-utf8/Makefile
recipes/mp4/to-mp3/Makefile
recipes/mp4/trim/Makefile
recipes/mp4/replace-audio/Makefile
recipes/png/to-gif/Makefile

When you call make-it-so on a particular file, you get completion for the recipes that are available for that file extension, along with an option to create a new recipe.

Example 1: convert pdf to txt

Suppose you want to convert a PDF file test.pdf to a text file test.txt.

In case the recipe is in your collection, you don't have to remember the command or the command switches to do it anymore:

  1. Navigate to test.pdf in dired and press , (bound to make-it-so).
  2. Select the recipe you want using completion: to-txt is already provided.
  3. Your file and the makefile recipe are moved to the staging area:

    ./to-txt_test.pdf/test.pdf
    ./to-txt_test.pdf/Makefile
    
  4. The makefile is opened in a new buffer with the following bindings:

    • f5 (mis-save-and-compile) will run compile, creating test.txt in the current directory.
    • C-, (mis-finalize) will finalize the operation, moving test.pdf and test.txt to the parent directory (where test.pdf was before), and deleting the staging directory.
    • C-M-, (mis-abort) will move test.pdf back to its initial location and delete all generated files. This command is effectively an undo for make-it-so.

It takes a large chunk of text to describe everything, but the key sequence for doing all this is quite short:

  1. , - make-it-so.
  2. RET - select to-txt.
  3. f5 - create test.txt.
  4. C-, - finalize.

Example 2: make a gif from a series of png images

I'll describe the process of creating a high quality gif like this one, which describes the effect of the C key in lispy:

lispy-convolute

First, I use kazam to take two png screenshots of my Emacs screen:

$ ls -1 *.png
Screenshot 2017-02-25 16:14:49.png
Screenshot 2017-02-25 16:15:10.png

I plan to use gifsicle to sequence the still images into a gif. But it only takes gif as the input format, so first I have to convert my png files to non-animated gif files.

I open the dired buffer where they are located and mark them with m (dired-mark). Then call make-it-so with , and select to-gif recipe. This recipe has no parameters, so there's nothing else to do but f5 C-,. Two new files are created:

$ ls -1 *.png *.gif
Screenshot_2017-02-25 16:14:49.gif
Screenshot_2017-02-25 16:14:49.png
Screenshot_2017-02-25 16:15:10.gif
Screenshot_2017-02-25 16:15:10.png

Note that the file names (the defaults of kazam) are problematic when used with makefiles, since they contain spaces and colons. The Elisp layer of make-it-so takes care of that. It renames the files back and forth so that the logic in the makefiles remains simple.

Next, I mark the two gif files using *% (dired-mark-files-regexp), press , once more and select the gifsicle recipe. I'm presented a makefile with the following contents:

# ——— parameters —————————————————————————————————

# delay between frames in hundredths of a second
delay = 60

# ——— implementation —————————————————————————————
DIRGIF = $(shell ls *.gif | grep -v anime.gif)

all: anime.gif

anime.gif: Makefile $(DIRGIF)
    rm -f anime.gif
    gifsicle --delay=$(delay) --colors=256 --loop $(DIRGIF) > $@
    echo $@ >> provide

clean:
    rm -f anime.gif provide

install-tools:
    sudo apt-get install gifsicle

.PHONY: all install-tools clean

The most commonly useful parameter, the delay between frames, is nicely documented at the top. I don't have to remember that the switch name is --delay or that the switch style --delay=60 is used. I simply change the number above until I get the result that I want.

Example 3: add a new recipe

As a sample scenario, assume you want to convert *.svg to *.png.

Step 1

An internet search leads to Stack Overflow and this command:

inkscape -z -e test.png -w 1024 -h 1024 test.svg

Navigate to the file(s) in dired and call make-it-so with ,. No default actions are available, so just type "to-png" and hit RET. The "to-" prefix signifies that this is a conversion, adapting the Makefile to this form:

# This is a template for the Makefile.
# Parameters should go in the upper half as:
#     width = 200
# and be referenced in the command as $(width)

# ____________________________________________

DIRSVG = $(shell dir *.svg)

DIRPNG = $(DIRSVG:.svg=.png)

all: clean Makefile $(DIRPNG)

%.png: %.svg
    echo "add command here"
    echo $@ >> provide

clean:
    rm -f *.png provide

# Insert the install command here.
# e.g. sudo apt-get install ffmpeg
install-tools:
    echo "No tools required"

.PHONY: all install-tools clean

If the action name doesn't have a "to-" prefix, the transformation is assumed to be e.g. "svg" -> "out.svg". You can change this of course by editing the Makefile.

Step 2

In case the command needs additional packages in order to work you might want to change echo "No tools required" to the appropriate package install instruction, e.g. sudo apt-get install inkscape.

When you're on a new system, this will serve as a reminder of what you should install in order for the Makefile to work. Simply call:

make install-tools

Step 3

Replace echo "add command here" with:

    inkscape -z -e $@ -w $(width) -h $(height) $^
  • The parameters width and height will go to the top of the Makefile, where they can be customized.

  • $@ refers to the output file, test.png in this case.

  • $^ refers to the input file, test.svg in this case.

That's it. You can see the final Makefile here. Test if the command works with f5 from the Makefile. If you're happy with it, call mis-finalize with C-, from dired. The Makefile will be saved for all future calls to make-it-so.

Outro

To summarize the advantages of make-it-so:

  • Write the recipe one time, never have to look up how to do the same thing a few months from now.
  • A chance to write the recipe zero times, if someone in the community has already done it and shared the recipe with you.
  • The Elisp layer takes care of hairy file names.
  • Parallel commands on multiple files, i.e. make -j8, are provided for free.

The most important usage tip: until you're sure that the command and the Makefile work properly make backups. In fact, make backups period. Happy hacking!

elf-mode - view the symbol list in a binary

Recently, I've been looking at libigl. I didn't manage to fully figure out their CMake build system for tutorials: although each tutorial has a CMakeLists.txt, it's only possible to build them all at once.

So I decided to replace CMakeLists.txt with a good-old Makefile; how hard can it be? Concerning includes, not at all hard: the missing files are found with counsel-locate and added to the include path.

But I had some trouble matching a missing ld dependency to a library file. Fixed it with a bunch of googling and guesswork; I still wonder if there's a better way. But in the process, I've found this useful command:

readelf --syms libGL.so

which produces e.g.:


Symbol table '.dynsym' contains 2732 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 000000000004faf0     0 SECTION LOCAL  DEFAULT    8
     2: 00000000000e8f20     0 FUNC    GLOBAL DEFAULT   11 glGetIntegerui64i_vNV
     3: 00000000000e13e0     0 FUNC    GLOBAL DEFAULT   11 glGetMultiTexEnvfvEXT
     4: 00000000000d7440     0 FUNC    GLOBAL DEFAULT   11 glProgramUniform2uiv
     5: 00000000000cfdc0     0 FUNC    GLOBAL DEFAULT   11 glMultiTexCoord3sv

This is a pretty good representation of a binary file: in this example, instead of one megabyte of gibberish I see a bit more than 2732 lines describing the functions this file uses and provides.

Viewing the symbol list automatically

I liked the above representation so much that I want to see it by default. In Emacs, it's pretty easy to do with auto-mode-alist:

(add-to-list 'auto-mode-alist '("\\.\\(?:a\\|so\\)\\'" . elf-mode))

The above code instructs Emacs to call elf-mode function whenever the file name ends in *.a or *.so.

And here's the body of elf-mode:

(defvar-local elf-mode nil)

;;;###autoload
(defun elf-mode ()
  (interactive)
  (let ((inhibit-read-only t))
    (if elf-mode
        (progn
          (delete-region (point-min) (point-max))
          (insert-file-contents (buffer-file-name))
          (setq elf-mode nil))
      (setq elf-mode t)
      (delete-region (point-min) (point-max))
      (insert (shell-command-to-string
               (format "readelf --syms %s" (buffer-file-name)))))
    (set-buffer-modified-p nil)
    (read-only-mode 1)))

The idea is very simple: elf-mode is a toggle function that replaces the buffer contents with the shell command output. It carefully uses read-only-mode and set-buffer-modified-p so that the file will not be overwritten by accident with the symbol names.

Using autoload to avoid overhead

As you might imagine, looking at binaries isn't really a common task. Is it worth to be dragging this code around from now on, loading it on each start? The answer is yes, of course. Since the actual cost is negligible until the feature is used.

If you look above, elf-mode has an ;;;###autoload cookie before it. The cookie results in this line in my loaddefs.el:

(autoload 'elf-mode "modes/ora-elf" "" t nil)

My init.el always loads loaddefs.el, but never loads ora-elf.el where the function is defined. That file is only loaded when the function elf-mode is called for the first time. The above autoload statement simply instructs Emacs to load a particular file when elf-mode needs to be called.

When you use the package manager, the autoloads file is generated and loaded for you automatically:

$ tree elpa/ace-link-20160811.112/

elpa/ace-link-20160811.112/
├── ace-link-autoloads.el
├── ace-link.el
├── ace-link.elc
└── ace-link-pkg.el

0 directories, 4 files

Here, the package manager will always load ace-link-autoloads.el, which instructs Emacs to load ace-link.el when one of the ;;;###autoload functions is called and ace-link.el isn't yet loaded.

As an example of how useful delayed loading is: my 6000 line config starts in 1.8 seconds. About 40% of that time is spent on (package-initialize), which I assume is the package manager loading all those *-autoloads.el files that I have in my elpa/.

Outro

Let me know if there's interest to have elf-mode on MELPA. Also, if anyone knows how to set mode automatically based on the first few chars of the file (all binaries seem to start with ^?ELF), I'd like to know that as well. Happy hacking!

Swipe all the files!

If you've ever tried the swiper-all from swiper, forget everything about it. The command was super-awkward, since it had to parse all your open files before giving you a chance enter anything, resulting in dozens of seconds before the prompt.

Recently, I've had some time to examine and improve it and the result looks very promising. The new command is now async, which means there's no delay before the prompt comes up. Here's a result I got with no delay while having around 50 buffers open:

swiper-all.png

The shortcut I'm using:

(global-set-key (kbd "C-c u") 'swiper-all)

For efficiency's sake a small trade off had to be made: the line numbers are no longer displayed. This actually results in an advantage that you can select different candidates on the same line.

There are still a few things I plan to try for the new command, like adding file-less buffers, caching for incremental regexes and maybe even newlines in wild cards, but even now it seems very usable. So give it a try, enjoy and happy hacking!

Bookmark the current window layout with Ivy

Today's post is about the newest feature related to the ivy-switch-buffer command. If you use ivy-mode, you're probably already using ivy-switch-buffer since it overwrites the built-in switch-to-buffer.

The cool thing about ivy-switch-buffer is that it's not only buffers that are offered for completion. Other buffer-like entities can be there as well: bookmarks, recently opened files (recentf), and finally, window layouts. Since all of those are relatively the same concept, it's very convenient to have them all in one place available for completion.

Here are the relevant settings:

;; Enable bookmarks and recentf
(setq ivy-use-virtual-buffers t)

;; Example setting for ivy-views
(setq ivy-views
      `(("dutch + notes {}"
         (vert
          (file "dutch.org")
          (buffer "notes")))
        ("ivy.el {}"
         (horz
          (file ,(find-library-name "ivy"))
          (buffer "*scratch*")))))

I did mention ivy-views before in the ivy-0.8.0 release post. But now, instead of setting ivy-views by hand, you can also bind ivy-push-view to a key and store as many window configurations as you like, really fast.

What gets stored:

  • The window list - all windows open on the current frame.
  • The window splits relative to each other as a tree. Currently, the size of the split isn't saved, all windows are split equally.
  • The point positions in each window. If you use just one window, you've got something similar to bookmark-set.

Here's what I use currently:

(global-set-key (kbd "C-c v") 'ivy-push-view)
(global-set-key (kbd "C-c V") 'ivy-pop-view)

Typical workflow

Suppose I have two files open: the file 2016-06-23-ivy-push-view.md and the _posts directory. By pressing C-c v I am prompted for a view name with the default being e.g. {} 2016-06-23-ivy-push-view.md _posts 2.

I can still name the view however I want, but I typically just press RET. The parts of the automatic view name are:

  • {} - this is a simple string marker to distinguish the views in the buffer view. If I enter only {} into ivy-switch-buffer prompt, the candidates will normally filter to only views, since very rarely will a file or a buffer name match {}.
  • 2016-06-23-ivy-push-view.md _posts is the list of buffers stored in the view. This view has only two buffers, but ivy-push-view can handle as many windows as you can cram into a single frame.
  • 2 means that I already have two views with the same buffers, each new view with the same buffers gets an increased number for the suggested name. And it's not useless to have many views for the same buffers, since the views also store point positions, not just the window list.

Here's the beauty of it for me: when I type _posts into ivy-switch-buffer I can chose to open the _posts directory in a variety of ways:

  • If the buffer is currently open, I can just switch there.
  • If the buffer is currently closed, I can re-open it, thanks to recentf.
  • I can open the buffer as part of a stored view(s) in ivy-views.

Finally, if I decide that I don't need a particular view any more, I can delete it with C-c V (ivy-pop-view). It's possible to delete many views at once by pressing C-M-m (ivy-call), as usual with most ivy completion functions.

Breaking API change

While implementing ivy-set-view I decided that the current way alist collections are handled together with actions is sub-optimal. Here's the new way of working:

(let (res)
  (ivy-with
   '(ivy-read "test: "
     '(("one" . 1) ("three" . 3))
     :action (lambda (x) (setq res x)))
   "t C-m")
  res)
;; =>
;; ("three" . 3)

Previously, the return result would be 3, i.e. the cdr of the selected candidate. Any code using ivy-read with an alist-type collection will break. I fixed all instances in counsel.el, and there actually aren't too many uses in the published third party packages.

A simple fix to the problem is to use cdr in the action function. Additionally, having more information available in the action function will serve to improve the code.