More productive describe-variable

I refer to counsel-describe-variable here, which you can get from the MELPA counsel package, or from Github. If you're not familiar, counsel contains specialized functions using ivy-mode completion. Being specialized means that you can do more than usual, for instance here are the exit points of counsel-describe-variable:

• RET calls describe-variable: a plain describe-variable with ivy-mode on looks like counsel-describe-variable, but has only this exit point.
• C-. calls counsel-find-symbol: jump to symbol declaration.
• C-, calls counsel--info-lookup-symbol: jump to symbol reference in *Info*.

These can also be extended by binding new commands in counsel-describe-map.

I'll just describe one useful scenario that happened just now. I was working on a checkdoc.el change and was testing it with avy.el. And I got a few style warnings for this code, which I got from a contributor (thanks for the code btw):

(defvar avy-translate-char-function #'identity
  "Function to translate user input key. This can be useful for
adding mirror key. E.g. one can make SPACE an alternative of 'a',
by adding:

\(setq avy-translate-char-function
      (lambda (c) (if (= c 32) ?a c)))

to allow typing SPACE instead of character 'a' to jump to the location
highlighted by 'a'.")

One of the warnings that avy-translate-char-function should be quoted while inside the docstring like so:

"`avy-translate-char-function'"

Of course that wouldn't play well with setq, so I wanted to see if any other variables that end in -function provide an example of use like here, and, if so, how they quote.

This is my personal configuration:

(global-set-key (kbd "<f1> v") 'counsel-describe-variable)

So I did <f1> v function$ to describe a variable that ends with function. Here, $ is a part of a regular expression, which should be a priority to learn if you want to be a (not just) Emacs Power User.

And now the key part: instead of pressing RET to describe just the current candidate, I repeatedly press C-M-n (ivy-next-line-and-call). After each C-M-n, the next candidate will be selected and the *Help* window will be updated.

With this trick I was able to skim through the 346 matches in seconds. The conclusion was that an example isn't usually provided, with nnmail-expiry-wait-function being an exemption. In the end, I decided not to provide such an explicit example and shorten the doc to this:

(defvar avy-translate-char-function #'identity
  "Function to translate user input key into another key.
For example, to make SPC do the same as ?a, use
\(lambda (c) (if (= c 32) ?a c)).")

counsel-git-grep, in async

Introduction

It took me quite a while to figure out, but finally counsel-git-grep, which I've described before works completely asynchronously, which means it's very fast and smooth even for huge repositories.

If you don't know what git grep is: it allows you to search for a regular expression in your Git repository. In a way, it's just a find / grep combo, but very concise and fast. It's particularly great for finding all uses and references to a symbol.

It can be useful even for non-programmers. One great application is to stick all your org-mode files into a Git repository. Then you can search all your files at once very fast: all my org stuff takes more than 1000,000 lines and there's no lag while searching. I really should remove some pdfs from the repository at some point.

The Video Demo

Check out the speed and the various command in this Video Demo. Each key stroke starts two asynchronous shell calls, while canceling the current ones if they are still running. Here are the example calls:

$ git --no-pager grep --full-name -n --no-color \
    -i -e "forward-line 1" | head -n 200
$ git grep -i -c 'forward-line 1' \
    | sed 's/.*:\\(.*\\)/\\1/g' \
    | awk '{s+=$1} END {print s}'

The Elisp side

I figure since I was interested in this topic, probably a few more people are as well. So I'll explain below how async processing is done (in this case):

(defun counsel--gg-count (regex &optional no-async)
  "Quickly count the amount of git grep REGEX matches.
When NO-ASYNC is non-nil, do it synchronously."
  (let ((default-directory counsel--git-grep-dir)
        (cmd (concat (format "git grep -i -c '%s'"
                             regex)
                     " | sed 's/.*:\\(.*\\)/\\1/g'"
                     " | awk '{s+=$1} END {print s}'"))
        (counsel-ggc-process " *counsel-gg-count*"))
    (if no-async
        (string-to-number (shell-command-to-string cmd))
      (let ((proc (get-process counsel-ggc-process))
            (buff (get-buffer counsel-ggc-process)))
        (when proc
          (delete-process proc))
        (when buff
          (kill-buffer buff))
        (setq proc (start-process-shell-command
                    counsel-ggc-process
                    counsel-ggc-process
                    cmd))
        (set-process-sentinel
         proc
         #'(lambda (process event)
             (when (string= event "finished\n")
               (with-current-buffer
                   (process-buffer process)
                 (setq ivy--full-length
                       (string-to-number
                        (buffer-string))))
               (ivy--insert-minibuffer
                (ivy--format ivy--all-candidates)))))))))

1. Since we're calling a shell command, it's important to set default-directory to a proper value: the shell command will be run there.
2. The output from the process will be channeled into a buffer. I start the buffer name with a space to make it hidden: it will not be shown in the buffer list and switch-to-buffer completion.
3. I leave the sync option and use shell-command-to-string for that. Sometimes it's necessary to know the amount of candidates without a delay.
4. If get-process and get-buffer return something it means that the shell command is still running from the previous input. But the input has changed and that data has become useless, so I kill both the process and the buffer.
5. Then I start a new process with start-process-shell-command- a convenience wrapper around the basic start-process that's useful for passing a full shell command with arguments.
6. Finally, I let counsel--gg-count return without waiting for git grep to finish. But something needs to be done when it finishes, so I use set-process-sentinel. Emacs will call my lambda when there's a new event from the process.

Outro

To check out the new stuff, just install or upgrade counsel from MELPA. Or clone the source. I hope that my Elisp explanations were useful, async stuff is awesome, and I hope to see more of it in future Emacs packages.

lispy 0.26.0 is out

Lispy 0.25.0 came out 2 months ago; 177 commits later, comes version 0.26.0. The release notes are stored at Github, and I'll post them here as well.

The coolest changes are the new reader-based M, which:

• Gives out very pretty output, with minor diffs for actual code, which is quite impressive considering all newline information is discarded and then reconstructed.
• Works for things that Elisp can't read, like #<marker ...> etc, very useful for debugging.
• Customizable rule sets; rules for Elisp and Clojure come with the package.

The improvements to g and G also great:

• Because of caching, the prettified tags can be displayed in less than 0.15s on Emacs' lisp/ directory, which has 21256 tags in 252 files.
• The tags collector looks at file modification time, so you get the updated tags right after you save.

The details for these and other features follow below.

Fixes

• C-k should delete the whole multi-line string.
• y should work for all parens, not just (.
• p should actually eval in other window for dolist.
• Prevent pairs inserting an extra space when at minibuffer start.
• ol works properly for active region.

New Features

Misc

• xf will pretty-print the macros for Elisp.
• M-m works better when before ).
• Fix ', ^ after a ,.
• Improve / (splice) for quoted regions.
• Z works with &key arguments.
• The new M is used in xf.
• Allow to flatten Elisp defsubst.
• c should insert an extra newline for top-level sexps.

Paredit key bindings

You can have only Paredit + special key bindings by using this composition of key themes:

(lispy-set-key-theme '(special paredit))

The default setting is:

(lispy-set-key-theme '(special lispy c-digits))

New algorithm for multi-lining

M is now bound to lispy-alt-multiline instead of lispy-multiline. It has a much better and more customizable algorithm.

See these variables for customization:

• lispy-multiline-threshold
• lispy--multiline-take-3
• lispy--multiline-take-3-arg
• lispy--multiline-take-2
• lispy--multiline-take-2-arg

They are set to reasonable defaults. But you can customize them if you feel that a particular form should be multi-lined in a different way.

lispy-multiline-threshold is a bit of ad-hoc to make things nice. Set this to nil if you want a completely rigorous multi-line. With the default setting of 32, expressions shorter than this won't be multi-lined. This makes 95% of the code look really good.

The algorithm has a safety check implemented for Elisp: if read on the transformed expression returns something different than read on the original expression, an error will be signaled and no change will be made. For expressions that can't be read, like buffers/markers/windows/cyclic lists/overlays, only a warning will be issued (lispy can read them, unlike read).

d and > give priority to lispy-right

For the expression (a)|(b), (a) will be considered the sexp at point, instead of (b). This is consistent with show-paren-mode. If a space is present, all ambiguities are resolved anyway.

b works fine even if the buffer changes

I've switched the point and mark history to markers instead of points. When the buffer is changed, the markers are updated, so b will work fine.

Extend Clojure reader

In order for i (prettify code) to work for Clojure, it must be able to read the current expression. I've been extending the Elisp reader to understand Clojure. In the past commits, support was added for:

• empty sets
• commas
• auto-symbols, like p1__7041#

Extend Elisp reader

It should be possible to read any #<...> form, as well as #1-type forms.

g and G get a persistent action for ivy

This is a powerful feature that the helm back end has had for a long time. When you press g, C-n and C-p will change the current selection. But C-M-n and C-M-p will change the current selection and move there, without exiting the completion.

This also means that you can call ivy-resume to resume either g (lispy-goto) or G (lispy-goto-local).

e works with defvar-local

As you might know, the regular C-x C-e or eval-buffer will not reset the values of defvar, defcustom and such (you need C-M-x instead). But e does it, now also for defvar-local.

Improve faces for dark backgrounds

I normally use a light background, so I didn't notice before that the faces looked horrible with a dark background.

The ` will quote the region

If you have a region selected, pressing ` will result in:

`symbol'

Customize the file selection back end for V

V (lispy-visit) allows to open a file in current project. Previously, it used projectile. Now it uses find-file-in-project by default, with the option to customize to projectile.

Fixup calls to looking-back

Apparently, looking-back isn't very efficient, so it's preferable to avoid it or at least add a search bound to improve efficiency. Also the bound became mandatory in 25, while it was optional before.

M-m will work better in strings and comments.

See the relevant test:

(should (string= (lispy-with "\"See `plu|mage'.\"" (kbd "M-m"))
                 "\"See ~`plumage'|.\""))

Thanks to this, to e.g. get the value of a quoted var in a docstring or a comment, or jump to its definition, you can M-m. Then, you can step-in with i to select the symbol without quotes.

Update the tags strategy

A much better algorithm with caching an examining of file modification time is used now. This means that the tags should be up-to-date 99% of the time, even immediately after a save, and no necessary re-parsing will be done. And it all works fine with the lispy-tag-arity modifications.

1% of the time, lispy-tag-arity stops working, I don't know why, since it's hard to reproduce. You can then pass a prefix arg to refresh tags bypassing the cache, e.g 2g or 2G.

Also a bug is fixed in Clojure tag navigation, where the tag start positions were off by one char.

The fetched tags retrieval is fast: less than 0.15s on Emacs' lisp/ directory to retrieve 21256 tags from 252 files. Which means it's lightning fast on smaller code bases (lispy has only 651 tags).

xj can also step into macros

lispy-debug-step-in, bound to xj locally and C-x C-j globally can now step into macros, as well as into functions. This command is very useful for Edebug-less debugging. Stepping into macros with &rest parameters should work fine as well.

p can now lax-eval function and macro arguments

When positioned at function or macro args, p will set them as if the function or macro was called with empty args, or the appropriate amount of nils. If the function is interned and interactive, use its interactive form to set the arguments appropriately.

Again, this is very useful for debugging.

Allow to paste anywhere in the list using a numeric arg

As you might know, P (lispy-paste) is a powerful command that:

• Replaces selection with current kill when the region is active.
• Yanks the current kill before or after the current list otherwise.

Now, you can:

• Yank the current kill to become the second element of the list with 2P
• Yank the current kill to become the third element of the list with 3P
• ...

It's OK to pass a larger arg than the length of the current list. In that case, the paste will be made into the last element of the list.

Update the way / (lispy-splice) works

When there's no next element within parent, jump to parent from appropriate side. When the region is active, don't deactivate it. When splicing region, remove random quotes at region bounds.

This change makes the splice a lot more manageable. For example, starting with this Clojure code, with | marking the current point:

(defn read-resource
  "Read a resource into a string"
  [path]
  (read-string
   |(slurp (clojure.java.io/resource path))))

A double splice // will result in:

(defn read-resource
  "Read a resource into a string"
  [path]
  |(read-string
   slurp clojure.java.io/resource path))

After xR (reverse list), 2 SPC (same as C-f), -> (plain insert), [M (back to parent and multi-line), the final result:

(defn read-resource
  "Read a resource into a string"
  [path]
  |(-> path
      clojure.java.io/resource
      slurp
      read-string))

This also shows off xR - lispy-reverse, which reverses the current list. Finally, reverting from the last code to the initial one can be done simply with xf - it will flatten the -> macro call.

Outro

Thanks to all who contributed, enjoy the new stuff. Would also be nice to get some more feedback and bug reports. Currently, it might seem that a large part of the features are either perfect or unused.

Ivy-mode 0.5.0 is out

At this point, swiper is only a fraction of ivy-mode's functionality. Still, it's nice to keep them all, together with counsel, in a single repository: counsel-git-grep works much better this way.

Anyway, I'll echo the release notes here, there are quite a few exciting new features.

Fixes

• TAB shouldn't delete input when there's no candidate.
• TAB should switch directories properly.
• require dired when completing file names, so that the directory face is loaded.
• TAB should work with confirm-nonexistent-file-or-buffer.
• TAB should handle empty input.
• work around grep-read-files: it should be possible to simply M-x rgrep RET RET RET.
• Fix the transition from a bad regex to a good one - you can input a bad regex to get 0 candidates, the candidates come back once the regex is fixed.
• ivy-switch-buffer should pre-select other-buffer just like switch-buffer does it.
• Fix selecting "C:\" on Windows.
• counsel-git-grep should warn if not in a repository.
• C-M-n shouldn't try to call action if there isn't one.
• Turn on sorting for counsel-info-lookup-symbol.
• ivy-read should check for an outdated cons initial-input.

New Features

Out of order matching

I actually like in-order matching, meaning the input "in ma" will match "in-order matching", but not "made in". But the users can switch to out-of-order matching if they use this code:

(setq ivy-re-builders-alist
          '((t . ivy--regex-ignore-order)))

ivy-re-builders-alist is the flexible way to customize the regex builders per-collection. Using t here, means to use this regex builder for everything. You could choose to have in-order for files, and out-of-order for buffers and so on.

New defcustom: ivy-tab-space

Use this to have a space inserted each time you press TAB:

(setq ivy-tab-space t)

ignore case for TAB

"pub" can expand to "Public License".

New command: counsel-load-library

This command is much better than the standard load-libary that it upgrades. It applies a sort of uniquify effect to all your libraries, which is very useful:

In this case, I have avy installed both from the package manager and manually. I can easily distinguish them.

Another cool feature is that instead of using find-library (which is also bad, since it would report two versions of avy with the same name and no way to distinguish them), you can simply use counsel-load-library and type C-. instead of RET to finalize.

Here's another scenario: first load the library, then call ivy-resume and immediately open the library file.

New command: ivy-partial

Does a partial complete without exiting. Use this code to replace ivy-partial-or-done with this command:

(define-key ivy-minibuffer-map (kbd "TAB") 'ivy-partial)

Allow to use ^ in swiper

In regex terms, ^ is the beginning of line. You can now use this in swiper to filter your matches.

New command: swiper-avy

This command is crazy good: it combines the best features of swiper (all buffer an once, flexible input length) and avy (quickly select one candidate once you've narrowed to about 10-20 candidates).

For instance, I can enter "to" into swiper to get around 10 matches. Instead of using C-n a bunch of times to select the one of 10 that I want, I just press C-', followed by a or s or d ... to select one of the matches visible on screen.

So both packages use their best feature to cover up the others worst drawback.

Add support for virtual buffers

I was never a fan of recentf until now. The virtual buffers feature works in the same way as ido-use-virtual-buffers: when you call ivy-switch-buffer, your recently visited files as well as all your bookmarks are appended to the end of the buffer list.

Suppose you killed a buffer and want to bring it back: now you do it as if you didn't kill the buffer and instead buried it. The bookmarks access is also nice.

Here's how to configure it, along with some customization of recentf:

(setq ivy-use-virtual-buffers t)

(use-package recentf
  :config
  (setq recentf-exclude
        '("COMMIT_MSG" "COMMIT_EDITMSG" "github.*txt$"
          ".*png$"))
  (setq recentf-max-saved-items 60))

Add a few wrapper commands for the minibuffer

All these commands just forward to their built-in counterparts, only trying not to exit the first line of the minibuffer.

• M-DEL calls ivy-backward-kill-word
• C-d calls ivy-delete-char
• M-d calls ivy-kill-word
• C-f calls ivy-forward-char

Allow to customize the minibuffer formatter

See the wiki on how to customize the minibuffer display to look like this:

100 Find file: ~/
  file1
  file2
> file3
  file4

When completing file names, TAB should defer to minibuffer-complete

Thanks to this, you can TAB-complete your ssh hosts, e.g.:

• /ss TAB -> /ssh
• /ssh:ol TAB -> /ssh:oleh@

More commands work with ivy-resume

I've added:

• counsel-git-grep
• counsel-git

Others (that start with counsel-) should work fine as well. Also don't forget that you can use C-M-n and C-M-p to:

• switch candidate
• call the action for the candidate
• stay in the minibuffer

This is especially powerful for counsel-git-grep: you can easily check the whole repository for something with just typing in the query and holding C-M-n. The matches will be highlighted swiper-style, of course.

Allow to recenter during counsel-git-grep

Use C-l to recenter.

Update the quoting of spaces

Split only on single spaces, from all other space groups, remove one space.

As you might know, a space is used in place of .* in ivy. In case you want an actual space, you can now quote them even easier.

Outro

Thanks to all who contributed, check out the new stuff, and make sure to bind ivy-resume to something short: it has become a really nice feature.

New on MELPA - define word at point

Doing things in Emacs is superlatively better than having to switch to another application.

In this case, "doing things" is getting the dictionary definition of word at point, and "superlatively" is a word that I didn't know - a straw that broke the camel's back and caused me to finally automate the process of getting the definition of a word that I encounter in an Emacs buffer.

The whole process of writing the define-word package took around 30 minutes, I just had to:

• See which engine DuckDuckGo uses.
• Follow to wordnik.
• Try to get an API key, read their draconian TOS and decide that I don't want to agree to it just to get their key.
• Examine the HTML that it returns and note that it's quite regular.
• Write a 10 line function with re-search-forward to extract the word definitions from a sample page that I saved with wget.

Then just wrap the function in an url-retrieve and done. It's a good thing that I learned to use url-retrieve when I wrote org-download.

Here's how it looks like in action, the word under point is "Authors" and instead of visiting this page, you can see it right away in your Echo Area:

The result is displayed simply with message, so it doesn't mess with your window config. You read it, press any key and the Echo Area popup will vanish automatically.

Install the package from MELPA or check it out at github. You just need to decide where to bind it:

(global-set-key (kbd "C-c d") 'define-word-at-point)
(global-set-key (kbd "C-c D") 'define-word)

At less than 50 lines, the source is very easy to understand. So if you're looking to write some Elisp that retrieves and parses some HTML from a web service, it's nice to look at a simple implementation of how it's done.