Complete Python symbols using Ivy

I had a fascination with Python at one point, until I got too annoyed with the indentation rules. I still have a few scripts left over, so I'm sympathetic to Python support in Emacs. The reason for today's post comes from this question on Emacs StackExchange. Essentially, the user wants to insert parens automatically when a function name is completed.

The completion candidates come from jedi plugin. To quickly see where the list of strings is coming from, I've examined ac-sources variable and saw that it contains ac-source-jedi-direct, which evaluates to:

((candidates . jedi:ac-direct-matches)
 (prefix . jedi:ac-direct-prefix)
 (init . jedi:complete-request)
 (requires . -1))

This means that jedi:complete-request has to be called at point, followed by jedi:ac-direct-matches to obtain the list of strings. So basically, this code:

(progn
  (deferred:sync!
   (jedi:complete-request))
  (jedi:ac-direct-matches))

Note the call to deferred:sync!. I had to add that one to make sure that jedi:complete-request completes before jedi:ac-direct-matches is called.

Testing with some Python code, where | is the point:

params = {"foo":"bar"}
params.i

The Elisp code above will return:

(#("items"
   0 5 (summary
        "function: __builtin__.dict.items"
        symbol
        "f"
        document
        "items(self)

D.items() -> list of D's (key, value) pairs, as 2-tuples"))
  #("iteritems"
    0 9 (summary
         "function: __builtin__.dict.iteritems"
         symbol
         "f"
         document
         "iteritems(self)

D.iteritems() -> an iterator over the (key, value) items of D"))
  #("iterkeys"
    0 8 (summary
         "function: __builtin__.dict.iterkeys"
         symbol
         "f"
         document
         "iterkeys(self)

D.iterkeys() -> an iterator over the keys of D"))
  #("itervalues"
    0 10 (summary
          "function: __builtin__.dict.itervalues"
          symbol
          "f"
          document
          "itervalues(self)

D.itervalues() -> an iterator over the values of D")))

So these strings come with the symbol documentation and symbol type encoded as string properties. After this, the rest of the Elisp code follows very easily, you can find it as part of counsel package on MELPA:

(defun counsel-jedi ()
  "Python completion at point."
  (interactive)
  (let ((bnd (bounds-of-thing-at-point 'symbol)))
    (if bnd
        (progn
          (setq counsel-completion-beg (car bnd))
          (setq counsel-completion-end (cdr bnd)))
      (setq counsel-completion-beg nil)
      (setq counsel-completion-end nil)))
  (deferred:sync!
   (jedi:complete-request))
  (ivy-read "Symbol name: " (jedi:ac-direct-matches)
            :action #'counsel--py-action))

(defun counsel--py-action (symbol)
  "Insert SYMBOL, erasing the previous one."
  (when (stringp symbol)
    (with-ivy-window
      (when counsel-completion-beg
        (delete-region
         counsel-completion-beg
         counsel-completion-end))
      (setq counsel-completion-beg
            (move-marker (make-marker) (point)))
      (insert symbol)
      (setq counsel-completion-end
            (move-marker (make-marker) (point)))
      (when (equal (get-text-property 0 'symbol symbol) "f")
        (insert "()")
        (setq counsel-completion-end
              (move-marker (make-marker) (point)))
        (backward-char 1)))))

Essentially, the last interesting part is (equal (get-text-property 0 'symbol symbol) "f") which test if the string corresponds to a function or not. The rest of the code just fiddles with symbol markers with ensure that the previous symbol is erased before the new symbol is inserted if you press C-M-n (ivy-next-line-and-call), or use ivy-resume. Just to describe how this would be useful for the Python code above: I call counsel-jedi, followed by C-M-n to get:

params = {"foo":"bar"}
params.iteritems(|)

Pressing C-M-n again will result in:

params = {"foo":"bar"}
params.iterkeys(|)

Once I'm satisfied with my selected candidate, I can press either C-m or C-j or C-g. Most of the above code can be used almost verbatim if you're a Helm fan and want to implement helm-jedi. Basically, the approach I described (going through ac-sources) can be used to implement alternative completion in a lot of cases where auto-complete-mode completion is already available.

Finally, if you like the idea of auto-inserting parens with completion and are using C/C++, have a look at function-args - this package does that too, with Ivy, Ido and Helm as available back ends.

Store all current ivy candidates into the kill ring

I'd like to highlight a new command added to ivy-mode today: ivy-kill-ring-save. It allows to store all current candidates to the kill ring. This could have a number of uses.

Scenario 1

Suppose you want to learn some Elisp, specifically all functions that start with forward-. Then call counsel-describe-function with "forward-" as input and press M-w. Then just yank this into some buffer, org-mode for instance, and go through the functions one-by-one takings notes on the way:

forward-comment
forward-paragraph
forward-thing
forward-symbol
forward-visible-line
forward-word
forward-same-syntax
forward-whitespace
forward-button
forward-line
forward-list
forward-sentence
forward-point
forward-to-indentation
forward-ifdef
forward-char
forward-sexp
forward-page

Scenario 2

Suppose you want to store a subset of projectile-find-file names that match a pattern. Just press M-w and those file names will be on the top of the kill ring. Of course, you need to have this setting on:

(setq projectile-completion-system 'ivy)

This scenario should apply to basically any place where you're completing file names, like counsel-locate or counsel-find-file. Also find-file-in-project, which uses ivy for completion if it's installed.

Outro

Note also that the command comes for free, since normally nothing is done by M-w when the region isn't active. When the region is active, this command will store the selected text instead of matched candidates.

Thanks to @drorbemet for giving me the idea in #197.

Ivy-mode 0.6.0 is out

For those who don't keep up, ivy-mode is a completion method that's similar to ido, but with emphasis on simplicity and customizability. Currently, there are two related packages on MELPA: swiper and counsel:

• swiper provides an isearch replacement, using ivy-read for completion, as well as the basic ivy-mode.
• counsel provides some extra commands, like -M-x, -ag, -load-theme etc.

The reasoning behind the split into two packages is that there's less update overhead if only one of them is updated.

completing-read-function conundrum

This is to explain a bit the presence of the counsel package. Initially, I hoped that ivy-mode could do most of the work, via the Emacs' completing-read-function interface. How it works: most built-in packages will call completing-read when they require completion. That function will forward the completion to completing-read-function if it's set. This is the way how things like icomplete-mode or ivy-mode or helm-mode work.

Unfortunately, the interface rather limits what you can do in completing-read-function. Essentially, you're given a list of strings and you have to return a string. You have no idea which function called you, so you can't do any fancy stuff depending on the caller, short of examining this-command which isn't very reliable. You have no idea what will be done with the one string that you return, so you can't do something fancy like select two or three strings and perform that action for each of them.

The result is that sometimes I have to replace the built-in functions, instead of re-using them. So instead of re-using find-file, I wrote my own counsel-find-file that looks like this:

(defun counsel-find-file ()
  "Forward to `find-file'."
  (interactive)
  (ivy-read "Find file: " 'read-file-name-internal
            :matcher #'counsel--find-file-matcher
            :action
            (lambda (x)
              (with-ivy-window
                (find-file (expand-file-name x ivy--directory))))
            :preselect (when counsel-find-file-at-point
                         (require 'ffap)
                         (ffap-guesser))
            :require-match 'confirm-after-completion
            :history 'file-name-history
            :keymap counsel-find-file-map))

It's still possible to call the original find-file, and you'll get ivy-read completion for it, but:

• ivy-resume won't work, since ivy-read doesn't know what to do with the string that you select.
• C-M-n (ivy-next-line-and-call) won't work to select another file within the same completion session, for the same reason.
• M-o (ivy-dispatching-done) may work with some customization, but since it dispatches on this-command to get the extra actions, it can cause a problem if the command wasn't called directly.

I think it would be cool to extend the built-in completing-read-function interface in the future Emacs versions. Both Ivy and Helm (which uses the same strategy of passing the action to the completion) would benefit from this.

Release summary and highlights

It's been two months and 150 commits since the last version. The full release notes can be found inside the repository in Changelog.org or at Github's release tab.

I recommend to look through the whole list to see if anything catches your attention. I'll highlight a few things that I think the users might be interested in most.

Fuzzy completion

This is off by default, since I think having less candidates is better than having more. You can customize this for all commands or per-command. For example:

(setq ivy-re-builders-alist
      '((t . ivy--regex-fuzzy)))

swiper has a case-fold-search optimization

Binds case-fold-search to t when the input is all lower-case:

• input "the" matches both "the" and "The".
• input "The" matches only "The".

Anzu things

To see not only the number of matched candidates, but also the index of the current one, set this:

(setq ivy-count-format "(%d/%d) ")

Customize additional exit points for any command

For any command that uses ivy-read that is. Example for ivy-switch-to-buffer:

(ivy-set-actions
 'ivy-switch-buffer
 '(("k"
    (lambda (x)
      (kill-buffer x)
      (ivy--reset-state ivy-last))
    "kill")
   ("j"
    ivy--switch-buffer-other-window-action
    "other")))

After this:

• Use M-o k to kill a buffer.
• Use M-o j to switch to a buffer in other window.

You can always use M-o o to access the default action. When there is only one action, M-o does the same as C-m.

More descriptions with a built-in Hydra

Toggle C-o in any completion session to get an overview of the things that you can do.

Many commands that interface with shell calls

The following commands will call the appropriate shell tool to give you a list of candidates:

• counsel-git-grep
• counsel-ag
• counsel-locate
• counsel-recoll

These commands will refresh after each new letter entered, and they account for the fact that the shell call will usually take more time than the time it takes to input a new letter. So the process will be killed and restarted, resulting in virtually no keyboard delay and no downtime.

Many commands that offer more options than the built-in counterparts

This list includes:

• counsel-find-file,
• counsel-M-x,
• counsel-load-theme,
• counsel-org-tag and counsel-org-tag-agenda.

Most extra things that you can do are:

• Doing stuff with multiple candidates through C-M-n like in this video.
• Using ivy-resume to resume the last completion session.

Make sure to try counsel-org-tag, this one is a bit tricky for completion, since it requires to return multiple candidates at once. So you can toggle each tag with C-M-m (ivy-call) and then exit with C-m (ivy-done). The currently selected tags will be displayed in the prompt.

Outro

A big thanks to all people who contributed code and issues towards this release. I'm really happy that you're helping me to refine this nice package.

Using Recoll desktop search database with Emacs

I know that most Emacs hackers love the simplicity and usability of grep, but sometimes it just doesn't cut it. A specific use case is my Org-mode directory, which includes a lot of org files and PDF files. There are just too many files for grep to be efficient, plus the structure of PDF doesn't lend itself to grep, so another tool is required: a desktop database.

I got into the topic by reading John Kitchin's post on swish-e, however, I just couldn't get that software to work. But as a reply to his post, another tool - recoll was mentioned on Org-mode's mailing list. In this post, I'll give step-by-step instructions to make Recoll work with Emacs.

Building Recoll

I'm assuming that you're on a GNU/Linux system, since it's my impression is that it's the easiest system for building (as in make ...) software. Also, it's the only system that I've got, so it would be hard for me to explain other systems.

If you want to toy around with the graphical back end of Recoll, you can install it with:

sudo apt-get install recoll

Unfortunately, the shell tool recollq isn't bundled with that package, so we need to download the sources. The current version is 1.20.6.

Extract the archive

After downloading the archive, I open ~/Downloads in dired and press & (dired-do-async-shell-command). It guesses from the tar.gz extension that the command should be tar zxvf. By pressing RET, I have the archive extracted to the current directory. I've actually allocated ~/Software/ for installing stuff from tarballs, since I don't want to put too much stuff in ~/Downloads.

Open ansi-term

I navigate to the recoll-1.20.6/ directory using dired, then press ` to open an *ansi-term* buffer for the current directory.

Here's the setup for that (part of my full config):

(defun ora-terminal ()
  "Switch to terminal. Launch if nonexistent."
  (interactive)
  (if (get-buffer "*ansi-term*")
      (switch-to-buffer "*ansi-term*")
    (ansi-term "/bin/bash"))
    (get-buffer-process "*ansi-term*"))

(defun ora-dired-open-term ()
  "Open an `ansi-term' that corresponds to current directory."
  (interactive)
  (let ((current-dir (dired-current-directory)))
    (term-send-string
     (ora-terminal)
     (if (file-remote-p current-dir)
         (let ((v (tramp-dissect-file-name current-dir t)))
           (format "ssh %s@%s\n"
                   (aref v 1) (aref v 2)))
       (format "cd '%s'\n" current-dir)))
    (setq default-directory current-dir)))

(define-key dired-mode-map (kbd "`") 'ora-dired-open-term)

Configure and make

Here's a typical sequence of shell commands.

./configure && make
sudo make install
cd query && make
which recoll
sudo cp recollq /usr/local/bin/

I was a total Linux newbie 5 years ago and had no idea about shell commands. Using only the first two lines, you can build and install a huge amount of software, so these are a great place to start if you want to learn these tools. I actually got by using only those two lines for a year a so.

After the first run or ./configure it turned out that I was missing one library, so I had to do this one and redo the ./configure step.

sudo apt-get install libqt5webkit5-dev

I think this one would work as well:

sudo apt-get build-dep recoll

Configuring Recoll

I only launched the graphical interface to select the indexing directory. It's the home directory by default, I didn't want that so I chose ~/Dropbox/org/ instead. Apparently, there's a way to make the indexing automatic via a cron job; you can even configure it via the graphical interface: it's all good.

Using Recoll from Emacs

Emacs has great options for processing output from a shell command. So first I had to figure out how the shell command should look like. This should be good enough to produce a list of indexed files that contain the word "Haskell":

recollq -b 'haskell'

And there's how to adapt that command to the asynchronous ivy-read interface:

(defun counsel-recoll-function (string &rest _unused)
  "Issue recallq for STRING."
  (if (< (length string) 3)
      (counsel-more-chars 3)
    (counsel--async-command
     (format "recollq -b '%s'" string))
    nil))

(defun counsel-recoll (&optional initial-input)
  "Search for a string in the recoll database.
You'll be given a list of files that match.
Selecting a file will launch `swiper' for that file.
INITIAL-INPUT can be given as the initial minibuffer input."
  (interactive)
  (ivy-read "recoll: " 'counsel-recoll-function
            :initial-input initial-input
            :dynamic-collection t
            :history 'counsel-git-grep-history
            :action (lambda (x)
                      (when (string-match "file://\\(.*\\)\\'" x)
                        (let ((file-name (match-string 1 x)))
                          (find-file file-name)
                          (unless (string-match "pdf$" x)
                            (swiper ivy-text)))))))

The code here is pretty simple:

• I don't start a search until at least 3 chars are entered, in order to not get too many results.
• I mention :dynamic-collection t which means that recollq should be called after each new letter entered.
• In :action, I specify to open the selected file and start a swiper with the current input in that file.

Outro

I hope you found this info useful. It's certainly pretty cool:

cd ~/Dropbox/org && du -hs
# 567M .

So there's half of a gigabyte of stuff, all of it indexed, and I'm getting a file list update after each new key press in Emacs.

If you know of a better tool than recoll (I'm not too happy that match context that it gives via the -A command option), please do share. Also, I've just learned that there's helm-recoll out there, so you can use that if you like Helm.

New Ivy multi-action exit

So many Emacs commands and bindings, but little time to learn them all. That's why many-in-one solutions like hydra are often a good deal: you only remember the base binding, and get more info along the way if you get lost.

I've wanted to optimize this approach when it comes to completion as well. In last month's post I described how you can get help / jump to definition / jump to info for the current F1 v (counsel-describe-variable) candidate with C-m / C-. / C-,. While that approach can be viable, it has a few problems. The first problem is that it's not discoverable: you may be using C-m option for ages and not know that C-. and C-, exist. The second problem is that it's not extensible: if you wanted more actions, there are hardly any good bindings left in the minibuffer to bind those actions.

In a more recent post, I described a solution that alleviates both problems. When you press C-o, you see how many actions are available, and you can cycle them with w and s. However, while discoverable, this approach is cumbersome if you already know what you want to do. Especially cycling can't be great when there are a lot of actions.

So today I've added a new approach in addition to the previous one. When you press M-o (ivy-dispatching-done), you get a hint showing you which actions are available. It's like C-m (ivy-done), only allows you to quickly select the action. Conveniently, the default action is bound to o, so M-o o is equivalent to C-m.

Here's how I add one more action to the default one of ivy-switch-buffer:

(ivy-set-actions
 'ivy-switch-buffer
 '(("k"
    (lambda (x)
      (kill-buffer x)
      (ivy--reset-state ivy-last))
    "kill")))

So now:

• M-o o will still switch to selected buffer.
• M-o k will kill the selected buffer.

If you're familiar with hydra, the format is the same: (key cmd hint). This approach is very extensible: you can add actions and bindings to commands without changing the code of the original command. It's really fast: you're typing just one extra key to select an action. And you'll hardly run out of keys to bind, with 25 lower case keys at your disposal (other bindings work as well, I just think that lower case keys are the fastest to press).

In case you wonder what (ivy--reset-state ivy-last) does, it's used to update the list of buffers, since one of them was deleted. This way, you can delete e.g. 4 buffers with C-x b C-o s gggg, and have the buffer list update after each g.

New functionality in action

Let's look at one package that has a multitude of actions available for each candidate - projectile:

(defvar helm-projectile-projects-map
  (let ((map (make-sparse-keymap)))
    (set-keymap-parent map helm-map)
    (helm-projectile-define-key map
        (kbd "C-d") #'dired
        (kbd "M-g") #'helm-projectile-vc
        (kbd "M-e") #'helm-projectile-switch-to-eshell
        (kbd "C-s") #'helm-find-files-grep
        (kbd "M-c") #'helm-projectile-compile-project
        (kbd "M-t") #'helm-projectile-test-project
        (kbd "M-r") #'helm-projectile-run-project
        (kbd "M-D") #'helm-projectile-remove-known-project)
    map)
  "Mapping for known projectile projects.")

Here's the basic Ivy function for selecting a project:

(defun ivy-switch-project ()
  (interactive)
  (ivy-read
   "Switch to project: "
   (if (projectile-project-p)
       (cons (abbreviate-file-name (projectile-project-root))
             (projectile-relevant-known-projects))
     projectile-known-projects)
   :action #'projectile-switch-project-by-name))
(global-set-key (kbd "C-c m") 'ivy-switch-project)

And now let's add all those actions:

(ivy-set-actions
 'ivy-switch-project
 '(("d" dired "Open Dired in project's directory")
   ("v" helm-projectile-vc "Open project root in vc-dir or magit")
   ("e" helm-projectile-switch-to-eshell "Switch to Eshell")
   ("g"
    (lambda (x)
      (helm-do-grep-1 (list x)))
    "Grep in projects")
   ("c" helm-projectile-compile-project "Compile project")
   ("r" helm-projectile-remove-known-project "Remove project(s)")))

Here's what I get now after pressing M-o:

Looks pretty good, I think. I hope you find the new approach useful. The multi-action exit is currently enabled by default for ivy-switch-buffer, counsel-locate and counsel-rhythmbox, but you can add actions yourself to almost any command that uses ivy-read. In the rare case when ivy-read isn't in the tail position, you can use ivy-quit-and-run inside the added action functions.