counsel-git-grep, in async

Introduction

It took me quite a while to figure out, but finally counsel-git-grep, which I've described before works completely asynchronously, which means it's very fast and smooth even for huge repositories.

If you don't know what git grep is: it allows you to search for a regular expression in your Git repository. In a way, it's just a find / grep combo, but very concise and fast. It's particularly great for finding all uses and references to a symbol.

It can be useful even for non-programmers. One great application is to stick all your org-mode files into a Git repository. Then you can search all your files at once very fast: all my org stuff takes more than 1000,000 lines and there's no lag while searching. I really should remove some pdfs from the repository at some point.

The Video Demo

Check out the speed and the various command in this Video Demo. Each key stroke starts two asynchronous shell calls, while canceling the current ones if they are still running. Here are the example calls:

$ git --no-pager grep --full-name -n --no-color \
    -i -e "forward-line 1" | head -n 200
$ git grep -i -c 'forward-line 1' \
    | sed 's/.*:\\(.*\\)/\\1/g' \
    | awk '{s+=$1} END {print s}'

The Elisp side

I figure since I was interested in this topic, probably a few more people are as well. So I'll explain below how async processing is done (in this case):

(defun counsel--gg-count (regex &optional no-async)
  "Quickly count the amount of git grep REGEX matches.
When NO-ASYNC is non-nil, do it synchronously."
  (let ((default-directory counsel--git-grep-dir)
        (cmd (concat (format "git grep -i -c '%s'"
                             regex)
                     " | sed 's/.*:\\(.*\\)/\\1/g'"
                     " | awk '{s+=$1} END {print s}'"))
        (counsel-ggc-process " *counsel-gg-count*"))
    (if no-async
        (string-to-number (shell-command-to-string cmd))
      (let ((proc (get-process counsel-ggc-process))
            (buff (get-buffer counsel-ggc-process)))
        (when proc
          (delete-process proc))
        (when buff
          (kill-buffer buff))
        (setq proc (start-process-shell-command
                    counsel-ggc-process
                    counsel-ggc-process
                    cmd))
        (set-process-sentinel
         proc
         #'(lambda (process event)
             (when (string= event "finished\n")
               (with-current-buffer
                   (process-buffer process)
                 (setq ivy--full-length
                       (string-to-number
                        (buffer-string))))
               (ivy--insert-minibuffer
                (ivy--format ivy--all-candidates)))))))))

1. Since we're calling a shell command, it's important to set default-directory to a proper value: the shell command will be run there.
2. The output from the process will be channeled into a buffer. I start the buffer name with a space to make it hidden: it will not be shown in the buffer list and switch-to-buffer completion.
3. I leave the sync option and use shell-command-to-string for that. Sometimes it's necessary to know the amount of candidates without a delay.
4. If get-process and get-buffer return something it means that the shell command is still running from the previous input. But the input has changed and that data has become useless, so I kill both the process and the buffer.
5. Then I start a new process with start-process-shell-command- a convenience wrapper around the basic start-process that's useful for passing a full shell command with arguments.
6. Finally, I let counsel--gg-count return without waiting for git grep to finish. But something needs to be done when it finishes, so I use set-process-sentinel. Emacs will call my lambda when there's a new event from the process.

Outro

To check out the new stuff, just install or upgrade counsel from MELPA. Or clone the source. I hope that my Elisp explanations were useful, async stuff is awesome, and I hope to see more of it in future Emacs packages.