Progress bars for apt in shell

Intro

For a couple years now, I use M-x shell as my main shell. Recently, I have fixed one of the minor annoyances that go along with using shell in Emacs. At least since Ubuntu 18.04, the terminal "progress bar" feature, displayed below is non-optional:

It uses terminal escape codes to display the progress bar, and shell-mode can't handle them well, so they clobber a lot of the output.

Initial work around

Previously, I was using this work around, since apt-get doesn't display the progress bar:

# sudo apt upgrade
sudo apt-get upgrade

Progress bar in the mode line

But typing 4 extra chars is hard. And apt-get will likely get these progress bars at some point as well. So I spent around an hour of my weekend hacking an Elisp solution. Here is the code:

(advice-add
 'ansi-color-apply-on-region
 :before 'ora-ansi-color-apply-on-region)

(defun ora-ansi-color-apply-on-region (begin end)
  "Fix progress bars for e.g. apt(8).
Display progress in the mode line instead."
  (let ((end-marker (copy-marker end))
        mb)
    (save-excursion
      (goto-char (copy-marker begin))
      (while (re-search-forward "\0337" end-marker t)
        (setq mb (match-beginning 0))
        (when (re-search-forward "\0338" end-marker t)
          (ora-apt-progress-message
           (substring-no-properties
            (delete-and-extract-region mb (point))
            2 -2)))))))

(defun ora-apt-progress-message (progress)
  (setq mode-line-process
        (if (string-match
             "Progress: \\[ *\\([0-9]+\\)%\\]" progress)
            (list
             (concat ":%s "
                     (match-string 1 progress)
                     "%%%% "))
          '(":%s")))
  (force-mode-line-update))

The solution will detect e.g. "\0337...Progress: [ 25%]...\0338", remove it from the shell buffer and display "25%" in the mode line instead.

Use the Echo Area instead of the mode line

The above is a good enough solution specifically for apt(8), but not for the generic case. Let's try to emulate how e.g. gnome-terminal handles these escape sequences. It takes sequences like "\0337.*\0338" and displays them in the bottom of the window. Kind of like the Emacs Echo Area. That's easy enough to do:

(defun ora-apt-progress-message (progress)
  (message
   (replace-regexp-in-string
    "%" "%%"
    (ansi-color-apply progress))))

Above, we use ansi-color-apply to get rid of any extra terminal escape codes. I decided to stay with the Echo Area version instead of the mode line version. Here's how it looks like:

You can find all of the above code in my config. Happy hacking!

Using exclusion patterns when grepping

Git

I like Git. A lot. After years of use it has really grown on me. It's (mostly) fast, (often) reliable, and (always) distributed. For me, all properties are important, but being able to do git init to start a new project in seconds is the best feature.

When it comes to working with Git day-to-day, a nice GUI can really make a difference. In Emacs world, of course it's Magit. Outside of Emacs (brr), git-cola looks to be the most promising one. If you're aware of something better, please share - I'm keeping a list of suggestions for my non-Emacs using colleagues.

Ivy integration for Git

The main two commands in ivy that I use for Git are:

• counsel-git: select a file tracked by Git
• counsel-rg: grep for a line in all files tracked by Git, using ripgrep as the backend.

There are many alternatives to counsel-rg that use a different backend: counsel-git-grep, counsel-ag, counsel-ack, counsel-pt. But counsel-rg is the fastest, especially when I have to deal with Git repositories that are 2Gb in size (short explanation: it's a Perforce repo with a bunch of binaries, because why not; and I'm using git-p4 to interact with it).

Using .ignore with ripgrep

Adding an .ignore file to the root of your project can really speed up your searches. In my sample project, I went from 10k files to less than 500 files.

Example content:

/TAGS
*.min.js*
/Build/Output/
/ThirdParty/

As you can see, both file patterns and directories are supported. One other nifty thing that I discovered only recently is that you can use ripgrep as the backed for counsel-git in addition to counsel-rg. Which means the same .ignore file is used for both commands. Here's the setting:

(setq counsel-git-cmd "rg --files")

And here's my setting for counsel-rg:

(setq counsel-rg-base-command
      "rg -i -M 120 --no-heading --line-number --color never %s .")

The main difference in comparison to the default counsel-rg-base-command is -M 120 which means: truncate all lines that are longer than 120 characters. This is really helpful when Emacs is accepting input from ripgrep: a megabyte long line of minified JS is not only useless since you can't see it whole, but it will also likely hang Emacs for a while.

Outro

I hope you found these bits of info useful. Happy hacking!

Using digits to select company-mode candidates

I'd like to share a customization of company-mode that I've been using for a while. I refined it just recently, I'll explain below how.

Basic setting

(setq company-show-numbers t)

Now, numbers are shown next to the candidates, although they don't do anything yet:

Add some bindings

(let ((map company-active-map))
  (mapc
   (lambda (x)
     (define-key map (format "%d" x) 'ora-company-number))
   (number-sequence 0 9))
  (define-key map " " (lambda ()
                        (interactive)
                        (company-abort)
                        (self-insert-command 1)))
  (define-key map (kbd "<return>") nil))

Besides binding 0..9 to complete their corresponding candidate, it also un-binds RET and binds SPC to close the company popup.

Actual code

(defun ora-company-number ()
  "Forward to `company-complete-number'.

Unless the number is potentially part of the candidate.
In that case, insert the number."
  (interactive)
  (let* ((k (this-command-keys))
         (re (concat "^" company-prefix k)))
    (if (cl-find-if (lambda (s) (string-match re s))
                    company-candidates)
        (self-insert-command 1)
      (company-complete-number (string-to-number k)))))

Initially, I would just bind company-complete-number. The problem with that was that if my candidate list was ("var0" "var1" "var2"), then entering 1 means:

• select the first candidate (i.e. "var0"), instead of:
• insert "1", resulting in "var1", i.e. the second candidate.

My customization will now check company-candidates—the list of possible completions—for the above mentioned conflict. And if it's detected, the key pressed will be inserted instead of being used to select a candidate.

Outro

Looking at git-log, I've been using company-complete-number for at least 3 years now. It's quite useful, and now also more seamless, since I don't have to type e.g. C-q 2 any more. In any case, thanks to the author and the contributors of company-mode. Merry Christmas and happy hacking in the New Year!

Comparison of transaction fees on Patreon and similar services

On December 7, Patreon made an announcement about the change in their transaction fee structure. The results as of December 10 speak for themselves:

December 2017 summary: -$29 in pledges, -6 patrons

All leaving patrons marked "I'm not happy with Patreon's features or services." as the reason for leaving, with quotes ranging from:

The billing changes are not great.

to:

Patreon's new fees are unacceptable

In this article, I will explore the currently available methods for supporting sustainable Free Software development and compare their transaction fees.

My experience

My experience taking donations is very short. I announced my fund raising campaign on Patreon in October 2017.

Here's what I collected so far, vs the actual money spent by the contributors:

• 2017-11-01: $140.42 / $162.50 = 86.41%
• 2017-12-01: $163.05 / $187.50 = 86.96%

The numbers here are using the old Patreon rules that are going away this month.

Real numbers

On Patreon

Just like everyone else, I'm not happy with the incoming change to the Patreon fees. But even after the change, it's still a better deal than OpenCollective, which is used quite successfully e.g. by CIDER.

Just to restate the numbers in the table, if all backers give $1 (which is the majority currently, and I actually would generally prefer 5 new $1 backers over 1 new $5 backer), with the old system I get $0.86, while with the new system it's $0.69. That's more than 100% increase in transaction fees.

On OpenCollective

It's more expensive than the new Patreon fees in every category or scenario.

On Flattr

Flattr is in the same bucket as Patreon, except with slightly lower fees currently. Their default plan sounds absolutely ridiculous to me: you install a browser plug-in so that a for-profit corporation can track which websites you visit most often in order to distribute the payments you give them among those websites.

If it were a completely local tool which doesn't upload any data on the internet and instead gives you a monthly report to adjust your donations, it would have been a good enough tool. Maybe with some adjustments for mind-share bubbles, which result in prominent projects getting more rewards than they can handle, while small projects fade away into obscurity without getting a chance. But right now it's completely crazy. Still, if you don't install the plug-in, you can probably still use Flattr and it will work similarly to Patreon.

I made an account, just in case, but I wouldn't recommend going to Flattr unless you're already there, or the first impression it made on me is wrong.

On Paypal

Paypal is OK in a way, since a lot of the time the organizations like Patreon are just middle men on top of Paypal. On the other hand, there's no way to set up recurring donations. And it's harder for me to plan decisions regarding my livelihood if I don't know at least approximately the sum I'll be getting next month.

My account, in case you want to make a lump sum donation: paypal.me/aboabo.

On Bitcoin

Bitcoin is similar to Paypal, except it also:

• has a very bad impact on the environment,
• is a speculative bubble that supports either earning or losing money without actually providing value to the society.

I prefer to stay away from Bitcoin.

Summary

Liberapay sounds almost too good to be true. At the same time, their fees are very realistic, you could almost say optimal, since there are no fees for transfers between members. So you can spend either €20.64 (via card) or €20.12 (via bank wire) to charge €20 into your account and give me €1 per month at no further cost. If you change your mind after one month, you can withdraw your remaining €19 for free if you use a SEPA (Single Euro Payments Area) bank.

If I set out today to set up a service similar to Liberapay, even with my best intentions and the most optimistic expectations, I don't see how a better offer could be made. I recommend anyone who wants to support me to try it out. And, of course, I will report back with real numbers if anything comes out of it.

Thanks to all my patrons for their former and ongoing support. At one point we were at 30% of the monthly goal (25% atm.). This made me very excited and optimistic about the future. Although I'm doing Free Software for almost 5 years now, it's actually 3 years in academia and 2 years in industry. Right now, I'm feeling a burnout looming over the horizon, and I was really hoping to avoid it by spending less time working at for-profit corporations. Any help, either monetary or advice is appreciated. If you're a part of a Software Engineering or a Research collective that makes you feel inspired instead of exhausted in the evening and you have open positions in EU or on remote, have a look at my LinkedIn - maybe we could become colleagues in the future. I'll accept connections from anyone - if you're reading this blog, we probably have a lot in common; and it's always better together.

Ivy 0.10.0 is out

Intro

Ivy is a completion method that's similar to Ido, but with emphasis on simplicity and customizability.

Overview

The current release constitutes of 280 commits and 8 months of progress since 0.9.0. Many issues ranging from #952 to #1336 were fixed. The number of people who contributed code as grown to 91; thanks, everyone!

Details on changes

Changelog.org has been a part of the repository since 0.6.0, you can get the details of the current and past changes:

• in org-mode
• in HTML

Highlights

Many improvements are incremental and don't require any extra code to enable. I'll go over a few selected features that require a bit of information to make a good use of them.

Selectable prompt

Off by default. You can turn it on like so:

(setq ivy-use-selectable-prompt t)

After this, your current input becomes selectable as a candidate. Press C-p when you're on the first candidate to select your input instead.

This solves the long standing issue of e.g. creating a file or a directory foo when a file foobar already exists. Previously, the only solution was to use C-M-j. It's still available, but now you can also select your input with C-p and press RET.

New global actions for ivy

ivy-set-actions was used to enable the following bindings:

• Press M-o w to copy the current candidate to the kill ring.
• Press M-o i to insert the current candidate into the buffer.

These bindings are valid for any completion session by default.

Use C-d in ivy-occur buffers

Here's an example use-case: search your source code for a variable name with e.g. counsel-rg and call ivy-occur (C-c C-o). Suppose you get 10 results, only 4 of which are interesting. You can now delete the uninteresting ones with C-d. Then maybe check off the others with C-d as well as you complete them one by one. A sort of a TODO list.

Similarly, if you want to go over variables to customize, you can call counsel-describe-variable with input ^counsel-[^-] and then check off the ones you have already examined with C-d.

Defcustoms to play with

Here's the list of new defcustom or defvar that might be interesting to review:

• counsel-async-filter-update-time
• counsel-async-ignore-re
• counsel-describe-function-function
• counsel-describe-function-preselect
• counsel-find-file-ignore-regexp
• counsel-fzf-dir-function
• counsel-git-grep-skip-counting-lines
• counsel-git-log-split-string-re
• counsel-url-expansions
• ivy-auto-select-single-candidate
• ivy-magic-slash-non-match-action
• ivy--preferred-re-builders
• ivy-truncate-lines

New Commands

14 new commands were added by me and many contributors. Here's the list:

• counsel-ack - completion for ack
• counsel-apropos - completion for apropos
• counsel-file-register - completion for file registers
• counsel-fzf - completion for fzf
• counsel-git-change-worktree - completion for git-worktree
• counsel-git-checkout - completion for git-checkout
• counsel-minibuffer-history - generalization of counsel-expression-history and counsel-shell-command-history
• counsel-org-capture - completion for org-capture
• counsel-org-file - browse all attachments for the current Org file
• counsel-org-goto - completion for Org headings
• counsel-org-goto-all - completion for Org headings in all open buffers
• counsel-switch-to-shell-buffer - switch to a shell buffer, or create one
• ivy-occur-delete-candidate - delete current candidate in ivy-occur-mode
• ivy-switch-view - select a window configuration, decoupled from ivy-switch-buffer

My personal favorites are counsel-fzf and counsel-org-file.

Outro

Again, thanks to all the contributors. Happy hacking!

P.S. Please consider joining my 74 patrons to give me the opportunity to work on Free Software a lot more. We are currently at 30% of the goal.