In the past couple of months, I’ve been using and, a first for me, regularly contributing to this open source project called Spacemacs.

Spacemacs is a new distribution of Emacs. Think what Ubuntu did for GNU/Linux – Spacemacs is doing the same for GNU Emacs. It combines all the existing great pieces and providing an easy-to-use good-looking package.

Spacemacs

I used to use my own emacs configuration and then switched to Prelude for it’s neat Clojure integration because Bozhidar Batsov wrote CIDER (the Clojure-Emacs package) as well. This was mostly helpful when I was working at Helpshift.

It all started when I was watching Sacha Chua’s Emacs Hangout video and Howard Abrams mentioned Spacemacs, and for a change, I immediately jumped into using it.

What attracted me to Spacemacs was that it was initially based on evil-mode, a full vi emulation layer inside Emacs. This was great because I was indeed having an Emacs pinky problem. And then the sane key binding hierarchy combined with guide-key for visually seeing that hierarchy was icing on the cake.

I first tweeted questioning whether it’ll be difficult to integrate the rest of Emacs ecosystem and then took it as a challenge and added a layer for ERC (IRC package in Emacs), was impressed with the layer system of Spacemacs and I was hooked. Then, I added org-pomodoro, org-present, etc.

I talked about Spacemacs at Emacs SF meetup, and I sometimes help and ask questions myself in the Gitter chatroom via the IRC bridge.

Minor annoyances led me to submit pull requests upstream – git-link, evil-org-mode, projectile, etc.

Elisp hacking has been fun.

Things that I’d like to see improved in Spacemacs:

  1. The default configuration file should be empty. Spacemacs installs a ~/.spacemacs file which is already full of stuff and this confuses new users. New users expect to just copy/paste snippets and it should just work, they will not take the time to read a large config file. For example, there is a layers config variable where users are supposed to add names of the layers they want to use, instead users should be able to copy/paste (enable-layer 'org) instead and it can work equivalently.
  2. The holy-mode should enable normal Emacs usage. For example, I cannot use meta-shift-right in OrgMode (used for indenting a heading and it’s content) in holy-mode whereas there is a key > in evil-mode. Spacemacs needs to make up its mind on whether it’ll fully support a holy-mode vanilla Emacs key bindings and I hope it does.
  3. The develop branch moves fast and many early adopters are using that whereas newbies are using the master branch, and there is often confusion in the chat room when someone asks for help. I wish there was a command in spacemacs that will generate the useful information as a text (which operating system, which emacs version, which layers are enabled, holy-mode or evil-mode, etc.) which can be pasted into the chat room and will assist others to offer advice much faster. Update: I contributed a change to make this happen, and happy to see it’s adoption, both in the chat as well as the default issue template.

It’s funny how was using XEmacs a decade ago, then dived fully into Vim (even wrote a book on it) and now I’m back into Emacs land.

On the same note, I am fascinated with newer editors like GitHub’s Atom which is gaining traction, also has a good package management system and UI using HTML/CSS which makes for easy extensibility – a hallmark of a great editor, and fascinating new possibilities such as integrating IPython/Jupyter in a Light-Table inspired way. My curiosity about Atom first piqued because Electron, the core of Atom editor is being used by the Slack desktop apps, Microsoft Visual Studio for Mac and Linux, etc.

I don’t know if / when / how I’ll make a switch to Atom, but until then I’m happy with Spacemacs.

Comments

@phoe6 says:

I am going to try this out. did not not about it’s existence. I have aliased vim to emacs, just to break my nature.

@phoe6 says:

I use IntellIiJ for packaged things, but vim, subl and emacs for one off. But like to use the powerful features of emacs more.

@spradnyesh says:

impressed (especially because of evil-mode). going to try it! @swaroopch thanks for sharing, and contributing!

@frankiesardo says:

Great post about spacemacs. Love the editor but I’m struggling with evil-lisp-state. How do you find CLJ develpment on it?

@k4rtik says:

I should try Spacemacs sometime. @swaroopch writes about it … Didn’t know about evil mode before, cc: @shrayasr

You may have already noticed the blog is active again because of me accidentally leaving the Welcome to Jekyll! entry on the website.That entry was added when I created a new Jekyll repository and wrote a long Ruby script with lots of regexes and invocations of curl and Nokogiri to convert from the WordPress-exported XML to the Jekyll structure.I’m happy to say that the website is now fully converted to Jekyll and it will help me avoid the whole problem of WordPress security updates.

So why did I want to migrate to Jekyll in the first place? Because ever since I have been assimilated by Orgmode and immersed in Emacs, I wanted to write in Emacs and no longer wanted to do the long “edit-publish-refresh” cycle of using the WordPress admin, especially because I would write in Markdown or Org and then convert to HTML and upload the generated HTML to WordPress. Every time I had to edit the post after publishing, I would have to then edit HTML.

So Jekyll seemed like the best option because it allows me to choose the format that I write in. So I wrote a small plugin to render AsciiDoc, my new syntax of choice which I used to rewrite A Byte of Python:

#encoding: utf-8
# http://jekyllrb.com/docs/plugins/#converters
module Jekyll

  class AsciidocConverter < Converter
    safe true

    def setup
      # http://asciidoctor.org/rdoc/Asciidoctor.html
      require 'asciidoctor'
    rescue LoadError
      STDERR.puts "Cannot load asciidoctor gem"
      raise FatalException.new("Missing dependency: asciidoctor")
    end

    def matches(ext)
      ext =~ /^\.asciidoc$/i
    end

    def output_ext(ext)
      ".html"
    end

    def convert(content)
      setup
      Asciidoctor::Document.new(content,
        {:attributes => {"source-highlighter" => "pygments"}}).render
    end
  end
end

So, voila, I wrote this blog post in AsciiDoc and Jekyll.

Now, all I need for hosting this website is just nginx http server and some rewrite rules (for backwards compatibility of URLs), and I’m done. No other software required.

With this setup, hopefully, I should be able to write often again. Hope.

Although it’s not yet the last month of the year, I was in a reflective mood, and jotted down programming lessons I learned this year:

  1. Spend time first on what you’re going to do, iterate several times on the “plan”, and when you get to the computer, it should just be about typing out what you have already decided.

    Of course, I do my own planning within OrgMode itself, it’s hard for me to do my planning on paper, but the idea is to do the iteration of the steps first, take feedback from stakeholders, etc. The actual part of typing should come last and should be boring.

    There are places where this should probably not be applicable such as explorations of technology and finding out what is possible, but for majority of things, I constantly have to remind myself that we are in the problem-solving business first, and in the programming business second.

    Part of the problem-solving is in figuring out the repercussions of making code changes and how to restrict the limitations and expand the possibilities of the code that we write, especially keeping the code as slim as possible. In simpler words, part of the time spent up front should be as much about the accidental complexity as well as the incidental complexity.

    Some call this hammock-driven development. Some call this killing busywork.

  2. “Working in strange environments is a CS life skill” is what Dan Grossman mentioned at the start of the Programming Languages Coursera course and it was a profound statement for me.

    The times when I have resisted learning new tools or getting used to new programming languages are the times I have atrophied.

    The times when I threw away familiarity and explored new tools was when I learned and enjoyed.

    Time and again, I have to make a commitment to throw familiarity away. That is what pushes me to learn to customize Emacs despite already having spent years mastering Vim and spent so much time using PyCharm, etc.

    The flip side is that learning how to create a mobile app has always been a point of procrastination for me because it is all about learning new tools, new APIs, new IDE, etc.

  3. Always keep raw original data around.

    You never know when you’ll need to reprocess it. It is always possible to reprocess but losing raw original data is a bad place to be. For example, if you’re auto-parsing some user input and it is not a selection that the user is choosing, then you’re probably better off storing the user input as-is and having a separate field where you store the result of your parsing the user input.

    This habit is especially important to scale to a Lambda Architecture.

  4. Always use camel-case for data.

    This is something I have learned the hard way when trying to use data passed between Postgresql database, MongoDB database, a Java server, a Django website server with JavaScript / Ajax-based workflows, a Django-based admin, and so on.

    Especially when JavaScript and Java & GSON enter into the picture, life will be much easier if the keys of your JSON data are camel-cased.

  5. Do logging-first development.

    No matter how well-written or well-indented or well-formatted the code is, ultimately, the run-time behavior of the code is what determines the value of the compile-time code.

    Always think about how to log the importants parts of the code which work based on time-sensitive data, so that your post-mortem on why something happened will be far easier.

    For example, storing every API request and response to payment gateways was one of the best things I ever did when writing the ecommerce platform for Automatic. That clarified so many behaviors when investigating things after-the-fact, and it reiterates point 3 where it was easy to reprocess some statuses based on unexpected API responses, etc.

  6. First make it possible. Then make it beautiful. Then make it fast.

    I’ve worked on tasks more efficiently and effectively whenever I have taken the time to get a quick “let’s get it working” version out there first, and then tweaking it constantly to get the software that the stakeholder wants.

    Some call this as how Apple rolls. Some call this as software is grown, not built.

  7. Queues are what makes things scalable.

    Well, queues and asynchronicity are what makes things scalable.

    Use queues wherever you can.

  8. Immutability is paramount.

    It is what will keep your system sane. As @ghoseb says: “One day, we incurred a loss of a million Rupees at my company because of a bug that got triggered by my (apparently pure) code due to our usage of mutable datastructures in Common Lisp.”

    Writing pure functions makes code far more predictable and easier to understand, and at the same time, keeps the prone-to-errors-because-of-data surface area of the code smaller.

    A side-effect (sic) of considering immutability is that you learn to data all the things.

Twitter Comments

@anuragwho says:

@swaroopch 1,2, 6 are probably the most important things in programming ever. And I can’t stress enough on how amazing Grossmans course was.

<!– "Do logging-first development." and other programming lessons @swaroopch learned in '13: http://t.co/ZpjzFXSRlI

+1 for logging-first. –>

Ever since I got back into using OrgMode full-time, I’ve been using Emacs every single minute of using a computer. In fact, these days, I can’t even function without Emacs and OrgMode.

Naturally, it follows, that just like the days when I dived deep into Vim, I’m diving into Emacs these days.

I got interested in Emacs again because of BG’s dotemacs, but to truly become familiar, I had to create my own configuration files. It was a great way to learn Emacs Lisp. I have breezed through the initial chapters of the Emacs Lisp Manual and after getting introduced to Clojure recently, Lisp and functional programming doesn’t seem that alien any more.

I bumped into Cask and Pallet last week (probably via Emacs-Reddit) and that gave me impetus to kickstart my own .emacs.d over the weekend.

$HOME
 |
 \-> .emacs.d
      |
      \-> install.sh
      \-> Cask
      \-> init.el

Contents of the file install.sh :

#!/bin/bash

if [[ ! -e ~/.cask ]]
then
    echo "Cloning Cask repo"
    git clone git@github.com:cask/cask.git ~/.cask
fi

if [[ $(grep "cask/bin" ~/.bash_profile) == "" ]]
then
    echo "Adding \$HOME/.cask/bin to \$PATH in ~/.bash_profile"
    echo '' >> ~/.bash_profile
    echo "# Added by ~/.emacs.d/install.sh" >> ~/.bash_profile
    echo "export PATH=\$HOME/.cask/bin:\$PATH" >> ~/.bash_profile
fi

export PATH=$HOME/.cask/bin:$PATH

cd ~/.emacs.d
cask install

# For Python / ELPY
# Prerequisite: Install Python as per:
# http://docs.python-guide.org/en/latest/#getting-started
pip install --upgrade elpy flake8 rope jedi ipython

Contents of the file Cask initially was (I’ve added more packages now, but this is the minimal list of packages that I would use, although ELPY is optional if you don’t use Python):

(source "gnu" "http://elpa.gnu.org/packages/")
(source "melpa" "http://melpa.milkbox.net/packages/")
(source "marmalade" "http://marmalade-repo.org/packages/")
(source "org" "http://orgmode.org/elpa/")

(depends-on "ack-and-a-half")
(depends-on "cask")
(depends-on "elpy")
(depends-on "exec-path-from-shell")
(depends-on "flx-ido")
(depends-on "magit")
(depends-on "pallet")
(depends-on "projectile")
(depends-on "yasnippet")
(depends-on "zenburn-theme")

A minimal init.el looks like:

;; /This/ file (~init.el~) that you are reading
;; should be in this folder
(add-to-list 'load-path "~/.emacs.d/")

;; Package Manager
;; See ~Cask~ file for its configuration
;; https://github.com/cask/cask
(require 'cask "~/.cask/cask.el")
(cask-initialize)

;; Keeps ~Cask~ file in sync with the packages
;; that you install/uninstall via ~M-x list-packages~
;; https://github.com/rdallasgray/pallet
(require 'pallet)

;; Root directory
(setq root-dir (file-name-directory
                (or (buffer-file-name) load-file-name)))

;; Theme
;; https://github.com/bbatsov/zenburn-emacs
(load-theme 'zenburn t)
(set-cursor-color "firebrick")

;; Font
;; https://www.mozilla.org/en-US/styleguide/products/firefox-os/typeface/#download-primary
(set-frame-font "Fira Mono OT-14" nil t)

;; Don't show startup screen
(setq inhibit-startup-screen t)

;; Show keystrokes
(setq echo-keystrokes 0.02)

;; Path
(require 'exec-path-from-shell)
(when (memq window-system '(mac ns))
  (exec-path-from-shell-initialize))

;; Git
(require 'magit)
(eval-after-load 'magit
  (progn '(global-set-key (kbd "C-x g") 'magit-status)))

;; flx-ido completion system, recommended by Projectile
(require 'flx-ido)
(flx-ido-mode 1)
;; change it if you have a fast processor.
(setq flx-ido-threshhold 1000)

;; Project management
(require 'ack-and-a-half)
(require 'projectile)
(projectile-global-mode)

;; Snippets
;; https://github.com/capitaomorte/yasnippet
(require 'yasnippet)
(yas-load-directory (concat root-dir "snippets"))
(yas-global-mode 1)

;; Python editing
(require 'elpy)
(elpy-enable)
(elpy-use-ipython)

And that’s it, a full-fledged Emacs setup is ready.

The best part is to play around with M-x list-packages and browse around and decide which packages you want to try out and use, when you install a package, it automatically gets added to your Cask file (that’s what Pallet does), so you can convert this .emacs.d folder into a Git repository and sync anywhere and have the same configuration available on multiple machines.

So far, I have resisted many things like evil-mode and started really enjoying using packages such as org-pomodoro and ace-jump-mode.

Getting back to Emacs after a decade has been so much fun.

1 Pro Tip

Pro Tip: Always set this in your REPL to prevent long data structures or infinite loops from hanging your REPL:

(set! *print-length* 50)
(set! *print-level* 10)

2 Workshop problem

Starting the day with this problem:

Given a text file, let’s find the n most common and uncommon words. Ignore certain stop words.

Lesson taught: Data > Code.

My solution:

(def text-file
  "http://www.gutenberg.org/cache/epub/19033/pg19033.txt"
  (str (System/getenv "HOME") "/Downloads/pg19033.txt"))

(def stop-words
  #{"you"
    "the"
    "and"
    "a"
    "to"
    "of"
    "was"
    "i"})

(defn get-words
  [stop-words text-file]
  (remove stop-words
          (map string/lower-case
               (re-seq #"\w+" (slurp text-file)))))

(defn count-words
  [words]
  (sort (map #(vector (val %) (key %))
             (frequencies
              words))))

(defn main
  []
  (let [freqs (count-words
               (get-words stop-words text-file))]
    (println (take 5 freqs))
    (println (take-last 5 freqs))))

Output:

([1 0] [1 000] [1 12] [1 1500] [1 1887])
([144 said] [177 alice] [203 it] [228 in] [241 she])

How to write your own frequencies function and introducing how to use reduce and update-in:

(defn freqs
  [words]
  (reduce (fn [res w]
            (update-in res [w]
                       (fnil inc 0)))
          {}
          words))

Why fnil is better than an if form – telling computer what vs. how.

A better way to sort was shown by BG:

(defn top-words
  [word-counts]
  (sort-by (comp - val) word-counts))

(top-words (frequencies ["a" "a" "b" "c"]))
#= (["a" 2] ["b" 1] ["c" 1])

Introduction to threading macros to simplify series of function compositions.

Amazing how much has been covered and BG hasn’t mention the let form at all so far (although I’m using it).

After seeing BG’s code, I realized why I couldn’t get juxt working in my own main function – I was trying to call map on juxt and messing things up, instead I should have just called the result of juxt:

(defn main*
  []
  ((juxt #(take 5 %) #(take-last 5 %))
   (count-words
    (get-words stop-words text-file))))

#= [([1 "0"] [1 "000"] [1 "12"] [1 "1500"] [1 "1887"])
#  ([144 "said"] [177 "alice"] [203 "it"] [228 "in"] [241 "she"])]

The star at the end of a name is a convention which means it is an alternative version or slightly different version.

3 Multimethods

How to have “objects”? Use multi-methods.

  • Dispatch on an arbitrary function
  • Ad-hoc hierarchies

Java uses static dispatch because method invocation, i.e. method of which type to call, is decided by the compiler at compile-time (at least for simple cases).

Java is also single-dispatch because it can dispatch on only one factor – the type (i.e. the class and the arity of the method).

Hierarchies and type concepts are tied together in Java.

Example of how to do multimethods in Clojure:

(def unix
  {:os ::unix
   :c-compiler "cc"
   :home "/home"})

(def osx
  {:os ::osx
   :c-compiler "gcc"
   :home "/Users"})

(defmulti home :os)

(defmethod home ::unix
  [m]
  (:home m))

(home unix)
;= "/home"

(home osx)
;= IllegalArgumentException No method in multimethod
;  'home' for dispatch value: :example1.core/osx

The ::keyword is used to confine the keyword to the current namespace.

The argument is a dispatch function which should not have side-effects.

Since OS X is a derivative of Unix, use derive:

(derive ::osx ::unix)

(home osx)
;= "/Users"

So this means that osx “derives the functionality” from unix because we are creating a hierarchy.

This is going to take some time to digest… the dispatch here is on a map which is the idiom in Clojure compared to dispatching on the class in Java.

Introducing multiple dispatch by dispatching on both :os and compiler.

Use :default to give a default multimethod implementation.

Introducing ancestors and descendants to introspect the hierarchy.

Introducing prefers to handle when multiple ancestors match for a multimethod call.

Recommendation to read http://clojure.org/multimethods

Discussion on what is the difference between a class and a type? In Java, there is no difference. But they are actually orthogonal concepts.

So use a type to differentiate / switch behavior and we can use a simple map data structure or a keyword or any simple values in Clojure as a “type”, instead of depending entirely on “classes” in traditional OOP languages. A class is a bag of data and behavior.

Taking duck-typing as an example, as long as a parameter matches some “behavior” (type), we can use that parameter regardless of what “class” it belongs to.

On the other side, a class can have many “types”, for example, a vector can also behave as a collection and can behave as a sequence.

Recommendation to read http://thinkrelevance.com/blog/2009/08/12/rifle-oriented-programming-with-clojure-2

Point is to keep data and functionality separate and not complect it into classes.

Multimethods are extensible, they are global and hence users of the code can extend the multimethod to more types.

Example of built-in print-method.

4 Protocols & Types

Introduction to protocols & types: How do you extend a third-party library without access to it’s source code?

In OOP languages, you can create a new class and subclass existing third-party class and interoperate bidirectionally.

In FP languages, you can create a new method and make it work on existing third-party classes.

You have to choose either one in traditional OOP and FP languages.

This is called the Expression Problem.

In current languages, the solution is usually monkey-patching (think of find_by* methods in Rails ActiveRecord) which can be full of surprises and brittleness.

Discussion on how you can use visitor pattern in Java to add new functionality on top of existing classes, but you lose identity – you may expect Student but you get MyStudent.

In Clojure, you can do both, in a clean manner.

We can use multimethods but the limitation is that it is global.

Multimethods have the advantage of multiple dispatch. Protocols are single-dispatch on type.

The separation between types and behaviors in Clojure enables the concept of protocols.

Protocols can be confined to one namespace. Multimethods are global.

Example uses records – records behave like a map but has a type (identity) attached to it.

(defprotocol IPalindrome
  (palindrome? [o]
    "Check whether o is a palindrome."))

(defrecord student
    [name email])

(extend-protocol IPalindrome
  java.lang.String
  (palindrome? [s]
    (= s (apply str (reverse s))))

  student
  (palindrome? [s]
    (palindrome? (:name s))))

(palindrome? "malayalam")
;= true

(palindrome? (->student "malayalam" "b@b.com"))
;= true

extend-protocol calls extend underneath which is just associating a type with a protocol with a map data structure of function names to implementations.

You can check if a type extends? a protocol or an object satisfies? a protocol.

Introducing reify – reification means given an abstraction, create a concretion. It allows you to create anonymous implementations of any protocol.

(def *anon
  (reify IPalindrome
    (palindrome? [_] true)))

(palindrome? *anon)
;= true

You can use lexical closures inside a function and returns an object of anonymous type (a dynamic implementation) that satisfies a protocol.

Limitation of reify is it cannot instantiate classes, it can only instantiate protocols and interfaces. For classes, you can use proxy which is mainly used for Java interoperability.

Advantage of protocol is that you can group functions and check if a type extends that protocol. That is the difference from multimethods.

Example of how to use protocols to create mixins with example of IOFactory implementations in the Clojure source code – data all the things!

Internally, Clojure uses interfaces and ClojureScript uses protocols. In future, Clojure will internally switch to protocols as well.

Example of ChainMap data structure in Python and BG’s ChainMap implementation in Clojure by using protocols.

5 Concurrency

Handle state properly and you’ll get concurrency for free. Get the basics right.

BG gave example of his experience in Cleartrip.com about state and mutability.

In Clojure, identity and value are separated.

value
immutable data in a persistent structure
identity
series of causally related values over time
state
identity at a point in time

Example: “bank balance” is an identity, it’s value changes over time and it’s current value is it’s current state. You can put the “bank balance” inside a container which can decide the semantics of how you can change it’s state. The “containers” are refs, atoms, agents and vars.

shared isolated
synchronous / refs / stm
coordinated
synchronous / atoms vars
autonomous
asynchronous / agents
autonomous

In Clojure, there is optimistic concurrency. Unified update model:

  • update by function application
  • readers require no coordination
  • readers never block anybody
  • writers never block readers

In single-threaded-view languages, you will use locks which prevents reading from others as well.

ref atom agent var
create ref atom agent def
deref deref/@ deref/@ deref/@ deref/@
update alter swap! send alter-var-root

Use deref function or @ reader macro to dereference a reference type and get the value behind it.

Vars are special because they are deref-ed automatically.

first
;= #<core$first clojure.core$first@2be06d39>

(var first)
;= #'clojure.core/first

#'first
;= #'clojure.core/first

@#'first
;= #<core$first clojure.core$first@2be06d39>

STM-related functionality like alter and commute have to be in a dosync form (transaction). vars can be rebound:

api scope
alter-var-root root binding
set! thread-local, permanent
binding thread-local, dynamic

Dynamic scope is imperative, not functional. Looking at the code, you cannot know what value it will have.

6 Parallelism

Parallelism is not same as concurrency.In concurrency, there is at least one resource being shared.

All along, we have multiple processes running on a single CPU core because we have concurrent processes, i.e. there is scheduling. Concurrent code can be sequential or parallel.

Parallelism is about running multiple processes at the same time across multiple CPU cores. Parallelism is highest when there are no shared resources.

If code is written properly w.r.t. concurrency, you will get parallelism for free.

Proper code written can run as fast as possible on a single-core machine or a multiple-core machine. But non-concurrent code (use locks, etc.) can run as fast as possible on a single-core machine but will not increase in performance on a multiple-core machine.

Example is Postgresql vs. MySQL performance comparison on scaling with more CPU cores.

See future, pmap, pcalls, pvalues and java.lang.concurrent.Executor.

7 Macros

DSLs via Macros

Macros are functions that run at compile-time that will generate data which will be treated as code at run-time.

  • Syntax quote (back-tick)
  • Unquote (tilde)
  • Unquote splicing (tilde and at-symbol)
    • Use the contents of the list directly
  • Variable capture
  • Gensym

Possible because code and data are same.

If you want to create an unless form which is semantically a boolean opposite of the when form (an if form with only the then form, i.e. no else form), you cannot use a function because the parameters will be evaluated before calling the function, so you will need a macro here.

(defmacro unless
  [test & body]
  `(when (not ~test)
     ~@body))

(def dead? false)

(unless dead? (println "Alive"))
;= "Alive"

(def dead? true)

(unless dead? (println "Alive"))
;= nil

Using a hash at the end of a new var name inside a macro means a universally unique name is generated so that there are no name clashes with other code that the macro expansion is part of. It is same as calling gensym function manually.

There is a Clojure built-in that does the same as unless but it is given a better name – when-not.

Example of a recursive macro:

(defmacro do-until
  "cond's lost cousin"
  [& clauses]
  (when clauses
    `(when ~(first clauses)
       ~(if (next clauses)
          (second clauses)
          (throw (IllegalArgumentException.
                  "do-until needs an even number of forms.")))
       (do-until ~@(nnext clauses)))))

(do-until true (prn 1) false (prn 2))
;= 1

Introducing a longer example of using macros and functions to create a quite complex DSL. And finally generated XML out of it! Wow.

8 Not Covered

  • ClojureScript
  • Reducers framework
  • core.async
  • core.typed
  • core.logic

Some Great Libraries:

  • enlive
  • instaparse
  • fluokitten
  • ring
  • carmine
  • http-kit
  • slingshot
  • tools.reader
  • a lot more

Twitter Comments

@pdvyas says:

@swaroopch Thanks for blogging this.

@sandgorgon1 says:

@swaroopch great post. Missed the clojure class in pune. Sounds amazing. Wishing for a delhi one @ghoseb

@pradeepto says:

@swaroopch Thank you for the posts. And meeting you was a pleasure as always.

@protoiyer says:

Experienced mind-expanding #clojure immersion thanks to the awesome workshop by @ghoseb. Gr8 wrap-up by @swaroopch: http://t.co/QzqEg3eKwA

Comments

Mayank says:

Might I add the following:
http://defnewsletter.com/
http://www.clojuregazette.com/

They are like digested planet.clojure material (not always).

Also there is this course that is about to start which is really helpful in grokking lisp:
https://www.coursera.org/course/programdesign

While that course is not actually about lisp but it teaches using a dialect of lisp.
PS – I took that course and its brilliant! Highly Recommend it.

Continuing my journey of learning Clojure, I am attending BG’s Clojure Course. Today was Day 1. ~30 people in one room and BG as teacher for the weekend. Of course, I was looking forward to it. Most people in the audience had a background of mixture of C/C++, Java/Android, Python and Ruby.

BG Clojure Workshop

BG Clojure Workshop

Below are my rough notes from the day:

Day 1

BG introduced Clojure as “invented in 2007 by a wicked-cool, guitar-playing, computer scientist cum Zen master”, referring to Rich Hickey.

Emphasized that it is an abstraction-oriented language.

First known production usage of Clojure was in 2009 for a message bus infrastructure for a veterinary hospital in Canada. In 2010, BG’s company launched paisa.com in Clojure. And BG’s company’s new product HelpShift is also built on Clojure platform.

Three tenets of Clojure:

  1. Simplicity
    1. Simplicity is objective, easy is subjective
    2. Frameworks (forced structure) vs. Libraries (opt-in functionality)
  2. Power
    1. Practical
    2. Leverage – designed to be a hosted language – using the underlying platform doesn’t require a wrapper or different API
      1. Platforms today are JVM, V8, CLR, Python, LLVM, etc.
  3. Focus
    1. Focus on my problem (readability of code), not the language

A language as powerful as Lisp is timeless. Lisp is based on Lambda Calculus, the ultimate abstraction (BG made me realize why the name of the website lambda-the-ultimate.org is called so).

Great example of indexOfAny from Apache Commons Lang to explain the concept of complecting, and how you would do it in Clojure which was a simple few lines and which was more generic and useful.

Simple introduction to Lisp syntax.

REPL is a puzzle / surprise for the Java guys in the audience.

REPL is to language what terminal is to your operating system.

Namespaces are reified vs. packages, so namespaces exist at runtime whereas in Java, packages are only used for loading classes, you cannot modify packages at runtime, so in Clojure, you can remove a symbol from a namespace at runtime (using ns-unmap), you cannot do that in Java.

Introduction to reader – converts literal forms to core Lisp syntax to hand over to compiler.

Introduction to symbols, keywords and namespaces.

Namespaces are a map data structure underneath.

Core data structures / “collections” are list, vector, map, set. All are immutable.

type properties
list singly linked,
insert at front
vector indexed,
insert at rear
map key/value
set key

Maps are one of the most-important data structures because it is pervasive.

Collections are persistent, immutable, abstraction-oriented.

Persistent means we get immutability with performance. It’s not about Hibernate / saving to database. Phil Bagwell wrote the paper on it called HAMT (Hash Array Map Trie) but could not implement it with performance because he always used branching factor of 2 and trees became very deep. Rich Hickey used bit-partitioned tries of branching factor 32 which was a great optimization on HAMT to get the performance.

Why 32, not 64? Because in JVM, 32 is the “cache line”, so 32 means the entire thing is loaded at once.

Introduction to conj, the magnificent.

Introduction to apply function – the difference between vec and vector.

into is used to convert sequences back into collections, the opposite of seq.

A sequence is guaranteed to have a first and a rest, there is no empty sequence, which is why (seq []) will return nil.

Learned about update-in which is related to assoc-in and realized I was abusing the latter to do the former in my own code.

Introduction to some and every? and ? as a convention for predicates (“test” functions that return boolean).

Prefer the fn form always over the #(+ %1 %2) lambda form. For example, #([%1, %2]) will fail because it is expecting a function in the first position, and you should use either #(vector %1 %2) or use (fn [x y] [x y]) (in this case though, simply use vec?).

Got people to write and think functionally which was challenging and exciting because it made people think, for example, find the 100th fibonacci number using iterate.

My naive solution was this:

(defn fib
  [[x y]]
  [y (+' x y)])

(defn nth-fib
  [num]
  (first (first (take 1 (drop (- num 1) (iterate fib [0 1]))))))
(nth-fib 100)
#= 218922995834555169026N

which was quite close to the example code that BG showed, after getting people to rack their brains, so that was nice. One improvement was he used nth instead of my combination of take and drop.

I got reminded of the Tutorial on Good Lisp Programming Style where Peter Norvig says that most algorithms are a combination of the following:

  1. Searching
  2. Sorting
  3. Filtering
  4. Mapping
  5. Combining
  6. Counting

Introduction to Java interop. Can’t believe how specifically well-designed Clojure is for hosted platform interoperability, esp. the ., (.method Class) and the doto forms.

Introduced more details of functions such as multiple arities and variadic functions.

Most people seemed enthralled by the idea of functional programming and succinct code in Clojure, but are still grappling with it, which was fine because it was just the first day of immersion and a lot of conceptual ground was covered by BG.

Looking forward to Day 2 where BG said we’ll look into protocols, macros, etc.

Clojure Lessons Learned So Far

As an aside, I think the lessons I have learned from Clojure in the past year could be summarized as:

  1. Model the data (core data structures – map, vector, set) + lots of functions to manipulate it : Simpler + Less code – compare JSON parsing in Clojure vs. Java (GSON) vs. Hide data with method wrappers inside opaque objects. I learned this first from Perl but this lesson was forgotten after C++ OOP, Python classes, Ruby on Rails, Java OOP, etc. over the years, and now re-learned through Clojure.
  2. Lisp syntax means no difference between built-in vs. user-defined functionality vs. Code becomes ugly if non-built-in function, hence the need for monkey-patching, e.g. things.get(things.size() - 1) in Java == things[-1] in Python == (last things) in Clojure, and in this case last could be either built-in or user-defined
  3. Separation between state, time, value, identity : Core philosophy of Clojure explained in Rich Hickey’s talk “Are We There Yet?” : Account for time (and hence avoid concurrency issues) : an identity corresponds to series of immutable values at each point in time, called as state. Observers pick and use a state. => Values and Identities are persistent data structures. Timelines and perception implemented using Agents, STM, MVCC, etc. This point still has to sink in, but something I should remind myself from time to time to fully grok it.
  4. Use lots of generic functions. Code that I used to write as is_ssl = p.startswith("/a") or p.startswith("/b") becomes is_ssl = any(p.startswith("/" + k) for k in ("a", "b")) and the latter code is much easier to modify later, esp. adding new items to that list.

Actually, I think that first point needs more elaboration:

@nedbat says:

Dear lazytwitter: what is a good name for “data that can be serialized to JSON”? jsonable? We need a name for this!

 

@nedbat says:

“JSONable”: interchange, jsonic, _asdict, serializable, json-serializable, structured, organized, external, plain-old-data. others? votes?

From https://news.ycombinator.com/item?id=3917695 :

> I’m not so sure I agree that simple hashes are the best choice for internal data representations

I’ve been contributing to the Clojure community lately. My experience working with hash-maps as the primary data structure has been entirely liberating.

At my startup, we’ve got an app with Sinatra services, a Rails API, a Node.js web frontend, and Backbone client code. JSON gets passed between them. Being forced to encode keys as strings is a mild annoyance that Clojure’s reader syntax avoids, but the real issue is that I’ve got raw JSON, Javascript domain objects (Backbone.Model), Ruby models (ActiveRecord), and Ruby hashes (hashie/mash/similar). Each has their own idiosyncrasies and interfaces. Of all of them, the raw JSON is most pleasurable to work with. CoffeeScript & Underscore.js roughly approximate 10% of the awesomeness that is Clojure’s core data structures, including maps, sets, vectors, and lazy-seqs.

ActiveRecord, for example, makes it super easy to tangle a bunch of objects up. If we had a big bag of functions, they could operate on in-memory hashes, or they could operate on database rows, or they could operate on the result of an API call. It would be so much simpler to reuse code between our main Rails API and our Sinatra service. And we could one-for-one translate functions for non-Ruby services. Instead of requiring a crazy tangled ness of polymorphism and mutable state.

> Every non-trivial program is going to have to define abstract datatypes

Absolutely true. However, Clojure has taught me that you really aught to only define a very small number of those. It’s been said that it’s much better to have 100 functions which operate on 1 data structures, than to have 10 functions that operate on 10. Clojure’s get-in function for example: (get-in some-hash [:some :key :path]) is glorious compared to Ruby’s somehash[:some][:key][:path] because you don’t need to go monkey patch in a getpath method. And even if you did monkey patch that in, it won’t work for the someobject.some.key.path case, unless you got fancy with object.send and yet another monkey patch.

Look at some of the substantial pieces of Clojure code out there. They may only define a small handful of data structures, but most of those are even defined with defrecord, which produces a hash-like object, which all those 100s of functions work on. The rest are tiny primitives that compose in powerful an interesting ways.

> I’m not sure how embedding and dispatching on a type tag in a hash is any better than using the more explicit support for dynamic dispatch you find in typical OO languages

Because you may want differing dispatch and single-dispatch inheritance doesn’t let you change your mind as easily. Those dynamic dispatches in Ruby/Python whatever are simply hash lookups anyway. You’ll get the same performance either way. Look at the output of the ClojureScript compiler for example. Most code paths dispatch on :op, but you could just as easily dispatch on some bit of metadata, maybe the [:meta :dynamic] key path to have a function that runs differently on static vars than dynamic ones. People are also working on advanced predicate dispatch systems.

> The real problem with most OO is that it mashes a lot of interdependent, mutable state together.

That’s a real problem. But it’s not the real one :-)

List, vectors, maps and sets seem sufficient to model most data, why would you want software constructs that are not easily JSON-able or EDN-able? Especially these days when we use polyglot languages and polyglot databases.

Lastly, Greenspun’s Tenth Rule Of Programming says:

“Every sufficiently complex application/language/tool will either have to use Lisp or reinvent it the hard way.”

 

@GKCDaily says:

Weak things must boast of being new, like so many German philosophies. But strong things can boast of being old.

 

@technomancy says:

Obviously Chesterton was talking about software; scholars are divided as to whether he was talking about lisp or Debian Stable.

Twitter Comments

@ghoseb says:

@swaroopch Great post, Swaroop! Thanks for the kind words.

 

@P7h says:

@swaroopch Brilliant post. Thanks. I was trying to decide between Go and Clojure. Chose Go for now. Will learn Clojure some day soon.

 

@P7h says:

@swaroopch Also, which language [excluding Lisp] do u think is more elegant n wonderful? Python, Scala, Clojure or any other?

 

@P7h says:

Thanks to @swaroopch’s Clojure post, reread Smashing Magazine’s interview with Doug Crockford on “How I work”. http://t.co/ETDkivAExu 1/2

Comments

Mohit says:

What are the benefits of using Clojure over other dynamically type languages — Python, Ruby or PHP, in production environments ?

Mayank says:

Nice share :)

swaroop says:

@Mohit I don’t think I’m qualified to answer that question, @ghoseb can do a much better job of answering this question, but I’ll give it a try – immutability, connecting to production runtime and investigating what is happening via nREPL, functional encouraging good concurrent code (and hence good parallelism), are all advantages in production environments.

Event Report of @GhoseB’s 2-Day Clojure workshop – by @SwaroopCH | punetech.com says:

[…] has written a detailed and insightful event report. Here is his report of day 1 and report of day […]