Wednesday, April 02, 2014

Hash Maps in LFE: Request for Comment

As you may have heard, hash maps are coming to Erlang in R17. We're all pretty excited about this. The LFE community (yes, we have one... hey, being headquartered on Gutland keeps us lean!) has been abuzz with excitement: do we get some new syntax for Erlang maps? Or just record-like macros?

That's still an open question. There's a good chance that if we find an elegant solution, we'll get some new syntax.

In an effort to (re)start this conversation and get us thinking about the possibilities, I've drawn together some examples from various Lisps. At the end of the post, we'll review some related data structures in LFE... as a point of contrast and possible guidance.

Note that I've tried to keep the code grouped in larger gists, not split up with prose wedged between them. This should make it easier to compare and contrast whole examples at a glance.

Before we dive into the Lisps, let's take a look at maps in Erlang:

Erlang Maps

Common Lisp Hash Tables

Racket Hash Tables

Clojure Hash Maps

Shen Property Lists

OpenLisp Hash Tables

LFE Property Lists

LFE orddicts

I summarized some very basic usability and aesthetic thoughts on the LFE mail list, but I'll restate them here:
  • Erlang syntax really is quite powerful; I continue to be impressed.
  • Clojure was by far the most enjoyable to work with... however, doing something similar in LFE would require quite a bit of additions for language or macro infrastructure. My concern here is that we'd end up with a Clojure clone rather than something distinctly Erlang-Lispy.
  • Racket had the fullest and most useful set of hash functions (and best docs).
  • Chicken Scheme was probably second.
  • Common Lisp was probably (I hate to say it) the most awkward of the bunch). I'm hoping we can avoid pretty much everything the way it was done there :-/
One of the things that makes Clojure such a joy to work with is the unified aspect of core functions and how one uses these to manipulate data structures of different types. Most other implementations have functions/macros that are dedicated to working with just maps. While that's clean and definitely has a strong appeal, Clojure reflects a great deal of elegance.

That being said, I don't think today is the day to propose unifying features for LFE/Erlang data types ;-) (To be honest, though, it's certainly in the back of my mind... this is probably also true for many folks on the mail list.)

Given my positive experience with maps (hash tables) in Racket, and Robert's initial proposed functions like map-new, map-set, I'd encourage us to look to Racket for some inspiration:
Additional thoughts:
  • "map" has a specific meaning in FPs (: lists map), and there's a little bit of cognitive dissonance for me when I look at map-*
  • In my experience, applications generally don't have too many records; however, I've known apps with 100s and 1000s of instances of hash maps; as such, the idea of creating macros for each hash-map (e.g., my-map-get, my-map-set, ...) terrifies me a little. I don't believe this has been proposed, and I don't know enough about LFE's internals (much less, Erlang's) to be able to discuss this with any certainty.
  • The thought did occur that we could put all the map functions in a module e.g., (: maps new ... ), etc. I haven't actually looked at the Erlang source and don't know how maps are implemented in R17 yet (nor how that functionality is presented to the developer). Obviously, once I have, this point will be more clear for me.
With this done, I then did a thought experiment in potential syntax additions for LFE. Below are the series of gists that demonstrate this.

Looking at this Erlang syntax:

My fingers want to do something like this in LFE:

That feels pretty natural, from the LFE perspective. However, it looks like it might require hacking on the tuple-parsing logic (or splitting that into two code paths: one for regular tuple-parsing, and the other for maps...?).

The above syntax also lends itself nicely to these:

The question that arises for me is "how would we do this when calling functions?" Perhaps one of these:

Then, for Joe's other example:

We'd have this for LFE:

Before we pattern match on this, let's look at Erlang pattern matching for tuples:

Compare this with pattern matching elements of a tuple in LFE:

With that in our minds, we turn to Joe's matching example against a specific map element:

And we could do the same in LFE like this:

I'm really uncertain about add-pair and update-pair, both the need for them and the names. Interested to hear from others who know how map is implemented in Erlang and the best way to work with that in LFE...

Wednesday, March 19, 2014

lfetool T-Shirts?!

So, yeah... waaaaay too early for a T-shirt; the code has just gone from 0.1.x to 0.2.x. But, how could we resist: the project logo is retro like woodblock prints!

Earlier today, lfetool converted to using plugins (easier for contributors to play!), and hopefully later tonight YAWS + Bootstrap support will land. Regardless, there's tons more work to do, and what's more motivating for new work than a T-shirt? As you can see, we had no choice.

So let's get some. T-shirts, that is.

The lfetool logo is on the front, and Robert Virding's example Fibonacci lfescript code is on the back (with the lfetool command that generates the script at the top). Here's the front of the T-shirt:

And here's the back:

We've got a CustomInk sign-up sheet that you can add your name to if you want a shirt. I'll coordinate with folks individually, once we meet our minimum number (we're going to need to use paypal with payment upfront). Due to the colors of the source code on the back, the minimum order is 15. This will put the shirts at $22 a piece + $5 to send it to you. I've just ordered 2; 13 more go!

Tuesday, March 18, 2014

lfetool 0.2 Released

This release of lfetool has a bunch of new interesting features -- too much to cover in a tweet, so it's getting a little blog post ;-)

Here are the high-level bits that should make users' lives better:
  • New yaws project type, provided for building basic web sites; includes the exemplar project as a dependency for generating HTML with S-expressions.
  • Every project now gets TravisCI support by default.
  • Unit tests, integration tests, and systems tests now each have their own dir.
  • Tests now get run in parallel for faster run times.
  • Added support for running on Linux and *BSD systems.
  • Added unit tests for lfetool itself (shUnit) (for checking skeleton-generation and tool options).
  • lfetool has a new logo :-)
Note that the version jumped from 0.1.2 to 0.2.0 due to the strong improvements in quality and feature set of the tool. Forth-coming features include:
  • Support for an e2 service skeleton project.
  • Support for a YAWS-RESTful service skeleton project.
  • Support for YAWS embedded in a skeleton OTP application.
  • Support for YAWS + Exemplar + Bootstrap projects
  • Support for slide decks with Exemplare + Reveal.js
If you've got more ideas for the lfetool wishlist, post a comment here or send an email to the LFE mailing list.

Thursday, March 13, 2014

The Future of LFE

Erlang Factory

First of all, Erlang Factory this year was just phenomenal: great talks, great energy, and none of the feared/anticipated "acquisition feeding frenzy." Instead, everyone was genuinely happy for WhatsApp and Cloudant, and with celebrations aside, they were ready to get on with the conference and dive into the tech :-)

And gosh, there was a bunch of good tech. If you don't believe me, check out the schedule. Also on that page are the speaker pics. For talks that have video or slides pushed up, the speaker pic is visually annotated and linked.

There's so much good stuff there -- I've definitely got my watching queue set up for the next few weeks ...

LFE Presentation

I gave a presentation on LFE which covered everything from motivational basics for using a Lisp in the 21st century, gave a taste of LFE in small chunks, and then took folks on a quick tour of creating projects in LFE. There was also some dessert of fun side/research projects that are currently in-progress. The slides for the presentation are here; also the slide source code is available (related demo project). I'd also like to give a shout out to the Hoplon crew for their help in making sure I could create this presentation in a Lisp (Clojure), and not HTML ;-) (It uses a Hoplon-based Reveal.js library.)

The Good Stuff

After the presentation, several of us chatted about Lisp and Erlang for a while. Robert and I later continued along these lines after heading over to the quiet of the ever-cool Marines Memorial 11th-floor library (complete with fireplace). Here we sketched out some of the interesting areas for future development in LFE. I'm not sure if I'm remembering everything (I've also added stuff that we didn't discuss in the library, like Sean Chalmers' recent experiments with types; blog and discussion):
  • getting the REPL to the point where full dev can happen (defining functions, macros, and records in the LFE shell)
  • adding macros (maybe just one) for easier use of Mnesia in LFE 
  • discussing the possibility of an LFE stdlib
  • gathering some of the best funcs and macros in the wild for inclusion in an LFE stdlib
  • possibly supporting both (: module func ...) and (module:func ...) syntax
  • possibly getting spec and type support in LFE
  • producing an LFE 1.0 release
Additional efforts planned:
  • building out an LFE rebar plugin
  • examining as another possible option
  • starting work on an LFE Cookbook
  • creating demos of LFE on Erjang
  • creating demos of LFE-Clojure interop via JInterface
  • creating more involved YAWS/REST examples with LFE
  • explore the possibility of an SOA tutorial with LFE + YAWS 
  • async tasks in LFE with rpc lib 
  • monad tutorial for LFE (practical, not conceptual)
  • releasing a planner demo
  • finishing the genetic programming examples
  • LFE style guide
  • producing a stdlib contribution guideline
  • continued work on the LFE user guide
I'll dump all these into github issues so they'll be easier to track.

If this stuff is exciting to you, feel free to jump into the low-volume discussions we have on the mail list.

More soon!

Sunday, January 12, 2014

Prefix Operators in Haskell

I wanted to name this post something a little more catchy, like "McCarthy's Dream: Haskell as Lisp's M-Expressions" but I couldn't quite bring myself to do it. If s-expressions had greater support in Haskell, I would totally have gone for it, but alas, they don't.

However, there is still reason to celebrate: many Haskell operators do support prefix notation! This was a mind-blower to me, since I hadn't  heard about this until last night...

At the Data Day Texas conference this past Saturday, O'Reilly/StrataConf had a fantastic booth. Among the many cool give-aways they were doing, I obtained a coupon for a free ebook and another for 50% off. Upon returning home and having made my free book decision, I was vacillating between an OCaml book and the famous Haskell one. I've done a little Haskell in the past but have never touched OCaml, so I was pretty tempted.

However, something amazing happened next. I stumbled upon a page that was comparing OCaml and Haskell, which led to another page... where Haskell prefix notation was mentioned. I know many Haskellers who might read this would shrug, or say "yeah, we know", but this was quite a delightful little discovery for me :-)

I don't remember the first page I found, but since then, I've come across a couple more resources:
That's pretty much it, though. (But please let me know if you know of or find any other resources!)

As such, I needed to do a lot more exploration. Initially, I was really excited and thought I'd be able to convert all Haskell forms to s-expressions (imports, lets, etc.), but I quickly found this was not the case. But the stuff that did work is pretty cool, and I saved it in a series of gists for your viewing pleasure :-)


The first test was pretty simple. Finding success, I thought I'd try something I do when using a Lisp/Scheme interpreter as a calculator. As you can see below, that didn't work (the full traceback is elided). Searching on Hoogλe got me to the answer I was looking for, though. Off to a good start:

More Operators

I checked some basic operators next, and a function. Everything's good so far:


Here are some basic list operations, including the ever-so-cool list difference operator, (\\). Also, I enjoyed the cons (:) so much that I made a t-shirt of it :-) (See the postscript below for more info.)

List Comprehensions

I'm not sure if one can do any more prefixing than this for list comprehensions:


Same thing for functions; nothing really exciting to see. (Btw, these examples were lifted from LYHGG.)


Things get a little more interesting with lambda expressions, especially when a closure is added for spice:

Function Composition

Using the compose operator in prefix notation is rather... bizarre :-) It looks much more natural as a series of compositions in a lambda. I also added a mostly-standard view of the compose operator for comparison:


I've saved the best for last, an example of the sort of thing I need when doing matrix operations in game code, graphics, etc. The first one is standard Haskell...

Wow. Such abstract. So brains

Note that to make the prefix version anywhere close to legible, I added whitepsace. (If you paste this in ghci, you'll see a bunch of prompts, and then the result. If you're running in the Sublime REPL, be sure to scroll to the right.)

And that pretty much wraps up what I did on Sunday afternoon :-)

Other Resources

Post Script

If you're in the San Antonio / Austin area (or SF; I'll be there in March) and want to go in on a t-shirt order, let me know :-) We need at least six for the smallest order (tri-blend Next Level; these are a bit pricey, but they feel GREAT). I'll collect payment before I make an order.

Tuesday, December 31, 2013

Joe Armstrong's Favorite Erlang Program... in LFE

It kind of shocked me to discover recently that a new post hasn't been pushed out on this blog since April. Looking at the drafts I've got backlogged -- 30 and counting -- I also realized that none of these were going to get finished in a single day. Of those 30, probably 6 will ever see the light of day, and all of those are drafts in various states of completion for the Lambda Calculus series.

There's a great Tiny Scheme example I want to write about, some cool Clojure stuff I've played with, and bunch of functional programming work I'd like to share, etc., etc., but again, none of these are anything I can complete in a few minutes (allowing me to do the other things that I'd like to do today!).

I'd given up, when I remembered that there was something short, sweet, and a bit of ol' fun that I wanted to share! Mr. Erlang himself recently blogged about it, and I wanted to convert it to LFE: Joe Armstrong's Favorite Erlang Program.

This little puppy is quite delightful -- and Joe shares a great little story about how he deployed it on Planet Lab in his "aside" section of that post :-)

After you read his post, come back and take a look at this code:
That's how you do it in LFE! (Also, did you notice that Github now knows how to colorize *.lfe files? That happened here.)

Ain't it just the purdiest thing you ever saw?

This code has also been submitted for inclusion in the LFE examples (thus the "examples/" below). Let's do a quick sanity check:
And now, let's run the example!

Happy New Year, everyone :-D

Tuesday, April 23, 2013

OpenStack Developer Summit: Heat Followup

Folks are finally starting to recover from the OpenStack Developer Summit that was held in Portland, Oregon recently. All reports indicate that it was a truly phenomenal experience, record-breaking in many ways, and something that has inspired incredible enthusiasm within the community. And that's great news, since there's an enormous amount of work to be done this release ;-)

Of particular importance to many in the community is the work around maturing the autoscaling feature in OpenStack Heat. There was a fantastic session at the summit, facilitated by the bow-tied and most dapper Ken Wronkiewicz (his notes from the Summit were published on the Rackspace blog).

In preparation for the session, the following resources were created:
That one in the middle is important, as it is also where notes were taken during the actual session itself (see the section entitled "ODS Session Notes"). Devs at Rackspace have started going through the notes from the session and started planning work around this -- all of which will be carried on in the open, on the OpenStack mail list (tagged with "[Heat]"), on Freenode, and on github/gerrit.

The discussion at the Summit indicated strong interest in building a REST API for the existing autoscaling feature. Needless to say, there is a lot involved in this, touching upon significant OpenStack components like Quantum, LBaaS, and Ceilometer. Once the appropriate code is in place, a REST API will need to be created, features will need to be expanded/added, etc., and we'll be off and running :-)

Lots to do, and lots of great energy and excitement around this to keep us all chugging through this cycle.

On that note, we'd like to send out a special "thanks" to all the countless folks who worked so hard to make ODS happen. This event anchors us in a most excellent way, providing the insight and fuel that supports future development work so well!

Tea Time at Rackspace SF

One of the Rackspace teams here in the SF office doesn't to standups; it does "tea time" instead. A delightful change, to be sure. One of my coworkers published a Rackspace blog post about this today.

More than a gratutious reblog, I wanted to highlight this bit of lovely coincidence I came across while reading Turing's Cathedral last night:
"Afternoon tea— a ritual introduced at Fine Hall by Oswald Veblen, who, according to Herman Goldstine, “tried awfully hard to be an Englishman”— was served on real china daily at exactly three o’clock. According to Oppenheimer, “tea is where we explain to each other what we do not understand.”

Dyson, George (2012-03-06). Turing's Cathedral: The Origins of the Digital Universe (p. 90). Knopf Doubleday Publishing Group. Kindle Edition. 
The part of the book where this quote occurred was discussing Fine Hall which was built for Von Neumann's computer team at the Princeton Institute for Advanced Studies.

And, yes, tea time is a great time for our team mates to explain to each other what they don't understand.

In case you missed it above, here's the link again:

Thursday, April 18, 2013

Cruising HTTP with LFE

In the last post, you learned how to get LFE running on Ubuntu. This one will give you some insight into how LFE can be used in something approaching real-world problems. In the next post, we're going to jump back into the lambda calculus, and we'll see some more LFE shortly after that.

Because Lisp Flavored Erlang is 100% compatible with Erlang Core, it has access to all the Erlang libraries, OTP, and many third-party modules, etc.  Naturally, this includes the Erlang HTTP client, httpc. Today we're going to be taking a look at how to use httpc from LFE. Do note, however that this post is only going to provide a taste, just enough to give you a sense of the flavor, as it were.

If you would like more details, be sure to not only give the official docs a thorough reading, but to take a look at the HTTP Client section of the inets Reference Manual.

Note that for the returned values below, I elide large data structures. If you run them in the LFE REPL yourself, you can view them in all of the line-consuming glory.

Synchronous GET

Let's get started with a simple example. The first thing we need to do is start the inets application. With that done, we'll then be able to make client requests:
Now we can perform an HTTP GET: This just makes a straight-forward HTTP request (defaults to GET) and returns a bunch of associated data:
  • HTTP version
  • status code
  • reason phrase
  • headers
  • body
All of that data is dumped into our result variable. Here's the same GET but with pattern matching set up so that we can easily access all that data:
For those not familiar with Erlang patterns, we've just told LFE the following:
  • the return value of the function we're going to call is going to be a tuple composed of an atom ('ok) and another tuple
  • the nested tuple is going to be composed of a tuple, some headers, and a body
  • the next nested tuple is going to be composed of the HTTP version, status code, and status code phrase
If you'd like to learn more about using patterns in LFE, be sure to view the patterns page of the LFE User Guide.

Once the request returns, we can check out the variables we set in the pattern:

That's great if everything goes as expected and we get a response from the server. What happens if we don't?

Well, errors don't have the same nested data structure that the non-error results have, so we're going to have to make some changes to our pattern if we want to extract parts of the error reason. Pattern matching for just the 'error atom and the error reason, we can get a sense of what that data structure looks like:

Looking at just the data stored in the reason variable, we see:
If you check out the docs for httpc request and look under "Types", you will see that the error returned can be one of three things:
  • a tuple of connect_failed and additional data
  • a tuple of send_failed and additional data
  • or just unspecified additional data
In our example our additional data is a tuple of the address we were trying to connect to and the specific error type that for our failed connection.

Async GET

Now that we've taken a quick look at the synchronous example, let's make a foray into async. We'll still be using httpc's request function, but we'll need to use one of the longer forms were extra options need to be passed, since that's how you tell the request function to perform the request asynchronously and not synchronously.

For clarity of introducing the additional options, we're going to define some variables first: You can read more about the options in the httpc docs.

With the variables defined, let's make our async call: The sender receives the results, and since we sent from the LFE REPL, that's the process that will receive the data. Let's keep our pattern simple at first -- just the request id and the result data:
Needless to say, parsing the returned data is a waste of Erlang's pattern matching, so let's go back and do that again, this time with a nice pattern to capture the results. We'll need to do another request, though, so that something gets sent to the shell:
Now we can set up a pattern that will allow us to extract and print just the bits that we're looking for. The thing to keep in mind here is that the scope for the variables is within the receive call, so we'll need to display the values within that scope:
This should demonstrate the slight differences in usage and result patterns between the sync and async modes.

Well, that about sums it up for an intro to the HTTP client in LFE! But one last thing, for the sake of completeness. Once we're done, we can shut down inets:

Sunday, April 14, 2013

Getting Started with LFE on Ubuntu

For those that don't know, there is a fully Erlang Core-compatible Lisp-2 that runs on the Erlang VM and produces .beam files that can be used in any Erlang application. This manna from heaven is LFE, or Lisp Flavored Erlang. It was started about 5 years ago by Robert Virding, one of the co-creators of the Erlang programming language, and has inspired other similar efforts: Elixir (a Ruby-alike) and Joxa (a Lisp-1). (Incidentally, Robert has also created Prolog and Lua implementations that run on top of the Erlang VM!)

The new LFE docs site (a continuous work in progress) has some good introductory materials for the curious reader:

This blog post aims to bring some of those hidden materials into the consciousness of Ubuntu users. If you are averse to Erlang syntax, LFE opens up a whole new world to you :-)

The examples below assume Ubuntu 12.10.

Getting Erlang

Erlang R15B01 comes with Ubuntu 12.10. If that's all you need, then this will suite you just fine:
$ sudo apt-get install erlang
If you are wanting to test against multiple versions of Erlang, you should check out the kerl project, which lets you install a wide variety of Erlang versions (including the latest releases) side-by-side.

You'll also need git, if you don't yet have it installed:

$ sudo apt-get install git
Currently, rebar is required to build all the LFE files. If you're going to be building LFE projects, you'll want this anyway ;-) Rebar will be in Ubuntu 13.04, but it's not in 12.10, so you'll need to get it:
$ wget
$ chmod 755 rebar
$ sudo mv rebar /usr/local/bin

Getting and Building LFE

Here's what you need to do to build LFE:
$ mkdir -p ~/lab/erlang && cd ~/lab/erlang
$ git clone
$ cd lfe
$ make compile
If you looked at your ./ebin directory when you cloned the repo, you would have seen that there were no .beam files in it. After compiling, it is full of .beams ;-)

Sidebar: A common pattern in Erlang applications is the use of a deps directory under one's project dir where dependencies can be installed without conflicting with any system-wide installs, providing versioning independence, etc. Managing these with rebar has been very effective for projects, where simply calling rebar compile puts everything your app needs in ./deps. Projects that depend upon LFE are doing this, but we'll cover that in a future blog post.

Using LFE

With everything compiled, we can jump right in! Let's fire up the REPL, and do some arithmetic as a sanity check:

How about a message to stdout?

Any form starting with : is interpreted as a call to a module. The full form is (: <module name> <function name> <arguments>). As such, you can see that we're calling the format function in the (built-in) io module.

Also, it's good to know that there are certain things that you can't do in the REPL, e.g., defining modules, macros, functions, and records. Erlang expects that these sorts of activities take place in modules. However, we can explore a little more before we write our first module. Let's use the REPL's set form and lambda to define a function anyway (albeit, in a somewhat awkward fashion):

That wasn't too bad ;-) We're seeing the external module call, again -- this time to the math library. Now let's use a module of our own devising...

Creating Modules

In another terminal (but same working directory), let's create a module in a file called my-module.lfe, with the following content:

Note that the module name in the code needs to match the file name (minus the extension) that you used for the module.

Back in the REPL terminal window, let's compile this module and run the defined function:

Let's add another function to the module that demonstrates the benefits of Erlang's multiple-arity support:

Re-compiling and running the functions, we are greeted with success:

Lastly, let's convert the power function we defined in the previous section using our REPL-workaround to a "real" function, defined in our new module:

And then let's try it out:


(Of course, it's rather absurd to redefine pow to exp, when there is basically nothing to gain by it ;-) It's just a quick demo...)


There's lots more to learn; this has been just a small sip from a hot mug o' LFE.

However, it's definitely enough to get you started and, should you be interested in following along in future LFE blog posts, you'll have everything you need to get the most out of those.