TracePoint essentially allows you to hook into Ruby’s events and listen for events.

Being curious and since I just started a brand new Rails 4/Ruby 2 app, I decided to write a little middleware and see what Rails is up to when handling incoming requests.

Here is my TracePoint Rack Middleware.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

class TracePoint
  class Middleware

    def initialize(app)
      @app = app
    end

    def call(env)
      stats = {}
      trace = TracePoint.new(:call) do |tp|
        stats[tp.defined_class] ||= {}
        stats[tp.defined_class][tp.method_id] ||= 0
        stats[tp.defined_class][tp.method_id] += 1
      end
      trace.enable
      response = @app.call(env)
      trace.disable

      puts "#{stats.keys.size} classes used"
      puts "#{stats.map{|k,v| v.keys}.flatten.size} methods used"
      puts "#{stats.map{|k,v| v.values}.flatten.sum} methods dispatched"
      response
    end

  end
end

(the gist shows a modified version so I could dump to disk the json representation of the calls)

I then inserted the middleware in Rails:

1
2

# in application.rb
config.middleware.insert_before(ActionDispatch::Static, TracePoint::Middleware)

I saved the output in json format for the curious: click here

On average, in production mode, using Ruby 2.0 and Puma on my laptop, my hello world index page takes 5ms.

To render my page, Rails uses (more or less):

• 250 classes
• 750 methods (not including C functions)
• and dispatches 2704 methods (not including calls to C functions)

Here is a small selection of some of the methods dispatched:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

"String": {
    "underscore": 1,
    "blank?": 14,
    "html_safe": 78
  },
"ActiveSupport::Inflector": {
  "underscore": 2,
  "inflections": 2
},
"Hash": {
  "with_indifferent_access": 2,
  "except": 1,
  "except!": 1,
  "stringify_keys": 5,
  "transform_keys": 13,
  "stringify_keys!": 1,
  "transform_keys!": 3,
  "extractable_options?": 4,
  "extract!": 2,
  "symbolize_keys": 8,
  "reverse_merge": 1,
  "slice": 2,
  "symbolize_keys!": 2
},
"ActionView::CompiledTemplates": {
  "_app_views_welcome_index_html_erb__4177595130715791755_70209827438920": 1,
  "_app_views_layouts_application_html_erb___652124533295419796_70209827456500": 1
},
"ActiveSupport::Notifications::Fanout": {
  "start": 4,
  "listeners_for": 12,
  "listening?": 5,
  "finish": 3
}

TracePoint is a great new addition and I hope to see some new crazy tools being developed (production dead-code analyzer, deprecation code path finder anyone?)

To end, this short post, here is an interesting quote from Chad Fowler Berliner by adoption:

Abstractions are expensive. The cost increases exponentially as you add them to a codebase. Chad Fowler

Turns out it wasn’t that hard, but the lack of documentation made me lost a couple hours. I therefore updated OmniAuth’s wiki and wrote this quick post so hopefully you won’t waste time looking for simple details.

Requirements

You actually only need to add 2 gems to your Gemfile:

1
2

gem 'omniauth-openid'
gem 'ruby-openid-apps-discovery'

Now, the second gem is the one I didn’t know about. The gem is actually provided by Google itself. It turns out, Google Apps use a custom discovery protocol. They monkey patched the popular OpenID Ruby libraries so you can just drop in their gem and their discovery system will magically work.

Setup

You need to require 4 files to get everything setup properly:

1
2
3
4

require 'omniauth-openid'
require 'openid'
require 'openid/store/filesystem'
require 'gapps_openid'

The first one is the omniauth extension for OpenID, the second one is the main Ruby OpenID library (needed so we can set our SSL cert). The third one allows us to store temporary data on disk instead of keeping it in memory (optional). And finally, the last one is Google’s magical gem to get their discovery system working.

Because we are going to communicate via SSL, we want to make sure that the OpenID library uses our certs to verify the SSL communications:

1

OpenID.fetcher.ca_file = "/absolute/path/to/ssl_cacert.pem"

(Obviously, you need to change the path to your own cert)

We are almost done with the setup, we just need two more things:

• make sure you are using a session.
• setup OmniAuth

I’m using Sinatra, so I’ll load the Rack::Session middleware before I set Omniauth:

1

use Rack::Session::Cookie, :secret => 'supers3cr3t'

(Rails turns that option by default, so you don’t need to worry about it)

Then I can finally setup OmniAuth:

1
2
3
4
5

use OmniAuth::Builder do
  provider :open_id,  :name => 'admin',
                      :identifier => 'https://www.google.com/accounts/o8/site-xrds?hd=aimonetti.net',
                      :store => OpenID::Store::Filesystem.new('/tmp')
end

There are two important things to notice. First, because I set the provider’s name to be ‘admin’, the magical paths provided by OmiAuth will use that name (/auth/admin). Secondly, and more importantly, notice how I added the name of my Google Apps domain at the end of the identifier:

1

'https://www.google.com/accounts/o8/site-xrds?hd=' + your_domain_name

Sinatra

In Sinatra, your just need to define the routes OmniAuth would use (same goes for Rails, just use the router for that).

By default, omniauth now offers you a /auth/admin endpoint that will push the user through Google Apps authentication. Once the authentication is over, the user will be redirected to the following endpoint:

1
2
3
4
5
6

# Callback URL used when the authentication is done
post '/auth/admin/callback' do
  auth_details = request.env['omniauth.auth']
  session[:email] = auth_details.info['email']
  redirect '/auth/admin/welcome'
end

You can access the authentication details from request.env['omniauth.auth'] and redirect the user to another page, like the admin landing page for instance.

1
2
3
4
5
6
7

get '/auth/admin/welcome' do
  if session[:email]
    erb :welcome_boss
  else
    redirect '/auth/admin'
  end
end

On the landing page, you need to verify that the user is logged in, in this case, during the previous step, we added the email of the user to her session. We can therefore verify the presence of that information to confirm the authentication status. If the user is authenticated, then we’ll render an ERB template otherwise we’ll redirect her back to the login page.

We should also provide an endpoint in case the authentication failed:

1
2
3
4

get '/auth/failure' do
  params[:message]
  # do whatever you want here.
end

Note that by default, in dev mode, Omniauth won’t redirect the user there. To enable this behavior, use the following snippet (works with any rack app, Rails, Sinatra or whatever):

1
2
3

OmniAuth.config.on_failure = Proc.new { |env|
  OmniAuth::FailureEndpoint.new(env).redirect_to_failure
}

Conclusion

Using Google Apps for authentication with OmniAuth is trivial as long you know two things:

• the identifier url: 'https://www.google.com/accounts/o8/site-xrds?hd=' + your_domain_name
• Google’s discovery service gem

This blog post was written using omniauth 1.1.1, omniauth-openid 1.0.1, rack-openid 1.3.1 and ruby-openid-apps-discovery 1.2.0. This might not apply to you if you come from the future :)

Today we are going to look at Google’s Go language. Go approaches concurrency from an interesting view point. But instead of digging into the history and reasoning which led to this approach, I’d like to show you the language constructs by actually writing real life code.

Fetching web resources concurrently

The following example is taken from my recent presentation Ruby vs. the World. I explored a few programming languages and showed how they changed my Ruby.

To show how Go addresses concurrency, I decided to build a program which would concurrently fetch various web resources, wait for all of them to be fetched, then process them all at once. In other programming languages, we could have used threads) and a semaphore), actors or callbacks. Go’s approach is slightly different, let’s walk through the code together.

The first part of our code gets us setup:

1
2
3
4
5
6
7
8
9
10
11
12
13

package main

import (
  "fmt"
  "net/http"
  "time"
)

var urls = []string{
  "http://www.rubyconf.com/",
  "http://golang.org/",
  "http://matt.aimonetti.net/",
}

The code above names our package then imports a few standard libraries that we are going to need. It then defines an array/slice of strings representing the urls we are going to fetch.

Next we define a type we will use a bit later:

1
2
3
4
5

type HttpResponse struct {
  url      string
  response *http.Response
  err      error
}

You can think of a struct type as a simple representation of a class. Technically, we are defining a structure with some typed attributes. We will later on, create instances of this defined type.

Go implements OOP slightly differently than other languages.

Methods in Go are more general than in C++, Java: they can be defined for any sort of data, even built-in types such as plain, “unboxed” integers. They are not restricted to structs (classes).

We can therefore define methods/functions for any type of data, including “any/all” types. This approach to types is called structural typing.

Here is the code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

func asyncHttpGets(urls []string) []*HttpResponse {
  ch := make(chan *HttpResponse)
  responses := []*HttpResponse{}
  for _, url := range urls {
      go func(url string) {
          fmt.Printf("Fetching %s \n", url)
          resp, err := http.Get(url)
          ch <- &HttpResponse{url, resp, err}
      }(url)
  }

  for {
      select {
      case r := <-ch:
          fmt.Printf("%s was fetched\n", r.url)
          responses = append(responses, r)
          if len(responses) == len(urls) {
              return responses
          }
      case <-time.After(50 * time.Millisecond):
          fmt.Printf(".")
      }
  }
  return responses
}

This is the meat of our application. And there is quite a lot of going on in just a few lines. Assuming you aren’t familiar with Go, I’ll walk though the code.

Let’s start with the signature:

• the function is named asyncHttpGets
• it takes an argument namedurls which is an “array” of strings (I used quotes around the word array because it’s technically what Go calls a slice)
• it returns an “array” of HttpResponse pointers

Then in the function body:

We start by creating an instance of a channel and assigning it to the ch variable name. Think of a channel as a pipe like in unix. We can write to and read from that channel.

In the next line we create an empty instance of a slice containing pointers to HttpResponse objects.

Then, using the for range language construct, we iterate through our urls, storing the current value being used into the scoped variable url. The url is then available within the block/lambda/closure marked by the curly braces.

Now this is where the async construct comes in. Using the go keyword, we define an anonymous function that takes a string argument representing a url.

The function prints this string, then uses the net/http library to fetch the web resource. We use the returned data to create an instance of our HttpResponse type and send it to the channel.

This part gets a bit confusing because I reused the name url. We call this anonymous function right away passing it the url variable set by the loop.

You might wonder why we bother to create an anonymous function and call it right away instead of just executing the code directly. The go keyword executes the code that is passed in as a goroutine which is well explained here

A goroutine is a function executing concurrently with other goroutines in the same address space. It is lightweight, costing little more than the allocation of stack space. And the stacks start small, so they are cheap, and grow by allocating (and freeing) heap storage as required.

In other words, you start a goroutine and you let the “system” handle how it wants to deal with the low level details. Technically, goroutines might run in one or multiple threads, but you don’t need to know. We trigger each http fetch in a separate goroutine and then each response is pushed down the channel.

The second block of code begins with another for loop containing a switch/case statement. The case statement checks if something is in the channel. If there is something, we…

• allocate the data to the r variable
• print the resource’s url
• append the resource to the slice we created at the beginning of the function.

If the length of the array is the same as the length of all urls we want to fetch, we are done fetching all our resources and can return. While still waiting for responses, we print a dot every 50ms.

Update: In the first version of this blog post I had used a ‘default’ case statement to print the dot and sleep for 50ms so the loop wouldn’t be too tight and the concurrency effect was more obvious. But some HN comments pointed out that it wasn’t needed and I shouldn’t block. For reference here is what I had before (don’t use this code):

1
2
3
4
5
6
7
8
9
10
11
12
13

for {
  select {
  case r := <-ch:
      fmt.Printf("%s was fetched\n", r.url)
      responses = append(responses, r)
      if len(responses) == len(urls) {
          return responses
      }
  default:
      fmt.Printf(".")
      time.Sleep(50 * time.Millisecond)
  }
}

Thank you HackerNews.

With that code constructed, our main can make use of it like this:

1
2
3
4
5
6
7

func main() {
  results := asyncHttpGets(urls)
  for _, result := range results {
      fmt.Printf("%s status: %s\n", result.url,
                 result.response.Status)
  }
}

Compiling and running the code look like this:

1
2
3
4
5
6
7
8
9
10

$ go build concurrency_example.go && ./a.out
.Fetching http://www.rubyconf.com/
Fetching http://golang.org/
Fetching http://matt.aimonetti.net/
.....http://golang.org/ was fetched
.......http://www.rubyconf.com/ was fetched
.http://matt.aimonetti.net/ was fetched
http://golang.org/ status: 200 OK
http://www.rubyconf.com/ status: 200 OK
http://matt.aimonetti.net/ status: 200 OK

As you can see from the print statements, the 3 urls are triggered in a sequential way, but the responses come back in different orders due to different server latencies and response transfer time.

Conclusion

Go was designed to make concurrency easy for developers. It is a very well documented language and you will find on this page a lot of information about its concurrency philosophy and details about each available constructs works.

I like that the language is very simple and the constructs explicit. If you want to write concurrent code, Go pushes you to do it in a specific style. That style is clear and comfortable for me. My code stays simple, I don’t go crazy with callbacks, and the conventions make it simple for everyone else to understand my code.

Whether or not Go appeals to you stylistically, but clearly the designers stayed close to the goal of developing to a 21st century C with a special focus on concurrency with the unix approach.

• Now is an awesome time to be a software engineer (probably the best time ever).
• Your job might not be well suited for you.

I’ll show you why we are lucky bastards, why we aren’t so good at picking the right jobs and some hints on how to solve this issue.

I remember a family friend telling me when I was a kid that computers are going to be the future and that there will be a lot of jobs in this field. I also remember that the idea of sitting all day, alone, in front of a minitel-like computer scared the hell out of me. But he was right and I now work from home, spending 12+ hours in front of a monitor. I get emails and phone calls from many people reaching out to me to help them find software engineers.

At least three things make “now” the best time to be a software engineer:

• demand
• projects
• prestige

Good time

There is a huge demand for engineers. There are many more job openings than candidates. But this is much better than in late 90s/early 2000 when anyone who could write a line of HTML would get a job. Now the projects people are building are way more interesting and have a real potential to change lives. Before, you had to work for a giant company to have a chance to do that. But now, with internet and smart mobile devices everywhere, almost any startup (fancy name for small company) can have a huge impact – look at Twitter for instance.

Lastly, being a geek is cool. Movies, cartoons, TV Shows now have geek heroes (granted they usually don’t represent real geeks, but hey it’s better than nothing).

The problem

Most software engineers I know, are really bad at choosing the right job for themselves. They don’t design a career. Engineers are good at solving technical problems in an objective way, but when it comes to our jobs and future, we seem to struggle.

Why?

I’m not an expert but I have a few guesses I’d like to share with you:

• We don’t know our real worth.
• We don’t make long term plans.
• We get paid well, so why bother changing?
• Changing job feels like betrayal.
• The system is broken.

Self worth

Most modern companies need solid engineers to be relevant in the short/medium term and they know it. Most engineers have no idea how their talent and dedication converts into real business value. Without that appreciation, they can’t easily estimate how much they are worth. The salary scale for software engineers is dramatically different from other jobs/industries. To your business, you might be worth twice or three times the salary of someone like a teacher. This is probably not fair because of the social value a teacher offers, but that’s the way the law of demand works in our society. Knowing how much you are worth to a company and how much to ask is critical to properly negotiate or renegotiate a contract.

I remember when I moved to America and was thankful to have a job. I had no idea that at $45k/year with basically no health coverage and no vacation, I was grossly underpaid and could have been paid twice that amount down the street. (Note: the average software engineer salary in the US is at $89,000 according to indeed.com). The average salary for a high school teacher is $47k/year so I wasn’t complaining. As a matter of fact, I didn’t leave this job because of the salary. I truly believe that “Money doesn’t buy happiness” and it shouldn’t be your primary reason to accept or leave a job. Money is nice and often makes life easier. But the point is that you have to understand how much you’re worth, so you can get paid and focus on your work.

Defining a vector

John Allspaw wrote a great blog post about what it means to be a senior engineer. I strongly recommend you read it. I often look back at it and pick up a couple points I need to focus on myself. John wrote a book which is a collection of essays and interviews regarding tech/web ops.

Here is a very interesting quote:

Not everyone can be senior. If, after five years, you are senior, are you at the peak of your game? After five more years will you not have accrued more invaluable experience? What then? “Super engineer”? Five more years? “Super-duper engineer.” I blame the youth of our discipline for this affliction. […] Given the dynamics of our industry many elected to move on to managerial positions or risk an entrepreneurial run at things.”

There are two very strong points in this quote:

• we don’t quite know what it means to be a senior engineer (and John’s post does a great job explaining his take on that).
• many of us end up in managerial positions or leading startups.

I might be a bit radical – but the day I stop learning/improving will be the day that I will quit, change jobs or careers. John’s post has great pointers to help us improve our skills. But the question I’m trying to raise is:

what do we want from our career?

“Career” sounds like a dirty word to many of us. It has a corporate, sleazy, back stabbing connotation. When I hear it, I picture a cliché stock photography of a bunch of smiling people wearing 80’s suits. In the context on this post, let’s take the Oxford English Dictionary definition: “course or progress through life (or a distinct portion of life)”. The word comes from from French via the Old Occitan word: “carriera” which means “street”.

A better word for career might be “path”.

I think we have a hard time knowing what kind of path we want to be on. When faced with the question, a lot of us answer: “solving problems”, “having fun”, “changing the world”. All these answers sound good, but they aren’t paths, they are just attributes.

I have to admit that I’m still struggling with this question and probably will for a while. I’m pretty good at defining short term goals but I have a hard time seeing the long term path. As a matter of fact, a little while back I was seated in front of Chad Fowler in his office in Washington, DC. Chad wrote a great book called “The Passionate Programmer: Creating a Remarkable Career in Software Development”.

I’ve read Chad’s book (which inspired me in many ways) and have known Chad for years. The point of our meeting was to decide what I was going to work on next. Unconsciously, I expected Chad to just tell me what I should be doing. I trusted him to pick the right “path” for me. But I was surprised when Chad told me: “you’re the kind of engineer who can do anything. You’re a generalist who can pick a topic and become a specialist. So what do you want to do?”

I wasn’t sure how to take it, it sounded like a compliment but, at the same time, the fact that Chad didn’t have a solution to my problem bothered me. I remember thinking, wait, he’s the expert and he’s deflecting the situation by asking me the question I came to ask him. Sure, the compliment was nice but that wouldn’t solve anything. What does that mean about me? If an expert can’t figure out what I should do, I might be screwed.

Then on my way back home, I realized that it didn’t matter how well Chad knew me, he couldn’t guess what I even didn’t know about myself.

My long term happiness depends on me finding a direction I want my professional life to take. In Chad’s book, there is a strong focus on finding a market, understanding it, developing skills and marketing yourself. However there was something I had missed.

The goal-oriented, destination-focused thinking that you usually do leads only from one goal to the next. It has no logical end. What most of us fail to realize is that the path is the end.

I’ve always known that the journey is more important than the destination in itself, but what I had missed is that you still need to define a destination or maybe more precisely a direction, a vector. My problem is that my path was just a bunch of scattered dots. Hopping from one to the other, I was hoping it was going to make a pretty drawing. The challenge is when I got to a spot, I was stuck not knowing what to do next. I ended up picking another short term goal/destination based on the opportunities available at that time.

What I should do instead was to define a general direction and then learn through the process. I believe this will help me enjoy my job more than running after goals. It will allow me to see the world differently and will help me make the right career choices when the time is right. To be honest, I think that’s the only way I can build endurance and not burn out in 5 years. That said, I’ll still have goals, deadlines and the usual – but they won’t define my own personal progress.

Why bother?

Changing jobs is a pain. As an engineer I weigh the pros and cons and try to logically pick the right choice – at least in theory. In practice I avoid dealing with questions that might result in challenging consequences.

Quiting a job is tough. You have to tell your current employer and your colleagues that you are leaving them for something you think is better for you. The nicer the people you work with, the harder it is. The better you are paid, the harder it is. If you work with nice people and you’re well paid, leaving is really hard (take note if you run a team).

In our profession, changing jobs isn’t usually seen as something bad. Recruiters might pressure you to take a new, better job. Beware impulsive changes though. Recruiters want their commissions so they’ll do anything they can do make you switch. They’ll try to convince you that the grass is greener on the other side. Maybe that’s only “bad” recruiters.

There are some recruiters who care about people and companies. Recruiters who will help you find the right job for you. However, they won’t be able to help you if you don’t know what direction you want to go to.

I remember being stuck in a pretty terrible work environment, being underpaid and the projects I was working on weren’t going anywhere. A friend gave me a Seth Godin’s book called “The Dip: A Little Book That Teaches You When to Quit (and When to Stick)”.

I’m not a big fan of self-help/business books, but this book raised a very good and simple question: how do the efforts compare to the returns? Which situation are you in:

Think about it. Will the effort you put into your work pay off? If you don’t think it will, then you should quit right away.

The logic is pretty simple but requires you to forecast. For that you need some sort of metrics helping you to see if you are getting closer or further from the direction you set for yourself.

Trust

Trust is the key element of any relationship. In The Five Dysfunctions of a Team, Patrick Lencioni explains that the base of management dysfunctions is absence of trust:

Turns out it’s the same thing for our careers. We need a team of people to help us define a vision/direction and keep us honest and accountable.

Find people who you can trust to talk to about your professional goals, your progress, failures and doubts. People who will be honest with you and tell you things you might not want to hear. Find mentors and honest people. These people don’t have to be working in the industry. They just have to be able to listen and care.

People like that are extremely hard to find, but so are good executives. I believe that having trustworthy friends (partners/family members..) who care is a big part of what makes someone successful.

My small contribution

As I explained earlier, I’m no expert and I also struggle with the issues I described. However, I’d be glad to provide a bit of my free time to help you think through these issues.

A lot of you are doing a great job without the rest of us noticing. If you don’t have a popular twitter account, blog, open source projects, published books or given talks at conferences, it might be hard to get yourself noticed or even know how much you’re worth. Trust is a big deal and if you are considering moving on, you probably don’t want your boss and your colleagues to know. You probably also don’t know good recruiters you can trust. You might not even be sure it’s worth investing too much time.

Most of you probably won’t consider it, but I’d like to offer my help if you’d like it. I promise full anonymity and no strings attached. Just

• email me
• tell me about yourself and what you do
• what direction you’d like your career to take
• ask any questions you might have

After exchanging a few emails , I’ll try to make good use of my network to find you a way to move in your desired direction. Or, if you work for an interesting company with current openings, feel free to contact me too.

Update: My inbox is overflowing with emails and I already spent literally days replying to as many people as possible. I’m sorry but at this time I can’t reply to any new enquiries. I’ll write a follow up blog post that covers what I learned from my interactions with so many readers.

I have no idea how this will turn out. Maybe I’ll get a couple of emails, zero, or way too many to handle – but it’s worth a try. I do have a full time job, so please don’t expect me to reply to your emails within the hour.

I finally took the time to enable the comments in this blog. Feel free to leave advice or feedback.

Description of the talk:

Ruby is an awesome programming language, it’s so pleasing you probably haven’t seriously looked at other languages since you switched. The programming world is evolving fast, new languages are created daily, new trends are emerging. Let’s take some time to look at a few languages from a Ruby developer perspective.

Slides

The slides are available on [Matt’s SpeakerDeck page]https://speakerdeck.com/matt_aimonetti/ruby-vs-the-world) and can be downloaded here.

Video

Presentation website

Matt’s presentation was filmed by Confreaks and posted here.

For more RubyConf 2012 talks, go there.

Description of the talk:

mruby is Matz’ new Ruby implementation, it’s not cooler than node.js, it doesn’t natively support Hypstermedia, it looks just like the good old Ruby. So why should we, as a community care?

Matt’s talk is divided in two parts, an introduction of mruby (embedded Ruby) and and revisiting Ruby.

Slides

The slides are available on [Matt’s SpeakerDeck page]https://speakerdeck.com/matt_aimonetti/mmmm-dot-mruby-everywhere-and-revisiting-ruby) and can be downloaded here.

Video

Presentation website

Matt’s presentation was filmed by Confreaks and posted here.

Description of the talk:

Programming languages affect the way one looks and solves problems. But comparing programming languages isn’t as simple as drawing a table comparing features.

In his talk, Matt shows what he likes, dislikes, the philosophy and concrete example of how to use 7 programming languages:

• Ruby
• JavaScript
• CoffeeScript
• Objective-C
• Clojure
• Scala
• Go

Slides

The slides are available on Matt’s SpeakerDeck and can be downloaded here.

Video

TBD

Instead of arguing the pros and cons of OOP vs FP and how each of the previously mentioned languages handle being OOP and FP, I’d like to introduce a very powerful Scala idiom: pattern matching. Note that pattern matching isn’t something Scala invented nor that it only exists in Scala. Pattern matching can be achieved many different ways. However, the majority of the popular languages don’t put this concept at the center of their language. A few languages way before Scala rested heavily on pattern matching such as Erlang, Haskell but that’s a different story. How does Scala offers Pattern Matching, what is it and finally why is it valuable?

Scala pattern matching by examples

As its name indicates, pattern matching is used to detect patterns. Here is an example that covers a few interesting cases:

1
2
3
4
5
6
7

def listAnalysis(list: List[Any]) = list match {
   case Nil => "empty"
   case 'a' :: tail => "starting by 'a'"
   case (head:Int) :: _ if head > 3 => "starting by an int greater than 3"
   case (head:Int) :: _ => "starting by an int"
   case _ => "whatever"
}

If you’ve never seen any Scala that probably looks like gibberish to you. Let me break it down:

1

def listAnalysis(list: List[Any]) = list match {}

I define a new function called listAnalysis which takes an argument named list which is of type List (this list could contain any kind of elements). The implementation of this function is a pattern match on the list argument. The body of this ‘pattern match’ looks like a classical switch statement. But it’s actually much more than a simple switch statement. Surely it could be used like one, but as we will see, it can do much more.

Note that you can apply a pattern match against more than one object at once as shown a bit later.

Let’s look at the statements inside the function.

1

case Nil => "empty"

In this case we are checking that the list is empty or nil. If that’s the case, the statement on the other side of the “fat arrow” is executed. In this case, we return a string but we could have called another function or so whatever.

Now the second statement is much more complex and much more powerful:

1

case 'a' :: tail => "starting by 'a'"

Remember that we are doing pattern matching against our list object. What we are doing here is use the :: operator (aka cons operator) to extract the head and the rest of the list and then we match the head against the 'a' character.

This statement could have been written different ways:

1

case 'a' :: rest => "starting by 'a'"

In that case we named the tail of the list rest, but really we don’t care how it’s called or its value, so the sensitive thing to do is to rewrite that statement like that:

1

case 'a' :: _ => "starting by 'a'"

This basically says we are looking for a list that starts by 'a' (and we don’t care about the rest).

Another statement:

1

case (head:Int) :: _ => "starting by an int"

In this case we type match the first element of the list and check that we have an integer. Note that using the cons operator in the match cases doesn’t seem to affect performance. It would seem that at compilation, the statement are rewritten to avoid creating uneeded objects (List also implements structural sharing of the tail list). I’m not a Scala expert so someone with more experience might be able to confirm/clarify.

Now let’s look at a variant of this statement:

1

case (head:Int) :: _ if head > 3 => "starting by an int greater than 3"

This is the same statement as above, but we are adding an extra condition after the match. This is quite useful when simple matching doesn’t cut it.

Finally we have a fallback statement:

1

case _ => "whatever"

For more information about the cons operator, read about the extractor objects and what they can do.

Here is the result of calling our function with different lists:

1
2
3
4
5
6

listAnalysis(List())                             //> java.lang.String = empty
listAnalysis("This is a test".toList)            //> java.lang.String = whatever
listAnalysis("abcde".toList)                     //> java.lang.String = starting by 'a'
listAnalysis(List(1,2,3))                        //> java.lang.String = starting by an int
listAnalysis(List(42,24,36))                     //> java.lang.String = starting by an int greater than 3
listAnalysis("a".toList)                         //> java.lang.String = starting by 'a'

Here is another example using 2 items for the match:

1
2
3
4
5
6
7
8
9
10
11
12
13

def doubleMatch(foo: Any, bar: Any) = (foo, bar) match {
  case ('a', 'b') => "a and b"
  case (1, 'b') => "1 and b"
  case (1, _) => "1 and "+ bar
  case (a:Float, _) => "foo float"
  case _ => "unknown case"
}

doubleMatch(1, "test")                           //> java.lang.String = 1 and test
doubleMatch(1, 'b')                              //> java.lang.String = 1 and b
doubleMatch(42, Nil)                             //> java.lang.String = unknown case
doubleMatch('a', 'b')                            //> java.lang.String = a and b
doubleMatch(4.2f, 42)                            //> java.lang.String = foo float

Why is pattern matching valuable?

In short, pattern matching allows the developer to deconstruct a structure to find specific elements, in other words the pattern, needed to then constuct an object/structure or trigger a function.

It’s the opposite process of calling a method on an object. Here we start from a structure (instead of the instance of an object), this structure is just a basic struct and based on a found pattern, we then trigger a function (with access to the data if we need it). When you have a stable and known data structure, it’s often very interesting to use the pattern matching approach because you can easily expand the operations you can execute. However, if your operations are stable but the data changes, then the Object Oriented approach seems more adequate.

Besides that, pattern matching will often make your code clearer than using if/else statements. Especially in a language like Scala where you can define pattern matching function within a function and you can also pass pattern matching functions around. Like eveything else, it needs to be used with caution so the intend of the code is still understandable. That said it’s a great tool to have handy and I’ve had a lot of fun rewriting my newbie Scala code using a more idiomatic approach based on pattern matching.

I hope you enjoyed this quick introduction. You can read more about pattern matching in Scala in the following articles: (note: Ikai’s post on how he uses regexps with pattern matching is a fun read.)

• http://www.scala-lang.org/node/120
• http://pragprog.com/magazines/2012-03/scala-for-the-intrigued
• http://kerflyn.wordpress.com/2011/02/14/playing-with-scalas-pattern-matching/
• http://ikaisays.com/2009/04/04/using-pattern-matching-with-regular-expressions-in-scala/
• http://www.artima.com/scalazine/articles/pattern_matching.html

An example of something that might seem simple is inheritance. Ruby, unlike C++, doesn’t support multiple inheritance. What that means is that a Ruby class can only have 1 parent class (superclass). However multiple inheritance can be achieved via modules used as a mixins. That’s a very common pattern, people put some code in a module and then mix it in/include it in a bunch of classes. The problem I see though, is that people abuse these concepts and don’t respect the difference between a class, a module and a module used a mixin.

The Class

Object Oriented Programming 101:

“In object-oriented programming, a class is a construct that is used to create instances of itself – referred to as class instances, class objects, instance objects or simply objects. A class defines constituent members which enable its instances to have state and behavior.”

So if you create a class and you don’t create instances, you are using the wrong construct. Here is an example of what I often see:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

class Settings

  @settings = {}
  def self.all
    @settings
  end

  def self.[](key)
    all[key]
  end

  def self.[]=(key, value)
    all[key] = value
  end

end

Which can be used as such:

1
2
3

Settings[:secret] = 42 * Math::PI * Time.now.to_f
p Settings[:secret]
# => 177243152913.2707

(granted this isn’t a great example since we could have used a subclass of Hash but just bear with me)

Actually, the developer who wrote the code above would probably also use some Ruby magic like method_missing to provide a more laxed API and allow for “nicer” getters such as Settings.secret and Settings['secret']. I have my own thoughts on the topic but it’s an entirely different subject.

Note also that the way class level methods are defined can also be different depending on who wrote the code, you might see the following variations (and other more esoteric ones):

1
2
3
4
5
6
7
8
9
10

class Settings

  def Settings.all; end

  # or
  class << self
    def all; end
  end

end

The Settings code above works, the code is simple, yet I will argue one thing: it’s an abuse of the class construct. We’re breaking the #1 rule of classes: “create instances of self”.

It’s easy, whenever you don’t create instances of a class, please don’t use a class.

That’s also true for slightly different examples such as:

1
2
3
4
5
6
7

class API

  def fetch(id)
    HTTP.get('http://matt.aimonetti.net/article/:id', :id => id)
  end

end

1

resource = API.new.fetch(42)

There is no need whatsoever to create an instance of API, using a class is picking the wrong construct. Also, I don’t care if you use the Singleton module to only allow 1 instance of the class, you still shouldn’t use a class in the above example.

The Module

In Ruby’s object hierarchy, the Class object actually inherits from the Module object.

1
2
3

ruby -v -e "p Class.ancestors"
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin11.3.0]
[Class, Module, Object, Kernel, BasicObject]

As per Ruby’s source code defintion:

“A Module is a collection of methods and constants.”

I like to think of modules as namespaced methods and constants. Whenever you want code that logically belongs together but that won’t require that you create instances of a ‘model’, then a module is the right construct to use.

As a matter of fact, the two examples above are great cases where a module should have been used.

The confusing bit is that modules can have module level methods but also instance level methods. Here is an example:

1
2
3
4
5
6
7

module API

  def self.fetch(id)
    HTTP.get('http://matt.aimonetti.net/article/:id', :id => id)
  end

end

This is a module level function, it could also be written like that:

1
2
3
4
5
6
7
8
9

module API

  module_function

  def fetch(id)
    HTTP.get('http://matt.aimonetti.net/article/:id', :id => id)
  end

end

And be used like that:

1

resource = API.fetch(42)

Until now, it makes sense. The weird thing is that even though, modules unlike classes aren’t meant to create instances, we have the possibility to define module instance methods.

1
2
3
4
5
6
7
8
9
10
11
12
13

module Settings

  DATA = {repo: 'http://github.com/mattetti'}

  def repository
    DATA[:repo]
  end

  def secret_key
    DATA[:key] ||= 42*Math::PI
  end

end

Great, but we can’t actually use these methods since they are instance methods and we don’t create instances of modules. Well, that isn’t quite true, there is a way to access them and that’s by using a module as a mixin. What that means is that we inject/copy the module code inside a class or an(other) object. Example in code:

1
2
3
4

a = Object.new
a.extend(Settings)
a.repository
# => "http://github.com/mattetti"

Or we can add the code to a class so the instances of this class can access our module instance methods:

1
2
3
4
5
6

class Foo
  include Settings
end

Foo.new.repository
# => "http://github.com/mattetti" 

The mixin modules

There a plenty of resources online about the many ways to use mixins in Ruby to achieve multiple inheritance and do cool stuff. But the point of this article is to try to demonstrate that mixins shouldn’t be abused.

My problem with the above example is that by mixing in the Settings module inside our Foo class, we created an uneeded, confusing extra level of abstraction. Instances of Foo now have access to two methods/objects: repository and secret_key. These methods or the objects they refer to don’t belong to the Foo class, but it seems convenient to not have to type Settings.repository so we mixed things in. Plus, a lot of Ruby developers seem to dislike adding class/module level methods so they feel that this approach ‘feels better’.

Here is the thing, the convenience of typing a few less characters isn’t worth it. Next time you or someone else will look at an instance of the Foo class calling repository, finding where it is defined is going to be a pain. That’s especially true if you have many mixins in your class. Settings will also probably grow and you will end up with a bunch of methods that have nothing to do with your class instances. In this case, I will call the use of a mixin, an abuse of construct. Sure, Ruby allows you to do it, but that doesn’t mean it’s the right thing to do. In Ruby, unlike in Python, there are 101 ways to do a simple thing. It doesn’t mean that the 101 ways are good, it just means that Matz wasn’t sure how people would use his programming language and chose to give us more freedom to messup/doing it our own way.

When to use mixins?

I have my own rule: use mixins whenever you need to share behaviors between different classes.

In the above example, we weren’t sharing behaviors, we were sharing objects, there was no need to actually use a mixin.

That said, rules aren’t rules without exceptions. A good example of this exception would be the Math module from the standard library. This module offers trigonometric and transcendental functions. You might think that this module would be designed to be a mixin so you can get log, cos, exp and friends available in your math related classes. It turns out, all Math’s methods are defined a module functions meaning that they are meant to be called from the Math module directly.

However, Ruby allows you to mixin module functions, but these functions become private. If you do include the module inside your class, your instance methods will be able to call hypot(x,y) directly, but these methods won’t be available from the outside (Foo.new.log(42) would raise an exception).

To conclude with mixins: mixins are great but don’t abuse them or you will endup with so much abstraction that your coworkers will secretely call you Kandinsky. Stick to simple mixins allowing you to share behaviors between at least a couple classes. See DataMapper for a great way to use mixins.

Modules: your secret functional programming weapon

I have to say that I do like functional programming. The idea of having functions not mutating the states of things around them pleases me. It just seems clean, you feed data to a function and you get another piece of data. No states were changed, maybe some temporary variables were allocated to process the data, but the only thing that matters is the input and the output. Easy to grasp, easy to follow, no magical states being changed by some code fairies.

The good news is that Ruby allows us to write code like that. And this is where modules are great. Very much like the Math module we discussed above, there are many cases where you want to have a bunch of functions that process an input and provide an output without keeping any states. A good example of such a module would be a param verification filter. The filter takes an input, takes some rules and verifies that the input matches the rules. Surely, we could create an instance for each verification, this would allow to keep states in our class and do the usual OOP things. But we could also simply use a module with a bunch of module (level) functions that would pass to each other the input they need to not need to keep states. The end result will be faster, nicer on the GC and easy to follow.

Mixing OOP and functional programming isn’t new, ask Scala developers! If done right, by adopting this approach we can simply our code base, make it faster, easier to maintain and not losing the chance to also use OOP paradigms when needed.

Compromise

As shown earlier, modules and classes have pros and cons. Classes are however much more natural to use for Object Oriented Programming. A compromise about the Settings examples was suggested by Evan Phoenix. The solution is elegant and simple. Use a class and an instance. Here is an implementation based on his suggestion:

1
2
3
4
5
6
7
8

class AppSettings < Hash

  def custom_method
  end

end

SETTINGS = AppSettings.new

The point here is not about the class implementation but that fact that we use a class and associate an instance of this class to a constant so it can be shared all over the place. Wow, a constant, this is so nasty you might think. Classes are constants too and so are modules, here we just allocate an instance of a class to a constant. Surpisingly simple and efficient.

They both create a shallow copy of an object (meaning that they don’t copy the objects that might be referenced within the copied object). However, #clone does two things that #dup doesn’t:

• copy the singleton class of the copied object
• maintain the frozen status of the copied object

Examples of the singleton methods not being copied.

dup:

1
2
3
4
5
6
7

a = Object.new
def a.foo; :foo end
p a.foo
# => :foo
b = a.dup
p b.foo
# => undefined method `foo' for #<Object:0x007f8bc395ff00> (NoMethodError)

vs clone:

1
2
3
4
5
6
7

a = Object.new
def a.foo; :foo end
p a.foo
# => :foo
b = a.clone
p b.foo
# => :foo

Frozen state:

1
2
3
4
5
6
7
8
9
10

a = Object.new
a.freeze
p a.frozen?
# => true
b = a.dup
p b.frozen?
# => false
c = a.clone
p c.frozen?
# => true

Looking at the Rubinius source code makes the difference extremely obvious.

Because of the extra steps, clone is a bit slower than dup but that’s probably not what will make your app too slow.

Just a quick note about shallow copies (true for clone and dupe). Notice how the array referenced by the bar attribute doesn’t get copied but shared between the original and the copied instances:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

class Foo
  attr_accessor :bar
  def initialize
    self.bar = [1,2,3]
  end
end

a = Foo.new
b = a.clone
p a.bar
# => [1, 2, 3]
p b.bar
# => [1, 2, 3]
a.bar.clear # clearing the array #bar points to
p a.bar
# => []
p b.bar
# => []

Both objects, a and b share the same reference to the array instance created when a is initiated. There are a few ways to do a deep copy of an object, but that’s a different matter.

What has changed in the last few years is that web applications start providing more than just dynamically rendered HTML templates. JavaScript is used to run more and more logic on the client side and JS usually talks to some backends via JSON. But web APIs are also more and more used to expose raw data to other software. The challenge though, is that we haven’t changed the way we write web applications to adapt to this new way of providing data. In this article, I’d like to explore why and how we need to rethink web API development and focus on communication and interaction.

Unlearning

Many are used to doing certain things a certain way and it’s really hard to unlearn old habits. The more concerning part of this fact is that it makes it hard for someone to step back and evaluate honestly if a technical decision is due to comfort or if it’s truly the right choice to achieve a given goal. Ruby on Rails revolutionized the way we wrote web applications almost 10 years ago and since then, many web frameworks have adopted the Rails philosophy. As a matter of fact, I personally think that when it comes to writing front end applications, Rails is still one if not the best web frameworks out there. But in its current state, I’m far from convinced that it’s a great tool to write web APIs mainly because it was not designed for that and because it comes with a lot of baggage.

Focusing on the API

Let’s take a step back and consider what is critical when developing a web API:

• Documentation for end users to know how to consume your services.
• Consistent and reliable output so you don’t break client applications relying on your services.
• Maintainability

Note that I didn’t mention performance because I consider performance almost always being important. I also didn’t mention the whole REST/RPC/Hypstermedia debate since I consider it being an implementation detail and a totally orthogonal discussion. But for the record, I personally don’t like solutions forcing you into a specific way of providing your data (I’m looking at you webmachine).

Documentation

Documentation isn’t as important when you consume your own APIs because you can look at your source code. However it gets much more tricky when you start working with other teams. They might not know the language/framework you use. They might not have time to go dig into your source code to figure out what your meta-magical piece of clever code does.

Lately more and more applications provide their raw data to the outside world via web APIs. The developers who will consume your resources don’t have access to your source code, probably don’t use the same programming language and don’t have much time to guess how your API work. Also your test suite won’t help communicate how your API works so we need to find a different approach. Also note that in this scenario, TDD/BDD won’t help us communicate better and tests can’t be used as documentation since your tests aren’t exposed.

Documentation is your #1 way to communicate with your human audience. Communication is key and even if as engineers we love to focus on code, if we can’t communicate about it, end users will have a hard time using our code and might just not even do it. The key is to communicate what your API does, why someone might want to use it and how to use it.

Consistent and reliable output

This point seems obvious but what we realize in reality is that for many, API’s consistency and reliability doesn’t include documentation. You find a lot of APIs out there poorly documented or out of sync with the actual implementation.

Documentation is a contract between you: the developer and the other developers consuming your APIs. As a developer, when you write any type of tests, they become some sort of quality contract. If someone changes your code and break your tests, they break the implicit contract. However, when talking about consuming data via an API, things get a bit more complicated. We need a way to ensure that the end user expectations are matching our implementation/documentation. Unit testing simply can’t guarantee that. Unit testing will guarantee that the logic of your units of code is intact but it can’t easily guarantee that the API output matches the documentation, however this is something that has to be done.

Maintainability

This is a tricky point since maintainability is very subjective. But I think we can agree that decoupling/separating concerns will make our code more maintainable.

That’s why I personally don’t think it’s a good idea to mix HTML rendering code and web API code. Consider using different controllers or different files depending on the way your code is structured.

I also strongly believe that the implementation details shouldn’t define the way you design your web APIs. Don’t just slap a CRUD API on top of your model and call it done. In most cases, you will pay a high price later on if you take this approach because whenever you will need to change your web API or your model, you will be stuck. This is because your interface is now used by a lot of people and you can’t easily change it. There are many ways to avoid API/model coupling, I don’t advocate one particularly, but whatever you do, be sure you can make your models and APIs can evolve separately.

My approach

I’ve been designing and developing web APIs for many years and I’ve been struggling with everything I mentioned until now. I don’t claim that I’ve found the solution but I’d like to think that I found one own way of addressing what are for me some of the most important parts of API design.

Being explicit

I believe that there is value in being explicit in the way we describe web APIs. Sometimes people get confused between being verbose and being explicit. These are two different concepts. Being explicit can sometimes seem verbose, but we need to evaluate the value that can be extracted from the provided information. If there isn’t any clear value and too many words are used, then we aren’t explicit, we are verbose.

When designing a web API, I don’t write it for myself, I do it for someone else. It’s crucial to consider who you are writing for and to expose what’s important for them. A “small” problem is that we don’t show our code to our end users, so we need to find a way to explicitly provide the important information and to have this information provided to our end users.

DSL

For that I use a Domain Specific Language (DSL) which is a fancy way to say that I have some specific code allowing me to explicitly define my web services. These services exist as objects that can then be used to process requests but also to validate inputs, and even more importantly to generate documentation.

The DSL allows me to define the following:

• details about consuming the service (uri, HTTP verb, authentication details, other service details…)
• incoming params (which ones are allowed, the type, are they required, optional?, what are they for)
• service output

The input details is really important to validate incoming requests and reject them before even dispatching them. This is done for data sanity and for security reason. It also defines a strong interface that is easier to develop against and to maintain.

The output details might sound quite surprising and redundant. After all, isn’t that a duplication of effort since we already have this information in the “view”? Well, if we consider the important points I highlighted in the first part of this article, we need a way to enforce a “contract” between the end users and our implementation. To do that, we can’t simply rely on our code since we can’t trust it. The output is important to document the expected output but also to validate that our services match our documentation and therefore our “contract”.

Here is an example of a Ruby DSL use to describe a hello world service:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

describe_service "hello_world" do |service|
  service.formats   :json
  service.http_verb :get
  service.disable_auth # on by default

  # INPUT
  service.param.string  :name, :default => 'World', :doc => "The name of the person to greet."

  # OUTPUT
  service.response do |response|
    response.object do |obj|
      obj.string :message, :doc => "The greeting message sent back. Defaults to 'World'"
      obj.datetime :at, :doc => "The timestamp of when the message was dispatched"
    end
  end

  # DOCUMENTATION
  service.documentation do |doc|
    doc.overall "This service provides a simple hello world implementation example."
    doc.example "<code>curl -I 'http://localhost:9292/hello_world?name=Matt'</code>"
  end
end

The DSL style isn’t really important, what’s important is that it allows us to address some of the concerns we discussed earlier such as clearly defined interface that can be communicated as documentation but also acts as code. The focus is really on the API and everything fits in one page. Because everything is an object and can be inspected, proper and up to date documentation can be generated. And the implementation can be tested against the documentation since everything is maintained as code.

Being agnostic

While I like Ruby for many reasons, I don’t think that it’s the only good language to implement great web services. It actually has its pros and cons and so have all the web frameworks out there. That’s why I wrote my DSL as a standalone library that could virtually run on top of any web engine since it only outputs a representation of services.

While I hope to one day create an interesting interop solution across programming language (by exporting the objects in a shared data structure for instance), I started by focusing on the various Ruby web frameworks.

The best starting point for me was to use Sinatra. I almost started just using rack, but Sinatra was providing me with a bit more feature for very little headache and little code to grasp. I wrote wd-sinatra which is a Ruby gem providing the WeaselDiesel DSL on top of a Sinatra app. It comes with a generator and all the needed hooks to design, implement, test and generate documentation for modern web APIs.

Using a simple rake command (Ruby’s version of make) one can generate documentation or test the APIs against the implementations. The “mini framework” is still very free form and should let you do whatever you want. I don’t even set a default ORM for you to use since this choice is highly personal. I do however leave you places to set these things.

Sinatra and my “freedom framework” might seem too free form for you. So I’m currently working on getting the DSL to run on top of Rails, and by goal is to actually get it to run with a normal Rails app.

Reconsidering Rails’ MVC

The Rails code base is very deeply marked with its own concept of MVC and having controllers and actions. The challenge I have is that I like my service to be self contained. I want my services to be simple and easy to grasp. I like having 1 service per file. Models, libraries and other optional presenters/decorators live separately but I like to have my service implementation code with the rest of my service description. I honestly don’t like telling developers that they need to go check a route file and that my simple service requires that you open 12 files to understand what’s going on. Simpler is often better and that’s why in wd-sinatra the DSL and the implementation live together which is quite harder to do with Rails.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

describe_service "hello_world" do |service|

  # [...] see the DSL section to see how the
  # service is described. The block below is being
  # being called in the context of the request
  # after the request was validated.

  # SERVICE IMPLEMENTATION
  # the returned value is used as the response body, all the Sinatra helpers
  # are available.
  service.implementation do
    {:message => "Hello #{params[:name]}", :at => Time.now}.to_json
  end
end

Learning from experience

While the DSL fits most of my needs, we all have different use cases. While it’s critical for a library to have a clearly defined objective, it’s also important to have many people help improve it. Part of the learning/vetting excercise is to test an approach against different needs to define when and why it works well in some cases and why sometinmes it doesn’t. This allows us to define a sweet spot that should match the clearly defined objective. However to be able to do that, a design needs to be tested by many people. So far my approach seems to work very well when an API needs to live outside an application and that 3rd parties need to consume the resources. It also seems to work well with edge cases that many API designers seem to encounter sooner or later.

Conclusion

At the end of the day, we have to remember that API stands for Application Programming Interface and that these interfaces have to be programmed so humans can write to comsume them. One of Ruby’s main design points has always been to try to address human needs more than computer’s. As API designers/implementers, it seems important to adopt the same approach and consider who will use our interfaces. I think the discussion should focus more on what to value when defining web APIs, instead of arguing about how to implement APIs. Standardization is a great concept but a really hard to implement. And even with standards, we have to find a way to communicate with the API users to express what standards we follow and where to find the various entry points. Think about your API, how well does it communicate with your future API users, is it good enough for them to get excited? Is it good enough for them to create something amazing with it? If not, why not?

Discussion

Comments are available on this HackerNews thread.

RubyMotion is a commercial, proprietary fork of MacRuby that targets iOS. This is not a small achievement, MacRuby relies on Objective C’s Garbage Collector (libauto) which is not available on iOS. Static compilation and new memory management solution was required to target the iOS platform . The new runtime had to be small and efficient. Furthermore, being able to run code on iOS isn’t enough, you need tools to interact with the compiler, to debug, to packages applications etc…

I don’t think anyone will contest the fact that RubyMotion is a well done product. The question however is, ”is it worth for you to invest some money, time and energy in this product instead of using Apple’s language and tools”. In this article, I’ll try to balance the pros and cons of RubyMotion so you can have a better understanding of what RubyMotion could mean for you. As a disclaimer I should say that I was beta testing RubyMotion, that they are strong ties between RubyMotion and the MacRuby project I’m part of and finally that having MacRuby on iOS has been something I’ve been looking forward for a very long time.

Over the last few months I’ve seen RubyMotion take shape and finally hit the big 1.0. As you can see from Twitter and HackerNews, the Ruby community is excited about being able to use their language to write statically compiled, native iOS apps. Spoiler alert, they are right, it’s a lot of fun.

What I like about RubyMotion:

Ruby Language

I don’t mind Objective-C, I think it’s a fine superset of C, with the arrival of blocks, new literals and automatic memory management via ARC, Objective-C is actually getting better over time. But frankly, it’s not Ruby. You still have to deal with headers, you always have to compile your code via some weird Xcode voodoo settings, testing is a pain, the language, even with the new literals is quite verbose. On the other hand, using Ruby syntax I can get much more flexibility, reuse my code via mixins, easily reopen existing classes etc… At the end of the day, I end up with some code that seems cleaner, easier to understand and maintain even though I’m calling the same underlying APIs. Ruby’s flexibility also allows developers to make their own higher level APIs, take a look at some of the wrappers/helpers I wrote while playing with RubyMotion.

MacRuby

RubyMotion is based on MacRuby, meaning that all the time and energy invested in the project will benefit RubyMotion’s users. All the concepts I explain in my MacRuby book apply to RubyMotion. You don’t have to find workarounds to work with native APIs, Ruby objects are Objective-C objects and performance is great. I do regret Apple didn’t decide to embrace MacRuby for iOS but at the same time, even though we lost the Open Source aspect of the project and Apple’s backing, we gained much more flexibility and freedom on Laurent’s part.

REPL/Interactive shell

RubyMotion doesn’t currently have a debugger, but it does have something Objective-C developers don’t have, a REPL working with the simulator. This feature is quite handy when debugging your application or learning the Cocoa APIs. You can click on a visual element in the simulator and start modifying the objects in real time in a terminal window and see the modifications in the simulator. It reminds me of the first time I used firebug to edit the html/css of a web page and saw the changes in real time.

Not dependent on Xcode

Xcode is fine when you write Objective-C code, but it crashes often, it has a complicated UI and never really worked well for MacRuby due to the fact that Objective-C and Ruby have different requirements and the that Xcode is not open source. It’s also fully controlled by Apple and doesn’t provide APIs for 3rd party developers. (That said, the Xcode team has often helped out when a new released of Xcode broke MacRuby, so thank you guys).

Being able to use simple rake tasks to compile, simulate and deploy applications is just really really nice. I’m sure we’ll end up with better IDE integration, nice GUIs for some who like that, but in the meantime, as a “hacker”, I really enjoy the simplicity of the Rake tasks and not being forced in using a specific IDE.

Memory management

Even though ARC made memory management much easier for Objective-C developers, when using RubyMotion you don’t have to worry about memory (well at least not explicitly, don’t be dumb and create a bazillion objects and hold references to them either). This includes the CoreFoundation objects that you still have to manually manage in Objective-C. Memory management is transparent and in most cases it’s really nice.

What I like less about RubyMotion

Here is a list of things that are cons to using RubyMotion, note that while the list is longer than my list of “pros”, I listed a lot of small things. I also think that most of these issues will get solved in the next few months.

Ruby language

There are some cases where Ruby just isn’t that great or is not an option. Examples include dealing with API relying heavily on pointers, when using some of the lower level APIs or when you have to interact with C++ (video game engines for instance). The good news is that within the same project, you can write part of your code in Objective-C and the rest in RubyMotion. The other thing that bothers me a little bit with writing Ruby code for iOS is that you can’t easily enforce argument types and therefore you are losing a lot of the features provided by Clang to the Objective-C developers. I dream of an optionally typed Ruby – but that’s a different topic.

Another downside of using Ruby is that Ruby developers will assume all standard libraries and gems will be compatible with RubyMotion. This isn’t the case. You need to think of RubyMotion as only offering the Ruby syntax (modulo a few differences). To be honest, most of the std libs and gems aren’t that useful when writing iOS apps. Even when I write MacRuby apps, I rarely rely on them and pick libraries designed to work in a non-blocking, multi-threaded environment (usually ObjC libs that I wrap).

Cocoa Touch

If you’re already an iOS/OS X developer, you know that most of the hurdles aren’t the language syntax but the Cocoa APIs. These APIs are what you need to interact with to create your application. Cocoa APIs are usually much lower-level compared to what you usually see in Python, Ruby or even Java. While they are quite consistent, the APIs still have a stiff learning curve and currently,  if you want to write iOS applications, even if you know Ruby, you still have to learn Cocoa.

However, I do think that with RubyMotion now building a userbase, we will start seeing more and more wrappers around these sometimes hideous APIs.

No Xcode/IDE

There are cases where an IDE is really practical, especially when learning new APIs. Being able to have code completion, quick access to the documentation, instrumentation, debugging, interface builder, refactoring tools are things that Objective-C developers might have a hard time with when switching to RubyMotion. If you don’t know either Ruby or Cocoa, getting started with RubyMotion might be quite hard and you are probably not currently in the target audience.

Writing UI code by hand

In some cases, it makes sense, in other, it should be much easier. I know that Laurent is working on a DSL to make that easier and I’m looking forward to it. But in the mean time, this is quite a painful exercise, especially due to the complexity of the Cocoa UI APIs. Using Xcode’s interface builder and Storyboards is something I know a lot of us wish we could do with RubyMotion when developing specific types of applications.

No debugger

Again, this is eventually coming but the current lack of debugger can be problematic at times, especially when the problem isn’t obvious.

Lack of clear target audience

It’s hard to blame a brand new product for not having clearly defined a target audience. But as a developer I find myself wondering “when should I use RubyMotion and for what kinds of problems?” Is RubyMotion great for quick prototypes I can then turn into production code? Or is good for throw away prototypes? Is it reserved for “fart and flash light” applications? Is it ready for prime time and should I invest and write my new awesome apps using it? Should I convert over my existing code base over from Titanium (or whatever other alternatives you used)? Should I use RubyMotion every time I would use Objective-C?

I guess we will see when the first applications start hitting the app store and people start reporting on their experience.

Documentation

I’m partially to blame here since I could have moved my butt and start writing a book but the point is nonetheless valid. All the iOS documentation out there is for Objective-C, all the APIs and samples provided by Apple are obviously only for Objective-C. Thankfully, you can use the 2 MacRuby books available out there to understand how to convert this existing documentation into something useful, but RubyMotion will need to provide better and more adapted documentation for beginners. I have no doubt that this is coming sooner than later.

Proprietary solution

RubyMotion isn’t open source and currently fully relies on the shoulders of a single man. If unfortunately, Laurent goes out of business or decides to do something else then we will have to rewrite our apps in Objective-C.  Using RubyMotion for a professional product represents a significant business risk, which is exactly the same as using proprietary technology from any vendor. Apple could also decide to switch to JavaScript or rewrite iOS in Java and deprecate Objective-C. Let’s just say that it is unlikely.

I usually favor open source solutions, from the programming language I use to the OS I deploy on. This isn’t always possible and if you want to write iOS applications, you don’t currently have a choice. I do wish Laurent had found a way to make money while keeping the source code open. But who knows – after he makes his first million(s), he might change his mind.

Conclusion

I would strongly suggest you consider giving RubyMotion a try. I can assure you that it will provide at least a few hours of ‘hacking fun’ (and you will be able to brag about havng written your own iPhone app).  It will also help support financially someone who’s taking a risk in trying to push mobile development to the next level.

RubyMotion is, by far, my favorite alternative to Objective-C. But it is hard to tell, just 48 hours after its release, what people will do with it. Can it transcend the programming language barriers and attract Python, PHP, Java, ObjC and JavaScript developers? What is the sweet spot for RubyMotion applications? Will it affect the native vs web app battle? Can it make iOS development more accessible to the masses? Only time will tell.

Update:

Since RubyMotion 1.0 was released, I spent quite a lot of my free time leading the development of BubbleWrap, a free and open source 3rd party library for RubyMotion. Unfortunately, as of July 2011 I took a break from this project and RubyMotion in general. I explained my motivations in this mailing list post.

• MRI
• REE
• JRuby
• Rubinius
• MacRuby
• Maglev
• IronRuby

And many other less known implementations.

This time, the main man behind the project is the Ruby creator himself: Yukihiro ‘Matz’ Matsumoto. I already covered the announcement, you can read more about it there.

Why mruby?

Following my previous article on mruby, some people seemed to be confused by the “raison d’être”/purpose of the project and the difference between MRI and mruby.

mruby is designed to be modular and to be embedded, which means that it’s expected to live inside other software applications. In other words mruby’s goal is to make Ruby an embedded language. mruby is library that you link in other applications.

When talking about embedded languages, Lua is probably the most well known interpreted language. Lua is usually used for the purpose of offering a higher level language (scripting language) inside a software written in a low level language (usually C/C++). This is why Lua is very popular in the game industry to allow a scriptable interface and have parts of the code interpreted instead of requiring to recompile the code base against the target platform. According to this survey from 2009, Lua was by far the most popular scripting language for game developers.

Of course, Lua is also popular outside of the game industry and the reasons are simple to understand:

• very easily embedded (simple and well documented API)
• small footprint
• portable (runs almost everywhere)

That gives you an idea of what mruby is trying to be. As a matter of fact, this definition of Lua applies very well to mruby:

“Lua is dynamically typed, runs by interpreting bytecode for a register-based virtual machine, and has automatic memory management with incremental garbage collection, making it ideal for configuration, scripting, and rapid prototyping.”

Why not Lua?

This is a matter of taste and requirements. It should eventually boil down to the difference in the language designs.

Lua combines simple procedural syntax with powerful data description constructs based on associative arrays and extensible semantics.

Ruby is a dynamic, open source programming language with a focus on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write.

So on one hand you have a very simple language with a focus on functional programming and on the other, a richer language (albeit more complex) with a main focus on Object Oriented programming.

If you don’t know Lua, it’s quite close to JavaScript in the sense that it’s a prototype based language and you can therefore write code that is Object Oriented.

Ruby being a full OO and richer language, you can organize your code differently and write more of your complex logic in a scriptable language. One could even potentially leverage the language to create its own Domain Specific Language in pure Ruby and create a “Rails like” experience within its application.

To be honest, I don’t think that Ruby via mruby will just replace Lua, but I do believe it will become a very interesting alternative for some use cases. Especially if mruby manages to perform as well as Lua with the same type of footprint. There is also the fact that mruby being a sponsored project from the Japanese government, I wouldn’t be surprised to see big Japanese companies experiment with embedding Ruby in their devices. After all, software is being added in every object we own, rapid development and being first to market is more critical than ever which makes mruby is a very attractive solution.

Getting started

I wrote a simple hello world example to give you a place to start:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

#include <stdlib.h>
#include <stdio.h>

/* Include the mruby header */
#include <mruby.h>
#include <mruby/compile.h>

int main(void)
{
  mrb_state *mrb = mrb_open();
  char code[] = "p 'hello world!'";
  printf("Executing Ruby code with mruby!\n");

  mrb_load_string(mrb, code);
  return 0;
}

Or the longer/more complex version:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

#include <stdlib.h>
#include <stdio.h>

/* Include the mruby headers */
#include <mruby.h>
#include <mruby/proc.h>
#include <mruby/data.h>
#include <mruby/compile.h>

int main(int argc, const char * argv[])
{
  struct mrb_parser_state *p;
  mrb_state *mrb = mrb_open();
  char code[] = "p 'hello world!'";
  printf("Executing code with mruby!\n");

  p = mrb_parse_string(mrb, code);
  int n;
  n = mrb_generate_code(mrb, p);
  mrb_run(mrb, mrb_proc_new(mrb, mrb->irep[n]), mrb_top_self(mrb));
  if (mrb->exc) {
      mrb_p(mrb, mrb_obj_value(mrb->exc));
  }

    return 0;
}

To compile and link the code:

1

$ gcc -Iinclude hello.c lib/libmruby.a -lm -o hello.out

To execute it:

1
2
3

$ ./hello.out
Executing Ruby code with mruby!
"hello world!"

To get started, you just need the mruby source code and a compiler. (I haven’t tried to compile mruby or my sample on Windows for this code, but I assume it would work just fine with Visual C++).

The above example is very trivial and takes a line of Ruby code:

1

p 'hello world!'

that the linked mruby lib interprets. You can take the compiled file and use it on other machines using the same platform and the code will run fine, just like normal C code.

For a more complete example, take a look at the mruby’s standalone interpreter

Future

It’s a bit too early to know if mruby will be successful or not. There are few things I will keep my eyes on.

Performance

To be a true alternative to Lua, mruby will have to interpret Ruby code much faster than what MRI is able to do right now. It will also need to keep the memory footprint really tiny. Ruby being a more complicated language than Lua, it might be tricky, but we’ll see.

Documentation

While the idea of using Ruby has a macro language might get a lot of people excited, if documentation lacks or if the process is painful, the very same people might fallback to another language. Historically speaking, the MRI team has had issues with documentation and communication due to various factors. I hope that thanks to Matz experience and to the strong Ruby community, mruby will become a easy and efficient alternative to Lua or people wanting to embed one of the greatest interpreted languages.

Ruby outside of Rails

While Ruby on Rails really made Ruby popular, I’m always a bit dissapointed when I see how many popular projects the Ruby community has outside of Rails. Think about it, we have some awesome implementation such as JRuby and MacRuby which allow you to use an amazing amount of libraries to do the craziest thing one could imagine using the language they like the most. But yet, Rails is still by far the #1 Ruby project. Of course, there are many non-Rails related projects out there, but they don’t benefit from Rails’ aura.

What I hope with mruby is that it will allow developers to leverage the beauty of Ruby and to create other niches for people to have fun. Some people already started:

MobiRuby

Yuichiro MASUI is working on having Ruby available on iOS and Android.

Ruby for Node.js

Yasuhiro Matsumoto is working on mruby-uv an interface for libuv Node.js’ platform layer.

mod_mruby

MATSUMOTO Ryosuke is working on an Apache module for mruby called mod_mruby which would be comparable to mod_lua

mruby REPL

Frank Celler started working and blogging about writing a REPL for mruby, go check out his excellent blog posts on the shell he’s working on and other stuff he’s doing with mruby.

But there is plenty more to do, redis for instance now has/is about to be scriptabled via Lua, what about trying to use mruby to also support Ruby? What about scripting game logic and even full games in Ruby? What about mruby on a Raspberry pi? Ruby on my TV, fridge, car, AC, solar panel controller…

I will certainly be looking forward to people trying to reproduce the success of Rails in a different domain thanks to the Ruby language.

• The Open Source release of Matz’ mruby on GitHub.
• The announce of MobiRuby, an upcoming solution to develop iOS and Android applications using Ruby.

Probably due to my involvement with the MacRuby project, people have been asking me what I thought of these news.

mruby

mruby is far from being a new project. It’s based on the RiteVM which is a sponsored project by the Japanese ministry of Economy, Trade and Industry and lead by Ruby’s creator: Yukihiro “Matz” Matsumoto and was explained in details during Matz’s RubyConf 2010 keynote.

Back in November 2011 Matz also explained mruby. His talk was recorded and he explains very well the current Ruby ecosystem and why mruby makes sense.

As explained, the main goal of mruby is to have a Ruby version that can be embedded and therefore have a smaller footprint, be compiled and linked within another application.

Hiroshi Nakamura gave a great 1 line definition of mruby:

mruby targets game developers (to use instead of Lua), embedded application developers (devices, TV, phones..) and small memory footprint server applications (instead of JS for instance).

I’m personally quite excited by mruby, it’s not there yet and there is still a lot of work to do to prove the value of the project but it’s certainly a great step in the right direction. What’s also really nice is that the project is released under an OSS license allowing for all of us to contribute and companies to improve the implementation based on their own needs.

Summary: mruby is a promising project even if it is still in its infancy. Besides being yet another Ruby implementation, the fact that the target audience and the project scope are well defined and that the project is lead by Ruby’s author and sponsored by the Japanese government makes me want to believe that it can be a successful project. That said Lua is a simpler language and it is already well implemented in the targeted market, so hopefuly Matz, his team and the Japanese government have a plan to advocate and champion this new technology. Good luck to them and I’ll keep an attentive eye on the project.

Update: I wrote a getting started with mruby guide.

MobiRuby

MobiRuby is being developed by Yuichiro MASUI who works for Appcelerator the company behind the popular Titanium platform to write native iOS, Android apps in JS.

MobiRuby is built on top of mruby making it the first demonstration of what motivated developers can do with Matz new implementation. Very much like mruby, MobiRuby will be released under an OSS license but unlike mruby, the Apache license was chosen.

So far this was just an announcement with a code sample and a screenshot. That was enough to make the front page of HackerNews. Apparently the author is planning on releasing a first version in a few months.

It might surprise some, but I’m quite glad to see this kind of projects even though, they compete to some extent against MacRuby. It proves two things:

• there is a strong interest in having Ruby on mobile devices.
• it’s technically possible to do so.

Now, this is not something new either, Lua developers have been able to write iOS apps for a while, yet the majority of the iOS developers still use Objective-C. What are the challenges facing implementations trying to replace Objective-C?

The replacement language might not fit the Cocoa design.

Developing an iOS/OS X app means that you spend your time using provided libraries (called frameworks in Apple’s jargon). These frameworks have specific patterns, a well defined syntax and usually work in a very consistent/constraining way. Or your language is quite similar (like Ruby) and the transition is easy, or you need to start writing and maintaining wrappers (titanium).

Bridged runtimes.

Having 2 runtimes running at the same time is quite challenging and not efficient. That’s one of the reasons why Apple pushed MacRuby to move from RubyCocoa being a bridge and to have a Ruby implementation running in Objective-C runtime itself. This allows something else, in MacRuby all objects are actually Objective-C objects which means you don’t need to convert anything and Cocoa APIs can be extended from Ruby code by just reopening them.

Support.

This one is critical for many. Often, you don’t want to have your next big project rely on a technology that doesn’t have a good backing and support. What happens if you build your app using an alternate implementation and all a sudden the developer(s) get bored and move on, or take another job? What about the updates needed as Apple/Google update their platforms? It might not be the best reason to not choose an alternative, but it’s a reasonable reason especially for companies who want to be “safe”.

Cocoa

Cocoa APIs represent probably 90% of the challenge when writing iOS/OS X applications. The APIs, while powerful and efficient, are often a pain to get used to and to learn.

You have the challenge of the documentation and the examples that are only in Objective-C, requiring that someone writes a book and/or that you convert and maintain an enormous amount of documentation.

You also have all the tools provided by Apple which, you often can’t fully use because you aren’t using their toolchain.

To be honest, after so many years using MacRuby, I think that the real value of such a project isn’t in the easier syntax but instead in the fact that you can easily build wrappers and higher level interfaces around repetitive tasks. Having a mix of a well designed DSL and yet access to the native object is something extremely powerful.

Objective-C is evolving.

Objective-C is evolving, with the introduction of ARC, memory management became much easier. The latest version of clang also granted Objective-C with a nicer syntax thanks to new literals and object subscripting (read more). As a matter of fact, Objective-C’s syntax is getting closer and closer to Ruby’s making the choice to use an alternate much harder.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

// character literals.
NSNumber *theLetterZ = @'Z';          // equivalent to [NSNumber numberWithChar:'Z']

// integral literals.
NSNumber *fortyTwo = @42;             // equivalent to [NSNumber numberWithInt:42]

// floating point literals.
NSNumber *piDouble = @3.1415926535;   // equivalent to [NSNumber numberWithDouble:3.1415926535]

// BOOL literals.
NSNumber *yesNumber = @YES;           // equivalent to [NSNumber numberWithBool:YES]

// Container literals
NSArray *array = @[ @"Hello", NSApp, [NSNumber numberWithInt:42] ];
id value = array[idx];

NSDictionary *dictionary = @{
  @"name" : NSUserName(),
  @"date" : [NSDate date],
  @"processInfo" : [NSProcessInfo processInfo]
};
id oldObject = dictionary[key];
dictionary[key] = newObject; // replace oldObject with newObject

Performance.

Even though devices are more and more powerful, performance is often critical and Apple optimized the performance of their solution for their language. If you have ever developed a Titanium app, you know that it can be an issue and you might have to find workarounds to get decent performance.

Summary: Based on all these things, once MobiRuby will be released, I will be able to make a better judgement. But based on what I have seen so far, I’m quite concerned by the syntax and the performance we will get out of the box. But time will tell and things can always be improved. Ruby on iOS/Android is something exciting and I’m looking forward to testing the first betas.

Authentication vs Authorization

I wish these two words didn’t share the same root because it surely confuses a lot of people. My most frequently-discussed example is OAuth. Every time I start talking about implementing a centralized/unified authentication system, someone jumps in and suggests that we use OAuth. The challenge is that OAuth is an authorization system, not an authentication system.

It’s tricky, because you might actually be “authenticating” yourself to website X using OAuth. What you are really doing is allowing website X to use your information stored by the OAuth provider. It is true that OAuth offers a pseudo-authentication approach via its provider but that is not the main goal of OAuth: the Auth in OAuth stands for Authorization, not Authentication.

Here is how we could briefly describe each role:

• Authentication: recognizes who you are.

• Authorization: know what you are allowed to do, or what you allow others to do.

If you are feel stuck in your design and something seems wrong, ask yourself if you might be confused by the 2 auth words. This article will only focus on authentication.

A Common Scenario

This is probably the most common structure, though I made it slightly more complex by drawing the three main apps in different programming languages. We have three web applications running on different subdomains and sharing account data via a centralized authentication service.

Goals:

• Keep authentication and basic account data isolated.

• Allow users to stay logged in while browsing different apps.

Implementing such a system should be easy. That said, if you migrate an existing app to an architecture like that, you will spend 80% of your time decoupling your legacy code from authentication and wondering what data should be centralized and what should be distributed. Unfortunately, I can’t tell you what to do there since this is very domain specific. Instead, let’s see how to do the “easy part.”

Centralizing and Isolating Shared Account Data

At this point, you more than likely have each of your apps talk directly to shared database tables that contain user account data. The first step is to migrate away from doing that. We need a single interface that is the only entry point to create or update shared account data. Some of the data we have in the database might be app specific and therefore should stay within each app, anything that is shared across apps should be moved behind the new interface.

Often your centralized authentication system will store the following information:

• ID

• first name

• last name

• login/nickname

• email

• hashed password

• salt

• creation timestamp

• update timestamp

• account state (verified, disabled …)

Do not duplicate this data in each app, instead have each app rely on the account ID to query data that is specific to a given account in the app. Technically that means that instead of using SQL joins, you will query your database using the ID as part of the condition.

My suggestion is to do things slowly but surely. Migrate your database schema piece by piece assuring that everything works fine. Once the other pieces will be in place, you can migrate one code API a time until your entire code base is moved over. You might want to change your DB credentials to only have read access, then no access at all.

Login workflow

Each of our apps already has a way for users to login. We don’t want to change the user experience, instead we want to make a transparent modification so the authentication check is done in a centralized way instead of a local way. To do that, the easiest way is to keep your current login forms but instead of POSTing them to your local apps, we’ll POST them to a centralized authentication API. (SSL is strongly recommended)

As shown above, the login form now submits to an endpoint in the authentication application. The form will more than likely include a login or email and a clear text password as well as a hidden callback/redirect url so that the authentication API can redirect the user’s browser to the original app. For security reasons, you might want to white list the domains you allow your authentication app to redirect to.

Internally, the Authentication app will validate the identifier (email or login) using a hashed version of the clear password against the matching record in the account data. If the verification is successful, a token will be generated containing some user data (for instance: id, first name, last name, email, created date, authentication timestamp). If the verification failed, the token isn’t generated. Finally the user’s browser is redirected to the callback/redirect URL provided in the request with the token being passed.

You might want to safely encrypt the data in a way that allows the clients to verify and trust that the token comes from a trusted source. A great solution for that would be to use RSA encryption) with the public key available in all your client apps but the private key only available on the auth server(s). Other strong encryption solutions would also work. For instance, another appropriate approach would be to add a signature to the params sent back. This way the clients could check the authenticity of the params. HMAC or DSA signature are great for that but in some cases, you don’t want people to see the content of the data you send back. That’s especially true if you are sending back a ‘mobile’ token for instance. But that’s a different story. What’s important to consider is that we need a way to ensure that the data sent back to the client can’t be tampered with. You might also make sure you prevent replay attacks.

On the other side, the application receives a GET request with a token param. If the token is empty or can’t be decrypted, authentication failed. At that point, we need to show the user the login page again and let him/her try again. If on the other hand, the token can be decrypted, the content should be saved in the session so future requests can reuse the data.

We described the authentication workflow, but if a user logins in application X, (s)he won’t be logged-in in application Y or Z. The trick here, is to set a top level domain cookie that can be seen by all applications running on subdomains. Certainly, this solution only works for apps being on the same domain, but we’ll see later how to handle apps on different domains.

The cookie doesn’t need to contain a lot of data, its value can contain the account id, a timestamp (to know when authentication happened and a trusted signature) and a signature. The signature is critical here since this cookie will allow users to be automatically logged in other sites. I’d recommend the  HMAC or DSA encryptions to generate the signature. The DSA encryption, very much like the RSA encryption is an asymmetrical encryption relying on a public/private key. This approach offers more security than having something based a shared secret like HMAC does. But that’s really up to you.

Finally, we need to set a filter in your application. This auto-login filter will check the presence of an auth cookie on the top level domain and the absence of local session. If that’s the case, a session is automatically created using the user id from the cookie value after the cookie integrity is verified. We could also share the session between all our apps, but in most cases, the data stored by each app is very specific and it’s safer/cleaner to keep the sessions isolated. The integration with an app running on a different service will also be easier if the sessions are isolated.

Registration

For registration, as for login, we can take one of two approaches: point the user’s browser to the auth API or make S2S (server to server) calls from within our apps to the Authentication app. POSTing a form directly to the API is a great way to reduce duplicated logic and traffic on each client app so I’ll demonstrate this approach.

As you can see, the approach is the same we used to login. The difference is that instead of returning a token, we just return some params (id, email and potential errors). The redirect/callback url will also obviously be different than for login. You could decide to encrypt the data you send back, but in this scenario, what I would do is set an auth cookie at the .domain.com level when the account is created so the “client” application can auto-login the user. The information sent back in the redirect is used to re-display the register form with the error information and the email entered by the user.

At this point, our implementation is almost complete. We can create an account and login using the defined credentials. Users can switch from one app to another without having to re login because we are using a shared signed cookie that can only be created by the authentication app and can be verified by all “client” apps. Our code is simple, safe and efficient.

Updating or deleting an account

The next thing we will need is to update or delete an account. In this case, this is something that needs to be done between a “client” app and the authentication/accounts app. We’ll make S2S (server to server) calls. To ensure the security of our apps and to offer a nice way to log requests, API tokens/keys will be used by each client to communicate with the authentication/accounts app. The API key can be passed using a X-header so this concern stays out of the request params and our code can process separately the authentication via X-header and the actual service implementation. S2S services should have a filter verifying and logging the API requests based on the key sent with the request. The rest is straight forward.

Using different domains

Until now, we assumed all our apps were on the same top domain. In reality, you will often find yourself with apps on different domains. This means that you can’t use the shared signed cookie approach anymore. However, there is a simple trick that will allow you to avoid requiring your users to re-login as they switch apps.

The trick consists, when a local session isn’t present, of using an iframe in the application using the different domain. The iframe loads a page from the authentication/accounts app which verifies that a valid cookie was set on the main top domain. If that is the case, we can tell the application that the user is already globally logged in and we can tell the iframe host to redirect to an application end point passing an auth token the same way we did during the authentication. The app would then create a session and redirect the user back to where (s)he started. The next requests will see the local session and this process will be ignored.

If the authentication application doesn’t find a signed cookie, the iframe can display a login form or redirect the iframe host to a login form depending on the required behavior.

Something to keep in mind when using multiple apps and domains is that you need to keep the shared cookies/sessions in sync, meaning that if you log out from an app, you need to also delete the auth cookie to ensure that users are globally logged out. (It also means that you might always want to use an iframe to check the login status and auto-logoff users).

Mobile clients

Another part of implementing a SSO solution is to handle mobile clients. Mobile clients need to be able to register/login and update accounts. However, unlike S2S service clients, mobile clients should only allow calls to modify data on the behalf of a given user. To do that, I recommend providing opaque mobile tokens during the login process. This token can then be sent with each request in a X-header so the service can authenticate the user making the request. Again, SSL is strongly recommended.

In this approach, we don’t use a cookie and we actually don’t need a SSO solution, but an unified authentication system.

Writing web services

Our Authentication/Accounts application turns out to be a pure web API app.

We also have 3 sets of APIs:

• Public APIs: can be accessed from anywhere, no authentication required

• S2S APIs: authenticated via API keys and only available to trusted clients

• Mobile APIs: authenticated via a mobile token and limited in scope.

We don’t need dynamic HTML views, just simple web service related code. While this is a little bit off topic, I’d like to take a minute to show you how I personally like writing web service applications.

Something that I care a lot about when I implement web APIs is to validate incoming params. This is an opinionated approach that I picked up while at Sony and that I think should be used every time you implement a web API. As a matter of fact, I wrote a Ruby DSL library (Weasel Diesel) allowing you describe a given service, its incoming params, and the expected output. This DSL is hooked into a web backend so you can implement services using a web engine such as Sinatra or maybe Rails3. Based on the DSL usage, incoming parameters are be verified before being processed. The other advantage is that you can generate documentation based on the API description as well as automated tests.

You might be familiar with Grape, another DSL for web services. Besides the obvious style difference Weasel Diesel offers the following advantages:

• input validation/sanitization

• service isolation

• generated documentation

• contract based design

Here is a hello world webservice being implemented using Weasel Diesel and Sinatra:

Basis test validating the contract defined in the DSL and the actual output when the service is called:

Generated documentation:

If the DSL and its features seem appealing to you and you are interested in digging more into it, the easiest way is to fork this demo repo and start writing your own services.

The DSL has been used in production for more than a year, but there certainly are tweaks and small changes that can make the user experience even better. Feel free to fork the DSL repo and send me Pull Requests.

Migration/upgrades

Migrating a Rails App from a version to the other is very much like playing the lottery, you are almost sure you will lose. To be more correct, you know things will break, you just don’t know what, when and how. The Rails team seems to think that everybody is always running on the cutting edge version and don’t consider people who prefer to stay a few version behind for stability reasons. What’s worse is that plugins/gems might or might not compatible with the version you are updating to, but you will only know that by trying yourself and letting others try and report potential issues.

This is for me, by far, the biggest issue with Rails and something that should have been fixed a long time ago. If you’re using the WordPress blog engine, you know how easy and safe it is to upgrade the engine or the plugins. Granted WordPress isn’t a web dev framework, but it gives you an idea of what kind of experience we should be striving for.

Stability vs playground zone

New features are cool and they help make the platform more appealing to new comers. They also help shape the future of a framework. But from my perspective, that shouldn’t come to the cost of stability. Rails 3’s new asset pipeline is a good example of a half-baked solution shoved in a release at the last minute and creating a nightmare for a lot of us trying to upgrade. I know, I know, you can turn off the asset pipeline and it got better since it was first released. But shouldn’t that be the other way around? Shouldn’t fun new ideas risking the stability of an app or making migration harder, be off by default and turned on only by people wanting to experiment? When your framework is young, it’s normal that you move fast and sometimes break, but once it matures, these things shouldn’t happen.

Public/private/plugin APIs

This is more of a recommendation than anything else. When you write a framework in a very dynamic language like Ruby, people will “monkey patch” your code to inject features. Sometimes it is due to software design challenges, sometimes it’s because people don’t know better. However,  by not explicitly specifying what APIs are private (they can change at anytime, don’t touch), what APIs are public (stable, will be slowly deprecated when they need to be changed) and which ones are for plugin devs only (APIs meant for instrumentation, extension etc..), you are making migration to newer versions much harder. You see, if you have a small, clean public API, then it’s easy to see what could break, warn developers and avoid migration nightmares. However, you need to start doing that early on in your project, otherwise you will end up like Rails where all code can potentially change anytime.

Rails/Merb merge was a mistake

This is my personal opinion and well, feel free to disagree, nobody will ever be able to know to for sure. Without explaining what happened behind closed doors and the various personal motivations, looking at the end result, I agree with the group of people thinking that the merge didn’t turn up to be a good thing. For me, Rails 3 isn’t significantly better than Rails 2 and it took forever to be released. You still can’t really run a mini Rails stack like promised. I did hear that Strobe (company who was hiring Carl Lerche, Yehuda Katz and contracted Jose Valim) used to have an ActionPack based, mini stack but it was never released and apparently only Rails core members really knew what was going on there. Performance in vanilla Rails 3 are only now getting close to what you had with Rails 2 (and therefore far from the perf you were getting with Merb). Thread-safety is still OFF by default meaning that by default your app uses a giant lock only allowing a process to handle 1 request at a time. For me, the flexibility and performance focus of Merb were mainly lost in the merge with Rails. (Granted, some important things such as ActiveModel, cleaner internals and others have made their way into Rails 3)

But what’s worse than everything listed so far is that the lack of competition and the internal rewrites made Rails lose its headstart.  Rails is very much HTML/view focused, its primarily strength is to make server side views trivial and it does an amazing job at that. But let’s be honest, that’s not the future for web dev. The future is more and more logic pushed to run on the client side (in JS) and the server side being used as an API serving data for the view layer. I’m sorry but adding support for CoffeeScript doesn’t really do much to making Rails evolve ahead of what it currently is. Don’t get me wrong, I’m a big fan of CoffeeScript, that said I still find that Rails is far from being optimized to developer web APIs in Rails. You can certainly do it, but you are basically using a tool that wasn’t designed to write APIs and you pay the overhead for that. If there is one thing I wish Rails will get better at is to make writing pure web APIs better (thankfully there is Sinatra). But at the end of the day, I think that two projects with different philosophies and different approaches are really hard to merge, especially in the open source world. I wouldn’t go as far as saying like others that Rails lost its sexiness to node.js because of the wasted time, but I do think that things would have been better for all if that didn’t happen. However, I also have to admit that I’m not sure how much of a big deal that is. I prefer to leave the past behind, learn from my own mistake and move on.

Technical debts

Here I’d like to stop to give a huge props to Aaron ”@tenderlove” Patterson, the man who’s actively working to reduce the technical debts in the Rails code base. This is a really hard job and definitely not a very glamorous one. He’s been working on various parts of Rails including its router and its ORM (ActiveRecord). Technical debts are unfortunately normal in most project, but sometimes they are overwhelming to the point that nobody dares touching the code base to clean it up. This is a hard problem, especially when projects move fast like Rails did. But looking back, I think that you want to start tackling technical debts on the side as you move on so you avoid getting to the point that you need a hero to come up and clean the piled errors made in the past. But don’t pause your entire project to clean things up otherwise you will lose market, momentum and excitement. I feel that this is also very much true for any legacy project you might pick up as a developer.

Keep the cost of entry level low

Getting started with Rails used to be easier. This can obviously argued since it’s very subjective, but from my perspective I think we forgot where we come from and we involuntary expect new comers to come with unrealistic knowledge. Sure, Rails does much more than it used to do, but it’s also much harder to get started. I’m not going to argue how harder  it is now or why we got there. Let’s just keep in mind that it is a critical thing that should always be re-evaluated. Sure, it’s harder when you have an open source project, but it’s also up to the leadership to show that they care and to encourage and mentor volunteers to  focus on this important part of a project.

Documentation

Rails documentation isn’t bad, but it’s far from being great. Documentation certainly isn’t one of the Ruby’s community strength, especially compared with the Python community, but what saddens me is to see the state of the official documentation which, should, in theory be the reference. Note that the Rails guides are usually well written and provide value, but they too often seem too light and not useful when you try to do something not totally basic (for instance use an ActiveModel compliant object). That’s probably why most people don’t refer to them or don’t spend too much time there. I’m not trying to blame anyone there. I think that the people who contributed theses guides did an amazing job, but if you want to build a strong and easy to access community, great documentation is key. Look at the Django documentation as a good example. That said, I also need to acknowledge the amazing job done by many community members such as Ryan Bates and Michael Hartl consistently providing high value external documentation via the railscasts and the intro to Rails tutorial available for free.

In conclusion, I think that there is a lot to learn from Rails, lots of great things as well as lots of things you would want to avoid. We can certainly argue on Hacker News or via comments about whether or not I’m right about Rails failures, my point will still be that the mentioned issues should be avoided in any projects, Rails here is just an example. Many of these issues are currently being addressed by the Rails team but wouldn’t it be great if new projects learn from older ones and avoid making the same mistakes? So what other mistakes do you think I forgot to mention and that one should be very careful of avoiding?

Updates:

1. Rails 4 had an API centric app generator but it was quickly reverted and will live as gem until it’s mature enough.

2. Rails 4 improved the ActiveModel API to be simpler to get started with. See this blog post for more info.

Before I show you the benchmark numbers and you jump to conclusions, it’s important to realize that in the grand scheme of things, the performance cost we are talking is small enough that it is certainly not the main reason why your application is slow. Spoiler alert: ActiveRecord is slow but the cost of initialization isn’t by far the worse part of ActiveRecord. Also, even though this article doesn’t make activeRecord look good, and I’m not trying to diss it. It’s a decent ORM that does a great job in most cases.

Let’s get started by the benchmarks number to give us an idea of the damage (using Ruby 1.9.3 p125):

                                                             | Class | Hash  | AR 3.2.1 | AR no protection | Datamapper | Sequel |
--------------------------------------------------------------------------------------------------------------------------------------
.new() x100000                                               | 0.037 | 0.049 | 1.557    | 1.536            | 0.027      | 0.209  |
.new({:id=>1, :title=>"Foo", :text=>"Bar"}) x100000          | 0.327 | 0.038 | 6.784    | 5.972            | 4.226      | 1.986  |

You can see that I am comparing the allocation of a Class instance, a Hash and some ORM models. The benchmark suite tests the allocation of an empty object and one with passed attributes. The benchmark in question is available here.

As you can see there seems to be a huge performance difference between allocating a basic class and an ORM class. Instantiating an ActiveRecord class is 20x slower than instantiating a normal class, while ActiveRecord offers some extra features, why is it so much slower, especially at initialization time?

The best way to figure it out is to profile the initialization. For that, I used perftools.rb and I generated a graph of the call stack.

Here is what Ruby does (and spends its time) when you initialize a new Model instance (click to download the PDF version):

This is quite a scary graph but it shows nicely the features you are getting and their cost associated. For instance, the option of having the before and after initialization callback cost you 14% of your CPU time per instantiation, even though you probably almost never use these callbacks. I’m reading that by interpreting the node called ActiveSupport::Callback#run_callbacks, 3rd level from the top. So 14.1% of the CPU time is spent trying to run callbacks. As a quick note, note that 90.1% of the CPU time is spent initializing objects, the rest is spent in the loop and in the garbage collection (because the profiler runs many loops). You can then follow the code and see how the code works, creating a dynamic class callback method on the fly (the one with the long name) and then recreating the name of this callback to call it each time the object is allocated. It sounds like that’s a good place for some micro optimizations which could yield up to 14% performance increase in some cases.

Another major part of the CPU time is spent in ActiveModel’s sanitization. This is the piece of code that allows you to block some model attributes to be mass assigned. This is useful when you don’t want to sanitize your incoming params but want to create or update a model instance by using all the passed user params. To avoid malicious users to modify some specific params that might be in your model but not in your form, you can protect these attributes. A good example would be an admin flag on a User object. That said, if you manually initialize an instance, you don’t need this extra protection, that’s why in the benchmark above, I tested and without the protection. As you can see, it makes quite a big difference. The profiler graph of the same initialization without the mass assignment protection logically ends up looking quite different:

Update: My colleague Glenn Vanderburg pointed out that some people might assuming that the shown code path is called for each record loaded from the database. This isn’t correct, the graph represents instances allocated by calling #new. See the addition at the bottom of the post for more details about what’s going on when you fetch data from the DB.

I then decided to look at the graphs for the two other popular Ruby ORMs:

Datamapper

and Sequel

While I didn’t give you much insight in ORM code, I hope that this post will motivate you to sometimes take a look under the cover and profile your code to see what’s going on and why it might be slow. Never assume, always measure. Tools such as perftools are a great way to get a visual feedback and get a better understanding of how the Ruby interpreter is handling your code.

UPDATE:

I heard you liked graphs so I added some more, here is what’s going on when you do Model.first:

Model.all

And finally this is the code graph for a call to Model.instantiate which is called after a record was retrieved from the database to convert into an Object. (You can see the #instantiate call referenced in the graph above).

Description of the talk:

Let’s be honest, Ruby became mainstream a few years back and it isn’t the cool underground programming language it once was. It’s quite likely that your cousin’s boyfriend who’s “into computers” knows what Ruby on Rails is. There are hundreds of books, conferences, training and meetups for Rubyists. Recruiters fight to hire whoever knows how to generate a scaffolded Rails app. But now cool kids can’t stop talking about node.js, CoffeeScript, Clojure, Haskell and pushing code to the UI layer. What does it mean for the new, existing and prospecting Ruby developers? Is it time to jump ship and move on to something else?

Slides

The slides are available on Matt’s SpeakerDeck and can be downloaded here.

Video

Presentation website

Matt’s presentation was filmed by Confreaks and posted here.

I was nicely surprised to see so many of my followers send recommendations. Here is a list of 25 books that like-minded people suggested I read. Hopefully you will find a book or two to read too. Feel free to send more recommendations via the comments.

1Q84 by Haruki Murakami

suggested by @mrb_bk and @chadfowler

The Floating Opera and The End of the Road by John Barth

suggested by @chadfowler

Into Thin Air by Jon Krakauer

suggested by @bradly

Cutting for Stone by Abraham Verghese

suggested by @bradly

Atlas Shrugged by Ayn Rand

suggested by @bradly

Cien años de soledad by Gabriel Garcia Marquez (es)

suggested by @romanandreg & @jrfernandez & @edgarschmidt

One Hundred Years of Solitude by Gabriel García Marquez

suggested by @romanandreg & @jrfernandez & @edgarschmidt

Jitterbug Perfume by Tom Robbins

suggested by @supaspoida

The Sisters Brothers by Patrick deWitt

suggested by @dennismajor1

The Glass Bead Game by Hermann Hesse

suggested by @dj2sincl

The Wind-Up Bird Chronicle by Haruki Murakami

suggested by @chadfowler

Mindfire by Scott Berkun

suggested by @lucasdicioccio

Les Fourmis by Bernard Werber (fr)

suggested by @twitty_tim

Perfume: The Story of a Murderer by Patrick Suskind

suggested by @twitty_tim

Les Miserables by Victor Hugo (en, free ebook)

suggested by @tutec

Song Of Ice and Fire by George R.R. Martin (Game of Thrones saga)

suggested by @eeppa & @jarin

Clockwork Century by Cherie Priest

suggested by @eeppa

The Darkness that Comes Before by R. Scott Bakker

suggested by @eeppa

Drood by Dan Simmons

suggested by @eeppa

This Is Water by David Foster Wallace

suggested by @atduskgreg

Anathem by Neal Stephenson

suggested by @jarin

Ender’s Game by Orson Scott Card (entire saga)

suggested by @jarin & @edgarschmidt

Snow Crash by Neal Stephenson

suggested by @jarin

Fixing the Game by Roger L. Martin

suggested by @jarkko

The Road by Cormac McCarthy

suggested by @mrreynolds

Process to develop a curriculum:

Purpose. Know why you’re doing what you’re doing.

• You know how to do this.

Product. Start with the end in mind.

• What does the student look like when they walk out the door at the end of the training.

• Usually, we break these down into Knowledge, Skills, or Attitudes.

• Sometimes it’s helpful to see a photograph or drawing of a someone who finished the program and just talk about what they can do that makes them successful.

• This “product” should be connected and help you accomplish your mission

Practices. Then ask yourself, “How do people become like this?”

• If you can break down your Product into 3-5 bit-sized chunks, then see how people learn each one of those skills, gain each one of those knowledge points, and how to they gain the attitudes you want them to have.

• This one is much easier the more experience you have in seeing people develop the “Product.”

• This is also easier to determine when you understand Learning Theory).

• The results from this section will result in a list of:

  •        Activities or experiences

  •        Resources. What books, website, teachers, software, etc. will help them learn more effectively and efficiently

  •        Assessments. How you would know if the activity was helpful?

Plans. Make your plans based on the practices you’ve determined you’ve needed.

On a related topic, Chad Fowler posted an interesting blog post about what LivingSocial is doing to change the software development education.