Having witnessed the transition from analog to digital, and from local to cloud computing, I see generative AI as yet another significant shift.

As an engineer, my focus is on enhancing human creativity and simplifying technical barriers to fun and self-expression. I advocate for a human-centric approach to AI, emphasizing user assistance over just output focus. Engaging in R&D is vital, but it should align with addressing real human challenges.

I’m hesitant about the current generative AI approach most companies take. This may stem from my passion for real-time 3D rendering, but I’ve observed a clear trend towards 3D (or 2.5D) in animation and I’m concerned that focusing solely on 2D would be a bad thing for the industry and for storytelling in general.

To help me think through the process, I took [Oshi no Ko] as a recent example of an outstanding manga adaptation.

Current parts of the creative process

• Character Design: Essential for translating manga, encompassing not just size and emotions, but also style elements like outlines, shadows, and blush.

• Storyboarding: Outlines the cuts, camera angles, lens, and motions.

• Environments: Backgrounds and props.

• Colorization:

• Art Approach/Compositing: Inklines, outlines, smears, shadows, color grading, particles, lights, basically putting everything together.

• Animation: I skipped the most time-consuming aspect: traditional animation, involving “hand-drawing” or mocap, like the dance scenes in this example.

This is the very well known/documented process of drawing keyframes and in-betweens. It’s a tedious process a lot of folks have been trying to improve over time. The challenge is to maintain a human touch while also going fast. We see more and more of mix of 3D rendered + 2D drawing on top.

A significant challenge arises in the later stages when directors often want to modify storyboards or change elements like camera angles and lighting. This leads to expensive revisions. In contrast, the video game industry benefits from real-time rendering, allowing for quicker adjustments in animation, lighting, and camera work.

In [Oshi no Ko], only the backgrounds and dance scenes were 3D modeled and animated; the rest was hand-drawn. While large studios are shifting to 3D, the transition is not trivial for 2D animators.

So, where does generative AI fit in?

1. Facilitating the 2D-to-3D Transition: Offering real-time previews and more flexible, faster, accessible workflows.
2. Streamlining Processes: Letting artists focus on impactful storytelling elements.

Facilitating the 2D-to-3D Transition

Tools such as Blender’s Grease Pencil or Goo’s Engine are making things easier, but artists have to think in 3D, which is harder than it sounds.

The reality is that newer artists will adapt more readily to 3D, but this transition won’t be instantaneous. AI can greatly simplify the process by automating some of the more intricate and tedious aspects. For instance:

• Transforming character art into a comprehensive 3D model, including a single face mesh, blendshapes, animation tracks, face normals, and detailed settings for materials, shaders, and shadows. It also involves overlaying 2D assets like outlines, inklines, and blush on the 3D mesh. While the character may not be perfect initially, this step completes a substantial portion of the work. (and IMHO they key to do that is to generate code, not directly 3D assets)

• Automating and reusing animations. Animation fundamentally involves keyframes that move specific bones. AI models can rapidly produce high-quality animations and smoothly transition between them. (and they MUST to be manually editable)

• Generating backgrounds, environments, and props. Instead of creating full-fledged 3D environments, various shortcuts like skyboxes can be used for static scenes, while normal maps and PBR textures applied to low-poly meshes (generated by diffusion models or extracted from diffusion outputs) can create dynamic settings. A notable (free) example of a partial approach is https://stableprojectorz.com/. In this specific case, processing manga pages to identify and extract recurring props can be immensely helpful. Training a model to recognize these objects and either find matches in an asset database or generate synthetic data to create 3D models and/or materials/textyres can save countless hours, especially for artists less versed in 3D modeling.

• Creating rudimentary 3D scenes that can be refined and incorporated into control networks for video-to-video conversions and producing complex animated backgrounds, not limited by current diffusion model constraints.

• Producing shaders and materials to ensure consistent and editable colorization.

These are just a few examples, and there’s much more potential, but time is limited.

Streamlining Processes

The focus should be on crafting a genuine “product” experience, where creators can concentrate on storytelling, liberated from technical complexities. Given that anime often follows recognizable patterns, this could be leveraged to offer immediate creative options or opportunities to deviate from the norm. The goal is for a director or even a mangaka to develop a solid episode prototype, which can then be refined by the team. All the elements mentioned in point #1 will underpin this product-focused approach. This strategic direction will define a startup’s success, distinguishing them from merely being a service provider or consultancy. It’s a challenging path, requiring disciplined decision-making and effective success metrics.

In conclusion, employing AI should go beyond mere generation of 2D images or vectors. It’s about reimagining the entire creative process, making each step more adaptable (with a strong emphasis on 3D), automating predictable elements (like materials, shapes, and animations), and providing creators the tools to manually refine and personalize their work. It’s crucial to allow for experimentation and discovery, which will prevent the end result from being just a less expensive version of existing content.

In traditional online applications, when you make a modification to the state of the object you work on, the change propagates to a central server that holds the “truth”. The server registers the change and sends back a confirmation that the change was indeed applied.

The problem happens when Alice and Bob try to update the same database field at the same time. At this point, there will be a data race and whichever request gets to the database last, will “win”, the other will be discarded. That’s quite a jarring experience especially because round-trip latency of up to a second aren’t unusual (especially on mobile), so Alice might have moved on thinking their change applied, only to realize later on that it was overwritten. This is quite regrettable especially since those two updates aren’t technically conflicting, they are modifying different parts of the same string.

This traditional approach wasn’t designed with real time collaboration in mind and it naturally induces latency and flow challenges, making the collaboration experience worse.

One way to work around the latency and some of the conflict resolutions challenges is to pretend that we don’t need to talk to a server and that all changes are local. This is the approach the Fluid framework chose. Fluid still needs to eventually have all collaborators agree on the current state of our object(s) and has to avoid all conflicts. For that, it relies on eventual consistency provided by distributed data structures.

Fluid abstracts the networking and backend layers so developers can focus on building an user experience by only thinking “local”. There isn’t a centralized canonical data truth to check against anymore, instead all connected users have the data locally and all these users always eventually end up with the exact same data. This “local” approach allows for a zero latency experience while keeping data consistent.

Eventual consistency using special data structures

Example: Alice and Bob are co-editing a document at the same time

Alice and Bob both load a Fluidified application. The User Interface uses Fluid to load and manage the state of its underlying data. The application page has a title for which the initial state is loaded by Fluid from a server, the title reads “hello world!”

Alice edits the title so it reads “hello Alice!”, the change is applied right away to the data structure representing the title and an operation representing the change is sent to the Fluid server to propagate to all connected collaborators.

A new operation is sent to the Fluid server: Replace characters at position 6, new content: "Alice"

At the very same time, Bob edits the title too. He injects an upside down exclamation mark so the title reads “¡hello world!”. The change is reflected right away on Bob’s screen and an operation is sent to the Fluid server to propagate to the connected collaborators.

A new operation is sent to the Fluid server: Insert character at position 0, new content: "¡"

Even though Alice and Bob edited the same text at the same time, both saw their change being applied right away and then saw their collaborator changes being applied without any conflicts. They eventually both see the same consistent title.

Summary

By breaking down content into smart linked structures that know how to deal with granular changes, and can handle potential conflicts on each user’s device, users never wait for anything to happen and therefore feel like there isn’t any latency . Those smart structures are referred as Distributed Data Structures (DDSes) in Fluid parlance and are the key to making Fluid feel real time. Fluid offers various DDSes to cover the diverse needs of developers besides working on a collaborative text. Most DDSes link one to one to known structures such as strings, maps or linked lists. Developers can focus solely on the creation of amazing User Experiences and data modeling, the underlying concerns such as connectivity, data transfer and applying of operations and conflict resolutions are handled by the framework.

Microsoft used to have an extremely negative view of Open Source but under a new leadership things have changed and they are making some drastic changes to be more welcoming to FOSS developers.

Could Windows provide me with a similar or maybe even better developer experience than macOs?

There is only way to know: I need to a Windows machine as my main driver. To be clear, I’m not trying to confirm/deny that Windows is becoming a better dev solution for developers in general. I’m only focusing on my own case: an open source developer coding using open source languages such as TypeScript, Go, Python, Rust and years of experience using Unix/Linux.

The first thing I wanted to try was to automate the setup of my Windows and Linux machine. I’ve been spoiled by Docker for years and I love reproducible environments. Here is my journey automating the set up of my Windows/linux dev environment.

Installing Linux/Ubuntu on Windows

The idea of having to use the Command Prompt or Regexit is reminding me of a distant past full of pain and anger. Let’s start by making this place a familiar home and install Ubuntu (Linux). Turns out you can install Linux from the app store thanks to a Windows 10 feature called the Windows Subsystem for Linux (WSL). The sales pitch is that you can run a GNU/Linux environment, including most command-line tools, utilities, and applications directly on Windows, unmodified, without the overhead of a traditional virtual machine or dualboot setup. You pick your Linux distribution from the Microsoft store, install it, and boom it runs locally. You can even invoke Windows applications using a Unix-like command-line shell and invoke GNU/Linux applications from Windows (you can even mix and match which is pretty crazy).

Bash command to copy the SSH key to the Windows clipboard

$ cat ~/.ssh/id_rsa.pub | clip.exe

For Machine Learning devs, there’s an extra good news: you can leverage your machine GPU from the Linux side (how many times did I wish I could use a GPU on my Mac?) WSL has been around for a little while and Microsoft recently released WSL2 to address some performance issues (and give Linux programs access to the GPU).

Enabling Windows Subsystem for Linux

To enable WSL2, you have to be on Windows 10 version 2004 (20H1). I know, the naming is super confusing, this isn’t the version from the year 2004 but the free update that was released last May. Windows stopped giving brand new names to updates. A bit like Mac OS X/MacOS but without the cute cat and location names. Since I was on this latest version, installing WSL and Ubuntu was trivial and took a few minutes. I just followed this very straightforward tutorial.

1. Launch PowerShell as Administrator (click the start menu, type PowerShell, expand the options and pick run as Administrator).
2. Type dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
3. Type dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart
4. Restart your machine manually

Installing Ubuntu

At this point, WSL2 is setup, and we need to pick a Linux flavor (aka distro or distribution). I picked Ubuntu from the Microsoft Store (it’s free and you don’t need to signup or anything).

Ubuntu installed automatically on my machine in a few seconds and I got a prompt to set my linux username/password and I was done.

All in all, it probably took me less than 10 minutes, I’m impressed! Furthermore, I really like that I can run multiple versions of Linux while maintaining my main OS very stable. I did mess up my mac stability a few time when installing dependencies for some of the software I was developing. Also, it’s nice to be on a true Linux system, the same as the one I deploy to and it’s much lighter/easier than having to run everything in Docker.

Tips

You might want to configure how much resource you allocate to WSL if you feel Windows doesn’t get enough resources (you can also shut down the linux instance(s)).

You can export/import WSL distributions by using the wsl command from the windows side:

In Command Prompt, start by creating the folders we need

> mkdir %HOMEPATH%\Data\wsl\exports %HOMEPATH%\Data\wsl\distros

Then export the target Distro (here called Ubuntu)

> wsl --export Ubuntu %HOMEPATH%/Data/wsl/exports/ubuntu_%DATE:~10,4%%DATE:~4,2%%DATE:~7,2%.tar

This created a distribution dump dated with today’s date.

We can then import it to create a new distribution/instance.

> wsl --import UbuntuReloaded %HOMEPATH%/Data/wsl/distros/ubuntuReloaded %HOMEPATH%/Data/wsl/exports/ubuntu_20200730.tar

The new distribution is called UbuntuReloaded and is a copy of the first instance. This is practical if you want to run a distro for Python and one for JS for instance. It’s also a nice way to backup an environment.

You can see the current distributions and their status by using:

> wsl -l -v

Note that exporting the distro requires shutting down the distro automatically, so save your work before running this command.

Note also that when importing an exported distro, wsl logs you in as root instead of the created admin user (which is obviously there).

Tooling & scripting

One of the many reasons I was interested in doing all my development on Linux is that I deploy my production code on Linux machines, so might as well develop on the same platform I deploy to. I’m also super familiar with Unix/Linux, not so much with Windows. And while I can run code via the command prompt or PowerShell, I really prefer not to and I miss my good old Terminal.

Terminal

Turns out Microsoft developed a great Terminal for Windows (and they open sourced it)! It just doesn’t ship with Windows but you can install it directly from the store.

Installation script

Besides Terminal, I need a few other tools, drivers and software to make my Windows machine truly useable. And like a lot of developers I like automating things. Partly because I’m a bit lazy but mainly because I hate to repeat the same task over and over. So I decided to write a script to set up the Windows and Linux side of things.

The idea is simple, I want to automate everything from the Linux side and for that I’ll use a bash script that I will backup on github as a gist so I will eventually be able to pull it down and run it from a new machine. The bash script is not designed to be generic but instead focuses on my machine with my dev env, my Windows apps, an option to install the device apps/drivers I might need.

dot files

My Linux config files (dotfiles) are shared between my machines using a tool called chezmoi and GitHub. I’m still planning on using my Mac and various other machines so managing my configs across machines makes a lot of sense.

Editor

I switched my editor to VSCode a long time and have been using a VSCode extension to keep my configuration/settings synchronized across machines. Note that I want VSCode to display on the Windows side but run code and tools from the Linux side (my programming tools won’t be installed on the Windows side). Note that the extension won’t be needed soon since VSCode is going to have settings synchronization built-in.

That said, there are still a bunch of things to coordinate so let me walk you through what My script does and how it does it.

Calling the Windows side from Linux

The first trick is to get the Windows home path since we are going to need it for a few things. This is done using WSL specific linux commands called wslpath and wslvar.

export WINHOME=$(wslpath "$(wslvar USERPROFILE)")

Then we wrap a calling winget from Linux:

# winget calls winget on the windows side
function winget { "cmd.exe /C winget.exe $1 $2 $3";}

Windows package manager

winget is the new Microsoft package manager, the equivalent of homebrew but for Windows and officially developed by Microsoft. As I’m writing this article, winget is still in preview but it’s maturing quickly. Winget must be installed on the Windows side for the script to work, so we are going to need to check if it’s available before going further.

The bash script to check if winget is available on the windows side looks like that:

wingetAlert() {
    echo "winget could not be found"
    echo "make sure winget is installed and available and restart this script - https://www.microsoft.com/en-us/p/app-installer/9nblggh4nns1"
    cmd.exe /C start "https://www.microsoft.com/en-us/p/app-installer/9nblggh4nns1"
    exit
}

# checking that winget is installed on the windows side
ifCMDNotAvail "winget.exe" wingetAlert

We are calling the ifCMDNotAvail function and passing 2 arguments, the name of the executable to check and the function to call if winget isn’t available. The alert prints some text and then opens the Microsoft store page for winget by using the Windows start.exe command.

ifCMDNotAvail is a function using command -v to see if a command available or not, if it’s not, then the second argument is called.

Installing Windows applications

Here is the block of code installing the apps I’d like if they aren’t already installed:

ifPathNotAvailWinInstall "/mnt/c/Program Files/PowerToys/PowerToys.exe" Microsoft.Powertoys
ifCMDNotAvail "wt.exe" Microsoft.WindowsTerminal
ifPathNotAvailWinInstall "$WINHOME/AppData/Local/1Password/app/7/1Password.exe" AgileBits.1Password
ifPathNotAvailWinInstall "/mnt/c/Program Files/VideoLAN/VLC/vlc.exe" VideoLAN.VLC
ifPathNotAvailWinInstall "/mnt/c/Program Files/7-Zip/7z.exe" 7zip
ifPathNotAvailWinInstall "/mnt/c/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/AcroRd32.exe" Adobe.AdobeAcrobatReaderDC

ifCMDNotAvail code installVSCode

Without going through the full explanation, we do a check on the command or path and call winget on the Windows side with the name of the package to install. Pretty straightforward.

Linux setup

Nothing really special here, just installing my tools the linux way locally :) Something interesting to show is how I setup my ssh key.

function setupSSHKey {
    echo "Setting up Git"
    ssh-keygen -t rsa -b 4096 -C "myemail@domain.com"
    echo "new SSH key generated"
    ssh-agent
    ssh-add ~/.ssh/id_rsa
    cat ~/.ssh/id_rsa.pub | clip.exe
    cmd.exe /C start https://github.com/settings/ssh/new
    echo "Your new ssh key was added to your clipboard, add it to GitHub (and consider turning on SSO)"
    echo "Press any key when your key was added to GitHub"
    while true; do
        read -t 3 -n 1
        if [ $? = 0 ] ; then
            break;
        else
            echo "waiting for the keypress"
        fi
    done
}

I’m definitely not a bash expert, far from it but the part that’s interesting here is how I load the ssh key into the windows clipboard and open GitHub so I can paste it right there. I accomplish that by piping the result of the linux cat command into the Windows clip.exe command.

cat ~/.ssh/id_rsa.pub  | clip.exe

Honestly, this blew my mind the first time I wrote this line and it just worked.

So Here it is, the full script. Probably a work in progress, but I’m quite happy with how convenient and maintainable this solution is. I got a new PC and was able to get it up and running in less than 30 minutes 🥳

I had a blast during my time off:

• I worked on a great project with Grimes which we started previewing but had to put on hold because of covid-19.
• I had the privilege to listened to, coach and mentor talented startup leaders.
• I hacked on so many research projects and learned a ton.
• Due to the lockdown, I became an unqualified homeschool teacher and was able to spend time with my kids.
• Finally, I was able to spend a lot of time reflecting and discuss with so many smart people. Having this time for myself has been invaluable given the current world situation.

Generic findings

My goal was to use this time to better understand myself and the world around me. Here are 4 things that I found exacerbated by the coronavirus crisis:

1. We want and need to be active, not passive (During the pandemic the increase in video gaming dwarfed the increase in Netflix/video streaming).
2. We want to develop ourselves, grow, we might not always have the tools to do so, but are driven by some sort of growth feeling (from playing Animal Crossing to DYI or learning new skills).
3. Ego is a powerful enemy. This bullet point would require a full post on its own, but the short version is that many not-ideal decisions are based on ego and results in missed opportunities or worse.
4. We’re less and less of a social society. We might have huge social networks, I think it’s pretty clear to everyone who suffered from the lockdown that we somewhat become so self-centered that we lost real/deep human connections.

Personal insights

Besides these generic point, here is what I learned about myself:

While I started my journey wanting to focus on impact, I realized that measuring personal impact is surprisingly subjective. This is a topic that would deserve more time, but the short version is that chasing impact seems to be a fool’s errand that might lead to megalomania. Maintaining and increasing one’s happiness while having a positive impact on others is probably a better goal (at least more measurable).

To increase my happiness I found a few required ingredients:

1. hard challenges that I haven’t yet accomplished
2. a team of people I respect and want to collaborate with
3. a sense of progress

Writing down those bullet points, I realize that they might not seem like big revelations. After all, aren’t we all after those 3 things? The thing is though, I can’t be happy unless I have those three things. I can tolerate not having of those items for a little bit but if the situation doesn’t get fixed, I quickly burnout or move on.

The other thing I realized is that I need to be active, I need to be focused and I need be seriously challenged or I eventually lose sense of “active purpose”. The covid19 lockdown made this very clear, even if I thought that I could just chill and hack on a new project every week. Instead I needed and wanted to focus on a new challenge with people that would push me to become even better.

My new adventure

The direction I chose might surprise some of you, I have to admit it’s a bit of an unorthodox and unexpected move but I think it’s the right thing for me. I’m not becoming a VC, developing modular synths nor jumping back on a new startup, instead I’m joining Microsoft and will work on on new technology focusing on collaboration (the tech will be free and open sourced). I’m particularly excited to rethink the concept of open source within a 2020 and enterprise context of 2020. Open Source evolved in surprising ways and there is an opportunity to help the overall tech community, I’m looking forward to this challenge. Finally, I’m very thankful that most of my work with be done transparently and in the open. So while It’s premature for me to share more details, I’m looking forward to blogging/posting more and hear your feedback.

The idea of organizing Braintrust LA came to us when we discussed what we could do to help the LA ecosystem. Between the two of us, Chang as a VC and Matt as a founder, we wanted to find a selfless way to support entrepreneurs beyond introducing them to our networks or helping them with their decks. We opted to follow an early Pixar key mechanism: the Braintrust. The concept is pretty simple, we put smart, passionate people in a room together and encourage them to be candid with one another. This process naturally pushes us towards excellence.

We handpicked Braintrust attendees because we believe each of you have tremendous value to bring to the discussions and because we trust you. This process can only work if there is trust. Two Entrepreneurs trusted us and will make themselves vulnerable so they can benefit from the collective brain power in the room. And we trust each of you to be candid and benevolent.

It’s natural for attendees to have self-preservation tendencies causing you to hold back. But candor is forthrightness and frankness. Chang and I will be there to moderate. Entrepreneurs are constantly lied to, all the time, from friends to investors to even customers. The reason we are bringing you all together is to help the Entrepreneurs see what they might have missed and/or to learn from your experience. Don’t hold back, let them hear what people are thinking but not telling them.

The Braintrust, unlike a potential board of directors, has no authority. The Entrepreneur does not have to follow any of the specific suggestions given. After a Braintrust session, it’s up to her or him to to figure out how to address the feedback. Braintrust meetings are obviously not top-down or do-this-or-else affairs. By removing from the Braintrust the power to mandate solutions, we affect the dynamic of the group in ways we believe will benefit entrepreneurs and Braintrust attendees.

The Braintrust feedback is never about asking for a specific remedy or answer. The Braintrust is benevolent. We are here to help and we have no selfish agenda.

Security, Privacy and Trust will be major themes in 2020. Looking at this arm report, 30% of surveyed folks were more concerned in 2019 over data privacy and security than they were in 2018. Add the elections, legal actions against Google & Facebook, Apple and others pushing for less tracking and safer data and we are going to see a huge need to push compute power to the edge (on device).

A big problem is that we spent the last 10+ years centralizing compute on the cloud and now we need to also run major computations on user devices and we are nowhere ready for that (ML for instance runs very poorly on CPUs and uses a lot of battery). Expect lots of excuses and delays around securing data. Excuses will be made for at least another year or until someone steps up and uses that as a commercial advantage.

In the meantime Google seems to start testing the industry appetite with ML dedicated edge co-processors. I’ll definitely keep an eye on this project.

ML isn’t magical, you need a lot of clean, labeled data. Then you need a specific use case to design and deploy a great model that can help your product. Most startups don’t have clean, reliable and labeled data that can be used to train models. That’s why you need data engineers.

The reality is that given a specific opportunity and the right data, your product & engineering team should be able to build a decently fast solution and get you to 80-90% there. ML is awesome at giving you the last 10-20% and to not require an entire eng team. But building, validating and deploying a ML model is time consuming. Plus, very few people have experience going through this process which means higher risks.

My recommendation:

1. hire data engineers and build an amazing data set
2. test your hypotheses quickly and in prod
3. invest in ML to replace the validated “80% prototype”

I originally posted this note on Linkedin, see insights and comments there

“The massive experiment in using capital as a moat to build startups into sustainable businesses has now played out and we can call it a failure for the most part.”

“The big four tech companies own monopolies or duopolies in their core markets and are using the power of those market positions to extend their reach into tangential markets and beyond.”

Startups are now focusing on becoming sustainable and big tech companies are focusing on consolidation via acquisitions or by taking over smaller markets by using their market positions. This leaves very little room for creative and innovative thinking. At the very least, we will see less resources and time dedicated to experimenting with new approaches. Revenue and more importantly net revenue is back to being the primary driver. While this is a healthy economical correction, it also means that we will see very small companies fight for their survival relying on differentiating themselves from big companies (strong values such as ethics, privacy, different business models, better support…) while big companies will spend money to grow as big as possible so they can’t fail.

This environment is unfortunately not conductive to crazy new ideas. Everything must be weighed against OKRs or KPIs which themselves are directly correlated to net revenue. It will be hard for independent entrepreneurs to innovate unless they already have a solid economical engine.

On the other hand, while the open-source movement was promising, the experience somewhat failed when big tech companies started taking over and used FOSS as a way to dominate or extend their market reach. Open-source turned into more of a marketing and recruiting strategy than a way to drive new ideas. Very few open-source maintainers can sustain that lifestyle developing new ideas while big companies make millions off of the work that the open source community has done over the last decades. Fascinating turn of event.

I might sound pessimistic but I believe defining a clear baseline of where we are at now, is the best way to find medium to long term opportunities. Things are changing, non revenue driven innovation might be negatively affected in the short term but new models are about to be revealed and put to the test. That’s something I am excited to see.

But in 2013, I was very lucky to be able to create and grow a company putting musicians first and trying to create a positive long lasting impact on our industry. Splice has been a fascinating journey. I’ve been able to bring my 20+ years of experience and empathy for producers and musicians, as well as my years of digging deep into the code base of the software our users use daily. It’s been an educational, emotional, passionate and certainly challenging ride at times. It’s been amazing to meet and hang with the people who developed the software that made me want to learn how to program a computer. It’s been humbling to be in studio with people shaping the world’s culture by translating their emotions into music. I can’t be prouder of what we accomplished and can’t wait to see what’s still to come. Nonetheless, I decided to transition out and stepped down from my operating role at Splice. I will remain an advisor to the company but It’s time for me to ride off into the sunset, take some time to relax and focus on some other things such as making music, hacking, spending time with my family here in beautiful Southern California and of course finding other ways to be impactful.

Why?

Being a founder, I have to admit that deciding to move on from your own project is a really hard decision. But a startup isn’t a baby you give birth to. As a parent myself, I realized that way before starting Splice, but seeing so many of my friends deeply hurt because of this misunderstanding made it clearer than ever. A startup is a project you care a lot about and want to see become reality but that doesn’t mean it can’t keep going without you. I believe founders should maximize for impact and I think that Splice will keep on reaching its potential and serve the mission Steve and I set on day one, even after my departure.

The team has been laser focused, learning from our past mistakes and moving towards great autonomy and increased quality. I can’t go to a studio session without everyone knowing and using Splice, from hollywood music composers to k-pop producers to Nashville greatests. Two things I care a lot about are impact and decisions and I measure my own impact by analyzing the ROI of my efforts in relation to the impact I have. Clearly our impact is visible every time I talk to users. But as I take a deep look at today and tomorrow, I believe my impact outside of Splice will be even greater than what it could be inside. I even believe that this impact has the potential to have positive consequences on Splice continuous success. As so many people before me said, your goal should be to make yourself redundant and I feel that’s exactly where we are now.

I also have to admit that I am personally motivated to explore things that aren’t and won’t be on Splice’s roadmap. I’m a doer, a maker, a dreamer, a hacker and I realized that lately I’ve been really enjoying being surrounded by creative people. I loved reverse engineering all the DAW formats, architecting our technical solutions, grow the engineering team and learning a ton along the way. But now the team has things under control and I have a strong desire to expand my horizons, to pour my energy into different projects: from crazy artistic concepts, MR, low level hardware hacking, building synthesizers with my daughter, video games and more. The list of crazy ideas I want to hack on is endless and I am so excited that it will be hard for me to prioritize. It’s therefore logical that to be able to explore those things, I need mental space and free time. I also I hope that through this journey I will be able to better discover who I am and who I want to become.

Thank you

I’m thankful for the splice team - from my co-founder Steve, to the the board, to the team: Juan-Pablo, thanks for everything, I learned a ton from you. Thank you to our early employees who gave everything they had, our colleagues that jumped in during the crazy growth phase and the entire team that allowed Splice to become what it is. But beyond all the people involved into building Splice and our partners who helped us get where we are, I want to thank our users, our deeply passionate users who supported us and gave me so much energy on a daily basis. Music is a really really hard space to operate in and musicians is the group of people the most lied to. I’m so glad you saw that we are truly here to advocate for you and help create more seats at the table, your support means the world, thank you 🙏🙏🙏

What’s next?

I don’t have a job lined up, nor am I looking for one :) As a matter of fact, I started a “sabbatical” during which I am exploring a lot of things I never had time for previously. As mentioned earlier, I have a few artistic projects lined up, I want to go deep into some technical skills I haven’t had the chance to develop yet and I am planning on reading, watching amazing shows, spending time with my family and taking naps, lots of naps. Last time I took a real break in between projects was more than 30 years ago. It’s exciting and scary at the same time. People who know me, know that it’s hard for me to not get passionate and jump straight into a project as soon as I have a little free time. But I am lucky to have close friends and a family who are keeping me accountable! I will still remain involved with Splice as an advisor and champion. I obviously really want Splice to continue with its success and I’ll do what I can to support the company I co-created. I’ll also eventually start new projects and hope to continue having a positive impact and assist others in their journey here in LA and around the world. You will certainly find me surrounded by creative people, trying to understand what makes the creative process so amazing and universal. I will forever remain an advocate to the people who make our hearts beat faster, our feet stomp and our heads spin with emotions. “Here’s to the crazy ones. The misfits. The rebels. The troublemakers. The round pegs in the square holes. The ones who see things differently.”

You can play with the web version of the UI here, thanks to Flutter web!

Why Flutter?

You might not be familiar with Flutter and might be surprised I’m using this technology to demonstrate how to draw waveforms. There are a few reasons behind my choice, first and foremost, drawing waveforms on the web is pretty well taken care of by a bunch of JS libraries. On the other hand, drawing waveforms in non-browser based environments isn’t as well documented. The second reason is that since Flutter now also targets the web (in beta), I can link to actual renderings from this blog. Finally, it’s a good excuse to write a fun Flutter tutorial :)

Setting up our project

I don’t expect that most of you to already know Flutter but the good news is that ramping up is quite fast and I’ll walk you through the steps. Start by heading to https://flutter.dev/ and install Flutter. You can use Flutter on Windows, Mac and Linux. If you want to target iOS, you need to have Xcode installed. Install Android studio if you want to target Android. The instructions are quite clear and the tooling is very helpful to make sure you have everything setup correctly (thanks $ flutter doctor). Note that you don’t need a physical device to test your code, you can run everything in the iOS simulator or the Android emulator.

Start by creating an empty project:

$ flutter create waveform_demo

Flutter will let you know that everything went well and tell you how to compile and run your project inside your connected device or simulator/emulator. It will also indicate where our entry point is located (that’s the main.dart file in the lib folder).

In order to run your application, type:

  $ cd waveform_demo
  $ flutter run

Your application code is in waveform_demo/lib/main.dart.

VSCode tips

I personally use VSCode as an editor and it has a great Flutter plugin with amazing tool integration from code completion to emulator launcher, debugger, profiler and refactoring tools. As a side note I also recommend Awesome Flutter Snippets and Bracket Pair Colorizer.

Tools you’d usually use

For this demo, we are going to keep our code simple so I won’t setup custom static analysis options, nor custom launcher icons and unfortunately, I won’t show how to write Flutter tests. Note that in a real life project, you’d want to set those things up (the good news is that you can find plenty of documentation on those topics).

Code organization

We are going to need two views, one that I will call painter_view and another I’ll call clipper_view. We’ll set the app to render the clipper_view view by default and add a bottom nav bar to go to the painter_view. In order to save us time, I created a git tag with the code base.

To retrieve the code locally, clone the repo and switch to the tag:

git clone git@github.com:mattetti/waveform_demo.git
git checkout tags/v1-setup

I organized the files as shown below:

There aren’t hard conventions in the Flutter community on how to organize your files but that’s the pattern I personally use. I isolate UI concerns and model/logic.

You can run the project (F5 in VSCode) to see our setup:

Loading the waveform data

We are almost to the drawing part of the article but we need to do one more thing, load the data. In a real life application we would probably make an HTTP call to load the json or binary file we generated server side but for this example, we will bundle our pre-generated file in the app and load it directly. For that we need to do 4 things:

1. generate a json file (covered in the previous article)
2. add it to the app
3. create a model to load the file content
4. create a loader function that we can use to load the content asynchronously.

To add json files to our app, we need to create an assets folder to move the files to and register the files in our pubspec.yaml manifest file.

I generated 2 json files called one-shot.json and loop.json and added them to assets/waveforms/.

We then need to register them into our pubspec.yaml manifest:

  assets:
    - assets/waveforms/loop.json
    - assets/waveforms/oneshot.json

The files are now bundle with the app and we can load them programmatically. Let’s add a loader service to do that. In lib/core/services let’s create a waveform_data_loader.dart file. This file looks more like a helper than a service but that’s fine for this demo. We want to setup an async function that will return a future of the parsed json data. Returning a future is key here, since we don’t know how long it takes to load and parse the json (especially if it was to come from the network) and we can rely on Flutter’s excellent FutureBuilder widget. The future builder builds itself once the data is available.

Here is what our code will look like:

Future<WaveformData> loadWaveformData(String filename) async {
  final data = await rootBundle.loadString("assets/waveforms/$filename");
  return WaveformData.fromJson(data);
}

While very straightforward, we need to first define our WaveformData class that will parse the JSON file.

Let’s create a model file as lib/core/models/waveform_data_model.dart and have that class represent the waveform data. I don’t usually write JSON serializers/deserializers by hand since it’s quite cumbersome and can be automated quickly and reliably. I like to use https://app.quicktype.io/ and paste the JSON data I want to parse and have it generate the dart code I can use in my app. The code on its own isn’t very interesting, but here is a quick snapshot of the class members:

class WaveformData {
  int version;
  // number of channels (only mono files are currently supported)
  int channels;
  // original sample rate
  int sampleRate;
  // indicates how many original samples have been analyzed per frame. 256 samples -> frame of min/max
  int sampleSize;
  // bit depth of the data
  int bits;
  // the number of frames contained in the data
  int length;
  // data is in frames with min and max values for each sampled data point.
  List<int> data;

  WaveformData({
    this.version,
    this.channels,
    this.sampleRate,
    this.sampleSize,
    this.bits,
    this.length,
    this.data,
  });

 // ...
}

And we are done with data loading. We now have a way to load the json files we bundled in the app load make their content available to our UI.

The code is available in this tag.

Drawing the actual waveform

We are finally getting to the core of this article.

What we want is to render a waveform like this but inside our app:

Painted view

This view is quite straightforward:

Center(
  child: FutureBuilder<WaveformData>(
    future: loadWaveformData("oneshot.json"),
    builder: (BuildContext context, AsyncSnapshot<WaveformData> snapshot) {
      if (snapshot.hasData) {
        return PaintedWaveform(sampleData: snapshot.data);
      } else if (snapshot.hasError) {
        return Text("Error ${snapshot.error}", style: TextStyle(color: Colors.red));
      }
      return CircularProgressIndicator();
    },
  ),
)

We are using a FutureBuilder widget, to render/build once the JSON data is parsed. The builder function will receive a snapshot containing the waveform data. Because the builder is called right away, we want to check the state of the snapshot and render a progress indicator or an error if the parsing failed! However if the snapshot has good data, we are going to render the waveform!

Note that the code above renders a widget called PaintedWaveform, this is a widget we need to create ourselves. Let’s write it!

class PaintedWaveform extends StatefulWidget {
  PaintedWaveform({
    Key key,
    @required this.sampleData,
  }) : super(key: key);

  final WaveformData sampleData;

  @override
  _PaintedWaveformState createState() => _PaintedWaveformState();
}

class _PaintedWaveformState extends State<PaintedWaveform> {
  double startPosition = 1.0;
  double zoomLevel = 1.0;

  @override
  Widget build(context) {
    return Container(
      color: Colors.black87,
      child: Column(
        mainAxisAlignment: MainAxisAlignment.spaceBetween,
        children: <Widget>[
          Flexible(
            flex: 4,
            child: LayoutBuilder(
              builder: (context, BoxConstraints constraints) {
                // adjust the shape based on parent's orientation/shape
                // the waveform should always be wider than taller
                var height;
                if (constraints.maxWidth < constraints.maxHeight) {
                  height = constraints.maxWidth;
                } else {
                  height = constraints.maxHeight;
                }

                return Container(
                  child: Row(
                    children: <Widget>[
                      CustomPaint(
                        size: Size(
                          constraints.maxWidth,
                          height,
                        ),
                        foregroundPainter: WaveformPainter(
                          widget.sampleData,
                          zoomLevel: zoomLevel,
                          startingFrame: widget.sampleData.frameIdxFromPercent(startPosition),
                          color: Color(0xff3994DB),
                        ),
                      ),
                    ],
                  ),
                );

              },
            ),
          ),

          Flexible(
            child: Slider(
              activeColor: Colors.indigoAccent,
              min: 1.0,
              max: 95.0,
              divisions: 42,
              onChanged: (newzoomLevel) {
                setState(() => zoomLevel = newzoomLevel);
              },
              value: zoomLevel,
            ),
          ),

          Flexible(
            child: Slider(
              activeColor: Colors.indigoAccent,
              min: 1.0,
              max: 95.0,
              divisions: 42,
              onChanged: (newstartPosition) {
                setState(() => startPosition = newstartPosition);
              },
              value: startPosition,
            ),
          )

        ],
      ),
    );
  }
}

It might seem like there is a lot going on in there but it’s not that bad, let’s break it down.

The first thing you should notice is that instead of rendering our waveform right away, we are using a LayoutBuilder widget so we can get the size of the rendering area at runtime and adjust the dimensions accordingly.

The builder creates the layout which contains a CustomPaint widget. That’s the key widget to render our waveform!

CustomPaint(
  size: Size(
    constraints.maxWidth,
    height,
  ),
  foregroundPainter: WaveformPainter(
    widget.sampleData,
    zoomLevel: zoomLevel,
    startingFrame: widget.sampleData.frameIdxFromPercent(startPosition),
    color: Color(0xff3994DB),
  ),
),

Here is a great quick video about CustomPaint:

As explained in the video, we need to implement a custom painter to tell the widget what to draw. For that, we are going to code WaveformPainter:

class WaveformPainter extends CustomPainter {
  final WaveformData data;
  final int startingFrame;
  final double zoomLevel;
  Paint painter;
  final Color color;
  final double strokeWidth;

  WaveformPainter(this.data,
      {this.strokeWidth = 1.0, this.startingFrame = 0, this.zoomLevel = 1, this.color = Colors.blue}) {
    painter = Paint()
      ..style = PaintingStyle.fill
      ..color = color
      ..strokeWidth = this.strokeWidth
      ..isAntiAlias = true;
  }

  @override
  void paint(Canvas canvas, Size size) {
    if (data == null) {
      return;
    }

    final path = data.path(size, fromFrame: startingFrame, zoomLevel: zoomLevel);
    canvas.drawPath(path, painter);
  }

  @override
  bool shouldRepaint(WaveformPainter oldDelegate) {
    if (oldDelegate.data != data) {
      debugPrint("Redrawing");
      return true;
    }
    return false;
  }
}

You probably expected to see the path drawing code in this class, however the paint method delegates most of the heavy lifting to the WaveformData class as shown here:

void paint(Canvas canvas, Size size) {
  if (data == null) {
    return;
  }

  final path = data.path(size, fromFrame: startingFrame, zoomLevel: zoomLevel);
  canvas.drawPath(path, painter);
}

I ended up moving the path generation to the WaveformData class since we need some logic to implement scrolling and zooming and we might want to use the path to render within a widget that might not take a painter.

Let’s jump to WaveformData and see how I implemented the path drawing:

class WaveformData {
  // [...]
  List<int> data;
  List<double> _scaledData;
  // [...]

  List<double> scaledData() {
    if (!_isDataScaled()) {
      _scaleData();
    }
    return _scaledData;
  }

  Path path(Size size, {zoomLevel = 1.0, int fromFrame = 0}) {
    if (!_isDataScaled()) {
      _scaleData();
    }

    if (zoomLevel == null || zoomLevel < 1.0) {
      zoomLevel = 1.0;
    } else if (zoomLevel > 100.0) {
      zoomLevel = 100.0;
    }

    if (zoomLevel == 1.0 && fromFrame == 0) {
      return _path(_scaledData, size);
    }

    // buffer so we can't start too far in the waveform, 90% max
    if (fromFrame * 2 > (data.length * 0.98).floor()) {
      debugPrint("from frame is too far at $fromFrame");
      fromFrame = ((data.length / 2) * 0.98).floor();
    }

    int endFrame = (fromFrame * 2 + ((_scaledData.length - fromFrame * 2) * (1.0 - (zoomLevel / 100)))).floor();
    return _path(_scaledData.sublist(fromFrame * 2, endFrame), size);
  }

  Path _path(List<double> samples, Size size) {
    final middle = size.height / 2;
    var i = 0;

    List<Offset> minPoints = [];
    List<Offset> maxPoints = [];

    final t = size.width / samples.length;
    for (var _i = 0, _len = samples.length; _i < _len; _i++) {
      var d = samples[_i];

      if (_i % 2 != 0) {
        minPoints.add(Offset(t * i, middle - middle * d));
      } else {
        maxPoints.add(Offset(t * i, middle - middle * d));
      }

      i++;
    }

    final path = Path();
    path.moveTo(0, middle);
    maxPoints.forEach((o) => path.lineTo(o.dx, o.dy));
    // back to zero
    path.lineTo(size.width, middle);
    // draw the minimums backwards so we can fill the shape when done.
    minPoints.reversed.forEach((o) => path.lineTo(o.dx, middle - (middle - o.dy)));

    path.close();
    return path;
  }

  // get the frame position at a specific percent of the waveform. Can use a 0-1 or 0-100 range.
  int frameIdxFromPercent(double percent) {
    if (percent == null) {
      return 0;
    }

    // if the scale is 0-1.0
    if (percent < 0.0) {
      percent = 0.0;
    } else if (percent > 100.0) {
      percent = 100.0;
    }

    if (percent > 0.0 && percent < 1.0) {
      return ((data.length.toDouble() / 2) * percent).floor();
    }

    int idx = ((data.length.toDouble() / 2) * (percent / 100)).floor();
    final maxIdx = (data.length.toDouble() / 2 * 0.98).floor();
    if (idx > maxIdx) {
      idx = maxIdx;
    }
    return idx;
  }

  bool _isDataScaled() {
    return _scaledData != null && _scaledData.length == data.length;
  }

  // scale the data from int values to float
  // TODO: consider adding a normalization option
  _scaleData() {
    final max = pow(2, bits - 1).toDouble();

    final dataSize = data.length;
    _scaledData = List<double>(dataSize);
    for (var i = 0; i < dataSize; i++) {
      _scaledData[i] = data[i].toDouble() / max;
      if (_scaledData[i] > 1.0) {
        _scaledData[i] = 1.0;
      }
      if (_scaledData[i] < -1.0) {
        _scaledData[i] = -1.0;
      }
    }
  }

}

So there is a lot going on in here, let’s unpack it and take a look at how we scale the samples. The BBC tool we use stores the peaks and valleys as integer values with a specific bit depth (8 or 16bit). We want to scale this data to a -1,+1 range and that’s what _scaleData() does if we haven’t yet scaled the samples. To access our scaled data, we can simply call scaledData() which will check if scaling is needed and return a scaled version of our samples.

Now that the data scaling is out of the way, let’s jump to our entry point:

Path path(Size size, {zoomLevel = 1.0, int fromFrame = 0}) {
  //...
}

This is the function that returns the waveform path based on a provided size and some optional zooming and scrolling arguments. I’ll skip the zooming and scrolling explanations in this post to focus on the drawing itself. That takes us to this line:

return _path(_scaledData.sublist(fromFrame * 2, endFrame), size);

We are delegating to a private method passing a potential subset of our samples and our size. This is where the path drawing is actually happening.

Path _path(List<double> samples, Size size) {
  final middle = size.height / 2;
  var i = 0;

  List<Offset> minPoints = [];
  List<Offset> maxPoints = [];

  final t = size.width / samples.length;
  for (var _i = 0, _len = samples.length; _i < _len; _i++) {
    var d = samples[_i];

    if (_i % 2 != 0) {
      minPoints.add(Offset(t * i, middle - middle * d));
    } else {
      maxPoints.add(Offset(t * i, middle - middle * d));
    }

    i++;
  }

  final path = Path();
  path.moveTo(0, middle);
  maxPoints.forEach((o) => path.lineTo(o.dx, o.dy));
  // back to zero
  path.lineTo(size.width, middle);
  // draw the minimums backwards so we can fill the shape when done.
  minPoints.reversed.forEach((o) => path.lineTo(o.dx, middle - (middle - o.dy)));

  path.close();
  return path;
}

The path drawing methods are close to the methods you’d find on the HTML canvas or other path drawing solutions, we want to draw a path showing the outline of the waveform and close it at the end.

Waveforms can be drawn as a stereo rendering where the top part is a channel and the bottom part is the other channel. But most of the time, the signal is converted into one channel (mono). There are 2 common approaches to drawing mono waveforms. The first one is to take the amplitude average of a time window and draw each window. Then we can mirror the graph to create the expected waveform shape.

This seems to be the approach SoundCloud went for:

Which seems to be confirmed by looking at the JSON data they use to draw the waveforms:

{
  "width":1800,
  "height":140,
  "samples":[4,9,9,9,9,9,9,9,9,9,9,9,10,9,9,8,9,9,9,10,9,22,101,100,100,94,91,85,75,69,60,53,46,45,42,42,39,34,34,35,34,32,26,27,57,55,55,53,52,48,42,35,33,57,61,57,58,5, ...]
}

This is an absolutely valid approach but it provides less details than the BBC/Audacity approach which might be used by audio editors or musicians.

This second approach has 2 amplitude ranges, the max values and the min values. The top part of the waveform shows the peak value for each sampled time windows, the bottom part shows the minimum value for the same time windows. This approach is nice if you want to highlight the dynamic range of an audio signal. Knowing that, we can can draw our two waveform shapes.

The drawing algorithm is quite simple, we convert each sample value into a coordinate knowing that the lowest value is at the middle of the display area and the max value is all the way to the top for max values, and all the way to the bottom for the min values. Another way to say the same thing is that we need to calculate the x and y values for each sample where:

x = (display size / number of samples) * index of specific sample y = half the display height - (half the display height * sample value)

In code where i is the index of the sample we want to convert to coordinates:

middle = size.height / 2;
t = size.width / samples.length;
Offset(
  dx: t * i,
  dy: middle - middle * samples[i]
);

Once we have our 2 vectors of points, we can draw our max line:

final path = Path();
path.moveTo(0, middle);
// draw a line going through each max value coordinate
maxPoints.forEach((o) => path.lineTo(o.dx, o.dy));

// back to zero
path.lineTo(size.width, middle);

This will draw a line going from middle left all the way to the right, going back to the zero value. To draw our minimum value contour, we want to continue the line we started. Reading our coordinates in the reverse order going from the last sample back to the first one would allow us to draw the line continuously. We also need to draw our line mirrored meaning that our max value should be at the bottom of the display/drawing area. This is something we can so on the fly by altering the y coordinate value.

// draw the minimums backwards so we can fill the shape when done.
minPoints.reversed.forEach((o) => path.lineTo(o.dx, middle - (middle - o.dy)));

path.close();

Here is how the path looks like if we don’t fill it:

If you wanted to customize the rendering style, you would do that in the WaveformPainter class.

Alternative renderings

I opted to draw a line going straight through each coordinate, note that if you wanted the SoundCloud style, you’d instead draw rectangles for each sample. There are a lot of options to make the rendering more interesting, for instance you could draw rectangles when zoomed out, draw lines when getting closer and draw curved lines and sample dots when zoomed-in really closely as shown here in Ableton Live:

Clipper view

Finally, I want to show another cool way to use our waveform path by creating a clipper view allowing us to use our waveform path as a mask for another image (a gradient in this case, but you could use an image or anything really). A concrete product example would be if you analyzed the spectrum of your signal and wanted to show the main frequency range of each window by assigning different colors. This feature is commonly seen in DJ software.

To do that, we are going to use the ClipPath widget which will render our waveform through a gradient container (also provided by built-in widgets).

return ClipPath(
  clipper: WaveformClipper(snapshot.data),
  child: Container(
    height: height,
    decoration: BoxDecoration(
      gradient: LinearGradient(
        begin: Alignment.centerLeft,
        end: Alignment.centerRight,
        stops: [0.1, 0.3, 0.9],
        colors: [
          Color(0xffFEAC5E),
          Color(0xffC779D0),
          Color(0xff4BC0C8),
        ],
      ),
    ),
  ),
);

We need to implement our CustomClipper which is super straightforward since we already extracted the path drawing logic into our WaveformData class. This refactoring results in allowing us get a clipper in just a few lines of code:

class WaveformClipper extends CustomClipper<Path> {
  WaveformClipper(this.data);

  final WaveformData data;

  @override
  Path getClip(Size size) {
    return data.path(size);
  }

  @override
  bool shouldReclip(WaveformClipper oldClipper) {
    if (data != oldClipper.data) {
      return true;
    }
    return false;
  }
}

Conclusion

Drawing waveforms isn’t that complicated but you need to assemble a lot of parts together to get there. First we need data processing to extract a reduced version of the information we need to draw an amplitude map. Then we need a way to load and parse this data. We need to have a way to position our rendered waveform in our layout in a dynamic way (based on screen size, device orientation etc..). And finally we need to draw the waveform.

Once we have everything setup, we have a flexible and performant solution to explore more creative ways to render our waveform such as using a clipper, animations or maybe going for a totally different style like circular waveforms.

In audio, a waveform represents the amplitude of the signal over time. In other words, it shows the maximum extent of a vibration or oscillation, basically the volume of the sound over time. Note that while commonly used, it’s not the only sound representation we have and we will see that even if waveforms look alike, they are often not calculated and rendered the same way.

By the end of this article, you will understand how waveforms are generated and how to create the data you need to more easily and efficiently draw waveforms yourself.

Understanding the audio signal

In the case of a waveform, we are looking at a sound source over a certain amount of time. Usually that content comes from an audio file (or a in-memory buffer). Audio content is usually stored in two different ways: compressed and uncompressed (AKA PCM). You probably encountered .wav or .aiff files, those are uncompressed files meaning that the amplitude of the signal is stored as is in the file. On the other hand files such as .mp3, .flac or .m4a are compressed meaning that an algorithm was used to pack the content more efficiently, kind of like zipping the audio file. I covered this topic a bit more in depth in this article, but the short version is that to record audio digitally, we sample the signal x times per second (44,100 times in CD quality) and we store this information in the file so we can then re-use the data to move speaker membranes. An audio file contains the amplitude of the signal, which is exactly what we need to draw a waveform.

But there are two problems.

Compressed audio files

If you have a MP3 file, you have to first decompress its content to get the amplitude. That’s a bit problematic because it’s a CPU intensive task and the file might be big so we don’t want to keep everything in memory. A simple approach might be to convert the mp3 file into a wav file and store it to disk so we can process it. It does require however that the code cleans up after itself and if the same file is reopened/rendered later on, the expensive operation will need to happen again.

Too much data

To draw a waveform, we don’t need as much samples as when playing it back. Processing so much data is expensive and slow, especially on big files where we need to reduce the data because we have more samples that pixels to draw the waveform on. That’s why software like Ableton Live and Audacity create summary files.

“If Audacity is asked to display a four hour long recording on screen it is not acceptable for it to process the entire audio each time it redraws the screen. Instead it uses summary information which gives the maximum and minimum audio amplitude over ranges of time. When zoomed in, Audacity is drawing using actual samples. When zoomed out, Audacity is drawing using summary information.”

You can read more about Audacity BlockFiles here

Generating a summary information file

From the BBC to Soundcloud most services offering audio playback need a way to efficiently render waveforms. This is usually done by pre-calculating a summary server side when the audio file is first seen. This pre-calculated summary can then be loaded by the client at the same time (or even before) the audio file is downloaded and ready to play.

There are various ways to create a summary file and store its data. But the overall concept is always the same, we need to reduce the amount of data so we group samples together and we extract a value representing the time window we are reducing. One option is to create an average value of that window. Let’s say we group 256 samples together, we could add all the values together and then divide by 256, that would give us the average amplitude during that time period. Another option is to follow Audacity’s approach and for each window, we get the min and max values. This is less efficient from a storage perspective because we will end up with twice the data size, but it gives us more resolution on the data to draw an arguably better waveform.

Luckily for us, the BBC R&D group wrote a free and open source tool to do just that: https://github.com/bbc/audiowaveform

The audiowaveform command line tool can generate summary files in binary and json formats and can also generage waveform images but I’m not interested in this last feature since fixed size waveforms don’t go well with modern responsive UI designs.

I picked this tool because it’s free, we can dig into the source code, wrap it to be used from another programming language and the default options are great to get started. It’s also well documented, written and maintained. Finally, the BCC also released JS tools to consume the data, while you might not care to use those libraries, it’s great to have a reference implementation to understand how things work.

I would suggest to spin a web service that would receive an audio file (more likely via a GCS/S3 reference), copy the file locally and summarize it using the audiowaveform tool:

$ audiowaveform -i input.mp3 -o test.json

Then copy the json or binary data to GCS/S3 so it can be easily accessible.

The window size (by default 256 samples) depends very much on what you are going to do with the waveform and the average duration and sample rates of your audio files. Grouping 256 samples together on a 44.1KHz audio files means that we still have more than 172 data points per second. But if you deal with very short sounds it might not be enough, here is an example if a short cymbal sample, the entire file is less than 2 seconds and the hit itself last around 500ms:

(the example waveforms are rendered in a Flutter app using path drawing, the same results can be achieved on the web using canvas, more on that in a later post)

As you can see the resolution isn’t great. So we can try again by using a smaller window. Here is the same rendering but with the grouping using half the number of samples: -z 128

It’s better but we can’t really zoom in, the resolution is still quite low. Let’s try with a window of 64 samples:

That’s much better!

Note that the BCC tool is only one of the many freely available tools out there. I chose it for this article because it’s easy to use and demo. However, depending on your use case, you might prefer to do normalized averages instead. To do that, you might want use other existing tools such as SoX or FFmpeg. If you don’t care about zooming, I’d suggest you pick a resolution that works for most of your files (let’s say 800 points per file) and average dynamically based on the source length. Also, as the BBC tool does, I’d suggest to keep the data in integer values instead of using floats and specify the bit depth so you can then convert the data during rendering on a range of -1/+1. This approach results in much smaller files and faster parsing. Finally you’re better off using a binary format but whatever you do, make sure to store the file gzipped to save network transfer time.

Conclusion

• Pre-calculate waveforms server side.
• It’s not that complicated, use FOSS such as the BBC’s or SoX/FFmpeg.
• Tweak the sampling window based on your use cases or the file duration.
• Leverage modern cloud solutions such as PubSub/SQS and on-demand cloud functions to make sure you can ingest a large volume and keep the costs low.

Next, we will cover rendering waveforms.

Programming languages

Safe bets and my preferred languages

Go

https://golang.org Free and open source, from Google.

Go has been my goto language for many years now, it’s a great server and system programming language that shines for its simplicity and performance. It works best when developed server side APIs/services and client side CLIs. It’s a language that is easy to transition to, supports large team working on the same code base and has great tooling and compilation time. The language could offer more flexibility (Go 2 means to address some of those complaints) and the C interop could be faster/better (Gophers would tell you not use C). Go is a very popular programming language in the “devops” world (Docker, Kubernetes, Terraform, CloudFoundry and many more). Recommendation: adopt as server side language if you agree with the strong philosophy.

TypeScript

https://www.typescriptlang.org/ Free and open source, from Microsoft.

TypeScript has been my recommended way to write JS for a few years now. It’s a superset of JavaScript that compiles to plain JS but adds optional static typing and a other features. I prefer it over React’s JSX and it looks like TS is slowly but certainly becoming a standard alternative to JS. Great documentation, tooling and debugging support. Recommendation: adopt if you have a decent team or decent amount of JavaScript and don’t have 10+ years of JS experience.

JavaScript

https://www.javascript.com/ spec with free and open source implementations, originally from Netscape.

JS is the language that runs almost everywhere, the language has a lot of challenges but it’s ubiquitous. It certainly provides a lot of flexibility and allows new developers to feel progress really quickly. It’s however a really hard language to master and the tooling around the language seems to be moving really fast without a change to mature. My biggest issue with the language its interpreted nature and flexibility often results in runtime issues that are hard to catch/debug. Recommendation: if you are targeting JS, I’d suggest to take a look at TS, while it requires a little more onboarding/setup, I feel the superset provides a lot of answers to the problems I see in JS.

Java

https://www.java.com/ Free and open source implementations, originally from Sun Microsystems.

Java is a sure bet, it really managed to catch-up after years without design evolution. Java is a language for the masses, it’s pretty easy to learn, lots of resources and libraries, it performs well but it uses a LOT of RAM. In a world where RAM still comes at a premium that’s a bummer, also having to spend days tweaking the GC settings because the language allocates so many objects isn’t ideal. I still prefer Go for server side tasks, but based on the team skills and the project being worked on, Java can be a great solution. Recommendation: not the most exciting language but it will get the job done and you will find talented devs, adopt if you are ok being conservative in your tech choices.

C

https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/ Free and open source implementations, from Microsoft.

C# is Microsoft version of Java but with better tooling, more language features and limited cross platform support. Very much like Java, C# is a solid choice, it’s a flexible language with a performing runtime, lots of skilled developers but unlike Java, it doesn’t run as well on other platforms than Windows. Thanks to mono and .NET Core, C# is becoming more popular outside of the Windows ecosystem and is the de facto language of Unity. Recommendation: unless you have very specific needs, I probably wouldn’t choose C# as server side programming language but would use it for Unity or Xamarin projects, especially if you have previous experience with C#.

Python

https://www.python.org/ Free and open source by Guido van Rossum.

Python is the standard scripting language these days and while the community is still having issues migrating developers from v2 to v3, it’s the default language of data scientists and machine learning developers for which no other programming languages comes close to having the same quality of libraries. Recommendation: stick to what Python is best at: data science and ML.

Ruby

https://www.ruby-lang.org Free and open source by Yukihiro Matsumoto.

Ruby can be considered a more flexible/dynamic version of Python. As a language, it isn’t very popular outside of the Rails framework and became very popular at the same time as Rails in the web 2.0 boom. While not as popular as it once was, it’s found in many companies with legacy Ruby code or Rails applications. Recommendation: use when Rails makes sense.

Hot takes on less safe choices

Rust: if Go is supposed to be an alternative to C, Rust is an alternative to C++. Very powerful system language especially to write safer low level code, but challenging to learn. Recommendation: only use for low level or embedded programs.

Swift: great replacement for Objective-C, fine language but stuck on the Apple ecosystem. Recommendation: use if you are focusing on native and Apple first.

Kotlin: was a promising language when Android started supporting it making it a viable Java replacement but now Google is moving to Flutter/Dart for Fuschia and mobile dev, I’m less excited about this language that sometimes feels a bit overly engineered. Recommendation: stick to writing Android apps with Kotlin and only if you are focusing on native (server side Kotlin is fine but sticking to modern Java would be a safer bet).

Scala: I tried, but nobody seems to agree on how good Scala should look like. Fascinating academic language that took a big hit when Twitter switched from Scala to Java. Recommendation: pick an alternative language.

Clojure: I loved the idea of writing Scheme/Lisp on top of the JVM then I realized I wasn’t smart enough to track the state/shape of the values through the processing chain. Process a big chunk of data to only realize 4 hours later than the data half way through the pipe wasn’t what you expected. Recommendation: learn for fun, pick an alternative language for a professional project.

Elixir: very interesting language on top of Erlang/OTP, it captivated the interest of early Ruby developers but didn’t become a standard, stayed niche and isn’t backed by a big name. Recommendation: play with it because it’s a great language and a fun web framework (Phoenix), but consider an alternative language for a professional project.

PHP: unless you work at Facebook, pick an alternative language.

Impactful innovation is what Research & Development (R&D) teams around the world are working hard towards. Yet, there don’t seem to be blueprints that are reused and trusted across companies. On one hand, we get very little visibility into this process and on the other, it seems very challenging to define and measure success.

I love challenges, the harder the better. And the R&D challenge while also building sustainable businesses or growth is something I’ve been thinking a lot about for a while. So here are my thoughts and some of the things I’m experimenting with at Splice. It will take some time for me to confirm, self-correct and codify a solid, empirical approach but I like the idea of documenting my findings and experiments as a way to break the cycle and to keep myself honest. I will start by mentioning some of the main challenges that R&D teams must surmount and will then share my own framework.

R&D is just a label for accelerated and impactful innovation

The first think I’d like to clarify is that R&D is just a label. It’s not a goal, it’s not even a strategy, it’s a label used to refer to things moved out of the normal process with specific expectations. The objective is clear tho: companies invest in “R&D” because they usually want accelerated and impactful innovation. The short version of the rational is that it’s really hard to focus on delivering on short term objectives (OKRs/KPIs) while also exploring risky hypotheses. So comes a time when companies decide to have teams focused solely on “innovation”. Hoping that by isolating them from the day-to-day concerns, they can find breakthroughs that will empower the business and make a difference.

Leonardo da vinci, Crossbow Machine

Before going further, I’d like to define a few useful terms used in the rest of this document:

• experiments: measurable procedure undertaken to make a discovery, test a hypothesis, or demonstrate a known fact without being sure of the eventual outcome. Concretely, that usually means building and running some sort of prototypes and measuring its impact.
• learnings: answers to questions we might have or discovered as we went. Learnings reduce the unknowns around a specific topic.
• local vs. global innovations: by local innovations, I refer to innovations being made by the teams working on the current roadmap. Global here, means outside of the scope of the roadmap, innovations that might span across teams or even outside the current roadmap.
• local exploration: refers to the discovery phase done by teams already focusing on a particular domain area.

Discussing with many industry R&D leaders and driving myself this effort at Splice, I defined three significant challenges:

Primary challenge: making decisions

This seems like an obvious challenge but yet it took me a while to grasp how critical it is to understand and plan for this challenge. Being in charge of bringing accelerated and impactful innovation sounds exciting, but where do we start, why, how do we communicate that? Are we too focused on “local innovations” vs big picture innovations? Making any kind of decisions, technical or not, is much harder in a context without clear directions, guide rails and metrics. Especially if we don’t have a way to evaluate success/failure.

Secondary challenge: what does success look like?

This is the biggest question I faced and one that I felt most compelled to answer before committing resources to R&D at Splice. Because innovation isn’t always tangible and because people within our organizations have different expectations, I wanted a clear framework of evaluation. Selfishly, I wanted to know if I was doing a good job or not, and I wanted to be able to have this discussion with others using a baseline we all agreed on. There is another aspect to this challenge: if we do find an adequate approach to R&D, we can create a framework by which to make decisions. Not all decisions and high fives in our framework would necessarily lead to “breakthroughs” or features shipped to prod. On the contrary, success lies also in the ability to decide on what shouldn’t be shipped or better yet, not worked on at all. Success is about decisions we manage to make, and in that sense, everything we prove we shouldn’t do, must also be celebrated as a huge win!

Third challenge: communication

There is so much stigma around R&D that communication is a huge part of what R&D teams should excel at. It might sound counter productive to many, especially if you imagine a R&D team as a bunch of crazy scientists running experiments in a basement lab. Turns out most R&D teams work very much in isolation and rarely interface with Product or Engineering. When they do, the relationship doesn’t have deep roots, and it is often perceived as a distraction. We end up with 2 common scenarios:

• R&D team doesn’t feel like they are shipping anything and aren’t valued.
• The Product & Engineering teams feel that they are forced to change their roadmap to ship (sometimes subpar) experiments that they have to maintain.

These 2 scenarios can be avoided if we all work on the same objectives, reduce our ego to a minimum and provide value to each other.

R&D is removed from the day-to-day so that the team can focus on out of band hypotheses. But they also want to see their work being shipped and be impactful. To accomplish that, finding amazing actionable learnings isn’t enough. Work needs to be done to have everyone in sync, expose valuable findings to other teams and build the relationships to make sure these findings make it to the roadmap. You can’t be impactful if you are totally isolated, building those bridges is absolutely critical so you can become and champion and find other champions within the organization.

Experiment: developing an R&D framework

Here is the framework I put together for Splice and something I am still tweaking. It’s far from perfect but it tries to address some of the challenges I highlighted earlier.

Process: Have stakeholders define one or more key hypotheses. For each hypothesis agree on desired learnings, budget and timeline.

Output: Regular communication, actionable (and documented) learnings, product briefs that summarize the research and make suggestions and finally prototypes/demos that helped us come to our conclusions and help others “feel” the potential.

Success evaluation: stakeholders evaluate the extracted learnings and how it impacts the roadmap.

Stakeholders

R&D is a service to the company and having a “Braintrust” to help steer, bring a different perspective and champion your work is a huge advantage. This is similar to Pixar’s approach to creative guidance. R&D doesn’t technically report to the stakeholders, they are a support mechanism. Those experienced experts work on their own challenges but come together to discuss how innovation can help the company, discuss hypotheses that aren’t explored but could be extremely impactful. Finally, they help refine the direction to maximize the impact on the work on the roadmap. Those stakeholders have to be influential people who can see the great big picture and can express candid feedback. I chose 3 stakeholders, depending on how your company works, you might pick other stakeholders:

Product Partner: Looks past our current roadmap and finds areas that if explored would help us make better decisions.> Business Partner: Looks at the current & future economical situation, helping define business opportunity areas to explore.> Transfer Partner: Makes sure learnings are properly transferred. This stakeholder is also the best judge of what should be explored locally vs by a dedicated future-looking team.

Main Hypothesis

Together we define a big hypothesis we want to explore. The hypothesis will be our North Star for a few months. From there, we define desired learnings to focus on. Those desired learnings will be used to extract initiatives but also to evaluate our work.

Let’s say our business provides services to the elderly. We’ve been focusing on a watch type device to monitor their health and falls. Our product is backed by a subscription service. We have a solid roadmap to execute on this vision, but we also believe that innovation could have a huge impact.

Hypothesis: “We believe that leveraging home automation integration would bring children and parents closer, thereby increasing the mental health of both groups.”

Context: The business partner is very interested in a potential new revenue line, the Product partner is thinking about next year’s roadmap and the feasibility of such an approach. Everybody is thinking about differentiating themselves from the competition as well as the potential ROI. Everybody is excited about the project but there are way too many unknowns to add this project to the roadmap.

Desired learnings

1. Who are the main home automation providers, what does it take to integrate with them?
2. What kind of experiences can we provide to our users?
3. What’s the technical investment required to build/ship a first experience on the biggest platform? How about integrating with other platforms?
4. How to measure usage, traction and the relationship between our 2 groups?
5. What are ways to monetize such a service if we were to ship it?

Those desired learnings come from discussions with the R&D stakeholders. They need to be able to answer the obvious questions the teams would have if the hypothesis is confirmed and the stakeholders decide to add a related project to the roadmap. The goal isn’t to have something ready to ship to users, but to have gathered enough facts so that the company has enough visibility to make the right planning decisions. The decision might be as simple as whether or not to pursue the opportunity pursue an opportunity, or should we not. Both outputs are great since they are decisions and decisions are what help companies move forward. Also, proving that we should not do something will save a huge amount of time and money, as well as solidify the direction the company is taking.

Budget and timeline

We defined a North Star and we have specific questions we’d like to answer but we also need to timebox our research. This is a very important phase because we are setting the expectations for the team and for the rest of the company. We aren’t building a product we are shipping to all our users, we are building confidence in whatever direction the company will take. This is an interesting exercise which helps the stakeholders and the R&D team define how much we want to invest in the hypothesis and each desired learning. Concretely that avoids treating R&D as a black box with results coming at unexpected times. It also gives an opportunity for the stakeholders to change focus at the end of a budgeted period. I hear Google X works a bit like that, each year, each project lead presents their team progress and asks for a renewal or increase of budget. This is somewhat similar except at a smaller scale. We need to evaluate more than one hypothesis per year and need to influence the current business instead of creating brand new ones.

Output

The goal is to have significant impact on the future of the company and we do that by influencing the roadmap. So our output has to be supporting this goal. In most companies, the roadmap is managed by a Product organization. We therefore need to provide them with what they need to understand and decide if altering the roadmap makes sense. There are 3 main forms of output that seem to help:

• Product Briefs— use the Product language to summarize learnings. It’s a bit awkward at times, since we are answering a question and making a recommendation but don’t have a full integration plan, that would be something the Product team would do later on. However awkward it might feel to non Product people, it is super important to speak a common language. I used a simple template that I’ll refine as I go, here are some of the sections: the idea in one sentence, goals, counter-goals, opportunity, hypothesis, use cases, key open questions, key open questions, suggestions. And finally a detailed breakdown of the experiment, with links to the research, code and demos.
• Prototypes/demos— whenever possible, providing people an opportunity to experience themselves what we could build together.
• Documentation, lots of documentation and examples — there are two reasons for that, first the product briefs don’t go into all the details of what we learned and tangents discoveries. Secondly, because if the learnings turn into a solution that will make it to the roadmap, you want to provide all teams (eng, design etc…) with as much information as possible so they don’t waste time and feel confident they can work within a certain timeline.

Conclusion

If you are looking into building an R&D team or are considering working with one, consider defining a process around it. Have clear discussions about expectations and accountability. While leaders often love investing in “Innovation”, the meaning and expected output are often too unclear to set an R&D team up for success. Instead of expecting such a framework to be delivered to you, consider offering the framework you think would help the company and the R&D team to stay aligned. It won’t be perfect, and will need to be tweaked, don’t overthink it, R&D the “R&D process”. Learn, evaluate and adapt, the company and the team will feel and be motivated by the impact.

I remember so clearly building the first prototype and hearing the excitement in the voice of my co-founder. The first pitches, the first hires, the first people we had let go 😥 The first arguments, the first fights, the first wins, the first 1:1s, the first dollars we made, the firsts disappointments. The first bugs, the first crashes, the first breakthroughs, the first townhome meetings. The first term sheets, the first board meetings, the first no, the first yes! The first employee departures, the first M&A, the first team off-site. The first exec meeting, the first OKR meetings, the first advisors, the first $10M paid to creators! Wow, so many memories, I certainly spent too much time thinking about tomorrow and not enough celebrating.

Kruskal-Szekeres spacetime diagram

It’s been and still is a wild ride. It feels like we started yesterday, yet it feels like it’s been forever. Often people want to know what it is like to be the founder of a startup. They have this glamorous silicon valley image clashing with the image reflected by Silicon valley, the HBO TV show. The reality is hard to describe, it’s full of pain, excitement, highs and lows. It’s a processes that forces you to reevaluate and reinvent yourself constantly. It’s lonely yet you are surrounded by others going through the same thing.

The experience seems to be different for each founder depending on their personality, cultural, social and economical background (and unfortunately also their gender and skin color). Running a VC backed company is like playing a board game where only a few select group of people know the rules. It’s not like the rules are secret or anything, it’s just that they aren’t the rules most of us grow up learning. Those rules seem very foreign at first. Worse, those rules are implicit, they are enforced guidelines that are believed to guide you to a predefined notion of success. Mind you, those rules change with the market. The way you were raising money in 2013, would probably not work in 2019. So you have to stay up to date with those implicit and changing rules. Let’s be honest, this part of the system is very unfair, it favors a small group of privileged people who have the social capital to be taught those rules. Talent is everywhere, opportunities are not. I have to admit that this is something that really bothers me. I had the chance to discuss this topic with some VC partners I trust. It’s clear they are very aware of the situation and in some cases try to induce local change. It’s clearly not enough but I honestly appreciate their effort when they try.

A thing I also learned the hard way is that you rarely have an opportunity to correct your mistakes. Running a VC backed startup is like strapping a rocket to your chair, it might get you to the finish line faster, but a few small mistakes and you are out (and a crappy rocket might blow up). It’s an exciting process, especially if you have little patience, but the g-force you have to endure will take its toll on most of us. Another funny thing about this analogy is that if you don’t pass you, you get used to the initial speed and when the rocket is slowing down because you are getting close to orbit, you miss the initial speed and thrill.

Robert Courter in the Bell Rocket Chair.

I’m not a nostalgic person and I don’t live in the past regretting things, but I often think of the many mistakes I made so I don’t make them again. I wish I could fix those mistakes now that I know the consequences, but the time window is often closed. The best I can do is assume the consequences and not making those mistakes again as well as helping others who are going through the same challenges. As much as we love saying that we embrace failure, ask a founder how they are doing, the most common answer is probably a version of: crushing it! Unfortunately, the reality is probably very different. Self-doubt is probably the best feeling describing being a founder. But we don’t want people to doubt us, otherwise investors won’t support us, we won’t be able to hire top talent and we will fail to make the “X under X” list. So many founders at all stages struggle with depression, substance abuse, deep feeling of loneliness. It’s quite sad to see peers going through that while the media love glorifying or at least “glamourising” our self destructive behaviors.

VCs are not your friends, but they aren’t your enemy either. Like you, they are doing their job and their job is to make sure you exponentially increase the financial value of their investment. You can have a close relationship with some investors, they can provide you with great insights and be supportive. But their primary duty is to their firm and they LPs. They are being kept accountable and have to show results. This is not a secret, it’s their duty. It’s too easy to blame VCs when things go bad. Don’t get me wrong, there is a fair share of bad VCs out there, but remember that you are the one who agreed to tape a rocket to your chair. They provided you with the rocket, they didn’t force you, being VC backed is certainly not for everyone or every project. You told them you’d manage the pressure, the growth etc.. But did you make sure you were aligned on the conditions attached to using the rocket? If they don’t think you will make it, don’t be mad if they don’t want to add more gas in the rocket or suggest you stop the race. If they feel that you can make it but you are slowing down too quickly on purpose, don’t be mad if they pressure you to add more gas and take bigger risks. That’s part of the game.

Another thing to remember, it’s all about the story. That’s probably the number 1 rule of the VC club. The same way we say that the journey is more important than the destination, the story of a startup is almost more important than its actual output. The story is constantly developing but is always promising that there is always better things coming soon. If bad things were to happen, we turn them into good things. The story must always sell a better future so the valuation keeps going up and the excitement doesn’t die. At least that has to be true on the outside. I don’t think this approach is good or bad, this is just the way it is. It’s just part of the culture and everyone in it knows it.

Finally, something I didn’t hear a lot of founders talk about. Non-CEO founders aren’t seen as equals to the CEO. That usually becomes more and more real over time when the company grows to a big enough stage where the CEO founder has to start being a 100% CEO, making harder and bigger decisions and focusing on “CEO’ing” the company. VCs will provide more support and resources to a CEO than any other C suite roles and it makes total sense. It’s pretty logical when you think about it, the CEO sets the direction for the company, they energize, drive and motivate the exec team to accomplish the vision. VCs are most impactful when they can support the person setting the direction. Now here is what’s not often talked about. This is both a curse and blessing. As we all know, the people who are at the start of a company aren’t often the right people for later stages. And that keeps staying true as long as the company evolves. The thing is, as non-CEOs, it’s easier to redefine our roles or even move on in some cases. It’s much much harder for a CEO-founder. That can be very tough to realize that the company you dreamed of, built, launched might have overgrown you. Actually, scratch that, I don’t like this term overgrowing. It implies that you might not have grown fast enough. In my experience, it’s the opposite, I saw amazing founders who grew so much that they realized that their current role was not what they wanted or should be doing and they went out there and hired amazing execs (presidents, CEOs, CTOs, COOs..) who were a better fit for the current company stage. Because you have a rocket pushing your back, you don’t have much time to think about it, you just need to have a way to evaluate things quickly and make those decisions as fast as possible.I hope I was able to give you a glimpse of what it might be to the founder of a startup. It’s exciting, thrilling and very often tough in ways you didn’t quite expect. Being VC backed is a choice, it’s a path and it comes with its set of rules. If it’s your first time, you might not know those rules and it’s ok to ask (also start reading blogs like Fred Wilson’s, Mark Suster’s and many more, you will find patterns). If your partners have more experience than you ask them to share, good VCs also know to explain those things clearly and the ones I know won’t judge you for asking but they will keep you accountable. If you learn how to ask questions, you will be able to catch-up very quickly.

Finally, for those who might think I sound a bit pessimistic, you probably don’t know me well 😎 Lately, there has been some increased backlash against being VC and VC backed startups. Some arguments were absolutely valid and some maybe less. My hope is that this post will bring some color to the discussion in a form that I hope is as objective as I could be.

There are a decent amount of books, events, talks and coaches for CEOs but we hear very little about the struggles of being a CTO. I get it, talking about mistakes we made learning on the job isn’t glamorous, it’s actually quite embarrassing and painful to think about. But part of this exercise is therapeutic, the other will hopefully help current or future CTOs. So here is one thing I’ve learned the hard way and was a very valuable lesson:

Set expectations and put together an evaluation framework

This probably sounds painful, unnecessary and something that might slow you down. Yet, I so wish I had done it a few years back, it would have saved me from so many mistakes. Much the same way a test suite guards against regressions, an evaluation framework can make sure we aren’t backsliding on our responsibilities. Writing tests helps us ensure the proper behavior of a system and to avoid regressions. But more importantly, it builds confidence in what we ship and the process around it.

As CTO, one of my responsibilities is to build confidence in the engineering org. We wouldn’t ship code without tests, yet we might ship an entire org without an objective way to define and evaluate if it matches our expectations. One very important detail about this point is that engineering isn’t the sole stakeholder of this initiative. Confidence must be provided to the rest of the organization to keep it healthy. It doesn’t matter if you feel great about your part of the org when the rest of the company doesn’t feel the same. We also owe that to the CEO who has to justify the speed/quality of the org and for that they need to be able to trust the team can deliver or handle hard situations.

To build confidence measure and monitor those 3 areas

1. Velocity
2. Quality
3. Organization Maturity

Velocity

Most startups who grow decently fast, start feeling they are getting slower. New employees, new processes, management and paying back technical debts are all well known reasons for a slower team throughput. But how do we determine if we are slow or fast? Having a baseline and a way to measure is critical to building confidence. I can’t feel stronger about the requirement of having a way to measure the throughput of the team. The metrics you will pick won’t be perfect, they actually probably won’t be great but you need this baseline to help build confidence, detect issues and celebrate progress. The metrics you pick will depend on your development style, from story points of delivered stories to development flow metrics such as the number of deployments per day.

As much as I hated on agile story points for years, even if deeply flawed, this approach is still one of the best compromises I know of. Alternatively, measuring the speed at which code is shipped isn’t a bad idea. Consider the following metrics:

• the number of pull requests opened per week
• the number of pull requests merged per week
• the average time-to-merge (or % of Pull Requests (PRs) merged under a certain threshold),
• the number of production deploys

Those could give you a sense of the constant throughput of engineering team. If that number stagnates as you hire more, there might be a problem related to a new process in place, lack of investment on the infrastructure or a technical debt that needs to be addressed. However, if it increases too quickly you might have a quality issue. Don’t forget that measuring the speed of a team without evaluating the quality of the work is extremely dangerous.

Quality (confidence in the code base)

As a team, we should aim for a good balance between speed and quality. Aiming at 100% test coverage and perfect APIs, data models and documentation isn’t quite rational when we need to also ship at a decent velocity. However, regressions should be exceptional, churn in our code base should be reduced to a minimum, new team members should quickly find their way in the code base and new experiments should be launched without fear of taking down the entire system. Quality isn’t a goal in an of itself, the confidence in being able to grow and change behaviors in a safe matter is what matters. Some of the metrics to consider:

• Test coverage ratio (no need for 100% coverage but knowing where you stand helps a lot)
• % of times a pull requests breaking the build or fail to pass the test suite
• % of merged vs rejected PRs
• Number of comments by PR (you don’t want a too low number but you also don’t want too high of number)
• Number of found bugs (or bug fixes vs feature shipped)
• “turbulence” churn/complexity ratio in the code base. If the same code area keeps changing over time, it’s usually a symptom of a technical challenge such as a bad abstraction, poor implementation or a tech/product/business misalignment
• How outdated are the dependencies used in your code base

Technical debt is normal, quality is quite subjective but you can define as a team, a set of objective criteria that help give you a limited perspective on the quality of the team work. It will also help define technical values which are crucial as the team grows. Like velocity, measuring quality is super flawed, but remember that it’s also super important for engineering and for the rest of the company.

Organization maturity

This is the part I was trying to avoid early on as a founder. You remember that dream of running a tiny, flat team of great engineers not needing much maintenance? It might be at the beginning but isn’t sustainable and we owe the team to mature the org as we grow the company. I don’t yet have a good set of metrics since that’s the area I’m still the least comfortable with but here are some ideas:

Hiring / retention

Do we have hiring goals? Salary brackets? A good hiring process, a good on-boarding process? How about voluntary and involuntary departures? Ideally, your people org should support you with that, but in some cases, you are growing the team before you have a people org in place (HR).

Ratio of managers per Individual Contributor (ICs)

This is a simple metric to start measuring: define a target goal of the number of ICs you want to report to a manager and keep measuring. Managers focus on IC growth, quality, retention, and watch for trouble spots in the process. If you have too many ICs per managers, those managers probably won’t be as effective and that decision might cost the org a lot in retention and happiness.

Responsibilities, expectations and performance evaluations

This is one of the things that most growing startups struggle with. It often requires experienced managers focusing on transitioning from a flat organization to clearly defined expectations and performance evaluation. It’s also a costly thing to do and it comes with serious consequences. However, if the individuals on the team don’t know what they are expected to achieve and when they fall short, they can’t perform at their best. Those expectations need to be clear to everyone else in the company. By the way, this is something that I wish I had done for myself much earlier. By not putting that in place for my own role, I failed to lead by example and I would have honestly done my job better with a properly communicated set of responsibilities and performance evaluation.

Collaboration and flexibility with the rest of the company

Engineering is part of the greater picture, if it only operates well within its org, then it is failing the company as a whole. Empathy, adaptability, and creativity when it comes to working with the rest of the growing organization is key to a successful company. It is true that very often, tech startups have a big focus on engineering but that doesn’t make us the center of the company. If we aren’t able to communicate and serve others, we aren’t playing our role in the success of the company. This is a big reason why all the engineering metrics we discussed above are so meaningful for the rest of the company. Building trust, collaboration and being flexible in the way we all work towards the same objective is what make things happen.

One on ones

One on ones are dedicated meetings between two people in the company. This is often an opportunity for a manager and an IC to dig deeper in a safe place about how to become more productive, happier or develop one’s career. It’s also a great way to get to know people, detect patterns and build empathy. One on ones should happen regularly and across disciplines. They don’t have to be formal 1:1s, sometimes they are scheduled lunches once a month or a monthly small group hangout to discuss how things are going.

Written communication, information organization

As the team grows, information can’t be solely shared orally, we need things written down to last and to be efficient. We also need to quickly have access to those things. How does one find access to the latest metrics, how about this weird piece of software that was written by this early stage employee and that very few people understand? How does one find out who’s on vacation and who’s on call? Where and how do I report a bug, how can I contribute a test or a bug fix? Those things are simple when you are 20 or less, they get way more complicated as you grow.

Conclusion

This is a vast topic and I am only scratching the surface, but this is also the kind of information I wish I could send to myself a few years back. I was lacking a framework of measurement and was evaluating the growth of our org from a gut feeling perspective. As an engineer, I should have known that there was a much better way. Note also that not all those metrics need to be driven or improved at all stages. Once you have an evaluation framework in place it’s much easier to talk about where and how to focus our energy and to see the result of your investment. Having this framework in place and evaluating the results together, builds something extremely powerful: confidence. Self confidence and organization confidence is what drives healthy teams to success because they can face more complex problems.

Source: https://www.bloomberg.com/view/articles/2016-08-09/amazon-and-google-change-the-r-d-race

But we rarely talk about R&D at a startup stage. As a matter of fact, if you google those terms you will find a lot of articles about R&D tax credits and deductions but little about how other startups tackled setting up a R&D team. As Splice CTO, I believe we reached the stage where it’s important to look ahead and start planning in advance instead of being purely responsive. But how can you plan if you can’t see 6 to 12 months ahead. If your product team (engineers, designers, PMs) are in good hands and executing quarterly objectives in a healthy manner, you probably should consider doing some R&D. I previously talked about why hiring a VP of engineering is crucial and how I (as the CTO) am delegating the day to day operations. This is critical since without that, I couldn’t even consider thinking ahead.

What I want to avoid

I have been on both sides of the R&D line, I know how frustrating it can be to receive a half done concept project that I’m in charge of shipping and maintaining ASAP. I also know how frustrating it feels to work on great research and realize it will never ship because the R&D team is so disconnected from the engineering or business team. Too often R&D teams end up being a group of misfits lead by someone who likes to tinker with new cool stuff and think they can recreate the genesis of the company in a vacuum. Often those people are seen are “smart” and left alone because “we don’t want to lose them” and who knows, maybe they will come up with something amazing? Another common situation is that the company has a special task force that tackles special projects they have to ship outside of the normal dev cycle. Those situations aren’t healthy, they create tension within the teams and put pressure on processes.

I really believe that R&D should be designed to be an integral part of the company and not as a special island where cool projects are being developed. I also think it’s important for engineers and designers to get room to explore different possibilities without relying on a team that did all the design for them. It is important to keep the team motivated but also to help them grow.

Setting goals and scope

Settings goals for a team is a great way to define expectations and making sure we are aligned on why we work a certain way and what we are doing. Let’s first start by breaking down what I hope R&D will accomplish:

• Explore technical & business opportunities
• Evaluate feasibility
• Evaluate technologies

The big question is how do we do that, especially in a software company that believes in small iterations and agile planning? And how do we leave room for the engineering team to discover adequate solutions on their own.

Operating timeline

Now that we have goals we need to understand where the team will operate. By that I mean what part of the company timeline should the team focus on. The way I see it, here is how I break down a yearly timeline:

1. Current quarter: clear roadmap to execute on to well defined objectives coming from the executive team. This is the main focus of the VP of engineering and Product. R&D is not involved.

2. Next quarter: in planning, execs have rough business goals that need to be converted into objectives and metrics. Discuss strategic approaches to get there with VP of product and VP of engineering leading the concretization of the roadmap. R&D to advise if needed, potentially researching something that comes up and that can’t be handled by the “execution team” (more on that later).

3. 6 months from now: R&D to explore opportunities & feasibility to help shape future objectives and strategies. R&D team’s main focus.

Execution and R&D teams

Let’s talk about the execution team in comparison to the R&D team. The execution team is the team in charge of shipping, it’s the frontline team that focuses on this quarter. That team’s primary objective is to deliver solutions to the problems the exec team decided we wanted to tackle this quarter. This is not a team made of “janitors” (term I heard from a manager in a previous company), it’s also not a team of “contractors” that go down a list of stories pre-defined for them. It’s a team made of awesome people who understand our products, necessary compromises, value delivery and see the rubber hit the road on a daily basis. This is the team constantly building and shipping.

In comparison, the R&D team doesn’t ship, or at least not to users. This service team is there to explore very hypothetical or deeply complex questions. Their output is what feeds into the decision making that helps us as a company to pick the problems we want to tackle next quarter or the one after. They help the other teams understand our potential needs for planning, staffing or strategic decisions such as acquisitions. This is not a team of crazy scientists working on whatever might be cool one day. This is a team exploring questions that might get closer to our North Star faster. This is a team that is technical, product and business oriented since they serve those departments. This is a team that always give partial answers to difficult questions so others can use to decide if a problem is worth tackling and when.

Scoping

Some level of research should happens at almost all levels. Very rarely do we have exact answers to questions without spending some time researching/discussing options. Both the execution and the research teams should have room to do that. To help us with terms let’s talk about discovery vs research.

Discovery: is becoming aware of the solution to a well scoped, measurable problem. Discovery is generally short (from a day to a week) and ready to implement almost right away. One way to think about it is that we know a solution is right there, but we need to make sure we discover it.

Research: as per the dictionary definition: “the systematic investigation into and study of materials and sources in order to establish facts and reach new conclusions.” In other words, the gathering and establishments of facts that help us make educated decisions.

The execution team needs discovery time, they are given problems to tackle and have a lot of context but sometimes need room to explore and find the right approach to a specific problem we are about to tackle.

The R&D team is working on longer term research projects, with a less concrete output and lots of throw away work to be able to extract facts and reach new conclusions.

Output

So what should be the output of the R&D team. We talked about extracting facts and reaching new conclusions, but what does that mean?

• The R&D team doesn’t ship code to production. At best the team output might live in a sandbox or as libraries.
• The R&D team doesn’t hand out half-way done projects for engineering & product to finish.
• The R&D team provides engineers, PMs and designers with documented research, prototypes and at times suggestions.

In other words, lots of research documentation that can be leveraged by the executive and execution teams as well as libraries, throw away prototypes and flows.

Misc

• The R&D team isn’t solely technical
• R&D doesn’t handle maintenance and support of their research with the exception of libraries that weren’t explicitly given to another engineering team.
• R&D priorities and backlog are maintained by the CTO based on input from the exec team & other team members.

Research and development isn’t about software architecture/design or solely about technical feasibility and cost. It is about exploring business opportunities ahead of schedule to help the company make educated decisions to define our backlog.

Researching and Developing in the open

It’s crucial that R&D doesn’t operate in a vacuum. The R&D team is a service offered to the rest of the company. While the team doesn’t usually ship things to users, there is a lot of information that can be very valuable to a lot of employees. The work has to be done in the open and make easily available for any employee to dig into or ask questions.

Research documents are organized and made available on an employee-only site.

Stakeholders

While the CTO is running the day to day of R&D, the team can be seen as an “oracle” with various people asking “what if” questions.

The questions should focus around two large themes:

• Business opportunities
• Feature inquires

Here is an example. Marketing is interesting in growing our Windows market since they think there is room for expansion in a specific vertical. They might word a question such as: “what if we wanted to make our service A more valuable to people using products from company B. What could be done to increase conversion of users of company B?”

The request should come with some sort of importance level but with the understanding that the priority of what is being worked on will be made transparent and negotiable at an exec level.

Feature requests are more specific and usually come from designers, PMs and engineers. We have a theory that doing something around X would really help our metrics Y. What if we wanted to work on X, what are our options, what’s the feasibility and potential cost?

We are still at an early stage of settings up R&D at Splice. There are still a lot of unknowns but by trying to define roles, expectations and scope, I believe we put all the odds on our side.

I took some time to write down my thought process when interviewing candidates. Of course, the process changes based on the candidate, position and how the interview goes, but this is a decent albeit generic break down of how I’m approaching the experience. You should also take a look at this great document talking about how to prepare to interview technical candidates. This was a good exercise for me, forcing me to be explicit about my thought process during the short amount of time I meet with a candidate. It was also a great way to share with the team the kind of expectations I have about interviews.

Initial technical question

Please describe the lifecycle of a web request going from the browser back to the browser.

Expect the candidate to understand what a web request is, the concepts of DNS, ips, web server/app dispatch. Often candidates with less experience see the web app as a black box and can’t really explain routing and template/response rendering as well as returned responses.

Pay attention to the level at which the candidate starts, based on that, ask them to go deeper or shallower and see how comfortable they can explain technical concepts. Don’t try to trick them, but listen to the vocabulary they use as well as their comfort level. Do they go in the nitty gritty details of what happens inside the trackpad when pressure is detected or do they skip entirely the request processing when it hits an application server.

Follow up technical question

Let’s pretend our web app is now getting a lot of traffic, how can we handle more loads, ideally without increasing cost?

Expect the candidate to **** understand the concepts of caching and load balancing.

Make sure to bring up HTTP caching and make them explain how it works, if they worked with it, dig a little around status codes etc.. Bonus points if they understand and can explain pro/cons of HTTP/2. If they bring up technologies such as memcached, varnish, squid or other caching solutions, ask them about pros/cons.

Pay attention to how they react when they get to a place they don’t know. For instance don’t hesitate to ask about HTTP/2 push and how the headers are different than HTTP 1.1 or if they know that, ask about DNS load balancing or whatever they might not know. Check if the candidate is defensive or interested in learning during the interview. Do they feel bad or curious. Don’t push too hard but get a sense of how you might interact with them when they are faced to something they don’t know yet. After all, the goal is for the candidate to become your coworker.

Getting a glimpse of the candidate technical world view

What’s the hardest technical problem you ever solved? Why was it so hard, what did you learn?

No specific expectations, listen and try to understand how the candidate shares a hard moment and talk about something you have little context about. As a few questions to see how they respond to you digging a little in a potentially hard memory. Imagine you are sitting with this candidate in a post-mortem. Is their answer helping the team move forward?

Pay attention to see if they bring up a hard technical problem or a difficult to debug situation. Can they explain in depth what the problem was, did they learn anything? Think about the role you are hiring for and how you’d be expected to answer the same question.

Getting a glimpse of the candidate generic world view

What’s the hardest non-technical problem you ever solved that you’d be comfortable sharing?

No specific expectations, listen to the problem the candidate is bringing up to your attention. Do they over communicate or under communicate. Do they pick a problem that is easy to talk about or a problem that is unique? Be sensitive that the candidate might not want to share and that’s totally ok. In that case, diffuse by asking how they dealt with the first time they got stuck learning programming. Make sure the candidate understands that this isn’t about any kind of personal problems, but more of a generic non-technical situation and how the solution was debugged/solved.

Pay attention to the maturity of the candidate. Do they have an optimistic or a pessimistic worldview, do they spend more time talking about the problem or the solution. Are they humble or show offs?

Cocky candidate special question

Come up with a question that you don’t expect the candidate to know the answer. Maybe something not related to tech such as the mother sauces, the capital of Mali or the country of Georgia.

Expectation here is to ask a question that confuses the candidate because it’s poorly asked and not on topic (about food, geography, anything not technology). An arrogant candidate will probably make things up instead of admitting they don’t know. A humble candidate will admit they don’t understand or are surprised by the question, or even better they might admit they don’t know the answer. Don’t aim to make the candidate feel bad, just find a way to get passed their attitude. Be supportive.

Pay attention to see if the candidate is asking for an answer or wait for you to move on. If the candidate doesn’t ask for the answer, volunteer the answer so they can see it was a bit of a trick question.

Shy candidate special question

Besides programming, do you have any passions? Maybe sport, music, craft, TV/movies, games? Can you please share your passion with me?

Expected behavior is that the shy candidate should be able to find a way to communicate why and how they consume their passion.

Pay attention to not cutting them off, spend more time listening. Ask questions if they seem to be stuck. Make sure they see you are interested in what they are sharing.

Going deeper on the technical level of the candidate

I like to take concrete examples of problems we actually encountered. Depending on the position we are hiring for, I pick a different problem we already solved and present the situation to the candidate and ask them to talk it through with me.

It’s important to note that I ask our candidates to do a technical exercise on their own as a first step. The exercise is meant as base to discuss technical topics and approaches, not to really evaluate the solution provided by the candidate. So at this point of the interview, I should have amore or less good understanding of their skillset. That said, sometimes a candidate can totally miss the point of the exercise or just have had a bad day. That’s why I have this optional step to see how deep they can go while being comfortable and where their safe technical zone is at.

Conclusion

There aren’t any magical recipes for a good technical interview. As an interviewer, you should certainly be well prepared and your team and you must have defined the kind of profile you are interested in hiring for. Too often, the interviewer is looking for a candidate matching their interests and values. They are looking for a clone and that’s usually a bad idea. Sharing the same philosophy around how to tackle technical problems is critical but you also want diverse skillsets and perspectives. Having a discussion with your team about why you think the candidate will do well and what kind of assistance they will need should also really help you make a decision quickly.

Realizing that some of our company’s early assumptions around our user base might have been a bit off, we adopted a slightly unconventional technical approach to our desktop applications.

Splice is the advocate, companion of the modern musician. We developed a creation hub that connects the musician workstation with the cloud meaning that we sit close by the musician as she goes through her creation process. We opted early on for a transparent, discreet, independent integration meaning that we have an app running in the background handling various events and presenting a UI only when needed (similar to Dropbox). More complex experiences were designed to happen on the web requiring the user to have a browser open. Long story short, Splice is amazingly successful but modern musicians feel that the browser is often too far away from their creation space, and it’s a source of distraction. We have native apps written in Objective-C and C# (Mac and Windows). Keeping them in sync, implementing rich UIs and QA testing have been challenging. The quality of the user interactions and limited feature set have prevented our users from fully benefiting from what we have to offer. After some research, an investigation period and lots of discussions we opted to go with a unified stack:

• GUI: Chromium wrapper (GitHub’s Electron)
• Local UI/UX: TypeScript + Angular (currently v4)
• Business logic / network / audio: Go

This stack is the same on Windows and Mac. We currently still have some legacy native code but we are quickly phasing it out. More on that below.

TL;DR summary

We are moving away from native code and switching to Go + local web based views. Here is why:

1. Simplify: Maintaining & keeping in sync 2 desktop code bases is seriously complicated, painful & expensive.
2. Empower: Unifying the technical stack allows the entire team to contribute & innovate.
3. Enable: the new stack allows us to more easily/faster build new features in a cross platform fashion, using cutting edge technology while still having access to native APIs.
4. Acceptable overhead: while there is certainly a RAM overhead, it can and should be minimized.

This decision is slightly controversial because of two things, the choice of using Electron and the fact that we don’t stay in the JavaScript world and instead bring in Go.

Electron is controversial

Electron often attracts bad comments on Hacker News and it is associated with very large RAM usage. The Slack app being the usual cited example. The issue here is more complex than it looks. Electron is a wrapper around Chromium which is the heart of the Chrome browser. In other words, the application renders a web view as its main view. Web views aren’t by default as optimized as native GUI code and that often means a great RAM cost than purely native UIs. Besides that, Chromium itself loads its fair share of code in memory. The raw cost of an Electron window is around 40MBs vs ~4MBs for a native UI. Not great… but how much RAM do you have on your computer? Do you realize that Chrome allocates around the same amount of RAM when you open a new empty Chrome tab? That’s not an excuse for the memory overhead but it helped me put some context around the cost of the raw overhead.

The Slack app problem

People complaining about Electron RAM usage usually refer to apps like Slack using 1GB or more. This is often due to a couple issues: the architecture of the app and the way the frontend is coded.

During the investigation phase, I realized that it was best to avoid using node.js modules, especially native modules. Those were hard to work with (odd behaviors at times, hard to debug, hard to test) and often memory heavy. The second part is the UI cost, load a web app in your browser and look at the memory usage go through the roof as you load pics, gifs, sounds and lots of data. Some apps are better than others and VSCode is a good example of an Electron based app with a very reasonable memory footprint. That said it is very true that most frontend developers don’t benchmark or track the memory usage of their apps.

So yes, an app that has a browser based UI will require a bit more ram but it should be reasonable if we expect people to have decently modern machines. Also, this quote from my colleague after I complained about how much RAM Chromium uses, made me smile:

RAM is meant to be used, stop being cheap trying to keep all of it free - Loren to Matt

He has a good point, but of course you don’t want to swap either and you don’t want apps you aren’t actively using to squat most of your RAM.

Web frontend: empower the team, share knowledge

Chromium

So on the cons side we saw that there is a bit of a RAM usage increase but what about the advantages of using a web based GUI?

The main advantage was obviously only developing the user interface once and reuse it everywhere. But it goes further than that. This decision also empowers a bigger part of the team: all our frontend developers can become desktop application developers. They certainly need to understand the difference between web and local (technical and behaviors) but they don’t need to learn Cocoa or WPF. Also modern web APIs such as a Web MIDI and Web Bluetooth are available via a cross-platform API in Chromium if we decide to go that way. Adding new UI features is now much faster than it used to be (even for our talented engineers).

Web framework

Because we already use Angular on the web, we decided to also use Angular for the GUI. Our team is familiar with the idioms, patterns, challenges and in theory elements can be shared. But what’s even more interesting to me is that a frontend developer who’s used to working on the web should be able to jump in on the desktop and feel at home. This consistency is very nice allowing us more flexibility and knowledge sharing. As a side note, we aren’t serving the application from the web, resources are available locally but we will probably start using service workers in the future.

Separation of concerns

Another interesting aspect of our approach is that our logic doesn’t live in Electron, it is common for applications to implement the logic so it runs in the Node.js process. We opted to not do that and instead are running a Go + native code process. The frontend and the “core” service communicate via gRPC. This design decision ensures that the separation of concerns is really enforced. In other words, we are implementing the same architecture we know very well and use everywhere else: client/server. The difference is that the GUI doesn’t talk to the web APIs but instead to a local server (which proxies some calls when needed and handles online/offline states). While that might sound like an over-engineered decision, we are planning on having other applications consume our services and having a local API is a huge win. We consider the app GUI one of the potential clients in an non-exclusive relationship. We can imagine that tomorrow, other applications, plugins or Digital Audio Workstations would want to communicate with our service in a standard, documented and cross platform interface. This separation of concern and design consideration would allow for that.

Unified code base using Go

We invested many years in the native applications and some of the details were implemented as we found out edge issues around advanced features. Porting that code to Go isn’t trivial and we didn’t want to block on the port to ship the code. That’s why we still have an hybrid “core” architecture with native + Go code:

Connecting Go to original native code allows us to rewrite components in pieces and gRPC allows us to not break the contract between the UI/frontend and the core/backend. We are going to rewrite piece by piece until there is no more native code left. It’s also decently trivial to have Go communicate with native code so we can also still reach out to OS level features if really needed.

To us, gRPC is the most effective way to send data across boundaries, it is comparable to Websockets but with better tooling. gRPC has other advantages that I might cover them in a future post. The short version is that while it is a bit cumbersome at times, it’s be reliable and ensures a thought through API.

The heavy lifting remains in Go and was ported from native code. Go’s simplicity, native concurrency and testability gave us more confidence, reduce the amount of bugs and is empowering the entire backend team to join the fun. Go’s rich ecosystem also gives us access to a lot of interesting libraries for future features. Obj-C, Swift and C# are great languages but they aren’t part of our main languages. I did consider using C#/Xamarin to create a single cross platform app. But we would have needed a specialized and isolated team and we wouldn’t be able to leverage as much of our knowledge and team expertise.

Conclusion

My bet is that bringing the well known cloud architecture to the desktop is the right move for our company. There is a slight memory overhead but I think that by paying extra attention to it and with the help of other community members we are going to keep that to a very minimum. After evaluating the cost, I prefer to pay that short term penalty but have a solid architecture to build upon. I also really think that empowering the team and unifying the tech stack is going to make a big difference. We already saw that when we were able to “surge” UI development. Finally, the design bet is on more than a simple port but a redefinition of what a desktop will become. Having a powerful cross-platform service written in a modern language opens the door to a lot of opportunities.

Shameless plug: If that’s a challenge you’re interested in working on, our team would love to chat with you.

When you start as a technical founder, you are really a developer, quickly becoming a team lead. The team lead does leadership things but still codes and does very little management tasks. Then depending on how the company grows, usually you become a manager and now you have very little time to code. I tried to do both coding and management for a little while and I quickly realized I wasn’t doing the team or the company a favor. That’s why I opted to hire a great technical manager and focused on my CTO role.

“My role is to make sure we are making the right business decisions and we have a plan on how to implement them”.— Me, in this blog post

If my job is to identify technical opportunities, make and/or support business decisions, should I still code? If I do, is it because I like it or because I need to? Should all CTOs continue or stop to code? If I keep on coding, what should I be coding, how do I relate to the rest of the company?

The answers to those questions greatly depends on the business you are in and the kind of decisions that need to be made and the stage of the company. I’ll go even further and say that it probably also depends on the kind of CTO you are. But at the end of the day, you should look at this quote from Camille Fournier and decide if coding is helping you with that or not:

“No matter what, the CTO must understand where the biggest technical opportunities and risks for the business are and focus on capitalizing on them.”— Camille Fournier, The manager’s path

If you keep on coding, you should never stand on the critical path. Your code output shouldn’t be as important as your leadership role. Your coding should only be there to help you make the right business decisions. Hopefully you can delegate most of the coding, but in some cases, that’s not feasible or recommended. Let’s take a quick look at why you might still be coding as a CTO:

My team is too small

You are probably a technical cofounder or you were given a CTO title to make you feel good. Your job is to build the right foundation for your company and help make the right business decisions so you can grow the team. By the way, you probably should start looking for a good team lead and maybe a good manager for your team.

I am the expert

Time to share the knowledge or hire other experts. You are a single point of failure, that’s dangerous! (Note: that’s the situation I’m trying to get out of, I have a lot to say about this)

I do a better job and deliver faster than the rest of the team

Maybe you didn’t hire the right people and should correct that. Or maybe you don’t have the right process in place yet, or finally, maybe you aren’t doing the rest of your job well and should focus on that. (Other option: you need an ego check)

I feel guilty to let the team struggle, they have so much on their plate

If they have too much on their plate, maybe the workload isn’t adequate for the team size. It’s your job to help define the business objectives and have a plan in place to execute. If the team needs your help with code, the planning wasn’t done right. It might happen from time to time but it should be really exceptional.

I need to prove to the company (or myself) that I still “got it”

That’s an obvious one, but more common than you think. Especially when you have a very strong engineer becoming a CTO. As engineers we value ourselves based on their capacities at designing solutions and all a sudden their actual coding abilities aren’t needed. Jordan Hubbard once told me that key to be a happy great engineer manager when you’ve been a great developer is to focus on helping others getting as good and even better than you.

So if coding helps you figure out the biggest technical opportunities and risks for the business, keep doing it but don’t stay on the shipping critical path! Otherwise, you probably shouldn’t code much anymore.

Gil Scott-Heron — The Bottle

I have to admit I was a bit anxious at the beginning of this journey. I was worried it might turn out to be a “feel good field trip” for privileged tech scene members. I didn’t want to experience a show put on to make me feel as though I was giving back. I was concerned things would quickly turn into sensationalism and objectification of black and brown people. To be very frank, for a good part of the program, I was still very skeptical.

But then I realized that while the event was as perfectly orchestrated as a well-choreographed TV show (timed segments, queued up music, MC routines), the men in front of me, their emotions and their experiences were very real. It suddenly hit me, the reason everything was “staged” was to facilitate human connections in a very short amount of time. Both the Entrepreneurs in Training (EITs) and the volunteers had their own preconceived ideas, expectations and anxieties about engaging with each other, and every aspect of the event was designed to break down those barriers and facilitate deeper connections.

As I got to know these guys, I realized there was something very special about them. Something beyond the way they looked, beyond their criminal past, beyond the fact that they were making themselves vulnerable and allowed themselves to cry. All these men had something in common, something a lot of us have struggled to hold on to since November 8th, 2016: hope. Most of them aren’t getting out anytime soon, yet they had toiled for months refining very well thought out business plans. They organized their thoughts and ideas, prepared their pitches and learned them by heart, even with the knowledge that, for most of them, the ability to bring those plans to life is at best a distant dream.

It’s their deep sense of hope that makes me sure that even the messed up society we are living in won’t be able to stop them once they are out. These men are able to see beyond racism, classism, biased and the unfair judiciary system. They are able to see beyond their sentences, their regrets, the pain they caused and their underdog status. What they see is so great it gives, them a glance at peace and happiness.

“What else do you have if you don’t have hope?” — Michelle Obama

These men acknowledge their crimes and the consequences they bear. But as a society, are we sending our parents, brothers, sisters, friends and loved ones to prison to be punished or rehabilitated? The reality is that those in charge of the judiciary system don’t send their own family, friends or loved ones to prison. When those being processed in the system don’t look like your friends or family, it is probably easier to see punishment as an appropriate solution.

Nonetheless, the men I met managed to find hope. To me, that was a shock. And my shock made me wonder if maybe deep down, I had given up on them before even entering the prison. What they taught me is that I have no right to lose hope because of the political climate and the anger of people around me. If they can defy the odds and are able to dream again, we all can.

In a day in age when fear is being leveraged to drive wedges between us and fuel hate, I was also reminded of how hope naturally leads to compassion and kindness. The prison is segregated by ethnicity: black, latino, white and others inmates are isolated. Yet the graduating group was mixed, and they were truly supportive of each other. When we shared difficult memories, you could feel, hear and see the support of people around you. Hugs, pats on the back and handshakes were given in multitude and in a non-discriminatory way.

During the pitch competition, as participants were eliminated from round to round, not a single person complained or had a frown on his face. To the contrary, they all cheered for the EITs who made it to the next round. I’ve honestly never experienced that. Take a group of founders, VCs, and engineers and make them go through the same process. I can guarantee that you’d see quite a bit of drama, and not nearly as much empathy and support.

As the graduation event wrapped up, the family of an exceptional EIT named Timothy was told that he would be released that night. The other EITs cheered so hard for him, without envy or jealousy. They wanted him to start living his dream. They believed in his success, and he was their ambassador. I thought to myself, “these are the kinds of leaders we need — people who lead by example and inspire others.”

That night, when Timothy passed the prison gate, he was accompanied by his mom and a friend. He also carried with him the hope and selfless encouragement of his fellow EITs and Defy team.

—

Acknowledgments: I can’t recall the first person who introduced me to Defy but thank you! I’m grateful for having been a part of this experience. Thanks Defy Ventures staff, Mark Suster and Upfront Ventures for putting this event together. Thank you also to Laura Gómez, Heidy Vega, Hamet Watt, Laura Zax and Juan Pablo Buriticá for their reviews and advice.

• Personality & skills of the engineering lead
• Business needs
• Team size
• Company finances

Should you do the job or hire someone?

Assuming you are currently the tech lead at your company (tech founder / CTO / lead engineer), you first need to take a honest look at what you are good at and what you want to do. Many founders aren’t best suited to be CTOs, they often are less driven by engineering design, architecture and long term business strategies but are more inclined to find “smart hacks” to get quick returns on small investments. This is a very useful skill when you start and aren’t quite sure where the company is heading. But this skill might not help you be the best CTO. Some other founders or early developers are amazing at selling their vision and getting people to join their cause. They convince very talented people to quit their cushy jobs to take a risk with them and they make sure everyone is both happy and produces amazing results. They create amazing teams and great cultures, great signs of potentially brilliant VP of engineers.

I was recently chatting with someone and was asking how they see the difference between a VP of eng. and a CTO and they were telling me that they currently lack experience so they should first be a VP and will then want to “climb up the career ladder” to become a CTO. That didn’t quite sound right to me, it would be like if someone would tell me they first needed to be a mobile software engineer before becoming a backend engineer. VPs and a CTOs are almost 2 separate tracks that require different skills. That’s why I think it’s very important to realize that it’s totally OK for founders or early engineers to hire CTOs (or CEOs for that matter) and take a VP of eng or tech lead role. But probably more importantly, it’s critical to know when you need help! Startups are intense and early employees and founders take on a lot of roles and while nobody likes to complain, saying you need help is not a sign of weakness, it shows strength and humbleness. If you know you need help, now is the time to hire! (or to reduce the team size)

Understand your business needs and capacity

The second important point is to understand the business needs. In startup land, you are often pushed to hire as many engineers as possible, as fast as possible. You look more credible when you say you have a team of 30 than when you say you have a team of 5 engineers. Founders (and VCs) want to prove they can hire and if they were to be acquired, they know that the team size factors in their company valuation. I personally find these arguments very silly, but I that’s unfortunately how many people think. Unless you are aiming for a acqui-hiring exit, I’d suggest to try to avoid falling in this trap. During the early phases of a startup, it’s the role of a CTO to define the real engineering needs and discuss with the other execs what kind of budget can be allocated to wages. We know that there will be some churn and therefore waste. If you can keep the overhead, low, you can run leaner and extend the runway. Furthermore, by having a lower head count, you can afford to pay your awesome engineers better (and recruit more experienced devs). I personally believe in running smaller team until you find the inflection point. For most startups, throwing more developers won’t make a huge difference since they first need to figure out their market fit. Focus on hitting those milestones and when you know that have to scale the team soon: then you will know it’s time to hire a VP of engineering. Don’t wait too long otherwise you are going to set your future VP for failure.

The company size is an interesting measure, as mentioned earlier, it’s often believed that the more the better. It gives you legitimacy. When you are well organized, have clear business goals, objectives and a clean code base, that’s usually true. But it also comes with a huge overhead. I approximate this overhead to represent around 10–15% of manager time per direct report. When you reach 6–10 engineers, it becomes a full time job (if you want to do it decently well). Certainly some teams can operate manager-free for longer than others but it comes at a different cost. My recommendation if you start seeing your team grow fast is to do one of the following:

• keep the engineering team small until you can hire a VP of engineering and properly scale the team
• act as a vp of engineering and let someone else be the lead engineer/CTO

Once you start seeing great traction and revenue, which can be via direct revenue or investment, that’s when you should start actively looking for the person that will be your team’s quarterback. Note that the time to get to that point might vary greatly from company to company and that’s totally normal. The hard part is what makes startups hard: timing. You don’t want to hire a VP too early, but you also don’t want to wait too long and bringing someone to “save” a team that’s falling apart.

So to summarize, you need to define how you will fit in the picture, pick the right team size once you reach the critical inflection point take the time you need to find the right person to help improve your process while scaling the team and keeping everyone happy and productive.

Splice now has its first VP of engineering: Juan Pablo Buriticá. There is often a lot of confusion between the roles of a VP of engineering and a CTO. So let me clarify, for me, what the VP should focus on:

• people
• now
• deliverables while the CTO should focus on:
• technology
• the future
• enabling business opportunities

Investing in a “people architecture”

These two sides of an engineering organization have always been a challenge for me. I invested a lot in our technology architecture (and it paid off) but I knew that we needed to invest the same energy in our “people architecture” if we wanted to scale. Having a good design and foundation is critical to good software. If you don’t have that and your company becomes successful, you will need to do a rewrite. Doing full rewrites is a dangerous, slow, expensive and scary thing. The same rule applies to building an engineering team. I won’t name any but I am sure you’ve seen your share of startups that hired a bunch of smart people, tried to keep everything flat and realized it wasn’t working out. Deliverables & morale went down, they had to rethink their entire organization… in other words, they had to do a full rewrite and it came at a huge cost.

“Head or gut?” — Bruce Willis

Doing day-to-day work as well as going deep in technical solutions & making long term technical decisions is really hard. These things are in constant conflicts. But at the beginning of a startup, you don’t have a choice and it comes at a cost that will directly affect one of these 2 areas: your technology or your people. Not a great choice is it? But it is the sad truth and the more people you add, the harder it gets. Which is one of the reasons I kept the team relatively small even if we had so much to do.

Being an enabler

Juan Pablo brings his technical and managerial expertise as well as his deep care for building up and supporting an amazing engineering team. With him on board I really feel that we can scale the team without putting more strain on each individual. But more than that, I feel that I wasn’t the manager I wanted to be and I didn’t have enough time to go deep on really hard technical problems that could be extremely beneficial to the company. Not being a great manager has always been one of my flaws and something I wanted to improve on. I never had great managers and I wanted to break the cycle. I’m told that part of being a leader means finding better people than you and empower them to do their job the best they can. I can’t be more thrilled to have Juan Pablo join us, scale and make our team and each individual member even better.

So Juan Pablo is going to take over the day-to-day: one on ones, backlog grooming, delivery process, ownership and accountability… These things honestly take a lot of my time, so with those gone, what’s left for me?

My mission is to discover and be responsible for the implementation of technical solutions that move the needle

Defining my role

To me, my #1 role is to leverage technology to serve our business. That sounds really vague so let me try to clarify what I mean: My mission is to discover and be responsible for the implementation of technical solutions that move the needle at a company level. But be warned, there is a big distinction between this and R&D. In R&D, you are trying things that might move the needle (and you aren’t usually responsible for implementing them) when as a CTO, I place bets that have to be closely related to business goals and deliver them in the short/medium term. My failures or successes have a direct effect on the company.

It’s all about business

Being a CTO is a strategic role, it’s not about picking the absolute best programming language, container solution or cloud vendor. It’s not even about evangelizing my engineering vision or reading hundreds of white papers to become the most knowledgeable engineer in the team, it’s about making the right business decisions. My role is not to “make it happen”, this is what a VP of engineering is responsible for. My role is to make sure we are making the right business decisions and we have a plan on how to implement them. My role is also to find new opportunities by leveraging technology. But I can’t do that without a great CEO, a great VP and a great team and lots of trust going around.

This is why it’s such a hard job, you have to be able to understand and appreciate things from a business and technical perspective. You have to deconstruct and reconstruct the technical challenges and solutions linked to each possibility and quickly evaluate the risk/reward factor of each. You often need to say no and explain in simple terms why you think the risk is probably too great. You have to have a great relationship with the CEO so these discussions don’t turn in continuous battles and you need to have a strong trust relationship with the VP of engineering so your decisions can be backed by the team and value can be delivered promptly.

On one hand, you are representing the technology and the engineers, fighting for low tech debts, exciting solutions and utopic technical perfection while on the other, you are representing the business with its objectives, revenue goals and market opportunities. As far as I’m concerned, the CTO role might the most schizophrenic C-suite position out there and I’m looking forward to continue exploring the essence of my role as Splice and I evolve.

Note: There is another entire discussion about knowing when your company should split these roles. What are the signs, how did it go for Splice/me. Let me know if you’d be interested in a follow up post. — Update: I posted a follow up post regarding timing.

The Go community can feel out of reach

Go was written and is developed by a group of very experienced people, in the context of a very mature tech company (Google). Because of that, the early community reflects its creators. Most are experienced developers, computer science focused and interested in specific technical problems. Compared to many other programming communities, our community can sometimes feel drier in the sense that we focus a lot on low level details and tooling and aren’t focusing much on carving out a path for newcomers.

We want Go to be more accessible

As a CTO and Co-Founder of a tech startup, I bet a lot on Go. It has been serving us great and to be honest, I love our community. But I have also heard many times that Go is an advanced language that is only for experienced developers. Or that the Go community is full of experts making it very hard for new developers to “break-in”. All communities need new people to challenge the “old-guard”, push new ideas and become tomorrow’s leaders. Without new members, communities end up aging, and eventually go away. We need to embrace new users and make sure we don’t leave less experienced developers high and dry due to minor things we didn’t think about.

Go isn’t for me, I just started learning how to code

This is something I’ve heard many times. You have a new developer who’s just starting to discover the wonderful world of programming and they are being told they should start with JavaScript (JS), Ruby or Python because the other languages such as Go are too complicated for them and a novice can’t and should not try to learn an “advanced” language.

I couldn’t disagree more. Sure, with these languages you will get your first running code experience faster, but you may also struggle very quickly with other things such as syntax and other errors that are reported only when your code runs. You will discover behaviors considered weird because you don’t understand them (yet). I wrote about that before and to me, this is similar to getting a credit line with bad terms. I do think JS, Ruby, Python etc. are great languages and should be learned, but I think a lot of developers, if not all, would benefit greatly from learning Go first.

Go isn’t revolutionary in itself; it does have a few paradigms that are somewhat unique, but its syntax and concepts are very similar to most programming languages. As a matter of fact, if you know Go fairly well, you can probably learn Ruby, Java, Python, JS, C# decently quickly since you already have the mental mapping of all the core concepts. Because Go is a small language (the language itself has a limited set of features), most of these other languages have similar programming concepts as Go (except for concurrency) but add a few extra layers on top. One can therefore easily argue that learning Go is easier and can be used as a foundation to explore other languages. Go isn’t the solution to all problems and many developers learning Go will end up preferring to move on to frontend languages such as JS/TypeScript, some will prefer a more classical Object Oriented approach, some will want more flexibility in the type language, some won’t have a choice and will write the programming language used at their job. This is totally fine! But learn Go so you can have a solid foundation and then you can move on to something else you might like better or better suit your needs.

Not enough content for beginners

It saddens me but it’s true: we have very little content for true beginners. Because our community was started by developers with a lot of experience, we created a lot of content for existing developers, not total newbies. On the other hand, communities such as Ruby and JS are larger and always had a big influx of totally new developers attracted by the promise of creating web applications quickly (at the cost of learning the fundamentals which sooner or later comes to bite you in the 🍑).

Swift came in as a replacement of Objective-C, but Apple and the community are doing an amazing job attracting new developers. Granted it’s a little bit easier for Apple to capture new developers since they are usually motivated to build something tangible as quickly as possible and Apple really wants revenue generating native apps on their platforms.

Improve the educational materials and new user experience

I don’t see any reasons why we can’t aim to do the same and be much much better at teaching the basics, sharing knowledge and welcoming new developers. After all, we brag that our language is simple; it’s time to show new developers that investing in Go early on is the right move for them. Let’s fix the gap in documentation and make onboarding exciting.

The great news is that we all started as newbies and we all learned something. If you’ve learned something, you should be able to explain it simply. Start writing small blog posts or give local talks on basic Go concepts. Maybe explain how to use the printf functions, how to organize your code in packages, or show how to use godoc or a feature of your favorite IDE. Our language is simple and the questions new developers are facing are finite; let’s make sure we create awesome content for new Gophers in the quest for knowledge!!

Thanks to Carlisia Campos, Tiffany Jernigan, Andrew Gerrand and Francesc Campoy Flores for reading drafts of this and providing feedback.

Video

In this blog post about team communication at Splice, I go into more details about synchronous vs asynchronous communication, the pros and cons and the process we explicitly defined.

I don’t know other companies using the EOD email approach yet but unless you’re a one man show, I can’t think of a reason not to do it. I also can’t stress enough that you need be mindful of others focus when using communication tools such as Slack.

TL;DR intro series teaching programmers the basic audio DSP concepts in a practical manner. No DSP knowledge is expected.

Analog to Digital conversion

To be able to access and manipulate audio data via our code, an audio signal needs to be converted from analog to digital. This is not something you usually implement yourself, in most cases it is done at the sound card level. Sound cards do the job of converting the signal, in other words, they are analog to digital converters (ADC).

It’s very important to understand how the conversion is done because it has direct repercussions on what we will do.

We have a continuous signal (the audio signal) and we need to convert it into numerical values. The numerical values needs to represent the sampled analog signal as closely as possible. To do that, a discrete signal needs to be created. A discrete signal is a fancy name for a time series representing the sampled signal. In other words, we measure the signal at a certain frequency (let’s say 44,100 times a second) and each value (an integer or a floating-point we call a sample) is added to our series. (This article assumes we are using PCM representation).

Sampling rate

The amount of time per second we take a sample from our analog signal is called the sampling rate. Common audio sampling rates are 44.1kHz, 48kHz and sometimes 96kHz.

There is a very important explanation about why we need to sample at a value greater than 40kHz.

The human hearing range falls into a frequency range of 20Hz to 20,000 H (that said you probably don’t hear very much above 17kHz). A requirement for our analog to digital conversion is that we can capture and then later on recreate an analog signal that will cover the entire frequency range we are interested in.

The reason we need to sample at a rate greater than 40kHz is due to the findings of Harry Nyquist. Nyquist, studied and wrote about conversion of continuous to discrete signals. One of the discoveries he made is called the Nyquist rate and simply put it says that you need to sample at at a greater frequency than twice the bandwidth you want. Since we are trying to sample a range of 20,000Hz we need to sample at 2x20000 = 40000 to cover the entire frequency range.

Fun fact, the default 44.1kHZ sampling rate comes Sony setting the standard by recording audio content on video equipment and wanting to have a system that was PAL/NTSC compatible (PAL runs at 25fps, NTSC at 30fps). They were able to store the exact same amount of 44,100 samples per seconds using different amounts of lines/frames.

Bit rate

The precision of the numerical value we use to store the sample is called the bit depth. The bit depth will change the range of values we capture, take for instance the sampling example below.

The analog signal in red is being sampled at a 4-bit bit depth meaning that each sample is represented by an integer taking 4 bits in memory. 4 bits give us a maximum range of 16 integer values (4*4) Our sample value will be between 0 and 15. 4-bit precision really isn’t enough to properly represent the reach dynamic range of audio signals. Most audio content is sampled at 16 or 24 bit (8, 20 and 32 bit depths are also sometime used). A 16-bit integer resolution offers a dynamic range of 96 dB (which can be improved using dithering but that’s outside of the scope of this article).

Too low of a resolution and you are missing auditive information (and getting extra noise), too high of a resolution and it’s a waste. 32 floating-point bit depth is often used later on in the audio chain when doing complex audio processing. This process allows for greater precision when manipulating audio data and rounding can have a negative effect on the output (DAWs, synthesizers, effects…). The data is usually converted back to the source bit depth making things easier/transparent to the caller.

Fun fact: The famous TR-909 drum machine only used a bit depth of 6-bit!

File formats: WAV & AIFF

Now that we learned how the audio signal is being converted to a digital stream, it’s interesting to see how it’s being stored. The 2 main uncompressed PCM formats are WAV and AIFF. These formats store the same data slightly differently and provides valuable information of the stream content via file headers and data chunks. To be able to do any kind of offline processing, we need to be able to read and write such formats. In a future post, we will write a coder/decoder that explains how these file formats work (and get a sneak peek into lossy compression formats such as MP3).

Back in November I was in Paris for dotGo, one of my favorite Golang conferences. I was invited to speak and wanted to share something concrete with the audience, something based on my experience with Go at Splice. Go is designed for simplicity with only one way of doing one thing. That makes preparing talks pretty challenging since a lot has already been said and there is not much more to add. A lot of us pick Go because it’s has concurrency built-in, but it doesn’t mean that writing efficient concurrent code is instinctive or obvious.

I walked the audience through the parallelization of code execution and all the mistakes we usually make. Using an Arduino and a set of LEDs, I show how goroutines operate and how to write working Go code.

Official post about the talk

Video

Slides

I chose the beautiful city of Medellin, Colombia to say my final goodbye to the awesome Ruby community. 10 years of being part of a community isn’t nothing and I took time to reflect on my personal experience and discuss the future of the language and its community.

Video

Slides

I found a solution to help us: coding and music!

My goal: to wake up my daughter with a custom music playlist starting a few minutes before I go see her.

We have an old Mac Mini at home that I recently upgraded it to run Yosemite. This recent version of OS X added JavaScript as an alternative to AppleScript. My goal is to write a little script that will play a given playlist and then I want to schedule this script to play every school morning. The Mac Mini is connected to an Airport Express with usb speakers located in my daughter’s bedroom.

JavaScript for Automation

Apple has some documentation about how to use JS to automate different tasks. To be honest the documentation isn’t really good in comparison to their usual doc. Also, the technical implementation is pretty hacky/buggy but that won’t prevent us to have a bit of fun.

The code to start iTunes and play our playlist if very straight forward:

app = Application('iTunes')
app.activate()
app.stop()
playlist = app.sources["Library"].userPlaylists["morning"]
try {
	playlist.play()
}
catch(err) {
	console.log("The playlist probably doesn't exist", err)
}

As you probably guessed, this script loads iTunes and then uses its scripting API to load the main library and look for a playlist called “morning”. I’m not proud of the ugly try/catch, but playlist is an instance of ObjectSpecifier which isn’t evaluated until a method is called on it. You can think of it as a lazy container. The problem is that if we try to call play() on a playlist we didn’t find, then an error is thrown. I didn’t find a way to check if the underlying value is null so I had to catch the error.

Once we have the above code, we have two options, put it in a script or convert it into an app. When developing your automation, it’s highly recommended to use Script Editor.app which ships with the OS. From within the app, you can run your script and test / “debug” it. Script Editor also allows you to export your script as script, script bundle and app. The easiest way is to export our code as an app:

Note that you can also write a script and create a osascript shebang, or even evalute your automation JS in the terminal:

$ say "you are listening to" `osascript -l JavaScript -e 'Application("iTunes").currentTrack.name()'`

So we have a script to start iTunes if needed and play our morning playlist. Now we need to schedule our app to start every school day:

Launchctl

OS X has a builtin scheduling system called launchd. It’s kind of like a cron scheduler but with more options. Unfortunately figuring something as simple as scheduling a recurring script is much harder than it should. So here is my plist file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
	<string>morning.playlist.itunes</string>
	<key>ProgramArguments</key>
	<array>
		<string>/usr/bin/open</string>
		<string>-a</string>
		<string>wakeup.app</string>
	</array>
	<key>RunAtLoad</key>
	<false/>
	<key>StandardErrorPath</key>
	<string>/tmp/morning.playlist.itunes.stderr</string>
	<key>StandardOutPath</key>
	<string>/tmp/morning.playlist.itunes.stdout</string>
	<key>StartCalendarInterval</key>
	<array>
		<dict>
			<key>Hour</key>
			<integer>6</integer>
			<key>Minute</key>
			<integer>55</integer>
			<key>Weekday</key>
			<integer>1</integer>
		</dict>
		<dict>
			<key>Hour</key>
			<integer>6</integer>
			<key>Minute</key>
			<integer>55</integer>
			<key>Weekday</key>
			<integer>2</integer>
		</dict>
		<dict>
			<key>Hour</key>
			<integer>6</integer>
			<key>Minute</key>
			<integer>55</integer>
			<key>Weekday</key>
			<integer>3</integer>
		</dict>
		<dict>
			<key>Hour</key>
			<integer>6</integer>
			<key>Minute</key>
			<integer>55</integer>
			<key>Weekday</key>
			<integer>4</integer>
		</dict>
		<dict>
			<key>Hour</key>
			<integer>6</integer>
			<key>Minute</key>
			<integer>55</integer>
			<key>Weekday</key>
			<integer>5</integer>
		</dict>
	</array>
</dict>
</plist>

Gist

Note that I called my app wakeup.app and I put it in my Applications folder. My launch agent starts the app when it’s called, but it doesn’t do that when the system loads the service (RunAtLoad is set to false). I’m also logging out stdout and stderr to tmp files so I can debug if something goes wrong. Finally the schedule is defined in the StartCalendarInterval key with a daily entry Monday to Friday at 6:55am.

Save the plist file as wakeup.playlist.itunes.plist and drop it in ~/Library/LaunchAgents/ and load it via launchctl:

$ launchctl load -w ~/Library/LaunchAgents/wakeup.playlist.itunes.plist

That’s it, everything should work fine, however if you want to make sure it will, you might want to unload the plist, edit it so RunAtLoad is set to true and reload it. At this point, your playlist should play. If it doesn’t, then check the log files to see what happened. When everything is good, unload, go back to the original version and reload.

There is plenty more you can do with JS Automation for Mac, if like me you are listening to a lot of music while coding, you might be interested in knowing that Spotify and VLC are scriptable (and so are most browsers).

Read more on the Go Challenge website

pattern_1.splice
Saved with HW Version: 0.808-alpha
Tempo: 120
(0) kick     |x---|x---|x---|x---|
(1) snare    |----|x---|----|x---|
(2) clap     |----|x-x-|----|----|
(3) hh-open  |--x-|--x-|x-x-|--x-|
(4) hh-close |x---|x---|----|x--x|
(5) cowbell  |----|----|--x-|----|

The challenge is going on for 2 weeks until March 15, a new challenge will be posted on April first. Prizes and conditions available on the website. Comments and discussions available on HackerNews

Note that I also wrote another challenge that we usually give our frontend candidates. If you’re not into Go and prefer JS, you might like this challenge better.

• how session authentication works in Ruby on Rails,
• the crypto used to protect the data,
• the security implications and
• how to share the session with applications written in other languages than Ruby (with an example in Go).

To illustrate this pretty dry topic, I told the story of Alice and Bob, two entrepreneurs building a succesful web application and facing some serious challenges.

Video

Slides

I gave a talk at Go SummerFest and then later on another one during I/O. For more information about the events, see this blog post from the Go team.

Slides

Full screen slides

Video

My talk was recorded but it is not yet online.

Very much like with many other programming languages, a challenging part of Go is to learn how to write idiomatic code. The good news is that Go makes refactoring easy (and already has a lot of conventions). I strongly recommend this post from Peter Bourgon about Go at SoundCloud and the extra conventions they follow (Splice also follows the same conventions).

One of my favorite Go projects is the gobot project by HybridGroup.

The Gobot project is pretty young and I noticed a few things that could be improved so I offered my help to Ron, Adrian and the rest of the team. Our discussion quickly turned into a fun group refactoring session (featuring @kytrinyx, @deadprogram, @codegangsta, @jnbeck, @adzankich )

Packages

Gobot is split into multiple packages, a core and a few other packages. The gobot team, out of habit chose to put a package per repo. After further discussions, we chose to bring all official packages inside the same repo to keep things easier and to keep the import paths clean and logical.

So instead of having:

github.com/hybridgroup/gobot
github.com/hybridgroup/gobot-sphero
github.com/hybridgroup/gobot-...

All the none-core packages are moved to subdirectories:

github.com/hybridgroup/gobot
github.com/hybridgroup/gobot/sphero
github.com/hybridgroup/gobot/...

This also allowed us to fix the package names gobot-sphero is now simply sphero

Which also allowed us to simplify the following code:

From:

type SpheroAdaptor struct {
	gobot.Adaptor
	sp io.ReadWriteCloser
}

To

type Adaptor struct {
  gobot.Adaptor
  sp io.ReadWriteCloser
}

We did that with a few other types and methods all over the packages.

We had a discussion about what lead to the multiple repos vs one repo. There are legitimate cases for both approaches but in this situation, the decision was based on a misunderstanding. The author thought that by importing the top package, all sub packages would also be somewhat included in the build, making the binary bigger than needed. Since Go only compiles and links packages imported, moving all packages within the same repo wouldn’t change the binary output. Note that this is not because in this specific case we have all packages in the same repo that this is the right thing to do every single time.

doc.go

By conventions, package should contain a doc.go file that contains an overview of the package and often some information so the developer trying to use the library can find the right entry points.

As usual, the standard libraries are a good example, here is the net/http doc.go file.

Using a constructor

We spent some time refactoring master.go which is the file implementing the code handling one or multiple robots (which can each have multiple devices).

The original function code looked like this:

func GobotMaster() *Master {
  m := new(Master)
  m.NumCPU = runtime.NumCPU()
  return m
}

There are a few things that aren’t really idiomatic in this code. The first thing is that by convention, constructors are usually called New<Type>. Secondly, the community seems to follow the following stylistic choice: only use new and make when you need to set the capacity (make([]string,3)) Finally we don’t need to allocate a variable. Here is the refactored code:

func NewMaster() *Master {
  return &Master{NumCPU: runtime.NumCPU()}
}

Cleanup package vars

In the original code, we had a variable called trap which was a function living at the top level of the package:

var trap = func(c chan os.Signal) {
  signal.Notify(c, os.Interrupt)
}

The func was then used to handle signals. The author chose to use a variable so he could mutate it in the test suite and avoid sending an interrupt when testing. We realized we could avoid having this function variable at the top of the package by moving it as a field on the Master type and setting the default func in the constructor.

func NewMaster() *Master {
	return &Master{
		NumCPU: runtime.NumCPU(),
		trap: func(c chan os.Signal) {
			signal.Notify(c, os.Interrupt)
		},
	}
}

The code still behaves the same and we can still overwrite the trap function in our tests (since the tests are part of the same packge, the non exported field is available) but we got rid of a top level var.

Reading from a channel

The following code was ranging over a predefined channel (c) of signals. and when a signal would arrive, all robots belonging to the master would be halted and disconnected.

for _ = range c {
  for r := range m.Robots {
  	m.Robots[r].haltDevices()
  	m.Robots[r].finalizeConnections()
  }
  break
}

The code above works well but could be cleaned up a little:

// waiting on something coming on the channel
<- c
for _, r := range m.Robots {
	r.haltDevices()
	r.finalizeConnections()
}

This code does the same thing but simpler. We are trying to read from the channel which will block (we don’t care about the result so we don’t capture or could have used an underscore). Then we loop through each robot and stop them. We managed to remove a for loop on the channel (with an odd break) and made the code intent clearer.

Chainable functions and typed nils

Next, we tackled the following method:

func (m *Master) FindRobotDevice(name string, device string) *device {
	robot := m.FindRobot(name)
	if robot != nil {
		return robot.GetDevice(device)
	}
	return nil
}

The funny thing about this method is that it’s not needed. We could get the same result by calling:

m.FindRobot("bot name").GetDevice("laser")

When I said that, someone suggested that it might be a bad idea since FindRobot() might return nil and now we would be calling GetDevice() on nil and bad things would happen. Looking at the code, it was actually easy to fix.

Here is the original code:

func (r *Robot) GetDevice(name string) *device {
	for _, device := range r.devices {
		if device.Name == name {
			return device
		}
	}
	return nil
}

Here is the refactored version:

func (r *Robot) GetDevice(name string) *device {
	if r == nil {
		return nil
	}
	for _, device := range r.devices {
		if device.Name == name {
			return device
		}
	}
	return nil
}

Did you spot the difference? We just added a check to see if the pointer (r) was nil, if it is, we just return nil. When I added the code above, the person who was worried about calling GetDevice() on nil was scratching his head.

Golang does something very interesting (and a bit surprising if you come from a dynamic language), it returns a nil pointer of the type we defined as return type. Let’s walk through the code by rewriting it slightly differently:

var bot *Robot
bot = m.FindRobot("unknown name")

At this point if FindRobot() didn’t find a robot, bot is still of type *Robot but the pointer is nil. Because we defined a method GetDevice() on *Robot, we can call:

bot.GetDevice("x-ray")

The GetDevice() method will execute and will return nil right away because we check if the pointer is nil.

The fact that nil pointers have types has 2 important implications, the first one is that you can nicely chain methods without checking at the caller site if the returned value is nil. The second is that your methods should expect to be potentially called on a nil pointer and should properly handle such cases.

Note: Go team member Andrew Gerrand suggested on Hacker News to name the method Device instead of GetDevice. The word Get is almost always redundant. In the same chain of thoughts, maybe we should rename FindRobot just Robot.

Collection types / type aliasing

I’m writing this post on my way back from GopherCon and there was one more thing I wanted to clean up and share with you. This is a nice pattern I use often to simplify my code.

Our Robot type has a connections field and a devices field:

type Robot struct {
  // .. fields removed to simplify the example
	devices       []*device
}

To avoid always having to manually loop through the slice, a method is defined on pointers to Robot. This method iterates over the devices and halts them:

func (r *Robot) haltDevices() {
	for _, device := range r.devices {
		device.Halt()
	}
}

This code is totally fine but from an API design perspective, wouldn’t it be nicer to use?:

r.devices().Halt()

One of the nice things with this approach is that the concept of halting, which really belongs to the devices, doesn’t need to leak into the Robot world.

To implement the suggested API change, we need to define a type alias:

type DeviceCollection []*device

We can now define methods on our new type:

func (c DeviceCollection) Halt() {
  for _, device := range c {
    device.Halt()
  }
}

We then need to update our Robot type:

type Robot struct {
  // .. fields removed to simplify the example
  devices       DeviceCollection
}

And we are done with our refactoring.

One last note, since we might need to call different methods on our collection we could create an iterator method.

func (c DeviceCollection) Each(f func(*device)) {
  for _, d := range c {
    f(d)
  }
}

// which can be called like so
r.devices.Each(func(d *device){
  d.Halt()
})

Conclusion

Needless to say that we had fun. The refactoring went much further and we removed the use of reflections, some sleeps and much more. The code is going through a nice cleanup before reaching 1.0 and I can only encourage everybody to play with Gobot, there are very few things as fun as Go and Robots! (The code is open sourced, look at it, add new drivers, send PRs!)

I’d like to thank Ron Evans and the Hybrid Group for open sourcing their code and sharing the fun with all of us. I can’t wait for the next LA Go + Robot hack night.

Finally, Splice is hiring, our stack uses a lot of different technologies but our backend is all in Go and we are always looking for talented engineers. Drop me a line if interested.

A big thanks to all the people who recommended these books, I’m looking forward to 2014!

Cover	Title	Type	Recommended by
	The Interestings	Novel	Andrew Weissman @aweissman
	Lexicon	Novel	Amos King @Adkron
	The Orphan Master's Son (Pulitzer Prize for Fiction)	Novel	Bryan Woods @bryanwoods
	The Flamethrowers	Novel	Andrew Weissman @aweissman
	Mr. Penumbra's 24-Hour Bookstore	Novel	Chris Conrey @conrey
	Lying Awake	Novel	Andrew Weissman @aweissman
	Perdido Street Station	Novel	Olivier Gutknecht @olg
	The Sisters Brothers	Novel	Andrew Weissman @aweissman
	Killing Floor (Jack Reacher, No. 1)	Novel	Dr Nic @drnic
	Jonathan Strange and Mr Norrell	Novel	Olivier Gutknecht @olg
	Thinking, Fast and Slow	Psychology / Business Decision-Making	Greg Borenstein @atduskgreg
	The Signal and the Noise: Why So Many Predictions Fail-but Some Don't	Business Planning & Forecasting	Mathias Meyer @roidrage
	The Life You Can Save: Acting Now to End World Poverty	Philanthropy & Charity	Theresa Preston-Werner @tpdubs2
	Thinking in Systems: A Primer	Problem Solving	Mathias Meyer @roidrage
	High Performance Browser Networking	Technical	Kevin Stewart @kstewart
	NOS4A2	Novel	Jason Rush @diminish7
	Basic Economics: A Common Sense Guide to the Economy, 4th Edition	Economics	Ben Hughes @rubiety

If you want more, you can check the previous book recommendation list

Since I own both apps, I thought it would be as simple as sharing the secret session key and re-implementing Rails crypto process in Go. It turned out to be a lot more interesting.

In a nutshell, here is what I discovered:

• It’s totally doable! Here is my Go package.
• If you are using a version of Rails older than 4.0, you’d better upgrade ASAP!
• Rails has been criticized for security issues, but the current solution has been vetted by many experts.
• Rails serializes session data using Ruby Marshal which means that someone with the secret key can inject arbitrary code in the session and it will execute server side. Switch to JSON, MessagePack or other safe serialization formats.
• Security is (still) hard.

Rails Cookies are Dangerous

Because Rails serializes and deserializes the session and any encrypted/signed cookies using Ruby’s Marshal library, someone with the app secret can wreak havoc. They can embed arbitrary Ruby code into the cookie, submit it with a request, and the server-side deserialization will execute that code without you noticing. Granted, this requires the attacker to have the app secret, but since 99% of the apps out there have the shared secret in their source code, anyone with access to the source code has this data. It’s not data you can easily rotate when employees leave or when you are done working with contractors. Anybody with the shared secret is a potential attacker. Start by moving this data out of the code base and into an environment variable.

Rails doesn’t let you change the default serializer directly. But Rails relies on ActiveSupport for its crypto work and AS supports swapping the serializer. Some people in the community are aware of this issue and monkey patch Rails to serialize their sessions using JSON or another alternative. Here is an Airbnb article and Rails 3 patch. Here is my Rails 4 monkey patch to switch the serialization to JSON. I’m using it in production with Rails 4, but it’s untested on Rails 3.

You can modify either solution to use MessagePack instead of JSON if you want to fit more data in the 4K cookie size.

Understanding Rails Session Encryption

Once I addressed the serialization issue, I had to reimplement the crypto work done by Rails to encode and/or sign the data.

Most of us just rely on our frameworks/libraries to do the right thing, but we rarely look under the hood. I ported the logic to Golang which has an amazing support for crypto (albeit lower level than Ruby). My Go package contains an explanation of the code logic and the examples needed to decode/verify as well as encode/sign sessions that are compatible with Rails.

Here is a high level summary of what Rails does when it encodes and signs your session data:

key_generator = ActiveSupport::CachingKeyGenerator.new(ActiveSupport::KeyGenerator.new(app_secret_key, iterations: 1000))
derived_secret = key_generator.generate_key("encrypted cookie")
sign_secret = key_generator.generate_key("signed encrypted cookie")

encryptor = ActiveSupport::MessageEncryptor.new(secret, sign_secret)
session_content = encryptor.encrypt_and_sign({hello: "world"})

The session_content string is then set as the session cookie value. Note that you could do that in any Ruby app using ActiveSupport, making it easy to share sessions between Ruby applications (like Rails & Sinatra).

Technically, there are a lot of things going on. To avoid using the same secret to sign and encode data, Rails relies on derived keys using PBKDF2 (password based key derivation function). It treats the app secret as a password and applies a pseudorandom function 1000 times (Rails default) using a default salt. The result is a derived key so the original password isn’t shared. The derived key can be regenerated identically if the salt and secret are known (because the function is pseudorandom).

The two derived keys are then passed to the MessageEncryptor class which uses MessageVerifier to do the signing. The generated keys are 64 bytes long. One key goes to the encryptor while the other goes to the verifier.

The verification is done via HMAC (SHA1) and it uses the full 64 byte key. The encoding is done via AES 256 CBC only using the first 32 bytes of the encryption derived key. Rails will only generate a 32-byte key since that’s the expected key length.

The session data is serialized (using Marshal by default) then encoded via AES. Both the encoded string and the IV are encoded using base64 and joined in a string using a predefined format.

At this point, the session is encoded but it could be tampered with. To avoid that, Rails signs the encoded data using the verifier (HMAC) and appends the base 64 encoded signature to the encoded data.

To decode and verify the data, Rails repeats the process in reverse using the serializer to deserialize the data.

Note that you can also rely on the the same crypto process to safely encode/sign any data you want to share. If you’re ok with the data being user-readable, sign it to make sure it isn’t tampered with along the way. If you don’t want it to be user-readable, encrypt it first then sign the encrypted data.

Sharing the Session with Non-Ruby Apps

Many apps are moving to an SOA approach. That often means multiple languages living together in production. Sharing a web session can be very useful, especially until you switch to a SSO solution.

The key is to start by having the session data serialized in a format that is available in all your relevent languages. JSON, XML MessagePack, and protobuf are good examples.

The second step is to reimplement the crypto dance I just explained above. The good news is that I’ve already done it for Go. Using that example, you should be able to port it to other languages (Node, Scala/Clojure/Java, Rust, Elixir, Python or whatever you fancy).

https://github.com/mattetti/goRailsYourself/tree/master/crypto

Even though the test suite isn’t perfect (yet), it should greatly help you through the porting process. To be honest the hardest part was understanding the process, not writing the code. Most languages have decent crypto libraries to do the hard parts for you. But for Go I had to implement lower level pieces like the PKCS7 padding for the AES CBC encryption/decryption.

Hopefully this article was helpful and you now better understand how Rails does its session encryption. Once you understand the process Rails uses, you can implement it in any language.

** Finally, if you interested in working on interesting and challenging problems like these ones, consider joining the Splice team! **

The format was a bit different than usual, I had a 40 minute slot and divided in 2, I made my points for 20 minutes and invited two special guests on stage to discuss the topic. The reason for this format is because I think we all learn better by looking at things from different perspectives. I can’t thank enough Sandi Metz and Katrina Owen for their contributions.

Here are a few points that I think are interesting and that we discussed during the talk.

There isn’t such thing as bad code

Bad code doesn’t exist, you have code that can be interpreted/compiled and code that doesn’t. “Good” and “bad” are moralistic designations, not scientific ones. Let’s try to stop using these terms to refer to talk. Let’s be more precise when arguing about code, “what do you mean by bad?” Is it hard to maintain, hard to understand, slow etc.. ? Always refer to the context in which the code was written. Don’t use it as an excuse to defend anybody’s potentially hurt ego, but instead to explain why the code was written caring about certain values instead of others. Most code turns emo after a little while, being able to understand the context, helps a lot the devision process when facing such code.

It’s all about expected outcome and context

As developers, we aren’t paid to write code, we are paid to build products, to convert ideas into something “concrete”. When I use my favorite app, I don’t care that the code is beautiful, I care that it works, that it’s stable and provides me with what I need. If the code is written in Go, Pascal, Erlang, VB or Ruby doesn’t matter at all. That the code has full test coverage and was written in an agile manner using weekly scrums and TDD is probably as important as knowing the Pantone color of a company’s logo: it only matters to the people deeply involved.

Don’t focus on how to build, focus on why you build things. Then the how will come as you learn from others, experiment and discover “how” to build the “why”. The “why” is often the constant, the “how” keeps on changing as we collect more information.

We love rules

When you learn, rules are easier. Katrina made a very good point during our discussion. When she teaches, she needs to have very strong rules that she can justify. That said, her rules can change, and as you become better, you can start challenging the rules. Rules are somewhere between training wheels and a guard-rail. They are very useful but shouldn’t be used to attack other people.

Ruby has a language doesn’t enforce many rules, as a matter of fact, Matz designed a language so you can set your own rules, or barely have any rules. This fact probably explains why so many people are after solid rules they can rely on. Rules are easy to follow and are a good reference. The Ruby language doesn’t have enough rules to some, so the community is helping by coming together to define them. This is by the way, the biggest difference with Python and Go where these 2 languages explicitly want to only have 1 way of doing one thing.

Communication and team work

The key to building a good product (which is what we are paid to do) is communication. Unfortunately, us developers, are on average, pretty terrible at that. We need to do a better job communicating with the rest of the organization (i.e: anybody who’s not an engineer). It usually starts by the designers. The expected outcome of the product should be clear, well understood by all and easily evaluable. Within the engineering team, coding values should be set explicitly. Not everyone will agree but when we write code, we should all care about the same overall values so our work is consistent. When something happens and we have a disagreement, it’s easier to refer to our values to decide what solution to pick.

Slides

Video

I’m starting this new project, what technology do you think I should use?

Categories = [] +++

Update: Speaking of HN, here is the thread for this post

I’m not yet super familiar with all the std lib packages, so when I wanted to test my Go web services, I wrote a few lines of code to create a multipart file upload function that was building the body from scratch. Once I was done messing with the various headers, boundary protocol etc.. I started testing some edge cases, I found some bugs in my code. Looking at Go’s packages, I realized that all the tools were already available for me to use. I was just lacking a good example. Walking through the test suite I finally figured out how to write a simple multipart file upload example with some extra query params.

Hopefully this example will be helpful to some of you.

package main

import (
	"bytes"
	"fmt"
	"io"
	"log"
	"mime/multipart"
	"net/http"
	"os"
	"path/filepath"
)

// Creates a new file upload http request with optional extra params
func newfileUploadRequest(uri string, params map[string]string, paramName, path string) (*http.Request, error) {
	file, err := os.Open(path)
	if err != nil {
		return nil, err
	}
	defer file.Close()

	body := &bytes.Buffer{}
	writer := multipart.NewWriter(body)
	part, err := writer.CreateFormFile(paramName, filepath.Base(path))
	if err != nil {
		return nil, err
	}
	_, err = io.Copy(part, file)

	for key, val := range params {
		_ = writer.WriteField(key, val)
	}
	err = writer.Close()
	if err != nil {
		return nil, err
	}

	req, err := http.NewRequest("POST", uri, body)
	req.Header.Set("Content-Type", writer.FormDataContentType())
	return req, err
}

func main() {
	path, _ := os.Getwd()
	path += "/test.pdf"
	extraParams := map[string]string{
		"title":       "My Document",
		"author":      "Matt Aimonetti",
		"description": "A document with all the Go programming language secrets",
	}
	request, err := newfileUploadRequest("https://google.com/upload", extraParams, "file", "/tmp/doc.pdf")
	if err != nil {
		log.Fatal(err)
	}
	client := &http.Client{}
	resp, err := client.Do(request)
	if err != nil {
		log.Fatal(err)
	} else {
		body := &bytes.Buffer{}
		_, err := body.ReadFrom(resp.Body)
    if err != nil {
			log.Fatal(err)
		}
    resp.Body.Close()
		fmt.Println(resp.StatusCode)
		fmt.Println(resp.Header)
		fmt.Println(body)
	}
}

Example’s source code on GitHub

All the work is done in the newfileUploadRequest function and really, the mime/multipart package hides all the complexity of creating a multipart request.

The key is to set a new multipart.Writer:

writer := multipart.NewWriter(body)

The writer will do all the work and will write directly to our body (which itself is a buffer of bytes).

We then create a part for the file form entry with the name of the file param and the name of the file (that we extracted using the path/filepath package). We need to add the content of the file to the file part, we use the io.Copy() to do so. In the first version of this article, I had used io/ioutil Readall to read the content of the file (see code here). However a few readers rightfully mentioned that I should instead copy content from the file to the part instead of temporarily loading the content of the file in memory. Here is an even more optimized version using goroutine to stream the data, and here is the full example using a pipe.

part, _ := writer.CreateFormFile(paramName, filepath.Base(path))
_, err = io.Copy(part, file)

The multipart.Writer takes care of setting the boundary and formating the form data for us, nice isn’t it?!

Then for any extra params passed as a map of string keys to string value, we use another function of the multipart.Writer type:

writer.WriteField(key, val)

Once again, the writer takes care of creating the right headers, and to add the passed value.

At this point, we just need to close our writer and use our body to create a new request.

writer.Close()
req, _ := http.NewRequest("POST", uri, body)

One last thing before triggering our request, we need to set the header that contains the content type including the boundary being used. Once again, the Go lib has us covered:

req.Header.Add("Content-Type", writer.FormDataContentType())

As a reference, here is the generated body:

--0d940a1e725445cd9192c14c5a3f3d30ea9c90f1f5fb9c08813b3fc2adee
Content-Disposition: form-data; name="file"; filename="doc.pdf"
Content-Type: application/octet-stream

%PDF-1.4
%????
4 0 obj
<</Type /Catalog
// removed for example
trailer
<</Size 18
/Root 4 0 R
>>
startxref
45054
%%EOF
--0d940a1e725445cd9192c14c5a3f3d30ea9c90f1f5fb9c08813b3fc2adee
Content-Disposition: form-data; name="title"

My Document
--0d940a1e725445cd9192c14c5a3f3d30ea9c90f1f5fb9c08813b3fc2adee
Content-Disposition: form-data; name="author"

Matt Aimonetti
--0d940a1e725445cd9192c14c5a3f3d30ea9c90f1f5fb9c08813b3fc2adee
Content-Disposition: form-data; name="description"

A document with all the Go programming language secrets
--0d940a1e725445cd9192c14c5a3f3d30ea9c90f1f5fb9c08813b3fc2adee--

Golang might not be as high level as Ruby or Python, but it’s not too far off and it certainly comes with some great std libs. I know I recently caught myself writing a lot of small scripts in Go, something I used to do in Ruby. I think this is mainly due to the fact that Go is compiled, designed for concurrency, has great std libs and is quite easy to write.

Hopefully this code sample illustrates how easy Go can be and can also serve as a reference point if you are looking for a way to do multipart upload.

Before optimizing, we need to measure.

Without a solid baseline, how can you say that the time you invested in making things better wasn’t a total waste?

True refactoring is done with a solid test suite in place. Developers know that their code behavior didn’t change while they cleaned things up. Performance optimization is the same thing: we need a good set of metrics before changing anything.

There are plenty of monitoring tools out there, each with its own pros and cons. The point of this article isn’t to argue about which one you should use, but instead to give you the some practical knowledge about Graphite.

Graphite is used to store and render time-series data. In other words, you collect metrics and Graphite allows you to create pretty graphs easily.

During my time at LivingSocial, I relied on Graphite to understand trends, issues and optimize performance. As my coworkers and I were discussing my recently announced departure, I asked them how I could help them during the transition period. Someone mentioned creating a Graphite cheatsheet. The cheatsheet turned into something much bigger than I expected and LivingSocial was nice enough to let me publicly publish this short guide.

For a more in depth dive into the statsd/graphite features, look at this blog post

Organizing metrics

There are many ways to feed Graphite, I personally used Etsy’s statsd (node.js daemon) which was being fed via the statsd RubyGem. The gem allows developers to push recorded metrics to a statsd server via UDP. Using UDP instead of TCP makes the metrics collection operation non blocking which means that while you might theoretically lose a few samples, your instrumented code performance shouldn’t be affected. (Read Etsy’s blog post to know more about why they chose UDP).

** Tip **: Doing DNS resolution on each call can be a bit expensive (a few ms), target your statsd server using its ip or use Ruby’s resolv standard library to only do the lookup once at initialization.

Note: I’m skipping the config settings about storage retention, resolution etc.. see the manual for more info.

Namespacing

Always namespace your collected data, even if you only have one app for now. If your app does two things at the same time like serving HTML and providing an API, you might want to create two clients which you would namespace differently.

Naming metrics

Properly naming your metrics is critical to avoid conflicts, confusing data and potentially wrong interpretation later on. I like to organize metrics using the following schema:

 <namespace>.<instrumented section>.<target (noun)>.<action (past tense verb)>

Example:

accounts.authentication.password.attempted
accounts.authentication.password.succeeded
accounts.authentication.password.failed

I use nouns to define the target and past tense verbs to define the action. This becomes a useful convention when you need to nest metrics. In the above example, let’s say I want to monitor the reasons for the failed password authentications. Here is how I would organize the extra stats:

accounts.authentication.password.failure.no_email_found
accounts.authentication.password.failure.password_check_failed
accounts.authentication.password.failure.password_reset_required

As you can see, I used failure instead of failed in the stat name. The main reason is to avoid conflicting data. failed is an action and already has a data series allocated, if I were to add nested data using failed, the data would be collected but the result would be confusing. The other reason is because when we will graph the data, we will often want to use a wildcard * to collect all nested data in a series.

Graphite wild card usage example on counters:

accounts.authentication.password.failure.*

This should give us the same value as accounts.authentication.password.failed, so really, we should just collect the more detailed version and get rid of accounts.authentication.password.failed.

Following this naming convention should really help your data stay clean and easy to manage.

Counters and metrics

StatsD lets you record different types of metrics as illustrated here.

This article will focus on the 2 main types:

• counters
• timers

Use counters for metrics when you don’t care about how long the code your are instrumenting takes to run. Usually counters are used for data that have more of a direct business value. Examples include sales, authentication, signups, etc.

Timers are more powerful because they can be used to analyze the time spent in a piece of code but also be used as a counters. Most of my work involves timers because I want to detect system anomalies including performance changes and trends in the way code is being used.

I usually use timers in a nested manner, starting when a request comes into the system, through each of the various datastores, and ending with the response.

Monitoring response time

It’s a well known fact that the response time of your application will both affect the user’s emotional experience and their likelihood of completing a transactin. However understanding where time is being spent within a request is hard, especially when the problems aren’t obvious. Tools like NewRelic will often get you a good overview of how your system behave but they also lack the granularity you might need. For instance NewRelic aggregates and averageses the data client side before sending it to their servers. While this is fine in a lot of cases, if you care about more than averages and want more detailed metrics, you probably need to run your own solution such as statsd + graphite.

I build most of my web-based APIs on wd_sinatra which has a pre_dispatch_hook method which method is executed before a request is dispatched.

I use this hook to both set the “Stats context” in the current thread and extract the client name based on HTTP headers. If you don’t use WD, I’ll show how to do the same thing in a Rack middleware.

def pre_dispatch_hook
  api_client = extract_api_client_name(env)
  Thread.current[:stats_context] = "#{api_client}.http.#{env['wd.service'].verb}.#{env['wd.service'].url}".gsub('/', '.')
  # [...]
end

Then using Sinatra’s global before/after filters, we set a unique request id and start a timer that we stop and report in the after filter. If we were using Rails we’d get the unique identifier generated automatically.

Before filter:

require 'securerandom'

before do
  Thread.current[:request_id] = request.env['HTTP_X_REQUEST_ID'] || SecureRandom.hex(16)
  response['X-Request-Id'] = Thread.current[:request_id]
  @instrumentation_start = Time.now
end

After filter:

after do
  stat = (Thread.current[:stats_context] || "http.skipped.#{env["REQUEST_METHOD"]}.#{request.path_info}") + ".response_time"
  $statsd.timing(stat, ((Time.now - @instrumentation_start) * 1000).round, 1) if @instrumentation_start
end

Note that this could, and probably should, be done in a Rack middleware like this (untested, YMMV):

# require whatever is needed and set statsd

class Stats
  class Middleware

    def initialize(app)
      @app = app
    end

    def call(env)
      request = Rack::Request.new(env)
      Thread.current[:request_id] = request.env['HTTP_X_REQUEST_ID'] || SecureRandom.hex(16)
      response['X-Request-Id'] = Thread.current[:request_id]
      api_client = extract_api_client_name(env)
      Thread.current[:stats_context] = "#{api_client}.http.#{request.request_method}.#{request.path_info}".gsub('/', '.')
      @instrumentation_start = Time.now

      response = @app.call(env)

      stat = (Thread.current[:stats_context] || "http.skipped.#{env["REQUEST_METHOD"]}.#{request.path_info}") + ".response_time"
      $statsd.timing(stat, ((Time.now - @instrumentation_start) * 1000).round, 1) if @instrumentation_start
      response
    end

  end
end

Note that the stats are organized slightly differently and will read like that:

<namespace>.<client name>.http.<http verb>.<path>.<segments>.response_time

The dots in the stats name will be used to create subfolders in graphite. By using such a segmented stats name, we will be able to use * wildcards to analyze how an old version of an API compares against a newer one, which clients still talk to the old APIs, compare response times, etc.

Monitor time spent within a response

We’re collecting stats on every request so we can see request counts and median average response times. But wouldn’t be better if we could measure the time spent in specific parts of our code base and compare that to the overall time spent in the request?

We could, for instance, compare the time spent in the DB vs Redis vs Memcached vs the framework. And what’s nice is that we could do that per API endpoint and per API client. In a simpler case, you might decide to monitor mobile vs desktop. The principle is the same.

Let’s hook into ActiveRecord’s query generation to track the time spent in AR within each request:

module MysqlStats
  module Instrumentation
    SQL_INSERT_DELETE_PARSER_REGEXP = /^(\w+)\s(\w+)\s\W*(\w+)/
    SQL_SELECT_REGEXP = /select .*? FROM \W*(\w+)/i
    SQL_UPDATE_REGEXP = /update \W*(\w+)/i

    # Returns the table and query type
    def self.extract_from_sql_inserts_deletes(query)
      query =~ SQL_INSERT_DELETE_PARSER_REGEXP
      [$3, $1]
    end

    def self.extract_sql_selects(query)
      query =~ SQL_SELECT_REGEXP
      [$1, 'SELECT']
    end

    def self.guess_sql_content(query)
      if query =~ SQL_UPDATE_REGEXP
        [$1, 'UPDATE']
      elsif query =~ SQL_SELECT_REGEXP
        extract_sql_selects(query)
      end
    end
  end
end

ActiveSupport::Notifications.subscribe "sql.active_record" do |name, start, finish, id, payload|
  if payload[:name] == "SQL"
    table, action = MysqlStats::Instrumentation.extract_from_sql_inserts_deletes(payload[:sql])
  elsif payload[:name] =~ /.* Load$/
    table, action = MysqlStats::Instrumentation.extract_sql_selects(payload[:sql])
  elsif !payload[:name]
    table, action = MysqlStats::Instrumentation.guess_sql_content(payload[:sql])
  end

  if table
    $statsd.timing("#{Thread.current[:stats_context] || 'wild'}.sql.#{table}.#{action}.query_time",
                        (finish - start) * 1000, 1)

  end
end

This code might not be pretty but it works (or should work). We subscribe to ActiveSupport::Notifications for sql.active_record and we extract the info we need. Then we use the stats context set in the thread and report the stats by appending .sql.#{table}.#{action}.query_time

The final stats entry could look like this: auth_api.ios.http.post.v1.accounts.sql.users.SELECT.query_time

• auth_api: the name of the monitored app
• ios: the client name
• http: the protocol used (you might want to monitor thrift, spdy etc..
• post: HTTP verb
• v1.accounts: the converted uri: /v1/accounts
• sql: the key for the SQL metrics
• users: the table being queried
• SELECT: the SQL query type
• query_time: the kind of data being collected.

As you can see, we are getting granular data. Depending on how you setup statsd/graphite, you could have access to the following timer data for each stat (and more):

• count
• lower
• mean
• mean_5
• mean_10
• mean_90
• mean_95
• median
• sum
• upper
• upper_5
• upper_10
• upper_90
• upper_95

Instrumenting Redis is easy too:

::Redis::Client.class_eval do

  # Support older versions of Redis::Client that used the method
  # +raw_call_command+.
  call_method = ::Redis::Client.new.respond_to?(:call) ? :call : :raw_call_command

  def call_with_stats_trace(*args, &blk)
    method_name = args[0].is_a?(Array) ? args[0][0] : args[0]
    start = Time.now
    begin
      call_without_stats_trace(*args, &blk)
    ensure
      if Thread.current[:stats_context]
        $statsd.timing("#{Thread.current[:stats_context]}.redis.#{method_name.to_s.upcase}.query_time",
                         ((Time.now - start) * 1000).round, 1) rescue nil
      end
    end
  end

  alias_method :call_without_stats_trace, call_method
  alias_method call_method, :call_with_stats_trace

end if defined?(::Redis::Client)

Using Ruby’s alias method chain, we inject our instrumentation into the Redis client so we can track the time spent there.

Applying the same approach, we can instrument the Ruby memcached gem:

::Memcached.class_eval do

  def get_with_stats_trace(keys, marshal=true)
    start = Time.now
    begin
      get_without_stats_trace(keys, marshal)
    ensure
      if Thread.current[:stats_context]
        type = keys.is_a?(Array) ? "multi_get" : "get"
        $statsd.timing("#{Thread.current[:stats_context]}.memcached.#{type}.query_time",
                         ((Time.now - start) * 1000).round, 1) rescue nil
      end
    end
  end

  alias_method :get_without_stats_trace, :get
  alias_method :get, :get_with_stats_trace

end if defined?(::Memcached)

Dashboards

We now have collected and organized our stats. Let’s talk about how to use Graphite to display all this data in a valuable way.

When looking at timer data series, the first thing we want to do is create an overall represention. Your first inclination is probably an average.

The problem with the mean is that it’s the sum of all data points divided by the number of data points. It can thus be significantly affected by a small number of outliers.

The median value is the number found in the center of the sorted list of collected data points. The problem in this case is that based on your data set, the median value might not well represent the real overall experience.

Neither median nor mean can summarize the whole story of your system’s behavior. Instead I prefer to use a 5-95 span (thanks Steve Akers for showing me this metric and most of what I know about Graphite). A 5-95 span means that we cut off the extreme outliers above 95% and below 5%.

Span

Here is a comparison showing how the graphs can be different for the same data based on what metric you use:

Of course the span graph looks much worse than the other two, but it’s also more representative of the real user experience and thus more valuable. Here is how you would write the graphite function to get this data.

Given that we are tracking the following data-series:

stats.timers.accounts.ios.http.post.authenticate.response_time

The function would be:

diffSeries(stats.timers.accounts.ios.http.post.authenticate.response_time.upper_95,
           stats.timers.accounts.ios.http.post.authenticate.response_time.upper_5)

Alias

If you try that function, the graph legend will show the entire function, which really doesn’t look great. To simplify things, you can use an alias like I did in the graph above:

alias(diffSeries(stats.timers.accounts.ios.http.post.authenticate.response_time.upper_95,
                 stats.timers.accounts.ios.http.post.authenticate.response_time.upper_5),
      "iOS authentication response time (span)")

Aliases are very useful, especially when you share your dashboards with others.

Threshold

Another neat feature you might add to your graph is a threshold. A threshold is a visual representation of expectations. Say, for example, that your web service shouldn’t be slower than 60ms server side. Let’s add a threshold for that:

alias(threshold(60), "60ms threshold")

and here’s how it would look in a graph:

Draw Null as Zero

Another useful trick is to change the render options of a graph to draw null values as zero. Open the graph panel, click on Render Options, then Line Mode and check the Draw Null as Zero box.

Here is a graph tracking a webservice that isn’t getting a lot of traffic:

You can see that the line is discontinued, that’s because the API doesn’t constantly receive traffic. If your data series gets only very few entries, you might not even see a line. This is why you want to enable the Draw Null as Zero.

SumSeries & Summarize or how to get RPMs

By default graphite shows data at a 10 second interval. But often you want to see less granular data, like the quantity of requests per second.

Let’s say we didn’t use a counter for the amount of requests, but because we used the middleware I described earlier, we are timing all responses. Graphite keeps a count of the timers we used, so we can use this count value with a wildcard:

stats.timers.accounts.*.http.post.authenticate.response_time.count

If we were to render a graph for this stat we would see a graph per client. Right now we only care about showing the total amount of requests. To do that, we’ll use the sumSeries function:

sumSeries(stats.timers.accounts.*.http.post.authenticate.response_time.count)

The graph looks pretty but it’s hard to understand what kind of request volume we are getting. We can summarize this data to show 1 min summaries instead:

summarize(sumSeries(stats.timers.accounts.*.http.post.authenticate.response_time.count), "1min")

We can now see the quantity of requests per minute. You could do the same to resolve by hour, day, etc.

Timeshift

Graphite has the ability to compare a given metric across two different time spans. For instance, let’s compare today’s quantity of logins vs those from last weeks.

To generate today’s graph:

alias(summarize(sumSeries(stats.timers.accounts.*.http.post.authenticate.response_time.count),"1min"), "today")

Then we use the timeShift function to get last week’s data:

alias(timeShift(summarize(sumSeries(stats.timers.accounts.*.http.post.authenticate.response_time.count), "1min"),"1w"), "last week")

Graphing both series in the same graph will give us that:

Wow, it looks like last week we had an authentication peek for a few hours. Why? It would be interesting to graph our promos and sales in the same graph to see if we can find any correlations.

Depending on your domain, you might want to compare against different time slices. Just change the second timeShift argument.

As percent

Another technique is to compare the percentage growth since last week. Let’s imagine we are looking at sales or signup numbers. We could graph today’s sales per minute vs those from last week.

To do that, Graphite has the asPercent function. This function takes a series representing 100% and second to compare against. The function call looks a bit scary so let me try to break it down over multiple lines:

asPercent(
  summarize(sumSeries(stats.timers.accounts.*.http.post.accounts.response_time.count),"1min")
  ,timeShift(summarize(sumSeries(stats.timers.accounts.*.http.post.accounts.response_time.count), "1min"),"1w")
)

The first argument is the summarized RPMs (requests per minute) and the second is last week’s summarized RPMs.

Here is how the graph looks:

Based on all the data we collect, we can now graph something like that:

This graph is basically the same as the one above, but we used the overall response time as the 100% value and we graphed all the different monitored sections of our code base.

You can now build some really advanced tools that look at trends, check pre- and post-deployment measurements, trigger alerts, and help you refactor your code.

Maybe you suspect that your app has a chokepoint at the database level. You can track the query types and the targeted tables per API endpoint. You can see where you spend most of the time and which code path is responsible for it. You can quickly see if adding indicies or other database-level techniques actually make a difference.

Other tips

Share a url into campfire/irc and see a preview

Campfire and many other chat tools offer image preview as long as they detect that the url has an image extension. Unfortunately, Graphite’s graph urls look more like this:

https://graphite.awesome.graphs.com/render?width=400&from=-4hours&until=-&height=400&target=summarize(sumSeries(stats.timers.accounts.*.http.post.accounts.response_time.count))&drawNullAsZero=true&title=Example&_uniq=0.11944825737737119

To get a preview, just append the with: &.jpg

Get the graph data in JSON format

You might want to do something fancy with the data like create alerts. For that you can ask Graphite for a json representation of the data by adding &format=json to the URL.

[
 {"target":
  "summarize(sumSeries(stats.timers.accounts.*.http.post.accounts.response_time.count))",
  "datapoints": [
    [20260.0, 137256960],[19513, 1372357020] //[...]
   ]
  }
]

The data points are the timestamped value of each graphed point. Note that you can also ask for the CSV version of the data then pass it on to some poor bastard using Excel.

Only show top graphs

Let say that you are graphing the response time of all your APIs. The amount of displayed graphs can be overwhelming.

To limit the displayed graphs, use one of the filters. For instance the currentAbove or averageAbove filters that can help you only display web services with more than X RPMs for instance. Using filters can be very useful to find outliers.

Get going with Graphite!

Hopefully this guide will help and inspire you to start using Graphite to easily collect and analyze your metrics. I’m sure there are great tricks I forgot to mention, please add your favorites in the comments.

Thanks to Jeff Casimir for reviewing this post before its publication!

I therefore wrote the same web API in Ruby (using Rails) and in Go (at first using Revel and then rewriting it without a framework since Go’s std lib have everything one might need). The API spec was simple:

• extract an authorization token contained in the request header
• use the token to query a MySQL database
• respond by sending back the MySQL row in json format
• return 401 if the token isn’t value

I didn’t try to optimize the Ruby code, nor the Go code. The idea wasn’t to get precise benchmark results, the goal was to get an idea of how much faster Go was in a real life situation. The other goal was to evaluate the amount of work needed to write web APIs in Go for someone who already knows the language.

At the end of the day the API implemented in Go is more than 50x faster than the Ruby version. Interesting enough, writing the code and tests for the Go API was pretty close to the Ruby experience (more on that later). 50X performance gain, including high concurrency support might be a very good argument to start using some Go when it makes sense.

I documented my experiment on Google+, click the following screenshot to read more.

TracePoint essentially allows you to hook into Ruby’s events and listen for events.

Being curious and since I just started a brand new Rails 4/Ruby 2 app, I decided to write a little middleware and see what Rails is up to when handling incoming requests.

Here is my TracePoint Rack Middleware.

class TracePoint
  class Middleware

    def initialize(app)
      @app = app
    end

    def call(env)
      stats = {}
      trace = TracePoint.new(:call) do |tp|
        stats[tp.defined_class] ||= {}
        stats[tp.defined_class][tp.method_id] ||= 0
        stats[tp.defined_class][tp.method_id] += 1
      end
      trace.enable
      response = @app.call(env)
      trace.disable

      puts "#{stats.keys.size} classes used"
      puts "#{stats.map{|k,v| v.keys}.flatten.size} methods used"
      puts "#{stats.map{|k,v| v.values}.flatten.sum} methods dispatched"
      response
    end

  end
end

(the gist shows a modified version so I could dump to disk the json representation of the calls)

I then inserted the middleware in Rails:

# in application.rb
config.middleware.insert_before(ActionDispatch::Static, TracePoint::Middleware)

I saved the output in json format for the curious: click here

On average, in production mode, using Ruby 2.0 and Puma on my laptop, my hello world index page takes 5ms.

To render my page, Rails uses (more or less):

• 250 classes
• 750 methods (not including C functions)
• and dispatches 2704 methods (not including calls to C functions)

Here is a small selection of some of the methods dispatched:

"String": {
    "underscore": 1,
    "blank?": 14,
    "html_safe": 78
  },
"ActiveSupport::Inflector": {
  "underscore": 2,
  "inflections": 2
},
"Hash": {
  "with_indifferent_access": 2,
  "except": 1,
  "except!": 1,
  "stringify_keys": 5,
  "transform_keys": 13,
  "stringify_keys!": 1,
  "transform_keys!": 3,
  "extractable_options?": 4,
  "extract!": 2,
  "symbolize_keys": 8,
  "reverse_merge": 1,
  "slice": 2,
  "symbolize_keys!": 2
},
"ActionView::CompiledTemplates": {
  "_app_views_welcome_index_html_erb__4177595130715791755_70209827438920": 1,
  "_app_views_layouts_application_html_erb___652124533295419796_70209827456500": 1
},
"ActiveSupport::Notifications::Fanout": {
  "start": 4,
  "listeners_for": 12,
  "listening?": 5,
  "finish": 3
}

TracePoint is a great new addition and I hope to see some new crazy tools being developed (production dead-code analyzer, deprecation code path finder anyone?)

To end, this short post, here is an interesting quote from Chad Fowler Berliner by adoption:

Abstractions are expensive. The cost increases exponentially as you add them to a codebase. Chad Fowler

Turns out it wasn’t that hard, but the lack of documentation made me lost a couple hours. I therefore updated OmniAuth’s wiki and wrote this quick post so hopefully you won’t waste time looking for simple details.

Requirements

You actually only need to add 2 gems to your Gemfile:

gem 'omniauth-openid'
gem 'ruby-openid-apps-discovery'

Now, the second gem is the one I didn’t know about. The gem is actually provided by Google itself. It turns out, Google Apps use a custom discovery protocol. They monkey patched the popular OpenID Ruby libraries so you can just drop in their gem and their discovery system will magically work.

Setup

You need to require 4 files to get everything setup properly:

require 'omniauth-openid'
require 'openid'
require 'openid/store/filesystem'
require 'gapps_openid'

The first one is the omniauth extension for OpenID, the second one is the main Ruby OpenID library (needed so we can set our SSL cert). The third one allows us to store temporary data on disk instead of keeping it in memory (optional). And finally, the last one is Google’s magical gem to get their discovery system working.

Because we are going to communicate via SSL, we want to make sure that the OpenID library uses our certs to verify the SSL communications:

OpenID.fetcher.ca_file = "/absolute/path/to/ssl_cacert.pem"

(Obviously, you need to change the path to your own cert)

We are almost done with the setup, we just need two more things:

• make sure you are using a session.
• setup OmniAuth

I’m using Sinatra, so I’ll load the Rack::Session middleware before I set Omniauth:

use Rack::Session::Cookie, :secret => 'supers3cr3t'

(Rails turns that option by default, so you don’t need to worry about it)

Then I can finally setup OmniAuth:

use OmniAuth::Builder do
  provider :open_id,  :name => 'admin',
                      :identifier => 'https://www.google.com/accounts/o8/site-xrds?hd=aimonetti.net',
                      :store => OpenID::Store::Filesystem.new('/tmp')
end

There are two important things to notice. First, because I set the provider’s name to be ‘admin’, the magical paths provided by OmiAuth will use that name (/auth/admin). Secondly, and more importantly, notice how I added the name of my Google Apps domain at the end of the identifier:

'https://www.google.com/accounts/o8/site-xrds?hd=' + your_domain_name

Sinatra

In Sinatra, your just need to define the routes OmniAuth would use (same goes for Rails, just use the router for that).

By default, omniauth now offers you a /auth/admin endpoint that will push the user through Google Apps authentication. Once the authentication is over, the user will be redirected to the following endpoint:

# Callback URL used when the authentication is done
post '/auth/admin/callback' do
  auth_details = request.env['omniauth.auth']
  session[:email] = auth_details.info['email']
  redirect '/auth/admin/welcome'
end

You can access the authentication details from request.env['omniauth.auth'] and redirect the user to another page, like the admin landing page for instance.

get '/auth/admin/welcome' do
  if session[:email]
    erb :welcome_boss
  else
    redirect '/auth/admin'
  end
end

On the landing page, you need to verify that the user is logged in, in this case, during the previous step, we added the email of the user to her session. We can therefore verify the presence of that information to confirm the authentication status. If the user is authenticated, then we’ll render an ERB template otherwise we’ll redirect her back to the login page.

We should also provide an endpoint in case the authentication failed:

get '/auth/failure' do
  params[:message]
  # do whatever you want here.
end

Note that by default, in dev mode, Omniauth won’t redirect the user there. To enable this behavior, use the following snippet (works with any rack app, Rails, Sinatra or whatever):

OmniAuth.config.on_failure = Proc.new { |env|
  OmniAuth::FailureEndpoint.new(env).redirect_to_failure
}

Conclusion

Using Google Apps for authentication with OmniAuth is trivial as long you know two things:

• the identifier url: 'https://www.google.com/accounts/o8/site-xrds?hd=' + your_domain_name
• Google’s discovery service gem

This blog post was written using omniauth 1.1.1, omniauth-openid 1.0.1, rack-openid 1.3.1 and ruby-openid-apps-discovery 1.2.0. This might not apply to you if you come from the future :)

Today we are going to look at Google’s Go language. Go approaches concurrency from an interesting view point. But instead of digging into the history and reasoning which led to this approach, I’d like to show you the language constructs by actually writing real life code.

Fetching web resources concurrently

The following example is taken from my recent presentation Ruby vs. the World. I explored a few programming languages and showed how they changed my Ruby.

To show how Go addresses concurrency, I decided to build a program which would concurrently fetch various web resources, wait for all of them to be fetched, then process them all at once. In other programming languages, we could have used threads and a semaphore, actors or callbacks. Go’s approach is slightly different, let’s walk through the code together.

The first part of our code gets us setup:

package main

import (
	"fmt"
	"net/http"
	"time"
)

var urls = []string{
	"https://www.rubyconf.org/",
	"https://golang.org/",
	"https://matt.aimonetti.net/",
}

The code above names our package then imports a few standard libraries that we are going to need. It then defines an array/slice of strings representing the urls we are going to fetch.

Next we define a type we will use a bit later:

type HttpResponse struct {
	url      string
	response *http.Response
	err      error
}

You can think of a struct type as a simple representation of a class. Technically, we are defining a structure with some typed attributes. We will later on, create instances of this defined type.

Go implements OOP slightly differently than other languages.

Methods in Go are more general than in C++, Java: they can be defined for any sort of data, even built-in types such as plain, “unboxed” integers. They are not restricted to structs (classes).

We can therefore define methods/functions for any type of data, including “any/all” types. This approach to types is called structural typing.

Here is the code:

func asyncHttpGets(urls []string) []*HttpResponse {
	ch := make(chan *HttpResponse)
	responses := []*HttpResponse{}
	client := http.Client{}
	for _, url := range urls {
		go func(url string) {
			fmt.Printf("Fetching %s \n", url)
			resp, err := client.Get(url)
			ch <- &HttpResponse{url, resp, err}
			if err != nil && resp != nil && resp.StatusCode == http.StatusOK {
				resp.Body.Close()
			}
		}(url)
	}

	for {
		select {
		case r := <-ch:
			fmt.Printf("%s was fetched\n", r.url)
			if r.err != nil {
				fmt.Println("with an error", r.err)
			}
			responses = append(responses, r)
			if len(responses) == len(urls) {
				return responses
			}
		case <-time.After(50 * time.Millisecond):
			fmt.Printf(".")
		}
	}
	return responses
}

This is the meat of our application. And there is quite a lot of going on in just a few lines. Assuming you aren’t familiar with Go, I’ll walk though the code.

Let’s start with the signature:

• the function is named asyncHttpGets
• it takes an argument namedurls which is an “array” of strings (I used quotes around the word array because it’s technically what Go calls a slice)
• it returns an “array” of HttpResponse pointers

Then in the function body:

We start by creating an instance of a channel and assigning it to the ch variable name. Think of a channel as a pipe like in unix. We can write to and read from that channel.

In the next line we create an empty instance of a slice containing pointers to HttpResponse objects.

Then, using the for range language construct, we iterate through our urls, storing the current value being used into the scoped variable url. The url is then available within the block/lambda/closure marked by the curly braces.

Now this is where the async construct comes in. Using the go keyword, we define an anonymous function that takes a string argument representing a url.

The function prints this string, then uses the net/http library to fetch the web resource. We use the returned data to create an instance of our HttpResponse type and send it to the channel.

This part gets a bit confusing because I reused the name url. We call this anonymous function right away passing it the url variable set by the loop.

You might wonder why we bother to create an anonymous function and call it right away instead of just executing the code directly. The go keyword executes the code that is passed in as a goroutine which is well explained here

A goroutine is a function executing concurrently with other goroutines in the same address space. It is lightweight, costing little more than the allocation of stack space. And the stacks start small, so they are cheap, and grow by allocating (and freeing) heap storage as required.

In other words, you start a goroutine and you let the “system” handle how it wants to deal with the low level details. Technically, goroutines might run in one or multiple threads, but you don’t need to know. We trigger each http fetch in a separate goroutine and then each response is pushed down the channel.

The second block of code begins with another for loop containing a switch/case statement. The case statement checks if something is in the channel. If there is something, we…

• allocate the data to the r variable
• print the resource’s url
• append the resource to the slice we created at the beginning of the function.

If the length of the array is the same as the length of all urls we want to fetch, we are done fetching all our resources and can return. While still waiting for responses, we print a dot every 50ms.

Update: In the first version of this blog post I had used a ‘default’ case statement to print the dot and sleep for 50ms so the loop wouldn’t be too tight and the concurrency effect was more obvious. But some HN comments pointed out that it wasn’t needed and I shouldn’t block. For reference here is what I had before (don’t use this code):

	for {
		select {
		case r := <-ch:
			fmt.Printf("%s was fetched\n", r.url)
			responses = append(responses, r)
			if len(responses) == len(urls) {
				return responses
			}
		default:
			fmt.Printf(".")
			time.Sleep(50 * time.Millisecond)
		}
	}

Thank you HackerNews.

With that code constructed, our main can make use of it like this:

func main() {
	results := asyncHttpGets(urls)
	for _, result := range results {
		if result != nil && result.response != nil {
			fmt.Printf("%s
                 result.response.Status)
		}
	}
}

Running the code looks like this:

$ go run concurrency_example.go
.Fetching https://www.rubyconf.org/
Fetching https://golang.org/
Fetching https://matt.aimonetti.net/
.....https://golang.org/ was fetched
.......https://www.rubyconf.org/ was fetched
.https://matt.aimonetti.net/ was fetched
https://golang.org/
https://www.rubyconf.org/
https://matt.aimonetti.net/

As you can see from the print statements, the 3 urls are triggered in a sequential way, but the responses come back in different orders due to different server latencies and response transfer time.

Conclusion

Go was designed to make concurrency easy for developers. It is a very well documented language and you will find on this page a lot of information about its concurrency philosophy and details about each available constructs works.

I like that the language is very simple and the constructs explicit. If you want to write concurrent code, Go pushes you to do it in a specific style. That style is clear and comfortable for me. My code stays simple, I don’t go crazy with callbacks, and the conventions make it simple for everyone else to understand my code.

Whether or not Go appeals to you stylistically, clearly the designers stayed close to the goal of developing to a 21st century C with a special focus on concurrency with a unix approach.

• Now is an awesome time to be a software engineer (probably the best time ever).
• Your job might not be well suited for you.

I’ll show you why we are lucky bastards, why we aren’t so good at picking the right jobs and some hints on how to solve this issue.

I remember a family friend telling me when I was a kid that computers are going to be the future and that there will be a lot of jobs in this field. I also remember that the idea of sitting all day, alone, in front of a minitel-like computer scared the hell out of me. But he was right and I now work from home, spending 12+ hours in front of a monitor. I get emails and phone calls from many people reaching out to me to help them find software engineers.

At least three things make “now” the best time to be a software engineer:

• demand
• projects
• prestige

Good time

There is a huge demand for engineers. There are many more job openings than candidates. But this is much better than in late 90s/early 2000 when anyone who could write a line of HTML would get a job. Now the projects people are building are way more interesting and have a real potential to change lives. Before, you had to work for a giant company to have a chance to do that. But now, with internet and smart mobile devices everywhere, almost any startup (fancy name for small company) can have a huge impact – look at Twitter for instance.

Lastly, being a geek is cool. Movies, cartoons, TV Shows now have geek heroes (granted they usually don’t represent real geeks, but hey it’s better than nothing).

The problem

Most software engineers I know, are really bad at choosing the right job for themselves. They don’t design a career. Engineers are good at solving technical problems in an objective way, but when it comes to our jobs and future, we seem to struggle.

Why?

I’m not an expert but I have a few guesses I’d like to share with you:

• We don’t know our real worth.
• We don’t make long term plans.
• We get paid well, so why bother changing?
• Changing job feels like betrayal.
• The system is broken.

Self worth

Most modern companies need solid engineers to be relevant in the short/medium term and they know it. Most engineers have no idea how their talent and dedication converts into real business value. Without that appreciation, they can’t easily estimate how much they are worth. The salary scale for software engineers is dramatically different from other jobs/industries. To your business, you might be worth twice or three times the salary of someone like a teacher. This is probably not fair because of the social value a teacher offers, but that’s the way the law of demand works in our society. Knowing how much you are worth to a company and how much to ask is critical to properly negotiate or renegotiate a contract.

I remember when I moved to America and was thankful to have a job. I had no idea that at $45k/year with basically no health coverage and no vacation, I was grossly underpaid and could have been paid twice that amount down the street. (Note: the average software engineer salary in the US is at $89,000 according to indeed.com). The average salary for a high school teacher is $47k/year so I wasn’t complaining. As a matter of fact, I didn’t leave this job because of the salary. I truly believe that “Money doesn’t buy happiness” and it shouldn’t be your primary reason to accept or leave a job. Money is nice and often makes life easier. But the point is that you have to understand how much you’re worth, so you can get paid and focus on your work.

Defining a vector

John Allspaw wrote a great blog post about what it means to be a senior engineer. I strongly recommend you read it. I often look back at it and pick up a couple points I need to focus on myself. John wrote a book which is a collection of essays and interviews regarding tech/web ops.

Here is a very interesting quote:

Not everyone can be senior. If, after five years, you are senior, are you at the peak of your game? After five more years will you not have accrued more invaluable experience? What then? “Super engineer”? Five more years? “Super-duper engineer.” I blame the youth of our discipline for this affliction. […] Given the dynamics of our industry many elected to move on to managerial positions or risk an entrepreneurial run at things.”

There are two very strong points in this quote:

• we don’t quite know what it means to be a senior engineer (and John’s post does a great job explaining his take on that).
• many of us end up in managerial positions or leading startups.

I might be a bit radical – but the day I stop learning/improving will be the day that I will quit, change jobs or careers. John’s post has great pointers to help us improve our skills. But the question I’m trying to raise is:

what do we want from our career?

“Career” sounds like a dirty word to many of us. It has a corporate, sleazy, back stabbing connotation. When I hear it, I picture a cliché stock photography of a bunch of smiling people wearing 80’s suits. In the context on this post, let’s take the Oxford English Dictionary definition: “course or progress through life (or a distinct portion of life)”. The word comes from from French via the Old Occitan word: “carriera” which means “street”.

A better word for career might be “path”.

I think we have a hard time knowing what kind of path we want to be on. When faced with the question, a lot of us answer: “solving problems”, “having fun”, “changing the world”. All these answers sound good, but they aren’t paths, they are just attributes.

I have to admit that I’m still struggling with this question and probably will for a while. I’m pretty good at defining short term goals but I have a hard time seeing the long term path. As a matter of fact, a little while back I was seated in front of Chad Fowler in his office in Washington, DC. Chad wrote a great book called “The Passionate Programmer: Creating a Remarkable Career in Software Development”.

I’ve read Chad’s book (which inspired me in many ways) and have known Chad for years. The point of our meeting was to decide what I was going to work on next. Unconsciously, I expected Chad to just tell me what I should be doing. I trusted him to pick the right “path” for me. But I was surprised when Chad told me: “you’re the kind of engineer who can do anything. You’re a generalist who can pick a topic and become a specialist. So what do you want to do?"

I wasn’t sure how to take it, it sounded like a compliment but, at the same time, the fact that Chad didn’t have a solution to my problem bothered me. I remember thinking, wait, he’s the expert and he’s deflecting the situation by asking me the question I came to ask him. Sure, the compliment was nice but that wouldn’t solve anything. What does that mean about me? If an expert can’t figure out what I should do, I might be screwed.

Then on my way back home, I realized that it didn’t matter how well Chad knew me, he couldn’t guess what I even didn’t know about myself.

My long term happiness depends on me finding a direction I want my professional life to take. In Chad’s book, there is a strong focus on finding a market, understanding it, developing skills and marketing yourself. However there was something I had missed.

The goal-oriented, destination-focused thinking that you usually do leads only from one goal to the next. It has no logical end. What most of us fail to realize is that the path is the end.

I’ve always known that the journey is more important than the destination in itself, but what I had missed is that you still need to define a destination or maybe more precisely a direction, a vector. My problem is that my path was just a bunch of scattered dots. Hopping from one to the other, I was hoping it was going to make a pretty drawing. The challenge is when I got to a spot, I was stuck not knowing what to do next. I ended up picking another short term goal/destination based on the opportunities available at that time.

What I should do instead was to define a general direction and then learn through the process. I believe this will help me enjoy my job more than running after goals. It will allow me to see the world differently and will help me make the right career choices when the time is right. To be honest, I think that’s the only way I can build endurance and not burn out in 5 years. That said, I’ll still have goals, deadlines and the usual – but they won’t define my own personal progress.

Why bother?

Changing jobs is a pain. As an engineer I weigh the pros and cons and try to logically pick the right choice – at least in theory. In practice I avoid dealing with questions that might result in challenging consequences.

Quiting a job is tough. You have to tell your current employer and your colleagues that you are leaving them for something you think is better for you. The nicer the people you work with, the harder it is. The better you are paid, the harder it is. If you work with nice people and you’re well paid, leaving is really hard (take note if you run a team).

In our profession, changing jobs isn’t usually seen as something bad. Recruiters might pressure you to take a new, better job. Beware impulsive changes though. Recruiters want their commissions so they’ll do anything they can do make you switch. They’ll try to convince you that the grass is greener on the other side. Maybe that’s only “bad” recruiters.

There are some recruiters who care about people and companies. Recruiters who will help you find the right job for you. However, they won’t be able to help you if you don’t know what direction you want to go to.

I remember being stuck in a pretty terrible work environment, being underpaid and the projects I was working on weren’t going anywhere. A friend gave me a Seth Godin’s book called “The Dip: A Little Book That Teaches You When to Quit (and When to Stick)".

I’m not a big fan of self-help/business books, but this book raised a very good and simple question: how do the efforts compare to the returns? Which situation are you in:

Think about it. Will the effort you put into your work pay off? If you don’t think it will, then you should quit right away.

The logic is pretty simple but requires you to forecast. For that you need some sort of metrics helping you to see if you are getting closer or further from the direction you set for yourself.

Trust

Trust is the key element of any relationship. In The Five Dysfunctions of a Team, Patrick Lencioni explains that the base of management dysfunctions is absence of trust:

Turns out it’s the same thing for our careers. We need a team of people to help us define a vision/direction and keep us honest and accountable.

Find people who you can trust to talk to about your professional goals, your progress, failures and doubts. People who will be honest with you and tell you things you might not want to hear. Find mentors and honest people. These people don’t have to be working in the industry. They just have to be able to listen and care.

People like that are extremely hard to find, but so are good executives. I believe that having trustworthy friends (partners/family members..) who care is a big part of what makes someone successful.

My small contribution

As I explained earlier, I’m no expert and I also struggle with the issues I described. However, I’d be glad to provide a bit of my free time to help you think through these issues.

A lot of you are doing a great job without the rest of us noticing. If you don’t have a popular twitter account, blog, open source projects, published books or given talks at conferences, it might be hard to get yourself noticed or even know how much you’re worth. Trust is a big deal and if you are considering moving on, you probably don’t want your boss and your colleagues to know. You probably also don’t know good recruiters you can trust. You might not even be sure it’s worth investing too much time.

Most of you probably won’t consider it, but I’d like to offer my help if you’d like it. I promise full anonymity and no strings attached. Just

• email me
• tell me about yourself and what you do
• what direction you’d like your career to take
• ask any questions you might have

After exchanging a few emails , I’ll try to make good use of my network to find you a way to move in your desired direction. Or, if you work for an interesting company with current openings, feel free to contact me too.

date = “Update:” slug = “Update:/engineers-suck-at-finding-right-jobs” literally days replying to as many people as possible. I’m sorry but at this time I can’t reply to any new enquiries. I’ll write a follow up blog post that covers what I learned from my interactions with so many readers. **

I have no idea how this will turn out. Maybe I’ll get a couple of emails, zero, or way too many to handle – but it’s worth a try. I do have a full time job, so please don’t expect me to reply to your emails within the hour.

I finally took the time to enable the comments in this blog. Feel free to leave advice or feedback.

##Description of the talk:

Ruby is an awesome programming language, it’s so pleasing you probably haven’t seriously looked at other languages since you switched. The programming world is evolving fast, new languages are created daily, new trends are emerging. Let’s take some time to look at a few languages from a Ruby developer perspective.

##Slides

The slides are available on [Matt’s SpeakerDeck page]https://speakerdeck.com/matt_aimonetti/ruby-vs-the-world) and can be [downloaded here]({{ page.slides }}).

##Video

##Presentation website

Matt’s presentation was filmed by Confreaks and posted here.

For more RubyConf 2012 talks, go there.

##Description of the talk:

mruby is Matz’ new Ruby implementation, it’s not cooler than node.js, it doesn’t natively support Hypstermedia, it looks just like the good old Ruby. So why should we, as a community care?

Matt’s talk is divided in two parts, an introduction of mruby (embedded Ruby) and and revisiting Ruby.

##Slides

The slides are available on [Matt’s SpeakerDeck page]https://speakerdeck.com/matt_aimonetti/mmmm-dot-mruby-everywhere-and-revisiting-ruby) and can be [downloaded here]({{ page.slides }}).

##Video

##Presentation website

Matt’s presentation was filmed by Confreaks and posted here.

##Description of the talk:

Programming languages affect the way one looks and solves problems. But comparing programming languages isn’t as simple as drawing a table comparing features.

In his talk, Matt shows what he likes, dislikes, the philosophy and concrete example of how to use 7 programming languages:

• Ruby
• JavaScript
• CoffeeScript
• Objective-C
• Clojure
• Scala
• Go

##Slides

The slides are available on Matt’s SpeakerDeck and can be [downloaded here]({{ page.slides }}).

##Video

TBD

Instead of arguing the pros and cons of OOP vs FP and how each of the previously mentioned languages handle being OOP and FP, I’d like to introduce a very powerful Scala idiom: pattern matching. Note that pattern matching isn’t something Scala invented nor that it only exists in Scala. Pattern matching can be achieved many different ways. However, the majority of the popular languages don’t put this concept at the center of their language. A few languages way before Scala rested heavily on pattern matching such as Erlang, Haskell but that’s a different story. How does Scala offers Pattern Matching, what is it and finally why is it valuable?

Scala pattern matching by examples

As its name indicates, pattern matching is used to detect patterns. Here is an example that covers a few interesting cases:

def listAnalysis(list: List[Any]) = list match {
   case Nil => "empty"
   case 'a' :: tail => "starting by 'a'"
   case (head:Int) :: _ if head > 3 => "starting by an int greater than 3"
   case (head:Int) :: _ => "starting by an int"
   case _ => "whatever"
}

If you’ve never seen any Scala that probably looks like gibberish to you. Let me break it down:

def listAnalysis(list: List[Any]) = list match {}

I define a new function called listAnalysis which takes an argument named list which is of type List (this list could contain any kind of elements). The implementation of this function is a pattern match on the list argument. The body of this ‘pattern match’ looks like a classical switch statement. But it’s actually much more than a simple switch statement. Surely it could be used like one, but as we will see, it can do much more.

Note that you can apply a pattern match against more than one object at once as shown a bit later.

Let’s look at the statements inside the function.

case Nil => "empty"

In this case we are checking that the list is empty or nil. If that’s the case, the statement on the other side of the “fat arrow” is executed. In this case, we return a string but we could have called another function or so whatever.

Now the second statement is much more complex and much more powerful:

case 'a' :: tail => "starting by 'a'"

Remember that we are doing pattern matching against our list object. What we are doing here is use the :: operator (aka cons operator) to extract the head and the rest of the list and then we match the head against the 'a' character.

This statement could have been written different ways:

case 'a' :: rest => "starting by 'a'"

In that case we named the tail of the list rest, but really we don’t care how it’s called or its value, so the sensitive thing to do is to rewrite that statement like that:

case 'a' :: _ => "starting by 'a'"

This basically says we are looking for a list that starts by 'a' (and we don’t care about the rest).

Another statement:

case (head:Int) :: _ => "starting by an int"

In this case we type match the first element of the list and check that we have an integer. Note that using the cons operator in the match cases doesn’t seem to affect performance. It would seem that at compilation, the statement are rewritten to avoid creating uneeded objects (List also implements structural sharing of the tail list). I’m not a Scala expert so someone with more experience might be able to confirm/clarify.

Now let’s look at a variant of this statement:

case (head:Int) :: _ if head > 3 => "starting by an int greater than 3"

This is the same statement as above, but we are adding an extra condition after the match. This is quite useful when simple matching doesn’t cut it.

Finally we have a fallback statement:

case _ => "whatever"

For more information about the cons operator, read about the extractor objects and what they can do.

Here is the result of calling our function with different lists:

listAnalysis(List())                             //> java.lang.String = empty
listAnalysis("This is a test".toList)            //> java.lang.String = whatever
listAnalysis("abcde".toList)                     //> java.lang.String = starting by 'a'
listAnalysis(List(1,2,3))                        //> java.lang.String = starting by an int
listAnalysis(List(42,24,36))                     //> java.lang.String = starting by an int greater than 3
listAnalysis("a".toList)                         //> java.lang.String = starting by 'a'

Here is another example using 2 items for the match:

def doubleMatch(foo: Any, bar: Any) = (foo, bar) match {
  case ('a', 'b') => "a and b"
  case (1, 'b') => "1 and b"
  case (1, _) => "1 and "+ bar
  case (a:Float, _) => "foo float"
  case _ => "unknown case"
}

doubleMatch(1, "test")                           //> java.lang.String = 1 and test
doubleMatch(1, 'b')                              //> java.lang.String = 1 and b
doubleMatch(42, Nil)                             //> java.lang.String = unknown case
doubleMatch('a', 'b')                            //> java.lang.String = a and b
doubleMatch(4.2f, 42)                            //> java.lang.String = foo float

Why is pattern matching valuable?

In short, pattern matching allows the developer to deconstruct a structure to find specific elements, in other words the pattern, needed to then constuct an object/structure or trigger a function.

It’s the opposite process of calling a method on an object. Here we start from a structure (instead of the instance of an object), this structure is just a basic struct and based on a found pattern, we then trigger a function (with access to the data if we need it). When you have a stable and known data structure, it’s often very interesting to use the pattern matching approach because you can easily expand the operations you can execute. However, if your operations are stable but the data changes, then the Object Oriented approach seems more adequate.

Besides that, pattern matching will often make your code clearer than using if/else statements. Especially in a language like Scala where you can define pattern matching function within a function and you can also pass pattern matching functions around. Like eveything else, it needs to be used with caution so the intend of the code is still understandable. That said it’s a great tool to have handy and I’ve had a lot of fun rewriting my newbie Scala code using a more idiomatic approach based on pattern matching.

I hope you enjoyed this quick introduction. You can read more about pattern matching in Scala in the following articles: (note: Ikai’s post on how he uses regexps with pattern matching is a fun read.)

• https://www.scala-lang.org/node/120
• https://pragprog.com/magazines/2012-03/scala-for-the-intrigued
• https://kerflyn.wordpress.com/2011/02/14/playing-with-scalas-pattern-matching/
• https://ikaisays.com/2009/04/04/using-pattern-matching-with-regular-expressions-in-scala/
• https://www.artima.com/scalazine/articles/pattern_matching.html

An example of something that might seem simple is inheritance. Ruby, unlike C++, doesn’t support multiple inheritance. What that means is that a Ruby class can only have 1 parent class (superclass). However multiple inheritance can be achieved via modules used as a mixins. That’s a very common pattern, people put some code in a module and then mix it in/include it in a bunch of classes. The problem I see though, is that people abuse these concepts and don’t respect the difference between a class, a module and a module used a mixin.

The Class

Object Oriented Programming 101:

“In object-oriented programming, a class is a construct that is used to create instances of itself – referred to as class instances, class objects, instance objects or simply objects. A class defines constituent members which enable its instances to have state and behavior.”

So if you create a class and you don’t create instances, you are using the wrong construct. Here is an example of what I often see:

class Settings

  @settings = {}
  def self.all
    @settings
  end

  def self.[](key)
    all[key]
  end

  def self.[]=(key, value)
    all[key] = value
  end

end

Which can be used as such:

Settings[:secret] = 42 * Math::PI * Time.now.to_f
p Settings[:secret]
# => 177243152913.2707

(granted this isn’t a great example since we could have used a subclass of Hash but just bear with me)

Actually, the developer who wrote the code above would probably also use some Ruby magic like method_missing to provide a more laxed API and allow for “nicer” getters such as Settings.secret and Settings['secret']. I have my own thoughts on the topic but it’s an entirely different subject.

Note also that the way class level methods are defined can also be different depending on who wrote the code, you might see the following variations (and other more esoteric ones):

class Settings

  def Settings.all; end

  # or
  class << self
    def all; end
  end

end

The Settings code above works, the code is simple, yet I will argue one thing: it’s an abuse of the class construct. We’re breaking the #1 rule of classes: “create instances of self”.

It’s easy, whenever you don’t create instances of a class, please don’t use a class.

That’s also true for slightly different examples such as:

class API

  def fetch(id)
    HTTP.get('https://matt.aimonetti.net/article/:id', :id => id)
  end

end

resource = API.new.fetch(42)

There is no need whatsoever to create an instance of API, using a class is picking the wrong construct. Also, I don’t care if you use the Singleton module to only allow 1 instance of the class, you still shouldn’t use a class in the above example.

The Module

In Ruby’s object hierarchy, the Class object actually inherits from the Module object.

ruby -v -e "p Class.ancestors"
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin11.3.0]
[Class, Module, Object, Kernel, BasicObject]

As per Ruby’s source code defintion:

“A Module is a collection of methods and constants.”

I like to think of modules as namespaced methods and constants. Whenever you want code that logically belongs together but that won’t require that you create instances of a ‘model’, then a module is the right construct to use.

As a matter of fact, the two examples above are great cases where a module should have been used.

The confusing bit is that modules can have module level methods but also instance level methods. Here is an example:

module API

  def self.fetch(id)
    HTTP.get('https://matt.aimonetti.net/article/:id', :id => id)
  end

end

This is a module level function, it could also be written like that:

module API

  module_function

  def fetch(id)
    HTTP.get('https://matt.aimonetti.net/article/:id', :id => id)
  end

end

And be used like that:

resource = API.fetch(42)

Until now, it makes sense. The weird thing is that even though, modules unlike classes aren’t meant to create instances, we have the possibility to define module instance methods.

module Settings

  DATA = {repo: 'https://github.com/mattetti'}

  def repository
    DATA[:repo]
  end

  def secret_key
    DATA[:key] ||= 42*Math::PI
  end

end

Great, but we can’t actually use these methods since they are instance methods and we don’t create instances of modules. Well, that isn’t quite true, there is a way to access them and that’s by using a module as a mixin. What that means is that we inject/copy the module code inside a class or an(other) object. Example in code:

a = Object.new
a.extend(Settings)
a.repository
# => "https://github.com/mattetti"

Or we can add the code to a class so the instances of this class can access our module instance methods:

class Foo
  include Settings
end

Foo.new.repository
# => "https://github.com/mattetti"

The mixin modules

There a plenty of resources online about the many ways to use mixins in Ruby to achieve multiple inheritance and do cool stuff. But the point of this article is to try to demonstrate that mixins shouldn’t be abused.

My problem with the above example is that by mixing in the Settings module inside our Foo class, we created an uneeded, confusing extra level of abstraction. Instances of Foo now have access to two methods/objects: repository and secret_key. These methods or the objects they refer to don’t belong to the Foo class, but it seems convenient to not have to type Settings.repository so we mixed things in. Plus, a lot of Ruby developers seem to dislike adding class/module level methods so they feel that this approach ‘feels better’.

Here is the thing, the convenience of typing a few less characters isn’t worth it. Next time you or someone else will look at an instance of the Foo class calling repository, finding where it is defined is going to be a pain. That’s especially true if you have many mixins in your class. Settings will also probably grow and you will end up with a bunch of methods that have nothing to do with your class instances. In this case, I will call the use of a mixin, an abuse of construct. Sure, Ruby allows you to do it, but that doesn’t mean it’s the right thing to do. In Ruby, unlike in Python, there are 101 ways to do a simple thing. It doesn’t mean that the 101 ways are good, it just means that Matz wasn’t sure how people would use his programming language and chose to give us more freedom to messup/doing it our own way.

When to use mixins?

I have my own rule: use mixins whenever you need to share behaviors between different classes.

In the above example, we weren’t sharing behaviors, we were sharing objects, there was no need to actually use a mixin.

That said, rules aren’t rules without exceptions. A good example of this exception would be the Math module from the standard library. This module offers trigonometric and transcendental functions. You might think that this module would be designed to be a mixin so you can get log, cos, exp and friends available in your math related classes. It turns out, all Math’s methods are defined a module functions meaning that they are meant to be called from the Math module directly.

However, Ruby allows you to mixin module functions, but these functions become private. If you do include the module inside your class, your instance methods will be able to call hypot(x,y) directly, but these methods won’t be available from the outside (Foo.new.log(42) would raise an exception).

To conclude with mixins: mixins are great but don’t abuse them or you will endup with so much abstraction that your coworkers will secretely call you Kandinsky. Stick to simple mixins allowing you to share behaviors between at least a couple classes. See DataMapper for a great way to use mixins.

Modules: your secret functional programming weapon

I have to say that I do like functional programming. The idea of having functions not mutating the states of things around them pleases me. It just seems clean, you feed data to a function and you get another piece of data. No states were changed, maybe some temporary variables were allocated to process the data, but the only thing that matters is the input and the output. Easy to grasp, easy to follow, no magical states being changed by some code fairies.

The good news is that Ruby allows us to write code like that. And this is where modules are great. Very much like the Math module we discussed above, there are many cases where you want to have a bunch of functions that process an input and provide an output without keeping any states. A good example of such a module would be a param verification filter. The filter takes an input, takes some rules and verifies that the input matches the rules. Surely, we could create an instance for each verification, this would allow to keep states in our class and do the usual OOP things. But we could also simply use a module with a bunch of module (level) functions that would pass to each other the input they need to not need to keep states. The end result will be faster, nicer on the GC and easy to follow.

Mixing OOP and functional programming isn’t new, ask Scala developers! If done right, by adopting this approach we can simply our code base, make it faster, easier to maintain and not losing the chance to also use OOP paradigms when needed.

Compromise

As shown earlier, modules and classes have pros and cons. Classes are however much more natural to use for Object Oriented Programming. A compromise about the Settings examples was suggested by Evan Phoenix. The solution is elegant and simple. Use a class and an instance. Here is an implementation based on his suggestion:

class AppSettings < Hash

  def custom_method
  end

end

SETTINGS = AppSettings.new

The point here is not about the class implementation but that fact that we use a class and associate an instance of this class to a constant so it can be shared all over the place. Wow, a constant, this is so nasty you might think. Classes are constants too and so are modules, here we just allocate an instance of a class to a constant. Surpisingly simple and efficient.

They both create a shallow copy of an object (meaning that they don’t copy the objects that might be referenced within the copied object). However, #clone does two things that #dup doesn’t:

• copy the singleton class of the copied object
• maintain the frozen status of the copied object

Examples of the singleton methods not being copied.

dup:

a = Object.new
def a.foo; :foo end
p a.foo
# => :foo
b = a.dup
p b.foo
# => undefined method `foo' for #<Object:0x007f8bc395ff00> (NoMethodError)

vs clone:

a = Object.new
def a.foo; :foo end
p a.foo
# => :foo
b = a.clone
p b.foo
# => :foo

Frozen state:

a = Object.new
a.freeze
p a.frozen?
# => true
b = a.dup
p b.frozen?
# => false
c = a.clone
p c.frozen?
# => true

Looking at the Rubinius source code makes the difference extremely obvious.

Because of the extra steps, clone is a bit slower than dup but that’s probably not what will make your app too slow.

Just a quick note about shallow copies (true for clone and dupe). Notice how the array referenced by the bar attribute doesn’t get copied but shared between the original and the copied instances:

class Foo
  attr_accessor :bar
  def initialize
    self.bar = [1,2,3]
  end
end

a = Foo.new
b = a.clone
p a.bar
# => [1, 2, 3]
p b.bar
# => [1, 2, 3]
a.bar.clear # clearing the array #bar points to
p a.bar
# => []
p b.bar
# => []

Both objects, a and b share the same reference to the array instance created when a is initiated. There are a few ways to do a deep copy of an object, but that’s a different matter.

What has changed in the last few years is that web applications start providing more than just dynamically rendered HTML templates. JavaScript is used to run more and more logic on the client side and JS usually talks to some backends via JSON. But web APIs are also more and more used to expose raw data to other software. The challenge though, is that we haven’t changed the way we write web applications to adapt to this new way of providing data. In this article, I’d like to explore why and how we need to rethink web API development and focus on communication and interaction.

Unlearning

Many are used to doing certain things a certain way and it’s really hard to unlearn old habits. The more concerning part of this fact is that it makes it hard for someone to step back and evaluate honestly if a technical decision is due to comfort or if it’s truly the right choice to achieve a given goal. Ruby on Rails revolutionized the way we wrote web applications almost 10 years ago and since then, many web frameworks have adopted the Rails philosophy. As a matter of fact, I personally think that when it comes to writing front end applications, Rails is still one if not the best web frameworks out there. But in its current state, I’m far from convinced that it’s a great tool to write web APIs mainly because it was not designed for that and because it comes with a lot of baggage.

Focusing on the API

Let’s take a step back and consider what is critical when developing a web API:

• Documentation for end users to know how to consume your services.
• Consistent and reliable output so you don’t break client applications relying on your services.
• Maintainability

Note that I didn’t mention performance because I consider performance almost always being important. I also didn’t mention the whole REST/RPC/Hypstermedia debate since I consider it being an implementation detail and a totally orthogonal discussion. But for the record, I personally don’t like solutions forcing you into a specific way of providing your data (I’m looking at you webmachine).

Documentation

Documentation isn’t as important when you consume your own APIs because you can look at your source code. However it gets much more tricky when you start working with other teams. They might not know the language/framework you use. They might not have time to go dig into your source code to figure out what your meta-magical piece of clever code does.

Lately more and more applications provide their raw data to the outside world via web APIs. The developers who will consume your resources don’t have access to your source code, probably don’t use the same programming language and don’t have much time to guess how your API work. Also your test suite won’t help communicate how your API works so we need to find a different approach. Also note that in this scenario, TDD/BDD won’t help us communicate better and tests can’t be used as documentation since your tests aren’t exposed.

Documentation is your #1 way to communicate with your human audience. Communication is key and even if as engineers we love to focus on code, if we can’t communicate about it, end users will have a hard time using our code and might just not even do it. The key is to communicate what your API does, why someone might want to use it and how to use it.

Consistent and reliable output

This point seems obvious but what we realize in reality is that for many, API’s consistency and reliability doesn’t include documentation. You find a lot of APIs out there poorly documented or out of sync with the actual implementation.

Documentation is a contract between you: the developer and the other developers consuming your APIs. As a developer, when you write any type of tests, they become some sort of quality contract. If someone changes your code and break your tests, they break the implicit contract. However, when talking about consuming data via an API, things get a bit more complicated. We need a way to ensure that the end user expectations are matching our implementation/documentation. Unit testing simply can’t guarantee that. Unit testing will guarantee that the logic of your units of code is intact but it can’t easily guarantee that the API output matches the documentation, however this is something that has to be done.

Maintainability

This is a tricky point since maintainability is very subjective. But I think we can agree that decoupling/separating concerns will make our code more maintainable.

That’s why I personally don’t think it’s a good idea to mix HTML rendering code and web API code. Consider using different controllers or different files depending on the way your code is structured.

I also strongly believe that the implementation details shouldn’t define the way you design your web APIs. Don’t just slap a CRUD API on top of your model and call it done. In most cases, you will pay a high price later on if you take this approach because whenever you will need to change your web API or your model, you will be stuck. This is because your interface is now used by a lot of people and you can’t easily change it. There are many ways to avoid API/model coupling, I don’t advocate one particularly, but whatever you do, be sure you can make your models and APIs can evolve separately.

My approach

I’ve been designing and developing web APIs for many years and I’ve been struggling with everything I mentioned until now. I don’t claim that I’ve found the solution but I’d like to think that I found one own way of addressing what are for me some of the most important parts of API design.

Being explicit

I believe that there is value in being explicit in the way we describe web APIs. Sometimes people get confused between being verbose and being explicit. These are two different concepts. Being explicit can sometimes seem verbose, but we need to evaluate the value that can be extracted from the provided information. If there isn’t any clear value and too many words are used, then we aren’t explicit, we are verbose.

When designing a web API, I don’t write it for myself, I do it for someone else. It’s crucial to consider who you are writing for and to expose what’s important for them. A “small” problem is that we don’t show our code to our end users, so we need to find a way to explicitly provide the important information and to have this information provided to our end users.

DSL

For that I use a Domain Specific Language (DSL) which is a fancy way to say that I have some specific code allowing me to explicitly define my web services. These services exist as objects that can then be used to process requests but also to validate inputs, and even more importantly to generate documentation.

The DSL allows me to define the following:

• details about consuming the service (uri, HTTP verb, authentication details, other service details…)
• incoming params (which ones are allowed, the type, are they required, optional?, what are they for)
• service output

The input details is really important to validate incoming requests and reject them before even dispatching them. This is done for data sanity and for security reason. It also defines a strong interface that is easier to develop against and to maintain.

The output details might sound quite surprising and redundant. After all, isn’t that a duplication of effort since we already have this information in the “view”? Well, if we consider the important points I highlighted in the first part of this article, we need a way to enforce a “contract” between the end users and our implementation. To do that, we can’t simply rely on our code since we can’t trust it. The output is important to document the expected output but also to validate that our services match our documentation and therefore our “contract”.

Here is an example of a Ruby DSL use to describe a hello world service:

describe_service "hello_world" do |service|
  service.formats   :json
  service.http_verb :get
  service.disable_auth # on by default

  # INPUT
  service.param.string  :name, :default => 'World', :doc => "The name of the person to greet."

  # OUTPUT
  service.response do |response|
    response.object do |obj|
      obj.string :message, :doc => "The greeting message sent back. Defaults to 'World'"
      obj.datetime :at, :doc => "The timestamp of when the message was dispatched"
    end
  end

  # DOCUMENTATION
  service.documentation do |doc|
    doc.overall "This service provides a simple hello world implementation example."
    doc.example "<code>curl -I 'https://localhost:9292/hello_world?name=Matt'</code>"
  end
end

The DSL style isn’t really important, what’s important is that it allows us to address some of the concerns we discussed earlier such as clearly defined interface that can be communicated as documentation but also acts as code. The focus is really on the API and everything fits in one page. Because everything is an object and can be inspected, proper and up to date documentation can be generated. And the implementation can be tested against the documentation since everything is maintained as code.

Being agnostic

While I like Ruby for many reasons, I don’t think that it’s the only good language to implement great web services. It actually has its pros and cons and so have all the web frameworks out there. That’s why I wrote my DSL as a standalone library that could virtually run on top of any web engine since it only outputs a representation of services.

While I hope to one day create an interesting interop solution across programming language (by exporting the objects in a shared data structure for instance), I started by focusing on the various Ruby web frameworks.

The best starting point for me was to use Sinatra. I almost started just using rack, but Sinatra was providing me with a bit more feature for very little headache and little code to grasp. I wrote wd-sinatra which is a Ruby gem providing the WeaselDiesel DSL on top of a Sinatra app. It comes with a generator and all the needed hooks to design, implement, test and generate documentation for modern web APIs.

Using a simple rake command (Ruby’s version of make) one can generate documentation or test the APIs against the implementations. The “mini framework” is still very free form and should let you do whatever you want. I don’t even set a default ORM for you to use since this choice is highly personal. I do however leave you places to set these things.

Sinatra and my “freedom framework” might seem too free form for you. So I’m currently working on getting the DSL to run on top of Rails, and by goal is to actually get it to run with a normal Rails app.

Reconsidering Rails' MVC

The Rails code base is very deeply marked with its own concept of MVC and having controllers and actions. The challenge I have is that I like my service to be self contained. I want my services to be simple and easy to grasp. I like having 1 service per file. Models, libraries and other optional presenters/decorators live separately but I like to have my service implementation code with the rest of my service description. I honestly don’t like telling developers that they need to go check a route file and that my simple service requires that you open 12 files to understand what’s going on. Simpler is often better and that’s why in wd-sinatra the DSL and the implementation live together which is quite harder to do with Rails.

describe_service "hello_world" do |service|

  # [...] see the DSL section to see how the
  # service is described. The block below is being
  # being called in the context of the request
  # after the request was validated.

  # SERVICE IMPLEMENTATION
  # the returned value is used as the response body, all the Sinatra helpers
  # are available.
  service.implementation do
    {:message => "Hello #{params[:name]}", :at => Time.now}.to_json
  end
end

Learning from experience

While the DSL fits most of my needs, we all have different use cases. While it’s critical for a library to have a clearly defined objective, it’s also important to have many people help improve it. Part of the learning/vetting excercise is to test an approach against different needs to define when and why it works well in some cases and why sometinmes it doesn’t. This allows us to define a sweet spot that should match the clearly defined objective. However to be able to do that, a design needs to be tested by many people. So far my approach seems to work very well when an API needs to live outside an application and that 3rd parties need to consume the resources. It also seems to work well with edge cases that many API designers seem to encounter sooner or later.

Conclusion

At the end of the day, we have to remember that API stands for Application Programming Interface and that these interfaces have to be programmed so humans can write to comsume them. One of Ruby’s main design points has always been to try to address human needs more than computer’s. As API designers/implementers, it seems important to adopt the same approach and consider who will use our interfaces. I think the discussion should focus more on what to value when defining web APIs, instead of arguing about how to implement APIs. Standardization is a great concept but a really hard to implement. And even with standards, we have to find a way to communicate with the API users to express what standards we follow and where to find the various entry points. Think about your API, how well does it communicate with your future API users, is it good enough for them to get excited? Is it good enough for them to create something amazing with it? If not, why not?

Discussion

Comments are available on this HackerNews thread.

RubyMotion is a commercial, proprietary fork of MacRuby that targets iOS. This is not a small achievement, MacRuby relies on Objective C’s Garbage Collector (libauto) which is not available on iOS. Static compilation and new memory management solution was required to target the iOS platform . The new runtime had to be small and efficient. Furthermore, being able to run code on iOS isn’t enough, you need tools to interact with the compiler, to debug, to packages applications etc…

I don’t think anyone will contest the fact that RubyMotion is a well done product. The question however is, “is it worth for you to invest some money, time and energy in this product instead of using Apple’s language and tools”. In this article, I’ll try to balance the pros and cons of RubyMotion so you can have a better understanding of what RubyMotion could mean for you. As a disclaimer I should say that I was beta testing RubyMotion, that they are strong ties between RubyMotion and the MacRuby project I’m part of and finally that having MacRuby on iOS has been something I’ve been looking forward for a very long time.

Over the last few months I’ve seen RubyMotion take shape and finally hit the big 1.0. As you can see from Twitter and HackerNews, the Ruby community is excited about being able to use their language to write statically compiled, native iOS apps. Spoiler alert, they are right, it’s a lot of fun.

What I like about RubyMotion:

Ruby Language

I don’t mind Objective-C, I think it’s a fine superset of C, with the arrival of blocks, new literals and automatic memory management via ARC, Objective-C is actually getting better over time. But frankly, it’s not Ruby. You still have to deal with headers, you always have to compile your code via some weird Xcode voodoo settings, testing is a pain, the language, even with the new literals is quite verbose. On the other hand, using Ruby syntax I can get much more flexibility, reuse my code via mixins, easily reopen existing classes etc… At the end of the day, I end up with some code that seems cleaner, easier to understand and maintain even though I’m calling the same underlying APIs. Ruby’s flexibility also allows developers to make their own higher level APIs, take a look at some of the wrappers/helpers I wrote while playing with RubyMotion.

MacRuby

RubyMotion is based on MacRuby, meaning that all the time and energy invested in the project will benefit RubyMotion’s users. All the concepts I explain in my MacRuby book apply to RubyMotion. You don’t have to find workarounds to work with native APIs, Ruby objects are Objective-C objects and performance is great. I do regret Apple didn’t decide to embrace MacRuby for iOS but at the same time, even though we lost the Open Source aspect of the project and Apple’s backing, we gained much more flexibility and freedom on Laurent’s part.

REPL/Interactive shell

RubyMotion doesn’t currently have a debugger, but it does have something Objective-C developers don’t have, a REPL working with the simulator. This feature is quite handy when debugging your application or learning the Cocoa APIs. You can click on a visual element in the simulator and start modifying the objects in real time in a terminal window and see the modifications in the simulator. It reminds me of the first time I used firebug to edit the html/css of a web page and saw the changes in real time.

Not dependent on Xcode

Xcode is fine when you write Objective-C code, but it crashes often, it has a complicated UI and never really worked well for MacRuby due to the fact that Objective-C and Ruby have different requirements and the that Xcode is not open source. It’s also fully controlled by Apple and doesn’t provide APIs for 3rd party developers. (That said, the Xcode team has often helped out when a new released of Xcode broke MacRuby, so thank you guys).

Being able to use simple rake tasks to compile, simulate and deploy applications is just really really nice. I’m sure we’ll end up with better IDE integration, nice GUIs for some who like that, but in the meantime, as a “hacker”, I really enjoy the simplicity of the Rake tasks and not being forced in using a specific IDE.

Memory management

Even though ARC made memory management much easier for Objective-C developers, when using RubyMotion you don’t have to worry about memory (well at least not explicitly, don’t be dumb and create a bazillion objects and hold references to them either). This includes the CoreFoundation objects that you still have to manually manage in Objective-C. Memory management is transparent and in most cases it’s really nice.

What I like less about RubyMotion

Here is a list of things that are cons to using RubyMotion, note that while the list is longer than my list of “pros”, I listed a lot of small things. I also think that most of these issues will get solved in the next few months.

Ruby language

There are some cases where Ruby just isn’t that great or is not an option. Examples include dealing with API relying heavily on pointers, when using some of the lower level APIs or when you have to interact with C++ (video game engines for instance). The good news is that within the same project, you can write part of your code in Objective-C and the rest in RubyMotion. The other thing that bothers me a little bit with writing Ruby code for iOS is that you can’t easily enforce argument types and therefore you are losing a lot of the features provided by Clang to the Objective-C developers. I dream of an optionally typed Ruby – but that’s a different topic.

Another downside of using Ruby is that Ruby developers will assume all standard libraries and gems will be compatible with RubyMotion. This isn’t the case. You need to think of RubyMotion as only offering the Ruby syntax (modulo a few differences). To be honest, most of the std libs and gems aren’t that useful when writing iOS apps. Even when I write MacRuby apps, I rarely rely on them and pick libraries designed to work in a non-blocking, multi-threaded environment (usually ObjC libs that I wrap).

Cocoa Touch

If you’re already an iOS/OS X developer, you know that most of the hurdles aren’t the language syntax but the Cocoa APIs. These APIs are what you need to interact with to create your application. Cocoa APIs are usually much lower-level compared to what you usually see in Python, Ruby or even Java. While they are quite consistent, the APIs still have a stiff learning curve and currently,  if you want to write iOS applications, even if you know Ruby, you still have to learn Cocoa.

However, I do think that with RubyMotion now building a userbase, we will start seeing more and more wrappers around these sometimes hideous APIs.

No Xcode/IDE

There are cases where an IDE is really practical, especially when learning new APIs. Being able to have code completion, quick access to the documentation, instrumentation, debugging, interface builder, refactoring tools are things that Objective-C developers might have a hard time with when switching to RubyMotion. If you don’t know either Ruby or Cocoa, getting started with RubyMotion might be quite hard and you are probably not currently in the target audience.

Writing UI code by hand

In some cases, it makes sense, in other, it should be much easier. I know that Laurent is working on a DSL to make that easier and I’m looking forward to it. But in the mean time, this is quite a painful exercise, especially due to the complexity of the Cocoa UI APIs. Using Xcode’s interface builder and Storyboards is something I know a lot of us wish we could do with RubyMotion when developing specific types of applications.

No debugger

Again, this is eventually coming but the current lack of debugger can be problematic at times, especially when the problem isn’t obvious.

Lack of clear target audience

It’s hard to blame a brand new product for not having clearly defined a target audience. But as a developer I find myself wondering “when should I use RubyMotion and for what kinds of problems?” Is RubyMotion great for quick prototypes I can then turn into production code? Or is good for throw away prototypes? Is it reserved for “fart and flash light” applications? Is it ready for prime time and should I invest and write my new awesome apps using it? Should I convert over my existing code base over from Titanium (or whatever other alternatives you used)? Should I use RubyMotion every time I would use Objective-C?

I guess we will see when the first applications start hitting the app store and people start reporting on their experience.

Documentation

I’m partially to blame here since I could have moved my butt and start writing a book but the point is nonetheless valid. All the iOS documentation out there is for Objective-C, all the APIs and samples provided by Apple are obviously only for Objective-C. Thankfully, you can use the 2 MacRuby books available out there to understand how to convert this existing documentation into something useful, but RubyMotion will need to provide better and more adapted documentation for beginners. I have no doubt that this is coming sooner than later.

Proprietary solution

RubyMotion isn’t open source and currently fully relies on the shoulders of a single man. If unfortunately, Laurent goes out of business or decides to do something else then we will have to rewrite our apps in Objective-C.  Using RubyMotion for a professional product represents a significant business risk, which is exactly the same as using proprietary technology from any vendor. Apple could also decide to switch to JavaScript or rewrite iOS in Java and deprecate Objective-C. Let’s just say that it is unlikely.

I usually favor open source solutions, from the programming language I use to the OS I deploy on. This isn’t always possible and if you want to write iOS applications, you don’t currently have a choice. I do wish Laurent had found a way to make money while keeping the source code open. But who knows – after he makes his first million(s), he might change his mind.

Conclusion

I would strongly suggest you consider giving RubyMotion a try. I can assure you that it will provide at least a few hours of ‘hacking fun’ (and you will be able to brag about havng written your own iPhone app).  It will also help support financially someone who’s taking a risk in trying to push mobile development to the next level.

RubyMotion is, by far, my favorite alternative to Objective-C. But it is hard to tell, just 48 hours after its release, what people will do with it. Can it transcend the programming language barriers and attract Python, PHP, Java, ObjC and JavaScript developers? What is the sweet spot for RubyMotion applications? Will it affect the native vs web app battle? Can it make iOS development more accessible to the masses? Only time will tell.

Update:

Since RubyMotion 1.0 was released, I spent quite a lot of my free time leading the development of BubbleWrap, a free and open source 3rd party library for RubyMotion. Unfortunately, as of July 2011 I took a break from this project and RubyMotion in general. I explained my motivations in this mailing list post.

• MRI
• REE
• JRuby
• Rubinius
• MacRuby
• Maglev
• IronRuby

And many other less known implementations.

This time, the main man behind the project is the Ruby creator himself: Yukihiro ‘Matz’ Matsumoto. I already covered the announcement, you can read more about it there.

Why mruby?

Following my previous article on mruby, some people seemed to be confused by the “raison d’être”/purpose of the project and the difference between MRI and mruby.

mruby is designed to be modular and to be embedded, which means that it’s expected to live inside other software applications. In other words mruby’s goal is to make Ruby an embedded language. mruby is library that you link in other applications.

When talking about embedded languages, Lua is probably the most well known interpreted language. Lua is usually used for the purpose of offering a higher level language (scripting language) inside a software written in a low level language (usually C/C++). This is why Lua is very popular in the game industry to allow a scriptable interface and have parts of the code interpreted instead of requiring to recompile the code base against the target platform. According to this survey from 2009, Lua was by far the most popular scripting language for game developers.

Of course, Lua is also popular outside of the game industry and the reasons are simple to understand:

• very easily embedded (simple and well documented API)
• small footprint
• portable (runs almost everywhere)

That gives you an idea of what mruby is trying to be. As a matter of fact, this definition of Lua applies very well to mruby:

“Lua is dynamically typed, runs by interpreting bytecode for a register-based virtual machine, and has automatic memory management with incremental garbage collection, making it ideal for configuration, scripting, and rapid prototyping.”

Why not Lua?

This is a matter of taste and requirements. It should eventually boil down to the difference in the language designs.

Lua combines simple procedural syntax with powerful data description constructs based on associative arrays and extensible semantics.

Ruby is a dynamic, open source programming language with a focus on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write.

So on one hand you have a very simple language with a focus on functional programming and on the other, a richer language (albeit more complex) with a main focus on Object Oriented programming.

If you don’t know Lua, it’s quite close to JavaScript in the sense that it’s a prototype based language and you can therefore write code that is Object Oriented.

Ruby being a full OO and richer language, you can organize your code differently and write more of your complex logic in a scriptable language. One could even potentially leverage the language to create its own Domain Specific Language in pure Ruby and create a “Rails like” experience within its application.

To be honest, I don’t think that Ruby via mruby will just replace Lua, but I do believe it will become a very interesting alternative for some use cases. Especially if mruby manages to perform as well as Lua with the same type of footprint. There is also the fact that mruby being a sponsored project from the Japanese government, I wouldn’t be surprised to see big Japanese companies experiment with embedding Ruby in their devices. After all, software is being added in every object we own, rapid development and being first to market is more critical than ever which makes mruby is a very attractive solution.

Getting started

I wrote a simple hello world example to give you a place to start:

#include <stdlib.h>
#include <stdio.h>

/* Include the mruby header */
#include <mruby.h>
#include <mruby/compile.h>

int main(void)
{
  mrb_state *mrb = mrb_open();
  char code[] = "p 'hello world!'";
  printf("Executing Ruby code with mruby!\n");

  mrb_load_string(mrb, code);
  return 0;
}

Or the longer/more complex version:

#include <stdlib.h>
#include <stdio.h>

/* Include the mruby headers */
#include <mruby.h>
#include <mruby/proc.h>
#include <mruby/data.h>
#include <mruby/compile.h>

int main(int argc, const char * argv[])
{
  struct mrb_parser_state *p;
  mrb_state *mrb = mrb_open();
  char code[] = "p 'hello world!'";
  printf("Executing code with mruby!\n");

  p = mrb_parse_string(mrb, code);
  int n;
  n = mrb_generate_code(mrb, p);
  mrb_run(mrb, mrb_proc_new(mrb, mrb->irep[n]), mrb_top_self(mrb));
  if (mrb->exc) {
      mrb_p(mrb, mrb_obj_value(mrb->exc));
  }

    return 0;
}

To compile and link the code:

$ gcc -Iinclude hello.c lib/libmruby.a -lm -o hello.out

To execute it:

$ ./hello.out
Executing Ruby code with mruby!
"hello world!"

To get started, you just need the mruby source code and a compiler. (I haven’t tried to compile mruby or my sample on Windows for this code, but I assume it would work just fine with Visual C++).

The above example is very trivial and takes a line of Ruby code:

p 'hello world!'

that the linked mruby lib interprets. You can take the compiled file and use it on other machines using the same platform and the code will run fine, just like normal C code.

For a more complete example, take a look at the mruby’s standalone interpreter

Future

It’s a bit too early to know if mruby will be successful or not. There are few things I will keep my eyes on.

Performance

To be a true alternative to Lua, mruby will have to interpret Ruby code much faster than what MRI is able to do right now. It will also need to keep the memory footprint really tiny. Ruby being a more complicated language than Lua, it might be tricky, but we’ll see.

Documentation

While the idea of using Ruby has a macro language might get a lot of people excited, if documentation lacks or if the process is painful, the very same people might fallback to another language. Historically speaking, the MRI team has had issues with documentation and communication due to various factors. I hope that thanks to Matz experience and to the strong Ruby community, mruby will become a easy and efficient alternative to Lua or people wanting to embed one of the greatest interpreted languages.

Ruby outside of Rails

While Ruby on Rails really made Ruby popular, I’m always a bit dissapointed when I see how many popular projects the Ruby community has outside of Rails. Think about it, we have some awesome implementation such as JRuby and MacRuby which allow you to use an amazing amount of libraries to do the craziest thing one could imagine using the language they like the most. But yet, Rails is still by far the #1 Ruby project. Of course, there are many non-Rails related projects out there, but they don’t benefit from Rails' aura.

What I hope with mruby is that it will allow developers to leverage the beauty of Ruby and to create other niches for people to have fun. Some people already started:

####MobiRuby Yuichiro MASUI is working on having Ruby available on iOS and Android.

####Ruby for Node.js Yasuhiro Matsumoto is working on mruby-uv an interface for libuv Node.js' platform layer.

####mod_mruby

MATSUMOTO Ryosuke is working on an Apache module for mruby called mod_mruby which would be comparable to mod_lua

####mruby REPL

Frank Celler started working and blogging about writing a REPL for mruby, go check out his excellent blog posts on the shell he’s working on and other stuff he’s doing with mruby.

But there is plenty more to do, redis for instance now has/is about to be scriptabled via Lua, what about trying to use mruby to also support Ruby? What about scripting game logic and even full games in Ruby? What about mruby on a Raspberry pi? Ruby on my TV, fridge, car, AC, solar panel controller…

I will certainly be looking forward to people trying to reproduce the success of Rails in a different domain thanks to the Ruby language.

• The Open Source release of Matz' mruby on GitHub.
• The announce of MobiRuby, an upcoming solution to develop iOS and Android applications using Ruby.

Probably due to my involvement with the MacRuby project, people have been asking me what I thought of these news.

mruby

mruby is far from being a new project. It’s based on the RiteVM which is a sponsored project by the Japanese ministry of Economy, Trade and Industry and lead by Ruby’s creator: Yukihiro “Matz” Matsumoto and was explained in details during Matz’s RubyConf 2010 keynote.

Back in November 2011 Matz also explained mruby. His talk was recorded and he explains very well the current Ruby ecosystem and why mruby makes sense.

As explained, the main goal of mruby is to have a Ruby version that can be embedded and therefore have a smaller footprint, be compiled and linked within another application.

Hiroshi Nakamura gave a great 1 line definition of mruby:

mruby targets game developers (to use instead of Lua), embedded application developers (devices, TV, phones..) and small memory footprint server applications (instead of JS for instance).

I’m personally quite excited by mruby, it’s not there yet and there is still a lot of work to do to prove the value of the project but it’s certainly a great step in the right direction. What’s also really nice is that the project is released under an OSS license allowing for all of us to contribute and companies to improve the implementation based on their own needs.

Summary: mruby is a promising project even if it is still in its infancy. Besides being yet another Ruby implementation, the fact that the target audience and the project scope are well defined and that the project is lead by Ruby’s author and sponsored by the Japanese government makes me want to believe that it can be a successful project. That said Lua is a simpler language and it is already well implemented in the targeted market, so hopefuly Matz, his team and the Japanese government have a plan to advocate and champion this new technology. Good luck to them and I’ll keep an attentive eye on the project.

** date = “I” slug = “I/mruby-and-mobiruby” **

MobiRuby

MobiRuby is being developed by Yuichiro MASUI who works for Appcelerator the company behind the popular Titanium platform to write native iOS, Android apps in JS.

MobiRuby is built on top of mruby making it the first demonstration of what motivated developers can do with Matz new implementation. Very much like mruby, MobiRuby will be released under an OSS license but unlike mruby, the Apache license was chosen.

So far this was just an announcement with a code sample and a screenshot. That was enough to make the front page of HackerNews. Apparently the author is planning on releasing a first version in a few months.

It might surprise some, but I’m quite glad to see this kind of projects even though, they compete to some extent against MacRuby. It proves two things:

• there is a strong interest in having Ruby on mobile devices.
• it’s technically possible to do so.

Now, this is not something new either, Lua developers have been able to write iOS apps for a while, yet the majority of the iOS developers still use Objective-C. What are the challenges facing implementations trying to replace Objective-C?

The replacement language might not fit the Cocoa design.

Developing an iOS/OS X app means that you spend your time using provided libraries (called frameworks in Apple’s jargon). These frameworks have specific patterns, a well defined syntax and usually work in a very consistent/constraining way. Or your language is quite similar (like Ruby) and the transition is easy, or you need to start writing and maintaining wrappers (titanium).

Bridged runtimes.

Having 2 runtimes running at the same time is quite challenging and not efficient. That’s one of the reasons why Apple pushed MacRuby to move from RubyCocoa being a bridge and to have a Ruby implementation running in Objective-C runtime itself. This allows something else, in MacRuby all objects are actually Objective-C objects which means you don’t need to convert anything and Cocoa APIs can be extended from Ruby code by just reopening them.

Support.

This one is critical for many. Often, you don’t want to have your next big project rely on a technology that doesn’t have a good backing and support. What happens if you build your app using an alternate implementation and all a sudden the developer(s) get bored and move on, or take another job? What about the updates needed as Apple/Google update their platforms? It might not be the best reason to not choose an alternative, but it’s a reasonable reason especially for companies who want to be “safe”.

Cocoa

Cocoa APIs represent probably 90% of the challenge when writing iOS/OS X applications. The APIs, while powerful and efficient, are often a pain to get used to and to learn.

You have the challenge of the documentation and the examples that are only in Objective-C, requiring that someone writes a book and/or that you convert and maintain an enormous amount of documentation.

You also have all the tools provided by Apple which, you often can’t fully use because you aren’t using their toolchain.

To be honest, after so many years using MacRuby, I think that the real value of such a project isn’t in the easier syntax but instead in the fact that you can easily build wrappers and higher level interfaces around repetitive tasks. Having a mix of a well designed DSL and yet access to the native object is something extremely powerful.

Objective-C is evolving.

Objective-C is evolving, with the introduction of ARC, memory management became much easier. The latest version of clang also granted Objective-C with a nicer syntax thanks to new literals and object subscripting (read more). As a matter of fact, Objective-C’s syntax is getting closer and closer to Ruby’s making the choice to use an alternate much harder.

// character literals.
NSNumber *theLetterZ = @'Z';          // equivalent to [NSNumber numberWithChar:'Z']

// integral literals.
NSNumber *fortyTwo = @42;             // equivalent to [NSNumber numberWithInt:42]

// floating point literals.
NSNumber *piDouble = @3.1415926535;   // equivalent to [NSNumber numberWithDouble:3.1415926535]

// BOOL literals.
NSNumber *yesNumber = @YES;           // equivalent to [NSNumber numberWithBool:YES]

// Container literals
NSArray *array = @[ @"Hello", NSApp, [NSNumber numberWithInt:42] ];
id value = array[idx];

NSDictionary *dictionary = @{
  @"name" : NSUserName(),
  @"date" : [NSDate date],
  @"processInfo" : [NSProcessInfo processInfo]
};
id oldObject = dictionary[key];
dictionary[key] = newObject;	// replace oldObject with newObject

Performance.

Even though devices are more and more powerful, performance is often critical and Apple optimized the performance of their solution for their language. If you have ever developed a Titanium app, you know that it can be an issue and you might have to find workarounds to get decent performance.

Summary: Based on all these things, once MobiRuby will be released, I will be able to make a better judgement. But based on what I have seen so far, I’m quite concerned by the syntax and the performance we will get out of the box. But time will tell and things can always be improved. Ruby on iOS/Android is something exciting and I’m looking forward to testing the first betas.

Authentication vs Authorization

I wish these two words didn’t share the same root because it surely confuses a lot of people. My most frequently-discussed example is OAuth. Every time I start talking about implementing a centralized/unified authentication system, someone jumps in and suggests that we use OAuth. The challenge is that OAuth is an authorization system, not an authentication system.

It’s tricky, because you might actually be “authenticating” yourself to website X using OAuth. What you are really doing is allowing website X to use your information stored by the OAuth provider. It is true that OAuth offers a pseudo-authentication approach via its provider but that is not the main goal of OAuth: the Auth in OAuth stands for Authorization, not Authentication.

Here is how we could briefly describe each role:

• Authentication: recognizes who you are.

• Authorization: know what you are allowed to do, or what you allow others to do.

If you are feel stuck in your design and something seems wrong, ask yourself if you might be confused by the 2 auth words. This article will only focus on authentication.

A Common Scenario

This is probably the most common structure, though I made it slightly more complex by drawing the three main apps in different programming languages. We have three web applications running on different subdomains and sharing account data via a centralized authentication service.

Goals:

• Keep authentication and basic account data isolated.

• Allow users to stay logged in while browsing different apps.

Implementing such a system should be easy. That said, if you migrate an existing app to an architecture like that, you will spend 80% of your time decoupling your legacy code from authentication and wondering what data should be centralized and what should be distributed. Unfortunately, I can’t tell you what to do there since this is very domain specific. Instead, let’s see how to do the “easy part.”

Centralizing and Isolating Shared Account Data

At this point, you more than likely have each of your apps talk directly to shared database tables that contain user account data. The first step is to migrate away from doing that. We need a single interface that is the only entry point to create or update shared account data. Some of the data we have in the database might be app specific and therefore should stay within each app, anything that is shared across apps should be moved behind the new interface.

Often your centralized authentication system will store the following information:

• ID

• first name

• last name

• login/nickname

• email

• hashed password

• salt

• creation timestamp

• update timestamp

• account state (verified, disabled …)

Do not duplicate this data in each app, instead have each app rely on the account ID to query data that is specific to a given account in the app. Technically that means that instead of using SQL joins, you will query your database using the ID as part of the condition.

My suggestion is to do things slowly but surely. Migrate your database schema piece by piece assuring that everything works fine. Once the other pieces will be in place, you can migrate one code API a time until your entire code base is moved over. You might want to change your DB credentials to only have read access, then no access at all.

Login workflow

Each of our apps already has a way for users to login. We don’t want to change the user experience, instead we want to make a transparent modification so the authentication check is done in a centralized way instead of a local way. To do that, the easiest way is to keep your current login forms but instead of POSTing them to your local apps, we’ll POST them to a centralized authentication API. (SSL is strongly recommended)

As shown above, the login form now submits to an endpoint in the authentication application. The form will more than likely include a login or email and a clear text password as well as a hidden callback/redirect url so that the authentication API can redirect the user’s browser to the original app. For security reasons, you might want to white list the domains you allow your authentication app to redirect to.

Internally, the Authentication app will validate the identifier (email or login) using a hashed version of the clear password against the matching record in the account data. If the verification is successful, a token will be generated containing some user data (for instance: id, first name, last name, email, created date, authentication timestamp). If the verification failed, the token isn’t generated. Finally the user’s browser is redirected to the callback/redirect URL provided in the request with the token being passed.

You might want to safely encrypt the data in a way that allows the clients to verify and trust that the token comes from a trusted source. A great solution for that would be to use RSA encryption with the public key available in all your client apps but the private key only available on the auth server(s). Other strong encryption solutions would also work. For instance, another appropriate approach would be to add a signature to the params sent back. This way the clients could check the authenticity of the params. HMAC or DSA signature are great for that but in some cases, you don’t want people to see the content of the data you send back. That’s especially true if you are sending back a ‘mobile’ token for instance. But that’s a different story. What’s important to consider is that we need a way to ensure that the data sent back to the client can’t be tampered with. You might also make sure you prevent replay attacks.

On the other side, the application receives a GET request with a token param. If the token is empty or can’t be decrypted, authentication failed. At that point, we need to show the user the login page again and let him/her try again. If on the other hand, the token can be decrypted, the content should be saved in the session so future requests can reuse the data.

We described the authentication workflow, but if a user logins in application X, (s)he won’t be logged-in in application Y or Z. The trick here, is to set a top level domain cookie that can be seen by all applications running on subdomains. Certainly, this solution only works for apps being on the same domain, but we’ll see later how to handle apps on different domains.

The cookie doesn’t need to contain a lot of data, its value can contain the account id, a timestamp (to know when authentication happened and a trusted signature) and a signature. The signature is critical here since this cookie will allow users to be automatically logged in other sites. I’d recommend the  HMAC or DSA encryptions to generate the signature. The DSA encryption, very much like the RSA encryption is an asymmetrical encryption relying on a public/private key. This approach offers more security than having something based a shared secret like HMAC does. But that’s really up to you.

Finally, we need to set a filter in your application. This auto-login filter will check the presence of an auth cookie on the top level domain and the absence of local session. If that’s the case, a session is automatically created using the user id from the cookie value after the cookie integrity is verified. We could also share the session between all our apps, but in most cases, the data stored by each app is very specific and it’s safer/cleaner to keep the sessions isolated. The integration with an app running on a different service will also be easier if the sessions are isolated.

Registration

For registration, as for login, we can take one of two approaches: point the user’s browser to the auth API or make S2S (server to server) calls from within our apps to the Authentication app. POSTing a form directly to the API is a great way to reduce duplicated logic and traffic on each client app so I’ll demonstrate this approach.

As you can see, the approach is the same we used to login. The difference is that instead of returning a token, we just return some params (id, email and potential errors). The redirect/callback url will also obviously be different than for login. You could decide to encrypt the data you send back, but in this scenario, what I would do is set an auth cookie at the .domain.com level when the account is created so the “client” application can auto-login the user. The information sent back in the redirect is used to re-display the register form with the error information and the email entered by the user.

At this point, our implementation is almost complete. We can create an account and login using the defined credentials. Users can switch from one app to another without having to re login because we are using a shared signed cookie that can only be created by the authentication app and can be verified by all “client” apps. Our code is simple, safe and efficient.

Updating or deleting an account

The next thing we will need is to update or delete an account. In this case, this is something that needs to be done between a “client” app and the authentication/accounts app. We’ll make S2S (server to server) calls. To ensure the security of our apps and to offer a nice way to log requests, API tokens/keys will be used by each client to communicate with the authentication/accounts app. The API key can be passed using a X-header so this concern stays out of the request params and our code can process separately the authentication via X-header and the actual service implementation. S2S services should have a filter verifying and logging the API requests based on the key sent with the request. The rest is straight forward.

Using different domains

Until now, we assumed all our apps were on the same top domain. In reality, you will often find yourself with apps on different domains. This means that you can’t use the shared signed cookie approach anymore. However, there is a simple trick that will allow you to avoid requiring your users to re-login as they switch apps.

The trick consists, when a local session isn’t present, of using an iframe in the application using the different domain. The iframe loads a page from the authentication/accounts app which verifies that a valid cookie was set on the main top domain. If that is the case, we can tell the application that the user is already globally logged in and we can tell the iframe host to redirect to an application end point passing an auth token the same way we did during the authentication. The app would then create a session and redirect the user back to where (s)he started. The next requests will see the local session and this process will be ignored.

If the authentication application doesn’t find a signed cookie, the iframe can display a login form or redirect the iframe host to a login form depending on the required behavior.

Something to keep in mind when using multiple apps and domains is that you need to keep the shared cookies/sessions in sync, meaning that if you log out from an app, you need to also delete the auth cookie to ensure that users are globally logged out. (It also means that you might always want to use an iframe to check the login status and auto-logoff users).

Mobile clients

Another part of implementing a SSO solution is to handle mobile clients. Mobile clients need to be able to register/login and update accounts. However, unlike S2S service clients, mobile clients should only allow calls to modify data on the behalf of a given user. To do that, I recommend providing opaque mobile tokens during the login process. This token can then be sent with each request in a X-header so the service can authenticate the user making the request. Again, SSL is strongly recommended.

In this approach, we don’t use a cookie and we actually don’t need a SSO solution, but an unified authentication system.

Writing web services

Our Authentication/Accounts application turns out to be a pure web API app.

We also have 3 sets of APIs:

• Public APIs: can be accessed from anywhere, no authentication required

• S2S APIs: authenticated via API keys and only available to trusted clients

• Mobile APIs: authenticated via a mobile token and limited in scope.

We don’t need dynamic HTML views, just simple web service related code. While this is a little bit off topic, I’d like to take a minute to show you how I personally like writing web service applications.

Something that I care a lot about when I implement web APIs is to validate incoming params. This is an opinionated approach that I picked up while at Sony and that I think should be used every time you implement a web API. As a matter of fact, I wrote a Ruby DSL library (Weasel Diesel) allowing you describe a given service, its incoming params, and the expected output. This DSL is hooked into a web backend so you can implement services using a web engine such as Sinatra or maybe Rails3. Based on the DSL usage, incoming parameters are be verified before being processed. The other advantage is that you can generate documentation based on the API description as well as automated tests.

You might be familiar with Grape, another DSL for web services. Besides the obvious style difference Weasel Diesel offers the following advantages:

• input validation/sanitization

• service isolation

• generated documentation

• contract based design

Here is a hello world webservice being implemented using Weasel Diesel and Sinatra:

{% gist 2300131 %}

Basis test validating the contract defined in the DSL and the actual output when the service is called:

{% gist 2300440 %}

Generated documentation:

If the DSL and its features seem appealing to you and you are interested in digging more into it, the easiest way is to fork this demo repo and start writing your own services.

The DSL has been used in production for more than a year, but there certainly are tweaks and small changes that can make the user experience even better. Feel free to fork the DSL repo and send me Pull Requests.

Migration/upgrades

Migrating a Rails App from a version to the other is very much like playing the lottery, you are almost sure you will lose. To be more correct, you know things will break, you just don’t know what, when and how. The Rails team seems to think that everybody is always running on the cutting edge version and don’t consider people who prefer to stay a few version behind for stability reasons. What’s worse is that plugins/gems might or might not compatible with the version you are updating to, but you will only know that by trying yourself and letting others try and report potential issues.

This is for me, by far, the biggest issue with Rails and something that should have been fixed a long time ago. If you’re using the WordPress blog engine, you know how easy and safe it is to upgrade the engine or the plugins. Granted WordPress isn’t a web dev framework, but it gives you an idea of what kind of experience we should be striving for.

Stability vs playground zone

New features are cool and they help make the platform more appealing to new comers. They also help shape the future of a framework. But from my perspective, that shouldn’t come to the cost of stability. Rails 3’s new asset pipeline is a good example of a half-baked solution shoved in a release at the last minute and creating a nightmare for a lot of us trying to upgrade. I know, I know, you can turn off the asset pipeline and it got better since it was first released. But shouldn’t that be the other way around? Shouldn’t fun new ideas risking the stability of an app or making migration harder, be off by default and turned on only by people wanting to experiment? When your framework is young, it’s normal that you move fast and sometimes break, but once it matures, these things shouldn’t happen.

Public/private/plugin APIs

This is more of a recommendation than anything else. When you write a framework in a very dynamic language like Ruby, people will “monkey patch” your code to inject features. Sometimes it is due to software design challenges, sometimes it’s because people don’t know better. However,  by not explicitly specifying what APIs are private (they can change at anytime, don’t touch), what APIs are public (stable, will be slowly deprecated when they need to be changed) and which ones are for plugin devs only (APIs meant for instrumentation, extension etc..), you are making migration to newer versions much harder. You see, if you have a small, clean public API, then it’s easy to see what could break, warn developers and avoid migration nightmares. However, you need to start doing that early on in your project, otherwise you will end up like Rails where all code can potentially change anytime.

Rails/Merb merge was a mistake

This is my personal opinion and well, feel free to disagree, nobody will ever be able to know to for sure. Without explaining what happened behind closed doors and the various personal motivations, looking at the end result, I agree with the group of people thinking that the merge didn’t turn up to be a good thing. For me, Rails 3 isn’t significantly better than Rails 2 and it took forever to be released. You still can’t really run a mini Rails stack like promised. I did hear that Strobe (company who was hiring Carl Lerche, Yehuda Katz and contracted Jose Valim) used to have an ActionPack based, mini stack but it was never released and apparently only Rails core members really knew what was going on there. Performance in vanilla Rails 3 are only now getting close to what you had with Rails 2 (and therefore far from the perf you were getting with Merb). Thread-safety is still OFF by default meaning that by default your app uses a giant lock only allowing a process to handle 1 request at a time. For me, the flexibility and performance focus of Merb were mainly lost in the merge with Rails. (Granted, some important things such as ActiveModel, cleaner internals and others have made their way into Rails 3)

But what’s worse than everything listed so far is that the lack of competition and the internal rewrites made Rails lose its headstart.  Rails is very much HTML/view focused, its primarily strength is to make server side views trivial and it does an amazing job at that. But let’s be honest, that’s not the future for web dev. The future is more and more logic pushed to run on the client side (in JS) and the server side being used as an API serving data for the view layer. I’m sorry but adding support for CoffeeScript doesn’t really do much to making Rails evolve ahead of what it currently is. Don’t get me wrong, I’m a big fan of CoffeeScript, that said I still find that Rails is far from being optimized to developer web APIs in Rails. You can certainly do it, but you are basically using a tool that wasn’t designed to write APIs and you pay the overhead for that. If there is one thing I wish Rails will get better at is to make writing pure web APIs better (thankfully there is Sinatra). But at the end of the day, I think that two projects with different philosophies and different approaches are really hard to merge, especially in the open source world. I wouldn’t go as far as saying like others that Rails lost its sexiness to node.js because of the wasted time, but I do think that things would have been better for all if that didn’t happen. However, I also have to admit that I’m not sure how much of a big deal that is. I prefer to leave the past behind, learn from my own mistake and move on.

Technical debts

Here I’d like to stop to give a huge props to Aaron “@tenderlove” Patterson, the man who’s actively working to reduce the technical debts in the Rails code base. This is a really hard job and definitely not a very glamorous one. He’s been working on various parts of Rails including its router and its ORM (ActiveRecord). Technical debts are unfortunately normal in most project, but sometimes they are overwhelming to the point that nobody dares touching the code base to clean it up. This is a hard problem, especially when projects move fast like Rails did. But looking back, I think that you want to start tackling technical debts on the side as you move on so you avoid getting to the point that you need a hero to come up and clean the piled errors made in the past. But don’t pause your entire project to clean things up otherwise you will lose market, momentum and excitement. I feel that this is also very much true for any legacy project you might pick up as a developer.

Keep the cost of entry level low

Getting started with Rails used to be easier. This can obviously argued since it’s very subjective, but from my perspective I think we forgot where we come from and we involuntary expect new comers to come with unrealistic knowledge. Sure, Rails does much more than it used to do, but it’s also much harder to get started. I’m not going to argue how harder  it is now or why we got there. Let’s just keep in mind that it is a critical thing that should always be re-evaluated. Sure, it’s harder when you have an open source project, but it’s also up to the leadership to show that they care and to encourage and mentor volunteers to  focus on this important part of a project.

Documentation

Rails documentation isn’t bad, but it’s far from being great. Documentation certainly isn’t one of the Ruby’s community strength, especially compared with the Python community, but what saddens me is to see the state of the official documentation which, should, in theory be the reference. Note that the Rails guides are usually well written and provide value, but they too often seem too light and not useful when you try to do something not totally basic (for instance use an ActiveModel compliant object). That’s probably why most people don’t refer to them or don’t spend too much time there. I’m not trying to blame anyone there. I think that the people who contributed theses guides did an amazing job, but if you want to build a strong and easy to access community, great documentation is key. Look at the Django documentation as a good example. That said, I also need to acknowledge the amazing job done by many community members such as Ryan Bates and Michael Hartl consistently providing high value external documentation via the railscasts and the intro to Rails tutorial available for free.

In conclusion, I think that there is a lot to learn from Rails, lots of great things as well as lots of things you would want to avoid. We can certainly argue on Hacker News or via comments about whether or not I’m right about Rails failures, my point will still be that the mentioned issues should be avoided in any projects, Rails here is just an example. Many of these issues are currently being addressed by the Rails team but wouldn’t it be great if new projects learn from older ones and avoid making the same mistakes? So what other mistakes do you think I forgot to mention and that one should be very careful of avoiding?

Updates:

1. Rails 4 had an API centric app generator but it was quickly reverted and will live as gem until it’s mature enough.

2. Rails 4 improved the ActiveModel API to be simpler to get started with. See this blog post for more info.

Before I show you the benchmark numbers and you jump to conclusions, it’s important to realize that in the grand scheme of things, the performance cost we are talking is small enough that it is certainly not the main reason why your application is slow. Spoiler alert: ActiveRecord is slow but the cost of initialization isn’t by far the worse part of ActiveRecord. Also, even though this article doesn’t make activeRecord look good, and I’m not trying to diss it. It’s a decent ORM that does a great job in most cases.

Let’s get started by the benchmarks number to give us an idea of the damage (using Ruby 1.9.3 p125):

                                                             | Class | Hash  | AR 3.2.1 | AR no protection | Datamapper | Sequel |
--------------------------------------------------------------------------------------------------------------------------------------
.new() x100000                                               | 0.037 | 0.049 | 1.557    | 1.536            | 0.027      | 0.209  |
.new({:id=>1, :title=>"Foo", :text=>"Bar"}) x100000          | 0.327 | 0.038 | 6.784    | 5.972            | 4.226      | 1.986  |

You can see that I am comparing the allocation of a Class instance, a Hash and some ORM models. The benchmark suite tests the allocation of an empty object and one with passed attributes. The benchmark in question is available here.

As you can see there seems to be a huge performance difference between allocating a basic class and an ORM class. Instantiating an ActiveRecord class is 20x slower than instantiating a normal class, while ActiveRecord offers some extra features, why is it so much slower, especially at initialization time?

The best way to figure it out is to profile the initialization. For that, I used perftools.rb and I generated a graph of the call stack.

Here is what Ruby does (and spends its time) when you initialize a new Model instance (click to download the PDF version):

This is quite a scary graph but it shows nicely the features you are getting and their cost associated. For instance, the option of having the before and after initialization callback cost you 14% of your CPU time per instantiation, even though you probably almost never use these callbacks. I’m reading that by interpreting the node called ActiveSupport::Callback#run_callbacks, 3rd level from the top. So 14.1% of the CPU time is spent trying to run callbacks. As a quick note, note that 90.1% of the CPU time is spent initializing objects, the rest is spent in the loop and in the garbage collection (because the profiler runs many loops). You can then follow the code and see how the code works, creating a dynamic class callback method on the fly (the one with the long name) and then recreating the name of this callback to call it each time the object is allocated. It sounds like that’s a good place for some micro optimizations which could yield up to 14% performance increase in some cases.

Another major part of the CPU time is spent in ActiveModel’s sanitization. This is the piece of code that allows you to block some model attributes to be mass assigned. This is useful when you don’t want to sanitize your incoming params but want to create or update a model instance by using all the passed user params. To avoid malicious users to modify some specific params that might be in your model but not in your form, you can protect these attributes. A good example would be an admin flag on a User object. That said, if you manually initialize an instance, you don’t need this extra protection, that’s why in the benchmark above, I tested and without the protection. As you can see, it makes quite a big difference. The profiler graph of the same initialization without the mass assignment protection logically ends up looking quite different:

Update: My colleague Glenn Vanderburg pointed out that some people might assuming that the shown code path is called for each record loaded from the database. This isn’t correct, the graph represents instances allocated by calling #new. See the addition at the bottom of the post for more details about what’s going on when you fetch data from the DB.

I then decided to look at the graphs for the two other popular Ruby ORMs:

Datamapper

and Sequel

While I didn’t give you much insight in ORM code, I hope that this post will motivate you to sometimes take a look under the cover and profile your code to see what’s going on and why it might be slow. Never assume, always measure. Tools such as perftools are a great way to get a visual feedback and get a better understanding of how the Ruby interpreter is handling your code.

UPDATE:

I heard you liked graphs so I added some more, here is what’s going on when you do Model.first:

Model.all

And finally this is the code graph for a call to Model.instantiate which is called after a record was retrieved from the database to convert into an Object. (You can see the #instantiate call referenced in the graph above).

##Description of the talk:

Let’s be honest, Ruby became mainstream a few years back and it isn’t the cool underground programming language it once was. It’s quite likely that your cousin’s boyfriend who’s “into computers" knows what Ruby on Rails is. There are hundreds of books, conferences, training and meetups for Rubyists. Recruiters fight to hire whoever knows how to generate a scaffolded Rails app. But now cool kids can’t stop talking about node.js, CoffeeScript, Clojure, Haskell and pushing code to the UI layer. What does it mean for the new, existing and prospecting Ruby developers? Is it time to jump ship and move on to something else?

##Slides

The slides are available on Matt’s SpeakerDeck and can be [downloaded here]({{ page.slides }}).

##Video

{% video https://cdn.confreaks.com/system/assets/datas/3173/original/816-larubyconf2012-time-to-move-away-from-ruby-small.mp4 640 360 /images/matt_aimonetti_larubyconf2012_video.png %}

##Presentation website

Matt’s presentation was filmed by Confreaks and posted here.

I was nicely surprised to see so many of my followers send recommendations. Here is a list of 25 books that like-minded people suggested I read. Hopefully you will find a book or two to read too. Feel free to send more recommendations via the comments.

1Q84 by Haruki Murakami

suggested by @mrb_bk and @chadfowler

The Floating Opera and The End of the Road by John Barth

suggested by @chadfowler

Into Thin Air by Jon Krakauer

suggested by @bradly

Cutting for Stone by Abraham Verghese

suggested by @bradly

Atlas Shrugged by Ayn Rand

suggested by @bradly

Cien años de soledad by Gabriel Garcia Marquez (es)

suggested by @romanandreg & @jrfernandez & @edgarschmidt

One Hundred Years of Solitude by Gabriel García Marquez

suggested by @romanandreg & @jrfernandez & @edgarschmidt

Jitterbug Perfume by Tom Robbins

suggested by @supaspoida

The Sisters Brothers by Patrick deWitt

suggested by @dennismajor1

The Glass Bead Game by Hermann Hesse

suggested by @dj2sincl

The Wind-Up Bird Chronicle by Haruki Murakami

suggested by @chadfowler

Mindfire by Scott Berkun

suggested by @lucasdicioccio

Les Fourmis by Bernard Werber (fr)

suggested by @twitty_tim

Perfume: The Story of a Murderer by Patrick Suskind

suggested by @twitty_tim

Les Miserables by Victor Hugo (en, free ebook)

suggested by @tutec

Song Of Ice and Fire by George R.R. Martin (Game of Thrones saga)

suggested by @eeppa & @jarin

Clockwork Century by Cherie Priest

suggested by @eeppa

The Darkness that Comes Before by R. Scott Bakker

suggested by @eeppa

Drood by Dan Simmons

suggested by @eeppa

This Is Water by David Foster Wallace

suggested by @atduskgreg

Anathem by Neal Stephenson

suggested by @jarin

Ender’s Game by Orson Scott Card (entire saga)

suggested by @jarin & @edgarschmidt

Snow Crash by Neal Stephenson

suggested by @jarin

Fixing the Game by Roger L. Martin

suggested by @jarkko

The Road by Cormac McCarthy

suggested by @mrreynolds

Process to develop a curriculum:

Purpose. Know why you’re doing what you’re doing.

• You know how to do this.

Product. Start with the end in mind.

• What does the student look like when they walk out the door at the end of the training.

• Usually, we break these down into Knowledge, Skills, or Attitudes.

• Sometimes it’s helpful to see a photograph or drawing of a someone who finished the program and just talk about what they can do that makes them successful.

• This “product” should be connected and help you accomplish your mission

Practices. Then ask yourself, “How do people become like this?"

• If you can break down your Product into 3-5 bit-sized chunks, then see how people learn each one of those skills, gain each one of those knowledge points, and how to they gain the attitudes you want them to have.

• This one is much easier the more experience you have in seeing people develop the “Product.”

• This is also easier to determine when you understand Learning Theory.

• The results from this section will result in a list of:

  •        Activities or experiences

  •        Resources. What books, website, teachers, software, etc. will help them learn more effectively and efficiently

  •        Assessments. How you would know if the activity was helpful?

Plans. Make your plans based on the practices you’ve determined you’ve needed.

On a related topic, Chad Fowler posted an interesting blog post about what LivingSocial is doing to change the software development education.

##Description of the talk:

Concurrency in Ruby and Python implementations is quite a hot topic nowadays. In this talk, Matt will explain how concurrency works for a language that uses a Global Interpreter Lock and it means concretely. Finally Aimonetti will also cover Ruby’s garbage collector (GC) and why it’s important to understand how automatic memory management works to get better performance out of your code.

##Slides

[]({{ page.slides }}) [The HTML5 slides can be seen online]({{ page.slides}}).

##Video

[Vimeo page of the presentation]({{ page.video_page }})

##Presentation website

Matt’s presentation was filmed by RubyConf Argentina and posted [here]({{ page.video_page }}).

##Matt’s articles related to this presentation

• About concurrency and the GIL
• Data safety and GIL removal
• Concurrency in Ruby explained

Showing obvious concrete examples of data corruption due to unsafe threaded code isn’t actually as easy at it sounds. First of all, even with a GIL, developers can write unsafe threaded code. So we need to focus only on the safety problems raised by removing the GIL. To demonstrate what I mean, I will try to create some race conditions and show you the unexpected results you might get. Again, before you go crazy on the comments, remember that threaded code is indeterministic and the code below might potentially work on your machine and that’s exactly why it is hard to demonstrate. Race conditions depend on many things, but in this case I will focus on race conditions affecting basic data structures since it might be the most surprising.

Example:

@array, threads = [], []
4.times do
  threads << Thread.new { (1..100_000).each {|n| @array << n} }
end
threads.each{|t| t.join }
puts @array.size

In the above example, I’m creating an instance variable of Array type and I start 4 threads. Each of these threads adds 100,000 items to the array. We then wait for all the threads to be done and check the size of the array.

If you run this code in C Ruby the end result will be as expected:

400000

Now if you switch to JRuby you might be surprised by the output. If you are lucky you will see the following:

ConcurrencyError: Detected invalid array contents due to unsynchronized modifications with concurrent users
        << at org/jruby/RubyArray.java:1147
  __file__ at demo.rb:3
      each at org/jruby/RubyRange.java:407
  __file__ at demo.rb:3
      call at org/jruby/RubyProc.java:274
      call at org/jruby/RubyProc.java:233

This is actually a good thing. JRuby detects that you are unsafely modifying an instance variable across threads and that data corruption will occur. However, the exception doesn’t always get raised and you will potentially see results such as:

335467
342397
341080

This is a sign that the data was corrupted but that JRuby didn’t catch the unsynchronized modification. On the other hand MacRuby and Rubinius 2 (dev) won’t raise any exceptions and will just corrupt the data, outputting something like:

294278
285755
280704
279865

In other words, if not manually synchronized, shared data can easily be corrupted. You might have two threads modifying the value of the same variable and one of the two threads will step on top of the other leaving you with a race condition. You only need 2 threads accessing the same instance variable at the same time to get a race condition. My example uses more threads and more mutations to make the problem more obvious. Note that TDD wouldn’t catch such an issue and even extensive testing will provide very little guarantee that your code is thread safe.

So what? Thread safety isn’t a new problem.

That’s absolutely correct, ask any decent Java developer out there, he/she will tell how locks are used to “easily” synchronize objects to make your code thread safe. They might also mention the deadlocks and other issues related to that, but that’s a different story. One might also argue that when you write web apps, there is very little shared data and the chances of corrupting data across concurrent requests is very small since most of the data is kept in a shared data store outside of the process.

All these arguments are absolutely valid, the challenge is that you have a large community and a large amount of code out there that expects a certain behavior. And removing the GIL does change this behavior. It might not be a big deal for you because you know how to deal with thread safety, but it might be a big deal for others and C Ruby is by far the most used Ruby implementation. It’s basically like saying that automatic cars shouldn’t be made and sold, and everybody has to switch to stick shifts. They have better gas mileage, I personally enjoy driving then and they are cheaper to build. Removing the GIL is a bit like that. There is a cost associated with this decision and while this cost isn’t insane, the people in charge prefer to not pay it.

Screw that, I’ll switch to Node.js

I heard a lot of people telling me they were looking into using Node.js because it has a better design and no GIL. While I like Node.js and if I were to implement a chat room or an app keeping connections for a long time, I would certainly compare it closely to EventMachine, I also think that this argument related to the GIL is absurd. First, you have other Ruby implementations which don’t have a GIL and are really stable (i.e: JRuby) but then Node basically works the same as Ruby with a GIL. Yes, Node is evented and single threaded but when you think about it, it behaves the same as Ruby 1.9 with its GIL. Many requests come in and they are handled one after the other and because IO requests are non-blocking, multiple requests can be processed concurrently but not in parallel. Well folks, that’s exactly how C Ruby works too, and unlike popular believe, most if not all the popular libraries making IO requests are non blocking (when using 1.9). So, next time you try to justify you wanting to toy with Node, please don’t use the GIL argument.

What should I do?

As always, evaluate your needs and see what makes sense for your project. Start by making sure you are using Ruby 1.9 and your code makes good use of threading. Then look at your app and how it behaves, is it CPU-bound or IO-bound. Most web apps out there are IO-bound (waiting for the DB, redis or API calls), and when doing an IO call, Ruby’s GIL is released allowing another thread to do its work. In that case, not having a GIL in your Ruby implementation won’t help you. However, if your app is CPU-bound, then switching to JRuby or Rubinius might be beneficial. However, don’t assume anything until you proved it and remember that making such a change will more than likely require some architectural redesign, especially if using JRuby.  But, hey, it might totally be worth it as many proved it in the past.

I hope I was able to clarify things a bit further. If you wish to dig further, I would highly recommend you read the many discussions the Python community had in the last few years.

I recently read a great management book called the five dysfunctions of a team by Patrick Lencioni . Instead of telling you what to do, the author highlights behavior patterns that are related to each other and when aggregated result in dysfunctional teams. I really liked the book because instead of a being a cookbook/playbook, this is more a fail book, in other words, it illustrates what you don’t want to do and explains why. It highlights very well the relation between various behaviors and nicely illustrates why teams of brilliant people can fail. The Kindle version is at less than $5, go get it and read it on your iPhone/iPad/computer/browser…

So this book somewhat changed my perception of management and leadership. Interesting enough, at Sony, my previous employer, they make a distinction between management and leadership. While they hope managers can be leaders, they don’t require them to be and to be honest very few are. I’m not sure that’s a good or a bad things, but I, for sure, was under different expectations. Finally, I spent a large amount of my life on the internet working on/with projects where meritocracy, respect and honor were key. The “ranking” is purely based on what your peers think of you and not based on your age/sex/origin/diploma/bank account. I do realize that this model has many pros but also some pretty major cons. My only point is that it did affect my worldview. In my world, seniority, a killer  job title or a fancy suit won’t buy you my automatic respect. On the other hand, job well done, great vision, honesty, over achievement will!

Taking these few trains of thoughts in consideration, I started thinking about my own expectations for a good manager/leader. I figured that if I were able to do that, I could possibly be able to define a work environment where I could thrive and maybe one day become a good “manager/leader”.

I’ve always questioned my ability to be a good leader. While most of the time, I have an opinion and can easily decide what I think should be done, I have a hard time relating to people who can’t see the “big picture”. While I usually can get decent results, I’m aware that it can unfortunately sometime be at the cost of a few bruised egos. I also know I have high expectations for myself and for others and I have a hard time understanding how some people can be ok with the “status-quo”. I’m a perfectionist who is only happy when he outperforms his previous achievement. I was raised to challenge and always push myself further, focusing on concrete end-results and achieved goals. And to be honest, that’s what I enjoy. But I also know for a fact, that many people are not like that and I can’t blame them for looking at things from a different angle and not sharing the same motivations. Furthermore, I know that most people actually don’t have the same driven temperament and that’s why I’ve questioned my abilities to lead others.

However, different temperaments can work together as long as there is respect. And by respect, I mean that everyone feel that they were being heard and know that their input was considered and addressed even though the outcome might not be as hoped for. But for respect to happen, you first need trust. And when people trust each other, Lencioni explains that “people don’t hold back one with another. They are unafraid to air their dirty laundry. They admit their mistakes, their weaknesses, and their concerns without fear of reprisal”. I think that as simple as it seems, it is the key to a successful team. A good leader should be able to create such an atmosphere where people can trust each other. In fact, I think that if a manger/leader/executive can manage to build trust as defined earlier, his technical skills or lack of vision don’t matter as much. He/she will be able to rely on people he trusts to help him make the right decisions. Of course, there is much more than to be a good leader, but I think that with this base, great things can be built, and without it, a much greater effort is required to get some good results.

Based on my findings, I think that I need to work on my communication so others don’t feel that they have to hold back and make sure everyone feels that their opinions were considered and addressed. To do that a key element is to admit my mistakes and weaknesses and asking others to help me improve. That’s it, sorry for the boring, not technical post. I promise the next one will have at least a code sample.

The GIL was probably even more of a hot topic now that Rubinius is about the join JRuby and MacRuby in the realm of GIL-less Ruby implementations.

During my RubyConf talk (slides here), I tried to explain how C Ruby works and why some decisions like having a GIL were made and why the Ruby core team isn’t planning on removing this GIL anytime soon. The GIL is something a lot of Rubyists love to hate, but a lot of people don’t seem to question why it’s here and why Matz doesn’t want to remove it. Defending the C Ruby decision isn’t quite easy for me since I spend my free time working on an alternative Ruby implementation which doesn’t use a GIL (MacRuby). However, I think it’s important that people understand why the MRI team (C Ruby team) and some Pythonistas feels so strongly about the GIL.

What is the GIL?

Here is a quote from the Python wiki:

In CPython, the global interpreter lock, or GIL, is a mutex that prevents multiple native threads from executing Python bytecodes at once. This lock is necessary mainly because CPython’s memory management is not thread-safe. (However, since the GIL exists, other features have grown to depend on the guarantees that it enforces.) […] The GIL is controversial because it prevents multithreaded CPython programs from taking full advantage of multiprocessor systems in certain situations. Note that potentially blocking or long-running operations, such as I/O, image processing, and NumPy number crunching, happen outside the GIL. Therefore it is only in multithreaded programs that spend a lot of time inside the GIL, interpreting CPython bytecode, that the GIL becomes a bottleneck.

The same basically applies to C Ruby. To illustrate the quote above, here is a diagram representing two threads being executed by C Ruby:

Such a scheduling isn’t a problem at all when you only have 1 cpu, since a cpu can only execute a piece of code at a time and context switching happens all the time to allow the machine to run multiple processes/threads in parallel. The problem is when you have more than 1 CPU because in that case, if you were to only run 1 Ruby process, then you would most of the time only use 1 cpu at a time. If you are running on a 8 cpu box, that’s not cool at all! A lot of people stop at this explanation and imagine that their server can only handle one request at a time and they they rush to sign Greenpeace petitions asking Matz to make Ruby greener by optimizing Ruby and saving CPU cycles. Well, the reality is slightly different, I’ll get back to that in a minute. Before I explain “ways to achieve true concurrency with CRuby, let me explain why C Ruby uses a GIL and why each implementation has to make an important choice and in this case both CPython and C Ruby chose to keep their GIL.

Why a GIL in the first place?

• It makes developer’s lives easier (it’s harder to corrupt data)

• It avoids race conditions within C extensions

• It makes C extensions development easier (no write barriers..)

• Most of the C libraries which are wrapped are not thread safe

• Parts of Ruby’s implementation aren’t threadsafe (Hash for instance)

Categories = [

##Description of the talk:

Programming languages, such as Ruby, are natural and elegant. But to achieve this elegance, things have to happen under the hood. Garbage Collection, concurrency, Global Interpreter Lock, metaprogramming, C extensions are just some of the things happening with or without our knowledge. Trying to understand these concepts, their implementations and their implications in daily coding might seem daunting. However, having a good understanding of these topics will make you a better developer. No CS degree or PhD required to attend this talk.

At the end of the talk, Matz (Ruby’s creator) came on stage to answer some questions related to what was explained.

##Slides [The HTML5 slides can be seen online]({{ page.slides}}) online.

##Video

{% video https://cdn.confreaks.com/system/assets/datas/2666/original/714-rubyconf2011-complex-ruby-concepts-dummified-small.mp4 640 360 /images/matt_aimonetti_presentation_ruby.jpg %}

##Presentation website

Matt’s presentation was filmed by confreaks and the dedicated page for the talk is available [here]({{ page.video_page }}).

##Matt’s articles related to this presentation

• About concurrency and the GIL
• Data safety and GIL removal
• Concurrency in Ruby explained

However, most browsers (IE 8+, Firefox 3.5+, Safari 4+, Chrome) implement a simple way to allow cross domain requests as defined in this w3C document.

Of course, if your users have an old version of their browser, you  might have to look into jsonp or something else such as cheating by using iframes & setting document.domain. Let’s pretend for a minute that 100% of your users are on Chrome. The only thing you need to do is set a response header listing the accepted domains or “*” for all. A simple Rack middleware to do that would look like that.

class XOriginEnabler
  ORIGIN_HEADER = "Access-Control-Allow-Origin"

  def initialize(app, accepted_domain="*")
    @app = app
    @accepted_domain = accepted_domain
  end

  def call(env)
    status, header, body = @app.call(env)
    header[ORIGIN_HEADER] = @accepted_domain
    [status, header, body]
  end
end

And to use the middleware you would need to set it for use:

use XOriginEnabler

To enable all requests from whatever origin, or pass the white listed domain(s) as shown below.

use XOriginEnabler, "demo.mysite.com demo.mysite.fr demo.techcrunch.com"

For a full featured middleware, see this project.

The challenge is that blocks come with their context and you can’t execute a block in the context of another one.

Here is a reduction of the challenge I was trying to solve:

class SolutionZero
  def initialize(origin, █)
    @origin = origin
    @block = block
  end

  def dispatch
    @block.call
  end
end

SolutionZero.new(42){ @origin + 1 }.dispatch
# undefined method `+' for nil:NilClass (NoMethodError)

The problem is that the block refers to the @origin instance variable which is not available in its context. My first workaround was to use instance_eval:

class SolutionOne
  def initialize(origin, █)
    @origin = origin
    @block = block
  end

  def dispatch
    self.instance_eval &@block
  end
end

SolutionOne.new(40){ @origin + 2}.dispatch
# 42

My workaround worked fine, since the block was evaluated in the context of the instance and therefore the @origin ivar is made available to block context. Technically, I was good to go, but I wasn’t really pleased with this solution. First using instance_eval often an indication that you are trying to take a shortcut. Then having to convert my block stored as a block back into a proc every single dispatch makes me sad. Finally, I think that this code is probably not performing as well as it could, mainly due to unnecessary object allocations and code evaluation. I did some benchmarks replacing instance_eval by instance_exec since looking at the C code, instance_exec should be slightly faster. Turns out, it is not so I probably missed something when reading the implementation code.

I wrote some more benchmarks and profiled a loop of 2 million dispatches (only the #disptach method call on the same object). The GC profiler report showed that the GC was invoked 287 times and each invocation was blocking the execution for about 0.15ms. Using Ruby’s ObjectSpace and disabling the GC during the benchmark, I could see that each loop allocates an object of type T_NODE which is more than likely our @block ivar converted back into a block. This is quite a waste. Furthermore, having to evaluate our block in a different context every single call surely isn’t good for performance.

So instead of doing the work at run time, why not doing it at load time? By that I mean that we can optimize the #dispatch method if we could “precompile” the method body instead of “proxying” the dispatch to an instance_eval call. Here is the code:

class SolutionTwo
  def initialize(origin, █)
    @origin = origin
    implementation(block)
  end

  private

  def implementation(block)
    mod = Module.new
    mod.send(:define_method, :dispatch, block)
    self.extend mod
  end
end

SolutionTwo.new(40){ @origin + 2}.dispatch
# 42

This optimization is based on the fact that the benchmark (and the real life usage) creates the instance once and then calls #dispatch many times. So by making the initialization of our instance a bit slower, we can drastically improve the performance of the method call. We also still need to execute our block in the right context. And finally, each instance might have a different way to dispatch since it is defined dynamically at initialization. To work around all these issues, we create a new module on which we define a new method called dispatch and the body of this method is the passed block. Then we simply our instance using our new module.

Now every time we call #dispatch, a real method is dispatched which is much faster than doing an eval and no objects are allocated. Running the profiler and the benchmarks script used earlier, we can confirm that the GC doesn’t run a single time and that the optimized code runs 2X faster!

Once again, it’s yet another example showing that you should care about object allocation when dealing with code in the critical path. It also shows how to work around the block bindings. Now, it doesn’t mean that you have to obsess about object allocation and performance, even if my last implementation is 2X faster than the previous, we are only talking about a few microseconds per dispatch. That said microseconds do add up and creating too many objects will slow down even your faster code since the GC will stop-the-world as its cleaning up your memory. In real life, you probably don’t have to worry too much about low level details like that, unless you are working on a framework or sharing your code with others. But at least you can learn and understand why one approach is faster than the other, it might not be useful to you right away, but if you take programming as a craft, it’s good to understand how things work under the hood so you can make educated decisions.  

Update:

@apeiros in the comments suggested a solution that works & performs the same as my solution, but is much cleaner:

class SolutionTwo
  def initialize(origin, █)
    @origin = origin
    define_singleton_method(:dispatch, block) if block_given?
  end
end

I’ll skip the intro to Rails 3.1 and how to use the new asset pipeline, refer to the Rails guide or one of the mainly posts referenced in this post (if I had properly read the guide, it would have saved me some valuable time, trust me, read it carefuly).

At that point last night, I had my app working great locally, Bruce created some awesome scss code using mixins and nested rules, the HTML was clean and working great, my CoffeeScript was brewing nicely, all was great until I tried to deploy to our QA environment.

JavaScript runtime dependency

The first thing you will notice is that you need the proper JavaScript runtime so the asset pipeline works properly. Not a big deal, you’ll find a lot of documentation about that. The problem is that you need to update your production environment or use depend on gem that will compile the required runtime (sounds dirty to me). So if you are deploying to many machines and you are using an image solution (EC2 AMI or other), you will need to update your image or spin new instances via updated chef/puppet recipes. In this case, the awesome team at LivingSocial had an image ready for me, so that wasn’t a big deal, but still, you need to take that in consideration as you are planning to update.

So the asset pipeline optimizes your asset management by processing/compiling asset files for you and optimizing their delivery. Instead of serving static files directly via public/images or public/javascripts you know serve them via the asset pipeline which will take care of compiling your CoffeeScript, grouping and minifying your JS and preprocessing all sorts of format. It also optimizes the caching process by giving a unique filename to each file based on the file metadata and gziping files. This is great, but you really, really, really don’t want to have your apps take care of that in production. Why wasting precious resources to serve assets when they can be prepared ahead of time. (by making Rails serve static assets, you are seriously reducing the throughput of  your app, please think of the children (or the dolphins/trees if you don’t like children))

Capistrano

Rails obviously has a preprocessor available as a rake task and you should update your deployment recipe to use that new feature. Here is my Capistrano code:

after 'deploy:update_code', 'deploy:compile_assets'
namespace :deploy do
  task :compile_assets do
    run "cd #{release_path}; RAILS_ENV=production rake assets:precompile"
  end
end

Well, my real code doesn’t hardcode the RAILS_ENV constant value, it’s in fact set in each env file, but I simplified it since most people only use 1 env outside of dev & test.

What that will do is compile all the files and dump them in public/assets/. But the file I had called bubble.png now becomes bubble-27543c671a3ab45141ee0d3216085009.png which means that my app is totally broken because images use in Bruce SCSS don’t load, my js files don’t load and the app is totally broken. Now this is least fun part, that I wish I had known before. This is where you go back and change your code so it uses magic to get the right file names.

Images

Fixing images was actually quite simple, in all my views, I just had to make sure I was using the image_tag helper everywhere.

CSS

SCSS files were a bit more tricky, I had to use the new scss preprocessor helpers you will find in the Rails guide (image_path and image_url). I first looked into using erb, but turned out it wasn’t needed and the end result is much cleaner.

Javascript/CoffeeScript

For the CoffeeScript files, I was referring to image assets in the code and of course all the links were broken. So I had to use ERB in my coffee which looked funky but it worked.

But to get that to work, you need to rename your coffee script and append erb at the end. For instance my feature.js.coffee script had to be renamed feature.js.coffee.erb. That made me cry a little inside, but oh well, at least its not a XML config file. Maybe soon we will start seeing code in filenames or filenames called my_feature.js.compressed.minified.coffee.erb.from_rails.mattetti.org Also, be careful about the order of the file extensions, otherwise it won’t work. I thought I was done, ready to deploy my apps and this time the assets will show up properly. Turns out I was wrong :(

Rails asset precompilation env specific configuration.

My css looked good, the precompiling task had run fine but I was missing some js files. I scratched my head as I could only see some of my js files. I then realized that all my JS files were there but some of my CoffeeScript files were missing. The answer was given to me by Bruce who asked me if I had updated my “config.assets.precompile” setting. Sometimes I feel that Rails is trying to compete with Struts and here I was really surprised that by default Rails, in production mode only precompiles all static JS and application.js files, but none of the other dynamic js files. Now it does precompile all the scss files, but for a reason I just don’t understand, it’s not the case for the JS files. So, you have to go edit production.rb in the config/environments folder and add the other js files you would like Rails to precompile for you.

After making all these changes, I was able to redeploy my app and everything was working again. (you might want to tweak your apache/nginx config as explained in the Rails guide)

Conclusion

Don’t be fooled like me and expect that because you have an app running locally, deployment will work right away. Make sure to read about the new features and what’s needed. Overall, I think that the asset pipeline is a nice addon to Rails and if you don’t feel like using it, just can put/leave all your files in the public folder and everything will work just like before. I do have to say that I was surprised to see that even in a brand new Rails 3.1 project, Rails isn’t running in threaded mode by default. But that’s a different (old) story and I guess people still get more excited about asset management than framework raw performance ;)

The challenge is that a lot of people using the Rails framework are used to doing page caching instead of relying on HTTP caching, even though this feature was added a long time ago. The major problem with page caching is that it doesn’t scale that well as soon as you run more than one server. Indeed you would need to store the page content to a shared drive between your servers or use memcached and do some work to avoid hitting your app every single time. On the other hand, HTTP caching is extremely easy to handle at the application level and it will dramatically reduce the amount of requests hitting your app. Let me explain a little more about HTTP caching.

Ryan Tomako wrote an excellent post about the details of caching, I strongly recommend you read it. In a nutshell, the HTTP caching layer (usually) seats before your application layer and allows you, the developer to store some responses that can be send back to the users based on optional conditions. That might still seem vague, let’s take a concrete example. If you look at https://cinematreasures.org’s home page you can see that it’s an agglomerate of various information:

And the bottom of the page contains even more dynamic data such as the popular movie theater photos, latest movie theater videos and latest tweets. One might look at that and say that this page can’t really be cached and that the caching should be done at the model layer (i.e. cache the data coming from the database). I would certainly agree that caching the data layer is probably a good idea, but you shouldn’t start by that. In fact without caching, this page renders fast enough. The problem is when someone like Roger Ebert tweets about CinemaTreasures the load on the app peaks significantly. At the point, the amount of concurrent connections your app can handle gets put to the challenge. Even though your page load is “fast enough”, requests will queue up and some will eventually time out. That’s actually a perfect case of HTTP caching.

What we want to do in that case is to cache a version of the home page in Varnish for 60 seconds. During that time, all requests coming to the site, will be served by Varnish and will all get the same cached content. That allows our servers to handle the non cached requests and therefore increase our throughput. What’s even better, is that if a user refreshes the home page in his/her browser during the first 60 seconds the requests won’t even make it all the way to our servers. All of that thanks to conditions set on the response. The first user hitting the HTTP cache layer (Varnish in this case) won’t find a fresh cached response, so varnish will forward the request to our application layer which will send back the homepage to varnish and tell Varnish that this content is good for a full minute so please don’t ask for it again until a minute from now. Varnish serves this response to the users' browser and let the browser know that the server said that the response was good enough for a minute so don’t bother asking for it again. But now, if during these 60 seconds another user comes in, he will hit Varnish and Varnish will have the cached response from the first user and because the cache is still fresh (it’s not been 60 seconds since the first request) and the cache is public, then the same response will be sent to the second user.

As you can see, the real strength of HTTP caching is the fact that it’s a conditional caching. It’s based on the request’s URL and some “flags” set in the request/response headers.

Setting these conditions in your app is actually very simple since you just need to set the response’s headers. If you are using a Ruby framework you will more than likely have access to the request object via the “request” method and you can set the headers directly like that: “response.headers[‘Cache-Control’] = ‘public, max-age=60’”. In Rails, you can actually use a helper method instead: expires_in 1.minute, :public => true.

You might have a case where you HAVE TO serve fresh content if available and can’t serve stale cached content even for a few seconds. In this case, you can rely on the Etag header value. The Etag is meant to validate the freshness of a cached response. Think of it as a signature (unique ID) that is set on the response and used by the client (or cache layer) to see if the server response has changed or not. The way it works is that the client keeps track of the Etag received for each request (attached to the cached response) and then sends it with the next requests. The HTTP layer or application sees the Etag in the request and can check if it is still valid and the content didn’t change. If that’s the case, an empty response can be sent with a special HTTP status code (304) to let know the client that the old cached value is still good to be used.  Rails has a helper called “stale?” that helps you do the Etag/last modified check and allows you to not fetch all the objects from the database by doing a cheap check on an attribute (For instance you can check the updated_at value and use that as a condition to pull an object and its relationships).

So I explain HTTP caching, I often hear people telling me: “that’s great Matt, but you know what, that won’t work for us because we have custom content that we display specifically to our users”. So in that case, you can always set the Cache-Control header to private which will only cache the response in the client’s browser and not the cache layer. That’s good to some extent, but it can definitely be improved by rethinking a bit your view layer. In most web apps, the page content is rendered by server side code (Rails, Django, node.js, PHP..) and sent to the user all prepared for him. There are a few challenges with this approach, the biggest one is that the server has to wait until everything is ready (all data fetched, view rendered etc…) before sending back a response and before the client’s browser can start rendering (there are ways to chunk the response but that’s besides the scope of this post). The other is that the same expensive content has to be calculated/rendered for two different users because you might be inserting the username of the current user at the top of the page for instance. A classic way to deal with that is often to use fragment caching, where the expensive rendering is cached and reused by different requests. That’s good but if the only reason to do that is because we are displaying some user specific data, there is a simpler way: async page rendering. The concept is extremely simple: remove all user specific content from the rendered page and then inject the user content in a second step once the page is displayed. The advantage is that now the full page can be cached in Varnish (or Squid or whatever you use for HTTP caching). To inject the user content, the easiest way is to use JavaScript.

Let’s stay on CinemaTreasures, when you’re logged in, the username is shown on the top of each page:

[caption id="" align=“aligncenter” width=“574” caption=“Once logged in, the username is displayed on all pages”][/caption]

The only things that differs from the page rendered when the user is not logged in and when he is, are these 2 links and an avatar. So let’s write some code to inject that after rendering the page.

In Rails, in the sessions controller or whatever code logs you in, you need to create a new cookie containing the username:

cookies[:username] = {
         :value => session[:username],
         :expires => 2.days.from_now,
         :domain => ".cinematreasures.org"
       }

As you can see, we don’t store the data in the session cookie and the data won’t be encrypted. You need to be careful that someone changing his cookie value can’t access data he/should shouldn’t. But that’s a different discussion. Now that the cookie is set, we can read it from JavaScript when the page is loaded.

document.observe("dom:loaded", function() {
  displayLoggedinUserLinks();
});

function readCookie(name) {
     var nameEQ = name + "=";
     var ca = document.cookie.split(';');
     for(var i=0;i < ca.length;i++) {
          var c = ca[i];
          while (c.charAt(0)==' ') c = c.substring(1,c.length);
          if (c.indexOf(nameEQ) == 0) return c.substring(nameEQ.length,c.length);
     }
     return null;
}

function displayLoggedinUserLinks() {
  var username            = readCookie('username');
  var loginLink           = $('login');
  var logout              = $('logout');
  if (username == null){
    loginLink.show();
    logout.hide();
  }else{
    // user is logged in and we have his/her username
    loginLink.hide();
    if(userGreetings){ userGreetings.update("<span id="username">username</span>"); }
    logout.show();
    showAvatar(username);
  };
  return true;
}

The code above doesn’t do much, once the DOM is loaded, the displayLoggedinUserLinks() function gets trigger. This function reads the cookie via the readCookie() function and if a username is found, the login link is hidden, the user name is displayed, as well as the logout link and the avatar. (You can also use a jQuery cookie plugin to handle the cookie, but this is an old example using Prototype, replace the code accordingly) When the user logs out, we just need to delete the username cookie and the cached page will be rendered properly. In Rails, you would do delete the cookie like that: cookies.delete(‘username’). Quite often you might even want to make an Ajax call to get some information such as the number of user messages or notifications. Using jQuery or whatever JS framework you fancy you can do that once the page is rendered. Here is an example, on this page, you can see the learderboards for MLB The Show. The leaderboards don’t change that often, especially the overall leaderboards so they can be cached for a little while, however the player’s presence can change anytime. The smart way to deal with that, would be to cache the  leaderboards for a few seconds/minutes and make an ajax call to a presence service passing it a list of user ids collected from the DOM. The service called via Ajax could also be cached  depending on the requirements.

Now there is one more problem that people using might encouter: flash notices. For those of you not familiar with Rails, flash notices are messages set in the controller and passed to the view via the session (at least last time I checked). The problem happens if I’m the home page isn’t cached anymore and I logged in which redirects me to the home page with a flash message like so:

The problem is that the message is part of the rendered page and now for 60 seconds, all people hitting the home page will get the same message. This is why you would want to write a helper that would put this message in a custom cookie that you’d pull JS and then delete once displayed. You could use a helper like that to set the cookie:

def flash_notice_cookie(msg, expiration=nil)
  cookies[:flash_notice] = {
    :value => msg,
    :expires => expiration || 1.minutes.from_now,
    :domain => ".cinematreasures.com"
   }
end

And then add a function called when the DOM is ready which loads the message and injects it in the DOM. Once the cookie read, delete it so the message isn’t displayed again.

So there you have it, if you follow these few steps, you should be able to handle easily 10x more traffic without increasing hardware or making any type of crazy code change. Before you start looking into memcached, redis, cdns or whatever, consider HTTP caching and async DOM manipulation. Finally, note that if you can’t use Varnish or Squid, you can very easily setup Rack-Cache locally and share the cache via memcached. It’s also a great way to test locally.

Update: CinemaTreasures was updated to use HTTP caching as described above. The hosting cost is now half of what it used to be and the throughput is actually higher which offers a better protection against peak traffic.

External resources:

• https://tomayko.com/writings/things-caches-do

• HTTP Caching at Heroku

• W3 caching protocol 

• Rack-Cache middleware

• Blog post covering HTTP Caching/Varnish/Rails

• jQuery cookie plugin

The Go Programming language is really cool language by Google. According to the sales pitch, it’s a “fast, statically typed, compiled language that feels like a dynamically typed, interpreted language”. Well, if you are like me, you don’t trust sales pitches because you know that people writing them don’t care about you, they care about their product. However cynical you are, you still have to check the facts. So here is a quick demonstration showing how to use Go’s reflection feature.

Installing Go is actually really straight forward on a Mac, and slightly harder on Linux, check this guide to see how to build Go in a few minutes.

Once all setup, you might want to read the documentation to see how to code in Go. Go is actually a kind of nice version of C with a simplified syntax, no header files, really fast compilation time, a garbage collector and a simple way to approach object inheritance without turning in the complicated mess C++ is. The language is designed around the concept of goroutines, a very nice way to handle concurrency. It also has some features that Rubyists, Pythonistas and Javascripters wouldn’t want to live without such as closures and some they probably wish they had such as defer. But of the things we are used to with dynamic languages is the concept of reflection. In a nutshell, at runtime, your code can reflect on the type of a given object and let the developer act accordingly. Depending on your programming background that might be obvious or you might not see the value. To be honest, that’s not the question here. What I’m interested in showing you is how it works.

For the sake of this demo, let’s pretend we want to have a “Dish” data model, each instance of the “Dish” type will have a few attributes, an id, a name, an origin and a custom query which really is a function that we store as an attribute. Here is how we would represent that model in Go:

// Data Model
type Dish struct {
  Id  int
  Name string
  Origin string
  Query func()
}

This is more or less the equivalent of the following Ruby code:

class Dish
  attr_accessor :id, :name, :origin, :query
end

Ruby works slightly differently in the sense that defining attribute accessors create getters and setter methods but doesn’t technically create instance variables until they are used. Here is what I mean:

shabushabu = Dish.new
shabushabu.instance_variables # => []
shabushabu.name = "Shabu-Shabu"
shabushabu.instance_variables # => ["@name"]
shabushabu.origin = "Japan"
shabushabu.instance_variables # => ["@name", "@origin"]

Another way of checking on the accessors is to check the methods defined on the object:

shabushabu.methods - Object.new.methods
=> ["name", "name=", "origin", "origin=", "id=", "query", "query="]

But anyway, this post isn’t about Ruby, it’s about Go and what we would like is to reflect on an object of “Dish” type and see its attributes. The good news is that the Go language ships with a package to do just that. Here is the full implementation:

package main

import(
  "fmt"
  "reflect"
)

func main(){
  // iterate through the attributes of a Data Model instance
  for name, mtype := range attributes(&Dish;{}) {
    fmt.Printf("Name: %s, Type %s\n", name, mtype.Name())
  }
}

// Data Model
type Dish struct {
  Id  int
  Name string
  Origin string
  Query func()
}

// Example of how to use Go's reflection
// Print the attributes of a Data Model
func attributes(m interface{}) (map[string]reflect.Type) {
  typ := reflect.TypeOf(m)
  // if a pointer to a struct is passed, get the type of the dereferenced object
  if typ.Kind() == reflect.Ptr{
    typ = typ.Elem()
  }

  // create an attribute data structure as a map of types keyed by a string.
  attrs := make(map[string]reflect.Type)
  // Only structs are supported so return an empty result if the passed object
  // isn't a struct
  if typ.Kind() != reflect.Struct {
    fmt.Printf("%v type can't have attributes inspected\n", typ.Kind())
    return attrs
  }

  // loop through the struct's fields and set the map
  for i := 0; i < typ.NumField(); i++ {
    p := typ.Field(i)
      if !p.Anonymous {
        attrs[p.Name] = p.Type
      }
     }

  return attrs
}

Unfortunately, my code highlighter doesn’t support the Go syntax, but GitHub does, so here is a pretty version.

There are ways of running Go source code like Ruby or Python scripts but in this case, we’ll use the compiler & linker provided with Go. I named my source file “example.go”, and here is how I compiled, linked and run it:

$ 6g example.go && 6l example.6 && ./6.out
Name: Origin, Type string
Name: Id, Type int
Name: Query, Type
Name: Name, Type string

As you can see each attribute is printed out with its name and type. The code might seem a bit odd if you never looked at Go before. Here is a quick rundown of the code:

In our main function, we create a new instance of type Dish on which we call attributes on. The call returns a map on which we iterate through and print the attribute name (key) and type (value). The attributes function is defined a bit below and and it takes any type of objects (empty interface) and returns a map, which is like a Hash or a Dictionary. The map has keys of String type and values of “Type” type. The “Type” type is defined in the reflect package. Inside the function, 23 then use the previously mentioned reflect package to check on the type and the name of each attribute and assign it to a map object. (note that I’m explicitly returning the map, but I could have done it in a more implicit way)

So there you go, that’s how you use reflection in Go. Pretty nifty and simple.

I was planning on posting this a bit later but since I was politely escorted out of the building by HR/security, I have more free time to let you know of my decision. Before you ask: No, my decision isn’t directly related to the recent PSN/Sony security breach events and no, I don’t have your credit card number. More seriously, my decision boils down to something much simpler & concrete: drive and passion.

The concept of drive is very well explained in this talk by Dan Pink and illustrated by RSA:

As explained, it is proven that when doing cognitive tasks, there are 3 factors that lead to better performance & personal satisfaction:

• Autonomy: engagement vs compliance

• Mastery: get better at stuff

• Purpose: be disruptive but make the world a better place

The challenge is that when you work for a big corporation, you rarely see these factors applied. The amount of red tape, management overhead, lack of recognition and accountability result in a low drive by most employees. I think it was the first time in my entire career that I was told by a manager to care less about the quality and end result of our products.

The second important concept that I think is critical when looking at your career is passion. This topic is very well covered in Chad Fowler’s book: The Passionate Programmer.

Here is a quote from Chad’s book:

Fulfillment and happiness don’t (often) come by chance. They require thought, intention, action, and a willingness to change course when you’ve made mistakes. […] It might be a technology or business domain that gets you excited. Or, on the other hand, it might be a specific technology or business domain that drags you down. Or a type of organization. Maybe you’re meant for small teams or big teams. Or rigid processes. Or agile processes. Whatever the mix, take some time to find yours. You can fake it for a while, but a lack of passion will catch up with you and your work.

It’s hard to summarize Chad’s book into just a few sentences, but what I got from his book is that if, for whatever reason, you lose your passion for your job, you should move on to another place where you can be passionate and excel. In my case, I’m still very much passionate about video game development but I find my passion seriously affected by an unhealthy work environment, bad communication and a lack of desire to change things in concrete ways. As the saying goes, “People don’t leave jobs…"

What’s next for me?

If you are in the software industry you know that everybody is hiring and that there is a real shortage of talent out there. You probably also receive half a dozen emails per week from recruiters offering you “the best job ever” with an obscene salary. Well, I receive them too. But in this market, you and I can allow ourselves to be picky and to choose the right job for ourselves. By choosing the right job, we are going to be more passionate, more driven, more efficient and bring a lot more value to our employers, who, in return, will hopefully do everything they can to make sure we grow within the company with a strong desire to do even better. As I was considering what I would want to do if I were to leave Sony, I came up with a few ideas about a job:

• I don’t want to live to work, but rather, work to live. (kind of an European cliché sentence)

• It’s not about the job title.

• It’s not about the pay check.

• It’s not about how glamourous an industry  is.

• What matters is :

  • the company culture.

  • the passion coming from the leaders.

  • the company’s ambitions.

  • how the company rewards and respects its employees.

  • the autonomy/trust given to the employees.

  • the purpose of the company and its potential to disrupt a market/change the world.

  • personal growth within the company.

Writing this list helped me realize that I was definitely not in the right place and helped me create a list of criteria to define what kind of job I should be doing instead. As I was thinking about all that, I was reminded of this quote:

“One should not pursue goals that are easily achieved. One must develop an instinct for what one can just barely achieve through one’s greatest efforts.” —Albert Einstein

This is probably a personality trait, but I like/need to be at the edge of my confort zone. I need to learn new things, experience new challenges. I need to try to solve unsolved problems. So part of me needs a company with a rich culture, a great a vision and leadership, but another part also needs to be allowed to think creatively, to push the existing boundaries, to challenge myself and to try to achieve things through my greatest efforts.

The challenge is that over the last 5-7 years I have become a Ruby specialist. I have learned to understand and master the language, learning its pros/cons and what goes on “under the hood”. I have built small and huge solutions for various domain spaces on top of Ruby. I have shared my knowledge giving talks, books, blog posts. I have also spent a lot of time expanding my own computer science knowledge, looking into other programming languages, frameworks, designs. My job at Sony was interesting due to the fact that Ruby is used in a quite singular way with very specific problems and a lot of C/C++ interaction. So while it is true that I am a Ruby specialist, I don’t like that label. I don’t like it because it’s very limiting. My goal is to solve problems and quite often Ruby is a great tool for that, but for some problems, it is not. So I need a job where my expertise is helpful but would not limit my desire to learn new approaches, languages and skills.

And this is why I will soon start working as a Code Alchemist in the LivingSocial R&D’s lab.

That might be a shocker at first glance. Going from working on video games to working for a daily-deal startup? It doesn’t seem like a very smart career move. To be honest, that was my first reaction. But LivingSocial’s VP of engineering is Chad Fowler (the author of The Passionate Programmer, mentioned earlier) and its VP of R&D is Rich Kilmer, InfoEther’s cofounder and recognized mad-scientist-coder. I’ve known Chad and Rich for many years, meeting at tech conferences that they organized or were invited to give talks to, and even getting to hack on some projects with Rich. So when they approached me to join them at LivingSocial, I wanted to know more about their own motivations, why I would be a good fit and how the company/job would rate against my list of criteria. We had long and honest discussions and the result is that I am excited to soon be working with one of the best teams I know of. More concretely, I will be working with Rich, Bruce and Michael in helping LivingSocial revolutionize the local market. LivingSocial is a fast growing and successful company with a strong focus on creating a great experience for both customers and merchants. The company’s vision is well defined and the desire to change the world the way we know it is palpable. Most of the founders have a technical background, they value good engineering practices and have created a nice, positive company culture. Chad has some really awesome, passionate and talented engineers on his team (too many to mention), for whom I have a lot of respect and with whom I look forward to working with.  I also value the fact that LivingSocial trusts and values me enough to let me work remotely. This is very important for me because it shows that my new employer really wants to work with me, trusts me but also is willing to embrace the challenges of having a remote team if that’s what’s needed to create the “right team”. After looking more deeply at what I need and what LivingSocial is, I believe that I can assist LivingSocial in making a very positive change to the way small businesses around the world are run. And I am looking forward to this.

Problem Domain

The video game industry is quite special, to say the least. It shares a lot of similarities with the movie industry. The big difference is that the movie industry hasn’t evolved as quickly as the video game industry has. But the concept is the same, someone comes up with a great idea, finds a team/studio to develop the game and finds a publisher. The development length and budget depends on the type of game, but for a AAA console game, it usually takes a least a few million and a minimum of a year of work once the project has received the green light. The creation of such a game involves various teams, designers, artists, animators, audio teams, developers, producers, QA, marketing, management/overhead etc.. Once the game gets released, players purchase the whole game for a one time fee and the studio moves on to their next game. Of course things are not that simple, with the latest platforms, we now have the option to patch games, add DLC etc.. But historically, a console game is considered done when it ships, exactly like a movie, and very little work is scheduled post release.

Concretely such an approach exposes a few challenges when trying to implement online features for a AAA console title:

• Communication with the game client network team

• Scalability, performance

• Insane deadlines, unstable design (constant change of requirements)

• Can’t afford to keep on working on the system once released (time delimited projects)

Communication

As in most situations, communication is one of the biggest challenges. Communication is even harder in the video game industry since you have so many teams and experts involved. Each team speaks its own jargon, has its own expertise and its own deadlines. But all focus on the same goal: releasing the best game ever. The team I’m part of has implementing online features as its goal. That’s the way we bring business value to our titles. Concretely, that means that we provide the game client developers with a C++ SDK which connects to custom web APIs written in Ruby. The API implementations rely on various data stores (MySQL, Redis, Memcached, memory) to store and retrieve all sorts of game data.

Nobody but our team should care about the implementation details, after all, the whole point of providing an API is to provide a simple interface so others can do their part of the job in the easiest way possible. This is exactly where communication becomes a problem. The design of these APIs should be the result of the work of two teams with two different domains of expertise and different concerns. One team focuses on client performance, memory optimization and making the online resources available to the game engine without affecting the game play. The other, focuses on server performance, latency, scalability, data storage and system contention under load. Both groups have to come together to find a compromise making each other’s job doable. Unfortunately, things are not that simple and game designers (who are usually not technical people) have a hard time not changing their designs and requirements every other week (usually for good reasons) making API design challenging and creating tension between the teams.

From this perspective, the API is the most important deliverable for our team and it should communicate the design goal while being very explicit about how it works, why it works the way it does, and how to implement it client side. This is a very good place where we can improve communication by making sure that we focus on making clear, well designed, well documented, flexible APIs.

Scalability, performance

On the server side, the APIs need to perform and scale to handles tends of thousands of concurrent requests. Web developers often rely on aggressive HTTP caching but in our case, the web client (our SDK) has a limited amount of memory available and 90% of the requests are user specific (can’t use full page HTTP cache) and a lot of these are POST/DELETE requests (can’t be cached). That means that, to scale, we have to focus on what most developers don’t often have to worry too much about: all the small details which, put together with a high load, end up drastically affecting your performance.

While Ruby is a great language, a lot of the libraries and frameworks are not optimized for performance, at least not the type of performance needed for our use case. However, the good news is that this is easily fixable and many alternatives exist (lots of async, non-blocking drivers for i.e). When obsessed with performance, you quickly learn to properly load test, profile, and monitor your code to find the bottlenecks and the places where you should focus your attention. The big, unique challenge though, is that a console game will more than likely see its peak traffic in the first few weeks, not really giving the chance to the online team to iteratively handle the prod issues. The only solution is to do everything possible before going live to ensure that the system will perform as expected. Of course if we were to write the same services in a more performant language, we would need to spend less time optimizing. But we are gaining so much flexibility by using a higher level programming language that, in my mind, the trade off is totally worth it (plus you still need to spend a lot of time optimizing your code path, even if your code is written in a very fast language).

Deadlines, requirement changes

That’s just part of the way the industry works. Unless you work for Blizzard and you can afford to spend a crazy amount of time and money on the development of a title; you will have to deal with sliding deadlines, requirement changes, scope changes etc… The only way I know how to protect myself from such things is to plan for the worst. Being a non-idealistic (read pessimistic) person helps a lot. When you design your software, make sure your design is sound but flexible enough to handle any major change that you know could happen at any time. Pick your battles and make sure your assumptions are properly thought through, communicated and documented so others understand and accept them. In a nutshell, this is a problem we can’t avoid, so you need to embrace it.

Limited reusability

This topic has a lot to do with the previous paragraph. Because scopes can change often and because the deadlines are often crazy, a lot of the time, engineers don’t take the time to think about reusability. They slap some code together, pray to the lords of Kobol and hope that they won’t have to look at their code ever again (I’m guilty of having done that too). The result is a lot of throw away code. This is actually quite frequent and normal in our industry. But it doesn’t mean that it the right thing to do! The assumption/myth is that each game is different and therefore two games can’t be using the same tech solution. My take on that is that it’s partly true. But some components are the same for 80% of the games I work on. So why not design them well and reuse the common parts? (A lot of games share the same engines, such as Unreal for example, and there is no reason why we can’t build a core online engine extended for each title)

My approach

When I joined Sony, I had limited experience with the console video game industry and my experience was not even related to online gaming. So even though I had (strong) opinions (and was often quite (perhaps even too) vocal about them), I did my best to improve existing components and work with the existing system. During that time, the team shipped 4 AAA titles on the existing system. As we were going through the game cycles, I did my best to understand the problem domain, the reasons behind some of the design decisions and finally I looked at what could be done differently to improve our business value. After releasing a title with some serious technical difficulties, I spent some time analyzing and listing the problems we had and their root causes. I asked our senior director for a mission statement and we got the team together to define the desiderata/objectives of our base technology. Here is what we came up with:

1. Stability

2. Performance / Scalability

3. Encapsulation / Modularity

4. Documentation

5. Conventions

6. Reusability / Maintainability

These objectives are meant to help us objectively evaluate two options. The legacy solution was based on Rails, or more accurately: Rails was used in the legacy solution. Rails had been hacked in so many different ways that it was really hard to update anything without breaking random parts of the framework. The way to do basic things kept being changed, there was no consistent design, no entry points, no conventions and each new game would duplicate the source code of the previously released game and make the game specific changes. Patches were hard to back port and older titles were often not patched up. The performance was atrocious under load, mainly due to hacked-up Rails not performing well. (Rails was allocating so many objects per request that the GC was taking a huge amount of the request cycles, the default XML builder also created a ton load of objects etc…) This was your typical broken windows scenario. Engineers were getting frustrated, motivation was fainting, bugs were piling up and nobody felt ownership over the tech.

Now, to be fair, it is important to explain that the legacy system was hacked up together due to lack of time, lack of resources and a lot of pressure to release something ASAP. So, while the end result sounds bad, the context is very important to note. This is quite common in software engineering and when you get there, the goal is not to point fingers but to identify the good and the bad parts of the original solution. You then use this info to decide what to do: fix the existing system or rewrite, porting the good parts.

Our report also came up with a plan. A plan to redesign our technology stack to match the desiderata previously mentioned. To put it simply, the plan was to write a new custom web framework focusing on stability, performance, modularity and documentation. Now, there are frameworks out there which already do that or value these principles. But none of them focus on web APIs and none of them are specific to game development. Finally, the other issue was that we had invested a lot of time on game specific code and we couldn’t throw away all that work, so the new framework had to support a good chunk of legacy code but had to make it run much faster.

Design choices

Low conversion cost

Using node.js & coffee script/Scala/whatever new fancy tech was not really an option. We have a bunch of games out there which are running on the old system and some of these games will have a sequel or a game close enough that we could reuse part of the work. We don’t want to have to rewrite the existing code. I therefore made sure that we could reuse 90% of the business logic by adding an abstraction layer doing the heavy lifting at boot time and therefore not affecting the runtime performance. Simple conversion scripts were also written to import the core of the existing code over.

Lessons learned: It is very tempting to just redo everything and start from scratch. However, the business logic implementation wasn’t the main cause of our problems. Even though I wish we could have redesigned that piece of the puzzle, it didn’t make sense from a business perspective. A lot of thought had to be put into how to obtain the expected performance level while keeping the optional model/controller/view combos. By having full control of the “web engine”, we managed to isolate things properly without breaking the old paradigms. We also got rid of a lot of assumptions allowing us to design new titles a bit differently while being backward compatible and have our code run dramatically faster.

Web API centric

This is probably the most important design element. If I had to summarize what our system does in just a few words, I would say: a game web API. Of course, it’s much more than that. We have admin interfaces, producer dashboards, community websites, lobbies, p2p, BI reports, async processing jobs etc… But at the end of the day, the only one piece you can’t remove is the game web API. So I really wanted the design to focus on that aspect. When a developer starts implementing a new online game feature, I want him/her to think about the API. But I also want this API to be extremely well documented so the developer working client-side understands the purpose of the API, how to use it, and what the expected response is right away. I also wanted to be able to automatically test our APIs at a very basic level so we could validate that there are discrepancies between what the client expects and what the server provides. To do that, I created a standalone API DSL with everything needed to describe your API but without any implementation details whatsoever. The API DSL lets the developer define a route (url), the HTTP verb expected, if the request should be authenticated or not, SSL or not, the param rules, default values and finally a response description (which was quite a controversial choice). All of these settings can be documented by the developer. This standalone DSL can then be consumed by different tools. For instance we have a tool extracting all the info into nicely formatted HTML doc for the game client developers. This tool doesn’t need to load the framework to just render the documentation. We also use this description at boot time to compile the validation rules and routes, allowing for a much faster request dispatch. And we also use these API description to generate some low level data for the client. Finally, we used the service description DSL to help create mocked service responses allowing the client team to test service designs without having to wait for the implementation streamlining the process.

Lessons learned: We had a lot of internal discussions about the need to define the response within the service description. Some argued that it’s a duplication since we already had a view and we could parse that to get most of what we needed (which is what the old system was doing). We ended up going with the response description DSL for a few critical reasons: testing and implementation simplicity. Testing: we need to have an API expectation reference and to keep this reference sane so we can see if something is changed. If we were to magically parse the response, we couldn’t test the view part of the code against a frame of reference. Implementation simplicity: magically parsing a view template is more tricky that it sounds, you would need to render the template with the right data to make it work properly. Furthermore, you can’t document a response easily in the view, and if you do, you arguably break the separation of concern between the description and the implementation. Finally, generated documentation isn’t enough and that’s why we decided to write English documentation, some being close to the code and some being just good old documentation explaining things outside of the code context.

Modularity

In order to make our code reusable we had to isolate each component and limit the dependencies. We wrote a very simple extension layer allowing each extension to registers itself once detected. The extension interface exposes the path of the extension, its type, models, services, controllers, migrations, seed data, dependencies etc.. Each extension is contained in a folder. (The extension location doesn’t matter much but as part of the framework boot sequence, we check a few default places.) The second step of the process is to check a manifest/config file that is specific to each title. The manifest file lists the extensions that should be activated for the title. The framework then activates the marked extensions and has access to libs, models, views, migrations, seed data and of course to load services (DSL mentioned earlier) etc…

Even though we designed the core extensions the best we could, there are cases where some titles will need to extend these extensions. To do that, we added a bunch of hooks that could be implemented on the title side if needed (Ruby makes that super easy and clean to do!). A good example of that is the login sequence or the player data.

Lessons learned: The challenge with modularity is to keep things simple and highly performing yet flexible. A key element to manage that is to stay as consistent as possible. Don’t implement hooks three different ways, try to keep method signatures consistent, keep it simple and organized.

Conclusion

It’s a bit early to say if this rewrite is a success or not and there are still lots of optimizations and technology improvements we are looking forward to doing. Only time will give us enough retrospect to evaluate our work. But because we defined the business value (mission statement) and the technical objectives, it is safe to say that the new framework meets the expectations quite well. On an early benchmark we noted a 10X speed improvement and that’s before drilling into the performance optimizations such as making all the calls non-blocking, using better connection pools, cache write through layer… However, there is still one thing that we will have to monitor: how much business value will this framework generate. And I guess that’s where we failed to define an agreed upon evaluation grid. I presume that if our developers spend more time designing and implementing APIs and less time debugging that could be considered business value. If we spend less time maintaining or fighting with the game engine, that would also be a win. Finally, if the player experience is improved we will be able to definitely say that we made the right choice.

To conclude, I’d like to highlight my main short coming: I failed to define metrics that would help us evaluate the real business value added to our products. What I consider a technical success might not be a business success. How do you, in your own domain, find ways to define clear and objective metrics?

MacRuby is so awesome that Apple is even using it in its upcoming OS. The only problem is that Apple apparently decided to not share MacRuby with other OS X developers and put MacRuby in the OS private frameworks. While this doesn’t affect the project itself, it does affect OS X developers like myself who can’t link to Lion’s private MacRuby framework and are forced to embed MacRuby with their applications.

That’s why I have opened a ticket on Apple radar system to ask that MacRuby be made a public framework.

If you also want Apple to make this change, please take a minute and let them know.

First off, if you are not really familiar with concurrency, take a minute to read the wikipedia article on the topic which is a great recap on the subject. But now, you should have noticed that my above example was more about parallel programming than concurrency, but we’ll come back to that in a minute.

The real question at the heart of the quest for concurrency is: “how to increase code throughput”.

We want our code to perform better, and we want it to do more in less time. Let’s take two simple and concrete examples to illustrate concurrency. First, let’s pretend you are writing a twitter client, you probably want to let the user scroll his/her tweets while the latest updates are  being fetched. In other words, you don’t want to block the main loop and interrupt the user interaction while your code is waiting for a response from the Twitter API. To do that, a common solution is to use multiple threads. Threads are basically processes that run in the same memory context. We would be using one thread for the main event loop and another thread to process the remote API request. Both threads share the same memory context so once the Twitter API thread is done fetching the data it can update the display. Thankfully, this is usually transparently handled by asynchronous APIs (provided by the OS or the programming language std lib) which avoid blocking the main thread.

The second example is a webserver. Let’s say you want to run a Rails application. Because you are awesome, you expect to see a lot of traffic. Probably more than 1 QPS (query/request per second). You benchmarked your application and you know that the average response time is approximately 100ms. Your Rails app can therefore handle 10QPS using a single process (you can do 10 queries at 100ms in a second).

But what happens if your application gets more than 10 requests per second? Well, it’s simple, the requests will be backed up and will take longer until some start timing out. This is why you want to improve your concurrency. There are different ways to do that, a lot of people feel really strong about these different solutions but they often forget to explain why they dislike one solution or prefer one over the other. You might have heard people conclusions which are often one of these: Rails can’t scale, you only get concurrency with JRuby, threads suck, the only way to concurrency is via threads, we should switch to Erlang/Node.js/Scala, use fibers and you will be fine, add more machines, forking > threading.  Depending on who said what and how often you heard it on twitter, conferences, blog posts, you might start believing what others are saying. But do you really understand why people are saying that and are you sure they are right?

The truth is that this is a complicated matter. The good news is that it’s not THAT complicated!

The thing to keep in mind is that the concurrency models are often defined by the programming language you use. In the case of Java, threading is the usual solution, if you want your Java app to be more concurrent, just run every single request in its own thread and you will be fine (kinda). In PHP, you simply don’t have threads, instead you will start a new process per request. Both have pros and cons, the advantage of the Java threaded approach is that the memory is shared between the threads so you are saving in memory (and startup time), each thread can easily talk to each other via the shared memory. The advantage of PHP is that you don’t have to worry about locks, deadlocks, threadsafe code and all that mess hidden behind threads. Described like that it looks pretty simple, but you might wonder why PHP doesn’t have threads and why Java developers don’t prefer starting multiple processes. The answer is probably related to the language design decisions. PHP is a language designed for the web and for short lived processes. PHP code should be fast to load and not use too much memory. Java code is slower to boot and to warm up, it usually uses quite a lot of memory. Finally, Java is a general purpose programming language not designed primarily for the internet. Others programming languages like Erlang and Scala use a third approach: the actor model. The actor model is somewhat a bit of a mix of both solutions, the difference is that actors are a like threads which don’t share the same memory context. Communication between actors is done via exchanged messages ensuring that each actor handles its own state and therefore avoiding corrupt data (two threads can modify the same data at the same time, but an actor can’t receive two messages at the exact same time). We’ll talk about that design pattern later on, so don’t worry if you are confused.

What about Ruby? Should Ruby developers use threads, multiple processes, actors, something else? The answer is: yes!

Threads

Since version 1.9, Ruby has native threads (before that green threads were used). So in theory, if we would like to, we should be able to use threads everywhere like most Java developers do. Well, that’s almost true, the problem is that Ruby, like Python uses a Global Interpreter Lock (aka GIL). This GIL is a locking mechanism that is meant to protect your data integrity. The GIL only allows data to be modified by one thread at time and therefore doesn’t let threads corrupt data but also it doesn’t allow them to truly run concurrently. That is why some people say that Ruby and Python are not capable of (true) concurrency.

However these people often don’t mention that the GIL makes single threaded programs faster, that multi-threaded programs are much easier to develop since the data structures are safe and finally that a lot of C extensions are not thread safe and without the GIL, these C extensions don’t behave properly. These arguments don’t convince everyone and that’s why you will hear some people say you should look at another Ruby implementation without a GIL, such as JRuby, Rubinius (hydra branch) or MacRuby (Rubinius & MacRuby also offer other concurrency approaches). If you are using an implementation without a GIL, then using threads in Ruby has exactly the same pros/cons than doing so in Java. However, it means that now you have to deal with the nightmare of threads: making sure your data is safe, doesn’t deadlock, check that your code, your libs, plugins and gems are thread safe. Also, running too many threads might affect the performance because your OS doesn’t have enough resources to allocate and it ends up spending its time context switching. It’s up to you to see if it’s worth it for your project.

Multiple processes & forking

That’s the most commonly used solution to gain concurrency when using Ruby and Python. Because the default language implementation isn’t capable of true concurrency or because you want to avoid the challenges of thread programming, you might want to just start more processes. That’s really easy as long as you don’t want to share states between running processes. If you wanted to do so, you would need to use DRb, a message bus like RabbitMQ, or a shared data store like memcached or a DB. The caveat is that you now need to use a LOT more memory. If want to run 5 Rails processes and your app uses 100Mb you will now need 500Mb, ouch that’s a lot of memory! That is exactly what happens when you use a Rails webserver like Mongrel. Now some other servers like Passenger and Unicorn found a workaround, they rely on unix forking. The advantage of forking in an unix environment implementing the copy-on-write semantics is that we create a new copy of the main process but they both “share” the same physical memory. However, each process can modify its own memory without affecting the other processes. So now, Passenger can load your 100Mb Rails app in a process, then fork this process 5 times and the total footprint will be just a bit more than 100Mb and you can now handle 5X more concurrent requests. Note that if you are allocating memory in your request processing code (read controller/view) your overall memory will grow but you can still run many more processes before running out of memory. This approach is appealing because really easy and pretty safe. If a forked process acts up or leaks memory, just destroy it and create a new fork from the master process. Note that this approach is also used in Resque, the async job processing solution by GitHub.

This solution works well if you want to duplicate a full process like a webserver, however it gets less interesting when you just want to execute some code “in the background”. Resque took this approach because by nature async jobs can yield weird results, leak memory or hang. Dealing with forks allows for an external control of the processes and the cost of the fork isn’t a big deal since we are already in an async processing approach.

Actors/Fibers

Earlier we talked a bit about the actor model. Since Ruby 1.9, developers now have access to a new type of “lightweight” threads called Fibers. Fibers are not actors and Ruby doesn’t have a native Actor model implementation but some people wrote some actor libs on top of fibers. A fiber is like a simplified thread which isn’t scheduled by the VM but by the programmer. Fibers are like blocks which can be paused and resumed from the outside of from within themselves. Fibers are faster and use less memory than threads as demonstrated in this blog post. However, because of the GIL, you still cannot truly run more than one concurrent fiber by thread and if you want to use multiple CPU cores, you will need to run fibers within more than one thread. So how do fibers help with concurrency? The answer is that they are part of a bigger solution. Fiber allow developers to manually control the scheduling of “concurrent” code but also to have the code within the fiber to auto schedule itself. That’s pretty big because now you can wrap an incoming web request in its own fiber and tell it to send a response back when it’s done doing its things. In the meantime, you can move on the to next incoming request. Whenever a request within a fiber is done, it will automatically resume itself and be returned. Sounds great right? Well, the only problem is that if you are doing any type of blocking IO in a fiber, the entire thread is blocked and the other fibers aren’t running. Blocking operations are operations like database/memcached queries, http requests… basically things you are probably triggering from your controllers. The good news is that the “only” problem to fix now is to avoid blocking IOs. Let’s see how to do that.

Non blocking IOs/Reactor pattern.

The reactor pattern is quite simple to understand really. The heavy work of making blocking IO calls is delegated to an external service (reactor) which can receive concurrent requests. The service handler (reactor) is given callback methods to trigger asynchronously based on the type of response received. Let me take a limited analogy to hopefully explain the design better. It’s a bit like if you were asking someone a hard question, the person will take a while to reply but his/her reply will make you decide if you raise a flag or not. You have two options, or you choose to wait for the response and decide to raise the flag based on the response, or your flag logic is already defined and you tell the person what to do based on their answer and move on without having to worry about waiting for the answer. The second approach is exactly what the reactor pattern is. It’s obviously slightly more complicated but the key concept is that it allows your code to define methods/blocks to be called based on the response which will come later on.

In the case of a single threaded webserver that’s quite important. When a request comes in and your code makes a DB query, you are blocking any other requests from being processed. To avoid that, we could wrap our request in a fiber, trigger an async DB call and pause the fiber so another request can get processed as we are waiting for the DB. Once the DB query comes back, it wakes up the fiber it was trigger from, which then sends the response back to the client. Technically, the server can still only send one response at a time, but now fibers can run in parallel and don’t block the main tread by doing blocking IOs (since it’s done by the reactor).

This is the approach used by Twisted, EventMachine and Node.js. Ruby developers can use EventMachine or an EventMachine based webserver like Thin as well as EM clients/drivers to make non blocking async calls. Mix that with some Fiber love and you get Ruby concurrency. Be careful though, using Thin, non blocking drivers and Rails in threadsafe mode doesn’t mean you are doing concurrent requests. Thin/EM only use one thread and you need to let it know that it’s ok to handle the next request as we are waiting. This is done by deferring the response and let the reactor know about it.