Tribune DataViz

Matters of interest, from the data reporters and developers across Tribune Publishing

Archive for the ‘Uncategorized’ Category

Side Conversations, or Best Listicle Ever

leave a comment »

Distraction is 9/10ths of the Internet, right? Liberté, fraternité and doge. So of course, it’s nice that text-editors ship with “distraction free-modes.” It’s nice that certain kinds of technology problems command the kind of focus that could block out a hurricane.

And yet. Talking about what’s going on in the world is part of journalism. It’s a requirement. An uninformed journalist is a very, very hungry one. “Hey, did you hear about this, what do you think of that, here’s the link.” It’s the water we swim in.

And to be honest, these two aspects of work on the News Apps team really complement each other. We solve problems and carry on conversations through our computers as a way of life, taking breaks from one to be rejuvenated by the other.

So, rather than be selfish and NOT share with you some of the wonderful things we’ve discussed on the side, I’ve decided to use this post for the greater good of all, and present them below. As a listicle. A listicle for good. Not evil.

1. “Hey, so, all of you have minds. If you enjoy exploding them while learning new things and staring at pictures, boy do I have a link for you.”

2. “I’m going to just ask her, it’s a data visualization of conservative and liberal hashtags on Twitter and how separated the two sides are – almost no overlap.”

3. “Until I meet a wysiwyg editor that can do everything I can do by hand, I’m not going to believe you.”


5. “‘We also can not take your money – money that you earned with your brain, knowing that taxes will be used to provide weapon for criminal group that will use it against you or to build prisons where you will be held for stating your civic position.'” -posted by the PyCon Ukraine organizers

6. “‘Want your name removed? It’s all public information, and we didn’t need your permission to build a website and put public information on it, but we do have a policy.'”

7. “Actually, here’s a way to make it without sous-vide (and wow, those brussels sprouts sound amazing):”

8. “Thanks to moving, I now know how to dispose of leftover latex paint safely.”

Chicago Garbage Etiquette

9. “I never want to walk like a regular human being again.”

10. Needs no explanation.

And, finally, this one goes to 11.

And now, back to work.


Written by jenlindner

April 14, 2014 at 9:54 am

Posted in Uncategorized

From weekend hack to actual project

leave a comment »

Sometimes, you get an idea so good you can’t wait for Monday to code it up and show your friends. (If you’re reading this, you have weird friends.) You tell yourself the idea is simple, pretty straightforward to implement and you have some time on Sunday afternoon for a fun project. Maybe you just want to get a prototype built to explore the idea, confirm it makes sense, demo it for your boss and team members. After building a few of these, you find yourself repeating common mistakes, coding yourself into the same corners over and over again. But it doesn’t have to be that way.

If you’re like me, your hard drive is littered with folders bearing now-obscure names, the tombstones of weekend hack projects that never came to fruition. Which is fine: most ideas are bad and most weekend hacks won’t go anywhere. But this post is for the ones that make it past the haze of beer and sleep deprivation to actually become something real. I have two pieces of advice, seemingly contradictory, that will make your projects better and help you enjoy them more.

The first and most important thing I’ve learned is that it’s always worth doing it right the first time. The old carpenters’ adage applies to the craft of software: measure twice, cut once. By writing code correctly when you first get started, you make it so much easier to maintain later. Generalize that set of lines you copy-pasted 3 times with slightly different variable names! Decompose that hundred-line function into smaller pieces you can reuse elsewhere! Take five seconds and write a comment explaining why that magic variable has to be set to 11 when the code first runs – you’ll be glad you did when you come back to it weeks later while tracking down a bug.

The reality is that not only does it rarely take that long to write clean code, it only gets easier and faster the more you do it. If you’re working on a passion project over the weekend, use the time to instill good habits that you’ll be happy to have later on. I know it’s easy to just copy-paste an answer from StackOverflow and be done with it, but take an extra few seconds – that’s really all it’ll take – to clean it up, harmonize the variable names and indentation levels and write a comment or two. The amount of time you’ll spend reading this code looking for bugs probably dwarfs the amount of time it takes to tap out a 10-word comment, and that comment could be a lifesaver later on.

That said, don’t worry about the boilerplate. When you have an idea and can’t wait to get started, it’s easy to lose momentum by dragging yourself through the muck of creating the right folders, installing libraries you won’t need until way later, and getting hung up on command-line syntax.

If you’re working on a weekend, or for fun, you have the freedom to explore what interests you. You’re not required to eat your vegetables before you get to dessert; it’s a Sunday! Start with the cake, and eat the carrots if there’s time.

Let’s say you want to learn the Twitter API, and you have an idea for a bot that tweets rugby scores. You can start by just trying to write code that sends a tweet. Then focus on the gritty scraper you’ve got to write in order to get the rugby data. That scraper is gonna be a huge pain, filled with corner cases and unexpected gotchas, and generally not that much fun to write. You might think you need it first – your tweets need data! – but spending an hour sucking down data and parsing it before you even get to the thing you actually wanted to do in the first place is just going to demoralize you. The energy you had when you first got started is valuable; don’t waste it on boilerplate code.

Those two points may seem contradictory, but they really boil down to the same thing: don’t be afraid to take a step back and prioritize. What are you trying to accomplish and what will help you keep your momentum going? Some tradeoffs are worth it – take the five seconds to document your weirdest code – but some tradeoffs don’t make sense. That scraper can wait.

Finally, don’t try to get it all done at once. Know how to tell when you’re at a good stopping point: when you’ve captured the basic idea you were trying to implement; when you’re about to start going down a rabbit hole of documentation for a new library; or when you just need to take care of a bunch of special one-off situations that you think won’t take that long, but inevitably suck up endless hours. Write up a bunch of notes to yourself in the comments, try to transcribe everything that’s currently stuck in your head, make explicit all the next steps you’ve got mentally planned out. You won’t remember them in the morning, because now the sun is coming up and you’re going to step back, go for a walk and try not to get burned out. You still have to show your coworkers the beautiful demo that, mere hours ago, was just a gleam in your eye.


Written by Abe Epton

April 7, 2014 at 9:37 am

Posted in Uncategorized

Note to self: Take some notes

leave a comment »

Don’t let the stereotypically laid-back atmosphere at technology firms fool you, because being a web developer is stupidly hard work. At least it is for us normal folks who have problems with abstract concepts like fundamental math and remembering hotkeys.

In fact, I don’t think it’s unreasonable to say programming is like the Contra of careers. Fortunately, much like Contra, it’s also really easy to cheat give yourself an advantage.

So hold onto your seats, because I’m about to blow your mind. That advantage? Taking really awesome notes. Boom. How is your mind? Blown to bits? Thought so.

I know what you’re thinking. Take notes? That’s your amazing advice? But just trust me. When I joined the News Apps team, I was a self-taught developer with only front-end web development experience. I had to start learning programming languages, and to do that, I had to take notes.

The idea behind taking good notes is a simple one, but sometimes it’s helpful to be reminded of those simple things, especially when you’re dealing with really complex problems you don’t totally understand. Well, soon after I started with News Apps, Ryan Mark reminded me. I got into the habit of taking really detailed notes every day thanks to his advice. Of all the awesome advice our fearless leader has given me over the past couple years, this has proved to be the most valuable.

Of course, if you’re just starting out with programming, it’s only natural for your programming notes to be terrible. (In one of my early notes, I literally told myself to take better notes next time. Thanks for nothing, Past Ryan.) That’s because, as Alex Bordens mentioned more eloquently in his recent post, sometimes you’re going to have to do things you don’t totally understand. It’s scary but it might just be the only way to learn. Most of the time you don’t break things anyway. Most of the time.

When you do break something, that’s also where Alex’s advice of crying or drinking a beer comes into the fold. Emotional outbursts and alcoholic beverages are a natural part of programming, and I’ve had more than my fair share of frustrating days. But I’ve also avoided exponentially more frustrating days by having really detailed notes as a reference.

All told, I’ve got about five documents on file containing over 33,300 words (not including links). That’s a lot of words, so if you’re going to be a programmer for an extended period of time, or–heaven forbid–make a career of it, it’s a good idea to have a method to the madness.

First, I have a file dedicated to the majority of commands I’ve run on the command line, along with a description of what each does. It’s a pretty straightforward file that acts as a handy cheat sheet when I need it.

Then I have a file full of all of the fancy links that are shared in our group’s tech chat. This includes everything from cool tutorials to posts on interesting new tech. If you can, find yourself a good group of nerds who will share these things with you. You’ll stay on top of your tech, and even if you don’t understand some of it now, it could help you learn something new in the future.

I have another file strictly dedicated to those tutorials I have taken, from Python to Javascript and beyond. Those are notes in the traditional sense, like I would have taken in school, had I actually taken notes in school.

My final set of files are my most important–they’re what I refer to as my daily diaries. These are files where I record what interesting things I did or learned in that day, as well as share my innermost secrets about all the cute boys at work. I record by date so I can match tasks I’ve completed to changes in our Unfuddle tracking system (and to my own recollection). For example, I’ll vaguely recall doing some work on a particular project last fall; since I’m recording by date, I know what section of my notes to check first if I need to reference that work again.

I put commands and particularly relevant chunks of code in bold, so they’re easier to spot as I scroll through the abhorrently long files. The writing is keyword rich for the same obvious reason (CTRL+F is the best). I take entire error strings I received and paste them into the file; if I come across this error message again, I can easily find it in my notes and see how I fixed it, before having to resort to Google and wading through Stack Overflow threads for solutions that might not even apply to me.

Most importantly, I put a little bit of pressure on myself to record notes each day and not slack off, even if I try to convince myself I “mostly understand” everything I did that day. It makes it more of a ritual and part of my standard workday.

I know this all sounds very simple, but trust me when I say that you’re going to need all the help you can get as you start programming. Taking good notes is paramount to your success, otherwise you’re going to be asking your nerd friends/coworkers way too many questions and, unless you funnel constant streams of coffee into them, they’re likely to get agitated with you.

Take notes. Don’t agitate the nerds.

Written by Ryan Asher

March 28, 2014 at 11:09 am

Posted in Uncategorized

From print to web and the lessons along the way

with 2 comments

About a year ago I moved from the Tribune’s graphics desk to the News Apps team. This was an initially jarring experience for a seasoned newspaperman who was used to creating whatever his mind imagined using Adobe Illustrator. Now it’s not like I had never written code before, but I certainly didn’t write code on a daily basis. So when all of a sudden code is the only way to do your job, it’s pretty easy to lose your mind and panic. After all, the expectation was that I would actually produce something, not flail about as I made my way through codecademy tutorials all day and read JavaScript: The Good Parts (seriously though you should read that book).

Today, after a year of banging my head against JavaScript errors, cussing at Internet Explorer and spinning my wheels trying to figure out how to transform data into usable JSON, I have a much clearer idea of what it takes to evolve. This journey is not unique and is one I know many of my brothers and sisters in news are making. So here is a road map for an efficient yet practical path to transitioning from print to web. Let’s start with two very important points:

First, the internet is hard and will likely make you either cry or drink heavily depending on your personality. That’s OK. It’s why you have friends. Stackoverflow, GitHub, Google and David Eads are all readily available when it’s 1 a.m. and you just broke something. I also suggest an SOS on Twitter.

Second, you already have a lot to bring to the internet besides your novice programming skills, even if it doesn’t seem that way. If you take nothing away from this article besides this point, I’m going to call it a win. No matter what your discipline, you already have a unique set of skills that apply to the web. You just need to figure out how to apply it right away while you learn to code. Trust me, it will help your confidence and growth. If you are a designer, you know how to organize information. That is something the internet needs. If you are an editor, you know how to manage a project and make sure no one forgot the ‘l’ in public. That is something the internet needs. See where I’m going with this? You’ll be surprised by how much you can already do.

Now that we have that out of the way, in no particular order:

The only way you will learn is by doing. How do you get to Carnegie Hall? Practice! You will write lots of code and most of it will be crappy and broken. That’s OK. Cry it out or drink a beer, then move on. The more you write the better you’ll get. Repeat after me: “The more you write the better you’ll get.” Find a project that starts with your idea then make it a real thing online. When you make something work the first time it will feel like performing a miracle and the internets will love you for it.

Plan on working more. You have to put in the time unless you are a natural and the only reason you’re reading this is because my words are simply that captivating. Plan on coming in early, staying late and working on your own time. You may not always be in the classroom but you should always be learning. It will be a lot of work, but learning to code should be exciting.

Don’t try to learn all the things at once. Pick a goal and stick with it. You will grow faster if you focus your time on a particular language or outcome. In my case, I focused more on Javascript than anything because I wanted to apply my data viz and design background. Focusing will also help you quickly discover the go-to sources to solve problems. Focusing right away will help you avoid becoming overwhelmed by all the possibilities. Once you are solving problems and making things, then you can start branching out.

Adjust your workflow. Break away from your print-centric processes. It’s easy to design your project with your favorite Adobe product, but I can tell you right now it’s going to hurt you. While it’s important to have a plan/sketches, you will ultimately spend more time than necessary trying to make your website mockup perfect. Remember, mockups rarely translate identically to the final product. Instead use a simple web sketchup tool like balsamiq or a pencil and paper. If your Stockholm Syndrome just won’t let you ditch Adobe, then simplify your process to use generic squares and remember it doesn’t have to be perfect.

Don’t cheat your audience. This is a particularly important point to keep in mind if you are developing on deadline. Know your abilities and what you can deliver. As journalists, we are here to tell clear stories. Shipping a half-baked, broken website doesn’t serve your readers. Look for solutions that fit what you can deliver and put you in a position to succeed. Don’t shy away from something hard because it’s hard. Just make sure it’s the right way to tell the story.

Don’t be afraid to be a fraud. When you first start writing code you certainly won’t know everything and the people around you will know a lot. So when someone starts describing technology with a bunch of acronyms that sound more like STDs than internet, just roll with it. Ask questions, but remember you can solve problems on your own. Pay attention to the point the person is trying to make then go back to your desk and Google the bejesus out of those acronyms. It’s normal to fake it till you make it. And sometimes you may never think that you made it.

The internet is hard. Don’t let it deter you. It’s hard for everyone. It’s hard to learn a new language or change your workflow or scale back your ideas or spend less time watching House of Cards, but it’s up to you to make that sacrifice if you want to evolve.

Written by Alex Bordens

March 12, 2014 at 4:11 pm

Posted in Uncategorized

Everything you ever wanted to know about the News Apps process

with 4 comments

I had never thought of the News Apps team as a particularly secretive group, especially given our insistence on transparency. We grant access to our Google Docs to practically anyone who shows interest. We’ve held demos and lunches to introduce the team and our work to colleagues. Our code is mostly open source and accessible in GitHub and, well, we post to this here blog you’re reading. I’d say we’re doing a pretty poor job of cultivating an aura of mystery.

Yet that aura of mystery persists. Part of this is inevitable—in a company as large as the Tribune Media Group, few employees can keep tabs on the activities of departments other than their own. And I’m sure outside the company even fewer people know what the News Apps team does. Sure, some folks may recognize our work, especially projects that have received widespread media attention or open-source tools we’ve built that other developers have adopted, but who knows how the team actually works?

Oh, wait—I do! I’ve found myself explaining who we are and what we do with some frequency lately, so I thought it might be useful to explain our process here, in an easily-shareable blog post. Then you, too, will know how the News Apps team works…and if any part of our process sounds useful, feel free to crib it for your own team!

We practice agile development, with a lowercase a. While there are more structured forms of capital-A Agile Development (terminology! acronyms! certifications!), we stick to the basics as outlined in the Agile Manifesto and keep our process as lightweight and straightforward as possible. We loosely follow the Menlo Way, adopting some key practices (like prioritization of personas) and discarding those elements that haven’t proven a good fit for our team. We’re big fans of eliminating unnecessary steps while ensuring everyone involved in a project has the information they need to make decisions and accomplish their tasks.

The team typically follows the same steps for all major projects. Small projects, bug fixes, and improvements to existing projects often skip some or most of these steps, depending on the scale of the work. However, from start to finish, this is our full process.

1) Idea/request

We receive a request for help or one of our team members suggests a project idea. Excellent! Ideas are good, and we like to help. This is the ideal time to loop in the project manager and/or the team director—approaching one of them directly is likely the fastest way to get the gears moving.

2) Description/documentation

But we need to know more about this idea. We request a document from the stakeholder explaining what the idea is, what need it fulfills, who the audience for the product would be, etc. Any available supporting research is great, too. (We have a doc explaining what we need for stakeholders to use as a guide.)

3) Evaluation

Once we receive the requested information (and possibly ask a few followup questions), we can start to evaluate the potential project. What does our schedule look like? What deadlines are involved? Does it sound like there’s a real need for this product? If we think it’s something we should pursue, we let the stakeholder know we will be able to tackle the project and we start planning in earnest.

4) Research/personas

Based on the research the stakeholder provided, we do some additional digging on our own and write representative personas for key audience members and stakeholders. These personas should feel like real people; they will help us put a human face on the project and make sure everything we’re building is *for* either a user or stakeholder. Once these are written, we check with the stakeholder to make sure the personas are accurate representations of the relevant stakeholders and target market and make any necessary adjustments.

Meet the (fictional) Dennisons, a sample persona. Sorry about the tape glare.

Meet the (fictional) Dennisons, a sample persona. Sorry about the tape glare.

5) Prioritization of personas

With the stakeholder, we determine who our key personas are. We draw a bullseye and place one primary persona in the center ring, two secondary personas in the second ring and three tertiary personas in the third ring. The primary persona is our key concern; nearly every feature we ultimately build should satisfy a need for that person. The other personas on the bullseye are important, too, but our focus is that primary persona. If we try to build something for everyone, our product will be unfocused and ultimately satisfy no one; if we focus on that representative primary persona, we ensure we’re building something useful for a specific market.

And these are our personas prioritized on a target. Note the Dennisons in the middle--they're our primary persona. Yeah, my circles are a little rough.

And these are our personas prioritized on a target. Yeah, my circles are a little rough.

6) Needs/wants

With the stakeholder, we brainstorm a list of each persona’s needs and wants as they pertain to the subject matter of our project. These are not features or tasks, simply what each person ideally wants. (A developer persona, for instance, might want a stable site that doesn’t require constant maintenance. If the project is a shopping site, for example, a suburban mom persona might want to know which stores in her area are having sales.)

7) Features

Once we have a list of each persona’s needs/wants, we (with the stakeholder) make a list of what features would address each of those needs/wants. It doesn’t matter how outlandish these features are—we know we’re not going to build them all—but we want to make sure we don’t overlook anything. For convenience, we write each feature on a separate index card.

8) Prioritization of features

With the stakeholder, we sort all the features into piles: Must Have, Really Want, Nice to Have and Meh. Must Have features are the features that are absolutely necessary for our minimum viable product, Meh features are things that we’ll probably never get around to building and Really Want and Nice to Have features are pretty self-explanatory. When we begin work, we tackle Must Have features first and work our way through Really Want and Nice to Have, depending upon how much time we have until launch.

9) Manifesto

Once we’ve established our audience, figured out their needs and determined what features our product should contain to satisfy those needs, we write a brief manifesto. Despite the intimidating name, this is just a short document that describes our high-level vision for the project: in brief, what features it will include, how it will be used, what key needs it will fulfill, etc. This isn’t a detailed, in-depth project requirements document, more of an overview of what we plan on building and the rationale behind the project. It’s written like an essay, vision statement at the top followed by narrative that describes the personas and key features. The narrative section is followed by the prioritized list of features. After we write this document, we will share it with stakeholders and incorporate any necessary input. Once we make any necessary adjustments, it’s important that all stakeholders sign off on the manifesto; this ensures we’re all on the same page and gives us a document we can refer to if new ideas or feature requests are proposed at a later date.

10) Kickoff

We have our personas, our prioritized list of features and our manifesto. Now is the time for an official kickoff meeting with stakeholders to make sure everyone’s on the same page. Whereas we try to keep brainstorming sessions and many other meetings quite small in order to move quickly, this meeting will likely involve all possible stakeholders. Any feedback? Reconsidering the priority of some features? Let’s discuss. This is also a good time to reconfirm deadlines and manage expectations in order to avoid confusion down the line.

11) Tickets

We can start entering tickets in Unfuddle any time after we’ve prioritized our features, but we need them in the system as soon as we kick off. In this stage, we make sure that all technical tasks encompassed in the features are entered as separate tickets. One feature may be broken down into several discrete tasks, each of which should have its own ticket.

12) Project plan (milestones, iteration reviews)

Once we’ve entered tickets in Unfuddle (or simultaneously), we need to make sure we have our project plan mapped out. The number and complexity of tasks, the number of team members working on the project and the amount of time we have to work with all figure into how many iterations we’ll schedule. Each iteration is typically a week long and culminates in a weekly iteration review with the project manager, the developers working on the project and the relevant stakeholders.

13) Development/iteration reviews

In order to begin development, we need to schedule tickets and assign them to team members. At the beginning of every project, we set up recurring weekly iteration reviews with the development team and the relevant stakeholders; in these iteration reviews, we cover:

  • what we did last iteration
  • what work was scheduled that we didn’t get to
  • what’s scheduled for the next iteration
  • any questions/comments/blockers.

These weekly reviews keep everyone in the loop and ensure we all have the opportunity to flag any potential issues in a timely fashion. Everything is described in a document and shared with the meeting attendees a few hours before the meeting so we can spend the meeting time giving a demo and collecting feedback.

14) Launch

Not on Fridays, just in case…. We always make sure to get stakeholder approval on launch dates and request any necessary support/changes from other departments as far in advance as reasonable. Most if not all of the developers assigned to a project should be in the office on launch days and should be prepared to stay late if necessary.

15) Post-launch iterations/support

Depending on the scope of the project, we will likely schedule at least one post-launch iteration of work and iteration review in order to make bug fixes and adjustments. Sometimes (particularly if we’ve launched a project on an aggressive timetable) we will plan on making multiple iterations’ worth of further improvements; we continue to schedule weekly iteration reviews with stakeholders as long as it’s a project with ongoing development (other than simply a support issue or bug fix here or there). Once we complete any post-launch development, we are still available to provide support on an as-needed basis. No further iteration reviews are necessary at this point, unless extensive additional development has been requested, in which case we will evaluate the new requests and schedule accordingly.

Situations where we don’t use our full process

For small projects or tight deadlines, we have a document template that encapsulates and streamlines steps 2-9. Someone on the team will complete this document, which the team director and/or project manager will review before determining if we can take on the project. If it’s a go, the project is scheduled and kicked off as soon as our schedule allows.

Bug fixes go straight into an Unfuddle ticket and are prioritized. Feature requests are vetted by the director, the project manager or a developer and either turned down or entered into an Unfuddle ticket and scheduled depending on priority and team availability.

Written by Kaitlen Exum

March 5, 2014 at 3:10 pm

Posted in Uncategorized

The Nuts and Bolts of Tribune News Apps Event Listings

leave a comment »

Screen Shot 2014-02-17 at 9.43.45 PM

A few months ago, we built a reusable Javascript app to provide an interface to a quirky events API. Internal stakeholders needed standalone, searchable event listings, embeddable calendar and upcoming events widgets, and integration with Google calendar, Facebook, and desktop calendar software.

Here’s how we built a client side app using a turbo-charged Backbone collection powered by LunrJS and a set of independent Backbone views powered by TypeaheadJS and FullCalendar to create a library that can be deployed in a wide variety of situations at the Tribune. After a solid year of using/loving/hating/fighting Backbone but uninspired by the other options, we found a style that our team can build on.

You likely have a different quirky API to deal with. Hopefully you can learn from our architecture, optimizations, struggles, and future plans.


Three Tribune properties — Blue Sky Innovation, Theater Loop, and Just Kidding — needed event listings pages and homepage widgets. The event information is entered by Tribune employees and is available via a web API provided by a third party vendor.

The vision was to build a simple list of events with a search box at the top and calendar next to the event list. We’ve had good success with using TypeaheadJS to provide simple search. The goal was to provide a single way to filter the data using Typeahead’s support for multiple datasets.

Screen Shot 2014-02-17 at 9.47.46 PM
Searching for opera

Of course, there was some bad news. The API was designed for a specific type of user interface that implements faceted search. It can filter specifically on venue, town, neighborhood, and other event properties. You’ve probably seen faceted search on shopping sites like Amazon, eBay, and millions of others.

Screen Shot 2014-02-17 at 9.53.18 PM

A never ending list of filtering options in the left rail are search “facets”

The lack of full text search was a real problem, particularly because event descriptions contain important information like review author and star rating (for lack of a better place to put them).

The solution

To get around this and other limitations of the API, we took a novel approach. Instead of using the API to execute searches, we instead load all the data for a specific time frame (typically the next three months) and index it in the user’s browser. Instead of using the API for search, we turn your browser into an event search engine using LunrJS.

The downside of this approach is a somewhat heavy initial page load. To minimize the pain, we gzip and aggressively cache results. The upside is that once the data is loaded, the search is very fast since all the data is indexed in memory on the client computer. And we can tune the search to return exactly the results we want for any given search term.

Building the app

Let’s look at the finished components and how to deploy a simple version of the app.

  • Event collection: Loads all event data for a given time range from the API, indexes it using LunrJS (a browser based full text search engine) and provides a search() method and event.

  • Event list view: A list of events that renders on every search event.

  • Event calendar view: A calendar of events that renders on every search event.

  • Event search view: A search box (powered by TypeaheadJS) that triggers search events.

The collection is the central component. All the views either listen for collection events or trigger collection events based on some input value, either through a Backbone router or the autocomplete search box.

Screen Shot 2014-02-17 at 10.17.43 PM
Search box embedded on Theater Loop homepage

The following code snippet is from a widget that provides a simple event search box that can be embedded on an external page.

What’s happening here?

  • Define the collection.

  • Set up a filter view that is bound to the collection.

  • Add an event listener that waits for the data to be loaded, then attach custom behavior to the autocomplete box to navigate to the external URL on selection.

  • Get the data! The filter view listens for the sync event, meaning the data is loaded, and then uses the data to set up a typeahead box.

The full blown event app

Showing all the code is likely to cause your eyes to cross, the architecture’s what matters anyway:

  • Set up helper functions to scroll around the results list.

  • Define the collection.

  • Define a Backbone router to manage and represent the application state via the URL.

  • Set up a filter view, calendar view, and event list view.

  • Bind search behavior to route change events: When the route changes, call

  • Bind route change behavior to view events: When a day in the calendar is clicked or a search is executed, change the URL and trigger a router event.

  • Get the data!

The event list and calendar are already listening for search events being emitted by the collection to render the appropriate slice of data.

Keeping the components loosely coupled

Each Backbone view is completely self contained and depends only on the collection. That means each deployment of the code must set up its own event listeners. This is annoying because it means some repeated boilerplate code across deployments.

That annoyance is outweighed by the flexibility of the system: Want a calendar on your homepage that links to the event search? We can do that! Want a calendar and an autocomplete box? We can do that too! How about an autocomplete and a list of events? Sure thing, holmes. Need to parse event descriptions as Markdown? We got this!

Digging into the collection code

The data that is provided by the API looks like this:

There’s a kind of compression going on in this data format. Instead of listing every event on every date as a unique object, only distinct events are provided, with a nested list of dates that the event occurs on.

For the purposes of display, this data structure will need to be “unspooled” into a flat list of events. Let’s take a look at the collection source and discuss a couple of key optimizations and notable features of the code:

There are three key functions here: initialize, parse, and search.

initialize is Backbone boilerplate to load in options and set up your view. It does extra duty here by immediately indexing the data using LunrJS.

The parse method massages the data structure above into something useful for display:

  • Call optional preprocess function on the data

  • For each row in the data:

    • Add row to rawData collection attribute to capture state of the data immediately after any preprocessing.

    • If the index option on the collection is true, index the row with LunrJS.

    • Collect venues, dates, and neighborhoods for use as TypeaheadJS datasets.

    • “Unspool” events by iterating over each repeating date and create a new object for that particular event, date, and time.

  • Remove duplicates from collected venues, dates, and neighborhoods.

The search function populates a collection.filtered property, which is the data we’ll want to display. The trick here was to make the filtered list itself an instance of the event collection. This instance does not need to be indexed by LunrJS (we already indexed the whole dataset) but it does need to be re-parsed.

Perhaps it sounds inefficient to re-parse the data on every search. But this design was born of necessity. Originally, the indexing happened after the raw data was unspooled into a big flat list of events. Unfortunately, a list with 400 events could easily become a 5,000+ element list, especially for theater events which tend to repeat in almost all cases.

It’s no surprise, but indexing 5,000 elements is a heck of a lot slower than indexing a few hundred elements. And pointless, given that the only difference between repeating event instances is the date of the event. The title, description, etc, will always be the same.

The code was refactored. Instead of searching against the flattened list, we search against the raw data, get the subset of the raw data that matches the search, and re-parse that subset. Despite the cost of slicing and re-parsing the raw data for every search, this approach was still on the order of hundreds of times faster and significantly more memory efficient than indexing every element after unspooling repeating events.

Also note the use of triggers at the start and end of parsing and searching. This allows application code to listen for these events and provide visual feedback like fading the results list in and out or showing a waiting spinner.

The proxy

There is one last piece of architecture to note. The events API we are using was not designed for direct access from public clients, nor does it support outputting iCal-formatted event data. So we wrote a tiny (55 line) proxy server using Python Flask. Our Fabric-based deployment rig makes deploying a little app like this behind nginx very easy.

That’s it! Our proxy goes out to the API and gets the data, then sends it on back to the client with aggressive caching and gzipping provided by nginx. If iCal-formatted data is needed, the proxy gets the data from the API as json, pumps that data into a template that uses the iCal format, and sends back the appropriate file.


We host the event listing library on our team’s shared assets Amazon S3 bucket. Currently, the library is integrated with a Django application and the main Chicago Tribune CMS. The standalone events pages are published with Tarbell. But hosting central, versioned copies of the library, we can deploy on practically any platform that allows a script tag.

Future optimizations

FullCalendar could use some improvement or an upgrade. It does a tricky job quite admirably, but with a fussy API and some rendering performance issues. The fussy API is acceptable, but a calendar that rendered a little faster and responded to media queries a little better would be a good improvement.

The project depends on a fork of TypeaheadJS to integrate LunrJS that still needs to be submitted back to the TypeaheadJS maintainers.

Data loading could be broken down into successive calls to cut down on initial page load.

The library could benefit from cutting down on repeated indexing. It may be possible to stash the index in local storage, especially because the underlying data source changes fairly infrequently.

Keep it flexible

By emphasizing a strong data API via our collection and keeping view code loosely coupled even at the expense of repeated app code, we were able to create a client side library that is flexible, stable, and easy to deploy. These are patterns to build on.

Written by David Eads

February 17, 2014 at 10:20 pm

Posted in Uncategorized

How to keep your data safe from even a supernova

leave a comment »

Intriguing headline, no? Check out the rest of the article News Apper Abe Epton wrote for the Chicago Tribune’s new(ish) section, Blue Sky Innovation. It’s a mercifully hysteria-free discussion of password management, encryption and data security.

Note: If we actually have to contend with a supernova, password protection is gonna be the least of our worries.

Written by Kaitlen Exum

February 14, 2014 at 11:22 am

Posted in Uncategorized