Architectures you've always wondered about

If you have ever wondered how the other guys approach large scale problems -- and solve them -- this track is for you. A perennial favorite through all QCon conferences, this track is the place for war stories and hard-won lessons, lofty principles and pragmatic realities, what went well and what blew up in our faces. This year's track will be Architectures You Have Always Wondered About -- New York City style. All speakers are in and around the New York technology scene, and represent the best of Big Apple talent. These speakers build and operate sites you have heard of -- or will soon. Come learn and share.

Day of week:

Wednesday

Location:

Grand Ballroom - Salon D

Host:

Randy Shoup

Presentations

Scaling Foursquare: From Check-ins to Recommendations

10:40am - 11:30am

By: Jon Hoffman
Foursquare Engineer (former head of Infrastructure Engineering)

Foursquare has grown from a simple check-in application to one that can tell you the best place to grab a burger. Along the way we've also become the best source of local data for apps like Instagram and Pinterest. I'll talk about the general architecture, storage systems and development practices that we've created to handle the ever increasing volume and complexity. Foursquare started out as a PHP MySQL app running on a single virtual server. Early on we switched to Scala and Mongo and have recently been aggressively splitting a monolithic application into multiple services while maintaining a single code base. I'll talk about the somewhat unique approach we've taken in our service oriented architecture. I'll also go over how we run Mongo and the additional storage systems we're running for more specialized tasks like search, key-value lookups, and offline tasks.

Solidifying the Cloud : How Google backs up the Internet

3:00pm - 3:50pm

By: Raymond Blum
Data integrity obsessed Site Reliability Engineer @ Google

Systems like GMail and Google Apps keep massive amounts of data in the cloud, all of which must be preserved despite the inevitable. Typical backup and recovery techniques don't scale, so Google has devised new methods for securing unprecedented volumes of data against every type of failure. There are many unique challenges, both obvious and subtle, in delivering durable storage systems at this scale; we'll discuss these and their solutions as well as some alternatives that didn't make the grade.

Scaling Chartbeat from 8 Million Open Browsers to Realtime Analytics and Optimization

4:15pm - 5:05pm

By: Wesley Chow
CTO @ Chartbeat

Chartbeat provides a real-time analytics platform that’s used by 80% of top US news organizations to inform their newsrooms on how visitors are engaging with content. We exceed 200,000 requests / sec on an average day, with each piece of real-time data making it into our dashboard within a fraction of a second. In addition to processing large amounts of data in real-time, we run Hadoop and Redshift jobs to produce reports and optimization tools for our customers.

I’ll discuss the overall architecture of the system and how we’re able to handle the requests in a cost efficient manner using a custom written analytics engine in C and Lua. I’ll also talk about some interesting numerical problems that arise from scale, and some algorithmic challenges and solutions we’ve encountered while measuring tens of billions of engaged minutes every month.

Presentations

Architectures you've always wondered about Open Space

11:55am - 12:45pm

Scaling Gilt: from Monolithic Ruby Application to Distributed Scala Micro-Services Architecture

1:45pm - 2:35pm

By: Yoni Goldberg
Lead Software Engineer at Gilt

Since its founding in 2007, Gilt--the innovative online shopping destination--has transformed its infrastructure from a monolithic Ruby application into distributed, fault-tolerant, scalable architecture based on hundreds of micro-services built with Scala. In this talk, Gilt Lead Software Engineer Yoni Goldberg will describe some of the technological innovations that have helped Gilt to reach its current size, and highlight some of the core challenges that the company's engineering team continues to face. He will also discuss what every tech team needs to consider and address before heading down the path of building a first-class micro-services architecture.

How Facebook Scales Big Data Systems

5:30pm - 6:20pm

By: Jeff Johnson
Core Data Engineer at Facebook

Facebook uses distributed storage systems that individually have tens of thousands of servers across multiple data centers around the world, receiving millions to billions of queries per second. At this scale, it is essential to find a malleable storage solution that is fast, easy for both engineers and operations staff to use, and provides strong safety guarantees.

Jeff Johnson, Software Engineer at Facebook New York, will share -- for the first time -- learnings on a new Facebook NoSQL project. The project's aim is to create a storage system simple in concept, implementation and operation, that offers a solution to the breadth of popular systems lacking one or more of these foundational elements. In comparison to many current NoSQL systems, this new system is based around using strong consistency as a core building block, implemented with the Raft consensus protocol. Jeff will detail how this approach is used to tackle storage problems at Facebook scale, and how issues of data availability and placement are handled.