Big Data

Big data is here to stay and to, well, get bigger. This track will cover the big data fundamentals, latest tools, tricks, and technologies to help you mitigate your big data challenges and to extrapolate value out of your data. We will move beyond just using Hadoop to best practices, scaling, and cover other large scale big data processing technologies like Storm.

Day of week:

Wednesday

Location:

Grand Ballroom - Salon A/B

Host:

Khawaja Shams

Presentations

Building Modern Applications using Apache Hadoop

10:20pm - 11:10pm

By: Eli Collins

The rapid adoption of Apache Hadoop and increasing capabilities of its ecosystem have made it a compelling platform for building applications. Hadoops architecture has allowed it to go beyond processing application-generated data to becoming a central component of products themselves, enabling a new generation of analytics-driven applications. In this presentation, Eli will give an overview of both how to build new applications with Hadoop, as well as how to integrate Hadoop with your existing applications. Eli will give an update on the state of Hadoop ecosystem for developers, covering the diverse set of frameworks and APIs useful for building different types of applications using Hadoop. Finally, Eli will cover issues like data management in the context of Hadoop-based applications.
Data Science of Love

4:00pm - 4:50pm

By: Vaclav Petricek

eHarmony was founded to give people a better chance to find someone for a long lasting relationship. As one of the first companies we have applied advanced technology what became known as Data Science these days to the age old problem of matchmaking.
Over the years eHarmony has accumulated vast amount of data on variety of romantic interactions. This data a is a treasure trove of entertaining tidbits and nuggets of insight into human nature. I will share some of those in hope that people may find them useful but more importantly I will also demonstrate how we actually use this data to make recommendations and give single people an upper hand in finding "The One".

In particular I will show how we utilize hadoop (YARN) to process billions of pairs of user profiles t o find ngrams and other features that are predictive of romantic attraction and how we use the features discovered for large scale machine learning using vowpal wabbit's allreduce parallell learning.

Finally I am going to describe an optimization technique that decides what matches to deliver to who and when but which is more broadly aplicable to other domains such as advertising or constrained recommendations.

At eHarmony, hadoop is hard at work looking for love. It gets a hand from hive, impala, and vowpal wabbit while solving three fundamental problems:

1) Compatibility Matching - who is right for who for the long term?

2) Affinity Matching - who is attracted to who?

3) Match Distribution - who should be introduced to who and when?

You will see how parallel processing and large scale machine learning allow us to sift through free text of user profiles and find ngrams that are predictive of romantic attraction.

Did you know that hadoop is now responsible for about 5% US marriages?
Lessons learned building Storm

1:40pm - 2:30pm

By: Nathan Marz

Storm is an open-source, distributed, realtime computation system. Storm is a multi-tenant system in which diverse and independent applications share the same resources on the cluster. Building Storm involved numerous challenges related to distributed systems, resource scheduling, resource contention, and capacity planning. We solved these problems by relying on a simple motto: the more complex the problem, the simpler the solution must be. In this talk you'll see what we learned building out Storm, both from a technical and process standpoint.

These lessons are hugely relevant to anyone working on complex systems or building out shared infrastructure.
Leveraging your hadoop cluster better - running performant code at scale

2:50pm - 3:40pm

By: Michael Kopp

Somebody once said that hadoop is a way of running highly unperformant code at scale. In this talk I want to show how we can change that and make map reduce jobs more performant. I will show how to analyze them at scale and optimize the job itself, instead of just tinkering with hadoop options. The result is a much better utilized cluster and jobs that run in a fraction of the original time running performant code at scale! Most of the time when speaking about Hadoop people only consider scale, however, when looking at it it very often runs highly unperformant jobs. By actually looking at the performance characteristics of the jobs themselves and optimizing and tuning those far better results can be achieved. Examples include small changes that cut jobs down from 15 hours to 2 hours without adding any more hardware. The concepts and techniques explained in the talk will be applicable regardless which tool is used to identify the performance characteristics, what is important is that by applying performance analysis and optimization techniques that we have used on other applications for a long time we can make hadoop jobs much more effective and performant! The attendees will be able to understand those techniques and apply them to their map/reduce/PIG/hive or other mapreduce jobs.
LinkedIn Endorsements: Bootstrapping a data product

5:10pm - 6:00pm

By: Sam Shah

How can large-scale machine learning be used to build products? At LinkedIn, the latest "data feature" in our portfolio is Endorsements, a mechanism to recognise someone for their skills and expertise. This ecosystem is generating a large graph of reputation signals: over 1B endorsements have been made in the few short months since the product launched.

How were we able to do this? In this talk, I'll deep dive into technical detail of our approach and the practical aspects of building a data feature like Endorsements. I'll go into how we extract a taxonomy of skills, how we determine if someone possesses a skill, and how we use that knowledge to recommend people to endorse. I'll also detail some of our open-source Hadoop-based infrastructure that allows us to put this into a productionized process.
Polyglot Programming Unconference

11:30pm - 12:40pm

By: Nitin Bharti

Meeting each other in the context of conversations we care about is why at QCon we are giving prime time to the unconference format; it creates a space for peer-to-peer learning, collaboration and creativity.

First we’ll father in a circle to be guided through creating an agenda using open space technology. Then there will be three three 20 minute time slots, each with 5 concurrent break out discussions that were designed by you.

Attendees often know more than the speakers! Bring your questions, ideas - be prepared to be surprised.