Presentation: Online Learning & Custom Decision Services

Track: Machine Learning 2.0

Location: Liberty, 8th fl.

Duration: 4:10pm - 5:00pm

Day of week: Wednesday

Level: Intermediate

Persona: Data Scientist

What You’ll Learn

  • Hear about recent developments in interactive learning.
  • Discuss which problems they are addressing and see the possibilities for this technology.
  • Understand how to leverage the Microsoft Decision Service for website personalization.


We have been on a decade long quest to make online interactive learning, a routine fact of life for programmers everywhere. Online learning systems can react quickly to changes in behaviour and have wide ranging applications such as fraud detection, advertising click-through rate prediction, etc. Doing this well has required fundamental research and development of one of the most popular open-source online learning algorithm systems ( ). Most recently, we have also created a system (, ) which automates exploit-explore strategies, data gathering, and learning to create stable and routinely useable online interactive learning. For well-framed problems such as content personalization sophisticated interactive online machine learning is now usefully available to programmers. For data scientists who can program, this is a fantastic new tool for solving previously unsolvable problems. As an example: How do you interpret an EEG signal so a user can type?


QCon: What are you going to discuss in your talk?

John: There are a lot of advances in terms of interactive learning. We have a cognitive service in public preview at Microsoft now where you can go and start doing personalization of web sites very easily. You set up an RSS feed, you add a little bit javascript to your web page and then it can start personalizing based upon whatever the user prefers.

There's a sense in which it will always work once you get enough information and that makes it a much more reliable algorithm to use in practice which allows you to insert it into a lot of pipelines where you would otherwise find the overhead of hiring a data scientist and doing AB testing to be far too high.

QCon: What does this service look like and what does that feedback loop look like that goes into it?

John: There are two ways you can use this decision service. One way is for kind of pre-framed problems. Content personalization which is the one that I will be talking about is pre-framed. It's very straightforward, you sign up and create an RSS feed with the content that you want to personalize. You add a little javascript to some web pages and then it kind of happens.

On the user's browser there's a javascript call which goes out and says: “How should things be ordered?” You put that javascript call on the top of your web page. What's happening in the background is that the the structure on the web page is getting boiled down to features which are getting turned into an example which is fed into Vowpal Wabbit once joined with a click or not signal. Essentially every individual event is treated in an experiment which has a duration of 10 minutes. If a click comes in that 10 minutes it will be joined in the example and that will be something positive saying oh yes it's good, or if it's not there then it'll be negative saying it's bad.

This is for the pre-framed things where a programmer can just do it. And then there are the not pre-framed things, there you need a programmer with some data science expertise to think about how you want to solve the problem.

We have probably half a dozen different framings that we've worked on that are in different levels of development in terms of making deployable applications. The first plan is getting the kind of personalization up because we have big success stories there. The second plan is enabling things so that data scientists / Dev can do their own framings and do their own fun stuff. That's happening over the summer and the documentation for how to do your own features for example will be there by the time you run QCon. The third plan is more about trying to fully capture some framings as we develop them.

QCon: Who's the persona for this talk, the developers, data scientists?

John: It's both to some extent. For the pre-framed things a developer can do it, it doesn't even have to be a back end developer, it could be just a web developer. We are creating a tool that just has not been available. For data scientists this is going to be a first kind of widely usable reinforcement based learning system.

We are reaching a point where these is a new class of problems which has never before been solvable by non PhDs via reinforcement learning and that can be a very amazing thing. Instead of taking a year of effort from somebody who has a PhD. it could be an hour of effort from somebody who knows how to program.

QCon: What do you want someone who comes to this talk to walk away with?

John: I want them to walk away knowing how to solve a new category of problems. For the ones who have those problems I want them to walk away ready to do it.

QCon: What should a developer who wants to stay ahead of the curve be focused on?

John: Machine learning is definitely very cool. Reinforcement like learning is extremely important in the long run. I expect that a decade from now we will think of machine learning as reinforcement learning because most of the problems are of that flavor.

Speaker: John Langford

Leading Machine Learning Researcher, Vowpal Wabbit Contributor

John Langford is a machine learning research scientist, a field which he says "is shifting from an academic discipline to an industrial tool". He is the author of the weblog and the principal developer of Vowpal Wabbit. John works at Microsoft Research New York, of which he was one of the founding members, and was previously affiliated with Yahoo! Research,Toyota Technological Institute, and IBM's Watson Research Center. He studied Physics and Computer Science at the California Institute of Technology, earning a double bachelor's degree in 1997, and received his Ph.D. in Computer Science from Carnegie Mellon University in 2002. He was the program co-chair for the 2012 International Conference on Machine Learning.

Find John Langford at

Speaker: Markus Cozowicz

Senior Research Software Development Engineer @Microsoft

Markus Cozowicz received an M.Sc. in Computer Science from Vienna University of Technology and an M.Sc. in Statistics from University of Vienna. He is an engineer working on the intersection of systems and machine learning. Prior to joining Microsoft Research New York he worked on time series anomaly detection in streaming systems such as Azure Stream Analytics at Microsoft Research Advanced Technology Labs Europe and on speech recognition for broadcast news at SAIL LABS Technology, Austria.

Find Markus Cozowicz at

Similar Talks

Software Engineer @Agrilyst
Developer Advocate @Couchbase
Principal Software Engineer @ Vistaprint
Cofounder & CTO, previously Co-Founder & CTO @Gilt
Senior Infrastructure Engineer @Heroku
Platform Director, "SeatGeek Open"​ @SeatGeek


Monday, 26 June

Tuesday, 27 June

Wednesday, 28 June