Streaming

Closer to the Wire: Real-time News Alerting @Bloomberg

What is the focus of the work you do today?

I'm working in the News Search Infrastructure at Bloomberg. In a nutshell, when a news story hits the News Search Infrastructure, it gets enriched by a set of metadata, e.g. classification, people tagging, sentiment analysis, etc, then it is indexed to be available for searching. What is interesting is that the incoming news stories are also matched against a set of stored search queries that they users want to be alerted on in a variety of ways. My team, in particular, is responsible for the News Alerting and Streaming backend system.

What's the motivation for this talk?

My motivation is to share the challenges and the lessons learned from our adventure on building the News Alerting system and also show an example of a scalable alerting system. Some of the lessons learned include what you need to be aware of before deciding to go with an open source project. Software architects and tech leads will be familiar with the complex set of trade-offs that need consideration. However, it’s not rare to hear people saying “If this solution is free, then I don't have to build a team around, like I would do while building a proprietary system”. There are many benefits to using open source but in most cases of production software there is no such thing as “free lunch”.