New Yorkers

Building and Operating a Serverless Data Pipeline

What is the focus of your work these days?

I'm the lead of the data platform team at Intent. Intent is a data science company that helps commerce sites maximize the value for each person who visits their site. Our main product is an ad network that runs on travel sites. On the data platform team we oversee our data pipeline, that collects data from ad servers and various other sources, the processing of that data and storage of that data to a data lake. We also oversee data warehouses, and aggregation jobs that compute summaries of that data;  making it available to various analysts and other business users. Our team is also responsible for automated reporting solutions for our publishers and advertisers.

What is the motivation for your talk?

To talk about our experiences at Intent building out our serverless data platform. Going from our legacy platform to this new data platform. Some of the wins that we experienced with that and also some of those pain points that we had during that transition. I also think that Serverless is becoming a bit of a buzzword so I think it’s worth clarifying what serverless is, and some of the benefits and tradeoffs.