Fluctuating prices. Changing weather. Increasing globalization. Two Sigma uses artificial intelligence and other advanced technologies to look for meaningful patterns in the world’s data. Then, we use those insights to create investment strategies.

In this talk, we’ll discuss how Two Sigma was able to scale up their research to harness tens of thousands of CPUs. First, we’ll look at the previous generation of Two Sigma’s compute resource scheduler, and the challenges a quota-based system brought with it. Then, we’ll learn about Cook, the Mesos-based scheduler Two Sigma developed and contributed to open source. We’ll also discuss some of the challenges faced along the road to increasing the number of machines by orders of magnitude, and the solutions we engineered to build a reliable, scalable platform. We’ll conclude by discussing the state of the infrastructure today, and how Two Sigma leveraged open source software to build their world-class platform.