Presentation: "High Volume Data Transformation"

Time: Tuesday 12:05 - 13:05

Location: Roebling/Gleason

Abstract:

Today's enterprise systems interact with multiple external systems and usually, because the world is like that, they each talk a different language. Even where standards exist (be they SWIFT, FpML, ISO20022 or FIX to name a few) each party has their own particular usage which requires custom handling. Internal collaborating systems are often little better as siloed business units pick data representations that are geared towards their own usage of the information.

How do you handle all these different data formats without polluting your application? Moreover, as data volumes grow, how do you acquire data from a 10GB zipped CSV file containing millions of messages as efficiently as you do individual XML messages read from a queue? Can we make the logic sufficiently consistent that the operations team can monitor and manage both in the same way?

Using a worked example loading data into MongoDB, this session will show how technologies such as Spring Integration, Spring Batch & C24 Integration Objects can be used to address the problems above (and the equivalent outbound ones), along with some practical examples to demonstrate that the obvious solutions are not necessarily the best performing. You will also see how languages such as Scala & Clojure provide a more descriptive way to wire them together.

Andrew Elmore, Head of Development, Incept5

 Andrew  Elmore

Andrew Elmore is the Head of Development at Incept5, an international consultancy specialising in enterprise application development and integration. Prior to joining Incept5 Andrew built and managed the front & middle-office systems at PrePay Solutions, a credit card issuer & processor. Earlier roles include development manager at SmartStream Technologies and head of new development at Quadstone Limited. He holds a BEng in Electronics & Computer Science from the University of Edinburgh and is a Chartered Engineer.