Machine learning and AI received a lot of attention in the last few years, but very little was published on how to architect real-life ML-based systems.

In this session we present our architecture that supports an AI-based system with numerous live models handling multiple types of data (audio, speech, text, numeric data) with multiple training and inference lifecycles, and multiple approaches (supervised vs. unsupervised, traditional vs. neural networks).

We discuss how we created a cloud architecture supporting security and privacy, how we built a system architecture that supports training and inference at scale, and how we package software in a polyglot (Java/Python/...) environment.