Automatic analysis of medical imaging is transforming medicine as we speak. However, it is no easy task - the medical imaging domain poses unique challenges in one's way. For example - privacy concerns make obtaining data and labels a major challenge, medical imaging data is not your standard RGB input, can be 3D or even 4D and can often be of huge size. In this talk I will describe some of the challenges we encountered at Viz.ai, how we tackled them, and how this work translates into saving lives.