This a course from deeplearning.ai which taught about distributed and parallel training of a ML model on GPUs and TPUs.
Have a look at my achievement on this course. DistributedTraining.pdf