Loading and processing large datasets

When you use deep learning, you often need a large dataset to train neural networks. It is not uncommon to use gigabytes of data to train your model. Included with CNTK is a set of components that allow you to feed data into the neural network for training.

Microsoft did its best to build specialized readers that will load data into memory in batches so you don't need a terabyte of RAM to train your network. We'll talk about these readers in greater depth in Chapter 3, Getting Data into Your Neural Network.