data partition within group

[nudles]

This training scheme partitions one batch data into sub-batches where each worker in the group process one sub-batch. It is implemented by partitioning the layers (except the data and parser layers) of the original neural network into sub-layers where each sub-layer holds a sub-batch of the features. These sub-layers share the same set of parameter objects. There is one worker who is the owner for each parameter object. Each worker owns a partition of the neural network and computes the gradients of parameters over it.

The work flow is,

• Each parameter object is initialized by its owner worker. A put request is then sent to the server by the owner worker.
• Each worker waits for the fresh parameters and runs the Back-propagation algorithm over its owned partitions (i.e., layers). Once getting the gradients, it send an update message to the main thread (i.e., the stub).
• The main thread averages the gradients from all workers for the same parameter and sends the update request the the server.
• The main thread handles the responses for the update requests and update the parameter version and data field.

TODO,

• optimization for single node case, where memory copy can be avoided by sharing memory between servers and workers.
• consider multiple nodes case.
• update the autoconf files to delete some files which are merged into other files.