Joone logo

the Joone site
SourceForge
 
   

The Distributed Training Environment (DTE)

PDF
PDF

Introduction

The idea that underlies Joone is to build a professional environment to create, train, distribute and use neural networks in a simple but powerful way. To accomplish this final goal, we have built a distributed training environment to train in parallel many neural networks.

Our dream is to provide a professional framework to permit everyone to build neural networks that could be trained on a lot of common problems of the real life suitable to be resolved by the neural networks (spoken and handwriting recognition, stock quotes forecasting, credit loan assessment, etc.), and then distribute these trained nets to interested users (final users, sales force, etc.) that can use the trained neural networks on their devices (desktops, palmtops, PDA, etc.).

Thanks to this framework it would be possible to originate a 'Neural Network Farm' that could fabricate neural networks by training them in parallel on several machines, and finally distribute them to the final users (e.g. for free, or as prepaid service), to 'train once, run anywhere' a neural network without being worried about the HW or SW platform owned by the user.

Why a distributed environment?

When we want to use the neural network technology to resolve a complex job, training only one neural network, often, doesn't resolve the problem, because the net can fall onto a local minima without finding the best results; thereby what must be developed is a mechanism to train many neural nets in parallel on the same problem, on several machines, governing the whole process from a central point.

Joone has an own complete and professional framework where it's possible to create a neural net (with the GUI editor), and then teach several copies of it on many machines where some agents are running (they also based on joone's engine, of course). Those agents (or Workers) reveive a copy of the neural net and train it, sending back the outcome (i.e. the trained net) to the starting machine where the user, or an automatic script, can control all the results of the distrubuted training process, being so able to choose, at the end of the process, the best net, by applying some discriminating mechanism: e.g. choosing the net with the lowest error, applying a genetic algorithm, etc.

Possible Applications

Maybe most of you will have already thought about some useful application suitable to be resolved by the DTE, but I think that it could be useful to give some ideas about possible applications of a distributed training environment like this one. Although the global optimization techniques illustrates here can be applied to any problem suitable to be resolved using neural networks, I want to use the most famous and, maybe, most appreciated example: global optimization for Financial Forecasting.

Input Selection

If you're thinking to use neural networks to make financial forecasting, maybe you need the DTE, because you could train several networks on several stocks, in order to find the most predictable one (i.e. that one having less noise – why waste time and money on a noisily stock?). Moreover, you could train in parallel several networks having as input different financial indicators in order to find the best combination of them to obtain good performances. The message is: do not choose yourself the input patterns to use, but instead let Joone DTE do it for you.

Parameter Selection

You could also use the DTE to train different neural networks architectures, in order to verify the influence of a different number of hidden neurons, a different weights initialization, a different initial learning rate, a different input temporal window' size, or... a whatever combination of them.

Committee of Experts

Moreover, you could train several neural networks on the same time series and, after that, make predictions by averaging the results of all the trained neural networks (or only using the fittest ones). If a single network could generate bad predictions, maybe a committee of experts could make better predictions. You could also decide to not trade in presence of conflicting outputs, avoiding in this case to make potentially losing trades by entering in the market when it is not very predictable.

Features

  • Console based interface
  • Centralized control
  • The Final results are logged into a XML file to permit to analyze the results from a custom program/script
  • The training process scale almost linearly adding more machines
  • No manual configuration needed to add or remove machines
  • Possibility to add or remove machines dynamically during the training process
  • The overall process controlled by XML parameters
  • Latest SUN's Jini 2.0 compliant
  • Easily expandible with new features/algorithms/control modules, etc.

How does it work?

The Jini Technology come to our aid, providing a mechanism to implement distributed parallel processing. It's name is the JavaSpaces technology. The core engine has been conceived with many features that make it suitable to be used easily in a distributed environment like Jini/JavaSpaces, in fact:

  • All the classes of the joone engine are serializable (the JavaSpaces transport mechanism is based on serialization)
  • The input synapses have a buffering mechanism to store input data, so a neural network can be transported along with its training data
  • The possibility to use a script to control the behaviour of a neural network represents a powerful mechanism to control remotely its training and validation phases (i.e. a neural network is transported along with its own control logic)

The underlying concept is very simple: there are mainly three components that take part in the whole process:

  1. The Jini based services: JavaSpaces, the Transaction Manager and a Lookup Service
  2. Several Workers running on different machines connected to a LAN
  3. A Master controller, or Job runner

Both the Workers and the Master use the Lookup Service to discover the JavaSpaces and the Transaction Manager services, and register themself as listeners of the JavaSpaces in order to be notified when a neural network is available on it to be elaborated.

The Master, by reading a parameter file written in XML, generates all the tasks to perform (i.e. the neural networks to train); the generated tasks are sent to the JavaSpaces to make them available to all the running Workers. After that the Master waits for a JavaSpaces event to be notified about the availability of trained neural networks.

The Workers are notified about the presence of a new neural network to elaborate. At the above notification event, one of the ready Workers 'takes' that neural network, train it on the remote machine where it lives, and at the end sends back the results to the JavaSpaces. After that the Worker returns in a 'suspended' state waiting for a new notification

When a trained network is available on the JavaSpaces, the Master is notified, so it can 'take' that network from the JavaSpaces and store it on the file system, writing also the results to a log file in XML format

To learn more, read the DTE Complete Guide, and download the binary packages.

Credits

The Joone DTE uses the following external packages:

We want to thank all the authors and contributors of the above packages. Please read the respective licenses contained in the distribution package.

A robust alternative to Jini: GreenTea

If you want to try an alternative to the Jini/JavaSpaces based DTE, download the GTJoone package, a porting of the previous version of the DTE built on GreenTea, a pure Java Peer-to-Peer (P2P) based Grid OS platform.

Resources