The Distributed Training Environment (DTE)
The idea that underlies Joone is to build a professional environment to create, train, distribute
and use neural networks in a simple but powerful way. To accomplish this final goal, we have built a distributed
training environment to train in parallel many neural networks.
Our dream is to provide a professional framework to permit everyone to build neural networks that could be trained on a
lot of common problems of the real life suitable to be resolved by the neural networks (spoken and handwriting recognition,
stock quotes forecasting, credit loan assessment, etc.), and then distribute these trained nets to interested users (final users, sales force, etc.)
that can use the trained neural networks on their devices (desktops, palmtops, PDA, etc.).
Thanks to this framework it would be possible to originate a 'Neural Network Farm' that could
fabricate neural networks by training them in parallel on several machines, and finally distribute them to the final users
(e.g. for free, or as prepaid service), to 'train once, run anywhere' a neural network without being worried about the HW or SW platform owned by the user.
Why a distributed environment?
When we want to use the neural network technology to resolve a complex job, training only
one neural network, often, doesn't resolve the problem, because the net can fall onto a local minima without finding
the best results; thereby what must be developed is a mechanism to train many neural nets in parallel on the same problem,
on several machines, governing the whole process from a central point.
Joone has an own complete and professional framework where it's possible to create a neural net (with the GUI editor),
and then teach several copies of it on many machines where some agents are running
(they also based on joone's engine, of course). Those agents (or Workers) reveive a copy of the neural net and train it,
sending back the outcome (i.e. the trained net) to the starting machine where the user, or an automatic script,
can control all the results of the distrubuted training process, being so able to choose, at the end of the process,
the best net, by applying some discriminating mechanism: e.g. choosing the net with the lowest error, applying a genetic algorithm, etc.
Maybe most of you will have already thought about some useful application suitable to be resolved by the DTE, but I think that it could be
useful to give some ideas about possible applications of a distributed training environment like this one.
Although the global optimization techniques illustrates here can be applied to any problem suitable to be resolved using neural networks,
I want to use the most famous and, maybe, most appreciated example: global optimization for Financial Forecasting.
If you're thinking to use neural networks to make financial forecasting, maybe you need the DTE, because you could train several
networks on several stocks, in order to find the most predictable one (i.e. that one having less noise – why waste time and money on a noisily stock?).
Moreover, you could train in parallel several networks having as input different financial indicators in order to find the best combination of
them to obtain good performances. The message is: do not choose yourself the input patterns to use, but instead let Joone DTE do it for you.
You could also use the DTE to train different neural networks architectures, in order to verify the influence of a different number of hidden neurons,
a different weights initialization, a different initial learning rate, a different input temporal window' size, or... a whatever combination of them.
Committee of Experts
Moreover, you could train several neural networks on the same time series and, after that, make predictions by averaging the results of all the
trained neural networks (or only using the fittest ones). If a single network could generate bad predictions, maybe a committee of experts could make
better predictions. You could also decide to not trade in presence of conflicting outputs, avoiding in this case to make potentially losing trades by
entering in the market when it is not very predictable.
- Console based interface
- Centralized control
- The Final results are logged into a XML file to permit to analyze the results from a custom program/script
- The training process scale almost linearly adding more machines
- No manual configuration needed to add or remove machines
- Possibility to add or remove machines dynamically during the training process
- The overall process controlled by XML parameters
- Latest SUN's Jini 2.0 compliant
- Easily expandible with new features/algorithms/control modules, etc.
How does it work?
The Jini Technology come to our aid, providing a mechanism to implement distributed parallel processing.
It's name is the JavaSpaces technology. The core engine has been conceived with many features
that make it suitable to be used easily in a distributed environment like Jini/JavaSpaces, in fact:
- All the classes of the joone engine are serializable (the JavaSpaces transport mechanism is based on serialization)
- The input synapses have a buffering mechanism to store input data, so a neural network can be transported along with its training data
- The possibility to use a script to control the behaviour of a neural network represents a powerful mechanism to control remotely its training
and validation phases (i.e. a neural network is transported along with its own control logic)
The underlying concept is very simple: there are mainly three components that take part in the whole process:
- The Jini based services: JavaSpaces, the Transaction Manager and a Lookup Service
- Several Workers running on different machines connected to a LAN
- A Master controller, or Job runner
Both the Workers and the Master use the Lookup Service to discover the JavaSpaces and the Transaction Manager services, and register themself as listeners
of the JavaSpaces in order to be notified when a neural network is available on it to be elaborated.
The Master, by reading a parameter file written in XML, generates all the tasks to perform (i.e. the neural networks to train); the generated tasks are sent
to the JavaSpaces to make them available to all the running Workers. After that the Master waits for a JavaSpaces event to be notified about the
availability of trained neural networks.
The Workers are notified about the presence of a new neural network to elaborate. At the above notification event, one of the ready Workers 'takes' that
neural network, train it on the remote machine where it lives, and at the end sends back the results to the JavaSpaces. After that the Worker returns in a 'suspended'
state waiting for a new notification
When a trained network is available on the JavaSpaces, the Master is notified, so it can 'take' that network from the JavaSpaces and store it on the file system,
writing also the results to a log file in XML format
To learn more, read the DTE Complete Guide, and download the binary packages.
The Joone DTE uses the following external packages:
We want to thank all the authors and contributors of the above packages. Please read the respective licenses contained in the distribution package.
A robust alternative to Jini: GreenTea
If you want to try an alternative to the Jini/JavaSpaces based DTE, download the GTJoone package,
a porting of the previous version of the DTE built on GreenTea, a pure Java Peer-to-Peer (P2P) based Grid OS platform.