SIGMAPI V 1.2 - 27th December, 2002 Mail: ewald72@yahoo.com Copyright (C) 2002 Ewald Kraemer mailto_sigmapi@yahoo.com WWW: sigmapi.sourceforge.net
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
SIGMAPI 1.2/1.2.1 TO 1.3 COMPATIBILITY
Nothing changed in the network file format
SIGMAPI 1.1 TO 1.2/1.2.1 COMPATIBILITY
Version 1.2 has local initial-range parameters. They are included at the end of
the connection declaration, 'S'
By default, append '-1 1' or '-1.0 1.0', so
S 0 3 1 1 1 0.175532 -5 5 0.5 0.5 10 0 0 1.2 0.6 1 0
becomes this:
S 0 3 1 1 1 0.175532 -5 5 0.5 0.5 10 0 0 1.2 0.6 1 0 -1 1
However, SigmaPi 1.2 will recognize the old format and automatically uses
-1 1 as default.
SIGMAPI 1.0 TO 1.1 COMPATIBILITY
Nothing changed in the network file format, so 1.1 is completely compatible with 1.0
SIGMAPI 0.9 TO 1.0 COMPATIBILITY
Nothing changed in the network file format, so 1.0 is completely compatible with 0.9. If you don't use the new activation function, 0.9 can even read 1.0 generated files.
SIGMAPI 0.8 TO 0.9 COMPATIBILITY
Version 0.9 introduces training with momentum parameters. To make network files
readable, extend every synapse declaration (lines beginning with S) with 0.0
S 0 3 1 1 1 0.175532 -5 5 0.5 0.5 10 0 0 1.2 0.6 1 0
should become
S 0 3 1 1 1 0.175532 -5 5 0.5 0.5 10 0 0 1.2 0.6 1 0 0.0
This declares a momentum of 0.0 (none), which is equivalent to the old networks.
SIGMAPI 0.7 TO 0.8 COMPATIBILITY
Version 0.8 uses different gate activation function. These are stored in the network file.
To make 0.7 network readable, insert 1 1 1 after the third parameter of all lines beginning
with N. Also insert the three corresponding FSE parameters (0.0 by default) at the end of the
line. So the line
N 3 3 0 0 0 0.1
should be changed to:
N 3 3 0 1 1 1 0 0 0.1 0.0 0.0 0.0
This tells the simulator to use the logistic activation function for all gates, which was
the case in versions 0.5 to 0.7 and to use no FSE on the gates.
SIGMAPI 0.6 TO 0.7 COMPATIBILITY
Version 0.7 uses more than one pattern-sets, so it needs to know the pattern-id of a signal.
All lines beginning with F have to be changed. The pattern ID has to be inserted as the fourth
parameter. By default, ID 1 should be used
So the line
F I 1 2 /home/signals/noisy.wav
should become
F I 1 2 1 /home/signals/noisy.wav
SIGMAPI V 0.5 TO 0.6 COMPATIBILITY
There is no mathematical equivalence. The networks are almost the same if used without forget-gates. However, the forget signal will be constantly 0.5, so a retraining will be necessary. The files are readable, so you won't have to change them this time.
SIGMAPI V 0.4 TO 0.5 COMPATIBILITY
SigmaPi 0.5 uses a different neuron model. However, it is possible to convert V 0.4 network files
by using an ASCII editor. All lines beginning with a 'S' (this stands for Synapse) need to be extended
by the integer 0 exactly AFTER S, leaving a space.
For instance,
S 3 1 1 1 0.175532 -5 5 0.5 0.5 10 0 0 1.2 0.6 1 0
should become
S 0 3 1 1 1 0.175532 -5 5 0.5 0.5 10 0 0 1.2 0.6 1 0
The second thing to do is to remove the 'Neuron Type' flag, which is the fourth parameter after 'N'.
For instance,
N 1 3 0 2 1 0 0
should become
N 1 3 0 2 0 0
Furthermore, all lines beginning with 'G' (the Gains) can be deleted (this is not necessary, since
SigmaPi will leave them out next time the network is saved).
Emulating old networks:
While the new neuron model is not an extension of the old one, it is possible to emulate old networks
(but you should really try the new InGate, OutGate mechanism, it kicks ass) as long as there are no
multiplicative units. This is simply done by loading an old network which has been converted. All
connection type should be zero now. Now create a unit with a constant nonzero output (a gaussian
with no inputs should be good, it delivers 1.0 all the time). Connect this unit's output to the
Input Gates of ALL neurons, using a constant connection of 9.999999 (the highest possible). This will
cause the Onput Gates to open constantly. Do the same with the Output Gates, the LOCK all this
connections (if you don't do this, SigmaPi will perform a training on them, too).
Now you have units who behave like the primitive ones of SigmaPi 0.4 or before. However, the gains are
constant. You will probably have to do some re-training.
SIGMAPI V 0.3 TO 0.4 COMPATIBILITY
The network files of SigmaPi 0.4 need an additional neuron description parameter (the 'Flat Spot
Elimination') fse. If fse==0, the neuron is equivalent to the 0.3 version.
To convert old networks, open the 0.3 network file with an ASCII editor, then extend all lines beginning
with the char 'N' with a space then type 0.0 (the added 0.0 denotes a noise amplitude of zero).
For instance,
N 3 3 0 1 0 0.1
should become
N 3 3 0 1 0 0.1 0.0
The network file should work now
SIGMAPI V 0.1/0.2 TO 0.3 COMPATIBILITY
While SigmaPi 0.1 network files were compatible to 0.2, this is no longer the case with 0.3.
Because the NOISE parameter was introduced, SIGMAPI 0.3 won't read your old network files any longer.
To convert the old files, open the network file with an ASCII editor, then extend all lines beginning
with the char 'N' with a space then type 0.0 (the added 0.0 denotes a noise amplitude of zero).
For instance,
N 3 3 0 1 0
should become
N 3 3 0 1 0 0.0
The network file should work now
INSTALLATION
You will need Trolltech QT 3.x for this. It is available from www.trolltech.com. At the time of this version of SIGMAPI, only the X11 and Embedded QT versions were available for free. However, there are commercial versions of QT3.x for Windows and Apple OSX. In order to get SIGMAPI running, it needs multithreading support, which is already enabled in the binary versions for OSX and Windows. On Unix, you should check whether libqt-mt is available, if not you will have to build your own multithreaded QT from the source.
To compile QT3/X11 on UNIX, do the following:
Get the QT3 development toolkit from www.trolltech.com
Unpack the tar.gz file, set $QTDIR to the directory
Set QMAKESPEC to $QTDIR/mkspecs/linux-g++ or to the corresponding platform/compiler directory
Extend PATH to $QTDIR/bin
Execute 'configure -thread' to configure the Makefile for multithreading-QT
Start 'make' and go for a Coffee
You don't need to install QT on your system, so no need for 'make install' now.
Go into the SIGMAPI directory
Type ./configure - if QT works, you will see a Makefile thereafter. You might want to edit it
to optimize code. Changing -O2 to -O4 works on every processor. But there are also processorspecific
options - your compiler's manual will contain more information.
If your QT is the QT/E (Embedded) version:
- Change -lqt-mt to -lqte-mt.
- Extend the C/CXXFLAGS by -DQWS and -fno-rtti.
- After the build, don't start SIGMAPI from X11, it may corrupt your display completely.
- You will need an active framebuffer with at least 800x600 pixels. On Linux, this is done with the vga=xxx setting on the bootloader prompt. The bootscreen should start with the 'Framebuffer Penguin' visible on the text screen. Some Linux-providers have their framebuffer already enabled. You will also need a setting for $QWS_MOUSE_PROTO. MouseMan is a good one for USB mouses. Kill GPM if it runs, since it will corrupt the graphics. In any case, don't start a framebuffer application from X11, this may corrupt your display If it is the first framebuffer application on your system, it needs to run the QT Windowserver. Do this by starting 'sigmapi -qws'
Now compile by typing 'make' - it will need a couple of minutes. Move the binary to a directory in your $PATH
That's all. If something goes wrong, please mail to: mailto_sigmapi@yahoo.com
WORKING WITH SIGMAPI
Probably you will have to read the paper to understand the method. The papers on the bibliography
are very useful, especially the Riedmiller, Zipser, Williams and Gers papers.
If you already know the RTRL method, the extension to LSTM is basically done by changing the neuron
model.
The training method is extended to learning rates local in space and time. They are adapted with
the update sign-change method (like in RPROP, Delta-Bar-Delta, etc)
IN ANY CASE: You should download the manual from the SigmaPi Homepage. SigmaPi is not a dumb end-user system. It allows you to change every single parameter, therefore is is complicated. If it behaves strange this is probably YOUT fault :) To reset it to the default state, you should quit it and delete the default.net and default.cfg files from the directory where you started SigmaPi.
DEMO
I've included a demonstration network 'smooth.net' with some neurons, top/left is the input neuron, top/right the output. The network was trained to approximate clear-shifted4.wav from noisy.wav.
The other demonstration is a sequenced XOR. It creates the XOR of the last two inputs in the sequence. The network classified nine out of ten pairs correctly.
CHANGES 0.1 TO 0.2
- The gui was redesigned, some widget groups were repositioned
- Training parameters can be overwritten using the 'Reset all' buttons in the 'Algorithm' tab
- The two algorithm families, gradient and gradient sign, are moved into one. The gradient is squashed using the tanh function. The amount of squashing (the 'gain' of this tanh function) defines the tendency to use the sign or the gradient SigmaPi will be able to read your old networks. The squashing amount parameter will be set to 1.0 if it is not defined in the network file.
- A Validation signal was introduced, furthermore a validation error display
- WAV file output now allows to define the sampling rate.
- New neuron icons
- Minor Bugfixes
CHANGES 0.2 TO 0.3
- Neuron output noise: the noise amplitude is added to the neuron potential AFTER the potential was gain-scaled, burt BEFORE the activation function. out=act((gain*net)+noise)
- Added the possibility to overwrite all neurons with the neuron default values from 'Editor/Defaults'. This makes it easier if you want to change the noise, the activation, etc.
- Global scaling value for learning rates allows learning rate fine-tuning during training
- Reduced memory consumption The standard float type 'double' (8Byte) was changed to 'float' (4Byte)
CHANGES 0.3 TO 0.4
- Flat spot elimination added. The fse parameter increases the absolute value of the derivative of the activation function. It the derivative is >=0, fse is added, else it is subtracted. This will avoid the negative influence of saturated neurons on the training speed.
CHANGES 0.4 TO 0.5
SigmaPi 0.5 uses the LSTM Neuron Model (Long Short-Term Memory, search the internet or
read the SigmaPi paper) to overcome RTRL limitations. Because of this, several things
are changed:
* There are three types of connections now: The conventional connections, the Input Gate
connections (they activate the neuron's 'Input Gate' to let external activation in
and the Output Gate connections (they active the 'Output Gate' and allow the neuron
to become active).
A Combobox in the left top of the Display defines the connection structure to work with.
* Both the 'Input Gate' signal and the 'Output Gate' signal are multiplied to the incoming
signals or to the internal neuron memory. Therefore, the multiplicative elements are
moved into the neuron, so there is no need for SigmaPi to emulate them. The only neuron
type it knows is the LSTM unit.
* The Gain of a unit emulated by the Output Gate signal. Therefore, gains have been removed
too.
* Because we have three types of connections now, the Algorithm Defaults Tab has three
subtabs for every type.
* A training cycle is seven times faster than in SigmaPi 0.4. This is possible because some
the 'Step Display' can be switched off now.
* Another benefit of Version 0.5 is the lower number of training cycles until the training
error target has been reached. This is of course only the case if the Gate connections are
used.
* The training window was extended by three switches and three sliders. The switches are
used to activate or deactivate the training on one of the connection types, the
sliders scale the corresponding global learning rate.
CHANGES 0.5 TO 0.6
- LSTM was extended to Forget-Gates. In version 0.5, the Input-Gate did also define the unit's forgetting behavior. This means that a high input gate not only passed input through to the memory, but also resetted the value inside of the memory. In this version, the two mechanisms are cleanly separated by introducing a fourth type of connection.
- Furthermore, a 'bug' was removed. It caused an input signal to loose two steps while passing through a unit. The new units only need one step. The change is documented in the paper. It is int the calculation of mem(t), which now used inp(t) instead of inp(t-1)
- The new connection types cause SigmaPi to need about 10% more processing power. However, the training needs less cycles now.
- The new connection type has it's own slider and on/off checkbox on the training screen.
CHANGES 0.6 TO 0.7
- Version 0.7 is able to use more then one input/target/validation signal per neuron. Use the pattern id selector (top left editor window) to select the pattern, then load the input/target/validation signals like in the version before During training, all pattern id's with at least one target signal will be learned. All pattern sets with at least one validation signal are validated. Pattern sets consisting only of input signals are not processed during training.
- In some cases LSTM needs a more precise gradient information. Old versions of SigmaPi did re-calculate the network after every cycle. However, each cycle contains many update-steps after which no recalculation was done. If the updates are small enough, this is no problem. However, it is sometimes a good idea to recalculate a certain number of steps. The 'Recalculation' slider allows this. For instance, a value of 2 means that after all parameters at time t were updated, the network's activations are recalculated at t-1 and t.
CHANGES 0.7 TO 0.8
- Version 0.8 is able to use other functions than logistic activation for the gate activation. Possible are the hyperbolic tangens and the gaussian bell.
- A Flat-Spot-Elimination parameter was introduced for every gate.
CHANGES 0.8 TO 0.9
- Version 0.9 introduces a momentum parameter. The momentum defines the influence of the last update. For instance, 0.1 means that 10% of the last update value will be added to the current update value. Of course, every single connection can have it's own momentum by selecting the connections and setting the parameter.
- A special training mode was implemented. I call it 'Careful Update'. In this mode, updates are made only if the local deviation has the same direction in two consecutive cycles. But be warned, this will slow down training, since many updates are calculated, but discarded.
CHANGES 0.9 TO 1.0
- This version can store and load the settings of the 'Algorithm' menu. With this, you can save and load different types of learning behavior. I've included an aggressive RPROP-like algorithm (rprop.alg) and an extremely careful and slow algorithm (slow.alg). After loading an algorithm, you should be aware that it won't change the connections by itself. Only connections generated from now-on will get the new settings. If you want to overwrite the existent connection's learning behavior, use 'Reset connections' (this will overwrite only the connections of the selected type) or use 'Set this algorithm' to overwrite the settings of all four connection types.
- Version 1.0 includes two new activation functions: the Mexican Hat and the first derivative of the Gaussian.
CHANGES 1.0 TO 1.1
- A new field 'Simultaneously Connect' in the 'Defaults' group allows to simultaneously repeat all connection-specific operations on different connection types. For instance, if you select IGate before creating the standard connections (select 'Con' in the Display-Bar) the topology created for the standard connections will be created for the Input-Gate connections, too. Network creation is much faster this way.
- You can overwrite the settings of a selected connection by loading an algorithm file. A new button 'Load algorithm' is available in the 'Parameter section'.
- Version 1.1 includes a statusbar counting the number of neurons and connections.
CHANGES 1.1 TO 1.2
- In versions 1.1 and before the initial connection range (the random value) was global for all connections of the network. Now it is local, every connection can have it's own init-range.
- Version 1.2 can read ASCII-numerical input. This means you can read files containing a sequence of values like: -1.0 1.1 .2 -.5 You can load the signal like a WAV-file, it is automatically recognized. The new format is also accepted as an output option, a new save-signal button was included.
- A split-pane was included to the user interface.
- A bug was removed - it caused a crash in the clear-network and load-network routines.
CHANGES 1.2 TO 1.2.1
- There was a severe error in the new signal object, SigmaPi 1.2 could not save WAV and ASCII files. I hope there are no more problems now.
CHANGES 1.2.1 TO 1.3
- The numeric error displays are colour-encoded. Green means error decrement, red means increment. This should make training easier.
- Early stopping checkboxes. If checked, the training algorithm stops after an error increment was detected.
- Since there were some problems with over/underflow, the new SigmaPi version uses double-precision floating point representation.
- A bug from the validation error display was removed.
LICENSE
SigmaPi is GPL-licensed. Read gpl.txt or go to the GNU homepage: http://www.gnu.org/licenses/licenses.html