LITERATURE SURVEY
on
MODULAR NEURAL NETWORKS

• Task allocation for multiple-network architectures : (Drabe, T. Bressgott, W., 1997)
• Genetic task clustering for modular neural networks : (T.; Bressgott, W.; Bartscht, E., 1996)
• Evolving a modular neural network-based behavioral fusion using extended VFF and environment classification for mobile robot navigation : (Kwang-Young Im; Se-Young Oh; Seong-Joo Han)
• A modular neural network for control of mobile robots : (S. Yamaguchi and H. Itakura, in Proc. Int. Conf. Neural Information Processing, vol. 2, 1999, pp. 661-666.)
• Learning piecewise control strategies in a modular neural network architecture, (Jacobs, R.A. Jordan, M.I., 1993)
• A Modular Neural Network Architecture with Additional Generalization Abilities for Large Input Vectors (Albrecht Schmidt,Zuhair Bandar)
• Modular Neural Networks: A Survey (GASSER AUDA and MOHAMED KAMEL)
• A Modular Neural Network Approach to Autonomous Navigation (Phd. Thesis, Ian Davis, CMU, 1996)
• Modeling and evolutionary learning of modular neural networks (Q. F. Zhao)
• The Design and Evolution of Modular Neural Network Architectures (Bart L.M. Happel, Jacob M.J. Murre, 1994, Neural Networks)

Article titles from citeceer

Task allocation for multiple-network architectures : (Drabe, T. Bressgott, W., 1997)

Genetic task clustering for modular neural networks : (T.; Bressgott, W.; Bartscht, E., 1996)

• Abstract: This paper introduces a method to cluster subtasks of a complex task to be learned by neural networks. The main objectives are minimization of the epochs needed to train the clusters up to a specified error limit and to maximize the generalization rates of the trained networks. To cope with the combinatorial optimization problem involved a genetic algorithm is developed. It starts with a set of random clusters which are trained up to a stopping criterion. Based on a fitness measure derived from the training results, new clusters are assembled using genetic operators. The approach of this work is relevant for all problems being decomposable into distinct subtasks, for example in robotics and plant control, where piecewise control strategies can be learned, and in image processing. Simulations for letter recognition indicate that the method is superior to both training all tasks on large monolithic network and to training randomly assigned clusters on small modular networks
• Abstract: Modular neural architectures pose the problem to find those subtasks of a complex task which can be efficiently trained together on the same network. We attack the involved combinatorial optimization problem by a genetic algorithm. For comparison a monolithic network and a modular architecture with random task distribution are considered. Letter recognition experiments show that the proposed method yields considerably better results concerning final convergence speed, generalization and completeness of solutions.

• Modularity is an indispensible tool for attacking complex problems in science and engineering.
• Modular networks have advantages as,
  • increased learning and relearning speed
  • improved generalization
  • better usability
  • interpretability
  • easier hardware implementation
• Problems:
  • task-decomposition, splitting task into subtasks.
  • training tasks in network modules
  • combination of network outputs
• Some solutions:
  • Maximum likelihood procedures
  • Mixture model estimations
  • Bayesian methods
  • Non-parametric techniques
  • Fuzzy systems
  • Dynamic modularization
  • multi-sieving networks??
• This work : finding tasks which can be jointly trained on the same network module..
• Problems:
  • Given a set of networks, not all combinations of subtasks are learnable
  • Minimization of the number of epochs.
  • Generalization rates for the clusters should be maximized.
  • Clustering should be complete and non-overlapping.
• Genetic algorithm:
  • Population of random clusters
  • The clusters of each generation are assigned fitness values based on seperate neural network training

  

• Each individual made up of Vmax clusters Ci, each Ci contains bits which determine if corresponding cluster is a part of a sub-task.

  

• There is a term which penalizes tasks not being one of an individuals active clusters.
• Mutation gene can be used to switch task or active bits to ON or OFF, and it can be mutated.
• Algorithm is applied to a letter recognition task. 20000 unique patterns. Recognition of 26 letters is the complete task and recognition of a single letter is a sub-task.
  • 16 numerical attributes
  • 26 letter recognition is complete task
  • recognition of single letter is subtask
  • backprop FFN is used with 2 hidden layers
  • population size is 8
  • max # of tasks = 10
  • 10% ON initially
  • after each generation, network weights are reset to intial values
  • 1000 generations

Evolving a modular neural network-based behavioral fusion using extended VFF and environment classification for mobile robot navigation : (Kwang-Young Im; Se-Young Oh; Seong-Joo Han)

• Abstract: A local navigation algorithm for mobile robots is proposed that combines rule-based and neural network approaches. First, the extended virtual force field (EVFF), an extension of the conventional virtual force field (VFF), implements a rule base under the potential field concept. Second, the neural network performs fusion of the three primitive behaviors generated by EVFF. Finally, evolutionary programming is used to optimize the weights of the neural network with an arbitrary form of objective function. Furthermore, a multinetwork version of the fusion neural network has been proposed that lends itself to not only an efficient architecture but also a greatly enhanced generalization capability. Herein, the global path environment has been classified into a number of basic local path environments to which each module has been optimized with higher resolution and better generalization. These techniques have been verified through computer simulation under a collection of complex and varying environments.
• obstacle avoidance, goal-seeking, and free-space attraction
• 
• In order to get past experience, sensor data S(t-4),S(t-2) used as inpu to the network.
• The various sensing patterns encountered along the way are classified or clustered into many local environments, which include the local minimum and narrow passage situations dealt with previously. Then, each behavior module is trained only for the corre-sponding local environment and a different permutation of these local environments can generate a great variety of navigation paths. Thus, this modular approach can adapt better to different environments than the monolithic case.
• Environment is classified according to adaptive-resonance-theory like network.

A modular neural network for control of mobile robots : (S. Yamaguchi and H. Itakura, in Proc. Int. Conf. Neural Information Processing, vol. 2, 1999, pp. 661-666.)

• Abstract: A new modular neural network architecture and its learning algorithm are proposed for a mobile robot controller. The learning algorithm for the proposed new network architecture is based on a feedback error learning procedure, which requires a feedback controller for training processes. It is not so easy, however, to obtain a robot feedback controller, when the robot control task is much more complex. In the present architecture, the complex robot control task is divided into a couple of small simple tasks, each of which is assigned to each of small network modules, respectively. By dividing the complex task, the simple feedback controllers are assigned to the network modules. Therefore, the neural network in each module can be trained by the feedback error learning scheme. The command to the robots is the weighted sum of the outputs of the modules. The weights for each module are obtained from a neural network which is one of the network modules in our proposed architecture. The present neural network architecture and learning algorithm are applied to a set of several robot controllers, whose task is to push a large box to a goal. It is confirmed through computer simulation experiments that the algorithm can train the robot controller skillfully.
• Target is a sensor value whom robot wants to obtain.Then the controller output is the command to obtain the target value of each robot controller.
• Complex task is divided into a number of simple tasks...Thus feedback error learning can be easily applied to simple tasks.
• 2 types of networks. Each task is assigned to expert network. After obtaining expert network output values, and the command to the robot is calculated by using the gating network output values.
• Tasks is to push a box cooperatively.

  

• v = sum(Gi . Vi)
• The language of the paper, and simplicity and clearance is not good.

Learning piecewise control strategies in a modular neural network architecture, (Jacobs, R.A. Jordan, M.I., 1993)

• Abstract: The authors describe a multinetwork, or modular, neural network architecture that learns to perform control tasks using a piecewise control strategy. The architecture's networks compete to learn the training patterns. As a result, a plant's parameter space is adaptively partitioned into a number of regions, and a different network learns a control law in each region. This learning process is described in a probabilistic framework and learning algorithms that perform gradient ascent in a log-likelihood function are discussed. Simulations show that the modular architecture's performance is superior to that of a single network on a multipayload robot motion control task.
• non-linear control system design. Design a piecewise controller that uses different control laws when the plant is operating in different conditions.
• Gain-scheduling: ...? usage of local models
• When network learns a task, when another task is given to be learnt, network is incompatible with the first one. Temporal crosstalk: alternating between tasks, slow learning rate.
• Competition different network modules learn different training patterns.
• Expert networks compete to learn the training patterns.
• Gaiting network mediates this competition.
• gaiting network output number = expert network module number.
• payload identity is given to gating network as INPUT!!!!!!

A Modular Neural Network Architecture with Additional Generalization Abilities for Large Input Vectors (Albrecht Schmidt,Zuhair Bandar)

• Abstract: This paper proposes a two layer modular neural system. The basic building blocks of the architecture are multilayer Perceptrons trained with the Backpropagation algorithm.Due to the proposed modular architecture the number of weight connections is less than in a fully connected multilayer Perceptron.The modular network is designed to combine two different approaches of generalization known from connectionist and logical neural networks; this enhances the generalization abilities of the network.The architecture introduced here is especially useful in solving problems with a large number of input attributes.

Modular Neural Networks: A Survey (GASSER AUDA and MOHAMED KAMEL)

• Abstract: Modular Neural Networks (MNNs) is a rapidly growing field in artificial Neural Networks (NNs) research. This paper surveys the different motivations for creating MNNs: biological, psychological, hardware, and computational. Then, the general stages of MNN design are outlined and surveyed as well, viz., task decomposition techniques, learning schemes and multi-module decision-making strategies. Advantages and disadvantages of the surveyed methods are pointed out, and an assessment with respect to practical potential is provided. Finally, some general recommendations for future designs are presented.
• Task decomposition:
  • Naturally defined modules (it is difficult in most cases)
  • Modularizing Learning:
    • Unsupervised feature extraction decrease the computational expense by decreasing the number of weights which are trained in Supervised Learning stage.
    • Supporting supervised learning by quantization: Transforming the nonlinear task into a linear one using a self-organizing quantization module. Task complexity is decreased instead of wieght number.
    • Decomposing the learning set: First, easy to learn data set is applied, then whole data set with difficult data is supplied.
    • Minimizing Interactions between weights: ??
  • Modularizing Structure: The application task is decomposed so that it would be distributed over several structurally separate modules. Decomposing a classi cation task involves clustering (grouping) the categories into a certain number of groups. Then, a separate classifier (neural module) can be applied to every group.
    • Task allocation using genetic algorithms : (papers)
      • Task Allocation for multiple-network architectures
      • Genetic Task Clustering for Modular Neural Networks
    • Faster and more systematic ways: (unsupervised clustering)
      • Self-Organized Maps
      • ART
      • K-means algorithms
      • LVQ ?
    • Module size is very important problem. No constraint is very bad since one module can dominate whole. If module sizes are taken same, NN will lose flexibility.
• Training Modules: In modular learning cases, training is performed sequentially. However, in Modularizing Structures case, modules are trained parallel, independent of each other.Some Modular Neural Networks (MNNs) adaptively define modules during learning process.
  • Using LVQ, a new branch is formed when similarity measure is less than a threshold.
  • SOM assigns each new training sample set to a sub-net.
  • As learning samples are introduced, the weights of the expert networks are modified so as to reduce the sum of the squared error between the output of the system and the desired output. The one which comes closer to producing the desired output is considered a winner. If the error is significantly improved, the gating modules will perform task-decomposition by assigning the input pattern to this winner expert. If the performance does not improve, the outputs of the gating network will approach neutral values, i.e., this sample is not yet clearly as- signed to one of the modules.
  • Merging different module weights, activation or nodes..
• Multi-model decision making:
  • complete decoupling: Decision boundaries must be defines. According to most activated module. There should be mechanisms not to activate wrong module at the same time.
  • competitive : A 2 layer network, each output node represents one module. Using unsupervised mechanisms (ART,SOM), complex tasks cannot be partitioned well. Different solutions, high vigilence ART...
  • cooperative : ... A large number of cooperative techniques.

A Modular Neural Network Approach to Autonomous Navigation (Phd. Thesis, Ian Davis, CMU, 1996)

• Abstract:
  • In this thesis we present both a novel neurla network paradigm and an approach for solving sensing and control tasks for mobile robots using this neural network paradigm. Real world tasks have driven the evolution of this methodology and its components, and we apply our methodology successfully to two robotic applications. we conclude that for some tasks, our novel modular neural network approach can achieve comparable or better performance than a traditional monolithic neural network in a much reduced training time.
  • We present the MAMMOTH (Modular Architecture Multi-Modality Theory) neural network paradigm, which is both an architectural blueprint and a training system for combining the internal representations of multiple neural networks each of which is trained to recognize different kinds of features. The modules in a MAMMOTH system are designed to provide functional decomposition of a task. That is, each module performs part of the task for a given in put, and the higher levels of the MAMMOTH network combine the results to get a solution; this is different from many modular neural network techniques in which the higher level arbitrates between complete answers provided by the modules.
  • We apply MAMMOTH networks to several tasks, which include vision for the alignment of an aircraft inspection robot, on-road navigation, and cross-country navigation. Through these tasks we see general applicability of MAMMOTH to real world sensing and control tasks. Ultimately, the greatest benefit of MAMMOTH is that for some tasks, low level features can be learned separately and in parallel, speeding the entire training process for a neural system, without losing any performance.
  • FUNCTIONAL decomposition, NOT TASK decomposition.

Modeling and evolutionary learning of modular neural networks (Q. F. Zhao)

• Abstract:
  • In the last decade, a number of neural network models have been proposed in the literature. Some of them have been successfully incorporated in different intelligent information processing systems. Among these models, a group of most successful ones are the modular neural networks (MNNs). This paper introduces a general model of MNNs, and proposes a neural network tree (NNTree) model. An evolutionary algorithm
    is also given for designing the NNTrees. The usefulness of the NNTrees and the effectiveness of the learning algorithm are verified through experiments with a digit recognition problem.
• The most distinctive feature : Modular Netork TREE...

The Design and Evolution of Modular Neural Network Architectures (Bart L.M. Happel, Jacob M.J. Murre, 1994, Neural Networks)

• Abstract:
  • To investigate the relations between structure and function in both artificial and natural neural networks, we present a series of simulations and analyses with modular neural networks. We suggest a number of design principles in the form of explicit ways in which neural modules can cooperate in recognition tasks. These results may supplement recent accounts of the relation between structure and function in the brain. The networks used consist out of several modules, standard subnetworks that serve as higher-order units with a distinct structure and function. The simulations rely on a particular network module called CALM (Murre, Phaf, and Wolters, 1989, 1992). This module, developed mainly for unsupervised categorization and learning, is able to adjust its local learning dynamics. The way in which modules are interconnected is an important determinant of the learning and categorization behaviour of the network as a whole. Based on arguments derived from neuroscience, psychology, computational learning theory, and hardware implementation, a framework for the design of such modular networks is laid-out. A number of small-scale simulation studies shows how intermodule connectivity patterns implement neural assemblies (Hebb, 1949) that induce a particular category structure in the network. Learning and categorization improves as the induced categories are more compatible with the structure of the task domain. In addition to structural compatibility, two other principles of design are proposed that underlie information processing in interactive activation networks: replication and recurrence.
  • Because a general theory for relating network architectures to specific neural functions does not exist, we extend the biological metaphor of neural networks, by applying genetic algorithms (a biocomputing method for search and optimization based on natural selection and evolution) to search for optimal modular network architectures for learning a visual categorization task. The best performing network architectures seemed to have reproduced some of the overall characteristics of the natural visual system, such as the organization of coarse and fine processing of stimuli in separate pathways. A potentially important result is that a genetically defined initial architecture cannot only enhance learning and recognition performance, but it can also induce a system to better generalize its learned behaviour to instances never encountered before. This may explain why for many vital learning tasks in organisms only a minimal exposure to relevant stimuli is necessary.
• Just as learning in neural networks is an autonomous self-organizing method for finding suitable weight values, we will use an autonomous self-organizing procedure to find optimal network architectures and parameter values.
• Biological Adaptation:
  • Evolution is the first level, Ontogenesis, or the development of an individual organism, is the second lower level of adaptation, operating on a time scale measured in years, months, or weeks. Learning forms the third level of adaptation of an organism to its specific environment. Learning can be seen as a fine-tuning of the neural structures which were phylogenetically and ontogenetically established. Learning processes operate at an even smaller time scale of minutes or seconds.
  • At the fourth level, we have the dynamics of neural activation which operate within milliseconds and which form a fast and specific adaptation mechanism that enables an organism to act on the demands of the moment.
  • An important conclusion was that learning may provide a decisive evolutionary advantage. Genes enhancing learning in organisms have a higher chance of survival than less adaptive variants.

GENETIC ALGORITHMS

GAVEL - a new tool for genetic algorithm visualization (Hart, E. Ross, P. 2001)

• Abstract:
  • This paper surveys the state of the art in evolutionary algorithm visualization and describes a new tool called GAVEL. It provides a means to examine in a genetic algorithm (GA) how crossover and mutation operations assembled the final result, where each of the alleles came from, and a way to trace the history of user-selected sets of alleles. A visualization tool of this kind can be very useful in choosing operators and parameters and in analyzing how and, indeed, whether or not a GA works. We describe the new tool and illustrate some of the benefits that can be gained from using it with reference to three different problems: a timetabling problem, a job-shop scheduling problem, and Goldberg and Horn's long-path problem. We also compare the tool to other available visualization tools, pointing out those features which are novel and identifying complementary features in other tools.