Suppose that the output of a network is the linear transformation of it input plus some output noise , i.e., , and both input and noise are Gaussian signals. Without losing generality, suppose that the network capacity is 1.
The optimal information transmition requires to choose the transformation which maximizes the information of the ouput about the input while keeping the output variance equal to the capacity. This can be easily solved by Lagrangian multiplier method through maximizing
in which the first term is the information of the output about the input and the second term is the Lagangian multiplier. The derivative of at the maximum is
The solution is and , i.e., the output activities are decorrelated.
The dynamics of feedforward connections is very similar to the feedback case. We are not going into the detail of that since our emphasis is on the feedback path. The dynamic stability and information optimization still hold. It is not difficult to show that the functions and shown earlier are still the Lyapunov function and Lagrangian function, respectively, for the joint dynamics of feedforward and feedback connections. The decorrelation of output activities is still the stable solution.