A feedforward architecture is always convex w.r.t. each input variables if every activation function is convex and the weights are constrained to be either all positive or all negative. A feedforward architecture with positive weights is a monotonically increasing function of the input for any choice of monotonically increasing activation function. The weights of a feedforward architecture must be constrained for the output of a feedforward network to be bounded. The bias terms in a network simply shift the output and have no effect on the derivatives of the output w.r.t. to the input.
Part II Sequential Learning
Chapter 8 Advanced Neural Networks
