shift the output and have no effect on the derivatives of the output w.r.t. to the input.

A feedforward architecture is always convex w.r.t. each input variables if every activation function is convex and the weights are constrained to be either all positive or all negative.  A feedforward architecture with positive weights is a monotonically increasing function of the input for any choice of monotonically increasing activation function.  The weights of a feedforward architecture must be constrained for the output of a feedforward network to be bounded. The bias terms in a network simply shift the output and have no effect on the derivatives of the output w.r.t. to the input.

Part II Sequential Learning

Chapter 8 Advanced Neural Networks