This article studies exponential families E on finite sets such that the information divergence D(P∥E) of an arbitrary probability distribution from E is bounded by some constant D>0. A particular class of low-dimensional exponential families that have low values of D can be obtained from partitions of the state space. The main results concern optimality properties of these partition exponential families. The case where D=log(2) is studied in detail. This case is special, because if D<log(2), then E contains all probability measures with full support.