In this paper a thermodynamically consistent phenomenological model for the anisotropic Mullins effect in filled elastomers is presented. The model takes into account anisotropic softening as well as permanent set. The formulation is based on an anisotropic three-dimensional softening criterion and a scalar damage function both formulated in terms of the principal stretches. The damage function describes the difference in stresses between the primary loading curve and unloading curve in uniaxial tension tests and is evaluated from experimental data. The predictive capabilities of the proposed model are examined in comparison to experimental data available in literature as well as to own experimental results on CR rubber presented in the paper. Good agreement with these experiments is observed. In particular, the characteristic S-shape of the stress softening curves is accurately captured.