We analyze the viscous damping of r-mode oscillations of compact stars, taking into account nonlinear viscous effects in the large-amplitude regime. The qualitatively different cases of hadronic stars, strange quark stars, and hybrid stars are studied. We calculate the viscous damping times of r-modes, obtaining numerical results and also general approximate analytic expressions that explicitly exhibit the dependence on the parameters that are relevant for a future spin-down evolution calculation. The strongly enhanced damping of large-amplitude oscillations leads to damping times that are considerably lower than those obtained when the amplitude dependence of the viscosity is neglected. Consequently, large-amplitude viscous damping competes with the gravitational instability at all physical frequencies and could stop the r-mode growth in case this is not done before by nonlinear hydrodynamic mechanisms.