nt.number theory – When does the double coset representative for a congruence subgroup contain a conjugation class $ text {SL} _2 ( mathbb {Z})?

In the newspaper L p-adic functions and p-adic periods of modular forms, Greenberg / Stevens say that if $ sigma_l: = begin {pmatrix} l & 0 \ 0 & 1 end {pmatrix} $ is the usual Hecke operator at $ l $ double coset representative, and $ Gamma $ is the congruence group $ Gamma_1 (N) $then $ g sigma_lg ^ {- 1} $ continues to be in the same double coset class if $ l equiv 1 pmod {N} $, for all $ g in text {SL} _2 ( mathbb {Z}) $. I thought it would be simple to see some kind of argument of the Euclidean / Restricted Lines operations, but I have a hard time doing that. What is the argument here, and what is wrong when $ N $ do not divide $ l-1 $?