Yes. Your criminal Defense Attorney is trying to present your case in the way that casts you in the best possible way. If your attorney does not know topics that may best be avoided you can allow your secret to become your downfall.
I recently was in a jury trial where my client was accused of rape. He had no criminal record and swore he never committed any of the crimes for which he was accused. He had, however, been in a voluntary sexual relationship with the woman making the accusation. She hid this fact from the prosecuting attorney and intended to lie to the Judge and Jury about this relationship as well.
If my client wanted to tell me less he could have denied ever having sex with the accuser at all. Instead he confided that the two were sexually active including AFTER the date she indicated she was raped. This important detail was not something my client wanted to talk about but he did because he trusted that his attorney was using the information to his advantage.