Yes, I believe everyone has the right to sexual privacy.
What you choose to share about your own activities is entirely your own choice. If being completely candid is what you crave, tell your partner everything. If you want to enjoy your own secrets, that’s fine too.
Of course, you wouldn’t want to lie if he looked directly into your eyes, and asked you if you’d touched yourself.
After all, everyone needs something to confess to, eventually…