When a man says he doesn't want to ruin the friendship, he's saying he likes the access but not the accountability. He isn't protecting the bond. He's protecting his ability to leave without feeling like the bad guy.
He doesn't want to ruin the friendship, that sounds noble until you realize you weren't being held as a friend. You were being kept close, but not chosen.
When he's protecting the friendship, ask yourself, whose comfort is actually being protected? Not wanting to ruin the friendship means the friendship felt safer to him than the possibility of something more.