the anime pacing is fine the manga is just a slow burn
It has to build up the relations between characters before sht hits the fan.
Also the protagonist isn't just a perv that wants one thing. Please read ahead in the manga or use critical thinking skills. He's been devoid of love and touch since he was a child and is actively being groomed. Ofc that's where his motives lie.
If you look at Maslow's Hierarchy of Needs our protagonists is currently at the "Love and Belonging" tier. This makes sense as he has attained his safety needs and physiological needs for the first time in his life.