Every now and again when I drag the little streetview character onto a street it will also scroll the wayfarer page down at the same time making me go to the wrong street on streetview. It usally scrolls me down to where i can press the appropriate rejection thing.
What type of phone and browser are you using? This has never been an issue for me, using my Galaxy S10 and Chrome.
I have the galaxy s22+ and I also use chrome