Can we remove the PayPal option?

No problem 🙂
Go to: Edit Invitation -> Invitation Features and disable the “Allow guests to send gifts” feature.