The actual problem in this case is, that “to” is a preposition here and NOT part of the infinitive!
Therefore the Gerund is required.
After verb+preposition, only the Gerund can be used:
I’m fond of doing sports. I’m looking forward to seeing you.
After some verbs, there’s only the infinitive with to that can be used:
I wanted to see you.
-> Here, “to” is part of the infinitive and no preposition.
(“I wanted seeing you” is not possible.)
And after a few verbs, either a preposition or an infinitive is possible;
In some cases, the meaning doesn’t change, while in others it does change:
a) I started to walk -or- I started walking. (same meaning)
b) I stopped to smoke cigarettes -/- I stopped smoking cigarettes. (different meaning)
-> There is already another thread about this change in meaning.