Any other species that is in direct competition for the same food base as the whitetail will be harmful to it. I think a small herd of the fallow deer could get a toe-hold in the area, but I'd bet most of the hunters who see then would take them out before the herd could get too large.

We have the same problem in certain areas of Texas. The hunters surrounding the big game ranches from where the exotics tend to escape from keep them in check by adding them to the other venison in the freezer, since no extra tags are needed to take these "non-game" species.