Bob Desimone and crew find that removal of the prefrontal cortex (PFC) reduces (but, notably, does not eliminate) the effects of attention on neurons in visual cortical area V4.  The modulation of attention on firing rates was weaker and onset was delayed relative to the hemisphere with an intact PFC and there was a reduction of gamma power and synchrony.  Thus, PFC is an important, but not the only, source of top-down modulation on visual cortex.

Lesions of prefrontal cortex reduce attentional modulation of neuronal responses and synchrony in V4
Georgia G Gregoriou, Andrew F Rossi, Leslie G Ungerleider & Robert Desimone
Nature Neuroscience 17, 1003–1011 (2014) doi:10.1038/nn.3742