Prevent users in AD to connect through RDC

I have created an OU (inside my AD) in my windows server 2008 R2. I have a few users inside it.

But all the users that i created inside the OU are able to connect to the server using remote desktop. 

I would like to give the access to only selected users

Please help me to do that

Thanks,

alagappan