The paper describes creation of a contention-aware environment in a large-scale distributed system where the contention occurs to access resources between external and local requests. To resolve the contention, we propose and implement a preemption mechanism in the InterGrid platform, which is a platform for large-scale distributed system and uses virtual machines for resource provisioning. The implemented mechanism enables the resource providers to increase their resource utilization through contributing resources to the InterGrid platform without delaying their local users. The paper also evaluates the impact of applying various policies for preempting user requests. These policies affect resource contention, average waiting time, and imposed overhead to the system. Experiments conducted in real settings demonstrate efficacy of the preemption mechanism in resolving resource contention and the influence of preemption policies on the amount of imposed overhead and average waiting time. Copyright © 2013 John Wiley & Sons, Ltd.