Citizens in rich countries should shoulder the burden of alleviating global poverty because they are harming the poor, or so many argue. But the baseline for assessing harm is often unclear. This paper recommends a baseline for harm as rights violations. This baseline makes it clear that many of the attributions of harm made by proponents of the harm argument, instead of representing cases of harms caused, are rather instances of benefits withheld from the poor. A moral case can be made that benefits should be extended by the rich countries toward poor ones, but this case will look very different from a case for responsibility for harm.