Since the 1980s it has been becoming increasingly clear that the Solar system’s irregular satellites are collisionally evolved. The current populations are remnants of much more massive swarms that have been grinding away for billions of years. Here, we derive a general model for the collisional evolution of an irregular satellite swarm and apply it to the Solar system and extrasolar planets. The model uses a particle-in-a-box formalism and considers implications for the size distribution, which allows a connection between irregular satellite populations and predicted levels in the resulting dust cloud. Our model reproduces the Solar system’s complement of observed irregulars well, and suggests that the competition between grain–grain collisions and Poynting–Robertson (PR) drag helps set the fate of the dust. In collision-dominated swarms most dust is lost to interplanetary space or impacts the host planet, while PR-dominated grains spiral in towards the planet through the domain of regular satellites. Because swarm collision rates decrease over time the main dust sink can change with time, and may help unravel the accretion history of synchronously rotating regular satellites that show brightness asymmetries, such as Callisto and Iapetus. Some level of dust must be present on au scales around the Solar system’s giant planets if the irregular satellites are still grinding down, which we predict may be at detectable levels. We also use our model to predict whether dust produced by extrasolar circumplanetary swarms can be detected. Though designed with planets in mind, the coronagraphic instruments on James Webb Space Telescope (JWST) will have the ability to detect the dust generated by these swarms, which are most detectable around planets that orbit at many tens of au from the youngest stars. Because the collisional decay of swarms is relatively insensitive to planet mass, swarms can be much brighter than their host planets and allow discovery of Neptune-mass planets that would otherwise remain invisible. This dust could have been detected by Hubble Space Telescope Advanced Camera for Surveys (HST ACS) coronagraphic observations, and in one case dust may have already been detected. The observations of the planet Fomalhaut b can be explained as scattered light from dust produced by the collisional decay of an irregular satellite swarm around a ∼10 M⊕ planet. Such a swarm comprises about 5 lunar masses worth of irregular satellites. Finally, we briefly consider what happens if Fomalhaut b passes through Fomalhaut’s main debris ring on a coplanar orbit, which allows the circumplanetary swarm to be replenished through collisions with ring planetesimals. This scenario, in which the planet is at least of the order of an Earth mass, may be ruled out by the narrow structure of the debris ring.