Abstract: A Bayesian approach to comparing the effects of accretion disks, dark matter or clouds of ultra-light bosons on gravitational waveforms from a black hole binary system concludes that detectors such as LISA can distinguish between these environments. Future gravitational wave interferometers such as the Laser Interferometer Space Antenna, Taiji, DECi-hertz Interferometer Gravitational wave Observatory and TianQin will enable precision studies of the environment surrounding black holes. These detectors will probe the millihertz frequency range, as yet unexplored by current gravitational wave detectors. Furthermore, sources will remain in band for durations of up to years, meaning that the inspiral phase of the gravitational wave signal, which can be affected by the environment, will be observable. In this paper, we study intermediate and extreme mass ratio binary black hole inspirals, and consider three possible environments surrounding the primary black hole: accretion disks, dark matter spikes and clouds of ultra-light scalar fields, also known as gravitational atoms. We present a Bayesian analysis of the detectability and measurability of these three environments. Focusing for concreteness on the case of a detection with LISA, we show that the characteristic imprint they leave on the gravitational waveform would allow us to identify the environment that generated the signal and to accurately reconstruct its model parameters.