The genetic consequences of species-wide declines are rarely quantified because the timing and extent of the decline varies across the species’ range. The sea otter (Enhydra lutris) is a unique model in this regard. Their dramatic decline from thousands to fewer than 100 individuals per population occurred range-wide and nearly simultaneously due to the 18th-19th century fur trade. Consequently, each sea otter population represents an independent natural experiment of recovery after extreme population decline. We designed sequence capture probes for 50 megabases of sea otter exonic and neutral genomic regions. We sequenced 107 sea otters from five populations that span the species range to high coverage (18-76x) and three historic Californian samples from 1500 and 200 years ago to low coverage (1.5-3.5X). We observe distinct population structure and find that sea otters in California are the last survivors of a divergent lineage isolated for thousands of years and therefore warrant special conservation concern. We detect signals of extreme population decline in every surviving sea otter population and use this demographic history to design forward-in-time simulations of coding sequence. Our simulations indicate that this decline could lower the fitness of recovering populations for generations. However, the simulations also demonstrate how historically low effective population sizes prior to the fur trade may have mitigated the effects of population decline on genetic health. Our comprehensive approach shows how demographic inference from genomic data, coupled with simulations, allows assessment of extinction risk and different models of recovery.