Mammalian chromosomes are organized in structural and functional domains of 0.1-10 Mb, which are characterized by high self-association frequencies in the nuclear space and different contact probabilities with nuclear sub-compartments. They exhibit distinct chromatin modification patterns, gene expression levels and replication timing. Recently, nucleolus-associated chromosomal domains (NADs) have been discovered, yet their precise genomic organization and dynamics are still largely unknown. Here, we use nucleolus genomics and single-cell experiments to address these questions in human embryonic fibroblasts during replicative senescence. Genome-wide mapping reveals 1,646 NADs in proliferating cells, which cover about 38% of the annotated human genome. They are mainly heterochromatic and correlate with late replicating loci. Using Hi-C data analysis, we show that interactions of NADs dominate interphase chromosome contacts in the 10-50 Mb distance range. Interestingly, only minute changes in nucleolar association are observed upon senescence. These spatial rearrangements in subdomains smaller than 100 kb are accompanied with local transcriptional changes. In contrast, large centromeric and pericentromeric satellite repeat clusters extensively dissociate from nucleoli in senescent cells. Accordingly, H3K9me3-marked heterochromatin gets remodelled at the perinucleolar space as revealed by immunofluorescence analyses. Collectively, this study identifies connections between the nucleolus, 3D genome structure, and cellular aging at the level of interphase chromosome organization.