Recent work has argued that large-scale neural recordings are often well described by low-dimensional ‘latent’ dynamics identified using dimensionality reduction. However, the view that task-relevant variability is shared across neurons misses other types of structure underlying behavior, including stereotyped neural sequences or slowly evolving latent spaces. To address this, we introduce a new framework that simultaneously accounts for variability that is shared across neurons, trials, or time. To identify and demix these covariability classes, we develop a new unsupervised dimensionality reduction method for neural data tensors called sliceTCA. In three example datasets, including motor cortical dynamics during a classic reaching task and recent multi-region recordings from the International Brain Laboratory, we show that sliceTCA can capture more task-relevant structure in neural data using fewer components than traditional methods. Overall, our theoretical framework extends the classic view of low-dimensional population activity by incorporating additional classes of latent variables capturing higher-dimensional structure.