Info
Background:
A central goal of systems neuroscience is to understand how neuronal dynamics give rise to complex behaviors1,2. This question is especially important for improving our understanding of neurological disorders (e.g. epilepsy, stroke and Parkinson’s disease) and guiding the development of adaptive, behavior-dependent therapies such as deep brain stimulation (DBS) to treat them3. Achieving this requires reliable mappings (i.e. decoding) between in vivo neuroimaging data (e.g. neuronal firing) and behavioral states. Recent advances in miniaturized microscopes (Fig. 1a), have made it possible to continuously image (≥24-hours) neuronal activity via genetically encoded fluorescent calcium sensors (i.e. GCaMP) in freely moving mice, while simultaneously recording their behavior. However, these experiments generate extremely large datasets (~100 GB) per experiment, and the complexity of natural behavior makes it challenging to directly correlate neuronal activity with behavioral state. Although conventional image processing and machine learning approaches have been applied to decode such neuroimaging data, they often exhibit poor predictive accuracy and can require extensive computational effort. Deep learning (DL) approaches that exploit Convolutional Neural Networks (CNNs)4,5 to extract spatial features and Recurrent Neural Networks (RNNs)6 to extract temporal dynamics offer a promising alternative because they can model both spatial and temporal patterns in sequential data and handle large datasets. Motivated by these advances, we explored the feasibility of decoding behavior from miniscope-acquired images of neuronal activity (i.e. “brain flashes”) in freely behaving mice via the use of a deep learning model that combines a ResNet-50 (i.e. CNN) with a Bidirectional Long Short-Term Memory (BiLSTM) module (i.e. RNN).

 

Hypothesis/Objective:
We hypothesized that behavioral states could be decoded directly from continuous 24-hour neuronal activity imaging data acquired in awake, freely behaving mice using a ResNet50-BiLSTM CNN-RNN framework, even under highly constrained imaging conditions, such as a limited 3×3 mm² sensorimotor cortical field-of-view and an ultra-slow sampling rate of ≤0.1 Hz. We further hypothesized that our model was generalizable across animals, enabling accurate behavior prediction in a completely unseen mouse during the model cross-validation process. If successful, real-time behavior prediction with such DL models could: (i) enable high-throughput estimation of “neurophysiology-to-behavior” transfer functions for relevant actions (running, whisking, eating, sleeping, etc.); (ii) power closed-loop therapeutics in which deviations from a patient-specific transfer function could trigger adaptive DBS for Parkinsonian symptoms; and (iii) accelerate theranostic and drug-discovery workflows via efficacy readouts.

 

Methods:
Using a multicontrast head-mounted miniscope (Fig. 1b)7, we acquired images of neuronal activity over 24-hours via the fluorescence channel (Fig. 1c) in five awake, freely behaving healthy mice (M1-M5). A second camera synchronously recorded behavior over the same period. For each animal, this behavioral movie was annotated into three behavioral states: minimally mobile (MiniMOB), eating or other non-mobile behavior (EoNB), or running (Run), using BehaviorCloud Software8 and then divided into 1-min blocks corresponding to a single behavioral state. Since mice remained inactive for extended periods, the overall dataset was class-imbalanced and heavily enriched for the MiniMOB state. To reduce this imbalance during training of the ResNet50-BiLSTM model, MiniMOB and EoNB samples were downsampled to ~300 blocks to match the number of Run samples. The preprocessing step involved resampling neuronal activity images from 512×512 to 224×224 pixels using nearest-neighbor interpolation with a 3×3 kernel. During training, the terminal layers of ResNet50, together with the BiLSTM and final fully connected layer, were fine-tuned to learn spatiotemporal patterns of neural activity associated with behavior. Model performance was evaluated using five-fold leave-one-mouse-out cross-validation, in which one mouse was withheld for testing in each fold to assess cross-animal generalizability. Performance on unseen animals was quantified using macro F1 score and average Precision Recall Area-Under-the-Curve (PR AUC). To further confirm representations learned by the DL network, we performed a negative-control analysis on unseen mice, M2 in Fold 2 and M4 in Fold 4, by randomly reordering frames within each 1-minute block. This shuffling disrupted temporal continuity, while preserving the spatial content of individual images. Finally, we used a 3D Uniform Manifold Approximation and Projection (UMAP) to visualize learned feature representations from the trained Fold 2 and Fold 4 models.

 

Results:
Across five leave-one-mouse-out folds, the model achieved a macro-F1 score of 0.63± 0.09 and an average PR AUC of 0.767±0.02. The MiniMOB class was classified most reliably, whereas Run and EoNB classes were frequently confused as shown in the precision-recall curves (Fig. 1e) and confusion matrices (Fig. 1f). This was likely due to class imbalance and inter-animal variability. UMAP analysis on Folds 2 and 4 showed unique label-specific clusters for the original data (Fig. 1g) but pronounced intermixing for the temporally shuffled data (Fig. 1h). These analyses collectively support the significance of temporal structure for behavioral decoding from neural images.

 

Conclusion:
In summary, this work introduces a novel DL framework that employs fluorescence images of neural activity to predict behavior in freely behaving animals. Moreover, we successfully characterized the potential generalizability of this AI framework across unseen (i.e. not seen by the DL network) animals. Finally, negative-control analyses confirmed that the DL model was learning behavior-relevant temporal neural dynamics, rather than static image features.

Acknowledgments: This work was supported by NIH grants 5R01CA237597-04, 5R01CA196701-09 and 5R21NS138938-02.

 

Image/Figure:
Click to view full size
Click to view full size

Image/Figure Caption:
Figure 1: AI can decode the behavior of freely moving animals from the language of neural activity: (a) Photograph of the lightweight miniscope used for multicontrast 24-hour neuroimaging. (b) Photograph of the miniscope mounted on a freely behaving mouse. (c) Representative fluorescence image acquired from the sensorimotor cortex, alongside a representative 24-hour time-series of mean neural activity from the region indicated by the yellow box in (c). (d) Schematic of the major steps in the DL-based behavior-classification workflow. Each sample consisted of a 1-minute image block composed of sequential frames of neural activity. Raw image blocks were first resampled from 512×512 to 224×224 pixels and then passed through a ResNet50 backbone to extract spatial features from individual frames. These frame-wise embeddings were subsequently fed into a bidirectional LSTM (BiLSTM) to learn temporal dynamics across the 1-min blocks, followed by a fully connected classification layer that predicted one of three behavioral classes: Run, Minimally Mobile (MiniMob), or Eating or Nest Building (EoNB). indicate trainable DL modules in the architecture. (e) Precision-recall curves for each behavioral class across the five leave-one-mouse-out folds. Curves are shown separately for Run (red), MiniMob (green), and EoNB (blue). Across folds, MiniMob generally achieved the strongest precision-recall performance, whereas Run and EoNB showed greater variability and lower separability, highlighting the challenge of discriminating more dynamic or overlapping behavioral states across animals. (f) Confusion matrices for all five folds, with rows representing true labels and columns representing predicted labels. Correct classifications are shown along the diagonal, and off-diagonal entries denote misclassifications. MiniMob was classified most reliably in all folds, with consistently high true positive counts. In contrast, substantial confusion was observed between Run and EoNB states in several folds, indicating an overlap in the image-derived temporal feature patterns associated with these behaviors. Fold-to-fold differences further reflected inter-animal variability and class imbalance in the dataset. (g) A 3D UMAP visualization of learned feature embeddings from the trained model for Fold 2 and Fold 4 using the original, temporally ordered 1-min blocks. Each point represents a 1-minute image block projected into low-dimensional space and color coded by behavioral label. Distinct structure and label-dependent organization are visible in the embedding space, indicating that the model learned discriminative latent representations from the original data. (h) A 3D UMAP visualization of embeddings for Fold 2 and Fold 4 after temporal shuffling of frames within each 1-min block as a negative-control analysis. Shuffling preserved spatial image content while disrupting temporal continuity. Compared with panel (g), the embeddings became markedly more intermixed and less behavior-specific, with reduced class-wise separations. This result indicates that the model strongly relied on temporal neural dynamics, rather than static spatial features alone, for behavioral classification, despite the ultra-slow sampling regime.

 

Author
Subhrajit Das
Johns Hopkins Univeristy