Studying temporal features of neural activities is crucial for understanding the functions of neurons as well as underlying neural circuits. To this end, recent researches employ emerging techniques including calcium imaging, Neuropixels, depth electrodes, and Patch-seq to generate multimodal time-series data that depict the activities of single neurons, groups of neurons, and behaviors. However, challenges persist, including the analysis of noisy, high-sampling-rate neuronal data, and the modeling of temporal dynamics across various modalities. To address these challenges, we developed NeuroTD, a novel deep learning approach to align multimodal time-series datasets and infer cross-modality temporal relationships such as time delays or shifts. Particularly, NeuroTD integrates Siamese neural networks with frequency domain transformations and complex value optimization for inference. We applied NeuroTD to three multimodal datasets to (1) analyze electrophysiological (ephys) time series measured by depth electrodes, identifying time delays among neurons across various positions, (2) investigate neural activity and behavioral time series data derived from Neuropixels and 3D motion captures, establishing causal relationships between neural activities and corresponding behavioral activities, and (3) explore gene expression and ephys data of single neurons from Patch-seq, identifying gene expression signatures highly correlated with time shifts in ephys responses. Finally, NeuroTD is open-source at https://github.com/daifengwanglab/NeuroTD for general use.