Abstract: Particle-scale tracking and breakage analysis are essential for understanding granular mechanics, but existing methods struggle with irregular/crushable particles and large interval tracking. To solve these challenges, this study presents a novel framework that formulates particle tracking as a graph optimization problem, solved through physics-informed spatiotemporal optimal transport (PSOT-Track). The proposed method uniquely integrates: (1) spatiotemporal graphs encoding particle neighborhood relationships, (2) physics-based costs (mass conservation and shape similarity) for transport optimization, and (3) breakage-aware matching via fragmented particle reassembly. Validation using X-ray micro-computed tomography (μCT) datasets (9,248 regular lentil particles, 4,765 irregular Fujian sand particles, and 1,727 irregular porous coral sand particles) demonstrates superior performance: 99.5%/93.1%/97.3% accuracy under small tracking interval (≤ 5% strain), maintaining over 70% accuracy at large tracking intervals (> 5% strain), achieving a 50% improvement over conventional methods. Furthermore, PSOT-Track successfully identifies 97.1% of splitting and 85.7% of chipping breakage, highlighting its potential for breakage analysis. The proposed approach offers a generic and robust solution for elucidating complex micromechanical phenomena in granular materials.