Domain adaptation in data scarce scenarios using Time Series Foundational Models

Summary

This thesis investigates domain adaptation strategies for time series forecasting in data-scarce scenarios using Time Series Foundational Models (TSFMs). The core problem addressed is whether fine-tuning a pre-trained model on a data-rich source domain improves performance when subsequently adapting to a related but data-scarce target domain, compared to directly fine-tuning on the target domain from scratch.

The research employs a two-stage progressive fine-tuning framework. In Stage One, the MOIRAI foundational model (a transformer-based architecture with approximately 13 million parameters) is fine-tuned on a data-rich source subdomain to create a domain-adapted model. In Stage Two, this adapted model undergoes incremental fine-tuning on the target subdomain as data becomes progressively available in small batches, simulating real-world data scarcity conditions. Three incremental fine-tuning strategies were evaluated: Progressive Incremental Fine-Tuning (which retained knowledge from previous iterations), Independent Incremental Fine-Tuning (which reset to the Stage One model at each iteration), and Independent Full Fine-Tuning (which performed thorough training at each step). The progressive approach proved most effective and computationally efficient.

The methodology was tested across three distinct domain pairs: Greek and Italian electricity load data (hourly observations over approximately 1.5 years), Bitcoin and Ethereum cryptocurrency prices (daily data spanning multiple years), and temperature records from Athens and Izmir (daily measurements over 11 years). Model performance was assessed using Mean Absolute Percentage Error (MAPE) for point forecasts and Mean Weighted Quantile Loss (MWQL) for probabilistic predictions, with evaluation conducted on held-out portions of the target domain datasets.

The results reveal that progressive fine-tuning effectiveness is highly domain-dependent. For cryptocurrency and weather temperature domains, the proposed method yielded substantial improvements, with average performance gains of approximately 7.6% in MWQL for cryptocurrency forecasting and 11.5% for temperature prediction. These improvements suggest effective knowledge transfer when source and target domains share strong statistical similarities. Conversely, the electricity load domain experienced significant performance degradation of roughly 30% in MWQL, highlighting challenges when transferring knowledge between domains with disparate consumption patterns and regulatory structures.

The key takeaway is that while progressive fine-tuning can accelerate model adaptation in low-data regimes and reduce training costs, its success depends critically on domain similarity. The methodology proves particularly valuable for closely related subdomains but may introduce harmful biases when fundamental data distributions differ substantially between source and target domains.