Food recognition and calorie estimation using computer vision

Summary

Calorie tracking is essential for a healthy diet, but traditional tools are tedious and prone to human error. The aim of this thesis is to develop a system capable of automatically estimating the calorie content of food based on images. The suggested method has two components: (1) a food classification model based on the 101-Food Dataset and (2) an estimation model of food portion size, which can be utilized to calculate the precise number of food calories. Food classification is implemented by training a CNN (with transfer learning) on a dataset containing images from 101 categories of food. Various approaches are studied for portion size estimation, such as food detection using the YOLO model, OpenAI’s CLIP model, and via ChatGPT’s textual reasoning abilities. While CNN-based classification performs robustly in most cases, the portion estimation task represents a more significant challenge. The same dish or ingredient can be arranged differently on the plate or even partially occluded by other objects. This thesis analyzes these limitations and proposes methods aimed at minimizing error propagation from the segmentation stage to the portion classification stage.