Improving The Performance of the Image Captioning Systems Using a Pre- Classification Stage

تحسين أداء أنظمة وصف الصور باستخدام مرحلة التصنيف المسبق للصور

المؤلفون

  • Rasha Mohammed Mualla, Jafar Alkheir, Samer Sulaiman

الكلمات المفتاحية:

Deep Learning
التعلم العميق
Natural Language Processing
معالجة اللغات الطبيعية
Arabic Language Image Captioning
وصف الصور باللغة العربية
English Language Image Captioning
وصف الصور باللغة الإنكليزية
Image Classification
تصنيف الصور
Image captioning
وصف الصور بتسميات توضيحية

الملخص

 

In this research, we introduce a novel image classification and captioning system by adding a classification layer before the image captioning models. The suggested approach consists of three main steps and inspired by the state- of- art that generating image captioning inside small sub- classes categories is better than the unclassified large dataset. In the first one, we have collected a dataset of two international datasets (MS- COCO and Flickr2k) including 10778 images in which 80% is used for training and 20% for validation. In the next step, dataset images have been classified into 11 classes (10 classes of indoor and outdoor categories and one class of "Null" category) and fed into a deep learning classifier. The classifier is re- trained again using our classes and learned to classify each image to the corresponding category. At the final step, each classified image is used as input of 11 pre- trained classified image captioning models, and the final captioning sentence is generated. The experiments show that adding the pre- classification step before the image captioning stage improves the performance significantly by (8.15% and 8.44%) and (12.7407% and 16.7048%) for Top- 1 and Top- 5 of English and Arabic systems respectively. The classification step achieves a true classification rate of 71.32% and 73.09% for English and Arabic systems respectively.

السيرة الشخصية للمؤلف

Rasha Mohammed Mualla, Jafar Alkheir, Samer Sulaiman

 

Rasha Mohammed Mualla

Mechanical & Electrical Engineering Faculty || Tishreen University || Syria

Jafar Alkheir

Artificial Intelligence, Computer Science || Tishreen University || Syria

Samer Sulaiman

Faculty Engineering || Al- Manara University || Syria

التنزيلات

منشور

2022-03-27

كيفية الاقتباس

1.
Rasha Mohammed Mualla, Jafar Alkheir, Samer Sulaiman. Improving The Performance of the Image Captioning Systems Using a Pre- Classification Stage: تحسين أداء أنظمة وصف الصور باستخدام مرحلة التصنيف المسبق للصور. jesit [انترنت]. 27 مارس، 2022 [وثق 9 أغسطس، 2022];6(1):150-64. موجود في: https://journals.ajsrp.com/index.php/jesit/article/view/4847

إصدار

القسم

المقالات