Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine

paper

Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, Zhaopeng Tu

Supporting Links:

Summary:

This article evaluates the machine translation capabilities of ChatGPT compared to commercial translation products like Google Translate and DeepL Translate. The evaluation covers three aspects: translation prompts, multilingual translation, and translation robustness.

The results show that ChatGPT performs comparably with commercial systems on high-resource languages but lags behind on low-resource languages. However, a pivot translation strategy that uses English as an intermediate language significantly improves translation performance for distant languages. Furthermore, the newly launched GPT-4 engine boosts ChatGPT’s translation abilities, making it competitive even for distant languages.

The article concludes that with GPT-4 as its foundation, ChatGPT has become a good translator. However, the evaluation has some limitations like the limited test data size and reproducibility issues due to ChatGPT’s latency.

Applicability to Applications Development: This evaluation demonstrates the quality and limitations of ChatGPT’s (and now GPT-4’s) machine translation capabilities. While it shows promising results for high-resource languages, it still lags behind commercial products on domain-specific or noisy texts. However, with large language models like GPT-4 as the backend, chatbot and conversational AI applications that require translation abilities have the potential to improve significantly.

The insights from this study into translation prompts, multilingual translation strategies, and robustness can help guide the development and optimization of such large language model powered applications. But more rigorous and extensive evaluations are needed to fully understand the capabilities and remaining gaps.