At the recent presentation of GPT-4, one of the standout features was its ability to engage in conversations enriched with images. However, this capability is yet to be integrated into OpenAI’s offering. While we previously highlighted Bing’s competence in this aspect, an open-source solution has now emerged in the form of the “Large Language and Vision Assistant” (LLaVA).
LLaVA’s Multimodal Potential
LLaVA is an open-source multimodal AI that combines language and vision processing. A demo of LLaVA can be accessed here.
We conducted a simple test by inputting a picture of Taxi and man, and LLaVA provided me with a descriptive analysis. However, my attempts to challenge it with mathematical problems, similar to those tackled by Bing, proved futile. LLaVA appears to struggle with mathematics, despite its proficiency in image recognition.
Challenges with Mathematical Tasks
We presented LLaVA with a mathematical problem involving trigonometry, akin to those successfully solved by Bing. Unfortunately, LLaVA could not provide the correct solution, returning an erroneous answer for the square root of 169 as 13.2.
An Unusual Perspective on Images
LLaVA excels in its ability to converse with images, although challenges persist, particularly in mathematical problem-solving. For example, when presented with an image of a man leaning out of a yellow taxi window, holding a clothesline with a white shirt, LLaVA provided an unusual perspective. It suggested that such a scene is atypical, as it is not common to witness individuals leaning out of car windows while holding clothing. The analysis indicated that the man may be attempting an unconventional and potentially unsafe method of drying his shirt while the taxi is in motion.
While LLaVA offers promising multimodal capabilities, particularly in conversing with images, it faces limitations in mathematical problem-solving. It’s worth noting that Google’s capabilities in this regard surpass LLaVA’s, as demonstrated by a more accurate solution to a similar mathematical problem.
The development of AI with multimodal capabilities is undoubtedly an exciting advancement, and LLaVA is a commendable open-source effort in this direction. However, improvements are needed to enhance its mathematical reasoning capabilities to match its proficiency in image analysis.
For a more accurate mathematical solution, Google’s capabilities are currently superior: Google’s Mathematical Problem Solver.
Source: mPost
