ChatGPT’s new viral image trend has a dark side — and it’s all about your data

Over the last few weeks, ChatGPT's native image generation capabilities had taken the internet by storm, enabling the chatbot to create Studio Ghibli-style images from real pictures of people, among other effects. However, OpenAI recently released its latest o3 and o4 Mini language models, which are now being used to power another social media trend that may not be so harmless.

What is the new ChatGPT update?

OpenAI explained in its launch blog that o3 and o4-mini are visual reasoning models, which means they can think with images in their chain of reasoning, allowing them to crop, zoom, memorise and use other image processing techniques to analyse the data on the image submitted by the user.

Similar to GPT-4o, the new visual reasoning capabilities of o3 and o4-mini are built natively, meaning that they do not rely on an external model such as DALL-E 3 to process the images.

What are users doing with ChatGPT's new abilities?

Social media users are already experimenting with OpenAI’s latest flagship model, o3, using it to identify the geographic location of places shown in photos. Posts on platforms like X (formerly Twitter) and Reddit highlight the model’s surprising accuracy—some even claim it can pinpoint exact coordinates.

While we didn't have access to OpenAI o3 since it's a paid model, we tried running a few images through o4 mini and the results were shocking to say the least. ChatGPT took a few seconds to think and zoomed and cropped the image in different directions after giving the correct answer in most cases. However, despite several attempts, we couldn't get an exact coordinate.

While a GeoGuessr-style game powered by ChatGPT’s new model may seem like harmless fun for now, the implications could become more serious over time. It’s not hard to imagine this capability being used to identify the locations of photos shared by random individuals—potentially putting their privacy, and even safety, at risk.

OpenAI risking safety to rollout new models faster?

Ever since the first public rollout of ChatGPT in late 2022, there have been concerns about AI being used for nefarious purposes, many of which have come to fruition recently. Although geo-guessing the location of images is very alarming in itself, the chatbot has also previously been used to generate fake Aadhaar and PAN cards, among other identification documents.

Although ChatGPT has historically blocked such prompts, OpenAI’s rapid release of new models—driven in part by rising competition from Chinese AI firms—appears to be raising fresh safety concerns. A recent Financial Times report revealed that the company has significantly reduced the time and resources devoted to safety testing, with new models being pushed to market without “sufficient safeguards”

Reportedly, staff and third-party groups were given just days to assess the risks and performance of the new models compared  to the several month timeframe for previous models. 

One OpenAI staffer told the publication that LLMs (the foundation models running ChatGPT) have become more capable of ‘potential weaponization’. 

“But because there is more demand for it, they want it out faster. I hope it is not a catastrophic mis-step, but it is reckless. This is a recipe for disaster.” FT report quoted the OpenAI staffer as saying.