Opinion
AI is turning search engines into thought drivers in mutual rivalry
Summary
- Upstarts like Perplexity and Arc Search are challenging Google with the help of AI that enables them to function as ‘thought engines’ which do much more than spout weblinks.
Many people would be surprised to learn that the foundational algorithm for Generative AI and Large Language Models (LLMs) was first ‘invented’ at Google, with eight researchers there writing a landmark 2017 paper, ‘Attention is All You Need.’ This laid the basis of the Transformer algorithm, one that made GenAI more efficient and effective. Surprisingly, Google dithered with this amazing discovery, and let a startup called OpenAI take the lead in leveraging it to create its series of GPTs, culminating with GPT3.5 or ChatGPT. The rest, as they say, is history.
Speculation abounds over why the world’s most innovative company did not capitalize on its discovery. People point to reputational risk arising from hallucinating LLMs; some speak of corporate inertia. Others say it had to do with the fear of its profitable business model of search getting cannibalized. AdWords and AdSense-driven ad revenue works very well in a deterministic, database-driven model like search, but would fail in a probabilistic LLM construct. A score of companies have tried to topple Google from its lofty perch, like Microsoft’s Bing and startups like DuckDuckGo and Neeva, but Google has been unassailable. The hallucinatory behaviour of even the best LLMs like GPT4 seems to preserve the status quo, with Google still the go-to tool for factual information. LLMs like ChatGPT are not optimized for the truth, but for factual believability.
This seems to be changing, though, with a couple of GenAI-based startups threatening Google’s mighty search engine. The best known is Perplexity, built on top of GPT and a proprietary LLM built on Meta’s open-sourced LlaMA. It takes a different approach. When you ask Perplexity’s ‘search’ bar a question, it does not return with 10 blue links plus advertising-pushed websites. Instead, Perplexity hunts the web and uses AI to write a succinct summary of its findings and annotates the same with the sources it used for its answer. It also offers a few suggested follow-up questions. So, when I asked Perplexity to give me the name of its founder, it spouted ‘Aravind Srinivas,’ told me that the sources were LinkedIn, Wikipedia, YouTube and Wired, and then checked whether it wanted me to know Perplexity’s mission or its co-founders’ background. Precise and useful. For $20 a month, it also opens up its Copilot, which helps you arrive at a precise query by asking you clarifying questions. If a student researching the impact of climate change on marine ecosystems were to ask it a question, for example, the Copilot would ask her for regions or species, and then scour scientific journals and environmental reports. It then summarizes the answer with deep insights and helps her build a comprehensive literature review. It also allows you to search within a specified set of sources, such as YouTube, Reddit or academic papers.
Another impressive contender is Arc Search from New York-based The Browser Company. Its philosophy is that “a browser, a search engine, an AI chatbot, and a website aren’t different things. They’re all just parts of an internet information finder, and they might as well exist inside the same app" (bit.ly/4a55DVC). So, again, rather than offer 10 links, Arc creates a kind of a webpage with relevant information around the query. Arc is also built on OpenAI and other LLMs and will improve as these models do.
These upstart Google competitors are not perfect. They hallucinate, though much less than bare-boned LLMs. They do not have a business model yet, though one could evolve. Publications fear that they will not send traffic to their sites, as Google does. However, Google should be worried; its search engine seems to have lost its simplicity and transparency along the way. It has shifted from being made for users to being optimized for advertisers, and its monopoly has let it squeeze the latter.
I believe that newer search constructs like Perplexity and Google will live together; I now have a browser tab permanently open for Perplexity and another for Google. Depending on what I am looking for, I choose which to use. It would be good for Google to have some semblance of competition; hopefully, it will make the firm pivot towards the consumer again, bring back the clarity of its search and shake off the corporate sclerosis which appears to have bogged it down. Shaan Puri, a tech thought leader, differentiated the two in an X post: “Google is a search engine. ChatGPT is a thought engine." Innovators like Perplexity and Arc seem to be blurring those boundaries.
