AI is taking over the search engine. This is the look under the hood

For decades, the way we find information on the Internet has only changed in small ways. Doing traditional Google searches today isn’t much different from asking Jeeves in the 1990s. Certainly, a lot has changed under the hood. The results are much more relevant and the interface has some new features, but you’re entering keywords to get a list of websites that may hold the answer.

That way of searching seems to be starting to follow the Altavista path.

In May, Google announced its new developments AI Mode Use a for search Generated AI Model (based on company’s Gemini Large language model) Provides conversational answers that make you feel like you’re chatting and looking through a set of links. Other companies, such as Perplexity and Openai, are also deploying search tools based on Gen AI. These tools that merge chatbots with traditional search engine capabilities are quickly becoming more and more steamy.

You can’t even escape AI by doing regular Google searches: AI overviews have appeared on top of those results pages since last year. Once in 5 A report from the Pew Research Center shows this type of summary. I’m surprised that it’s not more than that.

these NewFangled Search Tool It’s very similar to a typical chatbot chatgptbut they do something a little different. These differences share a lot of DNA with search engine ancestors. Here’s a look under the hood on how these new tools work and how to use them effectively.

Search Engine vs AI Search: What’s the difference?

The underlying technology of search engines is like an old library card catalog. The engine uses bots to crawl the vast spread of the internet, finding, analyzing and indexing an infinite number of web pages. Then, if you do a search to ask who played Dr. Angela Hicks in the ER, you will return pages such as the cast and actor biography of the ER, CCH Pounder, etc., as you remember what she saw elsewhere. From there, you can click on those pages, whether it’s Wikipedia, IMDB, or elsewhere, to find out you know CCH Pounder from an Emmy-winning guest star for an episode of X-Files.

“If a customer has a specific question, you can enter it into Google, then Google can run a ranking algorithm to find the best content for a particular query,” Eugene Levin, president of Semrush, marketing and SEO tool company.

In general, traditional searches require you to click on other websites to get the answer you are looking for. When I was trying to figure out where I recognized CCH Pounder, I clicked at least half a dozen different sites to track it. This includes using Google’s video search to attract video indexes across a variety of hosting platforms and find clips of her appearance on X-Files.

Multiple these searches do not necessarily occur. If you want to know the cast of the ER, enter “Cast of the ER” and Wikipedia Page On the top.

Usually there is another related, trusted site located at or near the top of the Wikipedia or the search results page. This is because the main way today’s search algorithms work is to track which sites and pages get most of the links from other places on the web. The “changed game” model when Google launched in the 1990s was more reliable than index systems that rely on how many times the keywords were displayed on the page, says Sauvikdas, an associate professor at the Human Computer Interaction Institute at Carnegie Mellon University.

“There are a lot of cookie recipes on the web, how do you know what to display first?” Das said. “Well, if many other websites link to this website due to the keyword “Cookie Recipe”, it’s pretty difficult to play the game. ”

AI-powered search engines work a little differently, but they work on the same basic infrastructure. In my quest to see where I recognized CCH Pounder, I literally asked Google’s AI mode, “Where do you recognize the actress who plays Dr. Angie Hicks in ER?” I narrowed it down in a conversation that was like chatting with a bot rather than doing a search. The first result gave me a list of shows and movies I’ve never seen before, so I sought an extensive list featuring guest appearances on other shows. I then could ask for details on her X-Files look and it narrowed it down.

Although the way we interacted with Google was different, the search mechanism was basically the same. The AI mode simply developed and processed numerous different web searches to gather the necessary information using the Gemini model, said Robby Stein, vice president of products at Google Search. “The user just queried each of those queries themselves.”

Essentially, AI mode did the same thing as me, but it got much faster.

Very many searches, little time

The approach here is called “query fan-out.” The AI model receives requests, breaks them down into a series of questions, and performs a search to answer these components of the request. We then receive all those searches and information we collect from our website and answer it for you. With a heartbeat.

These searches use the same index as traditional searches. “They work for the same foundation,” Levin said. “Change is how they pull information from this foundation.”

This fan-out process allows AI search to extract relevant information from sites that were not shown on the first page of traditional search results, and to extract good information paragraphs from pages with more unrelated information. Instead of going down the rabbit hole and finding a small part of the answer you want, the AI will descend a wide range of rabbit hole in seconds.

“They’re looking for this, what’s the next thing you might be interested in?” Levin said.

read more: AI Essentials: Our experts say 29 ways to make Gen AI work

The number of searches that an AI model does depends on the tool you are using and the complexity of the question. AI modes using Google’s deep search will spend more time and do more searches, Stein says. “More and more, when you ask a really difficult question, it responds using our most powerful model,” Stein said.

Large language models that power these search engines also have existing training data to guide searches. Much of the information comes from the latest content found by searching the web, but this training data could contain information from such websites to the entire library of books. The training data is so extensive that a lawsuit over whether AI companies actually owned it The right to use that information It’s hanging right away. (Disclosure: CNET’s parent company Ziff Davis filed a lawsuit against Openai in April, claiming it infringed Ziff Davis’ copyright in training and operating AI systems.)

AI Search is not just a chatbot

The underlying language model may be roughly the same, but the independence of training data is one of the reasons why an AI-powered search engine is set up separately from traditional chatbots. While ChatGpt searches scrutinize relevant sites and answers on the Internet, regular ChatGpt may rely on your own training data to answer your questions.

“The correct answer may be there,” Das said. “It could also hallucinate answers that may not be anywhere in pre-training data.”

AI search uses a concept called the searched high-end generation to incorporate what you find on the Internet into your answer. It collects information from a source pointing to (in this case the search engine index) and tells you to look at it rather than create something if it cannot be found in the training data. “You say AI has the answer here, I want you to find it,” Das said. “You got the top 10 Google results and you say AI has the answers here.”

Can you really trust AI search results?

These AI-powered search tools can be more reliable than using the chatbot itself. This is because it is derived from current related information and provides links. Think critically about that. Here are some tips from experts:

It brings your human skepticism

Think about how bad people there are to say when you’re sarcastically on the internet. Next, consider how bad a large-scale language model is. That’s the overview of Google’s AI: Place the adhesive on the pizza – By pulling information from humorous Reddit posts and repeating it as if it was real cooking advice. “AI doesn’t know what is real and humorous,” Das said. “We’ll treat all the information the same.”

Don’t forget to use your own judgment to find the source of information. They may not be as accurate as LLM thinks, and they don’t want to make important life decisions based on someone’s jokes on an internet forum where they thought robots were real.

AI can still make things

They should be drawn from search results, but these tools can make up for things even if there is no good information. This is how I started to create an AI overview False definition of meaningless proverb.

Searched generation may reduce the risk of completeness Hallucinations According to DAS, it doesn’t rule that out. Remember that LLM has no sense of what the correct answer to the question is. “This just predicts what the next English word will be after this previous stream of other English words and other language words,” Das said. “In that sense, there is no real concept of truth.”

Check the source

Traditional search engines are very hand-offs. They will provide you with a list of websites that you think are relevant to your search and you can decide whether you want to trust them or not. AI searches integrate and rewrite the information itself, so it may not be clear if you are using unreliable sources.

“These systems aren’t entirely error-free, but I think the challenge is losing their ability to catch them over time,” Levin said. “They’re very persuasive and you don’t know how to actually go and verify it. Otherwise you’d think you don’t need to go and verify it.”

However, you can see all the sources. But it’s exactly like the kind of job that you want to avoid using this new system designed to save you time and effort.

“The question is, if you do this analysis for all queries you run in ChatGPT, what is the purpose of ChatGPT?” Levin said.