By Marty Swant • August 26, 2024 •
Ivy Liu
Large language models powering AI search platforms like ChatGPT and Perplexity are still largely a black box, but some CRM and SEO providers are looking to give marketers more info about how brands appear on the inside — even from the outside.
Hubspot, for example, recently debuted a free new tool called AI Search Grader, which offers a way to test how brands are perceived by large language models — starting with OpenAI’s GPT-4o model. Through a partnership with OpenAI, the feature gives companies a quick report with benchmark metrics — such as brand sentiment and share of voice — along with personalized insights about product mentions, strengths and weaknesses within a category.
According to Hubspot CMO Kipp Bodnar, the goal is to help marketers better understand how brands appear in an LLM and help them start navigating an increasingly fragmented world of search beyond Google. It also uses some of the DNA from earlier Hubspot tools like its Website Grader. However, he said some aspects will get better in the coming months. The company also hopes to expand to other LLMs, but those deals are still in talks.
“We were trying to figure out if we can actually identify the competitors in the category and map that well,” Bodnar said. “And it turns out, for most established categories, you can. I think the sentiment is still the hardest part.”
To create AI Search Grader, the team started with qualitative analysis using prompts representative of how people use to find information about brands and companies. After that, the AI model’s answers are fed to Hubspot’s machine learning model to conduct a sentiment analysis before re-translating back into natural language for the final report.
Generative AI search platforms and chatbots still don’t provide the same granular info about search queries that brands are used to getting with traditional search. However, Hubspot’s tool aims to save companies the cost and time usually required to do a full LLM audit, according to Josh Blyksal, an engineer who worked on the AI Search Grader. He added that it’s also a very different process than an SEO audit. He thinks small businesses will benefit a lot from the tool and that between 50 and 100 were tested along with medium and larger companies.
“It’s a war against that out of control feeling like you’re not in the driver’s seat,” Blyksal said. “On even the best of days, marketers are feeling reactive. On the worst days, they’re literally running away from AI, they don’t want to know more about it, they’re not concerned about its impacts…Before you can do anything with AI search, you have to be able to understand your orientation.”
With or without the AI Search Grader, how LLMs operate are still mostly opaque and also a mystery to even those building them. Some say it’s hard to know what to do with the insights from AI Search Grader without knowing all the inputs and outputs used for the report. The questions asked determines the outcome, noted Isaac Gerber, head of insights and analytics at Captify. After testing the tool, he doesn’t think scores and rankings aren’t entirely helpful yet, but said it’s still a useful tool for brand benchmarking.
“If people haven’t tried [AI Search Grader] yet, brands should better understand this and pay attention even if it’s not perfect yet,” Gerbert said.
Perplexity ads enter the chat
Hubspot’s new feature comes as the race to overhaul search with AI heats up. Last week, Perplexity took aim at Google in a new and began circulating an advertising pitch deck to marketers before introducing ads in the fourth quarter of 2024.
According to a copy of the pitch deck obtained by Digiday, the plan is to integrate ads into the conversational answers within user queries. One example shows of Nike ad sponsoring a related question to user’s query about basketball shoes while and options for sponsored videos alongside a desktop version. Another example shows a sponsored Marriott video ad next to someone asking about “best European travel destinations for families.”
Perplexity is pitching an educated, affluent and engaged user base, with total monthly U.S. queries now totaling 230 million — eight times the total queries it had just a year ago. According to the deck, buying ads with Perplexity can help brands “reach, educate and spark curiosity of potential customers at high-leverage moments.”
Meanwhile, Google itself has been adding more features to AI Overviews and additional citations. There’s also still another key wildcard: How will Google’s search antitrust case loss change the landscape, depending on the court’s recommended remedies?
Google’s dominance hasn’t shifted marketers main priorities yet. However, experts say marketers need to start putting focus beyond just the giant. That’s partially because AI platforms vary in how they rank information based on information, prompts and other signals. However some say it’s still important to pay close attention to challengers like SearchGPT and Perplexity.
“What we’re seeing broadly with Gen AI and our clients broadly is we just have to amplify what we’ve been doing consistently on the SEO side,” said Chris Hansen, chief client officer at Power Digital. “Nothing has changed dramatically yet.”
Prompts, platforms and progress
To help navigate the new landscape, some SEO companies are conducting their own research about SearchGPT, Gemini and other AI platforms. BrightEdge last week released a new guide comparing results in SearchGPT, Google’s AI Overviews and Perplexity with information about categories like e-commerce, finance and education, and healthcare.
“What we’ve learned is these AI engines deal with each domain differently and different types of keyword intent very differently,” said BrightEdge founder and CEO Jim Yu. “That helps us understand the kinds of AI experiences where they’re building their differentiated experiences.”
The ways LLMs index pages will impact what companies should include on websites and in their overall content strategy, said Damian Rollison, director of Market Insights at SOCi. The SEO agency SoCi also released a new report last week about how location information affects answers in AI search results. Although AI Overviews is so far mostly impacting general search, he said companies might want to start thinking about providing different heading options depending on what someone might search for. He said AI answers only matched traditional search results 35% of the time for local queries.
“There’s no concept of ‘page two’ with AI results, so cracking the code of what will get your business into the initial answer is going to be critical,” Rollison said. “So far, we’re seeing that AI is looking for sources of deep information, so in addition to platforms like Google Maps businesses will have to ensure that they’re creating rich local landing pages for every store or office.”
Prompts & Products — AI News and announcements
- McAfee and Lenovo released a new tool that lets people detect audio deepfakes using their laptop. It also released a new platform to help educate people about AI scams.
- PilotDesk and LG Ad Solution released a new tool for AI workflow automation platform for advertising and yield operations.
- OpenAI announced another publisher deal, this time with Conde Nast.
- The Washington Post released a new AI tool for the newsroom to help reports analyze massive data sets.
- California state lawmakers agreed to a deal with Google rather than move forward with a bill that would’ve required Google to have ad revenue shares with local media.
- News Literacy Project launched a new election misinformation database with more than 500 pieces of content.
- The image AI startup Midjourney debuted a new platform that doesn’t rely solely on Discord.
- IBM and the US Tennis Association began another year of partnering for the US Open, with new features for AI-generated insights, content and commentary.
1s and 0s — Reports and other insights
- A new Forrester report says two thirds of B2C marketing execs in the US and nearly half of marketers in APAC are exploring use cases for genAI for marketing.
- IDC’s latest report says global spending on AI will more than 2x in the next five years $632 billion by 2028.
- A new report from Deloitte looks at generative AI adoption and investment by enterprise customers.
- A Stanford legal expert and former Google lawyer analyzed legal language in tech regulations to warn about unintended outcomes for free speech, AI and online content.
Other stories from across Digiday
- What Gartner’s 2024 digital ad hype cycle shows about marketing innovation and adoption
- How Amazon’s generative AI tool for developers is saving 4,500 years of work, $260 million annually
- ‘There needs to be an AI clause’: AI hype sparks influencer contract overhauls for name, image and likeness
- Media Briefing: The 2024 media glossary
- Five key signs that Reddit is getting ready to launch its own search ads business
https://digiday.com/?p=553464
