Why Google search upgrades make it harder for websites to win traffic

To publishers & businesses that have relied on Google to send users to their sites, subtle tweaks have siphoned off vital traffic & made it harder to reach customers.

Representational image | David Paul Morris/Bloomberg

New York: Type a query into the Google search bar on a smartphone and there’s a good chance the results will be dominated by advertising.

That stems from a decision in 2015 to test a fourth ad, rather than three, at the top of search results. Some employees opposed the move at the time, saying it could reduce the quality of Google’s responses, according to people familiar with the deliberations. But the company brushed aside those concerns because it was under pressure to meet Wall Street growth expectations, one of the people said.

By 2016, the extra marketing slot was a regular feature. It’s one of the many ways the search leader has altered how it presents results since its early days. Another example is the pre-packaged information Google often displays in a box at the top of a page, rather than sending users to other websites. Phased in gradually over years, changes like these have gone largely unnoticed by legions of consumers who regularly turn to Google to call up information and hunt for bargains. The company says these changes support its mission to organize the world’s information and make it useful and accessible to everyone.

But to many web publishers and other businesses that have historically relied on the internet giant to send users to their sites, Google’s subtle tweaks have siphoned off vital traffic and made it harder — and costlier — to reach customers online.

Debate over Google’s influence is gathering intensity as U.S. regulators prepare an antitrust case against the company in what will be one of the biggest legal clashes between the government and a corporation since the U.S. sued Microsoft Corp. in 1998. Google controls about 85% of the U.S. search market, and the changes it’s made have piled pressure on businesses to pay more to appear at the top of search results. That’s already a focus of regulators. Last year, David Cicilline, head of the House Subcommittee on Antitrust, asked Google if a 2004 statement from co-founder Larry Page that the company wants to get users “out of Google and to the right place as fast as possible,” still described its approach. In a written response, Google simply skipped the question.

The government is wading into a complex business with many competing interests that Google must balance. Users want the best answers. Web developers need eyeballs. Shareholders demand growth. From the beginning, websites have tried to trick Google’s algorithm into feeding them traffic, and they complain when the company cracks down. However, some web developers and advertisers say this balance has swung too far in Google’s favor.

In the early 2000s, the company’s search engine offered a simple deal: Produce quality information and Google will send you traffic so you can make money showing ads. Reinvest some of that cash to make better experiences, and the web will grow, giving Google more territory to explore and organize.

“Our search results are the best we know how to produce. They are unbiased and objective,” Page and co-founder Sergey Brin wrote when the company went public in 2004. Ads would be few, helpful and unobtrusive, they said.

This deal has been slowly changing, though. A turning point came in June 2019. That was when more than half of searches kept users on Google for the first time, rather than sending people to other sites through a free web link or an ad, according to data from digital marketing company Jumpshot.

“We’ve passed a milestone in Google’s evolution from search engine to walled-garden,” said Rand Fishkin, who has advised businesses on how to work with Google’s search engine for nearly two decades. “They used to be the good guys.”

On smartphones, the change has been more pronounced. From June 2016 to June 2019, the proportion of mobile searches that led to clicks on free web links dropped to 27% from 40%. No-click searches, which Fishkin says suggests the user found the information they needed on Google, rose to 62% from 56%. Meanwhile, clicks on ads more than tripled, Jumpshot data show.

When the search engine can give straightforward answers and save users a click, it will do that, and some sites have embraced this as a new way to gain traffic, according to Danny Sullivan, public liaison for Google’s search team. The company knows “the best information is coming from the web” and it wants to support the ecosystem, he added. Google also argues that ads keep the search service free for users and are confined to the small percentage of queries that suggest someone is looking to buy something.

In some cases, Google pays to summarize other companies’ content. Sports scores are one example, according to a May 20 blog post. The company is also planning to pay select media outlets in a news service later this year. But Google doesn’t think everyone’s content is worth paying for.

Mike Moloney runs FilterGrade, a marketplace for custom filters photographers use to edit their work. He gets most of his traffic from articles on his website, such as lens reviews and camera-related top 10 lists. Recently, he noticed Google pulling photos and text straight off the site and showing it at the top of search results. There’s a link to Moloney’s company at the bottom of the section, but clicking on any of the photos brings the searcher to another Google page full of shopping ads for film stock. None of these ads are related to, or benefit, FilterGrade — unless Moloney chooses to pay Google for placement.

“They’re doing a good job of making it subtle, almost like it’s an accident half the time,” Moloney said.

When Moloney tweeted his frustration in April, Google’s Sullivan said the company would review the practice. Several months later, the situation remained the same.

It’s often unclear who owns content online, especially when it’s relatively easy to scrape information from one site and re-purpose it quickly on a new web page. But even when ownership is not in dispute, Google’s combination of direct answers and extra ads has pushed free links to sites further down the search results page. Fishkin’s former colleague, Pete Meyers, has been testing the same list of 10,000 search terms for years. On average, users now have to scroll down twice as far to find the first organic free link, compared with 2013.

“This has been the slowest but most consistent march in tech,” venture capitalist Bill Gurley wrote on Twitter last year. “If you are still holding out hope for a SEO strategy you must be intentionally ignoring all of the data in front of you,” he added, referring to search engine optimization, a popular way of improving websites to rank higher in Google’s free results.

While businesses struggle to adjust, Google’s revenue and profit have surged. In 2009, the company generated sales of $24 billion and profit of $6.5 billion. Last year, parent Alphabet Inc. reported $162 billion in revenue and $34 billion in net income. The Search business alone brought in almost $100 billion in sales.

Much of that growth has come from adding more ads. On mobile phones, ads now take up the entire first screen for some searches. In 2015 and early 2016, when the company tested adding a fourth ad to the top of search results, there was push-back from some employees, according to people familiar with the situation who asked not to be identified discussing internal debates. The main concern was that the fourth ad was often lower quality than the first free web link right below, one of the people said. Google dismissed those worries and went ahead with the fourth ad slot because it was under pressure to keep revenue and profit growing to meet analysts’ expectations, this person added.

Google said it removed an ad slot on the side of the page when it added the fourth ad on top, leading to a lower number of overall ads for “highly commercial queries.” Fewer than 2% of all searches result in four or more text ads on the first page, according to the company.

Kevin Hickey, chief executive officer of Online Stores Inc., said these changes have forced him to spend more on Google search ads to keep traffic flowing to his e-commerce businesses. More than a decade ago, about two-thirds of Hickey’s Google traffic came from free, or organic, listings. But as Google increased ad slots to the top of results, that mix flipped. Organic results account for about 20% of visitors to his sites now, and he spends about 10% to 15% of his revenue on Google ads. He has raised prices, but his profit margins have shriveled.

“The prices that consumers are paying are now higher because of Google’s business model,” Hickey said.

Google doesn’t have a responsibility to pad the bottom lines of for-profit businesses. But one of the internet’s most beloved not-for-profit services has been caught up in this, too.

Wikipedia pages were some of the first that Google mined to answer search queries directly. The company would often fail to credit the digital encyclopedia clearly, leaving Wikipedia managers wondering if they were Google partners or simply bystanders, according to a person familiar with the situation. The concern inside Wikipedia is that its relevance will slip away the more its content is read in other places. The thousands of volunteers who write and edit the site’s articles may stop contributing if they see their hard work benefiting a trillion-dollar corporation instead of a non-profit, this person said. They asked not to be identified to preserve their relationship with Google.

“We regularly consider the impact of third-party use of Wikipedia’s information, especially as the public increasingly consumes content outside our sites,” a Wikipedia spokeswoman said. “We’ve worked with Google over the years to improve the way they credit content from Wikipedia in the knowledge panel so that the public clearly knows when they’re reading information from Wikipedia.”- Bloomberg