It’s summer vacation time. Tourists are saying no to America.

“There used to be thousands of people from Canada. They would give me Canadian dollars and I would change it at the bank," said Omar Tallat, 35 years old, who runs a corn dog stand near Times Square. “This year, business is very bad."

About 1.9 million foreigners arrived at the U.S.’s main airports in the past four weeks, down 6% from the same period last year, according to data from U.S. Customs and Border Protection. Airline bookings data for the summer suggest things won’t be picking up soon. Flight bookings to the U.S. from Europe are down by about 12% through August. San Francisco, Washington, D.C., and Los Angeles are seeing even larger declines, according to an analysis of online travel-agency booking data from Cirium.

Overseas travelers say they are swapping U.S. vacation plans for trips to Europe or in their own countries. Some cite the Trump administration’s immigration crackdown and reports of foreign visitors being detained or deported from the U.S. Others say they want to signal their discontent with the White House’s policies, echoing boycotts of American-made products by Canadians and Teslas in Europe.

“There’s not a lot politically I can do, but what I can do is not spend my money in the U.S.," said Marc Toews, a 53-year-old Canadian long-haul truck driver. He was planning a road trip through the U.S. that would take him to North Dakota, a Chicago Cubs game, and the Statue of Liberty.

“I was very excited about it. I’ve been talking about this trip for a few years," he said.

He said President Trump’s comments about annexing Canada changed his mind. Instead, he plans to do a road trip through Canada in the fall.

Canadians are the largest group of international visitors to the U.S., historically making up around a quarter of arrivals. They are now the largest weakness. Air travel from Canada to the U.S. in April dropped 20%, while land crossings dropped 35%, according to the Canadian government. Flight bookings to the U.S. for the summer are down 22% compared with last year, according to Cirium’s data, with Los Angeles and Miami down by a third.

In Plattsburgh, N.Y., about 25 miles south of the border with Canada, John Parmelee, the manager of American restaurant Naked Turtle, said he has seen a drop off in Canadian visitors since opening at the start of May. The city is in the North Country region of the state, where roughly 70% of summer visitors are Canadian.

To try to counter the trend, Parmelee recently put up a “Bienvenue Canadiens!" sign in front of the restaurant and plans to put up a giant Canadian flag next. He is also considering offering discounts to Canadian visitors.

“We are very concerned about what’s ahead of us," he said. “It’s a bit analogous to Covid. We don’t know where this is going to go."

Businesses in the region are rolling out discount deals for Canadians on everything from lodging to bike rentals, according to Kristy Kennedy, vice president of marketing and tourism at the North Country Chamber of Commerce.

The travel and tourism industry makes up about 3% of overall U.S. gross domestic product, and foreigners only contribute a sliver of that. A pullback would likely have a marginal impact on wider economic activity, according to a JPMorgan analysis.

But some areas could get hit hard. That includes towns near the Canadian border like Plattsburgh and beach towns along the Jersey Shore or along the south Atlantic, or in Maine, where generations of families from Canada and elsewhere have summered.

The tourism industry is dominated by small businesses, which are less able to absorb swings in demand, according to Aran Ryan, director of industry studies at Tourism Economics.

Tourism Economics initially expected the U.S. to see a 16% rise in international visitor spending in 2025. It now expects it to fall by about $8.5 billion, a decline of roughly 5%.

Many businesses say there is little they can do to reverse the tide. Some are hoping domestic tourism can fill the gap, but Americans have grown concerned about the economy and are also scaling back their vacation plans. Foreign tourists also are more lucrative, as they tend to stay for longer and spend more than Americans. Palm Springs, Calif., temporarily hung red “Palm Springs ‘loves’ Canada" street banners around the city in April.

A banner in support of Canada is seen in Palm Springs, Calif.

David Byrne, a retired executive who lives in London, had expected to spend about $7,000 for a trip to New York for the U.S. Open Tennis Championships in late summer. But he said he called it off, citing what he sees as antagonism from the Trump administration toward visitors. He’s booked a two-week trip to the Greek island Santorini.

“We end up spending quite small fortunes on holidays," he said. “We exercise our choice by going to other places. We’ll spend our money in Europe."

Scott Mills, general manager at the Garland Hotel in North Hollywood, Calif., said international visitors are down about 30% this year. Bookings for the coming months have fallen, too. The hotel counts Britons, Australians and Canadians among its customers.

“Part of it is the politics, part of it is the fear of visa issues, and part of it is the optics of the fires," said Mills, referring to the January wildfires that destroyed thousands of homes. Some international tourists mistakenly believe the whole city burned to the ground, he said.

Achim Diergarten, a 67-year-old lawyer who lives outside of Munich, had planned to spend about two weeks driving around the West Coast with his wife this summer, with stops in San Diego, Los Angeles and Las Vegas. Diergarten is critical of Trump on social media and canceled his trip after reading about Europeans who were deported or detained in the U.S. and had their phones searched.

“I have to worry about being stopped at immigration or even arrested," he said. Diergarten said he plans to go to Vancouver, British Columbia, instead.

“The U.S.A. is off limits for me for at least the next four years," he said.

Write to Chelsey Dulaney at chelsey.dulaney@wsj.com and Chao Deng at chao.deng@wsj.com