U.S. stocks fell Thursday, as concerns about slowing U.S. economic growth and rising pandemic cases weighed on investor sentiment.

All three major U.S. indexes sank, closing near their session lows, after erasing gains notched earlier in the day. The $S&P 500 index(.SPX.US)$ and $Dow Jones Industrial Average(.DJI.US)$ both fell for a fourth consecutive session, their longest losing streak since mid-June.

The benchmark index lost 20.79 points, or 0.5%, to finish at 4493.28 and the Dow shed 151.69 points, or 0.4%, to end at 34879.38. The technology-heavy $Nasdaq Composite Index(.IXIC.US)$ fell 38.38 points, or 0.3%, to 15248.25

Trading so far in September has been choppy, and major indexes have largely finished each session with small losses or gains. Money managers say that some of those moves have been driven by uncertainty surrounding the future pace of the U.S. economic recovery, especially with the continuing threat of rising pandemic cases and hospitalizations in many states.

There have some signs of slowdown. The Federal Reserve's Beige Book report showed Wednesday that economic growth softened slightly in early July through August, as consumers cut spending and businesses grappled with supply-chain bottlenecks and labor shortages. The report, which collects anecdotes from around the country, noted a pullback in activities such as travel and dining out due to safety concerns.

Still, a mix of positive data has helped offset some of those concerns. New figures Thursday showed that the number of Americans who applied for first-time unemployment benefits, a metric that is seen as a proxy for layoffs, fell to 310,000 in the week ended Sept. 4. Claims have trended lower since mid-July, a sign that employers are holding onto workers.

Some investors and analysts say they wouldn't be surprised if stocks retreated in the weeks or months ahead, with many noting that the S&P 500 hasn't suffered a 5% pullback since October. Still, many add that they see few alternatives outside of stocks.

"We have been among the voices saying...we are overdue for a correction, " said Lauren Goodwin, economist and portfolio strategist at New York Life Investments. But, she added, "What choices do investors have? Is there an alternative to equities when rates are so low and [forecasts show] the economy is still growing?"

Many economists expect U.S. gross domestic product to grow in the third quarter, though some have lowered earlier forecasts.

In the days and weeks ahead, investors say they will be closely watching for signs regarding when the Federal Reserve will begin to pare back easy-money policies implemented during the pandemic. In an interview with The Wall Street Journal, Federal Reserve Bank of Atlanta President Raphael Bostic said he believes the central bank will be able to pull back on its asset-buying this year, though he doesn't anticipate a decision to do so will come at the Fed's meeting this month.

"We're slightly more cautious," said Charles Hepworth, an investment director at GAM Investments. "It does feel that people are getting a bit freaked out by valuations. If you get tapering too soon, that risks derailing the recovery."

Among the S&P 500's 11 sectors Thursday, only three groups -- financials, energy and materials -- finished the day higher, with financials leading the way. For individual stocks, Moderna was was the best performer for the benchmark index on a percentage basis, adding 7.8%, or $33.02, to finish at $455.92, extending its blockbuster year-to-date gains to 336%.

Shares of $GameStop(GME.US)$ ended higher, adding 38 cents, or 0.2%, to finish at $199.18, reversing losses that came earlier in the day after the videogame retailer reported mixed second-quarter results. $Lululemon Athletica(LULU.US)$ shares gained $39.86, or more than 10%, to end at $420.71, a new all-time high, after it posted revenue that exceeded its guidance and analysts' expectations.

Overseas , Hong Kong's $Hang Seng Index(800000.HK)$ tumbled 2.3%, its biggest one-day drop in over six weeks. Shares of Chinese videogame giants $Tencent(00700.HK)$ and $NetEase(NTES.US)$ fell in Hong Kong after authorities summoned the companies and ordered them to follow new rules for the online-gaming industry.

In the U.S., NetEase's American depositary receipts pulled back 2.1%. Other Chinese stocks that trade in the U.S. also slid, with $Alibaba (BABA.US)$'s American depositary receipts slipping 2%.

The pan-continental Stoxx Europe 600 fell 0.1%. The European Central Bank said Thursday it will slow down its purchase of government bonds through an emergency program that was meant to bolster credit markets and growth during the pandemic.

In bond markets, the yield on the 10-year Treasury note fell to 1.300%, from 1.333% Wednesday, marking two consecutive days of losses.

Other major indexes in Asia broadly closed lower. South Korea's Kospi fell 1.5% and Japan's Nikkei 225 declined 0.6%.

China's Shanghai Composite Index bucked the trend, gaining 0.5%.

Write to Caitlin Ostroff at caitlin.ostroff@wsj.com and Caitlin McCabe at caitlin.mccabe@wsj.com