The regulations are expected to cut the supply of liquid nicotine and drive up costs for shop owners and prices for vapers. The big winners likely will be Big Tobacco. Companies like Altria Group Inc. MO 3.63 % and Reynolds American Inc., RAI 2.10 % which have their own e-cigarette brands, are better able to afford the approval process, said Kevin Altman, a consultant to the Council of Independent Tobacco Manufacturers of America.

Meanwhile, small vape businesses fret. “I can’t think of a manufacturer or a retailer it doesn’t keep up at night,” said Schell Hammel, a former smoker and hospital administrator who opened her first vape shop in McKinney, Texas, in 2011.

The FDA’s proposal has made Ms. Hammel’s plans to open more shops an “internal struggle.” She has negotiated leases that allow her to close her seven shops if new government rules damage business. She worries FDA rules would bring an end to the shops’ line of liquid-nicotine brands, which she recently spent more than $45,000 to test. But she plans to operate as long as she can.

“I’m not going to leave the business for financial reasons,” Ms. Hammel said. “That makes me crazy, I know.”

Some of the fiercest advocates of e-cigarettes are those like Ms. Hammel, who used them to kick a 24-year smoking habit. Researchers agree e-cigarettes are less harmful than traditional cigarettes, which release more than 60 carcinogens through combustion, and some studies have shown they can be effective in helping smokers quit.

But e-cigarettes, which arrived in the U.S. around 2008, are so new that their long-term effects remain unknown. Health officials, antitobacco groups and lawmakers have grown concerned about them because cases of nicotine poisoning in children from e-cigarette products have increased, and the Centers for Disease Control and Prevention released a study in April indicating teen e-cigarette use tripled in 2014. Health officials and antitobacco groups are concerned e-cigarettes could become a gateway to traditional cigarettes.

Contests called “cloud competitions” are being held across the U.S., where vapers try to blow the biggest cloud. Photo: Brandon Thibodeaux for The Wall Street Journal

The FDA said in a statement that the new rules will help it “learn more about these products and ultimately ensure that newly regulated products cause no worse harm to users.” The agency added that it plans to provide small businesses time to comply with a final rule.

Not everyone in the vapor industry opposes regulation. Vapeology L.A. owner John Hartigan, 54, said the absence of regulation today puts the onus on him to ensure the liquid nicotine he sells comes from manufacturers blending in controlled environments and doing chemical analysis. “The idea that you won’t have people out there putting juice together without a chemistry background is a good thing,” Mr. Hartigan said.

In the absence of regulation, more than 8,500 vape shops have sprung up in strip malls and stand-alone stores across the country.

Many have become places where former smokers congregate, learn about new devices and try different juices. The clientele ranges from tattooed millennials like Jeff Eldred, 34, a property maintenance worker from Wylie, Texas, to Jerry Davis, 53, a small-business owner in Atlanta.

Mr. Davis was dressed in a golf shirt and khaki shorts last month when he stopped by Vape Atlanta to sample a variety of liquid nicotine flavors like Fruit Loop-cereal inspired “Tucan Slam.” He said he has kicked his three-pack-a-day cigarette habit and estimated he is saving $8,000 a year in health insurance because he stopped smoking.

Vape shop owner Cheryl Richter, who smoked for 32 years, said helping people like Mr. Davis is why she opened Vape Den last year in Port Chester, N.Y. About 70% of the shop’s business comes from its liquid nicotine juices. If FDA rules force her to close the shop, she said she would return to her life as an advertising copy writer or the acting career she once pursued.

“It’s really a matter of just how crazy they will make it for this industry to exist,” Ms. Richter said.

Some liquid nicotine manufacturers are investing in the research tools necessary to apply for FDA approval.

Mr. Walsh, the CEO of PureBacco, estimated it sank $500,000 over the past year into upgraded manufacturing facilities and analytics equipment for his Flavorz juice brand. Five Pawns Inc., a California-based liquid-nicotine company, hired a half-dozen advisers to assist with regulatory, scientific, manufacturing and legal issues related to regulation.

“It’s going to become very expensive to play in the game,” said Five Pawns Chief Executive Rodney Jerabek. “This could mean the end for a lot of small companies.”