Celebrities are having a tough time making it big in the prepaid-card business.

Financial-advice personality Suze Orman and basketball legend Magic Johnson have shuttered their prepaid-card ventures. Rapper Lil Wayne has stopped accepting applications for his Young Money cards in the last few weeks, according to people familiar with the matter. Other celebrities, such as the Kardashians, have left the business as well after being criticized for charging high fees.

Their retreat shows how specialized cards endorsed by a single celebrity often struggle to compete with plain-vanilla cards issued by big banks that are pitched to a broader swath of consumers.

Prepaid cards typically are aimed at people without bank accounts or those looking to avoid bank fees. Customers load money onto the cards, including direct-deposit payroll checks, and can use the cards anywhere traditional credit or debit cards are accepted.

Ms. Orman, Lil Wayne and Mr. Johnson were part of a wave of celebrity-endorsed prepaid cards that peaked in the last two years. Other stars that have also attached their names and faces to plastic include Justin Bieber and George Lopez. The celebrities typically join forces with financial-services firms rather than invest much of their own capital.

But big financial institutions such as J.P. Morgan Chase JPM +0.75% JPMorgan Chase & Co. U.S.: NYSE ^$59.08 +0.44 +0.75% July 30, 2014 2:30 pm Volume (Delayed 15m) : 7.59M P/E Ratio N/A Market Cap $221.94 Billion Dividend Yield 2.71% Rev. per Employee $410,954 07/30/14 GE Set to Spin Off Synchrony A… 07/30/14 J.P. Morgan Launches Dedicated… 07/30/14 J.P. Morgan Tech Analyst Jumps… More quote details and news » JPM in Your Value Your Change Short position & Co. and American Express Co. AXP -0.59% American Express Co. U.S.: NYSE ^$91.17 -0.54 -0.59% July 30, 2014 2:30 pm Volume (Delayed 15m) : 3.21M P/E Ratio 17.87 Market Cap $97.08 Billion Dividend Yield 1.14% Rev. per Employee $565,016 07/30/14 Morning MoneyBeat: Midsummer M… 07/29/14 American Express Posts Higher … 07/29/14 Celebrities Struggle in Prepai… More quote details and news » AXP in Your Value Your Change Short position , hoping to generate new kinds of revenue from low-income consumers, have started rushing in as well.

That has made it tougher for celebrity cards to compete. Big banks have the advantage of scale, said consultants and others who track the industry. Banks already have the infrastructure to deal with an influx of new customers without having to spend much money ramping up.

Celebrity cards, on the other hand, often struggle because their potential customer base is limited to a group of fans that isn't always big enough to make the cards profitable for them or the financial firms that back the cards.

"Sometimes celebrities convince themselves that their brand has a bigger reach than it really does," said Paul Tomasofsky, president of Two Sparrows Consulting, Montvale, N.J.

Representatives for Ms. Orman, Mr. Johnson and Lil Wayne couldn't be reached for comment about the discontinuation of their cards. Financial details of celebrity cards typically aren't disclosed.

In theory, prepaid cards are an attractive business. They carry no credit risk for the card issuer because customers can't spend money they haven't put onto the card. Card issuers make money from the consumer fees that are tied to the card and from each time a merchants rings up a transaction.

The cards are catching on as more borrowers seek out cards that won't allow them to rack up debt and are cheaper and more convenient than high-fee check-cashing providers. In all, consumers made $132.9 billion of purchases using prepaid cards last year, more than double the $60.45 billion in sales made in 2009, according to the Nilson Report, a Carpinteria, Calif., newsletter.

But some issuers have been forced to reduce fees on the cards in the face of criticism from consumer advocates. Those fees are typically used to subsidize costs associated with maintaining the card, such as staffing customer-service centers. Bigger banks are able to absorb those fees easier than the smaller firms that typically back celebrity cards.

The Consumer Financial Protection Bureau, meanwhile, is a scrutinizing prepaid cards and is expected to issue new guidelines at the end of this year or in early 2015. The agency said Monday it would start accepting consumer complaints about prepaid cards.

The card endorsed by the Kardashian reality-television stars was pulled just weeks after its 2010 launch after consumer advocates lambasted its fee structure, which included a $99.95 purchase price. At the time they canceled the card, a lawyer for the Kardashians blamed the "negative spotlight."

"There is a stigma associated with these cards and the stigma has gotten worse instead of better," said Curtis Arnold, who runs a website called bestprepaiddebitcards.com that rates the cards.

Hip hop entrepreneur Russell Simmons in 2003 launched one of the first prepaid cards associated with a celebrity, called the RushCard. Mr. Simmons said that over the last decade, he has invested millions of dollars in the business, is a member of the company's board and has been vocal in defending and explaining prepaid cards to regulators and consumer advocates. The card has more than 500,000 active customers.

Most celebrities aren't as engaged, Mr. Simmons said. "The reason people get out of this space is because they can't make money at it," he said.

Mr. Simmons has overhauled the RushCard numerous times in the last decade. The company has lowered some fees and eliminated others after being criticized by consumer advocates for an initial fee structure that included a $19.95 activation fee and $1 charge tacked onto each purchase. The card now has two pricing plans that include initial fees of between $3.95 and $9.95 as well as monthly fees of $5.95 to $7.95.

"I have learned a lot and I'm still around and I'm still growing in a market that is new," Mr. Simmons said.

Lil Wayne's Young Money card, which has been sold since late 2011, stopped taking new applications within the last month, according to people familiar with the situation. Representatives for Sutton Bank, the Ohio lender that issued the card, couldn't be reached for comment.

Ms. Orman launched the Approved prepaid card in early 2012. The card cost $3 to buy and included other fees, such as a $3 monthly fee that was waived the first month, a $2 fee for automated teller machine withdrawals and a $1 fee to check the balance.

A customer-service representative who answered the card's toll-free number said the program went out of business on July 4. A representative of Bancorp Bank, the financial institution that issued the card, couldn't be reached for comment.

Mr. Johnson's card, called Magic, was launched in mid-2012 with an initial fee of $4.95, which was also its monthly fee. A statement on the card's website said the program was being "discontinued" on June 30.

A spokesman for OneWest Bank NA, which issued the Magic card, declined to comment.

Write to Robin Sidel at [email protected]