Inside the secret world of Trader Joe’s

But Trader Joe’s is no ordinary grocery chain. It’s an offbeat, fun discovery zone that elevates food shopping from a chore to a cultural experience. It stocks its shelves with a winning combination of low-cost, yuppie-friendly staples (cage-free eggs and organic blue agave sweetener) and exotic, affordable luxuries—Belgian butter waffle cookies or Thai lime-and-chili cashews—that you simply can’t find anyplace else.

Employees dress in goofy trademark Hawaiian shirts, hand stickers out to your squirming kids, and cheerfully refund your money if you’re unhappy with a purchase—no questions asked. At the Chelsea store opening, workers greeted customers with high-fives and free cookies. Try getting that kind of love at the Piggly Wiggly.

It’s little wonder that Trader Joe’s is one of the hottest retailers in the U.S. It now boasts 344 stores in 25 states and Washington, D.C., and strip-mall operators and consumers alike aggressively lobby the chain, based in Monrovia, Calif., to come to their towns. A Trader Joe’s brings with it good jobs, and its presence in your community is like an affirmation that you and your neighbors are worldly and smart.

The privately held company’s sales last year were roughly $8 billion, the same size as Whole Foods’ (WFM) and bigger than those of Bed Bath & Beyond, No. 314 on the Fortune 500 list. Unlike those massive shopping emporiums, Trader Joe’s has a deliberately scaled-down strategy: It is opening just five more locations this year. The company selects relatively small stores with a carefully curated selection of items. (Typical grocery stores can carry 50,000 stock-keeping units, or SKUs; Trader Joe’s sells about 4,000 SKUs, and about 80% of the stock bears the Trader Joe’s brand.) The result: Its stores sell an estimated $1,750 in merchandise per square foot, more than double Whole Foods’. The company has no debt and funds all growth from its own coffers.

You’d think Trader Joe’s would be eager to trumpet its success, but management is obsessively secretive. There are no signs with the company’s name or logo at headquarters in Monrovia, about 25 miles east of downtown Los Angeles. Few customers realize the chain is owned by Germany’s ultra-private Albrecht family, the people behind the Aldi Nord supermarket empire. (A different branch of the family controls Aldi Süd, parent of the U.S. Aldi grocery chain.) Famous in Germany for not talking to the press, the Albrechts have passed their tightlipped ways on to their U.S. business: Trader Joe’s and its CEO, Dan Bane, declined repeated requests to speak to Fortune, and the company has never participated in a major story about its business operations.

Some of that may be because Trader Joe’s business tactics are often very much at odds with its image as the funky shop around the corner that sources its wares from local farms and food artisans. Sometimes it does, but big, well-known companies also make many of Trader Joe’s products. Those Trader Joe’s pita chips? Made by Stacy’s, a division of PepsiCo’s Frito-Lay (PEP). On the East Coast much of its yogurt is supplied by Danone’s Stonyfield Farm. And finicky foodies probably don’t like to think about how Trader Joe’s scale enables the chain to sell a pound of organic lemons for $2.

To get inside the mysterious world of Trader Joe’s, Fortune spent two months speaking with former executives, competitors, industry analysts, and suppliers, most of whom asked not to be named. What emerged is a picture of a business at a crossroads: As the company expands into new markets and adds stores—analysts say the grocer could easily triple its size in the coming years—it must find a way to maintain its small-store vibe with customers. “They see themselves as a national chain of neighborhood specialty grocery stores,” says Mark Mallinger, a Pepperdine University professor who has done research for the company. “It means you want to create an image of mom and pop as you grow.” That’s no easy task. Just ask Starbucks (SBUX) CEO Howard Schultz, whose expansion has been a huge success but has come at the expense of credibility with some coffee aficionados. The alternative is to remain a small brand with unflagging devotees, like outdoor clothier Patagonia. If it can get the balance right, Trader Joe’s may be one of the few retailers to marry cult appeal with scale. Just don’t expect anyone from the company to talk about it.

Who’s a fan of Trader Joe’s? Young Hollywood types like Jessica Alba are regularly photographed brandishing Trader Joe’s shopping bags—but Supreme Court Justice Sonia Sotomayor reportedly is a fan too. “What’s not to like?” says Costco (COST) co-founder and CEO Jim Sinegal. “They’re very good retailers, and we admire them a lot.” Visit a Trader Joe’s early in the day, and there are senior citizens on fixed incomes shopping for bargains; on weekends and evenings a well-heeled crowd takes over. Kevin Kelley, whose consulting firm Shook Kelley has researched Trader Joe’s for its competitors, jokes that the typical shopper is the “Volvo-driving professor who could be CEO of a Fortune 100 company if he could get over his capitalist angst.”

The rise of Trader Joe’s reflects Americans’ changing attitudes about food. While Trader Joe’s is not a health food chain, it stocks a dizzying array of organics. It sells billions of dollars in food and beverages that years ago would have been considered gourmet but are now mainstays of the U.S. diet, such as craft beers and white-cheese popcorn. The genius of Trader Joe’s is staying a step ahead of Americans’ increasingly adventurous palates with interesting new items that shoppers will collectively buy in big volumes.

The retailer’s foodie roots and quirky in-store culture date to the original Joe. Joe Coulombe (pronounced COO-lomb), now 80, opened the first Trader Joe’s 43 years ago in Pasadena to serve a sophisticated—but strapped—consumer. He named the store Trader Joe’s to evoke images of the South Seas. He stocked it with convenience-store items and good booze, and at one time his shop boasted the world’s largest assortment of California wine. (Decades later Trader Joe’s would again become famous for wine, specifically its $1.99 Charles Shaw label, better known as “Two-Buck Chuck.”) Coulombe then added health food—a seemingly odd combination that totally worked in 1970s California. By the late 1970s he was operating more than 20 locations.

The company’s success did not go unnoticed. German grocery mogul Theo Albrecht, who died in July at age 88, coveted Trader Joe’s—not as part of a major U.S. expansion but as a smart financial investment. Even in the early days, Trader Joe’s appeal was its narrow but zany selection and loyal customers, recalls Dieter Brandes, who did due diligence on the company for Albrecht. “It was fantastic. It was different,” he says. In 1979, Coulombe sold his company to Albrecht. Coulombe tells Fortune he “can’t remember” the selling price.

The Albrechts, who own Trader Joe’s through a family trust, have generally stayed out of the business. They visit the U.S. operation about once a year, and word around the office spreads that “the Germans” are coming. Coulombe stayed on without a management contract for a decade; in 1987 he hired John Shields, a fraternity brother from his undergraduate days at Stanford, who was CEO until 2001. Under Shields’ reign, Trader Joe’s expanded outside California to Arizona in 1993 and to the Pacific Northwest in 1995. Although executives worried that Northeastern shoppers wouldn’t “get” Trader Joe’s, the company in 1996 leapfrogged the country and opened two stores in places crawling with college professors and other bargain-hunting elites: Brookline and Cambridge, both outside Boston.

Push your way into the bustling Trader Joe’s in Manhattan’s Union Square neighborhood, and it’s hard to believe that executives ever worried that East Coasters wouldn’t groove on the experience. Make no mistake: A typical family couldn’t do all its shopping at the store. There’s no baby food, toothpicks, or other necessities. But for this crowd of urbanites and college kids, Trader Joe’s is nirvana.

A closer look at its selection of items underscores the brilliance of Coulombe’s limited-selection, high-turnover model. Take peanut butter. Trader Joe’s sells 10 varieties. That might sound like a lot, but most supermarkets sell about 40 SKUs. For simplicity’s sake, say both a typical supermarket and a Trader Joe’s sell 40 jars a week. Trader Joe’s would sell an average of four of each type, while the supermarket might sell only one. With the greater turnover on a smaller number of items, Trader Joe’s can buy large quantities and secure deep discounts. And it makes the whole business—from stocking shelves to checking out customers—much simpler.

Swapping selection for value turns out not to be much of a tradeoff. Customers may think they want variety, but in reality too many options can lead to shopping paralysis. “People are worried they’ll regret the choice they made,” says Barry Schwartz, a Swarthmore professor and author of The Paradox of Choice. “People don’t want to feel they made a mistake.” Studies have found that buyers enjoy purchases more if they know the pool of options isn’t quite so large. Trader Joe’s organic creamy unsalted peanut butter will be more satisfying if there are only nine other peanut butters a shopper might have purchased instead of 39. Having a wide selection may help get customers in the store, but it won’t increase the chances they’ll buy. (It also explains why so often people are on their cellphones at the supermarket asking their significant other which detergent to get.) “It takes them out of the purchasing process and puts them into a decision-making process,” explains Stew Leonard Jr., CEO of grocer Stew Leonard’s, which also subscribes to the “less is more” mantra.

Customers accept that Trader Joe’s has only two kinds of pudding or one kind of polenta because they trust that those few items will be very good. “If they’re going to get behind only one jar of Greek olives, then they’re sure as heck going to make sure it’s the most fabulous jar of Greek olives they can find for the price,” explains one former employee. To ferret out those wow items, Trader Joe’s has four top buyers, called product developers, do some serious globetrotting. A former senior executive told me that Trader Joe’s biggest R&D expense is travel for those product-finding missions. Trade shows that feature the flavor of the moment “are for rookies,” a former buyer said. Trader Joe’s doesn’t pick up on trends—it sets them.

The other dozen or so buyers, or category leaders, spend more time in the office, fielding hundreds of cold calls a week from vendors tripping over themselves to make Trader Joe’s a customer. Trader Joe’s is a supplier’s dream account: It pays on time and doesn’t mess with extra charges for advertising, couponing, or slotting fees that traditional supermarkets charge suppliers to get their products onto the shelves. “It’s all transparent—no BS,” says a former executive. In exchange, suppliers have to agree to operate under Trader Joe’s cloak of secrecy. Fortune obtained a copy of a standard vendor agreement, which states, “Vendor shall not publicize its business relationship with TJ’s in any manner.”

Why the lockdown? Former executives say that Trader Joe’s wants neither its shoppers nor its competitors to know who’s making its products. And many suppliers aren’t that keen on consumers knowing that they produce a lower-cost version for Trader Joe’s either. Take Tasty Bite, which makes much of Trader Joe’s Indian food. The Tasty Bite Punjab Eggplant ran $3.39 at a Whole Foods in Manhattan. The seemingly identical Punjab Eggplant that the Stamford, Conn., company makes for Trader Joe’s is more than $1 cheaper.

Over the years Trader Joe’s has improved the way it distributes Joe’s-branded goodies to its stores. Management has sought to minimize the number of hands that touch a product; whenever possible, Trader Joe’s purchases directly from the manufacturers, which then ship their wares straight to Trader Joe’s distribution centers. A U.S.-made cheese, for example, is sent to distribution centers nationwide, where it’s sometimes cut and wrapped, taking another cost out of the equation. At a traditional supermarket, that same cheese would probably go through a distributor first, tacking on another cost. Trucks leave the distribution centers daily for the stores. Trader Joe’s small stores don’t have much of a back room, so ordering from the distribution centers has to be precise.

This distribution process helps determine where the company opens its stores. Texas and Florida have cities that boast consumers Trader Joe’s covets, but insiders say the current distribution infrastructure makes it difficult for the company to efficiently get products to those states. To pick their next locales, employees look at demographics such as education level. In the past they’ve even looked at who’s subscribing to high-end food and cooking magazines as a way of divining where the epicures are.

On a Tuesday evening just before dinnertime, retail expert Burt P. Flickinger III joins the steady hum of foot traffic at the Trader Joe’s in Larchmont, N.Y. Because Trader Joe’s won’t give Fortune any information on its stores, Flickinger, of consulting firm Strategic Resource Group, has agreed to walk through a few suburban locales and offer feedback. In Larchmont, Flickinger does a little bit of his own shopping. (It’s what happens when you walk into a Trader Joe’s—you get sucked into buying stuff you didn’t plan to.) An employee, noticing that he has his arms full, brings him a basket. At the register the perky cashier offers up that the mango sorbet Flickinger has selected is on her top 10 list of favorite Trader Joe’s items.

You can’t buy engagement from employees, but the pay at Trader Joe’s helps. Store managers, “captains” in Trader Joe’s parlance—the nautical titles are a holdover from Coulombe (newly promoted captains are commanders; assistant store managers are first mates)—can make in the low six figures, and full-time crew members can start in the $40,000 to $60,000 range. But on top of the pay, Trader Joe’s annually contributes 15.4% of employees’ gross income to tax-deferred retirement accounts.

All of that can lead to a better customer experience. A ringing bell instead of an intercom signals that more help is needed at the registers. Registers don’t have conveyor belts or scales, and perishables are sold by unit instead of weight, speeding up checkout. Crew members aren’t told the margins on products, so placement decisions are made based not on profits but on what’s best for the shopper. Every employee works all aspects of the store, and if you ask where the roasted chestnuts are he’ll walk you over instead of just saying “aisle five.” Want to know what they taste like? He can probably tell you, and he might even open the bag on the spot for you to try.

Can Trader Joe’s maintain that kind of charm as it expands? Former employees worry that the company is losing its entrepreneurial zeal and that CEO Dan Bane has made the place more corporate, adding more senior vice presidents, and creating new titles such as product developer. At headquarters Bane encourages employees to wear Hawaiian shirts and name tags. But putting systems in place isn’t necessarily a bad thing. “You have to grow up at some point,” says a former employee. “You have to start following rules. You have to start putting in checks and balances.” The stakes are higher now that Trader Joe’s has hundreds of stores. A buying error could cost the company millions.

Bane, 62, who has a background in accounting, graduated in 1969 from the University of Southern California, where he played baseball—or, as he’s said, “spent a lot of the time on the bench.” During a talk at USC, Bane said that he’s modeled his leadership style on his famed coach, Rod Dedeaux. Bane joined the company in 1998 as president of West Coast operations and became CEO only three years later.

A few former employees describe him as gruff, but he also has a softer side. In a video tribute to a sixth-grade teacher named Mrs. Bidwell, he talked about how she helped him adjust to life in El Dorado, Ark., after the Navy relocated Bane’s father there from Southern California.

Some former employees say Trader Joe’s has already lost its quirky cool. “In the early days we never tried to be the neighborhood store,” says a former employee. They didn’t have to: Trader Joe’s was the neighborhood store. And yet walk into the Chelsea location on a busy weekday night and you’ll see something you almost never see in Manhattan: strangers chatting with one another. Veteran customers tell newbies what products they absolutely have to try, and serious cooks share tips on how to spike sauces and semi-prepared foods to make them even tastier. If Trader Joe’s can maintain that kind of mojo, it could end up the biggest neighborhood store ever.