Toyota Research Institute’s robots leave home

“I think I’m probably just as guilty as everybody else,” Toyota Research Institute’s (TRI) senior vice president of robotics, Max Bajracharya, admits. “It’s like, now our GPUs are better. Oh, we got machine learning and now you know we can do this. Oh, okay, maybe that was harder than we thought.”

Ambition is, of course, an important aspect of this work. But there’s also a grand, inevitable tradition of relearning mistakes. The smartest people in the room can tell you a million times over why a specific issue hasn’t been solved, but it’s still easy to convince yourself that this time — with the right people and the right tools — things will just be different.

In the case of TRI’s in-house robotics team, the impossible task is the home. The lack of success in the category hasn’t been for lack of trying. Generations of roboticists have agreed that there are plenty of problems waiting to be automated, but thus far, successes have been limited. Beyond the robotic vacuum, there’s been little in the way of breakthrough.

TRI’s robotics team has long made the home a primary focus. That’s driven, in no small part, by it choosing eldercare as a “north star” for the same reason that Japanese firms are so far ahead of the rest of the world in the category. Japan has the world’s highest proportion of citizens over the age of 65 — trailing only Monaco, a microstate in Western Europe with a population of fewer than 40,000.

In a world where our health and wellness are so closely tied to our ability to work, it’s an issue bordering on crisis. It’s the kind of thing that gets Yale assistant professors New York Times headlines for suggesting mass suicide. That’s obviously the most sensationalistic of “solutions,” but it’s still an issue in search of meaningful solution. As such, many Japanese roboticists have turned to robotics and automation to address issues like at-home healthcare, food preparation and even loneliness.

Early, professionally produced videos showcased robotics in the home, executing complex tasks, like cooking and cleaning a broad range of surfaces. When TRI opened the doors of its South Bay labs to select press this week to show off a range of its different projects, the home element was notably lacking. Bajracharya showcased a pair of robots. The first was a modified off-the-shelf arm that moved boxes from a pile onto nearby conveyer belts, in a demo designed for unloading trucks — one of the more difficult tasks to automate in an industrial warehouse setting.

The second was a wheel robot that goes shopping. Unlike the warehouse example, which had standard parts with a modified gripper, this system was largely designed in-house out of necessity. The robot is sent out to retrieve different products on the shelf based on bar codes and general location. The system is able to extend to the top shelf to find items, before determining the best method for grasping the broad range of different objects and dropping them into its basket. The system is an outgrowth of the team’s pivot away from home-specific robots.

To the side of both robots is a mock kitchen, with a gantry system configured to the top of its walls. A quasi-humanoid robot hangs down, immobile and lifeless. It goes unacknowledged for the duration of the demos, but the system will look familiar to anyone who has watched the team’s early concept videos.

“The home is so hard,” says Bajracharya. “We pick challenge tasks because they are hard. The problem with the home is not that it was too hard. It was that it was too hard to measure the progress we were making. We tried a lot of things. We tried procedurally making a mess. We would put flour and rice on the tables and we would try to wipe them up. We would put things throughout the house to make the robot tidy. We were deploying into Airbnbs to see how well we were doing, but the problem is we couldn’t get the same home every time. But if we did, we would overfit to that home.”

Moving into the supermarket was an effort to address a more structured environment while still tackling a pressing issue for the elderly community. In testing the product, the team has moved from Airbnbs to a local mom-and-pop grocery store.

“To be totally honest, the challenge problem kind of doesn’t matter,” Bajracharya explains. “The DARPA Robotics Challenges, those were just made up tasks that were hard. That’s true of our challenge tasks, too. We like the home because it is representative of where we eventually want to be helping people in the home. But it doesn’t have to be the home. The grocery market is a very good representation because it has that huge diversity.”

In this instance, some of the learnings presented in this setting do translate to Toyota’s broader needs.

What, precisely, constitutes progress for a team of this nature is a difficult question to answer. It’s certainly one that’s top of mind, however, as large corporations have begun cutting roles in longtail research projects that have yet to deliver tangible, monetizable results. When I put the question to Gill Pratt yesterday, the TRI boss told me:

Toyota is a company that has tried very hard not to have employment follow business cycle. The car business is one that has booms and busts all the time. You may know that the history of Toyota is to try not to lay people off when times are tough, but instead go through a couple of things. One is shared sacrifice, where people take up the cause. The second is to use those times to invest in maintenance, plans and education to help people get trained.

Toyota is well-known in the industry for its “no layoffs” policy. It’s an admirable goal, certainly, especially as companies like Google and Amazon are in the midst of layoffs numbering in the tens of thousands. But when goals are more abstract, as is the case with TRI and fellow research wings, how does a company measure relevant milestones?

“We were making progress on the home but not as fast and not as clearly as when we move to the grocery store,” the executive explains. “When we move to the grocery store, it really becomes very evident how well you’re doing and what the real problems are in your system. And then you can really focus on solving those problems. When we toured both logistics and manufacturing facilities of Toyota, we saw all of these opportunities where they’re basically the grocery shopping challenge, except a little bit different. Now, instead of the parts being grocery items, the parts are all the parts in a distribution center.”

As is the nature of research projects, Bajracharya adds, sometimes the beneficial outcomes are unexpected: “The projects are still looking at how we ultimately amplify people in their homes. But over time, as we pick these challenge tasks, if things trickle out that are applicable to these other areas, that’s where we’re using these short-term milestones to show the progress in the research that we’re making.”

The path toward productizing such breakthroughs can also be fuzzy sometimes.

“I believe we kind of understand the landscape now,” Bajracharya. “Maybe I was naive in the beginning thinking that, okay, we just need to find this person that we’re going to throw the technology over to a third party or somebody inside of Toyota. But I think what we’ve learned is that, whatever it is — whether it’s a business unit, or a company, or like a startup or a unit inside of Toyota — they don’t seem to exist.”

Spinning out startups — akin to what Alphabet has done with its X labs — is certainly on the table, even though it isn’t likely to be the primary path toward productization. What form that path will ultimately take, however, remains unclear. Though robotics as a category is currently far more viable than it was when TRI was founded in 2017.

“Over the last five years, I feel like we’ve made enough progress in that very challenging problem that we are now starting to see it turn into these real-world applications,” says Bajracharya. “We have consciously shifted. We’re still 80% pushing the state of the art with research, but we’ve now allocated maybe 20% of our resources to figuring out if that research is maybe as good as we think it is and if it can be applied to real-world applications. We might fail. We might realize we thought we made some interesting breakthroughs, but it’s not anywhere near reliable or fast enough. But we’re putting 20% of our effort toward trying.”