Toyota Research Institute SVP on the difficulty of building the perfect home robot

Earlier this week, the Toyota Research Institute opened the doors of its Bay Area offices to members of the media for the first time. It was a day full of demos, ranging from driving simulators and drifting instructors to conversations around machine learning and sustainability.

Robotics, a longtime focus of Toyota’s research division, were on display, as well. SVP Max Bajracharya showcased a pair of projects. First was something more along the lines of what one would expect from Toyota: an industrial arm with a modified gripper designed for the surprisingly complex task of moving boxes from the back of a truck to nearby conveyor belts — something most factories are hoping to automate in the future.

The other is a bit more surprising — at least for those who haven’t followed the division’s work that closely. A shopping robot retrieves different products on the shelf based on bar codes and general location. The system is able to extend to the top shelf to find items, before determining the best method for grasping the broad range of different objects and dropping them into its basket.

The system is a direct outgrowth of the 50-person robotics team’s focus on eldercare, aimed at addressing Japan’s aging population. It does, however, represent a pivot away from their original work of building robots designed to execute household tasks like dishwashing and food prep.

You can read a lengthier writeup of that pivot in an article published on TechCrunch earlier this week. That was drawn from a conversation with Bajracharya, which we’re printing in a more complete state below. Note that the text has been edited for clarity and length.

TechCrunch: I was hoping to get a demo of the home robot.

Max Bajracharya: We are still doing some home robot stuff[…] What we’ve done has shifted. Home was one of our original challenge tasks.

Eldercare was the first pillar.

Absolutely. One of the things that we learned in that process is that we weren’t able to measure our progress very well. The home is so hard. We pick challenge tasks because they are hard. The problem with the home is not that it was too hard. It was that it was too hard to measure the progress we were making. We tried a lot of things. We tried procedurally making a mess. We would put flour and rice on the tables and we would try to wipe them up. We would put things throughout the house to make the robot tidy. We were deploying into Airbnbs to see how well we were doing, but the problem is we couldn’t get the same home every time. But if we did, we would overfit to that home.

Isn’t that ideal that you don’t get the same home every time?

Exactly, but the problem is we couldn’t measure how well we were doing. Let’s say we were a little better at tidying this one house, we don’t know if that’s because our capabilities got better or if that house was a little easier. We were doing the standard, “show a demo, show a cool video. We’re not good enough yet, here’s a cool video.” We didn’t know whether we were making good progress or not. The grocery challenge task where we said, we need an environment where it’s as hard as a home or has the same representative problems as a home, but where we can measure how much progress we’re making.

You’re not talking about specific goals to either the home or supermarket, but solving for problems that can span both of those places.

Or even just measure if we’re pushing the state of the art in robotics. Are we able to do the perception, the motion planning, the behaviors that are, in fact, general purpose. To be totally honest, the challenge problem kind of doesn’t matter. The DARPA Robotics Challenges, those were just made-up tasks that were hard. That’s true of our challenge tasks, too. We like the home because it is representative of where we eventually want to be helping people in the home. But it doesn’t have to be the home. The grocery market is a very good representation because it has that huge diversity.

There’s a frustration, though. We know how difficult these challenges are and how far off things are, but some random person sees your video, and suddenly it’s something that’s just over the horizon, even though you can’t deliver that.

Absolutely. That’s why Gill [Pratt] says every time, ‘reemphasize why this is a challenge task.’

How do you translate that to normal people? Normal people aren’t hung up on challenge tasks.

Exactly, but that’s why in the demonstration you saw today, we tried to show the challenge tasks, but also one example of how you take capabilities that come out of that challenge and apply it to a real application like unloading a container. That is a real problem. We went to factories and they said, ‘yes, this is a problem. Can you help us?’ And we said, yeah, we have technologies that apply to that. So now we’re trying to show coming out of these challenges are these couple of few breakthroughs that we think are important, and then apply those to real applications. And I think that that’s been helping people understand that, because they see that second step.

How large is the robotics team?

The division is about 50 people evenly split between here and Cambridge, Massachusetts.

You have examples like Tesla and Figure, which are trying to make all-purpose humanoid robots. You seem to be heading in a different direction.

A little bit. Something we’ve observed is that the world is built for humans. If you’ve just got a blank slate, you’re saying I want to build a robot to work in human spaces. You tend to end in human proportions and human-level capabilities. You end with human legs and arms, not because that’s the optimal solution, necessarily. It’s because the world has been designed around people.

How do you measure milestones? What does success look like for your team?

Moving from the home to the grocery store is a great example of that. We were making progress on the home but not as fast and not as clearly as when we move to the grocery store. When we move to the grocery store, it really becomes very evident how well you’re doing and what the real problems are in your system. And then you can really focus on solving those problems. When we toured both logistics and manufacturing facilities of Toyota, we saw all of these opportunities where they’re basically the grocery shopping challenge, except a little bit different. Now, the part instead of the parts being grocery items, the parts are all the parts in a distribution center.

You hear from 1,000 people that you know, home robots are really hard, but then you feel like you have to try for yourself and then you like, really, you make all the same mistakes that they did.

I think I’m probably just as guilty as everybody else. It’s like, now our GPUs are better. Oh, we got machine learning and now you know we can do this. Oh, okay, maybe that was harder than we thought.

Something has to tip it at some point.

Maybe. I think it’s going to take a long time. Just like automated driving, I don’t think there’s a silver bullet. There’s not just like this magical thing, that’s going to be ‘okay, now we solved it.’ It’s going to be chipping away, chipping away, incrementally. That’s why it’s important to have that kind of roadmap with the shorter timelines, you know, shorter or shorter milestones that give you the little wins, so you can keep working at it to really achieve that long-term vision.

What’s the process for actually productizing any of these technologies?

That’s a very good question that we are ourselves trying to answer. I believe we kind of understand the landscape now. Maybe I was naïve in the beginning thinking that, okay, we just need to find this this person that we’re going to throw the technology over to a third party or somebody inside of Toyota. But I think we’ve learned that, whatever it is — whether it’s a business unit, or a company, or like a startup or a unit inside of Toyota — they don’t seem to exist. So, we are trying to find a way of creating and I think that’s the story of TRI-AD, a little bit as well. It was created to take the automated driving research that we were doing and translate into something that was more real. We have the same problem in robotics, and in many of the advanced technologies that we that we work on.

You’re thinking about potentially getting to a place where you can have spinoffs.

Potentially. But it’s not the main mechanism by which we would commercialize the technology.

What is the main mechanism?

We don’t know. The answer is the diversity of things that we’re doing is very likely going to be different for different groups.

How has TRI changed since its foundation?

When I first started, I feel like we were very clearly just doing research in robotics. Part of that is because we were just so very far away from the technology being applicable to almost any real-world challenging application in a human environment. Over the last five years, I feel like we’ve made enough progress in that very challenging problem that we are now starting to see it turn into these real-world applications. We have consciously shifted. We’re still 80% pushing the state of the art with research, but we’ve now allocated maybe 20% of our resources to figuring out if that research is maybe as good as we think it is and if it can be applied to real-world applications. We might fail. We might realize we thought we made some interesting breakthroughs, but it’s not anywhere near reliable or fast enough. But we’re putting 20% of our effort toward trying.

How does eldercare fit into this?

I would say, in some ways, it’s still our north star. The projects are still looking at how we ultimately amplify people in their homes. But over time, as we pick these challenge tasks, if things trickle out that are applicable to these other areas, that’s where we’re using these short-term milestones to show the progress in the research that we’re making.

How realistic is the possibility of a fully lights-out factor?

I think if you were able to start from scratch in maybe in the future, that might be a possibility. If I look at manufacturing today, specifically for Toyota, it seems very unlikely that you could get anywhere close to that. We [told factory workers], we’re building robotic technology, where do you think it could apply? They showed us many, many processes where it was things like, you take this wire harness, you feed it through here, then you pull it out here, then you clip it here, and you clip it here, and you take it here, and you take it here, and then you run it like this. And this takes a person five days to learn the skill. We were like, ‘yeah, that’s way too hard for the robotic technology.’

But the things that are the most difficult for people are the ones you would want to automate.

Yes, difficult or potentially injury prone. For sure, we would like to make stepping stones to get to that eventually, but where I see robotic technology today, we’re quite far away from that.