Covariant is building ChatGPT for robots

Covariant this week announced the launch of RFM-1 (Robotics Foundation Model 1). Peter Chen, the co-founder and CEO of the UC Berkeley artificial intelligence spinout tells TechCrunch the platform, “is basically a large language model (LLM), but for robot language.”

RFM-1 is the result of, among other things, a massive trove of data collected from the deployment of Covariant’s Brain AI platform. With customer consent, the startup has been building the robot equivalent of an LLM database.

“The vision of RFM-1 is to power the billions of robots to come,” Chen says. “We at Covariant have already deployed lots of robots at warehouses with success. But that is not the limit of where we want to get to. We really want to power robots in manufacturing, food processing, recycling, agriculture, the service industry and even into people’s homes.”

The platform launches as more robotics firms are discussing the future of “general purpose” systems. The sudden onslaught of humanoid robotics firms like Agility, Figure, 1X and Apptronik has played a pivotal role in that conversation. The form factor is particularly suited to adaptability (much like the humans on which it’s modeled), though the robustness of on-board AI/software systems is another question entirely.

For now, Covariant’s software is largely deployed on industrial robotic arms doing a variety of familiar warehouse tasks, including jobs like bin picking. It isn’t currently deployed on humanoids, though the company is promising some level of hardware agnosticism.

“We do like a lot of the work that is happening in the more general purpose robot hardware space,” says Chen. “Coupling the intelligence inflection point with the hardware inflection point is where we will see even more explosion of robot applications. But a lot of those are not fully there yet, especially on the hardware side. It’s very hard to go beyond the staged video. How many people have interacted with a humanoid in person? That tells you the degree of maturity.”

Covariant doesn’t, however, shy away from human comparisons when it comes to the role RFM-1 plays in robots’ decision-making processes. Per its press material, the platform, “provides robots the human-like ability to reason, representing the first time Generative AI has successfully given commercial robots a deeper understanding of language and the physical world.”

This is one of those realms where we have to be careful with claims, both in terms of comparisons to abstract — or even philosophical — concepts and their actual real-world efficacy over time. “Human-like ability to reason” is a broad-sweeping concept that means a lot of different things to a lot of different people. Here the notion applies to the system’s ability to process real-world data and determine the best course of action to execute the task at hand.

This is a departure from traditional robotic systems that are a programmed to one job repeatedly, ad infinitum. Such single-purpose robots have thrived in highly structured environments, starting with automotive assembly lines. As long as there are minimal changes to the task at hand, a robot arm can do its work over and over again, unimpeded, until it’s time to call it a day and collect the golden pocket watch for its years of loyal service.

Things can break down quickly, however, with even the smallest deviations. Say the object isn’t placed exactly right on the conveyor belt, or there’s been an adjustment to lighting that impacts on-board cameras. These sorts of differences can have a huge impact on the robot’s ability to execute. Now imagine trying to get that robot to work with a new part, new material or even do a wholly different task. That’s even harder.

This is the point where programmers traditionally step in. The robot must be reprogrammed. More often than not, someone from outside the factory floor enters the picture. This is a big drain of resources and time. If you want to avoid this, one of two things needs to happen: 1) People working on the floor need to learn code or 2) You need a new, more natural method for interacting with the robot.

While it would be great to do the former, it seems unlikely that companies will be willing to invest the money and wait the necessary time. The latter is precisely what Covariant is attempting to do with RFM-1. “ChatGPT for robots” isn’t a perfect analogy, but it’s a reasonable shorthand (especially in light of the founders’ connection to OpenAI).

From the customer’s point of view, the platform presents as a text field, much like the current iteration of consumer-facing generative AI. Input a text command like, “pick up the apple” by typing or voice, and the system uses its training data (shape, color, size, etc.) to identify the object in front of it that most closely matches that description.

RFM-1 then generates video outcomes — in essence simulations — to determine the best course of action using past training. This last bit is similar to how our brains work out the potential outcomes of an action prior to executing.

During a live demo, the system reacts to inputs like “pick up the red object” and even the more semantically complex, “pick up what you put on your feet before you put on your shoes,” which caused the robot to correctly pick up the apple and a pair of socks, respectively.

A lot of big ideas are tossed around when discussing the system’s promise. At the very least, Covariant has an impressive pedigree among its founders. Chen studied AI at Berkeley under Pieter Abbeel, his Covariant co-founder and chief scientist. Abbeel also became an early OpenAI employee in 2016, a month after Chen joined the ChatGPT firm. Covariant was founded the following year.

Chen says the company expects the new RFM-1 platform will work with a “majority” of the hardware on which Covariant software is already deployed.