This is a somewhat freewheeling collection of my thoughts about the current, somewhat overheated excitement around “AI-powered robots” and “Embodied AI”, together with a brief explanation of the technology and a broad overview of the current robotics field. I’m a PhD student working on robotics, but the job I’m starting after this has little to do with robots. So I know the field pretty well from the inside, while being something of an outsider as far as my future career is concerned. So please take this as someone rambling from the sidelines. For all I know, writing this whole post might just be my way of justifying my decision to leave robotics.
(This is an AI-translated, human-tweaked English version of my original post in Japanese, which I wrote entirely on my own with no AI producing any part of the text)
There is a scene from Charlie and the Chocolate Factory that has stuck with me. Charlie’s dad works at a toothpaste factory, supporting his family by doing nothing but screwing caps onto toothpaste tubes all day. Then one day, the factory installs a robot that does this automatically, and he loses his job. (Apparently, the popularity of Wonka’s chocolate led to more cavities, which led to more toothpaste sales, which gave the factory enough money to invest in a robot.) These days we keep hearing that robots are about to take over all kinds of manual work. Are more people going to end up like Charlie’s dad?
At some point, robotics picked up a shiny new label, Embodied AI, and the industry’s scale started growing at an astonishing pace. I began my robotics PhD at ETH Zurich in Switzerland in 2022, the year ChatGPT came out. Back then, the only really well-known humanoid was Boston Dynamics’ Atlas, and if you wanted a human-like robot hand, there was the Shadow Hand, which cost about as much as a Porsche (and, from what I heard at the time, was quite a pain to maintain even after you bought it). Almost nobody around had ever seen either of them actually moving live, in person.
Fast forward four years, and a new robotics startup seems to appear every week, with new humanoids and robot hands coming out one after another. As someone who likes robots, this is exciting. As a PhD student, it also means that competition in my field has become much more intense. I can’t help wondering whether somebody somewhere is already doing the same thing as me, and whether there is any point in me working on it at all.
How did we get here? I think it comes down to the progress and excitement from AI based on large language models (LLMs), like ChatGPT and Claude. ChatGPT showed the world what LLMs could do in 2022, and the AI race was kicked off. The idea driving this race is that whichever company or country first achieves artificial general intelligence (AGI), and eventually artificial superintelligence (ASI), will get to dominate everything technologically, socially and politically. The investment is enormous: reportedly, an amount equivalent to nearly 2% of US GDP is being poured into AI development alone. For now, these companies aren’t even trying to turn a profit. They are competing for that future dominance, pouring the money they raise into more GPUs to make their models better and better.
Once this race to build AI that can do anything a human can do on a computer was underway, investors started looking for the next big thing. For many, that turned out to be Embodied AI: AI that can do any physical task a human can do. So, in a very rough way, the current robotics boom is fueled by trickle down cash from the ChatGPT boom.
Looking at the industry as a whole, China and the US are clearly the biggest players. China, with companies like Unitree, is particularly strong at robot hardware. Drawing on its manufacturing expertise and experience, it has dramatically shortened the path from development to an actual product. You could almost say that the only complete robots you can readily buy are Chinese. Many of you have probably seen a Chinese humanoid or quadruped at a public demonstration. What’s impressive is that these companies have committed to manufacturing at this scale when the robots are still mostly being bought as research platforms, without much real demand beyond that. South Korea is also doing well. Robotis’ Dynamixel servo motors are used as components in all kinds of robots, thanks to their reliability and ease of use.
The US is more focused on the software side: how to make robots smart using AI, and develop algorithms that can replace human work, rather than mass-producing and selling the hardware itself. Some representative companies are Physical Intelligence and Generalist AI for non-humanoid robots (relatively simple setups with two arms and grippers), and Figure and 1X for humanoids. (1X recently moved from Norway to the US.) These days, all of them seem to release impressive demo videos, again almost every week, showing robots doing dexterous tasks that would have been hard to imagine before.
Recently, conversations in our lab have started to sound like people following sports teams: who moved to which company, who raised another enormous round of funding, who just released a new robot demo. It reminds me of the Web 2.0 boom of the late 2000s and early 2010s, with Facebook, Twitter and Google competing with each other. Watching all this, I start to wonder whether the American dream shown in The Social Network, where an ordinary university student named Mark Zuckerberg builds Facebook and watches it take off, is still within reach today.
But there is a fundamental difference between building those online services and building robots that can replace human work. With the former, if you put enough programmers on it, you can technically build pretty much any kind of web service. The hard part is getting users, whether individuals or businesses, to actually stick around: product-market fit (PMF). AI robotics companies will eventually need to find PMF too. But there is a more fundamental problem: nobody yet knows whether the product they are trying to build is even possible.
If we collect huge amounts of data and scale things up, as we did with LLMs, will robots reach human-level dexterity?
There are certainly signs that this might work. Take this Generalist AI demo of a robot assembling a robot vacuum cleaner. It carefully adjusts the force as it fits parts together, and tries again when something goes a little wrong. This level of complex manipulation would have been unthinkable just a few years ago.
The basic approach today is to collect data, for example by having humans remotely operate a robot, then train the robot to imitate those movements with its own hands and arms. That way, it learns to perform the same task. Robot data collection is now happening at an enormous scale. People operate robots, or hold devices shaped like robot grippers, and repeat the same movements over and over to produce training data. No special knowledge or skills required. Just patience.
Some services will even pay you just to record yourself doing your job, betting that ordinary videos of humans at work will eventually be useful for teaching robots. This is still a research problem, and we don’t know whether it will work. Personally, I think video will only ever be useful as supplementary data, because it necessarily contains less information than data collected by operating the robot itself or a device like a UMI (Universal Manipulation Interface). There are even services offering to clean your home for free, in exchange for recording everything as training data.
If we keep pushing this approach of collecting data and using it to train robots, I do think we will eventually get to almost complete automation, provided that:
- The environment is sufficiently constrained.
- The task is specific, and physically achievable by the robot.
- Enough demonstration data has been collected.
But is that really as revolutionary as so many robotics startups make it out to be? Sure, a robot that repeatedly performs a particular assembly task in a controlled environment like a factory could be useful. But in many cases, building a dedicated machine using conventional automation would be far more efficient, even if the initial investment is higher, than using a complicated robot. Over years of operation, a robot with lots of moving parts also has more things that can break, and higher maintenance costs.
To give a somewhat extreme example: a dedicated paperclip-making machine would be hundreds of times more productive than a robot that carefully cuts a piece of wire, dexterously bends it in an impressive feat of autonomy, and makes each paperclip one by one.
A few months ago, Figure AI made headlines by livestreaming a humanoid doing “real work” for nine days straight. The task was to pick up irregularly shaped packages from one conveyor belt and transfer them to the next, turning them so the barcode faced upwards to be scannable. Apparently, this is a job that people actually do in factories. On day five, the robot even “competed” against a human and processed more packages.
My honest reaction was: why not just add more scanners? If each package passed through an area with a transparent bottom and scanners on all sides, there would be no need to turn the barcode upwards in the first place. If the packages also need to be spaced out as they move to the next conveyor, existing technology like a delta robot could handle that. Admittedly, you’d probably need a more dexterous gripper to deal with those irregularly shaped packages.
Another difference between AI in the digital world, like ChatGPT, and Embodied AI is the work they target. The former mainly targets white-collar work, where people work in offices rather than use their bodies, while Embodied AI targets blue-collar work, involving manual skills and physical labor. In many cases, white-collar jobs pay more, so there is more money to be saved by automating them. Embodied AI has a harder time making economic sense from the outset. And while ChatGPT can, for some tasks, produce results faster and more accurately than a human, Embodied AI generally works at about human speed or a little slower.
One obstacle to deploying robots, often brought up half-jokingly, is that “humans are just too cheap compared to robots”. But this deserves serious consideration. In the USA, for a little over $10 an hour, you could hire someone capable of more dexterous manipulation than any robot. And in India, the minimum wage is 179 rupees, or about $2, per day. I don’t think a humanoid with lower running costs than that is going to be possible anytime soon.
With a robot, you have the upfront hardware cost, then the maintenance costs of all those moving parts. And even a small change in the working environment might mean adjusting the controller or collecting the training data all over again.
Maybe, if we keep scaling up the current approach, we really will get robots that can autonomously do anything a human can do. That’s what all these companies are betting on. Personally, I’m skeptical. The real world has far too much uncertainty compared to the digital world, where clicking a button reliably executes an action and you can get all the debugging information you need.
It’s easy to watch a video from a leading robotics startup and think, “Wow, it can do anything!” But it’s worth taking a step back and looking carefully - as if you’re looking at a magic trick. For example, here is a Genesis AI robot making stir-fried eggs and tomatoes:
To be absolutely clear: this demo is really impressive. It is one of the best demonstrations of manipulation with five-fingered robot hands, and I have a lot of respect for what they have achieved. Showing the whole thing without cuts is also great. With that said, if I put on my nitpicking hat:
- 0:00: There are already several cracked and beaten eggs in the bowl at the start of the video. Maybe cracking several eggs reliably is still difficult.
- 0:08: A piece of eggshell gets stuck to the left thumb (or maybe it punctured the glove). The robot wipes its hand on a towel, but the shell doesn’t come off. It stays there, and the robot never tries to remove it again.
- 0:25: The top of the pepper mill looks quite loose. The pepper coming out is very fine, and there isn’t much of it. Grinding pepper properly takes quite a bit of force, so maybe they loosened the mill to make it easier to turn, and filled it with pepper that was already almost powder.
- 1:30: That’s a huge slice of tomato it throws away, just to remove the stem!
They also don’t disclose how many times someone had to operate the robot and cook eggs to collect the data for this demo. Their blog post does say that most tasks required less than an hour of robot data, but doesn’t give the actual figure for this one. And there are countless tasks necessary for cooking in a restaurant or at home that the demo doesn’t include. Getting each ingredient from the pantry. Carefully fishing out a piece of eggshell that fell into the bowl. Taking dishes and utensils out of a cupboard. Taking the pepper mill apart and refilling it when it runs out. Cutting away a spoiled patch on a tomato… The list goes on.
Unlike the world inside a computer, the real world is full of these prerequisite tasks and unexpected situations that need special handling which explode exponentially. I find it hard to imagine collecting enough data to cover all of them.
So, what should robotics aim for?
Robots are certainly getting better at doing things autonomously, even if they may never catch up with humans. A robot that repeatedly does a specific task, in a constrained environment, that was previously difficult to automate could be useful. About ten years ago, a Japanese company was developing Laundroid, a robot specifically for folding laundry. There were some questions about how close it actually was to being ready for sale: public demos only showed it folding T-shirts and towels, for example. But it does seem to have genuinely been able to fold certain types of clothing. (The company was quite secretive, but some of the internal mechanism and algorithm have been revealed.)
Unfortunately, the company shut down in 2019 without ever releasing the product. There may have been all kinds of technical and business issues behind the scenes, but the explanation given publicly was that it had trouble grasping thin fabrics like UNIQLO’s AIRism.
With today’s technology, the “robot that folds all your laundry” that Laundroid was aiming for seems actually quite achievable. Take your clothes out of the dryer, dump them in, and they come out folded. Even if it folds much more slowly than a human, I’m sure plenty of people would want one. It might not have the economic impact of something that can replace every task a human does, but it could become a market comparable to washing machines and dryers. (Whether robotics investors would be interested in a market that is destined to never get any bigger than the washing machine market, is another question.)
I think there is real potential in products like these, using robotic technology inside a dedicated appliance for a specific purpose, if you choose the market carefully.
Another possibility is to add some entertainment value, so that the fact that a robot is making something becomes part of what people pay for. With the soft-serve ice cream robot I worked on a while back, we even did a field study showing that moving its eyes increased customers’ reactions and interactions with it.
At that stage, a person still had to hand the robot a cone, and it occasionally failed, so a human staff member always had to be there. So to be honest, we weren’t actually saving any labor. Instead, we gave it a cute, bird-like exterior and had the robot call out to customers, adding value by getting people interested. And perhaps, if a robot is cute enough, people will be more forgiving when it messes up.
What will the world look like when I reread this post ten years from now? Maybe robots will be doing useful things within fairly limited settings, as I’ve suggested here. Or maybe I’ll have been hilariously wrong, and robots will be doing absolutely everything.
Anyway, there is more to the story of Charlie’s dad in Charlie and the Chocolate Factory. Towards the end, as everyone’s happy endings are being wrapped up, he gets his job back at the toothpaste factory, this time repairing the robot that screws on the caps. Maybe he even gets a higher hourly wage. The punchline: even the robot needed a human after all.
How to model tendon-driven robots: Calculating kinematics and forces through the tendon Jacobian
Cut off the Internet, and LLMs are still a hallucinating mess.
How to Set Up Isaac Sim on Non-Officially Supported AWS Instances with Remote Desktop Access