A lot of it is general complexity. A lot of situations require not only complex behaviour that's quite tricky to program into a robot, but the ability to deal with situations outside of their control. Like on a production line, that's not an issue. There's no weather, there are no humans or other animals in the nearby environment, the parts all get transported at the same pace and in the same state, etc.
The robot can basically run a macro that goes from the same few motions over and over again.
Now think about a self-driving car. They're a lot closer to being practical now, but they still have a lot of challenges to deal with that the robot arms in factories don't.
They have to worry about legal issues if they make a mistake. There are other vehicles on the road, including both human driven ones and ones by other companies. There might be pedestrians nearby, or wildlife. Weather has an effect on how they drive, since even a machine can't react as well to flooded or icy roads as normal ones. There may be natural disasters, or ones caused by human/robot mistakes. The programming for this sort of system can't just follow a simple routine, it has to accounts for dozens or even hundreds of possibilities.
Same situation in many other places robots could be used in. Shops and restaurants, construction sites, on the street doing law enforcement and traffic work, etc. Unless you have an incredibly well-designed, incredibly flexible machine that can deal with unforeseen circumstances, it may be cheaper, easier and less legally risky to have a human do the work.
The easiest solution to this is to retool the layout of the space the robot will occupy to fit the needs of said robot. So, a shop or restaurant would be designed in a way that the automated setup would be closer to an oven or dishwasher and the process wouldn't interact with humans beyond the ordering and receiving aspects.
But that's expensive and time consuming to do, and in a lot of cases may hurt the business needs of the company. I don't think a lot of people want to get their food from the equivalent of a vending machine at a high-end restaurant, or have it delivered by a glorified roomba.
Unless humans become extremely expensive to hire and robotics becomes extremely cheap, that probably won't be worth the hassle to change.