Certain types of driverless vehicles may not be safe، peers have warned.
Over-reliance on technology could mean drivers react slowly to taking back control of a semi-autonomous vehicle in an emergency، they said.
However، the Lords Science and Technology Committee noted that some technology could reduce accidents caused by human error.
The Department for Transport said driverless cars "have the potential to transform the way we travel."
Intel makes $15bn bet on driverless cars
Vehicles can be split into different levels of automation، according to industry body the Society of Motor Manufacturers and Traders (SMMT).
Level 0 is a vehicle with no automation، while level 5 is fully automated.
There is a "very dangerous" problem that lies with vehicles on the midway point of this scale، peers on the Lords Science and Technology Committee said.
Vehicles at level 3 still need a driver، but can shift critical operations to the car، and back again. The risk that the vehicle may need to hand back control to an unprepared driver in an emergency may be "too great to tolerate"، the Lords Science and Technology Committee said.
Professor Neville Stanton of the University of Southampton told the committee: "As vehicles become fully autonomous، even the most observant human driver's attention will begin to wane.
"Their mind will wander… This is particularly true if they are engaging in other activities such as reading، answering emails، engaged in conversations with passengers، watching movies or surfing the internet."
In general، research suggested drivers of automated vehicles were not as effective at dealing with emergencies as drivers of manual cars.
For example، they had been found to take six times longer to respond to emergency braking by other vehicles.