Zinedine Zidane's side dominated the game but succumbed to an 87th minute strike from Pablo Fornals, who chipped over goalkeeper Keylor Navas on the rebound after a lightning quick counterattack, giving Villarreal their first ever win at the Bernabeu in 19 visits in all competitions.
Real remain fourth in the Liga standings on 32 points and are only one point and one place above Villarreal. They could be a gaping 19 points behind runaway leaders Barcelona if Ernesto Valverde's side beat Real Sociedad on Sunday.
"We don't deserve this, we think we played well but the ball just didn't want to go in for us and I cannot explain why," Zidane told reporters.
"We then couldn't prevent them counter attacking us and it's really tough on the players."
Real's slim chances of getting back into the title race after their 3-0 defeat by Barcelona before Christmas had faded further with last week's 2-2 draw away to Celta Vigo and, coupled with the persistent rain, there was little enthusiasm for the game at the Bernabeu, which was three-quarters full.
The home side began brightly as Gareth Bale had a goal correctly ruled offside and Ronaldo hit the woodwork twice. Villarreal goalkeeper Sergio Asenjo tipped away a stinging distance strike from outside the area from Marcelo with little more than half an hour on the clock.
Madrid also had a penalty appeal waved away when visiting defender Alvaro Gonzalez appeared to block a shot from Bale with his arm, while Asenjo made his best save of an outstanding overall display by thwarting Ronaldo from close range on the stroke of halftime.
Villarreal slowly grew into the game as the second half progressed and capitalised on Real's desperation to try and win the game by outnumbering Zidane's side on the break after clearing Toni Kroos's corner.
"We knew that we could make history today and we always believed it," said Villarreal coach Javi Calleja.
"Madrid have many virtues but we thought that at this moment of the season and playing at home they might make more mistakes. We knew we had to keep the ball for long moments and couldn't do it in the first half but we managed it in the second, we got players in between the lines and caused them damage."
Second-placed Atletico Madrid were playing away to Eibar later on Saturday while third-placed Valencia visit Deportivo La Coruna.
At the other end of the table basement club Las Palmas were hammered 6-0 by Liga debutants Girona, while 19th-placed Malaga sacked hapless coach Michel a day after Friday's 1-0 defeat at Getafe, a fourth consecutive loss in the league.