The Australian, who won the French Open earlier this year, was beaten in straight sets by Wang Qiang, the 18th seed from China.
Below-par Barty was broken twice in the first set and saved two match points on serve in the second.
She had a chance to break back but the hugely impressive Wang, playing in the fourth round of a grand slam for the first time, held her nerve.
A fourth match point saw the 27-year-old over the line and into the quarter-finals.
“‘Q’ was very good, very solid,” said Barty. “I felt like she was able to put the ball with great depth in difficult positions for me.
“I still was able to create opportunities, it was just very frustrating that on the big points today, she played a lot better.
“I mean I had nine break points, and I wasn’t able to even get one of them, which is really frustrating.
“But it’s been a great season in grand slams, making the second week in every single one, which has been really special.
“Now we’ll sit back, reflect, and look forward to a big couple months to finish off the year.”
Wang will face 23-time grand slam winner Serena Williams in the quarter-finals.
The American recovered from losing her opening service game to beat Petra Martic of Croatia 6-3 6-4.
Williams suffered an injury scare when she hurt her ankle and required a medical time-out.
She said: “I just rolled it, I don’t know why. I was volleying and it went over so I just wanted to get it compressed really fast.
“It affected me a little mentally because I’ve had a tough year with injuries.”
Elina Svitolina, the fifth seed from Ukraine, dashed home hopes when she beat American 10th seed and former finalist Madison Keys 7-5 6-4.
Svitolina said: “It was really good serving from me, I was feeling good with my footwork as well. It was really special playing in this atmosphere.
“I’m playing quite solid. I’m just trying to take one match at a time and stay relaxed. Hopefully we can go far in the draw.”
Svitolina will face Britain’s Johanna Konta, who beat third seed Karolina Pliskova, in the last eight.
The Ukrainian said: “It’s going to be another great battle, we always have tough matches. It’s going to be another challenge which I will accept.