Jack Stephens' equaliser in the first minute of added time looked to have earned the hosts a point, but Cardiff substitute Kenneth Zohore struck two minutes later to inflict Saints' first Premier League loss of 2019.
Victory took Cardiff above their opponents in the table, leaving Ralph Hasenhuttl's side in the bottom three.
Sol Bamba had given Cardiff the lead against the run of play and Hojbjerg told his club's website: "I'm speechless. I don't quite know the words to put in.
"The only thing I can say that makes sense right now is that we, as a group, as a team and as a club, need these days to get together and analyse our small mistakes and details.
"Then we're going to improve, do better, work harder and come back as a stronger side.
"They (the coaches) prepared everything for us, we just had to put it into play. We did all right and in some parts of the game we created what we had to.
"You have to give credit to Cardiff for keeping believing, because in this case you have to believe until the end.
"We're going to work on what we have to work on and come back stronger – that's what this club is about, that's what the players are about and that's what the coaches are about.
"We have a big potential, but we still have to put it into excellence. This is what we will work on."
Saturday's match was Cardiff's first since the confirmation that Emiliano Sala, the club's record signing, had been killed in a plane crash.
The striker's body was recovered from underwater wreckage in the English Channel this week following the crash last month.
Bamba told Cardiff's website: "There was a lot going through my mind after scoring. I knew Emiliano was watching from up there, so it was so good to get the three points.
"It has been a tough couple of weeks, and we knew that we needed to come here and get a result. It wasn't easy, especially with the events that have unfolded recently.
"The result is a massive boost, especially at this stage of the season. We want to build up momentum and go on a run of games. With the matches coming up, we know that we can get three points from every game, especially at home."