The 'Supernatural' star also revealed Alba was there to comfort him following the death of a loved one.
The 43-year-old actor made the stunning revelation on Michael Rosenbaum's Inside of You podcast, which initially took the podcast host by surprise. At one point during the episode, Rosenbaum started running down Ackles' resume in Hollywood when he interrupted himself and asked, "By the way, was Jessica Alba cool to work with?"
"No," Ackles immediately responded. "She was horrible." The answer left Rosenbaum stunned in disbelief, prompting the host to ask Ackles if he was kidding. Ackles doubled down and again said "no" before telling Rosenbaum, "I've told this to her face."
Ackles then went on to explain what it was about their relationship that caused so much friction on the sci-fi show that debuted in 2000 and lasted two seasons.
"I love Jess, which I know kind of contradicts what I just said," Ackles explained. "She was under an immense amount of pressure on that show. She was young. She was in a relationship, and that was rocky and causing some undue stress, I believe, on set." Alba was 19 when the show premiered on Fox.
"I was the new fresh face on the set who wasn't really there in season 1," he continued. "I came in for one episode, but then they wrote me in as a series regular for season 2. And I was the new kid on the block and I got picked on by the lead."
When asked to explain how Alba allegedly picked on him, Ackles described it as "the worst kind of bickering a brother and sister" could have.
"She had it out for me," Ackles claimed. "It wasn't that she didn't like me. She was just like, 'Oh, here's the pretty boy that network brought in for some more window dressing 'cause that's what we need.'"
Ackles admitted he didn't take lightly to, in his words, being picked on, and he swung right back at Alba.
"Very quickly I was like, 'What the f**k? What did I do?'" he said. "So, very quickly I was like, 'Fire with fire.'"
After that, Ackles said they eventually "did build some mutual respect." Ackles pointed to two instances, for example, to show that it wasn't all sticks and stones. Ackles said Alba once invited him over to her house to keep her company because she apparently didn't feel comfortable being alone in a house she was renting at the time. There was also the time when Ackles said his grandfather died.
"My grandfather died while I was shooting and she literally just walked into my trailer and held me for a half an hour," he said. "So, it was that kind of a relationship. And, again, if she walked in, we'd be all hugs. But she didn't make it easy on me on set."
Alba opened up in a 2012 interview with Marie Claire and said she felt objectified as a young actress trying to make her stamp in Hollywood.
"I had a show [Fox's Dark Angel] that premiered when I was 19," she told the magazine. "And right away, everyone formed a strong opinion about me because of the way I was marketed. I was supposed to be sex, this tough action girl. That's what people expected."