Comedian Katt Williams on Saturday partly walked back an apology for making insensitive remarks to a Mexican member of his audience, telling CNN that he is not sorry for what he said, but apologizes to anyone who thought he was being hateful. "I meant what I said and I said what I meant," Williams said, referring to his back-and-forth with a heckler during a performance a week ago. A written statement apologizing for the remarks had been released by his publicist on his behalf this week, but the comedian said that it didn't come from him. Video of the exchange, delivered during a performance at the Celebrity Theater in Phoenix, was posted online. Williams singled out a man seated near the stage and asked him if he was Mexican. "Do you remember when white people used to say 'Go back to Africa,' and we'd have to tell them we don't want to? So if you love Mexico, (expletive), get the (expletive) over there!" Williams said to the man. It turned into a back-and-forth profanity-laced argument with the man, Williams exclaimed, "We were slaves, (expletive). You just work like that as landscapers." On Saturday, Williams said the man started heckling him first with an expletive aimed at the United States. "If a person starts their heckling with 'f' America, then that gives me the right to defend my country," Williams said. "I don't think I need to apologize for being pro-American." Williams, an African-American comedian known more for his erratic behavior than his jokes in recent months, said that he can't be anti-Mexican because many of them are his fans. "The guy said that all of (Arizona) is still Mexico, and I was giving him geography," Williams said. The conversation was with the one man and not with the rest of the crowd, he said. "If I had disrespected Mexicans, I wouldn't have been able to get out of there alive," he said. One Arizona activist, Rev. Jarrett Maupin, had called Williams' words "borderline hate speech and racially offensive." Maupin had accepted the written apology and called on the comedian to return to Arizona to show his commitment to the Latino community, much like comedian Tracy Morgan's return to Nashville, Tennessee, this summer to personally apologize for anti-gay remarks in a stand-up show there. But Williams told CNN that as a comic, he cannot apologize for his uncensored thoughts during a show. "That's for the Tracy Morgans of the world," he said.