After several years coaching high school basketball in New York and New Jersey and reviving a master's degree from the City College of New York, Frazier launched his college career as an assistant for Delaware State.[1]
He then joined the Illinois staff as an assistant in 1967, at a time when few African-American coaches could be found on Big Ten coaching staffs.[2]
In 1971, Frazier was named head coach at Morgan State.[1] One of his first recruits was seven-footer Marvin Webster, nicknamed “the Human Eraser” due to his shot-blocking prowess. In Webster's junior season of 1973–74, the Bears advanced to the NCAA Division II Final, where they defeated Southwest Missouri State 67–52 to win the school's first national championship. He was named the college division national coach of the year by the Associated Press.[3]
He then went on to serve as president and head coach of the Washington Metros of the short-lived Women's Professional Basketball League (WBL). Frazier returned to college coaching as head coach at Bowie State from 1980 to 1982. In 1984, he was hired as the head coach of Virginia Wave of the newly founded Women's American Basketball Association.[5] In 1985, he returned as head coach of Morgan State (now a Division I program) and served in that role for four seasons.