Carroll was the head basketball coach at Bloomfield College in Bloomfield, New Jersey from 1979 to 1982. The Deacons were 42–38 under Carroll and advanced to the NAIA District Finals twice in three seasons. He went on to become an assistant coach at Seton Hall University under P. J. Carlesimo for seven seasons before becoming the head basketball coach for Duquesne University from 1989 to 1995. He compiled a 73–98 career record with one winning season, in 1993–94, when the Dukes went 17–13 overall and 8–8 in the Atlantic 10 Conference. They earned a trip to the National Invitation Tournament (NIT), where they beat UNC Charlotte in the first round but lost to Villanova in second round. Carroll was the Atlantic 10 Coach of the Year for the 1990–91 season after leading the Dukes to a 10–8 conference record.
Carroll now coaches for the New England Playaz AAU program.