In 1965, Scherr joined IBM. He worked on the architecture for the IBM System/370 computers and did work that was instrumental in establishing the IBM System/360 Operating System. In 1971, he became a part of the IBM task force that proposed the creation of MVS, and managed the project until its release in 1974. In 1977, he took over the management of the development of the IBM DPPX for the IBM 8100 minicomputer. In 1991, he became vice president of technology of the IBM Consulting Group.[1] He was named an IBM Fellow in recognition of his contributions to IBM.[3]
After leaving IBM, he retired from the EMC Corporation in 2001, where he was senior vice president of software engineering.[3] In 2015 he ran for Town Board of Rhinebeck, New York.[4]