The Sawtooth Building, and later the School, were named for its distinctive jagged roof. The building started as a textile mill. The "teeth" hold 6 foot tall windows that provided light to the workers when the mill opened in 1911--before electric lights were widely available. The mill was built in 1911 for Shamrock Hosiery Co. and was started by John Wesley Hanes. The building had room for 200 knitting machines and Shamrock Hosiery employed an equal number of workers to make socks for men and boys. In 1914, Shamrock Hosiery was renamed Hanes Hosiery Mills Co. Hanes Hosiery later merged with Hanes Knitting to become Hanes Corp.

The mill began to specialize in women's hosiery by the late 1920's, outgrowing its space, so operations were moved to another location, between Northwest Boulevard and 14th Street. The Sawtooth Building later housed such businesses as car dealerships. In 1978, it was added to the National Register of Historic Places, and a few years later, after the building underwent its first major renovation, became home to the Arts Council of Winston-Salem and Forsyth County and the Sawtooth Center for Visual Art.

In Summer 2010, after a two-year renovation and reconstruction of the Sawtooth complex, the School resumed operating at its Marshall Street home, having held classes at satellite locations during those two years. The school is now part of the Milton Rhodes Center for the Arts, which opened in September 2010. The center's main entrance is now 251 N. Spruce Street.The school, now called Sawtooth School for Visual Art, offers instruction in ceramics, woodworking, drawing, painting, fibers, computer graphics, photography, glass and metals. The renovations to Sawtooth included a new floor plan, new equipment, more and better-equipped classrooms, and freshly painted studios with more natural light, thus using the "teeth" to the school's advantage.