Chronic low back pain is one of the most common reasons that people seek medical treatment, and the consequent disability creates a great financial burden on individuals and society. The etiology of chronic low back pain is not clear, which means it is often refractory to treatment. Acupuncture has been reported to be effective in providing symptomatic relief of chronic low back pain. However, it is not known whether the effects of acupuncture are due to the needling itself or nonspecific effects arising from the manipulation. To determine the effectiveness of acupuncture therapy, a meta-analysis was performed to compare acupuncture with sham acupuncture and other treatments. Overall, 2678 patients were identified from thirteen randomized controlled trials. The meta-analysis was performed by a random model (Cohen's test), using the I-square test for heterogeneity and Begg's test to assess for publication bias. Clinical outcomes were evaluated by pain intensity, disability, spinal flexion, and quality of life. Compared with no treatment, acupuncture achieved better outcomes in terms of pain relief, disability recovery and better quality of life, but these effects were not observed when compared to sham acupuncture. Acupuncture achieved better outcomes when compared with other treatments. No publication bias was detected. Acupuncture is an effective treatment for chronic low back pain, but this effect is likely to be produced by the nonspecific effects of manipulation.