U-strips of fiber reinforced polymer (FRP) are usually used for shear strengthening of reinforced concrete beams. A novel prestressing system with hybrid anchorage is proposed, in which the prestressed FRP is bonded to beam surfaces and the ends are anchored by interlocking. Eight rectangular beams were tested to validate the feasibility of the new method, including one control beam, one strengthened with bonded CFRP U-strips and six strengthened with hybrid-anchored and prestressed CFRP U-strips. The results show that the proposed prestressing method performs much better than externally bonding in the aspects of restraining the propagation of diagonal cracks and improving the strain behavior of steel stirrups. The end-debonding of CFRP can be shifted to rupture failure, the fiber-strength utilization can be significantly enhanced, and the shear capacity of RC beam can be increased with the maximum increase rate about 92%. The strengthening effect is related to the area ratio and the prestress of CFRP. Raising area ratio or prestress value is helpful to improve the comprehensive performance of the strengthened beams. Furthermore, a formula for calculating the effective strain of prestressed CFRP was proposed and then used to calculate shear capacities of the strengthened beams. The calculated capacities are all in good agreement with the test results, indicating that the formula can serve as a rational reference for engineering application.