A new algorithm for removing pixel cross-talk effect in CMOS image sensors with Bayer color filter array is proposed. With the assumption that color differences within small neighbors of an image is constant, the algorithm uses the R or B components of raw data to compensate the adjacent G component before color interpolation in order to eliminate the effects that R component interferes with Gr and B component interferes with Gb. The algorithm has low computational complexity. By simulation experimentation on Macbeth color chart and real images, the algorithm is proved to be effective and produces better quality images with conventional color interpolation methods. Finally, the paper design its VLSI frame and validates it with FPGA. The results of test indicate that the algorithm uses fewer hardware resources and can work in real-time completely.