Accurate cell counting in biomedical images is a fundamental yet challenging task for disease diagnosis. The early manual cell counting methods are mainly based on detection and regression, which are time-consuming and prone to errors. Benefiting from the advent of deep learning, convolutional neural network (CNN)-based cell counting has become the mainstream method. Despite the outstanding performance of CNN-based cell counting methods, the complex tissue background in medical images still hinders the accuracy of cell counting. In this paper, to solve the problem of complex tissue background and improve the performance of cell counting, an attentive recognition network (ARNet) is built. Specifically, the ARNet is composed of five convolution blocks and a channel attention (CA) module. The convolution blocks are employed to extract the basic features, and the CA module is introduced to suppress the complex background by recalibrating the weight of each channel to pay more attention to cells. Subjective and objective experiments on synthetic bacterial cells (SBC) dataset and modified bone marrow (MBM) dataset prove that the proposed ARNet outperforms the mainstream methods in accuracy and stability.