Flower pollination algorithm (FPA) is a new meta-heuristic optimization algorithm that mimics the real life processes of the flower pollination. FPA has been proven to be an effective tool in solving plenty of global optimization problems. In this study, visual tracking is considered to be a process of optimal reproduction of flowering plants. Meanwhile, an Improved Flower Pollination Algorithm (IFPA) is presented with the switch probability p changing dynamically with generation number. An IFPA-based tracking architecture is presented and the parameters’ sensitivity and adjustment of IFPA in tracking system are studied experimentally. To verify the tracking ability of the proposed tracker, comparative studies of tracking accuracy and speed of the proposed tracker with particle filter, mean shift and PSO are presented. Comparative results show that the IFPA-based tracker outperforms the other three trackers.