Particle filter algorithm has been proven to be a powerful tool in solving visual tracking problems. However, the problem of sample impoverishment which is brought by the procedure of re-sampling is a main handicap of the particle filter. In this work, an improved particle filter based on firefly algorithm is proposed to solve this problem. The particles in the particle filter are optimized using firefly algorithm before re-sampling. Thus, the number of meaningful particles can be increased, and the particles can approximate the true state of the target more accurately. Experimental results on visual tracking show that the proposed algorithm outperforms the standard particle filter and it can track targets robustly in various challenging conditions.