Among deep‐learning‐based trackers, the siamese‐based method inspires many researchers due to its effectiveness and simplicity. However, the traditional siamese tracker has not achieved satisfactory performance due to the limited representation ability and the lack of appropriate model update strategy. To cover the shortage of siamese models, we proposed a cross‐similarity‐based siamese network with three contributions. First, we introduce a novel cross similarity module into the SiameseFC framework, which could improve the matching ability of fully convolutional networks during the tracking process. Second, we propose a novel attention weighting layer to emphasize various contributions of matching scores in different positions. This adaptive attention weighting scheme makes our tracker well adapt to the appearance change caused by pose variation, partial occlusion, and so on. Third, we develop a simple yet effective model update strategy, which exploits an independent classification model to invoke the model fine‐tuning process. Experimental results on the standard tracking benchmark show that our tracker performs much better than the baseline SiameseFC method and also achieves promising results in comparisons to other state‐of‐the‐art algorithms.