Heterogeneous feature-aware Transformer-CNN coupling network for person re-identification

CNN-based person re-identification

Zheng, Zheng & Yang (2017a) proposed a CNN-based Identity Embedding (IDE) model to construct Re-ID as a multi-classification problem by treating each pedestrian ID as a different class during the training process. After that, most of the Re-ID methods (Luo et al., 2019a; Sun et al., 2018; Zhong et al., 2017) based on representation learning are developed from this IDE model. Since CNNs were initially applied to basic tasks such as image classification, early integration of advanced CNNs into ReID task is usually based on the direct learning of image global features, which is obtained from the whole image as the first choice (learning the best global features from the whole task features). However, such a network model tends to focus more on the part that contributes most to the classification performance, instead of considering all features of the person (Chen et al., 2020). Importantly, the neglected parts often have recognition value as well. So, the global features obtained in the above way are sensitive to the redundant background, and invalidly cope with the pedestrian pose misalignment problem.

To alleviate the influence of background clutter on global features, most existing pedestrian re-identification methods can be classified into three categories. (1) One way is to transform deep-level features into explicit body parts representation with the help of location information extracted by the pose estimation model or the human key point estimation network (Gao et al., 2020; Sarfraz et al., 2018; Su et al., 2017). Unfortunately, state-of-the-art pose estimation networks are not perfect. Such methods are easily limited by these additional auxiliary models. (2) Another type of methos is mainly based on attentional mechanisms, and typical schemes including ABD-Net (Chen et al., 2019), ConsAtt (Zhou et al., 2019), and SONA (Xia et al., 2019), which capture the relationship between different convolution channels, multiple feature maps, layers, different body parts/regions, and even multiple images to enhance the model’s learning of features in pedestrian regions. However, the attention in these works is embedded deep into the CNN network. Due to the inherent limitations of the convolution operator perceptual field, the attention obtained by local convolution in this way ignores the global information and implicit relationships of the images, thus, limiting the ability of attention to learn correlations between features. (3) Multi-branch networks are originally designed to allow the network to learn various features from different perspectives. For existing network models employing multiple branches (Ning et al., 2020; Wang et al., 2018; Zhang et al., 2019), the features of each branch are either replicated from the higher levels of the global branch or extracted from a homogeneous network, leaving the model lacking structurally diverse differential features.

The feature extraction of the proposed network for key regions is learned directly from the data and context and does not rely on manually defined partial, partial region proposals, nor on pose estimation models.

Transformer-based person re-identification

Transformer has become increasingly popular for computer vision tasks due to its superior interaction capabilities with global self-attention. He et al. (2021) designed a pure transformer architecture combining additional camera encoded information and jigsaw patches module to learn discriminative features. Zhang et al. (2021) took advantage of transformer ability to capture global context to aggregate multiscale features between different layers of CNN and align part features by self-attentiveness. The potential of pure transformer network structure for image based ReID is demonstrated, however, transformer is susceptible to the interference of patch neighborhood detail loss and cannot capture local fine-grained information (Yuan et al., 2021). It may limit the ability to discriminate pedestrians with similar attributes. In addition, the transformer does not have the hierarchical structure of CNN, which makes it difficult to generate local spatial contexts at each level. Moreover, the transformer lacks the inductive bias (Dai et al., 2021; Dosovitskiy et al., 2021) inherent to the CNN branch and has a strong dependence on the amount of data. One of the pain points in pedestrian re-identification research is that there is not enough data to satisfy this exorbitant dependence.

Both the vision transformer and CNN have complementary advantages and limitations in dealing with visual representations. Many current works try to combine CNN and vision transformer in different ways (Li et al., 2021; Peng et al., 2021; Zhang et al., 2021). Inspired by the above work, we design heterotypical networks and combine the complementarity of varying features extracted from different paradigms in ReID to help enhance model performance.

Overview

The proposed framework (TCCNet) is shown in Fig. 1, which consists of three main parts: heterogeneous feature extractor based on CNN-transformer, low-level feature coupling module (LFCM), and high-level feature coupling module (HFCM).

In terms of the overall architecture, TCCNet is a two-branch parallel network taking the advantages of both vision transformer and CNN. The vision transformer branch captures substantial long-range dependencies by stacking multi-headed attention modules, primarily to learn the potential spatial topological of pedestrians in images. The CNN branch aggregates local significant semantic information in a hierarchical manner through deep stacking of convolutional operations. Maximizing the structural advantages of complementary CNN and vision transformer reduces the influence of background clutter toward the representation learning, and obtains more discriminative representations on pedestrian body regions, alleviating the problem of feature misalignment. TCCNet proposed in this article is not an iterative stacking of two branches, but a symmetric interaction network using two bridging modules, i.e., LFCM and HFCM, to achieve association (i.e., features that flow into the transformer-branch also feed into the transformer-branch, vice versa). LFCM can enhance the ability of CNN branches to perceive global information at a shallow level, while alleviating the lack of discriminative properties of transformer-branch for human boundaries due to the lack of fine local details, and enabling the complementarity of overall pedestrian features and local fine-grained features such as texture and color. The HFCM is used to aggregate the semantic information of different preferences extracted from transformer-branch and CNN branches in depth, so that the final pedestrian representation is structurally diverse and semantically rich. Finally, the dual-stream network is trained end-to-end with duplicate losses jointly. In the testing phase, the final outputs of the two branches are connected for identification, and the dual-stream network is trained end-to-end with duplicate losses jointly. During the testing period, the final outputs of the two branches are concatenated for identification.

Heterogeneous feature representation based on CNN-Transformer

CNN-based salient feature extractor. The CNN-branch is structured with a feature pyramid. The whole branch is divided into three stages, denoted as $\mathrm{S}\mathrm{t}\mathrm{a}\mathrm{g}{\mathrm{e}}_1$, $\mathrm{S}\mathrm{t}\mathrm{a}\mathrm{g}{\mathrm{e}}_2$ and $\mathrm{S}\mathrm{t}\mathrm{a}\mathrm{g}{\mathrm{e}}_3$, as shown in Fig. 1. If the input feature of one stage is defined as $x\in {\mathbb{R}}^{H\times W\times C}$ (where C, H and W denote the number of the channel, width and height of features, respectively), the output feature of this stage is $x\in {\mathbb{R}}^{\frac{H}{2}\times \frac{W}{2}\times (2\times C)}$. Corresponding to the three stages, three hierarchical feature maps with different resolutions can be obtained.

Each stage is composed of a different number of convolution blocks, set to 7, 8, and 8, experimentally. Except for the first convolution block in $\mathrm{S}\mathrm{t}\mathrm{a}\mathrm{g}{\mathrm{e}}_1$, all subsequent convolution blocks appear in pairs. The outputs of the convolution blocks at the same stage have the same feature resolution. The convolution block is a stack of three convolutional layers (1 × 1, 3 × 3, 1 × 1). The CNN-branch ends with a batch normalization layer and a global max pooling layer to generate one vector as pedestrian locally salient feature.

Transformer-based interrelated feature extractor. For the vision transformer branch, the input image passed the Transformer Stem (TStem) unit is first represented as a sequence of N patches $\left\{x_i|i=1,2,\dots ,N\right\}$ and modify them as the input of transformer blocks. Each patch is defined as $x_i\in {\mathbb{R}}^{P\times P\times C}$, where P and C respectively denote the spatial dimension and the number of channels. The total number of patches N depends on kernel size, stride, and size of the image. Here, N can be easily calculated using the Eq. (1). $H,W$ are height and width of an image, respectively, equal to P. $K_H$, $K_W$ are height and width of the kernel, and S is kernel stride.

(1) $$N=\lfloor \frac{H-K_H}{S}+1\rfloor \times \lfloor \frac{W-K_W}{S}+1\rfloor$$

The patches are then projected into N E-dimensional vectors using the learnable embedding $\mathcal{F}$. After that, a learnable class token ( $x_{cls}$), which can be used as a discriminative representation, is concatenated with the above N E-dimensional embedding representation ( $\mathcal{F}\left(x_i\right)$). Pedestrian images with the same ID are taken across cameras, resulting in a data distribution gap in the embedding subspace. In addition to introducing a learnable position embedding (PE) for each patch to learn the spatial information between blocks and a learnable camera embedding (CE) He et al. (2021) added to learn the camera information, which aims to reduce the bias caused by changing camera settings. The sequence of embedded patches $z_0$ is:

(2) $$z_0=\left[x_{cls};\mathcal{F}\left(x_1\right);\mathcal{F}\left(x_2\right);\dots ;\mathcal{F}\left(x_N\right)\right]+PE+CE,$$where $x_{cls}\in {\mathbb{R}}^{N\times E},\mathcal{F}\left(x_1\right)\in {\mathbb{R}}^{N\times E},PE\in {\mathbb{R}}^{\left(N+1\right)\times E},CE\in {\mathbb{R}}^{\left(N+1\right)\times E}.$

Then, $z_0$ is feed to L repeated transformer blocks. Every transformer block consists of layer normalization (LN), multi-head self-attention (MSA), and multilayer perceptron (MLP) in sequential order. The input and output of the multi-headed attention layer are connected with residuals and layer normalization.

(3) $$z_l^{\mathrm{\prime }}=MSA\left(LN\left(z_{l-1}\right)\right)+z_{l-1}l=1,\dots ,L,$$

(4) $$z_l=MLP\left(LN\left(z_l^{\mathrm{\prime }}\right)\right)+z_l^{\mathrm{\prime }}l=1,\dots ,L.$$

The feature z is first passed to the MSA layer in every transformer block. MSA can be expressed as Eq. (5).

(5) $$MSA\left(z\right)=Concat\left({\mathit{H}}_1,{\mathit{H}}_2,\dots ,{\mathit{H}}_h\right){\mathit{W}}^o,$$where h is the number of heads in MSA, and concat (∙) denotes stacking on the embedding representation dimension of the patch embedding. Each header ${\mathbf{H}}_i(i=1,2,\dots ,h)$ can be expressed as follows:

(6) $$Attention(\cdot )=softmax\left(\frac{Q{K}^T}{\sqrt{d}}\right)V,$$

$$\begin{array}{c}Q=z{\mathit{W}}_i^Q\\ K=z{\mathit{W}}_i^K\\ V=z{\mathit{W}}_i^V\end{array}$$where Q, K, V are generated by three completely different linear projections, respectively. It is seen that Q, K, V are expressed as ${\mathit{W}}_i^Q,{\mathit{W}}_i^K\in {\mathbb{R}}^{E\times d_k}$, ${\mathit{W}}_i^V\in {\mathbb{R}}^{E\times d_v}$, which means they are more expressive. Attention (∙) is a function used to calculate the relevance and importance of the patch embeddings. $\sqrt{d}$, the scaling factor, is used to ensure numerical stability through normalization.

The MLP contains a two-level linear projection and a GELU activation function. It is denoted as:

(7) $$MLP\left(z\right)={\mathit{W}}_2\sigma \left({\mathit{W}}_{1}z\right),$$where ${\mathit{W}}_1$, ${\mathit{W}}_2$ are the parameters of the two linear projections and σ is the GELU activation function.

The last class token of representation captured by the vision transformer-branch serves as a representation of the overall pedestrian structure.

Separated double stem unit. Making full use of the feature base captured by lower of the network will achieve greater discriminability. We advocate the idea of using the Separated Double Stem, the transformer Stem (TStem) and CNN Stem (CStem), to perform down-sampling on the input image. In other words, TStem and CStem are constructed by two different convolution operations. CStem and TStem are both 16 × 16 filter with stride of 12 to get overlapping patches. The difference is that CStem contains 64 filters, while TStem contains 768 filters. Overlapping patches can introduce correlations between neighboring blocks, which in turn reduces the impact of the structure being corrupted and learns more local detail information. Heterogeneous networks have different ways of processing representations (Woo et al., 2018), leading to varying requirements for their feature response maps in the lower layers of the network. Specifically, the vision transformer-branch captures the holistic information from a global perspective at the low-level of the network. In contrast, the CNN-branch gradually learns global features as the depth of the network increases. Avoiding the limitations and maximizing the advantages of both network architectures is essential. The experimental results and analysis about this part are shown in “Analysis of separated double stem unit”.

Heterogeneous feature coupling

Global perception module. This is a plain module composed of only a fully connected layer, but the contribution of it is not negligible. Essentially, it is to allow the CNN branch to collect perceive information from a larger region ahead of the LFCM. Invalid features in complex backgrounds can have a suppressive effect on extracting discriminative pedestrian information. Features captured by convolution with limited receptive fields may compromise the representation obtained by transformers from self-attention, which leads to a limited ability to interact with heterogeneous features. Global Perception Module makes a contribution to alleviate this problem.

Feature coupling module. There are significant differences between CNN and transformer branches in the way of process representations and the ability to integrate semantic information in high-level. The CNN-branch strictly obeys the process of converging global features from local features. In contrast, both local and global information are captured in the lower and higher layers of transformer. As a result, the difference between the representations in the two branches increases layer by layer. In order to obtain a semantically rich pedestrian representation, the complementarity of the two branches needs to be enhanced. To ensure the effectiveness of the heterogeneous feature fusion, the strategy of integrating all layer features in a very extreme way is discarded. The heterogeneous feature coupling module, which is shown in Fig. 2, is applied to the low and high layers of the TCCNet (where correspond to the stage1 and stage 3), respectively. Specifically, inspired by conformer, the features extracted by the CNN-branch are dense three-dimensional, and the embedding representation dimension of the transformer-branch is two-dimensional. When CNN-branch features flow into transformer-branch, the channel dimension should first be transformed with a 1 × 1 convolution. The features in different channels can be understood as the response of the input image towards different feature patterns. There is a semantic gap between the features extracted from the vision transformer and CNN branch, so the features of the CNN branch use the global average pooling operation to retain some features that best express the image, and these focal features are normalized by Layer Normalization and then converged to the vision transformer branch to achieve detailed features in the key region. The representation obtained from the Transformer is also fed back to the CNN-branch by first aligning the channels with a 1 × 1 convolution, and then adding them to the feature of the CNN-branch after Batch Normalization and interpolation to align the spatial resolution of the features with the CNN-branch.

Supervision learning with duplicate loss

Loss function. In this article, the network is supervised using cross-entropy loss with BNNeck (Luo et al., 2019a) (called ID loss in this article) and triplet loss. ID loss is used for representation learning that can motivate the network to learn the implicit relationships that exist among identities. The purpose of the person re-identification task is to minimize the distance among the same identities while maximizing the distance among different identities. The core principle of triplet loss is to pull the distance between positive sample pairs and push away the distance between negative sample pairs. Therefore, triplet loss is introduced to explicitly learn the required similarity measure for recognition. ID loss and triplet loss have complementary advantages in learning discriminative pedestrian representations (Zheng, Zheng & Yang, 2017a).

The ID loss is defined as:

(8) $$L_{ID}=-\frac{1}{N}{\sum }_{i=1}^N{q}_i\log \left(p_i\right),$$where $q_i$ denotes the ground-truth label, $p_i$ is the probability that the network predicts identity i, and N is the number of images in each batch of the input.

The triplet loss is defined as

(9) $$L_{tri}=\log \left[1+\exp \left(d_{pos}-d_{neg}\right)\right],$$where $d_{pos}$ and $d_{neg}$ denote the distances of positive and negative sample pairs, respectively.

The loss of a single branch can be expressed as

(10) $$L_{CNN}=L_{transformer}=\alpha L_{tri}+\left(2-\alpha \right)L_{ID},$$where α is used to balance the two losses.

Duplicate loss. To motivate the two branches to learn their preferred features, we optimize the network by constructing duplicate loss, which is a separate classifier for each branch, as shown in Fig. 1. The total loss is expressed as

(11) $$L_{total}=\lambda L_{transformer}+\left(1-\lambda \right)L_{CNN},$$where λ is used to balance the losses of the two branches. In this article, α is set to 1 and λ is set to 0.5 based on experiments to optimize the network model by minimum $L_{total}$.

Similarity measurement

During the similarity measurement, the features of the images in the Query and Gallery sets are first extracted with the trained model, and then the recognition distance matrix is obtained by calculating the Euclidean distance between the two feature sets in the embedding space. The image with the smallest distance is considered as the most likely candidate person.

Given a set of test images is divided into Query and Gallery, with Query denoted as $Q={\left\{q_i\right\}}_{i=1}^m$ and Gallery denoted as $G={\left\{g_i\right\}}_{i=1}^n$, the feature vectors corresponding to Q and G are denoted by $F_Q$ and $F_G$, respectively, and shown as follows:

(12) $$F_Q=\left[f_{q1},f_{q2},\cdots ,f_{qm}\right]\in {\mathbb{R}}^{m\times d},$$

(13) $$F_G=\left[f_{q1,}{f}_{q2},\cdots ,f_{gn,}\right]\in {\mathbb{R}}^{n\times d},$$where $m$ denotes the number of images contained in Query, $n$ denotes the number of images contained in the Gallery, and $d$ denotes the image feature dimension. Because the Euclidean distance is the simplest and easiest to understand, the distance between Query image $q_i$ and Gallery image $g_i$ is expressed as:

(14) $$d\left(q_i,g_i\right)={\Vert \overrightarrow{f_{qi}}-\overrightarrow{f_{gi}}\Vert }_2=\sqrt{{\left(\overrightarrow{f_{qi}}-\overrightarrow{f_{gi}}\right)}^T\left(\overrightarrow{f_{qi}}-\overrightarrow{f_{gi}}\right)},i\le m,j\le n,$$where $f_{qi}$ and $f_{gi}$ denote the features obtained by inputting images $q_i$ and $g_i$ into the TCCNet network.

Datasets

The proposed model is evaluated on two public datasets, i.e., Market1501 (Zheng et al., 2015) and MSMT17 (Wei et al., 2018) datasets. Market1501 is a dataset consisting of 32,668 images of 1,501 pedestrians captured by six different cameras. MSMT17 is captured by 12 outdoor cameras and three indoor cameras, totally 126,441 images of 4,101 pedestrians. It is a large dataset closer to the real scene and is more reflective of the model performance. The details of two datasets are shown in Table 1.

Implementation details and evaluation metrics

All experiments were implemented on NIVDIA A100-SXM4-40G GPUs. Initial weights of Transformer were pre-trained on ImageNet-21K and then fine-tuned on ImageNet-1K.

Before training, the resolution size of the input image was adjusted to 256 × 128, and the training batch was set to 64, including 16 different pedestrians and four images each. The data were augmented by random horizontal flipping, padding, random cropping, random erasing and normalization strategies. The total number of training epoch was set to 150. The model was optimized using SGD, where the weight decay factor was set to 1e–4 and the momentum was set to 0.9. The initial learning rate was 0.009 with cosine learning rate decay.

In this article, the experimental performance was evaluated using the cumulative matching curve (CMC) at Rank-1 and the mean average precision (mAP). Compared to CMC. The mAP is a more comprehensive evaluation metric, which is calculated as:

(15) $$mAP=\frac{1}{m}{\sum }_{i=1}^{m}A{P}_i,$$where $m$ is the number of query images in the query set; $A{P}_i$ is the average precision of the i-th query image.

Analysis of separated double stem unit

Separated double stem unit. In order to verify the effectiveness of the proposed Separated Double Stem Unit, two sets of experiments are carried out. The experimental results are shown in Table 2. Shared Single Stem means that the CNN-branch and vision transformer branch share weights, while Separated Double Stem means that the CNN branch and vision transformer branch each have independent stem modules. The experimental results show that the use of large convolutional kernels for overlapping to slice the patches achieves the best results on both sets of experiments, and separated double stem achieves the best experimental results among all the experimental results by this slice.

Experiments in the Separated double stem section show that TStem and CStem achieve optimal results in over-lapping and non-overlapping modes using convolutional kernels of the same size. Convolutional kernels with the same size ensure the consistency of the perceptual fields (Sun et al., 2018; Zheng, Zheng & Yang, 2017b) of CStem and TStem, allowing the two branches to align features efficiently and facilitating feature fusion in the LFCM.

Analysis of global perception module. To enable the CNN branch to collect information over a relatively large area, the feature space resolution is adjusted using an up-sampling operation. In Table 3, different up-sampling strategies are compared. It is found that the linear approach provides a more significant contribution to the improvement of the model performance.

Analysis of feature coupling module

Analysis of LFCM. The first stage of the CNN branch contains three pairs of convolutional blocks, which means that, in this phase alone, there are multiple fusion modes. In order to design an effective low-level feature coupling module, we compare all possible effects of different feature coupling in the first stage, which is shown in Table 4. Here, we define whether the three pairs of convolutional blocks in the first stage interact with the transformer-branch with features as { ${\mathrm{b}}_1,{\mathrm{b}}_2,{\mathrm{b}}_3$}, and ${\mathrm{b}}_{\mathrm{i}}=1$ indicates that the i-th pair of the convolutional block in the CNN branch interacts with representation in the transformer-branch.

From the experimental results in Table 4, it is seen that the fusion method of index 5 is optimal. Moreover, the performance of the second pair of convolutional block features for interaction (index 5, 6, 7) is superior to that without interaction (index 1, 2, 3, 4), except for index 8, which shows the necessity of feature interaction between the second pair of convolutional blocks in the CNN-branch and the transformer-branch.

The reason why the performance of all three pairs of residual blocks with interaction (index 8) is worse than the performance of all other choices may be that it produces redundant features, overemphasizes certain non-accurate regions, misleads the learning of the network, and the model performance decreases.

Analysis of HFCM. Quantitative experiments are performed to find the appropriate number of feature coupling in high-level layers. The third stage of the network has four convolutional layers, which can be classified into four categories according to the number of fusion layers and subdivided into 16 options. The detailed results corresponding to this are shown in Table 5. The histogram in Fig. 3 represents the model performance with the different number of fusion layers. The line graphs indicate the mean value of mAP corresponding to each class of fusion layers. As shown in Fig. 3, it is found that model without coupling of high-level features are not robust enough, and performance of the model is the worst. The model performance rises gradually with increase of the number of coupling layers in HFCM. The best performance can be achieved when the number of fusion layers is set to four (i.e., the features of third stage are all fused). The HFCM can comprehensively use the overall features and local discriminative features of pedestrians to further locate the high response features of pedestrians on the basis of getting the complete features of pedestrians, which increases the structural diversity required for the model to learn the differential features. As a result, the final pedestrian representation has more valid information and can improve the generalization ability of features.

Ablation experiments of heterogeneous feature coupling modules. According to the experimental results in Table 6, with the same data augmentation and tricks (Luo et al., 2019a), a large gap between the model capacity of the ViT series and the CNN series can be observed, but the same gap in performance is obvious. The mAP achieves 56.7% using ResNet50 as the baseline network (ResNet50-BOT) and 61.8% using vision transformer as the baseline network (ViT-BOT). The mAP of our proposed TCCNet can reach 66.9%, which indicates that the heterogeneous network structure extracts comprehensive and highly significant features, largely reducing the effect of background noise on the model, thus achieving improved model performance. Furthermore, we set the deeper ResNet101 network as the backbone network. It brings some performance improvement (+2.1%/+0.8%) compared with ResNet50-BOT, but it is still far behind our TCCNet, indicating that the extra branch parameters are not the key contributors to the improvement, but rather the well-designed network architecture. We compare the inference times of several other models with ResNet50-BOT. Although it is larger than other models in capacity, the TCCNet model is superior to several other models in terms of performance, and we will further reduce the model params.

From the experimental results in Table 6, it is very essential for our model that HFCM and LFCM, and removing either module will cause a degradation in model performance. The LFCM and HFCM greatly enhance the global perception of local features of the pedestrian and the local details of the global representation of the network.

Analysis of duplicate loss

Choice of supervision strategy. Supervision strategy directly affects the performance of the heterogeneous networks with joint training. Several different supervision strategies and their corresponding feature representations at test stage are examined. Figs. 4A–4C represent single-loss learning. That is, the losses shown in Eqs. (8) and (9) are calculated once for the two-branch fusion feature, and then the two obtained loss values are summed, and the gradient is back-propagated to optimize the network. There are three ways of feature fusion in the single-loss strategy: including element-by-element summation (a), concatenate (b), and element-by-element multiplication (c). Fig. 4D illustrates the duplicate-loss joint learning process. The losses shown in Eq. (10) are computed once for each branch, and the four obtained loss values are summed to optimize the two branches. From the experimental results in Table 7, we can observe that the duplicate-loss joint training strategy performs better and the single-loss training approach is not effective in further improving the model. This is because the duplicate-loss can drive both branches to highlight their own strengths and learn their preferred features. In contrast, it is difficult to train the network stably with only single-loss constrained heterogeneous features.

Experiment with parameter sensitivity. The α defined in Eq. (10) is used to represent the weight of the triplet loss in each branch, and λ defined in Eq. (11) is used to represent the weight of the vision transformer branch loss. λ is selected from 0.1 to 1, taking a value every 0.1 interval. We fix each λ and vary α from 0.2 to 1.8, taking a value every 0.2 intervals. Table 8 shows the best performance produced by taking fixed values of λ, corresponding to one α. The total loss where λ is equal to 0.5 and α is equal to 1 has a better performance.

Figure 5 provides the graphical representation. (a) is the performance of the model with varied α when λ is fixed at 0.5. When α is set to 0.2, the performance of the model produces dramatic decreases. Since triplet loss explicitly learns the similarity metric needed for recognition. As the value of grows, the performance improves. The model performance decreases when α is more than 0.5. ID loss considers the variability between one sample and all samples, making the features near the classification layer more focused on the differences in training identities. The model can be driven to explore useful detailed information in the limited information available, and ID loss is indispensable in person re-identification tasks. (b) in Fig. 5 indicates the trend of λ when α is equal to 1. As the λ weight increases, the accuracy of pedestrian recognition increases and then decreases. The best accuracy is obtained at λ = 0.5. This indicates that the contributions of both branches are equally important in the learning process. The CNN branch is biased to focus on more significant discriminative features, while the vision transformer branch focuses on more regions of discriminative features than the CNN. When the weights of the two branches are equal, the model can adaptively filter among the features with high relevance to the pedestrians, and then capture the comprehensive and significant discriminative features of the pedestrians, allowing the model to achieve the best performance.

Comparison with existing relevant person ReID approaches

The results of the comparison with the existing relevant methods are shown in Table 9. The methods in the first group are local feature approaches based on fixed regions. The methods in the second group are semantic-based local feature methods using additional auxiliary models. The third group of methods is based on a CNN-based attention mechanism method, and the fourth group is a method that introduces vision transformer (where $\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{R}\mathrm{e}\mathrm{I}{\mathrm{D}}^{\ast }$ means without a sliding-window setting).

MSMT17. On this dataset, our framework obtains the best performance, as shown in Table 9. And it outperforms all other methods by at least 4.7% and 0.8% in both mAP and Rank-1 identification rate, respectively. The compared methods are divided into three groups, i.e., local feature extraction methods based on fixed regions, methods using additional auxiliary models to extract semantic-level local features, and methods using vision transformer-based methods, and TCCNet outperforms all of these methods by a considerable advantage. MSMT17 is a large-scale dataset that covers a long period of time and presents complex illumination variations. This validates that our proposed framework can extract to the effective semantic information as well as locally significant information to retrieve difficult samples.

Market1501. Table 9 shows the evaluation of the Market1501 dataset. We compare the recent CNN-based and Transformer-based approaches with our model. From the results, we can see that our framework also achieves the best performance in Rank-1 identification rate and mAP. Since the performance of this dataset is close to saturation, the results of these state-of-the-art methods are very close.