MA‐CAT: Misclassification‐Aware Contrastive Adversarial Training

Vulnerability to adversarial examples poses a significant challenge to the secure application of deep neural networks. Adversarial training and its variants have shown great potential in addressing this problem. However, such approaches, which directly optimize the decision boundary, often result in overly complex adversarial decision boundaries that are detrimental to generalization. To deal with this issue, a novel plug‐and‐play method known as Misclassification‐Aware Contrastive Adversarial Training (MA‐CAT) from the perspective of data distribution optimization is proposed. MA‐CAT leverages supervised decoupled contrastive learning to cluster nature examples within the same class in the logit space, indirectly increasing the margins of examples. Moreover, by taking into account the varying difficulty levels of adversarial training for different examples, MA‐CAT adaptively customizes the strength of adversarial training for each example using an instance‐wise misclassification‐aware adaptive temperature coefficient. Extensive experiments on the CIFAR‐10, CIFAR‐100, and SVHN datasets demonstrate that MA‐CAT can be easily integrated into existing models and significantly improves robustness with minimal computational cost.

examples closer and push those from different classes far away in the logit space.Note that we only use natural examples without any adversarial examples in the contrastive learning regularization term.The contrastive learning regularization term is a plug-and-play approach that can be effortlessly incorporated into existing models without significantly modifying the structure.Furthermore, the calculation of this term is a one-time process with low computational overhead.
Adversarial examples are commonly defined as perturbed versions of natural examples that a model has correctly classified.Nevertheless, there is no explicit definition of adversarial examples that are crafted from misclassified natural examples. [22]revious research [18,22]  Above all, we propose Misclassification-Aware Contrastive Adversarial Training (MA-CAT), a method that employs supervised contrastive learning during adversarial training to optimize the logit distribution of examples and increase example margin.MA-CAT also incorporates an instance-wise misclassification-aware adaptive temperature coefficient to dynamically adjust the training strength during the training phase.Extensive experiments conducted on the CIFAR-10, [23] CIFAR-100, [23] and SVHN [24] datasets using the PreActResNet [25] and WideResNet [26] networks demonstrate the effectiveness of MA-CAT in further enhancing the model's adversarial robustness with minimal computational overhead.
The main contributions of this paper are as follows: 1) We present a new viewpoint that optimizing the data distribution to learn a straightforward adversarial decision boundary is beneficial for improving robust generalization.2) We propose MA-CAT, which employs supervised decoupled contrastive learning to optimize the data distribution.Furthermore, we introduce an instance-wise misclassification-aware adaptive temperature coefficient to dynamically adjust the adversarial training strength for different examples during the training phase.
3) Extensive experiments demonstrate that MA-CAT can be effortlessly incorporated into existing popular adversarial training methods and further enhance their robustness with minimal computational cost.
The article adheres to a conventional structure with Section 2 reviewing prior research and existing approaches in the field of adversarial training and contrastive learning.Section 3 explains the notations and fundamental concepts, while Section 4 provides a detailed description of the proposed MA-CAT method including the incorporation of supervised decoupled contrastive learning and the instance-wise misclassification-aware adaptive temperature coefficient.Section 5 reports the experimental setups and presents and analyzes the empirical results to demonstrate the effectiveness of MA-CAT.The article's primary contributions are summarized in Section 6, along with potential research directions.

Adversarial Training
Adversarial training is considered to be one of the most effective defense methods against adversarial attacks.Some methods enhance model robustness by maximizing the distance between examples and the decision boundary.MMA [18] maximizes the shortest successful perturbation to increase the distance between the input and the decision boundary.IAT [17] aligns adversarial examples with their reverse adversarial counterparts during training to increase the distance between adversarial examples and the decision boundary.DyART [19] directly optimizes the distance through a closed-form expression for the gradient of the distance.Despite their success, these methods either learn complex adversarial decision boundaries or involve significant computational overhead, limiting the establishment of model robustness or the application of large models.Inspired by contrastive learning, we argue that incorporating it into adversarial training could increase the distance with minimal computational overhead by optimizing the data distribution.Some works have also utilized contrastive learning to enhance adversarial robustness, of which Jiang et al. [27] is the most representative.Jiang et al. [27] propose a robust pre-training method based on adversarial contrastive learning.However, their method is fundamentally different from ours in the following aspects: 1) Application goals.In our study, the goal of applying contrastive learning is to cluster examples of the same class, increase the margin, and increase the distances between different classes.However, their main focus is on using contrastive learning to learn invariant features of natural and adversarial views by aligning natural examples with their adversarial counterparts.
2) Contrastive learning methods.We utilize supervised decoupled contrastive learning, while they employ unsupervised contrastive learning.3) Application phases.We incorporate contrastive loss as a regularization term within the adversarial training phase.In contrast, they employ contrastive learning during the pre-training phase followed by adversarial fine-tuning.
Moreover, Mao et al. [28] incorporated a triplet metric learning regularization term into adversarial training.However, their focus primarily lies in aligning natural examples with adversarial examples, which is fundamentally different from our approach.It is noteworthy that supervised contrastive learning has advantages over metric learning in adversarial training.The former has the inherent ability to mine hard negative examples, whereas the latter requires an additional mining algorithm to do so. [29]dditionally, in contrast to metric learning, contrastive learning allows for the adaptable adjustment of the strength of adversarial training for both correctly classified and misclassified natural examples by adjusting the temperature coefficient during adversarial training.

Contrastive Learning
Contrastive learning is an important research direction in selfsupervised learning tasks that has made significant breakthroughs in recent years. [21,30,31]Its goal is to reduce intra-class variations and increase inter-class distances by bringing positive examples closer and pushing negative examples far away in the representation space.Wu et al. [30] regarded each example as a category and viewed the remaining examples as negative examples.They proposed using a memory bank to store representations of each example and randomly selecting examples from this bank during training.Tian et al. [21] expanded unsupervised contrastive learning to supervised one by using the natural categorization of labels.Moreover, Yeh et al. [31] discovered that the negative-positive-coupling (NPC) effect in contrastive loss makes the model highly sensitive to batch size and proposed a decoupled contrastive learning method.
In addition, there is a crucial hyperparameter in contrastive learning-the temperature coefficient τ. τ plays an important role in controlling the smoothness of the probability distribution over positive and negative examples. [29]A smaller τ sharpens the distribution, amplifying the differences among examples and making the model focus more on challenging examples.Conversely, a larger one makes the distribution smoother and treats all examples equally.This hyperparameter holds substantial importance as it directly affects the quality of the learned representations and influences the clustering performance within contrastive learning frameworks.Existing contrastive learning methods typically set τ as a small constant.However, recent research [32,33] from different tasks and perspectives has shown that a fixed τ may not always be the optimal choice.
The success of the dynamic τ mechanism in refs.[32,33]  inspires us to investigate the solution of the different levels of difficulty in different examples using adaptive τ.

Notations
We consider a C-class classification task over the training set where N is the total number of training data pairs.The classifier f θ ∶X !Y, parameterized by θ, maps from the input space X to the output labels Our study mainly concentrates on the l ∞ -norm.However, our approach can be generalized to other norms.Let ℒ CE ðf θ ðxÞ, yÞ denote the cross-entropy loss between f θ ðxÞ and y.

Vanilla Adversarial Training
The vanilla AT can be formulated as the following min-max optimization problem: [20] min θ E ðx,yÞ$D max where E½⋅ denotes the expectation function, δ is the adversarial perturbation, and x þ δ denotes the adversarial example in the l p -norm ball centered at x with radius ϵ.The inner maximization problem is employed to search for the worst-case adversarial examples in the norm ball.The outer minimization problem then updates the model to optimize the adversarial risk, based on the adversarial examples generated by the inner maximization.By optimizing the min-max saddle point problem, AT enhances the adversarial robustness of the model.

TRADES
TRADES incorporates a Kullback-Leibler (KL) divergence regularization term with the cross-entropy loss to alleviate the trade-off between natural accuracy and robustness.The objective is as follows: [15] min where ℒ KL denotes the KL divergence and λ is a balancing hyperparameter that balances the performance between the natural accuracy and robustness.The first term contributes to the natural accuracy, while the second term helps to improve the adversarial robustness.
Direct optimization of the decision boundary in AT and its variants often leads to overly complex decision boundaries, which hinders model generalization and increases training difficulty.From a different perspective, this article proposes to optimize the example distribution to move examples away from the decision boundary in the logit space, thereby increasing the margin to improve the robustness of the model.We provide a detailed explanation of our method in Section 4.

Method
In this section, we first define the supervised decoupled contrastive loss.We then introduce the novel misclassification-aware adaptive temperature coefficient, along with its update method.Finally, we present the overall loss function and the pseudocode of MA-CAT.

Supervised Decoupled Contrastive Learning
Yeh et al. [31] propose unsupervised decoupled contrastive learning to address the NPC issue.The loss function is as follows: where N is the total number of training examples, z ðkÞ i represents the feature vector of example x i from view k, and ⋅, ⋅ h idenotes the dot product indicating the similarity between two vectors, and τ is a fixed temperature coefficient.
In unsupervised contrastive learning, each example is considered as a distinct category, where positive examples are different views of the same example, and negative examples are different examples with different views.In this setting, the model learns to bring together different views of the same example, while pushing away all other examples.The goal is to learn invariant features between different views of the same example, and the negative examples are primarily to prevent the model from collapsing.This differs from our goal, where we focus on gathering examples from the same class while pushing away examples from different classes.Therefore, unsupervised contrastive learning cannot be directly applied in this context.To address this issue, we propose an extension from unsupervised decoupled contrastive learning to supervised decoupled contrastive learning, which can be formulated as follows: where I ½⋅ is the indicator function, if the condition is satisfied, it outputs 1, otherwise 0. s i,j denotes the cosine similarity between z i and z j .ℒ i contras determines the positive and negative examples based on the labels.The numerator of ℒ i contras mainly consists of the similarity between the anchor and the positive example, while the denominator primarily involves the similarity between the anchor and the negative examples.During the training phase, the model gradually increases the similarity between the same class examples, eventually causing them to cluster.In this way, the distance between the examples and the decision boundary is increased.If the current model can classify a natural example correctly, it indicates that the example is currently "trustworthy", i.e., it does not cross the decision boundary.When this example serves as an anchor in contrastive training, other examples from the same class can be safely brought closer to it.The temperature coefficient of the anchor is reduced to increase the strength of the adversarial training.Conversely, the misclassification of a natural example by the current model shows that it is "untrustworthy".In contrastive learning, its temperature coefficient is increased to reduce its loss and adversarial training strength, as shown in We begin by defining "trustworthy" and "untrustworthy" examples, and then present the formula used to update the instance-wise misclassification-aware adaptive temperature coefficient.
Definition 1.For a natural example x, if f θ ðxÞ ¼¼ y, it is a trustworthy example.Otherwise, it is an untrustworthy example.
Suppose that an adversarial training has reached epoch t, the temperature coefficient associated with the example x i at this stage is τ t i .Before starting the t þ 1 epoch, we need to update the τ t i for example x i based on its classification result by the current model.According to our previous analysis, we should decrease the τ t i if x i is trustworthy.Otherwise, we should increase the τ t i .To prevent τ t i from becoming too large or too small, we introduce two hyperparameters, τ max and τ min , as the upper and lower bounds of τ t i , respectively.The value of τ max and τ min will be determined in subsequent experiments.The proposed method for updating the adaptive temperature coefficient can be formulated as follows: (5)   where η denotes the update step size of the temperature coefficient.Note that the τ i can gradually converge to a proper range with training.

The Total Loss Function
MA-CAT is a plug-and-play method that can be effortlessly incorporated into existing popular adversarial training methods.Here, we introduce the total loss function of incorporating MA-CAT into TRADES as an example.In the experimental section, we will demonstrate that MA-CAT can be incorporated into other adversarial training methods.
When MA-CAT is incorporated into TRADES, the total loss function is where β is the coefficient of the MA-CAT regularization term, which will be determined in the experimental section.The pseudo-code is presented in Algorithm 1.

Experimental Section
In this section, we conducted extensive experiments on different datasets and models to demonstrate the effectiveness of MA-CAT.We began with an introduction to the experimental setups and then discussed the upper and lower bounds of the temperature coefficient and β through experiments.Furthermore, we evaluated the adversarial robustness of MA-CAT over different baselines and datasets.

Experimental Setups
We evaluated our MA-CAT on CIFAR-10, CIFAR-100, and SVHN datasets with PreActResNet-18 and WideResNet-34-10 as the target models.Based on the findings, [34] which suggest that the robustness of defense against targeted attacks is weaker than that against non-targeted attacks, we primarily focus on evaluating the adversarial robustness of MA-CAT against nontargeted attacks.All the experiments were conducted using PyTorch v2.0.0 on a GTX 3090 GPU, and the operating system used was Ubuntu 16.04 LTS.x 0 ← max

Adversarial Training Settings
Following the widely used settings, [35] we trained all the models for 100 epochs using SGD with batch size 128, momentum 0.9, weight decay 5 Â 10 À4 , and initial learning rate 0.

Evaluation Settings
We evaluated the adversarial robustness of MA-CAT mainly using white-box attack methods, including FGSM, [36] PGD-20, [20] CW-20, [37] and AutoAttack (AA). [38]AA combined three white-box attacks, namely APGD-CE, [38] APGD-DLR, [38] and FAB, [39] along with the black-box Square Attack. [40]By combining these attack methods, AA provided a comprehensive evaluation of the model's adversarial robustness in both white-box and black-box scenarios.This comprehensive evaluation enabled for a reliable evaluation of the model's robustness against adversarial attacks.

Sensitivity to τ max and τ min
In this section, we evaluated the sensitivity of MA-CAT to the upper and lower bounds of τ on CIFAR-10 using PreActResNet-18.τ played a critical role in MA-CAT as it controls the strength of adversarial training.To investigate their impact on the robustness of the model, we varied the τ.TRADES was employed as the baseline method with a parameter β set to 0.06.We reported both the natural accuracy and robust accuracy against FGSM, PGD-20, CW-20 (optimized by PGD), and AA attacks, as shown in Table 1.The highest natural accuracy and FGSM robust accuracy are achieved when τ min is set to 0.01 and τ max to 2. While the maximum PGD-20, CW-20, and AA robust accuracy is obtained when τ min is set to 0.07 and τ max to 1. Therefore, for the following experiments, we choose τ min as 0.07 and τ max as 1.Note that when both τ min and τ max were set to 0.2, it implied a fixed τ of 0.2.In this case, the MA-CAT degraded to a fixed temperature coefficient contrastive learning method.Existing fixed temperature coefficient contrastive learning methods often fix τ at 0.07, but we have encountered NaN loss using this value.This suggests that a fixed τ may not be appropriate in all circumstances and alternative values should be explored.
To better understand the sensitivity of MA-CAT to τ max and τ min , we conducted further analysis by alternately fixing τ min =τ max while observing the changes in robust accuracy as τ max =τ min varies, as shown in Figure 3 and 4. When τ min is fixed, both the natural accuracy and robust accuracy against FGSM attack increase as τ max increases.However, the robust accuracy against PGD-20, CW-20, and AA first increases and then decreases as τ max increases.On the other hand, when τ max is fixed, both the natural accuracy and robust accuracy against FGSM attack decrease as τ min increases.Robust accuracy against PGD-20, CW-20, and AA shows an initial increase and then decreases as τ min increases.The observations suggest that τ does not follow a simple "bigger is better" or "smaller is better" trend, but rather has an optimal range of values, which aligns with our previous analysis in Section 4.2.

Sensitivity to Regularization Parameter β
After determining the values for τ max and τ min , we investigated the impact of the parameter β in Equation ( 6) on the robustness of the model through numerical experiments in this section.We conducted contrast experiments between different β on CIFAR-10 using PreActResNet-18 and WideResNet-34-10 with τ max 1, τ min 0.07.We showed the robustness of all the models in Table 2.With variations in parameter β, a slight fluctuation was observed in both the natural accuracy and the robust accuracy of all models.When based on the PreActResNet-18 network, the natural accuracy and robustness against FGSM attack generally increase as β increases.However, the robust accuracy against AA tends to decrease as β increases.The overall trend for the robust accuracy against PGD-20 and CW-20 attacks is an initial increase followed by a decrease as β increases.Contrasting with the PreActResNet-18, when based on the WideResNet-34-10 network, there is a different trend observed.Except for the robust accuracy against PGD-20, which decreases with increasing β, the natural accuracy and all other robust accuracies generally exhibit an initial increase followed by a decrease as β increases.
As FGSM is a single-step attack with relatively low intensity, whereas PGD-20, CW-20, and AA attacks are iterative attacks that Table 1.The sensitivity of MA-CAT to the upper and lower bounds of τ on CIFAR-10 using PreActResNet-18 with TRADES as the baseline.All the attacks use l ∞ -norm with ε ¼ 8=255.We report both the natural accuracy (%) and robust accuracy (%).The best results are highlighted in bold.better represent the current attack paradigm, this experiment calculates the mean robust accuracy of the models against PGD-20, CW-20, and AA attacks.Subsequent experiments will utilize the β corresponding to the maximum mean accuracy.It can be observed that when using the PreActResNet-18 network, the model achieves the highest mean robust accuracy at β ¼ 0.007.When using the WideResNet-34-10 network, the highest mean robust accuracy is obtained at β ¼ 0.006.Considering both the natural accuracy and the robust accuracy, in the following experiments based on PreActResNet-18, β is set to 0.007, while for experiments using WideResNet-34-10, β is set to 0.006.

Comparison with SOTA Methods
We presented a comparative analysis of our method and state-ofthe-art methods (AT, TRADES, MART, MMA, DyART, as well as the most recent method from Jin et al., [16] which we abbreviate as RAT) on SVHN, CIFAR-10, and CIFAR-100, using PreActResNet-18 and WideResNet-34-10.With the exception of RAT, which uses a batch size of 100 on WideResNet-34-10 (due to its memory consumption exceeding the maximum GPU memory of the GTX 3090 at a batch size of 128), all baselines use the default settings as specified in the original paper.Note that we were only able to obtain the hyperparameters for MMA and DyART on a subset of the datasets used in our study.Therefore, we conducted experiments with MMA and DyART on these subset datasets.We evaluated both the natural accuracy and the robust accuracy against FGSM, PGD-20, CW-20, and AA attacks.The obtained results are shown in Table 3 and 4, providing a comprehensive comparison of the performance of various methods across different datasets and attack scenarios.Table 3 and 4 clearly demonstrate that MA-CAT has achieved state-of-the-art performance.Specifically, in terms of adversarial accuracy against PGD-20, CW-20, and AA attacks, MA-CAT  Table 2.The sensitivity of MA-CAT to regularization parameter β on CIFAR-10 using PreActResNet-18 and WideResNet-34-10.All the attacks use l ∞norm with ε ¼ 8=255.We report both the natural accuracy (%) and robust accuracy (%).Note that the mean column represents the average robust accuracy against PGD-20, CW-20, and AA attacks.The best results are highlighted in bold.outperforms AT, TRADES, MART, MMA, and DyART across all datasets, regardless of whether it is based on PreActResNet-18 or WideResNet-34-10.RAT shows higher adversarial accuracy against the PGD-20 attack, but the robust accuracy of MA-CAT against the more reliable AA attack surpasses it across all datasets and models.Particularly, when using the PreActResNet-18 model on SVHN, MA-CAT demonstrates a robust accuracy 2.63% higher than RAT against the AA attack, as shown in Table 3.It is noteworthy that the improvement in AA robust accuracy suggests that MA-CAT truly enhances the model's robustness, rather than merely obfuscating gradients or relying on inappropriate evaluation methods.Additionally, compared to RAT, MA-CAT can achieve similar robust accuracy against PGD-20 and CW-20 attacks, and even outperform against AA attacks, all with a smaller increase in computational overhead, as shown in Section 5.6.Meanwhile, MA-CAT exhibits a smaller variance against PGD-20, CW-20, and AA attacks than RAT, indicating greater stability in its robustness.In summary, MA-CAT is an effective method for improving the model's adversarial robustness.
It is also worth noting that AT achieves the highest natural accuracy, while TRADES, MART, MMA, DyART, RAT and MA-CAT show a slight reduction in natural accuracy but an improvement in robustness when compared to AT, as demonstrated in both Table 3 and 4.This phenomenon may be attributed to the trade-off between natural accuracy and robustness.Additionally, the robust accuracy against the FGSM attack exhibits a similar variation trend to that of the natural accuracy, as FGSM is a single-step attack with relatively low intensity.

Plug-and-Play Capability and Effectiveness of MA-CAT
In this section, we primarily validated the plug-and-play capability and effectiveness of the MA-CAT, i.e., MA-CAT can be readily incorporated into popular adversarial training methods to further enhance model robustness.We have confirmed the incorporation of MA-CAT into TRADES in Section 5.4.Building upon this, we focus on demonstrating that MA-CAT can also be incorporated into AT.The experiments were performed on the CIFAR-10 using the PreActResNet-18 and the results are presented in Table 5.It can be seen that the incorporation of Table 3.Comparison of MA-CAT with existing popular methods using PreActResNet-18.All the attacks use l ∞ -norm with ε ¼ 8=255 and step size 2=255 for CIFAR-10/100, 1=255 for SVHN.The variance column represents the variance of the robust accuracy against PGD-20, CW-20, and AA attacks.We report both the natural accuracy (%) and robust accuracy (%).Better results are highlighted in bold.
In the case of incorporation with TRADES, both natural and adversarial robustness are improved, resulting in an approximately 3% increase in robustness against CW-20 attacks.These results confirm the plug-and-play capability and effectiveness of MA-CAT.

Computational Efficiency
We investigated the computational efficiency of MA-CAT from two aspects: the time consumption of one epoch and the GPU memory usage during the training phase.The experiments were conducted on CIFAR-10 using PreActResNet-18 on a single GTX 3090 GPU with a batch size of 128.The results are presented in Table 6.Regarding the time consumption, one training epoch of TRADES with MA-CAT takes about 130 s, which is comparable to the original TRADES.However, RAT takes 237 s, almost twice as long as TRADES.In terms of GPU memory usage, TRADES with MA-CAT consumes an additional 900 MB compared to the original TRADES.This increase can be attributed to the inclusion of temporary computational variables and the establishment of individual temperature coefficients for each example.In contrast, RAT consumes 370 MB more GPU memory than MA-CAT.Overall, MA-CAT effectively enhances the adversarial robustness of the model, as it produces notable improvements while maintaining a similar training time and requiring only a small increase in computational and memory resources.

Conclusion and Future Work
In this article, we investigate the adversarial robustness of DNNs through the lens of the optimized example distribution.We propose the MA-CAT method, which aims to increase margins by optimizing the example distribution.To achieve this, MA-CAT employs supervised decoupled contrastive learning to  TRADES [15] 130.84 4033 RAT [16] 237.75 5287 TRADES þ MA-CAT 130.67 4915 shows that different examples can have different levels of difficulty in terms of adversarial training.Decreasing the training strength for misclassified natural examples and increasing it for correctly classified ones during the adversarial training phase have been demonstrated to enhance robustness.Inspired by this, we propose an adaptive temperature coefficient for contrastive learning to adjust the adversarial training strength in an instance-wise manner.The adaptive temperature coefficient updates according to the current model's classification performance on natural examples, offering more fine-grained control over the adversarial training strength for different examples.It amplifies the loss for correctly classified examples, thereby increasing their adversarial training strength and further enhancing the robustness of such examples.Conversely, it diminishes the loss for misclassified examples, thus reducing the adversarial training strength.Once the misclassified examples are correctly classified, the adversarial training strength is increased.This adaptation addresses the challenges posed by different levels of adversarial training difficulty in the training examples and facilitates the model to handle various levels of difficulty and enhance its robustness.

Figure 1 .
Figure 1.Diagram of the decision boundary.a) The decision boundary of natural training.b) The complex decision boundary of standard adversarial training.c) Contrastive learning clusters the same class examples and pushes away different classes of examples.d) The simple contrastive adversarial decision boundary.

4. 2 .
Instance-Wise Misclassification-Aware Adaptive Temperature Coefficient The classification difficulty varies for different natural examples, leading to different levels of difficulty in adversarial training for corresponding adversarial counterparts.It is essential to dynamically adjust the adversarial training strength based on the characteristics of the examples.This article proposes an instance-wise misclassification-aware adaptive temperature coefficient to finely adjust the adversarial training strength.Specifically, the temperature coefficient associated with each example in contrastive learning is dynamically adjusted, based on the current model's classification performance on the natural example.By amplifying or reducing the losses of correctly classified and misclassified examples, the adversarial training strength for each example is dynamically adjusted.

Figure 2 .
During the training phase, examples undergo dynamic shifts between "trustworthy" and "untrustworthy" states.The adversarial training strengths of the "trustworthy" examples progressively increase, eventually leading to a clustering of the same class examples and an increased distance between examples and the decision boundary, thereby enhancing the robustness of the model.

Figure 3 .
Figure 3.The changes in robust accuracy when τ max varies while τ min is fixed.Note that the red dots represent the maximum values of the curves.

Figure 4 .
Figure 4.The changes in robust accuracy when τ min varies while τ max is fixed.Note that the red dots represent the maximum values of the curves.
align and cluster the same class examples in the logit space, thus indirectly increasing the distance between the examples and the decision boundary.Furthermore, MA-CAT adapts the temperature coefficient for each example to address the challenges posed by different levels of adversarial training difficulty in the examples.This adaptation is based on the natural example classification performance of the current model, allowing for an adaptive adjustment of the adversarial strength for each example during the training phase.Extensive experiments have demonstrated the effectiveness of MA-CAT in improving model robustness with minimal computational and memory overhead.There is one limitation of MA-CAT.Inadequate settings of the regularization coefficient can cause unsuccessful model training as MA-CAT is sensitive to it.In the future, we will focus on improving the stability of MA-CAT.

Table 6 .
Comparison of computational efficiency from the time consumption of one training epoch and the GPU memory usage during the training phase.All the models are trained on CIFAR-10 using PreActResNet-18 on a single GTX 3090 GPU with a batch size of 128.The best results are highlighted in bold.Method PreActResNet-18 Training time/epoch (s) GPU memory (MB) Figure2.Diagram of the trustworthy and untrustworthy anchors in contrastive learning.An anchor is considered "trustworthy" if it can be correctly classified by the current model.In this case, other examples of the same class can be "safely" pulled closer to it.However, if the anchor is misclassified by the current model, it is "untrustworthy".This means that the process of pulling the examples toward the anchor is not safe, as some examples may cross the decision boundary during this process.

Table 4 .
Comparison of MA-CAT with existing popular methods using WideResNet-34-10.All the attacks use l ∞ -norm with ε ¼ 8=255 and step size 2=255.We report both the natural accuracy (%) and robust accuracy (%).The variance column represents the variance of the robust accuracy against PGD-20, CW-20, and AA attacks.The best results are highlighted in bold.

Table 5 .
The plug-and-play capability and effectiveness of MA-CAT on CIFAR-10 using PreActResNet-18 with 100 epochs.All the attacks use l ∞ -norm with ε ¼ 8=255 and step size 2=255.We report both the natural accuracy (%) and robust accuracy (%).The best results are highlighted in bold.