Transfer learning from T1 ‐ weighted to T2 ‐ weighted Magnetic resonance sequences for brain image segmentation

Magnetic resonance (MR) imaging is a widely employed medical imaging technique that produces detailed anatomical images of the human body. The segmentation of MR images plays a crucial role in medical image analysis, as it enables accurate diagnosis, treatment planning, and monitoring of various diseases and conditions. Due to the lack of sufficient medical images, it is challenging to achieve an accurate segmentation, especially with the application of deep learning networks. The aim of this work is to study transfer learning from T1 ‐ weighted (T1 ‐ w) to T2 ‐ weighted (T2 ‐ w) MR sequences to enhance bone segmentation with minimal required computation resources. With the use of an excitation ‐ based convolutional neural networks, four transfer learning mechanisms are proposed: transfer learning without fine tuning, open fine tuning, conservative fine tuning, and hybrid transfer learning. Moreover, a multi ‐ parametric segmentation model is proposed using T2 ‐ w MR as an intensity ‐ based augmentation technique. The novelty of this work emerges in the hybrid transfer learning approach that overcomes the overfitting issue and preserves the features of both modalities with minimal computation time and resources. The segmentation results are evaluated using 14 clinical 3D brain MR and CT images. The results reveal that hybrid transfer learning is superior for bone segmentation in terms of performance and computation time with DSCs of 0.5393 � 0.0007. Although T2 ‐ w ‐ based augmentation has no significant impact on the performance of T1 ‐ w MR segmentation, it helps in improving T2 ‐ w MR segmentation and developing a multi ‐ sequences segmentation model.


| INTRODUCTION
The segmentation of magnetic resonance (MR) images is a crucial task to perform various medical applications.Magnetic resonance images are segmented to either extract the required information from the MR modality itself or to perform mapping from another modality to MR modality such as pseudo-CT generation [1] and MR-based attenuation correction for positron emission tomography [2].With this increasing interest in MR modality in the clinical domain, multiple deep convolutional neural networks (CNN) have been successfully applied to perform MR images segmentation [3][4][5][6].However, CNNs have the limitations of heavy computations and the need of powerful computing resources with large memory allocation.Transfer learning is a promising approach to overcome this limitation and particularly with the existence of small number of medical images.
Transfer learning refers to either transferring knowledge or features from one domain to another domain or performing a different task within the same domain.Recently with the wide use of deep learning, transfer learning has been applied extensively by sharing the learnt weights or the set of extracted features of the pretrained model while changing the number of neurons in the output layer depending on the new task.Another view of transfer learning is training the deep model using the data of one domain then testing it with the data of another domain.Another approach is training and testing the deep model with the data of two different domains.The focus of this section is the application of transfer learning with deep neural networks to perform different tasks on medical images.
Despite the disparity between different domain images, the transfer learning from one domain to another has several advantages.It enables the training of deep networks with small datasets, reduces overfitting, and decreases the training time.
Transferring the knowledge from a pretrained network provides a ready set of generic features such as edges, boundaries, and colours which are useful for any image analysis task.
There are different approaches of transfer learning.First, the knowledge can be transferred by only using the pretrained weights to initialise the new model instead of random weights initialisation which tends to cause overfitting with the training data [7].Second, the pretrained model's weights can be finetuned using the new datasets.The fine tuning process requires freezing some layers of the network while retraining others.The fine-tuning process can be limited only to retraining the last fully connected layer to perform the new task or work with the new domain data.
Transfer learning can be applied to perform the same task but using data from a different domain.This type is called domain adaption, such as using datasets from different hospitals, different scanners, and different scanning protocols.On the other hand, transfer learning can convey the knowledge to perform different tasks from different domains.This approach has been widely applied by using the pretrained models with natural images to perform medical image classification and segmentation tasks.
This work aims to overcome the challenge of limited medical images for training and developing a robust CNN for brain segmentation by applying transfer learning and modalitybased augmentation using multi-MR sequences.Transferring the knowledge from T1-w to T2-w MR sequences is studied by investigating the optimal mechanism of fine tuning the pretrained CNN and the required size of the target datasets (T2-w MR) to fine tune the network.Four mechanisms of transfer learning have been studied which are: (1) transfer learning without fine tuning, (2) open fine tuning where CNN's weights are initialised using the pretrained network's weights, (3) conservative fine tuning by retraining some pretrained layers and freezing others, and (4) hybrid transfer learning which is the main contribution that aims at solving the issues of other types of fine tuning and enhances the transfer learning results.The hybrid transfer learning approach consists of two parallel encoders which are fused using an aggregation component.The output of both encoders is fed to a single decoder which consists of the mirrored layers of the encoder to learn the deconvolution.Another contribution is the augmentation approach where the T2-w MR images are considered an intensity based augmentation technique.The objectives of this augmentation are increasing the size of the training dataset and building a segmentation model for multi-MR sequences where both T1-w and T2-w MR images are used to train the CNN from scratch.

| RELATED WORK
In the medical domain, the application of transfer learning to perform the same task with datasets from different domains has been applied extensively to segment the MR images for different diseases.For instance, Kushibar et al. [8] have used brain MR volumes which are acquired with different scanners and protocols to segment sub-cortical brain structures.Transfer learning has been applied by training the model with one dataset then fine tuning the fully connected layers to segment the images of the second dataset.It has been concluded that transfer learning yields better results than the network which is trained from scratch using the same number of training images.It also has been found that the obtained results with less training images using transfer learning are similar to training a network from scratch.
The application of domain adaptation to segment different MR sequences has been investigated to segment left ventricle.For instance, Vesal and his team [9] have used three different cardiac MR sequences: late gadolinium enhancement (LGE), T2-w, and balanced steady-state free precession (bSSFP).Firstly, a network from scratch has been trained using 2 MR sequences (T2-w and bSSFP) then used to initialise another network that aims at segmenting the third MR sequence (LGE) using less number of images.This approach has been implemented on 2D images, yet has not performed well on 3D images.The results have shown the efficiency of domain adaptation with augmentation techniques to improve the segmentation results of the left ventricle.
Ghafoorian et al. [10] have used only 2 MR sequences: T1w and fluid attenuated inversion recovery images.The conducted experiments aim to study the optimal size of the new domain data.Several experiments have been performed to study the relation between the size of the new domain training data and the number of the network's layers that should be retrained to achieve optimal segmentation results.It has been found that with only a few training images of the new domain, the pretrained model with another MR sequence can be finetuned by just retraining the last fully connected layer to achieve an accurate domain adaptation model.
Additionally, Chen et al. [11] have performed the left ventricle segmentation on cine MR images by transferring the knowledge from a pretrained model on a public human cine MR dataset.The results of Dice similarity coefficient (DSC) have shown that the trained model with transfer learning outperforms the trained model from scratch.
Kessler et al. [12] have applied transfer learning from a different domain to carry out knee tissue segmentation using three datasets consisting of different MR sequences acquired by multiple scanners (GE, Siemens, Philips, Toshiba, and Hitachi).It has been concluded that transfer learning helps to improve the segmentation accuracy of a few knee joint tissues of the new domain data while it preserves the model's capability to segment the source domain data.
Zhao et al. [13] have used transfer learning from different MR sequences (T2-w and diffusion-weighted images) from four different medical centres for the detection and segmentation of lymph nodes.Hippocampus segmentation of two different T1-w MR datasets using domain adaptation has been proposed by Ataloglou et al. [14].The deep network is initially trained with one dataset and then fine-tuned using only 15 samples of the second dataset.It has been shown that transfer learning with fine tuning from one dataset to another improves the DSC of hippocampus segmentation.Besides the segmentation tasks, the domain adaptation has been successfully applied to perform MR images synthesis [15], reconstruction [16], and classification [17], On the other hand, transfer learning can be applied to transfer knowledge from models trained to perform different tasks on different domains.Many medical imaging tasks are conducted by transferring the knowledge from pretrained models on natural images such Imagenet dataset which consists of 14 million 2D images.
For instance, Grimm et al. [18] performed segmentation of brain volume and cerebrospinal fluid tissue on simulated T2-w MR images by transferring the knowledge from the pretrained Vsual Geometry Group (VGG) 16 network.An encoderdecoder Segnet architecture is preinitialised with layers and weights from a pretrained VGG 16 model and then fine-tuned by reducing the learning rate.
Brain tumour segmentation using transfer learning has been applied by Cui et al. [19] who have proposed a fully convolutional network which takes four input channels of four different MR sequences named FLAIR, T1-w, T2-w, and contrast enhanced T1-w (T1-c).The initial weights of the network are initialised using the weights of the pretrained model with ImageNet.Yet, the fourth input channel is initialised with the average weight of the three input channels.The model is fine-tuned afterwards using the SGD approach.
Kuzina et al. [20] have proposed a novel transfer learning approach which is based on deep weight prior concept.They have used a U-net-based generative Bayesian prior network which takes the trained convolutional filters with the source dataset to learn the prior on the target dataset.This method has been evaluated to perform segmentation for two different diseases: brain tumour and multiple sclerosis using two different datasets of MR images.It has been revealed that the trained network with the proposed approach outperforms the network that is initialised randomly or from a pretrained network.
The application of transfer learning has become a crucial step to build deep networks, especially with the availability of trained networks with millions of images.The pretrained networks with natural images have been used as a starting point to build new networks to perform various medical tasks.There are many studies that have applied transfer learning to transfer knowledge from MR images which acquired with different scanners or from different institutions.Moreover, transfer learning has been applied for different MR sequences such as transferring the knowledge from T1-w to FLAIR MR sequences.However, the transfer learning from the most common MR sequences such as from T1-w to T2-w has not been investigated yet in the literature.Additionally, to the best of our knowledge, the use of T2-w images as an intensity-based augmentation has not been reported in any literature.

| Data description
Fifty patients underwent T1-w MR and CT scanning and 14 patients out of the 50 underwent T2-w MR scanning.The patient's demographics and clinical characteristics are summarised in Table 1.All patients gave informed consent.
CT scans were acquired with the Biograph mCT and the Biograph 64 True Point Siemens scanners with dimensions of 512 � 512 � 150, and voxel size is 0.97 � 0.97 � 1.5 mm.

| Data preprocessing and labelling
Each volume consists of some black and blurry slices that do not provide any useful information.These slices which are located at the beginning and/or the end of each volume are removed to avoid unnecessary information.Therefore, 48 slices are forming each patient's volume.Moreover, each slice is resized to 256 � 256 matrix by removing the image's background.Another preprocessing step is unifying all volume's resolutions by applying bilinear interpolation to some volumes.Firstly, these matrices are resampled into 300 � 300 � 48 and then resized into 256 � 256 � 48 to obtain the same dimensions for all volumes.Finally, MR images are normalised using the local contrast normalisation approach.
Prior to generating labels, MR and CT images are coregistered to obtain common coordinates due the temporal gap between CT and MR scans acquisition.Afterwards, CT images are used to perform class labelling using intensity-based thresholding by segmenting each brain's pixel into one of the three tissue classes: bone, soft tissue, and air.The Hounsfield value thresholds for each tissue class are as follows: � Bone = Hounsfield value > 600 � Air = Hounsfield value < −500 � Soft tissue = other Hounsfield values

| Methodology
A convolutional excitation-based encoder-decoder network that was first suggested by the authors in refs.[29,30] is adopted in this work.In this network, two types of squeeze and excitation (SE) blocks, channel squeeze and spatial excitation block and spatial squeeze and channel excitation (cSE) block are incorporated in the architecture after each three convolutional layers.The architecture of this network is shown in Figure 1.
The proposed transfer learning methodology consists of four different strategies which are: (1) transfer learning using open fine tuning, (2) transfer learning using conservative fine tuning, (3) Hybrid transfer learning, and (4) multi-modalitybased augmentation.

| Transfer learning using open fine tuning
Fine tuning is the conventional way to conduct transfer learning task.The fine tuning process requires adjusting the weights of pretrained CNN layers.The most common way to fine tune networks is to initialise the new network with the weights of the pretrained network and fine tune all or some number of layers.The required size of the data to fine tune the model is much less than training a network from scratch.The open fine tuning strategy requires to initialise the network with the weights of the pretrained network with T1-w MR images and adjust them by unfreezing all pretrained layers using a small dataset of T2-w MR images.This mechanism uses the pretrained network's weights as initialisation values and updates the weights of the whole network using the target datasets.The main drawback of this network is the necessity of a big enough target dataset that can be used to retrain the network properly.This approach also requires a long training time until the model reaches convergence.Figure 2 illustrates the concept of open fine tuning.

| Transfer learning using conservative fine tuning
The conservative fine tuning is another type of transfer learning which entails using the weights of pretrained network and tuning them on a specific number of the network's layers.For instance, the first few layers of the CNN extract very generic features which can be common for all types of images.These layers should be frozen during the conservative fine tuning process and only the last network's layers can be retrained to adapt the weights to the new domain images.The optimal number of network's layers that should be retrained and the size of the dataset which can be used to conduct the conservative fine tuning process are experimentally determined.Figure 3 shows a general illustration of the conservative fine tuning concept.

| Hybrid transfer learning
The hybrid transfer learning consists of two parallel encoders (dedicated and adapted) which carry the features of both T1-w F I G U R E 1 Illustration of convolutional encoder-decoder architecture with multiple squeeze and excitation blocks.This figure is best viewed in colour, which is available in the online version.and T2-w MR images.The two sets of features are aggregated before passing to a single decoder.The main architecture of the hybrid transfer learning network is demonstrated in Figure 4.

Dedicated encoder
The dedicated encoder is specifically designed to extract T2-w features by training the encoder from scratch using only T2-w MR images as training datasets.The design of the dedicated encoder follows the design of the network's encoder shown in Figure 1 which consists of nine convolutional layers on the encoder path and three convolutional layers on the bottleneck path with four attention blocks placed after each three convolutional layers.

Adapted encoder
The adapted encoder is a pretrained encoder with T1-w images which generate features maps with smaller dimensions than the input image.All layers of this pretrained encoder are frozen to preserve the learnt features from T1-w images.The initially conducted experiments have shown that the segmentation accuracy of soft tissue and air classes of T2-w images using transfer learning without any fine tuning of the parameters is promising.This observation inspired me to adapt the pretrained T1-w encoder without any fine tuning.

Aggregation module
The output of the dedicated encoder (T2-w features) and the adapted encoder (T1-w features) are aggregated using a fusion block.There are different possible mechanisms of aggregation, such as concatenation, multiplication, maximum, and averaging which are tested experimentally.The segmentation results using different aggregation blocks are recorded in the experiments section.The aggregated features are passed to a decoder which retrieves the original size of the input images using nine convolutional layers, three unpooling layers, and three attention blocks.

| Multi-modality based augmentation
The most conventional MR sequences are T1-w and T2-w images.The main difference between these two sequences is the appearance of the tissue which is called tissue intensity.This variation of the intensity values between 2 MR sequences can be considered as an intensity-based augmentation technique.This idea is implemented by training the CNN with both sequences of T1-w and T2-w images.This augmentation technique firstly helps to increase the size of the training datasets and secondly aims at building a CNN which is able to segment multi-MR sequences.

| Model input
The proposed CNN takes a 3D matrix as input with a size of 256 � 256 � 48.This massive number of voxels overwhelms the memory of the graphical processing unit (GPU), causing limitations in network architecture design and hyperparameter selection, such as the number of filters per convolutional layer and batch size.To address this issue, each volume is divided into overlapped patches with dimensions of 128 � 128 � 48 to reduce input size while increasing training dataset size.In addition to that, the use of overlapped patches preserves the spatial contextual information for each volume.

| Training setup
Thirty patients are used for training, 10 patients are used for validation, and 10 patients are used for testing the CNN.The Xavier/Glorot Uniform initialisation scheme is used to initialise the network weights which relies on the number of input and output neurons to determine the scale of initialisation automatically.The network is trained with a batch size of 2 using the Adam optimiser, with an initial learning rate set to 0.0001, and when the training accuracy does not improve for five consecutive epochs, the learning rate is reduced by a factor of 0.75.Momentum decay is set to 0.9, and weight decay is set to 0.0005.The training process ends after 200 epochs or earlier if the training accuracy stops dropping for 10 continuous epochs.
The fine tuning training is conducted using three folds cross validation with a grid search mechanism to find the optimal learning rate.For hybrid transfer learning where the dedicated decoder is trained from scratch using T2-w images, the datasets are split as follows: training: 60%, validation: 20%, and testing: 20%, respectively.The same training setup of T1-w CNN is applied to train the dedicated encoder with T2-w as well as the T2-w based augmentation experiments where the network is trained with both T1-w and T2-w MR sequences using the size of datasets as described in Table 2.All experiments are conducted on Tesla V100 GPU with 16 GB RAM.

| Evaluation metrics
The comparison between the segmented MR and segmented CT (ground truth) images is performed using the following evaluation metrics: where TP stands for true positive, FN stands for false negative, and FP stands for false positive.The segmentation performance is assessed with 95% confidence intervals.

| Open fine tuning
The open fine tuning experiment shows the effect of transfer learning by initialising the weights of the CNN using the weights of the pretrained CNN with T1-w images.The weights are updated by retraining all layers of the CNN with T2-w images.Table 3 shows the comparison between the segmentation results of T2-w images without any fine tuning and three folds cross validation open fine tuning for bone, soft tissue, and air classes, respectively.The low accuracy of bone class segmentation generated by using a pretrained model without fine tuning indicates the inability of the model to segment the bone of two different MR sequences (T1-w and T2-w).On the other hand, the segmentation results of soft tissue and air classes using the transfer learning without fine tuning shows better performance than open fine tuning.The open fine tuning approach requires the update of CNN's weights using a limited number of training dataset of T2-w MR images.
Although the CNN weights are initialised with the pretrained model's weights, the process of open fine tuning needs more T2-w datasets to achieve promising segmentation results of the three brain tissue classes.

| Conservative fine tuning
The study of the performance of the conservative fine tuning approach requires conducting intensive experiments to find the relationship between the required size of the target dataset and the optimal number of CNN layers which need to be finetuned.Figure 5 illustrates the DSC of the three classes' segmentation by conducting several experiments.Each experiment involves fine tuning a different number of convolutional layers starting by retraining the last fully connected layer until the encoder's convolutional layers.
In general, the performance of the bone class segmentation is increased by updating the weights of more convolutional layers.Yet, the segmentation of the air and soft tissue classes are slightly improved by retraining more convolutional layers.Different evaluation metrics using three folds cross validation are shown in Table 4 for bone, soft tissue, and air classes.

Bone
The results show that the segmentation of bone and soft tissue classes is enhanced as more convolutional layers are retrained and the layer's parameters are updated.However, the segmentation of air class requires retraining only six convolutional layers to obtain the highest performance in terms of precision, DSC, and JSC.The highest score of air class sensitivity is achieved by retraining only the last fully connected layer.
The required size of T2-w MR images to train and fine tune the pretrained model is investigated by selecting different sizes of the training dataset and recording the DSC of the segmentation results of each size.The results reveal that the more convolutional layers are fined tuned, the highest DSC is obtained with larger training datasets especially for bone and soft tissue segmentation.In the case of using small training datasets (e.g. two patients only), less convolutional layers are required to be fine-tuned to avoid overfitting.The segmentation performance of fine tuning only the last fully connected layer to segment the three tissue classes is almost constant with different sizes of the training datasets.The highest DSC of the air class is achieved by fine tuning only three convolutional layers with a dataset of eight patients.The conducted experiments and the DSC of different classes reveal that fine tuning the pretrained model with T1-w MR images requires the largest target dataset (T2-w MR) in order to transfer the knowledge properly and avoid overfitting.

| Hybrid transfer learning
The design of the hybrid transfer learning network consists of an aggregation component where several experiments are conducted to choose the best aggregation mechanism.The operations of concatenation, multiplication, averaging, and maximum are applied to integrate the two parallel encoders.The concatenation operation requires more memory allocation to train the network and the batch size of this experiment is set to 1 instead of 2. The experiments of all other aggregation operations are conducted using a batch size of 2.
The three cross validation results of different aggregation mechanisms for the segmentation of bone, soft tissue, and air classes are shown in Table 5.The evaluation metrics of the bone class show that the multiplication of the feature maps of each encoder generates higher segmentation performance.However, the concatenation mechanism provides better segmentation results for the soft tissue and air classes.It is worth mentioning that the precision of the concatenation based results of the bone and air classes is better than the multiplication mechanism.

| Methods comparison
The comparison between the segmentation results using different transfer learning and fine tuning approaches is illustrated in Tables 6 to 8 for bone, soft tissue, and air classes, respectively.
The conservative fine tuning approach which requires retraining 18 convolutional layers using a dataset of 8 patients generates the most precise bone segmentation compared to other approaches.Yet, the hybrid fine tuning with the application of multiplication aggregation operation outperforms the other approaches in terms of sensitivity (recall), DSC, and JSC of the bone class.On the other hand, the segmentation of the air and soft tissue classes using the pretrained model with T1-w MR images is performing better than any fine tuning or transfer learning approach.The main advantage of the application of hybrid transfer learning and conservative fine tuning The Dice similarity coefficient (DSC) of three tissue classes using conservative fine tuning.

8
- to segment the soft tissue and air classes is the significant enhancement of the soft tissue sensitivity and the precision of the air class.
Figure 6 illustrates the segmentation results of three randomly selected 2D slices from the testing datasets of T2-w MR images.The segmentation results of the slice shown in the first row indicate the ability of the pretrained model without any fine tuning (column c) to segment the air cavities more accurately than the hybrid transfer learning (column h).Yet, the other approaches (dg) are not able to segment these regions of air.The visual comparison of the different segmentation approaches also shows that the transfer learning without fine tuning or with open or conservative fine tuning are generating many false positive pixels of the bone class as shown in the second row.The hybrid transfer learning using the multiplication aggregation mechanism is able to segment the bone class more precisely and accurately as illustrated on the slices of the second and third rows.

| Computation time
The computation time is an important factor while applying transfer learning.The conducted experiments show that the increase in the number of layers to be retrained increases the computation time of training and fine tuning the CNN.The T A B L E 4 The evaluation of the segmentation results of the three tissue classes by applying conservative fine tuning using four different evaluation metrics: precision, recall, Dice similarity coefficient (DSC), and Jaccard similarity coefficient (JSC).

| T2-w based augmentation
The segmentation results of the application of intensity-based augmentation by using the T2-w MR sequences as augmented images are illustrated in Table 9 for  - 11  and the ability of the model to preserve the T1-w features after the addition of T2-w images.
Firstly, the model is trained with only T1-w images without the application of augmentation and tested to segment T1-w and T2-w MR images.The evaluation metrics of the three classes show that the model is not able to segment T2-w images properly with low DSC for the bone class (0.3841).
Secondly, the model is trained with 30 T1-w and eight T2w MR images and tested to segment T1-w and T2-w MR images.The difference percentage of the segmentation results of the T2-w bone class is high.This enhancement indicates the ability of the augmentation technique to include the T2-w features which in turn enhances the segmentation results of T2-w images and builds a multimodality segmentation model.On the other hand, the T2-w-based augmentation technique is not affecting the segmentation results of T1-w images.The lack of segmentation improvement indicates that the model is able to preserve the features of T1-w images even after training the model with two different MR sequences.The sensitivity of the soft tissue class remains stable while the precision of the air class is improved by 0.03% after the application of T2-w-based augmentation.
Figure 8 illustrates the visual segmentation results of two randomly selected slices from the testing datasets of T1-w and T2-w MR images.The comparison between the segmentation results of T1-w MR images before and after the application of augmentation does not reveal a significant enhancement for the bone class.However, the segmentation of the air cavities before the augmentation is more accurate.Nevertheless, the segmentation of the T2-w images is highly improved after the T2-w-based augmentation where many false positives of the bone and air classes disappeared.

| CONCLUSION
The application of transfer learning from T1-w to T2-w MR sequences for MR segmentation has been proposed in this work by exploring different mechanisms of domain adaptation.The knowledge from a pretrained model with T1-w MR images is transferred to segment T2-w images using four approaches which are: transfer learning without fine tuning, open fine tuning, conservative fine tuning, and hybrid transfer learning.The mainly proposed approach which is hybrid transfer learning solves the issue of overfitting when fine tuning CNN models with a small dataset.This approach outperforms others in terms of bone segmentation in T2-w MR images when considering both performance and computation resources.The segmentation results of soft tissue and air classes in T2-w MR images using the transfer learning from T1-w MR images without any fine tuning show the best performance with minimal computation time.The experiments of conservative fine tuning reveal that retraining only six convolutional layers is able to produce better segmentation results for air classes with acceptable computation time.On the other hand, the segmentation results of the bone and soft tissue classes reveal better performance with fine tuning 18

T A B L E 9
The segmentation results of T1-w and T2-w three tissue classes by applying T2-w based augmentation using four different evaluation metrics: precision, recall, Dice similarity coefficient (DSC), and Jaccard similarity coefficient (JSC).

F I G U R E 4 6 -
An illustration of the hybrid transfer learning to segment T2-w Magnetic resonance (MR) images.This figure is best viewed in colour, which is available in the online version.MECHETER ET AL. 24682322, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/cit2.12270 by Bibliothèque de l'Université de Gen Division de l'information scientifi, Wiley Online Library on [05/10/2023].See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions)on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

T A B L E 2 3
The size of training, validation, and testing datasets of T1-w and T2-w Magnetic resonance (MR) sequences.The evaluation of the segmentation results of the three tissue classes by applying transfer learning without fine tuning and open fine tuning using four different evaluation metrics: precision, recall, Dice similarity coefficient (DSC), and Jaccard similarity coefficient (JSC).

- 9 conservative
fine tuning experiments show that the fine tuning of 12 or nine convolutional layers requires the same amount of computation time to train one epoch.However, they differ in the number of required epochs to achieve model convergence.The lowest number of epochs is 21 which is recorded by fine tuning nine convolutional layers.The total computation time is the computation time per epoch multiplied by the required number of epochs to train the model until the model reaches convergence.Figure7illustrates the total computation time for different fine tuning approaches.Although hybrid transfer learning requires a longer time to execute one epoch compared to other methods, only a few epochs are needed for the model to reach the convergence state.Overall, the hybrid transfer learning requires less amount of time to complete the training followed by conservative fine tuning by retraining 18 convolutional layers and then open fine tuning.The recorded computation time refers to using eight patients to train the model.
the bone, soft tissue, and air classes.The conducted experiments involve testing the model with T1-w and T2-w MR images to investigate the capability of the model to segment two types of MR sequences F I G U R E 6 The segmentation results of three randomly selected slices from the testing datasets of T2-w Magnetic resonance (MR) images.(a) MR images, (b) CT images which is used as ground truth, (c) the segmentation results using transfer learning without fine tuning, (d) the segmentation results using open fine tuning, (e) the segmentation results using conservative fine tuning (6 Conv), (f) the segmentation results using conservative fine tuning (18 Conv), (g) the segmentation results using hybrid transfer learning (multiplication aggregation), and (h) the segmentation results using hybrid transfer learning (concatenation aggregation).F I G U R E 7 The total computation time of different transfer learning approaches.MECHETER ET AL.