Applications of Knowledge Distillation in Remote Sensing: A Survey

The inclusion and exclusion criteria have been identified by including studies that are directly relevant to KD, particularly in the context of RS, with a focus on research published within the last 5 years to capture the latest advancements. Peer-reviewed articles, conference papers, preprints from reputable platforms and book chapters published in English are prioritized, in addition to to empirical studies, reviews, case studies, and theoretical papers. Studies that do not specifically address KD in RS or focus on unrelated technologies, as well as outdated research published more than 5 years ago unless it is seminal are excluded, as well as and non-peer-reviewed sources such as blog posts, opinion pieces, and non-academic publications.

The search strategy involves using multiple academic databases such as IEEE Xplore, Scopus, Web of Science, Elsevier, Springer Nature, Wiley, Taylor & Francis, MDPI, Google Scholar, etc. to conduct a comprehensive search using relevant keywords like "Knowledge Distillation", "Model Compression", "Model Distillation", "Feature Distillation", "Data Distillation", "Remote Sensing" "Urban Planning", "Precision Agriculture", "Land Cover Classification", etc., refined with Boolean operators (AND, OR, NOT). Initial screening of titles, abstracts, and keywords is performed manually to identify potentially relevant studies. Studies that meet the inclusion criteria are shortlisted for a full-text review, where a detailed evaluation confirms their relevance and quality. Additionally, reference lists of selected studies are screened to identify any further relevant studies that may have been overlooked. Fig. 1 explains the literature screening approach adopted in this study.

KD is a ML technique where a smaller, simpler model (known as the student) is trained to emulate the behavior of a larger, more complex model (known as the teacher) [19]. As shown in Fig. 2, KD relies on two deep neural network models, a more complex one that is called the Teacher and a simpler one that is called the Student. The core idea is to transfer the knowledge from the teacher model, which typically performs better due to its greater capacity, to the student model, which is less resource-intensive [47] and tries to mimic the teacher’s behavior. This can be achieved by aligning the student’s outputs with those of the teacher using a Distillation Loss function that compares the two outputs. Usually, the teacher’s soft target probabilities (the outputs from the softmax layer before applying the final decision function, as depicted in Fig. 2) are used for this purpose. These soft targets provide richer information than hard labels, as they contain insights about the relative probabilities of incorrect answers, giving the student model clues about the underlying data structure and feature relationships that the teacher model has learned [48]. However, apart from this response-based knowledge distillation tactic, the student also can learn the output of intermediate teacher layers, or other representations, making the KD approach very flexible and powerful.

The concept of KD can be traced back to earlier works in model compression and hints training, where simpler models were trained to mimic more complex ones using additional information from those models. However, the term “knowledge distillation” was popularized by Hinton et al. [49] in a seminal 2015 paper, where they demonstrated the effectiveness of using soft targets to train neural networks. Since then, the field has seen rapid development and broad applications across various domains of Artificial Intelligence. Originally, KD was primarily used to reduce the size and computational demands of large neural networks so that they could be deployed on devices with limited hardware capabilities, such as mobile phones and embedded systems. This was particularly valuable for applications that require real-time processing, such as speech recognition and mobile vision [50].

As research progressed, the scope of KD expanded beyond model compression. Researchers began exploring its potential to improve model generalization by smoothing the decision boundaries, making them more stable and improving generalization. Using ensembles of teacher networks to train a student network further stabilizes training by distilling the collective knowledge of multiple models into a single model, and facilitates transfer learning across different domains or tasks [51]. The technique has been adapted and refined to include not just direct output mimicry, but also feature-based and relation-based distillation, where intermediate representations and relationships between data points are also transferred from the teacher to the student [52]. Today, KD is an active area of research with ongoing innovations that aim to further enhance its effectiveness and expand its applicability. This includes cross-modal distillation for transferring knowledge between different types of data, such as video-to-text, and self-distillation, where a model is iteratively trained on its own softened outputs to refine its capabilities [53].

The training of the student network involves minimizing a loss function that typically comprises two terms: the distillation loss and the traditional hard target loss [56].

• •

  Distillation Loss: This loss measures the difference between the softened outputs of the teacher and the student, encouraging the student to mimic the teacher’s generalized behavior. It is often computed using the Kullback-Leibler ($KL$) divergence [57]:

  $$L_{\text{KD}}=T^{2}\cdot KL(\sigma(z^{T},T)\|\sigma(z^{S},T))$$

  (2)

  The factor of $T^{2}$ is used to scale the gradients appropriately, as the gradients produced by the softmax function are scaled by $\frac{1}{T}$ [57].

• •

  Hard Target Loss: This is a standard loss, such as Cross-entropy (CE), used in training neural networks, calculated between the student’s output (at $T=1$) and the true labels [58]:

  $$L_{\text{CE}}=CE(y,\sigma(z^{S},1))$$

  (3)

  where $y$ are the true labels.

The total loss function used to train the student model is a weighted sum of the distillation and hard target losses:

One of the primary benefits of KD is model compression. Traditional DL models often require substantial computational resources due to their depth and complexity, which limits their deployment on devices with restricted hardware capabilities such as mobile phones, IoT devices, and embedded systems. KD addresses this challenge by enabling the training of smaller, lighter models (students) that mimic the behavior of larger, more complex models (teachers). This process involves transferring the intricate knowledge and insights learned by the teacher model into a more compact form within the student model. The student thereby learns to approximate the function of the teacher but with fewer parameters and lower computational demands. This compression not only reduces the size of the model but also lessens the energy consumption and heat production, which are critical factors for battery-powered devices.

Efficiency in model training and inference is another significant advantage of KD. By distilling a cumbersome model into a smaller one, KD effectively reduces the time and computational power needed for training and deploying AI systems. This improved efficiency is particularly beneficial for applications requiring real-time data processing, such as autonomous driving and real-time surveillance. Smaller models also allow for more frequent updates and easier maintenance, which is crucial for systems that need to adapt to changing conditions or data streams.

KD not only compresses the size of the models but often also enhances their performance, especially in smaller models. Typically, smaller neural networks are prone to underfitting and may not capture the complex patterns in large datasets as effectively as their larger counterparts. However, when trained through the KD process, these smaller models inherit refined insights from the teacher models, which include soft probabilities and inter-class relationships that are not visible through traditional hard-label training. This enriched training set helps the student models to perform better than if they were trained independently from scratch. Moreover, the nuanced knowledge transferred includes the handling of edge cases and anomalies, which significantly improves the robustness and generalizability of the student models.

The advantages of KD extend beyond individual model improvements. In educational settings, distillation techniques can democratize access to advanced AI capabilities by enabling more institutions to deploy high-performing AI solutions without the need for expensive infrastructure. Furthermore, in a research context, KD facilitates greater experimental flexibility and faster iteration speeds, accelerating the pace of innovation in AI.

Heterogeneous KD (HKD) is a method of transferring knowledge from a teacher model to a student model where the teacher and student models have significantly different architectures [72]. Traditional KD methods typically assume that the teacher and student models have similar architectures, which allows for straightforward layer-by-layer transfer of knowledge. However, in HKD, the architectures may vary greatly, posing a challenge for direct knowledge transfer.

The study in [73] presents a Generalized KD (GKD) framework for multi-source Earth Observation analysis, specifically for land cover mapping using radar and optical satellite image time series data. This approach tackles data misalignment due to atmospheric conditions or acquisition costs, using radar data consistently and treating optical data as privileged information. This makes it a case of heterogeneous distillation, where different modalities (radar and optical) are involved, requiring the student model to adapt to a less data-rich environment at test time compared to training. The authors in [74] propose using information from deep convolutional networks to guide the training of shallow Grassmannian manifold networks, addressing the need for high-performance yet small-sized networks in resource-limited scenarios. The approach bridges DL with manifold learning, fitting well within the heterogeneous category, as it involves transferring knowledge between fundamentally different architectures. Moving forward, Yang et al. [75] introduce a two-way assistant distillation method for lightweight object detection in RS. This method incorporates compression and multiscale adaptive modules to address feature disparities and background noise, utilizing a heterogeneous distillation approach by applying complex operations from larger models to enhance smaller, simpler ones.

Besides, Nabi et al. [76] propose a compound loss computed on a Transformer-based student and a CNN teacher for single-label scene classification in RS. The use of heterogeneous architectures, where a CNN and a Transformer are involved, leverages the long-range visual capabilities of the Transformer and the inductive biases of the CNN, aiming to enhance classification accuracy in complex scenes. Similarly, the research in [77] involves a teacher-ensemble learning approach using KD in cross-source content-based image retrieval for high-resolution RS images. The method combines source-shared and source-specific classifiers, constructing an effective heterogeneous ensemble of teacher models to transfer useful information to the student model.

Cross-modal KD (CMKD) refers to the process of transferring knowledge from a model trained with superior modalities (e.g., depth maps or point clouds) to another model trained with weaker modalities (e.g., RGB images) [78]. The goal is to improve the performance of the student model trained on the weaker modality by leveraging the knowledge from the teacher model trained on the superior modality. This transfer is achieved by aligning the intermediate feature representations and activation maps between the teacher and student models [78]. In CMKD, the knowledge from the teacher model is used as an additional supervision signal to guide the training of the student model, enhancing its learning process and performance.

Expanding on this concept, Geng et al. [79] propose a topological space network for road extraction, where a denser teacher network focused on topological feature extraction guides a lighter student network. This distillation process transfers knowledge about complex road topology from a heavy network, illustrating a clear case of cross-modal architecture distillation by integrating high-dimensional topological features into a simplified network. Similarly, Xiong et al. [80] introduce a discriminative distillation network for cross-source Content-Based RS Image Retrieval (CBRSIR), addressing the challenge of harmonizing features between multispectral and panchromatic images, thereby further exemplifying cross-modal architecture by handling variations between different types of RS data sources. Additionally, Liu et al. [81] propose a cross-modal KD framework designed to improve multispectral scene classification by transferring knowledge from teacher models pre-trained on RGB images to a student model processing multispectral images. This approach highlights the adaptability of CMKD by addressing the differences between modalities and enhancing the student’s performance, particularly in scenarios with limited samples.

Furthermore, Pande et al. [82] contribute to the field with an adversarial training-driven hallucination architecture for modality distillation in RS image classification, focusing on learning discriminative feature representations from multiple sensor modalities, even in the presence of missing data during the model inference phase. This work aligns closely with cross-modal architectures as it effectively distills features across varying sensor modalities, enhancing model robustness. Lastly, Liu et al. [83] present a universal Super-Resolution-Assisted Learning (SRAL) framework aimed at improving the performance and efficiency of salient object detection in RS images. By incorporating super-resolution techniques into a multitask learning framework, this approach distills domain knowledge from the super-resolution task to significantly boost object detection performance, further showcasing the potential of cross-modal knowledge transfer in enhancing model accuracy and efficiency.

Response-based KD focuses on the knowledge extracted from the final layer of the teacher model. It aims to align the final predictions between the teacher and the student models. The primary goal is outcome-driven learning, which involves distilling the class probability distribution via a softened softmax function, known as ’soft labels.’ This method guides the student model by matching the output distributions of the teacher and student models using various distance functions such as Kullback-Leibler divergence, mean squared error, or Pearson correlation coefficient [84].

The study in [85] introduces a KD framework applied to RS scene classification. By using the high-temperature softmax outputs from a large, deep teacher model to train a smaller, shallow student model, the study showcases how KD can improve the performance of less complex models on multiple public datasets, increasing accuracy significantly even on smaller and unbalanced datasets. This approach directly employs the response-based distillation technique by leveraging the teacher’s softened output probabilities to enhance the student’s learning process. Moving on, Zhao et al. [86] introduces a novel pairwise similarity KD method for reducing the complexity of CNN models in RS image scene classification, maintaining accuracy while using less computational resources. This method focuses on distilling discriminative information between sample pairs.

Feature-based KD addresses the limitation of response-based KD by providing supervision at intermediate layers of the network. This method focuses on transferring intermediate feature representations, such as feature maps, attention mechanisms, activation boundaries, and probability distributions, from the teacher to the student model [84]. The goal is to ensure that the student model learns more meaningful semantic information throughout its hidden layers. The distillation loss in feature-based KD measures the similarity between the transformed intermediate features of the teacher and student models using various distance functions. For example, Chen et al. [87] introduce a new architecture for incremental object detection in RS, which utilizes a Feature Pyramid Network (FPN) to handle objects of various sizes and orientations. Importantly, the study incorporates KD to maintain previously learned information during incremental learning, applying it to outputs from different layers of the FPN. This approach is particularly aligned with feature-based KD, as it focuses on preserving and transferring detailed feature representations across various scales and model iterations, which is essential for detection tasks in dynamically changing datasets. Similarly, to address the efficiency challenge in lightweight object detectors, Yang et al. [88] propose a training method called adaptive reinforcement supervision distillation (ARSD). This method enhances lightweight models by using a multiscale core features imitation (MCFI) module and a strict supervision regression distillation (SSRD) module, improving feature selection and regression accuracy during training. Furthermore, Li et al. [89] introduce a dual KD model that incorporates dual attention and spatial structure modules to enhance the local feature extraction and high-level semantic representation abilities of lightweight CNN models for RS image scene classification. This approach, where knowledge in the teacher network about dual attention and spatial structure is transferred to the student network, directly targets the transfer of intermediate representations. In the same vein, Wang et al. [90] propose a fine-grained object recognition method for high-resolution RS images, which utilizes two stages of KD, emphasizing efficient fine-grained object recognition through feature learning and category correction.

Moreover, Shin et al. [91] focus on transferring detailed spectral feature representations from a teacher model to a student model, allowing the latter to perform complex scene classification tasks traditionally dependent on multispectral data inputs, using simpler RGB inputs. The approach involves intricate feature imitation and retention of critical spectral information, spanning across different data modalities, making it a clear example of feature-based KD. Similarly, Chi et al. [92] propose a self-supervised learning method with KD for hyperspectral image classification, focusing on generating soft labels for unlabeled samples by considering spatial and spectral distances, fitting well under feature-based distillation as it involves creating and transferring complex spectral features. Additionally, Jiang et al. [93] introduce the Deep Distillation Recursive Network (DDRN) for satellite image super-resolution, utilizing ultra-dense residual blocks and a multi-scale purification unit to enhance feature sharing and compensation, particularly focusing on high-frequency components in image super-resolution. Yuan et al. [94] also contribute by proposing a CNN framework for building change detection that uses self-attention KD strategies to refine features for detecting changes in building regions. This approach integrates globally changed information, emphasizing intermediate feature enhancement and integration for improved accuracy in change detection. Finally, Liu et al. [95] introduce ZoomInNet, a cross-scale KD method designed to improve the detection of small objects in drone-based imagery, which often presents complex and dynamic backgrounds. Utilizing a feature pyramid network, the method trains teacher and student networks with differently scaled images to enhance feature harmonization across scales, incorporating layer adaptation, feature level alignment, and an adaptive key distillation point algorithm to refine and distill essential features, showcasing significant advancements in the precision of object detection.

Relation-based KD explores the relationships between different data samples or across different layers within the neural network. Unlike response-based and feature-based KD, which typically handle individual samples, relation-based KD captures cross-sample or cross-layer relationships as meaningful knowledge [84]. This method constructs relational graphs to model dependencies and similarities between instances or layers and uses similarity metrics and distance functions to measure these relationships. The goal is to transfer structured knowledge that encapsulates higher-order dependencies and interactions within the dataset.

Chen et al. [96] develop consistency- and dependence-guided KD methods for object detection in RS images. They introduce modules that focus on extracting and transferring discriminative spatial locations and channels, as well as establishing the consistency and dependence of features between the teacher and student models. This approach utilizes relation-based distillation by focusing on the inter-layer and inter-feature relationships to guide the student model’s learning process. moving on, Li et al. [97] introduce an instance-aware distillation method, which combines feature-based and relation-based distillation techniques. The method enhances the student model’s performance by focusing on instance-related foreground information and constructing relationships between different instances to improve detection accuracy in complex remote-sensing images. Zhao et al. [98] propose a self-supervised KD network (SSKDNet) that uses feature maps of the backbone as supervision signals and transfers the "dark knowledge" through KD. This method focuses on enhancing the discriminative feature extraction capabilities by learning the relationships between different data layers in a self-supervised setting.

Dong et al. [99] present a cross-model KD framework, distilling segmenters from CNNs and transformers, which uses a channel-weighted attention-guided feature distillation and a target–nontarget KD module to guide the student model in learning complex representations and decision boundaries. This study distinctly focuses on relation-based distillation by leveraging the interdependencies of features and classification decisions between different network architectures. On the other hand, Zhou et al. [100] introduce the Multi-level Semantic Transfer Network (MSTNet), a KD framework designed for dense prediction of RS images. This network utilizes a Multi-level Semantic Knowledge Alignment (MSKA) framework to distill semantic information from a complex teacher model to a more compact student model. The MSKA framework emphasizes cross-layer semantic alignment, dynamic semantic aggregation, and softening learning to adaptively transfer knowledge and optimize the learning of semantic information, thus addressing the complexities of deploying models in practical scenarios.

Data distillation refers to techniques that aim to synthesize small, high-fidelity data summaries which capture the most important knowledge from a given dataset [101]. These distilled summaries are optimized to serve as effective substitutes for the original dataset in various data-usage applications such as model training, inference, and architecture search. The goal is to create a concise representation of the data that maintains its critical characteristics, allowing for faster and more efficient model training and evaluation [101].

Building on this concept, Zhang et al. [102] introduce a novel noisy label distillation method within an end-to-end teacher-student framework, which distills knowledge from labels across various noise levels. This approach exemplifies data distillation by effectively utilizing knowledge from noisy data to improve classification performance in RS image scene classification. Extending the application of data distillation, Zhao et al. [86] propose a pair-wise similarity KD method for RS image scene classification. By distilling discriminative information from a cumbersome model to a compact model, this study aims to maintain high accuracy while reducing model complexity, demonstrating another facet of data distillation. Furthermore, Yue et al. [103] contribute to this field with a self-supervised learning method that incorporates adaptive distillation for hyperspectral image classification. Their approach, which focuses on generating adaptive soft labels based on spatial-spectral similarity, underscores the importance of utilizing extensive unlabeled data in the data distillation process.

Model distillation refers to the process of replacing a complex ML model with a simpler model that approximates the original model’s performance [104]. This technique is used to improve computational efficiency by distilling large or ensemble models into smaller, more manageable models that maintain similar accuracy. The primary goal is to reduce the computational cost associated with deploying large models while preserving their predictive capabilities [104]. Model distillation also aids in model interpretability by converting “black-box” models, such as neural networks, into more transparent forms.

In the context of model distillation for RS applications, a variety of approaches have been developed to enhance the performance and efficiency of lightweight models. Zhang et al. [105] introduce a dynamic knowledge distillation (KD) framework that enables CNN models to be lightweight while maintaining high detection accuracy, with an emphasis on selective learning through a dynamic instance selection distillation module. Building on the concept of model distillation, Yang et al. [106] develop a lightweight semantic segmentation network that combines KD with a multiscale pyramidal pooling module and attention mechanisms, resulting in a pruned model that retains high accuracy. Similarly, Wang et al. [107] propose a change detection method that integrates prototypical contrastive distillation and channel-spatial-normalized distillation, allowing the student model to learn complex feature distributions from the teacher, thereby fitting into the model distillation framework.

Further advancing the field, Chen et al. [108] propose a multi-teacher collaborative distillation approach that uses adaptive weight and feature knowledge exchange to enhance the robustness of student models, while Gu et al. [109] introduce a Context-aware Dense Feature Distillation (CDFD) strategy for CubeSat-based RS object detection, integrating multiple teacher networks to optimize a lightweight detector. Chai et al. [110] contribute to the model distillation category with their Bidirectional Self-Attention Distillation (Bi-SAD) approach, aimed at enhancing cloud detection models by enabling compact models to learn detailed textural and semantic information.

Addressing the challenge of few-shot learning, Liu et al. [111] present a ranking-preserving KD method that improves the generalization capabilities of student models in RS scene classification. Similarly, Wang et al. [112] explore the enhancement of lightweight models through a Phase-shift encoded KD method (PseKD) that improves object orientation prediction. In a broader application, Chen et al. [113] propose a semi-supervised KD framework for global-scale urban object mapping, emphasizing the handling of urban diversity and large-scale sample growth.

Complementing these efforts, Zhao et al. [114] propose a weakly correlated distillation learning framework for RS object recognition with limited samples, leveraging large-scale natural image datasets to enhance small-scale RS datasets. Lin et al. [115] address the issue of denoising by presenting a lightweight model that uses KD to efficiently extract spatial and spectral features while maintaining computational efficiency. Yu et al. [116] focus on incremental learning, introducing a dual KD method to mitigate catastrophic forgetting, which aligns with the incremental learning approach proposed by Xu et al. [117] and Xu et al. [118], who use KD to enhance multimodal learning and hyperspectral image classification, respectively.

Lastly, Zhou et al. [119] introduce a graph semantic guided network (GSGNet) for optical RS scene analysis, utilizing knowledge refinement to maintain high inference speed and contextual inference capability. Zhao et al. [120] propose a target detection model distillation framework that uses feature transition and label registration to improve the learning ability of lightweight networks in RS imagery, further contributing to the body of work on model distillation.

Feature distillation refers to a method in which the student network learns to mimic the hidden feature values of a teacher network [121]. This process involves transferring the intermediate representations (features) learned by the teacher network to the student network. Unlike traditional KD that focuses on the output probabilities (logits), feature distillation emphasizes the transfer of internal activations or feature maps. The primary goal is to improve the student network’s performance by leveraging the knowledge encapsulated in the teacher’s feature representations [121].

Building upon this concept, Zhou et al. [122] propose a lightweight student network framework for semantic segmentation of high-resolution RS images. By employing a graph attention guidance network, they distill knowledge from a large teacher network to optimize image features, thereby enhancing segmentation accuracy. This method aligns with feature distillation, where the objective is to boost the student’s feature representation capabilities to closely match those of the teacher. Similarly, Zhang et al. [123] introduce a few-shot classification method for RS scene classification, which also falls under the feature distillation category. This approach utilizes a novel two-branch network and incorporates self-KD during training to generate powerful representations, prevent overfitting, and enhance overall performance.

In parallel, Hu et al. [124] contribute to the field with a variational self-distillation network designed for RS scene classification. This method hierarchically distills class entanglement information from deep to shallow layers, further illustrating the application of feature distillation by refining and transferring feature information across different network layers. Expanding on these ideas, Xing et al. [125] present a collaborative consistent KD method aimed at improving classification accuracy for RS image scenes on embedded devices. Their approach emphasizes feature distillation across multiple network branches, focusing on reducing parameter redundancy and enhancing model efficiency, thus reinforcing the relevance of feature distillation in RS applications.

Layer-to-layer distillation refers to the process where the teacher model’s intermediate layers directly guide the corresponding layers of the student model. This method ensures that the student model learns similar feature representations as the teacher model at different stages of its depth [126, 127].

Direct Mapping: In this approach, each layer of the teacher model is aligned with the corresponding layer in the student model. The outputs of each intermediate layer in the teacher model are used as targets for the corresponding layer in the student model. This direct mapping can help the student model learn hierarchical features similar to those learned by the teacher model [128].

Feature Representation: By mimicking the intermediate representations of the teacher, the student model can capture complex features and patterns, which might be difficult to learn solely from the final output. This method is particularly useful when the student model has a similar or reduced architecture compared to the teacher.

Loss Function: Often, additional loss terms are introduced to minimize the difference between the teacher’s and student’s intermediate layer outputs. This can include mean squared error (MSE) or other similarity measures.

Suppose a deep CNN is used as the teacher model with layers: $T_{1},T_{2},T_{3},\ldots,T_{n}$. The student model has corresponding layers: $S_{1},S_{2},S_{3},\ldots,S_{n}$. During training, the output of $T_{1}$ will guide $S_{1}$, $T_{2}$ will guide $S_{2}$, and so on, ensuring that each student layer learns to mimic the feature maps of the corresponding teacher layer.

Cross-layer distillation refers to the process where the teacher and student models do not have a direct correspondence between layers. Instead, the knowledge transfer happens between non-matching layers, for example, higher layers of the teacher model guiding lower layers of the student model or vice versa.

Non-Matching Layers: In this approach, there is no strict one-to-one correspondence between the layers of the teacher and the student. The knowledge from higher (more abstract) layers of the teacher model can be distilled into lower (more detailed) layers of the student model, allowing for flexible guidance. Chen et al. [129] propose Semantic Calibration for Cross-layer Knowledge Distillation (SemCKD), which automatically assigns target layers from a teacher model to each student layer using an attention mechanism. This method allows student layers to distill knowledge from multiple teacher layers rather than following a fixed, one-to-one correspondence. Building on this concept, Wang et al. [130] further refine the idea of non-matching layers by using a learned attention distribution to assign appropriate teacher layers to student layers, thereby enhancing cross-layer supervision and subsequently improving student model performance. In addition, Nath et al. [131] introduce Robust Neural Architecture Search by Cross-Layer Knowledge Distillation (RNAS-CL), which searches for the best teacher layer to supervise each student layer, thus allowing for non-matching layer associations that enhance robustness in neural architectures. Furthermore, Zhao et al. [132] develop Cross-Architecture Knowledge Distillation (CAKD), where non-matching layers are utilized to transfer knowledge from a Transformer-based teacher model to a CNN-based student model, involving the alignment of pixel-wise spatial information across different architectures and expanding the applicability of non-matching layers in cross-architecture scenarios.

Layer Interaction: This method leverages the hierarchical nature of neural networks, where different layers capture different levels of abstraction. By using high-level features from the teacher to guide the student’s learning process, the student can gain a richer understanding of the data. Yao et al. [133] propose Dense Cross-layer Mutual-distillation (DCM), which involves layer interaction by integrating auxiliary classifiers and bidirectional knowledge distillation operations across different layers of the teacher and student models, thereby enhancing knowledge representation and performance. Building on this concept, Su et al. [134] present Deep Cross-layer Collaborative Learning (DCCL), focusing on layer interaction through intermediate cross-layer supervision among peer student models, which integrates features from different layers to enhance representation and learning outcomes. Similarly, Zhu et al. [135] introduce Cross-layer Fusion for Knowledge Distillation (CFKD), which aggregates features from both teacher and student models, allowing for rich layer interactions that further enhance the student model’s learning process. In a related effort, Hu et al. [136] propose an online knowledge distillation method with layer-level feature fusion modules that connect sub-networks, thereby facilitating mutual learning through enhanced layer interaction among student networks. Expanding on the concept, Nguyen et al. [137] develop CLAFusion, a framework that employs cross-layer alignment for fusing neural networks with different numbers of layers, leveraging layer interaction to improve model accuracy and efficiency. Finally, Zhang et al. [138] propose Patch Aware Knowledge Distillation (PAKD), which emphasizes cross-layer patch alignment and interaction within and across instances, guiding the student’s learning of multi-level information and further reinforcing the importance of layer interaction in knowledge distillation.

Hierarchical Guidance: Cross-layer distillation can help in scenarios where the student model is significantly smaller or has a different architecture compared to the teacher. It allows the student to learn abstract representations earlier in its layers. Imagine a teacher model with layers: $T_{1},T_{2},T_{3},\ldots,T_{n}$, and a student model with layers: $S_{1},S_{2},S_{3},\ldots,S_{m}$. In cross-layer distillation, $T_{n}$ (the final layer of the teacher) might guide $S_{3}$ (a middle layer of the student), $T_{3}$ might guide $S_{1}$, and so on, depending on the distillation strategy and the specific architecture of the models. In this regard, Zou et al. [139] develop CoCo DistillNet, which utilizes cross-layer correlations to guide the student model in learning abstract representations from a teacher model in the context of pathological image segmentation, thereby enhancing the student model’s performance in resource-constrained environments. Building on this concept, Zou et al. [140] propose Graph Flow Distillation, a method that transfers cross-layer variations from a large teacher network to a compact student network in medical image segmentation, enabling the student model to learn from both high-level and low-level abstractions of the teacher. In a similar vein, Zhai et al. [141] introduce a method that uses the deepest feature maps from the teacher to guide the shallow layers of the student model, providing hierarchical guidance that effectively balances performance and efficiency. Furthermore, Guo et al. [142] propose Alignahead++, an online knowledge distillation framework for GNNs that transfers structure and feature information across layers, facilitating hierarchical guidance and significantly improving performance on edge devices. Together, these studies underscore the importance of hierarchical guidance in enhancing the efficiency and effectiveness of knowledge distillation across various architectures.

In the context of the classification of RS images/scenes, KD can be particularly beneficial. High-resolution satellite or hyperspectral images, which are rich in spatial and spectral information, can be computationally intensive to process online using large models. By employing KD, a large, powerful model (teacher) that has been trained on such images can pass on its learned representations and decision-making capabilities to a smaller, more efficient model (student). This allows the student model to achieve high classification accuracy while significantly reducing computational and storage requirements. Techniques such as spatial feature blurring can be incorporated to enhance the student’s learning by making the training data more challenging, which helps in better generalization and improved classification performance. Various studies have been proposed in the literature to enhance RS image classification, focusing on KD, model efficiency, feature extraction, and handling noisy or incomplete data. Table 3 provides the main features of these works. Building on this, Xu et al. [118] propose a hyperspectral image classification method based on class-incremental learning to learn new land-cover types without forgetting the old ones. This method uses a KD strategy to recall information of old classes and a channel attention mechanism to effectively utilize spatial-spectral information, demonstrating high accuracy on three hyperspectral image datasets. Similarly, Chi et al. [92] introduce a self-supervised learning method with KD for HSI classification, termed SSKD, which generates soft labels for unlabeled samples by considering spatial and spectral distances. This method significantly improves classification accuracy on three public HSI datasets. In addition, Xing et al. [125] address the challenge of using large deep neural networks on embedded devices by proposing a collaborative consistent KD (CKD) method. This method reduces the number of redundant parameters and improves the classification accuracy when tested on the SIRI-WHU and NWPU-RESISC45 datasets. Furthermore, Chen et al. [85] focus on scene classification using a KD framework to improve the performance of smaller and shallower network models. Their method increases the overall accuracy when tested on AID, UCMerced, NWPU-RESISC, and EuroSAT datasets. Along similar lines, Song et al. [143] present ERKT-Net, an efficient and robust knowledge transfer network designed for lightweight yet accurate CNN classifiers, demonstrating superior accuracy and compactness on three RSI datasets. Likewise, Wu et al. [1] propose the TAKD method, which reduces background disturbance and improves the accuracy of student models for RS scene classification on three benchmark datasets. Moreover, Ienco et al. [73] propose a Generalized KD (GKD) framework to manage information misalignment between training and test data, demonstrating improved classification results using radar and optical satellite image time series data. Similarly, Zhang et al. [102] address the challenge of noisy labels in RS image scene classification by proposing a noisy label distillation (NLD) method, which effectively distills knowledge from labels across a range of noise levels, achieving high accuracy on UC Merced Land-use, NWPU-RESISC45, and AID datasets.

In another approach, Zhao et al. [98] propose a self-supervised KD network (SSKDNet) that uses feature maps as supervision signals and dynamically fuses feature maps to extract discriminating features, showing excellent performance on three datasets. Furthermore, Yang et al. [75] introduce the TWA distillation method for RS object detection, reducing background information and addressing feature disparities, achieving superior performance on the LEVIR and SAR SSDD datasets. Additionally, Pande et al. [82] tackle the problem of missing modalities in RS image classification by proposing an adversarial training-driven hallucination architecture. This method shows that the student model can surpass the teacher model’s performance on HSI datasets. In a similar vein, Yu et al. [116] propose a two-stage training method for incremental learning that includes dual KD to prevent catastrophic forgetting, improving accuracy on CIFAR100 and RESISC45 datasets. Finally, Xie et al. [2] introduce an improved decoupled KD (DKD) strategy for HSI classification using a spatial feature blurring (SFB) module, achieving high overall accuracy on the Salinas dataset.

Moving forward, Zhang et al. [123] present RS-SSKD for few-shot RS scene classification, which uses Class Activation Maps (CAMs) and self-KD to generate powerful representations, achieving high accuracy on NWPU-RESISC45 and RSD46-WHU datasets. As the availability of airborne and satellite imagery increases, the challenge in RS (RS) scene classification has shifted from data scarcity to the lack of ground truth samples. Addressing these challenges, especially in unfamiliar environments with limited training data, few-shot classification offers a promising solution within meta-learning by extracting rich knowledge from minimal data. In [123], the authors introduce RS-SSKD, a method designed for few-shot RS scene classification that focuses on generating robust representations for downstream meta-learners. This approach features a two-branch network that uses three pairs of original-transformed images and incorporates Class Activation Maps (CAMs) to focus on the most relevant category-specific regions, ensuring the creation of discriminative embeddings. Additionally, a self-KD is applied to prevent overfitting and enhance performance (see Fig. 7).

In RS, object detection is crucial for identifying specific features such as buildings, vehicles, and vegetation. KD helps in creating lightweight models that maintain high accuracy, making it feasible to run these models on devices with limited computational power. Several studies focus on the use of KD for improving object detection in RS images, each introducing innovative strategies to address specific challenges. Algorithm 1 outlines a process for KD in RS object detection. It starts by training a teacher model on a dataset, and then defines a student model with a simpler architecture. The teacher model generates soft targets, which are probability distributions over classes, using a softened softmax function. The student model is trained using a combined loss function that includes the cross-entropy loss and the Kullback-Leibler divergence between the teacher’s and student’s outputs. The process iterates over several epochs, optimizing the student model to mimic the teacher while also learning from the original labels. Finally, the trained student model is deployed. The main works on the use of KD in the object detection task in RS images and their main features are summarized in Table 4.

For instance, Yang et al. [88] propose an adaptive reinforcement supervision distillation (ARSD) framework to enhance lightweight object detectors. This method focuses on multiscale core features imitation and strict supervision regression distillation to improve performance, especially for small objects in complex backgrounds. Zhang et al. [105] introduce a dynamic KD (DKD) framework, leveraging dynamic global distillation and instance selection distillation to enhance object detection in cluttered scenes. Another study by Zhang et al. [144] presents Orientation Distillation (OD) to address issues with boundary discontinuity and spatial feature ossification for detecting arbitrary-oriented objects in RS images, the authors further propose an adaptive composite feature generation (ACFG) strategy to improve feature mapping and handling of foreground and background loss in object detection [145].

Feng et al. [146] introduce an Instance-aware Distillation approach for Class-incremental Object Detection (IDCOD), which helps in preserving old class knowledge while learning new classes, thus mitigating catastrophic forgetting. Chen et al. [3] propose Discretized Position KD (DPKD), which focuses on transferring high-quality bounding box position and pose information to improve object detection performance. Pang et al. [4] present a pyramid KD (PKD) framework to handle the limitations of model compression, utilizing a hybrid online–offline smooth distillation strategy to enhance recognition accuracy while avoiding knowledge explosion and offset.

Du et al. [55] add a detection head specifically for small targets in the YOLOv5 model, proposing a network KD framework for improved small-scale target detection in RS images. Gao et al. [147] design a feature super-resolution fusion framework using cross-scale distillation to improve the detection accuracy of small objects by enhancing feature expression capability. Yang et al. [148] propose a weakly supervised object detection method using self-attention distillation and instance-aware mining to handle varying scales and dense object proximity in RS images.

Other studies address different aspects of RS. Zhang et al. [53] combine detection and tracking in a joint framework (OKD-JDT), using KD to improve tracking efficiency and accuracy. Sun et al. [149] develop the Efficient Multidimensional Global Feature Adaptive Fusion Network (MGFAFNET) for UAV platforms, incorporating a dual-branch multidimensional aggregation backbone and a localized compensation dual-mask distillation strategy to balance detection speed and accuracy. Yang et al. [150] introduce DC-KD, a distillation scheme for object detection in satellite images, addressing data distribution differences between aerial and satellite images. Song et al. [151] present HMKD-Net, a hybrid-model KD approach combining CNNs and vision transformers to enhance classification performance in RS images. Zhang et al. [152] propose a visual knowledge-oriented approach using pseudo labels to improve object detection in complex and dense RS images.

Lian et al. [153] introduce a multitask learning framework combining image translation and saliency detection networks with KD to enhance feature expressiveness and reduce model complexity. Zeng et al. [154] propose TDKD-Net, a tensor decomposition and KD-based network for UAV detection, focusing on small object detection and handling imbalanced issues. Wan et al. [155] develop a coarse-to-fine detection method integrating density-aware scale adaptation and KD to improve small object detection in UAV images. Jia et al. [156] suggest a multi-scale self-distillation approach to improve small target detection accuracy without using a teacher model. Lin et al. [157] propose DTCNet, a distillation Transform-CNN network for super-resolution reconstruction in RS images, enhancing reconstruction quality while maintaining a smaller parameter count. Tang et al. [158] introduce a text-guided tail-class generation network (TGN) to address long-tailed data distribution in RS datasets, improving tail-class accuracy by generating diverse and consistent tail-class images.

On the other hand, traditional KD-based object detection methods have limitations, such as ignoring crucial background information and focusing solely on global context. To overcome these issues, Attention-based Feature Distillation (AFD) is proposed in [159], which distills both local and global information. AFD enhances local distillation with a multi-instance attention mechanism and reconstructs pixel relationships, resulting in state-of-the-art performance in object detection while remaining efficient. Fig. 8 illustrates the architecture of the proposed Attention-based Feature Distillation (AFD) method. This framework improves upon traditional KD-based object detection by incorporating both local and global information from the teacher network. The multi-instance attention mechanism within AFD allows the model to distinguish between background and foreground elements effectively. Additionally, the method reconstructs pixel relationships, ensuring that both local details and broader context are accurately transferred from the teacher to the student detector, resulting in enhanced detection performance.

KD is beneficial for semantic segmentation in RS applications, which involves classifying each pixel in an image into predefined categories. The teacher model is first trained on the segmentation task using high-resolution RS data [160]. Due to its complexity and larger capacity, it can learn intricate patterns and detailed features from the data. Once trained, the teacher model’s predictions, along with its internal representations, are used to guide the training of the student model. The student model, being smaller and more efficient, aims to mimic the performance of the teacher model while maintaining lower computational costs and faster inference times [161]. Moreover, KD is particularly advantageous because it allows for the deployment of effective semantic segmentation models on edge devices or in scenarios with limited computational resources [162]. By leveraging the distilled knowledge from the teacher model, the student model can achieve high segmentation accuracy despite its reduced size. This is crucial for applications such as real-time environmental monitoring, disaster response, and agricultural analysis, where timely and accurate segmentation of satellite or aerial imagery is needed. The distillation process also helps the student model generalize better to new and unseen data, enhancing its robustness and reliability in diverse RS tasks [163].

The studies on semantic segmentation in RS show significant advancements but also face several limitations. For instance, Gao et al. [164] introduced the FoMA framework, which significantly improves segmentation performance by leveraging foundation models, but it struggles with data scarcity in novel classes and balancing segmentation performance across classes. Similarly, Zhou et al. [122] proposed a lightweight student network (GAGNet-S*) with KD that achieves excellent segmentation performance but faces challenges related to scalability and complexity in deployment on resource-limited equipment. Dong et al. [99] addressed the limitations of CNNs and transformers by proposing the DSCT framework, which enhances segmentation performance through cross-model KD. However, this approach requires high computational complexity and massive data resources.

Studies focusing on KD methods, such as the MGSAD by Zhang et al. [165], and MTKD by Li et al. [166] with MTKD, contribute innovative techniques but encounter challenges such as the need for extensive computation and handling of domain shifts. Liu et al. [167] proposed a three-stage UDA method that shows better performance but relies heavily on large-scale annotated data and struggles with domain shift handling. Similarly, Shi et al. [168] introduced DSANet, which effectively handles spatial and semantic feature enhancement but reduces the model’s characterization ability for these features.

Incremental learning and domain adaptation are other areas where significant contributions have been made but also face limitations. Rong et al. [169] proposed a generalized framework for CSS but struggled with the challenge of old classes collapsing into the background. Rui et al. [170] and Le et al. [171] focused on incremental learning methods but faced high computational costs and the complexity of adapting to incremental domains and partial multi-task learning, respectively. Shan et al. [172, 173] developed class-incremental segmentation methods that address catastrophic forgetting but require balancing old and new class learning and managing feature generation complexity. Li [174] proposed DSSN with weakly-supervised constraints to handle cross-domain segmentation, but the method heavily depends on labeled data and struggles with geographic variation.

Lastly, Guo et al. [175] and Cao et al. [176] proposed methods to balance effectiveness and compactness in segmentation models, but they face high computational demands and challenges in handling noise and redundant features. Zhou et al. [119] introduced GSGNet with high inference speed but had to balance this with contextual reasoning capabilities. Bai et al. [177] and Wang et al. [178] focused on domain adaptation, but they faced difficulties in aligning high-dimensional image representations and managing intermediate domain learning. Michieli et al. [179] addressed incremental learning with various KD techniques but struggled with catastrophic forgetting and internal feature representation complexity. Lastly, Pena et al. [180] introduced DeepAqua for water detection, which improves segmentation accuracy but lacks specific details on datasets and segmentation scenarios. Table 5 provides a summary of the works that use KD for semantic segmentation of RS images.

Besides, [164] produces a Foundation Model Assisted (FoMA) for Generalized Few-Shot Semantic Segmentation (GFSS) in RS images, aimed at improving segmentation performance under data scarcity conditions. FoMA leverages foundation models through three strategies: Support Label Enrichment (SLE) to enhance support labels, Distillation of General Knowledge (DGK) to transfer generalizable knowledge, and Voting Fusion of Experts (VFE) to combine zero-shot and few-shot predictions. The method demonstrates state-of-the-art performance on the OpenEarthMap few-shot challenge dataset. Fig. 9 illustrates the architecture of the FoMA framework, which effectively integrates a vision-language foundation model’s general knowledge into the GFSS task for RS images.

KD improves the classification of land cover types by refining the feature extraction capabilities of student models. This leads to better segmentation and classification of different land cover types, essential for environmental monitoring and urban planning. Several studies have proposed innovative methods to improve land cover classification and other RS tasks using KD and multimodal data fusion. For example, Xu et al. [117] developed a two-branch patch-based CNN with an encoder-decoder (ED) module to fuse multimodal RS (RS) data. They introduced a KD in model (DIM) module for better multimodal data fusion and a cross-model (DCM) module to enhance single-modal classification using multimodal knowledge. Their approach demonstrated superior performance on hyperspectral (HS) and light detection and ranging (LiDAR) data as well as HS and synthetic aperture radar (SAR) data. Fig. 10 depicts the approach proposed in [117]. Wang et al. [182] proposed the cross-modal graph knowledge representation and distillation learning (CGKR-DL) framework, which combines CNN and graph convolutional network (GCN) to enhance land cover classification. Their method addresses the limitations of traditional CNN-based cross-modal distillation methods and significantly improves performance on various multimodal RS datasets.

The Generalized KD (GKD) framework has been introduced by Ienco et al. [73] to handle data misalignment between training and test phases. Their method, applied to radar and optical satellite image time series data, improved land use land cover mapping, especially for agricultural classes. A multimodal online KD (MMOKD) framework that supports both multimodal and cross-modal learning, showing superior performance in both scenarios has been proposed by Liu et al. [11] for land use/cover classification using optical and SAR images. Finally, Li et al. [183] introduced the dynamic-hierarchical attention distillation network (DH-ADNet) with multimodal synergetic instance selection (MSIS) for land cover classification using missing data modalities. Their method emphasizes selective instance enhancement and hierarchical attention distillation, achieving state-of-the-art results.

Several other studies focused on specific RS challenges. For instance, Lu et al. [54] developed a weakly supervised change detection technique via KD and Multiscale Sigmoid Inference (KD-MSI), significantly improving change detection performance on multiple datasets. Similarly, Zhang et al. [184] proposed a transfer learning framework using teacher-student structure for better generalizability and performance in land cover classification. Kanagavelu et al. [185] and Gbodjo et al. [186] explored federated learning and multisensor data integration, respectively, to enhance land cover mapping and monitoring. The former work federated UNet model integrated KD to reduce communication costs while maintaining high accuracy whereas the later developed a self-distillation strategy within a CNN framework to combine multitemporal SAR and optical data for improved land cover classification. The works that use KD for the classification of land cover and their main characteristics are summarized in Table 6.

Besides, significant research has been devoted to enhancing land cover classification using multimodal RS data, which significantly outperforms single-modal methods due to its richer information content. To advance this field, a two-branch, patch-based CNN with an encoder-decoder (ED) module for effective multimodal data fusion is proposed in [117]. Typically, a KD in model (DIM) module to guide per-modality encoder learning is introduced, ensuring more efficient fusion. Additionally, we explored guiding single-modal learning with multimodal information through the KD cross-model (DCM) module. This approach treats the multimodal method as a teacher, transferring its knowledge to single-modal methods. Extensive experiments on the Houston2013 and Berlin datasets, combining hyperspectral (HS) with LiDAR and synthetic aperture radar (SAR) data, respectively, demonstrated the superiority of our multimodal fusion strategy over state-of-the-art methods. The DCM module also significantly enhances LC classification performance for single-modal methods.

KD is crucial in smart agriculture as it allows for the development of lightweight models that maintain high accuracy while being deployable on resource-constrained edge devices, such as drones or sensors. This is particularly important for precision agriculture tasks like early weed detection and crop monitoring, where efficient and accurate models are needed for real-time decision-making. By transferring knowledge from larger, more complex models to smaller ones, KD helps optimize these tasks, enhancing agricultural productivity and sustainability. Numerous studies have investigated the application of KD in precision agriculture. These efforts focus on enhancing model efficiency and accuracy in tasks such as crop monitoring, weed detection, and resource management. For instance, Liangde et al. [199] develop a model distillation approach to enhance agricultural named entity recognition, leveraging a BERT-based model enhanced by BiLSTM and CRF for precise entity detection from a constructed agriculture knowledge graph. Ghofrani and Mahdian Toroghi [200] focus on plant disease detection, using a KD approach to enable smaller CNN architectures, like MobileNet, to achieve near high-end model accuracy on the Plantvillage dataset. On the same line, Hu et al. [201] and Dong et al. [202] address crop disease detection. The former approach optimizes YOLOv5s for maize disease detection, while the latter uses ECA-KDNet for efficient apple disease diagnosis on mobile devices. Finally, Huang et al. [203] develop multistage KD to create lightweight models for diagnosing multiple crop diseases effectively.

In the task of image segmentation, Angarano et al. [204] introduce a method for robust crop segmentation using KD, aimed at improving the generalization across different environmental conditions for robotic field management. Similarly, Li et al. [205] employ KD for efficient panoptic segmentation, creating lightweight networks capable of detailed scene understanding at high speeds, whereas Jung et al. [206] improve plant leaf segmentation using KD to maintain high-quality instance segmentation. The work of Pagé-Fortin [207] investigates class-incremental learning methods to address the challenge of learning new plant species and diseases incrementally, focusing on mitigating catastrophic forgetting.

In the domain of precise image analysis and preventive detection in vineyards, Wang et al. [208] propose a lightweight semantic segmentation model for identifying grape picking points, enhancing the picking efficiency in vineyard environments, whereas Hollard and Mohimont [209] apply KD to enhance grapevine detection for early yield prediction, focusing on lightweight model deployment for embedded devices. As far as it concern disease detection, Musa et al. [210] propose a low-power DL model for detecting plant diseases in hydroponic systems, aiming at efficiency and reduced resource consumption and Zhang and Wang [211] improve plant leaf disease recognition using a novel data augmentation-based KD framework, enhancing recognition accuracy in natural environments.

More advanced and complex tasks have also been addressed using KD. In the aquaculture domain, Yin et al. [212] propose a novel fish individual recognition method using KD within a vision transformer framework, improving accuracy. In the same application domain, Li et al. [213] focus on underwater fish species classification using a novel two-tier KD method to enhance model accuracy and reduce computational demands. Back to the plants and trees images, Yang et al. [214] developed a fast pest detection algorithm using lightweight feature extraction and KD to enhance performance on edge devices and Wu et al. [215] presented Deep BarkID, a lightweight CNN for tree species identification from bark images, tailored for use in forest environments with limited computing resources. Finally, Yamamoto [216] utilized CNNs to distill crop models to accelerate understanding of plant physiology, applying DL to evaluate environmental impact on grain yield.

Researchers have also contributed to the development and deployment of lightweight student models on low capacity devices on the edge. Wenjie et al. [217] discuss structured model compression via KD, transferring knowledge from a complex VGG16 model to a lightweight MobileNet. This approach significantly reduces model size and improves performance, making it suitable for deployment on devices with limited resources. Wang et al. [218] explore lightweight model development for leaf image analysis, particularly for coffee leaf pest and disease identification. Using VGG as a teacher network, a student network is trained with KD, achieving high recognition accuracy and speed, which is crucial for real-time analysis. In the same line, Li et al. [219] delve into KD for instance-based semantic segmentation, particularly applying it to transform complex transformer models into more efficient DCNN architectures, which shows effectiveness in agricultural applications on datasets like BUP20 and SB20. Arablouei et al. [220] use KD to create compact models suitable for classifying animal behavior from accelerometry data on wearable devices. The models are optimized for real-time, in-situ performance on devices with limited computational resources.

Finally, there are works that take advantage of lightweight student models and combine them with heterogeneous remote sensor data to improve prediction accuracy. Castellano et al. [221] study the application of KD for mapping weeds using UAVs in precision agriculture, developing a lightweight Vision Transformer-based model that provides effective weed mapping with minimal computational resources, and Bansal et al. [222] develop a transformer-based network for plant growth monitoring, utilizing KD to enhance the model’s performance by fusing RGB and depth image data for more accurate growth predictions. Table 7 provides a summary of the main works that employ KD in the agriculture domain.

KD has emerged as a vital technique in urban planning, particularly in the context of enhancing the efficiency and accuracy of models used for complex tasks such as environmental monitoring, infrastructure management, and resource allocation. For instance, KD has been effectively used to improve the real-time detection of building defects, optimize building extraction from noisy datasets, and enhance the accuracy of traffic flow prediction and travel time estimation. This technique is particularly valuable in scenarios involving large-scale urban data, where it enables the deployment of sophisticated models on resource-constrained devices, such as UAVs and edge computing frameworks, facilitating more efficient management of urban infrastructure and services. By enabling the transfer of knowledge from powerful teacher models to more efficient student models, KD supports the development of robust, scalable solutions that are essential for modern urban planning and the creation of smarter, more responsive cities For instance, Rithanasophon et al. [228] proposed a method that leverages deep CNNs (DCNNs) and KD to evaluate QoL for pedestrians using walkability data collected through virtual reality tools, achieving significant improvements in model performance and computational efficiency. Similarly, Liu et al. [229] introduced UrbanKG, a knowledge graph system that integrates KD for urban data fusion, showing promising results in boosting performance across various urban computing applications. Xu et al. [230] also addressed the challenges of limited training samples in building polygon extraction by proposing BPDNet, a KD-based framework that effectively integrates generalization knowledge from large datasets with task-specific characteristics, resulting in superior performance in complex urban environments.

Federated learning frameworks have also benefited from KD, particularly in the context of land use monitoring and environmental impact assessment. Kanagavelu et al. [185] demonstrated the potential of integrating KD with federated UNet models for the semantic segmentation of satellite and street view images, achieving high accuracy and significant model compression. In a similar vein, Xu et al. [231] developed a KD-based building extraction method that reduces the impact of noise on model performance while maintaining generalization, achieving notable improvements in precision, recall, and IoU metrics. In the context of transportation systems, KD has been utilized to enhance travel time estimation (TTE) models and improve traffic flow prediction. Yang et al. [106] proposed KDTTE, a deep neural network model that employs KD to reduce computation and memory costs while increasing accuracy, significantly outperforming state-of-the-art baselines in TTE tasks. In a different but related task, Li et al. [232] applied deep KD to traffic flow prediction in spatio-temporal networks, demonstrating improvements in both local and global feature perception and achieving better accuracy in traffic predictions.

In the autonomous driving domain and the task of off-road environment segmentation, KD has been instrumental in improving model efficiency and accuracy. Pan et al. [233] developed an end-to-end lane detection method using KD to guide polynomial regression under complex road conditions, achieving competitive results in efficiency and accuracy. Similarly, Kim and An [234] proposed a KD method for segmenting off-road environment range images, resulting in a favorable trade-off between segmentation performance and computational cost, highlighting its effectiveness for autonomous systems.

Lee et al. [235] proposed a high-speed detection method for multi-class defects on residential building façades using KD. The study demonstrated that applying KD to a lightweight DL model significantly improved mean average precision (mAP) by approximately 20% and reduced inference time by 2.5 times, making it more suitable for real-time applications. Moving on, Chen et al. [108] introduced a novel approach to building extraction that utilizes KD to enhance the robustness of the distilled student model. The study employed a multi-teacher collaborative distillation strategy to transfer comprehensive feature knowledge from teacher networks to the student model. The approach demonstrated state-of-the-art performance on multiple datasets, including the Massachusetts Roads Dataset, LRSNY Roads Dataset, and WHU Building Dataset, achieving high IoU scores and improving learning capabilities. Geng et al. [79] developed a lightweight topological space network for road extraction from optical RS images, leveraging KD. The study addressed the challenge of extracting topological features from complex road networks by proposing a topological space loss calculation model. The method resulted in significant improvements in accuracy and computational efficiency, demonstrating a good balance between performance and model size.

Besides, Li et al. [236] proposed an off-policy imitation learning method for autonomous driving that employs task KD. This approach was designed to clone human driving behavior and transfer driving strategies to new, unseen scenarios. The method showed promising results in transferring knowledge to different illumination and weather conditions, enhancing route-following performance in realistic urban driving scenes. Hong et al. [237] introduced a hierarchical edge-decision framework for intelligent transportation systems (ITS) that incorporates KD. The framework enables vehicle-road-cloud cooperation to enhance real-time motion planning by distilling complex spatial-temporal event reasoning into efficient decision-making processes. The method was validated on autonomous driving scenarios, demonstrating improved adaptability to complex environments. Luo et al. (2022) [238] presented the KeepEdge framework, which integrates deep neural networks into an edge computing system for UAV-assisted parcel delivery. By employing KD, the study created a lightweight model that maintained high accuracy while reducing the computational load on UAVs. This approach proved effective in complex environments where traditional GPS-based positioning might fail. Pelizari et al. (2023) [239] developed a deep multitask learning (MTL) architecture for building characterization using street-level imagery. The study incorporated KD to encode cross-task interdependencies, which improved the generalization capabilities of the model across multiple natural hazards. The proposed MTL methods outperformed traditional single-task learning (STL) models, achieving higher accuracy and efficiency. The aforementioned studies that demonstrate the versatility of KD in enhancing the efficiency, accuracy, and scalability of models in various prediction and classification tasks in urban planning and intelligent transportation systems, using RS data are summarized in Table 8.