Optimization Strategies for Self-Supervised Learning in the Use of Unlabeled Data

Authors

  • Haopeng Zhao New York University USA, Illinois Institute of Technology USA, Northeastern University USA, Northeastern University USA, University Maine Presque Isle USA, Massachusetts Institute of Technology USA, Independent Researcher USA, Canoakbit Alliance Inc. Canada
  • Yan Lou New York University USA, Illinois Institute of Technology USA, Northeastern University USA, Northeastern University USA, University Maine Presque Isle USA, Massachusetts Institute of Technology USA, Independent Researcher USA, Canoakbit Alliance Inc. Canada
  • Qiming Xu New York University USA, Illinois Institute of Technology USA, Northeastern University USA, Northeastern University USA, University Maine Presque Isle USA, Massachusetts Institute of Technology USA, Independent Researcher USA, Canoakbit Alliance Inc. Canada
  • Zheng Feng New York University USA, Illinois Institute of Technology USA, Northeastern University USA, Northeastern University USA, University Maine Presque Isle USA, Massachusetts Institute of Technology USA, Independent Researcher USA, Canoakbit Alliance Inc. Canada
  • Ying Wu New York University USA, Illinois Institute of Technology USA, Northeastern University USA, Northeastern University USA, University Maine Presque Isle USA, Massachusetts Institute of Technology USA, Independent Researcher USA, Canoakbit Alliance Inc. Canada
  • Tao Huang New York University USA, Illinois Institute of Technology USA, Northeastern University USA, Northeastern University USA, University Maine Presque Isle USA, Massachusetts Institute of Technology USA, Independent Researcher USA, Canoakbit Alliance Inc. Canada
  • LiangHao Tan New York University USA, Illinois Institute of Technology USA, Northeastern University USA, Northeastern University USA, University Maine Presque Isle USA, Massachusetts Institute of Technology USA, Independent Researcher USA, Canoakbit Alliance Inc. Canada
  • Zichao Li New York University USA, Illinois Institute of Technology USA, Northeastern University USA, Northeastern University USA, University Maine Presque Isle USA, Massachusetts Institute of Technology USA, Independent Researcher USA, Canoakbit Alliance Inc. Canada

DOI:

https://doi.org/10.53469/jtpes.2024.04(05).05

Keywords:

Self-supervised learning, Unlabeled data, Optimization strategies, Data distribution imbalance, Label noise, Deep reinforcement learning

Abstract

This study explores optimization strategies for self-supervised learning in the use of unlabeled data. By deeply analyzing existing research, we propose a novel method that significantly enhances the performance of algorithms on unlabeled data, achieving improved accuracy and generalization capabilities. Our method is validated across multiple datasets, demonstrating superior performance compared to traditional approaches. We also discuss how to optimize self-supervised learning strategies in the use of unlabeled data. Through improvements and optimizations of self-supervised learning algorithms, we introduce a new method for effectively utilizing unlabeled data for model training. Experimental results show significant performance improvements across various datasets, highlighting the method's robust generalization ability. This research is significant for advancing self-supervised learning technologies, providing valuable insights for related fields.

References

Qin Tian, Teng Qifa, Jia Xiuyi. Weakly Supervised Label Distribution Learning Combining Local Marked Sequential Relations [J]. Journal of Intelligent Systems, 2023, 18(01): 47-55.

Jun Xin. Feature Extraction and Utilization of Graph Data Based on Self-Supervised Learning [D]. Advisor: Zhou Fengfeng. Jilin University, 2022.

Huang He, Wang Tengfei, Cheng Jiubing. Self-Supervised Learning-Based Reconstruction of Continuous Missing Seismic Data [A]. Proceedings of the 4th Annual Academic Conference on Oil and Gas Geophysics [C]. Committee on Oil and Gas Geophysics of the Chinese Geophysical Society: 2021: 373-376.

Sun Lang. Application of Self-Supervised Learning in Image Clustering [D]. Advisor: Chen Junfen. Hebei University, 2023.

Xiong Chuanchao. Research on Seismic Data Denoising Methods Based on Self-Supervised Learning [D]. Advisor: Song Chengyun. Chongqing University of Technology, 2023.

Fan Shaoshuai, Ma Xiaotian, Zhang Zeping. Application of Fuzzy Clustering in the Field of Deep Learning Self-Supervised [J]. China New Communications, 2021, 23(04): 166-167.

Tian Chunna, Ye Yanyu, Shan Xiao, Ding Yuxuan, Zhang Xiangnan. A Review of Self-Supervised Video Representation Learning [J]. Journal of Xidian University, 2021, 48(05): 222-230.

Zhu, Armando, et al. "Cross-Task Multi-Branch Vision Transformer for Facial Expression and Mask Wearing Classification." Journal of Computer Technology and Applied Mathematics 1.1 (2024): 46-53.

Li, Keqin, et al. "Utilizing Deep Learning to Optimize Software Development Processes." Journal of Computer Technology and Applied Mathematics 1.1 (2024): 70-76.

Wang, Jin, et al. "Research on Emotionally Intelligent Dialogue Generation based on Automatic Dialogue System." Journal of Computer Technology and Applied Mathematics 1.1 (2024): 1-5.

Li, Keqin, et al. "The Application of Augmented Reality (AR) in Remote Work and Education." Journal of Computer Technology and Applied Mathematics 1.1 (2024): 33-39.

Jin, Jiajun, et al. "Enhancing Federated Semi-Supervised Learning with Out-of-Distribution Filtering Amidst Class Mismatches." Journal of Computer Technology and Applied Mathematics 1.1 (2024): 100-108.

Song, Jintong, et al. "A Comprehensive Evaluation and Comparison of Enhanced Learning Methods." Academic Journal of Science and Technology 10.3 (2024): 167-171.

Wang, Jinyin, et al. "Research on Image Recognition Technology Based on Multimodal Deep Learning." arXiv preprint arXiv:2405.03091 (2024).

Liu, Hao, et al. "Adaptive speed planning for Unmanned Vehicle Based on Deep Reinforcement Learning." arXiv preprint arXiv:2404.17379 (2024).

Shen, Yi, et al. "Localization Through Particle Filter Powered Neural Network Estimated Monocular Camera Poses." arXiv preprint arXiv:2404.17685 (2024).

Yin, Zhengcong, Diya Li, and Daniel W. Goldberg. "Is ChatGPT a game changer for geocoding-a benchmark for geocoding address parsing techniques." Proceedings of the 2nd ACM SIGSPATIAL International Workshop on Searching and Mining Large Collections of Geospatial Data. 2023.

Li, Diya, and Zhe Zhang. "MetaQA: Enhancing human-centered data search using Generative Pre-trained Transformer (GPT) language model and artificial intelligence." Plos one 18.11 (2023): e0293034.

Li, Diya, et al. "A reinforcement learning-based routing algorithm for large street networks." International Journal of Geographical Information Science 38.2 (2024): 183-215.

Zhou, Yucheng, et al. "Improving zero-shot cross-lingual transfer for multilingual question answering over knowledge graph." Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Zhou, Yucheng, et al. "Modeling event-pair relations in external knowledge graphs for script reasoning." Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 (2021).

Zhou, Yucheng, et al. "Eventbert: A pre-trained model for event correlation reasoning." Proceedings of the ACM Web Conference 2022. 2022.

Zhou, Yucheng, et al. "Visual In-Context Learning for Large Vision-Language Models." arXiv preprint arXiv:2402.11574 (2024).

Zhou, Yucheng, et al. "Claret: Pre-training a correlation-aware context-to-event transformer for event-centric generation and classification." arXiv preprint arXiv:2203.02225 (2022).

Zhou, Yucheng, and Guodong Long. "Multimodal event transformer for image-guided story ending generation." arXiv preprint arXiv:2301.11357 (2023).

Chen, Jie, et al. "Full-Stack Machine Learning Development Framework for Energy Industry Applications." Abu Dhabi International Petroleum Exhibition and Conference. SPE, 2022.

Nagao, Masahiro, et al. "An efficient deep learning-based workflow for CO2 plume imaging considering model uncertainties with distributed pressure and temperature measurements." International Journal of Greenhouse Gas Control 132 (2024): 104066.

Yao, Changqing, Masahiro Nagao, and Akhil Datta-Gupta. A Deep-Learning Based Accelerated Workflow for Robust CO2 Plume Imaging at the Illinois Basin-Decatur Carbon Sequestration Project. National Energy Technology Laboratory (NETL), Pittsburgh, PA, Morgantown, WV, and Albany, OR (United States), 2023.

Fu, Jia, et al. "Field application of a novel multi-resolution multi-well unconventional reservoir simulation: History matching and parameter identification." Unconventional Resources Technology Conference, 13–15 June 2023. Unconventional Resources Technology Conference (URTeC), 2023.

Ren, Xiang, et al. "Data driven oil production prediction and uncertainty quantification for unconventional asset development planning through machine learning." Unconventional Resources Technology Conference, 13–15 June 2023. Unconventional Resources Technology Conference (URTeC), 2023.

Xin, Yi, et al. "VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 14. 2024.

Xin, Yi, et al. "Mmap: Multi-modal alignment prompt for cross-domain multi-task learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 14. 2024.

Xin, Yi, et al. "Self-Training with Label-Feature-Consistency for Domain Adaptation." International Conference on Database Systems for Advanced Applications. Cham: Springer Nature Switzerland, 2023.

Peng, Qucheng, Ce Zheng, and Chen Chen. "A Dual-Augmentor Framework for Domain Generalization in 3D Human Pose Estimation." arXiv preprint arXiv:2403.11310 (2024).

Peng, Qucheng, Ce Zheng, and Chen Chen. "Source-free domain adaptive human pose estimation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Peng, Qucheng, et al. "RAIN: regularization on input and network for black-box domain adaptation." arXiv preprint arXiv:2208.10531 (2022).

Pinyoanuntapong, Ekkasit, et al. "Gaitsada: Self-aligned domain adaptation for mmwave gait recognition." 2023 IEEE 20th International Conference on Mobile Ad Hoc and Smart Systems (MASS). IEEE, 2023.

Peng, Qucheng. Multi-source and Source-Private Cross-Domain Learning for Visual Recognition. Diss. Purdue University, 2022.

Downloads

Published

2024-05-23

How to Cite

Zhao, H., Lou, Y., Xu, Q., Feng, Z., Wu, Y., Huang, T., … Li, Z. (2024). Optimization Strategies for Self-Supervised Learning in the Use of Unlabeled Data. Journal of Theory and Practice of Engineering Science, 4(05), 30–39. https://doi.org/10.53469/jtpes.2024.04(05).05

Issue

Vol. 4 No. 05 (2024): JTPES-2023-4-5

Section

Articles

License

Copyright (c) 2024 Haopeng Zhao, Yan Lou, Qiming Xu, Zheng Feng, Ying Wu, Tao Huang, LiangHao Tan, Zichao Li

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.