IJMTES – PROTECTING SYSTEM CALL FROM ROOTKIT ATTACK USING RSA ALGORITHM

Journal Title : International Journal of Modern Trends in Engineering and Science

Author’s Name : L Shobanaunnamed

Volume 03 Issue 05 2016

ISSN no:  2348-3121

Page no: 91-95

Abstract – Windows is worldwide accepted user friendly operating system used by the majority of computer systems. In windows system, there is some limitation; it does not have kernel level security that leads to malicious activities to acquire the system services through code injection attacks. The rootkit is a special type of malicious software which normally hides itself and allows the attacker to steal the sensitive information from the system. It normally starts its activation before the operating system has completely booted up. The process of a user level application needs to prove its identity to the kernel before accessing the system services. And the information such as process name or process id is not enough to identify a process by an operating system. That’s results; malware may make another process to acquire the system service. To reduce the possibility of rootkit attacks, we first classify the incoming process request using an algorithm. If the requested service is legitimate means it is allowed to directly acquire the system service. If it is suspicious means it will be processed through customized ntdll.dll. The customization can be done by adding pre-processing and processing functions. By using RSA algorithm the scrambling and unscrambling of original process ID can be done in pre-validation and validation function. However, this system disallows the possibility of Brute force attack and avoids the kernel mode crashing.

Keywords— Rootkit, dispatch ID, hooking, API, Scrambling, Unscrambling 

Reference

  1. Hussain M.J. Almohri, Danfeng (Daphne) Yao and Dennis Kafura, “Process Authentication for High System Assurance”, IEEE transactions on dependable and secure computing, vol. 11, no. 2, march/April 2014.
  2. Pradnya Patil, Shubham Joshi,” Kernel Based Process Level Authentication Framework for Secure Computing and High Level System Assurance”, International Journal of Innovative Research in Computer and Communication Engineering (An ISO 3297: 2007 Certified Organization) Vol. 2, Issue 12, December 2014.
  3. Shadi R. Masadeh , Ahmad Azzazi, Bassam A. Y. Alqaralleh and Ali, Mousa.Al Sbou,” A NOVEL PARADIGM IN AUTHENTICATION SYSTEM USING SWIFI ENCRYPTION /DECRYPTION APPROACH”, International Journal of Network Security & Its Applications (IJNSA), Vol.6, No.1, January 2014.
  4. Jinpeng Wei1, Feng Zhu1, and Calton Pu2,” KQguard: Binary-Centric Defense against Kernel Queue Injection Attacks”, Springer-Verlag Berlin Heidelberg 2013.
  5. Jinku Li, Zhi Wang, Tyler Bletsch, Deepa Srinivasan, Michael Grace and Xuxian Jiang, “Comprehensive and Efficient Protection of Kernel Control Data”, IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 6, NO. 4, DECEMBER 2011.
  6. Mohd Anuar Mat Isa1, Jamalul-lail Ab Manan2, Raja Mariam Ruzila Raja Ahmad Sufian3, Azhar Abu Talib4 “An Approach to Establish Trusted Application”, 2010 Second International Conference on Network Applications, Protocols and Services.
  7. Deian Stefan1, Chehai Wu2, Danfeng (Daphne) Yao3, and Gang Xu4 Knowing Where Your Input is From:Kernel-Level Data-Provenance Verification”, This work has been supported in part by REU programs, NSF grant CCF-0728937,CNS-0831186, CNS-0953638.
  8. Takamasa Isohara, Keisuke Takemori, Yutaka Miyake,Ning Qu, Adrian Perrig.” LSM-based Secure System Monitoring Using Kernel Protection Schemes”, 2010 International Conference on Availability, Reliability and Security.
  9. Patrice Clemente, Jonathan Rouzaud-Cornabas, and Christian Toinard,” From a Generic Framework for Expressing Integrity Properties to a Dynamic MAC Enforcement forOperating Systems”, M.L. Gavrilova et al. (Eds.): Trans. on Comput. Sci. XI, LNCS 6480, pp. 131–161, 2010.c Springer-Verlag Berlin Heidelberg 2010.
  10. PENG Chao, YANG Xing-qiang,NIU Zhen-zhou, LIU Xiang-peng,” Research on Windows Operating System Education”, Supported by university relations of Microsoft research Asia.