Attacking and Defending Dynamic Analysis System-Calls Based IDS

Ishai Rosenberg (1) , Ehud Gudes (1, 2)

1 Open University of Israël
2 BGU - Ben-Gurion University of the Negev

Machine-learning augments today’s IDS capability to cope with unknown malware. However, if an attacker gains partial knowledge about the IDS’s classifier, he can create a modified version of his malware, which can evade detection. In this article we present an IDS based on various classifiers using system calls executed by the inspected code as features. We then present a camouflage algorithm that is used to modify malicious code to be classified as benign, while preserving the code’s functionality, for decision tree and random forest classifiers. We also present transformations to the classifier’s input, to prevent this camouflage - and a modified camouflage algorithm that overcomes those transformations. Our research shows that it is not enough to provide a decision tree based classifier with a large training set to counter malware. One must also be aware of the possibility that the classifier would be fooled by a camouflage algorithm, and try to counter such an attempt with techniques such as input transformation or training set updates.

Mots clés: Malware detection; Malware obfuscation; Decision trees; Behavior analysis; Camouflage algorithm; Machine learning

Domaines: Informatique [cs]

Ishai Rosenberg, Ehud Gudes. Attacking and Defending Dynamic Analysis System-Calls Based IDS. 10th IFIP International Conference on Information Security Theory and Practice (WISTP), Sep 2016, Heraklion, Greece. pp.103-119, ⟨10.1007/978-3-319-45931-8_7⟩. ⟨hal-01639619⟩