Evaluating the Performance of Dynamic Database Applications

This chapter explores the effect that changing access patterns has on the performance of database management systems. Changes in access patterns play an important role in determining the efficiency of key performance optimization techniques, such as dynamic clustering, prefetching, and buffer replacement. However, all existing benchmarks or evaluation frameworks produce static access patterns in which objects are always accessed in the same order repeatedly. Hence, we have proposed the dynamic evaluation framework (DEF) that simulates access pattern changes using configurable styles of change. DEF has been designed to be open and fully extensible (e.g., new access pattern change models can also be added easily). In this chapter, we instantiate DEF into the dynamic object evaluation framework (DoEF), which is designed for object databases, that is, object-oriented or object-relational databases such as multimedia databases or most eXtensible Mark-up Language (XML) databases. IDEA GROUP PUBLISHING This paper appears in the publication, Advanced Topics in Database Research, vol. 5 edited by Keng Siau © 2006, Idea Group Inc. 701 E. Chocolate Avenue, Suite 200, Hershey PA 17033-1240, USA Tel: 717/533-8845; Fax 717/533-8661; URL-http://www.idea-group.com ITB13075 Evaluating the Performance of Dynamic Database Applications 295 Copyright © 2006, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. INTRODUCTION In database management systems (DBMSs), architectural or optimisation choices, efficiency comparison, or tuning all require the assessment of system performance. Traditionally, this is achieved with the use of benchmarks, namely, synthetic workload models (databases and operations) and sets of performance metrics. To the best of our knowledge, none of the existing database benchmarks incorporate the possibility of change in the access patterns, whereas in real life, almost no application always accesses the same data in the same order repeatedly. Furthermore, the ability to adapt to changes in access patterns is critical to database performance. Highly tuning a database to perform well for only one particular access pattern can lead to poor performance when different access patterns are used. In addition, the performance of a database on a particular trace provides little insight into the reasons behind its performance, and thus is of limited use to database researchers or engineers, who are interested in the identification and improvement in the performance of particular components of the system. Hence, this chapter aims to present a new perspective on DBMS performance evaluation by exploring how to assess the dynamic behaviour of DBMSs. More precisely, this chapter presents a benchmarking framework that allows users to explore the performance of databases under different styles of access pattern change. In contrast, benchmarks of the Transaction Processing Performance Council (TPC) family aim to provide standardised means of comparing systems for vendors and customers. In this chapter, we take a look at how dynamic application behaviour can be modelled and propose the dynamic evaluation framework (DEF). DEF contains a set of protocols that define a set of styles of access pattern change. DEF by no means has exhausted all possible styles of access pattern change. However, it is designed to be fully extensible and its design allows new styles of change to be easily incorporated. Finally, DEF is a generic platform that can be specialized to suit the particular needs of a given family of DBMS (e.g., relational, object, object-relational, or XML). In particular, it is designed to be implemented on top of an existing benchmark so that previous benchmarking research and standards can be reused. In this chapter, we show the utility of DEF by creating an instance of DEF called the dynamic object evaluation framework (DoEF) (He & Darmont, 2003). DoEF is designed for object databases. Note that in the remainder of this chapter, we term object database management systems (ODBMSs) both object-oriented and object-relational systems, indifferently. ODBMSs include most multimedia and XML DBMSs, for example. To illustrate the effectiveness of DoEF, this chapter presents the results of two sets of experiments. First, it presents benchmark results of four state-of-the-art dynamic clustering algorithms (Bullat & Schneider, 1996; Darmont, Fromantin, Regnier, Gruenwald, & Schneider, 2000; He, Marquez, & Blackburn, 2000). There are three reasons for choosing to test the effectiveness of DoEF using dynamic clustering algorithms: 1. ever since the “early days” of object database management systems, clustering has been proven to be one of the most effective performance enhancement techniques (Gerlhof, Kemper, & Moerkotte, 1996); 2. the performance of dynamic clustering algorithms is very sensitive to c h a n g ing access patterns; and 3. despite this sensitivity, no previous attempt has been made to benchmark these algorithms in this way. 24 more pages are available in the full version of this document, which may be purchased using the "Add to Cart" button on the product's webpage: www.igi-global.com/chapter/evaluating-performancedynamic-database-applications/4398?camid=4v1 This title is available in InfoSci-Books, InfoSci-Database Technologies, Library Science, Information Studies, and Education, InfoSci-Library Information Science and Technology. Recommend this product to your librarian: www.igi-global.com/e-resources/libraryrecommendation/?id=1


INTRODUCTION
In database management systems (DBMSs), architectural or optimisation choices, efficiency comparison, or tuning all require the assessment of system performance.Traditionally, this is achieved with the use of benchmarks, namely, synthetic workload models (databases and operations) and sets of performance metrics.To the best of our knowledge, none of the existing database benchmarks incorporate the possibility of change in the access patterns, whereas in real life, almost no application always accesses the same data in the same order repeatedly.Furthermore, the ability to adapt to changes in access patterns is critical to database performance.Highly tuning a database to perform well for only one particular access pattern can lead to poor performance when different access patterns are used.In addition, the performance of a database on a particular trace provides little insight into the reasons behind its performance, and thus is of limited use to database researchers or engineers, who are interested in the identification and improvement in the performance of particular components of the system.Hence, this chapter aims to present a new perspective on DBMS performance evaluation by exploring how to assess the dynamic behaviour of DBMSs.
More precisely, this chapter presents a benchmarking framework that allows users to explore the performance of databases under different styles of access pattern change.In contrast, benchmarks of the Transaction Processing Performance Council (TPC) family aim to provide standardised means of comparing systems for vendors and customers.In this chapter, we take a look at how dynamic application behaviour can be modelled and propose the dynamic evaluation framework (DEF).DEF contains a set of protocols that define a set of styles of access pattern change.DEF by no means has exhausted all possible styles of access pattern change.However, it is designed to be fully extensible and its design allows new styles of change to be easily incorporated.Finally, DEF is a generic platform that can be specialized to suit the particular needs of a given family of DBMS (e.g., relational, object, object-relational, or XML).In particular, it is designed to be implemented on top of an existing benchmark so that previous benchmarking research and standards can be reused.
In this chapter, we show the utility of DEF by creating an instance of DEF called the dynamic object evaluation framework (DoEF) (He & Darmont, 2003).DoEF is designed for object databases.Note that in the remainder of this chapter, we term object database management systems (ODBMSs) both object-oriented and object-relational systems, indifferently.ODBMSs include most multimedia and XML DBMSs, for example.
To illustrate the effectiveness of DoEF, this chapter presents the results of two sets of experiments.First, it presents benchmark results of four state-of-the-art dynamic clustering algorithms (Bullat & Schneider, 1996;Darmont, Fromantin, Regnier, Gruenwald, & Schneider, 2000;He, Marquez, & Blackburn, 2000).There are three reasons for choosing to test the effectiveness of DoEF using dynamic clustering algorithms: 1. ever since the "early days" of object database management systems, clustering has been proven to be one of the most effective performance enhancement techniques (Gerlhof, Kemper, & Moerkotte, 1996); 2.
the performance of dynamic clustering algorithms is very sensitive to c h a n ging access patterns; and 3. despite this sensitivity, no previous attempt has been made to benchmark these algorithms in this way.