Benchmark frameworks and τBench
Article first published online: 7 APR 2013
Copyright © 2013 John Wiley & Sons, Ltd.
Software: Practice and Experience
Volume 44, Issue 9, pages 1047–1075, September 2014
How to Cite
2014), Benchmark frameworks and τBench, Softw. Pract. Exper., 44, pages 1027–1045, doi: 10.1002/spe.2189, and (
- Issue published online: 1 AUG 2014
- Article first published online: 7 APR 2013
- Manuscript Accepted: 26 JAN 2013
- Manuscript Revised: 24 JAN 2013
- Manuscript Received: 25 APR 2012
- NSF. Grant Numbers: IIS-0415101, IIS-0639106, IIS-0803229, IIS-1016205
- temporal databases
Software engineering frameworks tame the complexity of large collections of classes by identifying structural invariants, regularizing interfaces, and increasing sharing across the collection. We wish to appropriate these benefits for families of closely related benchmarks, say for evaluating query engine implementation strategies. We introduce the notion of a benchmark framework, an ecosystem of benchmarks that are related in semantically rich ways and enabled by organizing principles. A benchmark framework is realized by iteratively changing one individual benchmark into another, say by modifying the data format, adding schema constraints, or instantiating a different workload. Paramount to our notion of benchmark frameworks are the ease of describing the differences between individual benchmarks and the utility of methods to validate the correctness of each benchmark component by exploiting the overarching ecosystem. As a detailed case study, we introduce τBench, a benchmark framework consisting of ten individual benchmarks, spanning XML, XQuery, XML Schema, and PSM, along with temporal extensions to each. The second case study examines the Mining Unstructured Data benchmark framework, and the third examines the potential benefits of rendering the TPC family as a benchmark framework. Copyright © 2013 John Wiley & Sons, Ltd.