Filebench

From Filebench
Revision as of 17:28, 31 May 2011 by Vass-vass (Talk | contribs)

Jump to: navigation, search

Introduction

Filebench is a file system and storage benchmark that allows to generate a large variety of workloads. Unlike typical micro-benchmarks it allows to minutely specify applications' behaviour using extensive Workload Model Language (WML).

What is Filebench

Filebench is quick to set up and use unlike many of the commercial benchmarks which it can emulate. It is also a handy tool for micro-benchmarking storage subsystems and studying the relationships of complex applications such as relational databases with their storage without having to incur the costs of setting up those applications, loading data and so forth.

Filebench uses loadable workload personalities in a common framework to allow easy emulation of complex applications upon file systems. The workload personalities use a Workload Definition Language to define the workload's model.

Filebench Features

Filebench includes many features to facilitate filesystem benchmarking.

  • Configurable for multiple workload types via f scripts
  • Configurable for multi-threaded or multi-programmed
  • Integrated statistics for micro-second accurate latency and cycle counts per system call
  • Synchronization between threads
  • Workload rampup/rampdown to allow steady-state measurement, and exclusion of noise during file create/setup
  • Fractal directory hierarchies with depth, width, filesizes according to statistical distributions
  • Database Emulation features, including semaphores, O_SYNC etc

Linux Port

FSL are managing a Linux port of Filebench. Visit the FSL page here

Filebench News

Filebench continues to evolve. Read about the latest developments and planned features at Filebench_News.

Quick Start

If you're on OpenSolaris, there is no need to download the package - its already included (as of snv_76)! Just look under /usr/benchmarks/filebench. You'll note its installed as package SUNWfilebench. If you're not on OpenSolaris or aren't able to upgrade to snv_76, then follow the steps below to grab the package from sourceforge. Note: the package on sourceforge includes a few more things than the one bundled with OpenSolaris.

1. Download the Filebench package
It is on sourceforge here.
2. Install the Filebench package
It should be a standard package install for your OS. See here for a walkthrough.
3. Create a directory for workload files
This should be on the filesystem you would like to benchmark, and have plenty of spare room.
For example, you could mount a new filesystem under "/filebench" for testing.
4. Run filebench
The following executes a randomread workload, 
# cd /usr/benchmarks/filebench/config
# /usr/benchmarks/filebench/bin/filebench randomread
parsing profile for config: randomread2k
Running /tmp/centerville-tmpfs-randomread-Sep_26_2007-13h_42m_27s/randomread2k/thisrun.f
Filebench Version 1.0.0
 3941: 0.009: Random Read Version 1.12 2005/06/25 01:03:16 IO personality successfully loaded
 3941: 0.010: Creating/pre-allocating filesets
 3941: 0.011: Fileset largefile1: 1 files, avg dir = 10.0, avg depth = 0.0, mbytes=160
 3941: 0.011: Creating fileset largefile1...
 3941: 0.012: Preallocated 1 of 1 of fileset largefile1 in 1 seconds
 3941: 0.012: waiting for fileset pre-allocation to finish
 3941: 14.279: Running '/usr/benchmarks/filebench/scripts/fs_flush tmpfs /tmp'
filesystem type is: tmpfs, no action required, so exiting
 3941: 14.304: Change dir to /tmp/centerville-tmpfs-randomread-Sep_26_2007-13h_42m_27s/randomread2k
 3941: 14.304: Starting 1 rand-read instances
 3944: 15.316: Starting 1 rand-thread threads
 3941: 18.326: Running...
 3941: 78.775: Run took 60 seconds...
 3941: 78.776: Per-Operation Breakdown
rand-rate                   0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
rand-read1                311ops/s   0.6mb/s      3.2ms/op       51us/op-cpu

 3941: 78.776:
IO Summary:      18804 ops 311.1 ops/s, (311/0 r/w)   0.6mb/s,   3285us cpu/op,   3.2ms latency
 3941: 78.777: Stats dump to file 'stats.randomread2k.out'
 3941: 78.777: in statsdump stats.randomread2k.out
 3941: 78.777: Shutting down processes
Generating html for /tmp/centerville-tmpfs-randomread-Sep_26_2007-13h_42m_27s
file = /tmp/centerville-tmpfs-randomread-Sep_26_2007-13h_42m_27s/randomread2k/stats.randomread2k.out
The first run will spend a while "Creating/pre-allocating files" - for the above example this means creating a 160 Mbyte file.
5. Try running the seqread workload

Installing

Getting and Installing Filebench

Download the newest version of filebench package (Filebench) [here].

It must first be unpacked into a temporary directory: 

# gunzip (filebench-pkg-name).tar.gz 
# tar -xvf (filebench-pkg-name).tar

If you have a previous version of Filebench installed, you should back up any scripts you have created in the 'filebench' directory (including config, scripts and workloads subdirectories) and remove the Filebench package. The new package can then be installed from the temporary directory, answering yes to the question about installing setuid/setgid files. 

# pkgadd -d .

The following packages are available:
  1  filebench     Filebench
                   (sparc,i386) 15 Oct 04

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]: 1

Processing package instance  from 

## Processing package information.
## Processing system information.
   10 package pathnames are already properly installed.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.                          

Installing Filebench as 

## Installing part 1 of 1.
/usr/benchmarks/filebench/README
/usr/benchmarks/filebench/README.benchpoint
/usr/benchmarks/filebench/bin/Comm.pl
/usr/benchmarks/filebench/bin/benchpoint
/usr/benchmarks/filebench/bin/i386/fastsu
/usr/benchmarks/filebench/bin/i386/filebench
/usr/benchmarks/filebench/bin/i386/gnuplot
/usr/benchmarks/filebench/bin/sparcv9/fastsu
/usr/benchmarks/filebench/bin/sparcv9/filebench
/usr/benchmarks/filebench/bin/sparcv9/gnuplot
/usr/benchmarks/filebench/runscripts/runem.sh
/usr/benchmarks/filebench/tools/collect_iostat
/usr/benchmarks/filebench/tools/collect_lockstat
/usr/benchmarks/filebench/tools/collect_vmstat
/usr/benchmarks/filebench/tools/filebench_plot
/usr/benchmarks/filebench/tools/filebench_summary
/usr/benchmarks/filebench/workloads/bringover.f
/usr/benchmarks/filebench/workloads/createfiles.f
/usr/benchmarks/filebench/workloads/deletefiles.f
/usr/benchmarks/filebench/workloads/fileserver.f
/usr/benchmarks/filebench/workloads/multistreamread.f
/usr/benchmarks/filebench/workloads/oltp.f
/usr/benchmarks/filebench/workloads/postmark.f
/usr/benchmarks/filebench/workloads/randomread.f
/usr/benchmarks/filebench/workloads/singlestreamread.f
/usr/benchmarks/filebench/workloads/varmail.f
/usr/benchmarks/filebench/workloads/webproxy.f
/usr/benchmarks/filebench/workloads/webserver.f
[ verifying class  ]

Installation of  was successful.

Here Filebench was installed into the /usr/benchmarks/filebench directory.

What is in the Package

The package installs into /usr/benchmarks/filebench and contains the following 6 subdirectories:

  • bin: The binaries directory contains architecture specific subdirectories (amd64 etc) and binary stubs to execute the correct go_filebench binary via isaexec(5). filebench is a perl executeable and is the batch harness for go_filebench.
  • docs: The douments directory contains READMEs, licensing information and this manual.
  • config: This directory contains sample profiles used as input to filebench for running benchmarks and processing statistics in batches.
  • scripts: The scripts directory contains utilities for gathering various different system statistics.
  • workloads: Pre-defined f scripts representing micro and application level benchmarks of interest are contained in this directory.
  • xanadu: This directory contains Xanadu which is a separate included tool bundle - a collection of utilities for gathering, processing and graphically displaying system statistics.

Running

Filebench applies file system or disk based load by executing "workload personalities", which define the workload to apply to the system, and provide various tunables to customize the workload. Filebench is shipped with a library of these personalities, ready to use. They are located in /usr/benchmarks/filebench/workloads.

There are two ways to run Filebench: by using the filebench wrapper script or by invoking the go_filebench core program. The filebench script is the easiest way to run workloads, especially if a suite of workload runs is desired. go_filebench can be used in order to do interactive running (this is typically useful for development testing).

Running with filebench

In the getting started section, the suggested command line:

# /usr/benchmarks/filebench/bin/filebench randomread

actually ran the filebench wrapper script which used a profile file in /usr/benchmarks/filebench/config named randomread.prof to specify the actual workload model to run and set some run specific defaults for it. The randomread.prof profile supplied with Filebench specifies the randomread.f workload and sets the directory for its test file to /tmp, the I/O size to two kilobytes, and total test file size to 160 megabytes. There are currently five profiles supplied with Filebench:

  • randomread.prof - 2 KB random read using the randomread.f workload
  • seqread.prof - Four streams of one megabyte sequential reads using the multistreamread.f workload.
  • fileio.prof - A suite of benchmarks that measure basic I/O throughput
  • filemicro.prof - A suite of "micro" benchmarks
  • filemacro.prof - A suite of large and application oriented benchmarks.

All these profiles are located in the /usr/benchmarks/filebench/config directory.

Running interactively with go_filebench

Those that still need to access the executable (formerly called filebench) can do so by invoking go_filebench:

# /usr/benchmarks/filebench/bin/go_filebench

This will run Filebench in interactive mode, allowing loading and running of individual workload personalities, with full control over their parameters. The following example demonstrates interactively running the varmail workload personality.

Example varmail run

There are simple workload personalities, which configure the type of workload to simulate. An example is a /var/mail directory simulation (like postmark): 

$ /usr/benchmarks/filebench/bin/go_filebench
filebench> load varmail
 8395: 3.898: Varmail personality successfully loaded
 8395: 3.899: Usage: set $dir=<dir>
 8395: 3.900:        set $filesize=<size>    defaults to 16384
 8395: 3.900:        set $nfiles=<value>     defaults to 1000
 8395: 3.901:        set $dirwidth=<value>   defaults to 20
 8395: 3.901:        set $nthreads=<value>   defaults to 1
 8395: 3.902:        set $meaniosize=<value> defaults to 16384
 8395: 3.902:        run <runtime>
filebench> set $dir=/tmp
filebench> run 10
 8395: 14.886: Fileset mailset: 1000 files, avg dir = 20, avg depth = 2.305865, mbytes=15
 8395: 15.301: Preallocated fileset mailset in 1 seconds
 8395: 15.301: Starting 1 filereader instances
 8396: 16.313: Starting 1 filereaderthread threads
 8395: 19.323: Running for 10 seconds...
 8395: 29.333: Stats period = 10s
 8395: 29.347: IO Summary:      21272 iops 2126.0 iops/s, (1063/1063 r/w)  32.1mb/s,    338us cpu/op,   0.3ms latency
 8395: 29.348: Shutting down processes
filebench> stats dump "stats.varmail"
filebench> quit

This run did 21272 logical operations per second - e.g. 21272 open, close, read/write etc... 

$ more stats.varmail

Flowop totals:
closefile4                492ops/s   0.0mb/s      0.0ms/op       31us/op-cpu
readfile4                 492ops/s   7.1mb/s      0.6ms/op      171us/op-cpu
openfile4                 492ops/s   0.0mb/s      5.4ms/op      653us/op-cpu
closefile3                492ops/s   0.0mb/s      1.1ms/op      150us/op-cpu
appendfilerand3           492ops/s   7.7mb/s      0.0ms/op       55us/op-cpu
readfile3                 492ops/s   7.2mb/s      0.6ms/op      163us/op-cpu
openfile3                 492ops/s   0.0mb/s      5.1ms/op      623us/op-cpu
closefile2                492ops/s   0.0mb/s      0.9ms/op      136us/op-cpu
appendfilerand2           492ops/s   7.7mb/s      0.1ms/op       67us/op-cpu
createfile2               492ops/s   0.0mb/s      8.9ms/op     1151us/op-cpu
deletefile1               492ops/s   0.0mb/s      8.4ms/op     1007us/op-cpu

IO Summary:        649116 iops   5409.0 iops/s,      983/983 r/w    29.7mb/s,     1461uscpu/op

Workloads

Workload personalities define the workload to apply, and the default tunables for scaling that workload to the system.

Workloads are stored in workload definition files which Filebench loads, and can be easily customized from either the Filebench command line interface or by editing the files themselves (or copies of the files).

Filebench is shipped with a library of different workloads, which are described in the following sections.

Application Emulation

Application-Emulation Workloads Currently Available in Filebench package

varmail

A /var/mail NFS mail server emaulation, following the workload of postmark, but multi-threaded. The workload consists of a multi-threaded set of open/read/close, open/append/close and deletes in a single directory.

fileserver

A file system workload, similar to SPECsfs. This workload performs a sequence of creates, deletes, appends, reads, writes and attribute operations on the file system. A configurable hierarchical directory structure is used for the file set.

oltp

A database emulator. This workload performance transactions into a filesystem using an I/O model from Oracle 9i. This workload tests for the performance of small random reads & writes, and is sensitive to the latency of moderate (128k+) synchronous writes as in the log file. It by default launches 200 reader processes, 10 processes for asynchronous writing, and a log writer. The emulation includes use of ISM shared memory as per Oracle, Sybase etc which is critical to I/O efficiency (as_lock optimizations).

dss

DSS Database: this is future work TBD.

webserver

A mix of open/read/close of multiple files in a directory tree, plus a file append (to simulate the web log). 100 threads are used by default. 16k is appended to the weblog for every 10 reads.

webproxy

A mix of create/write/close, open/read/close, delete of multiple files in a directory tree, plus a file append (to simulate the proxy log). 100 threads are used by default. 16k is appended to the log for every 10 read/writes.

Micro-benchmark

copyfiles

A copy of a large directory tree. This workload creates a hierarchical directory tree, then measures the rate at which files can be copied from the source tree to a new tree. A single thread is used by default, although this is configurable.

createfiles

Create a directory tree and fill files. A populate of files of specified sizes a directory tree. File sizes are chosen according to a gamma distribution of 1.5, with a mean size of 16k.

randomread

A multi-threaded read of a single large file, defaulting to 8k reads. A single thread is used by default, although configurable by $nthreads.

randomwrite

A multi-threaded write of a single large file, defaulting to 8k writes. A single thread is used by default, although configurable by $nthreads.

singlestreamread

A sequential read of a large file. 1MB reads are used by default.

singlestreamwrite

A sequential write of a large file. 1MB writes are used by default.

multistreamwrite

A sequential read of 4 large files, each with their own reader thread. 1MB reads are used by default.

multistreamwrite

A sequential write of 4 large files, each with their own writer thread. 1MB writes are used by default.

Recommended parameters

The following recommmended parameters have been provided as suggestions; customize as appropriate.

Small Configurations

These may be suitable for servers with one or two disks.

Worklads Suggested Parameters
  • fileserver
  • bringover
  • createfiles
  • deletefiles
  • varmail
  • webproxy
  • webserver
set $nfiles=50000
  • randomread
  • singlestreamread
  • multistreamread
  • singlestreamwrite
  • multistreamwrite
set $filesize=1g
  • oltp
set $filesize=1g

Large Configurations

These may be suitable for servers with 20+ disks.

Worklads Suggested Parameters
  • fileserver
  • bringover
  • createfiles
  • deletefiles
  • mongo
  • varmail
  • webproxy
  • webserver
set $nfiles=100000
  • randomread
  • randomwrite
set $filesize=5g

set $nthreads=256

  • filemicro_createrand
  • filemicro_rread
  • filemicro_rwritefsync
  • singlestreamread
  • multistreamread
  • singlestreamwrite
  • multistreamwrite
set $filesize=5g
  • oltp
set $filesize=5g
  • filemicro_create
  • filemicro_delete
  • filemicro_seqwrite
  • filemicro_seqwriterand
  • filemicro_writefsync
set $count= 100000

Filebench Batch Run Harness

The filebench program calls go_filebench to automatically run multiple configurations. Each config section corresponds to one workload file (.f)

The command is as follows: 

# /bin/benchmarks/filebench/bin/filebench <profile name>

The statistics results are saved in the directory specified in the DEFAULT section.


Workload Model Language

The Workload Model Language defines the workload personalities. It is a concise text based language that has the capability to simulate a variety of complex workload patterns. Filebench is shipped with numerous examples in the workloads directory. These workload files have a .f extension.

If you don't find a appropriate workload model among those supplied with Filebench, you can write your own. See the Writing Workload Models page for a tutorial on how to write Filebench workload models, and the Workload Model Language page for a description of the language syntax. As Filebench is a work in progress, this is not yet a stable interface.

The following are examples of what the language currently looks like.

singlestreamwritedirect.f

The following is the most basic example of a workload file. This workload creates a file and writes to it from a single process and (by default) a single thread. 

#
# Copyright 2005 Sun Microsystems, Inc. All rights reserved.
#
# The contents of this file are subject to the terms of the
# Common Development and Distribution License.
# See the file LICENSING in this distribution for details.
#
 
set $dir=/tmp
set $nthreads=1
set $iosize=1m
 
define file name=largefile1,path=$dir
 
define process name=seqwrite,instances=1
{
  thread name=seqwrite,memsize=10m,instances=$nthreads
  {
    flowop write name=seqwrite,filename=largefile1,iosize=$iosize,directio
    flowop bwlimit name=limit
  }
}
 
echo "Single Stream Write Version 1.7 2005/06/21 21:18:53 personality successfully loaded"
usage "Usage: set \$dir=<dir>"
usage "       set \$filesize=<size> defaults to $filesize"
usage "       set \$nthreads=<value> defaults to $nthreads"
usage "       set \$iosize=<value> defaults to $iosize"
usage " "
usage "       run runtime (e.g. run 60)"

randomread.f

This is the definition file for a random read workload.

Filebench randomread Workload

varmail.f

The following workload file simulates /var/mail activity on a mail server. Note the variety of file operations that can be performed, such as readwholefile and appendfilerand. 

#
# Copyright 2005 Sun Microsystems, Inc. All rights reserved
#
# The contents of this file are subject to the terms of the
# Common Development and Distribution License.
# See the file LICENSING in this distribution for details
#
 
set $dir=/tmp
set $nfiles=1000
set $meandirwidth=1000000
set $filesize=16k
set $nthreads=16
set $meaniosize=16k
 
define fileset name=bigfileset,path=$dir,size=$filesize,entries=$nfiles,dirwidth=$meandirwidth,prealloc=80
 
define process name=filereader,instances=1
{
  thread name=filereaderthread,memsize=10m,instances=$nthreads
  {
    flowop deletefile name=deletefile1,filesetname=bigfileset
    flowop createfile name=createfile2,filesetname=bigfileset,fd=1
    flowop appendfilerand name=appendfilerand2,iosize=$meaniosize,fd=1
    flowop fsync name=fsyncfile2,fd=1
    flowop closefile name=closefile2,fd=1
    flowop openfile name=openfile3,filesetname=bigfileset,fd=1
    flowop readwholefile name=readfile3,fd=1
    flowop appendfilerand name=appendfilerand3,iosize=$meaniosize,fd=1
    flowop fsync name=fsyncfile3,fd=1
    flowop closefile name=closefile3,fd=1
    flowop openfile name=openfile4,filesetname=bigfileset,fd=1
    flowop readwholefile name=readfile4,fd=1
    flowop closefile name=closefile4,fd=1
  }
}
 
echo "Varmail Version 1.24 2005/06/22 08:08:30 personality successfully loaded"
usage "Usage: set \$dir=<dir>"
usage "       set \$filesize=<size> defaults to $filesize"
usage "       set \$nfiles=<value> defaults to $nfiles"
usage "       set \$nthreads=<value> defaults to $nthreads"
usage "       set \$meaniosize=<value> defaults to $meaniosize"
usage "       set \$meandirwidth=<size> defaults to $meandirwidth"
usage "(sets mean dir width and dir depth is calculated as log (width, nfiles)"
usage " dirdepth therefore defaults to dir depth of 1 as in postmark"
usage " set $meandir lower to increase depth beyond 1 if desired)"
usage " "
usage "       run runtime (e.g. run 60)"

List of Flowops

Here you can find the List of Flowops.

List of Attributes

Here you can find the List of Attributes.

Filebench for Programmers

See filebench_for_Programmers for details on the filebench code.

Retrieved from "http://filebench.sourceforge.net/wiki/index.php?title=Filebench&oldid=29"