SDR Recording: Recorders

Recorders

The recording module consists of the following data collectors:

Item	Description	Linux	Windows	Solaris	HP-UX
sysrec	overall system statistics	yes	yes	yes	NA
cpurec	per-cpu statistics	yes	yes	yes	NA
diskrec	per-disk statistics	NA	yes	NA	NA
nicrec	per-NIC statistics	yes	yes	yes	NA
hdwrec	system sfw, hdw inventory recorder	yes	yes	yes	NA

corerec	CMT T1, T2 processor statistics	NA	NA	yes	NA
jvmrec	garbage collection statistics	yes	NA	yes	NA
netrec	network protocol statistics	yes	NA	yes	NA
procrec	per process statistics	yes	NA	NA	NA
zonerec	zone statistics	NA	NA	yes	NA

sysrec

Linux

Parameters	42
Data Source	/proc
API	Sys::Statistics::Linux
Language	Perl5

sysrec records system metrics from a Linux x86/64 kernel system. The recorder requires a minimum version of Linux kernel: 2.6. The recorder uses Sys::Statistics::Linux a perl module which handles data collection from /proc interface. The recorder outputs its data as sdrd, sdr raw data format.

sysrec records system utilization along with certain other imortant metrics in two modes: standard and extended modes. The difference comes in displaying additional memory statistics, when executed in extended mode. The extended mode is activated when using -x option on the command line.

The following shows sysrec output parameters:

timestamp	cpupct	sumpct	headpct	userpct	nicepct	syspct	idlepct	iowaitpct	irqpct	softirqpct	stealpct
1335171430:	1.00:	1.00:	99.00:	0.00:	0.00:	1.00:	99.00:	0.00:	0.00:	0.00:	0.00:

runqsz	plistsz	memusedpct	memused	memfree	memtotal	buffers	cached	realfree	realfreepct
94.46:	723980:	42444:	766424:	288160:	136932:	467536:	61.00:	0:	105:

swapusedpct	swapused	swapfree	swaptotal	swapcached
0.01:	144:	2064200:	2064344:	0:

readReq	writeReq	totReq	readByt	writeByt	totByt
0:	0:	0:	0.00:	0.00:	0.00:

rxByt	txByt	ntByt	rxerr	txerr	rxdrp	txdrp
0.06:	0.31:	0.38:	0:	0:	0:	0:

load_avg1	load_avg5	load_avg15
0.00:	0.00:	0.00

Show fields description

timestamp: seconds since Epoch
cpupct: CPU utilization averaged across all CPUs, always <= 100%, percentage, gauge
sumpct: sum of all CPUs utilization always <= N*100%, percentage, gauge
headpct: headroom CPU available, all CPUs, always = N*100% - sumpct, percentage, gauge
userpct: CPU utilization User space in percent, averaged across all CPUs
nicepct: CPU utilization User space with nice priority, averaged across all CPUs
syspct: CPU utilization System space, averaged across all CPUs
idlepct: CPU utilization idle state, averaged across all CPUs
iowaitpct: CPU Percentage in idle state because an I/O operation is waiting to complete
irqpct: CPU Percentage servicing interrupts
softirqpct: CPU Percentage servicing softirqs
stealpct: CPU Percentage of time spent in other operating systems when running in a virtualized environment
runqsz: run queue length, number of tasks waiting for run time
plistsz: number of tasks in the task list
memusedpct: size of used memory in percent, gauge
memused: size of used memory in kilobytes, gauge, extended
memfree: size of free memory in kilobytes, gauge, extended
memtotal: size of memory in kilobytes, gauge, extended
buffers: size of buffers used from memory in kilobytes, gauge, extended
cache: size of cached memory in kilobytes, gauge, extended
realfree: size of memory is real free, memfree+buffers+cached, gauge
realfreepct:: size of memory is real free in percent of total memory, gauge
swapusedpct: size of used swap space in percent, gauge
swapused: size of swap space is used is kilobytes, gauge, extended
swapfree: size of swap space is free in kilobytes, gauge, extended
swaptotal: size of swap space in kilobytes, gauge, extended
swapcached: memory that once was swapped out, is swapped back in but still also is in the swapfile, gauge, extended
readReq: total disk read requests, counter
writeReq: total disk write requests, counter
totReq: total disk read+write requests, counter
readByt: total read bytes / sec, in KB
writeByt: total write bytes / sec, in KB
totByt: total read+write bytes / sec, in KB
rxByt: total received bytes /sec, in KB
txByt: total transmitted bytes /sec, in KB
ntByt: total received + transmitted bytes /sec, in KB
rxerr: Number of errors that happend while received packets/second
txerr: Number of errors that happend while transmitting packets/second
rxdrp: Number of rx packets that were dropped per second
txdrp: Number of rx packets that were dropped per second
avg_1: LA of the last minute
avg_5: LA of the last 5 minutes
avg_15: LA of the last 15 minutes

Solaris

Parameters	18
Data Source	kstat
API	Sun::Solaris::Kstat, Perl5

sysrec is a utility, part of K9Toolkit, author Brendan Gregg. The toolkit is a collection of free Perl scripts used to troubleshoot and observe (Open)Solaris systems. Check Appendix for more details. The recorder outputs its data as sdrd, sdr raw data format.

sysrec records system utilization and certain additional kernel statistics used as a starting point in observing the system's general health.

The following shows sysrec output parameters:

timestamp	cpupct	sumpct	headpct	userpct	syspct	idlepct	runqlen	pcount	tcount
1294052332:	1.98:	1.98:	98.02:	0.99:	0.99:	98.02:	0:	35:	167:

memusedpct	pscanner	diskpct	diskerrs	netpct	neterrs	load_avg1	load_avg5	load_avg15
70.69:	0.00:	0.00:	0.00:	0.00:	0.00:	0.00:	0.00:	0.00

Show fields description

Sometimes you might need to ask sysrec to monitor your systems using a sub-second interval. Whjy would you want to do such thing ? The interval is usually the input option to control the time period in seconds between each sample. SDR versions before 0.71 did not support interval lower than 1 sec. In latest releases we enhanced the sleeping between samples, using signals in Perl and we obtain a much finer degree of sleeping between each sample. See below:

Interval 1sec, 5 times: 1313952055:9.09:98.18:0.00:0:0:0:0.00:0.00:0.00:0.00:0.00:0.00:9.09:0.00:0.00:90.91:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00 1313952056:1.00:98.18:0.00:0:0:0:0.00:0.00:0.00:0.06:0.24:0.30:0.00:0.00:1.00:99.00:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00 1313952057:0.00:98.20:0.00:0:0:0:0.00:0.00:0.00:0.06:0.24:0.30:0.00:0.00:0.00:100.00:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00 Interval 0.5sec, 5 times: 1313952100.208:0.00:98.18:0.00:0:0:0:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0.00:100.00:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00 1313952100.710:1.96:98.18:0.00:0:0:0:0.00:0.00:0.00:0.13:0.47:0.60:1.96:0.00:0.00:98.04:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00 1313952101.210:0.00:98.20:0.00:0:0:0:0.00:0.00:0.00:0.13:0.47:0.60:0.00:0.00:0.00:100.00:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00

See the timestamp reporting a different time resolution. Be careful: dont play in PROD systems with values very low since the recorders will add an overhead to your system. Remember SDR is not DTrace or SystemTap.

Windows

Parameters	35
Data Source	Windows Management Instrumentation
API	WMI
Language	C/C++

Windows recorders are implemented as system services, extracting all parameters from WMI interface using C/C++ language. The reason to select C over Perl was that Windows has no default POSIX interface. All sdrd raw data files and transport utilities are similar to Solaris and Linux based operating systems. The recorder outputs its data as sdrd, sdr raw data format.

The following shows sysrec output parameters:

timestamp	cpupct	sumpct	headpct	userpct	syspct	idlepct	irqpct	runqsz	plistsz
1393141185:	2.68:	2.68:	97.32:	0.18:	2.49:	97.32:	1.17:	0:	39:

memusedpct	memused	memfree	memtotal	buffers	cache	realfree	realfreepct
45.99:	474552:	4104:	1031796:	27452:	580304:	557244:	54.01:

swapusedpct	swapused	swapfree	swaptotal	swapcached
25.74:	269912:	778664:	1048576:	0:

readReq	writeReq	totReq	readByt	writeByt	totByt
110:	0:	110:	2:	0:	2:

rxByt	txByt	ntByt	rxerr	txerr	rxdrp	txdrp
0:	0:	0:	0:	0:	71614314:	0

Show fields description

timestamp: seconds since Epoch
cpupct: CPU utilization averaged across all CPUs, always <= 100%, percentage, gauge
sumpct: sum of all CPUs utilization always <= N*100%, percentage, gauge
headpct: headroom CPU available, all CPUs, always = N*100% - sumpct, percentage, gauge
userpct: CPU utilization User space in percent, averaged across all CPUs
syspct: CPU utilization System space, averaged across all CPUs
idlepct: CPU utilization idle state, averaged across all CPUs
irqpct: CPU Percentage servicing interrupts
runqsz: run queue length, number of tasks waiting for run time
plistsz: number of tasks in the task list
memusedpct: size of used memory in percent, gauge
memused: size of used memory in kilobytes, gauge, extended
memfree: size of free memory in kilobytes, gauge, extended
memtotal: size of memory in kilobytes, gauge, extended
buffers: size of buffers used from memory in kilobytes, gauge, extended
cache: size of cached memory in kilobytes, gauge, extended
realfree: size of memory is real free, memfree+buffers+cached, gauge
realfreepct:: size of memory is real free in percent of total memory, gauge
swapusedpct: size of used swap space in percent, gauge
swapused: size of swap space is used is kilobytes, gauge, extended
swapfree: size of swap space is free in kilobytes, gauge, extended
swaptotal: size of swap space in kilobytes, gauge, extended
swapcached: memory that once was swapped out, is swapped back in but still also is in the swapfile, gauge, extended
readReq: total disk read requests, counter
writeReq: total disk write requests, counter
totReq: total disk read+write requests, counter
readByt: total read bytes / sec, in KB
writeByt: total write bytes / sec, in KB
totByt: total read+write bytes / sec, in KB
rxByt: total received bytes /sec, in KB
txByt: total transmitted bytes /sec, in KB
ntByt: total received + transmitted bytes /sec, in KB
rxerr: Number of errors that happend while received packets/second
txerr: Number of errors that happend while transmitting packets/second
rxdrp: Number of rx packets that were dropped per second
txdrp: Number of rx packets that were dropped per second

cpurec

Linux

Parameters	11
Data Source	/proc
API	Sys::Statistics::Linux
Language	Perl5

cpurec records per CPU metrics from a Linux x86/64 system. The recorder requires a minimum version of Linux kernel: 2.6. The recorder uses Sys::Statistics::Linux a perl module which handles data collection from /proc interface. The recorder outputs its data as sdrd, sdr raw data format.

The following shows cpurec output parameters:

timestamp	cpuid	userpct	nicepct	syspct	idlepct	totalpct	irqpct	softirqpct	iowaitpct	stealpct
1293371032:	cpu0:	0.00:	0.00:	0.00:	85.15:	14.85:	0.00:	14.85:	0.00:	0.00:
1293371032:	cpu1:	0.00:	0.00:	0.00:	91.09:	8.91:	0.00:	8.91:	0.00:	0.00:

Show fields description

Solaris

Parameters	13
Data Source	kstat
API	Sun::Solaris::Kstat, Perl5

cpurec is a utility, collecting per-CPU data from kstat. The recorder outputs its data under RRD format. cpurec used mainly to observe CPU activity and analyse how the CPUs are used in the system. Useful for capacity planning.

The following shows cpurec output parameters:

timestamp	cpuid	xcalls	intr	ithr	csw	icsw	migr	smtx	syscalls	usercpt	syspct	idlepct
1225039500:	1:	97:	568:	49:	1195:	69:	190:	40:	4183:	5.95:	4.37:	89.68
1225039500:	0:	98:	936:	513:	1169:	47:	190:	41:	4097:	6.17:	4.62:	89.21
1225039504:	1:	0:	90:	3:	219:	1:	28:	2:	289:	2.97:	0.00:	97.03
1225039504:	0:	0:	482:	378:	158:	3:	23:	3:	579:	0.00:	13.86:	86.14

Show fields description

Windows

Parameters	11
Data Source	Windows Management Instrumentation
API	Windows Management Instrumentation
Language	C/C++

Windows recorders are implemented as system services, extracting all parameters from WMI interface using C/C++ language. The reason to select C over Perl was that Windows has no default POSIX interface. All sdrd raw data files and transport utilities are similar to Solaris and Linux based operating systems. The recorder outputs its data as sdrd, sdr raw data format.

The following shows cpurec output parameters:

timestamp	cpuid	userpct	syspct	idlepct	totalpct	irqpct
1393106426:	0:	0.36:	3.38:	96.26:	3.74:	1.56:
1393106486:	0:	0.08:	2.36:	97.56:	2.44:	1.43:

Show fields description

diskrec

Windows

Parameters	22
Data Source	Windows Management Instrumentation
API	WMI
Language	C/C++

Windows recorders are implemented as system services, extracting all parameters from WMI interface using C/C++ language. The reason to select C over Perl was that Windows has no default POSIX interface. All sdrd raw data files and transport utilities are similar to Solaris and Linux based operating systems. The recorder outputs its data as sdrd, sdr raw data format.

The following shows diskrec output parameters:

timestamp	index	queue	totaltimepct	readtimepct	writetimepct	avgqueue	avgreadqueue	avgwritequeue
1293371032:	disk0:	0.00:	0.00:	0.00:	85.15:	14.85:	0.00:	14.85:

avgtotaltime	avgreadtime	avgwritetime	transfers	reads	writes	totalbytes	readbytes	writebytes
1293371032:	cpu0:	0.00:	0.00:	0.00:	85.15:	14.85:	14.85:	14.85:

avgtotalbytes	avgreadbytes	avgwritebytes	idle	splitio
0.00:	14.85:	0.00:	0.00:	0.00

Show fields description

timestamp:seconds since Epoch
index: disk id
queue: current disk queue length, the number of requests outstanding on the disk at the time the performance data is collected. Includes requests in service at the time of the collection
totaltimepct: disk time, the percentage of elapsed time that the selected disk drive was busy servicing read or write requests
readtimepct: disk read time, the percentage of elapsed time that the selected disk drive was busy servicing read requests
writetimepct: disk write time, the percentage of elapsed time that the selected disk drive was busy servicing write requests
avgqueue: average disk queue length, the average number of both read and write requests that were queued for the selected disk during the sample interval
avgreadqueue: average disk read queue length, the average number of read requests that were queued for the selected disk during the sample interval
avgwritequeue: average disk write queue length, the average number of write requests that were queued for the selected disk during the sample interval
avgtotaltime: average disk sec/transfer, the number of seconds it takes to do the disk transfer from disk
avgreadtime: average disk sec per read, the number of seconds it takes to do the a read from disk
avgwritetime: average disk sec per write, the number of seconds it takes to do the a write from disk
transfers: the rate of read and write operations on the disk
reads: disk reads per sec is the rate of read operations on the disk
writes: disk writes per sec is the rate of write operations on the disk
totalbytes: disk bytes per sec, is the rate bytes are transferred to or from the disk during write or read operations
readbytes: disk read bytes per sec, is the rate at which bytes are transferred from the disk during read operations
writebytes: disk write bytes per sec, is the rate at which bytes are transferred from the disk during write operations
avgtotalbytes: average disk bytes per transfer, is the average number of bytes transferred to or from the disk during write or read operations
avgreadbytes: average disk bytes per read, is the average number of bytes transferred from the disk during read operations
avgwritebytes: average disk bytes per write, is the average number of bytes transferred to the disk during write operations
idle: idle time reports the percentage of time during the sample interval that the disk was idle
splitio: split IO per sec, is the rate at which IOs to the disk were split into multiple IOs. A split IO may result from requesting data of a size that is too large to fit into a single IO or that the disk is fragmented

nicrec

Linux

Parameters	20
Data Source	/proc
API	Sys::Statistics::Linux
Language	Perl5

nicrec records per NIC statistics, receiving and transmitting metrics: the no. of KBytes received/sec, the no. of packets received/sec, the no. of KBytes transmitted/sec, the no. of packets transmitted/sec and many other metrics. The recorder uses Sys::Statistics::Linux to fetch all metrics. The recorder outputs its data as sdrd, sdr raw data format.

The following shows nicrec output parameters:

timestamp	interface	rxKB	rxpcks	rxerrs	rxdrop	rxfifo	rxframe	rxcompr	rxmulti
1313536832:	wlan0:	0.06:	1:	0:	0:	0:	0:	0:	0:

txKB	txpcks	txerrs	txdrop	txfifo	txcolls	txcarr	txcompr	ttpcks	ttKB
0.00:	0:	0:	0:	0:	0:	0:	0:	1:	0.06

Show fields description

Solaris

Parameters	10
Data Source	kstat
API	Sun::Solaris::Kstat
Language	Perl5

nicrec is a utility part of K9toolkit, author Brendan Gregg, printing network traffic, KB/s read and written. The recorder outputs its data under RRD format. A new version, a native C implementation, developed by Tim Cook of Sun Microsystems was offered some time ago. SDR deploys Tim's recorders since it is much lighter and better supported under Solaris and Linux systems. You can however have access to the original Perl version too.

nicrec used to observe the Kb/s transferred for all the network cards, including packet counts and average sizes.

The following shows nicrec output parameters:

timestamp	interface	reads	writes	rpks	wpks	ravg	wavg	utilpct	satpct
1225354256:	e1000g0:	72.29:	3.07:	72.09:	41.56:	1026.89:	75.58:	0.06:	0.00:
1225354256:	mac:	72.29:	3.07:	72.09:	41.56:	1026.89:	75.58:	0.06:	0.00:

Show fields description

Windows

Parameters	20
Data Source	Windows Management Instrumentation
API	WMI
Language	C/C++

The following shows nicrec output parameters:

timestamp	interface	rxKB	rxpcks	rxerrs	rxdrop	rxfifo	rxframe	rxcompr	rxmulti
1313536832:	wlan0:	0.06:	1:	0:	0:	0:	0:	0:	0:

txKB	txpcks	txerrs	txdrop	txfifo	txcolls	txcarr	txcompr	ttpcks	ttKB
0.00:	0:	0:	0:	0:	0:	0:	0:	1:	0.06

Show fields description

hdwrec

Linux

Parameters	15
Data Source	/proc
API	Sys::Statistics::Linux, Perl5, lsb_release

hdwrec records computer inventory information: number of physical and virtual CPUs, total physical memory installed, operating system name and release, kernel patch level, number of JVMs running, the uptime of the system. The recorder outputs its data as sdrd, sdr raw data format.

hdwrec records the following computer hardware and software information:

timestamp	hostname	platform	os	codename	distributor	kernel
1335101393:	sdrdemo:	x86_64:	Ubuntu 11.10:	oneiric:	Ubuntu:	Linux 3.0.0-17-generic:

pcpu	vcpu	memtotal	swaptotal	disks	nics	jvms	uptime
1:	1:	kB:	kB:	1:	1:	0:	56d 2h 35m 56s

Show fields description

Solaris

Parameters	15
Data Source
API	Sun::Solaris::Kstat, Perl5

hdwrec records the following computer hardware and software information:

timestamp	hostname	platform	os	pcpu	vcpu	pmem
1335101393:	sdrdemo:	x86_64:	SunOS 5.10:	1 x86 2400 MHz:	1:	1024 Megabytes:

disks	nics	zones	jvms	uptime
1:	1:	1:	0:	12 day(s)

Show fields description

netrec

Linux

Parameters	12
Data Source	netrec
API	Perl5

netrec records per protocol statistics TCP, UDP from netstat. The recorder uses netstat binary, so this utulity should be present on your Linux installation. The recorder outputs its data as sdrd, sdr raw data format.

netrec records TCP, UDP metrics:

timestamp	active	passive	failed	recv_rst	est	sent_rst	upks_recv	upks_sent
1293441496:	52:	20:	18:	0:	2:	25:	2009:	1616

Show fields description

Solaris

Parameters	26
Data Source	netrec
API	Perl5

netrec is a utility, reporting TCP, UDP and IP statistics from a running local or global zone. If the system deploys one or more zones and if all zones share same TCP/IP stack then you can simple use -s flag to report the numbers just once. The recorder outputs its data as sdrd, sdr raw data format.

netrec used to observe TCP, UDP and IP counters.

Show fields description

timestamp: seconds since Epoch
zonename: Solaris zone name
udpInDatagrams: No. of UDP input datagrams
udpInErr: No. of UDP input errors
udpOutDatagrams: No. of UDP output datagrams
udpOutErrors: No. of UDP output errors
tcpActiveOpens: No. of outgoing connections since boot, counter
tcpPassiveOpens: No. of incoming connections since boot, counter
tcpAttemptFails: No. of outgoing failures since boot, counter
tcpEstabResets: No. of resets to terminate established connections
tcpCurrEstab: No. of current established connections
tcpOutSegs: Total no. of segments sent, counter
tcpOutDataSegs: Sender total no. of data segments sent, counter
tcpOutDataBytes: Sender total no. of bytes in data segments sent, counter
tcpRetransSegs: Total no. of segments retransmitted
tcpRetransBytes: Sender total no. of bytes in segments retransmitted, counter
tcpOutRsts: No. of segments sent with RST flag, counter
tcpListenDrop: Total no. of connections refused, backlog full
tcpListenDropQ0: Total no. of connections refused, half-open queue full
tcpHalfOpenDrop: Total no. of connections dropped, full half-open queue
tcpOutSackRetrs: Total no. of retransmitted segments by SACK retrans
ipInHdrErr: No. of dg discards for iph error
ipInAddrErr: No. of dg discards for bad addr
ipInCksumErr: No. of bad IP header checksum
tcpInErr: Total no. of segments recv with error, counter
udpInCksumErr: No. of UDP packets with bad UDP checksum, counter

The output from netrec is displayed below:

# netrec global:1225354459:241756657:0:242661104:0:167385:475:54354:4477:17:209625:113456:407013:978714:913964897:0:0:0:435380:16:10 lobby:1225354459:241756657:0:242661104:0:167385:475:54354:4477:2:209627:113456:407013:978714:913964897:0:0:0:435380:16:10 wowza-test:1225354459:241756657:0:242661104:0:167385:475:54354:4477:0:209627:113456:407013:978714:913964897:0:0:0:435380:16:10 dss:1225354460:241756657:0:242661104:0:167385:475:54354:4477:0:209627:113456:407013:978714:913964897:0:0:0:435380:16:10 dss-test:1225354460:241756657:0:242661104:0:167385:475:54354:4477:0:209627:113456:407013:978714:913964897:0:0:0:435380:16:10 # netrec -s global:1225354465:241756657:0:242661105:0:167386:475:54354:4477:16:209627:113457:40701586:978715:913964897:0:0:0:435380:161:10 lobby:1225354465:2 wowza-test:1225354466:0 dss:1225354466:0 dss-test:1225354466:0

Note: If you observe for shared TCP/IP stacks we use -s flag in order to skip reporting same data for all zones

procrec

Data Source	/proc
API	Sys::Statistics::Linux, Perl5

procrec records per process statistics: owner, state, nice, the priority of the process, the no. of light weight processes, the no. of open file descriptors, the no. of minor and major faults the process made and many other metrics. The recorder uses Sys::Statistics::Linux to fetch all metrics. The recorder outputs its data as sdrd, sdr raw data format

procrec records per process metrics:

timestamp	pid	cmd	owner	state	nice	prior	nlwp	fds	minflt	majflt	utime	stime	ttime
1313446375:	1:	(init):	root	S:	0:	20:	1:	0:	0.00:	0.00:	0.00:	0.00:	0.00:

cutime	cstime	cpu	size	resident	nswap	cnswap	syscr	rchar	read_bytes	syscw	wchar	write_bytes
0.00:	0.00:	3:	24711168:	2347008:	0:	0:	0:	0:	0:	0:	0:	0

Show fields description

Example running procrec on a Linux Ubuntu 11.04 desktop:

$ procrec -u parvu 1313492134:1830:(gnome-keyring-d):parvu:S:0:20:5:17:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:161607680:3629056:0:0:0:0:0:0:0:0 1313492134:1849:(sh):parvu:S:0:20:1:4:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:4321280:655360:0:0:0:0:0:0:0:0 1313492134:1883:(ssh-agent):parvu:S:0:20:1:0:0.00:0.00:0.00:0.00:0.00:0.00:0.00:3:12382208:299008:0:0:0:0:0:0:0:0

zonerec

Parameters	8
Data Source	prstat
API	Perl5

zonerec is a simple script calling prstat utility to report zone utilization in human readable format. This data needs to be parsed and prepared in RRD format. Future versions will include a new recorder which will output its data to RRD format.

zonerec used to observe CPU and Mem utilization:

timestamp	zonename	zid	cpupct	mempct	nproc	swap	rss
1293441496:	global:	0:	0.30:	2.80:	29:	53477376:

Show fields description

corerec

Data Source	SPARC CPU, coreadm
API	Perl5

corerec is a utility using corestat, from Cooltools. The output is human readable format, it will require parsing and proper formating for RRD.

corerec used to observe core utilization from a T1 or T2 processor. Since T1 and T2 have different registers for keeping track of the usage the corestat utility has to be different for each case: corestat.t1 used for T1 processors and corestat.t2 for T2. Recording points:

%Usr: percentage user time
%Sys: percentage sys time
%Usr+Sys: both Usr + Sys
Avg: core utilization

The output from corerec is displayed below:

For a T2 processor: Core,Int-pipe %Usr %Sys %Usr+Sys 0,0 0.05 0.37 0.42 0,1 0.06 0.07 0.13 1,0 0.04 0.08 0.12 1,1 0.01 0.05 0.06 2,0 0.09 0.11 0.20 2,1 0.01 0.06 0.07 3,0 0.02 0.15 0.17 3,1 0.01 0.05 0.05 4,0 0.01 0.13 0.14 4,1 0.01 0.04 0.05 5,0 0.07 0.10 0.16 5,1 0.01 0.45 0.46 6,0 0.02 0.10 0.12 6,1 0.01 0.06 0.07 7,0 0.03 0.12 0.14 7,1 0.01 0.05 0.06 ------------- ----- ----- ------ Avg 0.03 0.12 0.15

Important to note here is that utilization for a T1 or T2 processor does not simple mean data from vmstat or mpstat alone. You have to use corerec in order to gather the correct utilization figures. See below Ravindra Talashikar's notes about mpstat vmstat on T1 processors!

jvmrec

Linux

Parameters	23
Data Source	/proc, jstat
API	Sys::Statistics::Linux, Perl5

jvmrec is a utility based on jstat, part of JDK helping to extract Garbage Collection statistics from a running virtual machine. This recorder will loop over all running zones found on the system and will fetch each java process found and extract its GC numbers. The recorder outputs its data as sdrd, sdr raw data format.

jvmrec records the GC statistics useful to understand how your JVMs are running.

timestamp	name.pid	s0pct	s1pct	edenpct	oldpct	permpct	mGC	tmGC	MGC	tMGC	totalGC
1225360607:	myjvm1.23699:	0.00:	93.76:	62.56:	40.04:	99.34:	5500:	123.901:	60:	86.513:	210.414
1225360668:	myjvm2.23699:	0.00:	83.33:	85.07:	19.05:	91.65:	1507:	2.997:	21:	3.082:	6.079

utime	stime	size	resident	nswap	syscr	rchar	read_bytes	syscw	wchar	write_bytes
0.12:	0.03:	4737872:	2133972:	0:	6:	589:	0:	3:	1299:	669
0.08:	0.02:	3440028:	286032:	0:	5:	280:	0:	2:	240:	724

Show fields description

Solaris

Parameters	12
Data Source	jstat
API	KSH

The output from Solaris jvmrec is displayed below:

timestamp	zone.pid	s0pct	s1pct	edenpct	oldpct	permpct	mGC	tmGC	MGC	tMGC	totalGC
1225360607:	global.23699:	0.00:	29.51:	10.03:	10.07:	60.44:	9:	0.100:	1:	0.058:	0.158
1225360668:	global.23699:	0.00:	29.51:	12.76:	10.07:	60.44:	9:	0.100:	1:	0.058:	0.158

Show fields description

SystemDataRecorder

Recorders

Linux

Solaris

Windows

Linux

Solaris

Windows

Windows

Linux

Solaris

Windows

Linux

Solaris

Linux

Solaris

Linux

Solaris