Using Async IO on AIX 5L

This article describes steps to verify if you have allocated enough aioserver processes for AIO on AIX 5L.

AIX 5L supports asynchronous I/O (AIO) for database files created both on file system partitions and on raw devices.

AIO on raw devices is implemented fully into the AIX kernel, and does not require database processes to service the AIO requests.

When using AIO on file systems, the kernel database processes (aioserver) control each request from the time a request is taken off the queue to the time it is completed. The number of aioserver servers determines the number of AIO requests that can be processed in the system concurrently. So, it is important to tune the number of aioserver processes when using file systems to store Oracle Database data files.

1) Find out how many aioservers are configured:


smitty -> Devices-> Asynchronous I/O-> Change/Show Characteristics of Asynchronous I/O-> Asynchronous I/O (Legacy)

MINIMUM number of servers                          [10]
MAXIMUM number of servers per cpu                  [20]
Maximum number of REQUESTS                         [4096]
Server PRIORITY                                    [39]
STATE to be configured at system restart            available
State of fast path                                  enable

Note:

Starting with AIX 5L version 5.2, there are two AIO subsystems available. Oracle Database 10g uses Legacy AIO (aio0), even though the Oracle preinstallation script enables Legacy AIO (aio0) and POSIX AIO (posix_aio0). Both AIO subsystems have the same performance characteristics.

An easier way to find setting on legacy AIO(aio0):


$ lsattr -El aio0

autoconfig available STATE to be configured at system restart True
fastpath   enable    State of fast path                       True
kprocprio  39        Server PRIORITY                          True
maxreqs    4096      Maximum number of REQUESTS               True
maxservers 20        MAXIMUM number of servers per cpu        True
minservers 10        MINIMUM number of servers                True

The ‘minservers’ setting (10) is the total number of aioserver kernel threads that start at boot – this is not a per CPU or per logical CPU value.

The ‘maxservers’ setting (20) is the maximum number of aioserver  per CPU or per logical CPU value.

The ‘maxreqs’ setting (4096) is the maximum number of outstanding or pending requests that the kernel will allow at any one time.

The default value for the minimum number of servers is 1. The default value for the maximum number of servers is 10. These values are usually too low to run Oracle Database on large systems with 4 CPUs or more, if you are not using kernelized AIO( for raw devices).

2) Check how many CPUs (or logical CPUs if SMT is enabled) are provided on system


$ lparstat

System configuration: type=Shared mode=Uncapped smt=On lcpu=8 mem=16384 psize=4 ent=3.00

%user  %sys  %wait  %idle physc %entc  lbusy   app  vcsw phint
----- ----- ------ ------ ----- ----- ------   --- ----- -----
 4.2   0.4    2.1   93.2  0.14   4.7   12.9  3.27 2541549655 92497349

The maximum aioserver kernel threads can reach 20*8=160. When aioserver kernel thread starts, it never disappears. Let’s check how many aioserver threads have been started since system boot.


$ ps -ek | grep aioserver | grep -v posix_aioserver | wc -l
 160

The ‘-k’ argument to the ps command lists kernel threads (also known as kprocs) in addition to user-space processes.

It appears that all 160 aioservers have been started, so sometime between when the system booted and this very instant, something needed all of them – and possibly more.

If the value of the maxservers or maxreqs parameter is set too low, then the following warning messages are repeatedly displayed in trace files in directory $BDUMP:


Warning: lio_listio returned EAGAINPerformance degradation may be seen.

3) increase number of maxservers and minservers


chdev -l aio0 -a maxservers='m' -a minservers='n'

You can change the AIO parameters for your system with the chdev command on the aio0 pseudo-device, or by running smitty aio on AIX 5.3 systems – you can even use the poorly documented aioo command. An important thing to remember though – is that you can set the values online, with no interruption of service – BUT – they will not take affect until the next time the kernel is booted.

4) Check CPU time of each aioserver

Another test is to see how much CPU time each of the aioserver. If all of the aioserver threads have a very similar amount of CPU time, and if the amount of CPU time consumed is significant – chances are you need to configure more of them. Depending on how long the OS has been running, you may need to add quite a few more.

$ ps -ek | grep aioserver | more
118880      –  1:31 aioserver
127062      –  1:32 aioserver

References:

Asynchronous disk I/O performance tuning

Asynchronous I/O tunable parameters

http://abstractinitiative.com/PerformanceJungle/2008/11/aix-53-aio-asynchronous-io-tuning/

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: