Wednesday, March 14, 2012

Collection Types in table columns may cause performance problems

As this is not one of the relational database rules supported by Boyce-Codd, in Oracle Database table columns can be defined as NESTED TABLES or COLLECTION TYPES. This functionality seems to be handy in some cases as you can fetch all the values listed in one column but there may be some consequences especially in performance.

To demonstrate the collection and nested table types, i will create three different users table. First table will use a COLLECTION/VARRAY TYPE as the column type, the second one will use the NESTED TABLE and the third table will use built-in VARCHAR type to hold the demo users phone numbers. After all, i will insert some random data to query and to examine the execution plans and statistics information.


--create the test array type
create type TYP_PHONE_VARRAY as varray(5) of varchar2(15); 

--create the test table type
create type TYP_PHONE_TABLE as table of varchar2(15); 


--create table which uses the varray type
drop table T_USER_VARRAY;
create table T_USER_VARRAY
(
username varchar2(25),
fullname varchar2(25),
phone TYP_PHONE_VARRAY default null
);

--create table which uses the nested table
drop table T_USER_TABLE;
create table T_USER_TABLE
(
username varchar2(25),
fullname varchar2(25),
phone TYP_PHONE_TABLE default null
)
NESTED TABLE phone STORE AS nt_t_user_table_phone;

--create table which uses the built-in varchar as column type
drop table U_EPEKER.T_USER_STR;
create table U_EPEKER.T_USER_STR
(
username varchar2(25),
fullname varchar2(25),
phone1 varchar2(15) default null,
phone2 varchar2(15) default null,
phone3 varchar2(15) default null,
phone4 varchar2(15) default null,
phone5 varchar2(15) default null
);



After creating the types and the tables, some random data would be very useful to query and examine the execution plans of the queries. And of course i should not forget to gather statistics, also on the nested table.



--fill the varray typed table with the test data.
declare

  i number;
  v_name varchar2(10);
  v_surname varchar2(10);
  v_username varchar2(25);
  v_phone varchar2(15);
  
begin

  execute immediate ('truncate table T_USER_VARRAY');
  i:=0;
 
  while i<10 loop
    v_name := DBMS_RANDOM.STRING('u', 10);
    v_surname := DBMS_RANDOM.STRING('u', 10);
    v_username := substr(v_name,1,1) || '_' || v_surname;
    v_phone := '+31' || round(DBMS_RANDOM.VALUE(60,70)) || 
                        round(DBMS_RANDOM.VALUE(1000000,9999999)); 
    
    insert into t_user_varray
    values
    (
    v_username,
    v_name || ' ' || v_surname,
    TYP_PHONE_VARRAY(v_phone)
    );
  i:=i+1;
  end loop;
  
  commit;
  
end; 


--fill the nested table with the test data.
declare

  i number;
  v_name varchar2(10);
  v_surname varchar2(10);
  v_username varchar2(25);
  v_phone varchar2(15);
  
begin

  execute immediate ('truncate table T_USER_TABLE');
  i:=0;
 
  while i<10 loop
    v_name := DBMS_RANDOM.STRING('u', 10);
    v_surname := DBMS_RANDOM.STRING('u', 10);
    v_username := substr(v_name,1,1) || '_' || v_surname;
    v_phone := '+31' || round(DBMS_RANDOM.VALUE(60,70)) || 
                        round(DBMS_RANDOM.VALUE(1000000,9999999)); 
    
    insert into t_user_table
    values
    (
    v_username,
    v_name || ' ' || v_surname,
    TYP_PHONE_TABLE(v_phone)
    );
  i:=i+1;
  end loop;
  
  commit;
  
end; 


--fill the conventional table with the test data.
declare

  i number;
  v_name varchar2(10);
  v_surname varchar2(10);
  v_username varchar2(25);
  v_phone varchar2(15);
  
begin

  execute immediate ('truncate table T_USER_STR');
  i:=0;
 
  while i<10 loop
    v_name := DBMS_RANDOM.STRING('u', 10);
    v_surname := DBMS_RANDOM.STRING('u', 10);
    v_username := substr(v_name,1,1) || '_' || v_surname;
    v_phone := '+31' || round(DBMS_RANDOM.VALUE(60,70)) || 
                        round(DBMS_RANDOM.VALUE(1000000,9999999)); 
    
    insert into t_user_str
    (
    username,
    fullname,
    phone1
    )
    values
    (
    v_username,
    v_name || ' ' || v_surname,
    v_phone
    );
  i:=i+1;
  end loop;
  
  commit;
  
end; 

--gather the statistics of the filled tables
exec sys.dbms_stats.gather_table_stats(ownname=>'U_EPEKER', tabname=>'T_USER_STR', estimate_percent=>100, cascade=>true);
exec sys.dbms_stats.gather_table_stats(ownname=>'U_EPEKER', tabname=>'T_USER_VARRAY', estimate_percent=>100, cascade=>true);
exec sys.dbms_stats.gather_table_stats(ownname=>'U_EPEKER', tabname=>'T_USER_TABLE', estimate_percent=>100, cascade=>true);
exec sys.dbms_stats.gather_table_stats(ownname=>'U_EPEKER', tabname=>'NT_T_USER_TABLE_PHONE', estimate_percent=>100, cascade=>true);


After the preparation of the tables and filling them with some test data we can examine the costs of the same identical queries on these tables.

Selecting all five of the columns and only the one column has both 8 bytes of "consistent gets" but the returned amount of data differs as expected. As the developer cannot guess how many phone number exists for an individual user, probably in the code all columns of the phones be selected to be sure of it.

But when selecting from the table which has the VARRAY type as the "phone" column there is an unexpected amount of bytes and "consistent gets" in the first look. As it is also explained in the Oracle Documentation columns and variables which are defined as VARRAY types are objects which should be instantiated once in the memory. This is the most probable reason of this excessive consistent gets which is almost 5 times more than the conventinal VARCHAR column. Even there is only one phone number stored in the VARRAY, read operation results as if there are five values in the list because the object instantiated as it is defined before.

On the other hand, when i examine the query on the table which uses the NESTED TABLE type for the phone column there is a considerable difference in the "consistent gets" and "bytes read" when comparing with the VARRAY type. Most probably this result indicates that NESTED TABLE types are not instantiated as objects in the memory. They are real tables in the database which are nested in another table and returns the results which they store physically in the database. If you investigate you will find the index and the table segments in the tablespace.


EXPLAIN PLAN FOR
SELECT 
  phone1, phone2, phone3, phone4, phone5 
FROM t_user_str 
where username='M_VKFHWFQOKL';  --1 row 48 bytes

/*
Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
          8  consistent gets
          0  physical reads
          0  redo size
        810  bytes sent via SQL*Net to client
        523  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed
*/

EXPLAIN PLAN FOR
SELECT 
  phone1
FROM t_user_str 
where username='M_VKFHWFQOKL';  --1 row 26 bytes

/*
Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
          8  consistent gets
          0  physical reads
          0  redo size
        534  bytes sent via SQL*Net to client
        523  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed
*/

EXPLAIN PLAN FOR
SELECT 
  phone 
FROM t_user_varray 
where username='V_RLQDMEPGBJ';  --1 row 35 bytes

/*
Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
         44  consistent gets
          0  physical reads
          0  redo size
       4619  bytes sent via SQL*Net to client
       1865  bytes received via SQL*Net from client
         11  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed
*/


EXPLAIN PLAN FOR
SELECT 
  phone 
FROM t_user_table 
where username='N_RNJASPGQOS';  --1 row 30 bytes

/*
Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
         16  consistent gets
          0  physical reads
          0  redo size
       1675  bytes sent via SQL*Net to client
        800  bytes received via SQL*Net from client
          4  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed
*/



As a conclusion; if you are not sure that the VARRAY typed column will not get filled properly then, instead of using VARRAY type using NESTED TABLES may be more convenient for the performance of the application. These small decreases of IO and consistent gets may be very valuable in a busy application for an enterprise environment.


Resources:
http://docs.oracle.com/cd/B19306_01/appdev.102/b14261/objects.htm
http://docs.oracle.com/cd/B28359_01/appdev.111/b28371/adobjdes.htm#i448939
http://docs.oracle.com/cd/B28359_01/appdev.111/b28371/adobjcol.htm#i454908


Tuesday, February 21, 2012

which audit options should be enabled

Enabling audit option for an Oracle Database is a smooth operation which needs a restart of the instance. But the real question comes after enabling the audit options: "Which audit options should we set?". As audit logging is a space consumptive operation it is important for the DBA's to carefully select the options to be logged. Otherwise the SYS.AUD$ table will grow unexpectedly. Moreover this table resides in the SYSTEM tablespace and even after changing the audit options and resizing the table will not help the tablespace to be resized which I dont prefer to have a large SYSTEM tablespace.

In my experience, i always hesitate to enable any DML (insert, delete, update and in this case also select) operation to be logged by database audit on application users (database users used by the application itself or the connection pooling). Depending on the intense usage of the application the DML logging may be disastrous as a lot of log will be produced in th SYS.AUD$ table. But the same DML commands may be logged on developer user accounts which is necessary in most of the cases. Of course enabling these kind of DML operations depends on the application itself or what is being expected from the database audit logs.

On the other hand, auditing DDL (create, drop, alter, truncate) operations should be enabled for auditing regardless of the user and object (Of course this also depends on the application behaviour but still should be forced to be audited).

To find which options to be audited exactly, the following query can be used. Which lists the most important System Privileges to be audited. Any user executing these kind of DML operations should be audited for further security surveillance.


SELECT 
  'audit ' || name || ';'
FROM 
  system_privilege_map
WHERE 
  (  name LIKE 'CREATE%'
  or name LIKE 'ALTER%'
  or name LIKE 'DROP%'
  or name LIKE 'EXECUTE%'
  or name LIKE 'GRANT%'
  or name LIKE 'BACKUP%'
  or name LIKE 'EXPORT%'
  or name LIKE 'IMPORT%'
  or name LIKE 'BECOME%'
  )
order by name;


After executing the output of the above script, enabled system privileges can be seen by selecting from the following dictionary view.


select * from dba_priv_audit_opts;


According to the application behaviour, even with the DML auditing only, application user may produce audit data which cannot be managed. In this case individual users should be audited on the DML operations accept the application user itself. It is as easy as adding "by " at the and of the audit statement.


audit drop any table by D_EPEKER;


Beyond auditing the system privileges, individual objects can also be audited. According to the application behaviour some of the tables may have significant importance and not only auditing the DML operations is sufficient but also the DDL operations should be audited to lower the security risks. As it is very hard to manage the operation of the audit data which will be produced by the application user on all tables, in this case individual database tables can be audited.

Keep in my that object level auditing can be both by session and by access. "By Access" audits every occurance of the event while "By Session" audits only the last occurance of the event within the same session. The decision of the level, again depends on what is expected from the audit logs.


audit update on SYS.T_TEST_TABLE by access;
audit update on SYS.T_TEST_TABLE by session;

select * from dba_obj_audit_opts;


Of course, these are my preferences while setting up the auditing option in oracle database. I usually always set the auditing of the privileges for all users as discussed above and leave the rest to the application developers and analysts as they are more aware of the application logic and the most ciritical objects to be audited.



Wednesday, October 19, 2011

Oracle Restart hands on

Starting from Oracle Database 11g, a new product (or functionality) called Oracle Restart comes with the part of the Grid Infrastructure installation. It seems Oracle decided to use crsctl, crs_stat, and srvctl like RAC commands for also managing the processes of the single instance databases. This standardization seems handy to me as i have already get used to manage RAC databases day by day.

After upgraded one of the development databases in our data center from 10.2.0.4 to 11.2.0.2 as well as the ASM instance, i decided to spend some of my time to play with this new functionality.

As on the RAC installations status of the services can be investigated with the crs_stat -t command. I think it is understandable that there is not vip, ons, gsd services here as this is not a RAC database.


[oracle@rhel6]:/oracle > crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora...._ASM.dg ora....up.type ONLINE    OFFLINE               
ora....ER.lsnr ora....er.type OFFLINE   OFFLINE               
ora.asm        ora.asm.type   OFFLINE   OFFLINE               
ora.cssd       ora.cssd.type  ONLINE    ONLINE    rhel6   
ora.diskmon    ora....on.type ONLINE    ONLINE    rhel6 

[oracle@rhel6]:/oracle > crs_stat
NAME=ora.DG_DB_ASM.dg
TYPE=ora.diskgroup.type
TARGET=ONLINE
STATE=OFFLINE

NAME=ora.LISTENER.lsnr
TYPE=ora.listener.type
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.asm
TYPE=ora.asm.type
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.cssd
TYPE=ora.cssd.type
TARGET=ONLINE
STATE=ONLINE on rhel6

NAME=ora.diskmon
TYPE=ora.diskmon.type
TARGET=ONLINE
STATE=ONLINE on rhel6 


The processes of the CRS (it is "HAS" for single instance) is again controlled by crsctl as it is in the RAC installations. You can use check, start, stop options to manage the processes as usual. A small note; CRS processes in the RAC installation is not installed for the single instance installations. For the single instance installations, there is the HAS processes stands for "High Availability Services" and covers the cssd and diskmon processes.


[oracle@rhel6]:/oracle > crsctl check has
CRS-4638: Oracle High Availability Services is online
[oracle@rhel6]:/oracle > crsctl check css
CRS-4529: Cluster Synchronization Services is online
[oracle@rhel6]:/oracle > crsctl check resource ora.cssd

[oracle@rhel6]:/oracle > crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6'
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6'
CRS-2677: Stop of 'ora.cssd' on 'rhel6' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'rhel6'
CRS-2677: Stop of 'ora.diskmon' on 'rhel6' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[oracle@rhel6]:/oracle > crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.



After my upgrade process Oracle Restart could not be able to manage the upgraded database. By using srvctl i added the database resource to the repository so that i can manage the database services by using srvctl command line tool. One of the nicest option is, by using the "-a" option and supplying dependent diskgroups of the database makes Oracle Restart to start the ASM and mount the related diskgroups before starting up the database.


[oracle@rhel6]:/oracle > crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora...._ASM.dg ora....up.type ONLINE    ONLINE    rhel6   
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rhel6   
ora.asm        ora.asm.type   ONLINE    ONLINE    rhel6   
ora.cssd       ora.cssd.type  ONLINE    ONLINE    rhel6   
ora.diskmon    ora....on.type ONLINE    ONLINE    rhel6   

[oracle@rhel6]:/oracle > srvctl add database -h          

Adds a database configuration to be managed by Oracle Restart.

Usage: srvctl add database -d db_unique_name -o oracle_home
  [-m domain_name] 
  [-p spfile] 
  [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY | SNAPSHOT_STANDBY}] 
  [-s start_options] 
  [-t stop_options] 
  [-n db_name] 
  [-y {AUTOMATIC | MANUAL}] 
  [-a "diskgroup_list"]
-d db_unique_name      Unique name for the database
-o oracle_home         ORACLE_HOME path
-m domain              Domain for database. Must be set if database has DB_DOMAIN set.
-p spfile              Server parameter file path
-r role                Role of the database (primary, physical_standby, logical_standby, snapshot_standby)
-s start_options       Startup options for the database. Examples of startup options are open, mount, or nomount.
-t stop_options        Stop options for the database. Examples of shutdown options are normal, transactional, immediate, or abort.
-n db_name             Database name (DB_NAME), if different from the unique name given by the -d option
-y dbpolicy            Management policy for the database (AUTOMATIC or MANUAL)
-a "diskgroup_list"    Comma separated list of disk groups
-h                     Print usage

[oracle@rhel6]:/oracle > srvctl add database -d ORCLT -o /oracle/orahome1
[oracle@rhel6]:/oracle >
[oracle@rhel6]:/oracle >

[oracle@rhel6]:/oracle > crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora...._ASM.dg ora....up.type ONLINE    ONLINE    rhel6   
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rhel6   
ora.asm        ora.asm.type   ONLINE    ONLINE    rhel6   
ora.ORCLT.db   ora....se.type OFFLINE   OFFLINE               
ora.cssd       ora.cssd.type  ONLINE    ONLINE    rhel6   
ora.diskmon    ora....on.type ONLINE    ONLINE    rhel6   

[oracle@rhel6]:/oracle > srvctl start database -d ORCLT
[oracle@rhel6]:/oracle > ps -ef | grep smon
oracle  9109530        1   0 15:46:08      -  0:00 ora_smon_ORCLT
oracle 11075806        1   0 15:43:50      -  0:00 asm_smon_+ASM
[oracle@rhel6]:/oracle >

[oracle@rhel6]:/oracle >                                                                                                     
[oracle@rhel6]:/oracle >

[oracle@rhel6]:/oracle > srvctl status database -d ORCLT
Database is running.

[oracle@rhel6]:/oracle >            
                    
[oracle@rhel6]:/oracle > crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora...._ASM.dg ora....up.type ONLINE    ONLINE    rhel6   
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rhel6   
ora.asm        ora.asm.type   ONLINE    ONLINE    rhel6   
ora.ORCLT.db   ora....se.type ONLINE    ONLINE    rhel6   
ora.cssd       ora.cssd.type  ONLINE    ONLINE    rhel6   
ora.diskmon    ora....on.type ONLINE    ONLINE    rhel6   

[oracle@rhel6]:/oracle >

[oracle@rhel6]:/oracle > srvctl stop database -d ORCLT

[oracle@rhel6]:/oracle > crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora...._ASM.dg ora....up.type ONLINE    ONLINE    rhel6   
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rhel6   
ora.asm        ora.asm.type   OFFLINE   ONLINE    rhel6   
ora.ORCLT.db   ora....se.type OFFLINE   OFFLINE               
ora.cssd       ora.cssd.type  ONLINE    ONLINE    rhel6   
ora.diskmon    ora....on.type ONLINE    ONLINE    rhel6


The second handy feature is "enable" and "disable" of the srvctl which configures the related objects restart options on host restart or restart of the process on failure.


[oracle@rhel6]:/oracle > srvctl enable -h 

The SRVCTL enable command enables the named object so that it can run under 
  Oracle Restart for automatic startup, failover, or restart.

Usage: srvctl enable database -d db_unique_name
Usage: srvctl enable service -d db_unique_name -s "service_name_list"
Usage: srvctl enable asm
Usage: srvctl enable listener [-l lsnr_name]
Usage: srvctl enable diskgroup -g dg_name
Usage: srvctl enable ons [-v]
Usage: srvctl enable eons [-v]



Shutting down everything nicely with Oracle Restart.


[oracle@rhel6]:/oracle > srvctl stop database -d ORCLT
[oracle@rhel6]:/oracle > srvctl stop diskgroup -g DG_DB_ASM
[oracle@rhel6]:/oracle > srvctl stop asm
[oracle@rhel6]:/oracle > srvctl stop listener

[oracle@rhel6]:/oracle > crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora...._ASM.dg ora....up.type OFFLINE   OFFLINE               
ora....ER.lsnr ora....er.type OFFLINE   OFFLINE               
ora.asm        ora.asm.type   OFFLINE   OFFLINE               
ora.ORCLT.db   ora....se.type OFFLINE   OFFLINE               
ora.cssd       ora.cssd.type  ONLINE    ONLINE    rhel6   
ora.diskmon    ora....on.type ONLINE    ONLINE    rhel6  




resources:
http://download.oracle.com/docs/cd/E14072_01/server.112/e10595/restart001.htm
$ srvctl -h