Tuesday, 10 July 2018

Setup PostgreSQL Using Source Based Installation


Download and setup PostgreSQL using source Based Installation
There are many ways to install & setup PostgreSQL like Package based installation, installer based installation and Source based installation. In this blog, I will show you, how we can setup PostgreSQL 10 using Source Based installation on OEL 6.5 64 Bit Machine.

Pre-requisites
1. Create a PostgreSQL Group and User on the OS.
[root@postgres221 ~]# groupadd postgres
[root@postgres221 ~]# useradd -g postgres postgres
2. Download and install the gcc zlib-devel readline-devel packages if not installed
Note: I installed from the Linux iso media
[root@postgres221 ~]# cd /media/OL6.5\ x86_64\ Disc\ 1\ 20131125/Packages/
[root@postgres221 Packages]# rpm -Uvh *gcc* --nodeps
Preparing...                ########################################### [100%]
   1:libgcc                 ########################################### [  8%]
   2:gcc                    ########################################### [ 17%]
   3:compat-gcc-34          ########################################### [ 25%]
   4:gcc-c++                ########################################### [ 33%]
   5:gcc-objc               ########################################### [ 42%]
   6:gcc-objc++             ########################################### [ 50%]
   7:compat-gcc-34-g77      ########################################### [ 58%]
   8:gcc-gfortran           ########################################### [ 67%]
   9:gcc-gnat               ########################################### [ 75%]
  10:gcc-java               ########################################### [ 83%]
  11:libgcc                 ########################################### [ 92%]
  12:compat-gcc-34-c++      ########################################### [100%]
[root@postgres221 Packages]# rpm -Uvh *zlib-devel* --nodeps
Preparing...                ########################################### [100%]
        package zlib-devel-1.2.3-29.el6.x86_64 is already installed
[root@postgres221 Packages]# rpm -Uvh *readline-devel* --nodeps
Preparing...                ########################################### [100%]
        package readline-devel-6.0-4.el6.x86_64 is already installed
Step 1: Download & extract PostgreSQL Source Code from the below site.
https://www.postgresql.org/ftp/source/

Step 2: Install postgreSQL
[root@postgres221 postgres]# tar -zxvf postgresql-10.0.tar.gz

Once untar the file, it will create a folder:
[root@postgres221 postgres]# ls -ltr
total 25252
drwxrwxrwx 6     1107     1107     4096 Oct  2  2017 postgresql-10.0

[root@postgres221 postgres]# cd postgresql-10.0
[root@postgres221 postgresql-10.0]# ls -ltr
total 672
-rw-r--r--  1 1107 1107   1212 Oct  2  2017 README
-rw-r--r--  1 1107 1107   1529 Oct  2  2017 Makefile
-rw-r--r--  1 1107 1107    284 Oct  2  2017 HISTORY
-rw-r--r--  1 1107 1107   3638 Oct  2  2017 GNUmakefile.in
-rw-r--r--  1 1107 1107   1192 Oct  2  2017 COPYRIGHT
-rw-r--r--  1 1107 1107  76410 Oct  2  2017 configure.in
-rwxr-xr-x  1 1107 1107 495611 Oct  2  2017 configure
-rw-r--r--  1 1107 1107    457 Oct  2  2017 aclocal.m4
drwxrwxrwx 55 1107 1107   4096 Oct  2  2017 contrib
drwxrwxrwx  2 1107 1107   4096 Oct  2  2017 config
drwxrwxrwx  3 1107 1107   4096 Oct  2  2017 doc
-rw-r--r--  1 1107 1107  71584 Oct  2  2017 INSTALL
drwxrwxrwx 16 1107 1107   4096 Oct  2  2017 src

[root@postgres221 postgresql-10.0]# ./configure
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking which template to use... linux
checking whether NLS is wanted... no
checking for default port number... 5432
checking for block size... 8kB
checking for segment size... 1GB
checking for WAL block size... 8kB
checking for WAL segment size... 16MB
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
.
.
.
config.status: linking src/backend/port/dynloader/linux.h to src/include/dynloader.h
config.status: linking src/include/port/linux.h to src/include/pg_config_os.h
config.status: linking src/makefiles/Makefile.linux to src/Makefile.port
[root@postgres221 postgresql-10.0]#

[root@postgres221 postgresql-10.0]# make
make -C src all
make[1]: Entering directory `/postgres/postgresql-10.0/src'
make -C common all
make[2]: Entering directory `/postgres/postgresql-10.0/src/common'
make -C ../backend submake-errcodes
make[3]: Entering directory `/postgres/postgresql-10.0/src/backend'
prereqdir=`cd 'utils/' >/dev/null && pwd` && \
          cd '../../src/include/utils/' && rm -f errcodes.h && \
          ln -s "$prereqdir/errcodes.h" .
make[3]: Leaving directory `/postgres/postgresql-10.0/src/backend'
  .
  .
  .
make -C config all
make[1]: Entering directory `/postgres/postgresql-10.0/config'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/postgres/postgresql-10.0/config'
All of PostgreSQL successfully made. Ready to install.

[root@postgres221 postgresql-10.0]# make install
make[1]: Entering directory `/postgres/postgresql-10.0/config'
/bin/mkdir -p '/usr/local/pgsql/lib/pgxs/config'
/usr/bin/install -c -m 755 ./install-sh '/usr/local/pgsql/lib/pgxs/config/install-sh'
/usr/bin/install -c -m 755 ./missing '/usr/local/pgsql/lib/pgxs/config/missing'
make[1]: Leaving directory `/postgres/postgresql-10.0/config'
PostgreSQL installation complete.

Step 3: Verify the postgreSQL directory structure
[root@postgres221 postgresql-10.0]#  ls -l /usr/local/pgsql/
total 16
drwxr-xr-x 2 root root 4096 Jul 11 00:23 bin
drwxr-xr-x 6 root root 4096 Jul 11 00:23 include
drwxr-xr-x 4 root root 4096 Jul 11 00:23 lib
drwxr-xr-x 6 root root 4096 Jul 11 00:23 share






Step 4: Change the group and ownership to postgres
[root@postgres221 postgresql-10.0]# chown -R postgres:postgres /usr/local/pgsql/
[root@postgres221 postgresql-10.0]# ls -ltr /usr/local/pgsql/
total 16
drwxr-xr-x 6 postgres postgres 4096 Jul 11 00:23 include
drwxr-xr-x 2 postgres postgres 4096 Jul 11 00:23 bin
drwxr-xr-x 6 postgres postgres 4096 Jul 11 00:23 share
drwxr-xr-x 4 postgres postgres 4096 Jul 11 00:23 lib

Step 5: Initialize postgreSQL data directory
[root@postgres221 postgresql-10.0]# su – postgres

[postgres@postgres221 ~]$ /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data/
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

creating directory /usr/local/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data/ -l logfile start
Note: Its created data directory by own in /usr/local/pgsql/data

Step 6: Validate the postgreSQL data directory

[postgres@postgres221 ~]$ ls -ltr /usr/local/pgsql/data/
total 112
-rw------- 1 postgres postgres     3 Jul 11 00:27 PG_VERSION
drwx------ 2 postgres postgres  4096 Jul 11 00:27 pg_twophase
drwx------ 2 postgres postgres  4096 Jul 11 00:27 pg_tblspc
drwx------ 2 postgres postgres  4096 Jul 11 00:27 pg_stat_tmp
drwx------ 2 postgres postgres  4096 Jul 11 00:27 pg_stat
drwx------ 2 postgres postgres  4096 Jul 11 00:27 pg_snapshots
drwx------ 2 postgres postgres  4096 Jul 11 00:27 pg_serial
drwx------ 2 postgres postgres  4096 Jul 11 00:27 pg_replslot
drwx------ 4 postgres postgres  4096 Jul 11 00:27 pg_multixact
drwx------ 2 postgres postgres  4096 Jul 11 00:27 pg_dynshmem
drwx------ 2 postgres postgres  4096 Jul 11 00:27 pg_commit_ts
-rw------- 1 postgres postgres 22778 Jul 11 00:27 postgresql.conf
-rw------- 1 postgres postgres    88 Jul 11 00:27 postgresql.auto.conf
-rw------- 1 postgres postgres  1636 Jul 11 00:27 pg_ident.conf
-rw------- 1 postgres postgres  4513 Jul 11 00:27 pg_hba.conf
drwx------ 2 postgres postgres  4096 Jul 11 00:27 pg_xact
drwx------ 3 postgres postgres  4096 Jul 11 00:27 pg_wal
drwx------ 2 postgres postgres  4096 Jul 11 00:27 pg_subtrans
drwx------ 2 postgres postgres  4096 Jul 11 00:27 pg_notify
drwx------ 2 postgres postgres  4096 Jul 11 00:27 global
drwx------ 5 postgres postgres  4096 Jul 11 00:27 base
drwx------ 4 postgres postgres  4096 Jul 11 00:27 pg_logical

Step 7: Start postgreSQL database

[postgres@postgres221 ~]$  /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data/ -l logfile start
waiting for server to start.... done
server started

Step 8: Create postgreSQL DB and test the installation
[postgres@postgres221 ~]$ /usr/local/pgsql/bin/createdb scott
[postgres@postgres221 ~]$ /usr/local/pgsql/bin/psql scott
/usr/local/pgsql/bin/psql: symbol lookup error: /usr/local/pgsql/bin/psql: undefined symbol:  PQsetErrorContextVisibility    ç error
To resolve the above error, I use the following command
[postgres@postgres221 ~]$ export LD_LIBRARY_PATH=/usr/local/pgsql/lib:${LD_LIBRARY_PATH}
[postgres@postgres221 ~]$ /usr/local/pgsql/bin/psql scott
psql (10.0)
Type "help" for help.

scott=#









Tuesday, 3 July 2018

Migration Oracle Database to PostgreSQL using Ora2Pg


ORACLE TO POSTGRESQL MIGRATION USING Ora2Pg
In this blog, I will explain how to migrate Oracle DB to PostgreSQL DB using ora2Pg tool.


Source Machine:  ora12c.ora.com        192.168.0.105          Oracle 12C
Target Machine   postgres222.ora.com   192.168.0.222          PostgreSQL 10.4

Pre-requisites
To install ora2pg we need following:
ora2pg-18.1.tar.gz : ora2pg archive
DBD-Oracle-1.75_2.tar.gz : Oracle database driver for the DBI module
DBD-Pg-3.6.0.tar.gz : PostgreSQL database driver for the DBI module
DBI-1.636.tar.gz : Database independent interface for Perl

Either way we can install directly using yum installation with below command.
yum install perl-DBD-Pg perl perl-devel  perl-DBI  perl-CPAN
Note: A version of perl > 5.6 require for Ora2pg as it is written in perl











Configure ora2pg and DB Module
I have downloaded all the software and moved to the oracle machine and untar
[root@ora12c ORA2PG]# ls -ltr  /ORA2PG
total 6340
drwxrwxr-x 6 1000  1000    4096 Apr 24  2014 DBD-Oracle-1.74
drwxr-x--- 5  502 games    4096 Apr 24  2016 DBI-1.636
drwxr-xr-x 5 1000  1000    4096 Apr 17  2017 DBD-Pg-3.6.0
drwxrwxr-x 6 root root     4096 Sep  1  2017 ora2pg-18.2

Install DBI module
[root@ora12c ORA2PG]# cd DBI-1.636
[root@ora12c DBI-1.636]# pwd
/ORA2PG/DBI-1.636
[root@ora12c DBI-1.636]# perl Makefile.PL
[root@ora12c DBI-1.636]# make
[root@ora12c DBI-1.636]# make install

Install DBD-Oracle
[root@ora12c ORA2PG]# cd DBD-Oracle-1.74
[root@ora12c DBD-Oracle-1.74]# pwd
/ORA2PG/DBD-Oracle-1.74
[root@ora12c DBD-Oracle-1.74]# export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
[root@ora12c DBD-Oracle-1.74]# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib
[root@ora12c DBD-Oracle-1.74]# export PATH=$ORACLE_HOME/bin:$PATH
[root@ora12c DBD-Oracle-1.74]# perl Makefile.PL
[root@ora12c DBD-Oracle-1.74]# make
[root@ora12c DBD-Oracle-1.74]# make install

Install DBD-Pg
[root@ora12c ORA2PG]# cd DBD-Pg-3.6.0
[root@ora12c DBD-Pg-3.6.0]# pwd
/ORA2PG/DBD-Pg-3.6.0
[root@ora12c DBD-Pg-3.6.0]# perl Makefile.PL
[root@ora12c DBD-Pg-3.6.0]# make
[root@ora12c DBD-Pg-3.6.0]# make install

Install ora2pg
[root@ora12c ORA2PG]# cd ora2pg-18.2
[root@ora12c ora2pg-18.2]# pwd
/ORA2PG/ora2pg-18.2
[root@ora12c ora2pg-18.2]# perl Makefile.PL
[root@ora12c ora2pg-18.2]# make
[root@ora12c ora2pg-18.2]# make install
[root@ora12c ora2pg-18.2]# ora2pg –version
Ora2Pg v18.2

For Migration we will use ora2pg.conf
Now that ora2pg is installed, we can proceed with the migration. The first step is to configure the ora2pg.conf file. We can do a copy of the default template and then modify the file ora2pg.conf. In our case the configuration file is located in /etc/ora2pg directory.

[root@ora12c ora2pg]# ls -ltr /etc/ora2pg/
total 120
-rwxrwxrwx 1 oracle oinstall 58356 Jul  2 05:13 ora2pg.conf  ç Change the ownership to oracle
-rw-r--r-- 1 root   root     58162 Jul  2 08:22 ora2pg.conf.dist
Note: Before change in the ora2pg.conf file. Take a backup of the original file.









Update the parameters in ora2pg.conf
Now I will be modify the /etc/ora2pgo/ora2pg.conf file like ORACLE_HOME, Schema name, password and connectivity details of the DB which we have to convert to PostgreSLQ.

Find the following four Oracle related parameters and change them accordingly:
ORACLE_HOME     /u01/app/oracle/product/12.1/dbhome_1
ORACLE_DSN      dbi:Oracle:host=192.168.0.105;sid=orcl;port=1521
ORACLE_USER     SYSTEM
ORACLE_PWD      manager

Find the following parameters and set them as descibed:
TYPE       TABLE  PACKAGE COPY VIEW GRANT SEQUENCE TRIGGER FUNCTION PROCEDURE TABLESPACE TYPE PARTITION
DISABLE_SEQUENCE 1
DISABLE_TRIGGERS USER
TRUNCATE_TABLE 1
DROP_FKEY

Set parameter OUTPUT to the desired value.
Set parameter SCHEMA to the source DB schema(s), I am here testing only on HR schema
Set parameter PG_SCHEMA to the target schema name
Add the following line to the configuration file:
ORA_INITIAL_COMMAND commit
####Put the below file in the source (oracle) server in the /etc/ora2pg/ora2pg.conf file
PG_DSN  dbi:Pg:dbname=orcl;host=192.168.0.222;port=5598
PG_USER hr
PG_PWD  temp1234





pg_hba.conf  make a entry of the source server for connectivity
Make an entry in the pg_hba.conf  in target side postgres222.ora.com, so that Oracle can be made connection to postgresql. I opened for all as per demo purpose but we cannot use this for prod.
# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
host    all             all             192.168.0.221/32            trust
host    all             all             192.168.0.222/32            trust
host    all             all             0.0.0.0/0            trust






















Run the report and check if any manual action required.
Before run the actual migration, let us check the report, if we need to take any action manually.
[oracle@ora12c ~]$ ora2pg -t SHOW_REPORT  --estimate_cost --cost_unit_value 10 --conf /etc/ora2pg/ora2pg.conf --dump_as_html > report.html
[========================>] 7/7 tables (100.0%) end of scanning.
[========================>] 9/9 objects types (100.0%) end of objects auditing.

We can also open the report.html file and check the summary of the report. Below is the sample of the report.



Once we will check the report and found everything is fine then we can start the actual migration. If there is required any manually intervention in the objects then we need to check those objects before migrate the data. We will do migration in two parts
 1 Migrate the schema structure
 2 Migrate the Actual



Export DDL from Oracle
[oracle@ora12c ~]$ ora2pg -d
Ora2Pg version: 18.2
Trying to connect to database: dbi:Oracle:host=192.168.0.105;sid=orcl;port=1521
Isolation level: SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
DEBUG: executing initial command to Oracle: commit
Looking forward functions declaration in schema HR.
Retrieving table information...
[1] Scanning table COUNTRIES (25 rows)...
[2] Scanning table DEPARTMENTS (27 rows)...
[3] Scanning table EMPLOYEES (107 rows)...
[4] Scanning table JOBS (19 rows)...
[5] Scanning table JOB_HISTORY (10 rows)...
[6] Scanning table LOCATIONS (23 rows)...
[7] Scanning table REGIONS (4 rows)...
Dumping table COUNTRIES...
Dumping table LOCATIONS...
Dumping table DEPARTMENTS...
Dumping table EMPLOYEES...
Dumping table JOB_HISTORY...
Dumping table REGIONS...
Dumping table JOBS...
Dumping RI COUNTRIES...
Dumping RI LOCATIONS...
Dumping RI DEPARTMENTS...
Dumping RI EMPLOYEES...
Dumping RI JOB_HISTORY...
Below file will be created. This file will be copied to target server and import the DDL
[oracle@ora12c ~]$ ls -ltr HR_output.sql
-rw-r--r-- 1 oracle oinstall 3599 Jul  2 17:38 HR_output.sql
Note: We provided the output file name and path in the /etc/ora2pg/ora2pg.conf file
OUTPUT          /home/oracle/HR_output.sql

.
Before import the data, I created the same database, user & schema in the Postgresql server
postgres=# create user hr WITH ENCRYPTED PASSWORD 'temp1234';
postgres=# alter user hr with SUPERUSER;
postgres=# create database orcl;
postgres=# grant all privileges on database orcl to hr;
postgres=# alter database orcl owner to hr;
postgres=# \l orcl
                          List of databases
 Name | Owner | Encoding |   Collate   |    Ctype    | Access privileges
------+-------+----------+-------------+-------------+-------------------
 orcl | hr    | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =Tc/hr           +
      |       |          |             |             | hr=CTc/hr

postgres=# \c orcl hr
orcl=# create schema  hr;

Import the DDL in PostgreSQL
-bash-4.1$ /opt/PostgreSQL/10/bin/psql -d orcl -U hr -W -p 5598 < /tmp/HR_output.sql
Note: The output file HR_output.sql, which were generated by ora2pg –d we copied from oracle to postgresql server










Check the objects in PostgreQL
postgres=# \c orcl hr
orcl=# \dt
          List of relations
 Schema |    Name     | Type  | Owner
--------+-------------+-------+-------
 hr     | countries   | table | hr
 hr     | departments | table | hr
 hr     | employees   | table | hr
 hr     | job_history | table | hr
 hr     | jobs        | table | hr
 hr     | locations   | table | hr
 hr     | regions     | table | hr
(7 rows)

orcl=# \di
                    List of relations
 Schema |       Name       | Type  | Owner |    Table
--------+------------------+-------+-------+-------------
 hr     | countries_pkey   | index | hr    | countries
 hr     | departments_pkey | index | hr    | departments
 hr     | employees_pkey   | index | hr    | employees
 hr     | job_history_pkey | index | hr    | job_history
 hr     | jobs_pkey        | index | hr    | jobs
 hr     | locations_pkey   | index | hr    | locations
 hr     | regions_pkey     | index | hr    | regions
(7 rows)

Now the structure has been transferred in Postgresql. This is empty structure.


Let copy the data now. For that we will directly run the command from oracle server and by network it will send the data to PostgreSQL server.
[oracle@ora12c ~]$ ora2pg -d -t COPY
Ora2Pg version: 18.2
Trying to connect to database: dbi:Oracle:host=192.168.0.105;sid=orcl;port=1521
Isolation level: SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
DEBUG: executing initial command to Oracle: commit
Retrieving table information...

Once it will be completed then verify the data at postgresql side.
orcl=# select count(*) from hr.countries;
LOG:  duration: 0.434 ms  statement: select count(*) from hr.countries;
 count
-------
    25
(1 row)

orcl=# select count(*) from hr.departments;
LOG:  duration: 0.275 ms  statement: select count(*) from hr.departments;
 count
-------
    27
(1 row)

We could see data copied without any issue.