Skip to main content

::Pg_dump With Load Balance::

Hi ,

Find the below steps to do pg_dump with load balance in PostgreSQL.

REQUIRED TABLES ON REMOTE:- [pg_dump is doing remotely]
==============================
CREATE TABLE LOAD_AVG(AVG REAL);
INSERT INTO LOAD_AVG VALUES(0.0);

REQUIRED SHELL SCRIPT FOR EVERY ONE MINUTE ON REMOTE:-
=======================================================
PGBIN="/opt/PostgreSQL/9.0/bin"
PORT=5432
USER="postgres"
PASSWORD="postgres"
DATABASE="postgres"
LOAD=$(cat /proc/loadavg|awk -F ' ' '{print $1}')
$PGBIN/psql -p $PORT -U $USER -d $DATABASE -c "UPDATE LOAD_AVG SET AVG='$LOAD'";


REQUIRED C PROGRAM ON LOCAL:-
================================
#include
#include
#include
#include
#include
#include
#include "/opt/PostgreSQL/9.0/include/libpq-fe.h"
int main(int argc,char *argv[])
{
FILE *fp,*remote_machine,*log;
char pid[10],cmd[30],con_string[110],host[40],port[4],dbname[20],user[20],password[20],pg_dump_cmd[1000];
int status,remote_avg,dump_initiated=0,dump_stopped=0;
PGconn *conn;
PGresult *res;
log=fopen("/tmp/remote_dump.log","w");
if(fp==NULL)
{
fprintf(log,"\n%s","remote_credentials file is not available");
return 1;
}
else
if(argc<5) { printf("\nProvide the following Arguments "); printf("\nhostaddress port dbname user password"); return 1; } sprintf(con_string,"hostaddr=%s port=%s dbname=%s user=%s password=%s",argv[1],argv[2],argv[3],argv[4],argv[5]); conn = PQconnectdb(con_string); if (PQstatus(conn) == CONNECTION_BAD) { fprintf(log,"\n Not able to connect to the database %s",PQerrorMessage(conn)); return 1; } while(1) { res=PQexec(conn,"BEGIN"); PQclear(res); res=PQexec(conn,"DECLARE find_cpu_load CURSOR FOR select round(avg) from public.load_avg"); if (PQresultStatus(res) != PGRES_COMMAND_OK) { fprintf(stderr, "Erorr %s", PQerrorMessage(conn)); PQclear(res); } PQclear(res); res = PQexec(conn, "FETCH ALL in find_cpu_load"); remote_avg=atoi(PQgetvalue(res,0,0)); PQclear(res); res=PQexec(conn,"END"); PQclear(res); if(!dump_initiated) { if(remote_avg<10) { fp=popen("/sbin/pidof -s pg_dump","r"); fgets(pid,10,fp); pid[strlen(pid)-1]='\0'; pclose(fp); if(strlen(pid)<2) { sprintf(pg_dump_cmd,"/opt/PostgreSQL/9.0/bin/pg_dump -f /opt/PostgreSQL/9.0/data/dump.sql -v -h %s -p %s -U %s %s >/tmp/remote_dump1.log 2>/tmp/remote_dump1.log &",argv[1],argv[2],argv[3],argv[4],argv[5]);
system(pg_dump_cmd);
fp=popen("/sbin/pidof -s pg_dump","r");
fgets(pid,10,fp);
pid[strlen(pid)-1]='\0';
pclose(fp);
dump_initiated=1;
}
while (waitpid(-1, &status, WNOHANG) > 0);
}
else
{
fprintf(log,"\nHigh Load Average .. So, Con't start pg_dump ... Load is %d",remote_avg);
PQfinish(conn);
fclose(log);
return 1;
exit(0);
}
}
else
{
fp=popen("/sbin/pidof -s pg_dump","r");
fgets(pid,10,fp);
pid[strlen(pid)-1]='\0';
pclose(fp);
if(strlen(pid)<2) { fprintf(log,"\nDump has completed .. hence closing the connections ... "); PQfinish(conn); return 1; } if(remote_avg<10&&!dump_stopped) { fprintf(log,"\nSleeping 10 seconds pid load is normal ==> %d",remote_avg);
sleep(10);
}
else if(remote_avg>10&&!dump_stopped)
{
fprintf(log,"\nSleeping 10 Seconds ****Load is high ****.. so stopping dump and Load is %d Processes is %d",remote_avg,atoi(pid));
sleep(10);
dump_stopped=1;
sprintf(cmd,"kill -s STOP %d",atoi(pid));
system(cmd);
}
else if(remote_avg>10&&dump_stopped)
{
fprintf(log,"\nSleeping 10 More Seconds Load is high ... Load is %d",remote_avg);
sleep(10);
}
else
{
fprintf(log,"\nLoad is Nomal Again so resumig the dump processes ... load is %d",remote_avg);
dump_stopped=0;
sprintf(cmd,"\nkill -s CONT %d",atoi(pid));
system(cmd);
}
}
while (waitpid(-1, &status, WNOHANG) > 0 );
}
fclose(fp);
}

HOW TO COMPILE ??
===============
1.export LD_LIBRARY_PATH=/opt/PostgreSQL/9.0/lib/
2.gcc -o mg MG.c -L/opt/PostgreSQL/9.0/lib -lpq -lssl -lcrypto

HOW TO RUN ??
============
./mg 172.24.35.67 5432 postgres postgres postgres
[host] [port] [user] [password] [database]

Comments

Popular posts from this blog

Parallel Operations With pl/pgSQL

Hi, I am pretty sure that, there will be a right heading for this post. For now, i am going with this. If you could suggest me proper heading, i will update it :-) OK. let me explain the situation. Then will let you know what i am trying to do here, and how i did it. Situation here is, We have a table, which we need to run update on “R” no.of records. The update query is using some joins to get the desired result, and do update the table.  To process these “R” no.of records, it is taking “H” no.of hours. That too, it’s giving load on the production server. So, we planned to run this UPDATE as batch process.  Per a batch process, we took “N” no.or records. To process this batch UPDATE, it is taking “S” no.of seconds. With the above batch process, production server is pretty stable, and doing great. So, we planned to run these Batch updates parallel.  I mean, “K” sessions, running different record UPDATEs. Of-course, we can also increase the Batch size here.  But

How To Send E-Mail From PostgreSQL

Hi , If you want to send E-Mails from PostgreSQL, then use the below Python 3.2 Script as below. I have used ActivePython 3.2 with PostgreSQL 9.1 for sending E-Mails from PostgreSQL. If you want to configure the Python 3.2 with PostgreSQL 9.1 then, please refer the below steps. http://manojadinesh.blogspot.in/2012/06/fatal-python-error-pyinitialize-unable.html Once, your Python 3.2 successful then follow the below steps to send an e-mail. Step 1 ===== postgres=# CREATE OR REPLACE FUNCTION public.send_email(_from Text,_password Text,smtp Text,port INT,receiver text, subject text, send_message text) RETURNS TEXT  LANGUAGE plpython3u AS $function$ import smtplib sender = _from receivers = receiver message = ("From: %s\nTo: %s\nSubject: %s\n\n %s"  % (_from,receiver,subject,send_message)) try:   smtpObj = smtplib.SMTP(smtp,port)   smtpObj.starttls()   smtpObj.login(_from, _password)   smtpObj.sendmail(sender, receivers,message)   print ('Successf

::Pipelined in Oracle as well in PostgreSQL::

Pipelined Table Functions:- [ORACLE] =========================== If you want to return multiple rows to the calling environment, then piplined table functions is prefred. It will increase the dbperformance as well. Ex:- Step 1: ----------- CREATE TABLE EMP(EMPNO INT,ENAME VARCHAR2(10),SAL INT); Step 2: ----------- Insert sample data. Step 3: ----------- Create an object for the row type casting. CREATE OR REPLACE TYPE emp_row AS OBJECT ( empno INT, ename VARCHAR2(20), SAL INT ); Step 4: ----------- Create a Return Type for the pipelined function. CREATE OR REPLACE TYPE emp_table_type AS TABLE OF emp_row; Step 5: ----------- CREATE OR REPLACE FUNCTION emp_pipe_function RETURN emp_table_type PIPELINED IS BEGIN FOR rec in (select * from emp) LOOP PIPE ROW (emp_row(rec.empno,rec.ename,rec.sal)); END LOOP; RETURN; END; Step 6: ---------- SQL> select * from table(emp_pipe_function); EMPNO ENAME SAL ---------- ----