Merging VMWare files to single file

May 27, 2010 Leave a comment

A few days back I had to use a VMWare image, being not really a fan of VMWare I wanted to use it with VirtualBox (http://www.virtualbox.org/) and found out that .vmdk files from VMWare are supported in VirtualBox which was a sigh of relief but then got hit with another requirement…

VirtualBox supports single VMWare .vmdk file and the VM I had was using multiple small vmdk files which means it will not work in VirtualBox.

Next step for me was to find a way if possible to convert those multiple files into a single “big” .vmdk file that I can use with VirtualBox and that was in the end a single VMWare commad from the folder where I had all the small vmdk files:

# vmware-vdiskmanager -r smallfiles.vmdk -t 0 bigsinglefile.vmdk
Creating disk 'bigsinglefile.vmdk'
Convert: 100% done.
Virtual disk conversion successful.
….Once this was done it was just a matter of adding this new .vmdk file as a disk in VirtualBox and booting a VM from that.
Shoaib Mir
shoaibmir[@]gmail.com
Categories: Operating Systems Tags: , ,

Reading XML file using shell script

May 2, 2010 2 comments

This was like the first time where I had to write something that will be able to read something out of a XML file using a shell script. Usually I would use Python/Perl as my favorite choices in such a scenario but in this one I really *had* to do all within a shell script.

This is an example of the type of XML file I had to read:

<Logentry>
<changelog>Fixed a new bug</changelog>
<name>Shoaib Mir</name>
<date>Sun, 02 May 2010</date>
<email>shoaibmir[@]gmail.com</email>
</Logentry>

I ended up having a shell script like this:

#!/bin/bash

#Looking for four keywords in here
for key in changelog name date email
do
OUTPT=`grep $key log.xml | tr -d '\t' | sed 's/^\([^<].*\)$/\1/' `
eval ${key}=`echo -ne \""${OUTPT}"\"`
done

# Getting the results in four specific arrays
changelogarr=( `echo ${changelog}` )
namearr=( `echo ${name}` )
datearr=( `echo ${date}` )
emailarr=( `echo ${email}` )

#Print all Arrays
echo ${changelogarr[@]}
echo ${namearr[@]}
echo ${datearr[@]}
echo ${emailarr[@]}

Which gives me an output:

shoaib@shoaib-desktop:~/Desktop$ ./readxml.sh
Fixed a new bug Shoaib Mir
Sun, 02 May 2010
shoaibmir[@]gmail.com


Shoaib Mir
shoaibmir[@]gmail.com

Using SchemaSpy for DB diagrams

May 1, 2010 10 comments

A couple of months back I came across this very useful tool “SchemaSpy” ( http://schemaspy.sourceforge.net/ ) which really helped me out in generating automated (nightly) database diagrams but never got enough time to share it with everyone.

“SchemaSpy is a Java-based tool that analyzes the metadata of a schema in a database and generates a visual representation of it in a browser-displayable format. It lets you click through the hierarchy of database tables via child and parent table relationships as represented by both HTML links and entity-relationship diagrams. It’s also designed to help resolve the obtuse errors that a database sometimes gives related to failures due to constraints”

It works great with PostgreSQL and very easy to cron as you just need to execute a single command like this:

java -jar schemaSpy.jar -t dbType -db dbName [-s schema] -u user [-p password] -o outputDir

…that will generate very nice and user-friendly DB diagrams and a whole lots of information about the database. You can have a quick look at the sample on http://schemaspy.sourceforge.net/sample/

Update: Also found an easy to use GUI for SchemaSpy http://www.joachim-uhl.de/projekte/schemaspygui/ where you don’t need to remember all the option but you can easily pick them up from the frontend.

Make sure you grab the latest beta from http://schemaspy.sourceforge.net/schemaSpy.jar as mentioned by John Currier.


Shoaib Mir
shoaibmir[@]gmail.com

Categories: PostgreSQL Tags: ,

Application lock file implementation

January 23, 2010 Leave a comment

Its always a good practice to put a check in the code to see if another instance of the same app is running.

This is helpful in case you have a cron job that is called every hour and single execution of an app can take more then a hour, in this case if we dont have a check in place then that means we can have more then one instance of the same at one time.

Recently I had to add the same kind of check for one of my Python scripts, sharing that if it might help someone trying to do the same:

if os.access(os.path.expanduser("~/.application.lock"), os.F_OK):
       lck_fl = open(os.path.expanduser("~/.application.lock"), "r")
       lck_fl.seek(0)
       oldpid = lck_fl.readline()
       if os.path.exists("/proc/%s" % oldpid):
           print "lck_fl exist! process is already running"
           print "Process is already running as pid %s," % oldpid
           sys.exit(1)
       else:
           print "lck_fl exists but no PID found as current process"
           print "Removing lck_fl of application process: %s" % oldpid
           os.remove(os.path.expanduser("~/.application.lock"))
lck_fl = open(os.path.expanduser("~/.application.lock"), "w")
lck_fl.write("%s" % os.getpid())
lck_fl.close

Shoaib Mir
shoaibmir[@]gmail.com

Categories: Python Tags: ,

Printing PDF file with images

January 23, 2010 Leave a comment

A few days back I was having a terrible time when I tried to print a few tickets that I had in a PDF document. It all started since I upgraded my desktop Ubuntu to 9.1, didnt have time at that moment so asked a colleague to do the print for me.

Once I got time, after a little bit of googling found out that it was a package that I was missing which is needed for the rendering of images and that needs to be installed before I go on printing PDF docs with images in them.

It was:

poppler-data

Hope it helps someone else who is going through the same pain of not able to print out PDF docs 🙂


Shoaib Mir
shoaibmir[@]gmail.com

Categories: Operating Systems Tags: , ,

Using pg_hba.d folder for HBA rules

January 23, 2010 Leave a comment

Its been quite a while since I did post something and that was due to a very busy schedule at work but got lucky today to get a few mins and sharing something I always wanted to write about.

We got an environment at work where we have a quite a huge number of databases on each PostgreSQL development database server and it is because of the static images of the production DBs we need to create for QA and Dev most of the time. In such a scenario if we keep on adding rules to the pg_hba.conf file it can be a big mess with around 200-300 lines in there and thats a nightmare to manage for a DBA.

Solution – What we ended up doing for this was by having a folder pg_hba.d like this:


pg_hba.conf
pg_hba.d
postgresql.conf
postmaster.opts

and under pg_hba.d folder:


host1-dbuser1-trust.db
host2-dbuser2-md5.db

The above two files then contain the list of databases that need to given access for those hosts.


host1-dbuser1-trust.db:
db1
db2
db3

This way you can have host files for each of your staging servers that are used by QA and then have a list of databases that need to be given access from those.

Last step is adding details for using these pg_hba.d files, this can be done in pg_hba.conf file the following way:


host @pg_hba.d/host1-dbuser1-trust.db dbuser1 192.168.2.5/32 trust
host @pg_hba.d/host2-dbuser2-trust.db dbuser2 192.168.2.6/32 md5

This way instead of having the whole list of databases in pg_hba.conf file we now just have two entries and making it very easy to manage. When ever you need to give a new database access to those hosts just add it to the specific file for that host under pg_hba.d folder and reload the DB server.


Shoaib Mir
shoaibmir[@]gmail.com

Categories: PostgreSQL Tags: , ,

PID lock file in python

December 14, 2009 2 comments

I have a python script which when running on large disks can (rarely) run on for like more then a hour, and then I have got this cron setup which runs this script and the cron is set to run every hour. This means I can have several instances of the same Python script at one time. In order to get rid of that kind of scenario I did basic style of pid lock file implementation today:

#This is to check if there is already a lock file existing#
if os.access(os.path.expanduser("~/.lockfile.vestibular.lock"), os.F_OK):
        #if the lockfile is already there then check the PID number 
        #in the lock file
        pidfile = open(os.path.expanduser("~/.lockfile.vestibular.lock"), "r")
        pidfile.seek(0)
        old_pd = pidfile.readline()
        # Now we check the PID from lock file matches to the current
        # process PID
        if os.path.exists("/proc/%s" % old_pd):
                print "You already have an instance of the program running"
                print "It is running as process %s," % old_pd
                sys.exit(1)
        else:
                print "File is there but the program is not running"
                print "Removing lock file for the: %s as it can be there
                because of the program last time it was run" % oldpid
                os.remove(os.path.expanduser("~/.lockfile.vestibular.lock"))

#This is part of code where we put a PID file in the lock file
pidfile = open(os.path.expanduser("~/.lockfile.vestibular.lock"), "w")
pidfile.write("%s" % os.getpid())
pidfile.close

Hope this can help someone doing the same.


Shoaib Mir
shoaibmir[@]gmail.com

Categories: Python Tags: , ,