Virtual Box 5 Broke My 12c RAC

After updating to Virtual Box 5 I noticed my 3 node demo RAC cluster had problems starting cluster resources. Any time I do any sort of upgrade with Virtual Box I cycle through my vm’s to make sure they are all working so I don’t have any nasty surprises whenever I need to use them.

After the upgrade, I started up the first node of the cluster and waited, came back to it a few minutes later and noticed no resources running and no crs. I tried to start it manually with crsctl start crs but it just hung for a while and nothing was happening. Next thing to do was to strace it and see if anything useful came up.

The segmentation fault on the /u01/app/12.1.0.2/grid/perl/bin/perl binary gave me a clue. The Clusterware scripts are heavy users of perl so there may be something wrong with the perl version that exists in the grid home. Any use of perl from the clusterware home caused a segmentation fault.

The default OS version of perl for OEL 6.6 was working ok but that is version is 5.10 while the clusterware perl installation is 5.14

Nevertheless I thought I’d try and start the cluster with it so I did the following:

This worked enough for the cluster to start and all resources were up. Unfortunately this is not a really good bandaid fix as the clusterware home also uses a number of perl modules which are tied to perl 5.14. So with my temp workaround if you try running something like asmcmd you’ll see the following error:

The best solution was to download perl 5.14 and then link it and copy into the clusteware home. Luckily Laurent Leturgez had hit a similar problem in Oracle 12c, VMWare Fusion and the perl binary’s segmentation fault so I had a pretty good guide as to how to get it done.

Once I got the perl compiled and copied to every node I cycled the servers with a reboot and all was well again.

Update

Thanks to Simon Coter, this is a bug in the perl binary as can be seen in the ticket here Oracle 12.1.0.2 Grid Installation fails to relink VB5 on Oracle Linux 6.7 guest,OSX host.

Leave a Reply

Your email address will not be published. Required fields are marked *

CAPTCHA * Time limit is exhausted. Please reload CAPTCHA.