Problem: Nbpem may crash after changing policy schedule type from Calendar to Frequency

Solution: The workaround is to be aware of the appropriate values in the schedule. Ultimately, delete the policy to get nbpem to load.

Here is a step by step recap:

8 RMAN jobs were failing with status 25 (cant connect) connection errors while bprd connected successfully. Found no transmission control protocol, host resolution, or network issues.

RMAN log showed that it was failing after sending policy info

dbclient

0:13:09.334 [3020] <2> async_connect: [vnet_connect.c:1477] connect in progress 1 0x1

00:13:09.334 [3020] <2> vnet_pbxConnect: pbxConnectEx Succeeded

00:13:09.334 [3020] <2> do_pbx_service: [vnet_connect.c:2116] via PBX bprd CONNECT FROM 10.162.244.101.61071 TO 10.162.244.242.1556 fd = 15

00:13:09.334 [3020] <2> async_connect: [vnet_connect.c:1644] connect async CONNECT FROM 10.162.244.101.61071 TO 10.162.244.242.1556 fd = 15

00:13:09.334 [3020] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO server01.domain.com 10.162.244.242 bprd VIA pbx

00:13:09.335 [3020] <2> logconnections: BPRD CONNECT FROM 10.162.244.101.61071 TO 10.162.244.242.1556 fd = 15

00:13:09.335 [3020] <2> vnet_check_vxss_client_magic_with_info: [vnet_vxss_helper.c:833] VxSS not supported 0 0x0

00:13:09.335 [3020] <4> serverResponse: entering serverResponse.

00:13:09.335 [3020] <4> serverResponse: initial client_read_timeout = <1800>

00:13:09.335 [3020] <4> readCommMessages: Entering readCommMessages

00:13:10.335 [3020] <4> serverResponse: read comm file:<00:13:09 Initiating backup>

00:13:10.335 [3020] <4> serverResponse: read comm file:<00:13:10 INF – Server status = 25>

00:13:10.335 [3020] <16> serverResponse: ERR – server exited with status 25: cannot connect on socket

00:13:10.335 [3020] <16> CreateNewImage: ERR – serverResponse() failed

00:13:10.335 [3020] <4> closeApi: entering closeApi.

00:13:10.335 [3020] <4> closeApi: INF – EXIT STATUS 6: the backup failed to back up the requested files

A shared memory and nbpem error was in the messages log that started intermittently at the same time the rman jobs started failing.

May  4 15:52:45 server01 SQLAnywhere(nb_servers01): Disconnecting shared memory client, process id not found

May  4 15:52:45 server01 SQLAnywhere(nb_servers01): Disconnected SharedMemory client’s AppInfo: IP=192.168.229.233;HOST=server01.domain.com;OSUSER=root;OS=’Linux 2.6.32-504.30.3.el6.x86_64 #1 SMP Thu Jul 9 15:20:47 EDT 2015 x’;EXE=/usr/openv/netbackup/bin/nbpem;PID=0x2f92;THREAD=0x7f51c5a17700;VERSION=16.0.0.2184;API=ODBC;TIMEZONEADJUSTMENT=-360

After rebooting the appliance, we discovered a NBPEM process coredumping with an assertion error in the nbpem log.

Cleaned NBPEM cache: Procedure for clearing the NetBackup Policy Execution Manager (nbpem) cache on NetBackup 7.x

Then discovered this technote regarding NBPEM assertion errors causing status 25 (cant connect): NetBackup 7.1 NBPEM process is core dumping on start with assertion error after LastBackupData::getWindowTimes and anything connecting to nbpem gets status 25.

Disabled poilicies and restarted services and nbpem stayed up.

One of the NBU admins admitted to copying a policy and a schedule issue at the time the nbpem shared memory error started.

We deleted the corrupt policy; restarted services; restarted jobs.

For more information, go to http://www.veritas.com/docs/000125511

Contact VAST IT Services to learn more about Veritas.