Not able to start the SAP HANA instance

[Former Member]

Hello

My SAP HANA instance is not coming up.

We are using Fujitsu HANA appliance.

Getting following error:

====================================================

punbnrhana01:HDB:tstadm /usr/sap/TST/HDB00 206> HDB start

StartService

Impromptu CCC initialization by 'rscpCInit'.

  See SAP note 1266393.

OK

OK

Starting instance using: /usr/sap/TST/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function StartWait 2700 2

1. 06.02.2014 17:16:46

Start

OK

1. 06.02.2014 17:20:02

StartWait

FAIL: process hdbdaemon HDB Daemon not running

====================================================

[Former Member]

Thanks to all for reply.

I have created new instance on HANA appliance. previous didn't come up.

[Former Member]

hi,

can u help me to delete the logfiles , to free the memory in hana drive ,

[Former Member]

Hi Sagar,

I get some error, can you tell me how to create new instance beside old instance?

Regards,

Deby Afandi

[Former Member]

Check basepath_datavolumes param in  /hana/shared/SID/global/hdb/custom/config/global.ini or global.ini->persistence->basepath_datavolumes

[Former Member]

Now I went little bit ahead and getting an error as

===============================

Cannot open file "/hana/data/TST/mnt00001/hdb00001/__DO_NOT_TOUCH_FILES_IN_THIS_DIRECTORY__", rc=13: Permission denied

exception throw location:

1: 0x00007f4e0780bd93 in FileAccess::LocalFileFactory::init()+0x280 at LocalFileFactory.cpp:82 (libhdbbasis.so)

2: 0x00007f4e077ecf41 in FileAccess::FileFactory::createInstance(FileAccess::FileFactory::Type, ltt::basic_string<char, ltt::char_traits<char> > const&, int, int, FileAccess::FileFactory::Usage)+0x3a0 at FileFactory.cpp:74 (libhdbbasis.so)

3: 0x00007f4e0a3c418e in PersistenceLayer::createPersistenceFileFactory(ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&)+0x2a at Persistence.cpp:118 (libhdbpersistence.so)

4: 0x00007f4e0a45045a in PersistenceLayer::PersistenceSystem::initialize(NameServer::ServiceStartInfo const&, bool, PersistenceLayer::PERSISTENCE_MODE)+0xa6 at PersistenceSystem.cpp:325 (libhdbpersistence.so)

===============================

Instance is not up yet.

Thanks,

Sagar

[Former Member]

Hello Sagar,

I am also facing the same issue in HANA instance.

Please let me know how the issue was resolved for you?

Thanks

Ankit

[Former Member]

which user you are using ?

are you suing SYSTEM user ?

Regards,

Pavan Gunda

[Former Member]

Hi,

Please check the corresponding trace file daemon_hostname.trc under /usr/sap/SID/Instance/hostname/trace folder for more detail description for the cause.

Regards,

Ram.

[Former Member]

Hi Ram,

I checked in daemon_hotname.trc file which sows failed error as:

=======================================

[5803]{-1}[-1/-1] 2014-02-07 11:38:57.933567 i Daemon           TrexDaemon.cpp(11900) : runlevel 0 completely started

[5803]{-1}[-1/-1] 2014-02-07 11:38:57.933575 i Daemon           TrexDaemon.cpp(10348) : line up of program group nameserver to instances 0.

[5803]{-1}[-1/-1] 2014-02-07 11:38:57.941234 i Daemon           TrexDaemon.cpp(10069) : start 'hdbnameserver' as process 5819

[5803]{-1}[-1/-1] 2014-02-07 11:42:00.730595 i Daemon           TrexDaemon.cpp(07470) : signo 3 SIGQUIT from user errno 0 code 0

[5803]{-1}[-1/-1] 2014-02-07 11:42:00.730624 i Daemon           TrexDaemon.cpp(07470) : sender pid 5819 real user id 1002

[5803]{-1}[-1/-1] 2014-02-07 11:42:00.730633 i Daemon           TrexDaemon.cpp(12013) : got shutdown event (stop)

[5803]{-1}[-1/-1] 2014-02-07 11:42:00.731097 i Daemon           TrexDaemon.cpp(12546) : comment file contains: nameserver: persistence initialization failed

stop process hdbnameserver with pid 5819

[5803]{-1}[-1/-1] 2014-02-07 11:42:00.732469 i Daemon           TrexDaemon.cpp(07207) : stopped child with pid 5819 (5819)

[5803]{-1}[-1/-1] 2014-02-07 11:42:00.786392 i Daemon           TrexDaemon.cpp(12189) : process hdbnameserver with pid 5819 exited normally with status 1

[5803]{-1}[-1/-1] 2014-02-07 11:42:00.786416 i Daemon           TrexDaemon.cpp(12242) : all instances in runlevel 1 stopped

[5803]{-1}[-1/-1] 2014-02-07 11:42:00.788555 i Daemon           TrexDaemon.cpp(11065) : writing started programs file /usr/sap/TST/HDB00/punbnrhana01/lock/started_programs.txt:

[5803]{-1}[-1/-1] 2014-02-07 11:42:00.841528 i Daemon           TrexDaemon.cpp(05630) : cleaning all

=======================================

Then I look into nameserver_hostname.trc file and which shows me error as:

=======================================

try to open existing persistence log volume, but filesystem path does not exists: /hana/log/TST/mnt00001/hdb00001/

exception throw location:

1: 0x00007fb06dc4bd23 in PersistenceController::checkVolumes(NameServer::ServiceStartInfo const*)+0x5d0 at PersistenceController.cpp:283 (libhdblogger.so)

2: 0x00007fb06dc4daf1 in PersistenceController::startup(PersistenceLayer::PERSISTENCE_MODE, NameServer::ServiceStartInfo*, bool, DataAccess::TablePreloadWriteCallback*, DataAccess::TablePreloadReadCallback*)+0xd20 at PersistenceController.cpp:431 (libhdblogger.so)

3: 0x00007fb06e2e4a9b in NameServer::Topology::initPersistence(NameServer::ServiceStartInfo&, bool, bool, TREX_ERROR::TRexError*, bool, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> >, NameServer::ServiceStartInfo::RequestAction)+0x407 at Topology.cpp:279 (libhdbns.so)

4: 0x00007fb06e2e5e99 in NameServer::Topology::initPersistence(bool, bool, TREX_ERROR::TRexError*, bool, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> >, NameServer::ServiceStartInfo::RequestAction)+0x235 at Topology.cpp:195 (libhdbns.so)

5: 0x00007fb06e24bf9f in NameServer::TREXNameServer::loadTopology(NameServer::LoadTopologyMode, NameServer::ServiceStartInfo&, Backup::Backup_ExtendedRecoveryInformation*)+0xfb at TREXNameServer.cpp:11159 (libhdbns.so)

6: 0x00007fb06e26336a in NameServer::TREXNameServer::assign(NameServer::ServiceStartInfo&)+0x2666 at TREXNameServer.cpp:1512 (libhdbns.so)

7: 0x00007fb06e2a3c57 in NameServer::SelfAssignThread::run(void*)+0x13 at TREXNameServer.cpp:283 (libhdbns.so)

8: 0x00007fb06cbe20cf in TrexThreads::PoolThread::run()+0x9fb at PoolThread.cpp:279 (libhdbbasement.so)

9: 0x00007fb06cbe3c38 in TrexThreads::PoolThread::run(void*&)+0x14 at PoolThread.cpp:125 (libhdbbasement.so)

10: 0x00007fb06b2b0dcb in Execution::Thread::staticMainImp(void**)+0x627 at Thread.cpp:475 (libhdbbasis.so)

11: 0x00007fb06b2b0f6d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:543 (libhdbbasis.so)

stopping service...

[5872]{-1}[-1/-1] 2014-02-07 11:42:00.728768 f NameServer       TREXNameServer.cpp(02757) : persistence initialization failed -> stopping instance ...

[5819]{-1}[-1/-1] 2014-02-07 11:42:00.731317 i Service_Shutdown TrexService.cpp(05683) : Preparing for shutting service down

[6054]{-1}[-1/-1] 2014-02-07 11:42:00.735924 i assign           TREXNameServer.cpp(02480) : shutdown, unassign from volume 1

[6054]{-1}[-1/-1] 2014-02-07 11:42:00.736186 i failover         TREXNameServer.cpp(03200) : setinactive(punbnrhana01) but no failover manager

[5872]{-1}[-1/-1] 2014-02-07 11:42:00.738987 e Basis            TREXNameServer.cpp(02787) : Process exited due to an error via explicit exit call with exit code 1 , no crash dump will be written

=======================================

Filesystem path is as expected.

Let me know what is wrong.

Thanks,

Sagar

[Former Member]

Hi Sagar,

Seems like your parameter file is corrupted. Can you please go to the following location and paste the contents of global ini file.

/usr/sap/<DB_Sid>/SYS/global/hdb/custom/config

Thanks

Suresh

[Former Member]

Below are my global.ini details:

========================================

[backup]

log_backup_parameter_file = /hana/shared/TST/profile/sap_parameter_file_dir/initSAP.utl

log_backup_using_backint = true

data_backup_parameter_file = /hana/shared/TST/profile/sap_parameter_file_dir/initSAP.utl

[communication]

listeninterface = .global

[persistence]

basepath_datavolumes = /hana/data/TST

basepath_logvolumes = /hana/log/TST

[storage]

ha_provider = fujitsu_connector_api

ha_provider_path = /opt/fujitsu/bwai/lib

[trace]

ha_fujitsu_connector_api = info

========================================

I am stuck and not able to do anything.

Thanks,

Sagar

[Former Member]

Hi Sagar,

As per the above trace seems  filesystem path does not exists: /hana/log/TST/mnt00001/hdb00001/.

Please check under the directory TST that /mnt00001/hdb00001 exists or not.

If the above directories are missing, DB should be restored from the latest backup with logs.

Regards,

Ram.

[Former Member]

Hi Ram,

Both the things are present there as:

===========================

/hana/log/TST/mnt00001 # ls -alrt

total 8

drwxr-x--- 3 hdbadm sapsys 4096 Jan 30 18:01 ..

drwxr-xr-x 2 tstadm sapsys 4096 Feb 11 14:30 .

-rwxrwxrwx 1 root   root      0 Feb 11 14:30 hdb00001

===========================

We have uninstall the third party backup tools.

But I tried for file based backup and got the error in hana studio.

Please help.

Thanks,

Sagar

[Former Member]

I tried with the command line as:

=================================================

/usr/sap/TST/HDB00 51> HDBSettings.sh recoverSys.py --command "recover data using file ('COMPLETE_DATA_BACKUP') clear log"

[139912255338240, 0.006] >> starting recoverSys (at Thu Feb 13 17:29:54 2014)

[139912255338240, 0.007] args: ()

[139912255338240, 0.007] keys: {'command': "recover data using file ('COMPLETE_DATA_BACKUP') clear log"}

own pid: 61027

recoverSys started: 2014-02-13 17:29:54

testing master: punbnrhana02

punbnrhana02 is master

shutdown database, timeout is 120

stop system

stop system: punbnrhana02

stop system: punbnrhana01

stop system: punbnrhana03

stopping system: 2014-02-13 17:29:54

recoverSys failed: sapcontrol failed:

13.02.2014 17:33:04

StopSystem

FAIL: HTTP error, HTTP/1.1 401 Unauthorized

2014-02-13 17:33:04

[139912255338240, 190.637] 1

[139912255338240, 190.637] << ending recoverSys, rc = 1 (RC_GENERAL_EXCEPTION), after 190.631 secs

=================================================

but it fails.

I tried by using all three nodes, but is giving me same error.

Thanks,

Sagar

[Former Member]

Hi Sagar,

As per the above update directory  hdb00001 is having root:root permission. Please change it to tstadm:sapsys and restart it again. As the system is getting with sidadm.

Regards,

Ram.

[Former Member]

Hi Ram,

I change the ownership as:

=================================

/hana/log/TST/mnt00001 # ls -lart

total 8

drwxrwxrwx 4 root   root   4096 Jul 16  2013 ..

drwxr-xr-x 2 tstadm sapsys 4096 Feb 11 14:30 .

-rwxrwxrwx 1 tstadm sapsys    0 Feb 11 14:30 hdb00001

=================================

try to start the instance, but getting below error:

=================================

/usr/sap/TST/HDB00 53> HDB start

StartService

Impromptu CCC initialization by 'rscpCInit'.

  See SAP note 1266393.

OK

OK

Starting instance using: /usr/sap/TST/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function StartWait 2700 2

13.02.2014 19:59:40

Start

OK

13.02.2014 20:12:11

StartWait

FAIL: process hdbdaemon HDB Daemon not running

=================================

Nameserver is failing with similar error as:

=================================

try to open existing persistence log volume, but filesystem path does not exists: /hana/log/TST/mnt00001/hdb00001/

exception throw location:

1: 0x00007f5081f57d23 in PersistenceController::checkVolumes(NameServer::ServiceStartInfo const*)+0x5d0 at PersistenceController.cpp:283 (libhdblogger.so)

2: 0x00007f5081f59af1 in PersistenceController::startup(PersistenceLayer::PERSISTENCE_MODE, NameServer::ServiceStartInfo*, bool, DataAccess::TablePreloadWriteCallback*, DataAccess::TablePreloadReadCallback*)+0xd20 at PersistenceController.cpp:431 (libhdblogger.so)

3: 0x00007f50825f0a9b in NameServer::Topology::initPersistence(NameServer::ServiceStartInfo&, bool, bool, TREX_ERROR::TRexError*, bool, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> >, NameServer::ServiceStartInfo::RequestAction)+0x407 at Topology.cpp:279 (libhdbns.so)

4: 0x00007f50825f1e99 in NameServer::Topology::initPersistence(bool, bool, TREX_ERROR::TRexError*, bool, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> >, NameServer::ServiceStartInfo::RequestAction)+0x235 at Topology.cpp:195 (libhdbns.so)

=================================

Something went wrong with Instance.

Your help is really appreciated.

[Former Member]

Hello Sagar,

Is this a scale out system or single node HANA ?

Thanks

Suresh

[Former Member]

This is scale out multinode Appliance

1- Master Node

1- Worker node

1- Standby Node

Thanks,

Sagar