SAP Standard issue and reason cased to it.

This document was designed to help SAP customers to troubleshoot and resolve the majority of SAP system startup problems own their own, minimizing the business impacts and costs attached to a standard support ticket life cycle. As time goes by, new topics and startup error scenarios will be added to this guide. Stay tuned!

1.1: IS DATABASE UP AND RUNNING?

This is *absolute* prerequisite for SAP SYSTEM to start. In order to test this, operating system command "R3trans -d" can be executed (as SIDadm user). Any return code different than 0 means that DB cannot be reached by DIALOG work processes, and so SAP system will not initiate. The following return indicates that database is up, running and reachable by SAP work processes:

<hostname>:SIDadm > R3trans -d

This is R3trans version X.xx (release 742 - 18.11.14 - 20:14:09).
unicode enabled version

R3trans finished (0000).

*** There is NO POINT in moving the analysis further while the return code is different than zero.

*** In case the return code is different than zero, please involve your DBA/Network/OS teams.

----------------------------------------------------------------------------------------------------------------------------------

1.2: IS SAP STARTUP SERVICE CORRECTLY DEFINED?

UNIX:

File "/usr/sap/sapservices" must contain the correct service definitions.

LD_LIBRARY_PATH=/usr/sap/<SID>/exe:$LD_LIBRARY_PATH;                                                                    
export LD_LIBRARY_PATH;                                                                                                 

/usr/sap/<SID>/<instance_nr>/exe/sapstartsrv pf=/usr/sap/<SID>/SYS/profile/<SID>_<instance_nr>_<hostname> -D -u <SIDadm>

WINDOWS:

 I. Run "services.msc";

 II. Look for the relevant service SAP<SID>_<instance_nr>;

 III. Check the command line assigned to it:

<DRIVE>:\usr\sap\<SID>\<instance>\exe\sapstartsrv pf=<DRIVE>:\usr\sap\<SID>\SYS\profile\<START_PROFILE>

----------------------------------------------------------------------------------------------------------------------------------

1.3: [START] PROFILES:

In order to correctly start the required processes such as message server, enqueue server and the disp+work, the SAP startup agent (sapstartsrv) needs to find a correct scenario for that. This scenario consists on a list of "Start_Program_<xx>" commands that should be present an the "START" profile (for all kernel releases up to 740) or at the "INSTANCE" profile (kernel release 740 onwards).

*** These "start" entries are mandatory in the profile being used by sapstartsrv process.

You can use the process list on UNIX and the service definition at WINDOWS to find out what is the exactly profile currently assigned to sapstartsrv process.

[EXAMPLES:]

Run " $ps -ef | grep sapstartsrv

•   Entries needed for starting the dispatcher + work processes: 

_DW=dw.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)
Execute_0X=local rm -f $(_DW)

Execute_0X=local ln -s -f $(DIR_EXECUTABLE)/disp+work$(FT_EXE) $(_DW)

Start_Program_0X=local $(_DW) pf=$(_PF)

• Example of message server start entries:

_MS = ms.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)
Execute_0X = local rm -f $(_MS)

Execute_0X = local ln -s -f $(DIR_EXECUTABLE)/msg_server$(FT_EXE) $(_MS)

Restart_Program_0X = local $(_MS) pf=$(_PF)

• Enqueue server related start commands:

_EN = en.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)
Execute_0X = local rm -f $(_EN)

Execute_0X = local ln -s -f $(DIR_EXECUTABLE)/enserver$(FT_EXE) $(_EN)

Restart_Program_0X = local $(_EN) pf=$(_PF)

--------------------------------------------------------------------------------------------------------------------------

1.4: LOGGING IN OS LEVEL DIRECTLY AS <SIDadm> USER:

        It is mandatory that SIDadm user should be able to log on at operating system level *WITHOUT ANY* other prompts such as:

1. Company policies;
2. User policies;
3. Next scheduled downtimes/System news;
4. Logging in as another user and SUdoing to SIDadm;

This can cause several problems throughout different scenarios, but mainly on upgrade procedures. Further details on SAP note 1301712.

#1301712 - UNIX sapstartsrv: Determining the environment

--------------------------------------------------------------------------------------------------------------------------

1.5: IS SAP STARTUP AGENT (SAPSTARTSRV) RUNNING?

There must be *ONLY ONE* sapstartsrv process running for each instance. Examples for each platform below:

UNIX:

As an example, we will use a centralized system with the following 03 instances running on the same host (ASCS, PAS and DI). The following output should return for the operating system level command:

[ ps -ef | grep <SID> | grep sapstartsrv ]

(...) /usr/sap/<SID>/ASCS00/exe/sapstartsrv   pf=/usr/sap/<SID>/SYS/profile/<SID>_ASCS01_<hostname> -D      
(...) /usr/sap/<SID>/DVEBMGS01/exe/sapstartsrv   pf=/usr/sap/<SID>/SYS/profile/<SID>_DVEBMGS00_<hostname> -D
(...) /usr/sap/<SID>/D02/exe/sapstartsrv   pf=/usr/sap/<SID>/SYS/profile/<SID>_D02_<hostname> -D            

• Profile pattern names returned with the command can be such as "START_<INSTANCE>_<hostname>" in older releases.

• There can be another sapstartsrv process (as below) but it will play no parts in the startup of the instance.
• When sapstartsrv service is not running you could run manually the above command for the desired instance
  If the service doesn't start and nothing it's written in "sapstartsrv.log" it might happen the binaries are damaged in the local exe folder. In this case has to be replaced

(...) /usr/sap/hostctrl/exe/sapstartsrv pf=/usr/sap/hostctrl/exe/host_profile -D

WINDOWS:

I. You shoud first configure task manager output so it displays the command line related to the selected process: 
Execute task manager >> Switch to tab "Processes" >> Menu "view" >> Select "columns" >> Select column 'PID' and 'COMMAND LINE';

II. Then using task manager tab "Processes", look for the instance sapstartsrv process with similar:

<DRIVE>:\usr\sap\<SID>\<instance>\exe\sapstartsrv pf=<DRIVE>:\usr\sap\<SID>\SYS\profile\<START_PROFILE>

--------------------------------------------------------------------------------------------------------------------------

1.6: IS ANY SAP PROCESS ALREADY RUNNING?

After trying to start the instance (either through the SAP MMC, the "startsap" command or using "sapcontrol ... -function Start"), no error is returned.

However, the SAP processes (Dispatcher - ABAP; jstart - Java) do not start.

No trace file is updated either.

Verify whether any process is already in "GREEN" or "YELLOW" status (e.g., the IGS).

hostname:sidadm 40> sapcontrol -nr 33 -function GetProcessList

12.10.2015 20:15:45
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
msg_server, MessageServer, GRAY, Stopped, , , 1234
disp+work, Dispatcher, GRAY, Stopped, , , 1111
igswd_mt, IGS Watchdog, GREEN, Running, 2015 07 31 11:59:15, 1:21:09, 2323
hostname:sidadm 41>

All processes must have the "GRAY" status. Otherwise, the SAP Startup Agent (sapstartsrv) will "ignore" the start command.

Stop the instance first, so all processes are stopped. Then, start the instance again.

--------------------------------------------------------------------------------------------------------------------------

1.7: PERMISSIONS:UNIX:

I. "/TMP" DIRECTORY:

"/tmp" directory's permissions must match the following. The final 't' for "temporary" is of ultimate relevance and should not be overlooked:

drwxrwxrwt  22 root   root  4096 Apr  6 20:16 tmp 

927637: Web service authentication in sapstartsrv as of Release 7.00

II. SAPSTARTSRV.LOG:

"/usr/sap/<SID>/<INSTANCE>/work/"<SIDadm>

"Unable to open trace file sapstartsrv.log. (Error 13 Permission denied)"


III. SAPUXUSERCHECK FILE: 

-rwsr-x       1 root     sapsys      1293660 Apr 06 2015  sapuxuserchk 

 There is one SAPUXUSERCHECK file per instance (/usr/sap/<SID>/<instance>/exe) and the permissions and ownership should look alike for all files. The following command can help tracking the files:

find /usr/sap/<SID>/* -name sapuxuserchk

 -------------------------------------------------------------------------------------------------------------------------

1.8: IP / HOSTNAME RESOLUTION:

LOCALHOST:

startup commands from sapcontrol will be using "localhost" implicitly (when no "-host" option is explicitly informed) and for such the loopback IP "127.0.0.1" should be resolved for it. What is the output for "niping -v -H localhost"? The loopback IP should be the only one returned. If not, please involve your network team.

Lookup of hostname: localhost (NiHostToAddr) 
--> IP-Addr.: 127.0.0.1 
Lookup of IP-Addr.: 127.0.0.1 (NiAddrToHost) 
--> Hostname: localhost

1476386 - Wrong localhost name resolution setting

HOSTNAME:

What IP is resolved for the local hostname (output of OS level command niping -v -H <hostname>)? There is a known error described in SAP note #1054467 where the loopback IP 127.0.0.1 is wrongly resolved for it.

1054467 - Local host name refers to loopback address

----------------------------------------------------------------------------------------------------------------------------------