{"id":326,"date":"2010-10-09T09:26:00","date_gmt":"2010-10-09T09:26:00","guid":{"rendered":"http:\/\/zensonic.dk\/?p=326"},"modified":"2017-04-09T10:03:43","modified_gmt":"2017-04-09T10:03:43","slug":"debugging-thread-exhaustion-on-hp-ux","status":"publish","type":"post","link":"https:\/\/zensonic.dk\/?p=326","title":{"rendered":"Debugging thread exhaustion on HP-UX"},"content":{"rendered":"<p>Thread exhaustion on an HP-UX machine manifests itself by one or more of the following errors in<\/p>\n<pre>\/var\/adm\/syslog\/syslog.log<\/pre>\n<pre style=\"padding-left: 30px;\">vmunix: kthread: table is full\r\nvmunix: WARNING: hponc_thread_create(): error creating thread for autofskd (12)\r\nsshd[8474]: fatal: fork of unprivileged child failed\r\nsshd[1895]: error: fork: Resource temporarily unavailable<\/pre>\n<p>Keywords being thread and fork and failed.\u00c2\u00a0 You should immediatly look at nkthread with kcusage<\/p>\n<pre style=\"padding-left: 30px;\">sudo kcusage nkthread\r\nTunable\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 Usage \/ Setting\r\n=============================================\r\nnkthread\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 5254 \/ 4096\r\n<\/pre>\n<p>You then get a descrption with<\/p>\n<pre>kctune<\/pre>\n<pre style=\"padding-left: 30px;\">sudo kctune -v nkthread\r\nTunable\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 nkthread\r\nDescription\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 Maximum number of threads on the system\r\nModule\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 pm_proc\r\nCurrent Value\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 4096\r\nValue at Next Boot\u00c2\u00a0 4096\r\nValue at Last Boot\u00c2\u00a0 4096\r\nDefault Value\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 8416\r\nConstraints\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 nkthread &gt;= 200\r\n nkthread &lt;= 4194304\r\n nkthread &gt;= max_thread_proc\r\n nkthread &gt;= nproc + 100\r\n nkthread &gt;= (5 * vx_era_nthreads)\r\nCan Change\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 Immediately or at Next Boot\r\n<\/pre>\n<p>You then resolve the problem with ie<\/p>\n<pre style=\"padding-left: 30px;\">sudo kctune nkthread+=8192<\/pre>\n<p>After that you would, as a good sysadmin start to look at the usage back in time. Mind you that the percentage you see is relative to the new tunable value you just set a moment ago, not what it was at the time of the measurement back in time!<\/p>\n<pre style=\"padding-left: 30px;\">sudo kcusage -m nkthread\r\nTunable:\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 nkthread\r\nSetting:\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 28051\r\nTime\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 Usage\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 %\r\n=============================================\r\nThu 09\/09\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6285\u00c2\u00a0\u00c2\u00a0 22.4\r\nFri 09\/10\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6403\u00c2\u00a0\u00c2\u00a0 22.8\r\nSat 09\/11\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6368\u00c2\u00a0\u00c2\u00a0 22.7\r\nSun 09\/12\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6150\u00c2\u00a0\u00c2\u00a0 21.9\r\nMon 09\/13\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6336\u00c2\u00a0\u00c2\u00a0 22.6\r\nTue 09\/14\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6436\u00c2\u00a0\u00c2\u00a0 22.9\r\nWed 09\/15\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6382\u00c2\u00a0\u00c2\u00a0 22.8\r\nThu 09\/16\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6416\u00c2\u00a0\u00c2\u00a0 22.9\r\nFri 09\/17\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6277\u00c2\u00a0\u00c2\u00a0 22.4\r\nSat 09\/18\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6157\u00c2\u00a0\u00c2\u00a0 21.9\r\nSun 09\/19\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6203\u00c2\u00a0\u00c2\u00a0 22.1\r\nMon 09\/20\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6319\u00c2\u00a0\u00c2\u00a0 22.5\r\nTue 09\/21\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6420\u00c2\u00a0\u00c2\u00a0 22.9\r\nWed 09\/22\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6306\u00c2\u00a0\u00c2\u00a0 22.5\r\nThu 09\/23\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6474\u00c2\u00a0\u00c2\u00a0 23.1\r\nFri 09\/24\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6567\u00c2\u00a0\u00c2\u00a0 23.4\r\nSat 09\/25\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6452\u00c2\u00a0\u00c2\u00a0 23.0\r\nSun 09\/26\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 5910\u00c2\u00a0\u00c2\u00a0 21.1\r\nMon 09\/27\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 8260\u00c2\u00a0\u00c2\u00a0 29.4\r\nTue 09\/28\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 8240\u00c2\u00a0\u00c2\u00a0 29.4\r\nWed 09\/29\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6617\u00c2\u00a0\u00c2\u00a0 23.6\r\nThu 09\/30\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6461\u00c2\u00a0\u00c2\u00a0 23.0\r\nFri 10\/01\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 5799\u00c2\u00a0\u00c2\u00a0 20.7\r\nSat 10\/02\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 5558\u00c2\u00a0\u00c2\u00a0 19.8\r\nSun 10\/03\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 5892\u00c2\u00a0\u00c2\u00a0 21.0\r\nMon 10\/04\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6983\u00c2\u00a0\u00c2\u00a0 24.9\r\nTue 10\/05\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6542\u00c2\u00a0\u00c2\u00a0 23.3\r\nWed 10\/06\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 6479\u00c2\u00a0\u00c2\u00a0 23.1\r\nThu 10\/07\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 12289\u00c2\u00a0\u00c2\u00a0 43.8\r\nFri 10\/08\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 11108\u00c2\u00a0\u00c2\u00a0 39.6\r\nSat 10\/09\/10\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 5292\u00c2\u00a0\u00c2\u00a0 18.9\r\n<\/pre>\n<p>Now you might be able to see what happend when and correlate it with your Change Management procedures to figure out what went wrong. I was not that lucky. This was on a database hotel consisting of 80 databases and database related applications and nearly no change control. So what to do? I needed to correlate the process information visible with ps with a thread. But how did you do that in HP-UX.<\/p>\n<p>First guess, the alway valuable tool glance. And lo and behold the capital Z will show you the thread (also called Light Weight Processes in HP-UX, or LWP for short) information. But on a screen by screen basis. Useless if you have thousands of processes. After a bit of ping pong with a fellow sysadmin we ended up with the pstack (print stack) tool. It works like<\/p>\n<pre style=\"padding-left: 30px;\">$ ps -ef | grep -i java | head -1\r\n dma65t7\u00c2\u00a0 6728\u00c2\u00a0 6708\u00c2\u00a0 0 17:48:17 ?\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 3:23 \r\n\/opt\/dma65t7\/product\/6.5\/classes\/com\/documentum\/jboss4.2.0\/jdk\/bin\/IA64N\/java \r\n-Dprogram.name=run.sh -server -Xms256m -Xmx512m -\r\n\r\nsudo pstack 6728 | grep -i lwpid | sed -e 's,-*,,g' | head -10\r\nlwpid : 7653569\r\nlwpid : 7653570\r\nlwpid : 7653572\r\nlwpid : 7653573\r\nlwpid : 7653574\r\nlwpid : 7653575\r\nlwpid : 7653576\r\nlwpid : 7653577\r\nlwpid : 7653578\r\nlwpid : 7653579<\/pre>\n<p>So basically I ended up with the following one-liner<\/p>\n<pre style=\"padding-left: 30px;\">ps -ef &gt; out ;\u00c2\u00a0 ps -ef | awk '{ print $1 \" \" $2 }' | grep -v root |\r\n while read user pid ; do sudo pstack $pid | egrep -i \"($pid|lwpid)\"|\r\n sed -e 's,-*,,g' ; done &gt;&gt; out 2&gt;&amp;1<\/pre>\n<p>Which gives a quick and dirty indication of which process eats up all the resources and you can go ask that application owner if that is normal. It wasn&#8217;t!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Thread exhaustion on an HP-UX machine manifests itself by one or more of the following errors in \/var\/adm\/syslog\/syslog.log vmunix: kthread: table is full vmunix: WARNING: hponc_thread_create(): error creating thread for autofskd (12) sshd[8474]: fatal: fork of unprivileged child failed sshd[1895]: error: fork: Resource temporarily unavailable Keywords being thread and fork and failed.\u00c2\u00a0 You should immediatly [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10,6,3],"tags":[],"class_list":["post-326","post","type-post","status-publish","format-standard","hentry","category-hp-ux","category-unix","category-work"],"_links":{"self":[{"href":"https:\/\/zensonic.dk\/index.php?rest_route=\/wp\/v2\/posts\/326"}],"collection":[{"href":"https:\/\/zensonic.dk\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zensonic.dk\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zensonic.dk\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/zensonic.dk\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=326"}],"version-history":[{"count":0,"href":"https:\/\/zensonic.dk\/index.php?rest_route=\/wp\/v2\/posts\/326\/revisions"}],"wp:attachment":[{"href":"https:\/\/zensonic.dk\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=326"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zensonic.dk\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=326"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zensonic.dk\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=326"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}