客户服务偶发性终止

reoyha

最近几天客户服务不稳定，但是无法排查出具体原因，希望平台技术组能给与一定的支持。

今天上午10：08~10：16访问速度慢，10：16系统自动终止。
今天下午15：07系统终止，重启服务后才回复正常。 logs.rar (1.88 MB, 下载次数: 2)

2018-12-3 16:17 上传

点击文件名下载附件
阅读权限: 100


这是附件，里边有日志，错误信息好像是：

1. javax.servlet.ServletException: com.justep.ui.exception.UIException: com.justep.ui.exception.UIException: com.justep.ui.exception.UIException: com.justep.ui.exception.UIException: ClientAbortException:  java.net.SocketException: Connection reset by peer: socket write error
   
2.         at com.justep.ui.servlet.MainServlet.service(Unknown Source)
   
3.         at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
   
4.         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   
5.         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   
6.         at com.justep.portal.filter.RedirectFilter.doFilter(Unknown Source)
   
7.         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   
8.         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   
9.         at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
   
10.         at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    
11.         at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    
12.         at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    
13.         at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)
    
14.         at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    
15.         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
    
16.         at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
    
17.         at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
    
18.         at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
    
19.         at java.lang.Thread.run(Thread.java:745)
    
20. Caused by: com.justep.ui.exception.UIException: com.justep.ui.exception.UIException: com.justep.ui.exception.UIException: com.justep.ui.exception.UIException: ClientAbortException:  java.net.SocketException: Connection reset by peer: socket write error
    
21.         at com.justep.ui.serializer.HttpSerializer.execute(Unknown Source)
    
22.         at com.justep.ui.util.UIUtils.httpSerializer(Unknown Source)
    
23.         at com.justep.ui.util.UIUtils.httpSerializer(Unknown Source)
    
24.         at com.justep.ui.WindowServer.service(Unknown Source)
    
25.         ... 18 more
    
26. Caused by: com.justep.ui.exception.UIException: com.justep.ui.exception.UIException: com.justep.ui.exception.UIException: ClientAbortException:  java.net.SocketException: Connection reset by peer: socket write error
    
27.         at com.justep.ui.serializer.HttpSerializer.output(Unknown Source)
    
28.         ... 22 more
    
29. Caused by: com.justep.ui.exception.UIException: com.justep.ui.exception.UIException: ClientAbortException:  java.net.SocketException: Connection reset by peer: socket write error
    
30.         at com.justep.ui.xml.XMLUtils.inputSourceToSAX(Unknown Source)
    
31.         ... 23 more
    
32. Caused by: com.justep.ui.exception.UIException: ClientAbortException:  java.net.SocketException: Connection reset by peer: socket write error
    
33.         at com.justep.ui.serializer.BinaryTextContentHandler.endDocument(Unknown Source)
    
34.         at orbeon.apache.xerces.parsers.AbstractSAXParser.endDocument(AbstractSAXParser.java:733)
    
35.         at orbeon.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(XMLDocumentScannerImpl.java:524)
    
36.         at orbeon.apache.xerces.impl.XMLEntityManager.endEntity(XMLEntityManager.java:1917)
    
37.         at orbeon.apache.xerces.impl.XMLEntityScanner.load(XMLEntityScanner.java:1763)
    
38.         at orbeon.apache.xerces.impl.XMLEntityScanner.skipSpaces(XMLEntityScanner.java:1249)
    
39.         at orbeon.apache.xerces.impl.XMLDocumentScannerImpl$TrailingMiscDispatcher.dispatch(XMLDocumentScannerImpl.java:1217)
    
40.         at orbeon.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:324)
    
41.         at orbeon.apache.xerces.parsers.XML11Configuration.parse(XML11Configuration.java:845)
    
42.         at orbeon.apache.xerces.parsers.XML11Configuration.parse(XML11Configuration.java:768)
    
43.         at orbeon.apache.xerces.parsers.XMLParser.parse(XMLParser.java:108)
    
44.         at orbeon.apache.xerces.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1201)
    
45.         ... 24 more
    
46. Caused by: ClientAbortException:  java.net.SocketException: Connection reset by peer: socket write error
    
47.         at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:330)
    
48.         at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:296)
    
49.         at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:98)
    
50.         at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
    
51.         at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
    
52.         at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
    
53.         ... 36 more
    
54. Caused by: java.net.SocketException: Connection reset by peer: socket write error
    
55.         at java.net.SocketOutputStream.socketWrite0(Native Method)
    
56.         at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
    
57.         at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
    
58.         at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:761)
    
59.         at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:448)
    
60.         at org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:318)
    
61.         at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:987)
    
62.         at org.apache.coyote.Response.action(Response.java:186)
    
63.         at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:325)
    
64.         ... 41 more
    
65. 2018-12-03 15:07:17 [http-18081-147] [com.justep.ui.JavaServer] ERROR:
    
66. javax.servlet.ServletException: Biz Action执行失败
    
67.         at BizAction.doPost(BizAction.java:17)
    
68.         at com.justep.ui.impl.JProcessorImpl.execute(Unknown Source)
    
69.         at com.justep.ui.JavaServer.runJava(Unknown Source)
    
70.         at com.justep.ui.JavaServer.doJ(Unknown Source)
    
71.         at com.justep.ui.JavaServer.service(Native Method)
    
72.         at com.justep.ui.servlet.MainServlet.service(Unknown Source)
    
73.         at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
    
74.         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    
75.         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    
76.         at com.justep.portal.filter.RedirectFilter.doFilter(Unknown Source)
    
77.         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    
78.         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    
79.         at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    
80.         at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    
81.         at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    
82.         at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    
83.         at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)
    
84.         at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    
85.         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
    
86.         at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
    
87.         at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
    
88.         at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
    
89.         at java.lang.Thread.run(Thread.java:745)
    
90. Caused by: com.justep.ui.exception.UIException: 编码: JUSTEP000000; 提示: 执行Action出错
    
91.         at com.justep.ui.exception.UIException.create(Unknown Source)
    
92.         at com.justep.biz.action.ActionBatchEngine.execute(Unknown Source)
    
93.         at com.justep.ui.system.service.commonUtils.postBizAction(Unknown Source)
    
94.         at com.justep.ui.system.service.commonUtils.bizAction(Unknown Source)
    
95.         at BizAction.doPost(BizAction.java:15)
    
96.         ... 22 more
    
97. Caused by: com.justep.biz.client.ActionException: 编码: JUSTEP120033; 提示: 输出Action结果出错
    
98.         at com.justep.biz.client.ActionException.create(Unknown Source)
    
99.         at com.justep.biz.client.StreamCallback.execute(Unknown Source)
    
100.         at com.justep.biz.client.ActionEngine.invokeActions(Unknown Source)
     
101.         at com.justep.biz.client.ActionEngine.invokeActions(Unknown Source)
     
102.         ... 26 more
     
103. Caused by: ClientAbortException:  java.net.SocketException: Connection reset by peer: socket write error
     
104.         at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:330)
     
105.         at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:296)
     
106.         at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:98)
     
107.         ... 29 more
     
108. Caused by: java.net.SocketException: Connection reset by peer: socket write error
     
109.         at java.net.SocketOutputStream.socketWrite0(Native Method)
     
110.         at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
     
111.         at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
     
112.         at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:761)
     
113.         at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:448)
     
114.         at org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:318)
     
115.         at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:987)
     
116.         at org.apache.coyote.Response.action(Response.java:186)
     
117.         at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:325)
     
118.         ... 31 more

复制代码

jishuang

服务停止是tomcat自动退出了还是其他什么现象？

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

确认内存空间是否够用

reoyha

jishuang 发表于 2018-12-3 17:16
服务停止是tomcat自动退出了还是其他什么现象？

Caused by: java.lang.OutOfMemoryError: GC overhead lim ...

不自动推出，只是无法访问，

jishuang

确认内存是否够用，日志中有2楼发的内存溢出的问题

在浏览器中输入http://ip:port/_index.html, 看访问是否正常

可以安装tomcat的监测工具监测一下相关的占用情况
http://docs.wex5.com/wex5-deploy-question-list-0004/

reoyha

jishuang 发表于 2018-12-4 08:43
确认内存是否够用，日志中有2楼发的内存溢出的问题

在浏览器中输入http://ip:port/_index.html, 看访问是 ...

刚刚11点29开始服务又无法访问了，确定内存足够，系统只占用了80%的内存。
服务就是这种状态，但是系统不能访问，连登录页面也无法打开，但是在我查看日志时，系统自动恢复了，未进行重启。
中间大概持续了20多分钟，一会我把日志传上来。
同服务器的其他服务可以访问。

logs2.rar (1.88 MB, 下载次数: 92)

2018-12-4 13:05 上传

点击文件名下载附件

jishuang

在服务器上安装http分析工具，看看卡的时候具体在执行什么请求

卡的是12-4的，发的日志是12-3的

reoyha

jishuang 发表于 2018-12-4 13:34
在服务器上安装http分析工具，看看卡的时候具体在执行什么请求

卡的是12-4的，发的日志是12-3的 ...

logs2.rar (2.95 MB, 下载次数: 78)

2018-12-4 14:34 上传

点击文件名下载附件

抱歉，上传错了

jishuang

两次都是在你说的卡之后输出这个


http://www.cnblogs.com/penghongwei/p/3603326.html

reoyha

jishuang 发表于 2018-12-4 15:32
两次都是在你说的卡之后输出这个

日志上，除了内存溢出以外，请问能看出是哪些功能导致的吗？这个系统功能点不下200个，一点点去排查时间也来不及，是任务表吗？客户这边任务表已经3GB了，大概100W数据量

jishuang

除了这个，没有其他会导致服务器死的问题