服务器宕机问题解决

zhang_xf

描述：服务器在linux上，2019年11月20日做过一次迁移，是因为原来将服务部署在了服务器的系统盘上，而系统盘空间小且不能扩展，所以将服务迁移到了同一服务器的一块后加入的可扩展的盘符上；这次迁移以前发生过两次问题，都是因为磁盘空间满了导致，清空一部分空间后服务都正成运转，迁移以后的一直到4月初也基本没有问题，这段时间因为疫情原因使用人员可能较少，4月初以后系统连续出现多次宕机，开始以为是系统盘满了，但是发现系统盘还有1个G的空间，我陆续清空空间，也连续重启几次都是用一会儿后就不行了，知道清空出3.8G后再重启就好了，节后五月8号，又出现宕机，查看系统盘空间是3.7G，重启后维持到下午再次宕机，于是再次重启，这次重启时我已经把系统盘空间清出了18个G，运行到现在没有出现问题。宕机表现：宕机后系统还是在运行中的，但是无法登录或者操作得不到响应。注：年后几个月都没有动过代码了，只是重启过。
希望起步技术人员能跟踪解决一下。

jishuang

先监控一下jvm、数据库连接等的占有情况
http://docs.wex5.com/wex5-deploy-question-list-0004/

宕机的是否看监控信息

zhang_xf

jishuang 发表于 2020-5-9 18:09
先监控一下jvm、数据库连接等的占有情况
http://docs.wex5.com/wex5-deploy-question-list-0004/

com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 1 milliseconds ago.  The last packet sent successfully to the server was 1 milliseconds ago.
        at sun.reflect.GeneratedConstructorAccessor358.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:400)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1038)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3434)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3334)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3774)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871)
        at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1665)
        at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1207)
        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2249)
        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2280)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2079)
        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:794)
        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)
        at sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:400)
        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:399)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at TaskNumber.getTaskORPass(TaskNumber.java:109)
        at TaskNumber.execute(TaskNumber.java:28)
        at com.justep.ui.JavaServer.runJava(Unknown Source)
        at com.justep.ui.JavaServer.doJ(Unknown Source)
        at com.justep.ui.JavaServer.service(Native Method)
        at com.justep.ui.servlet.MainServlet.service(Unknown Source)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.justep.portal.filter.RedirectFilter.doFilter(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.justep.ui.filter.MicroServiceAdminFilter.doFilter(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at com.googlecode.psiprobe.Tomcat60AgentValve.invoke(Tomcat60AgentValve.java:30)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2926)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3344)
        ... 44 more
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 1 milliseconds ago.  The last packet sent successfully to the server was 1 milliseconds ago.
        at sun.reflect.GeneratedConstructorAccessor358.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:400)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1038)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3434)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3334)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3774)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871)
        at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1665)
        at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1207)
        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2249)
        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2280)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2079)
        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:794)
        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)
        at sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:400)
        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:399)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at TaskNumber.getTaskORPass(TaskNumber.java:109)
        at TaskNumber.execute(TaskNumber.java:28)
        at com.justep.ui.JavaServer.runJava(Unknown Source)
        at com.justep.ui.JavaServer.doJ(Unknown Source)
        at com.justep.ui.JavaServer.service(Native Method)
        at com.justep.ui.servlet.MainServlet.service(Unknown Source)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.justep.portal.filter.RedirectFilter.doFilter(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.justep.ui.filter.MicroServiceAdminFilter.doFilter(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at com.googlecode.psiprobe.Tomcat60AgentValve.invoke(Tomcat60AgentValve.java:30)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:745)
com.mysql.jdbc.JDBC4ResultSet@34efb05d---------------------data------mayj----datas-----E10ADC3949BA59ABBE56E057F20F883E
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2926)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3344)
        ... 44 more
这是后台的报错信息，循环很多遍，这是数据库设置的问题吗