Connection pool is fully used error

Hello,

after some time creating and editing documents in Nuxeo 7.2, I get this error:

No ManagedConnections available within configured blocking timeout ( 100 [ms] ) for pool org.apache.geronimo.connector.outbound.SinglePoolMatchAllConnectionInterceptor@19397d12

              javax.el.ELException: org.nuxeo.ecm.core.api.local.LocalException: Failed to load repository default: Connection pool is fully used, consider increasing nuxeo.vcs.blocking-timeout-millis (currently 100) or nuxeo.vcs.max-pool-size (currently 20)
    at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:339)
    at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:348)
    at org.jboss.el.parser.AstPropertySuffix.invoke(AstPropertySuffix.java:58)
    at org.jboss.el.parser.AstValue.invoke(AstValue.java:96)
    at org.jboss.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:276)
    at org.nuxeo.ecm.platform.ui.web.rest.services.URLPolicyServiceImpl.navigate(URLPolicyServiceImpl.java:528)
    at org.nuxeo.ecm.platform.ui.web.rest.RestfulPhaseListener.beforePhase(RestfulPhaseListener.java:94)
    at com.sun.faces.lifecycle.Phase.handleBeforePhase(Phase.java:228)
    at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:99)
    at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:219)
    at javax.faces.webapp.FacesServlet.service(FacesServlet.java:647)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.wss.servlet.BaseWSSFilter.doFilter(BaseWSSFilter.java:127)
    at org.nuxeo.wss.servlet.FailSafeWSSFilter.doFilter(FailSafeWSSFilter.java:56)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
    at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
    at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
    at org.nuxeo.elasticsearch.web.sync.UIThreadMarker.doFilter(UIThreadMarker.java:46)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
    at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.core.management.jtajca.internal.Log4jWebFilter.doFilter(Log4jWebFilter.java:67)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilterInternal(NuxeoAuthenticationFilter.java:423)
    at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:36)
    at org.nuxeo.ecm.platform.ui.web.auth.oauth.NuxeoOAuthFilter.doFilter(NuxeoOAuthFilter.java:120)
    at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:34)
    at org.nuxeo.ecm.platform.ui.web.auth.oauth2.NuxeoOAuth2Filter.doFilter(NuxeoOAuth2Filter.java:68)
    at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:34)
    at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilter(NuxeoAuthenticationFilter.java:391)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.platform.web.common.exceptionhandling.NuxeoExceptionFilter.doFilter(NuxeoExceptionFilter.java:75)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.platform.web.common.encoding.NuxeoEncodingFilter.doFilter(NuxeoEncodingFilter.java:73)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748)
    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:486)
    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:411)
    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:338)
    at org.nuxeo.ecm.platform.ui.web.rest.FancyURLFilter.doFilter(FancyURLFilter.java:110)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoRequestControllerFilter.doFilter(NuxeoRequestControllerFilter.java:139)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.webdav.service.WIRequestFilter.doFilter(WIRequestFilter.java:58)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilterInternal(NuxeoAuthenticationFilter.java:578)
    at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:36)
    at org.nuxeo.ecm.platform.ui.web.auth.oauth.NuxeoOAuthFilter.doFilter(NuxeoOAuthFilter.java:120)
    at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:34)
    at org.nuxeo.ecm.platform.ui.web.auth.oauth2.NuxeoOAuth2Filter.doFilter(NuxeoOAuth2Filter.java:68)
    at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:34)
    at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilter(NuxeoAuthenticationFilter.java:391)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:168)
    at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:233)
    at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoCorsFilter.doFilter(NuxeoCorsFilter.java:47)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.platform.web.common.exceptionhandling.NuxeoExceptionFilter.doFilter(NuxeoExceptionFilter.java:75)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.platform.web.common.encoding.NuxeoEncodingFilter.doFilter(NuxeoEncodingFilter.java:73)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:503)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.nuxeo.ecm.core.api.local.LocalException: Failed to load repository default: Connection pool is fully used, consider increasing nuxeo.vcs.blocking-timeout-millis (currently 100) or nuxeo.vcs.max-pool-size (currently 20)
    at org.nuxeo.ecm.core.repository.RepositoryService.getSession(RepositoryService.java:228)
    at org.nuxeo.ecm.core.api.local.LocalSession.createSession(LocalSession.java:117)
    at org.nuxeo.ecm.core.api.local.LocalSession.getSession(LocalSession.java:107)
    at org.nuxeo.ecm.core.api.AbstractSession.resolveReference(AbstractSession.java:344)
    at org.nuxeo.ecm.core.api.AbstractSession.getDocument(AbstractSession.java:953)
    at org.nuxeo.ecm.webapp.context.NavigationContextBean.navigateToRef(NavigationContextBean.java:640)
    at org.nuxeo.ecm.webapp.context.NavigationContextBean.navigateTo(NavigationContextBean.java:727)
    at sun.reflect.GeneratedMethodAccessor1275.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.jboss.seam.util.Reflections.invoke(Reflections.java:22)
    at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:32)
    at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56)
    at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28)
    at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
    at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:79)
    at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
    at org.nuxeo.ecm.platform.ui.web.util.NuxeoExceptionInterceptor.aroundInvoke(NuxeoExceptionInterceptor.java:78)
    at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
    at org.jboss.seam.core.ConversationInterceptor.aroundInvoke(ConversationInterceptor.java:56)
    at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
    at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44)
    at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
    at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107)
    at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:196)
    at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:114)
    at org.nuxeo.ecm.webapp.context.NavigationContextBean_$$_javassist_seam_13.navigateTo(NavigationContextBean_$$_javassist_seam_13.java)
    at org.nuxeo.ecm.platform.ui.web.rest.RestHelper.initContextFromRestRequest(RestHelper.java:107)
    at sun.reflect.GeneratedMethodAccessor1274.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.jboss.seam.util.Reflections.invoke(Reflections.java:22)
    at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:32)
    at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56)
    at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28)
    at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
    at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:79)
    at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
    at org.nuxeo.ecm.platform.ui.web.util.NuxeoExceptionInterceptor.aroundInvoke(NuxeoExceptionInterceptor.java:78)
    at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
    at org.jboss.seam.core.ConversationInterceptor.aroundInvoke(ConversationInterceptor.java:56)
    at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
    at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44)
    at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
    at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107)
    at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:196)
    at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:114)
    at org.nuxeo.ecm.platform.ui.web.rest.RestHelper_$$_javassist_seam_12.initContextFromRestRequest(RestHelper_$$_javassist_seam_12.java)
    at sun.reflect.GeneratedMethodAccessor2295.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:335)
    ... 94 more
Caused by: org.nuxeo.ecm.core.api.DocumentException: Connection pool is fully used, consider increasing nuxeo.vcs.blocking-timeout-millis (currently 100) or nuxeo.vcs.max-pool-size (currently 20)
    at org.nuxeo.ecm.core.storage.sql.ra.ConnectionFactoryImpl.getSession(ConnectionFactoryImpl.java:168)
    at org.nuxeo.ecm.core.repository.RepositoryService.getSession(RepositoryService.java:226)
    ... 145 more
Caused by: org.nuxeo.ecm.core.storage.StorageException: Connection pool is fully used, consider increasing nuxeo.vcs.blocking-timeout-millis (currently 100) or nuxeo.vcs.max-pool-size (currently 20)
    at org.nuxeo.ecm.core.storage.sql.ra.ConnectionFactoryImpl.getConnection(ConnectionFactoryImpl.java:114)
    at org.nuxeo.ecm.core.storage.sql.ra.ConnectionFactoryImpl.getSession(ConnectionFactoryImpl.java:166)
    ... 146 more
Caused by: javax.resource.ResourceException: No ManagedConnections available within configured blocking timeout ( 100 [ms] ) for pool org.apache.geronimo.connector.outbound.SinglePoolMatchAllConnectionInterceptor@19397d12
    at org.apache.geronimo.connector.outbound.AbstractSinglePoolConnectionInterceptor.getConnection(AbstractSinglePoolConnectionInterceptor.java:86)
    at org.apache.geronimo.connector.outbound.ThreadLocalCachingConnectionInterceptor.getConnection(ThreadLocalCachingConnectionInterceptor.java:70)
    at org.apache.geronimo.connector.outbound.TransactionEnlistingInterceptor.getConnection(TransactionEnlistingInterceptor.java:49)
    at org.apache.geronimo.connector.outbound.TransactionCachingInterceptor.getConnection(TransactionCachingInterceptor.java:101)
    at org.apache.geronimo.connector.outbound.ConnectionHandleInterceptor.getConnection(ConnectionHandleInterceptor.java:43)
    at org.apache.geronimo.connector.outbound.TCCLInterceptor.getConnection(TCCLInterceptor.java:39)
    at org.apache.geronimo.connector.outbound.ConnectionTrackingInterceptor.getConnection(ConnectionTrackingInterceptor.java:66)
    at org.apache.geronimo.connector.outbound.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:77)
    at org.nuxeo.runtime.jtajca.NuxeoContainer$ConnectionManagerWrapper.allocateConnection(NuxeoContainer.java:838)
    at org.nuxeo.ecm.core.storage.sql.ra.ConnectionFactoryImpl.getConnection(ConnectionFactoryImpl.java:100)
    ... 147 more

I tried increasing nuxeo.vcs.blocking-timeout-millis and nuxeo.vcs.max-pool-size, but after a while, I get the same error.

I've tried searching how can I fix this and why it happens, but with no results. Can someone explain me why it happens and how can I solve the problem?

0 votes

0 answers

3220 views

ANSWER

I am facing the same issue. Did anybody encountered and got solution?
03/24/2016

Nuxeo 7.2 is obsolete and unsupported. Please use Nuxeo LTS 2015 (7.10).
03/31/2016

Using the LTS, doesn't fix the problem. It just means that it's supported and maybe somebody will look into it.
06/06/2016

I'm having the same problem with Nuxeo 7.10. As said above the respective parameters show no effect
07/11/2016

It means that there's a lot of work going on in the server and that all its threads are busy. You should use jstack to find out what's going on.
07/12/2016

I'm quite sure that 2 concurrent users of which one is browsing and the other is uploading a dozen files is not considered "a lot of work going on". It's a test server and therefore I can assure that there is no heavy traffic. Still the problem occurs quite often. After server restart it is gone and when working with the server for a few minutes, then the error occurs, especially when uploading multiple files at once via drag&drop. Multiple files means here 10-15 with an overall size of <20 MB.
07/12/2016

As you can imagine we have plenty of customers using 2 concurrent users not having this problem. So you must debug on your side to give us an idea of what is going on. To start with this the first tool is jstack.
07/12/2016

I've uploaded the jstack output and the server log to dropbox for everyone to read: https://www.dropbox.com/sh/afz5lccr3z2k2th/AADM21POyQmI_K5wRHhaVXr5a?dl=0
07/13/2016

A fix was recently pushed (https://jira.nuxeo.com/browse/NXP-20113), it may help your case.
07/25/2016

I cannot confirm it finally, since the bug was not always there from the beginning and sometimes appeared only later, but from our first tests it seems that Hotfix 13 really fixed our issue. It is very interesting that there are so many customers not having the issue, since it doesn't seem to be our special configuration but a general Nuxeo problem that causes the error. Nevertheless: thanks for the help. Much appreciated.
07/25/2016

hello ,I am getting same error as above in Nuxeo FT 8.2 Version,i am uploading document in nuxeo using document service and and retrieving through the NXQL query and it is working fine, but after some time i got the connection pool error and after restarting a nuxeo all operations are working. So is there any solution related to this problem? Is this possible to allow all connection in nuxeo?
01/11/2017

I am getting the same error above in Nuxeo 7.10 LTS 2015. Can somebody help me to fix this problem?
04/10/2017