Please read How to use an embedded Nuxeo from a Java application.

A more involved method but allowing you to control dynamically the distribution you use is described at Embedding Nuxeo.

Although Nuxeo provides pre-built applications (DM, DAM, etc.), the real power of Nuxeo IMO (apart from the ability to easily extend it) is being able to embed/wrap Nuxeo - either the core product (CAP) or one of the applications.

On the Nuxeo side you would make the customizations required in Studio or in the IDE, and then write a custom API, leverage pre-built capabilities (e.g. CMIS), or some combination thereof to provide access to your data. Then you wire in an existing application through the API or write your own presentation wrapper (which also leverages the API). I have done this for a number of clients with widely varying technologies.