Windows or Linux?
I'm quite comfortable in either environment, but are there any benefits for a Windows install or Linux? Also, is NTLM SSO working better/easier in Windows?
Some customers have reported up to 40% performance increase while running a Nuxeo server with PostgreSQL under heavy load and a couple of hundred of thousands of documents on Linux vs Windows on the same hardware.
Apparently this is mostly due to the fact that PostgreSQL can leverage the POSIX features of Linux very efficiently while this is obviously not the case under Windows. This anecdotal observation seems to be confirmed by this discussion on the PostgreSQL mailing list.
Nuxeo it-self should mostly behave the same. You should take care to install a 64bit OS and JDK to leverage all the available memory of your system.
As for the SSO, NTLM only works with Windows (clients). If you want a cross-platform SSO I would advise you to setup a CAS SSO server and the CAS authentication plugin of Nuxeo.