You cannot reliably enforce uniqueness of child titles within a folder using an automation chain. This must be done at a much lower level in the system which includes adding a unique conditional database constraint on the hierarchy table parentid and name columns for VCS-based Nuxeo deployments. A DBS-based deployment has a dependency on a Nuxeo Shared State Manager being implemented.

I am looking to contribute to the Nuxeo community a nuxeo-unicity-folder marketplace add-on that defines a new document type, 'Unicity Folder', that prevents its children from having duplicate dc:titles. This type of folder is intended to mimic the behavior of a directory in a traditional file system. A user that wants all folders in the domain to have this behavior could customize the Document Type configuration for the Domain accordingly.

Let me know if the nuxeo-unicity-folder add-on described above would be of interest to you.