You set up a dedicated Send connector and scope it for the sp.yourdomain.com domain, using the SharePoint server as smart host.
It works just fine.
Then you enable Hybrid Exchange, move a couple of mailboxes into the cloud, and you switch your MX record(s) to Exchange Online Protection.
In this scenario users with mailboxes in the cloud may fail to submit documents to SharePoint via e-mail and they'll get the following error:
LED=550 5.4.101 Proxy session setup failed on Frontend with '554 5.4.4 SMTPSEND.DNS.Non-ExistentDomain; nonexistent domain'
Here is what fixed it for me, and hopefully it will fix it for you too:
1. In the O365 Exchange Admin Center, under Mail flow / Accepted domains:
- Change your parent domain (yourdomain.com) from Authoritative to Internal Relay and enable Accept mail for all subdomains (see here - a tad old, due for an update).
- Edit the connector which routes mail from O365 to your organization and click Next until the Edit Connector page is displayed.
- Add the *.yourdomain.com domain to the list.
- Click next and work your way through the wizard. Don't change anything else. Save the settings. In the process you'll need to provide a valid e-mail address to validate the connection. Provide one in the yourdomain.com domain.
- Create a new accepted domain of type "External Relay" for sp.yourdomain.com:
- There should already be a dedicated Send connector. If not, create a new dedicated Send connector for the sp.yourdomain.com address space:
- On the Send connector, ensure that the SharePoint server is listed as smart host. Add it if it isn't.
E-mail will now start flowing as expected.
Your environment might be slightly different, but the principle still applies.