The provided/accepted answer explains the behavior. This answer provides a possible workaround.
As Split by Attributes is a Script tool, you can hack it a little bit. (I don't have 10.5 to test if this change will 'just work', but I know the hack will get you the behavior you want: to set the output in the tool to a feature dataset).
- With ArcToolbox or the Catalog Window toolbox, find Split by Attributes, right click and Copy
- Create a new toolbox in your working directory, right click and Paste the tool in.
- Right click the copied tool and select Properties
- Select the Target Workspace. The Data Type will be set to Workspace. In the drop down change this to Workspace or Feature Dataset
- Click ok to save changes and run the tool
If the tool doesn't "work", that means the code inside doesn't allow the writing of output to the feature dataset. Thus you could ping Esri to enhance the tool. If it does work you could still ping Esri and call the problem a bug (oversight that the wrong data type was selected in the tool design).