How can I stop students from submitting .pages files and other file types I can't open?
The Assignment tool has a file type restriction setting you can use. You can find this setting when creating a new Assignment or when editing the settings of an existing assignment. You can update this setting at any time. When editing or creating an Assignment, you can find this setting under the Submission & Completion section.
Annotatable Files is a good choice that allows for common file types like .doc, .docx, .pdf, .rtf etc. Compatible with Turnitin makes sense if Turnitin is required for the assignment. When Custom File Types is selected, you will need to enter the specific file extensions you wish to allow.

PDF only and Files that can be previewed without conversion (HTML files) are generally too restrictive and not recommended.