Making the Sublist Fields of a Transaction record mandatory is not possible through Form Customization.
We can do this by creating a Workflow with a Set Field Mandatory Action. The setup of the Set Field Mandatory would look like this:
Make the Columns Mandatory on Transactions Records