Yes. you should hold onto physical paper receipts, carrieannbooth.
Receipts are proof of your business expenses. In case you're audited or asked to show documentation, you can always submit the physical paper receipts anytime. However, if some are less important you can eventually discard them.