The ATO has stated there is some practical difficulty in denying such deductions as paying a tax debt is not capital in nature nor was used to gain exempt income. This is what the regulation says:
Subsection 51(1) of ITAA36 provides that: “All losses and outgoings to the extent to which they are incurred in gaining or producing the assessable income, or are necessarily incurred in carrying on a business for the purpose of gaining or producing such income, shall be allowable deductions except to the extent to which they are losses or outgoings of capital, or of a capital, private or domestic nature, or are incurred in relation to the gaining or production of exempt income.”
That is, such an expense is an expense incurred in carrying on that business and hence qualifies for deduction under the second limb of subsection 51(1) of the act. Care needs to be taken however, as the ruling would not apply to interest on borrowings that are not connected with the carrying on of a business for the purpose of producing assessable income.