When people want to over-rule you in a way that brooks no dissent they use the phrase: 'the simple fact is...'
The 'simple fact' is never a fact though, it is always an assumption or an opinion.
A proper simple fact might be the temperature on a thermometer or that you are singing or that Chelsea won the FA cup. In other words- something very hard to disagree with.
So what people are really doing is taking what they like- the incontrovertibility- and bolting it on to an opinion in order to get their own way.
The simple fact is: don't trust people who use this locution!