Why is Maths Hard?

Some slight thoughts about maths being hard.

Prompted by a couple of things I saw in quick succession.

First, this exchange on twitter about an interesting post about politeness. That post is interesting in its own right but not the point. The point is that Michiexile is a mathematician.

@quinnnorton @ftrain That first tip hits a bit of a personal nerve. As a mathematician, "that sounds hard" is top 3 reactions to my job. >.<
— michiexile (@michiexile) August 13, 2014

Second, this post about the lottery of what you happen to be interested in, where maths is definitely not in the author's bag.

This certainly strikes a chord with me. I happen to like maths and be good and it and it is seriously fascinating. But I have a couple of memories that are slightly discordant...

• I once went at a job interview at HP Labs Bristol, where I did their logical thinking test and they said I'd got their third highest score ever (amongst many smart people.)   But I didn't find it impressive because I thought "You gave me the kind of test I am good at, so what?"   It didn't help that the same day they had some charlatan psychologist who administered a personality test and nearly gave me a nervous breakdown. 
• A few years later I was talking with a good friend in a pub during my PhD, and he stopped and looked at me and said "You did Maths... at Cambridge... you must be very intelligent."  What was odd to me was that we had known each other for a couple of years, and surely his view of my intelligence shouldn't then have been judged by my CV.

The point I'm trying to make here is this.  Why should skill at Maths or Logic make you more intelligent that somebody skilled at something else.  Why do people think Maths is hard?

Is it because maths is the school subject with the highest ratio between compulsoriness (100% for a long time) and difficulty to some students (some reasonable percentage of people find it hard)?  So for example English language (in British schools) is similarly compulsory but not as obviously difficult.  Almost everyone can write essays even if they are not actually very good.  But in Maths, if you are really not very good at it, you can't do certain things at all.

I really don't know why people think Maths is hard but just asking the question.