If you are not maxing out your 401k, simple IRA, and a few other retirement benefits as a W2 there is no way I would consider paying off my house early.
For this main reason:
What happens if a financial sunami hits your house hold?
Two People:
Person A and Person B each own a $500k house putting $100k down. Note is $400k.
Person A maxes out there 401k and other retirement vehicles and does not pay down extra on the house.
Person B only does the minimum in their 401k to get the company match and pays down the rest on the house.
20 years later each person looses their job, has no income, the economy is in the dumps and neither person can pay their bills.
Person A has X dollars in their 401k account and owes Y dollars on their mortgage.
Person B has <X dollars in their account and owes <Y dollars on their mortgage.
They are both foreclosed on and go live with family.
In foreclosure the bank can take both homes and all their equity and go after all assetts that are not protected by the employee Retirement Income Security Act of 1974. This varies a little by state also.
Person A has X dollars in their 401k and no home. Person B has <X dollars in their 401k and no home.
Depending on what state you live in similar may apply if being sued for something and they go after your home equity.
Just some things to think about before going to pay off a mortgage.
In this low interest rate enviroment of <5% on most notes there is no way I would trade away my flexibility monthly to pay off my mortgage note early.
I am not saying go burn that money at the bar, but instead invest it in some other vehiclse align with your asset management goals.