There are several categories of reasons that lead to this.  They all boil down to one - the organization isn't fully choosing the change.  Here are the categories:

Corporate Beliefs:  These can be significant roadblocks.  For example, a ridiculously high financial payback "hurdle rate" can kill just about anything.  A belief that ideas from within the organization must be the result of someones agenda so they'll be ignored.  A deep distrust of a trade union can prevent well intentioned initiatives from the shop floor from going any further.  Finally, a deeply held belief that we can't do it without a lot of (expensive) outside help will stop you from even starting.

Business Planning Practises:  Budgeting is often done year to year.  If something wasn't thought of the year before, it isn't budgeted and won't be funded.  Even if the idea can pay for itself, there's no budget to invest!  That's absurd and we can all see it, yet we often work in organizations that don't.  The annual window is very limiting.  Great opportunities that arise that may not last through to the next budget season can die before they are born.  This arises from short term thinking.  It works for short term projects that fit the windows, but nothing longer.  It can't even capture multi-year initiatives.

Approaches to Managing Costs:  If your expenses go over budget for a new initiative it attracts excessive scrutiny.  This constrains people from doing what's right in the interests of doing what was budgeted. 

Management and Leadership:  If compensation or bonus is linked to short term results - as it almost always is, then the longer range slower payback ideas will suffer, even if they'll lead to significant long term and sustainable results.  We favour the short term over the long term.  Another limiting factor is the short "life span" of many managers and executives in their current roles.  These folks are constantly moving around within the company and even among companies.  They stay long enough to make substantial change but not to see it through.  The next person in the seat then makes other changes.  Nothing sticks and nothing is sustained.

Financial Constraints: Rigid adherence to financial reporting metrics such as ROI, EPS, ROA, EBITDA, etc., can limit investment opportunities.  Improvement initiatives from within will have a cost.  They almost always require some sort of cultural change and that takes long.  Managing to meet market driven objectives is a no-win method.  You simply won't drive something as slow as cultural change (getting a lot of people to behave differently) with a short term measure that keeps only shareholders and investment analysts happy.  They are not the ones undergoing the change - and your people know it.  The measures must be meaningful to them personally, simple to understand and realistic.

Culture:  Some organizations are simply too big to budge.  Large ships don't turn quickly and neither to large complex organizations.  Not only are the formal processes complex and difficult to change, the people are used to behaving in certain ways that often reflect executive and management preferences.  Some organizations are incredibly risk averse - they do nothing without analyzing it to death.  Some are prone to jump on the bandwagon of every improvement initiative, but never finish any of them.  Management is there to keep things stable.  Change rocks the boat.  Managers who are tasked with keeping things stable have no incentive to change. 

Getting over these hurdles requires a change of approach.

• Task senior leadership with the change - they must "own" it and truly support it, not just act as "figure heads".
• Re-evaluate measures that drive short term results - they may be working against you.  Look for long term metrics that are supportive of long term improvement ideas.
• Open up to all ideas no matter where they arise - stop assuming that everyone has their own selfish agenda.  Union members are employees too - they do care about your business' success.
• Slow down senior management and executive turnover.  Recognize that some of this is due to dissatisfaction in companies where they can't make significant change, so they move on.  If they start something they must see it through to completion and to the generation of results for more than a quarter!
• Move people around internally so that the reasons for various cultural barriers from within are understood by many.  The new perspective brought by people from elsewhere in the organization can help shed light on these barriers and eliminate them.
• Make financial measures and performance for change initiatives unique to the initiative.  Don't judge long range improvements with short term measures.  Protect your longer term sustainability goals from short term-ism.