Almost two-thirds of prisoners are unable to write or do sums properly, a report revealed.
The Prison Service needs to take action to better deliver education and skills, inspectors said.
Chief inspector Michael Maguire urged the Executive to help make improvements.
"While there have been pockets of excellence and innovative practice, inspectors found that the situation continues to deteriorate and this is unacceptable.
"One has to remember that the provision of learning and skills in our prisons is a major element in the rehabilitation of prisoners with a view to reducing reoffending.
"If that provision is inadequate in any way then the system not only lets the prisoners down but wider society as a whole."
The head of Criminal Justice Inspection Northern Ireland (CJINI) said with more than 60% of prisoners below the minimum required level in literacy and numeracy skills, too few were being helped to address their problems.
The report by CJINI and the Education and Training Inspectorate (ETI) said long-standing concerns about the quality and impact of the learning for inmates had been confirmed.
Noelle Buick, chief inspector at the ETI, said with the exception of the essential skills programme at Magilligan delivered by the North West Regional College the outcomes for prisoners were often poor.
"The report fully recognises that the real barriers to progressing the provision of learning and skills to the prisoner population, whether in relation to essential skills or to wider vocational training, are a number of difficult restrictive institutional and security issues," she warned.
Inspectors have already made numerous recommendations to improve learning and skills methods but progress has been negligible due to the low status afforded to the service, Ms Buick added.
The inspectors urged Justice Minister David Ford to consider outsourcing more of the education and training, building on good work done by further education centres.
"Effective collaborative partnerships with external providers are an important part of the way forward, in particular the delivery of essential skills within the unique context of a prison," they said.
"Collaborative partnerships also offer the Northern Ireland Prison Service a range of options which would prepare prisoners much better for competing in the employment market upon release."
Mr Ford pledged to address how the Prison Service delivers learning and skills.
"This latest report shines a light on the scale of the challenge the Prison Service and its educational partners face in delivering, on a consistent basis, meaningful learning and skills programmes for offenders," he said.
"While it finds pockets of excellence operating across the prison estate it also highlights a series of gaps in this service and that is something that I am determined to address."
Since the last CJINI learning and skills report, the Prison Service has commenced a review of the range of services available to prisoners and how they are delivered.
A draft employability strategy has been developed to ensure that the service provided to prisoners gives them a better opportunity to gain employment after their release and arrangements had been made to fill staff shortages.