A shortage of staff has led to the deterioration of conditions at HM Young Offenders' Institution in Aylesbury, a report has found.
It also revealed the Buckinghamshire institution was not safe enough, recording levels of violence as "high", with some incidents being described as "serious".
In his report, Nick Hardwick, Chief Inspector of Prisons, said more needed to be done to combat violence and address gang affiliations.
Half of all respondents reported feeling unsafe at some point during their stay and just under a quarter said they felt unsafe at the time of the inspection.
The unannounced inspection found that as well as the more predictable causes of violence, the long periods of lock up and inactivity caused frustrations that contributed to the likelihood of violence and aggression.
Between 30% and 40% of young prisoners were locked up during the working day.
Other findings included that the segregation unit environment and regime were poor, and that staff shortages were undermining offender management with heavy caseloads, a backlog of risk assessments and some limited sentence planning.
Drug usage was also found to be double the target, with many offenders thinking it was easy to get drugs into the unit.
There was also evidence of the availability of undetectable synthetic drugs.
Mr Hardwick said: "The population at Aylesbury presents risks but it is reasonably stable.
"The purpose and function of the prison was clear but the prison was uncertain about how to set about delivering its core functions in a coherent and joined-up way.
"For example, there was some good work taking place to address violence, but this was undermined by poor data, or by a very poor regime that fostered inactivity and indolence.
"The prison held long-term prisoners and yet many practices were punitive and regressive. Trust was too limited and relationships unpredictable.
"There was too little to motivate young men, or to encourage their personal investment in their futures while at the prison.
"Staffing shortages were a chronic weakness but it was hard to see how HMYOI Aylesbury could progress until there was a fundamental improvement in the quality of learning, skills and work offered."
HMYOI Aylesbury, a training prison, holds up to 444 young adult men aged 18 to 21 who are serving among the longest sentences for this age group in the country.
More than 80% of those held are serving more than four years and 30% are serving more than 10 years to life.
Commenting on the report, Juliet Lyon, director of the Prison Reform Trust said: "It is deeply distressing to read a report that describes too few staff, many of them temporarily assigned to the prison, trying to cope against unfair odds."
She continued: "Michael Gove's promise to change the fortunes of our prisons cannot come a moment too soon."