More than half of inmates have claimed it is easy to get drugs at a prison where many staff are accused of having a "complacent" attitude to the issue.
Urgent action must be taken at HMP Rochester in Kent, inspectors warned, following a surprise inspection in September last year.
Prisoners who were seen to be clearly under the influence of drugs went unchallenged by staff, the report said.
In the two years since it was last visited, the jail had not made enough progress on a range of issues, Chief Inspector of Prisons Nick Hardwick said.
A fifth of inmates said they felt unsafe at the jail, which houses 743 adult and young adult male prisoners on a site described as a sprawling mix of old and new accommodation.
Higher than expected levels of drug use were detected, inspectors said, adding that there was evidence of the presence of "considerable" amounts of as-yet undetectable drugs known as new psychoactive substances (NPS).
In his report, Mr Hardwick said: "There was much evidence to suggest that the availability of NPS was having a significant destabilising effect, and yet the prison's response lacked co-ordination and too many staff seemed complacent of the issue and its impact. Confronting this drug problem demanded urgent attention."
A survey found 57% of inmates said it was easy to access drugs in the prison, reporting that it was easier to get the synthetic cannabinoid Spice than tobacco.
The prison was found to be "not sufficiently good" in any of the four tests carried out by inspectors to decide whether it was functioning in a healthy way.
Living conditions were described as poor with "dirty accommodation", food was also found to be poor and inspectors noted that staff shortages were causing too many hospital appointments to be missed.
Explicit pornography was also found to be on display in many cells.
While prisoners were not spending too much time in their cells, inspectors found a third of inmates were doing nothing productive with their working day.
Incidents of self-harm were classed as high, as was the level of violence, especially against staff, the report said.
Inspectors recommended the jail focus on basic needs, rather than plans for the future - namely a better drug strategy, a cleaner environment for inmates, ensuring prisoners make better use of their work time and "joined-up thinking" on resettling offenders.