A peer told the Lords about his own experiences in jail as he called for better education for prisoners.
Lord Hanningfield said with half the prison population illiterate, greater priority needed to be given to teaching inmates to read and write.
The non-affiliated peer said most prisoners had no idea what the House of Lords was, and one fellow inmate assumed all members had a castle.
Lord Hanningfield was sentenced to nine months in prison in 2011 after being found guilty of nearly £14,000 worth of expenses fraud. He served a quarter of the sentence in jail.
Opening a debate on what measures the Government was taking to improve education in jail, Lord Hanningfield told fellow peers: "I think you are all aware that I have been in prison myself.
"When I was first sent to prison, after I got over the terrible shock of it, I thought I had better try and do something with myself, and so I spent a lot of time researching and talking to fellow inmates about how they got there.
"And I found that so many, particularly of the of the young ones, were really unable to read or write, and the illiteracy amount in prisons is over 50%, I have since been told. Education in prison needs to be brought-up the agenda enormously. Education is right at the bottom of the profile in prisons now."
The peer said one fellow inmate assumed he had a castle and asked if he could use it for a rave.
"I found it very difficult, as you might imagine, in my initial days in prison. It's quite difficult for me to talk about it now. I really found it quite extraordinary, for example, general knowledge is very absent in a lot of prisoners.
"Hardly anyone had ever heard of the House of Lords, and I am not really quite surprised at that. So many people, for example, asked me where it was and, what did it do?
"And someone imagined that every Lord has a castle, because someone asked me if they could borrow my castle for a rave later on.
"Some of these people are rather intelligent and they could have a much better future if we could only do more for them, and we need to think how we can do more, both in education and training in prison," he said.
The Bishop of Peterborough, the Right Rev Donald Allister, called for reform, saying: "We talk about a patient-centred NHS, what about a prisoner-centred prison service?"
Government whip Baroness Evans of Bowes Park thanked Lord Hanningfield for talking about his experiences as she insisted "education must be at the heart of our prison system if we are to rehabilitate effectively".
Lady Evans said the Government was investing £1.3 billion in prisons to improve standards of rehabilitation, and a review of education in jail was under way.