It was clear from the start that my interview with Lord Longford was going to be unusual. For some days telephones remained unanswered. Finally at 11.30 on Sunday morning (a post-Mass, pre-lunch window of opportunity), he answered. "What about this afternoon?" But I hadn't read his forthcoming Prison Diary yet, I said. "But you know something about my work. You know who I am." I did. "Well this afternoon would suit me. Tomorrow I'm in the House of Lords. Come at five. No at six. At six I have a glass of sherry. We can have a drink and you can spend half an hour reading the book before we start."

Francis Aungier Pakenham, a.k.a. Frank Longford, seventh Earl of that name, was born in London in 1905, although he now considers himself totally Irish. A gift of confiscated land in Co Westmeath was given to the Pakenham family by Cromwell in lieu of wages after Henry Pakenham changed sides in the English Civil War. An Irish barony soon followed, raised to an Earldom 30 years later in 1785.

During his childhood, the family lived most of the year in England. Holidays were in Westmeath but Pakenham House, as Tullynally was then called, "always remained home". To the Irish he was English, to the English he was Irish. This natural identification with the outsider goes some way to explain the apparent paradoxes which have mapped his life: the switch from uncommitted Protestant to committed Catholic, and from "unreflective" Tory to die-hard socialist; and the empathy of the aristocrat with the social outcast.

Longford says that he always considered himself Irish, but history relates that it was his elder brother, Edward, whose allegiance was never in dispute. At Eton Edward was bullied, and when both boys were at Oxford, the elder brother was physically attacked for his pro-IRA views, his rooms wrecked and his collection of Gaelic books destroyed. He never wavered and made his home in Ireland, at Tullynally, devoting his life to the theatre.

It was only when he met de Valera in 1932, that Frank Pakenham was fully converted to Irish nationalism. He was 26, and - fresh out of Oxford with a totally unexpected first - had landed a job with the Conservative Research Department. The "Irish Question" was not being given the attention he felt it deserved, so he decided to speak to de Valera personally.

"De Valera is the greatest man I ever met," he says, with a fierce passion that threatens to overwhelm his frail frame. "A wonderful man. When I was a boy in Ireland I remember the children were in rags. Now their standard of life is the same as the English. And it was de Valera who did it. I went to interview him and I fell under his spell, so to speak." (He later went on to write Peace by Ordeal, a study of the Anglo-Irish Treaty of 1921, and still the book he is most proud of.)

We are in Lord Longford's small sittingroom in his London flat. He opened the front door himself, moving inexpertly with a Zimmer frame in the confined space. Afternoon sun pours through the window via a small balcony; a silk square lies discarded on the back of his chair next to a white straw hat. Lady Longford has just returned from a lunch at the Tower of London. "They are laying some plaque on Wellington in the chapel in the tower. She wrote the classic book on Wellington and Queen Victoria you know." He wasn't well enough to go - a nasty cough, he explains - hence the free afternoon.

Far from giving me time to look over his Prison Diary, he has immediately poured me a hefty three inches of Amontillado sherry, the same for himself.

The room is a jumble of books, papers, black-and-white family photographs. In the corner one large classical landscape seals the image of genteel poverty. It could all do with a good tidy, as indeed could his Lordship. The shambolic nature of his appearance should come as no surprise: this is the fellow who while he was at Oxford would descend on the family of Lord Birkenhead - his best friend's father - walking from the local station dressed in all the clothes he considered necessary for a weekend away. Other guests would watch in astonishment as trousers would peel off to reveal a pair of shorts for tennis.

The wings of hair that frame the balding pate are still there, but the "terraces of curls" of his youth are now candy-floss wisps. Longford no longer wears the trademark pebble-glasses, and I suspect he can barely see.

"You're too young remember," he says, referring to an incident when he was in Harold Wilson's short-lived Labour government in the mid-1960s. I laugh. Sensing sport, he adds "How old are you? 25?" I laugh even louder and he beams with delight as his flirtatious remark hits home.

I tell him that I share a birthday with his youngest daughter, Catherine, who died when she was 23 in a car crash. He closes his eyes. Only after a long silence does he open them. She was a gifted journalist, he says. Did I know her? No, I reply. Was I married? Did I have children? He listens attentively as I answer as fully as he obviously requires. "Sounds like your husband was rather an ass. I will give you a signed document."

It's hard to keep the interview on track. He gets bored if asked something he feels he has said already. Take his thoughts on New Labour. He recently wrote an article that dealt with the subject in the Catholic Herald. "Why don't you call me after you've read it."

Lord Longford is of course Old Labour. When Tony Benn resigned his peerage, Longford hoped to do the same - when his elder brother had died, the Earldom quite unexpectedly had passed to him. Unfortunately this proved impossible as he had already accepted a separate peerage from Clement Attlee ("like a life peerage really before they had them") as Baron Pakenham of Cowley after he failed to win the safe Tory seat of Oxford in the Labour landslide of 1945. ("I didn't lose the seat you understand. I just didn't win it.")

Pakenham went on to become the Leader of the Labour Party in the House of Lords. Having been invalided out of the British army in 1940 for what he now claims was a nervous breakdown, Attlee made him Under Secretary of State at the War Office, Minister for Germany. His Catholicism was becoming increasingly overt, and contemporary accounts suggest that the eccentric English nobleman, who went down on his knees at the slightest opportunity and dressed like a gardener, was a baffling figure to both the British and Germans.

Asking about his past elicits an irritated "Haven't you read my autobiography?" quickly followed by an apology for this lapse in gentlemanly conduct. "What I don't like doing, is having to spell it all out, boasting about how I was in the cabinet. I don't like having to mention it and I don't particularly like trotting out my CV."

He is happy, though, to put the record straight in relation to his great political volte face. "I was a don at Oxford, teaching politics. I had done modern greats, PPE, politics philosophy and economics, so I knew all those subjects better than most people in politics. But I had originally intended to be an economist, and I never ceased to ask myself whether the vision of a socialist society was compatible with sound economic thinking. I was living in Cowley, a working-class area, and I became close to the Cowley [Labour] party through my wife, even before I became officially Labour myself. So I had the two influences, the social influences of the Cowley Labour party though my wife and my own academic studies. It wasn't sudden."

In 1931 he had married the beautiful and brilliant Elizabeth Harman, daughter of a Harley Street eye surgeon who had taken Oxford by storm. She is now 90 herself. "My wife is exceptional. I suppose everyone thinks that - if their marriage lasts anyway. She had eight children and 26 grandchildren and 16 great grandchildren, and she is very able and very good tempered. Perfect." (As a husband, however, he considers himself to be only "a good second-class husband".)

"My wife would say that she made me a socialist. When we were married, she was already Labour but I introduced her to the Labour party really, through the WEA (Workers' Educational Association). Anyway she fell in love with it and then I became a Catholic in 1940 when she was still very much against Catholicism, then she did."

Prison visiting began as an extension of his Labour party "social work". ("It became natural to go down further and further, and eventually prison.") His social conscience had been alerted while still a schoolboy by an uncle who, as well as being a director of Barings Bank, devoted his life to running Eton Manor in Hackney, the largest boys' club in Britain. Now, however, the impetus is clearly religious, taking as his text: "I was in prison and you came to see me."

"People say to me, why don't you do more for the victims? But I tried to do what I could. I introduced a Bill in the Lords for the victims. Of course if you're a mother and your child has been murdered you would feel differently. [Denis] Nilson, for example, strangled 12 people. I mean what he did was disgusting. But I don't hate him. I follow the motto of St Augustin: `hate the sin and love the sinner'. It's very, very difficult to do and society can't do it altogether. It's a rather complicated thing, especially when they don't know the person. Take my poor friend Myra Hindley."

In recent years Longford's name has become synonymous with the campaign to free "Moors Murderer" Myra Hindley, the woman still regularly described by the tabloid press as "the most reviled woman in Britain". The horrific murder of a number of small children on the Yorkshire Moors in 1965, by Hindley and Ian Brady, shortly after the abolition of capital punishment remains an open sore in the British consciousness, in part because some of the bodies have never been recovered.

Longford is convinced Hindley will eventually be released. "It will certainly happen, though probably not in my lifetime. She is no danger to anyone, and the present Lord Chief Justice, Woolf, the most liberal Lord Chief Justice there has ever been, the most enlightened man - and his appointment is the best thing the government has done so far - he's made some remarks. They don't bear on her directly, but he has said it is wrong to tell anyone that they can never come out."

Longford's interest in other people, whether murderers or visiting journalists, is unfeigned and completely genuine. "I like to converse with them. I enjoy their company and their conversation." We have spent a good deal of time talking about my personal involvement in the Craig and Bentley case. Christopher Craig murdered a policeman for which his accomplice Derek Bentley was hanged. (Bentley was exonerated in 1998.). Craig was underage for execution and was imprisoned instead. Lord Longford talks about the then-notorious Lord Chief Justice Goddard. "I remember him in a debate on the abolition of capital punishment saying `such a man should be destroyed'." (He mimics the over-the-top plummy voice.)

Lord Longford had visited Christopher Craig when he was in prison. "I knew him well enough so that when he got married he asked me to go to his wedding in 1965. But I was in the cabinet at that time, with a very small majority, and Wilson begged me not to go and so I didn't. But I gave him and his fiancee dinner at The White Tower the night before as compensation. I knew his brother, Niven, rather better."

Since he first began prison visiting in the 1930s, Lord Longford has averaged two visits a week, except for the period during the Attlee government and when he was in the cabinet. It's an impressive record by any standards. For anyone in their 90s it defies belief, especially as he travels on public transport, usually walking to his local railway station in Sussex when the Lords are not sitting, following the tortuous route of trains and buses that prisoners' families take. On the rare occasions he does travel by car it is usually "in a hired car provided by the generosity of Harold Pinter", who is married to Lonford's eldest daughter, Antonia Fraser. As frail as he is, Longford receives no special treatment and has to undergo the searches (checking he hasn't got drugs in his socks) and cavalier treatment meted out to any other visitor.

Prison Diary is an edited account of his visits to prisoners - always at their instigation - between 1995 and 1999. Although Hindley and Nilson are here, the majority of names will mean nothing to the general public. Although some maintain their innocence, the majority do not. What they want from Longford is help in transfers, education and health issues. Quite likely they are also seeking the company of someone who looks for the good, who is not after anything, and who is genuinely interested in them as people.

He doesn't go along with the current view that everything is a result of childhood abuse. "I'm too old. I don't think that it's the result of brutality. It's all very human. People who have committed crimes have to be punished and go to prison. Public opinion is hostile; people don't know anything about prisoners. They don't like them, but that's also human, it's just that they don't know anything about them. That's why I'm a penal reformer."

Doesn't he get angry at the system that still lays the emphasis on punishment rather than rehabilitation? "I don't waste my energy getting angry with people who don't agree with me. I'm not good at being angry, except at that," and he wags his finger at the Zimmer frame. "I'm not an angry person. I'd rather describe myself as Bonhoffer did. He was a pastor in Nazi Germany and everyone said he was a saint, because before being executed he said `I am a contemptible woebegone weakling.' "

Lord Longford is getting tired. I rise to leave and in spite of my protestations, he hauls himself to his unsteady feet. I extend my hand, but he does not shake it, just lifts it to his lips, kisses it and says: "Now what can I say that is nice. . . Considering you don't know anything about me, I think you've done a wonderful interview."

Lord Longford's Prison Diary will be published tomorrow by Lion, £14.99 in UK