Some people dream of owning a castle, complete with narrow slit windows and ramparts, spiral stairs and big baronial fireplaces. Maybe a priest's hole. When Aifric and Frank Gray found a stone tower for sale a few miles from Kilkenny with all this and more, they couldn't resist.

"There was a wee picture of Ballybur Castle, sitting in an auctioneer's window in Kilkenny in 1979," says Frank, who is Assistant Borough Engineer with Kilkenny Corporation. "Only in Ireland could you buy something like that for £20,000. I have a strong interest in building and conservation because I'm an engineer. I've always been interested in how people without engineering built such extraordinary structures, all by hand. We're so ruled and regimented, we'll never get that standard of building again.

"The tower is kind of pseudo-Norman, built by the Comerfords in the late 1500s and very defensive. The Papal Legate, Cardinal Rinuccini, is said to have lodged here on his way to the Confederation of Kilkenny in 1642. The cardinal's rosary, passed on through generations of castle owners, was donated to Rosse House museum.

"An architect from the OPW was bidding against us for the castle. He was from Scotland too. There we were, two Scotsmen bidding against each other for an Irish castle."

In the end, the Grays' bid won the day. For somewhere to live right away, they decided to do up the old stone outhouses at the back. But the draughts proved too much when their first child came along and they moved, first to Kells and then to Kenya for three years. "While we were in Kenya we drew up plans, so we knew exactly what we wanted to do when we came back in 1988. We did a complicated transfer with the farmer next door for a field behind the tower in return for the land in front with old farm buildings. Everybody was happy and we're now surrounded by farmland."

Ballybur Castle is at the end of a narrow laneway, dwarfing the pretty L-shaped cottage rebuilt in six months by the Grays when they came back from Kenya. They used old quoin stones and roof slates collected from demolished buildings. The arched sittingroom window was the original entrance to the building used to store the farm carts.

Inside, the cottage is all natural timber and bright primary colours. Clematis and climbing roses smother the walls of a suntrap terrace. The garden table is a slab of limestone left over from Kilkenny Arts Week. It's hard to imagine anyone wanting to exchange this, even for a castle.

"For the six months we were restoring the outhouses, we stayed in a house next to the pub in the village. We had three children by then, Mhairi, Ruan and Colm, who was born in Kenya. A flat was added on for Aifric's uncle, but he decided it was nicer in Bergamo, Italy. Mhairi has taken over the "uncle flat" now. Aifric had decided on curved walls in the hall, so the skirtings had to be cut to bend.

"We had been gathering lots of old slates from local builders and had a good stockpile. The roof slates were rescued from an old school which was being demolished. We were ripping off the slates while the builder was cutting the timbers underneath us. I've a love-hate relationship now with builders - I've been ten years with the same ones!"

Once the cottage was built, work started on the tower in earnest. A Heritage Council grant of £35,000 to redo the roof and battlements was a great help, but the job ended up costing £85,000. Previous owner Nicholas Marnell was born in the castle and his mother and sister had lived there until the mid-1970s without sewerage or running water.

"Everything had to be gutted. The beams were blackened and the floors were covered with six inches of dry clay which had to be swept away. We lime-plastered the tower from top to bottom because so much stone was oppressive. There's an Italian feel to the rooms now - I specially like the small rooms."

The tiny guards' room to the right of the main door was converted to a cloakroom and the main ground floor room is a kitchen, with a solid fuel stove and shortened church pews for seats. This room and the one at the very top have underfloor heating.

Here and there, small personal touches give the castle a lived-in feel. These include an Italian tile set into the wall and a colourful stained glass window by the Sheridans of Stoneyford, who made all the windows for the tower. There was no glass in the narrow windows originally, just shutters which were closed at night.

The most surprising thing about Ballybur Castle is the light. Though narrow, the windows have deep angled sills which deflect light very efficiently. On the first floor is the main bedroom, one smaller room and a simple and very elegant French-style shower room with a sloping floor. "Everything should be able to be stripped out and you would never know the difference," says Frank. "The loo is in the exact place of the original one, behind a hollow wall. You can see from the shape in the stone that it was a two-seater."

The third floor is a huge double-height dining room with an enormous circular light fitting from Willie Duggan's lighting shop in Kilkenny and a very stylish torch-shaped wall sconce from the same source. "I wanted a blend of old and modern. I've put a galley kitchen in here too because of all the stairs. We're toying with using natural paint colours in some of the smaller rooms."

One of these smaller rooms, on the mezzanine above the diningroom, looks as if it may have been a chapel. Because the original wattle and daub was too fragile to copy, they used a piece of old peat basket to roll a weave pattern onto the curved ceiling.

A magnificent baronial drawingroom on the top floor has a vaulted ceiling, left exposed to keep the room traditional and another giant chandelier. The fireplace, which was mostly in bits and pieces, was reclaimed. Off this is a secret room with a deep floor shaft, used as a priest's hole or for prisoners. A smaller flight of stairs leads up to the ramparts, where you can see as far as Mount Leinster and Slievenamon on a fine day. This precarious rooftop area is a favourite playground for Scruffy the dog.

Now Ballybur Castle is ready to furnish, Frank is looking forward to doing more work on the cottage. "The castle was my thing. We've been bringing back bits and pieces of French furniture over from our house in Brittany. It's really a multi-purpose place. We might rent it out, or it could be used for small weddings and gala dinners, craft exhibitions or workshops.

"We'll never move into it permanently, because there are 70 stairs to climb! But we had Christmas dinner in the tower and Ruan had his ninth birthday party disco there last February. It's a bit of fun for us all. I'd be happy to leave it as it is, if I could afford to, but that's not realistic. To me, its the kids' insurance. It'll be there for them rather than me."