Your Family Tree
Your Family Tree is a leading family history magazine available from all good newsagents. Click here to subscribe.Take a Look Inside The Genealogist
Richard Wentk investigates what goes on behind the scenes at one of the UK's Leading genealogy websites
Record offices and local archives will always be crucial to our family history research. However, our discoveries are increasingly taking place online using record websites. Many genealogy sites still assume that customers are primarily interested in access to digitised versions to paper records, delivered quickly in their home It's certainly true that being able to search millions of records on your PC from the comfort of your dining room is faster than accessing the smaller collection of records available from local and national archives. In fact to those used to reading records that were painstakingly assembled into difficult to find books, being able to search online at all can seem like a huge development. But so far, only a handful of sites are taking the next step and giving its customers smarter, context-aware search tools that can scan an entire collection of records and return individuals, sometimes with hints about their occupations, addresses and even their relationships.
Family affair
The Genealogist (www.thegenealogist.co.uk) is one such site. Set up as S&N Genealogy in 1992 by Sue and Nigel Bayley, it's a family business run by family historians. Mark Bayley is the head of the online division at S&N, and is responsible for many of the features and content of The Genealogist website. He explains how the business started, how it developed, and where it's planning to expand in the near future: "Nigel and Sue initially started selling a wide range of software and data products from Chilmark near Salisbury in Wiltshire – that was before the internet. But as the internet became more popular, they developed a range of online services from a new base in Jersey. The Chilmark offices are now the company HQ, and the offices in Jersey are the main administrative address."
The company employs Nigel and Sue's three sons and one daughter-in-law. David is the UK
Business: Sue and Nigel Bayley set up S&N Genealogy in 1992 and their current business HQ is at Chilmark, Wiltshire
office manager, Paul heads up the graphic design and digitisation departments, and there is also Becky, currently on maternity leave, to help manage the sales department. Mark continues: "As a family business, one of our main advantages is that we're not
Mark Bayley is head of the online division at S&N, responsible for many features and site content
very hierarchical. We have family members working closely with each other on a daily basis. The advantage of this is that S&N can afford to offer a great deal of customer service. "We differ a lot from other organisations, as all our staff engages directly with customers. We have a dedicated team doing telephone support, and that doesn't just mean you can ask us questions," explains Mark. "The support and the online development teams are close to each other, and that means that if a customer thinks of a feature they'd like to see, they can call us and suggest it. If it's practical, we can try to get that idea live online within a few days. It's one of the things our customers seem to like about us – we can get things to happen very quickly.
"Uploading raw scans isn't enough, records become valuable online when they are searchable"
Committed: Staff at The Genealogist are constantly giving the site a quality assurance check
High Technology
We reveal some of the software and systems that keep TheGenealogist up and running
TheGenealogist website looks deceptively simple. But as mark explains, there's a lot happening behind the scenes. "The site uses some huge image servers, with hardware and software we've designed and customised ourselves. We couldn't use off-the-shelf technology, it just wouldn't work for what we need," says Mark. The main challenge is the sheer volume of information that has to be stored, as well as the logistical task of keeping it backed-up in case of disaster. "Currently we have around 100 terabytes of image records and around half-a-billion text records," Mark reveals. "The records are stored on industrial computers that each have around 100 times more disk space than you'll find in a typical PC. All the data we keep is backed up, and then the back-ups are backed up again, so there are multiple copies of everything. This doesn't just keep the records safe; it means that if something goes wrong, a reserve machine switches in and the site keeps working before anyone notices there's a problem." Making the information accessible on the internet is another problem that the team had to solve, given that their rural Wiltshire base lacked the necessary internet connection to provide TheGenealogist users with a fast and usable search experience. Mark explains: "We looked at keeping everything in Chilmark, but it was too expensive to buy a fast internet connection to the middle of nowhere. Now we're looking at expanding into the rest of the country, so we'll have multiple machines in different locations. That will help speed up access and give our users a faster service."
Records are scanned and checked as they come into the offices
Searchable: Digitisation is done in-house at the Chilmark offices with a range of specialist equipment
Given that all of us have different genealogical needs, the process of picking which records to add to the site is a complex but important one.
And that's before the work of digitising the records for the website begins. Mark explains that it can be a long process, with the aim of keeping as many users as possible happy. "Customers sometimes ask us for record sets, and we do seek them out if we can. We can't always fulfil very specific or limited requests, but we have close relationships with various archives and societies and we're always looking for records that will be helpful and interesting" he says. We're currently negotiating with some record sources, so unfortunately I can't mention their names, but we can say that we talk regularly to The National Archives and the Federation of Family History Societies, as well as specific local societies. When it's been decided what collections have made the cut, the process of transferring them from paper to fully-searchable online records begins.
This huge undertaking actually takes place in-house, to ensure quality, as The Genealogist was keen to stress when it presented its high resolution scans of the 1911 Census earlier this year. "We have a range of film, microfiche and book digitisation and scanning equipment at our old converted barns in Chilmark", says Mark. "We scan the records as they come into us, and then do a first process quality assurance check. That involves checking that every record is in-focus, with the best possible contrast and resolution, and that there are no missing images. Book scanning can be a particular challenge. Some of the books are very fragile, and we use specialists & specialist equipment to ensure they're not damaged during scanning."
Detail checking
Scanning is just the start of the process. After the records are checked they have to be transcribed and processed for the website so users can search them, before being rechecked. Mark reveals how the small team at The Genealogist take on such a technical and manual challenge.
"Technically, we start with a file format called TIF, which gives the best possible resolution with no loss of quality. The images are compressed using a technology called JPEG2000, which reduces the massive TIF files to the smallest possible size losing as little quality as possible. We create different versions of each image, and the website checks to see if you're viewing the image on a PC at home over a fast broadband link, or on a mobile phone."
Mark adds that the volunteers are a valuable aid in the most important part of the process: making the records fully searchable. "It's not enough to simply upload the raw scans", he says, "records are valuable online when they're searchable."
"We create transcriptions of every record, because that makes it easier for our customers to search them. Transcription is partly voluntary. Anyone can take part via the UK Indexer site (www.ukindexer.co.uk). People who sign up can transcribe parish or other records that we send them. They can also voluntarily transcribe local records and send them to us. They can also do quality checks on records – for example to monitor the quality of our census records." The rewards for helping with such an important task include getting free credits to use on the website, or for other products – an incentive that means family history enthusiasts from all over the globe are involved in checking the quality of the documents.
Mark adds that experts are also on hand for adding the detailed knowledge needed in certain areas, such as checking UK place names.
The end of the process sees the records being presented to the world on TheGenealogist, but before that happens, they have to be expanded with search options so that users can find them easily. This is Mark's job, and he's keen to emphasise the TheGenealogist is about searches, not just records. "I manage a team of five developers, and between us we work on making searches as useful as possible. We have unique keyword searches that can scan our records quickly.
The search technology is what gives TheGenealogist a unique feature, and is something Mark and the team are building towards in the future, with the current goal that you'll soon be able to search all the site's records in one go. As Mark concludes: "Customers will get a much deeper insight into their ancestors. They'll be able to find everything we know about someone almost instantly with a single linked master search."
"Our new searches will allow people to get a much deeper understanding of their ancestors"
TheGenealogist has a dedicated team of problem-solving staff
Help is on hand!
You don't need to be a computer genius to use The Genealogists' clever online features
One of the criticisms of online searches is that they can be difficult to use. Mark is confident that TheGenealogist's customers don't find this a problem. "I go around the country visiting family history societies and explaining how to use the unique tools we offer" he says. "But you'll also see that our online help page is a full research guide. It has step-by-step instructions on how to use the tools. We also have a dedicated local support line. Our customers can click on a number and speak to someone in English who can answer their questions and make suggestions." The Genealogist's customer service is of paramount importance and staff are always happy to go that extra mile. "If the question is more technical or if it's a more specific customer suggestion, they can put callers through to the people who are actually working on the site," Mark explains. "We're proud of how that works, and of how quickly we can respond to suggestions."