The Digital Document Lifecycle

MARTYN RICHARD JONES

To begin at the beginning

This is a story of the life of a digital document. Its purpose is to explain the process of analysing, designing, building, testing and delivering content rich business artefacts in today’s digital age.

Many instant Enterprise Content Management experts will tell you that the lifecycle of a document consists solely of its creation, classification, storage, retrieval, distribution, use, archival and disposal. This doesn’t even tell us a fifth of the story, a tenth of the real requirements or a twentieth of the true objectives. It is essentially the naïve techno-bullshit of absolute beginners, and ignores a surfeit of business, legal and social aspects that are in play.

But I digress.

Now, whilst this piece might not make for the most compelling of reading material, it does focus on an increasingly important aspect of business life, one that I believe we should try to get right, first time, and every time. After all, this has been done many times and in many places in the past, so with all that accumulation of knowledge and experience, we should know how to do it right now, right?

In order to bring the explanation of the process to life, I have chosen to talk of a real life example of a business document rather than use a more abstract concept such as digital content or an even more ephemeral generalisation of all enterprise documents.

The document that I have chosen to illustrate this piece is one known to many millions of people throughout the world. In Spain it is called a factura, in Germany a Rechnung, in French it’s a facture d’achat, and in ‘the land of leeks, dragons, bards and daffodils’ it’s called an anfoneb. However, for the purpose of this piece I shall use the term Invoice.

What is an invoice?

We all have an idea what an invoice is, but what is it?

According to Wikipedia, an invoice is “a commercial document issued by a seller to a buyer, relating to a sale transaction and indicating the products, quantities, and agreed prices for products or services the seller had provided the buyer.”

So far, so good. But what about putting some more detail on the frame?

Items that might typically be found on an invoice are things like:

1. The word ‘Invoice’
2. The date of issue
3. An invoice serial number
4. The names and addresses of both buyer, seller and recipient (if different to buyer)
5. What has been provided (Such as services, goods or performance)
6. Date of delivery of what has been provided
7. The monetary amount charged for goods, services or performance provided. The amount owed.
8. Amounts resulting from the application of tax, sub-totals and gross amount
9. Currency and methods of payment
10. Identifications number(s) for taxation purposes
11. Other legal requirements

Clearly this is not an exhaustive list, but it serves as a means of orientation.

As we know, an invoice is not to be confused with a receipt. An invoice is a request for payment. A receipt is confirmation of payment.

Also worth mentioning is the fact that there are many variations on the theme of invoice, and of course, in most business to business and business to consumer settings it is a little bit more complex than the explanation given above, but this (again) gives a flavour of what a typical invoice is all about.

Scenario

Now I will set the stage, the scene, the setting for the story.

The Mendal Metatron Corporation, a global business that engage in providing comprehensive data architecture and management services to corporate clients on all seven geo-political continents, has decided to introduce a next generation architecture to homogenise – to the degrees possible – their enterprise information systems. As part of this initiative they have decided to use a Content Management system for the generation of all invoices, in all subsidiaries, and in all locations, across the globe.

It has been decided that the pilot project will be carried out in the welsh market-town of Caerfyrddin, the base of Mendal Metatron’s global Big Data, Analytics and EDW operations.

Walking through the process

Now I would like to walk you through the process that we would typically go through in order to explore the requirements mentioned in our scenario. It’s a relatively simple and intuitive process that makes sense on a number of levels.

Fig. 1 – The Digital Document Life-cycle

1. Identify – At a high-level identify the document you want to work with and what you want to do with it.
2. Plan – Based on high-level requirements work out a draft high-level plan, from analysis to production availability.
3. Analysis – Carry out an iterative formal analysis of requirements.
4. Design – Carry out design activities based on the analysis.
5. Done / Completion of analysis and design – When you think you’re done, check that you’re done. It helps to know beforehand what ‘done’ should look like.
6. Build acceptance – Check that your requirements make sense to the builders.
7. Build – Get the artefacts and the associated processes built.
8. Test – Test what has been built. From unit testing to production acceptance testing-
9. Off-specification – What happens if the test reveals an off-spec?
10. Design change – What happens if the test provokes a design change request?
11. Function change – What happens if the test provokes a functional change request?
12. Clarification / Feedback – the clarification and feedback of defect issues or change requests.
13. Deployment decision – Are we ready to deploy?
14. Sunset / shelve – Do we shelve the build? Do we sunset the artefact, process or any other aspect?
15. Deploy – Putting things into the ‘production’ environment.
16. Lessons learned – Discuss, compile and socialise lessons learned.
17. Reviews – Carry out period reviews of potential changes and change requests. Initiate an analysis, design, build, and test and deploy iteration if it makes business and operational sense.

In effect we identify the artefacts, the features and the associated processes that we wish to focus on, in our case the invoice document and the processes of invoicing. We create a high-level plan of action. Then we jointly enumerate and analyse the requirements coming up with ideas and solution designs as we progress. We finalise our design and also finalise the socialising of the requirements with the developers/configurators. The requirements are then turned into technical products. We innovatively test what has been produced as we develop and – more traditionally – after we have developed. We try to ensure that User Acceptance Testing is a mere formality, more of a social event than a true make or break point. We ensure that the products can be put into the business production environment, and then we go live with our products and processes.

Afterwards we review the lessons learned. On a periodic basis the products and processes in production will be reviewed.

It’s that simple.

Now I will address each of these aspects in a little more detail.

Identify

Here we identified the artefacts required and how they fit into an overall high-level business process. Here the emphasis is on having a clear idea rather than a detailed idea. The details will come later.

Most importantly, at least from the eclectic perspective of the author, the other key factors to identify and have well defined are:

1. Who is the real client
2. What does the real client want and expect (be very careful with this. It is not what you think they might want, it is what they want, regardless of the words they might use)
3. Who really has the final say, and
4. Who pays

Plan

In this scenario the plan is simple. The goal is to analyse the invoice document requirements in detail, to design the artefacts and process, and to build, test and deliver what is required.

A rough estimate of time required for the end-to-end development process is pegged at 2 months. Which will give IT Services time enough to put the pilot infrastructure in place.

The plan is developed jointly by the Project Manager, the Infrastructure Manager, the Business Analyst, the Content Management Architect and the business representatives.

Fig. 2 – The Draft Plan

Fig. 3 – The Draft Resource Plan

Analysis, Design and Done

The way we went about analysing and designing the invoice was as follows:

1. We gathered together examples of business as usual (BAU) and legacy invoices and analysed the content
2. We then carried out a series of requirements workshop to storyboard the content, function and form of the TO BE invoice.
3. In the workshops we explored different template designs, the fixed fields (such as “Invoice”, “Date” and “Address”), the variable fields containing, for example, the actual date and address and the conditionally displayed fields (fixed and variable).
4. We also discussed other wide ranging issues such as the adherence to corporate style guides, legal and regulatory requirements and customer acceptability.
5. We also looked at innovative ideas such as providing additional information to the client regarding their relationship with us and their purchase and accounts history. This also resulted in idea generation that was passed to the Data Warehousing/Statistics/Analytics/BI teams for further research and consideration.
6. Once we had the design fixed on paper we transferred the design to a pixel perfect document template designer. This was also very much a joint exercise. Laptop, projector and required attendants.
7. Around this point we involved the developer who would be responsible for ‘making our dream a reality’.
8. At the end of the exercise we had socialised our requirements and we had created three key artefacts.
   1. A business acceptable invoice template designed to pixel perfection that follows the corporate style guides
   2. Metadata for fixed, variable and conditional fields and graphics (such as table and cell borders, logos and other devices)

• Business rule set governing the variable and conditional data fields

In our example case, for the duration of analysis and design we reserved a dedicated meeting room papered from floor to ceiling with space to brainstorm ideas and designs. The following illustrates the sort of thing we were doing at the initial stages of analysis and design:

Fig. 4 – Brainstorming

From such diagrams, and with input from all – including developer/architect – we were able to produce the required analysis, design and development artefacts, including a clear, coherent and comprehensive requirements document.

You will notice that in our storyboarding that we mixed analysis and design, as both, are in reality, inextricably linked when it comes to producing great products that people will want to use – as Steve Jobs and Akio Morito knew full well.

As part of this phase we also used checklists and crib-sheets so that no significant factor was left unchecked.

For example, the following items were considered as part of this phase:

1. Fixed items – logos, fixed text such as legal items, descriptive text, such as ‘Date’, ‘Address’, ‘Email’, etc.
2. Variable text, such as the data itself, invoice line items and amounts.
3. Conditional text, such as text appearing in English or Welsh or both. Regulated by the application of business rules to content.

It’s also worthwhile to note that after each analysis and design iteration the artefacts that resulted from the sessions were all assigned product and product version numbers, so that they could be traced through the end to end process and the lifetime of the production artefacts. Versioning and change management is an absolutely fundamental part of Content Management and it cannot be ignored without taking on additional risks, costs and unintended consequences.

Here’s is an example view of the flow from analysis and design to development:

Fig. 5 – Analysis, Design and Development

Also, as stated before, the Content Management architect was involved (albeit over video-link) in parts of the analysis and design phase, to ensure that what we required could be delivered and that we were choosing optimum options where it mattered, and to ensure seamless continuity between analysis and design, and development. This is a very important aspect that cannot be over-emphasised.

One final comment on this part. The Mendal Metatron Corporation never ever skimps on providing an adequate infrastructure to support more than satisfactorily the process of analysis and design. Their belief is that if you can’t be bothered to adequately support analysis and design in the first place then you don’t really need the project deliverables, in which case it is a waste of valuable time, patience and money.

Build acceptance

We have golden rules at Mendal Metatron, and one related to IT development is that “Nothing is passed to development without it first being adequately socialised with the development management and staff.” This means that before the development team, any development team, whether in-house, outsourced or offshored, is fully aware of what is required, timescales, budgets, costs, quality and what fundamentally needs to be done, before they make any commitment to carry out the work.

Build

We also insist that development is done in phases, so that overruns are either avoided or accepted or mitigated.

In this particular case the developer was onsite for the first two weeks of development and was able to interact directly with both the BA/PM and the business. This phase of the development was an agile approximation to the Joint Application Development approach, but applied to the development of a digital content artefact i.e. an invoice.

Test

How did we do our testing?

We are dead-keen on the Risk and Requirements Based Testing model (RRBT), and we combine this approach with the philosophy of ‘test early and test often.’

Fig. 6 – RRBT Test Phases

Let’s just quickly go through each of these testing aspects.

1. Development testing. This is the type of testing that a developer will typically carry out as they build a solution component.
2. Unit testing is more rigorous testing of a unit of code or sub-component, and is ideally carried out by someone other than the builder of the component. In practice this very rarely happens.
3. Smoke test – This is a quick-kill test, quick and dirty, as the name might suggest. Rather than waste time with extensive Systems Integration Testing, each and every time something gets thrown over the wall, the idea is to catch some fundamental flaws quickly and then send it back for remediation even before SIT time is burnt and wasted.
4. Systems Integration Testing – A thorough and comprehensive test from a purely technical, operational and systems integration perspective. It is end to end testing for engineers and scientists.
5. User Acceptance Testing – In an ECM context this end-to-end business testing should be a formality. Why? Because you should have used User Participatory Testing from the get go, that’s why. If you didn’t do UPT then do UAT, and rue the day you chose to ignore sound advice.
6. Production Acceptance Testing – this testing is in place to ensure the correct hand-over of required products to those responsible for the day to day running, maintenance and availability of what has been built.

Design change

Design change that crops up in testing?

So you are in a test phase and someone realises they want a design change. It’s not beyond the realms of all possibilities, but considering the wiggle room already set aside in analysis and design, it really exposes some pretty sloppy upfront thinking.

Okay, if the design change is really important, like changing the iPod headphone colour from nausea-inducing puce to Alpen white, then just do it.

That said, make sure that the business agrees that the change is necessary. Otherwise, just don’t do it, and move on to the next stage.

If you decide to accept the design changes then you must go right back to the analysis and design phase and start from there (no passing go, and no collecting $200,000 USD, Roswell Monopoly style!).

Design change that crops up in production?

No problem, just start another iteration.

Function change

Function change that crops up in testing?

Okay, at the highest level of abstraction, we pretty much approach function change as we do for design change.

Here are factors and questions to consider:

• Do you really want this function change now?
• Does the business want this function change now?
• Are the changes really necessary?
• Three affirmatives and you’re back to the analysis and design phase.
• Are you sure you don’t want to ask the audience or call a friend?
• Are you absolutely certain about this?

Design change that crops up in production?

No problem. As for design changes, just go through another iteration of the whole process, skipping the superfluous bits, but including all of the players and actors and necessary aspects, as highlighted previously.

Clarification / Feedback

The clarification and feedback of information aspect provides for:

1. A point to collate and rationalise any requirements for functional or design change, or for feedback regarding defects.
2. A chance to think again about potential design and functional changes.
3. An opportunity to fully evaluate costs, impact and desirability of any changes before they are reintroduced into the process work-flow

Deployment decision

This is the single-point-of-decision, sometimes called the go/no-go decision point. It should be a formality. Most often it isn’t. With this approach to digital content it most definitely should be a formality. If not, you haven’t been paying attention.

Everyone should be involved in the deployment decision, but only the business and/or the operational/support team should have a veto. Which in real terms means that the business decides, and no one else.

Sunset / shelve

So you decide to sunset, upgrade or both. Okay, this is normal, and is a customary part of such a process.

Deploy

Okay, let’s get this baby off the ground.

This means deployment, but more than deployment it means use, use by users, business users.

Lessons learned

Organisations that learn and remember and recall do so creatively, consciously and conscientiously. That’s how winning corporations such as Mendal Metatron approach such opportunities to protect what they know and to expand their range of real and tangible possibilities.

Here was ask questions such as:

• What did we learn from the process?
• What things should we avoid in the future?
• What things could we do better in the future?
• What controls could we put in place to ensure that we do the right thing, right?

The Lessons Learned concept and process is explained more fully in literature that covers subjects such as Knowledge Management and Intellectual Capital. Ignore this aspect at your peril, especially if you are in an organisation that pretends that its business depends on knowledge and its effective and efficient acquisition and management.

Reviews

This is when a review is made of items that are being used on a habitual basis by the business. It is a continual process of reviews, suggestions and evaluations. For example:

• How are the products doing in production?
• Do we have any issues with the existing products?
• Do the existing products set too many artificial limits on business agility?
• Could we do things better?
• What does better mean?
• What degrees of satisfaction do we have with the products as they are?
• What would make the products better, more usable, less onerous and disruptive?

Reviews are tailored to the strategy, tactics and missions of the organisation, the needs of business users and the realities of each given situation.

Summary

So, there you have a complete end-to-end description of the Digital Document Lifecycle as envisaged at the amazingly prodigious and fabulously rational Mendal Metatron Corporation.

1. We started with the identification of a high-level identification of the document we wanted to work with and what we wanted to do with it. The document we chose was the Invoice, and what we wanted to do was to generate it, store it and distribute it.

Tip: Try and start with something tangible and realistic. Something like ‘providing for all the content needs of the corporation’ is not a good place to start from. Creating a strategy for content management should be done before creating digital document lifecycles. I know this rarely happens in practice, and this is why so many ECM products end up in the shit or as moderately unsuccessful and expensive database projects.

2. Then we drew up a high-level plan that would form the basis of taking us from ‘good idea’ to ‘done’. This is not a detailed plan, but it does contain all the high level elements, and of course the project products, both technical and management products.

Tip: Make a high level plan. Plans are good. Don’t fear them. Done right, they send out a strong, credible and respectable message of intent.

3. Next we went through an inclusive analysis and design cycle. The analysis being closely tied to the design.

Tip: Analyse in ways that stretch the corporation to think and imagine and innovate.

4. The design being closely aligned to the analysis. And both aspects involving business users, management, analysis and design and build staff in every step.

Tip: Design to innovate but ensuring multi-faceted correctness and fit, and don’t design to be so different that you seem to be doing so for a completely separate entity that has nothing to do with the culture, structure and orientation of the business in focus.

5. Then we defined what we meant by being done in terms of analysis and design. This means that we’ve conceptualised what we imagine we need then turned that into concrete, realistic and implementable ideas, which make business and technical sense. Here we should have achieved the goal of excellence in form following function, and in the architecture and design of both.

Tip: Ensure that you understand the meaning of ‘done’, and that everyone involved understands the same.

6. Next we ensured that what we wanted to be built, and all the relevant parameters associated with that, were also acceptable to the builders. We don’t just throw things over the wall to development and then hope for the best.

Tip: Keep the stakeholders, even the developers and testers, actively involved. They might not like it, because they are techies and shy away from more touchy-feely interactions, but just do it anyway. With sensitivity, of course.

7. Next up was the build process. Guess what? We also involved business users in this aspect as well, in the form of User Participatory Testing. Yes, I know, we’re not just good at this, we’re great.

Tip: Engage with the stakeholders and keep them involved. Yes, it is worth repeating.

8. After the build and build testing, and the use of many relevant aspects from agile development, including pair-development, we get to the final set of evaluation tests.

Tip: Drive out all the important defects and off-spec issues early and often. Turn the later tests until mere happy formalities. Yes, it can be done, even in your exceptional environment of exceptionally different peoples and processes and cultures.

9. We took a look at what happens when the build is not aligned with the requirements. Off-specification – What happens if the test reveals an off-spec?

Tip: Be rigorous with testing and fixing. Don’t mess around with and debase a proper, rigorous and coherent risk and requirements based testing regime.

10. Then we looked at what happens when design change requests come through. – What happens if the test provokes a design change request?

Tip: There is a place for everything and everything in its place.

11. Which was quickly followed by some wise words on functional changes.

Tip: Ensure that you get the analysis and design phase right. This isn’t rocket surgery, it’s about simple documents, content, events, actors and processes, at most.

12. Then we outlined the process of moving from off-spec, design change or functional change back down the process change to analysis and design. And please note, this process doesn’t have to be painful even in the most extreme of cases.

Tip: Never stop socialising. It’s simply about clarification, information, requirements and feedback. So don’t sweat the small stuff. That is, unless you really don’t know what you are doing.

13. I also mentioned the deployment decision and the singular question of “Are we ready to deploy?” Remember, never deploy anything that should have been strangled at birth. Not even to salve egos. It’s just not worth it.

Tip: Paying millions of Euros for a piece of crap that doesn’t deliver anything is one thing, pretending that you’ve put that piece of shit into production because you ran it once on a ‘production’ platform simply adds insult to injury, and contributes to the debasement of terms such as ‘professional’, ‘integrity’ and ‘expert’ and insults the ethically minded people who try and do a good professional job. Be professional! Be ethical! Be good!

14. I also mentioned product ‘sun-setting’, ‘shelve’ and ‘renewal’. Do we shelve the build? Do we sunset the artefact, process or any other aspect? Do we restructure, renovate and renew, or do we knock it down and start again? Or simply, knock it down and forget it?

Tip: Don’t think that anything will last forever. Be comfortable with that idea.

15. I also mentioned deployment and putting things into the ‘production’ environment. This is an aspect that very much changes from site to site.

Tip: Celebrate success… always.

16. Here’s a beauty! Lessons learned. This is where we discuss, compile and socialise lessons learned. I hope you know what lessons learned are by now, if not then look it up on google. But, let’s face it. This is a great and valuable exercise, however, if you find yourself in an organisation that cannot (or refuses to) do Lessons Learned, then the chances that they will want to follow a rational, reasonable and repeatable coherent-process are about zero. Here’s another fact you can take to the bank: IT services companies are almost invariably ‘Lessons Learned’ dodgers. Notoriously so.

Tip: Don’t disrespect ‘lessons learned’.

17. Last but not least I touched on ‘Reviews’. We should, of course, carry out periodic reviews of potential changes and change requests, and then initiate an analysis, design, build, and test and deploy iteration, that is, if it makes business and operational sense.

Tip: Don’t forget to never stop thinking about consolidating what you have, making things better and growing your ability to transform and innovate.

A final word of advice. Never, ever use Scrum for this type of development. It’s the worst possible choice of approach that one could ever contemplate using with respect to the analysis, design, development, testing and delivery of content rich artefacts and technical products. And that’s no joke.

That’s all folks!

So there you have it, the lifecycle of a digital document, in this case a humble yet powerful invoice. I hope you have find the journey worthwhile.

One last tip before I end this brief-hiatus away from the big brave world of Big Data, Analytics and 4^th Generation Enterprise Data Warehousing.

When it comes to a real-life challenge in Content Management and the analysis, design, development, testing and delivery of digital artefacts, then give it the importance that it has, not the importance that personal needs, foibles and marginal requirements might dictate, even against our better natures. It’s not as trivial as it may appear, but it is definitely not as important as some people pretend that it is.

Perplexed? Mystified? Baffled? Good!  It’ll give you something to think about.

Many thanks for reading.

 

MARTYN RICHARD JONES

Sanclêr, Caerfyrddin, Wales – 23^rd September 2014