About 20 years ago I started at a new employer as a Data Analyst with responsibility to prepare a data model for an application that had 2,000 tables and no documentation. As a newbie in a specialised industry it is not surprising that it was difficult to get access to subject matter experts for a task that was also not seen by them as being worthwhile.

So I changed my approach. I arranged to get a dump of the database metadata: all of the table and column definition data plus indexes and foreign key relationships etc.
From that I was able to:
- Prioritise and identify key tables and trace primary key fields to other tables.
- Establish what patterns and relationships existed (done manually as referential integrity not set up at all for performance reasons).
- Identify issues and anomalies to be followed up (and there were a few that surprised even the SMEs).

I learned a few things from that exercise:
- Respect the time of SMEs
- Try and access the facts
- Don’t give up
- Add value

A good outcome…
1. Produced data models for a substantial part of the system.
2. The SME’s and others learned the value of data modeling
3. Several unexpected bonuses (like when SME’s learned of comparisons between metadata downloads and how they documented ALL DB changes made in a release and not just the major ones the SMEs remembered).

What Nitin described is a scenario well-know. I’ve been through similar ones. You have a system that one needs to change, and no documentation is available. Reverse engineering is, sometimes, the only option left to go forward with one given project. What does it have to do with documentation?

If any form of documentation was available, maybe that task could have been easier to accomplish. By document I include: a picture of a whiteboard, a scanned piece of paper, a text file with instructions on how to obtain the numbers and their source, etc.

You got the idea. We need to break free of the concept of document as something formal, distant. We need to put the ideas in some form of long lasting format, and make it easily searchable and traceable.

Thanks again for posting

Laura

The team was struggling to get the flow and consolidate that to weekly, monthly & yearly reports. The task required interaction with tens of BPO staff without taking much of their time and yet understanding and documenting the data flow. The project went into red and the progress was escalated.

Being a damage control specialist, I had two weeks to do this and I was able to achieve this. The trick I used was Reverse Engineering.

What it requires is extreme dedication to the number crunching work and working backwards reading through all the macros and formulas used in these excel sheets until you reach the source file. Document these in different process flow diagrams and send these for validation to all relevant business users.

The benefits are as follows:
1. It saves lots of user time in explaining the flow and the data processing a group does.
2. It gives you in-depth understanding of the flow which makes your meeting time with the user quite effective and you can then restrict this to clarifications and probable enhancements requests.
3. You have documented and cross checked the data processing manually after the flow is documented in place which drastically reduces the scope for any significant corrections
4. In addition you have already delivered substantial part of your analysis to the business teams and IT teams when you have sent this for review, which would enable the IT team to start the thought process / work in parallel while you can still work on minor corrections or enhancements.

Now a days you do get tools that will give you all interfaces and data flow diagram but when you are dealing with hundreds of excel sheets, I believe reverse engineering is still the best bet.

As a business analyst, a tool in my bag of tricks that I’ve used often in addition to either functional decomposition diagrams or the process flow diagrams is the state transition diagram.

In my work, state transition diagram (std) comes in very handy in social services application or systems that are “Case based” and where formal “workflow” tools are not available. When creating, reviewing or simply presenting std to the user community (especially during requirements verification), I never refer to the diagram with its formal name. I often refer to the diagram as “case” flow diagrams (please do not confuse this with Use Cases) – here, a case refers to an actual “case folder” that a user maintains to track a “case” or and incident – example: a family seeking social services from an organization.

Using an std, we can look at the “triggers” or “actions” or events that occur for a case to move from one status (state) to another. It is also useful in answering the question why is this “case folder” not moving along?

One major benefits of an std is to clearly see where cycles emerge or are likely to emerge. Other benefits include: 1) existence of too many “states” or “Case Statuses”; 2) The presence of what I call “phantom” events – these are events that take up too much of the users time and effort but do not add useful information to the case or in moving the case along; 3) complexity of a business process. This third benefit becomes very useful during requirements validation in constructing case scenarios.

In summary then, State Transition Diagram is definitely one tool that is always in my bag of tricks.