What You (And I) Didn’t Know About Automating Pipeline Inspection Reporting

I thought I knew everything about the subject. I was wrong.

I’ve worked in data management since 1984, with decades of experience in data collection in the field and in both tabular and spatial models. Automating the pipeline construction inspection reporting process seems straight-forward: collect inspection data on a tablet device, upload it, and report the findings on a website. This is a typical mobile data collection scenario that’s been used across many industries for years. I thought I knew everything about the subject. I was wrong.

Here’s are a few lessons I learned after working with thousands of field users and tens of thousands of reports utilizing Mobile Inspection Forms.

1. No, we don’t have wi-fi or cellular connections all the time.

A right-of-way-based data collection solution must assume a total lack of connectivity. Designing a solution that allows for off line data collection that uploads data in the evening isn’t too difficult until we start considering the details. For example, an environmental inspector’s daily report records a damaged silt fence. This action needs to quickly trigger and assign a follow-up report to validate that the fence has been fixed. Today’s form answer dictates tomorrow’s form assignment. This complicates the model and presents a challenging workflow. Now we must heavily leverage connectivity at every moment we have it to pull and push data to where we need it before tomorrow morning.

2. This calculation is wrong.

Running calculations, like linear progress, looks so simple…. [end station] – [beginning station] = [progress]. But is it really? Are overlaps between inspectors counted twice, or eliminated for reporting? Are gaps between inspection progress assumed to be closed, or subtracted from the total? Every project is different, but people move from project to project and assume that the calculations are the same. If a user does not see what they expect to see, they think the dashboard is incorrect.

3. I need current progress reporting that I can trust.

How current is progress reporting? Everyone wants real-time reporting, but is it really a good idea? Real time reporting means using real time numbers that have usually not been validated. A dashboard that’s supposed to be current to yesterday’s progress report relies on the assumption that all inspectors have uploaded their reports from yesterday. It’s very important to learn exactly who will be using the reports, when they will be using them, and for what purpose. In one recent project, key decisions were made using roll-up reporting every Saturday morning. Processes and procedures were put into place ensuring the most up to date data was available to decision makers at that time each week.

4. The dashboard is wrong.

Incorrect dashboards are typically the result of unexpected data. Application developers see this as incorrect data (user error), and the field users see it as capturing what is really happening in the right of way (techies not getting it).

Roll-up data reporting is heavily reliant on structured data. Pipeline construction is an inherently dynamic environment where decisions and adjustments are made in response to real world situations.

The pressure from field users is to allow them to capture what is seen even if it doesn’t fit a pre-defined answer choice. The pressure from reporting users is for all responses and measurements to fall into pre-defined categories that can be automatically combined, counted, and summed to report overall progress. The balance point for these competing interests is different for every project.

5. I still need an inspection report.

I’m a data guy. In my mind, why would I want to move PDF documents around when I can just query the data when and how I need it? The requirement to have a physical inspection document adds complexity to the process. At first, this sounds easy: generate a PDF of the inspection form when the data is uploaded, file it into a content management system, and everyone is happy. Well, what if the validation process identifies data to be corrected? Is the data simply corrected making it inconsistent with the original document? Is the document updated after the fact as well, meaning that it’s different from when the inspector signed it? Is the document versioned with the update? What document metadata needs to be captured so it can be found later? We have found that audit trails are key, so that document changes can be tracked to the person, timing, and reason for the change.

6. I need help NOW.

Supporting a field inspector requires a different support model than typical help desk scenarios. When an inspector in the field needs help, they need help now. Opening a help desk ticket and hoping someone calls them back in the next business day is not an option. An inspection that cannot be completed, due to technical issues, can have expensive ramifications to a pipeline construction project. We’ve had to fundamentally re-think how we support our field solutions. One approach that has worked well is putting a support resource on the right of way in a highly accessible location by training an inspector in every spread to be a subject matter expert on the data collection tablet and forms. This qualified inspector supports the inspection team and provides significant cost savings to a project by increasing team productivity, improved consistency in forms usage between inspectors, and higher quality data reports and projections in roll-up dashboards.

7. I’m not a techy, I’m a construction inspector.

Building a solution that works well for the on-site inspector is very different than building technology that works well in an office. We have learned to focus on details like:

• Clearly stated form questions using terminology of the specific user and project.
• Simple and intuitive form navigation.
• Creating dashboards and forms that is already familiar to the user.
• Prevent incomplete forms from being submitted.
• Prevent invalid entries from being added.
• Use dynamic drop-down pick lists wherever possible.
• Flag suspicious data when it’s entered.
• Calculate anything for the user that can be calculated.
• Auto-fill anything that can be derived from another answer.
• Incorporating Training techniques when available.
• Eliminate the collection of duplicate information between forms.

Simply stated, if it doesn’t work for the inspector in the field, it doesn’t work. 

8. The best data QA is to keep bad data out of the system in the first place.

There are plenty of QA data checks that can be applied to pipeline inspection data. However, in most cases, QA checks take place after the data has been captured, uploaded, and distilled into reporting databases. The rule here is that the further down-stream you find a data error, the more expensive it is to correct. For example, the hours spent tracking down one bad data point that was throwing off calculations for an entire project. The main approach is to focus on keeping bad data out of the system in the first place through thoughtful form design providing the inspector with the flexibility needed, while also alerting the inspector to answers that appear erroneous (i.e. the heat number just entered is not present in a submitted MTR).

9. Were the issues found actually addressed? 

Finding a problem on a pipeline project is one thing, but making sure the problem is addressed is another.To do this, effective corrective action management for pipeline construction needs to be in place.Reporting technology needs to align with the approval workflows established by the project managers, provide visibility to open issues and how long are open, and assign thorough instructions to inspectors to revisit open issues.Most importantly, this process needs to be automated.With hundreds of inspections in flight at any given time, human intervention at each step is simply unrealistic.Human intervention should only be required at crucial decision points. Any other rule-based actions should be automated to help eliminate human error and improve efficiency.

10. Things change during a project. 

Here’s an example for mid-stream project change.A form question is designed to have four choices for weather.Half way through the project, two more choices are added, at the request of the project manager, because of disputes over what constitutes a weather delay.We are charged with creating reports detailing the impact of weather on daily progress. How do we compare forms submitted before the change with forms submitted after the change?How do dashboards account for the change without mis-leading the reader?Can a form that’s submitted before the change, but corrected after the change, use one of the new choices?

First, our pipeline experience allows us to vet questions at the beginning of a project to minimize mid-project form changes.Second, our business intelligence experience allows us to appropriately align disparate data for dash boarding when necessary.

11.“Why should I pay for form design when I already have inspection forms?

Just automate the forms we have… we know they work and it will keep the implementation cost down.” 

In just about every project we work, the conversation starts like this because the project team hasn’t been exposed to what can be done through intelligent form design.In every project that we have started with a pilot, the need for form design has been among the key learnings as the project team is exposed to the opportunities for improvement.

I have come to believe that form design is a simple math equation. On one recent project, about 100,000 inspection forms were processed.Often the small amount of time spent optimizing the form results in reducing the time spent to fill out each form. Just 5 minutes per form will save over 8,000 inspection hours on the project. Those hours can be spent focusing on collecting more and higher accuracy data with supporting pictures which will drive better project decisions which in turn increase pipeline integrity.

Given the math, I would argue that we can’t afford NOT to optimize inspection forms at the beginning of a project.

These are just a few “lessons learned” that have helped us to develop solutions, processes and procedures which are part of our best in class Mobile Inspection Forms solutions. We have found that experience matters. The best way to leverage digital data collection is to learn from the experience of others and get started on digitizing your existing processes and actively assess and tweak. Your patience and persistence will be rewarded.