What I've Learned in a Month of Enterprise Data Engineering

Since starting with Texas Instruments in October, I have been focused on developing data pipelines from our factories in Texas, Japan, and China. Each factory has many thousands of sensors recording sounds and flow rates of chemicals. Wrangling domain knowledge of those inputs has been challenging but once understood they provide incredible leverage to intelligent organizations.

Naming Things

Phil Karlton's quote, "“There are only two hard things in Computer Science: cache invalidation and naming things” rings true as ever. When I first began looking at these datasets, I saw the same column named 4 different ways in 6 different systems. The only way to identify the "ideal" name for the data is to have a close relationship with how the data is generated and what it looks like.

Building standard data models is predicated upon the standardization of data formatting. Taking an organization that's disparate into a streamlined, unified model has friction. Each team operating those sensors has already developed their own systems around those names; asking them to change is asking them to slow down for an abstract greater good. As we learned and showed factories the ideal format for their sensor outputs, our data models executed became easier to understand.

Because the end goal is supporting the company output in factories, our team has built the translation layer into our pipelines. The factories can focus on their roles while we use code to fill in the gaps of formatting. Our 5 people can save the work of 100 by standardizing data as we productize it.

Big Data

99.9% of 1,000,000 still leaves 1,000 missed items. Streaming live data from thousands of sensors leads to all kinds of problems for consumers. We often see a completed run arrive several hours after it's end. This confuses the upstream pipelines because the completion count for yesterday's process can show different numbers checking in the morning or in the evening.

To mitigate this change, we have initial and secondary levels of event listeners that can retrigger appropriate jobs as those late-comers arrive. If an event arrives 6 hours after its peers, the event listeners can retrigger aggregation pipelines to absorb this late event. This keeps the data quality up, while giving the source systems the margin they need to catchup on event emissions over time.

People

Tremendous work by the initial configuration teams has built raw data. Now that it is available, although tricky, a smaller 5-7 person team can ingest and clean up that data for an entire company to leverage. Smart people make other people's jobs easier, even when it is difficult to quantify just how impactful their design choices are.

When individuals make choices that are useful in the moment but also consider how other customers could use their work, integration is smoother. I worked with a dataset that had been specifically designed to change, akin to the object-oriented paradigm in software engineering. Although the dataset was set up decades ago, that design decision is still paying dividends in time today. People make choices that last well beyond their retirement.

Conclusion

Naming things is hard; so, understanding the data well enough to recognize it across various titles is essential. As the comprehension of source data increases, so can its utility. This lends itself well to smart people making good choices on quality data.