Erik: Welcome to the Industrial IoT Spotlight, your number one spot for insight from industrial IoT thought leaders who are transforming businesses today with your host Eric Walenza.

Welcome back to the Industrial IoT Spotlight podcast. I'm your host, Eric Walenza, CEO of IoT ONE, the consultancy that helps companies create value from data to accelerate growth. And our guest today is Mario Pereira, Edge DevOps engineer at Vopak. Vopak has 400 years of experience in storage and transshipment of oils, chemicals, gases, biofuels, and vege oils across 70 terminals in 23 countries.

In this talk, we discuss the path for operators to develop a sophisticated edge and cloud platform from defining the system infrastructure to prioritizing use cases and functionalities. We also explore challenges related to legacy technology to the build vers buy vers partner decisions for elements of the tech stack, and finally for the imperative to evolve teams and business processes in order to make best use of new technologies.

If you find these conversations valuable, please leave us a comment and a five-star review. And if you'd like to share your company's story or recommend a speaker, please email us at team@IoTone.com. Finally, if you have an IoT research, strategy, or training initiative that you would like to discuss, you can email me directly at erik.walenza@IoTone.com. Thank you.

Mario. Thank you so much for joining us on the podcast today.

Mario: Hi. Thank you for having me. It's a great opportunity to be here. Very excited to be. Thank you. My name is Mario. I'm from Portugal and currently I'm living in the Netherlands and I'm working as a senior engineer at Vopak.

Erik: Mario, you've listened to our podcast before you know that we typically interview tech companies. And obviously, Vopak is I guess it is a technology company, but it's a different type of technology than we normally deal with. I'm very interested in as an end user your vision for developing solutions internally.

But maybe before we get into the topic of the platform that you're building, just tell me a little bit more about the work that you do personally at Vopak.

Mario: So first you spoke about Vopak to give more context to the audience. Vopak is the leading independent liquid storage company. It's the oldest Dutch company in the world. It's 470 years old, very old company. And what we do, we store vital products in liquid state; we can say oil and gas, industrial chemicals, everything that humankind needs to run. Currently, we have 73 facilities, locations, we call it terminals, and they are located in 23 countries around the world that are placed in strategical routing places. Vopak as a company is going through a massive digital transformation program, replacing multiple tools internally, there’re PACRMs.

And during this transformation, the company itself, the management detected that we don't only store vital products, we start vital data for the company, for the partners for everyone. So at Vopak, we built our own industrial edge computing platform. So we built the tool that enables and supports all the industrial Iot initiatives inside of Vopak, MBI, EML, cybersecurity too.

So the product itself, for those that never heard about edge computing, the platform ingests, process at the edge, so on the facility data from the industrial sensors. We temporarily store data locally and before sending to the cloud and then we push it to the cloud. Then in the cloud, other teams use the data for their own initiatives.

Erik: It's interesting that you've developed this solution internally, or maybe I should ask how much of the solution was developed internally? Was this a fully internal driven? Did you have a strategic partner that was providing the architecture or were you working with a system integrator? What was the team that actually developed the solution?

Mario: So we have partners that helped us building the initial MVP solution. So our product is based using AWS IoT Greengrass, to give context it was what Greengrass uses, allows the capability to run in less lambdas and other cloud services at the edge. So extends the platform of the cloud to the edge.

We start building this in 2018. So at the beginning we went looking for these type of platforms and we contacted multiple partners, multiple advisors. We did multiple proof of concepts. And then we used Amazon partner to help us build the solution and the MVP in this case. But then all the rest of the platform was built by our own team. I was lucky to hire at this time because frankly, it's very hard to hire engineers because the market is very strong. We were able to hire two or three more engineers that helped me building this platform.

So the design was made in partnership, once again, with the partners of Amazon, and then we built the solution ourselves regarding that your question.

Erik: Are you still building on top of AWS Greengrass today or have you replaced that with internally built architecture?

Mario: We are still building on top of it to give an additional perspective or point of view of the architecture of what we build. So we have IT Greengrass core, which has three numbers running. For those that work in industrial IoT segment, they already know what is OPCA. So we have an OPCA lambda client running at the edge which ingests the data from OPCA server. And then we have a lambda that transforms the data at the edge, then we have a lambda that stores locally in the cloud.

And moving on now, we are building additional features on these platforms that we built. We are building streaming capabilities where we send real time data to other departments or we are building batch operations to other departments, other type of data. For example, previously, we were only ingesting OPCA operational data. So when a sensor changes, the measurement changes, a new tech changes, now we are ingesting OPCA alarms and events too. And currently, we are already assessing other types of data on site that enables other initiatives.

Erik: Help understand the thought process behind the selection of Greengrass and OPCA. But maybe if we start with Greengrass, I guess you evaluated also other solutions, what led you to conclude that this was the best infrastructure to build on?

Mario: We already using our team inside. Majority of our services are running in the West and we have a pattern to use. So we already have knowledge in-house. And the second, there are multiple factors. But unfortunately, at the edge we did multiple proof of concepts. In our view, it's very fragmented. So we have a product that is very good at ingesting that skills very well. It's very well to organize the data, structure the data, but then it’s not very good at giving additional context at the edge of flying at our scale because we have 73 facilities. Every facility is different. Everything is different. Every sensor is different.

We did multiple proof of concepts and every platform that we went through at their own pros and cons. We did a proof of concept with AWS Greengrass. The factor costs come in. The people models will come in. The usability for other products come in. And then that leads for what we are. We are now using IoT Greengrass But at the same time, we are already evaluating site twice edge from Amazon, which is the same as us. And I’m not saying that they copied us. We worked very closely with Amazon and product management of Amazon, and the product was kind of built based on our feedback.

So we are constantly evaluating other products. We are not going to stay using AWS Greengrass forever.

Erik: want to maybe get into what the platform looks like in terms of use cases. But before we go there, since we're now talking about the development, what were the big challenges along the way? I guess you started in 2018, so it's now four years in. What were the challenges here?

Mario: So first, we possess a lot of data. Currently, from the 73 locations we are live in production in 18 and we are producing at the edge, we are processing 65 million events per day. In the 18 locations, we have an average of 25,000 OPCA tags. So, 20,000, 30,000 tags, depends on the size of the terminal, we have a terminal that has 65,000 OPCA tags. We had a lot of challenges, not only technology, but majority was on processes, mind shifts. The one really big one is giving context to the data.

So the data that comes from OPCA from software engineering and BI and data science, it doesn't give a lot of information. And in our case, for example, a tank could be TK-004.CV and that doesn't provide which type of tank it is, which type of sensor it is, which type of measurement it is. We need to give that context, that information. And maintaining these business rules at the edge, at our scale, it's a big challenge; because every site is different, it can be very challenge.

The other problem is the challenge, it's the people itself. And you work with people around the world, multiple cultures with different expectations, different backgrounds. So I'm a software engineer by nature and by training, explaining and operator in the field how some tools work can be a little bit scary. In these times of cybersecurity attacks that we live on and even expectations, you say, okay, I'm going to ingest all these data and we're probably going to run machine learning and the maintenance or asset optimization and we'll help the operator. And they think, okay, me ingesting after 2 hours of ingestion data that I can already do it. And I know I need to say no, we need like five years of data in order to help you.

Another one is once again with the security I was telling you at the edge, every day you see like the colonial pipeline hack, last year, the offline security is a big challenge at our scale. The operational part of it, like the monitoring, logging, tracing, monitoring of the data quality, the monitoring of the processing time, the throughput: so we are sending a lot of data right in places where the Internet is not that good or the connection to outside world is not good.

Erik: There's many different challenges and at a very high level, you can break the challenges into IT challenges and OT challenges. So you have cybersecurity, you have high data volumes, and then you also have processes and the people challenges of getting the end users on board with the new applications. First of all, is this driven by the CIO office or the head of engineering or the CIO office? And then what does the actual project team look like? Who else is involved in the project team to address all these challenges?

Mario: Regarding who took the initiative, this comes from the top. We really want to become data driven, is one of our goals of the company itself, becoming data driven. We want to take smarter decisions. One of our big KPIs is safety and this data allows to make maintenance operations in more safe way. We can have drones. We can have all these technology that allows to reduce accidents in the terminals.

We are like a use case inside of Vopak. Our team is very small. From projects, so it's me or another DevOps background engineer. We have an industrial engineer, so a person that really understand [inaudible 14:15], all these protocols machine, OPCA, OPDA, these industrial protocols, these machine-to-machine communication protocols. I have done project manager team leader which comes with a technical background too. And we have an architect that helps us, solutions architect when the business, it’s like a request I want to do this, I want to build this dashboard, it translates the business needs to a technical needs. That's our team.

It's a very small team because we are very DevOps-approach, very agile, we develop, we tests, we deploy and everything is automated with test coverage of 99.5%. We have integration tests, unit test, regression test, load test, security test. It's a use case and maintaining this, we have on call we have 24/7. Our team is very senior and very good guys. So once again, I was lucky to be able to hire them. So it's a very small team.

Erik: But I think it's a luxury to be working on a small but very capable team.

Mario: To be honest, I've already been in other companies that process the same amount of data for the same amount of data and the team was ten times bigger.

Erik: So it's interesting that you really built this as an edge platform because I think that's where we're moving. But a lot of companies are still figuring out their edge strategy. There's the question of what's different between the edge on-premise and kind of traditional on-premise. And so talk to me about how do you balance processing, storage, etc., between the edge and the cloud? How do you decide where work needs to get done?

Mario: At the edge, we ingest once again the data, we transform the data, we give meaning to the data, we filter the data. So we use, for example, the dead band algorithm to filter. We use one that is very used in the operational domain, which is the swinging door compression algorithm  in order to compress the data. Locally, we store the whole data, so the data that comes from OPCA servers and the data that is normalized or transform data. Then we only send to the cloud the band data and data that passed through the swinging door algorithm goes from all terminal.

So as location, we have hard data and the data not filtered. And in the cloud, we have the data that is filtered but is transformed from all locations. In the cloud, we store it in the cluster, which then has an API that exposes data to ERP, CRMs. And we have the capability ones. I was always telling to stream the data, we streamed data through service from Amazon that is kinesis to an S3 bucket, which then the BI team uses to create dashboards.

For example, the OPCA alarms in events, they then do analytics over the alarms and events like which asset leaking more, which asset is vibrating more or giving problems. And then we have a batch [inaudible 17:59], another service from Amazon. You can do the same for the people that are listening this. They can do the same with any other cloud provider nowadays. They can use it with Azure. They can use it with Google platform. And then we have a batch service that vets the data and sends it to the data science team, where then the data science team uses these data to train their machine learning models.

Erik: Coming back to this organizational question, so you then have three teams. You have your team which owns the platform, you have the BI team, which is building dashboards and front end applications for different users, and then you have the data science team that's building the algorithms. How do these teams work together and make sure that they're coordinating well to bring the right solutions to the organization?

Mario: Our data department, data analytics is doing awesome work even before jumping to the teams. We need the use cases. And our data team is doing a huge, amazing work on setting up training courses and foundations to the terminal managers, senior managers, where they teach what they can do with the data. Because you can have all the data and all the teams can be working very well and our teams work very well because Vopak is big. But inside we look like a family and we are very agile inside. So the communication flows very well and we communicate very well. We follow the Agile practices. We have the Scrums of scrums. So it's a 407 years old company that is transforming from inside. It's changing itself. Sometimes it looks like a tech company. We have very good communication between departments.

But even before reaching to the teams, it's the use cases. And our data science team is creating these trainings and courses for everyone in Vopak to have some data information literacy where they can learn what they can do with the data.

Erik: And I've got to ask you a follow up question, because I was just helping a client kind of think through their industry 4.0 approach and we were talking about this question of how do you train the organization that doesn't need to be able to do data science, they don't need to be able to build dashboards or anything, but they need to understand how it can provide value to them?

And the question there was, how broad do we go? Do we really try to train everybody? Do we go down to the people on the shop floor. because they're also users right there? Or do we identify the 5% or 10% of people that are going to be most active users? So what was your approach? Was it rolling this out company-wide or identifying specific individuals?

Mario: What I can tell you right now is currently they are doing these four levels. They do when they release a new application or a new functionality, they give trainings to people on site. They train the people how to use, they explain what they can do with information. In our case of the Edge platform, what they can do with the data before going to one terminal, before going to one on site, we go there as a team.

We go to the terminal, we talk with the people on the floor, we talk with managers, with engineers on the floor, and we try to explain what the platform does, what they do, what they can do. But this is us, our team. As an organization, they're currently training mid-level, top levels, but then like it's organic, the other people will then train and teach and explain how those things work to the others. I can give you an example.

The digital twin, right, the modeling, the 3D and we have very good contact and we explain what we can do with the onsite team. They enjoy very much to see the physical representation in these types of people working from home and it's very nice that they can see what is real there when they are not there.

The capabilities of the analytics over alarms and events, it comes from the field. So the field itself asks for things We have a department that is called Innovation Department, and these Innovation Department is constantly talking with the field and field ask things, oh, can we do this with the data? So the people even in the field they are already aware what they can do with the data a little bit. We live in the world that software is eating everything. Software is everywhere. So the people are really understanding that they need data. Even giving us the training, people already know.

Erik: Probably 30 years ago it was the industrial or the B2B side that was pushing the technology forward maybe 40 years ago, and now it's consumer. So everybody kind of knows that you can use data for a lot of different ways to make decisions because you use that all the time just as a consumer. And then it becomes more the question of, okay, how can we leverage these internet technologies for my job right now?

Let's then do a quick walk through the use cases and I'm sure you have a long list, so maybe you can help us put some structure into it. Like how do you classify different use cases? Which ones are the really transformative ones? Which are the quick wins or the maybe the efficiency gains, but not necessarily transforming how you work?

Mario: The quick wins, the easy ones to win is the ones that have more customer impact or facing. For example, the CRMs or ERPs data, currently it's public customers can see the level of their tanks or what is the status of their product inside of the tanks. This data come from us. So this is a thing that is very important and non-priority is very easy to do and quick to do and it gives a lot of return. That's the first one, probably the most important, which has more impact on the customer.

Of course, internally we have the second one, probably more important is besides safety, it's the energy management or pollution levels reduction where we do peak shaving, for example, where we control how much energy because sometimes we have pumps or valves that are heating and they don't need to be heating. So we can do a management through our data. Of course, then we have the predictive maintenance because we can detect through vibration in the spectrum when something may be is get broken, we have a limit of vibration orders, equipment or even 3D printing equipment before it gets broken. Currently, Vopak is investing a lot in which public people can know.

We are acquiring companies that provide industrial sensors, certified which are able to put not smart like a pump or a valve which doesn't have sensors and we can just plug in or add a sensor. So it's another priority. So I'm trying to go by a list of priorities of impact and quickness because it's hard to retrofit all assets with new sensors. The predictive maintenance takes more time because you need a lot of data. But the energy management you don’t need so much data like one year of data or two years of data is enough. But the most important currently is the CRM in ERP because the customer can effectively see his data.

Of course, then we have other proof of concepts that we are doing, like autonomous robots like Boston Dynamics lookalike drones that do asset inspection. And recently, we invested in one of the companies that does that in Singapore. We collect that data too. And currently, we are doing proof of concepts or proof of values with 3D modeling, so the digital twin, which this is very nice to avoid problems or accidents in the terminal, for example, see how what is a temperature life and what is liquid state on my terminal in the jetty, in the tube, small things like that we are doing.

Erik: So, then my follow up here, I was just doing a bit of a benchmarking in the battery market, in one of the companies we looked at was a company called CATL, which is the market leader in EV battery manufacturing. And they are a little bit similar to you in that they've developed a lot of internal technology. They're quite sophisticated in terms of how they operate now and they've just put a $500 million into a software business, basically with the goal of taking the internal tools that they've built, developing those into a new company, and then selling those upstream and downstream to their suppliers and their customers and basically becoming a tech company.

Are you also looking at starting to monetize technology that you're developing? Or what's your thought on this?

Mario: That I don't know. It's about my background. That is something to my management team to reply if it will be a good thing. Market is so fragmented and then it's our software for our use case if it's possible to give this as a net platform to another company because it's just an OPCA. But we have a lot of business rules like on the sensors that probably does not apply to all companies, maybe to this type of company, storage liquid company, maybe. But once again, every sensor is different, every terminal is different.

So once again, the challenge was at the early stage and it still is. We are working very hard on it, is the stabilization of the data giving meaning to the data and maintaining these meaning on the data.

Erik: I see a lot of Chinese companies doing this like saying, okay, we're an oil and gas company like Sinopec or somebody. And we need to develop our own platform because and it might be a little bit policy-driven because I don't see so much American and European companies doing this, even though they have a lot of domain expertise and sophisticated solutions. So, it's just interesting to see how this is evolved a bit differently, maybe a bit policy oriented in in China in terms of these companies trying to become more tech oriented in their revenue as well.

Mario: The problem is the digital transformation process, I already spoke multiple times with other companies because our technology and they are curious, like a use case? When you do the digital transformation, this is a marathon. You're not going to do this in 2 years or three years or it's like a 10 years or 15 years plan. You need to hire a lot. You need to change all your structure. You need to change the processes. You need to change the structure of your company, the mindset.

If you now focus on technology, not on your business, Americans and European companies, if you are a tech company, your business is tech, right, you are selling, but it's tech. So you need to reshift your mentality and still keep your business working. Or you think okay, tech can enable my business, can give me new models of revenue or help me save money. But in order to do this, you need to invest first, not invest only on technology, but on people. You can say you buy software, right, but you still need to hire people that know the software, they know what to use, they know how to implement, they know how to maintain it and then you need to convince people that this technology is going to solve X amount problem.

I think digital transformation is not about tech, it's about changing the mentality, the mindset to achieve great things in order to explain that technology and even a non-technology company can be a technology company. You have a business to maintain, you are a business to run and now you need to worry about technology.

Culturally speaking, once again working with different people, with different cultural, I get that. Asian country is not China, but the Asian countries are more currently direct to technology, maybe because of resources. I was in Singapore, I see way more technology. I’ve already been in the United States multiple times. In San Francisco, you live and breathe technology. So, it's all about resources and mentality and mindset.

Erik: If you were talking to somebody that's been put in a project management or a leadership role, but maybe starting where you were mayor three or four years ago, what would be the two or three things that you would advise them to do to improve their chance of having success and then minimize the cost on the way?

Mario: Start small, very small. In our case, we started with a small prototype, a small sensor, it start small and make noise. So we start small, we build with only one sensor. We built the dashboard, it was quite funny because it was a sensor that it just a red bottom and people will pass by my desk and click in the bottom very hard and they will see stuff blinking in my screen. And we did an Alexa integration where people will ask, what is the status of the color of the button? It was Allen Bradley PLC. But we had the training kit from Allen Bradley. We did the proof of concept with that with Bradley PLC. We showed cases very fast. It took me two days with the help, of course, the partners, but two days, three days to build the MVP. Of course, more time it takes then you build a production ready platform.

Two days, three days, we build the MVP, we build the dashboards, we test it and say people, do you like this, do you don't like this? Automatically went to the field and talking with people, do you like this, do you don't like this? What do you think if you do this? Of course, you hear a lot of no's. I don't want that. I don't need that.

But I'm going to use for the question, if you ask customers if you want cars or faster horses, they will probably ask faster horses. It starts small, more lean. Start small, prototype fast, measure success, measure the type of feedback and iterate very quick. If it doesn't work, kill the project fast, work in something new, something different, something that they really want. For us, it was [inaudible 34:35] factor. The Alexa integration was quite funny in the dashboards. When I said it took two days, no, two days for this, I knew you can do the production quick and it is true it was quick to do.

Erik: And you basically followed that same methodology or mentality through, so you didn't say, okay, we're going to map out the architecture for this plan and we're going to devote 24 months to building it and then launch, but you were like step by step just incrementally building functionality or how did you go from that first MVP to real value in the operations?

Mario: Step by step, starting small and not the big bang change and here's the product done, here's the key. No, because that then people will start waiting, they become agitated, let's use this word. You need to listen to people and listen to what they want. And we started small, giving small features measuring that.

If you are a factory, with a lot of factories you need to building with automation and monitoring first because the retrofit of it is very hard then to do when you are already living like ten places with 10,000 sensors. We use that methodology to start small and continue to do that in the right direction continuously.

Erik: I think that's a very interesting case here. Anything else, Mario, that you'd like to share with the audience?

Mario: Vopak is hiring everywhere in the world, a lot of positions from OT to IT to non-OT to non-IT, visit our website. We have the Dutch version and English version. And if you have any question regarding what we build, feel free to reach me. The only social network that I use is LinkedIn, yeah, feel free to reach me.

Erik: It sounds like a great company and we'll put your link to it in the show notes. Thanks, Mario.

Mario: Thank you.

Erik: Thanks for tuning into another edition of the IoT spotlight podcast. If you find these conversations valuable, please leave us a comment and a five-star review. And if you'd like to share your company's story or recommend a speaker, please email us at team@IoTone.com. Finally, if you have an IoT research, strategy or training initiative that you would like to discuss, you can email me directly at erik.walenza@IoTone.com. Thank you.