Conversations with Paul Piong

By Chia Chu You and Joseph Khaw

Paul Piong has been serving as Staff Firmware Engineer at Tesla for the past seven years, and currently works on the firmware for the Battery Management Systems of the Model 3 and Model Y vehicles. Before that, he worked as a Software Engineer at Garmin International, specialising in low-level code and device drivers for several GPS-enabled fitness products. His career focus so far has been on software development with various hardware platforms, computer architecture and real-time embedded computing systems. Paul graduated with a Bachelor of Science (Computer Engineering) from the University of Michigan, and a Master of Science (Computer Engineering) from the University of Kansas.

Can you introduce yourself and the work you do?

I have been an embedded systems programmer for most of my career. During National Service (NS), I served in the Signals Corp where they taught us radio theory and systems networking; which motivated me to pursue engineering. After NS, I got accepted into the University of Michigan and studied Computer Engineering. Nearing graduation, I worked a part-time job building laser tag equipment and adding a radio to it, but it was a very small company and I wanted to learn more, which led me to Garmin and then to Tesla.

How is it like working at Tesla? What does your typical workday as a Staff Firmware Engineer look like?

I am currently on the battery management software team, which has around seven or eight staff and a few interns. Typically, my work is divided into three phases: day-to-day, short-term, and long-term.

Day-to-day matters include high-priority matters that we have to look at. I work on low-level code, so, for instance, if a part fails for a bigger sub-component that I am working on, I would have to work with the electrical engineers (EEs) to figure out what is going on, if it can be mitigated, and then issue a software fix. Unlike other car manufacturers, Tesla can push software fixes out to the whole fleet of cars, which makes for a very unique experience. The first time I made a change that got pushed out to the entire fleet I was very stressed as there were 25,000 cars then!

My short- or medium-term work involves improving how the battery management system works overall. For example, I am working on better current sensing accuracy, and as that improves, it feeds into other algorithms that track battery wear, energy usage, and related metrics, allowing us to make better optimisations to the performance of our vehicles.

The longer-term projects allow us to consolidate and apply our learning. For instance, for the upcoming semi-truck, I am the first one on the team to start working on it because I have to bring up the new hardware and make sure the low-level driver code works. As such, I have to work with the EEs and troubleshoot their designs to make sure they work correctly.

The nice thing about working at Tesla is that it is very cross-functional. When dealing with matters at the systems level, we could just walk across the aisle to talk to the relevant people. We still have an open office layout, but that has changed as everybody has to work from home.

Prior to joining Tesla, you were a Software Engineer at Garmin for over 5 years. What encouraged you to make the switch to Tesla?

After five years at Garmin, I felt I had reached a point where I needed to keep growing, but there were not many opportunities to do so. The downside of Garmin was that it was situated in Kansas and there were not many other tech companies there that you could go to if you did not like your job or wanted to try something else. Additionally, the weather was not great as it was quite cold. On the other hand, Silicon Valley is where all the tech stuff is. So those were the push and pull factors, where I got pulled towards Silicon Valley for opportunities and pushed away from Kansas because of the weather.

How has COVID-19 impacted your work given that remote work may be harder for Tesla due to the need to work in plants?

For now, our offices have reopened and people have started going back. However, it is not mandatory and, fortunately for me, I brought my hardware back home and it can fit on my desk. I am working on low voltage matters now but when I eventually transition to high voltage components, I will likely have to go in to work as well.

However, a positive aspect of the situation is that we knew, going into the semi-truck project, that we would not have the luxury of hopping into the semi-truck to test it due to a lack of commercial driving licenses. This has led to a push for more simulation testing and investments in the infrastructure for simulation work, which paid off in the current situation because we can do a lot of development on simulation instead of directly on the vehicle. Another positive is the situation has slowed things down a little, giving us more lead time on this project.

Having studied and worked overseas for a significant portion of your life, how would you describe the differences between working in Singapore and the US?

My three-month fellowship with GovTech was my only point of reference for working in Singapore. I did not know what to expect and was very pleasantly surprised as everyone I worked with was really smart and motivated. There was a good mix of very experienced people and fresh graduates with a lot of enthusiasm and different perspectives, which I appreciated.

The pace of work was definitely a lot slower. If you look at the Singapore government as well, their time frames may be in years, which was a jump from Tesla where I could get changes done in two days. I did not quite like that and I also felt that there was quite a rigid hierarchy between people in Singapore. At Tesla, I am friendly with my boss and generally everyone is quite approachable, including the directors. Elon used to walk around the office often and once there was a Taiwanese intern who intercepted him and asked, “Hey, why aren’t we selling in Taiwan?”, and Elon asked, “How many people live in Taiwan? How many cars do they have?”, and the intern had all the numbers and Elon replied, “Okay, I’ll think about it.” A few years later, we were in Taiwan. So that is the kind of culture at Tesla.

People are definitely more punctual in Singapore as they come and leave on time. At Tesla, people show up a little late or come in really early, but when it comes time to actually do something, we come together quickly. It may take slightly longer and sometimes people will stay late, but that is how it is, and none of my bosses have ever told me to stay late. If I do stay late, it is because I really want to get things done. We are a very mission-driven team as we are all in this together.

You have worked on refining and applying GPS technologies for a long time. What is your vision for the future of such technologies and the future of smart mobility?

GPS itself is quite a mature technology and the interesting thing is that while the US has been the dominant provider of GPS for a long time, China quickly caught up. They have launched the Beidou satellite constellation and are definitely going to put more satellites into space. It will be interesting to see how technological dominance develops globally, which is also seen in the race to 5G.

I think the next frontier of Smart Mobility is indoor mapping, which is technology that digitally positions people and objects inside offices, venues, and other buildings. The indoor mapping idea has been tossed around for quite a while, but I think that is where most companies are going to focus their resources moving ahead.

How long do you anticipate that it would take for autonomous vehicles to become mainstream globally or perhaps in San Francisco or in Singapore?

While I am not sure and cannot speak for the company, I think we will see some degree of autonomous driving by the end of 2020. However, full autonomy is going to take a while and I am not sure if infrastructure support will be needed. Many US companies have been targeting the “non-infrastructure” approach as it is unrealistic to expect such support in most countries.

That said, if infrastructure is what autonomous driving really needs to be successful, I feel that Singapore can actually support it. However, the downside is how do you attract companies to build a solution that will only work in Singapore? As the private-public partnership model does not work everywhere, it is a potential problem for the scaling of autonomous driving.

One key aspect of Singapore’s Smart Nation road map is the outdoor 3D mapping of Singapore’s landscape. How far do you think we are from the mass-market application of such technology?

It depends on what application you are thinking of. An idea I had when working for GovTech was that the government could do the heavy lifting of mapping and provide the data for companies to tap into for free as an incentive to develop applications. One application I thought would have been cool was drone delivery, as the high fidelity 3D mapping resource could easily plug on drone deliveries to locate someone precisely. However, the downside is the urban jungle shadowing effect, where dense concrete buildings cause GPS signals to constantly bounce around and results in GPS receivers getting less accurate. There are some potential solutions to this problem, though they usually require additional sensors or increased cost.

Regarding indoor mapping, I think sensor technology is still not sufficiently advanced yet. Getting the fidelity and coverage needed is a challenge for companies trying to build and sell a device at the lowest cost. Why put a sensor that not many people actually use to give you the additional mapping capability inside buildings? But that is what Google Street View tried to do with the people carrying the backpacks to gather interior shop views. If you want that to have more coverage, you need everybody to carry it and the best way to achieve that is to piggyback on cell phones. Yet, even then I do not think you will get the coverage that you expect.

So then is the key to have a system in which you can gather data from the environment in the first place?

Yes and I think the easiest way to incentivize people to do that mapping for you will be through augmented reality (AR). Most AR systems require accurate distance and location mapping, and with AR, you have people walking around trying to find things as they use it, which is a natural data gathering source. Moreover, companies already have the positional data needed for AR anyway. So that is my guess as to how Apple and Google are going to try to angle things ahead.

Do you believe that it is technologically feasible for a private tech firm in Singapore to drive that kind of project, given the amount of resources required, and gain ownership of that level of technology?

I feel a compelling application will be required to drive a lot of whether it is feasible. What Apple and Google are chasing is a broad-based application because they want the platform for indoor mapping. However, a simple and targeted use case would make more sense for Singapore. But maybe you can only get there with an AR platform, and I feel that developing a platform by yourself will be challenging.

I think once a bigger and widely adopted platform arises, there will be plenty of opportunities, especially for people in Asia, to leverage that in ways that the rest of the globe will try to adopt. One thing I have always wondered about Singapore is that while it has many good ideas, many startups in Singapore just mirror ideas from the US. I have always felt that there should be at least one good idea from the uniquely Singaporean or Southeast Asian culture that can become the next big thing. Not yet, although I think we will get there eventually.

Do you think that electric vehicles (EVs) are eventually going to replace all forms of vehicles in Singapore?

I actually met PM Lee once at a small gathering when he was in San Francisco. He responded to my question by stating that, firstly, the cars were too expensive for the average Singaporean (as we only had the Model S at that time). Secondly, having more cars on the road is the actual problem. Regardless of whether it is an EV or fuel-combustion vehicle, Singapore’s government generally dislikes private car ownership, which I can understand as cars are quite an inefficient mode of transportation and Singapore has a good public transport system.

In that case, would the move in Singapore be more tailored towards electrifying public transportation?

I think so. Alternatively, autonomous ride-sharing could be another palatable solution that would reduce road congestion. However, the sad truth is that Singapore burns quite a high grade of gas and petroleum, which is more expensive but burns cleaner, hence the amount of air pollution is not that obvious. In fact, compared to similar Asian cities, Singapore’s air pollution quality index is quite good. As such, I think the environmental factor might not be a large concern to consumers in Singapore unfortunately.

What do you think is the main obstacle for the average consumer when adopting electric vehicles?

I think charging speeds have to be faster for electric vehicles to become relevant.

In the US, Tesla has its own network of supercharging stations that provide high power, fast recharging, but that is not feasible for Singapore due to space constraints. It will make more sense to offer a portable alternative based on demand instead of permanent fast-charging stations altogether. During holidays for road trips here in the US, the supercharger locations get quite congested. So we have mobile versions of superchargers that are on the back of trucks, which are driven up to those stations for people to charge their cars from. They can also be tied into the grid, so that is a possibility.

Another potential solution is the pack swap. We have not done it widely because our design is not originally optimized for pack swaps, but other companies have done it successfully. For instance, Nio, in China, has done their first half-million pack swaps. It was nice to see somebody else take the idea and succeed with it, so that is definitely another option to cut down charging times here.

For now, I think the supercharger network works quite well for Tesla. In fact, we have just increased the power to around 250 kilowatts, which allows you to charge fully from 10% capacity in about half an hour. However, a challenge for charging in Singapore is that most Singaporeans do not live in their own landed property. If they need to park their car elsewhere, they may not have dedicated charging available to them. If they do, then how do you equitably charge them a price for charging up their car? If we dedicate spots in charging stations, people will complain about how EVs get preferential parking spots too. Thus, unfortunately, not all solutions can be solved with technology and sometimes some social engineering approaches are required as well.

Has the coronavirus pandemic slowed the development and reaching of the million-mile battery goal?

It mostly has not and we are ramping up with new projects. One of the good things at Tesla is that we do our own battery chemistry as well. So we are slightly ahead of the game compared to competitors that just integrate things that Original Equipment Manufacturers (OEMs) like LG Chem produces. We are also working on the China side to source new suppliers.

Interestingly, we are at a point where, to get to the million-mile drivetrain mark, we can either increase our battery capacity by adding more cells in or newer chemistries, pushing the limits on the thermals; or instead, make the rest of the car more efficient. At Tesla, we are evenly split on that and may actually lean towards finding efficiencies in the powertrain instead of solely improving the battery, which gives a better return on investment and cost

In fact, in the Model Y’s case, we introduced the heat pump. This is more complicated but you do not waste as much energy trying to heat up the battery to a specific temperature, as you can actually channel it around from other subsystems in the vehicle (and vice versa, like using battery heat to warm the cabin for example). That alone gives quite a significant efficiency advantage. So it is things like this, in combination with new battery technology, that will get us to the million-mile goal.

What is your evaluation of the Lidar approach that most other companies (Waymo, GM) are taking to tackle the autonomous driving problem? How would you compare it with the multi-camera approach taken by Tesla?

I cannot really speak for the companies that pick Lidar for their sensor approach, so this is just my personal opinion. I would say Lidar is similar to taking a very high-resolution photo with a DSLR camera. Essentially, it is very high-definition data. Once you have that high-definition data, you need to process it in real-time and processing all that data is very power-intensive.

Firstly, a high-energy laser is needed for Lidar imaging and, secondly, the high fidelity data retrieved must be interpreted. With such problems, you can throw more power at the solution to get the required processing speed, but every bit of power used for such an application takes away from the vehicle itself.

Tesla’s autopilot and Sentry Mode is already quite power-hungry. Thus, I think the camera approach makes more sense for us. It is the best compromise so far because cameras are comparatively less power-intensive and we are well versed with image processing. The extra depth provided by Lidar may not be particularly useful at that point. Instead, we lean a lot on the heavy lifting that the neural net will do the correlation and interpretation correctly.

As Elon has said, humans drive without Lidar or radar. We just use our eyes and we seem to do okay, so the algorithm just needs to be as good as a human driver, and then we can focus on getting better. But we still have quite a ways to go even if vision is all we really need.

What are you excited to work on in the foreseeable future?

Coming up next will be the semi-truck. If we can develop a compelling enough semi-truck for cargo and freight, there will be quite a significant impact as transportation is a large polluter in the US. Additionally, we are able to aggregate the lessons learnt from Model S, Model 3, and Model Y, and apply it to the semi-truck as we are trying to reuse a lot of things. The exciting part for me is examining if I had made the right design decisions such that I can reuse parts wholesale. If not, how can I change it so that we can reuse things that we have already built? That is baked into our culture because it is nice that we have been able to spin off a lot of projects from what we have built before, such as the superchargers, stationary energy storage products, and even ventilators, all which have roots in vehicle components.

What do you think will be the displacement fallout if Tesla truly automates away truckers’ jobs?

Unfortunately, if we go fully autonomous many of those people will lose their jobs, as truck driving is a top occupation in many states in the US. However, it is impossible to anticipate the full extent of what automation will displace. One way is to look at the pandemic now. Jobs that you had not thought were super stable, turned out to actually be, such as working at the grocery. One big competitive advantage is, if Tesla can get the fleet out, we will gather more data as well that can at least make roads safer for everybody, if not more autonomous.

What advice would you give to young potential engineers who would like to work in Tesla? What skills would you encourage them to pick up?

Get your engineering fundamentals in. Do not think that you will never use this in real life. Also, learn how to code efficiently. That is a big key, especially for embedded systems, but is less significant for bigger platforms and applications. Most importantly, you have to learn how to learn because there will always be new things coming up. As such, you have to figure out what you do not know and then where you think you need to focus your skills and time on.

Especially at Tesla, while knowledge is important, you also need to be passionate and have good communication skills because we have to work with people from different teams regularly. All my teammates are really motivated and smart people so you have to be able to keep up with them. You are also working with mechanical engineers who have their own lexicon and grammar. So “learning to learn” will ensure that you are able to understand how to talk to them or at the very least work better with people from other fields. Furthermore, you have to be able to communicate your ideas because that is where you gain the ability to collaborate well and be flexible with matters that crop up.

Do you think people who self-study coding are doing themselves a disservice?

Not really, I think there is a time and place for that kind of coding. I read an article about a lady who started a startup that is web-based, but she did not know how to code at all. She did it all without writing a single line of code, because there are many web services out there that can take inputs and she took the big building blocks and pieced together something that worked. The majority of the internet runs on AWS, but you do not need to know how AWS works to be able to use it for your business. It is more of adapting your mindset to appreciate the capabilities of different technologies and seeing what can be built out of them. That is where the true value actually comes from.

I think it is perfectly fine for people to learn the basics as it opens up more opportunities for them to decide what they want to do. I would say to at least try it out, as it is a relatively new skill that is useful in life and has low barriers to entry. At least, if it does not work out for you, you will know that it was not what you wanted to do.

That is also what is nice about programming languages like Python. All the heavy lifting is already done for you so you can spend your effort on building something new. For example, if you’re interested in Machine Learning using Python, the machine learning libraries are quite widely distributed, well maintained and well documented, and it really lowers the “activation energy” required to get started.

Is there anything else you would like to share with us or potential readers or even to potential job seekers who have just entered the job markets?

It is definitely a really tough time to be entering the job market now. But with that adversity comes an opportunity to explore career paths that you might not have thought of before. Personally, I was fortunate that my parents trusted my decision to enter engineering despite it not being seen as a prestigious profession in Singapore. I took the chance and it paid off. As difficult as it is right now, I think it is a good opportunity to try something new and explore, as that will let you see where other opportunities lie.