Software Update 2023.42.0 is out (⚠️ UPDATE: OTA fix rolling out today 11/15/23)

[tragicmonkey049]

Thanks for this. It confirms what I was speculating about earlier. They don't do a canary deploy to any vehicles running production software. Those "developer build" Rivian's they tested on probably had the developer certificates, so the build worked.

They need to test on vehicles that have never run developer or beta builds before they do a wider deploy.

... or maybe they DID do a canary deploy - but forgot to tell us 3 percenters that we're the canaries...

Welp, time to dust off the old strategy I used in the early Android OS days: decline updates for a couple of days while checking online for the experiences of others...

Sponsored

 

[bpamiri]

It's good to see that Rivian has a fix in place within two days.

I've held off posting about this while we were in the middle of the fire drill because I didn't have anything to add but now that there is a light at the end of the tunnel, here are my constructive thoughts and concerns.

This was a failure of the push process of an OTA and as others have suggested was probably impacted by a rush to resolve an earlier issue. That said, it shouldn't have happened and I would think this would be resolved and new procedures put into place to prevent it in the future.

My concern isn't that there was a mistake or the push process was fallible, my concern is the design of the ECU/OTA process in general. I've been in the IT industry for over three decades doing firmware updates on THINGS is pretty common. The more critical the THING the more sophisticated the update process usually is. Some of the more critical pieces of equipment have a dual ROM system. In such systems, you are running on one of the ROMs and the update is applied to the secondary ROM, if the update fails the THING can still reboot off the primary ROM if the update succeeds the THING switches to run off the secondary ROM. This is so ubiquitous in the industry that you don't give it a second thought. It is just assumed this is how you are going to reprogram the hardware that is running the THING you are trying to update. The current ECU/OTA process has a major flaw and may only be fixable for future vehicles as the whole firmware update process and chips may have to be changed out.

The second concern I have is that we have heard RJ and others point out how they are going to do some cost savings measures next year that will entail consolidating the many ECUs in the vehicle into fewer ECUs. Think of an ECU as a CPU, except your vehicle doesn't just have one of them it may have a dozen of them each concerned with a particular subsystem of the vehicle. They are looking for savings of thousands of dollars per vehicle when these consolidation efforts have been completed. As I understand it, in a push to get the vehicles designed and produced the many teams involved each had their own ECU. This made for a faster and more efficient development cycle and ultimately all these separate ECUs got packaged into the vehicles. So why is this important, as far as we know the infotainment subsystem was the only ECU effected with this botched push. But what if the break subsystem, or the traction control subsystem, or the chassis stabilization subsystem was one of the ECUs effected. The current state of having multiple ECUs actually worked in our favor this time by acting to isolate the failure to a less critical subsystem. What if the consolidation effort actually serves to create some critical singles points of failure in some critical subsystem. Think of all the times we've gotten updates that had tangible effect on the driving capabilities of the vehicles. Adding Zombie mode is on one level but adding "Snow mode" with an OTA is at a completely different level of complexity and risk.

I hope that Rivian will look at this botched OTA attempt and not only learn how to improve the process of actually pushing out the OTA but also redesign the entire ECU/OTA process for the R2 platform which has a clean slate but also figure out how to retrofit the R1 platform so a similar issue could never happen again.

 

[not_a_car_guy]

It’s a comin…

I don't have it yet. I guess I made the cut into the 3% of the overall ownership, but didn't make it into the 3% of the 3% bunch, LOL.

 

[beyond]

Cannot wait to press go! Since mine's sitting at the SC anyways; not like it'll do any harm . I'll check and make sure they haven't moved it of course first...

 

[R.I.P.]

It is afternoon now in most of the country, I take it nobody has gotten the fix yet?

I had a conversation with a Rivian tech today on an unrelated matter, who is not in the software department, but he painted a pretty grim picture of what's going on behind closed doors there: " It's a good day to be a mechanical engineer..."

 

[not_a_car_guy]

I don't have it yet. I guess I made the cut into the 3% of the overall ownership, but didn't make it into the 3% of the 3% bunch, LOL.

On second thought, are they sending the same update to all owners, or to just the ones that experienced failures with .0?

 

[DuoRivians]

In other news, USB thumbdrive sales executives are disappointed that they won’t be selling them to Rivian this week

 

[LevelHeaded]

It’s a comin…

Electrafi still only shows 1 installation

Though Electrafi also shows it as 2023.42.2 instead of 2023.42.02 ???

 

[ohseedee]

On second thought, are they sending the same update to all owners, or to just the ones that experienced failures with .0?

No, you are all getting .0 first, then 3 days later will get .2, which I think is fair ?

 

[tcole]

Cannot wait to press go! Since mine's sitting at the SC anyways; not like it'll do any harm . I'll check and make sure they haven't moved it of course first...

I did this once when my R1T was sitting at the SC. I pushed go on the update at like 7AM the day I was going to pick it up. While I was driving there, I got a kind of panicked call from one of the service advisors suggesting maybe I don't come, because my car seemed bricked, they couldn't even open the doors.

I told them what I'd done, and it was fine about 10 minutes later. I picked it up on schedule, but I certainly worried them!

However, the next time it was at the SC, they put it in service mode, and I no longer had access to any of the controls via the app. So I don't think I could have done the same thing.

 

[tcole]

It is afternoon now in most of the country, I take it nobody has gotten the fix yet?

I had a conversation with a Rivian tech today on an unrelated matter, who is not in the software department, but he painted a pretty grim picture of what's going on behind closed doors there: " It's a good day to be a mechanical engineer..."

Behind closed door in software? or all over the company?

 

[jcalabek]

T just started downloading the update

 

[Spork8]

T just started downloading the update

Looked like mine went to sleep, dropped off the wifi as active so I started the HVAC from the app to wake it. About 2 minutes later, it's now downloading as well.

 

[ThumprMN]

LOL I'm on with support chat now and they are scheduling me a mobile service appointment for the OTA fix as it's needed per chat. I trust that's not true and the email that just went out is correct. Hilarious.

I think they read all your forum posts and wanted to ensure you got the gold glove treatment and help you repack those emergency blankets in your first aid kit. ?

 

[FooF]

Update is Preparing...

Now it is installing

Sponsored