Software Update 2023.42.0 is out (⚠️ UPDATE: OTA fix rolling out today 11/15/23)

[SANZC02]

I'm a little surprised new software isn't rolled out to a known set of vehicles (employee cars?) about a week before the same software is sent to paying customers.

They do rollout to a beta list. The issue was not in the update they messed up the certificates when building the package for general release.

Sponsored

 

[ohseedee]

Got the email. Update 2023.42.02 should push later today. I will be downloading it as soon as I see the notification as I can’t be without my music any longer. lol

let us know how it goes, I'll wait until I hear back from you ?

 

[Dark-Fx]

Do Wassym's comments make it sound like they are using a backdoor to get the update pushed or is it just me?

 

[godfodder0901]

I never said in my case with the ID.4 it was an OTA, just said it was an update. From a perspective of an update failing, it no makes difference....failed physical vs. failed OTA update sucks and there should be safeguards in place so the vehicle still remains functional. In the case of Rivian, it looks like there was a fail safe in which you could at least still drive (was this by design or just a coincidence?). For my ID.4 it was completely bricked and completely inoperable for about 6 weeks.

My apologies; I misread your response. I inaccurately inferred that you referred to the update as an OTA.

 

[not_a_car_guy]

https://electrek.co/2023/11/15/rivian-fixes-infotainment-software-bug-via-ota-around-3-affected/

This interview was insightful, thanks for the link @cjones1130.

There is no way you can argue this wasn't a big mistake on Rivian's part, and that they will lose some goodwill among customers, and possibly lose some customers altogether over this. That's just life, mistakes have consequences.

However, after burning the midnight oil, I think it was great to see that the leader was willing to be interviewed about what happened instead of just a one way communication or relegating it to a "professional smooth talker". I didn't feel like there was any hedging or CYA in his answers. The only other thing I would have liked to see is an explanation of how they fixed their methodology for the future, but it's understandable that they might take a few days to figure that out.

Other than the mistake happening in the first place, there are lots of other signs of a healthy organization IMO.

 

[hevak]

Anybody else stalking their local network to see if the update got pushed out yet? ?
Not seeing anything significant yet.

How do you know which clients are the Rivians? The reverse mac address lookup doesn't seem to actually work for me.

 

[tcole]

I'm thinking they didn't want it to get overblown and picked up in the mainstream media in the off chance that they could rush a fix out before the severity of everything is revealed. An official tweet on their Twitter and other social media that's mainly used for positive marketing would be pretty embarrassing. It's a serious issue no doubt, but luckily for them (and owners) cars are still running and seems the big inflation news yesterday really took the heat off of this fiasco. If they can release a fix soon, hopefully the damage can be considered mitigated and crisis averted.

You could very well be right about Rivan not wanting to publicize this too widely because it looks bad.

I worked at a fairly well known Payments / Point of Sale company for many years and in the early days the executive team did not want to have a public status page because they were worried it would look bad. In engineering we strongly disagreed with this stance, and thought we were damaging customer trust with the lack of transparency. They eventually relented and we started being quite transparent about outages and even publishing public postmortems to explain what happened.

Now it's absolutely standard practice to have a public status page with frequent and public status updates for any important online service. And the best companies also publish a postmortem detailing what happened and what they plan to change to prevent it from happening again. All the big cloud providers do this, AWS, GCP, Azure. Salesforce does it. I honestly won't trust a company that doesn't do this.

Now, Rivan has a bit of a different product here, but they still have a long ways to go with frequent and transparent communication about issues. IMO they are far behind standard practice in the tech industry. It's a bit weird to me, because their software engineering is headquartered in Palo Alto so they should have a lot of Silicon Valley culture where this kind of thing is just expected.

 

[jcalabek]

How do you know which clients are the Rivians? The reverse mac address lookup doesn't seem to actually work for me.

Trail and error. I would let the truck go to sleep and the wake it up, compared what new device popped up in the list. One useful option is also look for device that tends to upload a ton of data to AWS.

 

[hevak]

Trail and error. I would let the truck go to sleep and the wake it up, compared what new device popped up in the list. One useful option is also look for device that tends to upload a ton of data to AWS.

Hah...yeah, but so many things use AWS these days that it can be hard to narrow it down.

 

[Spork8]

Trail and error. I would let the truck go to sleep and the wake it up, compared what new device popped up in the list. One useful option is also look for device that tends to upload a ton of data to AWS.

For my router, the charger has a hostname of LCET-EVSE and wifi module manufacturer of u-blox AG. The truck doesn't have a hostname but has same wifi module manufacturer. But YMMV depending on your router as to what it discovers.

 

[ohseedee]

Seems a bit like this is all happening in realtime. The front line support folks are likely learning about this as we are.

Really hope this OTA goes OK for all those effected.

I've been on both sides of these issues in the past as a developer with an OTA updatable product and as a consumer.

Yep. Also seems like there is not full awareness in general on the front lines. Support asked me what version I had, how I installed it, then asked me if I performed a reset yet.

Anyway, really happy to hear OTA.

 

[tcole]

They do rollout to a beta list. The issue was not in the update they messed up the certificates when building the package for general release.

Which would have been caught if they'd done a canary rollout to any (even 1) vehicle that had only ever run production builds. In my opinion that's a huge process miss by not doing a canary first.

A canary deploy can't catch all issues. But it really seems like it would have caught this one.

 

[jcalabek]

For my router, the charger has a hostname of LCET-EVSE and wifi module manufacturer of u-blox AG. The truck doesn't have a hostname but has same wifi module manufacturer. But YMMV depending on your router as to what it discovers.

That's what I am seeing as well. Strangely enough our R1S shows up as TP-Link RE305, and R1T as u-blox AG.

 

[SANZC02]

There should be a lot less disdain for Tesla. I've only owned a Tesla for 2 years so far, but I've probably done about 50 software updates since I bought it. Not a single issue with an update. These vehicles are highly software dependent, so the least an EV startup should be able to do is update software without a hitch. Otherwise, folks will be afraid to run a software update. VW addresses this issue by not releasing any OTA updates. Rivian should have had a much more stringent software update quality controls. This debacle cannot be undone easily. Time to make RAN chargers free of charge for another year as a token of good will.

Things happen, a few years ago Tesla sent out an update to our Model S that caused the AC compressor to stop working, since the compressor is required for the BMS we could not use the car until they sent out an update. ??

 

[tcole]

https://electrek.co/2023/11/15/rivian-fixes-infotainment-software-bug-via-ota-around-3-affected/

Thanks for this. It confirms what I was speculating about earlier. They don't do a canary deploy to any vehicles running production software. Those "developer build" Rivian's they tested on probably had the developer certificates, so the build worked.

They need to test on vehicles that have never run developer or beta builds before they do a wider deploy.

Sponsored