Software Update 2023.42.0 is out (⚠️ UPDATE: OTA fix rolling out today 11/15/23)

[DTown3011]

alright, well this morning's school drop off was indeed an adventure.

Started off fine as I pre-heated the cabin to a toasty 75 degrees with heated seats and steering wheel. However, since my last drive setting was fan/low vs auto the second I put it into reverse cold air started to blow on everyone. About 2 minutes later the kids start to complain how cold it is. With heated seats and wheel, I'm fine, but the kids in car seats are not. 5 minutes in the complaints turn to moans and before it turns to cries, I pull over remembering the first aid kit has emergency blankets. I bust those out and wrap the kids in them. Out of guilt, I turn off the heated seats and wheel so I can suffer with them. The blankets did a great job, and we make it after 20 minutes. However, I overcompensate for the solo trip back home and crank the pre-heat to 80 with full seats and wheel. I regret that decision for the first 10 minutes as my ass gets overheated, but levels off eventually with the cold air and I'm fine by the time I roll back into my driveway.

So, before the next person posts a cute comment about "boohoo, no Spotify!" or "so it's the same as every car I drove 10 years ago!" remember that heat and AC are pretty fundamental features and actually pretty dangerous not having them in certain conditions. If you had it set to auto and a decent temp, you are good. If you didn't, you are not. I'm fine as it's only 50 degrees where I am, but I'm sure plenty of others are in really bad situations right now without heat or defrost capabilities.

honestly your kids sound like weenies, I think they need to be toughened up a bit, after all “in my day” I walked to school in the snow uphill both ways! ?

Sponsored

 

[Dark-Fx]

honestly your kids sound like weenies, I think they need to be toughened up a bit, after all “in my day” I walked to school in the snow uphill both ways! ?

My personal favorite, there's no such thing as bad weather, only inadequate gear.

 

[beyond]

alright, well this morning's school drop off was indeed an adventure.

Started off fine as I pre-heated the cabin to a toasty 75 degrees with heated seats and steering wheel. However, since my last drive setting was fan/low vs auto the second I put it into reverse cold air started to blow on everyone. About 2 minutes later the kids start to complain how cold it is. With heated seats and wheel, I'm fine, but the kids in car seats are not. 5 minutes in the complaints turn to moans and before it turns to cries, I pull over remembering the first aid kit has emergency blankets. I bust those out and wrap the kids in them. Out of guilt, I turn off the heated seats and wheel so I can suffer with them. The blankets did a great job, and we make it after 20 minutes. However, I overcompensate for the solo trip back home and crank the pre-heat to 80 with full seats and wheel. I regret that decision for the first 10 minutes as my ass gets overheated, but levels off eventually with the cold air and I'm fine by the time I roll back into my driveway.

So, before the next person posts a cute comment about "boohoo, no Spotify!" or "so it's the same as every car I drove 10 years ago!" remember that heat and AC are pretty fundamental features and actually pretty dangerous not having them in certain conditions. If you had it set to auto and a decent temp, you are good. If you didn't, you are not. I'm fine as it's only 50 degrees where I am, but I'm sure plenty of others are in really bad situations right now without heat or defrost capabilities.

Did a 50 mile drive the SC today, when I left it was around 40 or so, I forgot to pre-warm or to turn on any of the heat stuff... When I got there the cabin was a toasty 53f. While not ideal, glad it didnt' happen in the summer when it's super hot.

 

[Donald Stanfield]

alright, well this morning's school drop off was indeed an adventure.

Started off fine as I pre-heated the cabin to a toasty 75 degrees with heated seats and steering wheel. However, since my last drive setting was fan/low vs auto the second I put it into reverse cold air started to blow on everyone. About 2 minutes later the kids start to complain how cold it is. With heated seats and wheel, I'm fine, but the kids in car seats are not. 5 minutes in the complaints turn to moans and before it turns to cries, I pull over remembering the first aid kit has emergency blankets. I bust those out and wrap the kids in them. Out of guilt, I turn off the heated seats and wheel so I can suffer with them. The blankets did a great job, and we make it after 20 minutes. However, I overcompensate for the solo trip back home and crank the pre-heat to 80 with full seats and wheel. I regret that decision for the first 10 minutes as my ass gets overheated, but levels off eventually with the cold air and I'm fine by the time I roll back into my driveway.

So, before the next person posts a cute comment about "boohoo, no Spotify!" or "so it's the same as every car I drove 10 years ago!" remember that heat and AC are pretty fundamental features and actually pretty dangerous not having them in certain conditions. If you had it set to auto and a decent temp, you are good. If you didn't, you are not. I'm fine as it's only 50 degrees where I am, but I'm sure plenty of others are in really bad situations right now without heat or defrost capabilities.

What did you dress your kids in, those paper gowns you get at the hospital? I'm not sure if you're aware but they make clothes for cold weather, even for climbing to the top of a mountain. I've spent days working outside in -40 temps and I didn't die I think I could manage 20 min at 50 degrees in the car.

I call bullshit on the claim of dangerous. Wear a coat, you're welcome.

 

[wpnelson]

alright, well this morning's school drop off was indeed an adventure.

Started off fine as I pre-heated the cabin to a toasty 75 degrees with heated seats and steering wheel. However, since my last drive setting was fan/low vs auto the second I put it into reverse cold air started to blow on everyone. About 2 minutes later the kids start to complain how cold it is. With heated seats and wheel, I'm fine, but the kids in car seats are not. 5 minutes in the complaints turn to moans and before it turns to cries, I pull over remembering the first aid kit has emergency blankets. I bust those out and wrap the kids in them. Out of guilt, I turn off the heated seats and wheel so I can suffer with them. The blankets did a great job, and we make it after 20 minutes. However, I overcompensate for the solo trip back home and crank the pre-heat to 80 with full seats and wheel. I regret that decision for the first 10 minutes as my ass gets overheated, but levels off eventually with the cold air and I'm fine by the time I roll back into my driveway.

So, before the next person posts a cute comment about "boohoo, no Spotify!" or "so it's the same as every car I drove 10 years ago!" remember that heat and AC are pretty fundamental features and actually pretty dangerous not having them in certain conditions. If you had it set to auto and a decent temp, you are good. If you didn't, you are not. I'm fine as it's only 50 degrees where I am, but I'm sure plenty of others are in really bad situations right now without heat or defrost capabilities.

good lord.

 

[tcole]

That is all entirely possible. But I know that early software versions are on employee and company vehicles because I've seen it with my own eyes. It was on those vehicles weeks before it was released to us. Like I said, I have no other insight into their process and obviously there must be some flaws in it or this would not have happened at all. Just sharing what I saw ?‍

Yeah, I don't know much about best practices in embedded software development or automotive computer architectures. But I know a whole lot about deploying and operating software for highly available online systems.

And from what I can tell from this incident, Rivian isn't doing a few basic best practices.

Some background, with me speculating a bit and piecing together little details from what Rivian has said or what I've heard.

1. From Rivian job postings and stuff, I think they run a QNX Hypervisor, which is a pretty standard automotive operating system. The center screen runs an Android VM, and that's what went wrong here. Other more critical software functions are probably run on a separate VM, or maybe something running directly on the QNX hypervisor. I don't know quite enough about this area to make an educated guess.

2. It sounds like that Android VM has one set of certificates for beta / internal and a different set for production. And they accidentally pushed out a software package with the beta certificates. I'd imagine this is kind of like when you enroll in Apple's beta program for iOS. You download a certificate package that lets your device run those beta builds. Without it, they would fail to install. They also have a shorter certificate expiration to force an upgrade because Apple doesn't want really old betas lurking on devices.

3. Rivian went straight from beta users on employee vehicles to a 10% production rollout, lots of people tried to install it, and it immediately went bad.

Here is what I think is wrong with their process:

1. If they have this capability to run Android VMs on the QNX hypervisor, they should be able to do an auto-rollback when an update goes badly for one of those VMs. Keep the old one around, reboot into the new one, if the new one doesn't complete the boot process and pass some health checks, you stop it, and boot back into the old one. I'm sure there are a lot of details to get right, but this is kind of standard practice in things like highly available networking hardware that gets deployed in a telco building miles from anyone who's operating the device.

2. Their CI (continuous integration) system that does the production builds should be testing for the production keychain. So they are missing a critical test here. And if there is any error prone manual process in choosing the cert, they need to fix that too.

3. It's standard practice for important systems to have a canary deploy before you roll out to a larger portion of production. A canary should be a real production system, but a very small portion. You deploy to it, watch it, make sure everything is good, then start in on your staged rollout, 5%, 10%, etc of your fleet. (I don't know a bunch of internal details, so I'm speculating here) Rivian could have avoided this whole debacle by having something as simple as a handful of "production canary" vehicles that get the latest release first. These could also be employee vehicles or just sitting on the lot in Normal or Palo Alto, but they need to never have run any beta software. Deploy to those first, wait 24 hours, then start pushing to customers.

4. Something seems quite wrong with the way their system validates new software before installing it. If it's really that the wrong certificate signed the builds, the vehicle software should have checked the certificate BEFORE it started installing it. Fail that check fast, and refuse to update. They could have caught that quickly. But it seems like it validates the certificates AFTER the new system is installed, then fails and gets into some kind of crash loop.


Anyway, that's my wild speculation from from applying cloud / SRE best practices to a fleet of embedded systems.

 

[Spork8]

Not to derail the thread but my truck does an auto/courtesy wipe virtually at every start if the wipers are set to auto.

I have had this happen every time I bring the truck out of park like many of you are saying. It comes and goes depending on the software version.

 

[LevelHeaded]

I pull over remembering the first aid kit has emergency blankets...

I turn off the heated seats and wheel so I can suffer with them

Location: Los Angeles

This is hilarious. I love it.

 

[ohseedee]

I think this thread has done a great job of putting Rivan customers into 3 camps.

1. Those that bitch and moan about every panel gap, lack of car play, delay, etc.
2. Normal people that love the features and capability of Rivan, that knew and accept there was going to be some issues, but expected and needed a working vehicle.
3. Those that are so in love with Rivan that they can do no wrong and dream of RJ and his dreamy Clark Kent look at night.

If you are #1, Rivian is probably not for you, and you should sell. If you are in #3 and not impacted by this update, maybe hold your comments on this thread that downplay what the rest of us are dealing with right now. If you are #2, hopefully Rivian gets this fixed ASAP and you can get back to enjoying your truck.

 

[COdogman]

Yeah, I don't know much about best practices in embedded software development or automotive computer architectures. But I know a whole lot about deploying and operating software for highly available online systems.

And from what I can tell from this incident, Rivian isn't doing a few basic best practices.

Some background, with me speculating a bit and piecing together little details from what Rivian has said or what I've heard.

1. From Rivian job postings and stuff, I think they run a QNX Hypervisor, which is a pretty standard automotive operating system. The center screen runs an Android VM, and that's what went wrong here. Other more critical software functions are probably run on a separate VM, or maybe something running directly on the QNX hypervisor. I don't know quite enough about this area to make an educated guess.

2. It sounds like that Android VM has one set of certificates for beta / internal and a different set for production. And they accidentally pushed out a software package with the beta certificates. I'd imagine this is kind of like when you enroll in Apple's beta program for iOS. You download a certificate package that lets your device run those beta builds. Without it, they would fail to install. They also have a shorter certificate expiration to force an upgrade because Apple doesn't want really old betas lurking on devices.

3. Rivian went straight from beta users on employee vehicles to a 10% production rollout, lots of people tried to install it, and it immediately went bad.

Here is what I think is wrong with their process:

1. If they have this capability to run Android VMs on the QNX hypervisor, they should be able to do an auto-rollback when an update goes badly for one of those VMs. Keep the old one around, reboot into the new one, if the new one doesn't complete the boot process and pass some health checks, you stop it, and boot back into the old one. I'm sure there are a lot of details to get right, but this is kind of standard practice in things like highly available networking hardware that gets deployed in a telco building miles from anyone who's operating the device.

2. Their CI (continuous integration) system that does the production builds should be testing for the production keychain. So they are missing a critical test here. And if there is any error prone manual process in choosing the cert, they need to fix that too.

3. It's standard practice for important systems to have a canary deploy before you roll out to a larger portion of production. A canary should be a real production system, but a very small portion. You deploy to it, watch it, make sure everything is good, then start in on your staged rollout, 5%, 10%, etc of your fleet. (I don't know a bunch of internal details, so I'm speculating here) Rivian could have avoided this whole debacle by having something as simple as a handful of "production canary" vehicles that get the latest release first. These could also be employee vehicles or just sitting on the lot in Normal or Palo Alto, but they need to never have run any beta software. Deploy to those first, wait 24 hours, then start pushing to customers.

4. Something seems quite wrong with the way their system validates new software before installing it. If it's really that the wrong certificate signed the builds, the vehicle software should have checked the certificate BEFORE it started installing it. Fail that check fast, and refuse to update. They could have caught that quickly. But it seems like it validates the certificates AFTER the new system is installed, then fails and gets into some kind of crash loop.


Anyway, that's my wild speculation from from applying cloud / SRE best practices to a fleet of embedded systems.

That all makes perfect sense and I bet you are pretty close with your assessment. Thanks for the detailed post!

Hopefully they learned from this and will make changes to ensure it doesn't happen again?

 

[Redline]

alright, well this morning's school drop off was indeed an adventure.

Started off fine as I pre-heated the cabin to a toasty 75 degrees with heated seats and steering wheel. However, since my last drive setting was fan/low vs auto the second I put it into reverse cold air started to blow on everyone. About 2 minutes later the kids start to complain how cold it is. With heated seats and wheel, I'm fine, but the kids in car seats are not. 5 minutes in the complaints turn to moans and before it turns to cries, I pull over remembering the first aid kit has emergency blankets. I bust those out and wrap the kids in them. Out of guilt, I turn off the heated seats and wheel so I can suffer with them. The blankets did a great job, and we make it after 20 minutes. However, I overcompensate for the solo trip back home and crank the pre-heat to 80 with full seats and wheel. I regret that decision for the first 10 minutes as my ass gets overheated, but levels off eventually with the cold air and I'm fine by the time I roll back into my driveway.

So, before the next person posts a cute comment about "boohoo, no Spotify!" or "so it's the same as every car I drove 10 years ago!" remember that heat and AC are pretty fundamental features and actually pretty dangerous not having them in certain conditions. If you had it set to auto and a decent temp, you are good. If you didn't, you are not. I'm fine as it's only 50 degrees where I am, but I'm sure plenty of others are in really bad situations right now without heat or defrost capabilities.

I really, really hope you made this whole thing up...

 

[Foobar]

I have had this happen every time I bring the truck out of park like many of you are saying. It comes and goes depending on the software version.

This has happened to me a handful of times in the past 8 months of it being on Auto. Usually because something was on the windshield, so I presume it uses the camera to check for any "debris" on the windshield and then it wipes it away. If this is happening every time for others, I wonder if it's related to the condition of the windshield or something.

 

[DuoRivians]

Yeah, I don't know much about best practices in embedded software development or automotive computer architectures. But I know a whole lot about deploying and operating software for highly available online systems.

And from what I can tell from this incident, Rivian isn't doing a few basic best practices.

Some background, with me speculating a bit and piecing together little details from what Rivian has said or what I've heard.

1. From Rivian job postings and stuff, I think they run a QNX Hypervisor, which is a pretty standard automotive operating system. The center screen runs an Android VM, and that's what went wrong here. Other more critical software functions are probably run on a separate VM, or maybe something running directly on the QNX hypervisor. I don't know quite enough about this area to make an educated guess.

2. It sounds like that Android VM has one set of certificates for beta / internal and a different set for production. And they accidentally pushed out a software package with the beta certificates. I'd imagine this is kind of like when you enroll in Apple's beta program for iOS. You download a certificate package that lets your device run those beta builds. Without it, they would fail to install. They also have a shorter certificate expiration to force an upgrade because Apple doesn't want really old betas lurking on devices.

3. Rivian went straight from beta users on employee vehicles to a 10% production rollout, lots of people tried to install it, and it immediately went bad.

Here is what I think is wrong with their process:

1. If they have this capability to run Android VMs on the QNX hypervisor, they should be able to do an auto-rollback when an update goes badly for one of those VMs. Keep the old one around, reboot into the new one, if the new one doesn't complete the boot process and pass some health checks, you stop it, and boot back into the old one. I'm sure there are a lot of details to get right, but this is kind of standard practice in things like highly available networking hardware that gets deployed in a telco building miles from anyone who's operating the device.

2. Their CI (continuous integration) system that does the production builds should be testing for the production keychain. So they are missing a critical test here. And if there is any error prone manual process in choosing the cert, they need to fix that too.

3. It's standard practice for important systems to have a canary deploy before you roll out to a larger portion of production. A canary should be a real production system, but a very small portion. You deploy to it, watch it, make sure everything is good, then start in on your staged rollout, 5%, 10%, etc of your fleet. (I don't know a bunch of internal details, so I'm speculating here) Rivian could have avoided this whole debacle by having something as simple as a handful of "production canary" vehicles that get the latest release first. These could also be employee vehicles or just sitting on the lot in Normal or Palo Alto, but they need to never have run any beta software. Deploy to those first, wait 24 hours, then start pushing to customers.

4. Something seems quite wrong with the way their system validates new software before installing it. If it's really that the wrong certificate signed the builds, the vehicle software should have checked the certificate BEFORE it started installing it. Fail that check fast, and refuse to update. They could have caught that quickly. But it seems like it validates the certificates AFTER the new system is installed, then fails and gets into some kind of crash loop.


Anyway, that's my wild speculation from from applying cloud / SRE best practices to a fleet of embedded systems.

Great points. This can’t happen again

 

[Dark-Fx]

I think this thread has done a great job of putting Rivan customers into 3 camps.

1. Those that bitch and moan about every panel gap, lack of car play, delay, etc.
2. Normal people that love the features and capability of Rivan, that knew and accept there was going to be some issues, but expected and needed a working vehicle.
3. Those that are so in love with Rivan that they can do no wrong and dream of RJ and his dreamy Clark Kent look at night.

If you are #1, Rivian is probably not for you, and you should sell. If you are in #3 and not impacted by this update, maybe hold your comments on this thread that downplay what the rest of us are dealing with right now. If you are #2, hopefully Rivian gets this fixed ASAP and you can get back to enjoying your truck.

What's your address? I'll buy your kids some Rivian long sleeved t-shirts they can wear until Rivian fixes your truck.

Which camp am I in?

 

[Cosmacelf]

alright, well this morning's school drop off was indeed an adventure.

Started off fine as I pre-heated the cabin to a toasty 75 degrees with heated seats and steering wheel. However, since my last drive setting was fan/low vs auto the second I put it into reverse cold air started to blow on everyone. About 2 minutes later the kids start to complain how cold it is. With heated seats and wheel, I'm fine, but the kids in car seats are not. 5 minutes in the complaints turn to moans and before it turns to cries, I pull over remembering the first aid kit has emergency blankets. I bust those out and wrap the kids in them. Out of guilt, I turn off the heated seats and wheel so I can suffer with them. The blankets did a great job, and we make it after 20 minutes. However, I overcompensate for the solo trip back home and crank the pre-heat to 80 with full seats and wheel. I regret that decision for the first 10 minutes as my ass gets overheated, but levels off eventually with the cold air and I'm fine by the time I roll back into my driveway.

So, before the next person posts a cute comment about "boohoo, no Spotify!" or "so it's the same as every car I drove 10 years ago!" remember that heat and AC are pretty fundamental features and actually pretty dangerous not having them in certain conditions. If you had it set to auto and a decent temp, you are good. If you didn't, you are not. I'm fine as it's only 50 degrees where I am, but I'm sure plenty of others are in really bad situations right now without heat or defrost capabilities.

Sorry to hear, thanks for the report, and thanks for the laugh about your roasted backside!

And gawd, the rest of you piling onto the frozen kids, stop it with the holier than thou sentiments. It isn't a good look. You weren't there. Socal kids normally dress in shorts and t-shirts, so 50 degree cold air will indeed be a frigging problem. Show some empathy instead of dumping on some dad doing the best he can. Sheesh.

Sponsored