My Deep Dive on R2 Autonomy with Rivian Engineers (LiDAR, RAP1 AI Chip, Cameras, Radar, Software)

[mpshizzle]

The video on my YouTube channel (Thunder Volt Auto) provides a lot more context and detail:



So I recommend watching that.


I just got to talk with a lot of Rivian’s Engineers at their AI and Autonomy Day in Palo Alto, and let me tell you: my brain is full.


It’s clear that for the upcoming R2, Rivian isn't messing around. This car *has* to be a hit, so they are throwing absolutely everything they have at it.


Here is the breakdown of the monster tech that’s about to end up in the R2:


1. Meet "RAP1": The Brain That Eats Gaming PCs for Breakfast
The biggest mic-drop moment? Rivian built their own processor. It’s called the Rivian Autonomy Processor 1 (RAP1), and on paper, it is an absolute beast.


If you’re a gamer, you know the Nvidia RTX 4090 is the king of the hill, pushing about 1,300 TOPS. Well, the new computer in the R2 uses 2 of these new RAP chips for a combined 1,600 TOPS.


Now, there’s a little catch here: We don’t know if they plan to run these two chips in parallel (maximum speed) or as a redundant pair (safety net). If one chip has a problem, the other one takes over instantly. But if that’s the case, you’re still getting 800 TOPS of performance, built on the same ultra-efficient 5nm tech as Apple’s M1 chips.


2. The Cameras: Similar, but NOT the same ones from R1
Rivian is sticking with the 11-camera layout we know from R1, but they’ve given everything a nice little spec bump. Slightly higher dynamic range, etc. But there are 10 megapixels of resolution somewhere. R1 G2 has 55MP of resolution, R2 has 65MP. My guess is those extra megapixels likely come from the interior cabin camera. In R1, my understanding (though not 100% confirmed) is that the driver monitoring camera hidden in the mirror is the only camera that is NOT a Rivian design. Since Rivian's new favorite thing is vertical integration, I'm guessing they swapped this for something of their own design, with some extra megapixels.


3. The Radars: The front imaging radar seems to be unchanged. But the corner radars have been upgraded to dual mode radars. That basically means they canfar away (for driving down the road) and up close (for parking). This replaces the sonar sensors in the bumper.


Honestly? If implemented well, I'll bet most drivers won't notice. And I think it's a smart chage. Fewer points of failure and less cost in manufacturing.


4. The BIG Change: LiDAR
Aside from the custom chip, LiDAR is the other big change everyone is talking about. But why? There seem to be 2 main reasons:


*The 99% Problem: Cameras are great for 99% of driving. But for that terrifying 1% (heavy fog, blinding glare, total darkness) different sensor types fill in the gaps. With 3 different types of sensors, each with varying strengths in different conditions, you get a lot more assurance you know what's going on around the vehicle in challenging conditions. To be clear, the cameras will still handle the heavy lifting and do most of the work, but having Radar and LiDAR fill in the edge cases. If Rivian wants to reach "Level 4", they need that safety net.
*The Penguin Effect: Rivian has a fleet of vehicles called "Penguins". R1 vehicles with crazy lidar rigs and extra sensors strapped to them. This helps them get what they call "Ground Truth". A very precise map and data set to assist in training the LDM (Large Driving Model). By putting LiDAR on R2, they effectively turn thousands of customer cars into data-gathering machines, allowing them to create even more robust training sets than the data that comes from R1 Gen 2 vehicles. This will even further accelerate their model training.


5. The Software Stack and the Future of Gen 2


Rivian (as with everything, it seems) owns the entire stack from top to bottom. Their custom middleware layer runs as a "hardware abstraction layer" that lets the Autonomy code run exactly the same way, no matter which hardware it's running on. So in theory, the same Autonomy stack and models can run on both Gen 2 and Gen 3 hardware with little to no modification.


As far as the current state of the software: I actually rode in a 100% factory (not modified) R1 Gen 2 running the new "Point-to-Point" autonomy prototype software, and it handled it like a champ. The engineers told me the current chips still have some headroom. So for the next while it's probable (and dare I say likely) that Gen 2 and Gen 3 will operate with the same feature set and capability.


That said, there is a ceiling. Rivian’s end goal is "Personal Level 4"—eyes off, hands off, car drives the kids to soccer practice alone. Because the R1 Gen 2 lacks that LiDAR safety net, it probably won't ever get *that* specific feature. Even though the cameras and (and possibly the compute) may be able to handle that task most of the time, it doesn't have the camera+radar+LiDAR safetey net for that 1% of the time.


But even so, gen 2 vehicles still benefit from all the data those R2s are collecting, making it a much more competent driver.

If you found this information useful or interesting, use my referral code to get up to 500 points in the Gear Shop and free charging! https://rivian.com/configurations/list?reprCode=MIKE4580282

Sponsored

 

[CharonPDX]

On the "parallel vs redundant" front - the fact that the two chips are right next to each other in a single housing makes redundancy semi-pointless. If something is impacting one of the two, it would likely impact the second. So it really sounds like it's parallel processing. (Plus "Jerry Rig Everything" commented in his video about them that Rivian has architected it "if two isn't enough, they can just add more in future vehicles.")

 

[portdirect]

On the "parallel vs redundant" front - the fact that the two chips are right next to each other in a single housing makes redundancy semi-pointless. If something is impacting one of the two, it would likely impact the second. So it really sounds like it's parallel processing. (Plus "Jerry Rig Everything" commented in his video about them that Rivian has architected it "if two isn't enough, they can just add more in future vehicles.")

From a physical standpoint, fully agree - however they still can provide computational redundancy if they chose to, would be interesting how they tiebreak or avoid split brain if taking this approach.

Ive been in a truck whist the ADAS has failed - and it’s terrifying; I’m very interested in how Rivian have ensured redundancy and safe operation during failure modes to avoid as many critical/instant SPOFs as possible.

 

[mpshizzle]

On the "parallel vs redundant" front - the fact that the two chips are right next to each other in a single housing makes redundancy semi-pointless. If something is impacting one of the two, it would likely impact the second. So it really sounds like it's parallel processing. (Plus "Jerry Rig Everything" commented in his video about them that Rivian has architected it "if two isn't enough, they can just add more in future vehicles.")

That's a great point! All of the marketing push certainly hints at using it in a parallel configuration. And that is true, they have what they are calling "RivLink" that can n theory link any number of these processors together. Gee that reminds of nv link

 

[DuoRivian]

Isn’t Apple M1 tech now 4-5 years old since they are on M5 which is faster and smaller nm scale than M1.

 

[portdirect]

Isn’t Apple M1 tech now 4-5 years old since they are on M5 which is faster and smaller nm scale than M1.

I think Rivian are pushing this narrative to help provide grounding for consumers. It’s almost certainly not the same 5nm process that Apple currently use but N5P which NXP and others have been using for ~5 years now; Apple also used this process for A13 and M2.

edit: Apple did use N5P.

 

[savethemanual]

Buying a piece of technology today, one has to assume it's outdated very shortly down the road. Still very usable, mind you...but all technology gets replaced with the latest and greatest within a few years time. I feel many don't have this mindset when buying and get frustrated when the next best thing comes along. Would be nice to see manufactures support modular upgradeable designs upfront, having the ability to simply swap out a new processor when the latest greatest comes along would be fantastic!

 

[zefram47]

Your specific ride may have "handled it like a champ", but there are other videos around where the safety driver had to perform multiple interventions where collisions likely would've occurred. It's definitely impressive how far they've come in a relatively short period of time, but they also have a long way to go. And just remember that all the claims they're currently making may never be realized on the current Gen2 or even the Gen3 hardware. There were a great many claims made about the Gen1 hardware that will never happen, including talk of SAE Level 3 capability.

 

[mpshizzle]

Your specific ride may have "handled it like a champ", but there are other videos around where the safety driver had to perform multiple interventions where collisions likely would've occurred. It's definitely impressive how far they've come in a relatively short period of time, but they also have a long way to go. And just remember that all the claims they're currently making may never be realized on the current Gen2 or even the Gen3 hardware. There were a great many claims made about the Gen1 hardware that will never happen, including talk of SAE Level 3 capability.

Later this week I will be releasing another video doing some "pixel peeping" so to speak on my demo drive. But I should clarify here. When I said "handled it like a champ" I was referring to the hardware. The point to point driving system was running on a completely stock Gen 2 with headroom to spare. And that's without them even trying to optimize the code yet. So from a hardware perspective, I have a fairly high level of confidence that Gen 2 will see at least a pretty good point to point system. But even so, the software clearly has a long way to go. My drive certainly did go a lot smoother than it did for others at the event. It's impressive how far they've come, and this rate of improvement bodes well. But there are certainly strides to be made

 

[portdirect]

‘Headroom to spare’ is an ominous quote @mpshizzle ! I hope it’s not got the same stark outlook gen1 had

Being more serious - it’s going to be interesting seeing the delta between gen2 (Orin) and gen3 (RAP1) manifest in capability. At some point it’s only natural they actually use the additional headroom provided by the next generation of hardware, otherwise why build it?

 

[captainjp]

Your specific ride may have "handled it like a champ", but there are other videos around where the safety driver had to perform multiple interventions where collisions likely would've occurred. It's definitely impressive how far they've come in a relatively short period of time, but they also have a long way to go. And just remember that all the claims they're currently making may never be realized on the current Gen2 or even the Gen3 hardware. There were a great many claims made about the Gen1 hardware that will never happen, including talk of SAE Level 3 capability.

 

[mpshizzle]

‘Headroom to spare’ is an ominous quote @mpshizzle ! I hope it’s not got the same stark outlook gen1 had

Well... There are no guarantees. That is for certain. What we do at least know for sure is that
1) They DO have working point to point software running on gen 2 (albeit with the need for improvement)
2) They do at least have complete control of the hardware/software stack, unlike mobileye in gen 1.

Beyond that... IDK don't look at me. I'm not a fortune teller haha

 

[jwanderson88]

When I heard about the "Rivian chip", I wondered about it. If I were a Rivian engineer, I would use the Rivian chip to catalog all the objects around. That's something that has to be done fast and often. It doesn't benefit from frequent hardware updates. Objects are either there or they aren't. Then the driving software would decide what to do about it. I would expect that to be constantly updated, so hardcoding it would be bad. Maybe that's the way it works, I haven't seen info on that.

 

[BrayBay]

Good video to distill that one hour presentation + real life demos into something more digestible and shareable.

On the "ground truth" topic, makes me wonder if the Amazon EDVs will get upgraded with LIDAR. I always thought from the onset of the Rivian-Amazon partnership that it be incredibly advantageous to have the EDVs gather the road data. Those vans have to go to practically every address after all.

While have a large number of R2 drivers out there will help with the data gathering, I can't help but think that the EDVs (especially if they hit 100,000 units) cover way more ground and see way more edge cases in a single delivery run that most drivers would see even in a month.

 

[HapticWagon]

Later this week I will be releasing another video doing some "pixel peeping" so to speak on my demo drive. But I should clarify here. When I said "handled it like a champ" I was referring to the hardware. The point to point driving system was running on a completely stock Gen 2 with headroom to spare. And that's without them even trying to optimize the code yet. So from a hardware perspective, I have a fairly high level of confidence that Gen 2 will see at least a pretty good point to point system. But even so, the software clearly has a long way to go. My drive certainly did go a lot smoother than it did for others at the event. It's impressive how far they've come, and this rate of improvement bodes well. But there are certainly strides to be made

IMO eyes-free will never come to the non-LiDAR models.
but time will tell! looking forward to point-to-point even if it’s just hands-free, eyes-on

Sponsored