The Ugly Truth about Software

[OldGoat]

As I read these threads, it occurs to me that there are a some who don't have a full appreciation for software/software development. This is not meant as a slight in the least. I find myself in the same situation when it comes to discussions around mechanical issues like suspension or technical threads dealing with batteries and electrical systems. As someone, now retired, who spent most of my life in the software world from code slinger to running software development to consulting, I thought it would be fun/interesting to share some war stories from the trenches. I know there are a number of others on here that could contribute with interesting lessons learned or humorous stories. (And please I DO NOT want this thread to derail into a discussion of how YOU would never do X or how YOU would solve Y...this is thread is not about YOU and how great YOU are! It's just meant to have some fun and give some perspective around the ugly truths of software!

Sponsored

 

[bael]

 

[OldGoat]

First lesson learned..It was 1979, just starting out working at a Fortune 500 company. I was writing a COBOL program. It was decent sized, ~10,000 lines of code. (For you youngsters out there, even using repetitive techniques like PERFORMS, 10k lines was not that unusual). It had a fairly complicated calculation in it. Not complicated in the operations, just a long, multi-nested equation. While testing it, for the life of me, I couldn't get the right answer....my boss at the time was extremely bright (had an advanced degree in computer science) so after awhile he and I tried to figure out what the hell was going on. FInally, we narrowed it down to an issue with the compiler. I remember it well as it was a fairly new compiler 1978 ANSI COBOL issued by IBM. IBM! Now in the 70's there was no more highly respected vendor in the commercial DP (that's what is was called back then) space than IBM. Anyhow, it was quite a rush for a neophyte to sit down with IBM to discuss a bug we found with their compiler...a compiler being used for millions of lines of code across darn near every Fortune 500 company.

Lesson learned: bug free software is hard, no matter who you are!

 

[OldGoat]

Many years later, I was working at a software company with a pretty esoteric client server application used for mission critical applications. We were very, very picky about the hardware used, certifying to a very limited set of clients (PCs). Anyhow, we had a client (company) that had a fairly large implementation that had been running fine for awile when suddenly they started having weird, flakey things happen. The software side was stumped, the network techs were scratching their heads. At some point, it was discovered that they had added some more users (which they did all the time). However, people far smarter than I determined that the PC manufacturer had made a minor tweak to the BIOS. Apparently, it created some type of latency issue which, in turn resulted in these intermittent issues.

Lesson learned, once you 'open' the system up to other components, the ability to identify problems increases dramatically.

As a sidenote, I believe I read that one of the drivers behind Rivian developing their own motors was the fact that the Bosch motors were really picky about the chipset used.

 

[OldGoat]

Last one for now...I always tell people the reason for testing is not to make sure your software works but, rather, to find what doesn't work. May seem like the same thing but it really isn't. Any decent developer/software engineer can make his or her code work.

Toward that end, as a final testing process, we'd bring in a bunch of people and have a contest with significant $ to those who found the most bugs. Of course, at this point the software had been beat to death by various automated and non-automated testing processes so there wasn't a lot to find. Occassionally, someone might find something that slipped through the cracks but they mostly found cosmetic issues. mostly...as there always seemed to be some bizarre issue someone would find.

This happened once where a tester was able to create an issue through a bizarre series of keystrokes, including having to hit the mouse with her elbow while holding down several buttons. Now if you have lived in the software world, you know that most of us are pretty soft-spoken and introverted. The person who wrote the offending code was such a person (brilliant guy that went on to get a PHD in ISD)...he never said much and never raised his voice. Well, when we went to show him the issue and, more importantly what was actually required to re-create....needless to say, he was not soft spoken. I think that may have been what drove him to go back to school to get his Phd!

Lesson learned and the most valuable one of all...the most heinous of variables one can introduce to software is....the user! You just never know what those knuckleheads are going to do!

 

[Dark-Fx]

Lesson learned: bug free software is hard, no matter who you are!

All software is bug free if you just say "it's supposed to do that".

 

[OldGoat]

All software is bug free if you just say "it's supposed to do that".

"It's a feature" (I admit I may have said those words a time or two.)

 

[SANZC02]

"It's a feature" (I admit I may have said those words a time or two.)

You can only get away with it if you call it an undocumented feature.

 

[zefram47]

Last one for now...I always tell people the reason for testing is not to make sure your software works but, rather, to find what doesn't work. May seem like the same thing but it really isn't. Any decent developer/software engineer can make his or her code work.

Toward that end, as a final testing process, we'd bring in a bunch of people and have a contest with significant $ to those who found the most bugs. Of course, at this point the software had been beat to death by various automated and non-automated testing processes so there wasn't a lot to find. Occassionally, someone might find something that slipped through the cracks but they mostly found cosmetic issues. mostly...as there always seemed to be some bizarre issue someone would find.

This happened once where a tester was able to create an issue through a bizarre series of keystrokes, including having to hit the mouse with her elbow while holding down several buttons. Now if you have lived in the software world, you know that most of us are pretty soft-spoken and introverted. The person who wrote the offending code was such a person (brilliant guy that went on to get a PHD in ISD)...he never said much and never raised his voice. Well, when we went to show him the issue and, more importantly what was actually required to re-create....needless to say, he was not soft spoken. I think that may have been what drove him to go back to school to get his Phd!

Lesson learned and the most valuable one of all...the most heinous of variables one can introduce to software is....the user! You just never know what those knuckleheads are going to do!

Lol! I'm sure any of us that have done front end / GUI work have stories like this. Ask a group of users to perform a particular task and it's amazing how many different ways people will go about something that weren't intended by the developer and some end up exposing bugs. With so few buttons in the R1 I do often find myself asking if I were the developer how I would go about doing something that wasn't obvious in the interface. Like how to cancel a turn signal or how to get a single wipe (mist) on the wipers...usually it winds up being the obvious (to me as a developer) thing.

 

[R1Tom]

So before I decided to go the mechanical route, I thought I wanted to be a programmer. Way back. Learned Fortran and even played with assembly language...whatever that was. Then said I am out...I will do mechanical. So to say....I am not software savvy.

What I am curious about....is my concept of how regarding works is cleary bad. But I would have thought that once a charger access door is programmed....it is programmed. Why after an update....does something like chop chop creap in?

Is it like sequential data stream and if one item is left blank then everything that follows is one spot off?

I just want to understand better and it seems many on here know how this stuff works...

 

[BigSkies]

Not software directly, but I used to work on the business side of a software startup. We would do all types of forensic data analysis on communications used in Fortune 500 companies. It was mostly related to active litigation.

Anyways, the data sets were so large that it made the most sense for customers to copy data to hard drives and mail us the hard drives.

So we ran our own internal data center. But as a bootstrapped startup that could barely make payroll, our data center was held together with metaphorical duct tape and bailing wire. Our IT guy was pretty good at scavenging up free components from other places that had upgraded or otherwise found a reason to throw stuff in the trash. They made it work, but it wasn't pretty. It was a mis-match of random hardware that spanned several decades of manufacture dates.

About a week before a major client was using our data in litigation that was high enough profile to be in the WSJ, the AC in the data center craps out. About half the machines were fried before the IT guy got there.

Through whatever miracle, there were enough good harddrives in the RAID arrays that customer data wasn't actually lost. This was a minor miracle, as statistically we should have lost something major.

All the client ever knew was that an "IT issue was creating intermittent slowness" for a few days.

 

[p3ck]

So before I decided to go the mechanical route, I thought I wanted to be a programmer. Way back. Learned Fortran and even played with assembly language...whatever that was. Then said I am out...I will do mechanical. So to say....I am not software savvy.

What I am curious about....is my concept of how regarding works is cleary bad. But I would have thought that once a charger access door is programmed....it is programmed. Why after an update....does something like chop chop creap in?

Is it like sequential data stream and if one item is left blank then everything that follows is one spot off?

I just want to understand better and it seems many on here know how this stuff works...

I thought about this one for a bit.. and if it hadn't been fixed so quickly I was going to try different scenarios to reproduce. One thought I had was I wonder if they are using a queue for tasks. They may do this for long running processes or just to keep too many operations from happening at the same time. anyway.. one thought was if you happened to touch the door open/close area it may add to the queue but can't act on it yet because the charger is plugged in. Who knows how the timed close worked, it could have cleared out the queue for door operations when unplugging?

I'm really just guessing of course.

 

[OldGoat]

Lol! I'm sure any of us that have done front end / GUI work have stories like this. Ask a group of users to perform a particular task and it's amazing how many different ways people will go about something that weren't intended by the developer and some end up exposing bugs. With so few buttons in the R1 I do often find myself asking if I were the developer how I would go about doing something that wasn't obvious in the interface. Like how to cancel a turn signal or how to get a single wipe (mist) on the wipers...usually it winds up being the obvious (to me as a developer) thing.

Truly one of my favorite things is usability testing. You bring in a person have them sit in a room with the computer signed in a dummy account and the tester will have them do various tasks...for example: "OK, go ahead and change your address." There is a one-way mirror so you can watch the person try to do the task. It can be hilarious watching people struggle...people will have to restrain themselves from screaming at the mirror 'it's right there on the left nav bar!" or "press account change, you moron!" It is amazing how something so intuitive to one person is so obscure to another.

 

[SANZC02]

So before I decided to go the mechanical route, I thought I wanted to be a programmer. Way back. Learned Fortran and even played with assembly language...whatever that was. Then said I am out...I will do mechanical. So to say....I am not software savvy.

What I am curious about....is my concept of how regarding works is cleary bad. But I would have thought that once a charger access door is programmed....it is programmed. Why after an update....does something like chop chop creap in?

Is it like sequential data stream and if one item is left blank then everything that follows is one spot off?

I just want to understand better and it seems many on here know how this stuff works...

The code flows in and out of different subroutines, because of this you can impact what at first glance would be an unrelated process when making any changes.

There is an axiom that the smaller and simpler the change the more likely there will be a failure. This comes from less forethought and analysis. A big change is heavily scrutinized from all angles, a small simple change is attacked with much less scrutiny.

 

[OldGoat]

I thought about this one for a bit.. and if it hadn't been fixed so quickly I was going to try different scenarios to reproduce. One thought I had was I wonder if they are using a queue for tasks. They may do this for long running processes or just to keep too many operations from happening at the same time. anyway.. one thought was if you happened to touch the door open/close area it may add to the queue but can't act on it yet because the charger is plugged in. Who knows how the timed close worked, it could have cleared out the queue for door operations when unplugging?

I'm really just guessing of course.

Yeah, seems like a reasonable guess, but who the heck knows what they may be doing. If one ever wants to experience the law of unintended consequences...get into software...sometimes it's like playing whack-a-mole...squash one bug and another pops up.

Sponsored