Twas the night of production, and all through the server, all the bugs were stirring... We've all heard the developer horror stories or have had our fair share of shocking experiences as a developer tackling an unfortunate event.
To keep up with the Halloween season, we wanted to hear some of the spooky encounters that developers dealt with. Here are some of the stories that you submitted to Rollbar. Make sure to turn on the light and read with caution!
"I learned the hard way never to use the “latest” tag for Docker images in production. One of my projects used a base image from an open source project that packaged an optical character recognition (OCR) model. I used the latest tag and my Continuous Delivery process would build the image from the ground up, whereas my local dev environment kept using its cache of the base image. This was fine for a year, but then the pandemic started and suddenly that open source project was on fire — causing major updates to the image every 12 hours. Things started changing too fast and the inevitable incompatibility with my code happened about a week in when my CD process merged my code with an incompatible update to the image."
- Anonymous Developer at Transport Company
"I'm most frightened by bugs for which the development environment behaves entirely differently than production, so I'll tell you one of those. One time, I had to work with some code that generated temporary files in the local filesystem and then had to process them in sorted order. In production, the routine kept processing the files in a different order, messing things up for users. It really perplexed me for a while - I just couldn't reproduce the issue in dev.
"Finally, I realized that the code used a system call to order the files, which left it open to the collation preference of the underlying operating system. Once I realized this, I quickly re-implemented the sort to *not* use the underlying system to do it."
- Anonymous Developer at FinTech Company
"The nightmare of a developer is when you deploy your changes but a bug is triggered due to that change, and a Rollbar storm and Pagerduty call come at night!"
- Ye, Software Engineer at VMWare
" I once synced production cache with my local DB data and this led to actual users getting weird data, which I had manually edited in my local data. Luckily, I caught it soon enough and the production cache was cleared!"
- Anonymous Developer at Transport Company
"During my first week on a new backend engineering team I pushed some code to staging using some internal dev-ops tools. Everyone soon started getting alerts that production login was down and nobody could access the site anymore. It was soon found that there was a copy + paste error in the dev-ops code where all deploys went to prod. My staging deploy went straight to prod. Yeah, that was scary"
- Taylor, Software Engineer at Tatari
"Our system contained a dashboard for business owners to view data related to problems in their factories, I accidentally added dummy data that are randomly generated for demo purposes to the production dashboard for the client to see. The charts showed non-existing problems and the process was displayed for over a month or so until, luckily, I notice the mixed data and removed it."
- Anonymous Developer from Field Service Company
"This experience is from my past employer. It would take 6 hours to build a release candidate from start to finish and if there was an error anywhere, it had to be restarted from the top. As a result, our release schedule was once a month and about a third of the time was dedicated to building and releasing to production..."
- Anonymous Developer at Real Estate Company
"We have a dropdown list on the home page and the list loads from a server side cache object at the application code. There are bugs every time the list gets added again and at the end, the dropdown at the UI showed a long repeated list."
- Anonymous Site Reliability Engineer
"I always with get scared when we have to do a system integration. Generally there are two sides connected via API, Enterprise Service Bus, etc. Something is always missed, whether due to budget or due to lack of communication. That then causes ten times the work for my team. The hardware challenges become software ones and we, the developer, always take the heat and blame. I just hope there was a better integration solution."
- Anonymous Student
"During site testing of a large Air Traffic Control System, the system was periodically failing. After investigation it was discovered that a constraint error occurred when the wind blew from 0 degrees. The interface would only accept 1-365. So, the system worked great depending on which way the wind blew!"
- Anonymous Engineer Manager at Government
"I was interning at one organization, basically only few weeks had passed after joining the organization and I was still learning how to work with Git and Github, so what happened is I was making changes to my own repo which changed other interns' repos!"
- Engineering Intern at Transport Company
Too many developers have stories that will make your skin crawl. Whether it is from undetected errors that cause you to stay awake all night or hidden bugs that make even the most skilled developers shiver, responding to issues in code is never an easy feat. Here at Rollbar we can help get rid of some of these fears.
