ChatGPT is Already Super Valuable

We need more thoughts from people who actually use it

A Limited Use Case

I apologize for the billionth article on this topic, but I haven’t read many pieces about ChatGPT from people who find it useful for their day-to-day lives. The articles I dislike fall into one of two categories:

1. ChatGPT sucks at something useless

2. ChatGPT might do something that it doesn’t do right now

I’ll start with Freddie deBoer. In a weekly digest post, he talked about the AI’s failure to produce a compelling tale in which Dracula fights Bruce Springsteen. Sure, it didn’t do a great job, but is there some unmet demand for fan fiction stories? I’ve never read fan fiction, but my understanding is that we’re not suffering from a shortage. Furthermore, most fan fiction readers and writers seem to enjoy the community aspect. Even if an AI could produce quality writing, most people would prefer to read stories written by a human author. I’m sure ChatGPT isn’t any good at writing original fiction either. Again, though, I don’t see much of a shortage for that. Kindle Unlimited probably releases more content in a month than you could read in your entire life.

Other times, people enjoy asking tricky questions to try to trip the chatbot up. Here’s a comment from a different deBoer article:

You can reliably make chat [GPT] fail the [Winograd] test. The trick [is] to split up the clauses into different sentences or [paragraphs]. Eg:

Person 1 has attributes A, B, and C. (More info on person 1).

Person 2 has attributes D, E, [and] F. (More info on person 2).

Person 1 wouldn’t (transitive verb) person 2 because they were (synonym for attribute[s] associated with person 1/2).

Chat GPT doesn’t understand, so it uses statistical regularities to disambiguate. It over indexes on person 1, because that’s the more common construction. Sometimes it can pattern[-]match on synonyms, because language models do have a concept of synonymy. But you can definitely fool it in ways you couldn’t fool a person.

Again… okay? You tried to mess it up and it messed up. My French press doesn’t make good coffee when I fill it with bleach or smash it with a hammer. I can’t think of any other product that’s analyzed in this way. If you want it to fail, have at it.

It might be worth remembering what a “chatbot” is. Think of the times you’ve used one. I recently logged into Fidelity, and the chatbot helped me locate tax forms and other information about my accounts. When I required more specific assistance, the bot transferred me to a human rep. A more advanced chatbot could have answered me without the need for a human. Still, both the human rep and chatbot helped me with small, discrete tasks. I didn’t plug “save for retirement” chatbot. If I provided such a vague instruction to a human, he or she would have tried to set up a time with the financial advisor. Furthermore, I certainly wasn’t trying to trip up the chatbot or human rep for the lulz.

With ChatGPT, then, we have a highly generalized chatbot. I’ve used it the same way I used the Fidelity chatbot: quick assistance with discrete, small-scale problems that I face on a day-to-day basis. I don’t need hockey commentary that sounds like it was written by Chaucer. Here’s what I have needed:

• One of my co-workers writes SQL queries in a counterintuitive way. Sometimes data sources change or we need to recalculate a metric, so I have to rewrite part of these queries. In the past, I’ve spent hours reading through them to make sure I understand each piece. Lately, I’ve just sent them to Chat GPT and it has rewritten them in a more standardized style

• I had to extract data from a URL. To explain this, consider the following URL from one of my articles: https://open.substack.com/pub/klaussimplifies/p/introduction-to-natural-language?r=lbykm&utm_campaign=post&utm_medium=web.

  • It contains a few pieces of information:

    • The blog name (klaussimplifies)

    • The title (introduction-to-natural-language)

    • Two UTM parameters (“post” and “web.”)

  • Extracting these values from the URL requires a “regular expression,” and I can never wrap my mind around these. I can usually put them together after a long period of trial and error, but ChatGPT wrote it in a few seconds.

• I asked it to create an empty Python “dataframe” (basically, a data table), run a series of statistical tests, and store the values in that dataframe. ChatGPT quickly wrote the code, and it ran more efficiently than what I would have written

• ChatGPT helped me create a visualization using Python’s notoriously confusing “matplotlib” package

• I needed to conduct a power analysis for a test with more than one treatment group. I knew which function to use, but I found some of the output ambiguous. ChatGPT explained the output better than the function’s own documentation did. It even helped me write the code for a post hoc statistical test (Tukey) for comparing the different samples.

I’m not cherry-picking or p-hacking. I’m not hiding 10 other examples where ChatGPT failed. These examples represent a few cases where I needed a bit of extra technical help and our friendly chatbot filled the void. Yes, I’ve solved a similar problem in other contexts. Without ChatGPT, I would have figured one out, but I would have produced less efficient code in more time.

The Competition

In the pre-ChatGPT area, I’d have started on Stack Overflow or similar tech sites. ChatGPT offers some advantages over these sites. For one, I don’t have to worry about wait times since ChatGPT answers in seconds. Nor will I hear that my question was already answered in a different thread, only to be linked to a thread that bares little resemblance to my problem. ChatGPT improves the process of finding code, too. Copy-pasted Stack Overflow code doesn’t work with your system or use case. With ChatGPT, I can send error messages and it will help me troubleshoot the issue.

Sure, the chatbot doesn’t replace a reliable mentor. In each of these cases, though, I don’t think I would have taken these problems to another person. Instead, I would have sat at my kitchen table and eventually powered through. Even if I reached out to a teammate, they could have been too busy to help or struggled to resolve the issue.

I’ve also heard people complain that ChatGPT provides incorrect answers. I’m sure it does. I also know, from experience, that tons of code on Stack Overflow doesn’t work.

The B2B Angle

I once listened to a podcast from a couple of executives at YCombinator, a firm that helps launch software startups. The podcast concerned common bad ideas for tech businesses. The hosts highlighted that this idea usually involved B2C (business-to-consumer) products. People encounter consumer products every day, and creators of these products can become celebrities. Tons of upcoming engineering admire Steve Jobs or Larry Page, but few of them could name the founders of Cisco. The host argued that, for the most part, there’s not as much demand for B2C products. If you upload a new social media or recommendation system onto the app store, few will notice. On the other hand, the market produces a ton of demand for B2B (business-to-business) products. As he put it, companies would pay a ton of money for something that makes their highest-paid employees more valuable.

And ChatGPT excels at that!

I’m not sure the average worker realizes the extent of B2B software. If you work at a large firm, they probably track call center information through apps like Zendesk or RingCentral. Your programmers, analysts, and IT professionals keep tabs on their progress through an app like Jira while your executives consult data visualization in Tableau or Power BI. Those datasets come from data engineering platforms like dbt or Airflow before being stored in cloud services like Google BigQuery or Amazon Redshift. If the firm wants customer feedback (the customers it keeps track of in Salesforce), it might run a survey through Survey Monkey. After receiving this feedback, a team might discuss design decisions in Figma. When I apply for jobs, I usually set up a meeting on Calendly, where the HR rep will enter some of my information into an applicant tracking system like Taleo. If they hire me, I’ll have to set myself up in an HR system like Dayforce. Every few weeks, someone messages me on LinkedIn to advertise a new pointless data science product.

Companies pay a lot for products that make their white-collar workforce more efficient. Most of these have no consumer uses. You probably wouldn’t get much out of Salesforce, but it’s still a multi-billion dollar company. In the examples mentioned above, ChatGPT probably saved me a couple of hours of work. Scale that to dozens or hundreds of employees, and you can see the value proposition. Imagine that the chatbot saves 2 hours per week. For a team of 20 employees, that amounts to 40 hours, a whole full-time employee, saved. That’s huge! For those of us who read about AI, I think we’ve gotten too accustomed to bold predictions about a revolutionary future. I don’t know if AI replaces lawyers or doctors or if a singularity will end humanity. I do know that saving a couple of hours per week, multiplies by millions of people, amounts to a massive increase in efficiency. Think of the increased fuel efficiency of cars. We didn’t see one day where they went from 10 MPG to 10,000. Over the decades, they’ve slowly ticked up from the low teens in the 1970s to the high 20s today. That accomplished a lot. It’s just not an epic revolution.

Granted, maybe I’m being too charitable to ChatGPT since it didn’t advertise itself as a B2B product. Again, though, I’m not cherry-picking a few test cases where it happened to work. ChatGPT has helped me in every scenario where I’ve sought its help, and those scenarios almost always occur at work.

But Let’s Not Get Carried Away

However, nothing I’ve seen from ChatGPT indicates some massive wave of automation. It’s fun to speculate about what it might do going forward, and one can argue for almost anything with an exponential line. This isn’t a speculation article, though. I’m only considering what our friendly chatbot does right now, and it doesn’t do anything that looks close to my day-to-day work.

That might sound confusing, given that I’ve listed examples where it automated tasks. However, I think there’s a point of confusion between the technical and lay audience when it comes to the meaning of “writing code.” I think a sports analogy will help. Let’s say we all agree that a given NFL coach excels at “calling plays.” We all understand, in this case, that the coach doesn’t boast any particular prowess at the physical calling of plays. Anyone with a mouth can say “Blue Dinosaur 52.” When one refers to the coach as a top-notch play caller, we mean that he’s talented at knowing which plays to call. The physical calling is the easy part. No one would confuse “stating the play’s name to your quarterback” with “knowing which plays to call.” Yet, people seem to make similar mistakes when it comes to “writing code.”

Writing the code (variables, classes, loops, etc.) is the easy part. The hard part is knowing what code to write. I’m making this statement from a data analytics angle, but I’ve heard similar thoughts from software developers. I also delved into this in an earlier piece about machine learning. To summarize, running a model involves little more than a “.fit” and “.predict” function. You don’t need a fancy AI for that. Someone at your local zoo could train a monkey to copy-paste code from the Machine Learning Mastery blog. The difference between quality data scientists and shoddy ones isn’t their knowledge of Random Forest models, it’s everything that occurs before and after creating the model.

Explaining what data analysts and data scientists do might require its own article, but I can try to run through a simple example. Earlier this week, someone noticed that a certain cohort of users stopped partway through the registration process, and he asked me if I could look into it. Notice that this differs from the discrete tasks mentioned above. I can’t solve that with a couple of lines of code, and it requires a greater understanding of the business. To complete this request, I had to check various data sources, each of which contain several counterintuitive nuances. I knew that major backend changes made certain data points unreliable before certain dates. In one dataset, a column’s naming convention provides incorrect information in some instances, but these can be corrected by searching for certain terms in another column. I produced a few summary tables with these tables and noticed a couple of oddities. I investigated these oddities by checking a different, more detailed, data source. I then selected the most important pieces of information and distilled them into a bulleted list. Finally, I anticipated follow-up questions and produced a couple of brief summaries to answer them.

I’m not sure if the preceding paragraph made any sense. Then again, if you didn’t understand it, imagine trying to ask a chatbot to perform it. I chose that example to illustrate that most of my work does not consist of discrete tasks with a simple right-or-wrong answer. Even the examples listed in the first section amounted to tiny parts of much larger projects. When I interviewed for this role, I didn’t sell the hiring manager my ability to write regular expressions or run power analyses on ANOVA models. I talked about how I could identify valuable projects and bring them to completion.

I can’t really speak too much to the software development angle, but I understand that the same logic applies. The difficulty in software engineering comes in converting abstract problems into discrete blocks of code and collaborating with your team to ensure that everything fits together. ChatGPT helps with the execution, but that only comes into play after the heavy lifting has been completed. It can’t architect a solution from scratch.

Yes, maybe exponential growth kicks in and AI automates everything. I don’t know how to evaluate these sorts of claims, so I’ll remain neutral on the question of AI’s potential. Still, right now, I can say that ChatGPT provides a ton of value. It’s not Skynet. It’s not replacing doctors, lawyers, and programmers. It’s not disrupting the mere notion of the knowledge worker. It solves a small number of common problems, and that’s all I ask for from technology.