This post originally was posted on Medium. Go check it out!
ChatGPT is already replacing jobs and information workers have real reasons to be worried. This week I discuss why I don’t think QA and Testing jobs will be replaced. Changed certainly, but not replaced.
If you haven’t read my initial thoughts on the impact of Large Language Models to engineering, I suggest you start there since this post builds on those thoughts. My thoughts here are not as fully formed because things are rapidly changing in our industry and I’m learning more daily.
I hope this sparks some new ideas for you and if it does, I’d love to chat about them!
All three bets are different angles of the same theory:
Software is about to be a lot cheaper, easier, and faster to produce in bulk. Without guardrails, it will also be much poorer quality.
If software is cheaper, easier, and faster to build there are a few implications:
It will be possible to solve problems that were too expensive or complicated to solve before. Many of these will be small internal efficiency gains but enough small gains add up to huge impacts on the bottom line. This feels similar to the gains made optimizing a lot of little steps in an assembly line.
Some of these will be much bigger things that a traditional software engineering team would be relevant but that were too much to tackle before with the team at hand. We’ll come back to this in a second on 3).
This has always been happening. I didn’t find a current number but in 2015 Microsoft said 1.1 billion people used their productivity tools including Excel. I bet the Google numbers are similar.
Now imagine a world where someone can ask ChatGPT to generate Python code to analyze data for them.
Actually, don’t imagine it. You can just ask ChatGPT.
(Note: I didn’t run this code but I have done similar experiments where I actually did and it was pretty close to working with a few minor changes)
The ability to ask ChatGPT for code won’t be limited to business users. Engineering teams will do the same things and integrate this code into public facing, mission critical apps where security, performance and reliability are critical.
Businesses won’t be happy only applying the efficiency gains we talked about above to small internal things. They’ll demand to see their big showcase software accelerate too.
Bet: Once executives see accounting or marketing get huge efficiency gains from using LLMs they’ll demand their highly compensated engineering teams do the same. They’ll be right to do so.
While professional software engineering artisans still produce bugs, we assume some level of inherent quality in the individual components they create. The most nuanced bugs are usually found when each piece of craftwork has to plug into a bigger system.
If the same software engineers are now producing 2x (or more!) code, how are they going to maintain any level of quality?
Now remove the artisan from the system whom we count on for some automatic level of quality and security and hand the tools to people with other skill sets. Then crank out 5x or 10x as much software. You can probably see where this is going.
I think there are two dimensions where things will change for people in quality focused roles.
Manual testing doesn’t scale well as teams grow today and it will scale even more poorly with GPT-driven development. Ideally, engineers will wrap their code with unit tests but more code means more regressions across a complicated system.
A lot of teams already use some combination of engineer-written tests and automated testers. Whatever the strategy here, the amount of testing will need to increase. A lot of teams will try to solve this by hiring more testers.
GPT can help with automated tests so the pool of people who can be an automated tester today just increased. You can ask GPT to write tests as easily as you can ask it to write code. This will be a huge help to anyone approaching testing from a less technical background.
It will be difficult to test everything, all of the time. In particular, it will be impossible to test the code being generated inside of business departments and outside of standard development processes.
Quality and Engineering leadership will need to have defined, published, and enforced architectural and quality strategy that everyone buys into. This strategy needs to describe how systems integrate, how software boundaries (where testing can easily occur) are defined, and what can and can’t integrate with critical systems.
Bet: Quality Assurance teams in software become more like QA on an assembly line.
They might not test everything but they will sample systems as they evolve and compare them to the strategy. When issues are found, they’ll work to improve processes to prevent that issue again.
I think these quality experts will have to partner with power users in the business to try to instill the same quality principles that are mandated in engineering. They will definitely need to police the boundaries to make sure nothing flows into mission critical systems that don’t follow quality practices.
For internal software this might not be the case, but for customer facing software (particularly with any sensitive data or in regulated industries) more QA people will be needed.
Software complexity and risk don’t scale linearly with lines of code. More code produces more complex interactions between units of code, and the surface area for defects increases exponentially.
Bet: Software teams will see more and more costly defects when they start using LLMs to accelerate their work. Many teams will respond by hiring more testers.
I don’t have a fully formed view of how this will work but I think we’ll also see a need for expanded and different quality teams on SaaS products and in regulated enterprises like healthcare or finance. These expanded quality teams will combine traditional testing with security and compliance awareness.
Bet: In the next couple of years we’ll hear of an entirely new role inside of software teams that’s the evolution of Quality Assurance.
I don’t know what this is, but I hope it’s not another terrible evolution of the AbbreviateEverythingOps naming convention like DevSecTestOps.
I expect to come back to the topic of GPT and other LLMs on Quality in the future when I’ve had more time to reflect.
Next week however, I want to dive deeper into the business impacts of this tech. I’ll touch on how I think executives should be thinking about how LLMs will eventually impact their businesses and what they should be doing right now to prepare.
]]>This post originally was posted on Medium. Go check it out!
ChatGPT is an amazing new tool that promises to help engineering teams do everything faster. But like using a microwave to make Thanksgiving dinner, faster is no guarantee of better.
Large Language Models (LLMs) like GPT are powerful text generation tools; you can think of them as really advanced autocomplete. They’ve seen millions of patterns. Most of the work a team does is not completely revolutionary, and they can usually find something that matches.
ChatGPT is a chatbot layered on top of GPT. It’s a human user experience for a complex piece of technology.
There are already specifically trained tools like GitHub’s Copilot (which is based on GPT’s tech) that are focused on matching the code it’s been trained on with what engineers are writing. ChatGPT is a more general text completion tool or “stochastic parrot” but it does its job well. Specialty tools like Copilot do similar things but embed closer to the developer experience. I expect to see many competing products in 2023.
None of these LLMs are new brilliant general AIs. They won’t solve your business problems or innovate something completely new. None of them are autocomplete for your business strategy.
The hyperbole today is that LLMs are “changing everything”. I think this is true, but not in the “telegraph ended the Pony Express in 18 months” sense. Engineering isn’t going away.
This changes everything today in the “Google made it possible to find code on the internet” way that instantly made every developer twice as effective. This is another step in the evolution of engineers gaining knowledge that went from expensive and huge books about software languages to using the internet.
Let’s take a simplified model of software engineering and say there are two basic modes of work:
In Mode 1, LLMs are useful for generating boilerplate code (i.e. structural code that’s necessary but doesn’t have business value) and for exploring technology or patterns developers aren’t familiar with.
Most of the examples being talked about are boilerplate. The actual value is not here.
The real value in Mode 1 is for giving engineering a head start on complicated technical tasks. When working on complicated work, engineers can spend hours or even days exploring a concept before they get close to writing the first line of useful code. LLMs can short circuit that.
Here’s an example of me exploring code to work with maps and plotting. I’ve never worked with projections (i.e. how we flatten the roundish Earth to flat images) and would have no idea where to start. ChatGPT gave me a starting point that did a few things:
ChatGPT is surprisingly good at telling you why it did something. You can ask it why it used a library, or ask it where a bug is likely going to happen. I’ve found that for routine tasks it does a good job telling you why and not just what, if you ask.
My bet — LLMs will quickly become the normal starting place for development teams starting a new project. Within 18 months engineers won’t think twice about using them, just like Google or StackOverflow now.
Mode 2 is harder to predict and a little different. GPT is already pretty good at laying out how you’d solve a specific problem but it’s not great at suggesting how to make decisions that will set you up for success 6/12/18 months from now.
This kind of information is probably not heavily represented in the training data because most companies don’t publish their strategy. It’s also highly unique to each team.
My bet — Lead engineers / architects become more valuable to a team but their roles will change. They’ll still have to set the technical strategy but they’ll spend more time reviewing the work of more junior team members to make sure it fits the strategy.
It’s unclear to me if this means team’s will need more senior people and become barbell shaped (lots of seniors, few mids, lots of juniors) or stay pyramid shaped.
Quality is going to take a hit and testing post-development is going to be even more important than it already is.
Why? None of the code ChatGPT gives you went through the normal learning cycle of engineering:
None of it is test-driven where the rules for how the code should (and shouldn’t) work are defined ahead of time. Engineers may not fully understand the inner workings of their code. In my example above, I have zero idea what the “projectionParameters” really mean. If they work, I probably won’t go figure it out either.
For internal projects, particularly small one off projects to serve a specific business function, this probably doesn’t matter at all. The ability to solve these projects just got much faster and cheaper.
For external projects and anywhere security, data integrity, and reliability are key, the need for QA just expanded. This code needs to be treated as untrustworthy until it’s heavily exercised by people trying to break it.
My bet — The ratio of Engineering to QA moves meaningfully to QA in the next 18 months; maybe 25–30% more QA. This might mean hiring more specialists, or it might mean your existing team shifts their priorities.
Next week I’ll write more on how LLMs will empower QA professionals and anyone testing code. In particular they will make automated testing cheaper and feaster.
After that, I’ll write about how a business leader should think about balancing speed and risk. I’ll also talk about how you can start future proofing your team to use tools like Copilot and ChatGPT.
]]>Indy.Code() was a great event, and I’m glad I finally got to visit. I wish I had been able to hang out a bit more, but I did get to meet a few new speakers and attendees. Interactions were great in both of my sessions, and I saw a few people I recognized who came to both. They must have gotten lost ;)
Click the images below to download the slides.
First, I got to brush off of my software estimation talk. My thinking here contnuously evolves, so I feel like it’s better and more useful now than when I first wrote it years ago.
Second, I gave my talk on the future (and past) of web standards. I got good feedback from attendees on this. One told me that he or she has been a web developer for years, and still walked away with some new technologies to research. Mission accomplished!
Thanks again to Indy.Code() for inviting me, and for everyone who came to my talks!!
]]>This was another first for me as a speaker as well. I’ve been considering running a panel on becoming a conference speaker for a few years, and CodeStock is the first conference that let it happen! Big thank you to the speaker committee for selecting the panel.
Special thanks to Chad Green, Kara Luton, and David Neal for doing all the real work!
I also gave my Build for the Future When Your Users Are in the Past talk. I had great conversations both durin and afterwards, and am grateful to everyone who attended.
If you’d like to get my slides for this, click here or the image below.
]]>Given at:
Given at:
Given at:
Most interesting web demos require a server of some sort, and I’ve gotten pretty good at writing bad Express applications quickly. But that’s a lot of work that’s tangential to the demo I’m trying to build. I cut corners and definitely don’t build enterprise RESTful APIs for these things, but I’d rather not build any API at all or worry about hosting.
While I was at the Microsoft Edge Summit last week, I had a lightbulb moment. I had spent much of the Summit thinking about PWAs and almost no time thinking about anything on servers. I needed to build the mileage PWA app, and while doing so it was finally time to skip the server.
The rest of this blog post deals with building a RESTful API using Microsoft Flow. I’ve got a post or two in the works about PWAs, but this isn’t one of them.
Side note: Edge Summit was awesome and you should try to attend or stream it next year! This year’s videos are on Channel 9.
Another side note: I love the Edge Summit logo. Whoever created it did a great job.
A third side note: Thank you to the Edge team for inviting me to the Summit and continuing to be an awesome team to work with as a Microsoft MVP.
At HMB we’ve been using Microsoft Flow quite a bit for business automation. It’s sort of like Microsoft’s version of IFTT, and it has great integrations with all sorts of other MS products as you’d expect. I feel less weird giving Flow access to my One Drive than other apps since they already have it, and it just works which is nice.
Flow supports a bunch of useful things that can be pretty advanced, but it also supports basic HTTP requests and responses. Those sound suspiciously like RESTful endpoints.
Flow allows you to create a Request endpoint based on a JSON Schema, which is pretty powerful. However, JSON Schema can be a little weird and complicated so they created a shortcut. You can simply copy/paste in a JSON object, and it will back into a relevant schema for you. This is an awesome user experience and one worth stealing copying if you build this sort of tool.
{ "reason": "Test", "dateTime": "2017-09-17T19:49:45.326Z", "startLocation": { "latitude": 40.1262, "longitude": -82.9291 }, "stopLocation": { "latitude": 39.9612, "longitude": -82.9988 } }
It ended up generating a nice JSON Schema for me, and we were ready to go.
Because I’m building a mileage app, the next thing I wanted to accomplish was to lookup the driving directions between the two locations to find the mileage. This was as easy as adding another step to my Flow and choosing a “Get route” action from Bing.
There was a little confusion becaues it listed “latitude” and “longitude” twice without giving any context about their location in the JSON, but I assumed (correctly) that the first choice in each list was the first one in the document. So this just worked.
I specified that I wanted driving directions in miles, and set each waypoint to be equal to “latitude, longitude”. That’s really it.
Finally, I needed to dump this data somewhere. I decided to create a spreadsheet on One Drive to use since Flow integrates easily there.
This took a little bit of work. You need to attach Flow to your One Drive account by logging into Office 365. Also, you need to creat an Excel spreadsheet with (this is important) a named table with appropriate column headers.
Flow read the POST and created parameters for each column. I then added the relevant data types from the Request object and the Bing response to the correct fields. It took a minute or two for the new spreadsheet on One Drive to show up in Flow but then we were up and running.
That’s it, and it just works. I’ve used Postman to send multiple requests over, and within a few seconds I have a new entry in my spreadsheet with the mileage.
This isn’t a production ready system. There’s no security and I wouldn’t try to scale it to 10 million users. Flow has a limited number of requests before you have to pay although the costs are reasonable after that. All that said though, for building demos, Flow is every bit as awesome as it is for building business automation.
Now I should probably get back to PWAs!
]]>Slides for the PWA talk are available here.
If you want to open the demos directly, you can find them on Azure here.
I don’t have any photos from this talk, so here is a picture of a corgi.
]]>If you’d like to download the slides, they are available right here.
It’s refreshing when we get a near-total replacement for something that is actually simpler. XMLHttpRequest (XHR) is the definition of an old, confusing, complex tool. Fetch API is the simpler replacement.
Fetch is a brand new way of making asynchronous calls to the server. Before we dive into the details, let’s look at an example of Fetch versus XHR. To make this simple, we’ll just request a file (jQuery in this case) from the Google CDN and just dump it to the console.
// Loading the jQuery code fetch('https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js') .then(function (response) { response.text().then(function (responseText) { console.log(responseText); alert('Done using Fetch!'); }); });
Now, let’s take a look at this using XHR.
// Loading the jQuery code request = new XMLHttpRequest(); request.open("GET", "https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js", true); request.send(null); request.onreadystatechange = function() { if(request.readyState === 4) { // What does this even mean? if(request.status === 200) { console.log(request.responseText); alert("Done using XHR!"); } } }
This is a pain to write and read. I had to Google what readyState 4 even meant (request finished and response ready) to write this post. Why do I have to send a null?
In reality, a lot of devs just include jQuery and write this:
$("#basicjQueryButton").on("click", function() { // Loading the jQuery code $.get("https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js", function(data) { console.log(data); alert("Done using jQuery!"); }); });
We download 34.5KB (as of 3.1.0), sometimes just for this functionality. You don’t need to do this anymore.
The Fetch API gives us a generic Request and Response that we can use to GET/POST/PUT/DELETE/HEAD data to and from the server. It uses promises which makes it easier to code, as well as much easier to read. Because it is promise based, it even chains nicely into other methods.
It’s useable today in Chrome, Edge and Firefox, and other browsers through a polyfill.
Before diving into the advanced features of Fetch, let’s take a brief look at promises. If you want to learn more, I recommend this really excellent post from Jake Archibald.
Promises are a new way of dealing with asynchronous code in JavaScript that are simpler than chains of callbacks. Promises make it easy to write code that reads like human language. For example, you could write something like this:
LoadMyData() .then(ProcessMyData) .then(DoSomethingElse) .catch(HandleError);
There is a lot more to promises, but for this article you should know the following:
The example shown above defaulted to a simple GET and didn’t set any advanced headers. However, Fetch allows you advanced control of your Request object before making a request, and lets you query the Response object after. For example, say your application wants to do the following:
var headers = new Headers(); // Set the content type to JSON headers.append("Content-Type", "application/json"); // Setup GET and disable cache var init = { method: "GET", cache: "no-store", mode: "cors", headers: headers } fetch("http://reqres.in/api/users", init) .then(function (response) { // Check that the response is a 200 if(response.status === 200){ alert("Content type: " + response.headers.get("Content-Type")); } });
You may have seen a call like this above:
response.text().then(function (responseText) { console.log(responseText); alert("Done using Fetch!");
What I didn’t tell you before was that .text() is actually a stream reader. The TL;DR of this is that the response data is a stream of bytes that you can read from. Stream readers pull in the entire stream and parse it for you.
There are a few methods available to you including:
If you need to parse the stream twice it’s a problem, because once you’ve read it, it’s empty. Fortunately there’s one more method:
Alternatively, you can parse the string manually and do what you want. You can find a good example here (with source), but note that this doesn’t work in all browsers yet.
Fetch can do nearly everything that XHR can do. The two things that are missing:
That’s really it. I told you this was a big improvement!
Fetch is a powerful tool you can use today (with or without the polyfill), and it’s better than XHR for almost any application. You probably weren’t writing XHR yourself, but if you were, life just got better. It’s more likely you are using a library like jQuery to simplify your AJAX calls which won’t be the case anymore. If you want to take a deeper dive into Fetch, the Mozilla Developer Network is the place to start.
Fetch really comes into its own when we start looking into Service Workers, but that’s a topic for another post.
Happy coding!
]]>