Evaluating ChatGPT for Mule 4
Hello World! This is the second post in a series of posts regarding ChatGPT and MuleSoft. If you have read the first post, thank you! If not, check it out after reading this post.
You clicked on this link to know if you can use ChatGPT to build Mule 4 flows. This post will answer that question based on experiments run during my appearance on the MuleSoft Community Twitch Stream on Feb 21, 2023 using ChatGPT Plus’ default model.
Disclaimer: These are my opinions and not the opinions of my current employer Salesforce.
Note: Interested in the ChatGPT MuleSoft and Mule 4 conversation? You can access it here. If you use a screen reader, I do not place the actual response in text form on this post. You will need to access the link to get the ChatGPT response.
Now the following conversational tests were run:
Test 1: What does ChatGPT know about async scope?
Prompt 1: What Mule 4 scope will allow me to process flows asynchronously?
Observations
Asynchronously was misspelled in the question. However, ChatGPT understood that I misspelled the word.
ChatGPT is intelligent enough to deduce from the question that the scope's name should have something to do with “asynchronous.” Therefore I assume it took an educated guess and came up with the term “asynchronous” scope.
ChatGPT answers this question confidently and goes as far as to provide a code snippet.
ChatGPT understands when to use the scope but does not get the name correct.
The code snippet is incorrect.
The tag “<asynchronous>” should be “<async>”
The use case is valid.
The chatbot thinks the code is PHP.
Prompt 2:Take the flow you created and add maxCurrency of 5 to it.
Observations
Instead of mentioning the “<asynchronous>” tag was wrong, ChatGPT modifies the code snippet to use the <async> tag.
After testing this flow in Anypoint Studio, I believe this flow is correct.
Notice. ChatGPT fixed the code snippet in the first prompt while adding the maxCurrency property.
Rating
Putting on the role of a computer science professor teaching an integration class. I would provide ChatGPT a C+. The chatbot was able to get the async scope theoretical concepts correct and explain how it worked; however, in applying those concepts, the chatbot struggled a bit.
Test 2: What does ChatGPT know about scatter-gather?
Prompt 1: Which Mule 4 component can I use to process three routes in parallel?
Observations
The response has a confident tone and appears to be correct, but it is not.
Each element in the collection (array) [1, 2, 3] will spin off a thread and sequentially process each route.
Execution should spin up three threads and execute in parallel.
The code is clean.
Prompt 2: Is there another way? (1st time)
Observations
The response gets better, but its still not the correct answer.
The code will spin up three threads sequentially.
Each thread will execute a route (sub-flow) in a fire-and-forget fashion.
Prompt 3: Is there another way? (2nd time)
Observations
The explanation and answer are correct.
Rating
The Chatbot came up with some clever responses that were ultimately wrong. I would expect a MuleSoft Developer without any experience but who has taken the Anypoint Platform Development: Fundamentals course to get this answer correct. Scatter-gather is the best way because the prompt mentions finite numbers (3) of routes. Scatter-gather is perfect for that case.
Other Considerations
Creating good prompts
While testing ChatGPT and other prompt-based AI tools, I have noticed a common theme, “the more descriptive the prompt is, the better the response.” To be fair, the inventors of these tools have taken time to write brief examples to help you before you start.
The ChatGPTs response is only as good as the prompt itself.
Adding details can increase the probability of ChatGPT getting the answer correct and limit the number of assumptions that the chatbot has to use.
Should you use ChatGPT to help build Mule 4 flows?
Right now, NO! At least not yet. However, I suspect future models will be pre-trained to write good Mule 4 flows.
If you use ChatGPT now to build Mule 4 flows, you will encounter the following issues, which may cause more frustration than help.
ChatGPTs default model makes mistakes on simple MuleSoft questions that an experienced developer should know after taking the Anypoint Platform Development: Fundamentals course.
The chatbot's confident tone makes inexperienced developers think they have the correct answer, but that is far from the truth.
ChatGPT in the future
If Open AI trains ChatGPT to build Mule 4 flows in the future, I think this tool will be good as a copilot for developers. This would be a fantastic tool for generating and inserting code templates. I know most want AI to write the entire code for you and remove the need for developers, but I double down on my belief that developers will be needed in the future.
By the way, I’ve watched a few “Explore Anypoint Code Builder” series on Twitch, and a few MuleSoft Community members mentioned wanting ChatGPT integrated with the tool. It's a great idea. I wonder Salesforce partner with Github to take advantage of the AI pair programming tool Github copilot, which is already an extension to VSCode.
Wouldn’t that be cool? What do you think?