GitHub CoPilot: The Missed Chance?
Is CoPilot really an "AI pair programmer"?
When ChatGPT gained popularity back in 2022, I was genuinely surprised by its capabilities. I hadn’t been closely following the development of generative AI. The last thing I saw was DaVinci, which was also very impressive. However, ChatGPT’s ability to write correct code was both exciting and frightening.
The euphoria has since subsided. Many people accepted that AI will change the future, whether they like it or not. Thus, I began to accept that generative AI could be a significant asset in my daily job. I used ChatGPT for private projects, and it helped me save some time.
Recently, I got the chance to try out GitHub CoPilot. However, I am currently disappointed with CoPilot, and here I explain why.
CoPilot’s promise
Let’s first examine what GitHub promises CoPilot to be. GitHub introduces CoPilot as “Your AI pair programmer,” which certainly sounds appealing. The features, as listed on their website, include:
Plugs right into your editor
Turns natural language prompts into code
Offers multi-line function suggestions
Speeds up test generation
Filters out common vulnerable coding patterns
Blocks suggestions matching public code
GitHub CoPilot generates code directly in your IDE. You write a comment or the initial lines of a function, and CoPilot attempts to complete it. Its functionality is reminiscent of the autocomplete feature commonly found in IDEs. I’d like to emphasize that all these features are indeed present. However, the pressing question remains: Does CoPilot offer better support for my daily tasks compared to ChatGPT and others? Is CoPilot the one tool which makes you a more productive software engineer?
What Works and What Does Not
It’s evident that GitHub Copilot can generate functional, and often accurate code from a comment or a function name. Its performance is on par to ChatGPT. Additionally, as promised by GitHub, it integrates directly into your IDE, provided you’re using a supported one. Indeed, CoPilot assists me in my daily tasks. No doubt about it. Yet, there’s a significant amount of untapped potential.
The Blind Spot: Overlooking Code Structure Context
One significant oversight with GitHub Copilot, in my opinion, is its apparent lack of in-depth code structure analysis. Given that it operates within an IDE and access to the whole codebase it’s positioned uniquely to have a comprehensive understanding of the project’s architecture. This could be a game-changer in terms of the suggestions it provides.
By not leveraging the full context, CoPilot sometimes suggests methods or fields that, while plausible in a generic sense, don’t exist in the current codebase. This can lead to confusion, especially for less experienced developers who might assume the suggestion is correct without double-checking. This is challenging in dynamically typed languages where the problem might only arise at runtime.
Moreover, this lack of context-awareness can hinder the coding process. Instead of seamlessly continuing with the AI’s suggestion, a developer might have to stop, verify the suggestion, and possibly correct it. This interrupts the flow of coding and diminishes the value proposition of an “AI pair programmer.”
An IDE inherently recognizes the structure of the code it’s working with. It knows the classes, their methods, the variables in scope, the libraries imported, and much more. This understanding is what powers features like refactoring, go-to-definition, and error detection.
If CoPilot tapped into this wealth of contextual information, its suggestions would not only be syntactically correct but also contextually relevant. For instance, when suggesting a method call, it will only use methods that are actually present in the class or its ancestors, rather than just providing generic or popular solutions.
Other Issues Worth Noting
GitHub Copilot in IntelliJ has its quirks. At times, its responsiveness lags, causing delays. Occasionally, code generation doesn’t trigger, raising reliability concerns. The activity indicator, subtly placed at the screen’s bottom, is easy to miss.
Moreover, the default settings of the plugin show the suggested code in gray, similar to IntelliJ autocomplete. I would change the color in the settings. These issues, while not major, do impact the overall user experience and need addressing.
Conclusion
Both GitHub Copilot and ChatGPT are powerful AI-driven tools for developers. While Copilot’s IDE integration is a useful feature, its code generation capabilities are comparable to ChatGPT. If one is already using ChatGPT, transitioning to Copilot might not offer significant advantages in terms of code quality. Ultimately, the choice between these tools hinges on individual preferences and needs.
The opinions expressed in this blog are solely my own and do not reflect the views of any organization or individual.