About Copilot pull request summaries
Copilot pull request summaries is an AI-powered feature that allows you to create a summary of the changes that were made in a pull request, which files they impact, and what a reviewer should focus on when they conduct their review.
When a user requests a summary, Copilot scans through the pull request and provides an overview of the changes made in prose, as well as a bulleted list of changes with the files that they impact.
Copilot pull request summaries uses a simple-prompt flow leveraging the Copilot API, with no additional trained models. This utilizes the generic large language model.
Response generation
The current process uses GPT 3.5 to initiate the auto-complete process and generate the pull request summary.
Pipeline approach
When a user requests a summary, a workflow is triggered. The workflow uses the code diffs to build a prompt call, which requests Copilot to generate a summary of the pull request. The summary request initiates a pipeline process which includes raw diffs in a prompt and requests Copilot to generate individual summaries of the various diff hunks. This process creates several individual summaries. Copilot then takes those diff hunk summaries and uses them to create a file-level summary. This file-level summary exists for each file that's summarizable, but excludes binary files and files Copilot deems not appropriate for summarization.
Copilot then takes the file-level summaries and uses them to request another prompt which then creates an overall summary for the pull request. Every summary request will generate a minimum of N+1 prompts, where N equals the number of summarizable files in the pull request.
Output formatting
A summary generated by Copilot will be in two parts:
- A three sentence overview, written in prose, to give a user an overview of what the changes in the pull request entail
- 3–5 of those changes listed in bulleted form that link out to the respective lines of code that they refer to
You can initiate this feature when creating a pull request, by editing the pull request description after creation, or in a comment in the pull request thread. This can take a couple of minutes on larger pull requests. You can share feedback directly from the UI.
Use case for pull request summaries
The goal of Copilot pull request summaries is to help optimize an author's ability to quickly provide context when they request a human review that requires sharing context of the changes that were made. It may help increase developer productivity by reducing the time taken to open a pull request.
For many users, it could provide more helpful context for the changes that were made within a pull request than would normally be available.
Improving performance of pull request summaries
Use Copilot pull request summaries as a tool, not a replacement
The feature is intended to supplement rather than replace a human's work to add context, and we encourage you to continue adding useful context and let Copilot do the busy work of parsing the code and linking to specific files. It remains your responsibility to review and assess the accuracy of information in a pull request that you create.
Provide feedback
This feature is currently in beta. If you encounter any issues or limitations with Copilot pull request summaries, we recommend that you provide feedback through the link that appears the UI after a summary is generated. You can provide feedback through the text link which takes you to our survey.
Limitations of pull request summaries
Currently, our team is aware that there are limitations to this feature. Many of them are expected in leveraging our Copilot API; however, there are a few that are specific to Copilot pull request summaries which pertain to limited scope, longer processing times, and inaccurate responses. We also note that users should expect terms used in their PR to appear in the AI-generated summary. This feature has been subject to RAI Red Teaming and we will continue to monitor the efficacy and safety of the feature over time. For more information, see Microsoft AI Red Team building future of safer AI on the Microsoft security blog.
Limited scope
Because of capacity, we know that larger pull requests that reference 30 or more files will require more time to be processed thoroughly. We don't have an exact threshold currently, but have observed the first 30 files being accounted for and then any additional files being omitted from the summarization. We are working to address this current scope limitation.
Processing time
In general, we expect a summary to be returned in 40 seconds or less after a user initiates the action. However, we have heard that this can take up to a minute, and in some cases a couple of minutes. We are working to decrease processing time and we know that users may not want to wait for this to finish before moving on to other parts of the pull request.
Inaccurate responses
The more inputs and context that Copilot can learn from, the better the outputs will become. However, since the feature is quite new, it will take time to reach exact precision with the summaries that are generated. In the meantime, there may be cases where a user's generated summary is less accurate and requires the user to make modifications before saving and publishing their pull request with this description. Reviewing is a requirement, and careful review of the output is highly recommended by our team during the beta.
Replication of pull request content
Because a summary is an outline of the changes that were made in a pull request, if harmful or offensive terms are within the content of the pull request, there is potential for the summary to also include those terms.
Further reading
- GitHub Copilot Trust Center
- "Creating a pull request summary with GitHub Copilot" in the GitHub Enterprise Cloud documentation.