Iffa Jayyana
In the modern enterprise landscape, the sprawling nature of digital infrastructure often leads to a common but perilous oversight: “repository rot.” As organizations scale, the number of repositories grows exponentially, bringing with it a chaotic web of outside collaborators, outdated permissions, and stale, dormant projects. For security teams and repository owners, this creates a significant attack surface and a governance headache. At GitHub, the internal solution to this persistent problem was the development of Evergreen, an automated system designed to bring order to the chaos without sacrificing developer productivity.
The Challenge of Distributed Collaboration
Working with outside collaborators—contractors, partners, and customers—is a cornerstone of open and collaborative development. However, it imposes a heavy administrative burden on repository owners. Maintaining the principle of least privilege requires owners to regularly audit who has access, why they have it, and whether that access is still necessary.
As projects evolve and business engagements conclude, access is frequently forgotten. Manual housekeeping is often the first task deferred by busy engineers, leading to a accumulation of "zombie" permissions. In a large organization, tracking these collaborators, managing expiration deadlines, and executing follow-ups manually is unsustainable. GitHub recognized that to keep their own organization secure and efficient, they needed to move away from manual interventions toward a system that treats organization hygiene as a first-class engineering problem.
Meet Evergreen: A Design Philosophy
Evergreen is a GitHub App designed to automate organizational health through two primary workflows: managing access for outside collaborators and archiving stale repositories. The system was built upon two rigid design pillars that define its efficacy:
- Fail Secure: The system assumes that access should be revoked unless explicitly renewed. If a collaborator or owner fails to respond to an audit, the default action is to remove access. This ensures that security isn’t left to chance or human forgetfulness.
- Never Destroy: While the system is aggressive about security, it is conservative about data. Evergreen never deletes repositories or their contents. Everything is archived and recoverable, and every access change is meticulously documented in a tracking issue, ensuring that any accidental revocation can be reversed via a simple audit trail.
The Chronology of Automation: A Quarterly Rhythm
Under the hood, Evergreen is a lightweight Go webhook server operating within a containerized environment. Its true brilliance, however, lies in its “choreography” of native GitHub features. It functions like a heartbeat for the organization, using a dedicated controller repository to track time and maintain state.
The system operates on a quarterly cadence, firing repository_dispatch events via GitHub Actions. The process is a carefully orchestrated sequence:
- Day 1 (Stale Repo Review): The system flags repositories that have lacked human activity for a specified period, automatically opening a warning issue for owners.
- Day 8 (Access Review): The system generates a snapshot of all outside collaborators and creates a checklist issue for every repository, prompting owners to confirm who still requires access.
- Day 16 (Stale Repo Cleanup): Repositories that remain inactive after the grace period are moved to an archived state.
- Day 23 (Access Cleanup): Any access review that remains unanswered is processed; the system automatically revokes access for those who failed to respond.
This two-week lead time between notification and action provides a sufficient buffer, ensuring that the system acts as a helpful nudge rather than an abrupt guillotine.
Supporting Data and State Management
One of the most innovative aspects of Evergreen is how it handles state. Instead of relying on an external, complex database, Evergreen uses the Git repository itself. Each repository’s status is tracked as a JSON file stored in the controller repo (state/repos/<repo_id>.json).
This file acts as a single source of truth, recording the last time a repo was seen, the status of access reviews, the names of collaborators, and the identity of the person who made the decision to renew or revoke access. Because this state is stored in Git, the commit history serves as an immutable audit trail. Security and compliance teams can use git log to investigate exactly when and why access was granted or removed, eliminating the need for separate, proprietary audit pipelines.
Issue-Ops: The Human Interface
Evergreen communicates through "Issue-Ops." When an access review begins, the system creates a checklist issue within the target repository. The workflow is intentionally simple: owners review the list, check the boxes for those who require continued access, and close the issue.
The system listens for the issues.closed webhook. Upon closure, it reads the checkboxes, retains the necessary permissions, revokes the rest, and posts a detailed comment providing a receipt of the actions taken. This interaction model keeps the "source of truth" within the context of the work itself, rather than forcing developers to navigate to a separate, third-party dashboard.
To prevent unauthorized access changes, the system is guarded by authentication logic. If a user without the proper authorization attempts to close a review issue, the application automatically reopens the issue and attaches a comment explaining the security restriction.
Informed Notifications and Ownership
A common failure point in automated systems is notifying the wrong people. Evergreen resolves this through a robust fallback chain. It first attempts to identify owners via repository metadata and team assignments. If that fails, it looks for code owners or, in the final resort, team-wide notifications. This ensures that the people responsible for the project are always the ones pinged, removing the need for maintainers to keep a manual, error-prone spreadsheet of ownership.
Fail-Closed Mechanisms
In security-critical automation, "failing open" is the worst-case scenario. Evergreen is engineered to "fail closed." If an API call to revoke a user’s access fails due to a transient error, the system does not mark the task as complete. Instead, it leaves the review pending, the issue open, and the system retries the operation in subsequent runs. A review is only marked "done" when every requested removal has been verified as successful. This granular tracking prevents the "quiet inconsistency" that often plagues large-scale automated cleanup scripts.
The Human Experience: Building Trust
Automated systems often suffer from a reputation for being disruptive. To ensure Evergreen was perceived as a helpful assistant, the development team focused on the user experience:
- Predictability: Notifications are timed to avoid surprises.
- Transparency: Every action taken by the bot is clearly logged and explained in the issue comments.
- Easy Reversals: If a mistake occurs, the audit trail makes it trivial to restore access or un-archive a repository.
Lessons in Large-Scale Deployment
Rolling out such a powerful tool across a massive organization requires extreme caution. The team employed several key strategies:
- Dry-Run Mode: The system was run in a
DRY_RUNmode for weeks, logging every intended action to a file rather than executing it. This allowed the team to verify the logic against real-world data without impacting any users. - Over-Communication: Before the first automated action, the team broadcasted the initiative multiple times. They found that in a large organization, repeated communication is necessary to ensure the message actually lands.
- Observability: By piping
logfmtlogs into Splunk, the team could monitor for edge cases—such as repositories with disabled issues or unusual permission configurations—and adjust the code before those edge cases caused widespread failures. - Respecting Rate Limits: Managing thousands of repositories involves hitting GitHub’s secondary rate limits. Evergreen was designed to pace itself, processing repositories in batches and introducing intentional delays to stay well within the platform’s API constraints.
Implications for Future Governance
The success of Evergreen demonstrates that the most effective way to manage a complex organization is to utilize the platform’s native capabilities rather than fighting them with external tools. By treating "hygiene" as code, organizations can move from reactive, sporadic cleanup sprints to a sustainable, automated state of cleanliness.
As organizations continue to grapple with the security implications of global collaboration, the "Evergreen" model offers a blueprint for the future. It proves that with the right orchestration of webhooks, actions, and issue management, companies can maintain a secure, compliant, and well-organized environment that empowers developers to focus on innovation rather than administrative maintenance. Routine hygiene is no longer a chore—it is a competitive advantage.
Dwi Wanna
Laily UPN
Ammar Sabilarrohman
Asep Darmawan
Asro
Layla Zulfa
Nila Kartika Wati
Muslim